Professional Documents
Culture Documents
Untitled Document
Untitled Document
We are forecasting different type of time data from 15min to year based on
Machin learning models.
Our Machine learning models are using previous actual data to forecast power
data.
Basically the method we are using is, we are taking previous input data from
postgres database saving into csv file inside test folder. We are using scheduler
python library In order to run models. We are using test file as input in ML
models and save output as predict file inside predict folder. Then final we’ll use
this forecasted data to store and display.
For displaying we are using javascript chart.js.
In this documentation, we have discussed 22 tables, and each table has their own
constraints and indexes.
● Auth_group :- this table contains the information of the authorized group.
Columns of auth_group :-
1. Id :- id of authorized group.
2. Name :- name of authorized group.
Constraints:-
1. name should be unique
2. id is used to uniquely identifies each record in table
● Django_migrations:- this table contains all the migration data that take place
in this application.
Columns of django_migrations:-
1. Id :- id assigned to the migration data.
2. App:- the application where migration has been labeled.
3. Name:- the changes have been made in the application.
4. Applied:- date & time of applied migration task.
● Django_sessions:- this table contains the sessions and their expiry date
Columns of django_sessions:-
1. Session_key:- key for particular session.
2. Session_data:- data of the particular session.
3. Expire_date:- date of session expiry.
● Stlf_main_empmast:- this table contains all the employees email and their
role
Columns of stlf_main_empmast:-
1. Id:- id of employee
2. Email:-email id of employee
3. role:- role of employee
● Stlf_main_menu:- this table defines that a user with their assigned role can
access specific tasks.
Columns of stlf_main_menu:-
1. Id:- id assigned to the options accessed by the role user.
2. Role:- role of the user.
3. Link:- link assigned to the particular role
4. Options:- task assigned to the particular role through the given link.
5. Parent:- the task belongs to a particular application.(e.g. Change
password is the part of account application)
● Stlf_main_onehour_missing_blocks:-this table contains the missing blocks
and their date and time.
Columns of stlf_main_onehour_missing_blocks:-
1. Id:- id for missing blocks
2. Timestamp:- time blocks of that missing blocks
● Stlf_main_roles:- this table contains the role for the stlf task.
Columns of stlf_main_roles:-
1. Id:- id for role
2. Roles:- name of the role
SLDC-Schedulers.py
Function:
1) start()
2) backup_files
3) forecast_daily
4) forecast_hourly
5) weather_forecast
6) hourly_weather
7) SCADA_Load
8) SCADA_Load_to_one_hour
9) forecast_daily_v2
10) forecast_daily_6pm_v2
11) forecast_hourly_v2
12) forecast_weekly
13) run_forecast_for_year
14) run_missing_data_check
15) run_backup_db
Function summary
1)start() : This function is starting schedulers at different time intervals. Common schedulers (line 29
-36) , 15 minutes model scheduler(line 43-46) ,weekly model schedulers (line 49) ,monthly model
schedulers (line 52) , missing data in load actual (line 55) , automatic database backup(line 58)
5) weather_forecast() : This function is common for 15 min and 1 hr models . this function is calling
and returning get_weather_forecast()(handlers.py 1232 -1322)
14) run_missing_data_check() : This function is calling and returning (handlers.py line 1939
-1955)
15) run_backup_db() : This function is calling and returning backup_db() (handlers.py line
1961-1979)
All your functions of one schedule, located inside handlers.py and views.py
So look at handles.py first.
SLDC – Handlers.py
This module contains brings together and houses all functions that perform
any computational or database related tasks.
Functions listed:
1. get_hour_ahead_forecast v1
2. get_hour_ahead_forecast v2
3. get_hour_ahead_forecast v5
4. get_daily_forecast v1
5. get_daily_forecast v2
6. get_day_ahead_forecast v1
7. get_day_ahead_forecast v2
8. reformat_day_ahead_forecast
9. reformat_hour_ahead_forecast
10. reformat_daily_forecast
11. reformat_daily_forecast_only_total
12. create_input_csv_model
13. insert_forecast_24Hr
14. insert_forecast_24Hr_v2
15. insert_forecast_3Hr
16. insert_forecast_3Hr_v2
17. remove_outliers
18. remove_outliers_from_database
19. get_SCADA_data_v1
20. insert_onehour_block_actual_data
21. display_weather_forecast
22. get_hourly_weather
23. get_weather_forecast
24. create_input_csv_weekly_model
25. get_and_insert_weekly_forecast
26. create_input_csv_yearly_model
27. get_and_insert_yearly_forecast
28. forecast_for_year
29. get_and_insert_monthly_actual
30. actual_for_month_lgbr
31. input_for_month_lgbr_stag2
32. forecast_for_month_lgbr_stag2
33. input_for_month_lgbr_stag1
34. forecast_for_month_lgbr_stag1
35. input_for_month_lgbr_stag1_april
36. forecast_for_month_lgbr_stag1_april
37. update_missing_data
38. check_loadactual
39. missing_data_check
40. backup_db
Types of functions:
o SCADA Function(s): 19
· get_hour_ahead_forecast:
Obtains hour ahead forecast
get_hour_ahead_forecast_v1()
Line: 116
get_hour_ahead_forecast_v2()
Line: 151
get_hour_ahead_forecast_v5()
Line: 132
· get_daily_forecast
Get_daily_forecast_v1()
Line: 176
get_daily_forecast_v2()
Line: 218
· get_day_ahead_forecast
Obtains forecast for next day using day ahead prediction models.
get_day_ahead_forecast_v1()
Line: 268
get_day_ahead_forecast_v2()
Line: 305
· reformat_day_ahead_forecast()
Line: 343
· reformat_hour_ahead_forecast()
Line: 382
· reformat_daily_forecast()
Line: 423
· reformat_daily_forecast_only_total()
Line: 491
Reformats data for saving into CSV file but only for total load and not
individual Discoms.
· create_input_csv_model()
Line: 531
· insert_forecast_24Hr()
Insert_forecast_24Hr():
Line: 633
**Not in use**
Insert_forecast_24Hr_v2():
Line: 699
· insert_forecast_3Hr()
Insert_forecast_3Hr():
Line: 772
**Not in use**
Insert_forecast_3Hr_v2():
Line: 894
· remove_outliers()
Line: 967
· remove_outliers_from_database()
Line: 894
· get_SCADA_data_v1()
Line: 1062
· insert_onehour_block_actual_data()
Line: 1201
Saves actual usage data (one hour block) to database, if data is missing,
saves that as well
· display_weather_forecast()
Line: 1286
Obtains weather data from forecast object and returns a weather object
**Not in use**
· get_hourly_weather()
Line: 1321
Obtains hourly weather data from Darksky API and stores to database
· get_weather_forecast()
Line: 1415
Obtains 24Hr weather data from Darksky API and stores to database
· create_input_csv_weekly_model()
Line: 1511
· get_and_insert_weekly_forecast()
Line: 1610
· create_input_csv_yearly_model()
Line: 1655
Creates CSV file for yearly model
· get_and_insert_yearly_forecast()
Line: 1766
· forecast_for_year()
Line: 1836
We predict data for the next year i.e. In April 2022 we will make
predictions for the time frame April 2023 – March 2024 using the data of
April 2022-March 2023, and two other data parameters.
The models that calculate LGBR report need to be fed data on a monthly
block basis, this is done by making a list of the months and then feeding
in data, the two other data parameters are dependent on the month in
concern, for month August 2022, we will use data of April 2022-March
2023 + August 2021 + August 2020.
Feeding this data month wise to the models generates LGBR predictions.
In October:
input_for_month_lgbr_stag2() ->
forecast_for_month_lgbr_stag2() ->
input_for_month_lgbr_stag1()->
forecast_for_month_lgbr_stag1()
· get_and_insert_monthly_actual()
Line: 1847
Creates actual_lgbr objects from actual load data and saves them to
database.
· actual_for_month_lgbr()
Line: 1881
· input_for_month_lgbr_stag2()
Line:1890
This function preps and formats the input for the prediction models of
LGBR, but since in October, the data till April is missing, we first make
prediction for the next 6 months i.e., till April. This functions preps and
formats data for feeding as input to those models and saves it as a CSV
file.
· forecast_for_month_lgbr_stag2()
Line: 1949
· input_for_month_lgbr_stag1()
Line: 2010
· forecast_for_month_lgbr_stag1()
Line: 2075
· input_for_month_lgbr_stag1_april()
Line: 2136
Takes all available data i.e., current year + last year + second last year as
required by LGBR models (requirement specified above in LGBR system
explanation), formats it for input as required. Saves it into a CSV file.
· forecast_for_month_lgbr_stag1_april
Line: 2197
· update_missing_data
Line: 2259
Every time some SCADA data is not transferred, that data is missing
from the actual load data, this function is used to upload this data
manually.
· check_loadactual
Line: 2443
Checks for missing data in the actual load data from given start time.
· missing_data_check
Line: 2534
· backup_db():
Line: 2545
SLDC- helpers.py
Functions:
datetime_list_hour
datetime_list_15min
past_datetime_list_hour
next_datetime_list_hour
past_datetime_list_15min
next_datetime_list_15min
clear_file
number_of_days
send_email
blocks - number of hourly blocks => blocks = 3 (at time 00:00) with start_time true will give 00:00,
01:00, 02:00
eg. if now time is 00:00 and we want time for 01:00 and so on then skip_blocks = 1
This function is updating datetime_list hourly covering all the blocks and returning the datetime_list
blocks - number of 15 min blocks => blocks = 3 (at time 00:00) with start_time true will give 00:00,
00:15, 00:30
eg. if now time is 00:00 and we want time for 01:00 and so on then skip_blocks = 4
# range(3) - 0,1,2 | range(1,3) 1,2 SO if start_time is false need to add a value to hours
This function is updating datetime_list every 15 minutes , covering all the blocks and returning the
7)def clear_file(filename,message) :
This function is deleting the content of a file named filename and printing a message. From line 96-99
8)def number_of_days(y,m):
This function is a simple function to calculate which year is the leap year.
9)def send_email(for_database=False):
This function is used to debug mail .
AT last your views function will work for display.
SLDC – Views.py
We have one function to detect outlier from data.
Outlier Case Handling involves detecting and removing outliers from data for better
prediction results.
- remove_outliers(df_load , time_for_outlier) :
The function uses a list of load data of last 287 15 min blocks of data and the
df_load[‘load’] is appended to the list.Moving averages with a window size of
48 and interquartile range is calculated for the above list of data and this is
used to create a list of upper and lower bounds using a mean_threshold
parameter of value 2.5 using the formulas :
upper_bounds[ ] = moving_averages[ ] + mean_threshold*interquartile_range
lower_bounds[ ] = moving_averages[ ] - mean_threshold*interquartile_range
Then,the current load data is checked if it lies within the upper bound and
lower bound and df_load[‘load’] is modified if it does not and the updated
dataframe is returned.