In load forecasting, especially long term load forecasting, documentation is probably the most important task. The ultimate test of documentation quality is whether the forecasting system has been described in detail so that other people with relevant education background and experience can reproduce the forecasts.
While forecasting is like an adventure, exploring an unknown trail, documentation is like walking the same trail again and again to record what happened in detail. Documentation often requires significant amount of efforts, sometimes more than forecasting itself.
Rather than having the forecaster explain the forecast in person every time, documentation helps simplify the communication.
2. Further improvement
As the forecasting process gets more and more complicated, it's impossible for a forecaster to memorize everything s/he has done in the past. Documentation serves as a reminder when the forecaster is trying to make improvement based on a formally developed model.
3. Business continuity
If the forecaster left the job, someone else with similar background can take over the work.
4. Auditing and defense
Industry regulation requires utilities to defend their forecasts as part of the rate case. The defense is mainly based on documentation.
Responsibilities of everybody getting involved in the entire load forecasting process, such as the ones who supply the raw data, pull the data from database, develop the models, convert the forecasts to the presentable format, review the forecast, suggest revision and approve the forecast.
2. Data
All the data sources involved in the forecasting process, such as load data, weather data, economy data, demographic data, outage data, etc. If the data needs modification, all major versions of the data should be stored with notes indicating why and how the data is being revised.
3. Forecasting methodology
Methodologies for weather station selection, outlier detection and data cleansing, model selection, hierarchical forecasting, forecast combination, etc.
4. Models and forecasts
The techniques (such as multiple linear regression, autoregression integrated moving average, etc.) being used, the resulting models and forecasts. If the models are being updated or revised in the forecasting process, all major revisions and versions of the models should be documented.
5. Error analysis
How accurate the forecasts are, including comprehensive measures on key decision periods, such as MAPE (Mean Absolute Percentage Error) of monthly peak demand, RMSE (Root Mean Square Error) by hour of a day, etc.
6. Judgmental changes
If the forecasts are being revised based on personal experience and/or business sense, details should be provided in documentation, such as who made the revision, how the forecasts are being changed, why and what the new forecasts look like.
7. Strength and limitation
Pros and cons of the model(s), forecast(s) and the forecasting process. For instance, is the model doing well during regular days but behave poorly during holidays?
8. Directions for future improvement
How the forecasting process can be improved in the next forecasting cycle.
The Global Energy Forecasting Competition 2014 is going on right now, with less than three weeks towards the end. The very last task after the marathon-like 15-week competition is documentation. The contestants are required to submit their code and a report illustrating how they did in the competition. Although we are not providing any formal template for the report, this blog post will server as a general guideline for the GEFCom2014 contestants in all tracks. In the final report, we would expect at least a summary of your scores and rankings (based on Inside Leaderboard), methodologies you've been using, a log of the 12 evaluation weeks showing evolution of your model(s) and key references that spark your brilliant ideas.
While forecasting is like an adventure, exploring an unknown trail, documentation is like walking the same trail again and again to record what happened in detail. Documentation often requires significant amount of efforts, sometimes more than forecasting itself.
Why documentation?1. Communication
Rather than having the forecaster explain the forecast in person every time, documentation helps simplify the communication.
2. Further improvement
As the forecasting process gets more and more complicated, it's impossible for a forecaster to memorize everything s/he has done in the past. Documentation serves as a reminder when the forecaster is trying to make improvement based on a formally developed model.
3. Business continuity
If the forecaster left the job, someone else with similar background can take over the work.
4. Auditing and defense
Industry regulation requires utilities to defend their forecasts as part of the rate case. The defense is mainly based on documentation.
What to document?1. Who
Responsibilities of everybody getting involved in the entire load forecasting process, such as the ones who supply the raw data, pull the data from database, develop the models, convert the forecasts to the presentable format, review the forecast, suggest revision and approve the forecast.
2. Data
All the data sources involved in the forecasting process, such as load data, weather data, economy data, demographic data, outage data, etc. If the data needs modification, all major versions of the data should be stored with notes indicating why and how the data is being revised.
3. Forecasting methodology
Methodologies for weather station selection, outlier detection and data cleansing, model selection, hierarchical forecasting, forecast combination, etc.
4. Models and forecasts
The techniques (such as multiple linear regression, autoregression integrated moving average, etc.) being used, the resulting models and forecasts. If the models are being updated or revised in the forecasting process, all major revisions and versions of the models should be documented.
5. Error analysis
How accurate the forecasts are, including comprehensive measures on key decision periods, such as MAPE (Mean Absolute Percentage Error) of monthly peak demand, RMSE (Root Mean Square Error) by hour of a day, etc.
6. Judgmental changes
If the forecasts are being revised based on personal experience and/or business sense, details should be provided in documentation, such as who made the revision, how the forecasts are being changed, why and what the new forecasts look like.
7. Strength and limitation
Pros and cons of the model(s), forecast(s) and the forecasting process. For instance, is the model doing well during regular days but behave poorly during holidays?
8. Directions for future improvement
How the forecasting process can be improved in the next forecasting cycle.
The Global Energy Forecasting Competition 2014 is going on right now, with less than three weeks towards the end. The very last task after the marathon-like 15-week competition is documentation. The contestants are required to submit their code and a report illustrating how they did in the competition. Although we are not providing any formal template for the report, this blog post will server as a general guideline for the GEFCom2014 contestants in all tracks. In the final report, we would expect at least a summary of your scores and rankings (based on Inside Leaderboard), methodologies you've been using, a log of the 12 evaluation weeks showing evolution of your model(s) and key references that spark your brilliant ideas.
No comments:
Post a Comment
Note that you may link to your LinkedIn profile if you choose Name/URL option.