Which Model is the Best?

Recently Rob Hyndman blogged about the history of forecasting competitions. I have read the post three times already. I learned something new each time. I wish every reader of this blog can also read that post and learn something from the history. Nevertheless, I would like to highlight a paragraph here:

[...] This reveals a view commonly held (even today) that there is some single model that describes the data generating process, and that the job of a forecaster is to find it. This seems patently absurd to me — real data comes from much more complicated, non-linear, non-stationary processes than any model we might dream up — and George Box famously dismissed it saying “All models are wrong but some are useful”.

I have been in the field load forecasting for a little more than 10 years. During the past decade, I received many "the best" questions: Which model is the best? Which technique is the best? Which software is the best? Which variable is the best? ...

There are some variants to these questions: Do you use neural networks? Do you use population in long term load forecasting? Do you use normal weather? Do you think demand response can reduce peak demand? Do you think a MAPE value of 20% is too high? ...

While the folks who asked these questions may expect a crisp answer of the best model, technique, software, variable, or a YES/NO answer, I had to disappoint them with "depends", sometimes followed by a lengthy elaboration.

It depends on the data, the business needs, the production environment, and many other factors... 

I don't have a single model to sell to my clients as "the best model". I recommend proper methodologies to the clients after making a comprehensive evaluation of their situations.

I think the root of these "the best" questions is that commonly held view:

There is some single model that describes the data generating process. 

I disagree with this view, and don't know where the view is originally coming from. I wish someone can write another history article to explain the source.

In load forecasting, there is no universally best model. We need empirical studies, many empirical studies, to show some evidence that one method is superior in some aspect. That's why I have been promoting reproducible research and benchmarking data pool and models.

In computational complexity and optimization, there is actually a theorem, no free lunch theorem. 

That Magic Ruler May Not Work This Year

It is December now. Those of us who live in the North Hemisphere have officially stepped into the winter season. So far this winter in North Carolina seems to be warmer than the last couple of years. Although I can't predict how severe or warm this winter will be, I think it is a good time to write this blog post about the magic ruler.

Documentation in Load Forecasting: 4 Reasons and 8 Elements

In load forecasting, especially long term load forecasting, documentation is probably the most important task. The ultimate test of documentation quality is whether the forecasting system has been described in detail so that other people with relevant education background and experience can reproduce the forecasts.

While forecasting is like an adventure, exploring an unknown trail, documentation is like walking the same trail again and again to record what happened in detail. Documentation often requires significant amount of efforts, sometimes more than forecasting itself.

Load Factor, Coincidence Factor, Diversity Factor and Responsibility Factor

Load factor is average load of a system divided by its peak load. The higher the load factor is, the smoother the load profile is, and the more the infrastructure is being utilized. The highest possible load factor is 1, which indicates a flat load profile.

In the old days, load factor is often used for long term peak demand forecasting. The forecasters first develop a energy forecast. They then calculate the average hourly load. Finally by dividing the forecasted average load by a predefined load factor, they can obtain the forecasted peak. However, I would avoid using this method for long term load forecasting in today's world where high resolution data is available for load forecasting.

Standard Time, Daylight Saving Time and Local Time

The earth is round like a ball. When it's night in the US, it's morning in China. To do business beyond a local region, people need a common reference to communicate time. Standard time is the synchronization of clocks in different geographical locations within a time zone to a common time standard, usually based on the meridian at the center of the time zone.

Prediction Interval and Confidence Interval

This is a pair of terms very difficult to distinguish, because statisticians and economists don't follow the same standard. Since load forecasting falls under the umbrella of forecasting, I'm following the terminology developed by the forecasting community. Special thanks to Rob Hyndman, who answered many questions from me during my preparation of this post. I highly recommend you his two blog posts The difference between prediction intervals and confidence intervals and Prediction intervals too narrow.

In short, there is a simple rule that tells where to use confidence or prediction interval:

A confidence interval is associated with a parameter, while a prediction interval is associated with a prediction. 

Below I'm using three examples to illustrate how to apply these two terms in load forecasting.

Weather, Climate and Temperature

Weather is the condition of the atmosphere, such as temperature, humidity and rainfall, at a particular place over a short period of time, i.e., a few days. For instance, a weather forecast usually goes a few days ahead. Climate refers to the weather pattern of a place over a long period, i.e., a few decades or more. A well-known term is "climate change".

In load forecasting, the most frequently used weather variable is temperature. A temperature station is often called weather station, though a weather station may measure many variables beyond temperature.

Many utilities also include other weather variables as predictors in short term load forecasting models, such as humidity, wind speed and cloud cover. In reality, because it is difficult to forecast these predictors with good accuracy, there is a trade-off between the information gained by adding these additional variables and the noise introduced by their forecast errors. I always go with the principle of parsimony. Unless rigorous tests have been conducted showing the benefits of adding additional variables, I would try to keep the model as lean as possible.

Back to Load Forecasting Terminology. 

Quantile, Quartile and Percentile

Suppose we have a set of data sorted in ascending order,  by dividing the data into q equal-sized pieces, we can get q-quantiles. The quantiles are the values marking the boundaries between two adjacent subsets.

Reliability (for Planning) and Reliability (for Forecasting)

My first job interview in the US was with Richard Brown. At that time, I knew virtually nothing about the electric power industry. To prepare for the interview, I checked Richard's profile, and found out that he is a Fellow of IEEE for his contribution in power systems reliability. I also browsed through his book Electric Power Distribution Reliability, a must-read book in power systems reliability. Although none of the things I prepared was actually used in the interview, I got to know about reliability before I even heard of load forecasting.

Weather Normalization and Load Normalization

In the electric power industry, there are two variables often associated with "weather normalization": reliability and load. If you are interested in reliability, you may refer to the activities of IEEE Working Group on Distribution Reliability. In this post, I'm going to focus on load.

Planning - the business driver of weather normalization

Training, Validation and Test

When developing models for forecasting or data mining (I will write a post about these two terms), we usually slice the data into three pieces, training, validation and test:

• Training data is used to estimate the parameters. 
• Validation data is used to select models. 
• Test data is used to confirm the model performance. 

Here let me use two representative techniques, regression analysis and Artificial Neural Networks (ANN) to illustrate how the process works.

Linear Models and Linear Relationship

Update 3/20/2015: sharing a blog post from Freakonometrics On Some Alternatives to Regression Models.
In many papers, we can find the statements similar to the one below:

Because linear models can hardly capture the nonlinear relationship between load and temperature, we use Artificial Neural Networks (or other black-box models) in this paper. 

The major conceptual error of the above statement is due to a common misunderstanding that linear models cannot capture nonlinear relationship.

Model, Variable, Function and Parameter

My first job was an engineer at an expert-based consulting firm. One of my first tasks was to develop models for a large Investor Owned Utility. I was quite exited when being assigned to this project - I thought it was a good opportunity to show off my modeling skills. During the project kick-off meeting, I realized that I misunderstood the scope of work. The "models" I was asked to develop are circuit models. "Modeling" was simply to draw the lines, fuses, switches and transformers on a distribution engineering software platform based on their physical specifications, which did not involve any math or statistics at all.

The predictive models in energy forecasting are different from the physical models mentioned above. A regression-based load forecasting model, for example, describes the relationship between load and the factors that drive the load. There are three components in such a model:

Very Short, Short, Medium and Long Term Load Forecasting

Load forecasting is so fundamental that it is being used across all sectors in the electric power industry for various business applications. Because of the wide spread of its applications, there are many ways to classify the various load forecasts:

• based on forecast horizon: very short, short, medium and long term load forecasts;
• based on resolution of the data or updating frequency (these two concepts are different!): hourly, daily, monthly, seasonal and annual load forecasts;
• based on business needs: operational, planning and retail load forecasts;

Forecasting, Forecast and Forecaster

The story starts with my graduate school days. After my PhD defense, Dr. David Dickey, who was one of the members on my doctoral committee, handed me a printout of my draft dissertation. Based on the edits, I can tell that he read it in detail and revised it carefully. One of the many things he suggested was to change "short term load forecaster" to "short term load forecasting system" to represent the solution, or to "short term load forecasting model" to represent the mathematical formula that captures the relationship between load and other explanatory variables.

It was probably the only time I didn't follow his advice. My reason was that some papers in the load forecasting literature used the term "load forecaster" to represent a load forecasting system. For instance, ANNSTLF, a system developed 15 years ago by EPRI, is the acronym of "Artificial Neural Network Short Term Load Forecaster".

Forecasting and Planning

Forecasting is to tell what the future would look like given one or more scenarios. Planning is to develop actions in order to reach a desirable stage in the future. Although the forecasting and planning functions often reside in one department of a utility, conceptually mixing the two often results in bad or even fraudulent forecasts and plans. Let me use a set of examples to illustrate the relationship between the two.

Background

The planning department of the Big City Power & Light (a distribution company) need to make a 5 year plan for system maintenance and upgrade. The forecasting team developed a 10-year ahead forecast showing an increasing trend of peak demand at about 2% per year. Majority of the load growth was due to the new tax incentives for attracting high tech companies to a technology park near downtown area. The technology park was expected to add 10,000 high tech jobs over the next 5 years. These jobs would help with the local economy growth.

Forecasting and Backcasting

Forecasting is the process of exploring the future events that have not been observed or determined. Backcasting "typically" refers to the process of exploring the past events given the information known to date. The two terms seem to be mutually exclusive, but are often used in an ambiguous way.

Load, Demand, Energy and Power

Let me start with the easy pair, power and energy. Power is the amount of energy consumed over time, while energy is the integral of power over time. In the electric power industry, the most frequently used units of power are kW (kilowatt) and MW (megawatt): 1MW = 1000kW = 1,000,000W. Consequently, the common units of energy are kWh (kilowatt hour) and MWh (megawatt hour). 

There are three types of power in an AC (alternating current) power system: active power or real power (in W, watt), reactive power (in Var, volt-ampere reactive) and complex power (in VA, volt-ampere). The magnitude of complex power is called apparent power (in VA). See Wikipedia for more details about AC power. 

Load Forecasting Terminology: a Growing List

Using the precise terminology is a necessary step towards effective communication. As mentioned in my Foresight article "Energy Forecasting: Past, Present and Future", the forecasting practice has a long history in the utility industry, in fact, over 100 years. While load forecasting plays a vital role in many business applications, the usage of different terms is quite messy. Everyone may have his/her own definition of everything in this area. I'm creating a new label "terminology" to host a series of posts on load forecasting terminology. I would like to cover as many ambiguous terms as possible. Here is a tentative list:

1. Load, demand, energy and power
2. Forecasting and backcasting
3. Forecasting and planning
4. Forecasting, forecast and forecaster 
5. Very short, short, medium and long term load forecasting
6. Model, variable, function and parameter
7. Linear models and linear relationship
8. Training, validation and test
9. Weather normalization and load normalization
10. Prediction interval and confidence interval
11. Probability forecasting and probabilistic forecasting
12. Reliability (for planning) and reliability (for forecasting)
13. Quantile, quartile and percentile
14. Resolution (for hierarchical load forecasting) and resolution (for probabilistic forecasting)
15. Forecasting and data mining
16. Statistical models and econometric models 
17. Load factor, coincidence factor, diversity factor and responsibility factor 
18. Calendar month and billing month
19. Energy efficiency, demand response and demand side management
20. Load forecasting and load research
21. Weather, climate and temperature
22. Standard time, daylight saving time and local time
23. Statistical learning and machine learning

If you can think of other confusing terms, please let me know. I will add the new ones after #12. As I'm moving along, I will also add links to the new posts. 

13 Lucky Tips for Energy Forecasting

I recently wrote this article for Intelligent Utility. They used the title "13 lucky tips to juggle the analytics of forecasting". I have been using the INFORMS definition of analytics, which includes descriptive analytics (or summary statistics), predictive analytics (or forecasting) and prescriptive analytics (or optimization). Since forecasting is part of analytics, I'm using "13 lucky tips for energy forecasting" here to be consistent with the definition from INFORMS.

Published in Intelligent Utility Magazine July/Aug 2014

Citation
Tao Hong, "How to Juggle the Analytics of Forecasting: 13 Lucky Tips", Intelligent Utility, pp. 11-13, July/August, 2014

How to Juggle the Analytics of Forecasting: 13 Lucky Tips

Tao Hong

Energy forecasting is one of those areas of great importance to electric grid that gets little attention—even from power industry insiders. But you need to know how to make the best of your forecasting process. Here are 13 tips to get you started.