Lagged Load Variables in Load Forecasting

This post was triggered by the email below:

I am a regular reader of your blog and website which is an inspiration to me as a forecasting analyst. I just have a very simple question for you, which I don’t understand as a practitioner. I have looked at 10-20 papers and almost every one has a lag variable in it for forecasting electricity demand. But in practice, if you are forecasting for a portfolio or a region and not the whole grid of a country, lag demand is simply not available until weeks or months later. Is this because academia is focused on the theoretical and not the practical, or is it because it focuses on the big picture, total demand and not by region/portfolio? And is there any way round this? You can always feed forecasts for D+1 as a lag into D+2 going forward, but this doesn’t give you a lag for D+0 and D+1.

This is an excellent and frequently asked question, but I don't have a simple answer. 

In practice, if you have lagged load as a variable in the model but don't have its observation for the forecasting period, you have to use the predicted value. 

Take day-ahead load forecasting for example, when forecasting hour ending 10am for tomorrow, we don't have the observation for hour ending 9am. If the model included the lagged load of the preceding hour, we have to predict the load of hour ending 9am first.  In order to make that prediction, we need the load of hour ending 8am, which has to be predicted as well. 

Let's say you are building is a multiple linear regression model, the regression models with lagged dependent variables are called dynamic regression models. 

To implement a dynamic regression model to forecast the period where the observations for the lags are not available, you will have to execute an iterative process to forecast those lags first. 

Now you may want to ask:

Are these dynamic regression models more accurate than the ones without lagged load?

Practically, it depends upon how far ahead you are forecasting and how far back the lagged variables go to. 

If you are using the load of the preceding hour in your model, you should expect some improvement for the next few hours comparing with the models without lagged load variables. The improvement diminishes as the forecast horizon stretches. Beyond 10 hours or so, you may not see any improvement. 

One way to get around this iterative process is to avoid using the load of preceding one or two hours. Instead, we can use the load of the same hour of yesterday. By doing so, you can expect some improvement for the next day or two comparing with the models without lagged load variables. Again, the improvement diminishes as the forecast horizon stretches. For the very short horizon, i.e., one or two hours ahead, the models with the load of the same hour of yesterday typically do not outperform the models with the load of the preceding hour. 

For long term load forecasting, adding lagged load variables doesn't help much but creates issues. 

One is on the interpretability of the model. Because the lagged load variables are highly correlated with the load series itself, most of the load variation is being "explained" by lagged load variables rather than the other explanatory variables such as weather and calendar variables. In other words, we can hardly answer "what if the next year is a hot year" if lagged load variables were in the model. 

Another issue is on the inflation of forecast accuracy. Many people are plugging in actual values of the lagged load when analyzing the long term load forecasting performance, which would result in a very low error. Be careful, this is not ex post forecasting! You should not assume the perfect knowledge of the dependent variable in ex post forecasting. 

To keep the answer short, this is what I have been doing: I use lagged load (see this MPCE paper) when the forecast horizon is less than two or three days, Sometimes I include residual forecasting (see the point forecasting portion of this IJF paper). I don't use lagged load for long term load forecasting. 

Hope this helps!

Temperature-based Models vs. Time Series Models

Last week, Spyros Makridakis asked me a question:

I have been reading your energy competition and I cannot find any clear statements about the superiority of explanatory/exogenous variables. Am I wrong? Is there a place where you state the difference in forecasting accuracy between time series and explanatory multivariate forecasting as it relates to the short as well as beyond the fist two or three days (not to mention the long term) that accurate temperature forecasting exist?

Today, Rob Hyndman asked me a similar question, which was routed originally from Spyros.

In fact, this has been quite a debatable topic in load forecasting. The answer is not straightforward. This subject could make a good master's thesis or even a doctoral dissertation. I was going to write a paper about it, but always had something more important or urgent to work on. Recently my research team has done some preliminary work along this direction. While the paper is still under preparation, let me start the discussion with this blog post, as part of the blog series on error analysis in load forecasting.

The literature is not vacant in this area. Various empirical studies have suggested different things.

Some earlier attempts were made by James Taylor. James has written many load forecasting papers. His best known work is on exponential smoothing models.

James' TPWRS2012 paper claimed that

Although weather-based modeling is common, univariate models can be useful when the lead time of interest is less than one day.

In Fig. 9 of the paper that depicted the MAPE values by lead time, the paper stated that

The exponential smoothing methods outperform the weather-based method up to about 5 hours ahead, but beyond this the weather-based method was better. 

Based on this paper, can we conclude that exponential smoothing models are more accurate than the weather-based methods for very short term ex ante load forecasting?

No.

This is my interpretation of the paper:

A world-class expert in exponential smoothing carefully developed several exponential smoothing models. These models generated more accurate forecasts than a U.K. power company's forecasts. 

The "weather-based method" used in that paper was devised by the transmission company in Great Britain using regression models. The paper briefly mentioned how the "weather-based method" worked, but the information was not enough for me to judge how accurate these weather-based models are. I don't know if this U.K. transmission company is using state-of-the-art models.

Some evidence came from recent load forecasting competitions, such as Global Energy Forecasting Competitions, npower forecasting challenges, and BigDEAL Forecasting Competition 2018.

In short, time series models, such as exponential smoothing and ARIMA models, never showed up as a major component of a winning entry in these competitions. On the other hand, regression models with temperature variables are always among the winning models.

In fact, ARIMA showed up in a winning method in GEFCom2014, where my former student Jingrui Xie used four techniques (UCM, ESM, ANN, and ARIMA) to model the residuals of a regression model (see our IJF paper).

Based on these competition results, can we conclude that time series models are not as accurate as regression models?

No.

In GEFCom2012, we let the contestants predict a few missing periods in the history without restricting the contestants to using only the data prior to each missing period. In my GEFCom2012 paper, I briefly mentioned that

This setup may mean that regression or some other data mining techniques have an advantage over some time series forecasting techniques such as ARIMA, which may be part of the reason why we did not receive any reports using the Box–Jenkins approach in the hierarchical load forecasting track.

In GEFCom2012, npower forecasting challenges, and the qualifying match of BFCom2018, actual temperature values were provided for the forecast period. In other words, these competitions were on ex post forecasting. Again, the temperature-based models have an advantage since perfect information of temperature is given for the forecast period.

GEFCom2014 and GEFCom2017 were on ex ante probabilistic forecasting. The temperature-based models dominated the leaderboards. This would be a fair evidence favoring temperature-based models.

For benchmarking purpose, I included two seasonal naive models in my recency effect paper per the request of an anonymous reviewer. Both performed very poorly compared with the other temperature-based models. I commented in the paper:

Seasonal naïve models are used commonly for benchmarking purposes in other industries, such as the retail and manufacturing industries. In load forecasting, the two applications in which seasonal naïve models are most useful are: (1) benchmarking the forecast accuracy for very unpredictable loads, such as household level loads; and (2) comparisons with univariate models. In most other applications, however, the seasonal naïve models and other similar naïve models are not very meaningful, due to the lack of accuracy. 

Here is a quick summary based on the evidence so far:

• For ex post point load forecasting, evidence favors temperature-based models.
• For ex ante point load forecasting, no solid evidence favoring either method. 
• For ex ante probabilistic load forecasting, evidence favors temperature-based models.

I'm not a fan of comparing techniques. In my opinion, it's very difficult to make fair comparisons among techniques. If I were good at ANN but bad at regression, I could build super accurate ANN models than regression models. Using exactly the same technique, two forecasters may build different models with distinct accuracy levels. My fuzzy regression paper offers such an example. In other words, the goodness of a model is largely depending upon the competency of the forecaster. The best way to compare techniques is through forecasting competitions. 

In practice, weather variables is must-have in most load forecasting situations. I'll elaborate this in another blog post. 

Factors Affecting Load Forecast Accuracy

In some of my papers, I tried to present fairly comprehensive case studies that cover various load zones. I often use a primary case study to illustrate the flow or components of a proposed methodology. After that I apply the same methodology to a secondary case study to show that the same methodology works well on other zones. A by product of this publication process is a series of benchmark results on various of load zones. You may have realized that the same methodology or model typically results in different forecast errors on different load zones.

That Magic Ruler May Not Work This Year

It is December now. Those of us who live in the North Hemisphere have officially stepped into the winter season. So far this winter in North Carolina seems to be warmer than the last couple of years. Although I can't predict how severe or warm this winter will be, I think it is a good time to write this blog post about the magic ruler.

Crystal Ball Lessons in Predictive Analytics

For a long time, I have had the idea of writing an article about "how much benefit are we getting from reducing load forecast errors". A few months ago I got a request from EnergyBiz to contribute an article. I thought this "valuation" topic would be a good fit. So here is Crystal Ball Lessons in Predictive Analytics.

Error Analysis in Load Forecasting: The Second Growing List

Over the past few years, THE most frequently asked question I have received is like the following:

My forecast error is x percent, how is the accuracy compared with my peers'?

Of course, each question was packaged with details about the jurisdictions and maybe model setups. At first I was trying to provide a customized response every time. Three years ago, I gave a webinar to cover very basic error measures used in load forecasting, which was apparently not enough to cover all the load forecast error/accuracy questions. Last week, after getting another email inquiry about forecasting accuracy, I decided to start this new series of blog posts. It will be the same style as the growing list of load forecasting terminology. The first 8 topics are listed below:

1. How accurate is my forecast? (recorded webinar)
2. Factors affecting load forecast accuracy
3. Temperature-based models vs. time series models
4. More about weather variables
5. How to measure load forecast errors?
6. Accuracy of ISO/RTO short-term forecasts
7. What a mess! Problems with reporting long term forecast errors
8. Models for benchmarking
9. Which techniques are "more accurate"?
10. How much are we saving from 1% error reduction? 
11. That magic ruler may not work this year

I will add the links as I finish the posts. If you have any feedback, just email me or leave your thoughts in the comment field below. I will update the list accordingly.