Energy Forecasting in the Big Data World

All papers for the International Journal of Forecasting special section on energy forecasting in the big data world have been published online. Out of 14 papers collected for this special section, eight are from GEFCom2017 documenting winning methods, while the other six non-GEFCom2017 papers cover diverse topics in the areas of energy supply, demand and price forecasting.

The guest editorial is HERE. Below is the list of special section papers:

1. Tao Hong, Jingrui Xie, and Jonathan Black. Global Energy Forecasting Competition 2017: Hierarchical probabilistic load forecasting. 
2. Florian Ziel. Quantile regression for the qualifying match of GEFCom2017 probabilistic load forecasting.
3. I. Dimoulkas, P. Mazidi, and L. Herre. Neural networks for GEFCom2017 probabilistic load forecasting.
4. Slawek Smyl and N. Grace Hua. Machine learning methods for GEFCom2017 probabilistic load forecasting.
5. Andrew J. Landgraf. An ensemble approach to GEFCom2017 probabilistic load forecasting.
6. Cameron Roach. Reconciled boosted models for GEFCom2017 hierarchical probabilistic load forecasting.
7. Julian de Hoog and Khalid Abdulla. Data visualization and forecast combination for probabilistic load forecasting in GEFCom2017 final match.
8. Isao Kanda and J.M. Quintana Veguillas. Data preprocessing and quantile regression for probabilistic load forecasting in the GEFCom2017 final match.
9. Stephen Haben, Georgios Giasemidis, Florian Ziel, and Siddharth Arora. Short term load forecasting and the effect of temperature at the low voltage level.
10. Jakob W. Messner and Pierre Pinson. Online adaptive lasso estimation in vector autoregressive models for high dimensional wind power forecasting. 
11. Dazhi Yang, Elynn Wu, Jan Kleissl. Operational solar forecasting for the real-time market.
12. Grzegorz Marcjasz, Bartosz Uniejewski, and Rafał Weron. On the importance of the long-term seasonal component in day-ahead electricity price forecasting with NARX neural networks.
13. Bartosz Uniejewski, Grzegorz Marcjasz, and Rafał Weron. Understanding intraday electricity markets: Variable selection and very short-term price forecasting using LASSO.
14. Xuerong Li, Wei Shang, and Shouyang Wang. Text-based crude oil price forecasting: A deep learning approach.

Citation 

Tao Hong and Pierre Pinson, "Energy forecasting in the big data world," International Journal of Forecasting, vol. 35, no. 4, pp. 1387-1388, 2019. 

Energy Forecasting in the Big Data World

Tao Hong and Pierre Pinson

Modern information and communication technologies have brought big data to virtually every segment of the energy and utility industries. While forecasting is an important and necessary step in the data-driven decision-making process, the problem of generating better forecasts in the world of big data is an emerging issue and a challenge to both industry and academia. This special section aims to collect top-quality forecasting articles that document cutting-edge research findings and best practices on a wide range of important business problems in the energy industry. Our emphasis is on big data, such as forecasting with high resolution data, the use of high-dimensional processes, forecasting in real-time, and the use of non-traditional data and variables. 

Guest Editorial: Big Data Analytics for Grid Modernization

IEEE Transactions on Smart Grid just published our special section on Big Data Analytics for Grid Modernization. The guest editorial is on IEEE Xplore with open access. The original Call for Papers is HERE.

While "big data" is quickly becoming a buzz word (see THIS POST), in this guest editorial we discussed our interpretation from four aspects:

1. The data involved in the analysis is big in at least one of its three defining dimensions: volume, variety or velocity. The big data used in the utility industry includes but is not limited to smart meter data, phasor measurement unit data, weather data, and social media data. 
2. The problem under investigation is to prepare for analyzing the big data, such as data compression and data security issues. 
3. The methodology requires customized modeling of individual components of a system, or leads to in-depth understanding of the individual components. For instance, estimating the invisible solar generation belongs to this category. 
4. The technology can be used to help reach the answer faster, or answer the questions otherwise difficult to answer. For example, a distributed platform can be used to speed up the analytic tasks.

Thanks to the diligent work from our guest editors, reviewers and the authors, we are able to present a high-quality collection of papers to the community. Below is the list of 17 special section papers:

1. D. Zhou, J. Guo, Y. Zhang, J. Chai, H. Liu, Y. Liu, C. Huang, X. Gui and Y. Liu, "Distributed Data Analytics Platform for Wide-Area Synchrophasor Measurement Systems," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2397-2405, Sept. 2016
2. P. H. Gadde, M. Biswal, S. Brahma and H. Cao, "Efficient Compression of PMU Data in WAMS," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2406-2413, Sept. 2016
3. X. Tong, C. Kang and Q. Xia, "Smart Metering Load Data Compression Based on Load Feature Identification," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2414-2422, Sept. 2016
4. J. Hu and A. V. Vasilakos, "Energy Big Data Analytics and Security: Challenges and Opportunities," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2423-2436, Sept. 2016
5. Y. Wang, Q. Chen, C. Kang and Q. Xia, "Clustering of Electricity Consumption Behavior Dynamics Toward Big Data Applications," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2437-2447, Sept. 2016
6. S. Ben Taieb, R. Huser, R. J. Hyndman and M. G. Genton, "Forecasting Uncertainty in Electricity Smart Meter Data by Boosting Additive Quantile Regression," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2448-2455, Sept. 2016
7. H. Shaker, H. Zareipour and D. Wood, "Estimating Power Generation of Invisible Solar Sites Using Publicly Available Data," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2456-2465, Sept. 2016
8. H. Shaker, H. Zareipour and D. Wood, "A Data-Driven Approach for Estimating the Power Generation of Invisible Solar Sites," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2466-2476, Sept. 2016
9. X. Zhang and S. Grijalva, "A Data-Driven Approach for Detection and Estimation of Residential PV Installations," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2477-2485, Sept. 2016
10. H. Wang and J. Huang, "Cooperative Planning of Renewable Generations for Interconnected Microgrids," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2486-2496, Sept. 2016
11. J. Peppanen, M. J. Reno, R. J. Broderick and S. Grijalva, "Distribution System Model Calibration With Big Data From AMI and PV Inverters," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2497-2506, Sept. 2016
12. Y. C. Chen, J. Wang, A. D. Domínguez-García and P. W. Sauer, "Measurement-Based Estimation of the Power Flow Jacobian Matrix," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2507-2515, Sept. 2016
13. H. Sun, Z. Wang, J. Wang, Z. Huang, N. Carrington and J. Liao, "Data-Driven Power Outage Detection by Social Sensors," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2516-2524, Sept. 2016
14. H. Jiang, X. Dai, D. W. Gao, J. J. Zhang, Y. Zhang and E. Muljadi, "Spatial-Temporal Synchrophasor Data Characterization and Analytics in Smart Grid Fault Detection, Identification, and Impact Causal Analysis," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2525-2536, Sept. 2016
15. M. Rafferty, X. Liu, D. M. Laverty and S. McLoone, "Real-Time Multiple Event Detection and Classification Using Moving Window PCA," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2537-2548, Sept. 2016
16. T. Jiang, Y. Mu, H. Jia, N. Lu, H. Yuan, J. Yan and W. Li, "A Novel Dominant Mode Estimation Method for Analyzing Inter-Area Oscillation in China Southern Power Grid," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2549-2560, Sept. 2016
17. B. Wang, B. Fang, Y. Wang, H. Liu and Y. Liu, "Power System Transient Stability Assessment Based on Big Data and the Core Vector Machine," IEEE Transactions on Smart Grid, vol. 7, no. 5, pp. 2561-2570, Sept. 2016

Citation

Tao Hong, Chen Chen, Jianwei Huang, Ning Lu, Le Xie and Hamidreza Zareipour, "Guest Editorial: big data analytics for grid modernization", IEEE Transactions on Smart Grid, vol.7, no.5, pp 2395-2396, September, 2016

Guest Editorial: Big Data Analytics for Grid Modernization

Tao Hong, Chen Chen, Jianwei Huang, Ning Lu, Le Xie and Hamidreza Zareipour

Probabilistic Energy Forecasting: Global Energy Forecasting Competition 2014 and Beyond

It is really hard to write an introduction for this paper, because there are too many things to highlight the outcome of hundreds of hours invested by the organizers of GEFCom2014 and thousands of hours spent by the GEFCom2014 contestants. In one sentence,

This is a MUST READ paper in energy forecasting. 

In this paper, we

• Summarized 7 papers collected through the special issue CFP process, including one on anomaly detection for gas demand data, two on load forecasting, two on price forecasting, one on wave energy forecasting, and one on wind speed forecasting;
• Introduced the GEFCom2014 together with the in-class probabilistic load forecasting competition I organized in 2015;
• Commented on the methodologies used by the winning teams of GEFCom2014;
• Published the 120MB data used in the four tracks of GEFCom2014 and the in-class competition;
• Made 12 predictions for the next decade of energy forecasting

We sincerely hope that you enjoy reading this paper and can help contribute to the energy forecasting community.

Citation

Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli and Rob J. Hyndman, "Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond", International Journal of Forecasting, in press. working paper available from http://www.drhongtao.com/articles.

Probabilistic Energy Forecasting: Global Energy Forecasting Competition 2014 and Beyond

Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli, and Rob J Hyndman

Abstract

The energy industry has been going through a significant modernization process over the last decade. Its infrastructure is being upgraded rapidly. The supply, demand and prices are becoming more volatile and less predictable than ever before. Even its business model is being challenged fundamentally. In this competitive and dynamic environment, many decision-making processes rely on probabilistic forecasts to quantify the uncertain future. Although most of the papers in the energy forecasting literature focus on point or single-valued forecasts, the research interest in probabilistic energy forecasting research has taken off rapidly in recent years. In this paper, we summarize the recent research progress on probabilistic energy forecasting. A major portion of the paper is devoted to introducing the Global Energy Forecasting Competition 2014 (GEFCom2014), a probabilistic energy forecasting competition with four tracks on load, price, wind and solar forecasting, which attracted 581 participants from 61 countries. We conclude the paper with 12 predictions for the next decade of energy forecasting. 

Guest Editorial: Special Section on Analytics for Energy Forecasting with Applications to Smart Grid

I'm glad to announce that the special section we edited last year has been published by IEEE Transactions on Smart Grid. I had a great pleasure working with and learning from my guest editors during this editorial process. The guest editorial is available on IEEE Xplore with open access.

Citation
Tao Hong, Shu Fan, Wei-Jen Lee, Wenyuan Li, Anil Pahwa, Pierre Pinson, Jianhui Wang and Hamidreza Zareipour, "Guest Editorial: special section on analytics for energy forecasting with applications to smart grid", IEEE Transactions on Smart Grid, vol.5, no.1, pp. 399-401, Jan 2014

Guest Editorial: Special Section on Analytics for Energy Forecasting with Applications to Smart Grid 

Tao Hong, Shu Fan, Wei-Jen Lee, Wenyuan Li, Anil Pahwa, Pierre Pinson, Jianhui Wang and Hamidreza Zareipour

Thanks to the help from the reviewers and editors, totally 15 high quality forecasting papers are included in this special section, which can be categorized into 5 groups:

1. Global Energy Forecasting Competition;
2. Load forecasting and analysis with high granular data;
3. Probabilistic energy forecasting;
4. Forecasting and analysis of emerging subjects;
5. Novel methods for wind power forecasting.

The links to the papers with the IEEE citation format are provided below with indicators of the groups they fall into:

• Y. Goude, R. Nedellec, and N. Kong, "Local short and middle term electricity load forecasting with semi-parametric additive models," IEEE Trans. Smart Grid, vol. 5, pp. 440-446, Jan. 2014. (1)
• D. Lee, and R. Baldick, "Short-Term Wind Power Ensemble Prediction Based on Gaussian Processes and Neural Networks," IEEE Trans. Smart Grid, vol. 5, pp. 501-510, Jan. 2014. (1, 5)
• M. Chaouch, "Clustering-based improvement of nonparametric functional time series forecasting: application to intra-day household-level load curves," IEEE Trans. Smart Grid, vol. 5, pp. 411-419, Jan. 2014. (2)
• J. Kwac, J. Flora, and R. Rajagopal, "Household energy consumption segmentation using hourly data," IEEE Trans. Smart Grid, vol. 5, pp. 420-430, Jan. 2014. (2)
• P. G. D. Silva, D. Ilić, and S. Karnouskos, "The impact of smart grid prosumer grouping on forecasting accuracy and its benefits for local electricity market trading," IEEE Trans. Smart Grid, vol. 5, pp. 402-410, 2014. (2)
• J. D. Black, and W. L. W. Henson, "Hierarchical load hindcasting using reanalysis weather," IEEE Trans. Smart Grid, vol. 5, pp. 447-455, Jan. 2014. (2)
• T. Hong, J. Wilson, and J. Xie, "Long term probabilistic load forecasting and normalization with hourly information," IEEE Trans. Smart Grid, vol. 5, pp. 456-462, Jan. 2014. (2, 3)
• J. Tastu, P. Pinson, P. Trombe, and H. Madsen, "Probabilistic forecasts of wind power generation accounting for geographically dispersed information," IEEE Trans. Smart Grid, vol. 5, pp. 480-489, Jan. 2014. (3, 5)
• D. Lee, and R. Baldick, "Future wind power scenario synthesis through power spectral density analysis," IEEE Trans. Smart Grid, vol. 5, pp. 490-500, Jan. 2014. (3, 5)
• C. Wan, Z. Xu, Y. Wang, Z. Y. Dong, and S. K. P. Wong, "A hybrid approach for probabilistic forecasting of electricity price," IEEE Trans. Smart Grid, vol. 5, pp. 463-470, Jan. 2014. (3)
• Z. Wang, M. Begovic, and J. Wang, "Analysis of conservation voltage reduction effects based on multistage SVR and stochastic process," IEEE Trans. Smart Grid, vol. 5, pp. 431-439, Jan. 2014. (2, 4)
• P. Wang, H. Zareipour, and W. D. Rosehart, "Descriptive models for reserve and regulation prices in competitive electricity markets," IEEE Trans. Smart Grid, vol. 5, pp. 471-479, Jan. 2014. (4)
• Z. Duan, B. Gutierrez, and L. Wang, "Forecasting plug-In electric vehicle sales and the diurnal recharging load curve," IEEE Trans. Smart Grid, vol. 5, pp. 527-535, Jan. 2014. (4)
• J. Shi, Z. Ding, W. Lee, Y. Yang, Y. Liu, and M. Zhang, "Hybrid forecasting model for very-short term wind power forecasting based on grey relational analysis and wind speed distribution features," IEEE Trans. Smart Grid, vol. 5, pp. 521-526, Jan. 2014. (5)
• L. Xie, Y. Gu, X. Zhu, and M. G. Genton, "Short-term spatio-temporal wind power forecast in robust look-ahead power system dispatch," IEEE Trans. Smart Grid, vol. 5, pp. 511-520, Jan. 2014. (5)

Enjoy reading!

Global Energy Forecasting Competition 2012

The paper is available on Science Direct.

Citation
Tao Hong, Pierre Pinson and Shu Fan, "Global Energy Forecasting Competition 2012", International Journal of Forecasting, vol.30, no.2., pp 357-363, April - June, 2014

Global Energy Forecasting Competition 2012

Tao Hong, Pierre Pinson and Shu Fan

Abstract

The Global Energy Forecasting Competition (GEFCom2012) attracted hundreds of participants worldwide, who contributed many novel ideas to the energy forecasting field. This paper introduces both tracks of GEFCom2012, hierarchical load forecasting and wind power forecasting, with details on the aspects of the problem, the data, and a summary of the methods used by selected top entries. We also discuss the lessons learned from this competition from the organizers’ perspective. The complete data set, including the solution data, is published along with this paper, in an effort to establish a benchmark data pool for the community.