Descriptive Analytics Based Anomaly Detection for Cybersecure Load Forecasting

Data quality has been a big challenge in load forecasting practice, but an underestimated issue in the academic literature. This work was supported by the U.S. Department of Energy through the Cybersecurity for Energy Delivery Systems Program. We were trying to detect anomalies so that accurate load forecasts can be produced even when the data is contaminated.

Citation

Meng Yue, Tao Hong, and Jianhui Wang, "Descriptive analytics based anomaly detection for cybersecure load forecasting," IEEE Transactions on Smart Grid, vol. 10, no. 6, pp. 5964-5974, November, 2019

Descriptive Analytics Based Anomaly Detection for Cybersecure Load Forecasting

Meng Yue, Tao Hong, and Jianhui Wang

Abstract

As power delivery systems evolve and become increasingly reliant on accurate forecasts, they will be more and more vulnerable to cybersecurity issues. A coordinated data attack by sophisticated adversaries can render existing data corrupt or outlier detection methods ineffective. This would have a very negative impact on operational decisions. The focus of this paper is to develop descriptive analytics-based methods for anomaly detection to protect the load forecasting process against cyberattacks to essential data. We propose an integrated solution (IS) and a hybrid implementation of IS (HIIS) that can detect and mitigate cyberattack induced long sequence anomalies. HIIS is also capable of improving true positive rates and reducing false positive rates significantly comparing with IS. The proposed HIIS can serve as an online cybersecure load forecasting scheme.

Big Data Analytics: Making Smart Grid Smarter

The May 2018 issue of the Power & Energy Magazine is on Big Data Analytics. My guest editorial is on IEEE Xplore with open access. The original articles are in English. The Spanish translation is also available. The links to these articles are listed below.

Citation

Tao Hong, "Big data analytics: making smart grid smarter" IEEE Power and Energy Magazine, vol.16, no.3, pp 12-16, May-June 2018. (IEEE Xplore)

Features in This Issue

Visualizing Big Energy Data
By Rob J. Hyndman, Xueqin (Amy) Liu, and Pierre Pinson

Distribution Synchrophasors
By Hamed Mohsenian-Rad, Emma Stewart, and Ed Cortez

Big Data Analytics for Flexible Energy Sharing
By Furong Li, Ran Li, Zhipeng Zhang, Mark Dale, David Tolley, and Petri Ahokangas

Weather Data for Energy Analytics
By Jonathan Black, Alex Hofmann, Tao Hong, Joseph Roberts, and Pu Wang

Big Data Analytics in China’s Electric Power Industry
By Chongqing Kang, Yi Wang, Yusheng Xue, Gang Mu, and Ruijin Liao

Training Energy Data Scientists
By Tao Hong, David Wenzhong Gao, Tom Laing, Dale Kruchten, and Jorge Calzada


Articulos de Mayo/Junio de 2018

Visualización de "big data" de energía
Por Rob J. Hyndman, Xueqin (Amy) Liu y Pierre Pinson

Sincrofasores en la distribución
Por Hamed Mohsenian-Rad, Emma Stewart y Ed Cortez

Análisis de "big data" para el intercambio flexible de energía
Por Furong Li, Ran Li, Zhipeng Zhang, Mark Dale, David Tolley y Petri Ahokangas

Datos meteorológicos para el análisis de energía
Por Jonathan Black, Alex Hofmann, Tao Hong, Joseph Robert y Pu Wang

Análisis de "big data" en la industria de la potencia eléctrica de china
Por Chongqing Kang, Yi Wang, Yusheng Xue, Gang Mu y Ruijin Liao

Formación de científicos de datos de energía
Por Tao Hong, David Wenzhong Gao, Tom Laing, Dale Kruchten y Jorge Calzada

Weather Data for Energy Analytics

Being an energy forecaster, I am genuinely interested in meteorology. I even recruited a master student who was a practicing meteorologist in Hawaii (see the blog post about Ying Chen). The more energy forecasting projects I conduct, the more I appreciate the value of weather data. In GEFCom2014, the top 1 place of the solar track was a team of meteorologists from Australia, who completely dominated the track. In GEFCom2017, the top 1 place of the final match was a team of meteorologists from Japan. I truly believe that the energy forecasting community can better leverage meteorology than what we do today. Here is an article about two use cases of weather data for energy analytics. In fact we merged two papers into one by removing the sophisticated mathematics and statistics to keep the story readable to a broad audience. The IEEE Power and Energy Society is so kind to offer the open access to this paper, so that people can read it for free.

Citation

Jonathan Black, Alex Hofmann, Tao Hong, Joseph Roberts, and Pu Wang, "Weather data for energy analytics: from modeling outages and reliability indices to simulating distributed photovoltaic fleets," IEEE Power and Energy Magazine, vol.16, no.3, pp 43-53, May-June 2018. (Open Access; IEEE Xplore)

Weather Data for Energy Analytics

From Modeling Outages and Reliability Indices to Simulating Distributed Photovoltaic Fleets

Jonathan Black, Alex Hofmann, Tao Hong, Joseph Roberts, and Pu Wang

Abstract

Weather impacts virtually all facets of our daily life. As a result, many business sectors are affected by weather conditions, and the power industry is no exception. Weather is a major influencer on system reliability and a key driver of both power supply and demand. In this article, we will demonstrate novel uses of weather data for energy analytics via two utility applications. We first use easily accessible weather data together with regression analysis to model distribution outages and construct a probabilistic view of reliability indices that helps reveal a utility’s reliability trend. We then use high-resolution, commercial-grade weather data to develop realistic simulations of anticipated behind-the-meter photovoltaic (PV) fleets

Training Energy Data Scientists

Traditional power engineering curriculum has been heavily focusing on the engineering aspects of power systems, such as power flow, state estimation, stability and control. Data science has never been a focus in the past. I saw that gap 5 years ago, predicted the shortage of data scientists in the power industry, and left a great place to work to come back to academia. Nowadays, when other business sectors are offering 6-figure salaries to fresh graduates, utilities are having a hard time to compete on the analytics talents. Recently I had the opportunity to collaborate with Prof. David Wenzhong Gao from University of Denver and three other utility executives to put our thoughts in a paper.

Citation

Tao Hong, David Wernzhong Gao, Tom Laing, Dale Kruchten, and Jorge Calzada, "Training energy data scientists: universities and industry need to work together to bridge the talent gap," IEEE Power and Energy Magazine, vol.16, no.3, pp 66-73, May-June 2018. (IEEE Xplore)

Training Energy Data Scientists 

Universities and Industry Need to Work Together to Bridge the Talent Gap

Tao Hong, David Gao, Tom Laing, Dale Kruchten, and Jorge Calzada

Abstract

The workforce crisis is nothing new to the U.S. power industry. It has been a growing concern of both governments and industry organizations since the early 2000s. Meanwhile, the growth of data during the past decade has led to a demand surge for data analytics across all business sectors. The shortage of an electricity workforce and the increasing demand for data analytics present an emerging challenge as well as opportunity for university power engineering programs to bridge the data analytics talent gap. After gathering various perspectives from members of academia, industry, and government, we propose an interdisciplinary and entrepreneurial approach to revising the traditional power engineering curriculum for training the next generation of energy data scientists.

Crystal Ball Lessons in Predictive Analytics

For a long time, I have had the idea of writing an article about "how much benefit are we getting from reducing load forecast errors". A few months ago I got a request from EnergyBiz to contribute an article. I thought this "valuation" topic would be a good fit. So here is Crystal Ball Lessons in Predictive Analytics.

Integrated Energy Forecasting: Improving T&D Planning and Operations

Comparing with my Foresight paper Energy Forecasting: Past, Present and Future, which was targeting the forecasting community, this paper was mainly written for the T&D engineers and managers. The web/digital version is available HERE.

Citation
Tao Hong, "Integrated Energy Forecasting: Improving T&D Planning and Operations", Electricity Today, pp. 58-62, January/February, 2015

13 Lucky Tips for Energy Forecasting

I recently wrote this article for Intelligent Utility. They used the title "13 lucky tips to juggle the analytics of forecasting". I have been using the INFORMS definition of analytics, which includes descriptive analytics (or summary statistics), predictive analytics (or forecasting) and prescriptive analytics (or optimization). Since forecasting is part of analytics, I'm using "13 lucky tips for energy forecasting" here to be consistent with the definition from INFORMS.

Published in Intelligent Utility Magazine July/Aug 2014

Citation
Tao Hong, "How to Juggle the Analytics of Forecasting: 13 Lucky Tips", Intelligent Utility, pp. 11-13, July/August, 2014

How to Juggle the Analytics of Forecasting: 13 Lucky Tips

Tao Hong

Energy forecasting is one of those areas of great importance to electric grid that gets little attention—even from power industry insiders. But you need to know how to make the best of your forecasting process. Here are 13 tips to get you started.

Utilities Dust off the Forecasting Playbook

This is a paper I wrote with my co-worker Alyssa Farrell. It was published by the Analytics Magazine. In this paper, we discussed several key issues related to forecasting in the utility industry, such as demand response, integration of electric vehicles and renewable energy, and how to build a modern utility forecasting team.

The web version is available HERE. The digital version is available HERE. The pdf version is available HERE.

Citation
Tao Hong and Alyssa Farrell, "Utilities Dust off the Forecasting Playbook: Smart Grid Data Brings Challenges and Opportunities", Analytics Magazine, pp.50-57, July/August, 2013.