Friday, March 23, 2018

Review of Smart Meter Data Analytics: Applications, Methodologies, and Challenges

About 10 years ago, the term "smart grid" was officially defined in the Energy Independence and Security Act of 2007 (EISA-2007). Soon after that, many power companies started their smart meter deployments. As of 2016, more than 70 million smart meters were installed in the united states. The world installed base was projected to reach 780 million by 2020, pushed by the mass roll-outs in China. We are now sitting on a gold mine of data collected by these smart meters. Last year I gave a forecast:
The energy companies will be moving more Gigabytes of data than GWh of electricity.
The scientific community has been trying to understand the smart meter data and get some actionable insights out of it. Thousands of papers have been published in the recent decade on the various aspects of smart meter data analytics. Last year, I worked with my collaborators in Tsinghua University to complete a review of smart meter data analytics. The paper was just put on the IEEE Xplore yesterday.

This is the longest paper ever published by the IEEE Transactions on Smart Grid. I'm sure reading this 24-page review article can save the readers significant amount of time from digging thousands of papers in the literature. Load forecasters may find some interesting stuff in Section III, which is dedicated to load forecasting in the smart grid era.


Yi Wang, Qixin Chen, Tao Hong, and Chongqing Kang, "Review of smart meter data analytics: applications, methodologies, and challenges," IEEE Transactions on Smart Grid, in press. (working paper; IEEE Xplore)

Review of Smart Meter Data Analytics: Applications, Methodologies, and Challenges

Yi Wang, Qixin Chen, Tao Hong, and Chongqing Kang


The widespread popularity of smart meters enables an immense amount of fine-grained electricity consumption data to be collected. Meanwhile, the deregulation of the power industry, particularly on the delivery side, has continuously been moving forward worldwide. How to employ massive smart meter data to promote and enhance the efficiency and sustainability of the power grid is a pressing issue. To date, substantial works have been conducted on smart meter data analytics. To provide a comprehensive overview of the current research and to identify challenges for future research, this paper conducts an application-oriented review of smart meter data analytics. Following the three stages of analytics, namely, descriptive, predictive and prescriptive analytics, we identify the key application areas as load analysis, load forecasting, and load management. We also review the techniques and methodologies adopted or developed to address each application. In addition, we also discuss some research trends, such as big data issues, novel machine learning technologies, new business models, the transition of energy systems, and data privacy and security.

Saturday, March 3, 2018

A look back at ISEA2017

About a year ago, I posted 7 reasons to attend ISEA2017. Last June, we had our very first International Symposium on Energy Analytics at the beautiful city Cairns, Australia. 50 researchers and practitioners from industry and academia across 16 countries attended the symposium. 36 of them also attended ISF2017.

The two-day agenda included 5 keynote presentations, 13 contributed presentations selected from 31 abstract submissions, and 5 GEFCom2017 finalists presentations. At the end of the first day, June 22, 2017, we hosted the GEFCom2017 award ceremony followed by a two-hour cocktail reception. The program agenda, presentations and pictures can be downloaded via THIS LINK.

We have been processing the full paper submissions since then. So far, three papers are currently in press with the Journal of Modern Power Systems and Clean Energy, one is in press with the Power & Energy Magazine. The remaining papers are going through the peer review system of the International Journal of Forecasting. The accepted ones together with the GEFCom2017 articles are going to be published in an energy forecasting special section edited by Pierre Pinson and myself.  

I want to thank many people who made ISEA2017 a successful event, the keynote speakers, the presenters, the GEFCom2017 winners, and all the attendees of ISEA2017. Special thanks to Rob Hyndman and George Athanasopoulos for the strong local support, our photographer Mitchell O'Hara-Wild for capturing the happy moments, and Pam Stroud for handling the registrations and helping keep things in order. 

Last but not least, big thanks to our sponsors, International Institute of Forecasters, Tangent Works and Journal of Modern Power Systems and Clean Energy, for their generous financial support that helped to keep the registration fees low. 

Many people have asked me about the next ISEA. I know there will be the next one, but I don't know when and where. If you have any idea or want to help with organizing ISEA, let me know! 

Wednesday, February 14, 2018

UNC Charlotte's M.S. Program in Engineering Management with Energy Analytics Concentration

Five years ago, I realized the big gap between academic offerings and industry needs in the energy analytics field. I took a brave step and a big salary cut to join UNC Charlotte, with the mission of "producing the next generation of finest analysts for the industry."

Since then, I have trained hundreds of students and working professionals, through the various courses I taught within and outside UNC Charlotte, and through the research activities at the BigDEAL lab that I founded to incubate the elites. I am pleased to see the growing interest in this emerging field of energy analytics and the enthusiasm from the my audience.

However, the gap did not really shrink despite my effort. In fact, the demand increase for energy analysts, which is now called energy data scientists, is more than the amount of graduates I can produce from my small shop!

To further enrich the teaching materials and serve a greater audience in the industry with diverse academic and professional backgrounds, I designed a 60-hour energy analytics curriculum. Many people who took those courses asked me the following question:
Is there a degree program that I can enroll to further my education in this area (energy analytics/forecasting)?
Previously, I directed them to BigDEAL and showed them the path to PhD. For those who were not ready for a PhD program, I didn't have much to offer, unfortunately.

Today, if you asked me the same question, I have a different and better answer:
We are launching a master program specifically designed for those who want to pursue a data science career in the energy industry. 
Upon graduation, the student receives a M.S. degree in Engineering Management with a concentration in Energy Analytics. All courses are offered both online and on campus. If you are an on campus student, you can come to the classroom, and enjoy the face-to-face interactions with the professors and students. If you are a remote student, you can take the courses at home or in your office, and enjoy the flexibility brought by the modern communication technologies.

The program requires a minimum of 31 credits to graduate. These credits can be split into three segments:

1. Required core courses (10 credits)

Students will take the following four core courses at the beginning of the program:
  • EMGT 6980 - Industrial and Technology Management Seminars (1)
  • EMGT 5201 - Fundamentals of Deterministic System Analysis (3)
  • EMGT 5202 - Fundamentals of Stochastic System Analysis (3)
  • EMGT 5203 - Fundamentals of Engineering Management (3)

2. Elective and concentration courses (15 credits for thesis option, or 18 credits for project option)

To claim the energy analytics concentration, students should complete at least four out of the following six courses:
  • EMGT 5961 - Introduction to Energy Systems (3)
  • EMGT 5962 - Energy Markets (3)
  • EMGT 5963 - Energy Systems Planning (3)
  • EMGT 5964 - Case Studies in the Energy Industry (3)
  • EMGT 6965 - Energy Analytics (3)
  • EMGT 6910 - Technological Forecasting and Decision Making (3)
In addition, I recommend the following elective courses to the energy analytics students:
  • EMGT 6905 - Designed Experimentation (3)
  • EMGT 6952 - Engineering Systems Optimization (3)
  • EMGT 6955 - Systems Reliability Engineering (3)
The on campus students also have the opportunity to take courses from other departments. Up to two of them can be recognized as electives. The remote students may also transfer up to two courses from other universities upon the approval of the graduate director.

3. Capstone (6 credits for thesis option, or 3 credits for project option)

Students interested in conducting research at BigDEAL should choose the 6-credit thesis option. Many real-world problems are good candidate topics for master thesis research. Here are two examples of master thesis research from former BigDEAL graduates (TSG2015; TSG2018). For those who are not interested in research, a 3-credit project option is available too.

We plan to launch the program this fall semester. The university has not updated the catalog yet, so this is a preview of the program. If you are interested, you may start the application HERE. Note that we offer a GRE waiver to the applicants with a bachelor degree in engineering from a U.S. ABET accredited school and two years of relevant industry experience.

Last but not least, Happy Valentine's Day! 

Thursday, February 8, 2018

Real-time Anomaly Detection for Very Short-term Load Forecasting

Many very short-term load forecasting (VSTLF) models in literature rely on lagged loads, while most of these VSTLF papers assume perfect information of the lagged loads. As a result, the accuracy reported in the VSTLF literature has been amazingly high. In reality, however, load forecasters may not have access to the load values of the most recent few hours. The imperfection of the recent load information would certainly affect the load forecast accuracy. This paper tackles a practical problem, how to detect the anomalies in the most recent load information.

Jian Luo, Tao Hong and Meng Yue, "Real-time anomaly detection for very short-term load forecasting," Journal of Modern Power Systems and Clean Energy, in press, available online. (open access)

Real-time Anomaly Detection for Very Short-term Load Forecasting

Jian Luo, Tao Hong and Meng Yue


Although the recent load information is critical to very short-term load forecasting (VSTLF), power companies often have difficulties in collecting the most recent load values accurately and timely for VSTLF applications. This paper tackles the problem of real-time anomaly detection in most recent load information used by VSTLF. This paper proposes a model-based anomaly detection method that consists of two components, a dynamic regression model and an adaptive anomaly threshold. The case study is developed using the data from ISO New England. This paper demonstrates that the proposed method significantly outperforms three other anomaly detection methods including two methods commonly used in the field and one state-of-the-art method used by a winning team of the Global Energy Forecasting Competition 2014. Finally, a general anomaly detection framework is proposed for the future research. 

Friday, February 2, 2018

Load Forecasting Using 24 Solar Terms

Can we use Chinese calendar to forecast the load in the U.S.? Since I started my load forecasting practice 10 years ago, this has been a question sitting in my mind. One year ago, we decided to the test this idea. In short, the answer is YES. In the big data era, this approach would fall in the category of leveraging a variety of data sources.

This paper will be collected in the MPCE special section "Forecasting in Modern Power Systems" (Call For Papers). The paper is open access, so you can read the full content and download the PDF file for free. Special thanks to the journal editorial office for the neat copy-editing work. I truly enjoyed the publication process. Unlike most other open access journals that charge the authors a big fee for publishing the papers, this one does not charge a dime. I would highly recommend this journal to those who are interested in publishing open access papers in energy forecasting but do not want to pay for the publication fees. 

Jingrui Xie and Tao Hong, "Load forecasting using 24 solar terms," Journal of Modern Power Systems and Clean Energy, in press, available online. (open access

Load Forecasting Using 24 Solar Terms

Jingrui Xie and Tao Hong


Calendar is an important driving factor of electricity demand. Therefore, many load forecasting models would incorporate calendar information. Frequently used calendar variables include hours of a day, days of a week, months of a year, and so forth. During the past several decades, a widely-used calendar in load forecasting is the Gregorian calendar from the ancient Rome, which dissects a year into 12 months based on the Moon’s orbit around the Earth. The applications of alternative calendars have rarely been reported in the load forecasting literature. This paper aims at discovering better means than Gregorian calendar to categorize days of a year for load forecasting. One alternative is the solar-term calendar, which divides the days of a year into 24 terms based on the Sun’s position in the zodiac. It was originally from the ancient China to guide people for their agriculture activities. This paper proposes a novel method to model the seasonal change for load forecasting by incorporating the 24 solar terms in regression analysis. The case study is conducted for the eight load zones and the system total of ISO New England. Results from both cross-validation and sliding simulation show that the forecast based on the 24 solar terms is more accurate than its counterpart based on the Gregorian calendar.

Tuesday, January 30, 2018

Integrated Facility Location and Production Scheduling in Multi-generation Energy Systems

Multiple energy systems integration has been attracting much attention during the recent years. At the PES General Meeting 2017, I co-chaired a session "Accommodating Intermittent Renewable Energy by Multiple Energy Systems Integration: Forecasting, Operations and Planning", which was very well attended.

In this paper, we tackle the problem from a systems perspective by investigating the network design philosophy and exploring the economic value of the multi-generation technologies under demand uncertainties. The work is certainly out of the mainstream energy forecasting research I have published in the past. Nevertheless, readers of this blog may find it intriguing, because some of the forecasts we produce are going to be fed into the decision making processes for locating power plants and generation scheduling.


Qiaochu He and Tao Hong, "Integrated facility location and production scheduling in multi-generation energy systems," Operations Research Letters, vol.46, no.1, pp 153-157, January 2018. (working paper; ScienceDirect)

Integrated Facility Location and Production Scheduling in Multi-generation Energy Systems

Qiaochu He and Tao Hong


In this paper, we investigate the energy system design problems with the multi-generation technologies, i.e., simultaneous generation of multiple types of energy. Our results illustrate the economic value of multi-generation technologies to reduce spatio-temporal demand uncertainty by risk pooling both within and across different facilities.

Friday, January 12, 2018

RTE Forecasting Challenge 2018

RTE, the French TSO is organizing its forecasting challenge for the second time. This short-term winter electricity demand challenge includes two parts, point forecasting and probabilistic forecasting. The registration will open until January 21, 2018, followed by both parts of the challenge running simultaneously from January 22 to February 10, 2018.

If you are interested, you can register at For non-French speakers, if you see the website in French, you may click the UK flag on the top to view the English version.

According to Geert Scholma, who took the 4th place in the first RTE forecasting challenge, it was "the most exciting competition so far". I guess this one will be very competitive too.

It's  nice to kick of the new year with such an interesting competition, isn't it? 

Thursday, December 21, 2017

Short-term Industrial Load Forecasting: a Case Study in an Italian Factory

Frequent readers of this blog know how much I dislike today's peer review system in the academic world. The experience this time was truly pleasant.

About two years ago, I submitted a paper to PMAPS2016, which later turned into a TSG paper. The quality of the review comments we received from PMAPS on the original submission was super high, way beyond my expectation. I managed to get the contact information from that reviewer, Antonio Bracale. I then reached out to him to express my appreciation. Later Antonio came back to me with a collaboration proposal on industrial load forecasting. This is the first paper from our collaboration. 

The load forecasting literature has been so dominated by forecasting at high voltage level. The smart grid initiatives stimulated many papers at medium or low voltage level. Nevertheless, industrial load forecasting is still an important area that has not yet been extensively studied. This is certainly not the first industrial load forecasting paper, but our findings from the real-world data collected at an Italian factory may be helpful to the others dealing with similar problems. 

Antonio Bracale, Guido Carpinelli, Pasquale De Falco and Tao Hong, "Short-term industrial load forecasting: a case study in an Italian factory," 2017 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe), Torino, Italy, September 26-29, 2017

Short-term Industrial Load Forecasting: a Case Study in an Italian Factory

Antonio Bracale, Guido Carpinelli, Pasquale De Falco and Tao Hong


Excellence in the planning and operations of power systems largely relies on accurate forecasts of loads. Although load forecasting has been extensively studied over the past several decades, the scientific community has not yet paid much attention to industrial load forecasting. The electricity demand of factories depends on many factors, of which some are uncommon or not as important in the classical load forecasting models. For instance, the scheduled processes and work shifts are very important to forecasting short-term industrial loads. In this paper, we offer some insights into modeling industrial loads. We develop a set of multiple linear regression models for an Italian factory that manufactures transformers. The proposed models outperform two other benchmark models for forecasting industrial loads 24 hours in advance.

Friday, December 8, 2017

Energy Analytics (Fall 2017)

This semester is the fourth time I'm teaching Energy Analytics at UNC Charlotte. I have been offering this course every Fall since 2014. Previously I blogged about the offering in Fall 2015 (see THIS POST).

Student Profile

The class started with 11 master students and 1 PhD student. The master students were from three programs: engineering management (8), applied energy (2), and economics (1). The PhD student was from the PhD program in infrastructure and environmental systems.

After the first mid-term exam, 4 master students from engineering management withdrew the class. The other 8 students completed the course at the end. Here is the group picture including the 8 students, graduate teaching assistant Masoud Sobhani, and myself.

Energy Analytics group picture (Fall 2017)