Friday, April 27, 2018

Weather Data for Energy Analytics

Being an energy forecaster, I am genuinely interested in meteorology. I even recruited a master student who was a practicing meteorologist in Hawaii (see the blog post about Ying Chen). The more energy forecasting projects I conduct, the more I appreciate the value of weather data. In GEFCom2014, the top 1 place of the solar track was a team of meteorologists from Australia, who completely dominated the track. In GEFCom2017, the top 1 place of the final match was a team of meteorologists from Japan. I truly believe that the energy forecasting community can better leverage meteorology than what we do today. Here is an article about two use cases of weather data for energy analytics. In fact we merged two papers into one by removing the sophisticated mathematics and statistics to keep the story readable to a broad audience. The IEEE Power and Energy Society is so kind to offer the open access to this paper, so that people can read it for free.


Jonathan Black, Alex Hofmann, Tao Hong, Joseph Roberts, and Pu Wang, "Weather data for energy analytics: from modeling outages and reliability indices to simulating distributed photovoltaic fleets," IEEE Power and Energy Magazine, vol.16, no.3, pp 43-53, May-June 2018. (Open AccessIEEE Xplore)

Weather Data for Energy Analytics

From Modeling Outages and Reliability Indices to Simulating Distributed Photovoltaic Fleets

Jonathan Black, Alex Hofmann, Tao Hong, Joseph Roberts, and Pu Wang


Weather impacts virtually all facets of our daily life. As a result, many business sectors are affected by weather conditions, and the power industry is no exception. Weather is a major influencer on system reliability and a key driver of both power supply and demand. In this article, we will demonstrate novel uses of weather data for energy analytics via two utility applications. We first use easily accessible weather data together with regression analysis to model distribution outages and construct a probabilistic view of reliability indices that helps reveal a utility’s reliability trend. We then use high-resolution, commercial-grade weather data to develop realistic simulations of anticipated behind-the-meter photovoltaic (PV) fleets

Friday, April 20, 2018

Training Energy Data Scientists

Traditional power engineering curriculum has been heavily focusing on the engineering aspects of power systems, such as power flow, state estimation, stability and control. Data science has never been a focus in the past. I saw that gap 5 years ago, predicted the shortage of data scientists in the power industry, and left a great place to work to come back to academia. Nowadays, when other business sectors are offering 6-figure salaries to fresh graduates, utilities are having a hard time to compete on the analytics talents. Recently I had the opportunity to collaborate with Prof. David Wenzhong Gao from University of Denver and three other utility executives to put our thoughts in a paper.


Tao Hong, David Wernzhong Gao, Tom Laing, Dale Kruchten, and Jorge Calzada, "Training energy data scientists: universities and industry need to work together to bridge the talent gap," IEEE Power and Energy Magazine, vol.16, no.3, pp 66-73, May-June 2018. (IEEE Xplore)

Training Energy Data Scientists 

Universities and Industry Need to Work Together to Bridge the Talent Gap

Tao Hong, David Gao, Tom Laing, Dale Kruchten, and Jorge Calzada


The workforce crisis is nothing new to the U.S. power industry. It has been a growing concern of both governments and industry organizations since the early 2000s. Meanwhile, the growth of data during the past decade has led to a demand surge for data analytics across all business sectors. The shortage of an electricity workforce and the increasing demand for data analytics present an emerging challenge as well as opportunity for university power engineering programs to bridge the data analytics talent gap. After gathering various perspectives from members of academia, industry, and government, we propose an interdisciplinary and entrepreneurial approach to revising the traditional power engineering curriculum for training the next generation of energy data scientists.

Saturday, April 14, 2018

Call For Papers: Forecasting for Social Good | International Journal of Forecasting

International Journal of Forecasting

Special Issue on Forecasting for Social Good

The area of forecasting and its link to decision making has been under research for decades. Whilst there have been many influential contributions seeking to examine the effects of forecasting under financial and economic motives, very little has been contributed (both in regular conferences and journal publications) on forecasting with social impact – that is forecasting for the social good, regardless of the financial implications, or optimizations attempted based on economic terms.

The International Journal of Forecasting (IJF) is excited to announce this Call For Papers for the special issue on “Forecasting for Social Good”. The purpose of the special issue is to attract high quality papers that are concerned with the social impact of forecasting.

Areas of interest include, but are not limited to:
  • Health and healthcare
  • Humanitarian operations
  • Disaster relief
  • Education
  • Social services
  • Environment
  • Sustainability
  • Sharing economy
  • Transportation
  • Urban planning
  • Fraud, collusion, and corruption
  • Government policy
  • Poverty
  • Privacy
  • Cyber security
  • Crime and terrorism

Submission Deadline: 31 October 2018.

Submission Guidelines:

To submit a paper for consideration for the Special Issue, please upload your paper online and include a cover letter clearly indicating that the paper is for the special issue “Forecasting for Social Good”. The webpage for online submission is Instructions for authors are provided at . All papers will follow IJF’s double-blind refereeing process. For further information about the Special Issue, please contact the guest editors.

Guest Editors

Bahman Rostami-Tabar, Cardiff University, UK

Michael Porter, University of Alabama, USA

Tao Hong, University of North Carolina at Charlotte, USA

Friday, April 13, 2018

Which Model is the Best?

Recently Rob Hyndman blogged about the history of forecasting competitions. I have read the post three times already. I learned something new each time. I wish every reader of this blog can also read that post and learn something from the history. Nevertheless, I would like to highlight a paragraph here:
[...] This reveals a view commonly held (even today) that there is some single model that describes the data generating process, and that the job of a forecaster is to find it. This seems patently absurd to me — real data comes from much more complicated, non-linear, non-stationary processes than any model we might dream up — and George Box famously dismissed it saying “All models are wrong but some are useful”.
I have been in the field load forecasting for a little more than 10 years. During the past decade, I received many "the best" questions: Which model is the best? Which technique is the best? Which software is the best? Which variable is the best? ...

There are some variants to these questions: Do you use neural networks? Do you use population in long term load forecasting? Do you use normal weather? Do you think demand response can reduce peak demand? Do you think a MAPE value of 20% is too high? ...

While the folks who asked these questions may expect a crisp answer of the best model, technique, software, variable, or a YES/NO answer, I had to disappoint them with "depends", sometimes followed by a lengthy elaboration.
It depends on the data, the business needs, the production environment, and many other factors... 
I don't have a single model to sell to my clients as "the best model". I recommend proper methodologies to the clients after making a comprehensive evaluation of their situations.

I think the root of these "the best" questions is that commonly held view:
There is some single model that describes the data generating process. 
I disagree with this view, and don't know where the view is originally coming from. I wish someone can write another history article to explain the source.

In load forecasting, there is no universally best model. We need empirical studies, many empirical studies, to show some evidence that one method is superior in some aspect. That's why I have been promoting reproducible research and benchmarking data pool and models.

In computational complexity and optimization, there is actually a theorem, no free lunch theorem

Sunday, March 25, 2018

Call For Papers: Data Analytics for Energy, Water, and Environment | IEEE Transactions on Engineering Management

IEEE Transactions on Engineering Management

Special Issue on Data Analytics for Energy, Water, and Environment

In 2008, the U.S. National Academy of Engineering proposed 14 grand challenges for engineering, of which five were related to energy, water, and environment. The world population has exceeded 7.6 billion in 2018, up from 6.7 billion in 2008. A common goal of the human beings is to meet the global need of energy and water without sacrificing the environment.

A growing interest in the recent decade is data analytics. Many industries have embarked analytics to extract actionable insights from the data and to make informed business decisions. This special issue of the IEEE Transactions on Engineering Management is aimed at highlighting the state of the art in data analytics applications in energy, water and environment.

The topics of interest include but are not limited to:
  • Quantitative models for water-energy nexus
  • Evaluating energy technology portfolios
  • Data integration and infusion for energy, water and environment applications
  • Cyber-physical systems for sustainable development
  • Power supply and air pollution
  • Building energy efficiency and indoor air quality
  • Energy efficiency analytics enabled by smart meter data
  • Clean energy/water access 
  • Energy systems planning under climate and water challenges
  • Risks and uncertainties in energy engineering projects
  • Energy management considering consumer behaviors
  • Sustainable supply chain design and inventory control
  • Water and energy management in sustainable agriculture
  • Environment, energy, and alternative transportation technologies

Submission Process

Please prepare the manuscript according to IEEE-TEM’s guidelines and submit it the journal’s Manuscript Central site. Please clearly state in the cover letter that the submission is for this special issue. 


Interested authors send 2-page extended abstracts to the Corresponding Guest Editor by August 31st 2018
Decisions on acceptance of abstracts by November 30th 2018
Full papers submitted by May 31st 2019

Corresponding Guest Editor

Dr. Tao Hong (
Department of Systems Engineering and Engineering Management 
University of North Carolina at Charlotte

Guest Editors

Dr. Hua Cai, Purdue University
Dr. Gang He, Stony Brook University
Dr. Guiping Hu, Iowa State University
Dr. Yueming Qiu, University of Maryland
Dr. Ekundayo Shittu, George Washington University
Dr. Hongliang Zhang, Louisiana State University
Dr. Ning Zhang, Tsinghua University


Dr. Tugrul Daim, Portland State University

Friday, March 23, 2018

Review of Smart Meter Data Analytics: Applications, Methodologies, and Challenges

About 10 years ago, the term "smart grid" was officially defined in the Energy Independence and Security Act of 2007 (EISA-2007). Soon after that, many power companies started their smart meter deployments. As of 2016, more than 70 million smart meters were installed in the united states. The world installed base was projected to reach 780 million by 2020, pushed by the mass roll-outs in China. We are now sitting on a gold mine of data collected by these smart meters. Last year I gave a forecast:
The energy companies will be moving more Gigabytes of data than GWh of electricity.
The scientific community has been trying to understand the smart meter data and get some actionable insights out of it. Thousands of papers have been published in the recent decade on the various aspects of smart meter data analytics. Last year, I worked with my collaborators in Tsinghua University to complete a review of smart meter data analytics. The paper was just put on the IEEE Xplore yesterday.

This is the longest paper ever published by the IEEE Transactions on Smart Grid. I'm sure reading this 24-page review article can save the readers significant amount of time from digging thousands of papers in the literature. Load forecasters may find some interesting stuff in Section III, which is dedicated to load forecasting in the smart grid era.


Yi Wang, Qixin Chen, Tao Hong, and Chongqing Kang, "Review of smart meter data analytics: applications, methodologies, and challenges," IEEE Transactions on Smart Grid, in press. (working paper; IEEE Xplore)

Review of Smart Meter Data Analytics: Applications, Methodologies, and Challenges

Yi Wang, Qixin Chen, Tao Hong, and Chongqing Kang


The widespread popularity of smart meters enables an immense amount of fine-grained electricity consumption data to be collected. Meanwhile, the deregulation of the power industry, particularly on the delivery side, has continuously been moving forward worldwide. How to employ massive smart meter data to promote and enhance the efficiency and sustainability of the power grid is a pressing issue. To date, substantial works have been conducted on smart meter data analytics. To provide a comprehensive overview of the current research and to identify challenges for future research, this paper conducts an application-oriented review of smart meter data analytics. Following the three stages of analytics, namely, descriptive, predictive and prescriptive analytics, we identify the key application areas as load analysis, load forecasting, and load management. We also review the techniques and methodologies adopted or developed to address each application. In addition, we also discuss some research trends, such as big data issues, novel machine learning technologies, new business models, the transition of energy systems, and data privacy and security.

Saturday, March 3, 2018

A Look Back at ISEA2017

About a year ago, I posted 7 reasons to attend ISEA2017. Last June, we had our very first International Symposium on Energy Analytics at the beautiful city Cairns, Australia. 50 researchers and practitioners from industry and academia across 16 countries attended the symposium. 36 of them also attended ISF2017.

The two-day agenda included 5 keynote presentations, 13 contributed presentations selected from 31 abstract submissions, and 5 GEFCom2017 finalists presentations. At the end of the first day, June 22, 2017, we hosted the GEFCom2017 award ceremony followed by a two-hour cocktail reception. The program agenda, presentations and pictures can be downloaded via THIS LINK.

We have been processing the full paper submissions since then. So far, three papers are currently in press with the Journal of Modern Power Systems and Clean Energy, one is in press with the Power & Energy Magazine. The remaining papers are going through the peer review system of the International Journal of Forecasting. The accepted ones together with the GEFCom2017 articles are going to be published in an energy forecasting special section edited by Pierre Pinson and myself.  

I want to thank many people who made ISEA2017 a successful event, the keynote speakers, the presenters, the GEFCom2017 winners, and all the attendees of ISEA2017. Special thanks to Rob Hyndman and George Athanasopoulos for the strong local support, our photographer Mitchell O'Hara-Wild for capturing the happy moments, and Pam Stroud for handling the registrations and helping keep things in order. 

Last but not least, big thanks to our sponsors, International Institute of Forecasters, Tangent Works and Journal of Modern Power Systems and Clean Energy, for their generous financial support that helped to keep the registration fees low. 

Many people have asked me about the next ISEA. I know there will be the next one, but I don't know when and where. If you have any idea or want to help with organizing ISEA, let me know! 

Wednesday, February 14, 2018

UNC Charlotte's M.S. Program in Engineering Management with Energy Analytics Concentration

Five years ago, I realized the big gap between academic offerings and industry needs in the energy analytics field. I took a brave step and a big salary cut to join UNC Charlotte, with the mission of "producing the next generation of finest analysts for the industry."

Since then, I have trained hundreds of students and working professionals, through the various courses I taught within and outside UNC Charlotte, and through the research activities at the BigDEAL lab that I founded to incubate the elites. I am pleased to see the growing interest in this emerging field of energy analytics and the enthusiasm from the my audience.

However, the gap did not really shrink despite my effort. In fact, the demand increase for energy analysts, which is now called energy data scientists, is more than the amount of graduates I can produce from my small shop!

To further enrich the teaching materials and serve a greater audience in the industry with diverse academic and professional backgrounds, I designed a 60-hour energy analytics curriculum. Many people who took those courses asked me the following question:
Is there a degree program that I can enroll to further my education in this area (energy analytics/forecasting)?
Previously, I directed them to BigDEAL and showed them the path to PhD. For those who were not ready for a PhD program, I didn't have much to offer, unfortunately.

Today, if you asked me the same question, I have a different and better answer:
We are launching a master program specifically designed for those who want to pursue a data science career in the energy industry. 
Upon graduation, the student receives a M.S. degree in Engineering Management with a concentration in Energy Analytics. All courses are offered both online and on campus. If you are an on campus student, you can come to the classroom, and enjoy the face-to-face interactions with the professors and students. If you are a remote student, you can take the courses at home or in your office, and enjoy the flexibility brought by the modern communication technologies.

The program requires a minimum of 31 credits to graduate. These credits can be split into three segments:

1. Required core courses (10 credits)

Students will take the following four core courses at the beginning of the program:
  • EMGT 6980 - Industrial and Technology Management Seminars (1)
  • EMGT 5201 - Fundamentals of Deterministic System Analysis (3)
  • EMGT 5202 - Fundamentals of Stochastic System Analysis (3)
  • EMGT 5203 - Fundamentals of Engineering Management (3)

2. Elective and concentration courses (15 credits for thesis option, or 18 credits for project option)

To claim the energy analytics concentration, students should complete at least four out of the following six courses:
  • EMGT 5961 - Introduction to Energy Systems (3)
  • EMGT 5962 - Energy Markets (3)
  • EMGT 5963 - Energy Systems Planning (3)
  • EMGT 5964 - Case Studies in the Energy Industry (3)
  • EMGT 6965 - Energy Analytics (3)
  • EMGT 6910 - Technological Forecasting and Decision Making (3)
In addition, I recommend the following elective courses to the energy analytics students:
  • EMGT 6905 - Designed Experimentation (3)
  • EMGT 6952 - Engineering Systems Optimization (3)
  • EMGT 6955 - Systems Reliability Engineering (3)
The on campus students also have the opportunity to take courses from other departments. Up to two of them can be recognized as electives. The remote students may also transfer up to two courses from other universities upon the approval of the graduate director.

3. Capstone (6 credits for thesis option, or 3 credits for project option)

Students interested in conducting research at BigDEAL should choose the 6-credit thesis option. Many real-world problems are good candidate topics for master thesis research. Here are two examples of master thesis research from former BigDEAL graduates (TSG2015; TSG2018). For those who are not interested in research, a 3-credit project option is available too.

We plan to launch the program this fall semester. The university has not updated the catalog yet, so this is a preview of the program. If you are interested, you may start the application HERE. Note that we offer a GRE waiver to the applicants with a bachelor degree in engineering from a U.S. ABET accredited school and two years of relevant industry experience.

Last but not least, Happy Valentine's Day! 

Thursday, February 8, 2018

Real-time Anomaly Detection for Very Short-term Load Forecasting

Many very short-term load forecasting (VSTLF) models in literature rely on lagged loads, while most of these VSTLF papers assume perfect information of the lagged loads. As a result, the accuracy reported in the VSTLF literature has been amazingly high. In reality, however, load forecasters may not have access to the load values of the most recent few hours. The imperfection of the recent load information would certainly affect the load forecast accuracy. This paper tackles a practical problem, how to detect the anomalies in the most recent load information.

Jian Luo, Tao Hong and Meng Yue, "Real-time anomaly detection for very short-term load forecasting," Journal of Modern Power Systems and Clean Energy, in press, available online. (open access)

Real-time Anomaly Detection for Very Short-term Load Forecasting

Jian Luo, Tao Hong and Meng Yue


Although the recent load information is critical to very short-term load forecasting (VSTLF), power companies often have difficulties in collecting the most recent load values accurately and timely for VSTLF applications. This paper tackles the problem of real-time anomaly detection in most recent load information used by VSTLF. This paper proposes a model-based anomaly detection method that consists of two components, a dynamic regression model and an adaptive anomaly threshold. The case study is developed using the data from ISO New England. This paper demonstrates that the proposed method significantly outperforms three other anomaly detection methods including two methods commonly used in the field and one state-of-the-art method used by a winning team of the Global Energy Forecasting Competition 2014. Finally, a general anomaly detection framework is proposed for the future research.