Thursday, October 8, 2015

NPower Forecasting Challenge, Again

RWE npower is hosting its forecasting challenge again to recruit summer interns for the next year. Same as the previous challenge, they are going to recruit students from UK but would like to open the competition the entire world. 

In the previous challenge, BigDEAL students did a great job. We also wrote a paper to summarize the winning method. I will again encourage my students to participate in this challenge. Meanwhile, I would encourage your participation as well. The registration is OPEN now. Look forward to seeing your name on the leaderboard!

Saturday, September 26, 2015

Fall 2015 In-class Short Term Load Forecasting Competition

This semester, I'm teaching my Energy Analytics course. The first exam is a short term load forecasting competition for the Dominion load zone under PJM. The competition rules are listed below:
  1. The competition will start on 10/5/2015, and end on 10/9/2015. 
  2. The exam is individual effort. Each student form a single-person team. No collaboration is allowed.
  3. The day-ahead hourly load forecast is due on 11:45am ET each day. The first forecast of 24 hourly loads for 10/6/2015 is due on 11:45am 10/5/2015. The fifth and last forecast of 24 hourly loads for 10/10/2015 is due on 11:45am 10/9/2015.
  4. The student can use any data (load, weather, calendar, economy, location, etc.) they can find to make the forecast as accurate as possible.
  5. MAPE is the error measure in this competition. The preliminary MAPE is calculated on daily basis using preliminary hourly load data published by PJM. The final MAPE is calculated using historical metered load data published by PJM in November.The final rankings and scores are based on the final MAPE. 
  6. The benchmark is the day-ahead load forecast released by PJM on 11:45am ET each day. A student receive no credit if not beating the benchmark nor ranking top 6 in the class. No late submission is allowed. 
I would like to open this competition to students and professionals outside my class. If you are interested in joining the competition, please contact me for detailed instructions. 

Friday, September 25, 2015

Job Opening at ERCOT - Planning/Load Forecasting Analyst

Posting a job for an ERCOT friend. So far the manager and all the new hires and in this team have taken load forecasting courses from me. If you have taken my course too, that's a good thing to highlight in your resume. The manager has a very strong forecasting background, so I'm sure you will learn a lot from him and the team on this job. If interested, please submit the application online HERE.

Planning/Load Forecasting Analyst 1, 2, 3, or Sr.

All times are in Eastern Daylight Time.
Job ID 2015-1575
# Positions 1
Location US-TX-Taylor
Posted Date 9/24/2015
Category  Engineering

More information about this job:

Saturday, September 19, 2015

BigDEAL Recruitment Process

I came back to academia to produce the finest data scientists for the energy industry. Over the past two years, I have established the factory, BigDEAL, offering golden opportunities for students to conduct cutting edge research through solving real world problems. As BigDEAL is getting increasingly recognized by the industry and academia, I'm receiving more and more requests from both sides too. While the employers are anxious to hire my students, many prospective students are eager to join BigDEAL.

There is no way for me to respond to every applicant. Here I'm sharing the BigDEAL recruitment process, so that the applicants can prepare accordingly.
  1. When an application reaches my inbox, I take a brief look to see if the person has followed the instructions. If so, and if the background seems to be relevant, I would forward the email to the lab manager of BigDEAL, typically a senior PhD student. 
  2. The lab manager will conduct the first round of interview. If s/he believes the person may be a good match, s/he will forward the application to several other lab members for the second round interview. After the second round interview, all interviewers will get together to give me their feedback. 
  3. If the feedback is overall positive, I will conduct the last round interview. The applicants who pass the final round of interview will be offered a position at the BigDEAL.
With the strong support from the industry, I have never had a budget constraint when making hiring decisions. Nevertheless, I do recognize that the quality of the raw materials plays an important role of driving the quality of the end product. Therefore, I have decided to maintain the highly selective recruitment process. So far, we have been hiring no more than one student each semester.

If you are interested in joining BigDEAL, I would strongly suggest that you read the blog posts for prospective students before contacting me. Best luck with your application!

Friday, September 11, 2015

Job Opening at AECC: Load Forecasting Analyst

Posting a job for a friend (and friend/colleague of Crystal Glidewell) at AECC. Interested applicants should apply at According to the hiring manager, the min Experience is really 0-2 years. The amount of experience will determine if they are an Analyst I or II.

Analyst - Load Forecasting
Job ID: 25036275
Position Title: Analyst - Load Forecasting
Company Name: Arkansas Electric Cooperative Corporation
Entry Level: No
Job Duration: Indefinite
Location(s): Little Rock, Arkansas, United States
Posted: September 11, 2015
Min Education: BA/BS/Undergraduate
Min Experience: 1-2 Years
Required Travel: 10-25%
Salary: $44,000.00 - $58,000.00 (Yearly Salary)

Job Description

Thursday, September 10, 2015

Where is My Dissertation?

I got my PhD with zero journal paper published or accepted. Since then, I have been wondering how many journal papers can eventually be published out of my dissertation. I think today, the end of the first five-year period, is a good time to check it out.

The list below maps the original chapters and sections to my papers:

The dissertation itself has been cited 41 times according to GoogleScholar

As I mentioned in the blog post (The Gap between Academic Research and Industry Needs) two years ago, the first two papers written from my dissertation materials were rejected by a journal in 2010. I got negative comments from 10 reviewers, of which most are nonsense. One of the reviewers loved to use the word "garbage" in the review comments. 

A couple of months ago, I got another rejection from the same journal. The similar garbage-style review comments showed up again. Shame on me - I should have stopped sending papers to this journal. Instead of yelling back to the editor-in-chief, I stayed calm this time and moved on, with the hope that the journal can clean up the garbage reviewers.

At least, the editor and editor-in-chief should do their homework: clean up the garbage comments before sending them out to the authors. 

Monday, August 31, 2015

Probabilistic Electric Load Forecasting: A Tutorial Review

Since 2012, Shu Fan and I have been trying to put together a review paper on load forecasting. We first started with a review on point load forecasting. Then both of us felt that our community may need a review paper on probabilistic load forecasting more than anything else, so we shifted the direction. The first submitted version was released at the beginning of GEFCom2014 for the contestants to have some basic idea about probabilistic load forecasting (see 10 Recommended Papers for GEFCom2014 Contestants). After going through two revisions, the paper went from 32 pages to 53 pages at its third submission. Many thanks to the valuable comments from Rob Hyndman, Pierre Pinson, Rafal Weron, and two anonymous reviewers, the quality of this review has gone up significantly. Today I'm very pleased to announce that this review paper on probabilistic load forecasting was just accepted by International Journal of Forecasting. We hope this paper is useful to the researchers and practitioners who are in the load forecasting community or interested in load forecasting.

Tao Hong and Shu Fan, "Probabilistic electric load forecasting: a tutorial review", International Journal of Forecasting, accepted. Working paper available online

Probabilistic Electric Load Forecasting: A Tutorial Review

Tao Hong and Shu Fan

Load forecasting is a fundamental business problem established since the inception of the electric power industry. Over the past 100 plus years, both research efforts and industry practices in this area are primarily focused on point load forecasting. In the recent decade, due to the increased market competition, aging infrastructure and renewable integration requirements, probabilistic load forecasting is becoming more and more important to energy systems planning and operations. This paper offers a tutorial review of probabilistic electric load forecasting, including notable techniques, methodologies, evaluation methods, and common misunderstandings. We also underline the need to invest in additional research, such as reproducible case studies, probabilistic load forecast evaluation and valuation, and consideration of emerging technologies and energy policies in probabilistic load forecasting process. 

Tuesday, August 18, 2015

Job Opening at NCEMC: Load Research Analyst

Posting a job for my friend at NCEMC. Its forecasting team has been at the leading edge of the load forecasting research and practice. Over the past several years, we have worked together to develop several load forecasting methodologies that are very useful in practice. Most of them are now being used by many other power companies worldwide. You may check out our recent papers on long term probabilistic load forecasting and normalization, weather station selection, and residual simulation for probabilistic load forecasting. We have also planned many exciting research activities for the next few years, of which some are listed in the job description below. I would like to add one more personal note about this job: I think the hiring manager is another reason that this position is so appealing. I haven't seen many managers who are as generous and supportive as this one in terms of professional development for the employees. If you are interested, please submit your resume online HERE.

Job Opening at NCEMC: Load Research Analyst

North Carolina Electric Membership Corporation, one of the largest generation and transmission cooperatives in the country, seeks a Load Research Analyst for its TSE Services Division.

The successful applicant will join a load forecasting team that is empowered with the latest technologies and cutting edge methods for both short- and long-term demand modelling.

Activities will also include measurement and verification of demand-response programs and the development of small area real-time forecasts for Distribution Systems Operations.

The position requires advanced knowledge of applied statistics, microeconomics, econometrics, time-series methods, statistical analysis of large data sets and load research methods.

A Bachelor of Science degree in engineering, economics, statistics, meteorology or equivalent is required and an advanced degree is preferred.

A minimum of five years professional experience in utility load forecasting or a related field and experience with forecasting tools from SAS is desired.

Applicants should submit their resume at

Friday, July 31, 2015

Reading for Writing

Writing skills are badly needed in the professional world, no matter in the industry or academia. I have seen many people (mostly international students, including myself) struggling with their writing skills. I don't believe there is any shortcut but constant and regular writing practice to improve writing skills. In addition, I think reading is a good compliment to writing practice. Here I'm putting together a list of recommended books.

The first two are on general writing (not necessarily scientific writing):
  • On Writing Well by William Zinsser
  • The Elements of Style by William Strunk and E. B. White
Then here are two books for scientific writing:
  • Scientific Writing and Communication: Papers, Proposals, and Presentations by  Angelika Hofmann
  • Writing Science: How to Write Papers That Get Cited and Proposals That Get Funded by Joshua Schimel 
I also got a list of classic novels from my colleague Jason Wilson (co-author of my TSG2014 paper):
  • Slaughterhouse Five by Kurt Vonnegut, Jr.
  • Gulliver’s Travels by Jonathan Swift
  • Ulysses by James Joyce
  • Hamlet by Shakespeare 
  • 1984 by George Orwell
  • Brave New World by Aldous Huxley
  • Animal Farm by George Orwell
  • The Grapes of Wrath by John Steinbeck
  • A Farewell to Arms by Ernest Hemingway
  • The Old Man and The Sea by Ernest Hemingway
  • Charlotte’s Web by E.B. White
  • War and Peace by Leo Tolstoy
  • Jurassic Park by Michael Crichton
Here I'm trying not to recommend a list of scientific papers in energy forecasting. I would rather suggest that we focus on the analytical and technical aspects of those papers.

Happy reading!

Saturday, July 25, 2015

Combining Load Forecasts from Independent Experts: Experience at NPower Forecasting Challenge 2015

Forecast combination is regarded as one of the best practices of forecasting. I think it is a straightforward and practical approach to improving existing forecasts. This paper describes the method my students took in the NPower Forecasting Challenge 2015. We will present the paper at the 47th North America Power Symposium.

Jingrui Xie, Bidong Liu, Xiaoqian Lyu, Tao Hong, and David Basterfield, "Combining load forecasts from independent experts: experience at NPower forecasting challenge 2015", the 47th North American Power Symposium (NAPS2015), October 4 - 6, 2015

Combining Load Forecasts from Independent Experts
Experience at NPower Forecasting Challenge 2015

Jingrui Xie, Bidong Liu, Xiaoqian Lyu, Tao Hong, and David Basterfield


The NPower Forecasting Challenge 2015 invited students and professionals worldwide to predict daily energy usage of a group of customers. The BigDEAL team from the Big Data Energy Analytics Laboratory landed a top 3 place in the final leaderboard. This paper presents a refined methodology based on the implementation during the competition. We first build the individual forecasts using several forecast techniques, such as Multiple Linear Regression (MLR), Autoregressive Integrated Moving Average (ARIMA), Artificial Neural Network (ANN) and Random Forests (RF). We then select a subset of the individual forecasts based on their performance on a validation period, a.k.a. post-sample. Finally we obtain the final forecast by averaging the selected individual forecasts. The forecast combination on average yields a better result than the forecast from a single technique.