Monday, February 8, 2016

Analytics, Smart Grid and Big Data: Are They Like Teenage Sex?

I can hardly find the original source for this quote about teenage sex:
Everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.
Over the past few decades, people have been inventing, abusing, reinventing and re-abusing various buzzwords. The title of this blog post is taken from a talk I gave last year.

In that talk, I was showing the audience how the public interest on these three terms has been evolving over time. For instance, "smart grid" on Google Trends look like this:

I also introduced my understanding of big data analytics using a series of research projects on load forecasting with NCEMC. At the end, I was making three points:
  • Forget about the buzzwords
  • Focus on what the industry needs
  • Solve real-world problems
The original presentation is available HERE, in case you are interested in taking a look.

p.s., when naming my lab two years ago, I almost used all of these three terms, analytics, smart grid and big data. Because I didn't really understand what smart grid is, I put "energy" instead of "smart grid" in my lab's name, making it BigDEAL - Big Data Energy Analytics Laboratory

Friday, February 5, 2016

Job Opening at DTU: Postdoc in High-Dimensional Renewable Energy Forecasting

Posting a job for my friend and colleague Pierre Pinson. If you are interested, please apply online HERE.

The Centre for Electric Power and Energy at the Department of Electrical Engineering seeks a highly talented applicant for a Postdoc position with focus on high-dimensional renewable energy forecasting. The period of employment is 1 year.

Responsibilities and tasks 

Forecasting of renewable energy generation has evolved from placing emphasis on single wind farms and solar power plants, with limited amount of data as input, to directly predict generation at a large number of sites (tens for portfolio management and hundreds for network management) with massive amount of varied data as input. Besides, while predictions were mainly issued in a deterministic framework still a few years ago, today one expects a complete probabilistic description of future power generation at each and every site of interest, but also of dependencies among sites and lead times.  

The central objective is to build on recent research works and achievements in the development of large-scale datasets and open-source platforms for wind and solar power forecasting, partly supported by EPRI (US) and EDF (France). This will translate to proposing cutting-edge methodological developments for simultaneously predicting power generation at hundreds to thousands of sites while also considering implementation and visualization aspects. The successful candidate will contribute to the internal collaboration at the department, external collaboration with industrial partners and possibly contribute to some of the teaching activities.


Candidates should have a PhD or equivalent.

In the assessment of the candidates consideration will be given to
  • scientific production and research potential at an international level
  • the ability to promote and utilize research results, through journal publications and conference presentation and all-round experience, preferably including international experience
  • strong publication record
  • innovative skills, ability to generate new ideas and to implement them in practice
  • independent problem solving skills
  • excellent communication and language skills
  • an all-round experience basis, prefably including international experience
For the specific position consideration will be given to documented experience (at high international level) with at least one of the following topics:
  • nonlinear and high-dimensional modeling with application to energy analytics and market-related problems
  • modeling and forecasting of renewable energy generation (or electric load, market prices, etc.)
  • programming skills in one or more of the following: R, Python, Matlab
We offer

We offer an interesting and challenging job in an international environment focusing on education, research, public-sector consultancy and innovation, which contribute to enhancing the economy and improving social welfare. We strive for academic excellence, collegial respect and freedom tempered by responsibility. The Technical University of Denmark (DTU) is a leading technical university in Northern Europe and benchmarks with the best universities in the world.

Salary and terms of employment

The appointment is based on the collective agreement with the Confederation of Professional Associations. The allowance is agreed with the relevant union.

Workplace is DTU Lyngby Campus, and the period of employment is 1 year and upon a positive evaluation, it will be extended to two years.

Further information 

Further information may be obtained from Professor Pierre Pinson, phone.: +45 4525 3541, or Professor Jacob Østergaard, phone : +45 25 13 05 01.

You can read more about the Centre for Electric Power and Energy at

Application procedure

Please submit your online application no later than 25 February 2016. Please submit the application as one PDF file containing all materials to be given consideration. Open the link "Apply online," fill in the online application form, and attach all your materials in English in one PDF file. The file must include:
  • Application (cover letter)
  • CV
  • Diploma (an official translation into English)
  • List of publications
  • Applications and enclosures received after the deadline will not be considered.
All qualified candidates irrespective of age, gender, disability, race, religion or ethnic background are encouraged to apply.

The Department of Electrical Engineering is the central department at the Technical University of Denmark in electrical, biomedical engineering and acoustic technology. It is our goal to ensure an engineering training and research, which matches that of other leading universities around the world. The department is organised into sections each covering important areas of electrical, biomedical engineering and acoustic technologies. Including PhD students we have a total of 260 staff members

DTU is a technical university providing internationally leading research, education, innovation and public service. Our staff of 5,800 advance science and technology to create innovative solutions that meet the demands of society; and our 10,300 students are being educated to address the technological challenges of the future. DTU is an independent academic university collaborating globally with business, industry, government, and public agencies.

Monday, January 25, 2016

Probabilistic Energy Forecasting: Global Energy Forecasting Competition 2014 and Beyond

It is really hard to write an introduction for this paper, because there are too many things to highlight the outcome of hundreds of hours invested by the organizers of GEFCom2014 and thousands of hours spent by the GEFCom2014 contestants. In one sentence,
This is a MUST READ paper in energy forecasting. 
In this paper, we
  • Summarized 7 papers collected through the special issue CFP process, including one on anomaly detection for gas demand data, two on load forecasting, two on price forecasting, one on wave energy forecasting, and one on wind speed forecasting;
  • Introduced the GEFCom2014 together with the in-class probabilistic load forecasting competition I organized in 2015;
  • Commented on the methodologies used by the winning teams of GEFCom2014;
  • Published the 120MB data used in the four tracks of GEFCom2014 and the in-class competition;
  • Made 12 predictions for the next decade of energy forecasting
We sincerely hope that you enjoy reading this paper and can help contribute to the energy forecasting community.


Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli and Rob J. Hyndman, "Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond", International Journal of Forecasting, in press. working paper available from

Probabilistic Energy Forecasting: Global Energy Forecasting Competition 2014 and Beyond

Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli, and Rob J Hyndman


The energy industry has been going through a significant modernization process over the last decade. Its infrastructure is being upgraded rapidly. The supply, demand and prices are becoming more volatile and less predictable than ever before. Even its business model is being challenged fundamentally. In this competitive and dynamic environment, many decision-making processes rely on probabilistic forecasts to quantify the uncertain future. Although most of the papers in the energy forecasting literature focus on point or single-valued forecasts, the research interest in probabilistic energy forecasting research has taken off rapidly in recent years. In this paper, we summarize the recent research progress on probabilistic energy forecasting. A major portion of the paper is devoted to introducing the Global Energy Forecasting Competition 2014 (GEFCom2014), a probabilistic energy forecasting competition with four tracks on load, price, wind and solar forecasting, which attracted 581 participants from 61 countries. We conclude the paper with 12 predictions for the next decade of energy forecasting. 

Sunday, January 10, 2016

Job Opening at Dominion: Load Research Analyst III

Posting a job opportunity for a friend from Dominion. If you are interested, please submit your application HERE.

Load Research Analyst III

Req ID  2015-7877
Job Locations US-VA-Richmond
Posted Date  12/22/2015
Location/Facility Name Richmond General Office
Category  031 - Transmission & Distribution

Friday, January 8, 2016

BigDEAL Backup and Archiving Process

One surprise I got over this winter holiday is the sudden death of my hard drive. The even bigger surprise is that I didn't have the backup. More precisely, I backed up the data every semester, but I accidentally deleted the backup during my last backup over the summer. That said, I have lost almost all the files created since I joined UNC Charlotte in August 2013.

This hard lesson reminded me to establish the BigDEAL backup and archival process below.

Storage options

Each BigDEAL member should have at least one external hard drive and one cloud drive for backup purpose. BigDEAL uses two external hard drives for backup purpose. One (M-drive) is managed by the lab manager (typically a senior PhD student on campus) for monthly backup. The other one (S-drive) managed by the BigDEAL director (Dr. Hong) for semester backup. BigDEAL has been using OneDrive as the cloud drive for archival purpose.

Backup process

BigDEAL backup process is executed on three levels:
  • Weekly backup. Each BigDEAL member is expected to perform weekly backup on the personal hard drive and through the cloud storage service.
  • Monthly backup. At the end of each month, each BigDEAL member should back up all the data on M-drive. 
  • Semester backup. At the end of each semester, each BigDEAL member should back up all the data on S-drive. 

Archiving process

BigDEAL archives the following items on the cloud:
  • Data
  • Code
  • Papers
  • Thesis and dissertations
  • Technical reports
  • Presentations
  • Administrative reports (weekly reports, semester summary, and semester plan, etc.)
In principle, once an item is acquired, updated or completed, the BigDEAL member in charge of the project or corresponding research subject should archive the item on the cloud and report the archival status to the BigDEAL director.

When archiving the revised version of the data, all changes and justifications should be documented and archived along with the new version. When archiving a paper, all the data, code, submitted versions, decision letters, response letters, and published version should be archived. 

Monday, January 4, 2016

GEFCom2014 Probabilistic Electric Load Forecasting: An Integrated Solution with Forecast Combination and Residual Simulation

Guest Blogger: Jingrui Xie

My adventure on load forecasting started in 2012, when I was the primary developer for the SAS Energy Forecasting solution. Our first customer was North Carolina Electric Membership Cooperation, who used the probabilistic load forecasts generated from our solution for long-term power supply planning.  That project was documented in my first TSG paper co-authored with Dr. Tao Hong and Jason Wilson, which was titled Long term probabilistic load forecasting and normalization with hourly information. Later on, the method proposed in that TSG paper was further investigated by analyzing the residuals. The findings were summarized in our recently published TSG paper On normality assumption in residual simulation for probabilistic load forecasting.

Sunday, January 3, 2016

Energy Forecasting @2015

Energy Forecasting just went through three full years. This is the time to look back at the statistics in 2015.

Where are the readers?

They are from 134 countries and SARs.

Monday, December 21, 2015

2015 Greetings from IEEE Working Group on Energy Forecasting

The holiday season is coming again. As always, I'm pleased to list the accomplishments made by our working group in 2015:

Wednesday, December 16, 2015

That Magic Ruler May Not Work This Year

It is December now. Those of us who live in the North Hemisphere have officially stepped into the winter season. So far this winter in North Carolina seems to be warmer than the last couple of years. Although I can't predict how severe or warm this winter will be, I think it is a good time to write this blog post about the magic ruler.

Thursday, December 10, 2015

Job Opening at CPS Energy: Data Scientist - Energy Forecasting and Predictive Analytics

Posting a very interesting job for a friend at CPS Energy. If you are interested, please apply HERE.

Data Scientist: Energy Forecasting and Predictive Analytics
CPS Energy - San Antonio, TX
2008433-Market Analytics Program Manager
Business Unit: Corporate Development and Planning
Department: Demand Side Analytics
Dates to apply: 12/4/2015 - 12/28/2015