Wednesday, April 22, 2020

Energy Forecasting Jobs During COVID-19

This blog has been viewed hundreds of thousands of times by tens of thousands of energy forecasters worldwide have been visiting this blog. Several years ago, I used to post jobs on weekly basis. Then I stopped doing that to avoid primary contents being buried by job posts. 

I changed my mind because of COVID-19.

Many companies have hiring freeze, while many are still searching for people to fill in the vacancies. 

Many students and job seekers are desperate and panic in the job market. 

As a forecaster, I have zero interest forecasting when this chaos would end. As an energy forecaster, maybe I can help other energy forecasters and energy companies during COVID-19.

This is what I'm going to do for the next few months...

To hiring companies/managers:
  • Send me an email with the link to your job description, as well as job location, restrictions about visa status / green card / citizenship, and other important details you think applicants should know. 
  • Leave your contact information if you want. 
  • If you haven't followed me on LinkedIn, do that now. 
I'll post the links on the JOBS page of this blog. Sometimes, I will also select some jobs to broadcast through my LinkedIn profiles, which are followed by thousands of energy forecasters. 

To job seekers:
  • Check this blog frequently. 
  • Apply to the jobs you are qualified for and interested in. 
  • Leave your name under my LinkedIn post.
  • If you haven't followed me on LinkedIn, do that now. 
If you think you are a superstar in the job market, you can send me more details about your experience. I may broadcast your profile through my LinkedIn profile as well. 

Hope we all work together to fight through this unprecedented period!

Tuesday, April 7, 2020

Deeksha Dharmapal - Well-rounded

Today (4/7/2020), Deeksha Dharmapal defended her master thesis on Gross Domestic Product for short-term load forecasting.

Deeksha received her B.S. in Civil Engineering from Bangalore Institute of Technology, India, in 2012. Prior to joining UNC Charlotte in Fall 2018, she had several years of industry experience in India, with Amazon and EMC2 (now DellEMC). During her graduate study at our program, she also completed a summer internship at Bosch Rexroth, Charlotte.

During the past few years, I gave a departmental seminar to new graduate students every year. Deeksha is definitely the one that left the best impression on me. Most students at the seminar stayed quiet, while Deeksha was one of the few asking questions. Her questions sounded genuine and intelligent. Nevertheless, I didn't think she would eventually join my group, because she appeared to have a bright future on the managerial track.

Among the students in our program, the self-motivated ones start looking for their faculty mentor rather early. Deeksha took an unique approach. She showed up at one of my MS student, Shreyashi Shukla's thesis defense. Later she signed up and completed my forecasting course, which was a surprise to me.

When she told me that she wanted to join my group, I gave her the same task, passing two SAS programmer certification exams. She completed the base one, but failed the advanced one in her first attempt. The outcome didn't surprise me, because I knew her programming background was weak at that time. Most students at this point would just give up and look for other professors. She didn't. Finally she got the SAS Advanced Programmer Certification. Since she didn't pass it within the time limit I assigned, I gave her an extra task, which she completed in time. I brought her to BigDEAL as my MS thesis student in the summer of 2019.

The thesis topic I gave her was non-trivial. Economic indicators are typically used for long term load forecasting but not short term load forecasting. I asked her to investigate what are the situations that we should consider economy, GDP to be specific, in short term load forecasting. This research involves a lot of programming skills as well as knowledge in statistics.  She picked up those things along the way, and had the thesis beautifully done.

As a forecaster, I love to investigate the things that I failed to predict. I was wondering how a little girl Deeksha surprised me multiple times by pushing herself out of her comfort zone and fighting such a tough uphill battle. Through some casual conversations, I learned that she was a student athlete. She ran track for the most part of her student life - short distance sprints and relay. She was on the basketball and volleyball teams. Post marriage, she has been playing competitive badminton.  I guess the sports experience must have built her a strong heart!

Congratulations, Deeksha!


Monday, April 6, 2020

Congratulations, Dr. Masoud Sobhani!

On March 12, 2020, Masoud Sobhani defended his doctoral dissertation on Delivery Point Load Forecasting. It was right before the coronavirus lockdown. Today, all his forms have gone through the approval chain.

Masoud joined UNC Charlotte's Master Program in Engineering Management in Spring 2016. He completed his M.S. degree in Engineering Management in December 2017 under my supervision. After that, he continued pursuing his PhD in Infrastructure and Environmental Systems.

Masoud is a great teacher. He is by far the only student in my lab being nominated for the Outstanding Graduate Teaching Assistant Award at the university level. As the graduate program director, I have to assign graduate teaching assistants to all graduate and undergraduate classes my department offers. Sometimes several colleagues ask me for Masoud to be their TA at the same time!  I had the fortune of having him help with four of my courses: forecasting, energy analytics, computational intelligence, and case studies in the energy industry. Sometimes I asked him to give lectures. During these coronavirus lockdown days, all of us professors have to move the course online. For some lectures, I can find no better resources other than recordings of Masoud's from last year!

Masoud is gifted for his leadership. During his tenure in my research group, he led two BigDEAL teams to win the NPower competitions. As a graduate student of UNC Charlotte, he is a leader of Iranian student body. As the teaching assistant, he led a team of students putting the course project into a high quality journal paper, which is the first and only class paper ever published from the courses I taught at UNC Charlotte. 

Upon graduation, Masoud published two journal articles. He also won the ISF travel grant to present his research at ISF 2018 in Boulder, CO. Masoud's research has advanced the state-of-the-art by several years. This is largely due to his two internships at NCEMC during the summers of 2018 and 2019. His research outcome has already been put in production environment by the cooperatives. Unfortunately, he experienced the dark side of today's peer review system, just like my experience 10 years ago. The core contribution of his dissertation was underappreciated by inexperienced reviewers.

Masoud passed his PhD Qualifying Exam in Fall 2018, completed his proposal defense in Spring 2019.  He was ready and going to defend his dissertation last semester, to keep up with my record of two-year PhD. Right after he came back from the summer internship, Duke Energy immediately took him before any other companies did,  which delayed the graduation a bit. Still, with a defense data in March 2020, he completed his PhD in 2.5 years, which is remarkably fast!

Now Masoud and his beautiful wife live in a luxury apartment in the city center, minutes away from Duke Energy. I wish I had that life style!

Again, congratulations, Dr. Masoud Sobhani!

Tuesday, February 18, 2020

Yike Li - Eager to Learn

Last Friday (2/14/2020), Yike Li defended his MS thesis, Optimal Weather Station Selection for Electric Load Forecasting.

Yike Li's MS thesis defense
From left to right: Dr. Tao Hong, Yike Li, Dr. Pu Wang and Dr. Linquan Bai

Yike received his B.S. degree in Applied Physics from Tianjin University, China, in 2010, and his M.S. degree in Electrical Engineering from North Carolina State University in 2012. He joined our MSEM program in Fall 2018. Meanwhile, he also enrolled in our INES PhD program. From 2012 to 2019, he had a progressive career in the power industry. His was promoted to a consulting manager at Accenture last year, and then decided to come back to school to pursue his PhD.

I got to know Yike since his days at NC State University. I was giving lectures on load forecasting and demand response, when he was one of the students in the class. At that time, he was definitely the student showing most interest in the subject. He was eager to learn, and asking me many questions about the software, models, and applications. Since then, we have been keeping in touch. Occasionally, he sent me greeting messages and updates about his progress in the industry. 

Couple years ago, Yike asked me about pursuing a PhD degree under my supervision. Since he didn't have thesis writing experience, I asked him to complete a master thesis first. Although I've known him for years, I still had him going through the BigDEAL interview process including the screening tests. He passed them without a surprise. His thesis topic is a continuation of my IJF paper on weather station selection. The task was to propose a method beating the one in my IJF paper. It was not an easy task, but he nailed it. He was able to complete the thesis research while working full time. 

Now he can focus on his dissertation research!

Congratulations, Yike!

Wednesday, January 8, 2020

Computational Intelligence

Last semester at UNC Charlotte, I taught a new graduate-level course Computational Intelligence.

Student Profile

The course started with 15 students on 8/21/2019, and ended with 10 students:

  • INES PhD students: 6 => 6;
  • ECE PhD student: 1 => 1;
  • Applied Energy student: 1 => 1;
  • MSEM on campus student: 6 => 1; 
  • MSEM remote student: 1 => 1.

The picture was taken at the last lecture with the on campus students, the Teaching Assistant, and myself.

Computational Intelligence Class 2019
From left to right: Tao Hong; Richard Alaimo; Vinayak Sharma; Sepehr Sabeti; Shreyashi Shukla; Deeksha DharmapalBhav Sardana; Zehan Xu; Masoud Sobhani (TA); Yike Li. Students not on the picture: Allison Campbell and Nima Nader

Topics

I developed this course to help the students better grasp the fundamentals in this hype of AI/ML. The following topics were covered in Fall 2019.

  • Mathematical programming and statistical forecasting
  • Fuzzy set, fuzzy logic, fuzzy regression, and fuzzy clustering
  • Support vector machine and support vector regression
  • Neural networks, neural fuzzy systems, recurrent neural networks, and deep learning
  • Metaheuristic search algorithms, A*, simulated annealing, and tabu search
  • Artificial immune systems
  • Genetic algorithms
  • Swarm intelligence, ant colony optimization, and particle swarm optimization
  • Bayesian network
  • Designing your tools

Assignments and Exams

The course has 4 homework assignments, a 3-phase course project, a mid-term exam and a final exam.

Traditionally, when this course is offered by Industrial & Systems Engineering faculty, the applications are various optimization problems. When I took Soft Computing during my PhD days at NC State University, we were mostly solving nonlinear optimization problems as homework, project and exam problems.

However, there are not many situations in daily life for us to find a global optimal solution of a sophisticated function made of several trigonometric functions. Instead of penetrating the homework and exam problems with unrealistic mathematical equations, I had the students work on realistic problems for most part of the semester.

For instance, the first homework was to predict my son Leo's jump rope performance. Leo is a very competitive jumper. In the national jump rope competition last year in Florida, he ranked top 5 among 10 and under kids for speed jump. But before the competition, I had to decide whether to have him participate or not. Students were asked to make that decision given his training records. I also taught some Texas Hold'em strategies when teaching Bayesian network after a light coverage of the traditional rain/sprinkler example. Some students used Texas Hold'em as their final project topic.

The final exam was jointly held with my collaborator Robertas Gabrys. The exam problem was on variable selection for forecasting, which I believe is a much more commonly seen problem than those traditional non-linear optimization problems.

Teaching Methods

For my other PhD level courses, I have minimized traditional lectures and maximized the homework. The classroom becomes a discussion forum, where the students learn from each other and myself by sharing their homework experience. The more efforts students put into the homework, the more they learn on their own and from each other. It was super effective, as many students improved their forecasting skills rapidly in a semester.

This course is different. There is a lot of theoretical contents to cover. My goal is not to have them be an operator of a black box. I want them to understand the details and fundamentals of those algorithms. Therefore, I was teaching them to hand-calculate parameters for a neural network, hand-calculate parameters for support vector regression, and so forth. I wanted to break down those fancy concepts, so that they can eventually build their own CI tools from scratch.

I greatly appreciate the students for their time being the first batch of this class and all the efforts they devoted to the course. This course is currently scheduled for every other year, so the next offering is Fall 2021. If you have any ideas or comments that can help me improve this course, please let me know!

Tuesday, January 7, 2020

Forecasting with High Frequency Data: M4 Competition and Beyond

M4 competition was a huge success. The International Journal of Forecasting just published a full issue covering all aspects of the competition. I was honored to be invited by the guest editors to write a commentary paper, which focused on the hourly series of the competition.

According to the organizers (see HERE), the M5 competition is coming soon!

Citation

Tao Hong, "Forecasting with high frequency data: M4 competition and beyond," International Journal of Forecasting, vol.36, no.1, pp.191-194, January, 2020. (ScienceDirect)

Forecasting with High Frequency Data: M4 Competition and Beyond

Tao Hong

Abstract

The M4 competition included 100,000 time series, with the frequencies ranging from yearly to hourly. The team rankings differ notably across frequencies for both point and probabilistic forecasting. I discuss the performances of these methods, with an emphasis on the hourly series of the M4 competition. I also discuss forecasting with high-frequency data in general.

Thursday, October 31, 2019

Descriptive Analytics Based Anomaly Detection for Cybersecure Load Forecasting

Data quality has been a big challenge in load forecasting practice, but an underestimated issue in the academic literature. This work was supported by the U.S. Department of Energy through the Cybersecurity for Energy Delivery Systems Program. We were trying to detect anomalies so that accurate load forecasts can be produced even when the data is contaminated.

Citation

Meng Yue, Tao Hong, and Jianhui Wang, "Descriptive analytics based anomaly detection for cybersecure load forecasting," IEEE Transactions on Smart Grid, vol. 10, no. 6, pp. 5964-5974, November, 2019

Descriptive Analytics Based Anomaly Detection for Cybersecure Load Forecasting

Meng Yue, Tao Hong, and Jianhui Wang

Abstract

As power delivery systems evolve and become increasingly reliant on accurate forecasts, they will be more and more vulnerable to cybersecurity issues. A coordinated data attack by sophisticated adversaries can render existing data corrupt or outlier detection methods ineffective. This would have a very negative impact on operational decisions. The focus of this paper is to develop descriptive analytics-based methods for anomaly detection to protect the load forecasting process against cyberattacks to essential data. We propose an integrated solution (IS) and a hybrid implementation of IS (HIIS) that can detect and mitigate cyberattack induced long sequence anomalies. HIIS is also capable of improving true positive rates and reducing false positive rates significantly comparing with IS. The proposed HIIS can serve as an online cybersecure load forecasting scheme.

Thursday, August 8, 2019

IEEE Working Group on Energy Forecasting in Good Hands

At IEEE Power & Energy Society General Meeting 2010 (GM'10), I proposed a future panel session on Practical Aspects of Electric Load Forecasting. The session was approved by the Power Systems Planning and Implementation (PSPI) Committee.

The following year at GM'11, we hosted the session with 6 speakers and a fully packed room of audience. The discussion was quite engaging. We shared ideas, experience, concerns and visions of the future for this field. At the same conference, Shu Fan and I proposed to establish the IEEE Working Group on Energy Forecasting to tackle a long list of challenges in the field. The proposal was approved at the PSPI committee meeting. I became the Chair, and Shu became the Vice Chair. 

After GM'11, we started working with a few other key players on several initiatives, such as GEFCom2012, and a TSG special session on forecasting. We brought in Pierre Pinson to both initiatives. Both turned out to be unbelievably successful. GEFCom2012 was a game-changing event that produced many valuable assets for the energy forecasting community and started the . Our TSG special section collected many high quality and highly cited papers.

At GM'12, I met Hamidreza Zareipour for the first time. Within seconds, we both sensed each other's strong passion in forecasting. Then Hamid became the Secretary of our working group. 

After GM'12, Shu, Hamid, Pierre, and I started another initiative: a tutorial on energy forecasting. We taught the tutorial for the next four consecutive years, from 2013 to 2016. 

During the last 9 years, we have completed virtually every task we promised to do back in 2011, and much more: the series of three Global Energy Forecasting Competitions, two special sections for IEEE Transactions on Smart Grid and one special issue for Power & Energy Magazine, a 25-page literature review, a full-day tutorial, and more than a dozen panel sessions at PES General Meetings.

Since GEFCom2017, I have been thinking about replicating the success beyond the power sector. Fortunately, the International Institute of Forecasters offered a great platform to reach out to the fields of gas, water, environment, and climate science. Long story short, I founded SWEET, IIF Section on Water, Energy, and EnvironmenT. We just had our first meeting at the International Symposium on Forecasting 2019, with 43 talks in 13 sessions. Our next meeting will be at ISF2020 in Rio, Brazil, July 5-8, 2020.

Running SWEET requires a lot of thoughts and efforts. To focus on this new challenge, I decided to take off my hat as the Chair of the Energy Forecasting Working Group. I believed the group also needed a new leader for its next chapter. I found no better successor than Hamid to take over the Chair position. Hamid is a full professor of electrical and computer engineering at the University of Calgary. He started as the Secretary of the group in 2012. After Shu Fan left academia to pursue his career in trading, he stepped up as the Vice Chair. Hamid has been a crucial contributor to the success of the group since its infancy. I'm sure that he will bring the group up to the next level.

Hamid and I searched for a new secretary and found a rising star Yi Wang, a postdoc researcher at ETH Zurich. Yi received his PhD from Tsinghua University. He has published more than 30 journal papers, of which most are on load forecasting and smart meter data analytics. In the past, he has helped us review many energy forecasting manuscripts for top scholarly journals. This year at GM'19, Yi is chairing a panel session on probabilistic energy forecasting. Yi has already brought up several innovative ideas to grow the group. As the Past Chair, I'll support Hamid and Yi to ensure a smooth transition.

IEEE Working Group on Energy Forecasting is in good hands!

Wednesday, August 7, 2019

Call for Proposals: IIF-SAS Grant to Promote Research on Forecasting

The International Institute of Forecasters is calling for proposals on how to improve forecasting methods and business forecasting practice. This is the 16th year of the financial support from SAS on this IIF-SAS award. In addition to the $10,000 funding from SAS, the IIF is adding another $10,000 to the award pool this year, so that two $10,000 grants are going to be awarded to the best proposals in methodology and practice/management categories.

The applications are due on September 30, 2019. The application must include: 
  • Description of the project (at most 4 pages) 
  • C.V./resume (brief, 4 page max) 
  • Budget and work-plan for the project (brief, 1 page max) 
Criteria for the award of the grant will include likely impact on forecasting methods and business applications. 

Details about the award can be found from the IIF website. For the frequent readers of this blog and SWEET members, I'm listing the energy related projects that were awarded in the last decade:
  • Robust kernel-free nonlinear support vector regression models for load forecasting. Jian Luo, Dongbei University of Finance & Economics, China. (2018-2019 grant, methodology category)
  • Hierarchy-based disaggregate forecasting using deep machine learning in power system time series. Cong Feng and Jie Zhang, The University of Texas at Dallas, USA. (2017-2018 grant, business applications category)
  • Convolutional neural networks for spatio-temporal wind speed forecasting. Fernando Cyrino and Bruno Q. Bastos, Pontifical Catholic University of Rio de Janeiro, Brazil. (2017-2018 grant, methodology category)
  • Short-term load forecasting using rule-based seasonal exponential smoothing incorporating special day effects. Siddharth Arora and James Taylor, University of Oxford, UK. (2010-2011 grant, business applications category)
Best luck!

Thursday, June 20, 2019

Energy Forecasting in the Big Data World

All papers for the International Journal of Forecasting special section on energy forecasting in the big data world have been published online. Out of 14 papers collected for this special section, eight are from GEFCom2017 documenting winning methods, while the other six non-GEFCom2017 papers cover diverse topics in the areas of energy supply, demand and price forecasting.

The guest editorial is HERE. Below is the list of special section papers:
  1. Tao Hong, Jingrui Xie, and Jonathan Black. Global Energy Forecasting Competition 2017: Hierarchical probabilistic load forecasting
  2. Florian Ziel. Quantile regression for the qualifying match of GEFCom2017 probabilistic load forecasting.
  3. I. Dimoulkas, P. Mazidi, and L. Herre. Neural networks for GEFCom2017 probabilistic load forecasting.
  4. Slawek Smyl and N. Grace Hua. Machine learning methods for GEFCom2017 probabilistic load forecasting.
  5. Andrew J. Landgraf. An ensemble approach to GEFCom2017 probabilistic load forecasting.
  6. Cameron Roach. Reconciled boosted models for GEFCom2017 hierarchical probabilistic load forecasting.
  7. Julian de Hoog and Khalid Abdulla. Data visualization and forecast combination for probabilistic load forecasting in GEFCom2017 final match.
  8. Isao Kanda and J.M. Quintana Veguillas. Data preprocessing and quantile regression for probabilistic load forecasting in the GEFCom2017 final match.
  9. Stephen Haben, Georgios Giasemidis, Florian Ziel, and Siddharth Arora. Short term load forecasting and the effect of temperature at the low voltage level.
  10. Jakob W. Messner and Pierre Pinson. Online adaptive lasso estimation in vector autoregressive models for high dimensional wind power forecasting
  11. Dazhi Yang, Elynn Wu, Jan Kleissl. Operational solar forecasting for the real-time market.
  12. Grzegorz Marcjasz, Bartosz Uniejewski, and Rafał Weron. On the importance of the long-term seasonal component in day-ahead electricity price forecasting with NARX neural networks.
  13. Bartosz Uniejewski, Grzegorz Marcjasz, and Rafał Weron. Understanding intraday electricity markets: Variable selection and very short-term price forecasting using LASSO.
  14. Xuerong Li, Wei Shang, and Shouyang Wang. Text-based crude oil price forecasting: A deep learning approach.

Citation 

Tao Hong and Pierre Pinson, "Energy forecasting in the big data world," International Journal of Forecasting, vol. 35, no. 4, pp. 1387-1388, 2019. 

Energy Forecasting in the Big Data World

Tao Hong and Pierre Pinson

Modern information and communication technologies have brought big data to virtually every segment of the energy and utility industries. While forecasting is an important and necessary step in the data-driven decision-making process, the problem of generating better forecasts in the world of big data is an emerging issue and a challenge to both industry and academia. This special section aims to collect top-quality forecasting articles that document cutting-edge research findings and best practices on a wide range of important business problems in the energy industry. Our emphasis is on big data, such as forecasting with high resolution data, the use of high-dimensional processes, forecasting in real-time, and the use of non-traditional data and variables.