Saturday, October 29, 2016

Instructions for GEFCom2017 Qualifying Match

The GEFCom2017 Qualifying Match means to attract and educate a large number of contestants with diverse background, and to prepare them for the final match. It includes two tracks: a defined-data track (GEFCom2017-D) and an open-data track (GEFCom2017-O). In both tracks, the contestants are asked to forecast the same thing: zonal and total loads of ISO New England. The only difference between the two tracks is on the input data.

Data 

The input data a participating team can use GEFCom2017-D should not go beyond the following:
  1. Columns A, B, D, M and N in the worksheets of "YYYY SMD Hourly Data" files, where YYYY represents the year. These data files can be downloaded from ISO New England website via the zonal information page of the energy, load and demand reports. Contestants outside United States may need a VPN to access the data. 
  2. US Federal Holidays as published via US Office of Personnel Management.
The contestants are assumed to have the general knowledge of Daylight Savings Time and inferring the day of week and month of year based on a date.

There is no limitation for the input data in GEFCom2017-O.

Forecasts

The forecasts should be in the form of 9 quantiles following the exact format provided in the template file. The quantiles are the 10th, 20th, ... 90th percentiles. The forecasts should be generated for 10 zones, including the 8 ISO New England zones, the Massachusetts (sum of three zones under Massachusetts), and the total (sum of the first 8 zones).

Timeline

GEFCom2017 Qualifying Match includes six rounds.

Round 1 due date: Dec 15, 2016; forecast period: Jan 1-31, 2017.
Round 2 due date: Dec 31, 2016; forecast period: Feb 1-28, 2017.
Round 3 due date: Jan 15, 2017; forecast period: Feb 1-28, 2017.
Round 4 due date: Jan 31, 2017; forecast period: Mar 1-31, 2017.
Round 5 due date: Feb 14, 2017; forecast period: Mar 1-31, 2017.
Round 6 due date: Feb 28, 2017; forecast period: Apr 1-30, 2017.
Report and code due date: Mar 10, 2017.

The deadline for each round is 11:59pm EST of the corresponding due date.

Submission

The submissions will be through email. Within two weeks of registration, the team leader should receive a confirmation email with the track name and team name in the email subject line. If the team registered both tracks, the team leader should receive two separate emails, one for each track.

The team lead should submit the forecast on behalf of the team by replying to the confirmation email.

The submission must be received before the deadline (based on the receipt time of the email system) to be counted in the leaderboard.

Template

The submissions should strictly follow the requirements below:
  1. The file format should be *.xls;
  2. The file name should be "TrackInitialRoundNumber-TeamName". For instance, Team "An Awesome Win" in the defined data track's round 3 should name the file as "D3-An Awesome Win".
  3. The file should include 10 worksheets, named as CT, ME, NEMASSBOST, NH, RI, SEMASS, VT, WCMASS, MASS, TOTAL. Please arrange the worksheets in the same order as listed above. 
  4. In each worksheet, the first two columns should be date and hour, respectively, in chronological order.
  5. The 3rdto the 11th columns should be Q10, Q20, ... to Q90. 
The template is HERE. The contestants should replace the date column to reflect the forecast period in each round.

Evaluation

In round i, for a forecast submitted by team j for zone k, the average Pinball Loss of the 9 quantiles will be used as the quantile score of the probabilistic forecast Sijk. A benchmark method will be used to forecast each of the 10 zones. We denote the quantile score of the benchmark method in round i for zone k as Bik.

In round i, we will calculate the relative improvement (1 - Sijk/Bik) for each zone. The average improvement over all zones team j accomplishes will be the rating for team j, denoted as Rij. The rank of team j in round i is RANKij.

The weighted average of the rankings from all 6 rounds will be used to rank the teams in the qualifying match leaderboard. The first 5 rounds will be weighted equally, while the weight for the 6th round is doubled.

A team completing four or more rounds is eligible to for the prizes. The ratings for the missing rounds will be imputed before calculating the weighted average of the ratings.

Prizes

Institute Prize (up to 3 universities): $1000
1st place in each track: $2000
2nd place in each track: $1000
3rd place in each track: $500
1st place in each round of each track: $200

For more information about GEFCom2017, please visit www.gefcom.org.

12 comments:

  1. Dear Tao, Thanks for organizing! Just to make sure: In Round 2 and 3 and in round 4 and 5 the same time periods have to be forecasted?

    ReplyDelete
  2. Just out of curiosity. What would be the big difference between rounds 2 and 3 or 4 and 5? We would have 15 days additional actuals or a lag of around 15 days instead of 30 days for the first day to forecast, but this should't have a large impact. Is there anything else I am overlooking?

    ReplyDelete
    Replies
    1. ISO NE typically publishes the load of the previous month in the first half of the current month, so you will have one more month of actual load and weather data. How much impact this additional data has is unknown. Secondly, this gives additional weight to Feb and Mar, as we want to give more weight to the later rounds. Finally, this gives some flexibility to the participants - if you don't have time to work on round 3, you may use the forecasts from around 2.

      Delete
  3. Tao, the holidays published on opm.gov start in the year 2011. I presume we are allowed to look up the same holidays from previous years from other sources for the D-track?

    ReplyDelete
  4. Specifically for the defined-data track (GEFCom2017-D), we are allowed to use historical temperature measurements (columns M and N) for training models. However, these features will not exist for the forecast horizon of our submittals. Will temperature forecasts for the forecast horizon be provided that all teams must use, or should we assume no further data will be provided for the test set? Thanks.

    ReplyDelete
    Replies
    1. No further data (i.e., weather in the forecasting period) will be provided for the defined data track. This is an ex ante forecasting problem.

      Delete
  5. Dear Tao,

    I have a few questions/comments to the GEFCom2017 rules:

    1) Concerning the data: We use the DEMAND column. There are some values exactly zero due to the clock-change (e.g. in the 2015 file the value for 2015-03-08 02:00:00 (entry 1587-D)). Do you expect us to put a zero in the corresponding March forecasts as well? How is this treated in the forecasting evaluation? (Do you ignore this value for computing the scores, or do you treat the error as 0?)

    2) For the evaluation of the forecasts, you decided that the final ranking depends on a benchmark method. Obviously, the performance of the benchmark (e.g. if it is better during night than during day time, or better in large zones than in smaller ones, ...) will influence the final leaderboard. Therefore, please make the used benchmark methodology completely public in advance of the competition to guarantee full transparency - or even better you publish the code (and datasets) that you will use to compute the benchmark.

    3) We have strict rules for public holiday information. Is it allowed to implement bridging effects, especially concerning the days before and after a public holiday? I interpret the rules as if it is allowed, but I want to be sure.

    Best,
    Florian

    ReplyDelete
    Replies
    1. Hi Tao,

      I believe question 1 has not been addressed. Do we need to predict hour 2 on the start of DST?

      Thanks,
      Andrew

      Delete
    2. You need to predict 23 hours of the first DST day.

      Delete
  6. Hi, I just wanted to clarify something about the forecasts we have to produce:

    "The forecasts should be generated for 10 zones, including the 8 ISO New England zones, the Massachusetts (sum of three zones under Massachusetts), and the total (sum of the first 8 zones)."

    For the TOTAL zone I have noticed that there is a very small difference between the ISO-NE control area values, and the sum of the 8 load zones.

    The differences are minimal (I've seen up to 3MWh either way), but I was wondering which will be used for evaluation purposes?

    ReplyDelete

Note that you may link to your LinkedIn profile if you choose Name/URL option.