GEFCom2014 had a super strong start. Within the first 4 weeks, we have had 225, 138, 126 and 135 solvers in the load, price, wind and solar tracks respectively. Totally 345 people have joined the LinkedIn group Global Energy Forecasting Competition. While many winning teams of GEFCom2012 came back to GEFCom2014 with very strong performance, several new faces also topped some of the tasks. Having been monitoring the competition from the back end, I am so excited about the ups and downs on the leaderboards. I really wish I could join the game in person.
The evaluation period of GEFCom2014 is starting in less than 2 days. I'd like to offer some recommendations, so that the contestants fully understand what makes a winning solution. Some of them may be overlapping with my previous post, GEFCom2014 is ON - 8 Tips before You Join the Game, but I think it's important to cover them again here.
1. One submission per task
We understand that human beings can make errors sometimes. In this competition, you may want to submit a file but mistakenly select another one. To accommodate human errors, we set the platform to accept up to 1 entry per day, namely 7 entries per week. The platform automatically picks up the lowest score from your entries as the score for the task. In reality, the future may not always pick your best forecast. To mimic the real-world forecasting operations, we expect you to submit only one entry per week. We will do an audit at the end. Instead of using the lowest score for each task, we will put a heavy weight on the earliest "valid" submission for each task. If you submitted multiple entries per week, it's up to us to decide which one is "valid". I will also release the detailed submission summaries for the top teams to keep the process transparent.
2. Participate in every task
This competition will last for another 12 weeks, which is not a short period. At the end, a team has to beat the benchmark for at least 9 times to be eligible for a position in the final leaderboard. It's very likely that some team(s) ranking higher than you would not be able to complete this marathon. Regardless where you rank right now, don't give up.
On the other hand, don't try to skip a week if you believe the score for that week would damage your average score. Let's say at the end, the top two teams completed 10 and 11 tasks respectively, with 9 tasks in common. We will give the preference to the team that wins on the common tasks and completed more tasks.
3. No external information
Again, do not try to guess where the data is from and take advantage of that information. Regardless how high your ranking is, you would be disqualified for breaking this rule.
4. Keep a weekly log
Documentation is an important skill of a good forecaster, because s/he has to keep a traceable log for the previous models and forecasts. In this competition, the report and code is due at the end as Task 16. It would be an easy task if you have kept a log every week to document what you have done and why. In GEFCom2012, several top contestants failed to submit the report, so they lost the chance to be recognized as a winner.
5. Registration, registration and registration
Don't forget to register. The deadline is 10/10. There are three registrations you have to go through to be eligible for being a winner: CrowdAnalytix, LinkedIn and GoogleForm. To submit an entry, you have to be a solver of CrowdAnalytix, so that's been taken care of. Don't forget to register on GoogleForm and join the LinkedIn group Global Energy Forecasting Competition. If you are part of a non-profit institute, and there are multiple teams from your site, please ask your academic advisor to register the institute as a candidate for the institute prize.
6. Help others
You don't have to disclose your secret sauce, but it doesn't hurt to give others a hand in the LinkedIn group. One goal of this competition is to enhance the forecasting practices in the energy industry. We don't have to make sure other people's forecasts are bad. Instead, all of us will win if we together contribute to some magic solution for solving the energy forecasting problem.
Happy forecasting and cheers!
The evaluation period of GEFCom2014 is starting in less than 2 days. I'd like to offer some recommendations, so that the contestants fully understand what makes a winning solution. Some of them may be overlapping with my previous post, GEFCom2014 is ON - 8 Tips before You Join the Game, but I think it's important to cover them again here.
1. One submission per task
We understand that human beings can make errors sometimes. In this competition, you may want to submit a file but mistakenly select another one. To accommodate human errors, we set the platform to accept up to 1 entry per day, namely 7 entries per week. The platform automatically picks up the lowest score from your entries as the score for the task. In reality, the future may not always pick your best forecast. To mimic the real-world forecasting operations, we expect you to submit only one entry per week. We will do an audit at the end. Instead of using the lowest score for each task, we will put a heavy weight on the earliest "valid" submission for each task. If you submitted multiple entries per week, it's up to us to decide which one is "valid". I will also release the detailed submission summaries for the top teams to keep the process transparent.
2. Participate in every task
This competition will last for another 12 weeks, which is not a short period. At the end, a team has to beat the benchmark for at least 9 times to be eligible for a position in the final leaderboard. It's very likely that some team(s) ranking higher than you would not be able to complete this marathon. Regardless where you rank right now, don't give up.
On the other hand, don't try to skip a week if you believe the score for that week would damage your average score. Let's say at the end, the top two teams completed 10 and 11 tasks respectively, with 9 tasks in common. We will give the preference to the team that wins on the common tasks and completed more tasks.
3. No external information
Again, do not try to guess where the data is from and take advantage of that information. Regardless how high your ranking is, you would be disqualified for breaking this rule.
4. Keep a weekly log
Documentation is an important skill of a good forecaster, because s/he has to keep a traceable log for the previous models and forecasts. In this competition, the report and code is due at the end as Task 16. It would be an easy task if you have kept a log every week to document what you have done and why. In GEFCom2012, several top contestants failed to submit the report, so they lost the chance to be recognized as a winner.
5. Registration, registration and registration
Don't forget to register. The deadline is 10/10. There are three registrations you have to go through to be eligible for being a winner: CrowdAnalytix, LinkedIn and GoogleForm. To submit an entry, you have to be a solver of CrowdAnalytix, so that's been taken care of. Don't forget to register on GoogleForm and join the LinkedIn group Global Energy Forecasting Competition. If you are part of a non-profit institute, and there are multiple teams from your site, please ask your academic advisor to register the institute as a candidate for the institute prize.
6. Help others
You don't have to disclose your secret sauce, but it doesn't hurt to give others a hand in the LinkedIn group. One goal of this competition is to enhance the forecasting practices in the energy industry. We don't have to make sure other people's forecasts are bad. Instead, all of us will win if we together contribute to some magic solution for solving the energy forecasting problem.
Happy forecasting and cheers!
No comments:
Post a Comment
Note that you may link to your LinkedIn profile if you choose Name/URL option.