Saturday, September 20, 2014

Inside Leaderboard

Update 11/22/2014:
Include a column showing the teams ineligible for the final leaderboard due to missing more than three submissions beating benchmark. 
The first evaluation week (Task 4) of GEFCom2014 just went by. CrowdAnalytix publishes the leaderboard based on the best score of each team in real time. Since some teams made multiple submissions, that leaderboard won't reflect the real positions. To enhance the transparency of our scoring process, I manually pulled the submission log of each track to come up with a more realistic leaderboard. Please understand that:
  • This leaderboard may not represent the final scores of each task nor the final position of each team. 
  • The positions and scores are subject to change based on the review of the final reports and codes.
  • The competition executive/award committee will be using this leaderboard as the starting point to determine the winning teams. 
  • The temporary position listed on my leaderboard is based on the first valid entry submitted by each team. That's the most important entry we are looking for. (Again, this is a forecasting competition, not a data mining one where you can try multiple entries to fit the solution data.) 
  • I also listed the scores of all submissions from all teams together with the timestamps. 
  • In case I made an error in my manual scoring process, please kindly point it out. 
  • I will try to update this leaderboard every two or three weeks, if not more often. I do have a full-time job aside from organizing GEFCom2014, so please be patient...
One more tip - I noticed that many teams submitted the entry in the last day of the task. Please consider submitting it sooner than later. In case you made an error in submission and got a failure message, you still have a chance in the coming day(s) before the task ends.

BTW, my leaderboard is HERE.

No comments:

Post a Comment

Note that you may link to your LinkedIn profile if you choose Name/URL option.