RecSys Challenge 2014

How to evaluate

The final evaluation will be performed by the organizers at the end of the challenge period. Before this period however, participants can calculate their intermediate progress on the provided public test set and upload the results on the leaderboard (see below). The intermediate results can be calculated using the RecSysChallenge Evaluator (Note: always use the most recent release of the evaluator):

java -jar rscevaluator-0.14-jar-with-dependencies.jar test_solution.dat participant_solution.dat

The test_solution.dat file can be found in the dataset, the participant_solution.dat file is the file generated by the participants with the predicted tweet rankings. The exact formatting specifications of this file (e.g., sorting order) can be found in the dataset section.

The nDCG@10 result as returned by the RecSyschallenge evaluator (e.g. 0.924849) may then be submitted on the leaderboard below. Contact the organizers if you should run into problems.

The Final Leaderboard

The final leaderboard

The Weekly Leaderboard

Here are the most up to date evaluation scores as reported by participants. Note that this leaderboard reflects only the intermediate progress, the final evaluation in the end will be performed by the organizers (using the private evaluation dataset) and may differ from this ranking. The winners will be announced during the RecSys conference.

The RecSys Challenge 2014 Leaderboard
RankLast updatedTeam namenDCG@10
110/06/2014 08:25:46TIPS0.9037241811257406
207/30/2014 19:59:29The Crone0.8958346633395474
308/13/2014 11:34:26Inria Sequel0.8759454091564973
408/17/2014 15:46:14BenHuns0.8743489392341603
509/19/2014 17:55:33UCD0.8725760626786041
609/19/2014 19:33:51SUD0.8700848804693523
708/08/2014 05:39:50Just for Fun0.8678869630011482
808/17/2014 19:51:17BDMOBBIES0.8597844419966622
908/04/2014 05:20:37Western0.8365268799686004
1007/24/2014 08:50:59ContentWisers0.8341420829408103
1109/25/2014 21:57:45Pilgrims0.8279531044818939
1208/01/2014 02:44:52UT-IIS0.827272745736101
1309/02/2014 09:17:35QUTrs0.8230342845478499
1407/30/2014 10:24:34Ahmad0.8230342845478497
1508/17/2014 17:10:08bigACL0.8228155591536459
1607/02/2014 00:04:33KaKo0.8218476880587547
1706/02/2014 18:30:45Opera Solutions0.8215184631072615
1810/07/2014 03:50:10PILOTS0.8176172946849103
1909/20/2014 00:28:28IITM0.8167582722829699
2008/10/2014 14:06:28SemWexMFF0.8166811439025655
2106/05/2014 18:52:41Gatv0.8066092597915455
2206/24/2014 15:15:02M.H.0.8051963729692464
2308/14/2014 12:52:23SIEL@IIIT0.8039003944578548
2406/18/2014 09:50:11Elvyra Team0.8020678417013893
2508/15/2014 14:52:51Unimatrix 010.7729808284948092
2610/06/2014 08:26:34Blockbuster0.772919900399079
2706/08/2014 15:43:50Nina and Tony0.7702572920271284
2808/08/2014 07:39:11Punam's Five0.7700033087917992
2906/04/2014 19:12:36Enigma0.7693008294306949
3008/14/2014 14:26:23recco0.763098468483586
3108/11/2014 13:43:00Weon0.763087798072605
3206/16/2014 09:41:29Pytha0.7630169918899049
3306/16/2014 16:03:12SU-FMI0.7621323203181151
3405/27/2014 22:41:59Prism team0.7528657501787783
3506/19/2014 14:20:02DeepInsight0.7514222997963794
3606/08/2014 14:37:21Volador0.7514222997963794