How to evaluate

The final evaluation will be performed by the organizers at the end of the challenge period. Before this period however, participants can calculate their intermediate progress on the provided public test set and upload the results on the leaderboard (see below). The intermediate results can be calculated using the RecSysChallenge Evaluator (Note: always use the most recent release of the evaluator):

java -jar rscevaluator-0.14-jar-with-dependencies.jar test_solution.dat participant_solution.dat

The test_solution.dat file can be found in the dataset, the participant_solution.dat file is the file generated by the participants with the predicted tweet rankings. The exact formatting specifications of this file (e.g., sorting order) can be found in the dataset section.

The nDCG@10 result as returned by the RecSyschallenge evaluator (e.g. 0.924849) may then be submitted on the leaderboard below. Contact the organizers if you should run into problems.

The Final Leaderboard

The Weekly Leaderboard

Here are the most up to date evaluation scores as reported by participants. Note that this leaderboard reflects only the intermediate progress, the final evaluation in the end will be performed by the organizers (using the private evaluation dataset) and may differ from this ranking. The winners will be announced during the RecSys conference.