INEX 2012 Social Book Search Track
Home |

INEX 2012 Social Book Search Track

Social Book Search Track home

Evaluation results for the Social Book Search task

The evaluation results shown below are based on the official INEX 2012 SBS topic set based on the LibraryThing discussion groups and the user profiles and catalogues of the topic creators..

These are the official Qrels:

  1. inex12SBS.LT.qrels.official: Qrels derived from the books recommended on the LibraryThing discussion threads of 101 topics. Each book suggested by a forum member is considered relevant to the requested topic and has relevance value rv=1. Books that are subsequently catalogued by the topic creator in their personal LibraryThing catalogue are considered the best suggestions and have a relevance value rv=4.
The Qrels set uses the LibraryThing work IDs as document IDs. For evaluation the ISBNs in the submitted runs are mapped to LT work IDs as well, with the highest ranked ISBN being mapped to the work ID and lower ranked ISBNs mapped to the same ID removed from the results list.

Deduplication: mapping ISBNs to LibraryThing work IDs



Evaluation using LT-forum suggestions:

The official evaluation measure is nDCG@10.

RunnDCG@10P@10MRRMAP
p4.xml_social.fb.10.50 0.1456 0.1376 0.3696 0.1249
p54.run2.all-topic-fields.all-doc-fields.baseline 0.1452 0.1248 0.3061 0.1059
p54.run3.all-topic-fields.QIT.alpha0.99 0.1447 0.1248 0.3057 0.1061
p62.mrf-booklike 0.1312 0.1376 0.3703 0.1138
p4.xml_social 0.1302 0.1248 0.3307 0.1173
p62.B_IT30_30 0.1285 0.1267 0.3291 0.1233
p54.run5.title.II.alpha0.94 0.1201 0.1178 0.2760 0.0973
p62.IOT30 0.1168 0.1337 0.3043 0.1077
p62.IT30 0.1112 0.1287 0.3106 0.1075
p54.run6.title.II.alpha0.97 0.0971 0.0891 0.2506 0.0730
p62.lcm-2 0.0878 0.0703 0.2227 0.0690
p54.run4.title.QIT.alpha0.65 0.0865 0.0782 0.1859 0.0604
p100.sb_g0 0.0844 0.0861 0.2347 0.0864
p100.sb_g_ttl_nar0 0.0711 0.0624 0.1655 0.0549
p54.run1.title.all-doc-fields.baseline 0.0691 0.0644 0.1508 0.0486
p100.sb_2xsh_ttl_nar0 0.0050 0.0020 0.0149 0.0023
p100.sb_2xsh0 0.0040 0.0020 0.0189 0.0033

Precision-recall curves, best system per group:

P-R curve

Precision-recall curves, all systems:

P-R curve




If you have questions, please send an email to marijn.koolen@uva.nl.





Imprint | Data protection | Contact someone about INEX