|
INEX 2012 Social Book Search Track
|
|
|
INEX 2012 Social Book Search Track
Social Book Search Track home
Evaluation results for the Social Book Search task
The evaluation results shown below are based on the official INEX 2012 SBS topic set based on the LibraryThing discussion groups and the user profiles and catalogues of the topic creators..
- Books suggested by members other than the topic creator are considered relevant suggestions (rv=1). Books that are catalogued by the topic creator after she created the topic are considered the best suggestions (rv=4).
- Only the first ISBN of a work is counted as a result. Lower-ranked ISBNs of the same work are ignored. If an ISBN at rank 7 maps to the same work as an ISBN at rank 4, we ignore the ISBN at rank 7 and move the ISBN at rank 8 to rank 7.
- The mapping of multiple ISBNs to the same work are based on the mapping from LibraryThing, from the file thingISBN.xml.gz (version of 27 January 2009). We filtered these mappings on the ISBNs in the Amazon/LibraryThing collection (the resulting file is amazon-lt.isbn.thingID.gz).
These are the official Qrels:
- inex12SBS.LT.qrels.official: Qrels derived from the books recommended on the LibraryThing discussion threads of 101 topics.
Each book suggested by a forum member is considered relevant to the requested topic and has relevance value rv=1. Books that are subsequently catalogued by the topic creator in their personal LibraryThing catalogue are considered the best suggestions and have a relevance value rv=4.
The Qrels set uses the LibraryThing work IDs as document IDs.
For evaluation the ISBNs in the submitted runs are mapped to LT work IDs as well, with the highest ranked ISBN being mapped to the work ID and lower ranked ISBNs mapped to the same ID removed from the results list.
Deduplication: mapping ISBNs to LibraryThing work IDs
- Only the first ISBN of a work is counted as a result. Lower-ranked ISBNs of the same work are ignored.
If an ISBN at rank 7 maps to the same work as an ISBN at rank 4, we ignore the ISBN at rank7 and move the ISBN at rank 8 to rank 7.
- The mapping of multiple ISBNs to the same work is based on the mapping from LibraryThing, from the file thingISBN.xml.gz (version of 27 January 2009).
From this file we extracted only the mappings for ISBNs in the Amazon/LibraryThing collection, which are available here:
amazon-lt.isbn.thingID.gz
- A perl script to map ISBNs to IDs in run is also available: deduplicate_simple.pl. This script requires amazon-lt.isbn.thingID.gz for the mappings.
Evaluation using LT-forum suggestions:
The official evaluation measure is nDCG@10.
Run | nDCG@10 | P@10 | MRR | MAP |
p4.xml_social.fb.10.50 | 0.1456 | 0.1376 | 0.3696 | 0.1249 |
p54.run2.all-topic-fields.all-doc-fields.baseline | 0.1452 | 0.1248 | 0.3061 | 0.1059 |
p54.run3.all-topic-fields.QIT.alpha0.99 | 0.1447 | 0.1248 | 0.3057 | 0.1061 |
p62.mrf-booklike | 0.1312 | 0.1376 | 0.3703 | 0.1138 |
p4.xml_social | 0.1302 | 0.1248 | 0.3307 | 0.1173 |
p62.B_IT30_30 | 0.1285 | 0.1267 | 0.3291 | 0.1233 |
p54.run5.title.II.alpha0.94 | 0.1201 | 0.1178 | 0.2760 | 0.0973 |
p62.IOT30 | 0.1168 | 0.1337 | 0.3043 | 0.1077 |
p62.IT30 | 0.1112 | 0.1287 | 0.3106 | 0.1075 |
p54.run6.title.II.alpha0.97 | 0.0971 | 0.0891 | 0.2506 | 0.0730 |
p62.lcm-2 | 0.0878 | 0.0703 | 0.2227 | 0.0690 |
p54.run4.title.QIT.alpha0.65 | 0.0865 | 0.0782 | 0.1859 | 0.0604 |
p100.sb_g0 | 0.0844 | 0.0861 | 0.2347 | 0.0864 |
p100.sb_g_ttl_nar0 | 0.0711 | 0.0624 | 0.1655 | 0.0549 |
p54.run1.title.all-doc-fields.baseline | 0.0691 | 0.0644 | 0.1508 | 0.0486 |
p100.sb_2xsh_ttl_nar0 | 0.0050 | 0.0020 | 0.0149 | 0.0023 |
p100.sb_2xsh0 | 0.0040 | 0.0020 | 0.0189 | 0.0033 |
Precision-recall curves, best system per group:
Precision-recall curves, all systems:
If you have questions, please send an email to marijn.koolen@uva.nl.