INEX 2014 Social Book Search Track

Operationalisation of forum judgement labels

Students from the Aalborg University Copenhagen, Royal School of Library and Information Science (Copenhagen) and the Oslo and Akershus University College have labelled the LibraryThing forum topic threads and the suggestions in those threads.

• The topic label were used to select topics for the 2014 SBS task. In total, 680 topics were selected.
• The suggestions labels, in combination with the user profiles, were used to determine the relevance value of each book suggestion in the thread.

Forum members can mention books for many different reasons. We want the relevance values to distinguish between books that were mentioned as positive recommendations, negative recommendations (books to avoid), neutral suggestions (mentioned as possibly relevant but not necessarily recommended) and books mentioned for some other reason (not relevant at all).

Furthermore, we want to differentiate between recommendations from members who have read the book they recommend and members who haven't. We assume the recommendation to be of more value to the searcher if it comes from someone who has actually read the book.

Finally, we distinguish between suggestions of books that the user already had in their catalogue versus books that the user added after getting a suggestion from others.

Terminology

• works added by topic creator before they were suggested on the forum are *Pre-catalogued*
• works added by topic creator after they were suggested on the forum are *Post-catalogued*
• first mention of work is *suggestion*
• subsequent mentions of work are *replies*
• has_read means suggester has read the book
• not_read means suggester hasn't read the book
• not_sure means labeller couldn't tell

Simplifying assumptions

• if creator has book catalogued before it's suggested: treated as known -> the creator already knows about this book so suggesting it has little value.
• if creator adds book to catalogue after it's suggested: treated as highly relevant -> this is signal that the creator acted upon the suggestion, so the suggestion has value to the creator.
• not sure if suggester has read book: treated as not read -> for creator its not clear to value not_sure recommendation more than not_read recommendation.
• not sure if suggester is positive/negative/neutral: treated as neutral -> for creator there is no signal this book is better/worse than neutral suggestions.
• has read overrules not read (opinion of someone who read the book is more valuable than of someone who didn't)
• pos + neg == neu (positive and negative cancel each other out).
• topic creator has read is rv=0 (if creator has already read the book, the suggestion has no value for search).
• Topic creator's judgements overrule others (we mainly care about topic creator's opinion.)
• Topic creator mentioning single work multiple times, last mention is used as judgement (assumption is that creator based last judgement on the thread discussion or having bought/read the book in the meantime).

Decision tree: determine which judgements to use

1 - Work mentioned once -> there is only one judgement, use that
2 - Work mentioned multiple times
  2.1 - topic creator mentions work
    2.1.1 - topic creator *suggests* neutral -> use replies (go to 2.2)
    2.1.2 - topic creator *suggests* pos/neg -> use creator judgement
    2.1.3 - topic creator *replies* -> use creator judgement only
  2.2 - topic creator doesn't mention work
    2.2.1 - there are some has_read suggestions/replies -> use has_read judgements
    2.2.2 - there are no has_read suggestions/replies -> use all judgements

Decision tree: turn judgements into relevance values

1 - catalogued by topic creator
  1.1 - post-catalogued -> rv=8
  1.2 - pre-catalogued -> rv=0
2 - single judgement
  2.1 - creator has_read judgement
    2.1.1 - creator pos/neg/neu -> rv=0
  2.2 - creator not_read judgement
    2.2.1 - creator positive -> rv= 8
    2.2.2 - creator neutral -> rv=2
    2.2.3 - creator negative -> rv=0
  2.3 - other has_read judgement
    2.3.1 - has_read positive -> rv= 4
    2.3.2 - has_read neutral -> rv=2
    2.3.3 - has_read negative -> rv=0
  2.4 - other not_read judgement
    2.4.1 - not_read positive -> rv= 3
    2.4.2 - not_read neutral -> rv=2
    2.4.3 - not_read negative -> rv=0
3 - multiple judgements
  3.1 - multi has_read judgements
    3.1.1 - some positive, no negative -> rv=6
    3.1.2 - #positive > #negative -> rv=4
    3.1.3 - #positive == #negative -> rv=2
    3.1.4 - all neutral -> rv=2
    3.1.5 - #positive < #negative -> rv=1
    3.1.6 - no positive, some negative -> rv=0
  3.2 - multi not_read judgements
    3.2.1 - some positive, no negative -> rv=4
    3.2.2 - #positive > #negative -> rv=3
    3.2.3 - #positive == #negative -> rv=2
    3.2.4 - all neutral -> rv=2
    3.2.5 - #positive < #negative -> rv=1
    3.2.6 - no positive, some negative -> rv=0