TY - GEN
T1 - On topic difficulty in IR evaluation
T2 - 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019
AU - Zampieri, Fabio
AU - Roitero, Kevin
AU - Shane Culpepper, J.
AU - Kurland, Oren
AU - Mizzaro, Stefano
N1 - Publisher Copyright:
© 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM.
PY - 2019/7/18
Y1 - 2019/7/18
N2 - In a test collection setting, topic difficulty can be defined as the average effectiveness of a set of systems for a topic. In this paper we study the effects on the topic difficulty of: (i) the set of retrieval systems; (ii) the underlying document corpus; and (iii) the system components. By generalizing methods recently proposed to study system component factor analysis, we perform a comprehensive analysis on topic difficulty and the relative effects of systems, corpora, and component interactions. Ourfi ndings show that corpora have the most significant effect on topic difficulty.
AB - In a test collection setting, topic difficulty can be defined as the average effectiveness of a set of systems for a topic. In this paper we study the effects on the topic difficulty of: (i) the set of retrieval systems; (ii) the underlying document corpus; and (iii) the system components. By generalizing methods recently proposed to study system component factor analysis, we perform a comprehensive analysis on topic difficulty and the relative effects of systems, corpora, and component interactions. Ourfi ndings show that corpora have the most significant effect on topic difficulty.
UR - http://www.scopus.com/inward/record.url?scp=85072724638&partnerID=8YFLogxK
U2 - 10.1145/3331184.3331279
DO - 10.1145/3331184.3331279
M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???
AN - SCOPUS:85072724638
T3 - SIGIR 2019 - Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval
SP - 909
EP - 912
BT - SIGIR 2019 - Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval
Y2 - 21 July 2019 through 25 July 2019
ER -