TY - GEN
T1 - Expertise screening in crowdsourcing image quality
AU - Hosu, Vlad
AU - Lin, Hanhe
AU - Saupe, Dietmar
N1 - Funding Information:
ACKNOWLEDGMENT We thank the German Research Foundation (DFG) for financial support within project A05 of SFB/Transregio 161.
Publisher Copyright:
© 2018 IEEE.
PY - 2018/9/13
Y1 - 2018/9/13
N2 - We propose a screening approach to find reliable and effectively expert crowd workers in image quality assessment (IQA). Our method measures the users' ability to identify image degradations by using test questions, together with several relaxed reliability checks. We conduct multiple experiments, obtaining reproducible results with a high agreement between the expertise-screened crowd and the freelance experts of 0.95 Spearman rank order correlation (SROCC), with one restriction on the image type. Our contributions include a reliability screening method for uninformative users, a new type of test questions that rely on our proposed database1of pristine and artificially distorted images, a group agreement extrapolation method and an analysis of the crowdsourcing experiments.
AB - We propose a screening approach to find reliable and effectively expert crowd workers in image quality assessment (IQA). Our method measures the users' ability to identify image degradations by using test questions, together with several relaxed reliability checks. We conduct multiple experiments, obtaining reproducible results with a high agreement between the expertise-screened crowd and the freelance experts of 0.95 Spearman rank order correlation (SROCC), with one restriction on the image type. Our contributions include a reliability screening method for uninformative users, a new type of test questions that rely on our proposed database1of pristine and artificially distorted images, a group agreement extrapolation method and an analysis of the crowdsourcing experiments.
KW - crowdsourcing
KW - expertise screening
KW - image quality assessment
KW - reliability screening
UR - http://www.scopus.com/inward/record.url?scp=85054368685&partnerID=8YFLogxK
U2 - 10.1109/QoMEX.2018.8463427
DO - 10.1109/QoMEX.2018.8463427
M3 - Conference contribution
AN - SCOPUS:85054368685
SN - 978-1-5386-2606-1
T3 - 2018 10th International Conference on Quality of Multimedia Experience, QoMEX 2018
BT - 2018 10th International Conference on Quality of Multimedia Experience, QoMEX 2018
PB - IEEE
T2 - 10th International Conference on Quality of Multimedia Experience, QoMEX 2018
Y2 - 29 May 2018 through 1 June 2018
ER -