Deja Vu: CharacterizingWorker Reliability Using Task Consistency

Alex C. Williams, Joslin Goh, Charlie G. Willis, Aaron M. Ellison, James H. Brusuelas, Charles C. Davis, Edith Law

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

11 Scopus citations

Abstract

Consistency is a practical metric that evaluates an instrument's reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker's reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task - such as placement, difficulty, and transformation - affect a workers task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.

Original languageEnglish
Title of host publicationProceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017
EditorsSteven Dow, Adam Tauman
Pages197-205
Number of pages9
ISBN (Electronic)9781577357933
StatePublished - Oct 27 2017
Event5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017 - Quebec City, Canada
Duration: Oct 24 2017Oct 26 2017

Publication series

NameProceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017

Conference

Conference5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017
Country/TerritoryCanada
CityQuebec City
Period10/24/1710/26/17

Bibliographical note

Publisher Copyright:
Copyright © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

Funding

We acknowledge and thank the participants recruited from Amazon Mechanical Turk for participating in our study. We also acknowledge the Egypt Exploration Society for providing access to the dataset of papyri images used in the experiment. This research was funded by an NSERC Discovery grant (RGPIN-2015-04543) and an NSF-DBI grant (EF1208835).

FundersFunder number
Amazon Mechanical Turk
NSF-DBIEF1208835
Natural Sciences and Engineering Research Council of CanadaRGPIN-2015-04543

    ASJC Scopus subject areas

    • Computational Theory and Mathematics
    • Human-Computer Interaction

    Fingerprint

    Dive into the research topics of 'Deja Vu: CharacterizingWorker Reliability Using Task Consistency'. Together they form a unique fingerprint.

    Cite this