Consistency is a practical metric that evaluates an instrument's reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker's reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task - such as placement, difficulty, and transformation - affect a workers task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.
|Title of host publication||Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017|
|Editors||Steven Dow, Adam Tauman|
|Number of pages||9|
|State||Published - Oct 27 2017|
|Event||5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017 - Quebec City, Canada|
Duration: Oct 24 2017 → Oct 26 2017
|Name||Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017|
|Conference||5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017|
|Period||10/24/17 → 10/26/17|
Bibliographical noteFunding Information:
We acknowledge and thank the participants recruited from Amazon Mechanical Turk for participating in our study. We also acknowledge the Egypt Exploration Society for providing access to the dataset of papyri images used in the experiment. This research was funded by an NSERC Discovery grant (RGPIN-2015-04543) and an NSF-DBI grant (EF1208835).
Copyright © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
ASJC Scopus subject areas
- Computational Theory and Mathematics
- Human-Computer Interaction