Deja Vu: CharacterizingWorker Reliability Using Task Consistency

Alex C. Williams; Joslin Goh; Charlie G. Willis; Aaron M. Ellison; James H. Brusuelas; Charles C. Davis; Edith Law

Deja Vu: CharacterizingWorker Reliability Using Task Consistency

Alex C. Williams, Joslin Goh, Charlie G. Willis, Aaron M. Ellison, James H. Brusuelas, Charles C. Davis, Edith Law

Modern and Classic Languages, Literatures, and Cultures

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

11 Scopus citations

Abstract

Consistency is a practical metric that evaluates an instrument's reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker's reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task - such as placement, difficulty, and transformation - affect a workers task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.

Original language	English
Title of host publication	Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017
Editors	Steven Dow, Adam Tauman
Pages	197-205
Number of pages	9
ISBN (Electronic)	9781577357933
State	Published - Oct 27 2017
Event	5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017 - Quebec City, Canada Duration: Oct 24 2017 → Oct 26 2017

Publication series

Name	Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017

Conference

Conference	5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017
Country/Territory	Canada
City	Quebec City
Period	10/24/17 → 10/26/17

Bibliographical note

Publisher Copyright:
Copyright © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

Funding

We acknowledge and thank the participants recruited from Amazon Mechanical Turk for participating in our study. We also acknowledge the Egypt Exploration Society for providing access to the dataset of papyri images used in the experiment. This research was funded by an NSERC Discovery grant (RGPIN-2015-04543) and an NSF-DBI grant (EF1208835).

Funders	Funder number
Amazon Mechanical Turk
NSF-DBI	EF1208835
Natural Sciences and Engineering Research Council of Canada	RGPIN-2015-04543

ASJC Scopus subject areas

Computational Theory and Mathematics
Human-Computer Interaction

Cite this

Williams, A. C., Goh, J., Willis, C. G., Ellison, A. M., Brusuelas, J. H., Davis, C. C., & Law, E. (2017). Deja Vu: CharacterizingWorker Reliability Using Task Consistency. In S. Dow, & A. Tauman (Eds.), Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017 (pp. 197-205). (Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017).

@inproceedings{f99903ea22414a11a4c0f1415671f302,

title = "Deja Vu: CharacterizingWorker Reliability Using Task Consistency",

abstract = "Consistency is a practical metric that evaluates an instrument's reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker's reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task - such as placement, difficulty, and transformation - affect a workers task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.",

author = "Williams, {Alex C.} and Joslin Goh and Willis, {Charlie G.} and Ellison, {Aaron M.} and Brusuelas, {James H.} and Davis, {Charles C.} and Edith Law",

note = "Publisher Copyright: Copyright {\textcopyright} 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.; 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017 ; Conference date: 24-10-2017 Through 26-10-2017",

year = "2017",

month = oct,

day = "27",

language = "English",

series = "Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017",

pages = "197--205",

editor = "Steven Dow and Adam Tauman",

booktitle = "Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017",

}

Williams, AC, Goh, J, Willis, CG, Ellison, AM, Brusuelas, JH, Davis, CC & Law, E 2017, Deja Vu: CharacterizingWorker Reliability Using Task Consistency. in S Dow & A Tauman (eds), Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017. Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017, pp. 197-205, 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017, Quebec City, Canada, 10/24/17.

Deja Vu: CharacterizingWorker Reliability Using Task Consistency. / Williams, Alex C.; Goh, Joslin; Willis, Charlie G. et al.
Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017. ed. / Steven Dow; Adam Tauman. 2017. p. 197-205 (Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Deja Vu

T2 - 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017

AU - Williams, Alex C.

AU - Goh, Joslin

AU - Willis, Charlie G.

AU - Ellison, Aaron M.

AU - Brusuelas, James H.

AU - Davis, Charles C.

AU - Law, Edith

PY - 2017/10/27

Y1 - 2017/10/27

N2 - Consistency is a practical metric that evaluates an instrument's reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker's reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task - such as placement, difficulty, and transformation - affect a workers task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.

AB - Consistency is a practical metric that evaluates an instrument's reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker's reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task - such as placement, difficulty, and transformation - affect a workers task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.

UR - http://www.scopus.com/inward/record.url?scp=85058120888&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85058120888&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85058120888

T3 - Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017

SP - 197

EP - 205

BT - Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017

A2 - Dow, Steven

A2 - Tauman, Adam

Y2 - 24 October 2017 through 26 October 2017

ER -

Deja Vu: CharacterizingWorker Reliability Using Task Consistency

Abstract

Publication series

Conference

Bibliographical note

Funding

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this