Multi-user input in determining answer sets (MIDAS)

Albert Kalim; Jane Huffman Hayes; Satrio Husodo; Erin Combs; Jared Payne

doi:10.1109/RE.2018.00070

Multi-user input in determining answer sets (MIDAS)

Albert Kalim, Jane Huffman Hayes, Satrio Husodo, Erin Combs, Jared Payne

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

Empirical validation is an important component of sound requirements engineering research. Many researchers develop a gold standard or answer set against which to compare techniques that they also developed in order to calculate common measures such as recall and precision. This poses threats to validity as the researchers developed the gold standard and the technique to be measured against it. To help address this and to help reduce bias, we introduce a prototype of Multi-user Input in Determining Answer Sets (MIDAS), a web-based tool to permit communities of researchers to jointly determine the gold standard for a given research data set. To date, the tool permits community members to add items to the answer set, vote on items in the answer set, comment on items, and view the latest status of community opinion on answer set items. It currently supports traceability data sets and classification data sets.

Original language	English
Title of host publication	Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018
Editors	Daniel Amyot, Walid Maalej, Guenther Ruhe
Pages	496-497
Number of pages	2
ISBN (Electronic)	9781538674185
DOIs	https://doi.org/10.1109/RE.2018.00070
State	Published - Oct 12 2018
Event	26th IEEE International Requirements Engineering Conference, RE 2018 - Banff, Canada Duration: Aug 20 2018 → Aug 24 2018

Publication series

Name	Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018

Conference

Conference	26th IEEE International Requirements Engineering Conference, RE 2018
Country/Territory	Canada
City	Banff
Period	8/20/18 → 8/24/18

Bibliographical note

Publisher Copyright:
© 2018 IEEE.

Keywords

Answer set
Data set
Evaluation
Gold standard

ASJC Scopus subject areas

Engineering (miscellaneous)
Safety, Risk, Reliability and Quality

Access to Document

10.1109/RE.2018.00070

Cite this

Kalim, A., Hayes, J. H., Husodo, S., Combs, E., & Payne, J. (2018). Multi-user input in determining answer sets (MIDAS). In D. Amyot, W. Maalej, & G. Ruhe (Eds.), Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018 (pp. 496-497). Article 8491177 (Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018). https://doi.org/10.1109/RE.2018.00070

@inproceedings{85d4063c8b124ef28c2fd0a5094d4a46,

title = "Multi-user input in determining answer sets (MIDAS)",

abstract = "Empirical validation is an important component of sound requirements engineering research. Many researchers develop a gold standard or answer set against which to compare techniques that they also developed in order to calculate common measures such as recall and precision. This poses threats to validity as the researchers developed the gold standard and the technique to be measured against it. To help address this and to help reduce bias, we introduce a prototype of Multi-user Input in Determining Answer Sets (MIDAS), a web-based tool to permit communities of researchers to jointly determine the gold standard for a given research data set. To date, the tool permits community members to add items to the answer set, vote on items in the answer set, comment on items, and view the latest status of community opinion on answer set items. It currently supports traceability data sets and classification data sets.",

keywords = "Answer set, Data set, Evaluation, Gold standard",

author = "Albert Kalim and Hayes, {Jane Huffman} and Satrio Husodo and Erin Combs and Jared Payne",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 26th IEEE International Requirements Engineering Conference, RE 2018 ; Conference date: 20-08-2018 Through 24-08-2018",

year = "2018",

month = oct,

day = "12",

doi = "10.1109/RE.2018.00070",

language = "English",

series = "Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018",

pages = "496--497",

editor = "Daniel Amyot and Walid Maalej and Guenther Ruhe",

booktitle = "Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018",

}

Kalim, A, Hayes, JH, Husodo, S, Combs, E & Payne, J 2018, Multi-user input in determining answer sets (MIDAS). in D Amyot, W Maalej & G Ruhe (eds), Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018., 8491177, Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018, pp. 496-497, 26th IEEE International Requirements Engineering Conference, RE 2018, Banff, Canada, 8/20/18. https://doi.org/10.1109/RE.2018.00070

Multi-user input in determining answer sets (MIDAS). / Kalim, Albert; Hayes, Jane Huffman; Husodo, Satrio et al.
Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018. ed. / Daniel Amyot; Walid Maalej; Guenther Ruhe. 2018. p. 496-497 8491177 (Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Multi-user input in determining answer sets (MIDAS)

AU - Kalim, Albert

AU - Hayes, Jane Huffman

AU - Husodo, Satrio

AU - Combs, Erin

AU - Payne, Jared

PY - 2018/10/12

Y1 - 2018/10/12

N2 - Empirical validation is an important component of sound requirements engineering research. Many researchers develop a gold standard or answer set against which to compare techniques that they also developed in order to calculate common measures such as recall and precision. This poses threats to validity as the researchers developed the gold standard and the technique to be measured against it. To help address this and to help reduce bias, we introduce a prototype of Multi-user Input in Determining Answer Sets (MIDAS), a web-based tool to permit communities of researchers to jointly determine the gold standard for a given research data set. To date, the tool permits community members to add items to the answer set, vote on items in the answer set, comment on items, and view the latest status of community opinion on answer set items. It currently supports traceability data sets and classification data sets.

AB - Empirical validation is an important component of sound requirements engineering research. Many researchers develop a gold standard or answer set against which to compare techniques that they also developed in order to calculate common measures such as recall and precision. This poses threats to validity as the researchers developed the gold standard and the technique to be measured against it. To help address this and to help reduce bias, we introduce a prototype of Multi-user Input in Determining Answer Sets (MIDAS), a web-based tool to permit communities of researchers to jointly determine the gold standard for a given research data set. To date, the tool permits community members to add items to the answer set, vote on items in the answer set, comment on items, and view the latest status of community opinion on answer set items. It currently supports traceability data sets and classification data sets.

KW - Answer set

KW - Data set

KW - Evaluation

KW - Gold standard

UR - http://www.scopus.com/inward/record.url?scp=85056855224&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85056855224&partnerID=8YFLogxK

U2 - 10.1109/RE.2018.00070

DO - 10.1109/RE.2018.00070

M3 - Conference contribution

AN - SCOPUS:85056855224

T3 - Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018

SP - 496

EP - 497

BT - Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018

A2 - Amyot, Daniel

A2 - Maalej, Walid

A2 - Ruhe, Guenther

T2 - 26th IEEE International Requirements Engineering Conference, RE 2018

Y2 - 20 August 2018 through 24 August 2018

ER -

Kalim A, Hayes JH, Husodo S, Combs E, Payne J. Multi-user input in determining answer sets (MIDAS). In Amyot D, Maalej W, Ruhe G, editors, Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018. 2018. p. 496-497. 8491177. (Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018). doi: 10.1109/RE.2018.00070

Multi-user input in determining answer sets (MIDAS)

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this