Guidelines for Benchmarking Automated Software Traceability Techniques

Yonghee Shin; Jane Huffman Hayes; Jane Cleland-Huang

doi:10.1109/SST.2015.13

Guidelines for Benchmarking Automated Software Traceability Techniques

Yonghee Shin, Jane Huffman Hayes, Jane Cleland-Huang

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

17 Scopus citations

Abstract

To comparatively evaluate automated trace ability solutions, we need to develop standardized benchmarks. However there is currently no consensus on how a benchmark should be constructed and used to evaluate competing techniques. In this paper we discuss recurring problems in evaluating trace ability techniques, identify essential properties that evaluation methods should possess, and provide guidelines for benchmarking software trace ability techniques. We illustrate the properties and guidelines using empirical evaluation of three software trace ability techniques on nine data sets.

Original language	English
Title of host publication	Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015
Pages	61-67
Number of pages	7
ISBN (Electronic)	9780769555935
DOIs	https://doi.org/10.1109/SST.2015.13
State	Published - Aug 5 2015
Event	8th IEEE/ACM International Symposium on Software and Systems Traceability, SST 2015 - Florence, Italy Duration: May 17 2015 → …

Publication series

Name	Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015

Conference

Conference	8th IEEE/ACM International Symposium on Software and Systems Traceability, SST 2015
Country/Territory	Italy
City	Florence
Period	5/17/15 → …

Bibliographical note

Publisher Copyright:
© 2015 IEEE.

Keywords

Traceability
benchmarks
evaluation metrics
measurement

ASJC Scopus subject areas

Software

Access to Document

10.1109/SST.2015.13

Cite this

Shin, Y., Hayes, J. H., & Cleland-Huang, J. (2015). Guidelines for Benchmarking Automated Software Traceability Techniques. In Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015 (pp. 61-67). Article 7181530 (Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015). https://doi.org/10.1109/SST.2015.13

@inproceedings{b07eebc1525d4f88908c0ddb253d386e,

title = "Guidelines for Benchmarking Automated Software Traceability Techniques",

abstract = "To comparatively evaluate automated trace ability solutions, we need to develop standardized benchmarks. However there is currently no consensus on how a benchmark should be constructed and used to evaluate competing techniques. In this paper we discuss recurring problems in evaluating trace ability techniques, identify essential properties that evaluation methods should possess, and provide guidelines for benchmarking software trace ability techniques. We illustrate the properties and guidelines using empirical evaluation of three software trace ability techniques on nine data sets.",

keywords = "Traceability, benchmarks, evaluation metrics, measurement",

author = "Yonghee Shin and Hayes, {Jane Huffman} and Jane Cleland-Huang",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; 8th IEEE/ACM International Symposium on Software and Systems Traceability, SST 2015 ; Conference date: 17-05-2015",

year = "2015",

month = aug,

day = "5",

doi = "10.1109/SST.2015.13",

language = "English",

series = "Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015",

pages = "61--67",

booktitle = "Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015",

}

Shin, Y, Hayes, JH & Cleland-Huang, J 2015, Guidelines for Benchmarking Automated Software Traceability Techniques. in Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015., 7181530, Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015, pp. 61-67, 8th IEEE/ACM International Symposium on Software and Systems Traceability, SST 2015, Florence, Italy, 5/17/15. https://doi.org/10.1109/SST.2015.13

Guidelines for Benchmarking Automated Software Traceability Techniques. / Shin, Yonghee; Hayes, Jane Huffman; Cleland-Huang, Jane.
Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015. 2015. p. 61-67 7181530 (Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Guidelines for Benchmarking Automated Software Traceability Techniques

AU - Shin, Yonghee

AU - Hayes, Jane Huffman

AU - Cleland-Huang, Jane

PY - 2015/8/5

Y1 - 2015/8/5

N2 - To comparatively evaluate automated trace ability solutions, we need to develop standardized benchmarks. However there is currently no consensus on how a benchmark should be constructed and used to evaluate competing techniques. In this paper we discuss recurring problems in evaluating trace ability techniques, identify essential properties that evaluation methods should possess, and provide guidelines for benchmarking software trace ability techniques. We illustrate the properties and guidelines using empirical evaluation of three software trace ability techniques on nine data sets.

AB - To comparatively evaluate automated trace ability solutions, we need to develop standardized benchmarks. However there is currently no consensus on how a benchmark should be constructed and used to evaluate competing techniques. In this paper we discuss recurring problems in evaluating trace ability techniques, identify essential properties that evaluation methods should possess, and provide guidelines for benchmarking software trace ability techniques. We illustrate the properties and guidelines using empirical evaluation of three software trace ability techniques on nine data sets.

KW - Traceability

KW - benchmarks

KW - evaluation metrics

KW - measurement

UR - http://www.scopus.com/inward/record.url?scp=84955497934&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84955497934&partnerID=8YFLogxK

U2 - 10.1109/SST.2015.13

DO - 10.1109/SST.2015.13

M3 - Conference contribution

AN - SCOPUS:84955497934

T3 - Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015

SP - 61

EP - 67

BT - Proceedings - 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, SST 2015

T2 - 8th IEEE/ACM International Symposium on Software and Systems Traceability, SST 2015

Y2 - 17 May 2015

ER -

Guidelines for Benchmarking Automated Software Traceability Techniques

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this