Research must be reproducible in order to make an impact on science and to contribute to the body of knowledge in our field. Yet studies have shown that 70% of research from academic labs cannot be reproduced. In software engineering, and more specifically requirements engineering (RE), reproducible research is rare, with datasets not always available or methods not fully described. This lack of reproducible research hinders progress, with researchers having to replicate an experiment from scratch. A researcher starting out in RE has to sift through conference papers, finding ones that are empirical, then must look through the data available from the empirical paper (if any) to make a preliminary determination if the paper can be reproduced. This paper addresses two parts of that problem, identifying RE papers and identifying empirical papers within the RE papers. Recent RE and empirical conference papers were used to learn features and to build an automatic classifier to identify RE and empirical papers. We introduce the Empirical Requirements Research Classifier (ERRC) method, which uses natural language processing and machine learning to perform supervised classification of conference papers. We compare our method to a baseline keyword-based approach. To evaluate our approach, we examine sets of papers from the IEEE Requirements Engineering conference and the IEEE International Symposium on Software Testing and Analysis. We found that the ERRC method performed better than the baseline method in all but a few cases.
|Title of host publication||Proceedings of the ACMSE 2018 Conference|
|State||Published - Mar 29 2018|
|Event||2018 Annual ACM Southeast Conference, ACMSE 2018 - Richmond, United States|
Duration: Mar 29 2018 → Mar 31 2018
|Name||Proceedings of the ACMSE 2018 Conference|
|Conference||2018 Annual ACM Southeast Conference, ACMSE 2018|
|Period||3/29/18 → 3/31/18|
Bibliographical noteFunding Information:
This work was supported in part by NSF grant CCF-1511117.
Recent work funded by the National Science Foundation developed a research framework called TraceLab . TraceLab is designed to “provide an experimental environment in which researchers can design and execute experiments .” While TraceLab allows researchers to easily reproduce experiments, it should first be determined if the work in a given research paper even has the possibility of being reproduced. While the ultimate goal of our research is to be able to quickly determine whether an experiment or study in a paper can be reproduced, this paper addresses antecedent questions to support that objective.
© 2018 Association for Computing Machinery.
- Empirical research
- Information retrieval
- Machine learning
- Reproducible research
- Requirements engineering
- Statistical analysis
- Supervised classification learning
- Text classification
ASJC Scopus subject areas
- Computational Theory and Mathematics
- Computer Science Applications
- Hardware and Architecture