Towards reproducible research: Automatic classification of empirical requirements engineering papers

Clinton Woodson, Jane Huffman Hayes, Sarah Griffioen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations


Research must be reproducible in order to make an impact on science and to contribute to the body of knowledge in our field. Yet studies have shown that 70% of research from academic labs cannot be reproduced. In software engineering, and more specifically requirements engineering (RE), reproducible research is rare, with datasets not always available or methods not fully described. This lack of reproducible research hinders progress, with researchers having to replicate an experiment from scratch. A researcher starting out in RE has to sift through conference papers, finding ones that are empirical, then must look through the data available from the empirical paper (if any) to make a preliminary determination if the paper can be reproduced. This paper addresses two parts of that problem, identifying RE papers and identifying empirical papers within the RE papers. Recent RE and empirical conference papers were used to learn features and to build an automatic classifier to identify RE and empirical papers. We introduce the Empirical Requirements Research Classifier (ERRC) method, which uses natural language processing and machine learning to perform supervised classification of conference papers. We compare our method to a baseline keyword-based approach. To evaluate our approach, we examine sets of papers from the IEEE Requirements Engineering conference and the IEEE International Symposium on Software Testing and Analysis. We found that the ERRC method performed better than the baseline method in all but a few cases.

Original languageEnglish
Title of host publicationProceedings of the ACMSE 2018 Conference
ISBN (Electronic)9781450356961
StatePublished - Mar 29 2018
Event2018 Annual ACM Southeast Conference, ACMSE 2018 - Richmond, United States
Duration: Mar 29 2018Mar 31 2018

Publication series

NameProceedings of the ACMSE 2018 Conference


Conference2018 Annual ACM Southeast Conference, ACMSE 2018
Country/TerritoryUnited States

Bibliographical note

Publisher Copyright:
© 2018 Association for Computing Machinery.


  • Empirical research
  • Information retrieval
  • Machine learning
  • Reproducible research
  • Requirements engineering
  • Statistical analysis
  • Supervised classification learning
  • Text classification

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Hardware and Architecture
  • Software


Dive into the research topics of 'Towards reproducible research: Automatic classification of empirical requirements engineering papers'. Together they form a unique fingerprint.

Cite this