A Study of Regression Test Selection in Continuous Integration Environments

Continuous integration (CI) systems perform the automated build, test execution, and delivery of the software. CI can provide fast feedback on software changes, minimizing the time and effort required in each iteration. In the meantime, it is important to ensure that enough testing is performed prior to code submission to avoid breaking builds. Recent approaches have been proposed to improve the cost-effectiveness of regression testing through techniques such as regression test selection (RTS). These approaches target at CI environments because traditional RTS techniques often use code instrumentation or very fine-grained dependency analysis, which may not be able to handle rapid changes. In this paper, we study in-depth the usage of RTS in CI environments for different open-source projects. We analyze 918 open-source projects using CI in GitHub to understand 1) under what conditions RTS is needed, and 2) how to balance the trade-offs between granularity levels to perform cost-effective RTS. The findings of this study can aid practitioners and researchers to develop more advanced RTS techniques for being adapted to CI environments.