CP-Detector: Using Configuration-related Performance Properties to Expose Performance Bugs

Haochen He; Zhouyang Jia; Shanshan Li; Erci Xu; Tingting Yu; Yue Yu; Ji Wang; Xiangke Liao

doi:10.1145/3324884.3416531

CP-Detector: Using Configuration-related Performance Properties to Expose Performance Bugs

Haochen He, Zhouyang Jia, Shanshan Li, Erci Xu, Tingting Yu, Yue Yu, Ji Wang, Xiangke Liao

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

Performance bugs are often hard to detect due to their non fail-stop symptoms. Existing debugging techniques can only detect performance bugs with known patterns (e.g., inefficient loops). The key reason behind this incapability is the lack of a general test oracle. Here, we argue that the performance (e.g., throughput, latency, execution time) expectation of configuration can serve as a strong oracle candidate for performance bug detection. First, prior work shows that most performance bugs are related to configurations. Second, the configuration change reflects common expectation on performance changes. If the actual performance is contrary to the expectation, the related code snippet is likely to be problematic. In this paper, we first conducted a comprehensive study on 173 real-world configuration-related performance bugs (CPBugs) from 12 representative software systems. We then derived seven configuration-related performance properties, which can serve as the test oracle in performance testing. Guided by the study, we designed and evaluated an automated performance testing framework, CP-DETECTOR, for detecting real-world configuration-related performance bugs. CP-DETECTOR was evaluated on 12 open-source projects. The results showed that it detected 43 out of 61 existing bugs and reported 13 new bugs.

Original language	English
Title of host publication	Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020
Pages	623-634
Number of pages	12
ISBN (Electronic)	9781450367684
DOIs	https://doi.org/10.1145/3324884.3416531
State	Published - Sep 2020
Event	35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020 - Virtual, Melbourne, Australia Duration: Sep 22 2020 → Sep 25 2020

Publication series

Name	Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020

Conference

Conference	35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020
Country/Territory	Australia
City	Virtual, Melbourne
Period	9/22/20 → 9/25/20

Bibliographical note

Funding Information:
problems [50], architectural impacts among metho26d]s, p[erfor-This research was supported by National Key R&D Program of manceanti-pattern6s5[]andtradeoffs21[]inORMapplications. China (Project No.2017YFB1001802); National Natural Science Foun-These works are effective in detecting certain types of performancedation of China (Project No.61872373, 61702534 and 61872375). problems, which are different to the configuration-handling per- formance bugs detected CbyP-Detector. Recent short position REFERENCES papers [37, 53, 54] have proposed a proof concept that using meta-[1] Categoriesandsubcategoriesincomputersciehntctep.s://en.wikipedia.org/wiki/ morphic testing to expose performance bug. While they did not[2] FitCDFfornormaldistributiohnt.tps://en.wikipedia.org/wiki/Cumulative_Category:Computin.g proposed an automatic approach or evaluate on large scale softwaredistribution_funct.ion systems. The difference between our work and metamorphic testing[4]Statisticalhypothesis[3] spaCy.https://spacy.i.o is that they typically use multiple test executions to infer metamor-_hypothesis_testin. g phic relations, and verify those relations on follow-up tests. W[h5]ilIeEEE standard glossary of software engineering terminIoElEoEgSyt.d 610.12 we conclude performance properties from bug study and generate[6]Linuxperftoolh.ttps://perf.wiki.kernel.org/index.php/Main_,P2a0g1e5.(1990). them from expert knowledge (e.g., user manuals). [7] httpd33605,Acc.:2020h. ttps://bz.apache.org/bugzilla/show_bug.cgi?id=3.3605 [8] httpd42065,Acc.:2020h.ttps://bz.apache.org/bugzilla/show_bug.cgi?id=4.2065 HotspotsDetection. Someworksfocusonpinpointinghotspots [9] httpd50002,Acc.:2020h.ttps://bz.apache.org/bugzilla/show_bug.cgi?id=5.0002 in programs via profiling: Per6f],[YourKit3[1]. Similarly, several [11]GCC17520,Accessed:2020.https://gcc.gnu.org/bugzilla/show_bug.cgi?id=1.7520[10] httpd58037,Acc.:2020h. ttps://bz.apache.org/bugzilla/show_bug.cgi?id=5.8037 following works address on generating the most time-consuming[12] GCC94957,Accessed:2020.https://gcc.gnu.org/bugzilla/show_bug.cgi?id=9.4957 workloads via profilers to help expose performance bottle2n8,ecks [[13] MariaDB 5802, Accessed: 2020h. ttps://jira.mariadb.org/browse/MDEV-5.802

Publisher Copyright:
© 2020 ACM.

Keywords

Performance bug detection
Performance property
Software configuration

ASJC Scopus subject areas

Artificial Intelligence
Software
Safety, Risk, Reliability and Quality

Access to Document

10.1145/3324884.3416531

Cite this

He, H., Jia, Z., Li, S., Xu, E., Yu, T., Yu, Y., Wang, J., & Liao, X. (2020). CP-Detector: Using Configuration-related Performance Properties to Expose Performance Bugs. In Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020 (pp. 623-634). [9286052] (Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020). https://doi.org/10.1145/3324884.3416531

He, Haochen ; Jia, Zhouyang ; Li, Shanshan et al. / CP-Detector : Using Configuration-related Performance Properties to Expose Performance Bugs. Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020. 2020. pp. 623-634 (Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020).

@inproceedings{7507346519e444e1a25528b14ff9f6b2,

title = "CP-Detector: Using Configuration-related Performance Properties to Expose Performance Bugs",

abstract = "Performance bugs are often hard to detect due to their non fail-stop symptoms. Existing debugging techniques can only detect performance bugs with known patterns (e.g., inefficient loops). The key reason behind this incapability is the lack of a general test oracle. Here, we argue that the performance (e.g., throughput, latency, execution time) expectation of configuration can serve as a strong oracle candidate for performance bug detection. First, prior work shows that most performance bugs are related to configurations. Second, the configuration change reflects common expectation on performance changes. If the actual performance is contrary to the expectation, the related code snippet is likely to be problematic. In this paper, we first conducted a comprehensive study on 173 real-world configuration-related performance bugs (CPBugs) from 12 representative software systems. We then derived seven configuration-related performance properties, which can serve as the test oracle in performance testing. Guided by the study, we designed and evaluated an automated performance testing framework, CP-DETECTOR, for detecting real-world configuration-related performance bugs. CP-DETECTOR was evaluated on 12 open-source projects. The results showed that it detected 43 out of 61 existing bugs and reported 13 new bugs.",

keywords = "Performance bug detection, Performance property, Software configuration",

author = "Haochen He and Zhouyang Jia and Shanshan Li and Erci Xu and Tingting Yu and Yue Yu and Ji Wang and Xiangke Liao",

note = "Funding Information: problems [50], architectural impacts among metho26d]s, p[erfor-This research was supported by National Key R&D Program of manceanti-pattern6s5[]andtradeoffs21[]inORMapplications. China (Project No.2017YFB1001802); National Natural Science Foun-These works are effective in detecting certain types of performancedation of China (Project No.61872373, 61702534 and 61872375). problems, which are different to the configuration-handling per- formance bugs detected CbyP-Detector. Recent short position REFERENCES papers [37, 53, 54] have proposed a proof concept that using meta-[1] Categoriesandsubcategoriesincomputersciehntctep.s://en.wikipedia.org/wiki/ morphic testing to expose performance bug. While they did not[2] FitCDFfornormaldistributiohnt.tps://en.wikipedia.org/wiki/Cumulative_Category:Computin.g proposed an automatic approach or evaluate on large scale softwaredistribution_funct.ion systems. The difference between our work and metamorphic testing[4]Statisticalhypothesis[3] spaCy.https://spacy.i.o is that they typically use multiple test executions to infer metamor-_hypothesis_testin. g phic relations, and verify those relations on follow-up tests. W[h5]ilIeEEE standard glossary of software engineering terminIoElEoEgSyt.d 610.12 we conclude performance properties from bug study and generate[6]Linuxperftoolh.ttps://perf.wiki.kernel.org/index.php/Main_,P2a0g1e5.(1990). them from expert knowledge (e.g., user manuals). [7] httpd33605,Acc.:2020h. ttps://bz.apache.org/bugzilla/show_bug.cgi?id=3.3605 [8] httpd42065,Acc.:2020h.ttps://bz.apache.org/bugzilla/show_bug.cgi?id=4.2065 HotspotsDetection. Someworksfocusonpinpointinghotspots [9] httpd50002,Acc.:2020h.ttps://bz.apache.org/bugzilla/show_bug.cgi?id=5.0002 in programs via profiling: Per6f],[YourKit3[1]. Similarly, several [11]GCC17520,Accessed:2020.https://gcc.gnu.org/bugzilla/show_bug.cgi?id=1.7520[10] httpd58037,Acc.:2020h. ttps://bz.apache.org/bugzilla/show_bug.cgi?id=5.8037 following works address on generating the most time-consuming[12] GCC94957,Accessed:2020.https://gcc.gnu.org/bugzilla/show_bug.cgi?id=9.4957 workloads via profilers to help expose performance bottle2n8,ecks [[13] MariaDB 5802, Accessed: 2020h. ttps://jira.mariadb.org/browse/MDEV-5.802 Publisher Copyright: {\textcopyright} 2020 ACM.; null ; Conference date: 22-09-2020 Through 25-09-2020",

year = "2020",

month = sep,

doi = "10.1145/3324884.3416531",

language = "English",

series = "Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020",

pages = "623--634",

booktitle = "Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020",

}

He, H, Jia, Z, Li, S, Xu, E, Yu, T, Yu, Y, Wang, J & Liao, X 2020, CP-Detector: Using Configuration-related Performance Properties to Expose Performance Bugs. in Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020., 9286052, Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, pp. 623-634, 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, Virtual, Melbourne, Australia, 9/22/20. https://doi.org/10.1145/3324884.3416531

CP-Detector : Using Configuration-related Performance Properties to Expose Performance Bugs. / He, Haochen; Jia, Zhouyang; Li, Shanshan et al.

Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020. 2020. p. 623-634 9286052 (Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - CP-Detector

AU - He, Haochen

AU - Jia, Zhouyang

AU - Li, Shanshan

AU - Xu, Erci

AU - Yu, Tingting

AU - Yu, Yue

AU - Wang, Ji

AU - Liao, Xiangke

N1 - Funding Information: problems [50], architectural impacts among metho26d]s, p[erfor-This research was supported by National Key R&D Program of manceanti-pattern6s5[]andtradeoffs21[]inORMapplications. China (Project No.2017YFB1001802); National Natural Science Foun-These works are effective in detecting certain types of performancedation of China (Project No.61872373, 61702534 and 61872375). problems, which are different to the configuration-handling per- formance bugs detected CbyP-Detector. Recent short position REFERENCES papers [37, 53, 54] have proposed a proof concept that using meta-[1] Categoriesandsubcategoriesincomputersciehntctep.s://en.wikipedia.org/wiki/ morphic testing to expose performance bug. While they did not[2] FitCDFfornormaldistributiohnt.tps://en.wikipedia.org/wiki/Cumulative_Category:Computin.g proposed an automatic approach or evaluate on large scale softwaredistribution_funct.ion systems. The difference between our work and metamorphic testing[4]Statisticalhypothesis[3] spaCy.https://spacy.i.o is that they typically use multiple test executions to infer metamor-_hypothesis_testin. g phic relations, and verify those relations on follow-up tests. W[h5]ilIeEEE standard glossary of software engineering terminIoElEoEgSyt.d 610.12 we conclude performance properties from bug study and generate[6]Linuxperftoolh.ttps://perf.wiki.kernel.org/index.php/Main_,P2a0g1e5.(1990). them from expert knowledge (e.g., user manuals). [7] httpd33605,Acc.:2020h. ttps://bz.apache.org/bugzilla/show_bug.cgi?id=3.3605 [8] httpd42065,Acc.:2020h.ttps://bz.apache.org/bugzilla/show_bug.cgi?id=4.2065 HotspotsDetection. Someworksfocusonpinpointinghotspots [9] httpd50002,Acc.:2020h.ttps://bz.apache.org/bugzilla/show_bug.cgi?id=5.0002 in programs via profiling: Per6f],[YourKit3[1]. Similarly, several [11]GCC17520,Accessed:2020.https://gcc.gnu.org/bugzilla/show_bug.cgi?id=1.7520[10] httpd58037,Acc.:2020h. ttps://bz.apache.org/bugzilla/show_bug.cgi?id=5.8037 following works address on generating the most time-consuming[12] GCC94957,Accessed:2020.https://gcc.gnu.org/bugzilla/show_bug.cgi?id=9.4957 workloads via profilers to help expose performance bottle2n8,ecks [[13] MariaDB 5802, Accessed: 2020h. ttps://jira.mariadb.org/browse/MDEV-5.802 Publisher Copyright: © 2020 ACM.

PY - 2020/9

Y1 - 2020/9

N2 - Performance bugs are often hard to detect due to their non fail-stop symptoms. Existing debugging techniques can only detect performance bugs with known patterns (e.g., inefficient loops). The key reason behind this incapability is the lack of a general test oracle. Here, we argue that the performance (e.g., throughput, latency, execution time) expectation of configuration can serve as a strong oracle candidate for performance bug detection. First, prior work shows that most performance bugs are related to configurations. Second, the configuration change reflects common expectation on performance changes. If the actual performance is contrary to the expectation, the related code snippet is likely to be problematic. In this paper, we first conducted a comprehensive study on 173 real-world configuration-related performance bugs (CPBugs) from 12 representative software systems. We then derived seven configuration-related performance properties, which can serve as the test oracle in performance testing. Guided by the study, we designed and evaluated an automated performance testing framework, CP-DETECTOR, for detecting real-world configuration-related performance bugs. CP-DETECTOR was evaluated on 12 open-source projects. The results showed that it detected 43 out of 61 existing bugs and reported 13 new bugs.

AB - Performance bugs are often hard to detect due to their non fail-stop symptoms. Existing debugging techniques can only detect performance bugs with known patterns (e.g., inefficient loops). The key reason behind this incapability is the lack of a general test oracle. Here, we argue that the performance (e.g., throughput, latency, execution time) expectation of configuration can serve as a strong oracle candidate for performance bug detection. First, prior work shows that most performance bugs are related to configurations. Second, the configuration change reflects common expectation on performance changes. If the actual performance is contrary to the expectation, the related code snippet is likely to be problematic. In this paper, we first conducted a comprehensive study on 173 real-world configuration-related performance bugs (CPBugs) from 12 representative software systems. We then derived seven configuration-related performance properties, which can serve as the test oracle in performance testing. Guided by the study, we designed and evaluated an automated performance testing framework, CP-DETECTOR, for detecting real-world configuration-related performance bugs. CP-DETECTOR was evaluated on 12 open-source projects. The results showed that it detected 43 out of 61 existing bugs and reported 13 new bugs.

KW - Performance bug detection

KW - Performance property

KW - Software configuration

UR - http://www.scopus.com/inward/record.url?scp=85099189983&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85099189983&partnerID=8YFLogxK

U2 - 10.1145/3324884.3416531

DO - 10.1145/3324884.3416531

M3 - Conference contribution

AN - SCOPUS:85099189983

T3 - Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020

SP - 623

EP - 634

BT - Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020

Y2 - 22 September 2020 through 25 September 2020

ER -

He H, Jia Z, Li S, Xu E, Yu T, Yu Y et al. CP-Detector: Using Configuration-related Performance Properties to Expose Performance Bugs. In Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020. 2020. p. 623-634. 9286052. (Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020). doi: 10.1145/3324884.3416531

CP-Detector: Using Configuration-related Performance Properties to Expose Performance Bugs

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this