Abstract
Because of the ever-increasing data being produced by today's high performance computing (HPC) scientific simulations, I/O performance is becoming a significant bottleneck for their executions. An efficient error-controlled lossy compressor is a promising solution to significantly reduce data writing time for scientific simulations running on supercomputers. In this paper, we explore how to optimize the data dumping performance for scientific simulation by leveraging error-bounded lossy compression techniques. The contributions of the paper are threefold. (1) We propose a novel I/O performance profiling model that can effectively represent the I/O performance with different execution scales and data sizes, and optimize the estimation accuracy of data dumping performance using least square method. (2) We develop an adaptive lossy compression framework that can select the bestfit compressor (between two leading lossy compressors SZ and ZFP) with optimized parameter settings with respect to overall data dumping performance. (3) We evaluate our adaptive lossy compression framework with up to 32k cores on a supercomputer facilitated with fast I/O systems and using real-world scientific simulation datasets. Experiments show that our solution can mostly always lead the data dumping performance to the optimal level with very accurate selection of the bestfit lossy compressor and settings. The data dumping performance can be improved by up to 27% at different scales.
| Original language | English |
|---|---|
| Title of host publication | Proceedings - 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019 |
| ISBN (Electronic) | 9781728147345 |
| DOIs | |
| State | Published - Sep 2019 |
| Event | 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019 - Albuquerque, United States Duration: Sep 23 2019 → Sep 26 2019 |
Publication series
| Name | Proceedings - IEEE International Conference on Cluster Computing, ICCC |
|---|---|
| Volume | 2019-September |
| ISSN (Print) | 1552-5244 |
Conference
| Conference | 2019 IEEE International Conference on Cluster Computing, CLUSTER 2019 |
|---|---|
| Country/Territory | United States |
| City | Albuquerque |
| Period | 9/23/19 → 9/26/19 |
Bibliographical note
Publisher Copyright:© 2019 IEEE.
Funding
ACKNOWLEDGMENTS This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations - the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nation’s exascale computing imperative. The material was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357, and supported by the National Science Foundation under Grant No. 1619253. This work was also supported by National Science Foundation CCF 1513201. We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory.
| Funders | Funder number |
|---|---|
| National Science Foundation CCF 1513201 | CCF 1513201 |
| National Science Foundation Arctic Social Science Program | 1619253 |
| National Science Foundation Arctic Social Science Program | |
| U.S. Department of Energy EPSCoR | DE-AC02-06CH11357 |
| U.S. Department of Energy EPSCoR | |
| Office of Science Programs | |
| National Nuclear Security Administration |
ASJC Scopus subject areas
- Software
- Hardware and Architecture
- Signal Processing