Abstract
With the ever-increasing volumes of data produced by today's large-scale scientific simulations, error-bounded lossy compression techniques have become critical: not only can they significantly reduce the data size but they also can retain high data fidelity for postanalysis. In this paper, we design a strategy to improve the compression quality significantly based on an optimized, hybrid prediction model. Our contribution is fourfold. (1) We propose a novel, transform-based predictor and optimize its compression quality. (2) We significantly improve the coefficient-encoding efficiency for the data-fitting predictor. (3) We propose an adaptive framework that can select the best-fit predictor accurately for different datasets. (4) We evaluate our solution and several existing state-of-the-art lossy compressors by running real-world applications on a supercomputer with 8,192 cores. Experiments show that our adaptive compressor can improve the compression ratio by 112∼165% compared with the second-best compressor. The parallel I/O performance is improved by about 100% because of the significantly reduced data size. The total I/O time is reduced by up to 60X with our compressor compared with the original I/O time.
Original language | English |
---|---|
Title of host publication | Proceedings of SC 2019 |
Subtitle of host publication | The International Conference for High Performance Computing, Networking, Storage and Analysis |
ISBN (Electronic) | 9781450362290 |
DOIs | |
State | Published - Nov 17 2019 |
Event | 2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019 - Denver, United States Duration: Nov 17 2019 → Nov 22 2019 |
Publication series
Name | International Conference for High Performance Computing, Networking, Storage and Analysis, SC |
---|---|
ISSN (Print) | 2167-4329 |
ISSN (Electronic) | 2167-4337 |
Conference
Conference | 2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019 |
---|---|
Country/Territory | United States |
City | Denver |
Period | 11/17/19 → 11/22/19 |
Bibliographical note
Publisher Copyright:© 2019 ACM.
Keywords
- Compression performance
- Data dumping/loading
- Error-bounded lossy compression
- Rate distortion
ASJC Scopus subject areas
- Computer Networks and Communications
- Computer Science Applications
- Hardware and Architecture
- Software