Significantly improving lossy compression quality based on an optimized hybrid prediction model

Xin Liang; Sheng Di; Sihuan Li; Dingwen Tao; Bogdan Nicolae; Zizhong Chen; Franck Cappello

doi:10.1145/3295500.3356193

Significantly improving lossy compression quality based on an optimized hybrid prediction model

Xin Liang, Sheng Di, Sihuan Li, Dingwen Tao, Bogdan Nicolae, Zizhong Chen, Franck Cappello

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

16 Scopus citations

Abstract

With the ever-increasing volumes of data produced by today's large-scale scientific simulations, error-bounded lossy compression techniques have become critical: not only can they significantly reduce the data size but they also can retain high data fidelity for postanalysis. In this paper, we design a strategy to improve the compression quality significantly based on an optimized, hybrid prediction model. Our contribution is fourfold. (1) We propose a novel, transform-based predictor and optimize its compression quality. (2) We significantly improve the coefficient-encoding efficiency for the data-fitting predictor. (3) We propose an adaptive framework that can select the best-fit predictor accurately for different datasets. (4) We evaluate our solution and several existing state-of-the-art lossy compressors by running real-world applications on a supercomputer with 8,192 cores. Experiments show that our adaptive compressor can improve the compression ratio by 112∼165% compared with the second-best compressor. The parallel I/O performance is improved by about 100% because of the significantly reduced data size. The total I/O time is reduced by up to 60X with our compressor compared with the original I/O time.

Original language	English
Title of host publication	Proceedings of SC 2019
Subtitle of host publication	The International Conference for High Performance Computing, Networking, Storage and Analysis
ISBN (Electronic)	9781450362290
DOIs	https://doi.org/10.1145/3295500.3356193
State	Published - Nov 17 2019
Event	2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019 - Denver, United States Duration: Nov 17 2019 → Nov 22 2019

Publication series

Name	International Conference for High Performance Computing, Networking, Storage and Analysis, SC
ISSN (Print)	2167-4329
ISSN (Electronic)	2167-4337

Conference

Conference	2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019
Country/Territory	United States
City	Denver
Period	11/17/19 → 11/22/19

Bibliographical note

Funding Information:
This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations - the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nation’s exascale computing imperative. The material was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357, and supported by the National Science Foundation under Grant No. 1619253. This work was also supported by National Science Foundation CCF 1513201. We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory.

Publisher Copyright:
© 2019 ACM.

Keywords

Compression performance
Data dumping/loading
Error-bounded lossy compression
Rate distortion

ASJC Scopus subject areas

Computer Networks and Communications
Computer Science Applications
Hardware and Architecture
Software

Access to Document

10.1145/3295500.3356193

Cite this

Liang, X., Di, S., Li, S., Tao, D., Nicolae, B., Chen, Z., & Cappello, F. (2019). Significantly improving lossy compression quality based on an optimized hybrid prediction model. In Proceedings of SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis [a33] (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). https://doi.org/10.1145/3295500.3356193

@inproceedings{4729371e239443b5b13feb250a1f29e1,

title = "Significantly improving lossy compression quality based on an optimized hybrid prediction model",

abstract = "With the ever-increasing volumes of data produced by today's large-scale scientific simulations, error-bounded lossy compression techniques have become critical: not only can they significantly reduce the data size but they also can retain high data fidelity for postanalysis. In this paper, we design a strategy to improve the compression quality significantly based on an optimized, hybrid prediction model. Our contribution is fourfold. (1) We propose a novel, transform-based predictor and optimize its compression quality. (2) We significantly improve the coefficient-encoding efficiency for the data-fitting predictor. (3) We propose an adaptive framework that can select the best-fit predictor accurately for different datasets. (4) We evaluate our solution and several existing state-of-the-art lossy compressors by running real-world applications on a supercomputer with 8,192 cores. Experiments show that our adaptive compressor can improve the compression ratio by 112∼165% compared with the second-best compressor. The parallel I/O performance is improved by about 100% because of the significantly reduced data size. The total I/O time is reduced by up to 60X with our compressor compared with the original I/O time.",

keywords = "Compression performance, Data dumping/loading, Error-bounded lossy compression, Rate distortion",

author = "Xin Liang and Sheng Di and Sihuan Li and Dingwen Tao and Bogdan Nicolae and Zizhong Chen and Franck Cappello",

note = "Funding Information: This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations - the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nation{\textquoteright}s exascale computing imperative. The material was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357, and supported by the National Science Foundation under Grant No. 1619253. This work was also supported by National Science Foundation CCF 1513201. We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory. Publisher Copyright: {\textcopyright} 2019 ACM.; 2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019 ; Conference date: 17-11-2019 Through 22-11-2019",

year = "2019",

month = nov,

day = "17",

doi = "10.1145/3295500.3356193",

language = "English",

series = "International Conference for High Performance Computing, Networking, Storage and Analysis, SC",

booktitle = "Proceedings of SC 2019",

}

Liang, X, Di, S, Li, S, Tao, D, Nicolae, B, Chen, Z & Cappello, F 2019, Significantly improving lossy compression quality based on an optimized hybrid prediction model. in Proceedings of SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis., a33, International Conference for High Performance Computing, Networking, Storage and Analysis, SC, 2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019, Denver, United States, 11/17/19. https://doi.org/10.1145/3295500.3356193

Significantly improving lossy compression quality based on an optimized hybrid prediction model. / Liang, Xin; Di, Sheng; Li, Sihuan et al.
Proceedings of SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis. 2019. a33 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Significantly improving lossy compression quality based on an optimized hybrid prediction model

AU - Liang, Xin

AU - Di, Sheng

AU - Li, Sihuan

AU - Tao, Dingwen

AU - Nicolae, Bogdan

AU - Chen, Zizhong

AU - Cappello, Franck

N1 - Funding Information: This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations - the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nation’s exascale computing imperative. The material was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357, and supported by the National Science Foundation under Grant No. 1619253. This work was also supported by National Science Foundation CCF 1513201. We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory. Publisher Copyright: © 2019 ACM.

PY - 2019/11/17

Y1 - 2019/11/17

N2 - With the ever-increasing volumes of data produced by today's large-scale scientific simulations, error-bounded lossy compression techniques have become critical: not only can they significantly reduce the data size but they also can retain high data fidelity for postanalysis. In this paper, we design a strategy to improve the compression quality significantly based on an optimized, hybrid prediction model. Our contribution is fourfold. (1) We propose a novel, transform-based predictor and optimize its compression quality. (2) We significantly improve the coefficient-encoding efficiency for the data-fitting predictor. (3) We propose an adaptive framework that can select the best-fit predictor accurately for different datasets. (4) We evaluate our solution and several existing state-of-the-art lossy compressors by running real-world applications on a supercomputer with 8,192 cores. Experiments show that our adaptive compressor can improve the compression ratio by 112∼165% compared with the second-best compressor. The parallel I/O performance is improved by about 100% because of the significantly reduced data size. The total I/O time is reduced by up to 60X with our compressor compared with the original I/O time.

AB - With the ever-increasing volumes of data produced by today's large-scale scientific simulations, error-bounded lossy compression techniques have become critical: not only can they significantly reduce the data size but they also can retain high data fidelity for postanalysis. In this paper, we design a strategy to improve the compression quality significantly based on an optimized, hybrid prediction model. Our contribution is fourfold. (1) We propose a novel, transform-based predictor and optimize its compression quality. (2) We significantly improve the coefficient-encoding efficiency for the data-fitting predictor. (3) We propose an adaptive framework that can select the best-fit predictor accurately for different datasets. (4) We evaluate our solution and several existing state-of-the-art lossy compressors by running real-world applications on a supercomputer with 8,192 cores. Experiments show that our adaptive compressor can improve the compression ratio by 112∼165% compared with the second-best compressor. The parallel I/O performance is improved by about 100% because of the significantly reduced data size. The total I/O time is reduced by up to 60X with our compressor compared with the original I/O time.

KW - Compression performance

KW - Data dumping/loading

KW - Error-bounded lossy compression

KW - Rate distortion

UR - http://www.scopus.com/inward/record.url?scp=85076172304&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85076172304&partnerID=8YFLogxK

U2 - 10.1145/3295500.3356193

DO - 10.1145/3295500.3356193

M3 - Conference contribution

AN - SCOPUS:85076172304

T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC

BT - Proceedings of SC 2019

T2 - 2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019

Y2 - 17 November 2019 through 22 November 2019

ER -

Liang X, Di S, Li S, Tao D, Nicolae B, Chen Z et al. Significantly improving lossy compression quality based on an optimized hybrid prediction model. In Proceedings of SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis. 2019. a33. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). doi: 10.1145/3295500.3356193

Significantly improving lossy compression quality based on an optimized hybrid prediction model

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this