Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets

Xin Liang; Sheng Di; Dingwen Tao; Sihuan Li; Shaomeng Li; Hanqi Guo; Zizhong Chen; Franck Cappello

doi:10.1109/BigData.2018.8622520

Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets

Xin Liang, Sheng Di, Dingwen Tao, Sihuan Li, Shaomeng Li, Hanqi Guo, Zizhong Chen, Franck Cappello

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

97 Citations (SciVal)

Abstract

Today's scientific simulations require a significant reduction of the data size because of extremely large volumes of data they produce and the limitation of storage bandwidth and space. If the compression is set to reach a high compression ratio, however, the reconstructed data are often distorted too much to tolerate. In this paper, we explore a new compression strategy that can effectively control the data distortion when significantly reducing the data size. The contribution is threefold. (1) We propose an adaptive compression framework to select either our improved Lorenzo prediction method or our optimized linear regression method dynamically in different regions of the dataset. (2) We explore how to select them accurately based on the data features in each block to obtain the best compression quality. (3) We analyze the effectiveness of our solution in details using four real-world scientific datasets with 100+ fields. Evaluation results confirm that our new adaptive solution can significantly improve the rate distortion for the lossy compression with fairly high compression ratios. The compression ratio of our compressor is 1.5X~8X as high as that of two other leading lossy compressors (SZ and ZFP) with the same peak single-to-noise ratio (PSNR), in the high-compression cases. Parallel experiments with 8,192 cores and 24 TB of data shows that our solution obtains 1.86X dumping performance and 1.95X loading performance compared with the second-best lossy compressor, respectively.

Original language	English
Title of host publication	Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018
Editors	Yang Song, Bing Liu, Kisung Lee, Naoki Abe, Calton Pu, Mu Qiao, Nesreen Ahmed, Donald Kossmann, Jeffrey Saltz, Jiliang Tang, Jingrui He, Huan Liu, Xiaohua Hu
Pages	438-447
Number of pages	10
ISBN (Electronic)	9781538650356
DOIs	https://doi.org/10.1109/BigData.2018.8622520
State	Published - Jan 22 2019
Event	2018 IEEE International Conference on Big Data, Big Data 2018 - Seattle, United States Duration: Dec 10 2018 → Dec 13 2018

Publication series

Name	Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

Conference

Conference	2018 IEEE International Conference on Big Data, Big Data 2018
Country/Territory	United States
City	Seattle
Period	12/10/18 → 12/13/18

Bibliographical note

Funding Information:
This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nations exascale computing imperative. The material was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357, and supported by the National Science Foundation under Grant No. 1619253. This research is also supported by NSF Award No. 1513201. We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory.

Publisher Copyright:
© 2018 IEEE.

ASJC Scopus subject areas

Computer Science Applications
Information Systems

Access to Document

10.1109/BigData.2018.8622520

Cite this

Liang, X., Di, S., Tao, D., Li, S., Li, S., Guo, H., Chen, Z., & Cappello, F. (2019). Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets. In Y. Song, B. Liu, K. Lee, N. Abe, C. Pu, M. Qiao, N. Ahmed, D. Kossmann, J. Saltz, J. Tang, J. He, H. Liu, & X. Hu (Eds.), Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018 (pp. 438-447). [8622520] (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018). https://doi.org/10.1109/BigData.2018.8622520

Liang, Xin ; Di, Sheng ; Tao, Dingwen et al. / Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets. Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. editor / Yang Song ; Bing Liu ; Kisung Lee ; Naoki Abe ; Calton Pu ; Mu Qiao ; Nesreen Ahmed ; Donald Kossmann ; Jeffrey Saltz ; Jiliang Tang ; Jingrui He ; Huan Liu ; Xiaohua Hu. 2019. pp. 438-447 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018).

@inproceedings{3c16ba73fd8644f48ae46d709da639b8,

title = "Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets",

abstract = "Today's scientific simulations require a significant reduction of the data size because of extremely large volumes of data they produce and the limitation of storage bandwidth and space. If the compression is set to reach a high compression ratio, however, the reconstructed data are often distorted too much to tolerate. In this paper, we explore a new compression strategy that can effectively control the data distortion when significantly reducing the data size. The contribution is threefold. (1) We propose an adaptive compression framework to select either our improved Lorenzo prediction method or our optimized linear regression method dynamically in different regions of the dataset. (2) We explore how to select them accurately based on the data features in each block to obtain the best compression quality. (3) We analyze the effectiveness of our solution in details using four real-world scientific datasets with 100+ fields. Evaluation results confirm that our new adaptive solution can significantly improve the rate distortion for the lossy compression with fairly high compression ratios. The compression ratio of our compressor is 1.5X~8X as high as that of two other leading lossy compressors (SZ and ZFP) with the same peak single-to-noise ratio (PSNR), in the high-compression cases. Parallel experiments with 8,192 cores and 24 TB of data shows that our solution obtains 1.86X dumping performance and 1.95X loading performance compared with the second-best lossy compressor, respectively.",

author = "Xin Liang and Sheng Di and Dingwen Tao and Sihuan Li and Shaomeng Li and Hanqi Guo and Zizhong Chen and Franck Cappello",

note = "Funding Information: This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nations exascale computing imperative. The material was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357, and supported by the National Science Foundation under Grant No. 1619253. This research is also supported by NSF Award No. 1513201. We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory. Publisher Copyright: {\textcopyright} 2018 IEEE.; null ; Conference date: 10-12-2018 Through 13-12-2018",

year = "2019",

month = jan,

day = "22",

doi = "10.1109/BigData.2018.8622520",

language = "English",

series = "Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018",

pages = "438--447",

editor = "Yang Song and Bing Liu and Kisung Lee and Naoki Abe and Calton Pu and Mu Qiao and Nesreen Ahmed and Donald Kossmann and Jeffrey Saltz and Jiliang Tang and Jingrui He and Huan Liu and Xiaohua Hu",

booktitle = "Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018",

}

Liang, X, Di, S, Tao, D, Li, S, Li, S, Guo, H, Chen, Z & Cappello, F 2019, Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets. in Y Song, B Liu, K Lee, N Abe, C Pu, M Qiao, N Ahmed, D Kossmann, J Saltz, J Tang, J He, H Liu & X Hu (eds), Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018., 8622520, Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018, pp. 438-447, 2018 IEEE International Conference on Big Data, Big Data 2018, Seattle, United States, 12/10/18. https://doi.org/10.1109/BigData.2018.8622520

Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets. / Liang, Xin; Di, Sheng; Tao, Dingwen et al.

Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. ed. / Yang Song; Bing Liu; Kisung Lee; Naoki Abe; Calton Pu; Mu Qiao; Nesreen Ahmed; Donald Kossmann; Jeffrey Saltz; Jiliang Tang; Jingrui He; Huan Liu; Xiaohua Hu. 2019. p. 438-447 8622520 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets

AU - Liang, Xin

AU - Di, Sheng

AU - Tao, Dingwen

AU - Li, Sihuan

AU - Li, Shaomeng

AU - Guo, Hanqi

AU - Chen, Zizhong

AU - Cappello, Franck

N1 - Funding Information: This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nations exascale computing imperative. The material was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357, and supported by the National Science Foundation under Grant No. 1619253. This research is also supported by NSF Award No. 1513201. We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory. Publisher Copyright: © 2018 IEEE.

PY - 2019/1/22

Y1 - 2019/1/22

N2 - Today's scientific simulations require a significant reduction of the data size because of extremely large volumes of data they produce and the limitation of storage bandwidth and space. If the compression is set to reach a high compression ratio, however, the reconstructed data are often distorted too much to tolerate. In this paper, we explore a new compression strategy that can effectively control the data distortion when significantly reducing the data size. The contribution is threefold. (1) We propose an adaptive compression framework to select either our improved Lorenzo prediction method or our optimized linear regression method dynamically in different regions of the dataset. (2) We explore how to select them accurately based on the data features in each block to obtain the best compression quality. (3) We analyze the effectiveness of our solution in details using four real-world scientific datasets with 100+ fields. Evaluation results confirm that our new adaptive solution can significantly improve the rate distortion for the lossy compression with fairly high compression ratios. The compression ratio of our compressor is 1.5X~8X as high as that of two other leading lossy compressors (SZ and ZFP) with the same peak single-to-noise ratio (PSNR), in the high-compression cases. Parallel experiments with 8,192 cores and 24 TB of data shows that our solution obtains 1.86X dumping performance and 1.95X loading performance compared with the second-best lossy compressor, respectively.

AB - Today's scientific simulations require a significant reduction of the data size because of extremely large volumes of data they produce and the limitation of storage bandwidth and space. If the compression is set to reach a high compression ratio, however, the reconstructed data are often distorted too much to tolerate. In this paper, we explore a new compression strategy that can effectively control the data distortion when significantly reducing the data size. The contribution is threefold. (1) We propose an adaptive compression framework to select either our improved Lorenzo prediction method or our optimized linear regression method dynamically in different regions of the dataset. (2) We explore how to select them accurately based on the data features in each block to obtain the best compression quality. (3) We analyze the effectiveness of our solution in details using four real-world scientific datasets with 100+ fields. Evaluation results confirm that our new adaptive solution can significantly improve the rate distortion for the lossy compression with fairly high compression ratios. The compression ratio of our compressor is 1.5X~8X as high as that of two other leading lossy compressors (SZ and ZFP) with the same peak single-to-noise ratio (PSNR), in the high-compression cases. Parallel experiments with 8,192 cores and 24 TB of data shows that our solution obtains 1.86X dumping performance and 1.95X loading performance compared with the second-best lossy compressor, respectively.

UR - http://www.scopus.com/inward/record.url?scp=85062612432&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062612432&partnerID=8YFLogxK

U2 - 10.1109/BigData.2018.8622520

DO - 10.1109/BigData.2018.8622520

M3 - Conference contribution

AN - SCOPUS:85062612432

T3 - Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

SP - 438

EP - 447

BT - Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

A2 - Song, Yang

A2 - Liu, Bing

A2 - Lee, Kisung

A2 - Abe, Naoki

A2 - Pu, Calton

A2 - Qiao, Mu

A2 - Ahmed, Nesreen

A2 - Kossmann, Donald

A2 - Saltz, Jeffrey

A2 - Tang, Jiliang

A2 - He, Jingrui

A2 - Liu, Huan

A2 - Hu, Xiaohua

Y2 - 10 December 2018 through 13 December 2018

ER -

Liang X, Di S, Tao D, Li S, Li S, Guo H et al. Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets. In Song Y, Liu B, Lee K, Abe N, Pu C, Qiao M, Ahmed N, Kossmann D, Saltz J, Tang J, He J, Liu H, Hu X, editors, Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. 2019. p. 438-447. 8622520. (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018). doi: 10.1109/BigData.2018.8622520

Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this