Use cases of lossy compression for floating-point data in scientific data sets

Franck Cappello; Sheng Di; Sihuan Li; Xin Liang; Ali Murat Gok; Dingwen Tao; Chun Hong Yoon; Xin Chuan Wu; Yuri Alexeev; Frederic T. Chong

doi:10.1177/1094342019853336

Use cases of lossy compression for floating-point data in scientific data sets

Franck Cappello, Sheng Di, Sihuan Li, Xin Liang, Ali Murat Gok, Dingwen Tao, Chun Hong Yoon, Xin Chuan Wu, Yuri Alexeev, Frederic T. Chong

Computer Science

Research output: Contribution to journal › Article › peer-review

70 Scopus citations

Abstract

Architectural and technological trends of systems used for scientific computing call for a significant reduction of scientific data sets that are composed mainly of floating-point data. This article surveys and presents experimental results of currently identified use cases of generic lossy compression to address the different limitations of scientific computing systems. The article shows from a collection of experiments run on parallel systems of a leadership facility that lossy data compression not only can reduce the footprint of scientific data sets on storage but also can reduce I/O and checkpoint/restart times, accelerate computation, and even allow significantly larger problems to be run than without lossy compression. These results suggest that lossy compression will become an important technology in many aspects of high performance scientific computing. Because the constraints for each use case are different and often conflicting, this collection of results also indicates the need for more specialization of the compression pipelines.

Original language	English
Pages (from-to)	1201-1220
Number of pages	20
Journal	International Journal of High Performance Computing Applications
Volume	33
Issue number	6
DOIs	https://doi.org/10.1177/1094342019853336
State	Published - Nov 1 2019

Bibliographical note

Funding Information:
We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory. The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations—the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nation’s exascale computing imperative. The material was based upon work supported by the US Department of Energy, Office of Science, under contract DE-AC02-06CH11357 and supported by the National Science Foundation under Grant No. 1619253.

Publisher Copyright:
© The Author(s) 2019.

Keywords

Lossy compression
applications
floating-point data
scientific data set
use cases

ASJC Scopus subject areas

Software
Theoretical Computer Science
Hardware and Architecture

Access to Document

10.1177/1094342019853336

Cite this

@article{e9e26e4316ac412a866f51a1b510b831,

title = "Use cases of lossy compression for floating-point data in scientific data sets",

abstract = "Architectural and technological trends of systems used for scientific computing call for a significant reduction of scientific data sets that are composed mainly of floating-point data. This article surveys and presents experimental results of currently identified use cases of generic lossy compression to address the different limitations of scientific computing systems. The article shows from a collection of experiments run on parallel systems of a leadership facility that lossy data compression not only can reduce the footprint of scientific data sets on storage but also can reduce I/O and checkpoint/restart times, accelerate computation, and even allow significantly larger problems to be run than without lossy compression. These results suggest that lossy compression will become an important technology in many aspects of high performance scientific computing. Because the constraints for each use case are different and often conflicting, this collection of results also indicates the need for more specialization of the compression pipelines.",

keywords = "Lossy compression, applications, floating-point data, scientific data set, use cases",

author = "Franck Cappello and Sheng Di and Sihuan Li and Xin Liang and Gok, {Ali Murat} and Dingwen Tao and Yoon, {Chun Hong} and Wu, {Xin Chuan} and Yuri Alexeev and Chong, {Frederic T.}",

note = "Funding Information: We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory. The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations—the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nation{\textquoteright}s exascale computing imperative. The material was based upon work supported by the US Department of Energy, Office of Science, under contract DE-AC02-06CH11357 and supported by the National Science Foundation under Grant No. 1619253. Publisher Copyright: {\textcopyright} The Author(s) 2019.",

year = "2019",

month = nov,

day = "1",

doi = "10.1177/1094342019853336",

language = "English",

volume = "33",

pages = "1201--1220",

number = "6",

}

TY - JOUR

T1 - Use cases of lossy compression for floating-point data in scientific data sets

AU - Cappello, Franck

AU - Di, Sheng

AU - Li, Sihuan

AU - Liang, Xin

AU - Gok, Ali Murat

AU - Tao, Dingwen

AU - Yoon, Chun Hong

AU - Wu, Xin Chuan

AU - Alexeev, Yuri

AU - Chong, Frederic T.

N1 - Funding Information: We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory. The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations—the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nation’s exascale computing imperative. The material was based upon work supported by the US Department of Energy, Office of Science, under contract DE-AC02-06CH11357 and supported by the National Science Foundation under Grant No. 1619253. Publisher Copyright: © The Author(s) 2019.

PY - 2019/11/1

Y1 - 2019/11/1

N2 - Architectural and technological trends of systems used for scientific computing call for a significant reduction of scientific data sets that are composed mainly of floating-point data. This article surveys and presents experimental results of currently identified use cases of generic lossy compression to address the different limitations of scientific computing systems. The article shows from a collection of experiments run on parallel systems of a leadership facility that lossy data compression not only can reduce the footprint of scientific data sets on storage but also can reduce I/O and checkpoint/restart times, accelerate computation, and even allow significantly larger problems to be run than without lossy compression. These results suggest that lossy compression will become an important technology in many aspects of high performance scientific computing. Because the constraints for each use case are different and often conflicting, this collection of results also indicates the need for more specialization of the compression pipelines.

AB - Architectural and technological trends of systems used for scientific computing call for a significant reduction of scientific data sets that are composed mainly of floating-point data. This article surveys and presents experimental results of currently identified use cases of generic lossy compression to address the different limitations of scientific computing systems. The article shows from a collection of experiments run on parallel systems of a leadership facility that lossy data compression not only can reduce the footprint of scientific data sets on storage but also can reduce I/O and checkpoint/restart times, accelerate computation, and even allow significantly larger problems to be run than without lossy compression. These results suggest that lossy compression will become an important technology in many aspects of high performance scientific computing. Because the constraints for each use case are different and often conflicting, this collection of results also indicates the need for more specialization of the compression pipelines.

KW - Lossy compression

KW - applications

KW - floating-point data

KW - scientific data set

KW - use cases

UR - http://www.scopus.com/inward/record.url?scp=85068877905&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068877905&partnerID=8YFLogxK

U2 - 10.1177/1094342019853336

DO - 10.1177/1094342019853336

M3 - Article

AN - SCOPUS:85068877905

SN - 1094-3420

VL - 33

SP - 1201

EP - 1220

JO - International Journal of High Performance Computing Applications

JF - International Journal of High Performance Computing Applications

IS - 6

ER -

Use cases of lossy compression for floating-point data in scientific data sets

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this