What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression

Franck Cappello; Robert Underwood; Yuri Alexeev; Alison Baker; Ebru Bozdağ; Martin Burtscher; Kyle Chard; Sheng Di; Kyle Gerard Felker; Paul Christopher O'Grady; Hanqi Guo; Yafan Huang; Peng Jiang; Sian Jin; Petter Johansson; Shaomeng Li; Xin Liang; Erik Lindahl; Peter Lindstrom; Zarija Lukić; Magnus Lundborg; Danylo Lykov; Masaru Nagaso; Kento Sato; Amarjit Singh; Seung Woo Son; Shihui Song; William Tang; Dingwen Tao; Jiannan Tian; Kazutomo Yoshii; Kai Zhao

doi:10.1145/3712285.3759856

What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression

Franck Cappello
, Robert Underwood
, Yuri Alexeev
, Alison Baker
, Ebru Bozdağ
, Martin Burtscher
, Kyle Chard
, Sheng Di
, Kyle Gerard Felker
, Paul Christopher O'Grady
, Hanqi Guo
, Yafan Huang
, Peng Jiang
, Sian Jin
, Petter Johansson
, Shaomeng Li
, Xin Liang
, Erik Lindahl
, Peter Lindstrom
, Zarija Lukić

Magnus Lundborg, Danylo Lykov, Masaru Nagaso, Kento Sato, Amarjit Singh, Seung Woo Son, Shihui Song, William Tang, Dingwen Tao, Jiannan Tian, Kazutomo Yoshii, Kai Zhao

Producción científica: Conference contribution › revisión exhaustiva

4 Citas (Scopus)

Resumen

Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications' data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications' needs and compressors' capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.

Idioma original	English
Título de la publicación alojada	Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025
Páginas	1966-1979
Número de páginas	14
ISBN (versión digital)	9798400714665
DOI	https://doi.org/10.1145/3712285.3759856
Estado	Published - nov 15 2025
Evento	2025 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025 - St. Louis, United States Duración: nov 16 2025 → nov 21 2025

Serie de la publicación

Nombre	Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025

Conference

Conference	2025 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025
País/Territorio	United States
Ciudad	St. Louis
Período	11/16/25 → 11/21/25

Nota bibliográfica

Publisher Copyright:
© 2025 Copyright held by the owner/author(s).

Financiación

This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations - the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering, and early testbed platforms, to support the nation's exascale computing imperative. The material was supported by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR), under contract DE-AC02-06CH11357, and supported by the National Science Foundation under Grant OAC-2003709/2303064, OAC-2104023/2247080, OAC-2311875/2311876/2311877, OAC-2312673, OAC-2034169, OAC-1751143, OAC-2330367, OAC-2313122, OAC-2311756, OIA-2327266 and OAC-2103621. We acknowledge the computing resources provided on Bebop (operated by the Laboratory Computing Resource Center at Argonne). Some of the experiments presented in this paper were carried out using the PlaFRIM experimental testbed, supported by Inria, CNRS (LABRI and IMB), Université de Bordeaux, Bordeaux INP, and Conseil Régional d'Aquitaine (see https://www.plafrim.fr). TEZip's work has been supported by the COE research grant in computational science from Hyogo Prefecture and Kobe City through the Foundation for Computational Science. XIOS-SZ - Mario Acosta and Xavier Yepes-Arbós have received co-funding from the State Research Agency through OEMES (PID2020-116324RA-I00). We thank the Texas Advanced Computing Center (TACC) at the University of Texas at Austin for providing computational resources on the 'Frontera' system [55]. Use of the Linac Coherent Light Source (LCLS), SLAC National Accelerator Laboratory, is supported by the U.S. Department of Energy, Office of Science, Office of Basic Energy Sciences under Contract No. DEAC02-76SF00515. This work has been supported in part by the Department of Energy, Office of Science, under Award Number DE-SC0022223, as well as by equipment donations from NVIDIA Corporation. This work has been co-funded by the European Union through 'MDDB: Molecular Dynamics Data Bank. The European Repository for Biosimulation Data [101094651], and The Swedish e-Science Research Center.

Financiadores	Número del financiador
Université Bordeaux
National Nuclear Security Administration
Conseil Régional Aquitaine
Hyogo Prefecture and Kobe City
State Agency for Research
CNRS Centre National de la Recherche Scientifique
SLAC National Accelerator Laboratory
INRIA Institut National de Recherche en Informatique et en Automatique
Texas Advanced Computing Center
Institut polytechnique de Bordeaux
Office of Science Programs
Nvidia
OEMES	PID2020-116324RA-I00
Advanced Scientific Computing Research	DE-AC02-06CH11357
DOE Basic Energy Sciences	DEAC02-76SF00515
U.S. Department of Energy	DE-SC0022223
European Commission	101094651
National Science Foundation Arctic Social Science Program	OAC-2330367, OAC-2311756, OIA-2327266, OAC-2311875/2311876/2311877, OAC-1751143, OAC-2034169, OAC-2104023/2247080, OAC-2003709/2303064, OAC-2312673, OAC-2313122, OAC-2103621

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Networks and Communications
Hardware and Architecture

Acceder al documento

10.1145/3712285.3759856

Otros archivos y enlaces

Citar esto

Cappello, F., Underwood, R., Alexeev, Y., Baker, A., Bozdağ, E., Burtscher, M., Chard, K., Di, S., Felker, K. G., O'Grady, P. C., Guo, H., Huang, Y., Jiang, P., Jin, S., Johansson, P., Li, S., Liang, X., Lindahl, E., Lindstrom, P., ... Zhao, K. (2025). What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression. En Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025 (pp. 1966-1979). (Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025). https://doi.org/10.1145/3712285.3759856

Cappello, Franck ; Underwood, Robert ; Alexeev, Yuri et al. / What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression. Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025. 2025. pp. 1966-1979 (Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025).

@inproceedings{1c5d072e5c9548b484d17b84430aeca5,

title = "What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression",

abstract = "Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications' data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications' needs and compressors' capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.",

keywords = "Error-Bounded Lossy Compression, High-Performance Computing, I/O Optimization, Requirements Analysis, State of Practice",

author = "Franck Cappello and Robert Underwood and Yuri Alexeev and Alison Baker and Ebru Bozdağ and Martin Burtscher and Kyle Chard and Sheng Di and Felker, \{Kyle Gerard\} and O'Grady, \{Paul Christopher\} and Hanqi Guo and Yafan Huang and Peng Jiang and Sian Jin and Petter Johansson and Shaomeng Li and Xin Liang and Erik Lindahl and Peter Lindstrom and Zarija Luki{\'c} and Magnus Lundborg and Danylo Lykov and Masaru Nagaso and Kento Sato and Amarjit Singh and Son, \{Seung Woo\} and Shihui Song and William Tang and Dingwen Tao and Jiannan Tian and Kazutomo Yoshii and Kai Zhao",

note = "Publisher Copyright: {\textcopyright} 2025 Copyright held by the owner/author(s).; 2025 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025 ; Conference date: 16-11-2025 Through 21-11-2025",

year = "2025",

month = nov,

day = "15",

doi = "10.1145/3712285.3759856",

language = "English",

series = "Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025",

pages = "1966--1979",

booktitle = "Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025",

}

Cappello, F, Underwood, R, Alexeev, Y, Baker, A, Bozdağ, E, Burtscher, M, Chard, K, Di, S, Felker, KG, O'Grady, PC, Guo, H, Huang, Y, Jiang, P, Jin, S, Johansson, P, Li, S, Liang, X, Lindahl, E, Lindstrom, P, Lukić, Z, Lundborg, M, Lykov, D, Nagaso, M, Sato, K, Singh, A, Son, SW, Song, S, Tang, W, Tao, D, Tian, J, Yoshii, K & Zhao, K 2025, What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression. En Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025. Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025, pp. 1966-1979, 2025 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025, St. Louis, United States, 11/16/25. https://doi.org/10.1145/3712285.3759856

What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression. / Cappello, Franck; Underwood, Robert; Alexeev, Yuri et al.
Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025. 2025. p. 1966-1979 (Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025).

Producción científica: Conference contribution › revisión exhaustiva

TY - GEN

T1 - What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression

AU - Cappello, Franck

AU - Underwood, Robert

AU - Alexeev, Yuri

AU - Baker, Alison

AU - Bozdağ, Ebru

AU - Burtscher, Martin

AU - Chard, Kyle

AU - Di, Sheng

AU - Felker, Kyle Gerard

AU - O'Grady, Paul Christopher

AU - Guo, Hanqi

AU - Huang, Yafan

AU - Jiang, Peng

AU - Jin, Sian

AU - Johansson, Petter

AU - Li, Shaomeng

AU - Liang, Xin

AU - Lindahl, Erik

AU - Lindstrom, Peter

AU - Lukić, Zarija

AU - Lundborg, Magnus

AU - Lykov, Danylo

AU - Nagaso, Masaru

AU - Sato, Kento

AU - Singh, Amarjit

AU - Son, Seung Woo

AU - Song, Shihui

AU - Tang, William

AU - Tao, Dingwen

AU - Tian, Jiannan

AU - Yoshii, Kazutomo

AU - Zhao, Kai

PY - 2025/11/15

Y1 - 2025/11/15

N2 - Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications' data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications' needs and compressors' capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.

AB - Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications' data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications' needs and compressors' capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.

KW - Error-Bounded Lossy Compression

KW - High-Performance Computing

KW - I/O Optimization

KW - Requirements Analysis

KW - State of Practice

UR - https://www.scopus.com/pages/publications/105023972839

UR - https://www.scopus.com/pages/publications/105023972839#tab=citedBy

U2 - 10.1145/3712285.3759856

DO - 10.1145/3712285.3759856

M3 - Conference contribution

AN - SCOPUS:105023972839

T3 - Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025

SP - 1966

EP - 1979

BT - Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025

T2 - 2025 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025

Y2 - 16 November 2025 through 21 November 2025

ER -

Cappello F, Underwood R, Alexeev Y, Baker A, Bozdağ E, Burtscher M et al. What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression. En Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025. 2025. p. 1966-1979. (Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025). doi: 10.1145/3712285.3759856

What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression

Resumen

Serie de la publicación

Conference

Nota bibliográfica

Financiación

ASJC Scopus subject areas

Acceder al documento

Otros archivos y enlaces

Huella

Citar esto