Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound

Sheng Di, Dingwen Tao, Xin Liang, Franck Cappello

Research output: Contribution to journalArticlepeer-review

15 Scopus citations

Abstract

An effective data compressor is becoming increasingly critical to today's scientific research, and many lossy compressors are developed in the context of absolute error bounds. Based on physical/chemical definitions of simulation fields or multiresolution demand, however, many scientific applications need to compress the data with a pointwise relative error bound (i.e., the smaller the data value, the smaller the compression error to tolerate). To this end, we propose two optimized lossy compression strategies under a state-of-the-art three-staged compression framework (prediction + quantization + entropy-encoding). The first strategy (called block-based strategy) splits the data set into many small blocks and computes an absolute error bound for each block, so it is particularly suitable for the data with relatively high consecutiveness in space. The second strategy (called multi-threshold-based strategy) splits the whole value range into multiple groups with exponentially increasing thresholds and performs the compression in each group separately, which is particularly suitable for the data with a relatively large value range and spiky value changes. We implement the two strategies rigorously and evaluate them comprehensively by using two scientific applications which both require lossy compression with point-wise relative error bound. Experiments show that the two strategies exhibit the best compression qualities on different types of data sets respectively. The compression ratio of our lossy compressor is higher than that of other state-of-the-art compressors by 17.2-618 percent on the climate simulation data and 30-210 percent on the N-body simulation data, with the same relative error bound and without degradation of the overall visualization effect of the entire data.

Original languageEnglish
Article number8421751
Pages (from-to)331-345
Number of pages15
JournalIEEE Transactions on Parallel and Distributed Systems
Volume30
Issue number2
DOIs
StatePublished - Feb 1 2019

Bibliographical note

Publisher Copyright:
© 2018 IEEE.

Keywords

  • Lossy compression
  • high performance computing
  • relative error bound
  • science data

ASJC Scopus subject areas

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics

Fingerprint

Dive into the research topics of 'Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound'. Together they form a unique fingerprint.

Cite this