Online data analysis and reduction: An important Co-design motif for extreme-scale computers

Ian Foster, Mark Ainsworth, Julie Bessac, Franck Cappello, Jong Choi, Sheng Di, Zichao Di, Ali M. Gok, Hanqi Guo, Kevin A. Huck, Christopher Kelly, Scott Klasky, Kerstin Kleese van Dam, Xin Liang, Kshitij Mehta, Manish Parashar, Tom Peterka, Line Pouchard, Tong Shu, Ozan TuglukHubertus van Dam, Lipeng Wan, Matthew Wolf, Justin M. Wozniak, Wei Xu, Igor Yakushin, Shinjae Yoo, Todd Munson

Research output: Contribution to journalArticlepeer-review

7 Scopus citations

Abstract

A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

Original languageEnglish
Pages (from-to)617-635
Number of pages19
JournalInternational Journal of High Performance Computing Applications
Volume35
Issue number6
DOIs
StatePublished - Nov 2021

Bibliographical note

Publisher Copyright:
© The Author(s) 2021.

Keywords

  • Data analysis
  • exascale computing
  • in situ
  • online data analysis and reduction

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture

Fingerprint

Dive into the research topics of 'Online data analysis and reduction: An important Co-design motif for extreme-scale computers'. Together they form a unique fingerprint.

Cite this