Online data analysis and reduction: An important Co-design motif for extreme-scale computers

Ian Foster; Mark Ainsworth; Julie Bessac; Franck Cappello; Jong Choi; Sheng Di; Zichao Di; Ali M. Gok; Hanqi Guo; Kevin A. Huck; Christopher Kelly; Scott Klasky; Kerstin Kleese van Dam; Xin Liang; Kshitij Mehta; Manish Parashar; Tom Peterka; Line Pouchard; Tong Shu; Ozan Tugluk; Hubertus van Dam; Lipeng Wan; Matthew Wolf; Justin M. Wozniak; Wei Xu; Igor Yakushin; Shinjae Yoo; Todd Munson

doi:10.1177/10943420211023549

Online data analysis and reduction: An important Co-design motif for extreme-scale computers

Ian Foster
, Mark Ainsworth
, Julie Bessac
, Franck Cappello
, Jong Choi
, Sheng Di
, Zichao Di
, Ali M. Gok
, Hanqi Guo
, Kevin A. Huck
, Christopher Kelly
, Scott Klasky
, Kerstin Kleese van Dam
, Xin Liang
, Kshitij Mehta
, Manish Parashar
, Tom Peterka
, Line Pouchard
, Tong Shu
, Ozan Tugluk

Hubertus van Dam, Lipeng Wan, Matthew Wolf, Justin M. Wozniak, Wei Xu, Igor Yakushin, Shinjae Yoo, Todd Munson

Producción científica: Article › revisión exhaustiva

8 Citas (Scopus)

Resumen

A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

Idioma original	English
Páginas (desde-hasta)	617-635
Número de páginas	19
Publicación	International Journal of High Performance Computing Applications
Volumen	35
N.º	6
DOI	https://doi.org/10.1177/10943420211023549
Estado	Published - nov 2021

Nota bibliográfica

Publisher Copyright:
© The Author(s) 2021.

Financiación

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This article reports on work supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and National Nuclear Security Administration. This research used resources of the Argonne and Oak Ridge Leadership Computing Facilities and NERSC, DOE Office of Science User Facilities supported under Contracts DE-AC02-06CH11357, DE-AC05-00OR22725, and DE-AC02-05CH11231, respectively.

Financiadores	Número del financiador
U.S. Department of Energy Office of Science and National Nuclear Security Administration
Office of Science Programs	DE-AC05-00OR22725, DE-AC02-05CH11231, DE-AC02-06CH11357
National Energy Research Scientific Computing Center

ASJC Scopus subject areas

Software
Theoretical Computer Science
Hardware and Architecture

Acceder al documento

10.1177/10943420211023549

Otros archivos y enlaces

Citar esto

Foster, I., Ainsworth, M., Bessac, J., Cappello, F., Choi, J., Di, S., Di, Z., Gok, A. M., Guo, H., Huck, K. A., Kelly, C., Klasky, S., Kleese van Dam, K., Liang, X., Mehta, K., Parashar, M., Peterka, T., Pouchard, L., Shu, T., ... Munson, T. (2021). Online data analysis and reduction: An important Co-design motif for extreme-scale computers. International Journal of High Performance Computing Applications, 35(6), 617-635. https://doi.org/10.1177/10943420211023549

@article{d3d5da4ad0e84da79919025351b399ee,

title = "Online data analysis and reduction: An important Co-design motif for extreme-scale computers",

abstract = "A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.",

keywords = "Data analysis, exascale computing, in situ, online data analysis and reduction",

author = "Ian Foster and Mark Ainsworth and Julie Bessac and Franck Cappello and Jong Choi and Sheng Di and Zichao Di and Gok, \{Ali M.\} and Hanqi Guo and Huck, \{Kevin A.\} and Christopher Kelly and Scott Klasky and \{Kleese van Dam\}, Kerstin and Xin Liang and Kshitij Mehta and Manish Parashar and Tom Peterka and Line Pouchard and Tong Shu and Ozan Tugluk and \{van Dam\}, Hubertus and Lipeng Wan and Matthew Wolf and Wozniak, \{Justin M.\} and Wei Xu and Igor Yakushin and Shinjae Yoo and Todd Munson",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2021.",

year = "2021",

month = nov,

doi = "10.1177/10943420211023549",

language = "English",

volume = "35",

pages = "617--635",

number = "6",

}

Foster, I, Ainsworth, M, Bessac, J, Cappello, F, Choi, J, Di, S, Di, Z, Gok, AM, Guo, H, Huck, KA, Kelly, C, Klasky, S, Kleese van Dam, K, Liang, X, Mehta, K, Parashar, M, Peterka, T, Pouchard, L, Shu, T, Tugluk, O, van Dam, H, Wan, L, Wolf, M, Wozniak, JM, Xu, W, Yakushin, I, Yoo, S & Munson, T 2021, 'Online data analysis and reduction: An important Co-design motif for extreme-scale computers', International Journal of High Performance Computing Applications, vol. 35, n.º 6, pp. 617-635. https://doi.org/10.1177/10943420211023549

TY - JOUR

T1 - Online data analysis and reduction

T2 - An important Co-design motif for extreme-scale computers

AU - Foster, Ian

AU - Ainsworth, Mark

AU - Bessac, Julie

AU - Cappello, Franck

AU - Choi, Jong

AU - Di, Sheng

AU - Di, Zichao

AU - Gok, Ali M.

AU - Guo, Hanqi

AU - Huck, Kevin A.

AU - Kelly, Christopher

AU - Klasky, Scott

AU - Kleese van Dam, Kerstin

AU - Liang, Xin

AU - Mehta, Kshitij

AU - Parashar, Manish

AU - Peterka, Tom

AU - Pouchard, Line

AU - Shu, Tong

AU - Tugluk, Ozan

AU - van Dam, Hubertus

AU - Wan, Lipeng

AU - Wolf, Matthew

AU - Wozniak, Justin M.

AU - Xu, Wei

AU - Yakushin, Igor

AU - Yoo, Shinjae

AU - Munson, Todd

PY - 2021/11

Y1 - 2021/11

N2 - A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

AB - A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

KW - Data analysis

KW - exascale computing

KW - in situ

KW - online data analysis and reduction

UR - https://www.scopus.com/pages/publications/85107798299

UR - https://www.scopus.com/pages/publications/85107798299#tab=citedBy

U2 - 10.1177/10943420211023549

DO - 10.1177/10943420211023549

M3 - Article

AN - SCOPUS:85107798299

SN - 1094-3420

VL - 35

SP - 617

EP - 635

JO - International Journal of High Performance Computing Applications

JF - International Journal of High Performance Computing Applications

IS - 6

ER -

Online data analysis and reduction: An important Co-design motif for extreme-scale computers

Resumen

Nota bibliográfica

Financiación

ASJC Scopus subject areas

Acceder al documento

Otros archivos y enlaces

Huella

Citar esto