BeeFlow: A workflow management system for in situ processing across HPC and cloud systems

Jieyang Chen; Qiang Guan; Zhao Zhang; Xin Liang; Louis Vernon; Allen McPherson; Li Ta Lo; Patricia Grubel; Tim Randles; Zizhong Chen; James Ahrens

doi:10.1109/ICDCS.2018.00103

BeeFlow: A workflow management system for in situ processing across HPC and cloud systems

Jieyang Chen, Qiang Guan, Zhao Zhang, Xin Liang, Louis Vernon, Allen McPherson, Li Ta Lo, Patricia Grubel, Tim Randles, Zizhong Chen, James Ahrens

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

13 Scopus citations

Abstract

In this paper, we propose BeeFlow - an in situ analysis enabled workflow management system across multiple platforms using Docker containers. BeeFlow can support both traditional workflows as well as workflows with in situ analysis. BeeFlow leverages Docker containers to provide a portable, flexible, and reproducible workflow management system across HPC and cloud platforms. We showcase how current in situ visualization workflows can apply BeeFlow with DOE production codes VPIC and Flecsale.

Original language	English
Title of host publication	Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018
Pages	1029-1038
Number of pages	10
ISBN (Electronic)	9781538668719
DOIs	https://doi.org/10.1109/ICDCS.2018.00103
State	Published - Jul 19 2018
Event	38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018 - Vienna, Austria Duration: Jul 2 2018 → Jul 5 2018

Publication series

Name	Proceedings - International Conference on Distributed Computing Systems
Volume	2018-July

Conference

Conference	38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018
Country/Territory	Austria
City	Vienna
Period	7/2/18 → 7/5/18

Bibliographical note

Publisher Copyright:
© 2018 IEEE.

Keywords

Cloud computing
Container
High performance computing
Scientific workflow

ASJC Scopus subject areas

Software
Hardware and Architecture
Computer Networks and Communications

Access to Document

10.1109/ICDCS.2018.00103

Cite this

Chen, J., Guan, Q., Zhang, Z., Liang, X., Vernon, L., McPherson, A., Lo, L. T., Grubel, P., Randles, T., Chen, Z., & Ahrens, J. (2018). BeeFlow: A workflow management system for in situ processing across HPC and cloud systems. In Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018 (pp. 1029-1038). (Proceedings - International Conference on Distributed Computing Systems; Vol. 2018-July). https://doi.org/10.1109/ICDCS.2018.00103

@inproceedings{4b8b3a570b7d46d19b3f418035e338ad,

title = "BeeFlow: A workflow management system for in situ processing across HPC and cloud systems",

abstract = "In this paper, we propose BeeFlow - an in situ analysis enabled workflow management system across multiple platforms using Docker containers. BeeFlow can support both traditional workflows as well as workflows with in situ analysis. BeeFlow leverages Docker containers to provide a portable, flexible, and reproducible workflow management system across HPC and cloud platforms. We showcase how current in situ visualization workflows can apply BeeFlow with DOE production codes VPIC and Flecsale.",

keywords = "Cloud computing, Container, High performance computing, Scientific workflow",

author = "Jieyang Chen and Qiang Guan and Zhao Zhang and Xin Liang and Louis Vernon and Allen McPherson and Lo, {Li Ta} and Patricia Grubel and Tim Randles and Zizhong Chen and James Ahrens",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018 ; Conference date: 02-07-2018 Through 05-07-2018",

year = "2018",

month = jul,

day = "19",

doi = "10.1109/ICDCS.2018.00103",

language = "English",

series = "Proceedings - International Conference on Distributed Computing Systems",

pages = "1029--1038",

booktitle = "Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018",

}

Chen, J, Guan, Q, Zhang, Z, Liang, X, Vernon, L, McPherson, A, Lo, LT, Grubel, P, Randles, T, Chen, Z & Ahrens, J 2018, BeeFlow: A workflow management system for in situ processing across HPC and cloud systems. in Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018. Proceedings - International Conference on Distributed Computing Systems, vol. 2018-July, pp. 1029-1038, 38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018, Vienna, Austria, 7/2/18. https://doi.org/10.1109/ICDCS.2018.00103

BeeFlow: A workflow management system for in situ processing across HPC and cloud systems. / Chen, Jieyang; Guan, Qiang; Zhang, Zhao et al.
Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018. 2018. p. 1029-1038 (Proceedings - International Conference on Distributed Computing Systems; Vol. 2018-July).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - BeeFlow

T2 - 38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018

AU - Chen, Jieyang

AU - Guan, Qiang

AU - Zhang, Zhao

AU - Liang, Xin

AU - Vernon, Louis

AU - McPherson, Allen

AU - Lo, Li Ta

AU - Grubel, Patricia

AU - Randles, Tim

AU - Chen, Zizhong

AU - Ahrens, James

PY - 2018/7/19

Y1 - 2018/7/19

N2 - In this paper, we propose BeeFlow - an in situ analysis enabled workflow management system across multiple platforms using Docker containers. BeeFlow can support both traditional workflows as well as workflows with in situ analysis. BeeFlow leverages Docker containers to provide a portable, flexible, and reproducible workflow management system across HPC and cloud platforms. We showcase how current in situ visualization workflows can apply BeeFlow with DOE production codes VPIC and Flecsale.

AB - In this paper, we propose BeeFlow - an in situ analysis enabled workflow management system across multiple platforms using Docker containers. BeeFlow can support both traditional workflows as well as workflows with in situ analysis. BeeFlow leverages Docker containers to provide a portable, flexible, and reproducible workflow management system across HPC and cloud platforms. We showcase how current in situ visualization workflows can apply BeeFlow with DOE production codes VPIC and Flecsale.

KW - Cloud computing

KW - Container

KW - High performance computing

KW - Scientific workflow

UR - http://www.scopus.com/inward/record.url?scp=85050985435&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050985435&partnerID=8YFLogxK

U2 - 10.1109/ICDCS.2018.00103

DO - 10.1109/ICDCS.2018.00103

M3 - Conference contribution

AN - SCOPUS:85050985435

T3 - Proceedings - International Conference on Distributed Computing Systems

SP - 1029

EP - 1038

BT - Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018

Y2 - 2 July 2018 through 5 July 2018

ER -

Chen J, Guan Q, Zhang Z, Liang X, Vernon L, McPherson A et al. BeeFlow: A workflow management system for in situ processing across HPC and cloud systems. In Proceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018. 2018. p. 1029-1038. (Proceedings - International Conference on Distributed Computing Systems). doi: 10.1109/ICDCS.2018.00103

BeeFlow: A workflow management system for in situ processing across HPC and cloud systems

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this