TY - JOUR
T1 - Improving I/O Performance for Exascale Applications through Online Data Layout Reorganization
AU - Wan, Lipeng
AU - Huebl, Axel
AU - Gu, Junmin
AU - Poeschel, Franz
AU - Gainaru, Ana
AU - Wang, Ruonan
AU - Chen, Jieyang
AU - Liang, Xin
AU - Ganyushin, Dmitry
AU - Munson, Todd
AU - Foster, Ian
AU - Vay, Jean Luc
AU - Podhorszki, Norbert
AU - Wu, Kesheng
AU - Klasky, Scott
N1 - Publisher Copyright:
© 1990-2012 IEEE.
PY - 2022/4/1
Y1 - 2022/4/1
N2 - The applications being developed within the U.S. Exascale Computing Project (ECP) to run on imminent Exascale computers will generate scientific results with unprecedented fidelity and record turn-around time. Many of these codes are based on particle-mesh methods and use advanced algorithms, especially dynamic load-balancing and mesh-refinement, to achieve high performance on Exascale machines. Yet, as such algorithms improve parallel application efficiency, they raise new challenges for I/O logic due to their irregular and dynamic data distributions. Thus, while the enormous data rates of Exascale simulations already challenge existing file system write strategies, the need for efficient read and processing of generated data introduces additional constraints on the data layout strategies that can be used when writing data to secondary storage. We review these I/O challenges and introduce two online data layout reorganization approaches for achieving good tradeoffs between read and write performance. We demonstrate the benefits of using these two approaches for the ECP particle-in-cell simulation WarpX, which serves as a motif for a large class of important Exascale applications. We show that by understanding application I/O patterns and carefully designing data layouts we can increase read performance by more than 80 percent.
AB - The applications being developed within the U.S. Exascale Computing Project (ECP) to run on imminent Exascale computers will generate scientific results with unprecedented fidelity and record turn-around time. Many of these codes are based on particle-mesh methods and use advanced algorithms, especially dynamic load-balancing and mesh-refinement, to achieve high performance on Exascale machines. Yet, as such algorithms improve parallel application efficiency, they raise new challenges for I/O logic due to their irregular and dynamic data distributions. Thus, while the enormous data rates of Exascale simulations already challenge existing file system write strategies, the need for efficient read and processing of generated data introduces additional constraints on the data layout strategies that can be used when writing data to secondary storage. We review these I/O challenges and introduce two online data layout reorganization approaches for achieving good tradeoffs between read and write performance. We demonstrate the benefits of using these two approaches for the ECP particle-in-cell simulation WarpX, which serves as a motif for a large class of important Exascale applications. We show that by understanding application I/O patterns and carefully designing data layouts we can increase read performance by more than 80 percent.
KW - IO performance
KW - Parallel IO
KW - WarpX
KW - data access optimization
KW - data layout
UR - http://www.scopus.com/inward/record.url?scp=85112654543&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85112654543&partnerID=8YFLogxK
U2 - 10.1109/TPDS.2021.3100784
DO - 10.1109/TPDS.2021.3100784
M3 - Article
AN - SCOPUS:85112654543
SN - 1045-9219
VL - 33
SP - 878
EP - 890
JO - IEEE Transactions on Parallel and Distributed Systems
JF - IEEE Transactions on Parallel and Distributed Systems
IS - 4
ER -