Collating time-series resource data for system-wide job profiling

V. K.Cody Bumgardner; Victor W. Marek; Ray L. Hyatt

doi:10.1109/NOMS.2016.7502958

Collating time-series resource data for system-wide job profiling

V. K.Cody Bumgardner
, Victor W. Marek
, Ray L. Hyatt

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

Through the collection and association of discrete time-series resource metrics and workloads, we can both provide benchmark and intra-job resource collations, along with system-wide job profiling. Traditional RDBMSes are not designed to store and process long-term discrete time-series metrics and the commonly used resolution-reducing round robin databases (RRDB), make poor long-term sources of data for workload analytics. We implemented a system that employs Big-data (Hadoop/HBase) and other analytics (R) techniques and tools to store, process, and characterize HPC workloads. Using this system we have collected and processed over a 30 billion time-series metrics from existing short-term high-resolution (15-sec RRDB) sources, profiling over 200 thousand jobs across a wide spectrum of workloads. The system is currently in use at the University of Kentucky for better understanding of individual jobs and system-wide profiling as well as a strategic source of data for resource allocation and future acquisitions.

Original language	English
Title of host publication	Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium
Editors	Sema Oktug Badonnel, Mehmet Ulema, Cicek Cavdar, Lisandro Zambenedetti Granville, Carlos Raniery P. dos Santos
Pages	1043-1048
Number of pages	6
ISBN (Electronic)	9781509002238
DOIs	https://doi.org/10.1109/NOMS.2016.7502958
State	Published - Jun 30 2016
Event	2016 IEEE/IFIP Network Operations and Management Symposium, NOMS 2016 - Istanbul, Turkey Duration: Apr 25 2016 → Apr 29 2016

Publication series

Name	Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium

Conference

Conference	2016 IEEE/IFIP Network Operations and Management Symposium, NOMS 2016
Country/Territory	Turkey
City	Istanbul
Period	4/25/16 → 4/29/16

Bibliographical note

Publisher Copyright:
© 2016 IEEE.

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems and Management

Access to Document

10.1109/NOMS.2016.7502958

Cite this

Bumgardner, V. K. C., Marek, V. W., & Hyatt, R. L. (2016). Collating time-series resource data for system-wide job profiling. In S. O. Badonnel, M. Ulema, C. Cavdar, L. Z. Granville, & C. R. P. dos Santos (Eds.), Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium (pp. 1043-1048). Article 7502958 (Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium). https://doi.org/10.1109/NOMS.2016.7502958

Bumgardner, V. K.Cody ; Marek, Victor W. ; Hyatt, Ray L. / Collating time-series resource data for system-wide job profiling. Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium. editor / Sema Oktug Badonnel ; Mehmet Ulema ; Cicek Cavdar ; Lisandro Zambenedetti Granville ; Carlos Raniery P. dos Santos. 2016. pp. 1043-1048 (Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium).

@inproceedings{5bea9feeb4c848b590250694367a5b4b,

title = "Collating time-series resource data for system-wide job profiling",

abstract = "Through the collection and association of discrete time-series resource metrics and workloads, we can both provide benchmark and intra-job resource collations, along with system-wide job profiling. Traditional RDBMSes are not designed to store and process long-term discrete time-series metrics and the commonly used resolution-reducing round robin databases (RRDB), make poor long-term sources of data for workload analytics. We implemented a system that employs Big-data (Hadoop/HBase) and other analytics (R) techniques and tools to store, process, and characterize HPC workloads. Using this system we have collected and processed over a 30 billion time-series metrics from existing short-term high-resolution (15-sec RRDB) sources, profiling over 200 thousand jobs across a wide spectrum of workloads. The system is currently in use at the University of Kentucky for better understanding of individual jobs and system-wide profiling as well as a strategic source of data for resource allocation and future acquisitions.",

author = "Bumgardner, \{V. K.Cody\} and Marek, \{Victor W.\} and Hyatt, \{Ray L.\}",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 2016 IEEE/IFIP Network Operations and Management Symposium, NOMS 2016 ; Conference date: 25-04-2016 Through 29-04-2016",

year = "2016",

month = jun,

day = "30",

doi = "10.1109/NOMS.2016.7502958",

language = "English",

series = "Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium",

pages = "1043--1048",

editor = "Badonnel, \{Sema Oktug\} and Mehmet Ulema and Cicek Cavdar and Granville, \{Lisandro Zambenedetti\} and \{dos Santos\}, \{Carlos Raniery P.\}",

booktitle = "Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium",

}

Bumgardner, VKC, Marek, VW & Hyatt, RL 2016, Collating time-series resource data for system-wide job profiling. in SO Badonnel, M Ulema, C Cavdar, LZ Granville & CRP dos Santos (eds), Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium., 7502958, Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium, pp. 1043-1048, 2016 IEEE/IFIP Network Operations and Management Symposium, NOMS 2016, Istanbul, Turkey, 4/25/16. https://doi.org/10.1109/NOMS.2016.7502958

Collating time-series resource data for system-wide job profiling. / Bumgardner, V. K.Cody; Marek, Victor W.; Hyatt, Ray L.
Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium. ed. / Sema Oktug Badonnel; Mehmet Ulema; Cicek Cavdar; Lisandro Zambenedetti Granville; Carlos Raniery P. dos Santos. 2016. p. 1043-1048 7502958 (Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Collating time-series resource data for system-wide job profiling

AU - Bumgardner, V. K.Cody

AU - Marek, Victor W.

AU - Hyatt, Ray L.

PY - 2016/6/30

Y1 - 2016/6/30

N2 - Through the collection and association of discrete time-series resource metrics and workloads, we can both provide benchmark and intra-job resource collations, along with system-wide job profiling. Traditional RDBMSes are not designed to store and process long-term discrete time-series metrics and the commonly used resolution-reducing round robin databases (RRDB), make poor long-term sources of data for workload analytics. We implemented a system that employs Big-data (Hadoop/HBase) and other analytics (R) techniques and tools to store, process, and characterize HPC workloads. Using this system we have collected and processed over a 30 billion time-series metrics from existing short-term high-resolution (15-sec RRDB) sources, profiling over 200 thousand jobs across a wide spectrum of workloads. The system is currently in use at the University of Kentucky for better understanding of individual jobs and system-wide profiling as well as a strategic source of data for resource allocation and future acquisitions.

AB - Through the collection and association of discrete time-series resource metrics and workloads, we can both provide benchmark and intra-job resource collations, along with system-wide job profiling. Traditional RDBMSes are not designed to store and process long-term discrete time-series metrics and the commonly used resolution-reducing round robin databases (RRDB), make poor long-term sources of data for workload analytics. We implemented a system that employs Big-data (Hadoop/HBase) and other analytics (R) techniques and tools to store, process, and characterize HPC workloads. Using this system we have collected and processed over a 30 billion time-series metrics from existing short-term high-resolution (15-sec RRDB) sources, profiling over 200 thousand jobs across a wide spectrum of workloads. The system is currently in use at the University of Kentucky for better understanding of individual jobs and system-wide profiling as well as a strategic source of data for resource allocation and future acquisitions.

UR - https://www.scopus.com/pages/publications/84979747402

UR - https://www.scopus.com/pages/publications/84979747402#tab=citedBy

U2 - 10.1109/NOMS.2016.7502958

DO - 10.1109/NOMS.2016.7502958

M3 - Conference contribution

AN - SCOPUS:84979747402

T3 - Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium

SP - 1043

EP - 1048

BT - Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium

A2 - Badonnel, Sema Oktug

A2 - Ulema, Mehmet

A2 - Cavdar, Cicek

A2 - Granville, Lisandro Zambenedetti

A2 - dos Santos, Carlos Raniery P.

T2 - 2016 IEEE/IFIP Network Operations and Management Symposium, NOMS 2016

Y2 - 25 April 2016 through 29 April 2016

ER -

Bumgardner VKC, Marek VW, Hyatt RL. Collating time-series resource data for system-wide job profiling. In Badonnel SO, Ulema M, Cavdar C, Granville LZ, dos Santos CRP, editors, Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium. 2016. p. 1043-1048. 7502958. (Proceedings of the NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium). doi: 10.1109/NOMS.2016.7502958

Collating time-series resource data for system-wide job profiling

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this