Hadoop-EDF: Large-scale Distributed Processing of Electrophysiological Signal Data in Hadoop MapReduce

Yuanyuan Wu; Xiaojin Li; Jinze Liu; Licong Cui

doi:10.1109/BIBM47256.2019.8983371

Hadoop-EDF: Large-scale Distributed Processing of Electrophysiological Signal Data in Hadoop MapReduce

Yuanyuan Wu
, Xiaojin Li
, Jinze Liu
, Licong Cui

Computer Science

Producción científica: Conference contribution › revisión exhaustiva

5 Citas (Scopus)

Resumen

Rapidly growing volume of electrophysiological signals has been generated for clinical research in neurological disorders. European Data Format (EDF) is a standard format for storing electrophysiological signals. However, the bottleneck of existing signal analysis tools for handling large-scale datasets is the sequential way of loading large EDF files before performing signal analyses. To overcome this, we develop Hadoop-EDF, a distributed signal processing tool to load EDF data in a parallel manner using Hadoop MapReduce. Hadoop-EDF uses a robust data partition algorithm making EDF data parallelly processable. We evaluate Hadoop-EDF's scalability and performance by leveraging two datasets from the National Sleep Research Resource and running experiments on Amazon Web Service clusters. The performance of Hadoop-EDF on a 20-node cluster achieved about 26 times and 47 times faster than the sequential processing of 200 small-size files and 200 large-size files, respectively. The results demonstrate that Hadoop-EDF is more suitable and effective in processing large EDF files.

Idioma original	English
Título de la publicación alojada	Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019
Editores	Illhoi Yoo, Jinbo Bi, Xiaohua Tony Hu
Páginas	2265-2271
Número de páginas	7
ISBN (versión digital)	9781728118673
DOI	https://doi.org/10.1109/BIBM47256.2019.8983371
Estado	Published - nov 2019
Evento	2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019 - San Diego, United States Duración: nov 18 2019 → nov 21 2019

Serie de la publicación

Nombre	Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019

Conference

Conference	2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019
País/Territorio	United States
Ciudad	San Diego
Período	11/18/19 → 11/21/19

Nota bibliográfica

Publisher Copyright:
© 2019 IEEE.

Financiación

This work was supported by the US National Institutes of Health under grants R24HL114473 and U01NS090408. Correspondence: [email protected]

Financiadores	Número del financiador
National Institutes of Health (NIH)	U01NS090408, R24HL114473
National Institutes of Health (NIH)

ODS de las Naciones Unidas

Este resultado contribuye a los siguientes Objetivos de Desarrollo Sostenible

Good health and well being

ASJC Scopus subject areas

Biochemistry
Biotechnology
Molecular Medicine
Modeling and Simulation
Health Informatics
Pharmacology (medical)
Public Health, Environmental and Occupational Health

Acceder al documento

10.1109/BIBM47256.2019.8983371

Otros archivos y enlaces

Citar esto

Wu, Y., Li, X., Liu, J., & Cui, L. (2019). Hadoop-EDF: Large-scale Distributed Processing of Electrophysiological Signal Data in Hadoop MapReduce. En I. Yoo, J. Bi, & X. T. Hu (Eds.), Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019 (pp. 2265-2271). Artículo 8983371 (Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019). https://doi.org/10.1109/BIBM47256.2019.8983371

Wu, Yuanyuan ; Li, Xiaojin ; Liu, Jinze et al. / Hadoop-EDF : Large-scale Distributed Processing of Electrophysiological Signal Data in Hadoop MapReduce. Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019. editor / Illhoi Yoo ; Jinbo Bi ; Xiaohua Tony Hu. 2019. pp. 2265-2271 (Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019).

@inproceedings{c7e7f584283e45848cb4616aac00a695,

title = "Hadoop-EDF: Large-scale Distributed Processing of Electrophysiological Signal Data in Hadoop MapReduce",

abstract = "Rapidly growing volume of electrophysiological signals has been generated for clinical research in neurological disorders. European Data Format (EDF) is a standard format for storing electrophysiological signals. However, the bottleneck of existing signal analysis tools for handling large-scale datasets is the sequential way of loading large EDF files before performing signal analyses. To overcome this, we develop Hadoop-EDF, a distributed signal processing tool to load EDF data in a parallel manner using Hadoop MapReduce. Hadoop-EDF uses a robust data partition algorithm making EDF data parallelly processable. We evaluate Hadoop-EDF's scalability and performance by leveraging two datasets from the National Sleep Research Resource and running experiments on Amazon Web Service clusters. The performance of Hadoop-EDF on a 20-node cluster achieved about 26 times and 47 times faster than the sequential processing of 200 small-size files and 200 large-size files, respectively. The results demonstrate that Hadoop-EDF is more suitable and effective in processing large EDF files.",

keywords = "Cloud Computing, Electrophysiological Signals, European Data Format, Hadoop MapReduce",

author = "Yuanyuan Wu and Xiaojin Li and Jinze Liu and Licong Cui",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019 ; Conference date: 18-11-2019 Through 21-11-2019",

year = "2019",

month = nov,

doi = "10.1109/BIBM47256.2019.8983371",

language = "English",

series = "Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019",

pages = "2265--2271",

editor = "Illhoi Yoo and Jinbo Bi and Hu, \{Xiaohua Tony\}",

booktitle = "Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019",

}

Wu, Y, Li, X, Liu, J & Cui, L 2019, Hadoop-EDF: Large-scale Distributed Processing of Electrophysiological Signal Data in Hadoop MapReduce. En I Yoo, J Bi & XT Hu (eds.), Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019., 8983371, Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019, pp. 2265-2271, 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019, San Diego, United States, 11/18/19. https://doi.org/10.1109/BIBM47256.2019.8983371

Hadoop-EDF: Large-scale Distributed Processing of Electrophysiological Signal Data in Hadoop MapReduce. / Wu, Yuanyuan; Li, Xiaojin; Liu, Jinze et al.
Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019. ed. / Illhoi Yoo; Jinbo Bi; Xiaohua Tony Hu. 2019. p. 2265-2271 8983371 (Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019).

Producción científica: Conference contribution › revisión exhaustiva

TY - GEN

T1 - Hadoop-EDF

T2 - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019

AU - Wu, Yuanyuan

AU - Li, Xiaojin

AU - Liu, Jinze

AU - Cui, Licong

PY - 2019/11

Y1 - 2019/11

N2 - Rapidly growing volume of electrophysiological signals has been generated for clinical research in neurological disorders. European Data Format (EDF) is a standard format for storing electrophysiological signals. However, the bottleneck of existing signal analysis tools for handling large-scale datasets is the sequential way of loading large EDF files before performing signal analyses. To overcome this, we develop Hadoop-EDF, a distributed signal processing tool to load EDF data in a parallel manner using Hadoop MapReduce. Hadoop-EDF uses a robust data partition algorithm making EDF data parallelly processable. We evaluate Hadoop-EDF's scalability and performance by leveraging two datasets from the National Sleep Research Resource and running experiments on Amazon Web Service clusters. The performance of Hadoop-EDF on a 20-node cluster achieved about 26 times and 47 times faster than the sequential processing of 200 small-size files and 200 large-size files, respectively. The results demonstrate that Hadoop-EDF is more suitable and effective in processing large EDF files.

AB - Rapidly growing volume of electrophysiological signals has been generated for clinical research in neurological disorders. European Data Format (EDF) is a standard format for storing electrophysiological signals. However, the bottleneck of existing signal analysis tools for handling large-scale datasets is the sequential way of loading large EDF files before performing signal analyses. To overcome this, we develop Hadoop-EDF, a distributed signal processing tool to load EDF data in a parallel manner using Hadoop MapReduce. Hadoop-EDF uses a robust data partition algorithm making EDF data parallelly processable. We evaluate Hadoop-EDF's scalability and performance by leveraging two datasets from the National Sleep Research Resource and running experiments on Amazon Web Service clusters. The performance of Hadoop-EDF on a 20-node cluster achieved about 26 times and 47 times faster than the sequential processing of 200 small-size files and 200 large-size files, respectively. The results demonstrate that Hadoop-EDF is more suitable and effective in processing large EDF files.

KW - Cloud Computing

KW - Electrophysiological Signals

KW - European Data Format

KW - Hadoop MapReduce

UR - https://www.scopus.com/pages/publications/85084340165

UR - https://www.scopus.com/pages/publications/85084340165#tab=citedBy

U2 - 10.1109/BIBM47256.2019.8983371

DO - 10.1109/BIBM47256.2019.8983371

M3 - Conference contribution

AN - SCOPUS:85084340165

T3 - Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019

SP - 2265

EP - 2271

BT - Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019

A2 - Yoo, Illhoi

A2 - Bi, Jinbo

A2 - Hu, Xiaohua Tony

Y2 - 18 November 2019 through 21 November 2019

ER -

Wu Y, Li X, Liu J, Cui L. Hadoop-EDF: Large-scale Distributed Processing of Electrophysiological Signal Data in Hadoop MapReduce. En Yoo I, Bi J, Hu XT, editores, Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019. 2019. p. 2265-2271. 8983371. (Proceedings - 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019). doi: 10.1109/BIBM47256.2019.8983371

Hadoop-EDF: Large-scale Distributed Processing of Electrophysiological Signal Data in Hadoop MapReduce

Resumen

Serie de la publicación

Conference

Nota bibliográfica

Financiación

ODS de las Naciones Unidas

ASJC Scopus subject areas

Acceder al documento

Otros archivos y enlaces

Huella

Citar esto