Curated variation benchmarks for challenging medically relevant autosomal genes

Justin Wagner; Nathan D. Olson; Lindsay Harris; Jennifer McDaniel; Haoyu Cheng; Arkarachai Fungtammasan; Yih Chii Hwang; Richa Gupta; Aaron M. Wenger; William J. Rowell; Ziad M. Khan; Jesse Farek; Yiming Zhu; Aishwarya Pisupati; Medhat Mahmoud; Chunlin Xiao; Byunggil Yoo; Sayed Mohammad Ebrahim Sahraeian; Danny E. Miller; David Jáspez; José M. Lorenzo-Salazar; Adrián Muñoz-Barrera; Luis A. Rubio-Rodríguez; Carlos Flores; Giuseppe Narzisi; Uday Shanker Evani; Wayne E. Clarke; Joyce Lee; Christopher E. Mason; Stephen E. Lincoln; Karen H. Miga; Mark T.W. Ebbert; Alaina Shumate; Heng Li; Chen Shan Chin; Justin M. Zook; Fritz J. Sedlazeck

doi:10.1038/s41587-021-01158-1

Curated variation benchmarks for challenging medically relevant autosomal genes

Justin Wagner
, Nathan D. Olson
, Lindsay Harris
, Jennifer McDaniel
, Haoyu Cheng
, Arkarachai Fungtammasan
, Yih Chii Hwang
, Richa Gupta
, Aaron M. Wenger
, William J. Rowell
, Ziad M. Khan
, Jesse Farek
, Yiming Zhu
, Aishwarya Pisupati
, Medhat Mahmoud
, Chunlin Xiao
, Byunggil Yoo
, Sayed Mohammad Ebrahim Sahraeian
, Danny E. Miller
, David Jáspez

José M. Lorenzo-Salazar, Adrián Muñoz-Barrera, Luis A. Rubio-Rodríguez, Carlos Flores, Giuseppe Narzisi, Uday Shanker Evani, Wayne E. Clarke, Joyce Lee, Christopher E. Mason, Stephen E. Lincoln, Karen H. Miga, Mark T.W. Ebbert, Alaina Shumate, Heng Li, Chen Shan Chin, Justin M. Zook, Fritz J. Sedlazeck

Research output: Contribution to journal › Article › peer-review

128 Scopus citations

Abstract

The repetitive nature and complexity of some medically relevant genes poses a challenge for their accurate analysis in a clinical setting. The Genome in a Bottle Consortium has provided variant benchmark sets, but these exclude nearly 400 medically relevant genes due to their repetitiveness or polymorphic complexity. Here, we characterize 273 of these 395 challenging autosomal genes using a haplotype-resolved whole-genome assembly. This curated benchmark reports over 17,000 single-nucleotide variations, 3,600 insertions and deletions and 200 structural variations each for human genome reference GRCh37 and GRCh38 across HG002. We show that false duplications in either GRCh37 or GRCh38 result in reference-specific, missed variants for short- and long-read technologies in medically relevant genes, including CBS, CRYAA and KCNE1. When masking these false duplications, variant recall can improve from 8% to 100%. Forming benchmarks from a haplotype-resolved whole-genome assembly may become a prototype for future benchmarks covering the whole genome.

Original language	English
Pages (from-to)	672-680
Number of pages	9
Journal	Nature Biotechnology
Volume	40
Issue number	5
DOIs	https://doi.org/10.1038/s41587-021-01158-1
State	Published - May 2022

Bibliographical note

Publisher Copyright:
© 2022, This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply.

Funding

We thank the Genome Reference Consortium for their curation efforts of GRCh37 and GRCh38 (https://www.genomereference.org), especially V.A. Schneider and P.A. Kitts from the National Institutes of Health (NIH)/NCBI for developing the falsely duplicated regions that should be masked in GRCh38. We thank S. Miller at NIST for helping make available benchmark sets and READMEs. Certain commercial equipment, instruments or materials are identified to adequately specify experimental conditions or reported results. Such identification does not imply recommendation or endorsement by NIST, nor does it imply that the equipment, instruments or materials identified are necessarily the best available for the purpose. C.F. was funded by Instituto de Salud Carlos III (PI20/00876) and Ministerio de Ciencia e Innovación (RTC-2017-6471-1; AEI/FEDER, UE), cofinanced by the European Regional Development Fund ‘A Way of Making Europe’ from the European Union, and Cabildo Insular de Tenerife (CGIEU0000219140). J.M.L.-S. was funded by Consejería de Educación-Gobierno de Canarias and Cabildo Insular de Tenerife (BOC 163, 24/08/2017). F.J.S. and M.M. was supported by the NIH (UM1 HG008898). C.X. was supported by the Intramural Research Program of the National Library of Medicine, NIH. K.H.M. was supported by the NIH/National Human Genome Research Institute (R01 1R01HG011274-01 and U01 1U01HG010971). H.L. was supported by the NIH (R01 HG010040 and U01 HG010961). C.E.M. thanks funding from the WorldQuant Foundation, NASA (NNX14AH50G), the National Institutes of Health (R01MH117406, R01CA249054, R01AI151059, P01CA214274) and the Leukemia and Lymphoma Society (LLS) (MCL7001-18, LLS 9238-16, LLS-MCL7001-18). We thank the Genome Reference Consortium for their curation efforts of GRCh37 and GRCh38 ( https://www.genomereference.org ), especially V.A. Schneider and P.A. Kitts from the National Institutes of Health (NIH)/NCBI for developing the falsely duplicated regions that should be masked in GRCh38. We thank S. Miller at NIST for helping make available benchmark sets and READMEs. Certain commercial equipment, instruments or materials are identified to adequately specify experimental conditions or reported results. Such identification does not imply recommendation or endorsement by NIST, nor does it imply that the equipment, instruments or materials identified are necessarily the best available for the purpose. C.F. was funded by Instituto de Salud Carlos III (PI20/00876) and Ministerio de Ciencia e Innovación (RTC-2017-6471-1; AEI/FEDER, UE), cofinanced by the European Regional Development Fund ‘A Way of Making Europe’ from the European Union, and Cabildo Insular de Tenerife (CGIEU0000219140). J.M.L.-S. was funded by Consejería de Educación-Gobierno de Canarias and Cabildo Insular de Tenerife (BOC 163, 24/08/2017). F.J.S. and M.M. was supported by the NIH (UM1 HG008898). C.X. was supported by the Intramural Research Program of the National Library of Medicine, NIH. K.H.M. was supported by the NIH/National Human Genome Research Institute (R01 1R01HG011274-01 and U01 1U01HG010971). H.L. was supported by the NIH (R01 HG010040 and U01 HG010961). C.E.M. thanks funding from the WorldQuant Foundation, NASA (NNX14AH50G), the National Institutes of Health (R01MH117406, R01CA249054, R01AI151059, P01CA214274) and the Leukemia and Lymphoma Society (LLS) (MCL7001-18, LLS 9238-16, LLS-MCL7001-18).

Funders	Funder number
Cabildo Insular de Tenerife	CGIEU0000219140
Consejería de Educación-Gobierno de Canarias and Cabildo Insular de Tenerife	24/08/2017, BOC 163, UM1 HG008898
NCBI
National Institutes of Health (NIH)
Author National Institute on Drug Abuse DA031791 Mark J Ferris National Institute on Drug Abuse DA006634 Mark J Ferris National Institute on Alcohol Abuse and Alcoholism AA026117 Mark J Ferris National Institute on Alcohol Abuse and Alcoholism AA028162 Elizabeth G Pitts National Institute of General Medical Sciences GM102773 Elizabeth G Pitts Peter McManus Charitable Trust Mark J Ferris National Institute on Drug Abuse	U01DA053941
Author National Institute on Drug Abuse DA031791 Mark J Ferris National Institute on Drug Abuse DA006634 Mark J Ferris National Institute on Alcohol Abuse and Alcoholism AA026117 Mark J Ferris National Institute on Alcohol Abuse and Alcoholism AA028162 Elizabeth G Pitts National Institute of General Medical Sciences GM102773 Elizabeth G Pitts Peter McManus Charitable Trust Mark J Ferris National Institute on Drug Abuse
National Human Genome Research Institute	U01 1U01HG010971, U01 HG010961, R01 1R01HG011274-01, R01 HG010040
National Human Genome Research Institute
U.S. National Library of Medicine
National Aeronautics and Space Administration	R01AI151059, R01CA249054, R01MH117406, NNX14AH50G, P01CA214274
National Aeronautics and Space Administration
National Institute of Standards and Technology
Leukemia and Lymphoma Society	MCL7001-18, LLS 9238-16, LLS-MCL7001-18
Leukemia and Lymphoma Society
WorldQuant Foundation
European Commission
Instituto de Salud Carlos III	PI20/00876
Instituto de Salud Carlos III
Ministerio de Ciencia, Innovación y Universidades	RTC-2017-6471-1
Ministerio de Ciencia, Innovación y Universidades
European Regional Development Fund

ASJC Scopus subject areas

Biotechnology
Bioengineering
Applied Microbiology and Biotechnology
Biomedical Engineering
Molecular Medicine

Access to Document

10.1038/s41587-021-01158-1

Cite this

Wagner, J., Olson, N. D., Harris, L., McDaniel, J., Cheng, H., Fungtammasan, A., Hwang, Y. C., Gupta, R., Wenger, A. M., Rowell, W. J., Khan, Z. M., Farek, J., Zhu, Y., Pisupati, A., Mahmoud, M., Xiao, C., Yoo, B., Sahraeian, S. M. E., Miller, D. E., ... Sedlazeck, F. J. (2022). Curated variation benchmarks for challenging medically relevant autosomal genes. Nature Biotechnology, 40(5), 672-680. https://doi.org/10.1038/s41587-021-01158-1

@article{0b5bc3b4136d46048a71ef6faeaab242,

title = "Curated variation benchmarks for challenging medically relevant autosomal genes",

abstract = "The repetitive nature and complexity of some medically relevant genes poses a challenge for their accurate analysis in a clinical setting. The Genome in a Bottle Consortium has provided variant benchmark sets, but these exclude nearly 400 medically relevant genes due to their repetitiveness or polymorphic complexity. Here, we characterize 273 of these 395 challenging autosomal genes using a haplotype-resolved whole-genome assembly. This curated benchmark reports over 17,000 single-nucleotide variations, 3,600 insertions and deletions and 200 structural variations each for human genome reference GRCh37 and GRCh38 across HG002. We show that false duplications in either GRCh37 or GRCh38 result in reference-specific, missed variants for short- and long-read technologies in medically relevant genes, including CBS, CRYAA and KCNE1. When masking these false duplications, variant recall can improve from 8\% to 100\%. Forming benchmarks from a haplotype-resolved whole-genome assembly may become a prototype for future benchmarks covering the whole genome.",

author = "Justin Wagner and Olson, \{Nathan D.\} and Lindsay Harris and Jennifer McDaniel and Haoyu Cheng and Arkarachai Fungtammasan and Hwang, \{Yih Chii\} and Richa Gupta and Wenger, \{Aaron M.\} and Rowell, \{William J.\} and Khan, \{Ziad M.\} and Jesse Farek and Yiming Zhu and Aishwarya Pisupati and Medhat Mahmoud and Chunlin Xiao and Byunggil Yoo and Sahraeian, \{Sayed Mohammad Ebrahim\} and Miller, \{Danny E.\} and David J{\'a}spez and Lorenzo-Salazar, \{Jos{\'e} M.\} and Adri{\'a}n Mu{\~n}oz-Barrera and Rubio-Rodr{\'i}guez, \{Luis A.\} and Carlos Flores and Giuseppe Narzisi and Evani, \{Uday Shanker\} and Clarke, \{Wayne E.\} and Joyce Lee and Mason, \{Christopher E.\} and Lincoln, \{Stephen E.\} and Miga, \{Karen H.\} and Ebbert, \{Mark T.W.\} and Alaina Shumate and Heng Li and Chin, \{Chen Shan\} and Zook, \{Justin M.\} and Sedlazeck, \{Fritz J.\}",

note = "Publisher Copyright: {\textcopyright} 2022, This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply.",

year = "2022",

month = may,

doi = "10.1038/s41587-021-01158-1",

language = "English",

volume = "40",

pages = "672--680",

number = "5",

}

Wagner, J, Olson, ND, Harris, L, McDaniel, J, Cheng, H, Fungtammasan, A, Hwang, YC, Gupta, R, Wenger, AM, Rowell, WJ, Khan, ZM, Farek, J, Zhu, Y, Pisupati, A, Mahmoud, M, Xiao, C, Yoo, B, Sahraeian, SME, Miller, DE, Jáspez, D, Lorenzo-Salazar, JM, Muñoz-Barrera, A, Rubio-Rodríguez, LA, Flores, C, Narzisi, G, Evani, US, Clarke, WE, Lee, J, Mason, CE, Lincoln, SE, Miga, KH, Ebbert, MTW, Shumate, A, Li, H, Chin, CS, Zook, JM & Sedlazeck, FJ 2022, 'Curated variation benchmarks for challenging medically relevant autosomal genes', Nature Biotechnology, vol. 40, no. 5, pp. 672-680. https://doi.org/10.1038/s41587-021-01158-1

TY - JOUR

T1 - Curated variation benchmarks for challenging medically relevant autosomal genes

AU - Wagner, Justin

AU - Olson, Nathan D.

AU - Harris, Lindsay

AU - McDaniel, Jennifer

AU - Cheng, Haoyu

AU - Fungtammasan, Arkarachai

AU - Hwang, Yih Chii

AU - Gupta, Richa

AU - Wenger, Aaron M.

AU - Rowell, William J.

AU - Khan, Ziad M.

AU - Farek, Jesse

AU - Zhu, Yiming

AU - Pisupati, Aishwarya

AU - Mahmoud, Medhat

AU - Xiao, Chunlin

AU - Yoo, Byunggil

AU - Sahraeian, Sayed Mohammad Ebrahim

AU - Miller, Danny E.

AU - Jáspez, David

AU - Lorenzo-Salazar, José M.

AU - Muñoz-Barrera, Adrián

AU - Rubio-Rodríguez, Luis A.

AU - Flores, Carlos

AU - Narzisi, Giuseppe

AU - Evani, Uday Shanker

AU - Clarke, Wayne E.

AU - Lee, Joyce

AU - Mason, Christopher E.

AU - Lincoln, Stephen E.

AU - Miga, Karen H.

AU - Ebbert, Mark T.W.

AU - Shumate, Alaina

AU - Li, Heng

AU - Chin, Chen Shan

AU - Zook, Justin M.

AU - Sedlazeck, Fritz J.

PY - 2022/5

Y1 - 2022/5

N2 - The repetitive nature and complexity of some medically relevant genes poses a challenge for their accurate analysis in a clinical setting. The Genome in a Bottle Consortium has provided variant benchmark sets, but these exclude nearly 400 medically relevant genes due to their repetitiveness or polymorphic complexity. Here, we characterize 273 of these 395 challenging autosomal genes using a haplotype-resolved whole-genome assembly. This curated benchmark reports over 17,000 single-nucleotide variations, 3,600 insertions and deletions and 200 structural variations each for human genome reference GRCh37 and GRCh38 across HG002. We show that false duplications in either GRCh37 or GRCh38 result in reference-specific, missed variants for short- and long-read technologies in medically relevant genes, including CBS, CRYAA and KCNE1. When masking these false duplications, variant recall can improve from 8% to 100%. Forming benchmarks from a haplotype-resolved whole-genome assembly may become a prototype for future benchmarks covering the whole genome.

AB - The repetitive nature and complexity of some medically relevant genes poses a challenge for their accurate analysis in a clinical setting. The Genome in a Bottle Consortium has provided variant benchmark sets, but these exclude nearly 400 medically relevant genes due to their repetitiveness or polymorphic complexity. Here, we characterize 273 of these 395 challenging autosomal genes using a haplotype-resolved whole-genome assembly. This curated benchmark reports over 17,000 single-nucleotide variations, 3,600 insertions and deletions and 200 structural variations each for human genome reference GRCh37 and GRCh38 across HG002. We show that false duplications in either GRCh37 or GRCh38 result in reference-specific, missed variants for short- and long-read technologies in medically relevant genes, including CBS, CRYAA and KCNE1. When masking these false duplications, variant recall can improve from 8% to 100%. Forming benchmarks from a haplotype-resolved whole-genome assembly may become a prototype for future benchmarks covering the whole genome.

UR - https://www.scopus.com/pages/publications/85124358447

UR - https://www.scopus.com/pages/publications/85124358447#tab=citedBy

U2 - 10.1038/s41587-021-01158-1

DO - 10.1038/s41587-021-01158-1

M3 - Article

C2 - 35132260

AN - SCOPUS:85124358447

SN - 1087-0156

VL - 40

SP - 672

EP - 680

JO - Nature Biotechnology

JF - Nature Biotechnology

IS - 5

ER -

Curated variation benchmarks for challenging medically relevant autosomal genes

Abstract

Bibliographical note

Funding

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this