The design, deployment, and evaluation of the CORAL pre-exascale systems

Sudharshan S. Vazhkudai; Bronis R. De Supinski; Arthur S. Bland; Al Geist; James Sexton; Jim Kahle; Christopher J. Zimmer; Scott Atchley; Sarp Oral; Don E. Maxwell; Veronica G.Vergara Larrea; Adam Bertsch; Robin Goldstone; Wayne Joubert; Chris Chambreau; David Appelhans; Robert Blackmore; Ben Casses; George Chochia; Gene Davison; Matthew A. Ezell; Tom Gooding; Elsa Gonsiorowski; Leopold Grinberg; Bill Hanson; Bill Hartner; Ian Karlin; Matthew L. Leininger; Dustin Leverman; Chris Marroquin; Adam Moody; Martin Ohmacht; Ramesh Pankajakshan; Fernando Pizzano; James H. Rogers; Bryan Rosenburg; Drew Schmidt; Mallikarjun Shankar; Feiyi Wang; Py Watson; Bob Walkup; Lance D. Weems; Junqi Yin

doi:10.1109/SC.2018.00055

The design, deployment, and evaluation of the CORAL pre-exascale systems

Sudharshan S. Vazhkudai, Bronis R. De Supinski, Arthur S. Bland, Al Geist, James Sexton, Jim Kahle, Christopher J. Zimmer, Scott Atchley, Sarp Oral, Don E. Maxwell, Veronica G.Vergara Larrea, Adam Bertsch, Robin Goldstone, Wayne Joubert, Chris Chambreau, David Appelhans, Robert Blackmore, Ben Casses, George Chochia, Gene DavisonMatthew A. Ezell, Tom Gooding, Elsa Gonsiorowski, Leopold Grinberg, Bill Hanson, Bill Hartner, Ian Karlin, Matthew L. Leininger, Dustin Leverman, Chris Marroquin, Adam Moody, Martin Ohmacht, Ramesh Pankajakshan, Fernando Pizzano, James H. Rogers, Bryan Rosenburg, Drew Schmidt, Mallikarjun Shankar, Feiyi Wang, Py Watson, Bob Walkup, Lance D. Weems, Junqi Yin

School of Electronics, Electrical Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

138 Citations (Scopus)

Abstract

CORAL, the Collaboration of Oak Ridge, Argonne and Livermore, is fielding two similar IBM systems, Summit and Sierra, with NVIDIA GPUs that will replace the existing Titan and Sequoia systems. Summit and Sierra are currently ranked No. 1 and No. 3, respectively on the Top500 list. We discuss the design and key differences of the systems. Our evaluation of the systems highlights the following. Applications that fit in HBM see the most benefit and may prefer more GPUs; however, for some applications, the CPU-GPU bandwidth is more important than the number of GPUs. The node-local burst buffer scales linearly, and can achieve a 4X improvement over the parallel file system for large jobs; smaller jobs, however, may benefit from writing directly to the PFS. Finally, several CPU, network and memory bound analytics and GPU-bound deep learning codes achieve up to a 11X and 79X speedup/node, respectively over Titan.

Original language	English
Title of host publication	Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	661-672
Number of pages	12
ISBN (Electronic)	9781538683842
DOIs	https://doi.org/10.1109/SC.2018.00055
Publication status	Published - 11 Mar 2019
Event	2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 - Dallas, United States Duration: 11 Nov 2018 → 16 Nov 2018

Conference

Conference	2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018
Country/Territory	United States
City	Dallas
Period	11/11/2018 → 16/11/2018

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Networks and Communications
Hardware and Architecture
Theoretical Computer Science

Access to Document

10.1109/SC.2018.00055

Cite this

Vazhkudai, S. S., De Supinski, B. R., Bland, A. S., Geist, A., Sexton, J., Kahle, J., Zimmer, C. J., Atchley, S., Oral, S., Maxwell, D. E., Larrea, V. G. V., Bertsch, A., Goldstone, R., Joubert, W., Chambreau, C., Appelhans, D., Blackmore, R., Casses, B., Chochia, G., ... Yin, J. (2019). The design, deployment, and evaluation of the CORAL pre-exascale systems. In Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 (pp. 661-672). Article 8665810 Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SC.2018.00055

@inproceedings{eda817d6544f47a6b7049e9138dc5df8,

title = "The design, deployment, and evaluation of the CORAL pre-exascale systems",

abstract = "CORAL, the Collaboration of Oak Ridge, Argonne and Livermore, is fielding two similar IBM systems, Summit and Sierra, with NVIDIA GPUs that will replace the existing Titan and Sequoia systems. Summit and Sierra are currently ranked No. 1 and No. 3, respectively on the Top500 list. We discuss the design and key differences of the systems. Our evaluation of the systems highlights the following. Applications that fit in HBM see the most benefit and may prefer more GPUs; however, for some applications, the CPU-GPU bandwidth is more important than the number of GPUs. The node-local burst buffer scales linearly, and can achieve a 4X improvement over the parallel file system for large jobs; smaller jobs, however, may benefit from writing directly to the PFS. Finally, several CPU, network and memory bound analytics and GPU-bound deep learning codes achieve up to a 11X and 79X speedup/node, respectively over Titan.",

author = "Vazhkudai, {Sudharshan S.} and {De Supinski}, {Bronis R.} and Bland, {Arthur S.} and Al Geist and James Sexton and Jim Kahle and Zimmer, {Christopher J.} and Scott Atchley and Sarp Oral and Maxwell, {Don E.} and Larrea, {Veronica G.Vergara} and Adam Bertsch and Robin Goldstone and Wayne Joubert and Chris Chambreau and David Appelhans and Robert Blackmore and Ben Casses and George Chochia and Gene Davison and Ezell, {Matthew A.} and Tom Gooding and Elsa Gonsiorowski and Leopold Grinberg and Bill Hanson and Bill Hartner and Ian Karlin and Leininger, {Matthew L.} and Dustin Leverman and Chris Marroquin and Adam Moody and Martin Ohmacht and Ramesh Pankajakshan and Fernando Pizzano and Rogers, {James H.} and Bryan Rosenburg and Drew Schmidt and Mallikarjun Shankar and Feiyi Wang and Py Watson and Bob Walkup and Weems, {Lance D.} and Junqi Yin",

year = "2019",

month = mar,

day = "11",

doi = "10.1109/SC.2018.00055",

language = "English",

pages = "661--672",

booktitle = "Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

note = "2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 ; Conference date: 11-11-2018 Through 16-11-2018",

}

Vazhkudai, SS, De Supinski, BR, Bland, AS, Geist, A, Sexton, J, Kahle, J, Zimmer, CJ, Atchley, S, Oral, S, Maxwell, DE, Larrea, VGV, Bertsch, A, Goldstone, R, Joubert, W, Chambreau, C, Appelhans, D, Blackmore, R, Casses, B, Chochia, G, Davison, G, Ezell, MA, Gooding, T, Gonsiorowski, E, Grinberg, L, Hanson, B, Hartner, B, Karlin, I, Leininger, ML, Leverman, D, Marroquin, C, Moody, A, Ohmacht, M, Pankajakshan, R, Pizzano, F, Rogers, JH, Rosenburg, B, Schmidt, D, Shankar, M, Wang, F, Watson, P, Walkup, B, Weems, LD & Yin, J 2019, The design, deployment, and evaluation of the CORAL pre-exascale systems. in Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018., 8665810, Institute of Electrical and Electronics Engineers Inc., pp. 661-672, 2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018, Dallas, United States, 11/11/2018. https://doi.org/10.1109/SC.2018.00055

The design, deployment, and evaluation of the CORAL pre-exascale systems. / Vazhkudai, Sudharshan S.; De Supinski, Bronis R.; Bland, Arthur S. et al.
Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 661-672 8665810.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - The design, deployment, and evaluation of the CORAL pre-exascale systems

AU - Vazhkudai, Sudharshan S.

AU - De Supinski, Bronis R.

AU - Bland, Arthur S.

AU - Geist, Al

AU - Sexton, James

AU - Kahle, Jim

AU - Zimmer, Christopher J.

AU - Atchley, Scott

AU - Oral, Sarp

AU - Maxwell, Don E.

AU - Larrea, Veronica G.Vergara

AU - Bertsch, Adam

AU - Goldstone, Robin

AU - Joubert, Wayne

AU - Chambreau, Chris

AU - Appelhans, David

AU - Blackmore, Robert

AU - Casses, Ben

AU - Chochia, George

AU - Davison, Gene

AU - Ezell, Matthew A.

AU - Gooding, Tom

AU - Gonsiorowski, Elsa

AU - Grinberg, Leopold

AU - Hanson, Bill

AU - Hartner, Bill

AU - Karlin, Ian

AU - Leininger, Matthew L.

AU - Leverman, Dustin

AU - Marroquin, Chris

AU - Moody, Adam

AU - Ohmacht, Martin

AU - Pankajakshan, Ramesh

AU - Pizzano, Fernando

AU - Rogers, James H.

AU - Rosenburg, Bryan

AU - Schmidt, Drew

AU - Shankar, Mallikarjun

AU - Wang, Feiyi

AU - Watson, Py

AU - Walkup, Bob

AU - Weems, Lance D.

AU - Yin, Junqi

PY - 2019/3/11

Y1 - 2019/3/11

N2 - CORAL, the Collaboration of Oak Ridge, Argonne and Livermore, is fielding two similar IBM systems, Summit and Sierra, with NVIDIA GPUs that will replace the existing Titan and Sequoia systems. Summit and Sierra are currently ranked No. 1 and No. 3, respectively on the Top500 list. We discuss the design and key differences of the systems. Our evaluation of the systems highlights the following. Applications that fit in HBM see the most benefit and may prefer more GPUs; however, for some applications, the CPU-GPU bandwidth is more important than the number of GPUs. The node-local burst buffer scales linearly, and can achieve a 4X improvement over the parallel file system for large jobs; smaller jobs, however, may benefit from writing directly to the PFS. Finally, several CPU, network and memory bound analytics and GPU-bound deep learning codes achieve up to a 11X and 79X speedup/node, respectively over Titan.

AB - CORAL, the Collaboration of Oak Ridge, Argonne and Livermore, is fielding two similar IBM systems, Summit and Sierra, with NVIDIA GPUs that will replace the existing Titan and Sequoia systems. Summit and Sierra are currently ranked No. 1 and No. 3, respectively on the Top500 list. We discuss the design and key differences of the systems. Our evaluation of the systems highlights the following. Applications that fit in HBM see the most benefit and may prefer more GPUs; however, for some applications, the CPU-GPU bandwidth is more important than the number of GPUs. The node-local burst buffer scales linearly, and can achieve a 4X improvement over the parallel file system for large jobs; smaller jobs, however, may benefit from writing directly to the PFS. Finally, several CPU, network and memory bound analytics and GPU-bound deep learning codes achieve up to a 11X and 79X speedup/node, respectively over Titan.

UR - http://www.scopus.com/inward/record.url?scp=85064105525&partnerID=8YFLogxK

U2 - 10.1109/SC.2018.00055

DO - 10.1109/SC.2018.00055

M3 - Conference contribution

AN - SCOPUS:85064105525

SP - 661

EP - 672

BT - Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018

Y2 - 11 November 2018 through 16 November 2018

ER -

Vazhkudai SS, De Supinski BR, Bland AS, Geist A, Sexton J, Kahle J et al. The design, deployment, and evaluation of the CORAL pre-exascale systems. In Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 661-672. 8665810 doi: 10.1109/SC.2018.00055

The design, deployment, and evaluation of the CORAL pre-exascale systems

Abstract

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Bronis de Supinski

Cite this

The design, deployment, and evaluation of the CORAL pre-exascale systems

Abstract

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Profiles

Bronis de Supinski

Cite this