Detailed modeling and evaluation of a scalable multilevel checkpointing system

Kathryn Mohror, Adam Moody, Greg Bronevetsky, Bronis R. De Supinski

Research output: Contribution to journalArticlepeer-review

16 Citations (Scopus)


High-performance computing (HPC) systems are growing more powerful by utilizing more components. As the system mean time before failure correspondingly drops, applications must checkpoint frequently to make progress. However, at scale, the cost of checkpointing becomes prohibitive. A solution to this problem is multilevel checkpointing, which employs multiple types of checkpoints in a single run. Lightweight checkpoints can handle the most common failure modes, while more expensive checkpoints can handle severe failures. We designed a multilevel checkpointing library, the Scalable Checkpoint/Restart (SCR) library, that writes lightweight checkpoints to node-local storage in addition to the parallel file system. We present probabilistic Markov models of SCR's performance. We show that on future large-scale systems, SCR can lead to a gain in machine efficiency of up to 35 percent, and reduce the load on the parallel file system by a factor of two. Additionally, we predict that checkpoint scavenging, or only writing checkpoints to the parallel file system on application termination, can reduce the load on the parallel file system by 20 × on today's systems and still maintain high application efficiency.

Original languageEnglish
Article number6494566
Pages (from-to)2255-2263
Number of pages9
JournalIEEE Transactions on Parallel and Distributed Systems
Issue number9
Publication statusPublished - 01 Jan 2014


  • evaluation
  • Fault tolerance
  • measurement
  • modeling
  • simulation of multiple-processor systems

ASJC Scopus subject areas

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics


Dive into the research topics of 'Detailed modeling and evaluation of a scalable multilevel checkpointing system'. Together they form a unique fingerprint.

Cite this