The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index

Michael Walsh; Sadeesh K. Srinathan; Daniel F. McAuley; Marko Mrkobrada; Oren Levine; Christine Ribic; Amber O. Molnar; Neil D. Dattani; Andrew Burke; Gordon Guyatt; Lehana Thabane; Stephen D. Walter; Janice Pogue; P. J. Devereaux

doi:10.1016/j.jclinepi.2013.10.019

The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index

Michael Walsh^*, Sadeesh K. Srinathan, Daniel F. McAuley, Marko Mrkobrada, Oren Levine, Christine Ribic, Amber O. Molnar, Neil D. Dattani, Andrew Burke, Gordon Guyatt, Lehana Thabane, Stephen D. Walter, Janice Pogue, P. J. Devereaux

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

553 Citations (Scopus)

676 Downloads (Pure)

Abstract

Objectives

A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes.

Study Design and Setting

A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result.

Results

The 399 eligible trials had a median sample size of 682 patients (range: 15-112,604) and a median of 112 events (range: 8-5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0-109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up.

Conclusion

The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results.

Original language	English
Pages (from-to)	622-628
Number of pages	7
Journal	Journal of Clinical Epidemiology
Volume	67
Issue number	6
Early online date	05 Feb 2014
DOIs	https://doi.org/10.1016/j.jclinepi.2013.10.019
Publication status	Published - 01 Jun 2014

Keywords

Lost to follow-up
Randomized controlled trials
Research methodology

ASJC Scopus subject areas

Epidemiology

Access to Document

10.1016/j.jclinepi.2013.10.019

The statistical significance of randomized controlled trial results is frequently fragile
2014 The Authors. Published by Elsevier Inc. This is an open access article published under a Creative Commons Attribution-NonCommercial-NoDerivs License (https://creativecommons.org/licenses/by-nc-nd/3.0/), which permits distribution and reproduction for non-commercial purposes, provided the author and source are cited.
Final published version, 472 KB

Cite this

Walsh, M., Srinathan, S. K., McAuley, D. F., Mrkobrada, M., Levine, O., Ribic, C., Molnar, A. O., Dattani, N. D., Burke, A., Guyatt, G., Thabane, L., Walter, S. D., Pogue, J., & Devereaux, P. J. (2014). The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index. Journal of Clinical Epidemiology, 67(6), 622-628. https://doi.org/10.1016/j.jclinepi.2013.10.019

@article{5b53307736034dee9c48ed02a2e5be05,

title = "The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index",

abstract = "Objectives A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes. Study Design and Setting A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result. Results The 399 eligible trials had a median sample size of 682 patients (range: 15-112,604) and a median of 112 events (range: 8-5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0-109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up. Conclusion The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results. ",

keywords = "Lost to follow-up, Randomized controlled trials, Research methodology",

author = "Michael Walsh and Srinathan, {Sadeesh K.} and McAuley, {Daniel F.} and Marko Mrkobrada and Oren Levine and Christine Ribic and Molnar, {Amber O.} and Dattani, {Neil D.} and Andrew Burke and Gordon Guyatt and Lehana Thabane and Walter, {Stephen D.} and Janice Pogue and Devereaux, {P. J.}",

year = "2014",

month = jun,

day = "1",

doi = "10.1016/j.jclinepi.2013.10.019",

language = "English",

volume = "67",

pages = "622--628",

journal = "Journal of Clinical Epidemiology",

issn = "0895-4356",

publisher = "Elsevier Inc.",

number = "6",

}

Walsh, M, Srinathan, SK, McAuley, DF, Mrkobrada, M, Levine, O, Ribic, C, Molnar, AO, Dattani, ND, Burke, A, Guyatt, G, Thabane, L, Walter, SD, Pogue, J & Devereaux, PJ 2014, 'The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index', Journal of Clinical Epidemiology, vol. 67, no. 6, pp. 622-628. https://doi.org/10.1016/j.jclinepi.2013.10.019

TY - JOUR

T1 - The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index

AU - Walsh, Michael

AU - Srinathan, Sadeesh K.

AU - McAuley, Daniel F.

AU - Mrkobrada, Marko

AU - Levine, Oren

AU - Ribic, Christine

AU - Molnar, Amber O.

AU - Dattani, Neil D.

AU - Burke, Andrew

AU - Guyatt, Gordon

AU - Thabane, Lehana

AU - Walter, Stephen D.

AU - Pogue, Janice

AU - Devereaux, P. J.

PY - 2014/6/1

Y1 - 2014/6/1

N2 - Objectives A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes. Study Design and Setting A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result. Results The 399 eligible trials had a median sample size of 682 patients (range: 15-112,604) and a median of 112 events (range: 8-5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0-109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up. Conclusion The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results.

AB - Objectives A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes. Study Design and Setting A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result. Results The 399 eligible trials had a median sample size of 682 patients (range: 15-112,604) and a median of 112 events (range: 8-5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0-109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up. Conclusion The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results.

KW - Lost to follow-up

KW - Randomized controlled trials

KW - Research methodology

UR - http://www.scopus.com/inward/record.url?scp=84899543437&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2013.10.019

DO - 10.1016/j.jclinepi.2013.10.019

M3 - Article

C2 - 24508144

AN - SCOPUS:84899543437

SN - 0895-4356

VL - 67

SP - 622

EP - 628

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

IS - 6

ER -

The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this