Methods: Participatory action research including a modified Delphi consensus building technique with frontline hospital staff, quality and safety staff, health systems researchers, local and national emergency medicine stakeholders was the method employed in this study. In Stage One, a workshop was held with the participatory action research team including frontline hospital staff, quality and safety staff and health systems researchers to gather suggestions regarding the evaluation measures. In Stage Two, an electronic modified-Delphi study was undertaken with a panel consisting of the workshop participants, key local and national emergency medicine stakeholders. Descriptive statistics were used to summarise the characteristics of the panellists who completed the questionnaires in each round. The mean Likert rating, standard deviation and 95% bias-corrected bootstrapped confidence interval for each variable was calculated. Bonferroni corrections were applied to take account of multiple testing. Data were analysed using Stata 14.0 SE.
Results: Using the Institute for Healthcare Improvement framework, 12 process, outcome and balancing metrics for measuring the effectiveness of an ED-specific early warning score system were developed.
Conclusion: There are currently no published measures for evaluating the effectiveness of an ED early warning score system. It was possible in this study to develop a suite of evaluation measures using a modified Delphi consensus approach. Using the collective expertise of frontline hospital staff, quality and safety staff and health systems researchers to develop and categorise the initial set of potential measures was an innovative and unique element of this study.
Keywords: Emergency department, Early warning score system, Longitudinal patient monitoring, Evaluation measures