Balancing measures or a balanced accounting of improvement impact

a qualitative analysis of individual and focus group interviews with improvement experts in Scotland

Madalina Toma, Tobias Dreischulte, Nicola M Gray, Diane Campbell, Bruce Guthrie (Lead / Corresponding author)

Research output: Contribution to journalArticle

3 Citations (Scopus)
40 Downloads (Pure)

Abstract

Background: As quality improvement (QI) programmes have become progressively larger scale, the risks of implementation having unintended consequences are increasingly recognised. More routine use of balancing measures to monitor unintended consequences has been proposed to evaluate overall effectiveness, but in practice published improvement interventions hardly ever report identification or measurement of consequences other than intended goals of improvement.

Methods: We conducted 15 semistructured interviews and two focus groups with 24 improvement experts to explore the current understanding of balancing measures in QI and inform a more balanced accounting of the overall impact of improvement interventions. Data were analysed iteratively using the framework approach.

Results: Participants described the consequences of improvement in terms of desirability/undesirability and the extent to which they were expected/unexpected when planning improvement. Four types of consequences were defined: expected desirable consequences (goals); expected undesirable consequences (trade-offs); unexpected undesirable consequences (unpleasant surprises); and unexpected desirable consequences (pleasant surprises). Unexpected consequences were considered important but rarely measured in existing programmes, and an improvement pause to take stock after implementation would allow these to be more actively identified and managed. A balanced accounting of all consequences of improvement interventions can facilitate staff engagement and reduce resistance to change, but has to be offset against the cost of additional data collection.

Conclusion: Improvement measurement is usually focused on measuring intended goals, with minimal use of balancing measures which when used, typically monitor trade-offs expected before implementation. This paper proposes that improvers and leaders should seek a balanced accounting of all consequences of improvement across the life of an improvement programme, including deliberately pausing after implementation to identify and quantitatively or qualitatively evaluate any pleasant or unpleasant surprises.

Original languageEnglish
Pages (from-to)547-556
Number of pages10
JournalBMJ Quality & Safety
Volume27
Issue number7
Early online date21 Oct 2017
DOIs
Publication statusPublished - Jul 2018

Fingerprint

Scotland
Focus Groups
Interviews
Quality Improvement
Costs and Cost Analysis

Cite this

@article{6e4a8ef1170f4877a74e008f3dbca504,
title = "Balancing measures or a balanced accounting of improvement impact: a qualitative analysis of individual and focus group interviews with improvement experts in Scotland",
abstract = "Background: As quality improvement (QI) programmes have become progressively larger scale, the risks of implementation having unintended consequences are increasingly recognised. More routine use of balancing measures to monitor unintended consequences has been proposed to evaluate overall effectiveness, but in practice published improvement interventions hardly ever report identification or measurement of consequences other than intended goals of improvement.Methods: We conducted 15 semistructured interviews and two focus groups with 24 improvement experts to explore the current understanding of balancing measures in QI and inform a more balanced accounting of the overall impact of improvement interventions. Data were analysed iteratively using the framework approach.Results: Participants described the consequences of improvement in terms of desirability/undesirability and the extent to which they were expected/unexpected when planning improvement. Four types of consequences were defined: expected desirable consequences (goals); expected undesirable consequences (trade-offs); unexpected undesirable consequences (unpleasant surprises); and unexpected desirable consequences (pleasant surprises). Unexpected consequences were considered important but rarely measured in existing programmes, and an improvement pause to take stock after implementation would allow these to be more actively identified and managed. A balanced accounting of all consequences of improvement interventions can facilitate staff engagement and reduce resistance to change, but has to be offset against the cost of additional data collection.Conclusion: Improvement measurement is usually focused on measuring intended goals, with minimal use of balancing measures which when used, typically monitor trade-offs expected before implementation. This paper proposes that improvers and leaders should seek a balanced accounting of all consequences of improvement across the life of an improvement programme, including deliberately pausing after implementation to identify and quantitatively or qualitatively evaluate any pleasant or unpleasant surprises.",
author = "Madalina Toma and Tobias Dreischulte and Gray, {Nicola M} and Diane Campbell and Bruce Guthrie",
note = "{\circledC} Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.",
year = "2018",
month = "7",
doi = "10.1136/bmjqs-2017-006554",
language = "English",
volume = "27",
pages = "547--556",
journal = "BMJ Quality & Safety",
issn = "2044-5415",
publisher = "BMJ Publishing Group",
number = "7",

}

TY - JOUR

T1 - Balancing measures or a balanced accounting of improvement impact

T2 - a qualitative analysis of individual and focus group interviews with improvement experts in Scotland

AU - Toma, Madalina

AU - Dreischulte, Tobias

AU - Gray, Nicola M

AU - Campbell, Diane

AU - Guthrie, Bruce

N1 - © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

PY - 2018/7

Y1 - 2018/7

N2 - Background: As quality improvement (QI) programmes have become progressively larger scale, the risks of implementation having unintended consequences are increasingly recognised. More routine use of balancing measures to monitor unintended consequences has been proposed to evaluate overall effectiveness, but in practice published improvement interventions hardly ever report identification or measurement of consequences other than intended goals of improvement.Methods: We conducted 15 semistructured interviews and two focus groups with 24 improvement experts to explore the current understanding of balancing measures in QI and inform a more balanced accounting of the overall impact of improvement interventions. Data were analysed iteratively using the framework approach.Results: Participants described the consequences of improvement in terms of desirability/undesirability and the extent to which they were expected/unexpected when planning improvement. Four types of consequences were defined: expected desirable consequences (goals); expected undesirable consequences (trade-offs); unexpected undesirable consequences (unpleasant surprises); and unexpected desirable consequences (pleasant surprises). Unexpected consequences were considered important but rarely measured in existing programmes, and an improvement pause to take stock after implementation would allow these to be more actively identified and managed. A balanced accounting of all consequences of improvement interventions can facilitate staff engagement and reduce resistance to change, but has to be offset against the cost of additional data collection.Conclusion: Improvement measurement is usually focused on measuring intended goals, with minimal use of balancing measures which when used, typically monitor trade-offs expected before implementation. This paper proposes that improvers and leaders should seek a balanced accounting of all consequences of improvement across the life of an improvement programme, including deliberately pausing after implementation to identify and quantitatively or qualitatively evaluate any pleasant or unpleasant surprises.

AB - Background: As quality improvement (QI) programmes have become progressively larger scale, the risks of implementation having unintended consequences are increasingly recognised. More routine use of balancing measures to monitor unintended consequences has been proposed to evaluate overall effectiveness, but in practice published improvement interventions hardly ever report identification or measurement of consequences other than intended goals of improvement.Methods: We conducted 15 semistructured interviews and two focus groups with 24 improvement experts to explore the current understanding of balancing measures in QI and inform a more balanced accounting of the overall impact of improvement interventions. Data were analysed iteratively using the framework approach.Results: Participants described the consequences of improvement in terms of desirability/undesirability and the extent to which they were expected/unexpected when planning improvement. Four types of consequences were defined: expected desirable consequences (goals); expected undesirable consequences (trade-offs); unexpected undesirable consequences (unpleasant surprises); and unexpected desirable consequences (pleasant surprises). Unexpected consequences were considered important but rarely measured in existing programmes, and an improvement pause to take stock after implementation would allow these to be more actively identified and managed. A balanced accounting of all consequences of improvement interventions can facilitate staff engagement and reduce resistance to change, but has to be offset against the cost of additional data collection.Conclusion: Improvement measurement is usually focused on measuring intended goals, with minimal use of balancing measures which when used, typically monitor trade-offs expected before implementation. This paper proposes that improvers and leaders should seek a balanced accounting of all consequences of improvement across the life of an improvement programme, including deliberately pausing after implementation to identify and quantitatively or qualitatively evaluate any pleasant or unpleasant surprises.

U2 - 10.1136/bmjqs-2017-006554

DO - 10.1136/bmjqs-2017-006554

M3 - Article

VL - 27

SP - 547

EP - 556

JO - BMJ Quality & Safety

JF - BMJ Quality & Safety

SN - 2044-5415

IS - 7

ER -