Reliability of assessment of medical students' non-technical skills using a behavioural marker system: does clinical experience matter?

TY - JOUR

T1 - Reliability of assessment of medical students' non-technical skills using a behavioural marker system

T2 - does clinical experience matter?

AU - Clarke, Benjamin

AU - Smith, Samantha E.

AU - Phillips, Emma Claire

AU - Hamilton, Ailsa

AU - Kerins, Joanne

AU - Tallentire, Victoria R.

N1 - Publisher Copyright: © 2021 BMJ Publishing Group. All rights reserved.

PY - 2021/7/1

Y1 - 2021/7/1

N2 - Introduction Non-technical skills are recognised to play an integral part in safe and effective patient care. Medi-StuNTS (Medical Students' Non-Technical Skills) is a behavioural marker system developed to enable assessment of medical students' non-technical skills. This study aimed to assess whether newly trained raters with high levels of clinical experience could achieve reliability coefficients of >0.7 and to compare differences in inter-rater reliability of raters with varying clinical experience. Methods Forty-four raters attended a workshop on Medi-StuNTS before independently rating three videos of medical students participating in immersive simulation scenarios. Data were grouped by raters' levels of clinical experience. Inter-rater reliability was assessed by calculating intraclass correlation coefficients (ICC). Results Eleven raters with more than 10 years of clinical experience achieved single-measure ICC of 0.37 and average-measures ICC of 0.87. Fourteen raters with more than or equal to 5 years and less than 10 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC of 0.59. Nineteen raters with less than 5 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC 0.65. Conclusions Using 11 newly trained raters with high levels of clinical experience produced highly reliable ratings that surpassed the prespecified inter-rater reliability standard; however, a single rater from this group would not achieve sufficiently reliable ratings. This is consistent with previous studies using other medical behavioural marker systems. This study demonstrated a decrease in inter-rater reliability of raters with lower levels of clinical experience, suggesting caution when using this population as raters for assessment of non-technical skills.

AB - Introduction Non-technical skills are recognised to play an integral part in safe and effective patient care. Medi-StuNTS (Medical Students' Non-Technical Skills) is a behavioural marker system developed to enable assessment of medical students' non-technical skills. This study aimed to assess whether newly trained raters with high levels of clinical experience could achieve reliability coefficients of >0.7 and to compare differences in inter-rater reliability of raters with varying clinical experience. Methods Forty-four raters attended a workshop on Medi-StuNTS before independently rating three videos of medical students participating in immersive simulation scenarios. Data were grouped by raters' levels of clinical experience. Inter-rater reliability was assessed by calculating intraclass correlation coefficients (ICC). Results Eleven raters with more than 10 years of clinical experience achieved single-measure ICC of 0.37 and average-measures ICC of 0.87. Fourteen raters with more than or equal to 5 years and less than 10 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC of 0.59. Nineteen raters with less than 5 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC 0.65. Conclusions Using 11 newly trained raters with high levels of clinical experience produced highly reliable ratings that surpassed the prespecified inter-rater reliability standard; however, a single rater from this group would not achieve sufficiently reliable ratings. This is consistent with previous studies using other medical behavioural marker systems. This study demonstrated a decrease in inter-rater reliability of raters with lower levels of clinical experience, suggesting caution when using this population as raters for assessment of non-technical skills.

KW - Medical Student

KW - Non-Technical Skills

KW - Reliability

KW - Simulation Based Education

UR - https://www.scopus.com/pages/publications/85092543688

U2 - 10.1136/bmjstel-2020-000705

DO - 10.1136/bmjstel-2020-000705

M3 - Article

AN - SCOPUS:85092543688

SN - 2056-6697

VL - 7

SP - 285

EP - 292

JO - BMJ Simulation and Technology Enhanced Learning

JF - BMJ Simulation and Technology Enhanced Learning

IS - 5

ER -