Constructing and evaluating a validity argument for the final-year ward simulation exercise

TY - JOUR

T1 - Constructing and evaluating a validity argument for the final-year ward simulation exercise

AU - Till, Hettie

AU - Ker, Jean

AU - Myford, Carol

AU - Stirling, Kevin

AU - Mires, Gary

N1 - No funding info

PY - 2015/12

Y1 - 2015/12

N2 - The authors report final-year ward simulation data from the University of Dundee Medical School. Faculty who designed this assessment intend for the final score to represent an individual senior medical student’s level of clinical performance. The results are included in each student’s portfolio as one source of evidence of the student’s capability as a practitioner, professional, and scholar. Our purpose in conducting this study was to illustrate how assessment designers who are creating assessments to evaluate clinical performance might develop propositions and then collect and examine various sources of evidence to construct and evaluate a validity argument. The data were from all 154 medical students who were in their final year of study at the University of Dundee Medical School in the 2010–2011 academic year. To the best of our knowledge, this is the first report on an analysis of senior medical students’ clinical performance while they were taking responsibility for the management of a simulated ward. Using multi-facet Rasch measurement and a generalizability theory approach, we examined various sources of validity evidence that the medical school faculty have gathered for a set of six propositions needed to support their use of scores as measures of students’ clinical ability. Based on our analysis of the evidence, we would conclude that, by and large, the propositions appear to be sound, and the evidence seems to support their proposed score interpretation. Given the body of evidence collected thus far, their intended interpretation seems defensible.

AB - The authors report final-year ward simulation data from the University of Dundee Medical School. Faculty who designed this assessment intend for the final score to represent an individual senior medical student’s level of clinical performance. The results are included in each student’s portfolio as one source of evidence of the student’s capability as a practitioner, professional, and scholar. Our purpose in conducting this study was to illustrate how assessment designers who are creating assessments to evaluate clinical performance might develop propositions and then collect and examine various sources of evidence to construct and evaluate a validity argument. The data were from all 154 medical students who were in their final year of study at the University of Dundee Medical School in the 2010–2011 academic year. To the best of our knowledge, this is the first report on an analysis of senior medical students’ clinical performance while they were taking responsibility for the management of a simulated ward. Using multi-facet Rasch measurement and a generalizability theory approach, we examined various sources of validity evidence that the medical school faculty have gathered for a set of six propositions needed to support their use of scores as measures of students’ clinical ability. Based on our analysis of the evidence, we would conclude that, by and large, the propositions appear to be sound, and the evidence seems to support their proposed score interpretation. Given the body of evidence collected thus far, their intended interpretation seems defensible.

KW - Clinical ability

KW - Clinical performance

KW - Evaluating medical students’ fitness to practice

KW - Generalizability theory

KW - Multi-facet Rasch measurement

KW - Score interpretation

KW - Simulated ward

KW - Simulation in assessment of clinical performance

KW - Validity argument

KW - Validity evidence

KW - Validity propositions

UR - https://www.scopus.com/pages/publications/84947037993

U2 - 10.1007/s10459-015-9601-5

DO - 10.1007/s10459-015-9601-5

M3 - Article

C2 - 25808311

SN - 1382-4996

VL - 20

SP - 1263

EP - 1289

JO - Advances in Health Sciences Education

JF - Advances in Health Sciences Education

IS - 5

ER -