Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: Meta-regression of six UK longitudinal studies

Objective To assess the construct-level predictive validity of measures of educational attainment and intellectual aptitude used in medical student selection for predicting future assessments of clinical knowledge in undergraduate and postgraduate medical training. A major problem for studies of selection is that predictor-outcome correlations can only be measured in those who have been selected, whereas selectors need to know how those measures would predict in the entire pool of applicants, with their much wider range of predictor scores. Construct-level predictive validity estimates the true predictor-outcome correlation across the whole range of candidate abilities.
Design Meta-regression of estimates of construct-level predictive validity of A-levels, GCSEs and tests of intellectual aptitude (AH5 and UKCAT) in predicting undergraduate performance in basic medical science and finals assessments, and postgraduate performance at MRCP(UK) results and gaining entry to the Specialist Register. Construct-level predictive validities were calculated with a modification of the method of Hunter, Schmidt and Le (2006), using the MCMC algorithm, which corrected for right-censorship of examination results due to grade inflation, range restriction, and attenuation due to unreliability.
Setting Medical schools in the UK.
Participants Applicants and entrants to UK medical schools in six large, prospective, longitudinal
cohort studies in which students entered medical school between 1972 and 2009.
Results 57 separate predictor-outcome correlations and construct-level predictive validities were analysed using meta-regression. Construct-level predictive validities are substantially higher (mean = .450) than simple predictor-outcome correlations (mean=.171). For all undergraduate and postgraduate assessments A-levels had a higher predictive validity (.656; CI .575 to .726) than did GCSEs/O-levels (.342; CI .258 to .420) and intellectual aptitude tests (.208; CI .124 to .289). Overall construct-level predictive validity of educational attainment measures probably declines a little during training, but continues to predict postgraduate performance. In predicting first year examinations, A-levels had higher construct-level predictive validities (.809; CI= .501 to .935) than GCSEs/O-levels (.332; CI= .024 to .583) and UKCAT, the sole test of intellectual aptitude (.245; 95% CI = .207 to .276).
Conclusions Educational attainment has strong construct-level predictive validity for both undergraduate and postgraduate performance, accounting for perhaps 65% of true variance in first year basic medical science performance. That validity justifies the continuing use of measures of educational attainment in selection, but also raises the key theoretical question of the nature of the remaining 35% of variance, since measurement error, range restriction and right-censorship have all been taken into account. As in astrophysics, where ‘dark matter’ and ‘dark energy’ are posited to exist in order to balance various theoretical equations, so medical student selection must also have its ‘dark variance’, whose nature is not yet properly known, but accounts for a third of variation in performance of doctors in training. Some of this variance may relate to factors which cannot be predicted at selection, such as student illness or other life events, but a proportion may also be associated with personality, motivation or study skills which may be measurable during student selection.