Implementation Fidelity in Computerised Assessment of Book Reading

Keith Topping (Lead / Corresponding author)

Research output: Contribution to journalArticle

1 Citation (Scopus)
16 Downloads (Pure)

Abstract

Measuring the implementation fidelity (IF) or integrity of interventions is extremely important, since without it a positive or negative outcome cannot be interpreted. However, IF is actually measured relatively rarely. Direct and indirect methods of measurement have been used in the past, but tend to over-emphasize teacher behaviour. This paper focuses on student behaviour collated through computers - an interesting alternative. It deals with the reading of real books and reading achievement, for which variables a very large amount of computerised data was available – on 852,295 students in 3243 schools. Reading achievement was measured pre-post with STAR Reading, a computerised item-banked adaptive norm-referenced test of reading comprehension. IF came from the Accelerated Reader (AR), which measures understanding of independent reading of real books the student has chosen by a quiz. Results showed higher IF was related to higher achievement. Neither IF nor reading achievement related to socio-economic status. Primary (elementary) schools had higher IF and achievement than secondary (high) schools. Females had higher IF and achievement than males. Students of higher reading ability implemented AR at a higher level, but did not gain in reading at a higher level. However, this computerised method of measuring IF with book reading showed limited reliability, no greater than methods emphasising teacher behaviour. In future, IF measures emphasising student response and those emphasising teacher behaviour need to be blended, although the latter will never generate the sample size of the former. This may be true of implementation fidelity in areas other than book reading.
Original languageEnglish
Pages (from-to)176-190
Number of pages15
JournalComputers & Education
Volume116
Early online date23 Sep 2017
DOIs
Publication statusPublished - Jan 2018

Fingerprint

Students
teachers' behavior
student
Economics
quiz
school
elementary school
integrity
primary school
comprehension
ability
economics

Keywords

  • Evaluation methodologies
  • Gender studies
  • Improving classroom teaching
  • Pedagogical issues
  • Teaching and learning strategies

Cite this

@article{770bbccac600407fb0103b5a75592ba5,
title = "Implementation Fidelity in Computerised Assessment of Book Reading",
abstract = "Measuring the implementation fidelity (IF) or integrity of interventions is extremely important, since without it a positive or negative outcome cannot be interpreted. However, IF is actually measured relatively rarely. Direct and indirect methods of measurement have been used in the past, but tend to over-emphasize teacher behaviour. This paper focuses on student behaviour collated through computers - an interesting alternative. It deals with the reading of real books and reading achievement, for which variables a very large amount of computerised data was available – on 852,295 students in 3243 schools. Reading achievement was measured pre-post with STAR Reading, a computerised item-banked adaptive norm-referenced test of reading comprehension. IF came from the Accelerated Reader (AR), which measures understanding of independent reading of real books the student has chosen by a quiz. Results showed higher IF was related to higher achievement. Neither IF nor reading achievement related to socio-economic status. Primary (elementary) schools had higher IF and achievement than secondary (high) schools. Females had higher IF and achievement than males. Students of higher reading ability implemented AR at a higher level, but did not gain in reading at a higher level. However, this computerised method of measuring IF with book reading showed limited reliability, no greater than methods emphasising teacher behaviour. In future, IF measures emphasising student response and those emphasising teacher behaviour need to be blended, although the latter will never generate the sample size of the former. This may be true of implementation fidelity in areas other than book reading.",
keywords = "Evaluation methodologies, Gender studies, Improving classroom teaching, Pedagogical issues, Teaching and learning strategies",
author = "Keith Topping",
note = "Funding: none.",
year = "2018",
month = "1",
doi = "10.1016/j.compedu.2017.09.009",
language = "English",
volume = "116",
pages = "176--190",
journal = "Computers & Education",
issn = "0360-1315",
publisher = "Elsevier",

}

Implementation Fidelity in Computerised Assessment of Book Reading. / Topping, Keith (Lead / Corresponding author).

In: Computers & Education, Vol. 116, 01.2018, p. 176-190.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Implementation Fidelity in Computerised Assessment of Book Reading

AU - Topping, Keith

N1 - Funding: none.

PY - 2018/1

Y1 - 2018/1

N2 - Measuring the implementation fidelity (IF) or integrity of interventions is extremely important, since without it a positive or negative outcome cannot be interpreted. However, IF is actually measured relatively rarely. Direct and indirect methods of measurement have been used in the past, but tend to over-emphasize teacher behaviour. This paper focuses on student behaviour collated through computers - an interesting alternative. It deals with the reading of real books and reading achievement, for which variables a very large amount of computerised data was available – on 852,295 students in 3243 schools. Reading achievement was measured pre-post with STAR Reading, a computerised item-banked adaptive norm-referenced test of reading comprehension. IF came from the Accelerated Reader (AR), which measures understanding of independent reading of real books the student has chosen by a quiz. Results showed higher IF was related to higher achievement. Neither IF nor reading achievement related to socio-economic status. Primary (elementary) schools had higher IF and achievement than secondary (high) schools. Females had higher IF and achievement than males. Students of higher reading ability implemented AR at a higher level, but did not gain in reading at a higher level. However, this computerised method of measuring IF with book reading showed limited reliability, no greater than methods emphasising teacher behaviour. In future, IF measures emphasising student response and those emphasising teacher behaviour need to be blended, although the latter will never generate the sample size of the former. This may be true of implementation fidelity in areas other than book reading.

AB - Measuring the implementation fidelity (IF) or integrity of interventions is extremely important, since without it a positive or negative outcome cannot be interpreted. However, IF is actually measured relatively rarely. Direct and indirect methods of measurement have been used in the past, but tend to over-emphasize teacher behaviour. This paper focuses on student behaviour collated through computers - an interesting alternative. It deals with the reading of real books and reading achievement, for which variables a very large amount of computerised data was available – on 852,295 students in 3243 schools. Reading achievement was measured pre-post with STAR Reading, a computerised item-banked adaptive norm-referenced test of reading comprehension. IF came from the Accelerated Reader (AR), which measures understanding of independent reading of real books the student has chosen by a quiz. Results showed higher IF was related to higher achievement. Neither IF nor reading achievement related to socio-economic status. Primary (elementary) schools had higher IF and achievement than secondary (high) schools. Females had higher IF and achievement than males. Students of higher reading ability implemented AR at a higher level, but did not gain in reading at a higher level. However, this computerised method of measuring IF with book reading showed limited reliability, no greater than methods emphasising teacher behaviour. In future, IF measures emphasising student response and those emphasising teacher behaviour need to be blended, although the latter will never generate the sample size of the former. This may be true of implementation fidelity in areas other than book reading.

KW - Evaluation methodologies

KW - Gender studies

KW - Improving classroom teaching

KW - Pedagogical issues

KW - Teaching and learning strategies

U2 - 10.1016/j.compedu.2017.09.009

DO - 10.1016/j.compedu.2017.09.009

M3 - Article

VL - 116

SP - 176

EP - 190

JO - Computers & Education

JF - Computers & Education

SN - 0360-1315

ER -