Some recommendations for developing multidimensional computerized adaptive tests for patient-reported outcomes

Niels Smits (Lead / Corresponding author), Muirne C. S. Paap, Jan R. Böhnke

    Research output: Contribution to journalComment/debate

    3 Citations (Scopus)
    111 Downloads (Pure)

    Abstract

    Purpose: Multidimensional item response theory and computerized adaptive testing (CAT) are increasingly used in mental health, quality of life (QoL), and patient-reported outcome measurement. Although multidimensional assessment techniques hold promises, they are more challenging in their application than unidimensional ones. The authors comment on minimal standards when developing multidimensional CATs.

    Methods: Prompted by pioneering papers published in QLR, the authors reflect on existing guidance and discussions from different psychometric communities, including guidelines developed for unidimensional CATs in the PROMIS project.

    Results: The commentary focuses on two key topics: (1) the design, evaluation, and calibration of multidimensional item banks and (2) how to study the efficiency and precision of a multidimensional item bank. The authors suggest that the development of a carefully designed and calibrated item bank encompasses a construction phase and a psychometric phase. With respect to efficiency and precision, item banks should be large enough to provide adequate precision over the full range of the latent constructs. Therefore CAT performance should be studied as a function of the latent constructs and with reference to relevant benchmarks. Solutions are also suggested for simulation studies using real data, which often result in too optimistic evaluations of an item bank's efficiency and precision.

    Discussion: Multidimensional CAT applications are promising but complex statistical assessment tools which necessitate detailed theoretical frameworks and methodological scrutiny when testing their appropriateness for practical applications. The authors advise researchers to evaluate item banks with a broad set of methods, describe their choices in detail, and substantiate their approach for validation.

    Original languageEnglish
    Pages (from-to)1055-1063
    Number of pages9
    JournalQuality of Life Research
    Volume27
    Issue number4
    Early online date23 Feb 2018
    DOIs
    Publication statusPublished - Apr 2018

    Fingerprint

    Psychometrics
    Benchmarking
    Calibration
    Mental Health
    Quality of Life
    Research Personnel
    Guidelines
    Patient Reported Outcome Measures

    Keywords

    • Computerized adaptive testing
    • Item bank
    • Multidimensional item response theory
    • Patient-reported outcomes
    • Quality of life

    Cite this

    @article{d5bfc557535941d9bc4edeedc4080873,
    title = "Some recommendations for developing multidimensional computerized adaptive tests for patient-reported outcomes",
    abstract = "Purpose: Multidimensional item response theory and computerized adaptive testing (CAT) are increasingly used in mental health, quality of life (QoL), and patient-reported outcome measurement. Although multidimensional assessment techniques hold promises, they are more challenging in their application than unidimensional ones. The authors comment on minimal standards when developing multidimensional CATs.Methods: Prompted by pioneering papers published in QLR, the authors reflect on existing guidance and discussions from different psychometric communities, including guidelines developed for unidimensional CATs in the PROMIS project.Results: The commentary focuses on two key topics: (1) the design, evaluation, and calibration of multidimensional item banks and (2) how to study the efficiency and precision of a multidimensional item bank. The authors suggest that the development of a carefully designed and calibrated item bank encompasses a construction phase and a psychometric phase. With respect to efficiency and precision, item banks should be large enough to provide adequate precision over the full range of the latent constructs. Therefore CAT performance should be studied as a function of the latent constructs and with reference to relevant benchmarks. Solutions are also suggested for simulation studies using real data, which often result in too optimistic evaluations of an item bank's efficiency and precision.Discussion: Multidimensional CAT applications are promising but complex statistical assessment tools which necessitate detailed theoretical frameworks and methodological scrutiny when testing their appropriateness for practical applications. The authors advise researchers to evaluate item banks with a broad set of methods, describe their choices in detail, and substantiate their approach for validation.",
    keywords = "Computerized adaptive testing, Item bank, Multidimensional item response theory, Patient-reported outcomes, Quality of life",
    author = "Niels Smits and Paap, {Muirne C. S.} and B{\"o}hnke, {Jan R.}",
    note = "No funding info",
    year = "2018",
    month = "4",
    doi = "10.1007/s11136-018-1821-8",
    language = "English",
    volume = "27",
    pages = "1055--1063",
    journal = "Quality of Life Research",
    issn = "0962-9343",
    publisher = "Springer Verlag",
    number = "4",

    }

    Some recommendations for developing multidimensional computerized adaptive tests for patient-reported outcomes. / Smits, Niels (Lead / Corresponding author); Paap, Muirne C. S.; Böhnke, Jan R.

    In: Quality of Life Research, Vol. 27, No. 4, 04.2018, p. 1055-1063.

    Research output: Contribution to journalComment/debate

    TY - JOUR

    T1 - Some recommendations for developing multidimensional computerized adaptive tests for patient-reported outcomes

    AU - Smits, Niels

    AU - Paap, Muirne C. S.

    AU - Böhnke, Jan R.

    N1 - No funding info

    PY - 2018/4

    Y1 - 2018/4

    N2 - Purpose: Multidimensional item response theory and computerized adaptive testing (CAT) are increasingly used in mental health, quality of life (QoL), and patient-reported outcome measurement. Although multidimensional assessment techniques hold promises, they are more challenging in their application than unidimensional ones. The authors comment on minimal standards when developing multidimensional CATs.Methods: Prompted by pioneering papers published in QLR, the authors reflect on existing guidance and discussions from different psychometric communities, including guidelines developed for unidimensional CATs in the PROMIS project.Results: The commentary focuses on two key topics: (1) the design, evaluation, and calibration of multidimensional item banks and (2) how to study the efficiency and precision of a multidimensional item bank. The authors suggest that the development of a carefully designed and calibrated item bank encompasses a construction phase and a psychometric phase. With respect to efficiency and precision, item banks should be large enough to provide adequate precision over the full range of the latent constructs. Therefore CAT performance should be studied as a function of the latent constructs and with reference to relevant benchmarks. Solutions are also suggested for simulation studies using real data, which often result in too optimistic evaluations of an item bank's efficiency and precision.Discussion: Multidimensional CAT applications are promising but complex statistical assessment tools which necessitate detailed theoretical frameworks and methodological scrutiny when testing their appropriateness for practical applications. The authors advise researchers to evaluate item banks with a broad set of methods, describe their choices in detail, and substantiate their approach for validation.

    AB - Purpose: Multidimensional item response theory and computerized adaptive testing (CAT) are increasingly used in mental health, quality of life (QoL), and patient-reported outcome measurement. Although multidimensional assessment techniques hold promises, they are more challenging in their application than unidimensional ones. The authors comment on minimal standards when developing multidimensional CATs.Methods: Prompted by pioneering papers published in QLR, the authors reflect on existing guidance and discussions from different psychometric communities, including guidelines developed for unidimensional CATs in the PROMIS project.Results: The commentary focuses on two key topics: (1) the design, evaluation, and calibration of multidimensional item banks and (2) how to study the efficiency and precision of a multidimensional item bank. The authors suggest that the development of a carefully designed and calibrated item bank encompasses a construction phase and a psychometric phase. With respect to efficiency and precision, item banks should be large enough to provide adequate precision over the full range of the latent constructs. Therefore CAT performance should be studied as a function of the latent constructs and with reference to relevant benchmarks. Solutions are also suggested for simulation studies using real data, which often result in too optimistic evaluations of an item bank's efficiency and precision.Discussion: Multidimensional CAT applications are promising but complex statistical assessment tools which necessitate detailed theoretical frameworks and methodological scrutiny when testing their appropriateness for practical applications. The authors advise researchers to evaluate item banks with a broad set of methods, describe their choices in detail, and substantiate their approach for validation.

    KW - Computerized adaptive testing

    KW - Item bank

    KW - Multidimensional item response theory

    KW - Patient-reported outcomes

    KW - Quality of life

    UR - http://www.scopus.com/inward/record.url?scp=85044774040&partnerID=8YFLogxK

    U2 - 10.1007/s11136-018-1821-8

    DO - 10.1007/s11136-018-1821-8

    M3 - Comment/debate

    C2 - 29476312

    VL - 27

    SP - 1055

    EP - 1063

    JO - Quality of Life Research

    JF - Quality of Life Research

    SN - 0962-9343

    IS - 4

    ER -