Reliability and validity of methods to assess undergraduate healthcare student performance in clinical pharmacology courses

Research output: Contribution to conferenceAbstractpeer-review

Abstract

Strategies to assess student performance include traditional closed-book, time-limited written examinations comprising single best answer (SBA), short answer and longer essay and case-based questions. During the COVID-19 pandemic, many assessments adapted to open-book, extended duration format. We compared reliability and validity of these differing approaches, in assessment of clinical pharmacology learning outcomes within medicine and dentistry. In both courses, assessment strategies map to professional learning outcomes, ensuring construct validity.
The second year dental cohort comprises 58+5.6 students annually (mean+SD, n=7) who undertake two written assessments: class test (30 open-ended very short answer questions) and paper (short answers, essay and case study). Both assessments are reliable: during 2014-2019, Cronbach’s alpha was 0.84+0.05 and 0.74+0.09 (n=6) for the class test and paper, respectively. The class test is taken mid-year whereas the paper focuses on the entire year’s content, increasing content validity. Correlation between closed-book class test and paper scores was 0.626+0.045 (n=6, p-value <0.001), inferring criterion validity. During COVID-19 restrictions, the paper was delivered using a 24-hour open-book format: Cronbach’s alpha was 0.65 and correlation with the closed-book class test completed earlier that year was 0.252 (p-value 0.064).
The first year medical cohort comprises 261+5.0 students annually (n=8). Clinical pharmacology is integrated within a larger course and contributed one section (5 questions) of a larger closed-book paper containing short answers and case studies during 2012-2016. Since 2017, clinical pharmacology has contributed 20 SBA questions within a larger mid-year class test and one section (3 questions) of a larger paper. Reliability of the paper correlated with number of questions: mean alpha was 0.79+0.06 and 0.66+0.10 for 5 and 3 questions, respectively. Mean alpha for the SBA paper was 0.74+0.002. Correlation between closed-book SBA and short answer scores was 0.625 (p-value <0.001). Performance in clinical pharmacology correlated with other sections of the paper (biochemistry 0.53, p-value <0.001; pathology 0.57, p-value <0.001). During COVID-19 restrictions, the paper was delivered using a 24-hour open-book format: correlation with a closed-book SBA completed earlier that year was 0.213 (p-value <0.001) and with the pathology section of the open-book paper (5 questions) was 0.30 (p-value <0.001). Alpha was 0.70 (5 questions).
Overall, our experience suggests that free response written assessments are at least as reliable as SBA for closed-book time-limited assessments. Reliability is correlated with number of questions used. Dependence on open-book extended duration assessments during the COVID-19 restrictions resulted in only modest reduction in criterion validity and reliability.

Original languageEnglish
Number of pages1
Publication statusPublished - 23 Jun 2022
EventInternational Assessment in Higher Education Conference - MacDonald Manchester Hotel, Manchester, United Kingdom
Duration: 22 Jun 202224 Jun 2022
https://ahenetwork.org/

Conference

ConferenceInternational Assessment in Higher Education Conference
Country/TerritoryUnited Kingdom
CityManchester
Period22/06/202224/06/2022
Internet address

Fingerprint

Dive into the research topics of 'Reliability and validity of methods to assess undergraduate healthcare student performance in clinical pharmacology courses'. Together they form a unique fingerprint.

Cite this