Enhancing student support through pre-course predictive analysis of aptitude data

Support for new students in higher education is of primary importance in addressing retention and progression issues. For Computer Science courses, this has proved to be a problem for larger classes, where some students can have difficulties in accessing required support, leading to disengagement and unrealized potential. While some attrition is normally expected, the high rate will usually include a proportion of students who have an aptitude for the subject but disengage after a few weeks of course commencement. This is particularly evident in courses where programming is a major constituent.

The use of aptitude testing can potentially help with this and has been successfully deployed to support the admissions process for the postgraduate (conversion course) master’s in software development at Queen’s University Belfast (QUB). However, such testing is not currently used in admission to Computer Science undergraduate degree pathways at QUB, and given the attrition rate, there is justification for post-enrolment analysis. This is of particular importance as the recent Covid 19 pandemic has had a significant impact on the admissions process, with public examination within the education system for Northern Ireland replaced (albeit temporarily) with teacher-managed assessment to determine the grades used for university entry.

Consequently, this work considers assessment strategies for students enrolled on undergraduate pathways with significant programming content, aimed at early (predictive) identification of groups who may benefit from additional support. Three distinct cohorts are considered in this study:
1. Students with no prior programming qualifications or experience
2. Students with no prior qualification but with some experience (claimed) in programming
3. Students with prior qualifications in programming

New students from all cohorts were encouraged to participate in the general aptitude tests currently applied in the postgraduate admission process. While these tests do not assume previous programming experience, they provide a baseline for language analysis, symbolic reasoning and application of logic. Furthermore, students from cohorts 2 and 3 were presented with an additional test, specifically aimed at assessing basic knowledge in procedural programming data, organization, constructs and techniques.

Performance and engagement with formative and summative assessments in the context of initial testing applied to all cohorts, together with initial qualifications profile is considered. Initial results indicate that standard aptitude testing is useful as a broad tool for identifying the weakest group, but anomalies persist when comparing against students with measured or claimed experience in programming. This paper provides a statistical analysis on completion of introductory programming modules, contrasting general versus targeted testing in performance prediction and considers the potential impact of Covid 19 on assessing student grades prior to admission.