Within-school segregation of pupils by attainment remains prevalent, despite evidence that these practices detrimentally impact outcomes for those in low attainment groups. This article explores the hypothesis that ‘ability grouping’ by setting impacts pupil self-confidence, precipitating a self-fulfilling prophecy. Survey data from 11,546 11/12 year old pupils in ‘ability sets’, and individual interviews and focus groups with 66 pupils, are drawn upon to explore this hypothesis. We find a significant correlation between perceived set placement and self-confidence in the set subject. More importantly, we also find a correlation between set placement and general self-confidence in learning. Pupils’ qualitative responses illustrate how setting can promote self-fulfilling prophecy, and worrying evidence of internalisation of ‘ability’ labels among pupils.