<img src=" https://secure.leadforensics.com/31510.png " style="display:none;">

The Digital Assessment News

30th October 2019

The science behind Adaptive Comparative Judgement

RM Results

As a method of assessment for learning, the benefits of using Adaptive Comparative Judgement (ACJ) are clear. It leads to a richer understanding of pupil performance, less time spent on the overall assessment process (marking and moderation), and clearer insight for both learners and teachers into the quality of work, from poor to good and, indeed, exceptional. Deployments of ACJ in schools, colleges and universities, both in the UK and overseas, have proven that its application leads to improved learner attainment.

So how exactly does it work? Is the process of comparing pairs of student work more effective than marking each piece of work individually? If so, what is the scientific evidence to support this?

The concept of comparative judgment emerged in the 1920s through the work of psychometrician Louis Thurstone. Thurstone studied how people measured the severity of crimes and found that, whilst they struggled to evaluate the severity of a crime in isolation, they could easily compare two crimes and say which was more serious. From his work, he formulated the ‘Law of Comparative Judgement’, which found that humans are more accurate, exact and reliable when comparing two stimuli, rather than passing absolute judgement on just one stimulus.

In an assessment context, rather than marking individual pieces of work against a set of marking criteria, comparative judgement (also known as ‘pairwise marking’) allows assessors to compare two pieces of work side-by-side and decide which is better. This is done multiple times by multiple assessors to ensure a high level of reliability. Adaptive Comparative Judgement makes this process even more efficient, as an algorithm adapts how often each piece of work needs to be seen and judged based on the results of previous rounds of comparison. This helps achieve reliable professional consensus on the rank order of all of the pieces of work being assessed much more quickly.

The comparison process – together with the collaborative, multi-assessor approach that allows comparisons to be made on a national and international scale – improves teachers’ assessment skills, and enables teachers and learners alike to understand how criteria for success is interpreted. In a recent study conducted by Dr Eva Hartell and Professor Inga-Britt Skogh, looking at the use of comparative judgement in Swedish technology education, the participating teachers reported that they liked being exposed to work other than just their own students’. It made them consider their classroom practices, and refine how they articulate criteria for success – ultimately benefitting their own pupils.

For learners, ACJ can help learning goals to be understood more clearly. By going through a process in which teachers can provide more clarity on what they expect of their students, and what good work looks like, learners can understand what direction they need to move in. Previous deployments of ACJ have shown it to be particularly beneficial for lower achievers, helping to bridge the gap between those at the top and bottom of the class.

When used in a peer-to-peer assessment context, ACJ can also offer clarity of learning goals, allowing students to identify for themselves what specific components comprise a better piece of work. In a study by Seery et al (2011), conducted by the Irish Technology Education Research Group (TERG) and the University of Limerick, technology education students were asked to participate in a comparative judgement peer review process before final assessment. Afterwards, the students reported that the intervention improved their understanding of the technology, and was more effective than referring to rubrics.

In summary, the human brain is more capable of assessing and forming valid judgements when it is comparing two stimuli. Within education, ACJ can play a crucial role in assessment for learning, helping teachers clarify, validate and moderate their own interpretations of ‘good’ and shed light on areas they may need to focus their pupils on. For learners, ACJ can help to demystify learning, identifying and paving the way to success.

Download our case study to find out how 14 primary schools collaborated to improve their formative assessment of writing using Adaptive Comparative Judgement.

 

© 2019 RM Education Ltd. All rights reserved.