As the education industry seeks innovative assessment approaches to cater to the changing needs of the workforce, the interest in Comparative Judgement (CJ) continues to increase. To date, its potential to contribute to the assessment mix, especially in summative high stakes examination, has been hindered by an inability to scale effectively.  

An article from the Home of Assessment & Qualifications Insight in 2021 asks a simple question: ‘Could Comparative Judgement replace traditional exam marking?’ and concluded it as unlikely because of the workload needed to implement the Comparative Judgement process at the time. In the article, AQA provide an illustrative example focusing on the assessment of English Literature. Although there are several underlying assumptions behind the CJ figures used such as the time it takes to make the judgement and number of judgements needed per script, they provide an effective use case:  

These figures illustrate that the total hours needed for a traditional approach to CJ is five times greater than for traditional marking methods. Assuming the number of examiners available (4500) could not be increased and that the marking window (5 weeks) remains the same, the additional workload quickly becomes un-manageable for an individual. Markers, who are typically practicing teachers, currently complete on average 15 hours of marking. A switch to 75 hours of CJ judging is clearly impractical.  

So, what can be done? 

Introducing Comparative Judgement On-Demand 

The team at RM have been focused for some time on 3 key questions regarding Comparative Judgement to see if we can overcome the workload challenge.

• How can we make it more efficient? 

• How can we make it more predictable? 

• How can we make it have a much higher throughput and output? 

The answer, we believe, is in the creation of an ‘On-Demand' model.  

This approach allows us to deliver game changing efficiencies and would look something like this against the previous figures in Table 1: 

CJ On-Demand offers the potential to realise the numerous benefits of Comparative Judgement without increasing workload for markers.  

The benefits of Comparative Judgement have been well explored over many years. Here are a just a few examples:  

Increased reliability: Comparative judgment can provide more reliable results compared to traditional grading methods. By directly comparing items or performances, it focuses on the relative quality of items rather than absolute scores, leading to more consistent and accurate evaluations.  

Improved objectivity: Comparative judgment helps mitigate the inherent biases and preconceptions that can affect traditional grading. It shifts the focus from trying to assign absolute scores or grades to making relative comparisons, which minimises the influence of personal biases and allows for a more objective assessment.  

Enhanced efficiency: This method can be more time-efficient compared to traditional grading. Instead of assigning scores or grades to each individual item, evaluators only need to make comparative judgments between pairs of items. This streamlined process can save time and effort while still producing reliable results.  

Valuable feedback: Comparative judgment can generate valuable feedback for individuals being assessed. Rather than receiving a simple grade or score, they can gain insights into how their performance or work compares to others. This feedback can be more informative and actionable, helping individuals understand their strengths and areas for improvement.  

Now imagine these benefits at your fingertips, anytime you want, at a scale appropriate to your needs... 

Find out more 

We are working with partners around the world in a wide range of contexts who are as excited as we are. As we move into a new BETA phase, we will be able to offer the new On-Demand functionality even more widely and cannot wait to see the new use cases this encourages. 

If you are attending the e-Assessment Association conference next month, come along to our interactive session at 11:30 on 7th June to learn about how we are using the power of adaptive comparative judgement on-demand to empower learners and markers, and streamlines the assessment process at scale. This will be a hands-on session so that you can explore this technology for yourself.