Digital Assessment Adoption: changing hearts and minds from the experience of CUPA and RM

Digital Assessment Adoption: changing hearts and minds from the experience of CUPA and RM

RM Assessment was pleased to sponsor the e-Assessment international conference and awards last week (10-12 June). We took a novel approach to the event and invited some of our clients along to join us to discuss their experiences of digital assessment.

Following on from his webinar on ‘Schools engagement with digital assessment’, Ian Castledine (Head of Proposition at RM) was joined by Cerys Burcher, Head of Digital Product Development at Cambridge University Press and Assessment (CUPA) and Katherine Lancaster (Head of Product at RM, and former computer science teacher) to talk about life on the frontlines of digital adoption. RM and CUPA have worked together for many years, and digital adoption is an exciting part of their journey.

Schools’ engagement with digital assessment and the transformation of the future of qualifications is underway, but organisations will be at different stages on their journey to adopt digital assessment.

Cerys, Katherine and Ian spoke about some of the emerging themes of organisations adopting digital:

Capture hearts and minds

Considering the perceptions of the different stakeholder groups, Cerys pointed out that it is important to assess how ready and willing they are, address concerns, highlight benefits and decide how best to support the transition to digital assessment.

Some teachers are keen to embrace digital, but others are anxious. Whilst they see the opportunity, they also see challenges. For Katherine, it is about how to get early adopters onboard. Looking across the curriculum, digital assessment can be adopted:

Adopt digital by subject

Some subjects are easier than others to digitalise, such as textual or essay-based subjects. Cerys mentioned computer science as a subject that lends itself well to digital, but it is currently being assessed on paper. Other subjects are harder to transform, in order to make sure you are actually testing what you think you are testing and are not just implementing ‘paper under glass’.

Transitioning a science subject onto the screen is more than just putting in a drawing tool and hoping for the best. The exciting thing is when new forms of assessment that are more relevant and authentic become unlocked using technology.

Adopt digital by candidate experience

For some subjects, candidate experience is driving change. In England, we are seeing the adoption of digital for subjects that make logical sense, like computer science, where digital assessment can add additional value to the candidate experience such as being able to create and execute code under exam conditions. It adds authenticity, Cerys pointed out.

“I know that in Cambridge, there's a real push for 21st century skills and competences. How is that having a bearing on the nature of assessment and the way that that you see that emerging in the future?”, asked Ian.

Cerys replied that there are multiple tracks in terms of timeline. “We are moving towards on-screen assessment in the near-term, which is about what you can do in terms of benefits, but in the longer term, it’s about what we are doing with assessment as a sort of concept. What are we trying to assess? Is it about knowledge as much as it used to be when you know you can Google anything you like any time of day? Cambridge talks a lot about 21st Century competencies or skill sets around critical thinking, collaboration, communication, all these sorts of skill sets, competencies that students or 21st century kids are going to need in higher education and in their careers.”

“The assessment landscape does drive the curriculum,” explained Katherine. “So, for example one of the reasons why my kids that I taught recently actually had some Excel skills embedded into the year seven eight and nine curriculums was because we did the BTEC that has a big unit all around Excel. A lot of schools don't teach basic IT literacy skills like Excel, and therefore they get into the office space, and they ask, “What’s a spreadsheet?” Katherine also shared that although there has been a huge focus in primary schools around algorithmic thinking, year sevens don’t actually know how to work a mouse or move files around on a computer – basic digital literacy skills. She said that exams need to be crafted to allow those skills to be assessed.

Cerys agreed. The job of the exam board is to take the curriculum and develop the teaching and learning, building an entire pathway, not just to start or finish with the assessment. Whether it’s GCSE or A Level, or right the way back to early years, the pathway needs to be of benefit to the learners.

The way to get schools excited about digital adoption would be for them to start to see the appreciation of the core skills coming into the curriculum and start to get the recognition from higher education and careers as well for developing assessments that will be bringing those skills along that are going to be a benefit to the learners in the long run.

Assessment designed for digital

In addition to adoption by subject and candidate experience discussed at the eAA conference, we are now seeing a global shift to assessments which are designed for digital right from the outset, rather than being constrained to capabilities that are only possible on paper. As we get more familiar and comfortable with the use of digital assessments, we are expecting that designed for digital will become ever more relevant as we look at subjects that are currently harder to digitise.

Change management

As we adopt new capabilities, particularly within general qualifications, we start to see challenges with early adopters, who particularly embrace change and are driving it forwards, and those who are in danger of being left behind. CUPA recognises that we are in a world where some learners are ready for digital assessment, and some aren’t. They are looking at what they can continue to offer on paper alongside on-screen exams, but also to think about digital only. The dual running is important.

Across the UK, there are differences from school to school. Katherine compared her experience of teaching in the Highlands of Scotland where they had gone completely digital and every child got either a Chromebook or some other device, to the school she had just left in Wolverhampton where they were using exercise books and photocopies. There is nothing disparaging about schools who are still working on paper, as it is very common, but Katherine made the point that she was giving out GCSE computer science past papers where learners were having to write out Python and have the work peer reviewed, but then they would work in an online environment.

School readiness for digital assessment

Schools are going to be nervous about whether they have the right technology, the right tools. Some are ready and some are not. Thinking about preparedness and equipment, it even comes down to whether there are enough power sockets, enough computers or whether the computer labs are set up in the right way. A lot of computer labs were set up for collaborative learning, so they are not set up for assessment. It is part of the challenge that we suddenly have laptops in halls requiring power and emergency backups with contingencies in case anything goes wrong. Some exam officers and School Leadership Teams (SLTs) are keen and willing to work around and adopt, but others are going to wait and see how it goes first.

CUPA doesn’t have all the answers, but it is walking schools through the early adopter programme and has strategies and mechanisms in place to address issues, so everyone feels comfortable, confident and ready adopting digital. Working with schools has been part of its process for a very long time, even pre-digital. It consults with schools around new processes, how subjects are taught and how they evolve.

Teacher engagement

The pandemic accelerated change in a way that hadn’t been experienced in the education sector before. There was a huge amount of formative assessment going on and acceleration around how teachers could use digital assessment embedded in teaching, which brought CUPA ideas. It helped it to build a bi-directional relationship where teachers were telling it how they wanted to work, so they gained customer, or rather teacher, insight. “There’s nothing that annoys teachers more than experts coming in and telling them how to teach,” commented Katherine, “Teacher insight and that teacher research is critical.”

The teaching community is quite innovative. Some use ChatGPT to come up with assessment questions and using Microsoft forms, even in the schools where they are mainly using pen and paper, observed Katherine. Teachers and students are learning from each other. It’s a real collaboration.

Learner engagement

Thinking about the learners themselves, some are highly technically savvy, capable and equipped, but there are many who aren’t confident. Ian asked Cerys how CUPA approached that landscape. She responded that proportions are higher in terms of student engagement and enthusiasm, as they are readier. CUPA had a sample from the mock service, of which 92% of them preferred typing to handwriting exams.

According to independent research carried out on behalf of RM Assessment, many learners feel more relaxed and less anxious with digital assessments, preferring typing to writing. From a group of over 2,400 learners, 59% prefer digital assessment compared to just 22% for pen & paper, almost 3:1. Perception of accuracy is marginally higher for digital (68%) compared to pen and paper assessment (65%). The difference was more marked among learners who have had a previous positive experience with digital assessment. There, 72% agree for pen and paper and 79% agree for digital.

Cerys said that the high proportion of students with a preference for digital depends on ensuring that nothing goes wrong and that they are familiar with the system before the exam. Unusually, we have the situation where often, learners are more willing to adopt digital and are pushing for the teachers to do the same, which is an interesting situation because in a school, it’s usually the other way around.

CUPA is working out how to help teachers and learners understand how to apply critical thinking to what they find online that has been written by a machine, how they understand, how they analyse and to make sure that they understand about cheating mechanisms and AI.

More relevant and more authentic assessment content

More relevant and more authentic assessment methods are a very good way of driving user engagement. Katherine again used the example of computer science to say that the ability to type in a bit of Python and see it run, compile, execute and even give an error message back in a digital assessment is much more realistic and what developers do in the real world. Students are learning authentic 21^st century skills, which they prefer. It also makes it easier and less frustrating for markers, as they are assessing the syntax, and understanding of the constructs as opposed to whether someone has forgotten to put in a semicolon.

Ian asked Cerys if introducing digital was opening new opportunities that paper-based exam systems haven’t been able to provide. She explained that there had been a huge amount of learning during the piloting phase. Katherine asserted that tech, collectively as a group, could overcome challenges.

CUPA looks at things through different lenses, in the near term – what is problematic now in the paper space, e.g. things like having videos or audio playing in the corner or language listening, which are obvious pains – and at things at the other end of the spectrum that could be transformational, like trust and confidence.

Ian summarised the key takeaways from the talk for encouraging the adoption of digital assessment in schools.

Work closely with stakeholders to build confidence and trust

Work closely with all of your stakeholders – students, educators, parents, regulators and government bodies – to build confidence and trust.

From her perspective as Head of Product at RM, Katherine is excited about some of the innovative item types they are looking at and driving an authentic assessment experience.

Cerys closed by saying that, when looking at the future and what is coming, CUPA is excited about the innovation in assessment and what it can do in terms of how this impacts on student outcomes from the point of view of helping them develop skill sets and competencies which the assessment system doesn’t currently do, and how those competencies are connected to digital and how they live and work as students and in their careers as they go forward.

A big thank you to Cerys, Katherine and Ian for sharing their insights and experience around digital assessment adoption.

Looking to the future of digital assessment

For further information around digital assessment adoption: changing hearts and minds, download RM Assessment’s Guide: Encouraging the adoption of digital assessment in schools, published 14th June.