If we can use AI to write essays, should we find another way to assess?

Cast your mind back to late November 2022 when ChatGPT took over the internet and caused a frenzy across industries. There is no doubt that Artificial Intelligence (AI) and the likes of ChatGPT will change the world as we know it. What is up for debate however, is whether this is for the better. And as they say, there are two sides to every coin...

New York City's education board took an adverse stance and almost immediately banned ChatGPT from use in schools. Others were quick to follow, and the reaction from educators across the world was divided. Those that were positive about the technology saw value in opportunities to support learning, while others were concerned with the threat of cheating increasing and becoming more difficult to detect.

In response to this LinkedIn post by Sam Shead, BBC News reporter, one educator said, “I have been playing with this tool and can see many positive opportunities with supporting learning. Potentially to support writing skills and critical narratives” (Suzanne Fergus, Associate Professor, University of Hertfordshire).

Another said, “In higher education this development is causing some concerns, after testing it we found that the results wouldn’t pass one of our university assignments, it lacks critical analysis and appropriate referencing. However, we need to keep an eye on how this develops.” (Paula Shaw, Associate Professor of Teaching and Learning at the University of Derby).

But what is ChatGPT?

ChatGPT is a type of artificial intelligence (AI) technology known as a natural language processing (NLP) model, created by the AI research and development company OpenAI. The acronym ChatGPT, stands for chat-based Generative Pre-trained Transformer. It is a powerful tool that can be used by anyone, and in a variety of ways.

You must really see it to believe it, and you can, if it’s not too busy when you decide to log on. ChatGPT (at the time of this writing) is freely accessible via the web at chat.openai.com. To use it, you simply create an account, login, and begin typing your prompt into the ChatGPT chat box.

The system uses machine learning algorithms to assess and comprehend written or spoken language. It will then generate responses based on that input, so try to be simple and succinct with your prompt. That way you will get the best response.

Ultimately, the software can be used to gather, modify, refine, summarise, and even validate swathes of information quickly and in a relatively precise manner. Although, it is important to remember that ChatGPT is simply a large neural net that’s been trained to predict the next word. Despite the hype, it still comes with the same challenges seen in other large language models (that’s right, there are others), one being: it may make up facts.

Even so, ChatGPT has got some educators worried.

“Its output only has to be better than the 50% level to be tempting to a student who would otherwise be at risk of failing. Plagiarism detectors... are of very little use here because the text from ChatGPT is created from scratch, not copied from existing sources. For several years now in universities we’ve been fighting contract-cheating (students who pay someone else to write a coursework essay/report for them, and then submit it as if they had written it); in this context, ChatGPT could be characterised as offering easily-accessible and hard-to-detect mechanised contract-cheating, for zero fee.” (Dave Cliff, Professor of Computer Science, University of Bristol).

Professor Dave Cliff makes a good point, so we asked ChatGPT: If we can use AI to write essays – should we find another way to assess? This was its response:

“The use of AI to write essays raises questions about the effectiveness of current methods of assessment. If AI can produce written responses that are indistinguishable from those written by humans, it may be difficult to use written essays as a reliable means of assessing student understanding and abilities.

One potential solution could be to focus on other forms of assessment, such as oral exams, problem-solving tasks, or group projects, which may be less susceptible to being replicated by AI. Additionally, other methods like peer-review, self-assessment, and formative assessments can also be used to evaluate student progress.

Another possible solution could be to use AI-detection tools, which could detect if the work submitted is generated by AI or not.

Ultimately, the use of AI in essay writing may necessitate a shift in how we think about assessment, and a re-evaluation of the types of tasks and activities that can accurately measure student understanding and abilities.” (ChatGPT).

In addition to the suggestions generated, another approach to assessing could be correlating students’ results from a particular piece of work with other evidence. By building a broader picture of a student’s ability in this way, it could help to identify pieces of work that appear to be outliers.

In conclusion

AI is not new technology – it has been around since the 40s, when Warren McCulloch and Walter Pits proposed a model of artificial neurons. Donald Hebb then demonstrated a rule for modifying the connection strength between neurons in 1949. In 1950, Alan Turing pioneered machine learning in the form of a test that can check a machine's ability to exhibit behaviour equivalent to that of human intelligence, now affectionately referred to as the Turing test.

The point being that whilst the tech isn’t new, how we use it could be.

In an interview for Time magazine, Mira Murati, chief technology officer at OpenAI said: “Right now, it’s in the research review stage, so I don’t want to speak with high confidence on what problems it is solving. But I think that we can see that it has the potential to really revolutionize the way we learn”.

She continued by stating that “This is a unique moment in time where we do have agency in how it shapes society. And it goes both ways: the technology shapes us and we shape it.”

The way we think about assessment is changing, and AI is one of the key players of influence as we move forwards. But if there is a way to capitalise on these advancements to provide higher quality support and educational opportunities, then why not?

So, let's get back to work.

Further reading:

Why do we tolerate human over machine error?

FE News | Artificial intelligence: What’s the impact for educators?

How Large Language Models Will Transform Science, Society, and AI (stanford.edu)

The emerging types of language models and why they matter | TechCrunch