I got asked if a student could use Chat-GPT

Early February 2023, I was giving an exam for a course I teach on web data collection. The exam is a competence-based online oral exam, usually starting with a coding session before we move on to discuss specific problems related to a submitted task on web scraping.

On that day, I asked the student to write a regular expression for extracting a URL from a website. First reaction from the student: “Can I use ChatGPT?”

Well…

Knowing that I’ve been working with generative models for at least 5 years, and that I know quite well GPT-3 - which our research group recently used to generate a speech for Swiss Bundespräsident Ignasio Cassis - I am very well aware of the potential and the limitations of ChatGPT. Having participated in a recent evaluation project on e-assessment forms, I was also painfully aware that they are many more ways to cheat at exams than to use ChatGPT.

So, full of confidence, I nodded: “go ahead”.

The student typed my question in, and his face on the screen turned happy, annoyingly really happy, just before he pasted the following into the online python notebook:

'https?://\S+'

I felt surprisingly relieved (you never know, if the the set generation temperature could generate some thing "unexpected"): the answer was bad, almost really bad! Nevertheless, it was a good start to ask him further questions.

“- What are the cases this doesn't cover?” I asked. http://. is a valid URL for that regular expression, so from there I could ask the cases where I could see what are the basic expressions the student knew and how he approached the task. It was not faster but I thought I could show him that ChatGPT can be a good starting point.

However, it won't indicate its sources, it won't start with the most precise answer, it can't store all the information it saw, nor reproduce it from every perspective accurately, and it can be just plain wrong : so if you are unlucky, ChatGPT might be a waste of time altogether.

I am happy, though, that the student was open enough to disclose his use of ChatGPT.

I love to automatize processes (I hate to repeat myself), create new views and test new approaches. In that regard, ChatGPT is something I always hoped for, but we need to adapt when teaching and assessing students, to focus more than ever on problem-solving and critical thinking skills.

Using ChatGPT in assessment such as oral exams or even in programming exams can move the questions to a higher level of reflection. But we also have to determine which basic knowledge the student should have, what can be derived in the conversation with ChatGPT and how efficient to evaluate the learning of a student? These questions will keep us busy for several more years.

And to those who’re afraid that ChatGPT’ omnipotence is likely to demotivate students to learn the “stuff” themselves, well… fear not: in the aftermath of this exam, the very same student just decided to enrol in my natural language processing class.