I ask students ‘why?’ (more often) in the era of ChatGPT

(NOTE: Patricia is opening up her MGRK1010 classes during Session 1 to allow other teaching staff to join a live class and observe teaching in action as part of the Open for Observation program.)

---

The explosion of Generative AI tools left most educators wondering about the impact of these tools on assessment design. For me, the advent of generative AI presented an unexpected opportunity to shift away from conventional language assessment exercises. Engaging with AI has opened doors to developing fresh and non-traditional opportunities for authentic language learning, teaching and assessment in the realm of lifelong learning.

The holy grail of learning: critical thinking

Why move away from traditional language assessment exercises?

While traditional grammar exercises such as ‘Identify and correct grammatical errors’ and ‘Fill in the blanks with the correct form of verbs/nouns’ are beneficial, they have limitations, especially in today’s AI-driven world. Here’s why:

• Limited contextual understanding: focus on isolated grammar points hinders application in real-life situations.
• Rote memorisation: memorizing rules without understanding inhibits deep learning.
• Lack of creativity: little room for creative expression, hindering meaningful and engaging communication.
• Insufficient development of intercultural skills: fail to address cultural awareness needed for effective language use.
• Demotivation: repetitive exercises can be demotivating without clear relevance to real-life communication.
• Not reflective of authentic language use: fails to prepare students for the complexities of genuine language use. Real communication involves understanding pragmatics (the use of language in social contexts), sociolinguistics (how language varies and changes in social groups), and discourse strategies (ways to structure spoken or written language).

My 2023 assessment trial to critically engage students with their submitted work

So, I re-designed online assessment tasks, both online quizzes and written assignments, by adding a part that would ask students to reflect critically on a part of their submitted work (written assignments) or to ask students to justify a provided correct answer (online quizzes) or even to correct ChatGPT’s wrong answers (as part of the interactive feedback in class). This re-design goal aligned with the objective that ‘assessment tasks should provide meaningful feedback to the students that will subsequently develop their critical thinking, judgement and decision-making. These are components of the (learning) process that AI is less able to simulate’ (Assessment Reform for The Age of Artificial Intelligence, 2023, p.4). At the same time this assessment design would minimise plagiarism.

Adding metalanguage to rubrics

Metalanguage is the language we use to talk about language. This concept, as described by McArthur in 1992, is a way of stepping back and thinking about how language works, rather than just using it. When learners engage with metalanguage, they’re tapping into their explicit knowledge – the kind of knowledge they are consciously aware of and can articulate. This process, as highlighted by researchers like Hulstijn (2005) and Roehr (2006), involves bringing what they know to the forefront of their minds, discussing it and reflecting on the learning process. In the language learning process, metalanguage becomes a crucial player, and it comes into play when learners are asked to justify why something is correct or incorrect.

Thus, I revised the rubrics for reflective writing, online quizzes and the final online test. This involved the incorporation of explicit assessment criteria related to metalanguage by the following criterion under the category of ‘Metalanguage of grammar and syntax’: assessment of accuracy, meaningfulness and appropriacy: demonstrates awareness of correct language use in grammatical and syntactical choices.

View the Reflective writing rubric below (scroll to view or click the arrow to download).

Familiarising students with the concept of metalanguage by asking them ‘why?’

I provided a concise explanation of the concept of metalanguage. This introductory phase aimed to familiarise students with the term, setting the groundwork for its application throughout the course. To facilitate a better understanding, I employed illustrative examples each time we delved into a new grammatical or syntactical aspect within the context of Modern Greek during our class sessions. This strategic approach served a dual purpose: first, to ensure that students grasped the theoretical underpinnings of metalanguage, and second, to seamlessly integrate its practical application into their understanding of language phenomena.

Asking ‘why?’ in language choices in students’ essays

Students were asked in their written assignments to reflect in English on their Greek grammatical, syntactical, and/or lexical choices from their submitted Greek assignment. This enhanced their critical understanding of the language phenomena, but also their reflections on their learning process.

An example of an effective student’s reflection on a grammar choice is the following (16/10/23, 02:50):

I used the superlative of the adjective ‘τις πιο δύσκολες’ (meaning the most difficult) instead of the positive form ‘τις δύσκολες’ (meaning the difficult) in order to make the degree of the difficulty, that is the attribute of the noun, more emphatic. I couldn’t use the comparative form in this sentence as I was not comparing the difficulty of two things. It’s feminine accusative plural has to agree with the noun ‘έννοιες’. The structure is definite article + πιο + adjective.

Asking ‘why?’ in language problem-solving exercises in online quizzes

An example of a beginner level exercise from the first online quiz that examines metalanguage is the following:

“Θέλεις την μεγάλη τηλεόραση;” (=Do you want the big TV?) –“Ναι, την μεγάλη.” (Yes, the big one). Explain in English why ‘την μεγάλη’ (accusative) is used in the answer instead of ‘η μεγάλη’ (nominative).

In other words, the students in this metalinguistic question had to explain the use of the accusative case rather than forming it mechanically.

I tested this exercise on ChatGPT multiple times. All responses presented by ChatGPT were verbose, redundant, and incorrect. Due to the chatbot’s assertive tone, students might have presumed accuracy, even when it was providing an inaccurate explanation. Bowman (2022) notes, ‘There are still many cases where you ask [ChatGPT] a question, and it’ll give you a very impressive-sounding answer that’s just dead wrong’.

Asking ‘why?’ ChatGPT is wrong while giving feedback in class

Professor Rorden Wilkinson in the workshop ‘Education for Success’ (April 2023) said that: “Feedback gives students a genuine transferable experience”. Significant feedback plays a crucial role in education, serving as a pivotal stage that provides teachers with the unique opportunity to inquire about students’ reasoning behind their answers and allows students to reflect on the learning process.

Real-time feedback in class also provides the opportunity to train the students on the effective interaction of human critical thinking and AI as AI lacks metalanguage skills. Thus, while giving feedback on the students’ submitted answers in the online quizzes, I challenged their understanding by exposing them to ChatGPT’s incorrect answers to engage them in a discussion where they would loudly articulate their understanding of language. An example of an exercise designed to assess students’ understanding (and not the knowledge) of the Greek verb conjugation system is the question, “Which combinations of Greek personal pronouns match with the verb μένουμε (=we live)?”. This exercise focuses not on verb formation, but rather on the understanding of pronoun-verb pairings. I tested this question on ChatGPT multiple times, and the answer was incorrect every time.

Takeaway:

Critical thinking and evaluation skills are a lot more important now than pre-ChatGPT. We can easily imagine a time when AI tools will be doing the ‘first draft’ of many things, and the added value of humans would be to evaluate the output and improve on it. The small baby steps that I took last year are aimed at helping students develop AI literacy skills and better prepare them for the AI-enhanced world where they need to be more critical than ever of what they see in front of them.

Acknowledgements:
I would like to express my sincere gratitude to Dr Bénédicte André for providing valuable insights at the challenging onset of Session 1 2023. I am also thankful to Dr Sarah Keith for inviting me to present the application of ChatGPT in my units at the MCCALL education workshop on AI tools in teaching and assessment where I received effective feedback from my colleagues. Additionally, I wish to convey my deep appreciation to Dr Olga Kozar for her invaluable feedback during the drafting process of this article.

Banner image: Photo by metamorworks on Shutterstock
Lightbulb image: Photo by Diego PH on Unsplash
Thinking image: Photo by patpitchaya on Shutterstock