Tips and tricks from the ALP on AI and writing

Tips and tricks

• Podcast Nooit meer schrijven?: Conversations about AI and writing

  As part of our project, we started a podcast series: Nooit meer schrijven? In each episode, Gea talks to a guest about one of the many topics related to AI and writing skills. Guests include Piek Vossen, professor of Computational Lexicology, who explains the Large Language Models that ChatGPT is based on; Christine Moser, Associate Professor of Organization Theory, who reflects on the role of writing in education; and Maaike van den Haak, who talks about the opportunities and threats that machine translation presents. Episodes are in Dutch (with a translated transcript) or English.

• The writing pie: Mapping writing skills

  HWriting an academic text is a complex cognitive task that requires not only the ability to produce complex and accurate sentences, but also a sophisticated knowledge of audience and genre conventions. And at the same time as thinking about the text itself, the writer needs to develop ideas, understand concepts, and engage with source materials. To conceptualise the writing process, we created a “writing pie”, with the different skills required to write an academic text displayed in each slice of the pie.

  Mapping writing skills this way gives a broader perspective on the value of writing: writing is not just about the end product, but it is also a tool for thinking, for understanding, and for putting thoughts into words. Another benefit of the writing pie is that it allows for a more detailed look at the potential issues that the new AI present to writing assignments. For this purpose, we added the outer ring, which indicate our assessment of the relative ability of AI to replicate these skills (at least on a surface level).

• Three things everyone needs to know about (generative) AI

  1. Input = output. Language models are trained on existing texts. This means that ChatGPT does not just make something up, but 'learns' from a huge database of texts. The more people have written about a topic, the better ChatGPT works. It also means that it is important to know what is in that database of texts, but this is precisely something that, for instance, OpenAI does not disclose.
  2. Language models are based on probability. ChatGPT calculates what the most likely next letter (and word) is based on what is already there. This means ChatGPT is especially good at common combinations of words and phrases. But it also means that it is very superficial: it produces combinations of words and sentences that are commonly used about a topic or in a specific type of text, but does not understand what it writes.
  3. Programming is human work. Even though ChatGPT looks like an intelligent computer, there is also a lot of human programming involved behind the scenes. For instance, the interactive part, so this sense that ChatGPT actually talks to you, is programmed as a sort of extra layer on top of the language models output. It’s good to realise this is a deliberate choice of the programmers.

• For teachers: take a close look at your writing assignments

  A roadmap to review writing assignments:

  1. Take a writing assignment that you currently are using in one of your courses. Which of the sub-skills from the 'writing pie' are part of the learning goals of the assignment?
  2. Then consider to what extent ChatGPT is a threat to this skill. Can a student have ChatGPT do this part very easily? Will it be difficult for you to see if the student has done it themselves?
  3. Take the information from steps 1 and 2 and use it to revise your assignment.
     • If the goal for the student is to gain knowledge and you assess this through a writing assignment, then given the ease with which ChatGPT can mimic this, a writing assignment is not a wise choice. An oral examination or a midterm can be a good solution here.
     • Is learning to write part of the learning objectives? Then see if you can make the assignment more specific, so that it can be imitated less easily by ChatGPT. Or you could think about assessing sub-tasks: for example, a student can show through an argumentation scheme or text plan that they can divide a text logically.

  By the way, it is important not only to go through these steps for specific courses, but also to consider them at curriculum level. If a student has to write a thesis, writing skills should be addressed somewhere in the curriculum.