What did I do?
| Old learning practice | New learning practice |
| I used to ask my students to choose a recent empirical study, read it, and write a short, critical essay about it (half a page or so). The results were often more like a summary than a critical essay. | I asked my students to 1) choose a recent empirical study, 2) ask the ChatGPT to summarize the paper for them (keeping a record of their prompt and the summary), and 3) read the paper in order to criticize the summary of the ChatGPT (with clear reference to the paper and the ChatGPT’s summary). |
Results?
Examining 36 attempts by my students on different papers (70% quantitative), all from academic journals, I have observed three patterns:
1. Students vividly recognize the flaws and limitations of the ChatGPT summaries
Students were able to identify various flaws in the ChatGPT's, incorrect statements, and violations of the content of the studies, for instance,
- Wrongly reporting the methods: such as ChatGPT saying “187 employees from three organizations in India”, whereas the study is conducted in Bangladesh and the sample size is 403!. ChatGPT fell short, especially when the studies used some less popular methods such as ethnography or reported their methods different from the common templates.
- Conveying a message that is not the focal point of the study: They also frequently reported incompleteness and missing aspects of the ChatGPT summaries, leading to a different message than the intended focus of the paper. This was notable when ChatGPT summaries emphasized "main effects" while the study's primary contribution focused on "moderating" effects of variables. In these situations, the summary of ChatGPT was not completely wrong, but if someone would take it as the summary of “that” paper, they would easily miss on the main contribution of the study.
- Overlooking study context and boundary conditions: ChatGPT summaries often missed the boundary conditions and contexts of the studies (especially if they are not reflected formally in the variables and models) were missed in the summary. As a result, the ChatGPT summary presented the findings of the studies as generalizable findings (creating a wrong sense of generalizability).
2. Students engage deeply with the content of the studies
Students engaged deeply with not only the content, but also the implicit messages, writing styles, methodological aspects, boundary conditions, and the broader meaning and implications of the studies. This way, they were triggered to not only look into the papers, but also see beyond the study itself and consider the broader context. They reflected on whether the ChatGPT summary would convey the same message as reading the paper itself. This prompted critical analysis of contexts, details, and nuances absent from algorithmically generated summaries.
3. Students became reflective of and critical to Generative AI
Engaging with critical reflection of ChatGPT’s summaries enabled students to enhance their own critical reflection capabilities. They became alert to the different ways of (mis-)interpreting the findings, mindful of the context of the studies and interpreting the results against that, and aware of how seemingly trivial details (e.g., the procedure of collecting data) could influence the interpretation of the studies.
What lessons have I learned?
- Prioritize the fundamental learning objectives: In my case, “the capability of critically reflecting on scientific materials” was essential. It was helpful to see how a new configuration of tasks and technologies can support such an objective. When new (transformative) technologies arrive to our domains, we must reflect on what are our essential objectives, what values we are creating, and what are the end results we want to produce.
- Go beyond the established teaching and learning practices: I was reminded of the power of not taking the current ways of teaching (and especially examination) for granted. Rather, we must think what are the novel possibilities that we can generate for more effective and even deeper learning. Starting from current practices, routines, protocols, habits can easily take us into a mode of policing the new technologies in order to safeguard our established practices.
- Empower students (and other actors) as the responsible learners: Assigning the role of “evaluator” of one of the most advanced technologies was empowering for my students. This aligns with the evidence showing that we learn more deeply and effectively when we are in the “active” state of mind, than being passively treated.
- Embrace the (technological) innovation and learn how to work with them creatively: Again, I was reminded of the classic examples of innovations, and how much we laugh (today) at the way we were scared of them or were overly enthusiastic and hypnotized by their promises! ChatGPT is another (sharp) knife that is in the hand of everyone (even before we as parents notice, our kids are holding this knife and moving around with it!). It stands to reason that we empower and train everyone about the pros and cons of this knife and together be prepared for using it effectively.
Notes
1) I am grateful for the efforts of my students and their generosity for giving their consent for drawing on their attempts,
2) students used the free version of the ChatGPT, which provides shorter outputs,
3) students may have used different prompts to figure out how they can get a good summary of the papers.
By dr. Mohammad H. Rezazade Mehrizi,
Associate Professor,
School of Business & Economics,
KIN Center for Digital Innovation
https://www.linkedin.com/in/mohammad-hosein-rezazade-mehrizi-1734b010a/