The event focused on a practical problem: as AI becomes more accessible, the risk of receiving automated or AI-assisted responses in surveys increases, which threatens the reliability of scientific data.
We find ourselves in a “arms race” between AI capabilities and detection technologies. Traditional safeguards like CAPTCHA, which once were a reliable tool against automated responses ,are rapidly becoming obsolete. A crucial nuance addressed was distinguishing between truly malicious actors deploying bot swarms to exploit paid survey panels for profit, and ordinary participants who paste open-ended responses into ChatGPT for convenience. Both contaminate data but require different countermeasures. Tactics such as “poison pill” sentences, metadata tracking of keystrokes and response times, and copy-paste restrictions offer partial solutions, but the panellists agreed these are temporary and not structural fixes.
While discussing possible solutions, a rather philosophical question emerged. If an AI’s response is indistinguishable from a human’s, does it matter who or what answered? The panel’s answer was a clear yes. LLM’s are trained on existing human data, but they do not reflect the unique lived experience of any individual. They tend to exaggerate effect sizes, over-confirm hypotheses, and systematically misrepresent minority populations. For instance, when non-native speakers use AI to polish their open-ended responses, something is lost, the unfiltered tone, sentiment, and voice that qualitative researchers depend upon for gathering more interesting insights from the data. The panellists also raised a broader concern, that as AI-generated content floods the internet, training data for future models becomes increasingly homogenised, risking an epistemic feedback loop that could further distort scientific knowledge at scale.
Large-scale online sampling remains irreplaceable for certain research designs, for instance, you cannot bring 2,000 nationally representative participants into a lab, and therefore the continuous reliance of online surveys. However, the speed and scale that online surveys allow, once seen as virtues of digital research, now amplify certain vulnerabilities. The panel questioned the acceleration of academic output itself, with publications proliferating at a rapid rate and reviewers increasingly relying on AI, which raises the question what does the scientific community gain if entire pipelines become automated? Measures to address this can span multiple levels, researchers should check their data and report detection methods transparently, journals should enforce integrity standards comparable to open-science practices, and commercial survey-participant providers must take accountability for respondent authenticity. Practices such as auditing for AI responses are not administrative inconveniences, but rather core requirements of rigorous scientific methodology.
The panel closed with cautious optimism as AI is also a powerful augmenter of research by allowing automating text and video analysis, enabling questionnaire generation and validation at scale, and opening methodological avenues previously too costly to pursue. The consensus is that online surveys as a methodology is not obsolete, but it is at an inflection point. The path forward requires continuous research into AI’s evolving capabilities, broader institutional awareness of the threats, and a collective commitment to prioritise data authenticity.