Perception of Artificial Intelligence - generated texts among university students and faculty

TY - CONF

T1 - Perception of Artificial Intelligence - generated texts among university students and faculty

AU - Kuzminov, Serhii

A2 - Smith, Julie

A2 - Duke-Williams, Emma

A2 - Holme, Richard

A2 - Srinivasan, Sundar

A2 - Spark, Gordon

A2 - Martindale, Linda

A2 - Kinnear, Susan

N1 - © 2025 The Author.

PY - 2025/10/6

Y1 - 2025/10/6

N2 - This study explores the ability of university students and teachers to distinguish between essays written by humans and those generated by ChatGPT 3.5, an early yet powerful version of a large language model. Conducted in spring 2024 at Alfred Nobel University (Dnipro, Ukraine), the experiment involved 16 student volunteers and 2 teachers. Each student submitted two essays: one written independently, and one mainly generated by ChatGPT. The essays were anonymized and randomly redistributed among participants for evaluation. Evaluators were asked to identify the likely authorship of each essay (AI or human) and assign scores based on a 100-point scale, regardless of perceived authorship. Results revealed that neither students nor instructors could reliably identify AI-generated texts, with notable errors occurring across both language groups (Ukrainian and English). Interestingly, students demonstrated a measurable bias: essays suspected to be AI-generated received significantly lower scores, even when not actually produced by AI. This bias (quantified through score deviations and comparison with actual authorship) suggests a tendency among students to undervalue texts believed to be artificial, treating them as “inferior goods.” In contrast, instructors largely maintained objectivity, with minimal deviations in their evaluations. The findings raise important questions about the reliability of traditional assessment tools in the context of generative AI. They highlight the challenges of maintaining fairness and validity in student evaluations, as well as the need for enhanced assessment literacy and critical awareness of AI’s growing role in academic writing. The study concludes with recommendations for future experimental designs and suggests directions for research into human-AI interaction.

AB - This study explores the ability of university students and teachers to distinguish between essays written by humans and those generated by ChatGPT 3.5, an early yet powerful version of a large language model. Conducted in spring 2024 at Alfred Nobel University (Dnipro, Ukraine), the experiment involved 16 student volunteers and 2 teachers. Each student submitted two essays: one written independently, and one mainly generated by ChatGPT. The essays were anonymized and randomly redistributed among participants for evaluation. Evaluators were asked to identify the likely authorship of each essay (AI or human) and assign scores based on a 100-point scale, regardless of perceived authorship. Results revealed that neither students nor instructors could reliably identify AI-generated texts, with notable errors occurring across both language groups (Ukrainian and English). Interestingly, students demonstrated a measurable bias: essays suspected to be AI-generated received significantly lower scores, even when not actually produced by AI. This bias (quantified through score deviations and comparison with actual authorship) suggests a tendency among students to undervalue texts believed to be artificial, treating them as “inferior goods.” In contrast, instructors largely maintained objectivity, with minimal deviations in their evaluations. The findings raise important questions about the reliability of traditional assessment tools in the context of generative AI. They highlight the challenges of maintaining fairness and validity in student evaluations, as well as the need for enhanced assessment literacy and critical awareness of AI’s growing role in academic writing. The study concludes with recommendations for future experimental designs and suggests directions for research into human-AI interaction.

U2 - 10.20933/100001443

DO - 10.20933/100001443

M3 - Paper

T2 - University of Dundee Scholarship Conference 2025

Y2 - 2 September 2025 through 3 September 2025

ER -