A study that claimed OpenAI’s ChatGPT could have a positive impact on student learning has been retracted nearly a year after publication. The journal publisher, Springer Nature, cited “inconsistencies” in the analysis and a lack of confidence in the findings — but that was before the paper collected hundreds of citations and circulated on social media.
“The authors of the paper make some very eye-catching claims about the benefits of ChatGPT on learning outcomes,” Ben Williamson, a senior lecturer at the Center for Research in Digital Education and the Edinburgh Futures Institute at the University of Edinburgh, Scotland, said in an email to Ars. “Many people on social media considered this to be one of the first pieces of solid, gold standard evidence that ChatGPT, and Generative AI more broadly, benefits learners.”
The retracted paper attempted to measure “the impact of ChatGPT on students’ learning performance, learning perception, and higher-order thinking” by analyzing the results of 51 previous research studies. Its meta-analysis calculated effect sizes between experimental groups of different studies that used ChatGPTS in education and control groups that did not use AI chatbots.
According to the researchers who wrote the paper, that analysis showed how ChatGPT has a “modest positive effect on enhancing learning perception” and “promoting higher-level thinking”, as well as “a large positive effect on improving learning performance”. The now withdrawn results first appeared in the journal Humanities and Social Sciences Communications, published by Springer Nature, on May 6, 2025.
“In some cases it appears it was synthesizing very poor quality studies, or mixing together findings from studies that could not be accurately compared because of very different methods, populations and samples,” Williamson told Ars. “It really seemed like a paper that shouldn’t have been published in the first place.”
Williamson also questioned the timing of the paper’s publication, just two and a half years after OpenAI released ChatGPT in November 2022. Williamson said, “It is unlikely that dozens of high-quality studies about ChatGPT and learning performance could have been conducted, reviewed, and published at that time.”
A legacy that can outlive
Since its publication, the study has been cited 262 times in other papers published by Springer Nature’s peer-reviewed journals and has received a total of 504 citations from both peer-reviewed and non-peer-reviewed sources. It also attracted nearly half a million readers and received enough online attention to rank in the 99th percentile for journal articles in terms of attention score.
<a href