If there is a burning question that parents, teachers, principals, and K-12 observers all want an answer to, it’s this one: Does AI in the classroom improve learning?
If you were to pull recently from any number of research syntheses that have appeared in reputable, peer-reviewed journals put out by major scholarly publishers, you would likely conclude the answer is a settled yes. They show almost uniformly positive findings—sometimes with stunningly large effects and sweeping recommendations for adopting the technology.
And you would also be getting terrible advice, a new audit of those papers concludes.
Many of the early research syntheses about AI in education are deeply flawed and too unreliable to guide decisionmaking about the tools, said Patrick O’Neill, the author of the audit and an associate professor at Ivy Tech Community College in Bloomington, Ind. But the papers have still been widely influential, cited more than 3,000 times, according to Google Scholar. One of them, which appeared in the journal Nature in May 2025, was retracted a year later.
“There is a hunger for this evidence which is driving people to publish quickly and overstate the solutions,” O’Neill said in an interview.
High-quality, smaller-scale studies—what should be the backbone of any kind of research synthesis—are only now beginning to generate results, he said: “The people who are going to do the best research on AI and education probably haven’t published yet.”
In his audit, O’Neill looked at 14 meta-analyses—a type of study that analyzes results from many others to arrive at an average effect size—from nearly 40 that all concluded AI had positive effects on learning. He found that all of them fail basic standards for this type of research by pooling a mismash of studies on different populations and deploying disparate measures of effectiveness, often not related to academic achievement.
And the analyses suffered from a fundamental conceptual problem: They didn’t clearly define what type of AI use they were studying and limit the papers they included to those matching that definition.
The rush to publish on AI outcomes and pressure on academics to do so quickly has fueled the problem, said Justin Reich, an associate professor of digital technology at the Massachusetts Institute of Technology, who hosted a webinar late last month on O’Neill’s new research.
“There’s demand for these studies. There’s huge incentives to do them,” he said during the webinar. “There’s maybe not enough expertise in review communities to be able to evaluate them. It’s kind of a perfect storm of factors to motivate and incentivize problematic research.”
Garbage in, garbage out
O’Neill’s paper comes as ed-tech ventures are clamoring to get schools to adopt AI tools in schools and classrooms and have begun issuing tailored versions, like Google Classroom, which incorporates the Gemini AI technology, and ChatGPT for Teens.
Meanwhile, the K-12 field is only beginning to categorize the various ways that the tools are being used for teaching and learning: One recent study of 20 tools found big differences in the pitfalls and potential benefits of using a generic large-language model compared to using an instructionally focused tool designed specifically, for example, to teach writing.
The meta-analyses O’Neill reviewed found that most of them were incoherent in how they defined learning and attempted to measure it. A study of AI use in elementary PE was included in the same meta-analysis of a study looking at simulations for medical students. A few included studies that didn’t even use AI tools. In terms of outcomes, the meta-analyses pooled measures of knowledge alongside motivation, engagement, mental health, and perceived learning all in the same bucket.
All of those inconsistencies pose major challenges to what’s called construct validity—having a clear sense of what’s being studied. During the webinar, Elizabeth Tipton, a professor of statistics and data science at Northwestern University, drew a parallel to illustrate why construct validity matters.
One would not, she said, conduct a meta-analysis on something as broad as “math education;" a proper one would look for evidence on a specific math topic—such as whether spatial reasoning interventions can help boost learning. But the AI meta-analyses smush everything together and end up with something unusable, she and other panelists said.
The meta-analyses also didn’t account for other common sources of error, such as publication bias, a longstanding problem in social science in which journals tend to favor research that shows positive effects over null or negative effects. And all but two of them failed to control for dependent variables that might have influenced the findings.
O’Neill also examined a subset of the studies that were included in the meta-analyses and found those had severe drawbacks, too, in what amounts to a “garbage in, garbage out” problem. Some of those studies had “impossible” statistics; others had references that were likely hallucinated (one cited studies on ChatGPT that were dated before that technology was publicly released).
In the webinar, the panelists agreed that the research community needs to collaborate on coming up with a valid set of discrete research questions on AI in education, and to generate evidence on those topics, before attempting to conduct a meta-analysis with the goal of reaching more broad-scale conclusions.
“When we say ‘AI in education,’ I’m not even sure I know how to classify that into five problems,” Tipton said. “Do we mean people are using apps and programs that include AI in them, like Khan Academy? Do we mean people using giving students access to Claude? What exactly do we mean?”
When will we have better research on AI in education?
O’Neill’s findings also raise ongoing problems about the strength of peer review, a quality-control mechanism that is supposed to catch problems before publication.
“Given the number of serious errors that should have been caught, or at least cause for a correction and probably in some cases weren’t reparable, this is already a serious problem,” he said. “They’re at 11 different journals that are very highly reputed. These just should not have gotten through.”
It’s not clear when educators will have access to better big-picture answers on AI education, in part because it takes a while to generate enough high-quality studies, even as use of the technology accelerates and evolves at a rapid clip. An explosion of new research is now beginning to be reviewed and sorted.
The AI Hub for Learning, part of Stanford University’s Accelerator for Learning research group, has been attempting to categorize the evidence on AI learning. In its 2026 review of more than 800 studies, just 20 of them had a strong cause-and-effect design, and not a single one involved K-12 students. The organization is now reviewing dozens of additional papers and hopes to release an update soon, said Susanna Loeb, a professor at Stanford, during the webinar.
Though it’s too early to gain insight from a meta-analysis, Loeb said she hopes the field can learn from the limited amount of high-quality research on AI in education.
“But it requires that each of us are really honest in what papers and research studies can do and what they can’t do, and that trusted voices can emerge in here,” she said.
