As generative AI has swept into classrooms across the country, the promoters and the skeptics have offered starkly different visions of the technology’s effects on student learning.
In the best-case scenario, AI could finally personalize instruction for every student, working with kids on the specific skills they need and giving targeted feedback. In the worst-case scenario, AI outsources the learning process and flattens student creativity and voice.
But a new review of the emerging research on AI in education finds that the reality isn’t so simple, or so binary.
Researchers at the AI Hub for Education, part of Stanford University’s Accelerator for Learning, analyzed 52 randomized controlled trials and quasi-experimental studies that tested the effect of student and teacher use of generative AI on students’ academic achievement in K-12 and higher education, perception of their learning experience, and social-emotional outcomes. The paper is an update of a similar review conducted in March of this year.
Some use cases stood out as more promising than others, but almost all offered caveats.
When students used generative AI for learning support, like generating explanations of concepts or practice problems, it generally led to better learning gains than when students used the tools to complete their work. Across the studies, though, neither use led to consistent positive outcomes.
Teachers’ generative AI use can increase the quantity and quality of student feedback, even leading to improved student outcomes. But it can also have unintended consequences. At least one study found teacher AI use reduced student motivation and confidence.
“You can use this tool in so many different ways. Some of these ways can be good, and some of them can be bad for student learning and development,” said Susanna Loeb, a professor at the Stanford University Graduate School of Education and an author on the report.
Evaluating the effect of AI on education is sort of like evaluating the effect of paper on learning, she continued.
“Is paper good or bad? Well, it depends on how you’re using it,” she said.
Complicating the picture further is the shortage of high-quality studies conducted in the United States’ K-12 schools. Only 10 of the 52 studies examined were conducted with U.S. K-12 students or educators, and as such the review’s findings shouldn’t be treated as broadly generalizable, the authors write.
Research in this field is still very early, said Loeb. “It will be a long time before we’re at a point where we can say, ‘We know what the effects of AI tools are on students.’”
Review features new findings on student- and teacher-facing AI
Relatively few studies on generative AI use in education can make cause-and-effect claims about student outcomes. Of the 1,686 papers in the AI Hub’s research repository database, only 52 tested the impact of specific interventions.
In general, the research base on AI in education is muddy. A recent audit of meta-analyses on AI’s effect on classroom learning found the papers flawed and unreliable as guides for decisionmakers.
That makes the AI Hub’s work combing through the evidence stand out in the field, said Elizabeth Tipton, a professor of statistics and data science at Northwestern University who was not involved in the meta-analysis. “Not just for the findings,” she said, “but for shining a light that there are so few studies that meet the criteria for the standards we should be upholding for our schools.”
New papers added since the last review reinforce the idea that schools should be cautious about AI chatbots, Loeb said.
“Unless we’re careful with chatbots, they really can take away from learning, because it is so tempting to give over the learning to the chatbot to produce something,” she said. Students can think that they’re learning more with AI support, even when they aren’t, other findings from the review show.
This updated scan also probes variation in AI tool design and delivery: Do certain features make a tool more effective? At what point in the learning process should students turn to AI? Do tools designed for educational use outperform general-purpose tools? There aren’t clear answers to any of these questions, but some studies suggest adaptive capabilities, interactivity, and structured guidance could improve learning.
It’s not surprising that research hasn’t yet identified a concrete list of features that distinguish helpful AI tools from harmful ones, said Sina Rismanchian, a doctoral candidate at the University of California, Irvine School of Education. Rismanchian has conducted research on AI in math education that was included in Stanford’s review.
Those features change depending on the learning task, and how the user wants AI to support the learning goal, he said. Now, Rismanchian is conducting a new study, working with a company that allows teachers to create their own chatbots. But so far, it’s been hard to identify commonalities shared by the more effective chatbots, compared with those that don’t support student learning, he said.
Tipton also said it makes sense that no one kind of tool emerged as a “quick fix.”
“It’s really hard to improve student learning,” she said, with or without technology.
Stanford’s review also found some conflicting results on educator use of AI.
“We do have evidence that the tools for teachers can save time and produce high-quality work,” Loeb said. “But this time, we saw some evidence, similar to the evidence for students, that if new teachers are using the tools, sometimes the students won’t like that as much, or won’t be as motivated by the lessons that come out. And we aren’t quite sure why.”
It’s possible that developing lessons and materials on their own helps teachers build important skills, Loeb said, and using AI for those purposes short-cuts that learning process.
How should schools navigate scant evidence?
Stanford’s findings offer implications for researchers looking into AI’s use in education—and for schools making decisions about how, or whether, to introduce new technologies.
The causal research conducted in U.S. K-12 schools, comprising only a fifth of the review’s contents, is “such a small number of studies,” said Tipton.
“There’s just massively more work that’s needed,” she said, on the benefits but also potential harms. She added that such work would require accelerated funding.
Leaders in the field are calling for the same. Last month, the Alliance for Learning Innovation, a coalition of nonprofits, researchers, practitioners, funders, and private-sector leaders, issued a report arguing that the federal government should invest in research to develop safe and effective AI tools for classroom use.
Decisionmakers need more research on how AI tools operate when they’re used in practice, in real K-12 settings, Loeb said. That could mean more closely examining the roles of educators in mediating students’ interactions with AI, how to leverage tools for their specific strengths, and how to embrace variability among students, she said.
“The fact that we’re having to push to make decisions right now amid this emerging technology is a problem,” said Tipton. “Why are we willing to accept this risk in schools if we don’t know?”
Rismanchian, of UC Irvine, said he thought the New York City schools’ approach struck an interesting balance, banning student-facing generative AI for elementary and middle schoolers while limiting access for high schoolers.
“One thing I would advocate for is going with caution, and having this idea of experimentation in mind,” he said. “It’s not this one-off decision that we’re just done with.”
Loeb echoed the point. If schools choose to use AI, they need to do it strategically, and closely monitor implementation, she said. “There is plenty of evidence that unless carefully used, these tools can reduce learning.”
