A Harvard experiment involving 194 students found that a custom AI tutor produced roughly twice the learning gains of instructor-led active learning. Students using the tutor also reported greater engagement and motivation while spending less time studying the material.
The findings concern a system that faculty carefully designed around teaching principles, rather than unrestricted access to a chatbot.
Harvard first reported preliminary results in September 2024. The final peer-reviewed study appeared in Scientific Reports on June 3, 2025.
How Harvard tested the AI tutor
Harvard lecturers Gregory Kestin and Kelly Miller led the research in Physical Sciences 2, an introductory physics course for life sciences majors.
The experiment took place in autumn 2023 and covered two lessons over consecutive weeks. Students studied surface tension and fluid flow.
One group initially attended an instructor-guided active learning lesson, while the other used the AI tutor at home. The groups switched methods the following week, allowing students to experience both approaches.
Researchers used tests before and after each lesson to measure content mastery. They also surveyed engagement, enjoyment, motivation and growth mindset.
The comparison involved an established active learning classroom that already used research-based teaching methods. Students participated in activities rather than simply listening to lectures.
“It was shocking, and super exciting,” Miller said of the early findings.
The published study found that median learning gains with AI exceeded twice those of the classroom group. Students spent a median of 49 minutes using the tutor, compared with approximately 60 minutes of classroom learning.
Seventy per cent of AI users finished within an hour. Engagement and motivation improved, but enjoyment and growth mindset showed no statistically significant differences.
Faculty expertise shaped the chatbot
Kestin and Miller created detailed instructions to make the tutor behave like an experienced teacher.
The system used a framework built on the GPT application programming interface. Faculty supplied content-rich prompts, checked conversations and structured feedback.
The published research describes a GPT-4-based tutor with carefully sequenced activities and prepared solutions to improve accuracy.
Researchers identified personalised feedback and control over learning pace as likely advantages.
Students familiar with a concept could move ahead quickly. Those struggling could ask additional questions and spend longer working through explanations.
That flexibility could help teachers reserve classroom time for advanced problem-solving, projects and group work.
Researchers stress teachers’ continuing role
Kestin cautioned that AI could support or undermine learning, depending on its use. The objective was to develop students’ critical thinking rather than let a chatbot think for them.
The researchers did not present AI as a replacement for teachers. Their findings also do not establish that AI tutoring will outperform classroom instruction across every subject or learning task.
Harvard’s September 2024 report outlined plans to adapt the tutor for multivariable calculus. The Derek Bok Center for Teaching and Learning and Harvard University Information Technology also planned pilots in other large introductory courses.
The results point to the importance of instructional design: expert guidance, structured feedback and clear learning goals shaped this tutor’s performance.
