What Research Measured on AI-Written Teaching Materials and AI Tutors: Content Error Rates, Human Screening, Student Ratings, and Tutoring Experiments

🌏 中文版

This is Part 28 of the "Writing Articles That Teach" series, in its sixth section, "how to know it works, and how to make it". Earlier parts covered the signals in an author's hands, large-scale experiments on platforms, and whether re-explaining someone else's course is worth it. This part looks at a practice that keeps getting more common: using AI to write teaching materials, or putting an AI tutor next to them.

The question comes from this site. Many of its course guides are drafted by an AI agent from lecture notes and recording subtitles, then handed for review to another agent that took no part in writing them. Part 22 mentioned this in passing; what this process catches and what it misses will be covered fully in Part 29. Each article page also has an Ask AI sidebar. Part 21 discussed it: readers ask the questions, and a model answers. Under the default settings, it first searches this site's articles and answers from the passages it retrieves. When it can't find a reliable source, the prompt tells it to say the knowledge base lacks enough evidence, rather than fall back on the model's own general knowledge. When a draft fails the format or quality checks in both rounds, it is shown anyway, with a warning in front, in Chinese, that the answer may be incomplete. So this part answers two questions. First, how reliable is teaching content written by AI, and how should it be checked? Second, if you put an AI tutor next to the material, do readers learn more?

This site already has a piece written from the learner's side, AI Makes Things Smooth Exactly Where They Should Be Hard, about what happens when learners use AI themselves. That piece also covered the retraction of a heavily cited meta-analysis by Jin Wang and Wenxiang Fan (2025), which is a different study from the work of Rose Wang et al. and Yizhou Fan et al. cited here. That meta-analysis is not cited again in this part. This part moves to the side of the people who make content, and breaks down what each experiment actually compared.

How sources were chosen, and their biases. I scanned the literature first, then picked sources to read closely: three studies that examine AI-generated teaching content, and nine randomized experiments on AI tutors or AI-assisted learning. Nearly half of the results are either no detected difference or negative. The biases are clear. First, six of the nine are preprints or working papers that have not gone through journal review. Second, the models are GPT-3, GPT-3.5, Codex, and GPT-4, from 2022 to 2024. Error rates for current models are likely different, but these studies can't tell us by how much. Third, the subjects are concentrated in math, programming, and physics, and most participants are university students. There are also high school students in Turkey, first-year senior secondary students in Nigeria, elementary and middle school students in the United States, adults on Mechanical Turk (median age bracket 36 to 40), and adult learners in a free online course (mean age 31.4). Fourth, none of the studies looks at self-learners reading web articles. I read every item in the references from start to finish and kept an extraction record; studies I only knew existed, without a legally public full text to read, are not cited.

The short version

The standard deviations in the table are the difference between two group means divided by the standard deviation of the scores. By Cohen's convention, around 0.2 counts as small and around 0.5 as medium; these are customary reference lines, not claims the studies make. No difference detected means the difference was not statistically distinguishable; it does not mean the two are equally good.

1. Teaching content written by AI

How many errors, and how to reduce them: a study of math hints

The 2024 study by Zachary Pardos and Shreya Bhandari is the only one I found that measured both how many errors AI-generated teaching content has and what the learning effect is after screening. The problems came from four OpenStax algebra and statistics textbooks, all without figures. The researchers gave the problems to ChatGPT 3.5 (the December 15, 2022 version) as they were, with no special prompt design, and used each whole response as a solution hint.

The first result is the error rate. Six raters checked each hint for three things: whether the answer was correct, whether the steps were correct, and whether there was inappropriate language. 32% of problems failed. Every failure was a wrong answer with wrong steps; none was a problem of language, spelling, or grammar. In other words, the errors don't show up as spelling or grammar mistakes. The rate varied by subject; the highest was Intermediate Algebra, at nearly half. This number comes with conditions: a late-2022 model, a single response to a problem given as is, and math from secondary school to introductory university level without figures. The problem counts in the paper's text and its table don't quite match; details are in the appendix.

The second result is a way to reduce errors. When the model generated 10 responses for each problem and the most common answer was taken, the error rate dropped to 0% to 2% for algebra and 13% for statistics. This technique is called self-consistency in the research literature. Two caveats: the grouping and judging were done by hand, and the paper doesn't describe an automated version; and this procedure was used only to estimate error rates, not in the learning experiment below.

The third result is the learning effect, on the condition of human screening. Before the experiment, the second author checked every ChatGPT hint, and if any of the three checks failed, the whole problem was removed from the item pool. So every AI hint participants saw was correct. The human check took 37.60 seconds per hint on average.

The experiment ran on Amazon Mechanical Turk and analyzed 274 adults (median age bracket 36 to 40) in three groups: no hints, hints written by human tutors, and screened ChatGPT hints. The sequence was a 3-item pre-test, 5 practice problems (the only place hints appeared), and a 3-item post-test; the post-test used the same items as the pre-test and was given immediately, in the same session. The ChatGPT group gained significantly more than the no-hint group. No difference was detected between the human-hint group and the no-hint group, and none between the ChatGPT group and the human-hint group. The title says "equivalent", but the paper ran tests of difference, not equivalence tests.

A pilot by the same authors (77 people, two algebra subjects) pointed the opposite way: human hints produced larger gains, and the difference was significant. The authors attributed the pilot result to the ChatGPT group's near-ceiling pre-test scores.

The authors wrote one recommendation in their discussion: "Educators and education technology organizations should therefore be cautious when integrating raw ChatGPT output into their pedagogy without the use of error mitigation processing". They also said ChatGPT hints took one-twentieth of the time to produce that human hints did; the paper doesn't show how this was calculated.

How students rate it: two perception studies

Two other studies asked whether students thought AI-written material was good. Both took place in the same 2022 first-year C programming course at the same university (University of Auckland). Neither paper says whether they involved the same students, so they can't be treated as two independent samples.

The 2023 study by Paul Denny et al. had Codex generate about 100 code examples with explanations, modeled on 6 examples from the teaching team, and mixed them without screening into a pool of resources that students rated for each other, without knowing the source. Across 987 resources and 4,499 ratings, no difference was detected between student-written and AI-written resources on any of five measures: code correctness, explanation correctness, helpfulness, overall quality, and rating confidence. The closest to the threshold was overall quality, with student-written resources slightly higher. The title asks "Can We Trust AI-Generated Educational Content?", but nobody in the study checked whether the content was actually correct; correct is a score that students gave. The comparison was also with first-year classmates, not instructors. One more result: the AI output stuck closely to the examples it was given, with shorter code and fewer kinds of syntax, while student-written resources varied much more.

The 2023 study by Juho Leinonen et al. used a different kind of content: explanations of three C functions, half written by GPT-3 and half by students, rated by nearly a thousand students. Students found GPT-3's explanations easier to understand and more like accurate summaries; both differences were significant, but the authors themselves said the effect sizes were weak. No difference was detected in the length of the explanations, either perceived or actual. Keep three conditions of this study in mind together:

• The student half was sampled in strata by word count, with two-thirds coming from the shortest 10% and the longest 10%. It is not a representative sample of student explanations.
• The paper doesn't say whether students knew which explanations were written by AI.
• The authors themselves said it is "possible – even likely" that explanations written by stronger students or instructors would be rated higher than GPT-3's.

This study has another observation useful to people who write explanations. The paper lists the highest- and lowest-rated examples for one function. The lowest-rated one from GPT-3 and the lowest-rated one from a student were each just one sentence or phrase. The GPT-3 one was "This code snippet returns the largest value in an array of integers." The content isn't wrong, yet its accurate-summary score was clearly lower than that of the top-rated explanation for the same function. In their thematic analysis, the authors also said students would rate short explanations low even when they hit the point. About a fifth of the open-ended responses asked, unprompted, for line-by-line explanations; the authors noted that this runs against earlier research in which short, abstract explanations were rated higher.

Can readers tell when AI got something wrong

This is the question people who write teaching articles most want answered: if AI-drafted content contains an error, can readers catch it themselves? None of the studies I read tested this directly. Denny et al. and Leinonen et al. measured scores that students gave; they didn't plant known errors in the content, and they didn't ask students whether they could tell the source. In Pardos's experiment, the incorrect hints never reached participants at all.

Only two things are certain. First, the incorrect hints in Pardos's batch had no spelling or grammar problems, so the errors don't show on the surface of the text. Second, in Leinonen's study, students rated short explanations low even when they were on point; the one-sentence explanation with no errors got only 3.7 for accurate summary. Putting the two together to say that readers can't spot the errors is my inference; no study supports it directly. Pardos and Bhandari suggested framing AI feedback as coming from an "imperfect robot" so that students look at it critically. That too is a suggestion; the study didn't test it.

2. AI as a tutor, and what each of nine randomized experiments compared

Research on AI tutors is easy to reduce to it works or it doesn't. After reading all nine, I think three things matter more than the results: who the AI group was compared with, how the AI was set up, and when the test happened. The table below lays these out first; details for each study follow in the next sections.

(The first arXiv versions of Lehmann, Nie, and Wang are from 2024; I read the v2 marked in the table.)

An unguarded chatbot: a Turkish high school

The 2025 PNAS experiment by Hamsa Bastani et al. randomized by class, was preregistered, and used blinded external graders. About 50 classes at one Turkish high school took four 90-minute math lessons, each in three parts: the teacher's review, assisted practice, and a closed-book exam without computers. The three groups differed only in the middle, practice segment:

• The control group had only the textbook and notes, with no device.
• GPT Base: GPT-4 in an interface imitating ChatGPT, with a prompt that gave only the problem and told it to act as a tutor.
• GPT Tutor: the same interface, with two more things in the prompt: an instruction not to give full solutions and only give incremental hints, and the teacher's correct solution, common mistakes, and matching hints for each problem.

Scores in both AI groups rose sharply during practice, but this comparison was not in the preregistered analysis. On the exam, the GPT Base group scored lower than control; the authors' regression estimate puts the gap at 17% of the control group's mean. The preregistered t-test version shows a smaller gap: from its appendix table, I calculated about 11%, with the same direction and significance. No difference was detected between the GPT Tutor group and control, and the point estimate was still negative. In other words, the guarded version was not detected as worse than control, and not detected as better either.

Why would the unguarded version cause harm? After the study, the researchers asked GPT Base the question students sent most often, "What is the answer?", 10 times for each of 57 practice problems, and it was right only 51% of the time. The authors' analysis suggests that most students simply copied the answers. One point is easy to miss here: GPT Tutor's guardrails changed two things at once, withholding answers and inserting the teacher's correct solutions. GPT Base got about half wrong in the researchers' rerun. GPT Tutor, by contrast, had the correct solutions in its prompt, and the paper didn't measure its error rate. So the difference between the two groups can't be credited to withholding answers alone. And even with guardrails, judging from the paper's figure of chat logs, about half of the GPT Tutor group's conversations still contained messages asking for the answer or copying the problem.

The limitations the authors list: one school, one subject, a deployment in fall 2023, and "we focus on short-term outcomes due to limitations imposed by our partner school". The exam came right after practice, and its items were very similar to the practice problems.

An unrestricted chat window: a German lab

The preprint by Matthias Lehmann et al. ran two preregistered experiments, and most participants had never written Python. The treatment group had an unrestricted GPT-3.5 chat window beside a 45-minute learning phase, but not during the pre-test or post-test. Neither experiment detected a difference on the post-test. In the second experiment, the AI group solved about 2.6 more problems during the learning phase, but this gap didn't carry over to the post-test.

The abstract says, in effect, that asking for answers lowers understanding and asking for explanations raises it. The authors labeled this an exploratory analysis, and it is thinner than I expected. The asking-for-answers half comes from a comparison between the two experiments (in the first experiment, the computers accidentally blocked copy and paste; in the second they didn't), and it no longer reaches .05 once the total number of messages is added as a control. The asking-for-explanations half is a correlation with how many times students chose to ask. Neither kind of use had a significant effect on the overall post-test. Understanding here isn't a separate test either; it is what remains of the post-test difference after controlling for practice progress. Another result, also exploratory: people below the median on the pre-test showed worse understanding with AI, while those above it did better.

A structured tutor beats the classroom: Harvard physics

The 2025 experiment by Greg Kestin et al. is the only study in this batch that used in-class active learning as the control. The setting was a Harvard introductory physics course for life science students, with a crossover design: two topics over two weeks, with each group using the AI tutor one week and attending class the other, then switching. The control was an active-learning class: the instructor led activities, students worked in small groups, and the instructor gave feedback. The AI group did the same activities at home on their own, with the introduction replaced by videos pre-recorded in a studio and all feedback coming from GPT-4. On the post-test given right after class, the regression estimate put the AI group 0.63 standard deviations higher. The authors said a quantile regression estimated an even larger effect but gave no table for it.

The paper is specific about how much human work this result relied on:

• The system prompt for each problem contained the problem, the answer, and a step-by-step solution. The authors wrote that "accuracy of our AI tutor relied on pre-written answers".
• Prompts alone weren't enough: "a system prompt could not reliably provide enough structure to scaffold problems with multiple parts", so the platform made students complete each sub-question in order.
• The prompts for each class took several days of running and revising problem by problem; the platform software took months to develop; the videos were recorded in a studio by a presenter with ten years of experience making popular-science videos.

The authors themselves listed these as conditions the effect might depend on, and wrote "we do not presume that structured AI tutoring will always outperform in-class active learning in all contexts". A few things the paper does not list as limitations also affect how to read it. The test came only right after class. There was no preregistration. The post-test counts were 142 and 174, which don't match the 194 included students. And the two conditions differed in several things at once: location, pre-recorded videos, self-pacing, and who students interacted with. The abstract says students "learn significantly more in less time"; the time comparison is the AI group's measured median of 49 minutes against an assumed 60 minutes for the classroom group, and 30% of the AI group spent more than 60 minutes.

An after-school program with a teacher present: Nigeria

The World Bank working paper by Martín De Simone et al. (I read the December 2025 updated version) ran in 9 public schools in Benin City, Nigeria. First-year senior secondary students volunteered and were chosen by lottery; those selected attended an after-school English program for 6 weeks, up to 12 sessions of 90 minutes each. Two students shared a computer to use Microsoft Copilot, a teacher circulated in the room, and the research team wrote a starting prompt for each session in advance, telling the model to set questions, wait for students' answers, and give a hint first when an answer was wrong.

The primary outcome the authors designated was English, with an effect of 0.24 standard deviations. The 0.31 that appears first in the abstract and on the title page is a total score that includes two other parts, AI knowledge and digital skills, and the treatment group's first session taught what Copilot is. On the school's own English exam, the effect was 0.21.

The most important thing to remember about this result is that the control group had no after-school activity at all: the treatment group got up to 18 extra hours of learning time, with a teacher present and content aligned with the curriculum. The authors themselves said the effect comes from the whole intervention, and that separating "additional instructional time and the interaction with the chatbot with teacher support" would require additional groups. Also, the share of students who completed the final assessment differed between groups; from its counts, I calculated about 64% for treatment and about 50% for control. The authors handled attrition in two ways, one of which lowered the effect on the school exam to 0.13. All outcomes were measured right after the program ended; the abstract's equivalence to 1.5 to 2 years of schooling is a conversion, not something measured.

Restricting when AI can be used: a Berlin lab

The IZA discussion paper by Mira Fischer et al. directly tested a common intuition: read on your own first, then ask AI. 334 university students read a 7-page excerpt from an intermediate microeconomics textbook for 25 minutes, then immediately took a 25-item multiple-choice test, without the materials or AI. There were three groups: PDF only; a GPT-4 tutor with retrieval over the materials available throughout; and reading only for the first 10 minutes, with the tutor appearing for the last 15.

The two AI groups combined scored 0.23 standard deviations higher than reading only. Split apart, the ask-throughout group was 0.34 higher, and no difference was detected for the read-first group. The preregistered hypothesis was that the read-first group would do better; the result went the opposite way, with a difference of 0.21 between the two groups. In the text, the authors call this only marginally significant (p = 0.066), but the abstract says "significantly".

The title says the AI tutor works "Without Crowding Out Reading Effort", but the study didn't measure reading. What it calls reading time is the time elapsed before the first question was sent, and both AI groups had much less of it than control; the ask-throughout group started asking after 5.7 minutes on average. Two more things to keep in mind. The restriction here only delays the tutor by 10 minutes; it is not a guardrail on response content like Bastani's, and the paper doesn't describe the system prompt. And the task was reading a textbook to prepare for a multiple-choice test, not practicing problem solving. So what it can say is that forcing 10 minutes of reading first was no better than open access throughout; it can't say that guardrails don't work.

Just making AI available: two large-sample experiments

These two studies are closest to this site's situation: they compare offering one more AI tool and letting learners decide whether to use it against not offering it.

The preprint by Allen Nie et al. ran in Code in Place, Stanford's free online programming course. The 5,831 people who finished the first week's assignment were randomly split into two groups. In week 4, the treatment group received an email, and their home page gained a button leading to a GPT-4 chat interface the course built itself, styled to look like ChatGPT. Before entering, they had to read a page explaining the benefits and drawbacks; the prompt had the assignment problems loaded in advance so it wouldn't easily hand out answers. A voluntary exam came 9 days later.

• Only 14.2% of the treatment group used it before the exam. The authors said this is a typical usage rate for voluntary tools in this course.
• The treatment group's exam participation was 4.3 percentage points lower than control's, and its week 6 assignment completion was also lower; for section attendance, no difference was detected after correction. The drop was in the whole group that was notified and given access, and 85.8% of the treatment group never used the tool, so it can't be described as engagement falling after people used AI.
• Among those who took the exam, no difference in scores was detected between groups. The authors estimated that people who actually used it may have scored 6.9 points higher (out of 100), but this estimate didn't survive multiple-comparison correction and depends on assumptions that can't be verified.

The authors said plainly that their data can't explain why engagement fell, and offered only four hypotheses. One bears directly on practice: they suspect that the benefits-and-drawbacks page shown before use itself created friction: "perhaps the existence of any handout, even if positive and short, could have been a point of friction that impacted outcomes".

The OSF preprint by Weijiao Huang et al. ran in an introductory marketing course at a US state university, with 253 students randomized. From week 4 to week 14, the treatment group had unlimited access to a tutor with retrieval over the course materials; whether to use it and how much was up to them, and it wasn't graded. For course interest, self-efficacy, and exam scores, no group-by-time interaction was detected on any of the three. Usage was low: of the 95 treatment-group students who completed both the pre- and post-surveys, 18 never logged in, 43 logged in during only one period, and "Only 15 students interacted continuously with the Tutor." Logins clustered around exams. Both groups were encouraged to use the same set of Quizlet flashcards, and the authors speculated that the tutor had limited value to add in this course. The authors call this a preliminary study, and in several places the paper's sample sizes and degrees of freedom don't match.

For tutors, not students: US elementary and middle school online tutoring

Tutor CoPilot, by Rose Wang et al., puts AI in a different position: it doesn't face students directly. Instead, it is embedded in an online tutoring platform. When a tutor presses a button, the system returns a few suggestions for how an expert would reply, based on the last 10 messages, and the tutor decides whether to edit and send them. Tutors were the unit of randomization. The students were elementary and middle school students in a school district in the US South who had scored below grade level on the previous year's state test, and the study lasted two months.

The primary outcome was the exit ticket at the end of each session (tutors could not help): the pass rate for students of treatment-group tutors rose from 62% to 66%. This is an intention-to-treat effect; the tool was actually used in only about 29% of treatment-group sessions. No difference was detected on any of the five questions in the student post-session survey or the four in the tutor survey. The end-of-year standardized test is in Appendix K: the math coefficient was negative ("math scores trending negative") and reading was positive, both not significant. Students were not randomized, and this analysis used the share of sessions a student had with treatment-group tutors, so it can't tell us whether students improved by the end of the year; the authors themselves said future work should randomize students. The abstract's figure of only 20 US dollars per tutor per year is just the language-model API cost, not including development and training.

AI helping revise essays: the knowledge tests were unlikely to move anyway

The 2025 BJET experiment by Yizhou Fan et al. (I read the pre-publication submitted manuscript) had 117 university students with English as a second language (most of the authors are at Peking University in Beijing) read English materials for two hours, write a short essay, then spend an hour revising it. The four groups differed only in that revision hour: they could ask a ChatGPT limited to giving suggestions, chat in text with a human expert, use a writing checklist tool, or have no extra support. The ChatGPT group's essays improved by about 2 points more (out of 25) than the no-support group's, while no difference from the no-support group was detected for the human expert or the checklist tool. No difference was detected among the four groups on the knowledge test or the transfer test.

This study is easy to read as using AI made the writing better without more learning, but the design doesn't support that reading: the knowledge and transfer tests covered the material read in the first two hours, and the four groups had exactly the same conditions in those two hours. In their discussion, the authors also mentioned that some people in the lab got around the limits and pasted example sentences from ChatGPT into their essays to cater to the rubric; they didn't give a count. The "metacognitive laziness" in the title is not a measured result; in the limitations, the authors wrote "The last limitation is the lack of targeted and matured measures for assessing metacognitive laziness within the study’s design."

What students felt they learned, and what was actually tested

Bastani, Fischer, and Lehmann also asked students how they felt. Part 25 already went through the same numbers, so I won't repeat them here. Only one point is relevant to this part: in all three studies, some group with AI rated itself higher than its actual scores could explain. For example, Bastani's GPT Tutor group did not actually score better on the exam, yet felt significantly more strongly that it had done well. But most of these comparisons set two separate results side by side, and Fischer's ask-throughout group even appears to have slightly underestimated itself.

Where the sources disagree

The following sets of results point in different directions, or look contradictory while actually measuring different things. I set them side by side without picking a side.

Whether to add guardrails. Bastani et al.: the unguarded GPT-4 group did worse on the exam than control, and no difference from control was detected for the guarded version. Fischer et al.: the group restricted to reading for 10 minutes first was no better than open access throughout; the direction was opposite. The two studies tested different restrictions: the first restricted response content (no answers, plus the teacher's correct solutions), the second restricted when access opened. The tasks also differed: the first was practicing problem solving, the second was reading a textbook to prepare for a multiple-choice test.

Is the effect of AI tutors positive, zero, or negative? Positive: Kestin (0.63 standard deviations, against in-class active learning), De Simone (0.24 in English, against no after-school activity), Fischer (0.34 for ask-throughout, against reading only), Wang (exit ticket +4 percentage points, against tutors teaching as usual). No difference detected: both Lehmann experiments, Huang, Bastani's GPT Tutor, and Nie's scores. Negative: Bastani's GPT Base. In none of the four positive studies was the AI a general-purpose model used straight out of the box: Kestin wrote worked solutions for each problem, De Simone's research team wrote the prompts and a teacher was present, Fischer used retrieval over the materials, and Wang's suggestions were shown only to tutors. Lehmann and Bastani's GPT Base, by contrast, were unrestricted general-purpose models. Fischer's paper doesn't describe its system prompt, and it also differs from the others in having only 25 minutes of learning with 7 textbook pages as material. I can't separate out from these studies which factor caused the difference, because each study changed several things at once.

Does AI make students more engaged or less? Kestin et al.: the AI group reported higher engagement and motivation (two of four questions; no difference was detected on the other two). Nie et al.: after AI was offered, the share of people taking the exam and submitting assignments fell. The two measured different things: the first is a survey after one class, the second is actual behavior over several weeks; in the first, all students were required to use it, while in the second only 14% did.

Do students think AI-written material is good? Denny et al.: when students rated examples written by AI and by classmates, no difference was detected on any of five measures. Leinonen et al.: students found GPT-3's explanations easier to understand and more correct, with weak effect sizes. The two studies differ in content (code examples versus code explanations) and model (Codex versus GPT-3); Leinonen's student explanations were sampled in strata by length, and Denny's AI output was not screened.

How large was Bastani's harm? This site's 2026-08-04 piece and the paper's abstract both say 17%, which is a regression specification the authors presented separately. The preregistered primary analysis is a t-test, with a gap of 0.035 points (out of 1), which I calculated as about 11% of the control group's mean. The direction and significance are the same in both.

Human hints or AI hints: which is better? Pardos and Bhandari's main experiment: no difference was detected between the two, with AI numerically higher. The same authors' pilot: human hints were significantly better. The authors attributed the difference to the AI group's high pre-test scores in the pilot.

What you can do

Every item below is my (this site's) judgment, with the results it rests on in parentheses. They are split between people who draft with AI and people who want to put an AI tutor next to their articles.

If you draft with AI:

1. Have a person or a program independently recompute every calculated answer and every derivation; don't just read it through. (Basis: all of Pardos's 32% errors had both a wrong answer and wrong steps, and none had a spelling or grammar problem, so the errors don't show on the surface of the text. Whether readers can tell has not been tested; see the section above. Limits: only one study, one old model, only math problems.)
2. If a recomputation doesn't match, remove or rewrite the whole problem; don't change a few words and keep it. (Basis: Pardos removed whole problems, and the learning experiment used the pool after removal. This is what that study did, not a condition that was compared.)
3. For important calculations, have the model generate several independent responses, and treat inconsistent answers as a warning sign. (Basis: with multiple generations and a majority answer, the algebra error rate fell to 0–2%, while statistics remained at 13%. Limits: the grouping and judging were done by hand, and this didn't enter the learning experiment.)
4. Don't take readers' praise as evidence that the content is correct. (Basis: the correctness in Denny and Leinonen was a score students gave, and nobody checked it; in Leinonen, a one-sentence explanation with nothing wrong in it was rated low.)
5. Examples written by AI tend to look like the examples you gave it, so add a few of different kinds yourself. (Basis: Denny et al.'s AI output stuck closely to the examples, with little variety. Only one study, and the content was short C functions.)

If you want to put an AI tutor next to your articles:

6. Expect most readers not to use it. (Basis: Nie 14.2%; in Huang, 61 of 95 people never logged in or logged in during only one period. Both were voluntary, ungraded tools, conditions similar to a sidebar next to a blog; but both were formal courses, and no study has measured usage rates among blog readers. Part 10 covered whether readers answer the questions in an article, which is the same kind of problem.)
7. If you want it to answer practice problems, give it the correct answers first. (Basis: Bastani's unguarded version got about half wrong in the researchers' rerun; both GPT Tutor and Kestin's tutor had worked solutions for each problem in the prompt, and Kestin's authors wrote that accuracy relied on pre-written answers. Limits: both studies involved math and physics problem solving.)
8. Don't assume that read a section yourself first, then open AI is necessarily better. (Basis: in Fischer, the only direct test, the direction was opposite. Limits: only one 25-minute experiment, with the difference only at the 10% level.)
9. Keep warnings and explanation pages shown before use short, or put them inside the conversation; don't block the entrance with them. (Basis: Nie et al. suspect the explanation page itself created friction. This is the authors' hypothesis and was not tested; between a positive and a cautionary version of the explanation page, no difference in long-term engagement was detected.)
10. Don't treat it as something that can judge for readers whether they understood. (Basis: in Bastani, Fischer, and Lehmann, some AI group rated itself higher than its actual scores could explain; Bastani and Fischer set two separate results side by side, Fischer's comparison wasn't tested statistically, and Lehmann's was an exploratory analysis; Fischer's ask-throughout group slightly underestimated itself.)

Checking this site's practice

Three things you can do tonight

1. Pick an article drafted with AI that contains calculations or derivations, and recompute every number yourself or rerun it in code. Rewrite any passage that doesn't match.
2. In the same article, pick the most important example and ask the model to generate it independently three times with the same prompt. Compare whether the three answers agree; if they don't, go back to the original material.
3. If your site or course has an AI Q&A next to it, look again at the explanation or warning in front of it and ask yourself: do readers have to get through this before they can start asking? If so, try shrinking it to one sentence and putting it in the placeholder text of the input box.

Limits of this evidence

• None of the studies looks at self-learners reading web articles. The closest are Nie's free online course and Huang's optional tutor, and both are still courses with assignments and exams.
• No delayed tests. Every experiment tested right after learning, at the end of the session, or within a day. Of the two that tested later, Huang's in-class exam in week 14 covered the last segment's content, and Wang's end-of-year test was not a randomized comparison. Nie's exam came 9 days after access opened, which is not a delayed test either. These studies can't answer whether the effects of AI tutors last.
• The models are old. GPT-3, GPT-3.5, Codex, and GPT-4, all versions from 2022 to 2024. Pardos's 32% can't be taken as the error rate of current models, and these studies can't tell us how different things are now.
• Many have not gone through journal review. Lehmann, Nie, Wang, Huang, and Denny are preprints; De Simone and Fischer are working papers. Kestin's authors say they do not presume structured AI tutoring will win in all contexts, Fischer's authors limit their conclusion to "at least in our setting", and Huang's authors call theirs a preliminary study.
• Each experiment changed several things at once. Bastani's control group had no device, and the guardrails combined withholding answers with the teacher's correct solutions; Kestin's AI group changed location, videos, and pacing at the same time; De Simone's AI group got 18 extra hours of class and a teacher; Nie's treatment group received an email, read an explanation page, and could use the tool, all at once.
• Researchers have ties to the tool developers. What the papers state: the first author of Wang et al. worked part-time at the partner tutoring company during the study; Fischer et al. thank and collaborated with a co-founder of the tutoring platform; in Kestin et al.'s supplementary materials, the platform is labeled TeachGPT.com, and the author contributions say the software was conceived and developed by the first author. Kestin et al. declare no conflicts of interest; the other two papers have no conflict-of-interest statement. These are verifiable facts; I don't speculate about their effect on the results.
• No study has directly tested whether readers can spot AI's errors.
• Several details that affect interpretation are missing from the papers: Denny et al. don't give the number of students who rated; Leinonen et al. don't give GPT-3's prompt, temperature, or whether outputs were selected; Kestin et al.'s post-test counts (142 and 174) don't match the 194 included students, with no explanation, and the paper doesn't say when the question on which 83% said the AI's explanations were as good or better was asked or how many answered; Huang et al. give neither the original assignment counts nor attrition by group; Fan et al. don't say whether essays were scored by people blind to group.
• My inference appears in several places: the section on whether readers can tell when AI got something wrong; the ten things you can do and the three things for tonight; connecting the perception studies to this site's review process; and connecting the studies of optional tools to the Ask AI sidebar.
• Studies not cited: the 2025 meta-analysis by Jin Wang and Wenxiang Fan (a different study from the work of Rose Wang and Yizhou Fan cited here) was retracted on 2026-04-22; see AI Makes Things Smooth Exactly Where They Should Be Hard.

Appendix: design and numbers for each study

The main text keeps only one or two numbers per item; the rest are here.

Pardos and Bhandari (2024), PLOS ONE 19(5). Content quality: single responses from ChatGPT 3.5 "Dec 15 Version" to problems from four OpenStax textbooks; six undergraduate raters, Fleiss' κ 0.857 to 0.929. Failures: Elementary Algebra 6 of 24, Intermediate 7 of 15 (47%), College 3 of 11, Statistics 7 of 24; all with both a wrong answer and wrong steps. The text gives an average of 24 of 75 (32%); adding up the table gives 23 of 74 (31.1%). Self-consistency (10 responses per problem, taking the modal answer): 0%, 2%, 2%, 13%. Median length of ChatGPT hints 355 words, human hint sequences 277. Learning experiment: 2023-02-28 to 05-10; 394 recruited, 120 excluded, 274 analyzed (control 90, human hints 86, ChatGPT 98), 16 to 28 per cell, against a planned 35 per cell. Overall gain (percentage points on the 3-item test): control 1.85, human 11.62, ChatGPT 17.00. Main effect of condition F(2, 262) = 5.037, p = 0.0071; condition × subject not significant. Dunn's test: ChatGPT versus control p = 0.011; human versus control p = 0.087; ChatGPT versus human p = 0.416. Both hint groups took longer to answer than control, with no difference detected between them. By subject, the only significant pre-to-post gains were in all three Elementary groups (including control, at 22.66%) and the Statistics ChatGPT group. The ChatGPT group's overall pre-test was 43.51%, about 10 percentage points below the other two groups; the authors report pre-test balance only within each subject. Human checking took 37.60 seconds per hint on average; the paper doesn't report quality-check time for human hints. Pilot: 77 people, human hints significantly better.

Denny, Khosravi, Hellas, Leinonen, Sarsa (2023), arXiv:2306.10509 v2. University of Auckland first-year C programming course, 950 enrolled in 2022. Codex (code-davinci-002), temperature 0.7, few-shot with 6 examples from the teaching team, generating 100 resources (101 in the table), not screened. Student resources: 886, with 4,053 ratings; AI resources: 101, with 446 ratings. Five rating measures (all with a median of 4): code correctness 4.09 versus 4.03 (p = 0.46), explanation correctness 4.01 versus 4.10 (p = 0.13), helpfulness 3.91 versus 3.89 (p = 0.72), overall quality 3.94 versus 3.85 (p = 0.07), rating confidence 3.95 versus 3.97 (p = 0.43), students first. Median code length: AI 224 characters, students 377; explanation length: AI 395, students 303. Ratings were analyzed per rating, without handling repeated ratings by the same rater or of the same resource.

Leinonen et al. (2023), ITiCSE 2023. University of Auckland first-year C programming course, about 1,000 enrolled in 2022. Three C functions that traverse arrays, each receiving 963 student explanations; split into three strata by word count (shortest 10%, middle, longest 10%), with 3 sampled per stratum, 27 in total; GPT-3 wrote 9 per function, 27 in total. 954 students each rated 4 explanations. Ease of understanding 3.75 versus 4.12 (p = 3.0 × 10⁻³¹, RBC 0.206); accurate summary 3.78 versus 4.0 (p = 2.9 × 10⁻¹², RBC 0.123); perceived length 2.75 versus 2.66 (p = 0.025, not past the Bonferroni threshold of 0.0125); actual length 811 versus 760 characters (p = 0.764), students first. Of 100 open-ended responses: 57 focused on specific aspects and 65 on general aspects; line-by-line explanations 21%, variable names and inputs/outputs 36%, defining terms 8%; 40% mentioned length.

Bastani, Bastani, Sungu, Ge, Kabakcı, Mariman (2025), PNAS 122(26), including the appendix. One Turkish high school, fall of the 2023–2024 school year, grades 9 to 11. Main sample 839 (GPT Tutor 277, control 320, GPT Base 242), excluding gifted classes and students who didn't fill in the survey; 943 including gifted classes. Four sessions of 90 minutes each, together about 15% of that semester's math classes. Model GPT-4-0613. Scores from 0 to 1. Practice (not preregistered): GPT Base +0.137, GPT Tutor +0.361, control mean 0.284. Exam (regression): GPT Base −0.054 (SE 0.022), GPT Tutor −0.004 (SE 0.013), control mean 0.321. Preregistered t-test: GPT Base −0.035 [−0.057, −0.012], p = 0.003; GPT Tutor −0.006, p = 0.613. Across robustness specifications, GPT Base falls between −0.041 and −0.069, and GPT Tutor is never significant. GPT Base on practice problems: 51% correct, 42% logic errors, 8% arithmetic errors. Perceptions (Appendix Table B.8): GPT Tutor self-rated exam performance +0.130 (p < .05), value of practice +0.271 (p < .01), self-rated amount learned not significant; GPT Base not significant on either. Heterogeneity (previous year's GPA, private tutoring, hours of self-study) was not significant on the exam. Time on platform: GPT Base averaged 15.5 minutes, GPT Tutor about 2 minutes more.

Lehmann, Cornelius, Sting (2025), arXiv:2409.09047 v2. Study 2: June 2024, N = 107, 79% had never written Python; Study 3: three weeks later, N = 69, copy and paste allowed. A 20-item pre-test, a 45-minute learning phase with 24 units, and a 20-item post-test, with GPT-3.5-turbo-0125 only in the learning phase. Study 2 post-test 7.9 versus 9.0 (p = 0.206), regression p = 0.580; Study 3 post-test 6.8 versus 8.8 (p = 0.092), regression p = 0.298, learning phase +2.614 (p = 0.013). Message types: asking for solutions 54.1%, asking for explanations 24.4%. Exploratory: copy and paste on understanding −1.407 (p = 0.009), on overall post-test −0.249 (p = 0.737); number of solution requests on understanding −0.312, and −0.223 (p = 0.090) after adding total messages; number of explanation requests on understanding +0.249 (p = 0.05). Above-median pre-test on understanding +1.613 (p = 0.034), below-median −0.950 (p = 0.043). Self-rated learning +0.338 (SE 0.166, p = 0.044, N = 176). Field data (Study 1, 113 people) used the similarity between students' code and ChatGPT's solutions as a proxy; per unit of cumulative similarity, scores on later problems were −0.02 to −0.06.

Kestin, Miller, Klales, Milbourne, Ponti (2025), Scientific Reports 15, including the supplementary materials. Harvard PS2, weeks 9 and 10 of fall 2023, 233 enrolled and 194 included (Group 1 96, Group 2 98), assigned by peer group. Baseline FCI 13.3 versus 15.2 (p = 0.28). Classroom condition 75 minutes, of which 15 were the test; the authors assumed 60 minutes of learning time. Model GPT-4-0613. Post-test median AI 4.5 (N = 142), classroom 3.5 (N = 174); pre-test median 2.75 (both groups combined); from the medians, I calculated gains of 1.75 versus 0.75. Standardized regression coefficients: condition 0.63, pre-test 0.18, prior AI experience −0.15, time on task 0.1 (not significant), R² = 0.21. AI group median time on task 49 minutes, with 30% over 60 minutes. Perceptions: engagement 4.1 versus 3.6, motivation 3.4 versus 3.1, both significant; no difference detected for enjoyment or growth mindset; none of the four questions asked how much students learned. 83% of students said the AI's explanations were as good as or better than the classroom instructor's.

De Simone, Tiberti, Barron Rodriguez, Manolio, Mosuro, Dikoru (2025), World Bank PRWP 11125, December 2025 updated version. 9 public schools in Benin City with computer labs, 2024-06-03 to 07-11. Lottery among first-year senior secondary volunteers: treatment 657, control 671; 422 and 337 completed the final assessment. Average attendance in the treatment group about 72%. Main regression N = 654: weighted total 0.310 (SE 0.068), IRT 0.263, English 0.238 (SE 0.068), AI knowledge 0.309, digital skills 0.139 (10% level); school third-term English exam 0.206 (SE 0.067, N = 636). Inverse probability weighting: total 0.272, English 0.194, school exam 0.134, digital not significant. Lee bounds: English 0.202 to 0.238. Dropping one school at a time: 0.156 to 0.360, significant only at the 10% level when the largest school is dropped. Heterogeneity: interaction for girls 0.420 (boys −0.039 in the same specification, not significant; the authors say this is driven by one girls' school), baseline scores 0.151, socioeconomic status 0.113, all p < .05. Each additional day attended about 0.033 standard deviations (with the lottery as an instrument). Cost about 48 US dollars per student, not including electricity and school facilities. Two schools were unbalanced between groups in school composition.

Fischer, Rau, Rilke (2025), IZA DP 18338. Technical University of Berlin lab, February 2025, 334 analyzed (control 112, throughout 102, read-first 120). Materials were two excerpts from Varian's Intermediate Microeconomics with Calculus; note-taking was not allowed. GPT-4 retrieval tutor on the acemate platform. Standardized test 2 scores: combined 0.227 (SE 0.106); throughout 0.337 (SE 0.116); read-first 0.132 (SE 0.122); difference between the two p = 0.066. Time before the first question: read-first group about 11.2 minutes, throughout group about 5.7 minutes. Number of questions 6.1 versus 7.0, no difference detected. Beliefs: the control group thought one should read for about 17 minutes first; the two AI groups, about 15.5 minutes. Enjoying preparing for and taking the test: read-first group +0.308 (p < .05), throughout group not significant. Post-test self-estimate: throughout +0.195 (p < .1), read-first +0.217 (p < .05). Of five moderators in the heterogeneity analysis, only one had a significant interaction; the authors say there was no multiple-comparison correction and it can only be taken as a hint.

Nie, Chandak, Suzara, Malik, Woodrow, Peng, Sahami, Brunskill, Piech (2025), arXiv:2407.09975 v2. Code in Place, spring 2023: 8,762 registered, and the 5,831 still active after the first week entered the experiment, 60% treatment (3,581) and 40% control (2,250). Intervention lasted 9 days. Notification email opened 77.6%, link clicked 15.1%, used before the exam 14.2% (510 people). Exam participation 44.1% versus 48.5% (−4.3 percentage points, corrected P = .020; another place in the paper says .006); week 6 assignment −4.6 percentage points; section attendance not significant after correction. All 15 tests in the paper were Bonferroni-corrected. Among exam takers, mean 87.0 versus 86.1, with a median of 95.9 in both groups. Intention-to-treat effect for everyone: counting only exam takers +0.91 [−0.00, 1.89], with imputation +0.67 [−0.37, 1.72], both 90% confidence intervals. Effect on adopters (instrumental variable, with imputation) +6.86, 90% CI [0.30, 14.13], not significant after correction. Conversation topics: conceptual clarification 37.6% to 53.3%, trying to get assignment answers 1% to 4%. In low Human Development Index countries (180 people), participation went the opposite way (+14.8 percentage points), not significant after correction. The paper has several inconsistent numbers; for example, the group labels for participation are reversed in one place.

Huang, Thoeni, Fryer (2025), OSF preprint v2. An introductory marketing course at a state university in the US Southeast, one online section and two in-person sections. Final 253 (control 126, treatment 127). The tutor combined the ChatGPT-4o API with course vocabulary, chapter summaries, and example questions (another place says it was a RAG system built on Microsoft Azure). Group × time: course interest β = −0.02 (p = .645), self-efficacy −0.01 (p = .739), exam 0.04 (p = .418). Raw scores: control pre-test 77.9% → post-test 80.5%, treatment 76.2% → 80.1%; from the table, I calculated a difference of 1.3 percentage points; the paper reports no effect size or confidence interval for the between-group difference. The pre-test and post-test covered different content. Within the treatment group, students were split into four categories by number of login periods (None 18, One-time 43, Occasional 19, Regular 15), with no difference detected on any of the three outcomes; this part is observational. In-person students logged in more (p < .001); students without jobs logged in 6.02 times on average, those with jobs 2.77.

Wang, Ribeiro, Robinson, Loeb, Demszky (2025), arXiv:2410.03017 v2, preregistered. FEV Tutor and a school district in the US South, two months starting at the end of March 2024. The district designated 1,787 students; sessions with data were only in grades 3 to 6. The paper gives the number of tutors as 900, 874, and 879 in different places; 782 were teaching when the study began. Each tutor taught about 5 sessions on average, about 200 minutes. 4,136 sessions analyzed. Exit ticket passed: +0.04 (SE 0.01), control 0.62; exit ticket attempted +0.02 (p < 0.1); conditional on attempting +0.03 (the table says p < 0.05, the text says p < 0.01). Using assignment as an instrument to estimate the effect of the tool being used in a session: +0.14. The lowest-rated third of tutors by quality +9 percentage points, the least experienced third +7, with no tables or test statistics. MAP math −0.35 (SE 0.88), reading +1.44 (SE 1.45); without imputing baselines, −0.59 and +1.56; none significant, n about 1,000. About 29% of treatment-group sessions used the tool. API cost totaled 1,419.66 US dollars over two months. No difference was detected on any of the five student survey questions, the four tutor survey questions, or the engagement score.

Fan et al. (2025), BJET 56(2); I read the submitted manuscript, arXiv:2412.09315. 117 university students with English as a second language, July to September 2023. Procedure: 30-minute pre-test, 120 minutes of reading and writing, 60 minutes of revision, post-test. Group sizes are inconsistent within the manuscript (the methods section gives CN 30, AI 35, HE 25, CL 27). No difference was detected in essay scores before revision. Improvement scores F = 4.549, p = 0.005; AI exceeded no support by 1.970 [0.083, 3.858], human expert by 2.120, and checklist tool by 2.200, all significant after Tukey correction; no difference was detected for the other three pairs. Knowledge post-test F = 0.913, p = 0.438; transfer F = 0.019, p = 0.996. No difference was detected on any of the four motivation subscales. Process logs: during revision, the AI group showed more orientation than the no-support group, with no difference detected in evaluation; only the checklist group's evaluation increased. In the discussion, the authors say the AI group had fewer metacognitive processes, which doesn't match this figure. An appendix table contains a pasting error (the checklist group's post-revision scores are identical to its pre-revision scores); I did not check whether the published version corrected it.

References

I read each item from start to finish and kept an extraction record; the version I read is noted in parentheses.

• Pardos & Bhandari (2024), ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills, PLOS ONE 19(5): e0304013 (published version, archived via eScholarship; I did not download the figshare data).
• Denny, Khosravi, Hellas, Leinonen & Sarsa (2023), Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources (arXiv preprint v2).
• Leinonen et al. (2023), Comparing Code Explanations Created by Students and Large Language Models, ITiCSE 2023 (I read the author's version posted at arXiv:2304.03938).
• Kestin, Miller, Klales, Milbourne & Ponti (2025), AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting, Scientific Reports 15 (published version and the publisher's supplementary materials; I did not read the data on GitHub).
• Bastani, Bastani, Sungu, Ge, Kabakcı & Mariman (2025), Generative AI without guardrails can harm learning: Evidence from high school mathematics, PNAS 122(26), doi:10.1073/pnas.2422633122 (the published version on PMC and the SI appendix; the 2025-08-20 correction only changes author affiliations; I did not read the preregistration document).
• Lehmann, Cornelius & Sting, AI Meets the Classroom: When Do Large Language Models Harm Learning? (arXiv preprint v2, 2025-03-08; I did not read the preregistration document).
• Nie et al., The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters' Exam Performances (arXiv preprint v2, 2025-07-15).
• Wang, Ribeiro, Robinson, Loeb & Demszky, Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise (arXiv preprint v2, 2025-01-26, including Appendices A–M; I did not read the preregistration document).
• De Simone, Tiberti, Barron Rodriguez, Manolio, Mosuro & Dikoru (2025), From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria, World Bank Policy Research Working Paper 11125 (working paper, December 2025 updated version).
• Huang, Thoeni & Fryer (2025), Interact more, learn more? A randomized controlled trial of GenAI tutors on student course interest, self-efficacy, and learning performance (OSF preprint v2).
• Fischer, Rau & Rilke (2025), AI Tutoring Enhances Student Learning Without Crowding Out Reading Effort, IZA Discussion Paper No. 18338 (discussion paper, the legal deposit copy at the University of Bonn library; I did not read the preregistration document).
• Fan et al. (2025), Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance, British Journal of Educational Technology 56(2) (I read the submitted manuscript at arXiv:2412.09315 and did not compare it with the published version).
• Articles on this site: AI Makes Things Smooth Exactly Where They Should Be Hard, Should a Teaching Article Have a Comment Section?, Writing for Others, Learning for Yourself, Do Readers Know Whether They Understood?, Do Readers Answer the Questions in an Article.