Readers Saying It's Well Written Doesn't Mean They Learned: What Self-Ratings, Liking, Viewing Time, Practice Scores, and Proxy Metrics Each Measure

🌏 中文版

This is Part 25 of the "Writing Articles That Teach" series and the first part of its sixth section, "how to know it works, and how to make it". Part 15 looked at this from the reader's side: readers are poor at telling which articles they understood and which they didn't. Without any intervention, the within-person correlation between their judgments and their scores averages only about 0.18; answering questions before judging, or writing keywords after reading everything, makes the judgments somewhat more accurate. This part moves to the author's side. If readers' judgments are inaccurate, how much can the things readers tell the author, and the behavioral traces they leave, tell the author?

The question comes from what this site has on hand. Once a course guide is published, these are the only things the author can see:

• Traffic statistics. The production site loads at least Cloudflare Web Analytics, which Cloudflare injects automatically at the edge; it is not in the site's code. According to Cloudflare's documentation, its metrics are aggregate page views, visits, page load time, and Core Web Vitals. A visit is determined by whether the page view came from a different website or was opened directly (the referrer differs from the site's hostname). Its FAQ says custom events are not yet supported: "Not yet, but we may add support for this in the future." So it can't recognize the same reader twice, and it can't record whether anyone answered a question.
• Glossary lookup counts. Each time the site's glossary feature is used, the database adds one to the row for that term, that article, and that level, and keeps the context of the most recent lookup. Within a single page load, if the same term is opened again at the same level with the same sentence as context, the browser reuses the explanation it already has and does not count it again. There is no reader identity.
• Site chat content. After an answer in the site's Q&A chat has been fully generated, the question and answer are written to the database, along with the conversation thread ID, the source IP, and whether the site owner was logged in when asking. If a question is similar enough to an earlier one and is answered from the cache, it is not recorded; answers that never finished generating are not recorded either. So this record is not every question, and it is mixed with the site owner's own test questions (which the flag can separate out). The first 240 characters of each question also stay in the back-end step logs, which the site owner can read one by one; when Langfuse is configured, the full question and answer are sent there too.
• Review reports and checkers. The course-guide spec requires each article to get a semantic review from a reviewer who did not take part in writing it, and there are automated checks such as pnpm verify (code style, internal citations, skill sync, and so on). Both measure the article, not the readers.

Besides these, a reader occasionally says something like this one was so clear. It is easy for the writer to take that remark, or a jump in an article's traffic, as evidence that this article taught people something. This part asks what each of these signals measures, which ones are related to how much readers actually learn, and how strongly. After reading it, you should be able to say of each kind of feedback in your hands what it measures and what it doesn't, and decide whether to base a revision on it.

What this part does not cover: large-scale randomized experiments on platforms, and whether course-quality rubrics can stand in for outcome studies, are topics for a later part; for readers judging whether they understood, see Part 15; for whether readers answer questions, see Part 10.

How sources were chosen, and their biases. I scanned the literature first, then picked sources to read closely. The criterion was that a single study measured both some signal an author can see and learning measured separately, and the set should include studies with inconsistent results where possible. I ended up reading these: five experiments that measured both feelings and test results (the crossover experiment in a Harvard physics course, the generative AI experiment in a Turkish high school, the Python experiment in a German lab, the AI tutor experiment at Harvard, and the AI tutor experiment in Berlin); an experiment on video playback speed; a log analysis of a Japanese history MOOC; a teaching-practice project report from National Dong Hwa University; an A/B experiment in an MIT edX course; a Taiwanese MOOC study that predicts passing from click logs; a log analysis of a psychology MOOC; and an official Duolingo blog post about proxy metrics. The biases are clear. Most participants are university or high school students, most studies are in classrooms or labs, learning was almost always measured right after the session, and not one studies readers who go online on their own to read long articles. I read every item in the references from start to finish and kept an extraction record for each; studies I only know exist, without having read a legitimately public full text, are flagged as such in the text, and I don't cite their results.

The short version

In one sentence: the signals an author can see fall roughly into two kinds, what readers say and what readers do. What readers say moves with fluency, liking, and whether they had a tool at hand; what readers do mostly measures how much was done. Either kind may move in the same direction as learning, but neither can serve on its own as evidence of learning.

Readers say they learned a lot

The class where they learned more got the lower self-rating

Louis Deslauriers and colleagues' 2019 experiment was covered in detail in Part 8. It involved 149 students in Harvard's introductory mechanics course, randomized in a crossover at the individual level, with each student getting one class in each teaching style. In one, the instructor fluently worked through example problems; in the other, students first solved problems in small groups while the instructor circulated, and then got the same solutions. Right after class they took a 12-question multiple-choice test, on which the active class scored 0.46 standard deviations higher. On the questionnaire filled in just before the test, the active class was rated 0.56 standard deviations lower.

What matters for this part is which items made up that self-rating. The authors combined four five-point items into one score: enjoying the class, feeling that one learned a lot, the instructor being effective, and wishing all physics classes were taught this way. Only one item asks directly how much was learned. The authors say that using only that item, the results were "virtually identical", but they did not print the numbers. In other words, the questionnaire mixed in did you like it and was it explained well, and those are exactly the things readers most often say about an article.

Add explained well to the model, and teaching method stops being significant

The authors also ran a supplementary regression: they used the item the instructor was effective to predict the item feeling of having learned a lot. Students who saw the instructor as highly fluent were 0.51 standard deviations higher on learned a lot; once this item was included, the teaching method itself (active or passive) was no longer significant. In the original: "Notably, the type of instruction (active vs. passive) was not significant in predicting FOL; only the perceived fluency of the instructor was relevant."

The basis for interpreting this item as fluency is a separate set of interviews: students said they understood the item as meaning the explanation was clear, well organized, and smooth. In addition, in the main set of 17 interviews, 15 students found the active class disjointed and not smooth. On this basis the authors argue that students took a fluent explanation for having learned.

Three limitations need to stay attached. The two items come from the same questionnaire, answered at the same time, so the direction can't be separated out. The instructor was effective is itself one of the items in the composite score. Fluency was not measured separately; the interviews were used to interpret that item as fluency. So what can be said is this: in this experiment, feeling it was explained well and feeling of having learned a lot were clearly related (people higher on the former were 0.51 standard deviations higher on the latter), and once the former was included, teaching method no longer predicted the latter; neither tracked the actual test. Carried over to writing articles, this is the most direct reminder: when readers say this is very clearly written, what it measures may mainly be how smooth it felt.

The authors also include a clue pointing the other way: in the model of test scores, the interaction between pretest score and feeling of having learned a lot was positive (p < .05, coefficient not printed). The more prior knowledge students had, the better their self-rating matched their actual performance. Part 15 covered this point.

With AI available, self-ratings run high

Two generative AI experiments show self-ratings and actual results coming apart from another direction.

Hamsa Bastani and colleagues' 2025 experiment, at a high school in Turkey, randomized classes into three groups for four rounds of math practice: textbook only, an interface mimicking ChatGPT, and an AI tutor with guardrails (it does not give full solutions, and its prompt contains teacher-written correct answers and common mistakes). Right after practice came a closed-book exam, followed by a five-item questionnaire about how students felt. The group using the unguarded interface did worse on the exam, but no difference was detected in their self-ratings; the guarded group's exam results showed no detected difference from the textbook-only group, yet they rated themselves as having done better by a significant margin. In the original:

While students in the GPT Base arm performed worse on the exam (relative to the control arm), they did not perceive that they performed worse or learned less. Similarly, while students in the GPT Tutor arm did not perform better on the exam (relative to the control arm), they perceived that they performed significantly better.

This is a comparison of group averages, not a follow-up of how the students who did badly each rated themselves. For the unguarded group, did not feel they learned less means no difference was detected (the point estimate is negative); it does not mean they felt they did just as well. The numbers on feelings appear only in tables in the supplementary materials, not in the main text.

Matthias Lehmann and colleagues' preprint ran two preregistered experiments in a lab at a German university. Most participants had no Python experience and spent 45 minutes learning introductory Python; one group had an unrestricted chat window beside them. In both experiments, no difference was detected on the posttest. After the posttest, participants answered one question on how much they felt they had improved: controlling for actual improvement (posttest minus pretest), people with the LLM still rated themselves about a third of a point higher (on a five-point scale). In the authors' words: "Thus, LLMs increase perceived learning by more than can be explained by actual differences in learning."

This is an exploratory analysis combining both experiments, 176 people, with a p-value of .044. The same table also shows that each additional question of actual improvement raised the self-rating by 0.154 points, and this was significant. So this is not learned less but felt they learned more; self-ratings broadly tracked actual improvement, and people with the tool were shifted upward overall.

Mira Fischer and colleagues' discussion paper had 334 students in a lab at Technische Universität Berlin read an excerpt from an economics textbook for 25 minutes and then take a multiple-choice test. The group that had to read for 10 minutes before they could use the AI tutor raised their post-test estimate of how many questions they got right by about 0.2 standard deviations, and also enjoyed the preparation more, yet their actual scores were not significantly better than the group that only read the material. The group with access throughout raised their estimates by about the same amount, and their actual scores were 0.34 standard deviations higher. On this basis the authors say the former were "somewhat overconfident" and the latter "slightly underconfident". This comes from setting coefficients from two tables side by side, with no test. Going by the authors' untested side-by-side comparison, the full-access group does not seem to have rated itself too high, so I do not take self-ratings run high with AI as holding for every group.

How far this section can go

All four experiments measured right after the session, in classrooms or labs. The claim that holds up is this: how much readers say they learned moves with things other than learning, including how smooth the explanation was and whether they had a tool at hand. It may move in the same direction as actual learning or the opposite direction, and the author can't tell which from what readers say. No study has tested how much readers actually learned when they say they learned a lot from a web article.

Readers say they liked it, felt engaged, want to use it again

Questionnaire items on liking and engagement relate to scores in all three directions across different studies.

Same direction. Greg Kestin and colleagues' 2025 experiment used two classes over two weeks in a crossover, in a Harvard physics course for life-science students: one class used an AI tutor at home, the other did active learning in the classroom. The AI tutor class had higher posttest scores (regression coefficient 0.63 standard deviations), and students reported higher engagement (4.1 versus 3.6) and motivation; no difference was detected in enjoyment of the class or growth mindset. Note that none of the four items asks how much do you think you learned. This paper is often cited as a counterexample to Deslauriers, but the two do not measure the same thing: what ran opposite in Deslauriers is self-rated learning, and what ran in the same direction in Kestin is a sense of engagement.

Worse ratings, no detected difference in scores. Toru Nagahama and Yusuke Morita's 2017 experiment had 75 Japanese university students watch a roughly 9-minute video from a high school information-studies class (slides and audio only), split into three groups at 1x, 1.5x, and 2x speed, and tested them right after. The main effect of speed was not significant for any of the three scores. On the questionnaire, 2x speed was rated significantly lower than the other two speeds on seven items, including I understood the learning content and the lecturer's explanation was easy to understand, and eye strain and difficulty following the text were significantly higher.

This can't be read as students found 2x speed harder but actually scored the same. First, the power the authors themselves report is only 0.17 to 0.65, and mean scores on the recall questions did decline with speed, just not significantly. Second, the ratings and the test did not measure the same experience: in the test, each group watched the full video at one speed, while for the ratings, after the test, every participant watched short clips at all three speeds and then rated them. Third, 1.5x speed got the highest ratings on concentration, appropriate length, and wanting to use it in future, and its scores were no worse; that is an example of ratings and scores not coming apart.

No detected difference in ratings or scores. Chih-Hung Lai (賴志宏) of National Dong Hwa University wrote a teaching-practice research project report comparing two cohorts of an online general-education Python course: in the experimental cohort, multiple-choice questions were embedded in the videos, which stopped at that point to pose the question; in the previous year's cohort, the questions sat outside the videos. No difference was detected in learning motivation, self-efficacy, satisfaction, or the achievement test. This report comes up again in the next section.

These three studies asked different questions (engagement, ease of understanding, satisfaction), with different participants and settings, so they can't be combined into one correlation between liking and learning. All that can be said is this: readers liking something and feeling engaged may come along with learning more, or may not.

How long readers watched

This is the number an author can get most easily, and the number most easily read as it works.

Viewing time went up, no detected difference on the achievement test. Back to the Dong Hwa report. In the cohort with questions embedded in the videos, the system-recorded viewing time averaged 11,400, against 8,898 the year before, a significant difference (t = 2.03). On the achievement test, adjusted with the pretest as a covariate, the scores were 10.42 versus 10.31 out of 15, with no detected difference. The report's abstract describes the increase in viewing time as having met the study's goal.

The evidence level of this report should be recorded as it is: the comparison group was the previous year's class, not a concurrent random assignment; only 49 and 41 people had complete data; it is a project outcome report without peer review, and it gives the number of valid questionnaires inconsistently (74 in one place, 90 in another). The report does not say whether the time the video sat paused waiting for an answer counted as viewing time; if it did, part of the increase in viewing time may have been caused by the mechanism itself.

Viewing time predicts completion, not completers' thinking ability. Hiroki Oura and colleagues' 2018 study analyzed a four-week MOOC on medieval Japanese history on gacco. Among the 2,900 learners who scored at least 1 point and gave valid answers to the required items on the course-start questionnaire, those who watched at least 6 hours of video (about one pass through all of it) had an odds ratio of 8.35 for completing. Separately, the authors asked learners at the start and end of the course to write a 300-character historical argument, which two experts coded as historical thinking ability. Among the 1,224 learners who completed and gave valid answers on both the start and end questionnaires, viewing time had no significant effect on thinking ability at the end (odds ratio 1.15, p = .161), and neither did the number of days active on the discussion board.

The authors' reading is that the assignments required for completion were based on the video content, so watching the videos naturally made completion easier, while watching the videos alone was not enough to develop thinking ability. Three conditions need to travel with this. These are observational data, not an experiment. The thinking-ability analysis includes only completers, whose median viewing time already exceeded the length of all the videos; the paper does not discuss how much variation in viewing time remained in this group. Thinking ability was measured with a single free-response argument, and only about 7% of registrants filled in both questionnaires.

What this means for an author: finishing and learning something else are two different outcomes within the same course, and behavioral logs that predict the former don't necessarily predict the latter.

A different filming style, no consistent direction in engagement or learning. Zhongzhou Chen and colleagues' 2016 paper ran two A/B experiments in MIT's edX mechanics course, one of which compared two styles of filming videos. The abstract says the new style increased engagement and had no significant effect on learning. The tables are messier than the abstract. Of four videos, only one had a significantly higher share of watched at least 70% in one uninterrupted viewing; average viewing time was longer for the old style in two videos and for the new style in one; and on the final correct rate for the same question after watching, two videos showed significant differences but in opposite directions. The authors' own results section says they "did not observe any consistent trend".

The thing most worth remembering from this paper is one remark by the authors: "Unfortunately, most MOOC video studies made no attempt to determine whether students learned from the videos." This is the authors' view, not a result of these two experiments.

How much readers did, how much they got right in practice

Among behavioral logs, doing is closer to learning than watching; Part 8 and Part 9 both covered this. Kenneth Koedinger and colleagues' 2015 study analyzed a psychology MOOC: doing one standard deviation more interactive practice went with 0.44 standard deviations higher total weekly quiz scores, more than six times the figure for watching more video or reading more pages (about 0.065 each). Two conditions relevant to this part need adding. The quiz questions and the interactive practice came from the same question bank. On the final exam written by the instructor instead, the share of high scorers among people who only did more practice (16%) was about the same as among people who only watched more video (15%); only people who did more of both reached 31%. So how much was done moves with the quizzes, and its relationship to a different exam is less simple.

Two other experiments show that how much was right in practice comes apart from learning when a helper is available.

• In Bastani and colleagues' experiment, the two groups with AI available during practice had sharply higher practice scores (relative to the textbook-only group, +48% unguarded and +127% guarded). On the closed-book exam right after, the unguarded group actually did worse, and no difference was detected for the guarded group. The comparison of practice scores was not in the preregistered analysis.
• In Lehmann and colleagues' second experiment, people with the LLM got about 2.6 more questions right during the learning phase (significant), and no difference was detected on the posttest.

Carried over to articles: if readers answer questions while reading with a chat window or the answers open, getting answers right measures able to find it, not able to do it. Part 15 covered the same point from the angle of judgment accuracy.

Models that predict passing

Predicting from click logs whether students will pass is a very common topic in learning analytics. Chen-Hsiang Yu and colleagues' 2019 study used logs from three runs of one course on Taiwan's OpenEdu platform to predict with a neural network whether learners earned a certificate, with an accuracy of about 0.95; using the model from the first two runs to predict the third, about 0.98. The paper is titled "Predicting Learning Outcomes with MOOC Clickstreams".

These numbers can't be read as video-watching behavior predicts learning very well, for three reasons.

1. What is predicted is pass or fail; the paper's definition is literally "pass (1) and fail (0) of the course", with no measure of learning independent of course grades.
2. Of the 14 features in the model, 4 relate to tests, including the total unit-test score. Using test scores to predict a pass or fail that is determined by test scores makes high accuracy partly inevitable; the paper did not build a model without the test features, so there is no way to see how accurately video-watching behavior alone predicts.
3. There is no comparison with the simplest guess. I calculated from the paper's counts: in the third run, 370 of 427 people did not earn a certificate, and in another course, 891 of 977 did not; guessing fail for everyone already gives an accuracy of about 0.91 in the latter.

Oura and colleagues' study has the same relationship: the assignments required for completion are based on the video content, so completion is itself a test of the video content. What gets called a learning outcome needs to be looked at closely first.

Proxy metrics a platform picks for itself

Duolingo's Rachel Salvador explained in a 2024 post on the official blog what they use to stand for learning in everyday A/B tests, and why the metric was switched twice, with one more adjustment in between. The post says formal efficacy research is done by a separate team, but those studies can't be used for daily A/B tests, so a proxy metric is needed.

1. The first was the total number of completed sessions. In the original: "The idea was more sessions = better for learning. If you do more, you learn more, right? Not quite!" It was replaced because session length varies a lot, and this metric favors people who repeatedly grind short, easy lessons.
2. The second was time spent on learning activities. The total was driven by a small group of very diligent people, so they switched to tracking the share of people who learn at least 15 minutes a day.
3. The third is Time Spent Learning Well: minutes in lessons that move down the learning path count in full; minutes in other lessons, such as review and challenges, count half. The author calls this a "rough formula", and the post does not explain where the 0.5 weight comes from.

One result in the post is worth remembering: they tried shortening lessons in the hope that people would do more of them, and this metric went down instead.

This post is a product team's own account, not research: there is no sample size and no test, and nowhere in the post is this metric compared with learning measured separately. Its only external support is a white paper it cites as saying the number of completed lessons is the strongest predictor of learning gains, and lesson counts are exactly the kind of metric the post's first section says was retired. The first two changes (dropping session counts, and changing total time to the share of people who hit a daily threshold) were made because the old metric could be inflated by certain behaviors (grinding short lessons, a few heavy users). For the switch to a 15-minute daily threshold, the post says it drew on "several formal studies", but gives no citation or link. The last switch, to Time Spent Learning Well, was made because different activities have different learning value. None of the changes came from comparing the metric with separately measured learning and finding a mismatch. For someone who writes articles, what this post shows is that even a company with a dedicated efficacy research team uses time-based proxy metrics for everyday decisions, and has to keep revising them.

Time on page, scroll depth, Was this page helpful?

For the three numbers blog authors see most often, I found no study that measured both the number and readers' understanding. This round, I only searched in the direction of course platforms, not web usability research, so I can only say we didn't find any, not there are none. For the Was this page helpful? buttons common on documentation sites, I found no study testing their validity; I didn't read the full text of the few practitioner articles about them, so I don't cite them.

Two related meta-analyses exist, but I didn't read a legitimately public full text, so I don't cite their results: Klose and colleagues (2022) on the relationship between online learning log data and grades, and Uttl and colleagues (2017) on the relationship between teaching evaluations and learning (Deslauriers and colleagues cite the latter at the end to support using teaching evaluations with caution).

What each of this site's signals measures

Below is my judgment from mapping the sections above onto this site, item by item. Closest research is only an analogy in direction; none of it measures readers of web articles.

Of the first five rows, the closest thing to something readers produced is the chat questions: at least the reader wrote them. But readers who speak up select themselves, and their questions mostly point to where they got stuck. The back end has a page for reading step logs one by one and a count of questions over a period, but I found no code that analyzes what the questions say.

No row in this table can measure whether readers learned. This is not a gap peculiar to this site: unless readers hand in something that can be scored independently, and the author can record it, the author of a public article simply doesn't have this column.

Where the sources disagree

Are feelings and learning opposite, aligned, or aligned but shifted upward? In Deslauriers's crossover experiment, opposite; in Kestin's crossover experiment, aligned; in Lehmann's experiment, self-ratings correlated positively with actual improvement, and people with the tool were shifted upward overall. The three measured different feelings: Deslauriers used a four-item composite (including enjoyment and the instructor was effective), Kestin used engagement and motivation (without asking how much was learned), and Lehmann used a single self-rated improvement item. What was compared also differed: passive lecture versus group problem solving, classroom versus AI at home, with or without a chat window. They don't directly conflict, but they also can't be combined into one answer to are feelings accurate.

Do self-ratings run high with AI? Lehmann's pooled data and Bastani's guarded group ran high; Bastani's unguarded group did worse on the exam with no detected drop in self-rating; for Fischer's full-access group, the authors say they slightly underestimated. The four groups differ in tools, tasks, and measures.

Abstracts and tables say different things. Chen and colleagues' abstract says the new filming style increased engagement and had no significant effect on learning; in the tables, the long-viewing share was significant for only one video, average viewing time differed significantly for three videos but not in a consistent direction, and learning was significant for two videos but in opposite directions. Nagahama and Morita's Japanese abstract says speed does not affect learning outcomes, while the results section uses the word "示唆された" (was suggested); when the English abstract discusses the possibility of learning twice as much in the same time, it drops the qualifier in the Japanese abstract, and the experiment did not measure this. Yu and colleagues' abstract says they found a correlation between video-watching behavior and learning outcomes, but the full text contains no correlation coefficient. For these three papers I go by the body text and tables, not the abstracts.

Do practice scores count as a learning outcome? In Chen and colleagues' other experiment, the authors called drag-and-drop questions leading to more correct answers on later practice questions (looking only at students who got it wrong at first) "quicker learning", but on a quiz with conventional questions, no difference was detected between drag-and-drop and multiple choice. Koedinger's quizzes share a question bank with the practice, and the final exam gives a different picture. In Bastani and Lehmann, practice scores rose and exam scores didn't follow. Each paper defines learning outcome differently; when citing one, state which score it is.

What you can do

The five points below are this site's judgment, and each states which results it rests on.

1. Record readers' very clear as readers feeling it was smooth, not as the article working. How: when you get this kind of feedback, put it in a reader feelings column, separate from what readers were able to do; when revising, don't use everyone says it's clear as a reason not to change something. Basis: in the Deslauriers experiment, the explained well item alone predicted learned a lot, and neither tracked the test; in the Lehmann and Bastani experiments, self-ratings drifted from actual scores. Distance: all classrooms or labs, and the readers there were talking about a teacher, not an article.

2. Don't use time on page, page views, or viewing time to judge whether a revision is better. How: if you look at traffic after a revision, use it only to answer how many people reached this page; don't write sentences like readers read longer after the revision, so it's easier to understand. Basis: in the Dong Hwa report, viewing time rose with no detected difference on the achievement test; in Oura and colleagues, viewing time predicted completion but not thinking ability; Duolingo itself says total learning time is driven by a few heavy users, which is why it changed the metric. Distance: all videos or apps, no web articles.

3. Read glossary lookup counts as which terms readers want to look up, and use them to decide where to add explanations. How: check them every so often, and for the most-looked-up term in an article, add a one-sentence plain-language definition where it first appears; don't treat lookups went down as readers understanding. Basis: this data has no reader identity and no record of anything after the lookup; in other studies, behavioral measures mostly measure what was done, not what was learned. Distance: no study has tested this point directly.

4. If you want a learning signal, readers have to produce something without seeing the answer and without a helper. How: in the synthesis question at the end of an article, state explicitly close the article, don't open the chat, write your answer down first and then check; see the approach in Part 15. Basis: in the Bastani and Lehmann experiments, practice scores rose with a helper while exam scores didn't follow; Koedinger's quizzes shared a bank with the practice, and a different exam gave different results. Distance: this site can't record readers' answers, so this point improves readers' own judgment, and the author still gets no data.

5. Before adding any event logging, write down which non-learning behaviors would also make it go up. How: write a short table; for example, expanding an answer goes up because of curiosity, skimming, or going straight for the answer. If you can't write one, you haven't yet worked out what it stands for. Basis: Duolingo's total lessons favored people grinding short lessons and total learning time was driven by a few heavy users, so both were replaced; the Dong Hwa report's viewing time may include time paused waiting for an answer. Whether to add logging is a separate decision for this site (changes to the site's data collection need the site owner's approval first); this point only says what to think about before adding any.

Checking this site's course-guide spec

This site's course-guide spec (course-guide-contract.md), in its section on completion evidence and preventing recurrence, says three things related to this part: each article must get a semantic review from a reviewer who did not take part in writing it, explaining item by item how the learning goals, the mechanisms in the body, the examples, the exercises, and the answers line up; a green verify run or fully confirmed sources cannot substitute for teaching acceptance; and no one may claim that lint can automatically judge whether something was learned.

The spec does not treat traffic or reader feedback as acceptance criteria, which points in the same direction as the evidence read for this part.

Limits of this evidence

• No study's participants are readers of public web articles. All the studies are in classrooms, labs, or course platforms, and the learners are students who were assigned or who registered; readers of long web articles are not assigned and take no test. Every step from these studies to this site is an inference.
• Learning was almost always measured on the spot. Deslauriers, Bastani, Lehmann, Kestin, Fischer, and Nagahama and Morita all tested right after learning, with no delayed test. The author's own signals (what readers say) also mostly appear right after reading, but no study has measured how the two relate after some time has passed.
• Feelings were measured in different ways. A four-item composite, a single self-rating item, engagement, liking, and ratings of playback speed can't be converted into one another.
• The evidence is uneven. The Dong Hwa report was not peer reviewed and its comparison group is the previous year; Fischer is a discussion paper; Lehmann is a preprint; Duolingo is a company blog; Oura and colleagues and Yu and colleagues are observational data; the Kestin paper does not mention preregistration, and its posttest counts don't match the number included.
• Tests that detected no difference often lack power. Nagahama and Morita report their own power as 0.17 to 0.65; the Dong Hwa report has 40-odd people per group; Lehmann's two experiments had 107 and 69 people. No difference was detected does not mean the same.
• Details the sources don't report: the separate set of interviews on which Deslauriers and colleagues based reading that item as fluency has no reported size; the Dong Hwa report gives no unit for viewing time; Nagahama and Morita don't report group sizes.
• My inferences: that readers' very clear mainly measures smoothness (extended from the classroom to articles); that viewing time with embedded questions may include paused time; that getting answers right with a helper measures being able to find them; that chat questions mostly point to where readers got stuck; how to read glossary lookup counts; how to read the spec's teaching acceptance; and the five points under what you can do.
• Known to exist, but no legitimately public full text read: Klose and colleagues (2022), a meta-analysis of online activity and grades; Uttl and colleagues (2017), a meta-analysis of teaching evaluations. No results are cited from either, and neither is in the references.
• Not found: the relationship of time on page, scroll depth, or return visits to readers' understanding; the validity of buttons like Was this page helpful?; the relationship between readers' unprompted praise and actual learning.

Appendix: design and numbers of each study

The body keeps only one or two numbers per item; the rest are here.

Deslauriers, McCarty, Miller, Callaghan, Kestin (2019), PNAS 116(39), published version and supplementary materials. Design and main results: see Part 8. Additions relevant to this part: the composite self-rating (FOL) weights four five-point items by principal components: enjoying the class, feeling that one learned a lot, the instructor being effective, and wishing all physics classes were taught this way. Main effect −0.56 (robust standard error 0.13); test +0.46 (0.10). Using only the learned a lot item, the authors say the results were nearly identical, without printing numbers. Supplementary regression 1: those higher on the instructor was effective were 0.51 standard deviations higher on learned a lot (p < 0.001), and with it included, teaching method was not significant. 17 one-on-one interviews: 15 found the active class disjointed and not smooth; all but 1 found the passive lecture more enjoyable. The instructor in the passive condition was asked to make the explanation as fluent as possible. The higher the midterm grade, the lower the FOL (−0.20) and the higher the test score (0.26); the body text does not discuss these coefficients.

Bastani, Bastani, Sungu, Ge, Kabakcı, Mariman (2025), PNAS 122(26), published version and supplementary materials; the correction notice changes only author affiliations. One high school in Turkey, grades 9 to 11, main sample 839 (excluding gifted classes), randomized by class; four 90-minute sessions, each with a closed-book exam right after practice. Practice: unguarded +0.137, guarded +0.361 (control mean 0.284, maximum 1). Exam: unguarded −0.054 (SE 0.022, about 17%), guarded −0.004 (0.013); the preregistered t-test versions are −0.035 and −0.006. Feelings (supplementary Table B.8, ordered probit or OLS): self-rated learning, unguarded −0.078 (0.080), guarded 0.071 (0.055); self-rated exam performance, −0.022 (0.077) and 0.130 (0.056, p < .05); value of practice for solving exam questions, 0.113 and 0.271 (p < .01); minutes of exam time willing to give up for AI help, 2.245 and 3.710 minutes (both p < .01). In the supplementary materials the authors write that students "potentially overvalue its benefits".

Lehmann, Cornelius, Sting (2024; preprint v2 dated 2025-03-08), arXiv:2409.09047. Study 2: 107 people in a lab at a German university, with a GPT-3.5 chat window beside them during learning; afterward the authors discovered that the lab computers blocked copy and paste. Study 3: the same experiment with copy and paste allowed, 69 people. Both preregistered. Posttest regression coefficients 0.396 (p = 0.580) and 0.959 (p = 0.298); Study 3 learning phase +2.614 (p = 0.013). Exploratory analysis (Table 13, both experiments pooled, 176 people): controlling for posttest minus pretest, self-rating with an LLM +0.338 (SE 0.166, p = 0.044); also controlling for progress in the learning phase, +0.348 (p = 0.043); each additional question of posttest minus pretest, self-rating +0.154 (p < .01); people who had used LLMs before, self-rating −0.672 (p < .01).

Fischer, Rau, Rilke (2025), IZA Discussion Paper 18338, a discussion paper not reviewed by a journal, preregistered. 334 people in a lab at Technische Universität Berlin; read an excerpt from Varian's textbook for 25 minutes, then immediately took a 25-question multiple-choice test. AI tutor available throughout +0.337 standard deviations (p < .01); access opened after 10 minutes of reading +0.132 (not significant); difference between the two groups 0.21 (p = 0.066). Post-test estimate of questions answered correctly: throughout +0.195 (p < .1), delayed +0.217 (p < .05); enjoying the preparation: delayed +0.308 (p < .05), throughout +0.199 (not significant). The authors set the estimates and actual results side by side, without a test.

Kestin, Miller, Klales, Milbourne, Ponti (2025), Scientific Reports 15, published version and supplementary materials. Harvard PS2, 194 people included, assigned in a crossover in groups of 2 to 3, one class in each of two weeks; the AI tutor was used at home and included prerecorded videos and worked solutions written for each question. Posttest medians 4.5 versus 3.5; regression 0.63 standard deviations, and the authors say quantile regressions gave 0.73 to 1.3, without a table. Feelings, four five-point items: engagement 4.1 versus 3.6 (t(311) = −4.5), motivation 3.4 versus 3.1 (t(311) = −3.4); no difference detected in enjoyment or growth mindset (no numbers in the body). There is also a statement that 83% of students said the AI's explanations were as good as or better than a human teacher's, without saying when it was asked or how many answered. The paper does not mention preregistration; posttest counts of 142 and 174 don't match the 194 included.

Nagahama, Morita (2017), Japan Journal of Educational Technology 40(4), published version. 75 people at a private university in Tokyo, divided into three groups by pretest and memory test. Video 9 minutes 12 seconds (at 1x), slides with audio; note-taking, pausing, fast-forwarding, and rewinding not allowed. Posttest total (out of 20) 13.00, 12.20, 11.92, main effect of speed F(2,72) = 0.92, power 0.38; recall (out of 11) 6.52, 6.08, 5.32, power 0.65; application (out of 9) 6.48, 6.12, 6.60, power 0.17. Subjective ratings on 25 five-point items, rating short clips at the three speeds watched separately after the test (within subjects): 19 items significant at the 1% level; I understood the learning content F = 60.27, with 1x and 1.5x higher than 2x. On five single-choice questions, significantly more people chose 1.5x as easiest to understand, best for concentration, and most wanted for future use (percentages only in a figure). Limitations the authors list: the viewing procedure doesn't match normal conditions; no reading speed or eye-tracking data.

Oura, Ikejiri, Fushikida, Anzai, Yamauchi (2018), Japan Journal of Educational Technology 41(4), published version. gacco's 日本中世の自由と平等 (my translation: Freedom and Equality in Medieval Japan), April 2014, four weeks, 34 videos totaling 5.81 hours; 20,025 registrants. Completion model, 2,900 people (2,847 online, plus 53 in a flipped group with two face-to-face sessions): watching at least 6 hours, odds ratio 8.35 (95% CI 6.74–10.35); each additional day active on the discussion board 1.41; flipped learning 1.63 (p = .569); the model failed the Hosmer–Lemeshow test (p < .001), AUC 0.85. Thinking-ability model (1,224 completers, 1,189 online, 35 in the flipped group): viewing time (square root) 1.15 (p = .161); discussion board 1.01 (p = .348); prior thinking ability 5.16; flipped learning 5.08 (95% CI 1.98–13.05), where the flipped group were high school students who signed up and paid themselves or were selected; motivation toward history 1.58. Thinking-ability coding by two experts, weighted κ = 0.86. Limitations the authors list: only completers were evaluated; a single course.

Lai (2023), Ministry of Education Teaching Practice Research Program outcome report (PEE1100902), public on the website of the teaching excellence center of National Dong Hwa University; the report cover is marked delayed release. An online general-education Python course for non-computing majors; in the experimental group, multiple-choice questions were embedded in the videos, and the control group was the previous year's class with questions outside the videos; enrollment 109 and 114, complete data 49 and 41. Self-efficacy 3.38 versus 2.98 (t = 1.89), motivation 3.42 versus 3.04 (t = 1.75), satisfaction 3.36 versus 3.21 (t = 0.59), none significant; viewing time 11,400 versus 8,898 (unit not given, t = 2.03, p < .05); achievement test of 15 questions, pretest as covariate, 10.42 versus 10.31 (F = 0.40). Completion rate and replay and rewind counts are listed among the study's goals, but no results are reported for them. The report has no limitations section.

Chen, Chudzicki, Palumbo, Alexandron, Choi, Zhou, Pritchard (2016), RPTEL 11, published version (PMC). edX 8.MReVx mechanics review course, about 11,000 registrants; seven experiments ran in this offering, and the paper reports two. Filming-style experiment (all figures below are new style versus old style): for each version of each video, only people who watched the video before their last attempt were analyzed (36 to 113 people, under 30% of those who attempted). Share with a long viewing: video 3, 45% versus 27% (p = 0.02); the rest not significant. Average viewing time (as a share of video length): video 1, 75% versus 96% (p = 0.02, old style longer); video 2, 88% versus 108% (p = 0.05); video 3, 101% versus 79% (p = 0.006). Final correct rate: video 2, 93% versus 83% (p = 0.04); video 4, 84% versus 98% (p = 0.02, old style higher). The authors' overall assessment: "did not result in eye opening discovery or solid conclusions".

Yu, Wu, Liu (2019), Education Sciences 9(2), published version (publisher PDF indexed in ERIC). One OpenEdu course, 6 weeks, 55 videos; enrollment across three runs 590, 346, 427, with 264, 137, and 57 earning certificates; another course had 977 people, 86 of whom earned certificates. 14 features: units logged into, videos watched, connection sessions, counts of seven video actions, quizzes available to answer, answers submitted, attempts, total unit-test score. ANN on the first two runs (80/20 split) averaged 0.951; predicting the third run averaged 0.978; the other course 0.943. No baseline, precision, recall, or confusion matrix, and no model using only video features. The paper has no limitations section.

Koedinger, Kim, Jia, McLaughlin, Bier (2015), L@S 2015, published version. For the numbers, see the appendices of Part 9 and Part 8. Relevant to this part: standardized coefficient for doing practice on total weekly quiz score 0.44, about 0.065 each for watching videos and reading pages; quiz questions came from the interactive materials' question bank, while the final exam was written by the instructor; share with excellent final exam performance, 16% for only doing more practice, 15% for only watching more video, 31% for doing more of both. Observational data; the authors argue that causal modeling supports a causal effect of doing problems, while acknowledging there was no random assignment and attrition was high.

Salvador (2024), Duolingo Blog, published 2024-06-13, modified 2024-12-02. Three metrics: total completed sessions → learning time (later changed to the share of people learning at least 15 minutes a day) → Time Spent Learning Well = minutes learning on the path + minutes learning in other lessons × 0.5. The figure legend for the XP reallocation experiment says about 1.1 million and 1.8 million more minutes per day, with no year, sample size, or description of a control group on the figure, and what it measures is the metric itself. Shortening lessons made the metric drop; no numbers given.

Cloudflare Web Analytics documentation. High-level metrics page: Visits are page views that came from a different website or were opened directly (the documentation says "A page view that originated from a different website or direct link", determined by the referrer differing from the hostname), and one visit can include several page views; Page views are successful responses with an HTML content type; there are also page load time and Core Web Vitals. FAQ: custom events are not yet supported.

References

I read each item from start to finish and kept an extraction record; parts I couldn't read are noted in parentheses.

• Deslauriers, McCarty, Miller, Callaghan & Kestin (2019), Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom (published version; PMC full text and supplementary materials).
• Bastani, Bastani, Sungu, Ge, Kabakcı & Mariman (2025), Generative AI without guardrails can harm learning: Evidence from high school mathematics (published version, PMC full text and supplementary materials; correction notice; preregistration documents and data not read).
• Lehmann, Cornelius & Sting, AI Meets the Classroom: When Do Large Language Models Harm Learning? (arXiv preprint v2, 2025-03-08; for the interface screenshots in the appendix only the captions were read; preregistration documents not read).
• Fischer, Rau & Rilke (2025), AI Tutoring Enhances Student Learning Without Crowding Out Reading Effort (IZA Discussion Paper No. 18338, discussion paper, copy deposited with the University of Bonn library; preregistration documents not read).
• Kestin, Miller, Klales, Milbourne & Ponti (2025), AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting (Scientific Reports open-access published version and supplementary materials; data on GitHub not read).
• 長濱澄、森田裕介 (2017)〈映像コンテンツの高速提示による学習効果の分析〉 (in Japanese; Japan Journal of Educational Technology 40(4), J-STAGE published version; bar heights and percentages in figures not estimated).
• 大浦弘樹、池尻良平、伏木田稚子、安斎勇樹、山内祐平 (2018)〈歴史をテーマにした MOOC における反転学習モデルの評価〉 (in Japanese; Japan Journal of Educational Technology 41(4), J-STAGE published version; for figures, only captions and numbers cited in the text were read).
• 賴志宏 (2023)〈線上教學影片穿插即時測驗對學習投入與學習成效影響之研究：以非資訊相關科系之程式設計課程為例〉 (in Chinese; Ministry of Education Teaching Practice Research Program outcome report, public on the website of the teaching excellence center of National Dong Hwa University, not peer reviewed; for system screenshots only captions were read).
• Chen, Chudzicki, Palumbo, Alexandron, Choi, Zhou & Pritchard (2016), Researching for better instructional methods using AB experiments in MOOCs: results and challenges (published version, PMC full text; the figures themselves are not in the full text obtained, so only captions were read).
• Yu, Wu & Liu (2019), Predicting Learning Outcomes with MOOC Clickstreams (published version, publisher PDF indexed in ERIC).
• Koedinger, Kim, Jia, McLaughlin & Bier (2015), Learning is Not a Spectator Sport: Doing is Better than Watching for Learning from a MOOC (ACM open-access published version, also on the authors' lab publications page; numbers in figures could not be read, so only the body text is used).
• Salvador (2024), Behind the metric: how we developed "Time Spent Learning Well" (official Duolingo blog; the three external links in the post were not read, including the white paper it cites).
• Cloudflare Web Analytics documentation: High-level metrics and FAQ (checked 2026-10-15), used to describe what this site's aggregate traffic statistics measure and don't measure.