Do Readers Know Whether They Understood? Judgment Accuracy, the Illusion of Explanatory Depth, Practice Questions, and Delayed Summaries

🌏 中文版

This is part 15 of the series "Writing Articles That Teach," in its third section, "what readers bring with them." The earlier posts in this section covered prior knowledge, things readers already believe that are wrong, and usefulness and interest. This one covers something else readers bring: their own judgment of how much they understood.

The trigger was this site's course-guide spec. It requires a question at the end of every major section, with an answer or a criterion, and says a self-check must be more than "think about it." A cross-section question goes at the end of the article. The rule rests on an unstated assumption: after a section, readers roughly know whether they understood it, and the question only helps them confirm. Part 9 reviewed what questions and feedback do for learning. Part 8 described a classroom experiment in which students gave lower self-ratings in the class where they actually learned more, and left "how accurate are readers' judgments" for a later post. This is that post: on average, how accurately do readers judge whether they understood, and what can an article do to make that judgment more accurate?

This has direct consequences for writers. A reader who thinks they understood moves on and does not go back. If their judgment does not track their actual understanding, then lines like "after this section you should be able to…", plus a check question whose answer is visible without answering, may all reinforce a wrong judgment. This site's post on Learning How to Learn covered how "fluency lies" from the learner's side; this post takes the writer's side. What the signals visible to authors (readers saying it was well written, traffic) actually measure is the topic of part 25; this post covers only the reader's side.

How the sources were chosen, and their bias. The source list came from a scan of the literature followed by a selection for close reading. The rule was that every conclusion that uses a number needs a source I could read from start to finish, and at least one source had to show an opposite or null result. What I read: two meta-analyses of metacomprehension accuracy (studies that pool many experiments into one estimate), one classic paper on the "illusion of explanatory depth," three experiments on how learners judge which study method works, one experiment on misconceptions and judgment accuracy, and one on headings and judgment accuracy. Also one meta-analysis of prequestions, one review, one log analysis of in-video quizzes, and the two recommendations on judgments of learning in a practice guide from the US Institute of Education Sciences (IES). The bias is obvious: English-language literature, college students, short materials in the lab. Every item in the references was read from start to finish and has an extraction record. Studies I know exist but could not read in a legal public full text are flagged as such in the text, and I do not cite their results.

The conclusions first

"Strong" here means there are meta-analyses and the direction is consistent. It does not mean the number will show up as is in a long web article; the reasons are in the limitations section.

"Judging accurately" means two different things

A reader's judgment of their own understanding is called metacomprehension in the research, part of metacognition. There are two ways to compute judgment accuracy, and mixing them up gives opposite impressions.

• Relative accuracy: the same reader reads several texts; do their ratings of "how much I understood" for each text rank in the same order as their test scores on each text? It measures whether they can tell which text they understood and which they did not. It is usually a Gamma or Pearson correlation: 1 means a perfect match, 0 means no relationship.
• Absolute accuracy (bias): the judgment minus the actual score. Positive means overestimation, negative means underestimation.

The two can move independently. A person can overestimate every text by two points and still rank them perfectly; or be unbiased on average with a random ranking. The two meta-analyses below compute relative accuracy only. The evidence on over- and underestimation comes from other experiments.

Both matter to a writer. Relative accuracy decides whether the reader picks the right section to reread; absolute accuracy decides whether they feel they "got all of it" and never go back.

Can readers tell which text they understood?

Chunliang Yang and colleagues' 2023 meta-analysis (I read the author accepted manuscript in UCL's repository) included 115 studies. The typical experiment: participants read several expository texts, rate how well they understood each one after each text or after all of them, then take a test on all texts. Each person's judgments are correlated with their scores, and the correlations are averaged across participants.

With no intervention, the average correlation is 0.178. The authors convert this with their own simulation to about 54.5% judgment accuracy, against a chance level of 50%; the simulation method is in the supplement, which I could not read. The introduction gives an example computed for the 0.27 often cited in earlier literature (not for 0.178): someone who has read 40 textbook sections and wants to reread the 20 they understood least would, on average, pick only 11 that they actually did not understand. 0.178 is lower than 0.27.

Anja Prinz and colleagues' 2020 meta-analysis (published version) is narrower. It includes only studies of students reading expository texts, with tests of inference or application, and a control condition with no intervention at all. In these studies the control conditions average .14. The two numbers are close.

Three conditions go with these numbers:

• Plain text only. Both excluded texts with illustrations or animations; Yang and colleagues say this was to focus on plain text.
• Mostly in the lab, with materials that were not the participants' own coursework. Yang and colleagues list this as a limitation.
• Correcting for publication bias makes the number higher, not lower. Yang and colleagues ran six publication-bias checks; three gave adjusted values, all higher than 0.178. They also estimated a "true" correlation of about 0.31 after correcting for measurement error, and called even that "still quite poor."

Yang and colleagues also found that study conditions with higher average scores had more accurate judgments. This is a relationship across study conditions, using average test scores as a proxy for comprehension; it cannot be read as "the more a given reader understands, the more accurate their judgment."

Do readers overestimate themselves?

The illusion of explanatory depth

Leonid Rozenblit and Frank Keil's 2002 paper (the author manuscript on PMC) reports twelve studies, mostly with Yale students. The core procedure has three steps. First, participants rate on a 7-point scale how deeply they understand things like zippers, flush toilets, and cylinder locks. Next, for a few of these items they write a detailed, step-by-step causal explanation, marking "GAP" wherever they cannot continue, and rate again. Finally, they answer a diagnostic question that requires the key mechanism (for example, explain step by step how to pick a cylinder lock), and rate again.

After writing the explanation, self-ratings dropped clearly. In Study 2, the one with reported numbers, the average fell further after the diagnostic question, but that step was not tested separately, and in Study 1 it was not significant. In Study 5, 12 students from the same university rated these explanations; their ratings were lower than the writers' first self-ratings. The self-ratings right after writing the explanation were still significantly higher than the outside ratings; only after the diagnostic question was there no significant difference. The numbers are in the appendix.

The illusion is not general overestimation. With other kinds of knowledge:

• Natural phenomena (earthquakes, tides, rainbows) behaved like devices, with a clear drop.
• Facts such as national capitals showed a small but significant drop.
• Everyday procedures (tying a tie, driving from New Haven to New York) showed no drop; if anything, ratings went slightly up.
• The drop for movie plots was not significant.

The authors therefore argue that the illusion is concentrated in explanatory knowledge, not that people overestimate everything. In Study 6, participants were warned in advance that they would write explanations and answer diagnostic questions. The drop got smaller, but because the ratings after writing were higher; the initial self-ratings did not get lower. The authors say themselves that this pattern was unexpected.

There is an important distance when applying this to writing, which the authors state in their introduction:

the current studies are concerned with people’s ability to assess the knowledge they have before coming into the lab, rather than things learned in the course of an experiment. The implications of our research are different: they tell us less about how people learn when reading…

In other words, these studies measure knowledge participants already had, not understanding just after reading an article; and the paper has no objective test at all, so "how much they actually understood" rests on the self-rating drop and on outside ratings. I treat it as indirect evidence: for mechanism-type content, asking readers to write out a step-by-step explanation lets them see the gaps.

Reading with misconceptions

Prinz, Golke, and Wittwer's 2021 experiment had 92 German college students who had taken statistics read a text on covariance. The text targeted four common misconceptions (for example, "covariance indicates causation"). After reading and before answering, participants predicted how many questions they would get right.

All three groups received the misconceptions, in different forms. The refutation-text group read a version that states each misconception and refutes it. The think-sheet group read the standard text while filling in a three-column sheet: the first two columns listed questions and the matching misconceptions, and participants wrote the correct explanation in the third. The control group read the standard text plus a sheet listing only the questions and misconceptions.

In the control group, participants with more prior misconceptions overestimated their comprehension more. In the think-sheet group this relationship disappeared, and the difference from the control group was significant. In the refutation-text group the relationship was also near zero, but its difference from the control group did not reach .05. Pooling all three groups, the number of misconceptions was not significantly correlated with overestimation, and at the average number of misconceptions the three groups' prediction bias did not differ. The authors themselves say several results were only marginally significant and power was below .80; separately, each dependent measure had only 4 items, which I noticed in the design and which the authors do not list as a limitation.

So what this experiment can say is: under one way of presenting misconceptions, readers with more misconceptions overestimated more; when readers had to write the correct explanation against each misconception themselves, that relationship disappeared in this experiment. It has no group without any mention of misconceptions, so it cannot answer whether adding a "common misconceptions" passage beats leaving it out. The effect of refuting misconceptions itself is covered in part 13.

Headings

Part 1 covered Anja Prinz-Weiß and Aline König's 2023 experiment. 51 college students read five science texts of about 500 words; half got a version with a question heading before each paragraph. After each text and before seeing the questions, participants predicted how many inference questions they would get right. The plain-text group significantly overestimated their comprehension, while the headings group's predictions were close to their actual scores; the difference in bias between groups was d = 0.68.

Part 1 mentions in a single line of its appendix that the other kind of accuracy (relative accuracy) did not differ significantly between groups; the authors wrote "In contrast to the metacomprehension hypothesis." Here is one thing part 1 did not cover: the predictions themselves did not differ significantly between groups, and if anything the headings group predicted higher; the difference was in comprehension scores, which were clearly higher in the headings group. Reading the numbers, I think bias shrank mainly because the headings group actually understood more, not because their judgments became more cautious. This is my reading; the paper does not break it down this way. One sentence in the authors' limitations fits this reading: "it is unclear to what extent learners who read texts with headings were indeed aware of their improved comprehension when making their metacomprehension judgments."

How readers judge which study method works

Besides judging whether they understood, readers also judge whether a way of reading or teaching works. This matters for writing because a reader's sense of whether an article taught them something is also a judgment.

Part 8 covered Louis Deslauriers and colleagues' 2019 classroom crossover experiment in detail: the active-learning class scored higher on the test and lower on self-ratings. Here I add only one point relevant to this post. The authors added an interaction of pretest score × perceived learning to the test-score model; it was positive, p < .05: the more prior knowledge students had, the better their perceived learning matched their actual scores. The paper does not report the coefficient.

Two other sets of experiments used different materials to ask the same question.

Veronica Yan and colleagues' seven experiments from 2016 had participants learn to recognize the styles of 12 painters from landscape paintings. Half of the painters' works appeared consecutively (blocked) and half were mixed with other painters' works (interleaved); participants were then tested on new paintings. Interleaved painters were classified better in every experiment (only marginally in 1B), yet of the 149 participants in 1A and 1B combined, 72% said blocking was more effective; among those who actually did better with interleaving, 75% said so too. In neither experiment was a relationship detected between the judgment and which schedule actually worked better for the participant.

Three other results matter most for writing:

• Seeing your own scores does not mean you believe them. Experiment 2 told participants explicitly how many items they got right in each condition. Of the 45 who did better with interleaving, only 36% attributed the difference to interleaving being more effective, and 44% said those painters were easier; of the 20 who did better with blocking, 60% attributed it to blocking being more effective. The same kind of evidence was read asymmetrically.
• Telling them directly helps only partly. After the test, Experiments 3 to 5 told participants that previous research had found 90% of people learn better with interleaving, and some groups were also told why. Pooling the three experiments, 12% chose interleaving with no information, and 35% to 44% in the information conditions; none reached a majority. In every information condition, no relationship was detected between judgments and which schedule actually worked better for the participant. The 90% figure was explanatory text shown to participants, not a finding of this paper.
• Separating the two experiences is what makes judgments match scores. Experiment 6 split blocked and interleaved study into two independent study-test cycles. With no information, 46% already chose interleaving; with the full explanation, 75%; and among those who experienced interleaving first and blocking second, 93%. Judgments also began to correlate with participants' own scores. Experiment 6 changed several things at once, and the comparison with the earlier experiments is across experiments.

Jessica Macaluso and colleagues' two experiments from 2022 used bird photos for category learning, with University of Pittsburgh undergraduates taking psychology courses. Interleaving was rated more effortful and less familiar, and also as producing less learning; before being told which was better, 76.6% chose blocking as their future method. The second experiment had a posttest: bird families learned interleaved were classified correctly 60% of the time, blocked 55%, an effect the authors call relatively small; participants who chose blocking also did better with interleaving.

The authors' interpretation: participants did not choose blocking to save effort; they misread effort as learning less. This is a mediation analysis, the relationships among variables are correlational, and the authors say themselves that causation cannot be established. After being told that prior studies found 90% of people learn better when examples are not grouped, half of participants did not answer whether they would switch; among those who had chosen blocking, about six in ten said they would not. The four answer categories add up to about 100%, so the denominator appears to be those who answered, but the paper does not say so.

Both sets of experiments involve inducing categories from images viewed for a few seconds each, not reading an article, and not judging whether one understood a passage. I treat them as an analogy: a reader's sense of which way of reading works can diverge from their actual scores, and just telling them changes it only a little.

What makes judgments more accurate

Both meta-analyses compared many interventions. In the table below, the left column is from Yang and colleagues: the average correlation for each type of intervention minus the average correlation of all no-intervention conditions, a cross-study comparison. The right column is from Prinz and colleagues: the effect size g of the intervention group against the control group within each study. Numbers in parentheses are the number of effects.

Points useful for writing:

• Both "delay" and "generating it yourself" are needed. Listing keywords after reading all texts was clearly better than listing them right after each text; listing your own keywords was clearly better than reading keywords supplied by the experimenter. Prinz and colleagues cite a 2005 study by Thiede and colleagues: delay without generation (thinking about the text, reading supplied keywords), or delaying only the judgment, did not affect accuracy. The "delay" here means a few minutes after reading all texts, not the next day; studies with judgments a day after reading were excluded by Prinz and colleagues.
• Questions help only if you answer them yourself. The condition with practice questions answered before judging improved accuracy; just laying out questions and answers to read showed no detectable benefit, direction negative; the difference between the two was 0.264, significant. The authors' sentence is "covertly answering practice questions, but not passively reading questions and answers, boosts metacomprehension accuracy." Covertly literally means silently, in one's head; the contrasting description in the same passage is reading questions and answers "without the requirement to actively generate answers." So the analysis compares generating an answer yourself against passively reading answers, and it does not separate whether the included studies required written answers; if anything, the authors' wording is closer to "answering in your head counts." Part 9 also notes that the authors of another prequestion meta-analysis (King-Shepard et al. 2025) cautioned that participants told only to read the questions may have answered them in their heads. These are lab studies, with practice questions taken after reading and before judging, not embedded mid-article.
• Knowing what level the test will be has a precondition. Prinz and colleagues define it as giving both an explanation that the questions will test inference and sample questions on a practice text; its point estimate is the largest of the seven, but all three effects come from the same paper and the same research group, and the overall difference among interventions was not significant. Yang and colleagues define it as being told in advance whether the test is on memory or inference; a match helps a little, a mismatch makes things worse.
• Combinations have the largest effect, still far below 1. Yang and colleagues say "additive" is only descriptive; they did not test whether the effects actually add.

Two caveats that change how to use this:

• More accurate judgment does not mean better learning. Prinz and colleagues also analyzed these interventions' effect on comprehension scores: the three delayed generation tasks combined were near zero, with no detectable effect, and delayed summary writing was in the negative direction; the overall effect was significant but small. Yang and colleagues also write in their limitations that higher accuracy does not guarantee better learning.
• The effective interventions mostly ask readers to work harder. Yang and colleagues write in their limitations that self-directed learners may not want to use them.

What signal can questions in an article give readers?

This site's end-of-section questions are written into the article, not handed out as practice sheets in a lab. This section looks at the studies closest to "putting questions in an article," and how far each can go.

Prequestions: judgment accuracy was not measured. Kyle St. Hilaire and colleagues' 2024 meta-analysis reviews studies where learners guess answers before reading. Its conclusion was covered in part 1: the benefit is concentrated on the content that was asked about. It did not measure any metacognition or self-assessment. "Prequestions let readers see what they do not know" is only a theoretical account in this analysis, and the authors judged its support to be weak. Only one cell is relevant here: in the condition where participants only read the questions and were not asked to answer, the confidence interval crosses zero.

Learners do not think testing first helps. Steven Pan and Shana Carpenter's 2023 review has a section on metacognition. The studies it cites say that many learners, just after benefiting from pretesting, still did not think it helped; in one survey 78% thought reading was more effective, and in another only 56% chose "guess first, then study." The review also cites two studies showing that explaining the benefit or giving score feedback can correct this misjudgment. I did not read any of these original studies.

What readers actually do when a question appears. Geza Kovacs's 2016 log analysis looked at in-video quizzes in a Coursera machine learning course: a question pops up at a certain point in the video, with a skip button, ungraded. In the 92 videos with a single quiz, 74% of people who started the video submitted an answer. Three findings relate to judgment accuracy:

• Six in ten backward seeks from a quiz happened before the first answer: look at the question, go back to find the answer, then answer.
• Most people who answered wrong on the first try answered again within a dozen or so seconds (median 13 seconds); among those who got it right within 30 minutes, only 9.1% went back to the video before getting it right.
• This is descriptive data from a single course, with no statistical tests and no learning outcomes; the motive for going back is inferred from behavior.

Putting this together with the previous section is my inference: when readers can look for the answer in the article while answering, a correct answer tells them "I can find it," not "I remember it and understand it"; after a wrong answer, most just try another option rather than reread. The 2007 practice guide from the US Institute of Education Sciences (IES) says the same in the checklist for its recommendation on teaching students to judge what they have learned: "Remind students to complete judgments of learning without the answers in front of them." That recommendation's level of evidence is low, because the evidence comes from the lab and there are no classroom experiments yet.

Does asking readers to stop and reflect help? Zengilowski and colleagues published a large-scale randomized trial at L@S 2025 on using prompts to encourage metacognitive reflection in a computer-based learning platform. The study exists; I could not read a legal public full text, so I do not cite its results.

Where the sources disagree

The following results point in different directions, or look contradictory but measure different things. I set them side by side without picking a side.

What the no-intervention baseline is. Yang and colleagues found 0.178; the control conditions in Prinz and colleagues average .14. In their own introduction, Prinz and colleagues cite +.24 from another meta-analysis (Prinz et al. 2020, Educational Research Review) as the baseline, without discussing the gap from their own .14; Yang and colleagues argue that .24, and the .27 often cited in earlier literature, mix in conditions with interventions and so overestimate. I could not read a legal public full text of the .24 meta-analysis.

Whether self-explanation helps. Self-explanation is significant in Yang and colleagues' analysis (7 effects). In Prinz and colleagues' analysis it is not significant, with only two experiments, one strongly positive and one negative; the authors call it "unreliable," not ineffective, and speculate that explaining while reading may overload readers. The inclusion criteria differ (Prinz includes only studies testing inference or application, with a control condition with no intervention).

How large the effect of knowing the test format is. In Prinz and colleagues' analysis it has the largest point estimate of the seven; in Yang and colleagues' it is one of the smaller effective interventions. The definitions differ: Prinz requires an explanation plus sample questions on a practice text; Yang only requires being told in advance whether the test is on memory or inference, and treats "told one thing, tested on another" as a separate category.

Does a warning that you will be tested keep you from overestimating at the start? In the two meta-analyses in the previous item, knowing the test format improves relative accuracy. Rozenblit and Keil's Study 6 warned participants in advance that they would write explanations and answer diagnostic questions, and measured absolute self-ratings: the initial ratings did not get lower; the drop got smaller because the later ratings got higher. The two measure different things, so this cannot be called a direct contradiction, but the directions differ.

Can telling learners which method works correct the misjudgment? The two studies cited by Pan and Carpenter say yes. In Yan and colleagues' experiments, the explanation raised agreement with interleaving to three or four in ten, never a majority, and no relationship was detected between judgments and participants' own scores; only with the two experiences separated did it pass seven in ten. In Macaluso and colleagues' open-ended question, about six in ten of those who had chosen blocking said they would not switch (the denominator appears to be those who answered), but half did not answer.

Whether prequestions need to be answered (this item measures learning, not judgment accuracy). In St. Hilaire and colleagues' 2024 meta-analysis, conditions requiring a guess had a clearly larger effect than conditions where participants only read the questions; the latter's confidence interval crosses zero, and the difference is significant. In King-Shepard and colleagues' 2025 meta-analysis, cited in part 1 and part 9, reading without answering and the questions-alone condition had similar effects, with no detectable difference; the authors cautioned that participants may have answered in their heads and did not conclude that answering is unnecessary. The inclusion criteria differ (St. Hilaire excluded prequestions with answers and questions embedded during study), and the two share only 35 records. In a comparison table in their appendix, King-Shepard and colleagues matched the moderators the two analyses share and say several did not agree, possibly because of different included records and coding definitions. For whether the questions were answered, they marked the outcome as matching, but the two analyses' numbers point in different directions; I did not find an explanation for marking it as matching. This post's recommendation agrees in direction with St. Hilaire, but neither analysis measured judgment accuracy.

Did headings make judgments more accurate? In the same experiment by Prinz-Weiß and König, bias (overestimation) improved, while relative accuracy showed no detectable difference. This is not two sources disagreeing but one dataset giving different conclusions under two kinds of accuracy; say which one when citing it.

What you can do

The five items below are this site's judgment. Each states which results it rests on and how far it is from them.

1. Collapse the answers to end-of-section questions by default, and ask readers to answer before expanding. Putting the answer right below the question amounts to having readers "read ready-made questions and answers"; in Yang and colleagues' analysis that showed no detectable benefit, direction negative, while answering before judging helped. How: put the answer in a collapsible block, and next to the question write "Answer it first (on paper is better), then open." "Writing it down" is this site's choice, and goes a step beyond the authors' wording, which is about answering in your head. The IES checklist adds that judgments should be made without the answers in front of you. Distance: in the studies, practice questions came after reading and before judging, not mid-article; neither "collapsing" nor "writing it down" has been tested directly.

   This item differs from what parts 9 and 5 say, and that needs stating up front. Part 9 also first wanted to recommend "write down your answer before reading on," then, after reading its sources, downgraded it to a personal practice without direct evidence, and listed "whether to collapse end-of-section answers" and "whether to require readers to answer first" as questions research has not answered and the spec should leave open for now. Part 5 recommends having readers write out their own answer before moving on. Part 9 asks about learning: the self-explanation meta-analysis it relies on excluded the "answer first, then see the answer" combination, and in the prequestion meta-analysis it cites (King-Shepard et al. 2025), reading without answering and answering had similar effects, so it judged the evidence insufficient. But the other prequestion meta-analysis cited in this post (St. Hilaire et al. 2024) found the opposite, that answering is needed; see "Where the sources disagree." Part 5 relies on two sets of video experiments, which also measured learning (the following is based on part 5's summary; I did not read the originals for this post): Szpunar and colleagues compared testing after every segment with testing only at the end, and their Experiment 2 also had a group that reread questions and answers after each segment; Lawson and Mayer compared writing one's own explanation after each segment, reading a ready-made explanation, or doing nothing. These were college students watching short videos, tested on what they had just watched. This post asks about judgment accuracy: Yang and colleagues directly compared "answering first" with "reading ready-made questions and answers" in the same type of experiment, a difference of 0.264, a row of results part 9 did not use. The three posts can all hold at once: for learning, the sources disagree on whether answering first is needed; for whether readers can see where they do not understand, one meta-analysis supports answering first. As for the two concrete practices, "collapsing" and "writing it down," none of the three posts has direct evidence. Whether to change the spec is in the comparison table in the next section; it is a proposal for the site owner to decide.

2. Add a "close the article and write keywords" step at the end. Keywords and summaries generated by the reader after reading everything were effective in both meta-analyses. How: at the end, write "When you finish, do something else for a few minutes, then come back and, without looking at the article, write one keyword for each section and one sentence on what it is about. The section you cannot write, or that you got wrong when you check back, is the one to reread." The IES guide's procedure is almost the same: after reading, do other homework first, then take a blank sheet and write four keywords and definitions without looking at the text. Distance: the delay in the experiments was a few minutes, and the texts were a few expository passages; this step improves judgment, not comprehension itself.

3. For mechanism-type content, write the check question as "explain step by step," followed by a question that requires the key mechanism. How (an example I wrote myself, not tested): "In your own words, explain why parameters jump farther and farther away when the learning rate is too large, writing the cause and effect for each step; at any step you cannot write, mark a 'gap'." Then ask a question that cannot be answered without the mechanism, such as "If you halve the learning rate, will training that used to diverge necessarily converge? Why?" The basis is Rozenblit and Keil's procedure: self-ratings dropped after writing the explanation, and matched outside ratings only after the diagnostic question. Distance: they measured knowledge participants already had, not content just read; and no illusion was detected for procedural knowledge (things done by following steps), so for hands-on content a check of "do it and compare with the expected result" fits better; that is my inference.

4. At the start of the article, say what level the end-of-section questions test, and make the first question an example of that level. How: at the start, write "Each section ends with a question asking 'what happens if this condition changes,' not asking you to recite a definition," and then make the first section's question actually that kind. The basis is the knowing-the-test-format conditions in both meta-analyses; in Yang and colleagues' analysis, judgments got worse when the announced and actual test formats did not match, so if you say it, do it. Distance: the largest effect size comes from three experiments by a single research group.

5. Label each question with the section it tests, and link to it. A reader who gets it wrong can go straight back to that section, instead of only knowing "I got one wrong." There are two bases. After Yan and colleagues' Experiment 6 split the two experiences into two cycles with separate scores, judgments began to correlate with scores; Experiment 2 told participants explicitly how many items they got right in each condition, and their choice of "which would you use if you were a teacher" was also related to their actual performance, only asymmetrically (95% of those who did better with blocking chose blocking; only 51% of those who did better with interleaving chose interleaving). And the IES recommendation on using quizzes to find content not yet learned says to state which pages to reread after a wrong answer. Mapping "being able to attribute a result to its cause" to "linking a question back to its section" is my inference.

Checking this site's spec item by item

The parts of the course-guide spec (course-guide-contract.md) relevant to this post are compared below. The "evidence" column lists only what this post read; the original basis for per-section self-checks in the spec is experiments with quizzes inserted between video segments or with learners writing their own explanations, summarized in the video post, and those measured learning, not judgment accuracy.

One thing the spec does not mention but this post's evidence points to: the distance between question and answer. The spec currently requires only "an answer or criterion," and says nothing about what readers should do before seeing the answer. This post proposes adding it from the angle of judgment accuracy; part 9 recommends leaving it open from the angle of learning. The two angles rest on different evidence; whether to change the spec is the site owner's call, and this is not presented as a settled spec change.

Limitations of this evidence

• No study has directly tested readers of long web articles. The two meta-analyses used plain expository text, mostly in the lab; Yang and colleagues state that the materials were not participants' own coursework. Carrying these results over to a five-thousand-word guide is my inference.
• The relative-accuracy studies do not answer whether readers change their behavior as a result. Yang and colleagues write in their limitations that higher accuracy does not guarantee better learning, and that effective interventions take more effort, which self-directed learners may not put in.
• More accurate judgment and better learning are two different things. In Prinz and colleagues' analysis, the delayed generation tasks had an effect near zero on comprehension scores, with no detectable effect.
• Most of the overestimation evidence is not about reading articles. Rozenblit and Keil measured existing knowledge, with no objective test; Yan and colleagues and Macaluso and colleagues studied inducing categories from images.
• Small samples and marginal results. Prinz et al. 2021 had about 30 per group and 4 items per dependent measure; Prinz-Weiß and König had 51 participants; the test-expectancy result in Prinz et al. 2020 rests on three effects from a single paper.
• Participants were mostly college students taking part for credit or pay; six of Yan and colleagues' seven experiments used MTurk crowdworkers.
• Cross-study comparisons. Yang and colleagues' effect for each intervention is against "all no-intervention conditions," not a within-study control; the comparison between Yan and colleagues' Experiment 6 and Experiments 3 to 5 is across experiments.
• Supplements not read. Yang and colleagues' supplement (including the method of the accuracy simulation) is not in the accepted manuscript, nor in the OSF project; Macaluso and colleagues' mediation path coefficients are only in figures, and the file I read had no figures.
• My inferences, eight in all: carrying the results over to long web articles; that the bias improvement in Prinz-Weiß and König comes mainly from higher comprehension; using Rozenblit and Keil as indirect evidence that writing explanations reveals gaps; using Yan and colleagues and Macaluso and colleagues as an analogy for how readers judge an article's effectiveness; that a correct answer while able to look things up means "I can find it"; comparing results for hands-on check questions; that linking questions back to sections helps attribution; and the five items under what you can do. Each is marked in the text.
• Known to exist, but no legal public full text read: Zengilowski et al. (2025) on metacognitive prompts, Prinz et al. (2020, Educational Research Review) on baseline accuracy, Dunlosky and Lipko (2007), a short review. None of their results are cited, and none are in the references.
• Secondhand citations: Thiede et al. (2005), as cited by Prinz et al. 2020; the metacognition studies cited by Pan and Carpenter (Huelser and Metcalfe 2012 and others); and the video experiments by Szpunar et al. (2013) and by Lawson and Mayer (2021), as summarized in part 5. I did not read any of these originals for this post.
• Not found: the effect on judgment accuracy of check questions that appear the next day or week; metacomprehension accuracy for code or math content; a direct test of whether check questions an author attaches to an article improve judgment accuracy.

Appendix: design and numbers for each study

The main text keeps one or two numbers per point; the rest are here.

Yang, Zhao, Yuan, Luo, Shanks (2023), Review of Educational Research 93(2); I read the author accepted manuscript.

• Scope: searched 30 databases; 115 studies, 502 effects, 15,889 participants, published 1985–2020; dissertations contributed 166 effects. Inclusion: text as the main material, excluding texts with decorative or supporting pictures; prospective, global judgments; English and Chinese studies. One place in the manuscript's introduction says 508 effects; the results and tables say 502.
• Method: multilevel random-effects model, relative accuracy only (mean within-person correlation, mostly Gamma).
• Accuracy: all effects MC = 0.242; no intervention 0.178 (95% CI 0.155–0.200, 243 effects), converted by the authors' simulation to 54.5%; with intervention 0.332 (259 effects). Heterogeneity Q(501) = 3279.
• Publication bias: six checks run on the no-intervention effects only, four showing no signal; the three giving adjusted values were PET-PEESE 0.201, Trim-and-Fill 0.229, 3PSM 0.196. A rough correction for measurement error using reliabilities from Kelemen et al. 2000 gives 0.312 (about 58%). Bias from missing outcome data was related to MC (Q(2) = 17.57).
• Comprehension level: the meta-regression slope of test scores on accuracy ranged from 0.298 (all) to 0.500 (no intervention, multiple choice, immediate test, young adults only), a relationship at the level of effects.
• Difference from no intervention for each intervention (Table 2): combined 0.316; delayed summary 0.210; concept maps and similar 0.196; delayed keywords 0.184; self-explanation 0.179; practice tests 0.169; letter deletion 0.112; consistent test expectancy 0.090; short-delay rereading 0.063; question generation 0.042 (p = .406); immediate summary 0.030; other 0.023; analogies 0.010; immediate keywords 0.009; reading questions and answers −0.095 (p = .065); reading experimenter-provided keywords −0.096 (p = .050); inconsistent test expectancy −0.117 (p = .019).
• Direct contrasts between interventions: delayed vs immediate keywords 0.175; delayed keywords vs reading supplied keywords 0.280; delayed vs immediate summary 0.179; practice tests vs reading questions and answers 0.264; combined vs single 0.198.

Prinz, Golke, Wittwer (2020), Educational Psychology Review 32(4), published version. Built on the search from the authors' other meta-analysis, adding this paper's criteria (students, expository text, tests of inference or application, a control condition with no intervention, at least two studies per intervention): 17 records, 28 effect sizes, 2,236 participants, 2000–2020. Excluded texts with illustrations or animations, second-language learners, and studies with predictions a day after reading. Control conditions r = .14 (95% CI .09–.20), intervention conditions .41, overall g = 0.46 (0.33–0.60, k = 26), I² = 65%; trim-and-fill adjusted 0.41. By intervention: delayed summary 0.49, delayed keywords 0.34, delayed diagram completion 0.72 (k = 2), self-explanation 0.34 ([−0.20, 0.89], I² = 91%), concept mapping 0.47 (p = .097), rereading 0.32, test expectancy 0.84 (all three effects from Griffin, Wiley, Thiede 2019b). The moderator analysis across interventions was not significant (Q(6) = 5.71, p = .456), and no pair was significant after Bonferroni correction. On comprehension scores: overall g = 0.13 (p = .046); delayed summary −0.17 (not significant); retrieval-based (the three delayed tasks) −0.03; concept mapping 0.67; test expectancy 0.36 (p = .054). The overall effect did not differ significantly between school and college students.

Rozenblit and Keil (2002), Cognitive Science 26(5), PMC author manuscript. Twelve studies of 12–54 participants each, all Yale students except Study 3 and part of Study 9. Study 2 (33 undergraduates, 7-point scale): 3.89 before explaining, 3.10 after writing, 2.49 after the diagnostic question, and 2.62 when re-rating "how much I knew before reading" after reading the expert explanation. In Study 5, 12 raters from the same university rated Study 2's explanations at 2.72 and 2.44 (once before and once after reading the expert explanation), significantly lower than the self-ratings before explaining and right after writing, and not significantly different from the self-ratings after the diagnostic question. Drop from before to after explaining by domain: devices .918, natural phenomena .860, narratives .351 (not significant), facts .291 (significant), procedures −.173 (not significant). Study 3's regional-university students started nearly a point higher and dropped more. In Study 6, after the warning, the drop was still significant but smaller (interaction p = .007). Study 12: the ratio of visible to hidden parts of a device was the main predictor of overconfidence; the predictors were highly collinear, and this is a correlational analysis. The author manuscript has several degrees-of-freedom and typesetting errors; in Studies 3, 4, and 9 the degrees of freedom do not match the sample sizes.

Prinz, Golke, Wittwer (2021), Instructional Science 49, published version. 92 German college students who had taken statistics: refutation text 31, think sheet 29, control 32; texts of 536 (standard) and 665 (refutation) words, 12 minutes total study time in every group. Number of misconceptions was measured, not manipulated. Bias in comprehension predictions (positive is overestimation): control-group slope b = 0.55 (p = .012); refutation group < −0.01 (p = .991); think-sheet group −0.13 (p = .557); slope difference refutation vs control p = .078, think sheet vs control p = .031; overall interaction p = .069. Pooled correlation between misconceptions and bias r = .11 (not significant). On transfer predictions all three groups underestimated on average, and in the think-sheet group more misconceptions went with more underestimation (p = .052). Response-confidence accuracy (Gamma) was .72 in the think-sheet group, higher than the other two groups (post hoc contrast p = .039), with no moderation by misconceptions. At the delayed test 6 to 8 days later, none of the regressions were significant, and misconceptions in the refutation group rose significantly. The authors acknowledge that some results were only marginally significant and power was below .80.

Prinz-Weiß and König (2023), Applied Cognitive Psychology 37(4), institutional repository version. 51 education-science undergraduates (headings group 25, plain-text group 26), taking part online from home during the pandemic; five science texts of 482–505 words, 10 minutes per text, question headings in bold on their own line; very low prior knowledge. After each text and before seeing the questions, participants predicted how many detail and inference questions they would get right (0–3). Comprehension (inference questions): headings 1.54, plain text 0.82 (d = 1.00); the memory difference was not significant (d = 0.52, p = .069). Comprehension predictions 1.44 vs 1.24, not significant. Comprehension bias: plain text +0.42 (significant overestimation), headings −0.10 (not significant), between groups d = 0.68. Relative comprehension accuracy: headings .60 (significantly above zero), plain text .33 (not significant), difference between groups not significant (p = .248). On memory, the headings group significantly underestimated itself (−0.33).

Deslauriers, McCarty, Miller, Callaghan, Kestin (2019), PNAS 116, published version and supplement. Design and main results are in part 8. The addition relevant here: adding the interaction of pretest score (FCI) and "I feel like I learned a great deal" to the test-score model gave a positive interaction, p < .05, with no coefficient reported; the authors interpret it as students with more prior knowledge showing a stronger correlation between perceived learning and actual performance.

Yan, Bjork, Bjork (2016), Journal of Experimental Psychology: General 145(7), published version.

• Design: seven experiments; 1A used 109 UCLA undergraduates, the rest MTurk workers (analyzed n from 43 to 155). 72 paintings, 12 painters, 3 seconds each.
• Actual performance: classification accuracy interleaved vs blocked ranged from .63 vs .42 (1A) to .74 vs .64 (Experiment 6); 1B was only marginally significant.
• 1A and 1B combined: 72% said blocking was more effective, 15% equal, 13% interleaving; no relationship was detected between judgment and which schedule actually worked better (p = .12 and .09 in the two experiments). In 1B, post-study judgments of individual painters were actually higher for interleaving (68.77 vs 60.00).
• Experiment 2 (75 participants): 45 did better with interleaving, 20 with blocking, 10 equal. "If you were a teacher": only 51% of those better with interleaving chose interleaving, and 95% of those better with blocking chose blocking; this choice was related to actual performance (χ²(2) = 16.08, p < .001).
• Experiments 3 to 5 combined: interleaving chosen by 12% with no information, 40% with the percentage, 35% with percentage plus fluency reminder, 44% with everything (χ²(6) = 32.35); within each information condition, no relationship was detected between judgment and actual performance (smallest p = .17). Experiment 4: after being told that 90% of people do better with interleaving, 31 of 40 (78%) still said blocking was as good or better.
• Experiment 6: no information 46%, everything 75%, interleaving first then blocking plus everything 93%.
• Caveats the authors state: about 30 per group in Experiments 3 to 5, underpowered; the information conditions had strong demand characteristics; later actual choices were not measured.
• Inconsistencies in the paper: the 1A/1B panel labels in Table 1 appear to be swapped, so only the combined row is cited; the general discussion gives 44% for the "everything" condition of Experiment 5, which conflicts with 36% in that experiment's results; 44% is the figure for the three experiments combined.

Macaluso, Beuford, Fraundorf (2022), Journal of Intelligence 10(4), published version. Two online experiments, Study 1 with 328 and Study 2 with 377 University of Pittsburgh students, within-subjects. Bird photos at 3 seconds each, 3 families and 45 images per phase. Immediate ratings (1–6): interleaving was more effortful than blocking (d = 0.55, 0.41), felt like less learning (d = −0.47, −0.40), and less familiar (d = −0.45, −0.45). Chose blocking: 76.6%, 76.9% (before seeing the 90% explanation). Study 2 posttest (11 new images per family, three choices): interleaved 60%, blocked 55%, d = 0.18; the schedule × preference interaction was not significant (p = .28). Odds ratios for choosing interleaving per point of "interleaving felt like more learning": immediate scale 4.25, 4.79; retrospective scale 15.94, 12.87 (one retrospective item is itself "which would you use in the future"). The direct effect of effort on choice was not significant in three of four models; in Study 2 the indirect effect of familiarity on the immediate scale was p = .06, and the discussion section swaps the two scales. After the explanation: half did not answer; among those who had chosen blocking and answered, 63.7% (Study 1) and 60.6% (Study 2 table; 60.3% in the text) would not switch, and the paper does not state the denominator clearly. Limitations the authors list: psychology undergraduates only, only one study decision examined, mediation analysis is correlational.

St. Hilaire, Chan, Ahn (2024), Psychonomic Bulletin & Review 31(2), published version and author manuscript. Preregistered; 76 records. Content asked about g = 0.54 (k = 97), content not asked about 0.04 (k = 91, p = .349). Reading questions without answering 0.22 (95% CI −0.02 to 0.45, k = 29), required to guess 0.65. Excluded studies with answers given before reading and with questions embedded during study. No metacognition was measured; the predictions of the curiosity account received only weak support (negative relationship between pretest accuracy and effect, p = .095, with outliers). One classroom study and four online studies, too few to analyze.

Pan and Carpenter (2023), Educational Psychology Review 35, published web version (the page I obtained lacks the tables and figures). A narrative review that does not describe its search or inclusion methods. Its metacognition section cites: participants in Huelser and Metcalfe (2012) ranked pretesting as the least effective method; in a survey by Yang et al. (2017), 78% thought reading was more effective; in a survey by Pan et al. (2020), 56% chose guessing first and 44% chose reading only, and 81% in the same survey agreed that avoiding errors while learning is important; two studies showed that explaining the benefit or giving score feedback can correct the misjudgment. Its conclusion lists unsupervised online settings as an open question: when students can choose, will the experience of errors or the belief that it does not help lead them to skip it?

Kovacs (2016), L@S 2016, ACM published version. Coursera Machine Learning, fourth run: 96,195 registered, 61,453 started at least one video, 42,437 answered at least one quiz; 113 videos, 109 quizzes. Quizzes could be skipped, were ungraded, and showed the answer after three wrong attempts. In single-quiz videos, 74% of people who started answered and 66% finished; among certificate earners, 79% and 72%. First-try correct 76.0%. Among those wrong on the first try and right within 30 minutes, 90.9% did not go back before getting it right, with a median retry time of 11 seconds; the 9.1% who went back had a median of 115 seconds. Backward seeks from a quiz: 60.5% before the first answer, 19.4% never answered, 14.4% after a wrong answer, 3.0% after a correct answer; those who went back before answering were right on the first try 62.4% of the time, lower than the overall 76.0%. No statistical tests, no learning outcomes; several numbers for quiz counts and completion rates do not match between text and tables.

IES practice guide (Pashler et al., 2007). Recommendation 6a, "Teach students how to use delayed judgments of learning to identify content that needs further study," and 6b, "Use tests and quizzes to identify content that needs to be learned," both have a low level of evidence, because the evidence comes mainly from the lab. The procedure for 6a: after a delay, look only at the question, not the answer, and estimate the chance of answering correctly; the reading version is to do other homework after reading, then write four keywords and definitions on a blank sheet without looking at the text, and reread if you cannot. Checklist: "Remind students to complete judgments of learning without the answers in front of them." 6b recommends stating which pages to reread after a wrong answer.

References

Each was read from start to finish and has an extraction record; parts I could not read are noted in parentheses.

• Yang, Zhao, Yuan, Luo & Shanks (2023), Mind the Gap Between Comprehension and Metacomprehension: Meta-Analysis of Metacomprehension Accuracy and Intervention Effectiveness (author accepted manuscript on UCL Discovery; supplement not obtained).
• Prinz, Golke & Wittwer (2020), To What Extent Do Situation-Model-Approach Interventions Improve Relative Metacomprehension Accuracy? Meta-Analytic Insights (open-access published version on Springer).
• Prinz, Golke & Wittwer (2021), Counteracting detrimental effects of misconceptions on learning and metacomprehension accuracy: The utility of refutation texts and think sheets (open-access published version on Springer).
• Rozenblit & Keil (2002), The misunderstood limits of folk science: an illusion of explanatory depth (PMC author manuscript; figures read only through their captions, and the link to the online supplement is dead).
• Prinz-Weiß & König (2023), Caption it! The impact of headings on learning from texts (KITopen institutional repository version; supplement not read).
• Deslauriers, McCarty, Miller, Callaghan & Kestin (2019), Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom (published version, full text and supplement on PMC).
• Yan, Bjork & Bjork (2016), On the Difficulty of Mending Metacognitive Illusions: A Priori Theories, Fluency Effects, and Misattributions of the Interleaving Benefit (published PDF on the APA website; some figures read only through their captions).
• Macaluso, Beuford & Fraundorf (2022), Familiar Strategies Feel Fluent: The Role of Study Strategy Familiarity in the Misinterpreted-Effort Model of Self-Regulated Learning (published version on PMC; figures and OSF data not viewed).
• St. Hilaire, Chan & Ahn (2024), Guessing as a learning intervention: A meta-analytic review of the prequestion effect (published version and table pages on the publisher's site; abstract and footnotes from the author manuscript in Iowa State's repository; figures read only through their captions).
• Pan & Carpenter (2023), Prequestioning and Pretesting Effects: a Review of Empirical Research, Theoretical Perspectives, and Implications for Educational Practice (open-access published version; the page I obtained lacks the tables and figures).
• Kovacs (2016), Effects of In-Video Quizzes on MOOC Lecture Viewing (published version in the ACM Digital Library).
• Pashler et al. (2007), Organizing Instruction and Study to Improve Student Learning (IES practice guide).
• Posts on this site: Writing the Examples and Openings of a Teaching Article, Writing Questions and Wording in a Teaching Article, What the Evidence for Learning by Doing Measures, Pairing a Teaching Article with Video, Learning How to Learn.