Building Spaced Review into Teaching Articles: Quantum Country's Data, Online Course Experiments, and Why It Hasn't Caught On

🌏 中文版

This is part 17 of the series "Writing Articles That Teach," in its fourth section, "after some time has passed." Part 16 looked at how much of the benefit of distributed practice remains once it leaves the lab and goes into real courses. This post goes one step further: if spaced review works, can people who write articles build it into the article itself, so readers come back and answer again days or weeks later?

The trigger was this site's course guides. The spec asks for one self-check question at the end of each main section, with an answer or a criterion; in practice, the answer usually sits in a block that only opens when clicked (part 9 covers how to write the questions, and part 10 covers whether readers answer them). All of these questions are answered on the spot, at the point of reading. Once readers finish a post, the article has nothing more to do with them, and this site has no idea when, or whether, the next post in the series gets read.

Some people have tried to fill that gap. The best-known attempt is Quantum Country, built by Andy Matuschak and Michael Nielsen in 2019: a long essay on quantum computing with questions embedded in the text, followed by emails a few days later asking readers to come back and review. The two authors call this form the "mnemonic medium." This post goes through the data they published, then looks at what happened when the same idea moved into online courses, a language app, a programming course site, and medical students' self-study, and finally collects the reasons the builders themselves give for why it is hard to spread.

First, a gap. This site has no data on reading behavior. It does not load any analytics, so I do not know whether readers come back or how long they wait before doing so. I also did not find any research on how often readers of web-based teaching articles return. All the data below comes from other settings.

How these sources were chosen. I did a first scanning pass, then picked 11 sources and read them in full. The criterion was whether a source reported what actually happened after review was built into the material. The selection has two clear biases. First, the most detailed data mostly comes from the people who built the products: the authors of Quantum Country, Duolingo employees, and the founder of Execute Program. Matuschak's working notes and the Execute Program forum comment were never peer reviewed. Second, the only randomized studies are from online courses, a language app, and one middle school classroom. Matuschak later wrote more about Quantum Country's data in letters to sponsors; those letters are marked not to be shared or cited, so they are not used here.

The conclusions first

"No detectable difference" in the table does not mean "no effect." Among these studies, only two looked at review after a gap, used random assignment, and measured learning: the MOOC study found no detectable difference, and the classroom study found an effect, but in the classroom study the teacher scheduled review into class time, so students did not have to decide whether to come back.

The forms "review built into an article" has taken

The sources below all say "spaced review," but they did very different things. The differences come down to two points: who decides when review happens, and whether readers have to come back on their own.

There is one more approach that often gets lumped in: a question appears at some point while a video plays, testing what was just watched. The Coursera course analyzed in Kovacs 2016 works that way, and part 10 already covers its response rates. Those questions are answered on the spot, with no gap in time, so this post does not count them as spaced review. The self-check questions in each section of this site's articles belong to the same category.

What Quantum Country published

How it works

Quantum Country's first essay, "Quantum Computing for the Very Curious," embeds 112 questions. Readers create an account and answer each question as they reach it; each question and its answer form a pair, which the interface calls a card. A few days after reading, readers get an email asking them to log in and review.

The schedule works like this: after a card is answered in the essay, a correct answer moves the interval to 5 days, then 2 weeks, then 1 month, and so on; the authors write that after five successful reviews the interval is at four months. A wrong answer drops it one level, for example from 2 weeks back to 5 days. In a sidenote, the authors say the 5-day first interval was a recent change, and that for most of its life Quantum Country used a more conservative schedule.

The two authors describe all this in "How can we develop transformative tools for thought?" (October 2019). It is a self-published long essay, not a research report. It has no methods section, and its numbers are the authors' descriptions of their own backend data.

How much people who finish the reviews remember

The authors' measure is "demonstrated retention": the longest interval between a successful review of a card and the review before it. In other words, if the schedule pushed a card's interval to 1 month and the reader got it right that time, the card has 1 month of demonstrated retention. It is not a separate retention test.

For users who had reached their 6th review, the authors read from a graph that "most users" were at about 6,000 days summed across all 112 cards, or about 54 days per question. Getting there typically took about 95 minutes of total review time, and reading the essay takes about 4 hours, so the authors say the extra time is less than half. Matuschak's working notes give 90 minutes and a 35% to 50% overhead, slightly different from the essay.

The authors themselves state two conditions on this number, in another note:

• The data covers only readers who answered at least 80% of the questions in the essay before their first review session, on the more conservative schedule used in the first half of 2019.
• The note asks: "But is this just a survivorship effect?" It goes on to ask whether these readers would have remembered without review, or whether they already knew the material.

Another frequently cited number is 195: six months after the prototype was released, 195 users had demonstrated one month of retention on at least 80% of the cards in the essay. The essay does not say how many people read it or created accounts, so the number cannot be turned into a percentage. The authors describe it as "an extraordinary level of commitment to the process."

How much worse it is without review

The comparison that addresses this question is what the essay calls an "informal experiment." Eight cards were chosen. Some users, after answering them during the initial read, could not review them for at least two weeks; other users reviewed as usual. The essay does not say how users were put into each group.

• Delayed group, 25 people: accuracy went from 91% on the initial read to 87% two weeks later.
• Usual group, 16 people: from 89% to 96%.
• Person by person: everyone in the usual group stayed the same or improved; 40% of the delayed group got worse.

This comparison used an even earlier schedule (1 day, 3 days, 1 week, 2 weeks), and no significance test was run. The confidence intervals for the delayed group's two numbers (given by the authors as 91 ± 4% and 87 ± 5%) overlap; I noticed this by comparing the numbers, and the authors do not mention it. The authors also say that the delayed group kept reviewing other cards during that period, whose content overlapped with the 8 cards, and that this almost certainly boosted their final performance.

Questions inside the article, or a separate set afterward

The question writers most likely want answered is: do the questions have to be embedded in the text? If an article offered a downloadable question set for readers to review afterward, what would be lost? The published data I read does not answer this. The essay argues that embedding spaced repetition in a narrative is what lets this form handle abstract, conceptual knowledge, but it never compares "embedded in the article" with "a separate question set." The two authors also mention that they had previously argued users should write their own cards, while in Quantum Country the authors wrote them; they say preliminary user feedback suggests "it has worked out adequately" and that they want to look into it further.

How the authors rate it themselves

In one note, Matuschak lists the four claims Matuschak would like to be able to make: if you complete the review sessions, you will reliably remember what you read; you would remember much less without them; it will not cost much time; and the effect generalizes to many domains. Then comes the self-assessment:

Within Quantum Country, we have strong evidence of 1, moderate evidence of 3, and emerging evidence of 2. We have no evidence of 4 yet.

Read this carefully: the only claim the author considers strongly supported is "people who complete the reviews remember." The causal comparison, "people would remember much less without the reviews," is only emerging, and there is no evidence for generalizing to other domains. The essay carries the same caution: "These are preliminary results, and need more investigation." Its strongest conclusion is "At a minimum, it seems likely the mnemonic medium is genuinely helping people remember."

There are two more things to watch when citing the essay. First, it originally had a design principle that the opening questions should be very easy; in December 2019 the authors added a note retracting it, saying it came from an error in their data analysis. When the essay later lists three principles, this one is still there, not updated. Second, the essay's summary claims more than its body: the summary says effective memory can be increased by an order of magnitude and that genuine conceptual understanding can be built, while the corresponding passages in the body say "we don't quite mean those numbers literally" and that the authors do not yet know what readers are actually learning.

Moving into courses and products

Review questions pushed to MOOC learners

The 2018 experiment by Dan Davis, René Kizilcec, Claudia Hauff, and Geert-Jan Houben is the closest to a system scheduling review for learners. It ran in a geoscience MOOC on edX, and the 1,047 people who entered the course at least once were randomly split into two groups. In weeks 2 to 6, the treatment group saw a small window pop up automatically on lecture video pages, with a question drawn at random from earlier weeks' question bank; it could be closed from the top-right corner and was not graded. The control group got no pop-up questions, but both groups got the same weekly page explaining the benefits of self-testing and how to do it. The same experiment also has an extended journal version (Davis et al. 2018, JLA); part 10 cites that version, which has the details on response rates.

Of the 523 people in the treatment group, only 102 interacted with a pop-up question at least once. Comparing the groups as assigned, earning a certificate, getting a non-zero grade, and the continuous grade all showed no detectable difference; using assignment as an instrumental variable to estimate the effect on actual users also found none. The authors' boxed conclusion reads:

The causal analysis yields no evidence that ARPS raised learning, performance, or self-regulatory outcomes in this course.

The 102 who interacted with the questions had a 34% pass rate, far above the 8% for the whole control group. The authors say plainly that this cannot be read causally, because it compares people who chose to answer against everyone in the control group; they also attribute the lack of a detected effect to the small sample and low compliance. My judgment is that this experiment had very low statistical power: pass rates in both groups were only 7% to 8%, and the median grade was 0. So it shows "no evidence that it works," not "proof that it doesn't."

The paper also has exploratory data directly about time passing. For learners who passed the course, accuracy on these older questions (with skipped questions counted in the denominator) fell from 67% in week 2 to 49% in week 6; counted by how many weeks earlier the question's material came from, accuracy was 65% at 1 week and 42% at 5 weeks. The authors attribute this to the growing amount of material being tested each week, and to forgetting over time.

The same team's 2016 experiments in two other MOOCs are often cited alongside, but they did something else: after the last video each week, learners were asked to write 3 to 5 sentences on the key points, right then, with no delay. Of the people assigned to that condition, 22% clicked on any of the prompts; in the other course, 14.7% clicked on a study-planning prompt. The authors report no significant difference in grades for the summary-writing group among more engaged learners, but give no statistics. This study cannot be used to say "spaced review doesn't work." What it can say is that most people do not click on optional prompts placed in a course.

Review scheduled into class time

The 2014 experiment by Robert Lindsey, Jeffery Shroyer, Harold Pashler, and Michael Mozer is the only source in this set that used random assignment, measured a delayed exam, and found a positive result. It took place in a U.S. middle school: 179 eighth graders in six classes with the same teacher, learning a semester of Spanish vocabulary. Students used web-based flashcard software in class. Each chapter's items were randomly divided into three sets, each given a different review schedule: keep practicing the current chapter (massed), always review the previous chapter (generic spaced), or let a model estimate how much each student had forgotten of each item and pick items accordingly (personalized spaced). All three schedules got the same number of reviews.

On the cumulative exam 28 days after the end of the semester, items on the personalized schedule scored 16.5% higher than massed review and 10.0% higher than generic spacing. The paper does not say whether these are percentage points or relative differences; from the bars in the figure they look relative, which is my reading. On end-of-chapter quizzes, though, massed review scored highest (89.4%, against 87.2% and 88.1%), because it gave the most practice on the chapter being quizzed. In the same data, massing did better on the immediate test, and spacing did better on the delayed one.

Three things to keep in mind when carrying this over to articles:

• The teacher scheduled review into class time, about 30 minutes a week, roughly a tenth of the time students spent on the course. Students did not have to decide whether to come back.
• The personalized schedule used every timestamped response from every student on every item, plus a server that kept re-running the model. By the end of the semester one round of inference took half an hour, and the backend crashed at least three times.
• The authors acknowledge that personalization and "spacing reviews further apart" cannot be separated: "We acknowledge that our design confounded personalization and the coarse temporal distribution of review." So what can be said is that this personalized schedule beat always reviewing the previous chapter, not how much personalization itself contributed.

Duolingo's scheduling experiments measured return visits

Burr Settles and Brendan Meeder's 2016 paper proposes a model that predicts how quickly each word is forgotten and runs two user experiments on Duolingo. Both authors were at Duolingo when they did this research, and the product being tested was their own.

Both experiments measured only next-day return: the share of students who did an activity on one day and came back to do it again the next day. The paper measures no learning outcomes at all.

• The first experiment compared the new model with the original rule, over six weeks and just under 1 million students. Next-day return for overall activity and for new lessons were both +0.3%, not significant; next-day return to practice dropped 7.3%, significant. Before the experiment, many students had said they would practice just to keep their skill tree gold, but that practice didn't review what they felt they needed most. Combined with positive anecdotal feedback from the new-model group about the quality of the strength meter, the authors concluded the new model was better for engagement, "so we deployed it for all students."
• The second experiment, a few months later, compared a simplified version of the new model with one set of features removed against the full new model. After launch, students had complained that certain words decayed quickly no matter how much they practiced, which the authors traced to overfitting. The simplified version raised next-day return for overall activity by +12.0%.

The abstract's claim to "improve Duolingo daily student engagement by 12%" comes from the second experiment, which compared a fix for a problem in their own model, not the new model against the old rule. The paper does not say whether these percentages are relative changes or percentage points.

Allowing only a few lessons a day

Execute Program, a programming course site run by Gary Bernhardt, goes the other way: it limits the reader directly. In March 2021, in a Hacker News thread about addictive design, Bernhardt described it: on the first day of a course you can do only 5 lessons; on each following day you first do short reviews of the previous day's lessons, then a few new lessons, and then you are stopped until the next day. "You can't binge/cram even if you want to." Review intervals grow exponentially, and performance on reviews decides which lessons unlock.

Bernhardt says new users occasionally write in to complain that they want to finish a course in a day, while long-term users write in to say they like it: being able to answer the next day is what shows they really understood, and they no longer worry about whether they did enough today. The only number given is the median time to finish a course, 8 to 18 days depending on the course. Bernhardt writes, "Obviously I'm very biased here," and also says it has been hard to explain to people why the courses can't be binged.

The comment contains no learning outcome data. In the replies, one person says they would not want to be forced into a pace if they were paying, and another asks "who are you to judge how I learn ?" The longest line of objection is that locking people out each day and telling them to come back tomorrow is exactly how mobile games drive retention. None of the people replying says they are a paying or long-term user of the product.

Students using Anki on their own

The last form is readers doing it themselves. The 2026 systematic review by Nicholas Frappa and colleagues covers 11 studies of medical students' Anki use and exam performance, 1,135 people in total.

• None of the 11 studies was randomized. Five were cross-sectional; the rest were cohort studies, quasi-experiments, and mixed methods.
• "Using Anki" was measured in different ways: 5 studies used software logs, 4 relied on student self-report, and 2 had teachers hand out decks without measuring individual use.
• The three positive results on Step 1 of the U.S. medical licensing exam all come from cross-sectional self-report surveys; the only study that looked at Step 2 found no detectable difference.
• The two quasi-experiments with a comparison group point in opposite directions: the one where teachers handed out a four-week deck found improvement (d = 0.8), and the one using the previous year's class as the comparison found no detectable difference.

The authors conclude that "these findings should not be interpreted as causal," and that the observed associations more likely reflect retrieval-based, spaced study strategies than anything about Anki itself. On the pattern of more use going with higher scores, they say part of it may be self-selection: students who start out stronger, more motivated, or with more time are more likely to keep using it.

Why it hasn't become common practice

I did not find any data counting how widely this kind of practice has been adopted. In 2020, Matuschak and Nielsen wrote in "Timeful Texts" that spaced repetition systems are "not yet widely adopted." The reasons below are the builders' own accounts or my inferences from the data above; no study has directly compared why people do or don't adopt this.

Readers have to build a habit of coming back. This is the point the most sources lead to. Matuschak's note says "adopting any new regular habit is a burden for most people," that flashcard review reminds people of high school, and that the benefits don't show up quickly: "So people churn." "Timeful Texts" imagines a review session that includes a question about a book the reader has read, and the authors admit such a question may seem like a burden on its own, and that it depends on a situation that does not currently exist:

Such prompts might seem onerous or strange on their own, but what if most educated people maintained a regular review practice of the kind we’ve described?

The matching piece of data is the Davis experiment: the system had already delivered the questions to the page learners were viewing, and about 80% of those assigned still never touched them.

Good questions are hard to write. From "Timeful Texts": "One important barrier to adoption is that it’s surprisingly difficult to write good prompts." Matuschak's index note lists, among the barriers on the author's side, that writing questions for other people may be harder than writing them for yourself. The essay also describes a failure: for the only question in the essay that started with "Who," many users learned to recite the answer as soon as they saw "Who." "This is a common failure mode in memory systems, and it's deadly to understanding." The authors say they plan to try multiple equivalent phrasings to counter it, which they had not yet done.

Readers feel it is like homework. When Matuschak applied the mnemonic medium to non-technical essays, readers left comments that the note itself labels anecdotal, and the note groups one set of them as feeling like homework and stressful. One reader wrote that the prompts in one essay remind them of school homework. Another left after reading about 30% and said "it felt strangely stressful," but the same comment also says "I totally recognize that my retention of the core concepts is likely to increase from going through those prompts," and ends with "Definitely not a bad thing, albeit uncomfortable." Another note opens with "The initial mnemonic medium is implicitly authoritarian in premise." Matuschak argues that the current design gets in the way of browsing and skimming and makes the questions feel like someone else's, and proposes letting readers decide which questions to collect. The complaints from Execute Program's new users and the objections on Hacker News are the same kind of reaction.

It needs accounts, emails, and a scheduling system. Quantum Country requires readers to create an account and sends emails; Lindsey's personalized schedule needs a server that keeps re-running the model. Matuschak's index note lists another barrier on the user side: you need enough material before you get going, or review is boring.

The evidence itself is not there yet. The authors of the most complete project rate the causal evidence as only emerging; Duolingo's large experiments measured no learning; the Anki studies were not randomized. For people writing articles, the cost is substantial and the payoff uncertain. This one is my inference from the sections above.

The range where it fits may be narrow. Matuschak thinks the medium is especially suited to linear primers in established fields, where readers don't yet know enough and are willing to follow the path the author lays out: "I think it’s also important that the primer be covering a well-established field, so that its table of contents is broadly not up for much dispute or authorial idiosyncrasy." That note is entirely the author's reasoning, with no data. In the index note, next to the entry on retrieval practice and transfer learning, Matuschak adds: "I don’t think the MM performs very well here."

Where the sources disagree

The following results point in different directions, or look like they conflict while actually measuring different things. I set them side by side without picking one.

Quantum Country's summary versus its body. The essay's summary says effective memory can be increased by an order of magnitude and conceptual understanding can be built; the body says those multiples are not meant literally and that it is not yet clear what readers learn; Matuschak's later note rates the causal evidence as emerging and generalization as having no evidence.

Is enforced spacing a learning design or a retention tactic? Bernhardt says readers spend the same amount of time as they would cramming (the comment says "wall-clock time"; from context it means total study hours) and remember more, but gives no data. Objectors on Hacker News say locking people out each day and calling them back is the mobile game playbook. In Duolingo's experiment, next-day return to practice dropped after the schedule changed, and the company launched it anyway based on anecdotal feedback. None of the three has learning outcome data.

People who answer do well. In the Davis experiment, people who interacted with the questions had a 34% pass rate against 8% in the control group; in the 2016 Davis paper, people who actually filled in a study plan also had higher grades. The authors of both papers point out that this is self-selection. Quantum Country's 54 days also counts only people who finished. All three sources' "people who finish do well" is the same kind of data.

Immediate versus delayed tests. In Lindsey's data, massed review scored highest on end-of-chapter quizzes, and personalized spacing scored highest on the exam after the winter break.

Anki and different exams. The three positive results on Step 1 of the licensing exam all come from cross-sectional surveys; of the 6 studies on school course exams, 3 found no detectable difference, and a fourth found no overall difference. The review authors open their discussion with "modest but consistent improvements," but "consistent" only fits some of the studies.

This site's practice, compared

This site's course guides already do the first layer of Quantum Country: questions embedded in the text, answered when reached. What is missing is the reviews after that. The English speaking page adds a schedule, but like Quantum Country it relies on readers coming back on their own, and it lacks the email step.

What to do

Each item below is this site's judgment, with its basis.

1. Don't add a scheduling system to article pages for now. Basis: for the most complete project, the authors rate the causal evidence as only emerging; in the version that pushed questions to MOOC learners, about 80% of those assigned never touched them and grades showed no detectable difference; the classroom experiment that worked relied on class time, every student's responses, and a server. This site also has no reader accounts or email, and no channel for notifying readers other than RSS. In practice: put the effort into the in-article questions themselves (parts 9 and 10).

2. At the start of the next post in a series, include one or two questions on the previous post's core concept. This continues the second recommendation in part 16, with the same number of questions. It is the cheapest form of "answering again after a gap" I can think of: no accounts, no emails. It assumes readers actually reach the next post, and part 16 also warns against assuming readers will come back on their own. The basis is thin: Matuschak's note imagines serializing a text in small sections timed to readers' recall, which is an idea with no data; in Lindsey's study, "always review the previous chapter" had only a small advantage over massed review, and the paper reports no separate test for it; Davis pushed questions from earlier weeks and found no detectable difference in grades. So this is a low-cost experiment, not an evidence-based practice. In practice: put a link back to the previous post after the question, and keep the answer in a collapsed block as usual.

3. Avoid questions readers can answer by spotting a single word. Basis: Quantum Country's observation about the "Who" question, which is the authors' description with no numbers. In practice: if the same concept is tested twice across a series, phrase it differently the second time, for example asking for a definition the first time and giving a situation and asking which method applies the second time.

4. If the English speaking page's schedule is ever evaluated, measure whether people can answer, not just whether they come back. Basis: Duolingo's experiments measured only return visits; in both Davis and Quantum Country, "people who finish do well" may be self-selection. In practice: compare whether people can answer the same card when they return at the scheduled time, and remember that you are only seeing the people who came back.

5. Don't force readers to follow a spacing schedule. Basis: complaints from Execute Program's new users (as reported by the founder), the objections in the forum thread, and Matuschak's argument for reader control. None of these is outcome data, but there is no data on the effect of enforced spacing either. This site doesn't do this; the point is a reminder not to add it later.

Three things to try tonight

Pick a series you have written and make these changes; it takes about an hour:

1. At the start of the second post, add one question on the first post's core concept, put the answer in a collapsed block, and link back to the first post.
2. Read through the self-check questions and find any that can be answered by spotting a word in the question; rephrase them.
3. Make the review question from step 1 something readers can answer by choosing an option or writing a word or two, rather than calculating a number or writing a paragraph. The basis is Davis 2018: among the pop-up questions, about 70% of multiple-choice questions were attempted, against about 40% of those requiring a numeric entry. But the two question types were different questions and not randomly assigned, and answering more does not mean learning more (see part 10).

Limits of this evidence

• No study has tested whether readers of web-based teaching articles come back. This site has no reading behavior data. Carrying results from MOOCs, a language app, and a middle school classroom over to articles is my inference.
• The most detailed data comes from the people who built the products. Quantum Country's numbers come from the authors' essay and working notes, which are not peer reviewed, and the working notes keep being revised; the authors of the Duolingo paper are company employees; for Execute Program there is only the founder's forum comment.
• Most of Quantum Country's numbers have no denominator. The 195 people, the 54 days, and the 95 minutes all count only people who reached a certain point, with no figure for how many people read the essay or created accounts. The essay's figures were not in the version I read, so numbers "read from the graph" rely on the authors' own description in the text.
• Quantum Country's delay experiment is small: 8 cards and 41 people, with no description of how groups were formed, no significance test, and an earlier schedule.
• How much more in-essay questions help than a separate question set, and what happens when readers review too late, are not answered by Quantum Country's published data.
• Only two studies looked at review after a gap, used random assignment, and measured learning: Davis 2018 (no detectable difference; low statistical power is my judgment) and Lindsey 2014 (an effect, but with review scheduled into class time). The retrieval prompt in Davis 2016 asked for an immediate summary, not review after a gap; Duolingo's two experiments had no learning measure; none of the 11 Anki studies was randomized.
• Lindsey's two main t values are printed inconsistently with their p values in the published version itself, so I don't cite them; for tests, I use the analysis of variance in the supplementary material, with numbers in the appendix. The paper does not say whether its percentages are relative or percentage points.
• Timing. Execute Program's design is as described in March 2021, and I don't know whether it still works that way; Duolingo's experiments took place before 2016.
• Frappa et al.'s review itself was not preregistered, has no meta-analysis, and contains some inconsistent numbers (for example, it says both 5 and 6 cross-sectional studies); the abstract's "4–13 points" has no source for the 13-point figure in the text or tables.
• My inferences and judgments: that the end-of-article questions are not spaced; grouping the sources' adoption barriers into six categories; that weak evidence is one reason this hasn't spread; that Davis 2018 had very low statistical power; that Lindsey's percentages look relative; the five items under "What to do"; the three things to try tonight. Each is marked where it appears.
• Data not used: Matuschak's letters to sponsors are marked not to be shared or cited. Several of Matuschak's working notes analyzing Quantum Country data (on the 2019 informal experiment, in-essay questions, reviewing too late, and the overall impact on readers) currently cannot be opened on the author's site (404), so they are not used here; for the 2019 experiment, only that note is excluded, and the essay's description of the same experiment is still cited. One of Matuschak's notes has a title giving a retention rate for the first half of 2019; the file I had for that note was empty, so it is not cited.
• Reading gaps: I didn't read Davis 2018's preregistration; I couldn't get the publisher's PDF layout of Frappa et al., so the tables were read from the structured full text on PubMed Central; I didn't read the other roughly 220 comments in the thread where the Execute Program comment appears; for most of the other notes linked from Matuschak's notes, I read only the titles.
• Not found: research on whether readers of teaching articles return; peer-reviewed experiments on the mnemonic medium; studies that randomly compare the same questions placed in the article versus in a separate set; adoption statistics.

Appendix: study designs and numbers

The body keeps one or two numbers per item; the rest are here.

Matuschak and Nielsen (2019), "How can we develop transformative tools for thought?", a self-published essay, CC BY 4.0. Quantum Country's first essay embeds 112 questions and takes about 4 hours to read. Schedule: in-essay → 5 days → 2 weeks → 1 month…, dropping one level on a wrong answer. At the 6th repetition, "most users" among those who had reached it were at about 6,000 days (summed over 112 cards), or about 54 days per question; after the 1st review, the average per card was just over 2 days. Reaching the 6th repetition took about 95 minutes of total review time; the authors say the overhead is less than 50%. Six months after release, 195 people had one month of demonstrated retention on at least 80% of cards, with no denominator. Delay experiment: 8 cards; delayed group of 25 people went from 91% (± 4%) to 87% (± 5%); usual group of 16 people went from 89% (± 5%) to 96% (± 3%); the confidence intervals assume binomial, independent, identically distributed variables, which the authors call approximate; 100% of the usual group stayed the same or improved, while 40% of the delayed group got worse and 60% stayed the same or improved; the schedule was the earlier 1 day, 3 days, 1 week, 2 weeks. The principle that opening questions should be easy was retracted on 2019-12-09. The 25 million monthly active Duolingo users and 50 million for Quizlet cited by the authors are outside figures, not data from this essay.

Matuschak's public working notes, notes.andymatuschak.org, undated and not peer reviewed, as of 2026-10-13. Self-assessed evidence: 1 strong, 3 moderate, 2 emerging, 4 none. The 90 minutes and 35% to 50% come from readers in the first half of 2019; the same note says readers in the second half of 2019 (with a more aggressive schedule) may already have cut the cost by 25%. The 54 days covers only readers who answered at least 80% of the questions in the essay before their first review. As of the first quarter of 2020, "several hundred" users had demonstrated one month or more.

Matuschak and Nielsen (2020), "Timeful Texts", July 2020, published in The Future of Text, CC BY-NC 4.0. No data in the essay. In interviews with readers, the authors noticed that regular review did not just build detailed retention but also kept readers in contact with the material and changed their relationship to it; the number of interviews and the method are not described. Quantum Country's final chapter added a kind of question that changes over time in a set sequence.

Davis, Kizilcec, Hauff, Houben (2018), LAK '18, pp. 1–10. edX GeoscienceX, 2017-05-23 to 07-26; 2,324 registered, and the 1,047 who entered at least once were split into control 524 and treatment 523; preregistered, but the analysis was changed because of the small sample and low compliance, and the planned exclusion criteria were not applied. Question bank of 132 questions (112 multiple-choice, 20 numeric). In the treatment group, 102 interacted at least once. Intention to treat: certificate logOR = −0.215 (p = 0.357), non-zero grade logOR = 0.143 (p = 0.280), continuous grade p = 0.442; instrumental variable estimates p = 0.371, 0.267, 0.374. Compliers versus the whole control group: pass rates 34% versus 8%, which the authors say has no causal interpretation. Pass rates of 7% and 8% in the two groups, with grade quartiles of [0, 0, 2] in both. Per question: 69% of multiple-choice questions attempted, 41% of numeric ones; median time to dismiss, 7 seconds. Passing learners' accuracy (skips included) went from 67% in week 2 to 49% in week 6, and their error rate from 4% to 25%; non-passing learners' skip rate went from 47% to 61%. By question delay: passing learners' accuracy went from 65% at 1 week to 42% at 5 weeks; non-passing learners' skip rate went from 50% to 72%. A 10-question test two weeks after the course: control 6.2, treatment 6.6, N = 20 (response rates 7.3% and 8.8%, with the two groups sampled differently), p = 0.66. The extended journal version (JLA 5(3)) reports the same experiment.

Davis, Chen, van der Zee, Hauff, Houben (2016), EC-TEL 2016, LNCS pp. 57–71, read as the author manuscript. Two MOOCs from TU Delft, randomly assigned using edX's cohort feature. Retrieval practice: Functional Programming, 9,836 people in the cohorts, three groups (no intervention, write 3–5 sentences of key points, read a provided summary); of 3,262 in the writing group, 66.4% watched a video and 22% clicked a prompt. Compared only within the subsample who watched more video than average in week 1, the authors report no significant differences in weekly quizzes or final grades, with no statistics. Study planning: Industrial Biotechnology, 14.7% of 998 in the treatment group clicked a planning module; exposure alone made no significant difference; the 146 who actually filled in a plan had a final grade of 46.42 against 36.44 for highly engaged control learners, a self-selected comparison. The denominators for 22% and 14.7% are the whole group, although the paper's sentence reads as if only video watchers were counted.

Lindsey, Shroyer, Pashler, Mozer (2014), Psychological Science 25(3). A middle school in suburban Denver, 179 eighth graders, 6 classes, one teacher, third-semester Spanish; 409 words and phrases grouped into 221 knowledge components. The three schedules had the same number of reviews (about 8 per item), with average days between reviews of 0.12 for massed, 1.69 for generic spaced, and 4.70 for personalized. End-of-semester exam: personalized was 12.4% higher than massed (d = 1.38) and 8.3% higher than generic (d = 1.05); 28 days later, 16.5% (d = 1.42) and 10.0% (d = 0.88). Forgetting rates over the winter break of 18.1%, 17.1%, and 15.7%, not defined or tested in the paper. In the supplementary analysis of variance, the main effect of schedule was F(2, 328) = 52.3 on exam 1 and F(2, 340) = 55.1 on exam 2, both p < .001. Among students who took both exams, only 22.3% scored higher on generic-spaced items than on personalized ones, and 13.5% scored higher on massed items than on personalized ones. End-of-chapter quizzes: massed 89.4%, generic 87.2%, personalized 88.1% (156 people). The effect was concentrated in chapters from early in the semester. 597,990 responses over the semester.

Settles and Meeder (2016), ACL 2016, pp. 1848–1858. Historical data: two weeks, 12.9 million student-word records; mean absolute error in predicting recall, 0.128 for the new model, 0.235 for the original Leitner rule, and 0.175 for always guessing the average; ranking quality (AUC) was highest for Leitner at 0.542, with the new model at 0.538, both only slightly above the random 0.5. User experiment 1 (new model versus Leitner, six weeks, just under 1 million people) found overall activity +0.3, new lessons +0.3, and practice −7.3 (only the last had p < 0.001). Experiment 2 (simplified versus full, two weeks, 3.3 million people) found +12.0, +1.7, and +9.5, all p < 0.001. The table caption reads "Change (%) in daily student retention," with no baseline return rate, group sizes, or confidence intervals.

Bernhardt (2021), Hacker News comment 26349754, 2021-03-04. The founder's own report: 5 lessons on the first day, then reviews each day before new lessons; exponentially growing review intervals, with lessons unlocked based on review performance; median time to finish a course of 8 to 18 days; paid after the 16th lesson. The 40 replies from other users came from about 30 accounts, and none says they are a paying or long-term user of the product.

Frappa, Chernov, Dillon, Alben (2026), Medical Science Educator 36(2). Seven databases, searched up to 2025-10-12, English only; 4,600 records screened down to 11 studies from 2015 to 2025 (9 in the U.S., 1 in Ireland, 1 in Pakistan), each with 25 to 206 people. Quality scores (MERSQI, out of 18) ranged from 8 to 11.5, all single-institution. Step 1: self-reported heavy use 241.1 versus 235.5 (p = 0.012); daily use 238 versus 233.5 (medians, p = 0.039); the number of cards seen predicted scores. Step 2 CK: 252.5 versus 247.0 (p = 0.440), and Anki use was not an independent predictor after controlling for Step 1. Course exams: Durrani 2024, a quasi-experiment with 115 people, d = 0.8; Gilbert 2023, students who volunteered for training scored higher; Levy 2023, Sun 2021, and Magro 2024 (p = 0.43) found no detectable difference or association; Haughey 2025 found no overall difference, but did in physiology.

Kovacs (2016), L@S 2016, pp. 31–40. A log analysis of the fourth run of Coursera's Machine Learning course, where in-video quizzes test the content just watched; across the 92 videos with one quiz, 74% of people who started a video answered its quiz. No random assignment, no statistical tests, no learning outcomes. Details in part 10.

References

Each source was read from start to finish, with an extraction record; parts I couldn't read are noted in parentheses.

• Matuschak & Nielsen (2019), How can we develop transformative tools for thought? (public version on the authors' site; images and videos not available).
• Matuschak & Nielsen (2020), Timeful Texts (public version on the authors' site).
• Andy Matuschak's public working notes (as of 2026-10-13): Mnemonic medium, Effects of the mnemonic medium on reader memory, Mnemonic essays may offer detailed retention of their contents in exchange for 35-50% reading time overhead, Quantum Country readers reliably develop detailed retention for embedded questions, Lessons learned from applying the mnemonic medium to non-technical texts, The mnemonic medium works well for linear primers in established fields, The mnemonic medium should give readers control over the prompts they collect, Regular spaced repetition memory practice is an onerous habit to adopt, Mnemonic texts could be serialized in small sections over time, timed to reader recall, Substantive examples of the mnemonic medium, Execute Program (for most other notes linked from these, only the titles were read).
• Davis, Kizilcec, Hauff & Houben (2018), The Half-Life of MOOC Knowledge: A Randomized Trial Evaluating Knowledge Retention and Retrieval Practice in MOOCs (ACM-typeset version on the first author's site; preregistration not read).
• Davis, Kizilcec, Hauff & Houben (2018), Scaling Effective Learning Strategies: Retrieval Practice and Long-Term Knowledge Retention in MOOCs (extended journal version of the same experiment; published version, open access).
• Davis, Chen, van der Zee, Hauff & Houben (2016), Retrieval Practice and Study Planning in MOOCs: Exploring Classroom-Based Self-regulated Learning Strategies at Scale (author manuscript on co-author Hauff's site).
• Lindsey, Shroyer, Pashler & Mozer (2014), Improving Students' Long-Term Knowledge Retention Through Personalized Review (published reprint on corresponding author Mozer's site, including supplementary material; the model equations in the supplement were not checked one by one).
• Settles & Meeder (2016), A Trainable Spaced Repetition Model for Language Learning (published version on Duolingo's research page).
• Bernhardt (2021), Hacker News comment: Execute Program's daily limit (the comment plus the replies below it, 44 in all; other comments in the same thread not read).
• Frappa, Chernov, Dillon & Alben (2026), Anki Use and Academic Performance in Medical Education: A Systematic Review of Evidence and Learning Theory (published version, open access; the publisher's PDF layout was not available, so tables were read from the PubMed Central full text).
• Kovacs (2016), Effects of In-Video Quizzes on MOOC Lecture Viewing (L@S 2016 published version, ACM layout).
• Posts on this site: Writing Questions and Wording in a Teaching Article, Do Readers Answer the Questions in an Article, How Much of the Spacing Effect Survives in Real Courses, English speaking practice (Chinese interface).