Can Experiments Test How a Teaching Article Is Written? Evidence from Platform A/B Tests, Scale-Up Replications, Rubrics, and Design-Based Research

🌏 中文版

This is Part 26 of the "Writing Articles That Teach" series, in its last section, "how to know it works, and how to make it". The previous part looked at an author with no exam to give: what each of the signals in their hands (readers saying it was well written, how long they read) actually measures. This part goes one level up. If there are exams, large numbers of learners, and the ability to randomize people into groups, what can experiments tell someone who writes teaching materials? And when experiments aren't possible, how much support do the two common substitutes have?

The question comes from how this site checks its own work. After a course guide or research roundup is written, it has to pass two gates. The first is a writing specification plus a few checker scripts, which together act as a checklist: a complete example needs conditions, reasoning, a result, and what the result means; a self-check needs an answer or a criterion; the verification process must not appear in the body text; and so on. The second gate is a reviewer who took no part in writing, who checks the article against the sources and the specification item by item. Both gates check the article itself. Neither measures what readers learn from it. So this part asks three things:

1. What have randomized experiments produced on the online learning platforms that can run them?
2. Have rubrics (checklists that score materials item by item) themselves been validated? Are rubric scores related to learning outcomes?
3. Design-based research (DBR) is an approach in education research: design something in a real setting, implement it, revise it based on the results, implement it again. It looks a lot like this site's cycle of writing, reviewing, revising the specification, and writing again. How is this method actually used?

Part 8 discussed whether observational data can show that doing practice has a causal effect, and Part 19 covered what the 247-course scale-up study means for completion rates. When this part cites the same data, it only adds their role in the question of how to validate teaching materials.

How sources were chosen, and their biases. I scanned the literature first, then picked sources to read closely. There are 11 references; for each I read the whole of the version I obtained and kept an extraction record. The biases are clear. Of the five platform-experiment sources, three come from the same platform, ASSISTments, and two of those share three authors. Apart from the 247-course study, the materials are all mathematics, physics, or psychology, and the learners are mostly US K-12 students and MOOC participants. The two rubric sources come from the same research team. All three design-based research sources review how other people's research reports are written; none is an effectiveness study. None of them measures readers of a blog or a public teaching article.

The short version

Randomized experiments on platforms

50 homework experiments on ASSISTments

ASSISTments is an online math homework platform used in US K-12 schools. It has a testing system for researchers (E-TRIALS): researchers adapt existing assignments and add experimental conditions, teachers assign the homework as usual, and students are randomized individually to different versions. A 2022 paper by Ethan Prihar and five coauthors pooled the experiments run since February 2019 to look for trends across experiments.

Start with the denominator. In that period, 103 experiments were deployed on the platform; 50 entered the analysis, covering 30,408 students. The inclusion condition was that at least one condition resembled a condition in another experiment and the data were complete enough. The 53 excluded experiments were of two kinds: too unique, with no other experiment to pool them with; or broken by build errors or dead video links, so students were never actually assigned. The paper gives no separate count for each kind, so it can't be described as half the experiments being broken.

The method was to pool the data of students under the same condition across different experiments and run regressions on the pooled data, not to compute an effect per experiment and combine them as a meta-analysis would. A few results are fairly clear:

• Support beats showing only the answer. Five kinds of support all beat simply showing the answer: giving the full solution to a similar problem (a worked example), step-by-step hints, breaking the current problem into a few smaller problems, and recording the worked example or the smaller steps as a video. Their d ranged from 0.155 to 0.267. The only one where no difference was detected was giving the full solution to the current problem, with only 85 students and a wide confidence interval.
• Differences between completion thresholds come almost entirely from completion rates. Lowering the threshold from three correct in a row to two gave a higher overall standardized score (d = 0.142). But among those who finished, no difference was detected at any threshold. The authors themselves say the difference comes almost entirely from more students in the higher-threshold groups not finishing.
• Motivational messages went the other way. Groups whose homework included motivational messages and videos performed worse (d = −0.072, 12,243 students). The authors guess the messages were distracting to read, but they didn't record how students felt about them.
• Giving choices showed no detected difference. Letting students choose their own problems or versions: 9 experiments, 12,789 students, d = 0.014. The body text doesn't discuss this result.

To read these numbers, first know what they measure. All five dependent variables are performance during or within the homework: how many problems it took to reach mastery, a post-test inside the assignment, learning gain, accuracy, and whether the assignment was finished. There is no delayed test and no test outside the platform. And students who didn't finish were all scored 0, on the authors' reasoning that they presumably learned the least, so learning in the paper is mixed with whether students finished.

In their limitations section, the authors acknowledge that with 19 research questions and pooled experiments that were similar but not identical in design, the risk of false positives went up. Their standard was that a result is probably credible if d is above 0.1 and the p-value is on the order of one in a thousand. I went through the results item by item with that standard; the ones that hold up are several kinds of support versus only the answer, two correct in a row versus three or four, and video versus text. Yet the conclusion section also presents three results with d below 0.1 as findings (adaptivity, fill-in questions outperforming multiple choice, and motivational messages going the other way).

How precisely experiments on the same platform can measure

That A/B tests on such platforms are underpowered is a premise the paper below states in its abstract; it does not test that claim with data.

The 2023 preprint by Adam Sales and coauthors is a statistical methods paper, not a study of teaching effectiveness. It uses 227 pairwise comparisons drawn from 68 A/B tests on the same platform (38,035 students) to test a method. The method trains a prediction model on the historical records of more than 190,000 students who weren't in the experiments, puts the predictions into the analysis as a covariate, and checks how much the standard errors shrink. There is only one outcome variable: whether the assignment was finished. The paper has no learning test and no power calculation anywhere.

The main result is that the method helps, but not by much. Compared with an unadjusted t-test, the median improvement in standard error is equivalent to about 20% more sample; compared with an adjustment using only within-experiment data, about 10%. Broken down into subgroups, 11% to 13% of cases actually got worse; every case where adjusting only with the predictions was more than 10% worse than the t-test occurred in a subgroup of fewer than 80 students.

More relevant to this part's question is a table that comes with it. Of the 227 comparisons, 38 were significant at the .05 level with an unadjusted t-test. After controlling the false discovery rate, only 3 remained; with a correction that also holds when comparisons are dependent on each other, 2 remained. The authors themselves point out that even if nothing had any effect, this many tests would be expected to produce about 11 significant results, and that "Counting significant p-values is a problematic approach". Few significant results could mean low power, or interventions that simply have no effect; without knowing the truth, you can't tell which. When estimates were broken down by subgroup, a t-test with false-discovery-rate control left none at all.

So what can be said is this: on a platform with tens of thousands of students and individual random assignment, with was the assignment finished as the outcome, the great majority of comparisons could not detect a difference after correction. This paper doesn't answer which cause is behind it.

Two experiments in one MOOC

A 2016 paper by Zhongzhou Chen and six coauthors reports on a mechanics review course MIT ran on edX. That run had about 11,000 registrants and slightly more than 500 earned certificates. The authors say seven A/B experiments were run in that course; this paper reports only two, and the results of the other five are not given.

The first experiment compared three kinds of homework: deliberate practice homework that breaks problem solving into single skills to practice (in a drag-and-drop format or a multiple-choice format), and traditional end-of-chapter problems. On each unit's quiz, no difference was detected among the three groups; with the three units combined, the multiple-choice version of deliberate practice was 4.7 percentage points lower than traditional homework. For the 20% to 40% of students who got the opening problems wrong at first, the drag-and-drop format led to more correct answers on later problems within the sequence than the multiple-choice version did. That is a selected subsample.

The second experiment compared two ways of filming explanation videos. Of the students who answered, fewer than 30% opened the video before their last attempt, so the analysis sample was only 36 to 113 students per video per version. Of the two engagement measures, one went in inconsistent directions across the four videos and the other was significant for only one video. The learning measure (whether students got the same problem right on their final attempt after watching) was significant for two of the four videos, in opposite directions. Accuracy ranged from 66% to 98%; the authors themselves say there was a ceiling effect, and they didn't test transfer.

The authors' overall assessment is blunt:

Our first attempts at conducting and analyzing learning from AB experiments in a MOOC environment did not result in eye opening discovery or solid conclusions.

They listed three difficulties with running experiments in MOOCs. In my view, all three apply to people writing teaching materials too:

1. Learners are too varied. In the first experiment, 60% to 70% of students already had the skills, and the authors say any teaching method might only make a difference for a small group among them. They suggest including a pretest in experiments to pick out the people likely to respond.
2. Assessment amounts to an open-book exam. Students can learn from elsewhere in the course, from other websites, or from books, and can even take the test before the treatment. The authors had almost no control over when or in what order students answered.
3. The data are too abundant and too messy. Performance on the same problem can be measured by first-attempt accuracy, last-attempt accuracy, number of attempts, or time spent. Which to choose rests "more on intuition plus trial and error", with no research basis.

One more point, which the authors themselves wrote: the significant results in both experiments came from selected subsamples, much smaller than the registration numbers would lead you to expect.

Putting the whole platform into schools

The three sources above all change one feature inside a platform. The 2016 study in AERA Open by Jeremy Roschelle and coauthors works at a different level: it randomized schools and asked whether students in schools that used ASSISTments for math homework learned more.

The setting was Maine, which gives every middle school student a laptop they can take home. There were 43 schools and 2,850 seventh graders. The treatment group got a whole package: the platform itself; university students who, over the summer, typed in all the homework problems from each school's textbook; skill-practice problem sets the control group didn't have; and teacher professional development (3 days in the summer, about 6 hours of in-class coaching during the school year, and 2 more days the next summer). Teachers also used the platform for a year first, and the next year's new cohort of students was the one analyzed. The control group assigned homework as usual. The outcome was a nationally normed standardized test at the end of the year, scored by the publisher without knowing group membership.

The treatment group scored 8.84 points higher after adjustment, Hedges's g = 0.18. The effect was larger for students who were below the median in sixth grade (g = 0.29 versus 0.12; these two numbers come from a 2017 correction, because the original table misprinted both cells). Students actually spent less time on the platform than the authors expected.

This is the only study in this part that used an end-of-year test outside the platform, with random assignment in real schools, and also the strongest evidence. The price is that it can't answer which part worked: the authors state in their limitations that they can't identify which mechanism in the intervention was most related to the effect. So 0.18 can't be taken as the effect of the platform itself, much less added to or compared with Prihar's d values for single features inside the platform. The title says only that online math homework increases student achievement; it doesn't mention the teacher training or the warm-up year.

What worked in a few courses, scaled up to 247

The 2020 study in PNAS by René Kizilcec, Justin Reich, Michael Yeomans, and six coauthors put four light-touch writing activities into 247 courses offered by Harvard, MIT, and Stanford from September 2016 to May 2019: writing a study plan, writing about how the course connects to values one cares about, mental contrasting plus an if-then plan, and writing down who could hold you accountable. The focal sample was 269,169 people, in four waves, each with a preregistered analysis plan. The first three activities had previously worked in a few MOOCs, and the original studies were all done by these same authors; the last was tested in online courses for the first time.

Details of the results are in Part 19. For the question of how to validate teaching materials, three points matter:

• None of the earlier effects replicated. Earlier studies of planning and mental contrasting raised completion by about 2 to 4 percentage points in the corresponding subgroups, and the values connection had larger effects in two courses. After scale-up, none of these main hypotheses showed a detected difference, and neither did social accountability, tested for the first time. The planning interventions increased activity in the first and second weeks, but no difference was detected in final completion.
• The one result that replicated is much smaller, and depends on the course. In courses where students from less developed countries had clearly lower completion in the control group, values-connection writing raised these students' completion by about 2.8 percentage points, and this replicated in both years. In courses without such a gap, the same group of students had about 1.6 to 1.7 percentage points lower completion. Predicting in advance from course features which courses would have a gap was 54.3% accurate, about the same as the 50% of random guessing.
• An order of magnitude smaller holds for only one intervention. The abstract says scaling up makes average effects an order of magnitude smaller. In the body text, this corresponds only to the values connection: 24 percentage points earlier (a study of 64 people) versus 2.79 now. The earlier studies of the other interventions weren't small (2,053 to 4,628 people), and after scale-up no difference was detected, rather than a tenth of the effect remaining.

The authors withdrew their earlier recommendation. In the original studies they had recommended that policymakers and online instructors adopt these interventions in their own MOOCs. Their conclusion now:

we now conclude that further research is necessary to predict in advance when these interventions will help populations of students in need.

The denominator for this result is people who reached the later part of the survey at the start of a course and were randomized, about 8.8% of registrants (I calculated this from the counts in the supplementary materials); more than 70% had a bachelor's degree. The authors themselves say the results shouldn't be extended to less engaged students or other educational settings.

Without random assignment

When a platform doesn't run experiments, the most common approach is to analyze existing usage logs. A 2015 study by Kenneth Koedinger and coauthors analyzed a psychology MOOC: one standard deviation more interactive practice went with a 0.44 standard deviation higher total score on weekly quizzes, more than six times the figure for watching more videos or reading more pages. The authors used a causal model search algorithm to support reading this as causal, while acknowledging that naturally occurring usage data plus very high attrition make causal inference more uncertain; what it can provide is "initial, if not confirming, evidence". The quiz questions and the practice came from the same question bank. On the final exam written by the instructors, the share of excellent performance among those who only did more practice (16%) was about the same as among those who only watched more video (15%).

The advantage of this kind of analysis is that no separate experiment has to be designed. The drawback is that it can't separate practice works from conscientious people do more of everything. Part 8 covered how the same authors later dealt with this problem, and Part 9 used this study's numbers.

Putting it together

Taking the sources above together:

• When people can be randomized and the outcome is performance during the homework, the clearest result is that support beats showing only the answer, with d of about 0.16 to 0.27. This is immediate support in math homework, some distance from how an article is written.
• What the outcome measures almost determines the conclusion. The effect of completion thresholds is really an effect on completion rates; Sales's 227 comparisons have only finished or not; Chen's learning measure hit a ceiling. Only Roschelle used an end-of-year test outside the platform, and that was a whole package of interventions.
• Many comparisons can't detect a difference, even on a platform with tens of thousands of students. Sales's table and Chen's analysis samples both show this. In scale-up replications, most earlier effects didn't replicate.

Rubrics: have the rubrics themselves been tested?

For people who can't run experiments, the most common substitute is a rubric: a list of what good teaching materials should have, item by item, for someone to score or check off. This site's writing specification is one of these. The question is whether such rubrics have themselves been tested, and whether materials with high rubric scores lead people to learn more.

Where rubrics come from, and whether they were tested

A 2015 review by Min Yuan and Mimi Recker looked for rubrics that evaluate the quality of open educational resources (OER), searching through April 2014 and including 14. They checked whether each rubric's public documents reported certain things:

This table is easy to misread as saying most rubrics have no empirical support. Its criterion is whether the documents report testing, not whether testing happened, and it doesn't distinguish whether reliability, validity, or usability was tested, or how well. Two numbers stand on firmer ground than no reported testing: 11 of the 14 had no detailed scoring guidance, and 11 were never revised. The paper also has no limitations section; it calls itself a systematic review but doesn't explain how 200-plus articles were narrowed to 14, or how many people did the coding.

A 2020 study by Ji Eun Lee, Mimi Recker, and Min Yuan first reviewed 10 rubrics for evaluating the quality of college online courses, with a stronger conclusion: "none of the reviewed rubrics publicly reported the results of reliability or construct validity tests". Only two mentioned empirical testing (pilot use, inter-rater agreement), with no details reported. Most rubrics were adapted from other rubrics rather than derived from theories or models of online learning.

Testing one rubric

The same paper then tested one rubric. The teaching support unit at a mid-sized US public university adapted Quality Matters (QM, one of the 10 above, with 42 weighted items) into a 51-item rubric, with each item scored as present or absent. Instructional designers sampled 202 online courses offered from 2012 to 2016 and scored them at the start of the semester, with only one rater per course.

The first result is that the rubric itself had problems. After removing items step by step and examining the factor structure, 12 of the 51 items were dropped (the authors say "almost one-fourth"). Of the first 8 dropped, 4 were about accessibility (captions for audio and video, alternative text for images, and the like) and 4 were about access to technology and resources. The authors attribute the problem to imprecise wording; for example, phrases like unreasonable software requirements and sufficient instructions were never defined. Among the remaining 39 items, the authors considered one (that course fees, if any, are explained) not to be measuring the same thing as the other items in its factor.

The second result concerns the relation between rubric scores and outcomes. Among the 121 courses with usage logs and grades, covering 5,240 students:

• Of the nine dimensions, only Learner Engagement and Interaction was related to the amount of interaction on the learning platform (students viewing materials, students posting to each other, instructors posting). The other eight dimensions had no significant relationship with any of the four kinds of interaction.
• Only student-content interaction was linked to course pass rates (β = .358).
• None of the nine dimensions had a significant direct path to pass rates. The only direct path left from a dimension to pass rates was course facilitation → pass rate, and it wasn't significant.
• The whole model explained 16.3% of the variance in pass rates.

These results come with four conditions. The data are correlational, yet the authors call the path analysis "causal relationships" throughout. The final model was obtained by running the full model, deleting the non-significant paths, and rerunning. Pass rates include D grades and average 0.90, and the mean scores for the rubric's dimensions also range from 0.76 to 0.95, so both sides sit near the ceiling. Pass rate is not a separately measured learning test.

So what this paper can say is: in one university, nearly a quarter of the items in a rubric adapted from QM had problems, and no significant direct path was detected from dimension scores to pass rates. It can't say rubric scores are unrelated to learning, because it didn't measure learning.

There is one more related study: a 2016 paper in Computers & Education by Shanna Jaggars and Di Xu examining how several aspects of online course design relate to student grades. The study exists, but I didn't find a legitimately public full text, so I don't cite its results.

Design-based research: can repeated revision replace experiments?

What the method does

Design-based research starts from the view that classroom interventions are hard to control the way a laboratory can. Rather than run one controlled experiment, you build a design in a real setting, implement it, look at the results, revise it, and implement it again, improving the design while distilling design principles that can be used elsewhere. This closely resembles this site's cycle: write an article, review it, write the problems the review caught back into the specification, write the next article. So it is worth seeing how the method is actually used.

First, what this section can answer. The three papers I read are all reviews: they look at what published design-based research reports say and leave out. None of them compares whether materials made with design-based research lead people to learn more than materials made some other way. Among the sources I read, no study has directly tested this question so far.

162 studies from 2004 to 2013

A 2015 review by Lanqin Zheng looked at design-based research published in 219 SSCI education journals from 2004 to 2013 and screened it down to 162 studies. Three raters coded independently, with agreement above 0.9.

• Half implemented only one cycle: 81 of the 162.
• A quarter (42) didn't revise the intervention at all; 98 explained how they revised it and 64 didn't.
• 72.84% of the research methods were purely qualitative, and none was purely quantitative. Among data sources, outcome data such as tests and student work made up only about a fifth.
• About 70% of the studies completed design and testing within a year.

Two caveats. The inclusion criteria also took in conceptual papers discussing design-based research, yet the six categories for number of iterations add up to exactly 162; the paper doesn't explain how the conceptual papers were coded. Also, one cycle is the number of implementations reported in a single paper, and a multi-year project might write one paper per cycle.

163 studies from 2013 to 2020

A 2022 review by Luís Tinoca and four coauthors picked up where Zheng left off. Following the PRISMA process, it looked at design-based research indexed in Scopus and Web of Science from 2013 to 2020 in K-12 and teacher-education settings, including only open-access papers in English, and arrived at 163 studies.

• 103 of the 163 didn't state methodological limitations; 148 made no recommendations at all about using design-based research.
• 40% (65) didn't say how many iteration cycles they ran. Of the 98 that did, none ran only one.
• Only 25% (40) reported how each cycle was revised; 61 gave no explanation, and 50 said there were revisions without giving details.
• 32% (52) didn't say how long each cycle lasted.
• Sorted by the main results each study reported itself, the largest category was potential to improve learning (73 studies); 24 reported that learning outcomes actually improved.

The authors attribute part of the incomplete reporting to journal word limits, with the consequence that others find it hard to replicate or adapt these interventions.

27 articles in one journal

A 2024 paper by Alexa Brase takes a different angle: what arguments the authors of design-based research use to defend their results. Brase looked only at the 57 articles from 2017 to 2022 in one journal devoted to design-based research (EDeR, the journal that published this paper), kept 27 research articles, and coded them alone. Brase states plainly that this is not a systematic review and that it has a European and German bias.

• The 27 articles share no common section structure: 3 have case-style headings, 6 look generic but differ from one another, and 18 are mixed.
• The author identified eight kinds of key argument, such as robust because it holds up across different settings, feasible because practitioner partners took part, refined through iteration, and triangulated across multiple data sources. The four most common were each named in only 3 to 6 articles.
• Most authors only outlined their methodological choices, and not in detail.
• One case the author relays is instructive: Brahm (2017) ran a pretest and posttest with a control group plus a transfer measure, and the results were mixed, but these results "did not seem to have played a major role" in later revisions of the intervention. The outcomes were measured but weren't fed back into the design.

The author warns readers of reviews like this:

Outcomes described in a single article must therefore not be confused with outcomes of a DBR project.

This sentence also limits how far reviews like Zheng's and Tinoca's, which count how many cycles each paper ran, can go.

Another related assessment is a 2012 article in Educational Researcher by Terry Anderson and Julie Shattuck, which examined a decade of design-based research. The article exists, but I didn't find a legitimately public full text, so I don't cite its content.

What people writing teaching materials can borrow

What the reviews of design-based research show is what a method of repeated revision often leaves out of actual reports: what was revised, why, and what limitations the method has. They don't show that what repeated revision produces is more effective, nor that it is less effective. Treating it as a substitute for experiments currently has no evidence for or against it.

Where the sources disagree

Between sources

Do design-based research reports explain their revisions? In Zheng's table (2004–2013), 60% explained how they revised; in Tinoca et al. (2013–2020), only 25% reported the revisions in each cycle. The two papers differ in coding categories, population (whether higher education is included, whether only open access is included), and search terms, so this can't be read as a drop from 60% to 25% over a decade. The number of iterations is the same story: in Zheng, one cycle accounts for 50%; in Tinoca, zero studies ran one cycle and 40% didn't say. Tinoca et al.'s conclusion says "It is also common to find studies reporting only one of the intervention cycles", meaning papers that report just one cycle of a larger project; the table does not show whether such papers were counted as "didn't say" or classified some other way.

Does support on a platform help? In the pooled analysis by Prihar et al., hints, smaller steps, and worked examples all beat showing only the answer; in the MOOC studied by Chen et al., deliberate-practice homework did no better than traditional homework, and the multiple-choice version did slightly worse. The two compare different interventions, learners (K-12 students versus adults with some background in mechanics), and outcomes (performance within the homework versus open-book quizzes). I place them side by side to show that whether experiments on platforms find effects has no single answer, not because the two refute each other.

Earlier studies versus the scale-up study. Of the main hypotheses in Kizilcec et al., the three with earlier studies had all been positive and significant in the corresponding subgroups; after scale-up, no difference was detected for any of them. The authors overlap across the two, and the intervention wording, exclusion criteria, and subgroup definitions were also adjusted between waves.

Within the same paper: abstract versus body

Of the 11 sources, 7 have an abstract or title that claims more than the body text. The three that would most change a conclusion: the order of magnitude smaller in Kizilcec et al. holds in the body for only the values connection; the completion thresholds in Prihar et al. move the completion rate, with no difference detected among those who finished; Zheng's abstract says revision methods weren't explained in detail, while the table shows 60% explained them. The paper-by-paper comparison is in the appendix.

Checking this site's practice

The following is based on the versions in the working directory on 2026-10-14. This site's writing specification and review process are an instance of a checklist and of repeated revision. This section and the next are my judgment, checking this site against the research above and then extending it to a single teaching article. What errors the reviews actually caught will be covered in the last part of the series.

One clause in the specification points the same way as the recommendations from rubric research: after changing the specification or a checker, rerun a set of representative tasks, keeping good and bad samples side by side, with an independent reviewer pointing to evidence in the body text. This checks whether a rule can tell good articles from bad ones. It is a validity check of the rubric, not a check of learning outcomes.

What can be done

Each item gives its basis; all the bases come from platforms, courses, and research reports, not from single teaching articles.

1. Before citing a platform experiment, write down three things: who, what was measured, and when. Basis: all of Prihar's results are performance during the homework, with unfinished work scored 0; Sales has only finished or not; Roschelle used an end-of-year test, but for a whole package of interventions. How: in your notes, turn video beats text, d = 0.123 into K-12 math homework: support inside the homework made as video versus as text, within-homework performance d = 0.123, no delayed test. Then decide whether it can support the way you write.
2. Before citing, find the abstract's number in the tables. Basis: seven sources have an abstract or title that claims more than the body (the comparison table is in the appendix). How: when an abstract says significant improvement, most, or an order of magnitude smaller, turn to the results table and find which group, which outcome, and what sample size that number refers to.
3. Write each checklist item as a sentence that can be judged, and note where it came from. Basis: the items dropped from Lee's rubric clustered around undefined words like unreasonable and sufficient; 11 of the 14 rubrics in Yuan and Recker had no detailed scoring guidance. How: change examples should be complete to an example needs conditions, reasoning or calculation, a result, and what the result means; if any is missing, send it back, and note beside it which mistake led to the item and whether it has research support.
4. Don't describe passing the checklist as teaching well. Basis: none of Lee's nine dimensions had a significant direct path to pass rates; none of the reviews of design-based research compares effectiveness. How: conclude a review report with meets items A, B, and C of the specification, not teaching quality is acceptable. This site's specification already says a green light from the checker scripts can't replace teaching acceptance; the specification makes an independent review report a condition for acceptance (without a report, an article only counts as awaiting acceptance), and the review doesn't measure what readers learn either.
5. When revising repeatedly, record what changed and why in each round. Basis: about 40% of Zheng's 162 studies didn't explain their revisions, and only a quarter of Tinoca's 163 reported revisions for each cycle; Brase's case measured outcomes but didn't use them to revise. How: each time you change a writing rule, note beside the clause the date, which article had what problem, and which sentence changed.
6. Occasionally have two reviewers review the same article and compare verdicts. Basis: a single rater is exactly what Lee lists as a limitation; research cited by Yuan and Recker says scoring guidance can improve reliability. How: for one in every ten articles, use two reviewers, and count how many must fix items both listed and how many only one listed. No study has directly tested whether it helps with teaching articles.

Three things to do tonight

1. Open the education study you cite most often and find its results table. Write down the sample, outcome variable, and time of measurement behind the number you cite. Compare it with how your article puts it.
2. Take your own writing checklist and pick three items containing words like appropriate, enough, or clear. Rewrite each as a sentence that two people would judge the same way after reading it.
3. Beside each checklist item, mark one label: R (research support; name the study), M (added after a particular mistake), or P (personal preference). When you're done, count how many items carry each label.

Limits of this evidence

• None of the sources measures readers of public teaching articles. The platform experiments are K-12 math homework and MOOCs, the rubrics are college online courses and open materials, and the design-based research reviews cover schools and teacher education.
• The platform experiments concentrate on one platform. Three of the five come from ASSISTments, and the Prihar and Sales papers share three authors; the Roschelle study has a different author group.
• Outcome measures are narrow. Prihar's are all within-homework performance, with unfinished work scored 0; Sales has only whether the homework was finished; Chen's learning measure hit a ceiling and didn't test transfer; Kizilcec's main outcome is completion; Lee's is pass rate. Only Roschelle used an end-of-year test outside the platform.
• Multiple comparisons generally went uncorrected. Prihar had 19 research questions, Chen several videos and several measures, Kizilcec several subgroups; none mentions a correction. Sales is the only one that demonstrates correction, and after correction few significant results remain.
• Selected subsamples and post-hoc subgroups. Chen's significant results come from selected subsamples; Prihar's only those who finished is a subgroup formed after random assignment (different thresholds mean different numbers of finishers, so the two groups are no longer a randomized comparison); Kizilcec's courses with a gap were defined after the fact in the first year, which the authors themselves point out.
• Only some experiments are reported. Chen's course ran seven experiments and reported two; Prihar analyzed only 50 of 103 deployments.
• The rubric evidence comes from one team. The Yuan and Recker paper and the Lee et al. paper share authors; Yuan and Recker searched only through April 2014.
• All three design-based research sources review research reports, and Brase points out that a single report is not the whole project. Several chi-square values reported in Zheng's body text don't match the significance levels marked beside them. I recomputed six of them from the counts in the tables, and only the increase in distance-education settings was significant at the .05 level; the drop the paper reports as significant, in the share explaining how revisions were made, comes out at p ≈ .09 by my calculation. The paper doesn't say which test it used.
• Numbers I calculated myself: Kizilcec's 8.8%; the recomputed chi-square values for Zheng; the totals of experiments and students across Prihar's research questions; the indicator-count statistics for Yuan and Recker, which were computed from 17 entries rather than 14 rubrics (see the appendix).
• My inferences: carrying these conclusions over to a long web article; the three points in Putting it together; that design-based research can be called neither more effective nor less effective; that what this site most lacks is an outcome measure; the six items in What can be done and the Three things to do tonight.
• Known to exist, but no legitimately public full text read: Jaggars and Xu (2016) on aspects of online course design and grades, and Anderson and Shattuck (2012) on a decade of design-based research. Neither has its content cited, and neither is listed in the references.
• Secondhand citations: the claim cited by Chen et al. that most online A/B tests show no effect (Kohavi and Longbotham 2015), the research on scoring guidance and reliability cited by Yuan and Recker, and the Brahm (2017) case relayed by Brase. I didn't read the original studies.

Appendix: design and numbers for each study

Prihar, Syed, Ostrow, Shaw, Sales, Heffernan (2022), EDM 2022 proceedings, pp. 27–38. E-TRIALS on ASSISTments, with students randomized individually; K-12 math skill builder assignments (problems on the same skill in a row, by default three correct in a row counts as completion). 103 experiments deployed from February 2019, 50 included; 30,408 students and 50,752 participation records. I added up the experiment counts and student counts across the 19 research questions separately and got 87 and 61,640, which means the same experiments and students enter several questions. Distribution of dependent variables: problems to mastery 44%, within-assignment post-test 44%, learning gain 7%, assignment accuracy 3%, completion 1%. Statistics: Cohen's d plus OLS (experiment fixed effects and student, class, teacher, and district characteristics). Completion thresholds (overall d / d among finishers only): 2 versus 3 in a row, 0.142 (p = .004) / 0.036 (p = .745); 2 versus 4, 0.139 (.007) / 0.006; 2 versus 5, 0.18 (.015) / 0.055; 3 versus 4, 3 versus 5, and 4 versus 5 were all non-significant overall. Support versus only the answer: video of smaller steps 0.267 (1,033 students), video worked example 0.237 (366 students, p = .029), smaller steps 0.176 (2,010 students), hints 0.155 (1,192 students, p = .003), worked example 0.237 (765 students), full solution to the current problem −0.061 (85 students, p = .576). Splitting students into high and low groups by prior accuracy, three of the four kinds of support had larger point estimates in the low group, but the paper ran no interaction test, and the two groups' confidence intervals overlap heavily. Other results: video versus text 0.123 (2,492 students, p = .002); motivational messages −0.072 (12,243 students, p < .001); adaptivity 0.054 (7,754 students, p = .015); feedback on common wrong answers −0.041 (7,046 students, p = .055, not discussed in the body); emotional messages −0.013 (p = .807); giving choices 0.014 (12,789 students, p = .472); fill-in versus multiple choice 0.085 (4,057 students, p = .009). Breaking video versus text down by kind of support, the three d values are 0.125, 0.121, and 0.125, and only smaller steps was significant (p = .017). The authors take this to mean that different kinds of support benefit differently from being made into video, but the three point estimates are nearly identical; the difference is in sample size. Confidence interval endpoints appear only in figures.

Sales, Prihar, Gagnon-Bartsch, Heffernan (2023), arXiv:2306.06273v1, preprint. 84 A/B tests split into 383 pairwise comparisons; after removing those with zero variance, those whose assignment probability did not appear to be 1/2, and those with fewer than 56 students in any condition, 227 pairs from 68 tests remained, with 38,035 students. The smallest pair had 16 students. Records of 193,218 students who weren't in the experiments trained four neural networks; the combined model predicting whether an assignment was finished had AUC 0.770 and R² 0.184. Number of significant results among the 227 pairs (uncorrected / Benjamini-Hochberg / Benjamini-Yekutieli): t-test 38 / 3 / 2; adjustment using only within-experiment data 39 / 11 / 2; adjustment using only predictions 41 / 8 / 2; predictions plus covariates 41 / 10 / 2. 4,318 subgroup estimates: t-test 375 / 0 / 0; predictions plus covariates 421 / 25 / 8. In the subgroup analysis, adjusting only with predictions did worse than the t-test in about 11% of cases, and predictions plus covariates did worse than within-experiment adjustment in about 13%; every case where adjusting only with predictions was more than 10% worse than the t-test was in a subgroup of fewer than 80 students. The authors also guessed gender from names (a process they call "faulty and inexact") to test whether the method still helps when the historical data and the experimental sample come from different populations: in most cases it still improved precision.

Chen, Chudzicki, Palumbo, Alexandron, Choi, Zhou, Pritchard (2016), Research and Practice in Technology Enhanced Learning 11:9. The 2014 run of 8.MReVx, a mechanics review course on edX. Experiment 1: in the last three units, three groups rotated through drag-and-drop deliberate practice, multiple-choice deliberate practice, and traditional end-of-chapter problems, followed by the same quiz; the analysis included only students who completed at least 70% of the homework and quizzes (205 to 280 per unit). Three-group ANOVA on each unit's quiz: p = .23, .52, .22; combined, the multiple-choice version was 4.70 percentage points below traditional problems (p = .026), drag-and-drop versus multiple choice p = .27. On later problems within sequences, restricted to the subsample who first answered wrong, first attempts were significant in two of five sequences (+0.73, p = .007; +0.75, p = .02). Experiment 2: two videos each in units 8 and 9, with groups A and B crossing over between the new and old filming styles. Final-attempt accuracy: video 1, new 74% versus old 66% (p = .61); video 2, new 93% versus 83% (p = .04); video 3, new 82% versus 84% (p = .8); video 4, new 84% versus 98% (p = .02). Average viewing time as a share of video length: the old versions of videos 1 and 2 were longer (p = .02, .05), and the new version of video 3 was longer (p = .006). The share with long viewing was significant only for video 3 (45% versus 27%, p = .02). The two kinds of video were filmed in different years, had different lengths, and had only one lecturer.

Roschelle, Feng, Murphy, Mason (2016), AERA Open 2(4); correction in AERA Open 3(2), 2017. School-level cluster randomization with a delayed-treatment design, assigned after matching. 46 schools recruited, 43 analyzed; student-level attrition 10.1%, differential attrition 5.6% (more attrition in the control group, 178 versus 129); analysis sample 1,621 in treatment and 1,229 in control. 92.6% white, 38.7% eligible for free or reduced-price lunch. Two-level HLM, with covariates for prior math and reading, lunch eligibility, special-education status, school prior mean, and matching indicators. Main effect 8.84 points (t(20) = 2.992, p = .007), g = 0.18; with 122 transfer students removed, 8.57 points, g = 0.17. Students used the platform an average of 14 hours and 967 problems over the school year; the authors had expected 18 to 24 hours. Interaction of prior achievement and treatment p = .015; below-median group: control 650.94, treatment 665.29, difference 14.35, g = 0.29; above-median group: 711.95 versus 717.80, difference 5.84, g = 0.12 (after the correction; the original table printed 50.10 in both cells, and another place in the body gave 14.35 as 13.35). The authors' other way of stating 0.18: it moves a student at the 50th percentile to the 58th. Limitations: a state where everyone has a laptop, a more rural and homogeneous population, measurement only after teachers had used the platform for a year, no way to separate mechanisms, and practice problem sets in the treatment group that the control group didn't have.

Kizilcec, Reich, Yeomans, Dann, Brunskill, Lopez, Turkay, Williams, Tingley (2020), PNAS 117(26). Four waves, each preregistered separately; intention-to-treat analysis, regardless of whether students actually wrote anything. Total registration across the 247 courses was 3,546,206; 312,221 reached random assignment; the focal sample was 269,169. In the first year, 14% of those who saw the planning intervention wrote nothing at all. Long-term planning (committed and fluent in English): earlier +3.9 percentage points (2,053 people); this study +0.19 in the first year (95% CI −0.69 to 1.07) and −0.23 in the second. MCII (individualist countries): earlier +1.8 and +3.9 (4,628 and 3,248 people); this study +0.25. Social accountability +0.89 (p = .115). Values connection (less developed countries, all courses): +0.14 and −0.04. Students from less developed countries in courses with a global gap: +2.79 in the first year (p < .001, 5,974 people) and +2.74 in the second (p = .026); in courses without a gap: −1.62 (p = .004) and −1.71 (p = .031). The gap was defined after the fact in the first year. A model predicting whether a course would have a gap was 54.3% accurate (95% CI 44.1 to 64.4), versus 50.0% for random guessing; among 79 courses that ran again, only 60.8% had the same gap status as the previous run. The estimated completion rate under a personalized assignment policy was 13.38%, versus 12.81% with no intervention, a difference that was not statistically significant. In the focal sample, 71.6% to 77.7% had a bachelor's degree, and the average age was 32 to 35.

Koedinger, Kim, Jia, McLaughlin, Bier (2015), Learning at Scale 2015. A psychology MOOC offered by Georgia Tech on Coursera, with CMU's interactive materials embedded; 27,720 registered and 1,154 took the final. In the causal model (Tetrad), the standardized coefficient from interactive practice to total weekly quiz score was 0.44, about 0.065 each for watching videos and reading pages, 0.65 from quizzes to the final, and 0.25 from the pretest to quizzes; model fit chi-square df = 7, p = 0.39. Share with excellent final exam performance: only more practice 16%, only more video 15%, more of both 31%. For the other numbers, see the appendix of Part 9.

Yuan and Recker (2015), IRRODL 16(5). Searched four databases including ERIC through April 2014 and found 200-plus articles; inclusion criteria were that the rubric evaluates quality, evaluates the open materials themselves, is a concrete rubric, comes with an explanation, and was published after 2000 (the paper says four criteria but lists five). 14 rubrics: 5 had both dimensions and indicators; rating scale binary for 2, four-point for 2, five-point for 5, unspecified for 5; detailed scoring guidance for 3; reported results of empirical testing for 6; revised several times for 3; general-purpose 7, specific to one website 5, specific to one subject 2; suitable for automated evaluation 2. Indicators per rubric: mean 15.53, median 15 (I recomputed these from the appendix; these statistics treat four versions of one rubric as four entries, 17 entries in total).

Lee, Recker, Yuan (2020), Online Learning 24(1). Literature review: 10 of 31 rubrics kept by three criteria; QM has 42 weighted items, with nearly 30% of the weight on learning objectives and assessment and 11% on interaction and collaboration. Field test: the AS rubric, 51 binary items, 202 courses (173 undergraduate, 29 graduate), one rater per course; initial KR-20 = .82, after dropping 8 items, 43 items with KR-20 = .87; exploratory factor analysis with nine factors explaining 73% of the variance, then 4 more items dropped, leaving 39. Path analysis on 121 courses: initial model CFI = .37; after removing non-significant paths, χ²(6) = 14.26, p < .05, CFI = .91, RMSEA = .11. Learner Engagement and Interaction → student-content interaction β = .286, student-student .333, student-instructor .365; student-content interaction → pass rate .358; course facilitation → pass rate .155 (not significant). Indirect effects were not tested. Mean pass rate 0.90 (SD 0.12); 169 of 5,240 students withdrew. Limitations: a single institution, a single rater, binary scoring, and mixed disciplines.

Zheng (2015), Journal of Computers in Education 2(4). 219 SSCI journals; the search found 479 articles, screened by six criteria down to 162 (45 from 2004–2008, 117 from 2009–2013); the appendix list has two duplicate titles. Number of iterations: one 81, two 52, three 21, four 4, five 2, more than five 2. Revised 120 (74.07%), not revised 42; explained how revised 98 (60.49%), not explained 64. Iteration duration: one month 31, six months 38, one year 42, two years 25, three years 12, more than three years 7, unstated 7. Type of intervention: technology interventions 86 (53.09%). Research methods: qualitative 118, mixed qualitative and quantitative 44, quantitative 0. Data sources (a paper can have several, 392 entries in total): process data 54, outcome data 82, questionnaires, interviews, and notes 254. Outcomes measured: cognitive 78, attitude 7, motor skills 7, multiple 46, other 24. Most chi-square values reported in the paper are below the .05 critical value for one degree of freedom yet are marked significant. I ran 2×2 chi-square tests (without correction) from the counts in the tables: explaining revision methods, 71.11% versus 56.41%, χ² = 2.94, p = .086; distance-education settings, χ² = 4.43, p = .035. The paper doesn't say clearly whether coding was based on full texts or abstracts. I read the web version, with tables from the publisher's table pages, and didn't check against the PDF layout.

Tinoca, Piedade, Santos, Pedro, Gomes (2022), Education Sciences 12(6):410. 396 records from Scopus plus 260 from Web of Science, 442 after deduplication, screened by title, abstract, and keywords down to 163; five coders, agreement above 85%. Number of iterations: unstated 65, two 32, three 28, four 12, five 3, more than five 23. Revisions: reported 40, said revised without details 50 (the body says 51), another category 12, unstated 61. Iteration duration: unstated 52, up to six months 44, six months to a year 28, one to two years 21, two to three years 13, more than three years 5. Research methods: qualitative 80, mixed qualitative and quantitative 75, quantitative 8. Data sources (each paper coded in one category only): questionnaires, interviews, and similar 130, process data 18, outcome data 12. Of the 60 papers that stated methodological limitations, 30 named context specificity. The list of 163 papers isn't attached to the paper; one place in the conclusion gives the count as 162.

Brase (2024), EDeR – Educational Design Research 8(1), Article 62. 57 articles from a single journal, 2017–2022; 30 excluded as editorials, practice examples, book reviews, responses, and theory papers not tied to a single case, leaving 27, coded by the author alone in MAXQDA. Of the 27, 1 reported no intervention; design principles appeared in more than half the articles. Number of articles naming each of the eight key arguments: robust across settings 6, collaboration and dialogue 5, iterative refinement 4, data triangulation 3, integrating stakeholder perspectives 2, and 1 each for the other three. Articles mentioning methodological standards or explaining individual methods in detail: 4 named. Limitations the author lists: not a systematic review, analysis by one person, provisional categories, a European and German bias, a journal that itself specializes in design-based research, and published articles not being the same as whole projects.

Appendix: abstract or title versus body text.

Two more sources overstate to a lesser degree: Yuan and Recker's abstract says the rubrics have had "little accompanying evaluation", while the table shows 6 of 14 reported empirical testing; Brase's abstract says authors "often" justify their work with variation, iteration, cooperation, and triangulation, while the body names 3 to 6 of the 27 articles for each.

References

For each item, I read the whole of the version I obtained and kept an extraction record; parts I couldn't read are noted in parentheses.

• Prihar, Syed, Ostrow, Shaw, Sales & Heffernan (2022), Exploring Common Trends in Online Educational Experiments (final version in the EDM 2022 proceedings, doi:10.5281/zenodo.6853041; confidence intervals appear only in figures, and endpoints were read by eye; interface screenshots not examined).
• Sales, Prihar, Gagnon-Bartsch & Heffernan (2023), Using Auxiliary Data to Boost Precision in the Analysis of A/B Tests on an Online Educational Platform (arXiv preprint v1; formulas not checked one by one).
• Chen, Chudzicki, Palumbo, Alexandron, Choi, Zhou & Pritchard (2016), Researching for better instructional methods using AB experiments in MOOCs: results and challenges (published version, read as the full-text XML in PMC; figures available only as captions).
• Roschelle, Feng, Murphy & Mason (2016), Online Mathematics Homework Increases Student Achievement (published version), together with the correction (2017).
• Kizilcec, Reich, Yeomans et al. (2020), Scaling up behavioral science interventions in online education (published version and SI Appendix, from the PMC open-access dataset; the two SI tables of baseline model coefficients were read only as text files).
• Koedinger, Kim, Jia, McLaughlin & Bier (2015), Learning is Not a Spectator Sport: Doing is Better than Watching for Learning from a MOOC (published version, ACM open access; numbers in the figures could not be read).
• Yuan & Recker (2015), Not All Rubrics Are Equal: A Review of Rubrics for Evaluating the Quality of Open Educational Resources (published version).
• Lee, Recker & Yuan (2020), The Validity and Instructional Value of a Rubric for Evaluating Online Course Quality: An Empirical Study (published version).
• Zheng (2015), A systematic literature review of design-based research from 2004 to 2013 (published version and table pages, free to read on the publisher's website; not checked against the PDF layout).
• Tinoca, Piedade, Santos, Pedro & Gomes (2022), Design-Based Research in the Educational Field: A Systematic Literature Review (published version).
• Brase (2024), Knowledge generation between design, data and theory: Argumentation in design-based research (published version and appendix).
• Articles on this site: What the evidence for learning by doing measures, Why most online self-learners don't finish, Writing questions and wording in a teaching article.