How Much Guidance Should a Project in a Teaching Article Give? Problem-First, Discovery, Scaffolding, and Project-Based Learning

🌏 中文版

This is part 11 of the series "Writing Articles That Teach," and the last post in its second section, "what to ask readers to do." Part 8 asked what the evidence for learning by doing measures, and part 9 covered questions and feedback in the text. Whether readers actually answer them is the subject of a later post. This post deals with something much larger than a single self-check: a project that takes several hours. It asks three questions. How much should an article explain before the reader starts? What kinds of guidance can an article give, and should they be withdrawn as the reader progresses? And of everything the teacher and classmates do in classroom studies, how much survives when it moves into an article?

The trigger was the project sections of this site's seven-week Harvard CS50 AI course guides. In a rewrite on 2026-10-13, the project sections stopped pasting a full reference implementation up front. Each now gives the task, the distribution code, the approach, pseudocode, where it goes wrong, and how to check your work, with the full implementation collapsed at the end. The reason was the course's academic honesty policy: it counts looking at someone else's solution before submitting your own as not reasonable, and counts discussing solutions with diagrams or pseudocode, but not actual code, as reasonable. That was an integrity decision. It left a learning question behind: is it better to give the approach and then let the reader write the code, or to let the reader get stuck first and then give the approach? How far the code itself should go was already checked against the same spec in part 3; this post looks only at ordering and guidance.

How the sources were chosen, and their biases. The source list came from a literature scan followed by a selection for close reading, in four strands: problem solving before instruction (one meta-analysis, one set of experiments with the opposite result, one online experiment); discovery and guidance (two meta-analyses, one position paper); scaffolding (one meta-analysis); and project-based learning (one meta-analysis, one literature review). I also read the article that defines cognitive apprenticeship, to check which kinds of guidance an article can provide. The biases: English-language literature, studies with full text available, mostly school and college students, mostly mathematics and natural science. I read the full main text of every source in the references and kept an extraction record; the parts I did not read (reference lists, appendices, the two theory chapters of Ashman's dissertation, and so on) are noted in parentheses in the references. For three works I could not find a legal full text from the authors or publisher, so I only note that they exist and cite no numbers: the 2017 review by Loibl, Roll, and Rummel on when problem solving before instruction works, the 2019 meta-analysis of project-based learning by Chen and Yang, and the 2020 journal article by Ashman, Kalyuga, and Sweller (for the same experiments I read the first author's public doctoral dissertation instead).

The short version

Problem solving before instruction

What the meta-analysis compares

In problem solving followed by instruction (shortened here to "problem first"), students who have not yet learned a concept get a problem that needs the concept to solve well, and try it on their own; only afterward does the teacher present the canonical solution. Its proponent Manu Kapur calls the carefully designed version productive failure, meaning that failing to solve is itself useful. The comparison group does the reverse: instruction first, then problems.

Tanmay Sinha and Manu Kapur's 2021 meta-analysis covers 53 studies and 166 comparisons. Across conceptual knowledge and transfer, the problem-first groups scored higher by g = 0.36 on average (g is the difference between groups divided by the standard deviation; 0.36 is roughly a third of a standard deviation). No difference was detected on procedural knowledge, meaning applying a learned procedure to problems of the same kind. Taking only the randomized experiments, where the two groups differ only in order, the point estimate is 0.25; the difference between randomized and quasi-experimental comparisons was not significant.

Three things affect how to read this number:

• The search. It used only Google Scholar and only took studies citing one of five productive failure papers; all five include this paper's second author (one is co-authored), plus 20 studies taken from the 2017 review by Loibl and colleagues.
• All classrooms and labs. Grades 6–10 are the largest group, the content is mostly mathematics, physics, and medicine, and nearly nine in ten comparisons measured only one of conceptual knowledge or transfer.
• The comparison group's problem phase is mostly practice. In the instruction-first groups, the problems are for practicing the method just taught, not for inventing one. The authors list this as a limitation: the two orders may have different learning goals and may not be directly comparable.

What the effective versions look like

Sinha and Kapur coded the problem-first designs against seven criteria: the problem allows multiple solutions, there is evidence that students actually generated multiple solutions, the problem has affective draw (story problems, contrasting cases, and simulations all count), problem solving is done in groups, instruction builds on student solutions, and the social environment of the problem-solving and instruction phases.

Taken one at a time, comparisons that met each criterion had larger effects, but only four of the seven differences were significant or close to it. The authors ranked the criteria with two methods; only two came out near the top in both and passed the authors' own weight threshold: evidence that students generated multiple solutions, and group problem solving. "Instruction builds on student solutions" had the most significant subgroup difference on its own (p = .02), but once placed in a multiple regression no independent contribution could be seen; the paper cannot separate its effect from the other criteria. The model chosen by stepwise selection explains only 13.51% of the variance in effect sizes.

Group work also has a result in the other direction: in short interventions lasting a few hours, effect sizes grew more when students worked individually rather than in groups. The authors' wording is that, in short interventions, individual work better predicts the observed effect sizes; this is a correlation, not a recommendation the authors tested.

All of this is correlation between studies. How fully the seven criteria were implemented is also tied to study design: quasi-experiments implemented far more of them on average than randomized experiments, and interventions lasting several days more than those lasting a few hours. The authors' own advice is not to force all seven: "it is important to consider what is feasible."

Another frequently cited review on this question is the 2017 one by Katharina Loibl and colleagues. It exists, but I could not find a legal full text from the authors or publisher, so I cite none of its content.

Who sees no benefit

In Sinha and Kapur's subgroup analysis, two subgroups have point estimates leaning toward instruction first: grades 2–5 (25 comparisons) and domain-general skills (control of variables, water-jar problems, and the like; 8 comparisons). Both point estimates are small, and what is significant is "this subgroup is lower than the others," not "instruction first is significantly better in this subgroup." The safer statement: in these two subgroups, no problem-first advantage is visible.

The first-hand experiments I read that directly push against productive failure are Greg Ashman's five randomized experiments. They are not the only ones. Ashman's dissertation itself cites several studies unfavorable to problem first: Hsu and colleagues (2015) compared placing worked examples before or after problems, and examples first did better; Fyfe and colleagues (2014) and others added an exploration phase before instruction and found either no advantage or an effect in the opposite direction (the dissertation does not say which for each). I have not read any of these. Ashman, Kalyuga, and Sweller's 2020 journal article reports two of the five and has no legal public full text; I read Ashman's 2022 doctoral dissertation (UNSW institutional repository, CC BY), and the numbers and experiment numbering below follow the dissertation.

• The participants were year 5 and 6 students (about ten to eleven years old) at a private school in Australia, learning to calculate the energy efficiency of light bulbs. Both groups did the same things in a different order: one group did 15 minutes of individual paper problem solving and then heard the lesson, the other heard the lesson first and then solved problems.
• Differences showed up in experiments 4 and 5. The instruction-first group did better on similar problems, by about 0.56 SD in both experiments; on transfer problems one experiment was significant and the other was p = .06, in the same direction. The first three experiments detected no difference, which the author attributes to flaws in procedure and testing. None of the results in the five experiments significantly favored problem first (in experiments 1 and 2 the problem-first group's means were slightly higher, not significantly).
• The author attributes the result to high element interactivity (many steps that have to be handled at once). But element interactivity was not manipulated within an experiment; the author classed both experiments as "high." The author's own word for the result is "suggestive," and the dissertation says the results cannot be used to support or reject problem first for content low in element interactivity.

Checked against Sinha and Kapur's seven criteria (my own comparison), the problem-first version in these experiments is a low-fidelity design: individual, 15 minutes, no feedback, no groups, and the lesson did not build on students' solutions on the spot. The author says the lesson slides addressed a common misconception known in advance, and that this meets Kapur's design requirements. Supporters of problem first can answer with "low fidelity." Looked at the other way, this kind of version is probably about all one person reading an article can get (this is my inference).

One experiment moved online

Marci DeCaro and colleagues' 2023 experiment is the closest to a remote setting I found. In one section of an introductory college physics course, students were randomly assigned to the two orders, with 78 analyzed (the number originally assigned was not reported), in a live video class on Blackboard lasting 50 minutes. The activity was a set of contrasting cases: three figures and ten points, where students matched gravitational field vectors to the points and then devised their own formula for magnitude and direction. The explore-first group scored lower on the activity and higher on the posttest overall (76% vs. 66%).

This experiment is still three layers away from one person reading an article: the activity was done in small groups in video breakout rooms; the teacher was present and students could ask questions at any time; and a whole-class review followed. The posttest was done outside class, untimed, and graded for effort, and the conceptual subscale's internal consistency was only 0.30. In their limitations the authors write that whether this can be done, and whether it works, in an asynchronous online course needs separate research; their suggestion is to have students submit the exploration activity on the learning platform before the lecture video is unlocked.

Unassisted discovery and guided discovery

Two meta-analyses in one paper

Louis Alfieri and colleagues' 2011 paper (I read the author manuscript in the repository of co-author Tenenbaum's current institution, plus the supplementary tables the publisher made public) contains two meta-analyses.

The first compares unassisted discovery (no guidance and no feedback during learning: manipulating materials alone, trial and error, working practice problems, inventing strategies) with explicit instruction. Across 108 studies, explicit instruction did better, d = −0.38. There are some exceptions: no difference was detected in the computer-skills domain (72 comparisons), nor for discovery through computer simulation. Among the forms of explicit instruction used for comparison, the largest gap was for worked examples (problems with every solution step written out).

The second compares enhanced discovery with other teaching methods; across 56 studies, enhanced discovery did better, d = 0.30. Broken down by type of guidance, the differences are large:

• Guided discovery with scaffolding or feedback at each stage did best (0.50).
• Asking learners to explain their own ideas came next (0.36), with no difference detected from the former.
• Asking learners only to generate rules, strategies, or answers did worse than other methods (−0.15).
• No difference was detected between enhanced discovery and worked examples.

A few points need to travel with this paper: the comparison-level analysis counts the same participants in several comparisons; study quality and randomization were not coded; every moderator result is a cross-study comparison; the included studies go back as far as 1941; and the paper has no limitations section. The last sentence of the abstract says that "feedback, worked examples, scaffolding, and elicited explanations" help learners, which merges results from the two analyses into one sentence (details under "Where the sources disagree").

The authors' own teaching advice is cautious: "Although direct teaching is better than unassisted discovery, providing learners with worked examples or timely feedback is preferable."

Which guidance to add to inquiry learning

Ard Lazonder and Ruth Harmsen's 2016 meta-analysis asks a different question. All students are doing inquiry (running experiments or simulations and inferring principles from data); one group gets an added type of guidance and the other does not. It covers 72 studies, participants aged 5 to 22, almost all in natural science. The groups with guidance had better learning outcomes, d = 0.50.

The paper sorts guidance into six types by specificity: from only constraining the task or showing progress, through prompting what to do and suggesting how to do it, to doing the harder parts for the student (scaffolds), and finally explaining directly how to do it (explanations). On learning outcomes, no difference was detected among the six types, but two of them have only two studies each, with confidence intervals crossing zero. On what students produced during inquiry, more specific guidance had larger effects, but the two types with the largest effects have only one study each, and the authors say no definite conclusion can be drawn.

The authors' explanation for "a difference during the process, none on learning outcomes" is that specific guidance improves immediate performance and the gap disappears quickly once the guidance is withdrawn. That is the authors' interpretation, not a separately measured result. The paper has no delayed tests and does not analyze whether guidance was withdrawn.

A position paper

Paul Kirschner, John Sweller, and Richard Clark's 2006 article is often cited as the source for "discovery learning doesn't work." It is a position paper with no data of its own, and it does not explain how the cited studies were chosen. It treats discovery, problem-based, inquiry, experiential, and constructivist learning as one pedagogically equivalent group. Its central argument: novices have limited working memory when handling new information, the load of searching for a solution on their own contributes nothing to learning, so novices should be told directly what to do and how to do it.

It also states a boundary for this claim: the benefit of worked examples disappears as learners gain expertise and then reverses so that problem solving does better: "That guidance can be relaxed only with increased expertise." The paper predates the productive failure studies and never discusses the order of problem solving and instruction.

The discussion section of Lazonder and Harmsen answers it directly: "strong guidance" is not the same as "specific guidance," and the frequency and duration of guidance may matter more. The two papers take opposite positions and cannot be grouped as one camp.

Scaffolding, and whether to fade it

Scaffolding originally meant the temporary support a teacher gives while a student does a task they cannot yet do alone, withdrawn step by step as the student improves (fading).

Brian Belland and colleagues' 2017 meta-analysis covers computer-based scaffolding: conceptual prompts, question prompts, hint buttons, and feedback from intelligent tutoring systems given by software while students work on an ill-structured STEM problem. The comparison group follows the same problem-centered instruction without scaffolds. Across 144 studies and 333 outcomes, g = 0.46; taking only the 181 outcomes from individually randomized designs, it is 0.40.

The comparison between fading and not fading needs careful reading:

• Only 12 outcomes moved only toward less support (with those that both added and faded, 16.5% in total). Their point estimate is higher than for no change, but the interval is wide; no difference between groups was detected, and the paper only reports p > .05. The authors say the intervals would narrow with more studies and a difference might emerge.
• Some of the 216 outcomes coded "no change" just reflect papers that did not describe it; the authors say in their limitations that such codes "indicate a lack of description."
• The same authors' 2015 pilot meta-analysis went the other way (not fading did better).
• This is a comparison between studies, and the ones with changing scaffolds were mostly intelligent tutoring systems, so system type and fading are mixed together. The paper also did not measure whether students could work independently once the scaffolds were gone.

So what this paper supports is "no difference detected," not "fading makes no difference." Data pointing the other way come from a classroom study (McNeill and colleagues, 2006) as summarized in Barbara Condliffe and colleagues' literature review: in randomly selected classes, prompts for writing scientific explanations gradually lost detail, while the other classes kept the same prompts, and posttests suggested that the classes with faded prompts were better at writing explanations on their own. I have not read the original study.

One more point matters for articles: Belland and colleagues' inclusion criteria excluded interventions given before learners encountered the problem. Whether an article's approach section, written before the reader starts, counts as scaffolding under this analysis's definition is something the paper does not address.

Project-based learning

Project-based learning makes a project with a finished product the backbone of instruction, so that learning happens while doing the project rather than finishing with a capstone assignment after learning.

Lu Zhang and Yan Ma's 2023 meta-analysis covers 66 studies and 190 effect sizes, comparing project-based learning with "traditional teaching" (undefined in the paper). The pooled effect on academic achievement is SMD = 0.65 (40 effect sizes). I do not recommend treating this as a reliable estimate, and the reasons are all in the paper: no screening for study quality or design; multiple effect sizes from the same study treated as independent; a significant Begg test for publication bias, while the authors conclude from fail-safe N that there is no significant bias; individual study data not published, so readers cannot recompute; and many self-contradicting numbers in the tables, with none of the confidence intervals in the country-and-region block matching their point estimates. The school-level breakdown has one clue more relevant to this site's readers: the effect at the university level is the smallest of the four levels (0.12), and it too can only serve as a clue.

The other meta-analysis on this topic is the 2019 one by Chen and Yang. It exists, but I could not find a legal full text from the authors or publisher, so I cite none of its numbers.

Condliffe and colleagues' 2017 literature review (an MDRC working paper, not a peer-reviewed journal article) pools no effect sizes; it goes through K–12 classroom studies one by one. Its overall verdict appears only in the abstract:

The working paper suggests that the evidence for PBL's effectiveness in improving students' outcomes is "promising but not proven."

The main text says that, apart from a few, most quantitative studies use designs that do not allow causal inference; even with a comparison group, differences may come from student characteristics before the intervention. The authors also say a review of published research may be biased toward positive findings. Some details useful for writing articles:

• There is no consensus on what must be present in a classroom for it to count as project-based learning. Collaboration is not an agreed-upon requirement: of the seven sets of design principles the authors reviewed, only two list it as essential.
• For English and language arts, the authors could not find a single outcome study.
• Teachers often lack the time to give high-quality feedback or to guide students through self-assessment and reflection.
• Out-of-school programs are explicitly outside the scope.

Neither source includes a single case of a learner doing a project alone. Condliffe and colleagues limited their scope to K–12 classrooms from the start, so their review cannot show that such studies do not exist, only that it did not include any.

What is left without a teacher

Every piece of evidence above had a teacher or experimenter present, and most had classmates too. To answer how much an article can do, I went through two checklists item by item: Sinha and Kapur's seven criteria, and the six methods of cognitive apprenticeship defined in the 1991 article by Allan Collins, John Seely Brown, and Ann Holum. Cognitive apprenticeship carries the master-apprentice relationship over to invisible cognitive skills: the master demonstrates, the apprentice tries, and the master supports from the side and then lets go step by step.

The article is a framework piece written for teachers, with no data of its own; the effect numbers it reports from other studies come with no description of comparison groups. In the definitions of all six methods, the subject is the teacher. The column "Can an article do it?" below is my inference from the definitions; the original does not discuss written materials.

The original says "the most important difference" runs both ways: the teacher's thinking must be made visible to students, and the students' thinking must be made visible to the teacher. An article can only do the first half.

The seven criteria come out the same way (my inference). A problem that allows multiple solutions and has affective draw is something an article writer can design. Whether the reader actually generated multiple solutions, the writer cannot see. Group work and the social environment of the two phases, an article does not have. As for instruction that builds on student solutions, an article can only write out common errors in advance; Ashman says the lesson in those experiments did exactly this, and in those experiments instruction first did better. Whether prewritten errors can stand in for building on student solutions has not been tested.

Collins and colleagues wrote one passage about reading and listening, the part of the article most directly relevant to self-learners:

Active listeners or readers, who test their understanding and pursue the issues that are raised in their minds, learn things that apprenticeship can never teach. To the degree that readers or listeners are passive, however, they will not learn as much as they would by apprenticeship, because apprenticeship forces them to use their knowledge.

The authors continue: "Moreover, few people learn to be active readers and listeners on their own, and that is where cognitive apprenticeship is critical." That is, most people do not teach themselves to read actively, and this is exactly where apprenticeship-style guidance is needed. These are claims, with no data attached. Read together, the two sentences say that whether an article's guidance gets used depends on the reader, and that the authors believe most readers need someone to lead them (the authors' view, not a measured result).

Finally, whether doing a project alone, with no teacher and no group, works: none of the studies read here tested it, so neither "self-learners benefit from projects" nor "they don't" is supported.

Checking this site's CS50 AI project sections

The spec for the project sections lives in the working directory (.work/cs50ai-rewrite/PROJECT-SECTION-SPEC.md, not under version control). Each project follows a fixed order: task, distribution code, approach, pseudocode, where it goes wrong, how to check your own work, and the full reference implementation (collapsed, with a warning about the honesty policy). Take week 1 as an example: the approach for the Knights project explains how to write "X said S" as two implications, the pseudocode gives the structure in prose, and the "where it goes wrong" part describes symptoms, such as nothing printed under Puzzle 3, and how to notice them.

Judged by the research above, this order is instruction first, then problem solving: the approach comes before the reader starts. And even if the approach section moved to the end, the order would still be instruction first. The problem-first studies compare the order of conceptual instruction and problem solving, and in the CS50 guides the lecture walkthrough sits as a whole before the project sections. The approach section is closer to Alfieri's axis, "whether discovery gets guidance," than to the ordering axis of the problem-first studies. Below is an item-by-item check against the version in the working directory on 2026-10-14; the rightmost column is this site's judgment. How far the code should go was checked in part 3 and is not repeated here.

The re-review reports also record a problem that undoes this structure: solutions exposed outside the collapsed blocks. Exactly one week had a project solution exposed outside the collapsed block, flagged as must-fix: week 6's changelog, outside the collapsed block, gave the numbers of the two attention heads the assignment asks students to submit. Two more cases were answers to lecture-walkthrough self-checks seen early (in week 0 a pruning diagram sat before its question; a cross-week synthesis worked out the answer to a week 6 question in advance), and the concerns raised for weeks 3 and 5 were judged by the reviewers not to be leaks. The design of approach first and solution collapsed assumes the reader really tries before opening the block, and an answer leaked outside the block removes that step (this is my inference). None of these review reports has reader data; this site has not measured whether readers actually try first or when they open the collapsed block.

As of 2026-10-14, the spec has not been changed because of this post.

Where the sources disagree

The following results point in different directions, or look contradictory while actually measuring different things. I list them side by side without taking sides.

Problem first or instruction first. Sinha and Kapur's meta-analysis leans toward problem first on average, and the grades 2–5 and domain-general skill subgroups show no such advantage. Ashman's experiments lean toward instruction first, and their participants were in years 5 and 6. The two explain the difference differently: Sinha and Kapur speculate that for young grades it is a matter of prior knowledge, metacognitive strategies, or design fidelity; Ashman attributes it to the high element interactivity of the content. In Ashman's experiments, age and content complexity cannot be separated, and the problem-first version had low fidelity, so neither explanation has yet been tested on its own.

Did Sinha and Kapur account for publication bias? The abstract says "after accounting for publication bias" the effect is g = 0.87, and the conclusion also speaks of accounting for significant evidence of publication bias. In the main text the funnel plot is symmetric and the Egger test is not significant, and the authors write that small-study bias "could be therefore ruled out"; 0.87 is an estimate back-calculated by p-curve from significant results only, and the main text does not explain why it is more than twice the 0.36 from all 166 comparisons. The abstract also says intervention length and quasi-experimental design "contributed," while in the main text neither subgroup difference is significant.

Does group work help? In Sinha and Kapur's overall data, group problem solving has larger effects; in the short interventions in the same data, individual work shows larger gains. DeCaro's effective example had groups exploring; Ashman's ineffective example had individuals working. Condliffe and colleagues say there is no consensus on collaboration in the design principles of project-based learning.

Is more specific guidance better? Kirschner and colleagues argue that novices need direct, strong guidance. Lazonder and Harmsen's data detected no difference by specificity of guidance on learning outcomes, and the authors conclude from this that strong guidance is not the same as specific guidance. In Alfieri's data, discovery that only asks learners to generate rules did worse than other methods, scaffolding and feedback did best, and no difference was detected between worked examples and enhanced discovery. The three compare different things: Alfieri compares discovery with other methods, Lazonder compares inquiry with and without a given type of guidance, and Kirschner has no data.

What guidance does for experts. Kirschner and colleagues' abstract says the advantage of guidance "begins to recede"; the main text says the worked-example effect first disappears and then reverses; the conclusion says that for students with considerable prior knowledge, strong guidance is "most often found to be equally effective." The three differ in strength, and all relay other people's studies.

Fading scaffolds or not. Belland and colleagues' meta-analysis of computer-based scaffolding detected no difference, with a higher point estimate for fading; the same authors' 2015 pilot analysis found not fading better; McNeill and colleagues (2006), as summarized by Condliffe and colleagues, found fading better. The settings differ: software scaffolds, ill-structured STEM problems, and a cross-study comparison, versus paper writing prompts in one classroom study.

What Alfieri's abstract says. The abstract says "feedback, worked examples, scaffolding, and elicited explanations" help learners. Of these, feedback and worked examples were comparison conditions on the explicit-instruction side of the first analysis, not support added to discovery; scaffolding and elicited explanation are the enhanced discovery types of the second analysis. The paper has no result that "discovery with worked examples does better."

Where DeCaro's benefit falls. The authors hypothesized that exploring first would help mainly with conceptual knowledge; the result was a non-significant interaction between condition and subscale, with the numerical gap actually larger on the procedural items. The abstract does not say that the result differed from the hypothesis. Sinha and Kapur's average result is no difference detected on procedural knowledge; DeCaro's procedural subscale shows a larger numerical gap, but it was not tested separately.

Regional differences in project-based learning. Zhang and Ma's abstract says effects were larger in Asia than in Western Europe and North America; none of the confidence intervals in their region table match the point estimates, and there is no test between groups, so the claim has no statistical basis that can be checked. The other meta-analysis of project-based learning (Chen and Yang, 2019) I could not read in full, so it cannot be used for comparison.

What you can do

The following are this site's judgments, each with its basis; "basis" names the results that led me to do this, not a study that tested the practice directly.

1. Don't write "research shows getting stuck first works better" or "project-based learning is more effective" in an article. If you cite it, name the comparison and the setting: classrooms, a teacher, usually groups. Basis: Sinha and Kapur's 166 comparisons and the studies in Zhang and Ma and in Condliffe were all in school classrooms; Ashman's experiments found the opposite.
2. When readers try first, always give feedback they can use on their own. Give small cases they can work by hand with answers, sample inputs with expected outputs, and which symptoms point to which mistakes. Basis: in Alfieri's data, discovery with no guidance and no feedback lost to explicit instruction, and guided discovery with feedback at each stage did best. This is a cross-study comparison.
3. Give the full solution, but place it after the reader has tried. Basis: in Alfieri's data, worked examples clearly beat unassisted discovery; in the problem-first studies, instruction is always given, just later.
4. For a problem readers try first, pick one that allows several approaches and where mistakes produce visible symptoms. Basis: of Sinha and Kapur's criteria, "the problem allows multiple solutions" was met in more than nine in ten of the 166 comparisons, and "evidence that students generated multiple solutions" had the more stable association with effect size. An article cannot see what the reader generated, so I chose "mistakes have visible symptoms" as a substitute the reader can see for themselves; this is my inference.
5. Don't treat prewritten common errors as a substitute for building instruction on student solutions. Keep writing such sections, but don't claim the problem-first research supports them. Basis: no independent contribution of building on student solutions could be seen in the multiple regression; Ashman says the lesson addressed a common error known in advance, and in those experiments instruction first did better.
6. Don't set a rule yet on whether guidance should shrink over a series. Basis: Belland's data detected no difference between fading and not fading, and the authors say power may be too low; McNeill's classroom study found fading better, and I have not read the original; Kirschner and colleagues' claim has no data. If you want to try, start by removing the pseudocode from the projects in later posts and keeping only the approach and the checks.

Almost every practice in this section carries a "depends on the reader's level" caveat: the young-grade and domain-general skill subgroups, the expertise reversal Kirschner and colleagues describe, and the age differences in Alfieri's data. The section on reader level in part 1 covered a meta-analysis of expertise reversal; a later post on prior knowledge will bring these caveats together.

Three things to do tonight

Pick a teaching article you wrote that has exercises or a project; it takes about an hour:

1. Find the biggest task in the article and, before its explanation of the solution, add a sentence: "If you want to try first, skip to the 'how to check' section, write your code, then come back for the approach." Then make sure the "how to check" section really works without reading the approach: it should contain at least one small case that can be worked by hand, with its answer. (This adds an option for readers rather than changing the order. The basis: in Alfieri's data, exploring first beat unassisted discovery only with feedback, so the precondition is that the "how to check" section works on its own.)
2. Take the pseudocode out of the solution explanation and ask yourself: if I typed this up as code, would I still have to make any decisions? If not, delete the most detailed layer and keep only what each step needs to achieve.
3. Search the whole article for answers stated outside the collapsed blocks (titles, figures, changelogs, the tldr); move any you find into a collapsed block or delete them.

Limitations of this evidence

• No study has directly tested one person reading a web article and doing a project. The research on problem first, discovery, scaffolding, and project-based learning was all in classrooms, labs, or live online classes with a teacher. Carrying any of it over to web course guides is my inference.
• The problem-first meta-analysis leans toward its proponents' own body of research. The search only took studies citing one of five papers by the second author; how fully the seven criteria were implemented is tied to study design and intervention length, and all of it is correlation between studies.
• The opposing first-hand evidence I read is one set of experiments. Ashman's experiments used one school, one topic, and students around ten years old, at the school where the first author teaches; I did not read the journal article itself.
• The online evidence is one section and one class session, with live video, small-group exploration, and an unproctored posttest.
• The discovery and guidance meta-analyses are cross-study comparisons. Alfieri has no coding of study quality and its comparison-level analysis is not independent; in Lazonder, several guidance types have only one or two studies, and there are no delayed tests.
• The evidence on fading scaffolds is thin. Only 12 outcomes in Belland only faded; McNeill and colleagues (2006) is relayed by Condliffe and colleagues, and I have not read the original.
• The project-based learning meta-analysis has quality problems, the other one I could not read in full, and the literature review is a working paper that does not describe its search or screening.
• The cognitive apprenticeship comparison is entirely my inference from the definitions. Collins and colleagues' article has no data of its own, and the authors themselves say the three domains in their examples (reading, writing, mathematical problem solving) rest on relatively little conceptual and factual knowledge, unlike knowledge-heavy technical content. The authors also say the model does not fit rote content.
• My inferences and judgments: that Ashman's problem-first version is closest to what an article can do; which cognitive apprenticeship methods an article can provide and which of the seven criteria remain; the analogy from Lazonder to pseudocode granularity; the collapsed implementation working as a worked example after the attempt; leaked answers removing the try-first step; the six items in "What you can do"; and the verdicts in the table on this site's spec. Each is marked in the text.
• Secondhand: McNeill and colleagues (2006, relayed by Condliffe and colleagues), Fyfe and colleagues (2014) and Hsu and colleagues (2015) (cited by Ashman), and Schoenfeld's remark and the reciprocal teaching numbers (relayed by Collins and colleagues); I have not read any of the originals.
• Known to exist, but no legal public full text found: Loibl, Roll, and Rummel (2017), Chen and Yang (2019), and the journal article by Ashman, Kalyuga, and Sweller (2020).
• Reading gaps: Sinha and Kapur's online supplement (per-study effect sizes) was not obtained; Belland's supplementary tables were not obtained, and the moderator numbers were read off the figures. The parts of Ashman's dissertation I did not read: chapters 1 and 2 (background on cognitive architecture and cognitive load theory, pp. 4–43), the references, appendices 1–3 and 5 (materials for experiments 1–3 and 5), the posttest items in appendix 4, and appendix 6 (the reading cloze task).
• This site has no reader data. Whether readers try first and when they open the collapsed block have not been measured.

Appendix: study designs and numbers

The main text keeps one or two numbers per item; the rest are here.

Sinha and Kapur (2021), Review of Educational Research 91(5). Three-level random-effects model. The search used only Google Scholar, taking literature citing any of Kapur 2008, Kapur and Bielaczyc 2012, Kapur 2014, 2015, and 2016 (1,212 records), plus the 20 studies in Loibl et al. 2017, for 908 records after deduplication, cut off on 2019-06-26. Included: 45 articles, 53 studies, 166 comparisons (81 experimental, 85 quasi-experimental); total number of participants not reported. Grades: 2–5, 25; 6–10, 75; undergraduate, 61; other, 5. Content: mathematics 75, basic science 47, medicine 32, domain-general skills 8, other 4. Conceptual knowledge and transfer g = 0.36 (95% CI 0.20–0.51), Q(165) = 295.91; procedural knowledge g = −0.03 (−0.20 to 0.15, 51 comparisons). Bayesian re-analysis 0.36 (0.19–0.52). Share of comparisons meeting each criterion: problem allows multiple solutions 92.77%, evidence of multiple solutions generated 59.64%, affective draw 61.45%, groups 53.01%, building on student solutions 44.58%, mainly dialogic instruction phase 33.73%; the problem-solving social environment was unreported for 57.2%. Subgroup point estimates (met vs. not met): evidence of multiple solutions 0.47 vs. 0.16 (p = .05), groups 0.49 vs. 0.19 (p = .04), building on student solutions 0.56 vs. 0.20 (p = .02), dialogic instruction 0.55 vs. 0.24 (p = .05), affective draw 0.44 vs. 0.19 (not significant). In Tables 3–5 many point estimates fall outside their own intervals; working back from the numbers, the intervals of all but the reference rows appear to be intervals for differences, so only point estimates are listed here. Multiple regression R² = 13.51%; in multimodel inference only "evidence of multiple solutions" (0.93) and "groups" (0.85) pass the Akaike weight threshold of 0.8, with "building on student solutions" at 0.40. Each one-percentage-point increase in the overall fidelity score raises the effect size by 0.0065; mean fidelity was 32.36% for experiments and 76.78% for quasi-experiments. In short interventions (a few hours), the gain without groups was β = 0.7395 (p = .0023). Subgroups: grades 2–5 −0.09, domain-general skills −0.17, grades 6–10 0.50, undergraduate 0.28; quasi-experimental 0.46, experimental 0.25 (not significant); several days 0.41, a few hours 0.32 (not significant). Publication bias: Egger intercept 0.63, p = .219; p-curve estimate 0.87. The authors use Kraft's 2019 benchmarks to say 0.36 "can be considered rather large"; Kraft's benchmarks come from randomized trials with standardized tests as outcomes, while most outcomes here are researcher-made tests. Limitations the authors list: nearly three quarters of comparisons are mathematics and science; uneven numbers of research teams in Europe and North America versus Asia and Australia; "generating multiple solutions" is a weak proxy for failure; no data on individual differences or students with learning disabilities; the two orders may have different learning goals; all criteria dichotomized.

Ashman (2022), UNSW Sydney doctoral dissertation, supervised by Kalyuga and Sweller. Five randomized experiments, all at the school where the first author teaches, with no student in more than one experiment. Experiment 1 (48 year 6 students), experiment 2 (72 year 5), and experiment 3 (69 year 6) detected no difference; experiment 3 had a ceiling effect, and a wrong heuristic could earn full marks on the posttest. Experiment 4 (71 year 5, tested the next day): similar problems, instruction first 31.06/48, problem first 21.40, d = .57; transfer 15.22/27 vs. 10.20, d = .56. Experiment 5 (64 year 5, tested six days later): similar problems 24.68/35 vs. 17.57, d = .56; transfer 11.41/28 vs. 8.33, d = .47, p = .06. In experiments 2–5 the two groups were offset by 15 minutes and heard the same lesson together, and the teacher gave no feedback during the lesson. No pretest. The author counted element interactivity as numbers of steps (8 and 12), with no independent measure and no measure of cognitive load. The two experiments listed in the journal abstract correspond to experiments 5 and 4 of the dissertation; this mapping was made during extraction by matching sample sizes and results, and the dissertation gives no correspondence table. Limitations the author lists: the two groups' treatments were offset in time; the instruction-first group had already done 15 minutes of reading before starting; the different posttest delays in the two experiments resulted from scheduling; the conclusions apply only to content high in element interactivity.

DeCaro, Isaacs, Bego, and Chastain (2023), Frontiers in Education 8, Brief Research Report. One section of an introductory physics course at the University of Louisville, N = 78 (explore first 37, instruct first 41); 3 students who lost connection for over a minute and 1 who scored 0 on the procedural items were excluded, and the number originally assigned was not reported. The two conditions met on different days with the same instructor. The explore-first group did a 13-minute activity in breakout rooms, then a 24-minute lecture, then a 5-minute review; the session opened with a "tips for effective exploration" slide, and the activity instructions included the hint to start by comparing PA and PB if stuck. Activity scores 4.59 vs. 6.73 (out of 10), F(1, 76) = 11.12. Posttest overall 76.49% vs. 66.36%, F(1, 76) = 5.79, p = 0.019, ηp² = 0.07; procedural subscale 83.78% vs. 70.04%, conceptual subscale 69.19% vs. 62.68%; condition × subscale interaction p = 0.164. The 10 conceptual items were all true/false, α = 0.30; procedural subscale α = 0.79. No delayed test, no pretest.

Alfieri, Brooks, Aldrich, and Tenenbaum (2011), Journal of Educational Psychology 103(1). Read from the author manuscript and the publisher's public supplementary tables. First analysis: unassisted discovery vs. explicit instruction, 108 studies (N = 5,226), 580 comparisons, study-level d = −0.38 (−0.50 to −0.25). Domains: mathematics −0.16, computer skills 0.07 (not significant, 72 comparisons), science −0.39, problem solving −0.48, physical/motor −0.01 (not significant), verbal/social −0.95. Age: children −0.44, adolescents −0.53, adults −0.26; only adolescents vs. adults was significant. Discovery types: general unassisted −0.41 (476 comparisons), simulation −0.13 (not significant). Explicit comparison conditions: direct teaching −0.29, feedback −0.46, worked examples −0.63, explanations provided −0.28; against "no instruction" 0.21 (not significant, 17 comparisons). Second analysis: enhanced discovery vs. other methods, 56 studies (N = 4,243), 360 comparisons, study-level d = 0.30 (0.15–0.44). Types: generation −0.15 (87 comparisons), elicited explanation 0.36 (128), guided discovery 0.50 (142). Comparisons: direct teaching 0.26, worked examples 0.06 (not significant, 22), explanations provided 0.33. Age: adults 0.44 (largest), children 0.24, adolescents 0.19. Reaction time −0.72 (enhanced discovery groups slower). Publication bias assessed only with fail-safe N. In several tables the k values do not add up to the totals.

Lazonder and Harmsen (2016), Review of Educational Research 86(3). Search 1993–2013, 72 studies (41 randomized, 31 quasi-experimental), participants aged 5–22, science 66, mathematics 5. All comparison groups were doing inquiry and differed only in whether they got the type of guidance studied. Learning activities (20 studies) d = 0.66; performance success (17) d = 0.71; learning outcomes (60) d = 0.50 (0.37–0.62). Learning outcomes by type of guidance: process constraints 0.52 (4), status overviews 0.61 (2, interval crosses zero), prompts 0.55 (8), heuristics 0.22 (2, interval crosses zero), scaffolds 0.34 (19), explanations 0.61 (25), Qb(5) = 4.05, p = .542. Performance success by type Qb(5) = 14.34, p = .014, with specificity correlating with effect size at r = .53; explanations and heuristics each had only 1 study. On learning outcomes, inquiry skills 0.78 and domain knowledge 0.37, a significant difference. No age differences were detected. Publication bias assessed only with fail-safe N. The authors say worked examples would be classified as explanations in this scheme. Limitations the authors list: 6 studies with incomplete statistics dropped; few studies in the learning-activity and performance-success analyses; some studies entered two or three analyses; multiple testing; mathematics and science only.

Kirschner, Sweller, and Clark (2006), Educational Psychologist 41(2). Position paper with no data, no search method, no tables or figures. The empirical work it cites is mostly given as conclusions without numbers. It calls the worked-example effect one of the strongest pieces of evidence for guidance and states two conditions under which it fails: when the examples themselves impose heavy load, and when learners' expertise grows, so that the effect first disappears and then reverses. The conclusion's stated audience is "novice to intermediate learners." No limitations section.

Belland, Walker, Kim, and Lefler (2017), Review of Educational Research 87(2). Search 1993–2014, 144 studies, 333 outcomes. Inclusion: a control group without scaffolds, cognitive outcomes measured, learners working on ill-structured STEM problems, support meeting Wood et al.'s 1976 definition of scaffolding; interventions given before learners encountered the problem were excluded. Overall g = 0.46, Q = 1096.96, I² = 69.7%; Egger test not significant. Design: individual randomization 0.40 (181), group randomization 0.48 (54), quasi-experimental 0.56 (98), with quasi-experimental significantly higher than individual randomization. Scaffold change: none 0.45 (216), adding only 0.46 (62), fading only 0.62 (12), both 0.50 (43), between groups p > .05; 16.5% of outcomes included fading. Basis for change: performance-adapted 0.47, self-selected 0.45, fixed schedule 0.62, none 0.45, not significant. Generic vs. context-specific 0.48 vs. 0.46, p = .778. Education level: middle school 0.37 (lowest), college 0.49; the adult cell has only 1 outcome. The 5 outcomes with the largest effects were excluded as outliers. 64% of outcomes reported no test reliability.

Zhang and Ma (2023), Frontiers in Psychology 14. 2003–2023, 66 studies, 190 effect sizes, compared with "traditional teaching." Random-effects SMD = 0.441, I² = 87.4%; Begg test Z = 5.082, p < 0.05; fail-safe N = 2,546; trim and fill is mentioned in the methods but no result is reported. Academic achievement 0.650 (0.453–0.846, 40); affective attitudes 0.389 (50); thinking skills 0.386 (100), with decision making, critical thinking, and question posing not significant. School level: primary 0.527, junior high 0.520, senior high 0.720, university 0.116 (57). Group size was reported for only 81 effect sizes. Table problems: the lower bound of the fixed-effect interval exceeds the point estimate; none of the nine country-and-region rows has an interval containing its own point estimate; the Z for creative thinking does not match its interval. No between-group test statistics anywhere in the tables. The authors list only two limitations: some studies lacked statistical information, and most studies did not break down learning outcomes.

Condliffe et al. (2017), MDRC working paper, ERIC ED578933, an update of a 2016 first version. Narrative review with no databases, search terms, or screening process described, and no quality-coding table. The only effect sizes in the whole paper come from Finkelstein et al. 2011, a teacher-level randomized trial (a problem-based economics curriculum plus five days of teacher professional development and year-round support; teachers dropped from 128 to 64): economic literacy 0.32, economic problem solving 0.27. The exceptions allowing causal inference are listed as two studies in section V and four in section VI. The first version was funded and published by Lucas Education Research, and the update was funded by the Kresge Foundation; the George Lucas Educational Foundation is also a partner in one of the curricula reviewed. No limitations section; limitations are scattered through the text (publication bias, most studies not supporting causal inference, different models across studies, a lack of measures for deeper learning).

Collins, Brown, and Holum (1991), American Educator, Winter 1991; another version is a 1989 book chapter. A framework article with no data. The framework has four dimensions with 17 items: content (domain knowledge, heuristic strategies, control strategies, learning strategies), method (the six in the table above), sequencing (global before local, increasing complexity, increasing diversity), and sociology (situated learning, community of practice, intrinsic motivation, exploiting cooperation). The three cases are methods others developed independently, which the authors read through the framework afterward: reciprocal teaching for reading (groups of two to seven, or one-on-one), procedural cue cards for writing (elementary students), and Schoenfeld's mathematical problem-solving course (college students). The numbers it relays (for example, a one-on-one pilot study's comprehension scores rising from 15% to 85%) come with no description of a comparison group. The authors' stated boundaries: not the only way to learn, not suited to rote content, no formula.

References

I read the full main text of each source and kept an extraction record; the parts I did not read are in parentheses.

• Sinha & Kapur (2021), When Problem Solving Followed by Instruction Works: Evidence for Productive Failure (SAGE version of record, open access CC BY; online supplement not obtained).
• Ashman (2022), Should problem solving precede explicit instruction when element interactivity is high? (UNSW Sydney doctoral dissertation) (UNSWorks institutional repository, CC BY 4.0; read the introduction, chapters 3–6, and the lesson slides and problem booklet in appendix 4; did not read the theory background in chapters 1 and 2, the references, appendices 1–3, 5, and 6, or the posttest items in appendix 4). The journal article on the same experiments, Ashman, Kalyuga & Sweller (2020), has no legal public full text and is not listed.
• DeCaro, Isaacs, Bego & Chastain (2023), Bringing exploratory learning online: problem-solving before instruction improves remote undergraduate physics learning (version of record, CC BY; read the posttest items in the supplementary appendix; raw data on OSF not downloaded).
• Alfieri, Brooks, Aldrich & Tenenbaum (2011), Does Discovery-Based Instruction Enhance Learning? (read the author manuscript in the University of Surrey repository and the publisher's public supplementary tables 1–10; the two per-study sample tables were not read row by row).
• Lazonder & Harmsen (2016), Meta-Analysis of Inquiry-Based Learning: Effects of Guidance (version of record, PDF in the University of Groningen repository; references only skimmed).
• Kirschner, Sweller & Clark (2006), Why Minimal Guidance During Instruction Does Not Work (Taylor & Francis version of record, free access; references only skimmed).
• Belland, Walker, Kim & Lefler (2017), Synthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education: A Meta-Analysis (version of record in PMC, CC BY-NC; moderator numbers read from the figures; supplementary tables S1 and S2 not obtained).
• Zhang & Ma (2023), A study of the impact of project-based learning on student learning effects: a meta-analysis study (version of record, CC BY; no supplementary material could be found).
• Condliffe et al. (2017), Project-Based Learning: A Literature Review (MDRC Working Paper) (full version on ERIC; references only skimmed).
• Collins, Brown & Holum (1991), Cognitive Apprenticeship: Making Thinking Visible (web reprint on the publisher's site, American Educator; volume and page numbers not verified).
• CS50's Introduction to Artificial Intelligence with Python: Academic Honesty (course website, read 2026-10-13).
• Posts on this site: What the Evidence for Learning by Doing Measures, Writing Questions and Wording in a Teaching Article, How Far Should the Code in a Teaching Article Go, Writing the Examples and Openings of a Teaching Article, Harvard CS50 AI Week 0, Harvard CS50 AI Week 1.