Should a Teaching Article Use Interactive Figures: Classroom Simulations, Interaction Added to Charts, Reader Usage, and Production Cost

🌏 中文版

This is part 7 of the "Writing Articles That Teach" series and the last post in its first section, on what goes on the page. Earlier posts covered text, diagrams, where to place video, and how to lay out video next to text. This one asks about the most expensive option: figures or simulations that readers can drag, adjust, or click. Are they worth building?

The question came from a cost table in part 2. That post looked at when this site's course guides should use a diagram, and in its section on static pictures, animation, and interactive figures it listed a cost comparison. The row for manipulable figures said: no learning-outcome comparison that I had read, and high cost. This site's guides mostly explain things that change step by step, such as search, pruning, and gradient descent. Draggable figures like the ones in Red Blob Games' introduction to A* cover the same kind of material, so that row had to be filled in eventually. This post fills it in through three questions: what readers learn when interaction is added, how many readers actually use it, and what it costs to build.

The sources include meta-analyses (studies that pool the results of many experiments), experiments, click logs from real readers, and slides, blog posts, and production notes written by people who make interactive content. The biggest gap comes first: I found no study that turned the same content into an interactive article and a static article and compared how much readers learned. What follows rests on three kinds of indirect evidence: classroom simulations, interaction added to data charts, and reader behavior logs. Participants were mostly students or workers on crowdsourcing platforms, the materials were mostly short, and no study directly tested a self-learner reading a long guide.

Conclusions first

No row directly answers "should my guide have interactive figures?" The closest is the fifth row, and it is about what readers did, not whether the page had interactive components. The section "Whether interaction helps depends on what readers do with it" pulls this together.

What readers learn when interaction is added

No direct comparison, and the makers of interactive articles say so

Fred Hohman, Matthew Conlen, Jeffrey Heer, and Duen Horng Chau's 2020 Distill article "Communicating with Interactive Articles" is a review written by people who make interactive articles. Conlen and Heer created Idyll, a tool for writing interactive articles, and Hohman and Conlen founded the interactive publication The Parametric Press. The introduction says interactive articles "have been shown" to be more engaging and to help recall and learning, and neither sentence cites anything. At the end, the authors write:

The usefulness of interactive articles is predicated on the assumption that these interactive articles actually facilitate communication and learning. There is limited empirical evaluation of the effectiveness of interactive articles.

They go on to say that large publishers will not share internal metrics and that laboratory studies may not generalize to real reading. If you cite this article on the evidence, cite the closing sentence. The two sentences in the introduction cannot be passed on as claims with evidence behind them.

The same article contains a line that is rarely quoted: not everything needs to be interactive, and in the worst case interactivity distracts readers or goes unused, and the author has wasted the time spent building it.

Classroom simulations: a large effect that cannot all be credited to interaction

Cynthia D'Angelo and colleagues' 2014 report for SRI International is the only meta-analysis I found whose subject is interactive simulation. It covers K–12 STEM studies published in peer-reviewed journals from 1991 to 2012: 59 articles and 128 effect sizes. The report itself was published by the institute and was not peer reviewed.

Its definition of a simulation requires interaction: learners can set parameters and see outputs. Non-interactive animations and visualizations were excluded at screening. Students taught with simulations scored 0.62 standard deviations higher on achievement tests than students taught some other way. (An effect size is the difference between two group means divided by the standard deviation; around 0.2 counts as small and around 0.5 as medium.)

That number comes with conditions:

• What the comparison groups did was not coded. The report describes the comparison only as "non-simulation instruction," which could be lecture, hands-on labs, or reading a textbook; it is not broken down. So 0.62 is the gap between a whole package of instruction that includes a simulation and some other instruction. It cannot be attributed to interactivity as a feature.
• It holds only in science. The 42 science effect sizes average 0.67. Math has only 4, and their average is negative (−0.15) and not significant. Outcomes on scientific inquiry and reasoning (6 effect sizes) were also not significant.
• The setting is far from an article. Participants were K–12 students, and more than half the studies were in high school. The report does not give the distribution of settings (classroom, after school, out of school). Most assessments were written by the researchers, and only 5 were given more than two weeks later. Whether the two groups spent the same amount of time is not reported separately.

The same report has one result that bears directly on design. When comparing the same simulation plus some enhancement against the original simulation, and counting only science studies, highly structured simulations (0.88, 9 effect sizes) did significantly better than partly structured ones (0.42). Adding the math studies made the difference non-significant, and the authors say more research is needed. They also state that the articles reporting studies rarely describe what the simulation looked like, so they could not analyze which simulation features work.

Making the figure move: covered in part 2, with two additions

Part 2 already covered the meta-analyses of animation versus static pictures. Sandra Berney and Mireille Bétrancourt's 2016 meta-analysis pooled 140 comparisons for an overall effect size of 0.226, which the authors classify as small. The advantage of animation concentrated in three conditions: system-controlled pacing, narration, and no accompanying text. I won't repeat that here. Two points relevant to interaction are new.

First, when learners could control playback, animation showed no advantage. Berney and Bétrancourt grouped studies into four levels of learner control: none, a play button only, play and stop, and full control with rewind, fast-forward, and replay. With no learner control, animation beat static pictures by 0.309, which was significant. The other three levels came out at 0.129, 0.061, and −0.120, none significant. This is a comparison of subgroups across studies, not a manipulation of control within one experiment, so it cannot be read as "letting readers control the pace makes them learn worse." What it does say is that in the studies where readers controlled playback, animation did not beat static pictures. The meta-analysis also did not analyze whether learners could manipulate the content on screen, only playback control.

The same table has three subgroups that match this site's conditions: abstract representations (charts, formulas, symbols) at 0.006, informatics at 0.108, and math at −0.067, all non-significant. These are also cross-study comparisons, and the authors say the tests have low power and that "domain" may not be a valid moderator at all.

Second, the second meta-analysis is not an independent replication. Juan Castro-Alonso and colleagues' 2019 meta-analysis included only randomized experiments, 46 articles and 82 comparisons, and arrived at 0.23, which looks consistent with Berney's. But 8 of its articles were added from the reference lists of three earlier meta-analyses including Berney's, and when I compared the two lists I found at least 14 studies in both, without counting the full overlap. So the two cannot be treated as independent evidence. Castro-Alonso's "dynamic" category also includes static pictures shown one after another. The authors state in their limitations that they did not analyze interactive features.

Part 2 also covered Rolf Ploetzner and colleagues' 2021 experiment (one picture, four sequential pictures, or an animation). Its animation is explicitly non-interactive and all three groups only watched, so it says nothing about whether to build interaction.

The same chart, with interaction added

The experiment closest to "does adding interaction to a static chart help?" is Ab Mosca, Alvitta Ottley, and Remco Chang's 2021 study (I read the preprint of the CHI 2021 paper).

Participants were workers on Amazon Mechanical Turk, and each solved one Bayesian reasoning problem: a short text in frequency format next to an icon array. None of the interactions added information. They only hid or highlighted subgroups already on the chart, or linked the text to the chart. Every condition had both text and chart.

• The first experiment, with 472 participants, compared a chart whose subgroups could be toggled with checkboxes against the same static chart: 53% versus 57% correct, with no difference detected.
• The second experiment, with about two thousand participants, tried five interactions (checkboxes starting all on or all off, dragging labels onto the chart, highlighting when hovering over the text, and a tooltip when hovering over an icon). None beat the static chart.

The authors conclude that despite their best efforts, none of the interaction techniques they tested significantly improved accuracy, and they recommend caution with interaction that adds no information.

The abstract says interaction can "significantly detract" from performance in some cases. In the body, only one result passed a significance test: participants with high spatial ability did worse with the version that highlights on hover over the text than with the static version (about 60% versus 80%). The other declines were not significant.

Four conditions go with this paper. It measured accuracy on one problem answered immediately, not learning, retention, or transfer. The interactions revealed no new information, and the authors say that interaction which does reveal new information (panning a map, for example) has undisputed value. The static control group in the second experiment was borrowed from the first, with different pay and procedure, and the only significant negative result came from exactly this cross-experiment comparison. For the overall comparison the authors report p = 0.05 and call it non-significant. I recomputed it from the chi-square distribution and got about 0.045, so the accurate wording for this one is "the authors judged it non-significant," not "clearly no difference."

What readers do with the figure

Part 2 cited Christopher Hundhausen, Sarah Douglas, and John Stasko's 2002 review. It covers 24 controlled experiments on algorithm visualization and concludes that what students do with a visualization matters more than what the visualization shows. Of the 14 experiments that manipulated student activity, 10 found a significant difference; of the 10 that manipulated how the display was drawn, 3 did.

Reading the full paper adds three points, all of which weaken that conclusion:

• The gap disappears when the comparison group also gets an activity. In Hansen and colleagues' third study, the text-only group also did paper-and-pencil exercises, and the result did not differ significantly from the group using the visualization system. Gurka set out to replicate Byrne and colleagues' experiment but dropped the prediction step; both groups had small-group discussion, and again there was no significant difference. The authors' own reading is that the researchers made the text group as active as the visualization group, so no difference showed up.
• "Predict first" is unstable on its own. Jarc and colleagues compared animation software with and without predicting the next step. Post-test scores did not differ; the prediction group only spent significantly more time. In Byrne and colleagues' two significant results, the effects of prediction and animation cannot be separated. I checked Table 2 row by row: the "10 of 14" above counts these two as well, which the paper does not state; without them it is 8 of 14. The manipulation closest to "ask first" is Hansen and colleagues' sixth study, which removed the interactive questions from the system and also found no significant difference.
• Method limits. This paper is not a meta-analysis. The authors call it a meta-study and count significant versus non-significant results, with no effect sizes and no description of how they searched. By my count from Table 2, 14 of the 24 experiments come from two sources. Which experiments count as manipulating student activity is the authors' own judgment. Their conclusion uses the word "appears," and they answer the question of whether visualization works with "a qualified 'yes'."

Yea-Seul Kim, Katharina Reinecke, and Jessica Hullman's 2017 experiment took "predict first" to data charts. About 370 online participants looked at 2008 US presidential election data by ethnicity for two states. There were five techniques: just look at the data; type explanations while looking; draw your guess for one group on the chart, then see your guess overlaid on the real data; draw then explain; draw then see system feedback. The data was shown either as a line chart or as text, for nine conditions in total (draw-then-feedback had no text version). Three minutes later, participants had to recall 12 percentages.

• With the chart, all four techniques reduced recall error compared with just looking, and they did not differ from one another. Prediction alone had the smallest effect.
• With text, both prediction techniques had no effect; only typing an explanation helped.
• With data participants knew well (fast-food calories), no technique had any effect; with very unfamiliar data (a blood-sugar experiment in mice), prediction alone also had no effect.
• Participants who predicted recalled values biased toward their original guesses.

The title and abstract speak of recall and comprehension, but the paper has no comprehension questions; both outcome measures are recall errors. The control group retyped an unrelated passage rather than doing nothing. So the experiment compares an activity related to the data against unrelated typing, not interaction against no interaction. The authors also did not directly test the difference between chart and text.

Taking the two papers together, my inference is that having readers do something related to the content with the figure may help more than just looking, but that activity does not need an interactive component. In Kim's experiment, typing an explanation helped with both chart and text, while prediction showed an effect only in the chart conditions. In the experiments Hundhausen reviewed, giving the text group equivalent exercises made the visualization's advantage disappear. The evidence on asking readers to explain for themselves is covered in part 1.

How many readers actually use the interaction

Click logs from three interactive articles

Matthew Conlen, Alex Kale, and Jeffrey Heer's 2019 paper instrumented three interactive articles written in Idyll: a rhythm lesson on 3/4 versus 6/8 time, an introduction to dimensionality reduction, and a guitar-tuning lesson. The paper reports over fifty thousand reading sessions; adding up the three articles' own figures, I get about seventy-two thousand. Part 2 cited two of its numbers; here is the full picture.

"How many readers interact" has no single answer. It depends on the component. All of the following are desktop readers:

The authors' reading is that readers are willing to interact but mostly with the one component on the main line of the narrative. The last row is the use the authors designed for, and only about 5% used it that way; the authors themselves say this presentation was not as effective as intended.

Mobile readers' median time on page was 23% to 52% of desktop readers'. In the guitar article, 23% of desktop readers tuned all six strings, against 6% on mobile. But all three articles were designed for desktop, and many features were disabled or made static on mobile, so the gap cannot all be put down to the device. The authors' advice to designers is that the narrative should stay intact and understandable even if a reader never touches the interactive widgets.

This data has three limits. It measures only behavior, not understanding or learning, and there is no static version to compare, so it cannot say interaction made people read longer. Of the three articles, the guitar one was written by two of the paper's coauthors, and according to the references the dimensionality-reduction one was written by Conlen and Hohman. Readers mostly came from Hacker News, Reddit, and Twitter, and traffic was concentrated in the two days after publication. Unless noted, all statistics count desktop readers only.

Putting interaction on the page does not mean anyone uses it

Mosca and colleagues measured this too. In the group that could toggle subgroups, only 43% touched the checkboxes in the first experiment, and 44% in the second. Those who did were correct 70% of the time, against 40% for those who did not. Participants chose for themselves whether to interact; they were not randomly assigned, and the authors draw no causal conclusion. Interacting may help, or people who were more engaged to begin with may be the ones who interact. The 70% versus 40% cannot be written up as "people who interact learn better."

Where The New York Times number comes from

The claim usually passed around, as Aisch quotes it, is that "85% of the Times' page visitors online simply ignore interactive infographics altogether". I read the two original sources, and the number means something different.

The 2016 slides by Archie Tse, deputy graphics director at The New York Times, are titled "Why We Are Doing Fewer Interactives," and they contain no numbers. They say readers were not getting to all the content and readers just want to scroll, and they give three rules: "If you make the reader click or do anything other than scroll, something spectacular has to happen." "If you make a tooltip or rollover, assume no one will ever see it." And: "When deciding whether to make something interactive, remember that getting it to work on all platforms is expensive."

The number comes from Gregor Aisch's 2017 blog post. Writing "we" about the Times' work, he says he measured it himself: the share of readers who clicked a prominent button on a couple of graphics published in 2015. He mentioned it at a conference, and from there it passed through another talk and a Medium post before turning into the sentence above in a design publication. His correction:

the 85% of readers didn't ignore the graphic, they just didn't click the button.

His conclusion comes with conditions: don't hide important content behind interaction, the static version has to stand on its own, and interaction serves the minority of readers who want to dig deeper; "15% of readers isn't nobody." None of his three examples (tooltips that let readers fact-check the numbers, navigation through the full dataset, and an election calculator where readers set their own scenarios) comes with usage figures.

Hohman and colleagues also took this up in 2020: the Times' statement has become a rule of thumb for designers, and many choose not to use interactivity because of it, even though later discussion clarified the original point. They call the statement an anecdote and say more research is needed to find when interactivity is worth its cost.

What it costs to build

There is no research data on hours or money for this question. All I have is what people who make interactive content say, and they are all high-end makers.

• Amit Patel of Red Blob Games (the pages are signed only "Red Blob Games"; his name appears in the page source) wrote in his 2014 production notes: "It can take 1 to 5 months to make a tutorial." That includes the time to learn the algorithm, and he doesn't say whether it is full-time or spare time, so it cannot be converted into hours. He puts the slowness down to an unstructured process: he doesn't know what he wants beforehand. While making the A* page, he rewrote the diagram code twice partway through.
• Hohman and colleagues say that building a successful interactive article is closer to building a website than writing a blog post, often taking more time than an academic paper. It takes editorial, design, and programming skills, and many interactive articles are made by a team. Web technology changes fast enough that interactive content can break within a few years. It has to work on desktop and mobile and across connection speeds. Making it usable by readers with disabilities is still an open problem. For the first issue of The Parametric Press they worked with the authors for four months. The article gives no figures for hours.
• Distill's editorial team, announcing a hiatus in 2021, said the main bottleneck to more articles of this kind is the effort it takes to produce them and the unusual combination of scientific and design expertise required. The only number of hours in the announcement: for some early articles, editors spent more than 50 hours helping authors with diagrams and writing. That is editors' help on top of the authors' own work, not the time an author needs to write one. The people who burned out were volunteer editors, and they attribute it mainly to the structure of running a journal (mentors who also had to accept or reject), not to such articles being something nobody should make.
• Tse's third rule is also about cost: getting things to work on all platforms is expensive.

Hundhausen and colleagues already noted in 2002 that algorithm visualization experiments only looked at the period when students used the tools; none measured how long instructors spent preparing and setting them up. I still have not found any data on how long a simple chart with one adjustable parameter takes to make.

Two lines from Patel's notes are worth more than the hours. After finishing his first draggable figure, he wrote:

Great! Now I have an interactive diagram. But this isn't an explanation.

He then had to break the algorithm into step-by-step code before it became an explanation. The other is a question he leaves unanswered: some points in the parameter space are more interesting or educational than others, so how do you guide the reader to them? That matches Conlen's logs: in the dimensionality-reduction article, most readers either never touched the slider or dragged it all the way to the left.

Whether interaction helps depends on what readers do with it

Putting the answers to the three questions together, here is my inference, with the evidence after each point.

1. The interactive component itself showed no effect. Adding interaction to the same chart beat the static version in no case (Mosca and colleagues); when learners controlled playback, animation did not beat static pictures (Berney and Bétrancourt, a cross-study comparison).
2. What worked was readers doing something related to the content. In algorithm visualization experiments, manipulating student activity was more likely to produce a difference, but the gap disappeared when the comparison group got equivalent exercises (Hundhausen and colleagues). In the data-chart experiment, typing an explanation helped with both chart and text (Kim and colleagues).
3. Most readers will not use interaction the way the author intends. Many people use the main component that responds at a touch; most never touch what requires a click, an adjustment, or the idea to try something themselves (Conlen and colleagues; Mosca and colleagues).
4. The cost is high and has not been measured. The only time figure comes from an experienced maker and includes learning the content itself (Patel; Hohman and colleagues; Distill).

So "should I build interaction?" can be swapped for another question: what do I want the reader to do here, and does that need a moving component? The evidence on doing itself, from large meta-analyses of active learning to whether it carries over to one person reading an article, is the subject of the next section of the series; a later post on what the evidence for "learning by doing" actually measures picks it up.

Where sources differ

The pairs below either point in different directions or look contradictory while measuring different things. I list them side by side without choosing.

Has it been shown that interactive articles are better? Hohman and colleagues' introduction says interactive articles have been shown to be more engaging and to improve learning, without citations; the end of the same article says there is limited empirical evaluation of their effectiveness.

Do readers use interactive components? Tse's slides say readers just want to scroll and that no one will see a tooltip. In Conlen and colleagues' logs, 85% of desktop readers of the rhythm lesson used the main component. The two measure different things: Tse published no data, and the number Aisch describes is the click rate on one prominent button; Conlen's 85% is for a component that responds on hover, and in the same logs the median for the slider was 0.

Does predicting first help? In Kim and colleagues' experiment, drawing a prediction on the chart before seeing the real data reduced recall error compared with just looking. In the same experiment, prediction showed no effect when the data was shown as text, had no effect on familiar data, and prediction alone without explanation had no effect on unfamiliar data either. Among the experiments Hundhausen reviewed, the only one where prediction can be separated out is Jarc and colleagues', and the prediction group did no better on the post-test; in Byrne and colleagues' two experiments, prediction and animation cannot be separated.

More structure or more freedom? Counting only science studies, D'Angelo and colleagues found highly structured simulations had a significantly larger effect; adding math made it non-significant. In Berney and Bétrancourt's subgroups, the more control learners had, the smaller the point estimate of animation's advantage (no pairwise tests). Both are cross-study comparisons, and they do not measure the same kind of freedom: the first is how much structure the simulation's operation has, the second is playback control.

Does interaction hurt performance? Mosca and colleagues' abstract says interaction can significantly detract from performance in some cases. Only one result in the body passed a significance test (high-spatial-ability participants with the hover-highlight version); right after mentioning it, the discussion says none of these findings are significant. The conclusion says the effect of interaction depends largely on the design of the underlying chart, but all four tests of the interaction between chart design and interaction technique were non-significant.

Do the two animation meta-analyses confirm each other? Berney and Bétrancourt found 0.226, and Castro-Alonso and colleagues 0.23. The two include overlapping studies (at least 14 appear in both; I did not finish counting), and their definitions differ (the first excludes video and includes only expository animation; the second includes video and counts static pictures shown in sequence as dynamic). They cannot be treated as two independent pieces of evidence.

How this site does it

This site's posts are all Markdown files, and I found none with a manipulable figure. The only moving parts relevant here are two: course-guide self-check questions keep their answers in an expandable <details> block, and video timestamp links play the video in the page. Below, each practice is compared with the evidence, marking which have support and which are this site's own choices.

The last row needs one more sentence. Aisch says not to hide important content behind a click or hover, with an exception in parentheses: "unless you really want to." His example is a Times piece that has readers draw their guess before revealing the data. A self-check answer is that kind of thing, so folding it away does not conflict with the row above. The real conflict arises when an answer contains an explanation the body has not given, because readers who don't open it never see it. So the rule is: answers can be folded, but any explanation they need goes into the body first, and the answer is only for checking. Part 9 found no direct research on whether to fold answers, and asking readers on the question to write down their answer first is also a practice without direct evidence. Whether to leave a hint outside the fold, and whether a hint gives the answer away, depends on whether readers answer at all; part 10 takes that up.

Part 2's cost table lists "ask before showing the figure" as the cheapest form of interaction, and its evidence column says: the broad category of having students do something has support; the one experiment that can separate out prediction (Jarc and colleagues) found no difference, and the experiment that removed the interactive questions (Hansen's sixth study) found none either. This post reads it the same way and adds one point: when the text-only group got equivalent exercises, the visualization's advantage disappeared (Hansen's third study, Gurka). So that row supports having readers do something, not the idea that it has to be done with a figure.

What you can do

If you are thinking about adding interaction to a teaching article, here are the practices I drew from the sections above, each with its basis.

1. Write the static version first and check that it reads without the interaction. This is Conlen and colleagues' conclusion for designers and the premise shared by Aisch's three examples. How: remove the interactive component, keep only its default view, have someone who hasn't read the piece read it, and ask whether they can state the main point of the section.
2. Decide what you want the reader to do before deciding whether to build a moving component. The basis is Hundhausen's and Kim's results, point 2 in the previous section, and this is my inference. How: write in the draft, "When readers reach this figure, they should ___." If the blank says "guess the next step," "say why," or "work it out once," start with a self-check question and a <details> block. If it says "try several parameter values and see how the output changes," that is what a manipulable figure actually adds.
3. If you build it, put it on the main line of the narrative and make it respond at a touch. The basis is Conlen's 85% versus a median of 0. How: place the component in the middle of the text with its default state set to the example the section is about; don't put it behind "click here to see more."
4. Lead readers to meaningful parameter values. Patel has no answer for this himself, and Conlen's slider data shows most readers won't find them on their own. This one is my inference. How: next to the slider, name two or three values in words, such as "try setting the learning rate to 1.5 and see what happens," and explain that value's result in the next paragraph.
5. Give mobile a fallback. The basis is Conlen's mobile data and Tse's third rule. How: open the page in a 390px-wide window; if the component doesn't work, there should at least be a static picture of its default state.

Three things to do tonight

1. Find the figure in your article that you most want to make interactive, and add a question before it. Ask readers to say why, not just guess the answer, for example: "If the frontier changes from a stack to a queue, which node comes out next, and why?" Put the answer in <details>, and make sure any explanation the answer relies on is already in the body.
2. Go through everything in your article that only appears on click or hover. If it is needed to understand the section, move it into the body.
3. If you already have an interactive component, take it out and read the page once. Whatever stops making sense is what the static version is missing; add it as text or as a picture of the default state.

Limits of the evidence

• No direct comparison. I found no study that compared learning from the same content as an interactive article and as a static one. The conclusions here are inferred from classroom simulations, data-chart experiments, and behavior logs, and every inference is marked where it appears.
• I could not read the meta-analyses on learner control. Karich and colleagues (2014) and Landers and Reddock (2017) each published a meta-analysis on giving learners control; I could not find legally available full texts, so this post does not cover the average effect of control itself. The series plan lists this as the sticking point for a later post on series order.
• Narrow outcome measures. Mosca measured accuracy on one reasoning problem, Kim measured recall after three minutes, Ploetzner measured recognition right after viewing, and Conlen measured only behavior. Mosca, Kim, and Ploetzner had no delayed tests; of D'Angelo's 76 assessments, only 5 were given more than two weeks later.
• Participants are far from self-learners. D'Angelo's were K–12 students; Mosca's and Kim's were online workers paid to take part; Conlen's readers came from tech communities, across three articles from a single tool's community.
• The algorithm visualization evidence is old. All 24 of Hundhausen's experiments predate 2002, and the review only counts significant results; by my count from Table 2, 14 of the 24 come from two sources.
• Cost comes only from high-end makers' own accounts. Patel, Hohman and colleagues, and Distill's editors all make large interactive pieces; there is no data on how long a simple adjustable chart takes.
• Interests. Both Hohman and colleagues' and Conlen and colleagues' papers include authors of the interactive-article tool Idyll, and two of Conlen's three articles are connected to the paper's authors. This doesn't affect the data they published, but their qualitative assessment of interaction is a supporter's assessment.
• Cross-study comparisons. Berney's playback control, D'Angelo's degree of structure, and Castro-Alonso's subgroups are all comparisons between different studies, not manipulations within one experiment.
• Usage in interactive textbooks. One study of how much each component in an interactive e-book actually gets used will be covered in the later post on whether readers answer questions; this post does not cite its numbers.
• Gaps in what I read. I did not read Mosca's supplementary materials, Conlen's raw logs and supplementary materials, Kim's supplementary materials, or the list of interactive articles in Hohman's Table 1, and I did not look at the interactive figures on the Red Blob Games pages themselves.
• My inferences and this site's choices: the four-point synthesis in "Whether interaction helps depends on what readers do with it"; putting no new explanation in self-check answers; points 2 and 4 under "What you can do"; and the three things to do tonight. Each is marked where it appears.

Appendix: study designs and numbers

The main text keeps one or two numbers per item; the rest are here.

D'Angelo, Rutstein, Harris, Bernard, Borokhovski, Haertel (2014), SRI International report funded by the Bill & Melinda Gates Foundation, not peer reviewed. Searched ERIC, PsycINFO, and Scopus; included only peer-reviewed journal articles from 1991–2012, no grey literature. 2,722 abstracts; 59 articles and 128 effect sizes (achievement 96, scientific inquiry and reasoning 17, non-cognitive 15). Of the 59 articles, K–5 8, grades 6–8 15, grades 9–12 32; science 49, math 6. Random-effects model. Simulation versus no simulation, achievement: g = 0.62 (95% CI 0.45–0.79, k = 46, N = 2,947, I² = 78.55); science 0.67 (k = 42), math −0.15 (k = 4, non-significant), difference Q = 15.28. Inquiry and reasoning 0.26 (k = 6, non-significant); non-cognitive 0.69 (k = 12, I² = 92.92). Enhanced versus original simulation, achievement: 0.49 (k = 50). In the science-only moderator analysis, the only significant moderator was number of sessions (Q = 12.32, p = .006): one session 0.49 (k = 11), two to three 0.39, four to six 0.91 (k = 9), eleven or more 0.44 (k = 8), with 11 not reported. Enhanced versus original, science only: highly structured 0.88 (k = 9) versus partly structured 0.42 (k = 22), Q = 7.00, p = .008; p = .11 with math included. Assessment type: technology-based assessments not embedded in the simulation 0.11 (k = 4, non-significant), Q = 12.42, p = .002. Randomized and quasi-experimental studies did not differ on achievement (0.52 versus 0.57). Publication bias: trim-and-fill added no studies; the population included only journal articles. Of 76 assessments, 58 were researcher-designed; 5 were given more than two weeks later, and 24 did not report timing. Limits listed by the authors: no moderator analysis of specific simulation features was possible; too few studies in math, technology, and engineering; 83 studies with only qualitative data were not included. The report has ten numerical inconsistencies (two sets of inter-rater agreement figures, screening counts that don't add up, and others), plus two places where wording or counting units need care.

Berney and Bétrancourt (2016), Computers & Education 101. Included only computer-presented expository animation, excluding video; searched through the end of 2013, in English, French, and German, including unpublished dissertations; did not require random assignment. 50 articles, 61 experiments, 140 comparisons, 7,036 participants. Overall g = 0.226 (95% CI 0.12–0.33), I² = 78.38; of 140 comparisons, animation was significantly better in 43, static significantly better in 14, and no significant difference in 83. Pacing control: system-paced 0.309 (k = 96), play button only 0.129 (k = 17), play and stop 0.061 (k = 15), full control −0.120 (k = 12), between-groups QB(3) = 8.921, p = 0.03 in the table (the text says 0.003; the table is used here); no pairwise comparisons. The 12 full-control comparisons come from 6 articles, one of which contributes 4, all negative (I counted this from the appendix; the paper gives no such figure). Representation: abstract 0.006 (k = 26), iconic 0.245 (k = 91). Modality of accompanying explanation: written text 0.105 (k = 58, non-significant), narration 0.320, no text at all 0.886 (k = 14, 6 of them from one article). Domain: informatics 0.108 (k = 13), math −0.067 (k = 11), both non-significant. Publication bias assessed only with fail-safe N (2,999). Multiple outcomes from the same study were treated as independent estimates, which the authors acknowledge makes standard errors too small. The control column labels in appendix Table A1 do not match Table 3 and cannot be used directly.

Castro-Alonso, Wong, Adesope, Ayres, Paas (2019), Educational Psychology Review 31. Included only studies stating random assignment, published 1990–2017 in English peer-reviewed journals, and reporting the sample's gender ratio. 46 articles, 82 effect sizes, 5,474 participants; 77% of comparisons involved university students. Overall g+ = 0.23 (95% CI 0.17–0.28), I² = 68%. University subset 0.19; samples 60% or more female 0.07 (non-significant). With different media (computer or television dynamic versus paper or slide static, k = 7) the direction reversed, −0.20, and the authors say no strong conclusion can be drawn. Egger's test p = 0.42. Limits listed by the authors include not analyzing realism and interactive features. The paper does not say whether it used a fixed- or random-effects model and did not code whether tests were immediate or delayed.

Ploetzner, Berney, Bétrancourt (2021), Instructional Science 49. 88 university students in teacher education, three groups of 30, 28, and 30; the material was a six-bar linkage; the animation could not be manipulated and had no narration. Participants viewed for 2 minutes and answered immediately. Identifying motions (out of 5): one picture 1.77, four pictures 1.64, animation 3.80. Identifying arrangements (out of 4): one picture 3.93, four pictures 2.93, animation 2.93. The authors say they studied only perceptual-level processing. The motion items were four-option multiple choice, so the two static groups' scores (35% and 33%) were only a little above the 25% expected by guessing; I calculated this from the table, and the paper does not mention it.

Mosca, Ottley, Chang (2021), CHI 2021 (read as the arXiv:2103.01701v2 preprint). Amazon Mechanical Turk, US only, approval rate above 90%. Each participant solved one Bayesian reasoning problem (of 100 people, 6 have the disease and 4 of them test positive; 94 do not and 16 of them test positive); both sub-questions had to be right to count as correct; three base charts (grouped, aligned, randomly scattered). Experiment 1: checkboxes versus static × three base charts, N = 472 after excluding 58; base pay $1.80 plus $0.10 per correct answer; an instruction page. Accuracy 52.85% versus 57.08%, χ²(1) = 0.69, p = 0.41. Of 246 in the checkbox group, 43% interacted; those who did were correct 70.48% of the time and those who didn't 39.72% (self-selected within the group, χ²(1) = 21.63). On the randomly scattered base, the interactive version scored 42.68% and the static 58.57%; the six-group test gave p = 0.27, and this pair was not tested separately. Experiment 2: five interactions × three base charts, N = 1,980, base pay $0.80, no instruction page; the static group was the 226 participants from experiment 1, for a total of 2,206. Accuracy across six conditions: static 57.08, checkboxes all on 56.60, all off 56.85, drag 49.71, hover-highlight 48.16, hover-tooltip 53.33; χ²(5) = 11.33, reported as p = 0.05 and judged non-significant. Of the 371 with checkboxes all on, 44% interacted; the other interactions had "significantly higher" rates, with no figures given. Only the drag group showed a significant difference between those who interacted and those who didn't. The base chart had a significant main effect (χ²(2) = 8.09, p = 0.02), grouped beating randomly scattered. In the high-spatial-ability group (N = 1,114), the six conditions differed significantly, hover-highlight 60.45% versus static 80.36% (χ²(1) = 12.44, p < 0.001); not in the low-spatial-ability group (N = 1,092). High and low spatial-ability participants using the static chart scored 80% and 34%; the paper does not test this gap. NASA-TLX was collected but not reported in the text. Gender composition differed between the two experiments (38.3% versus 54.1% female).

Hundhausen, Douglas, Stasko (2002), Journal of Visual Languages and Computing 13(3). The authors call it a meta-study and compute no effect sizes. 24 between-subjects experiments; Hansen and colleagues (2000) contribute 8 and Lawrence's 1993 dissertation 6. 11 significant, 10 non-significant, 2 significant but with prediction and animation inseparable (Byrne and colleagues), and 1 in the opposite direction (people using a graphical tracing tool solved significantly fewer problems). By theory: cognitive constructivism 10 of 14 significant (I checked Table 2; this includes Byrne's 2, which the paper does not state), epistemic fidelity 3 of 10, dual coding and individual differences 1 of 2 each. By whether the two groups' effort was equal: of 9 with equal effort, 6 were non-significant; of 14 with unequal effort, the higher-effort group did significantly better in 10. Unequal effort but no significant difference: Stasko and colleagues (text versus text plus animation), Hundhausen and Douglas (constructing a visualization versus actively viewing a ready-made one), and Jarc and colleagues (prediction versus none; the prediction group spent significantly more time). In Lawrence's chapter 7 experiment, black-and-white beat color. Limits listed by the authors: no assessment of each experiment's validity; classifications are the authors' judgment; the pre-to-post-test gain result comes almost entirely from one line of research; a narrow range of outcome measures; no measurement of preparation and setup costs. The lists in Table 5 don't match the 9 and 14 in the text; my guess is a typo in the table.

Kim, Reinecke, Hullman (2017), CHI 2017. Amazon Mechanical Turk, 42 per condition, 378 recruited and 5 excluded, paid $1.50. Five techniques × two presentations (no text version of prediction with feedback), nine conditions. The data were the Republican vote shares in the 2008 US presidential election in Colorado and Connecticut, for three ethnic groups in two income brackets, 12 data points; predictions covered only one randomly chosen ethnic group. After viewing, participants did a three-minute paper-folding test, then recalled the values. Chart conditions, reduction in absolute error compared with control: predict then explain 4.08 percentage points, predict with feedback 3.54, explain only 3.52, predict only 2.30, with no differences among the four. Only the first two reduced trend error. Chart conditions overall had 3.76 percentage points (36%) more error than text conditions. Text conditions: explain only reduced error by 2.00 percentage points (p < .05); both prediction techniques had no effect; none affected trend error. For 75.9% of data points, recall was biased in the same direction as the prediction. Neither the number nor the quality of self-explanations was related to error. Low-familiarity data (mouse blood sugar): predict only non-significant, the other three effective, no effect on trend error; high-familiarity data (fast-food calories): no differences between any conditions. The sample sizes of the two replications are not given in the text. The interaction between presentation and technique was in the model but not reported in the text.

Conlen, Kale, Heer (2019), EuroVis 2019, Computer Graphics Forum 38(3). Three Idyll articles: rhythm lesson 10,000 sessions, dimensionality reduction 7,055, guitar tuning 55,193. No control group and no measure of learning. Unless noted, statistics cover desktop readers only; many features were disabled on mobile. Rhythm lesson: median time on page 86 seconds; 85% interacted with the interactive content; 73% reached the end; mobile median 20 seconds, and 10% triggered the example audio at the end. Dimensionality reduction: median 116 seconds; 76% reached the end, of whom 59% clicked a button to switch algorithms, about 45% of all readers; median interactions with the slider and the details button 0; about 5% alternated between the slider and hovering over the images; about 15% clicked a details button and then interacted with the figure; on mobile 48% reached the end, of whom 38% used the feature to see other algorithms. Guitar tuning: median 78 seconds; median 22 plucks; 23% tuned all six strings; 14% played the beat-frequency example and 2% changed its setting; only 45% reached the last section; on mobile 34% tried tuning and 6% tuned all six strings. Mobile median time on page was 23–52% of desktop. The discussion section rounds the 59% of readers who reached the end to "roughly half"; the roughly 45% of all desktop readers is a different denominator.

Hohman, Conlen, Heer, Chau (2020), Distill. A review with practitioners' reflections and no data of its own; the material is over sixty interactive articles chosen by the authors (selection not described). The two "have been shown" sentences in the introduction carry no citations; the end says there is limited empirical evaluation. When citing Kim 2017 it does not mention the text-condition results; it cites a study on showing other people's predictions, which found no benefit to recall. Costs are described only qualitatively; the only time figure is four months of collaboration per issue of The Parametric Press. The authors describe themselves as practitioners and tool builders, and close by saying they believe in the potential of interactive articles.

Tse (2016), conference slides by The New York Times' deputy graphics director, 24 slides, text only, with no speaker notes and no numbers. Three rules plus "Most visuals are static" and "We still do interactives, but the bar is now VERY high." The title is marked "2016 edition."

Aisch (2017), personal blog. Source of the number: his own measurement of the click rate on a prominent button on a couple of graphics from 2015; he doesn't say which pieces or how large the sample was. Path of the retelling: conference → another talk → Medium → FastCo.Design. None of the three examples has usage figures.

Distill editorial team (2021), "Distill Hiatus," drafted by Chris Olah, Nick Cammarata, Sam Greydanus, and Janelle Tam. A one-year hiatus that may be extended indefinitely. The 50-plus hours were editorial help on early articles and have since come down. No figures for submissions, publications, or number of editors.

Red Blob Games (Amit Patel), three production pages. Making of the A* introduction (2014, not updated after the diagrams were redone in 2020): 1 to 5 months per tutorial, including learning the algorithm; about 20 diagrams on that page; the diagram system was rewritten twice partway through. Making of the line-drawing tutorial (2017): "a medium sized project," with no time given; the basic form of interaction is letting the reader change inputs and see outputs; two of the twelve finishing touches concern touch input (a drag handle made of a small visible circle plus a large invisible circle that catches events; scrubbable numbers being too small on touch devices). Diagram structure (2018): start by redrawing everything and rerunning everything, and optimize only when it gets slow. Open questions the author lists include: which problems suit this kind of explanation; whether letting readers manipulate data but not code shortchanges them; how to guide readers to educational points in the parameter space; how even the text has to adapt for mobile; and whether to have a version that doesn't rely on JavaScript.

References

Each source was read in full and has an extraction record; parts I could not read are noted in parentheses.

• D'Angelo et al. (2014), Simulations for STEM Learning: Systematic Review and Meta-Analysis (SRI International report, no DOI; screenshots in Appendix C not viewed).
• Berney & Bétrancourt (2016), Does animation enhance learning? A meta-analysis (published version in the University of Geneva repository; text conversion drops minus signs, so numbers follow the PDF layout).
• Castro-Alonso, Wong, Adesope, Ayres & Paas (2019), Gender Imbalance in Instructional Dynamic Versus Static Visualizations: a Meta-analysis (published version in the Erasmus University repository; the last page of Table 2 not checked against the layout).
• Ploetzner, Berney & Bétrancourt (2021), When learning from animations is more successful than learning from static pictures: learning the specifics of change (open-access published version; OSF data not downloaded).
• Mosca, Ottley & Chang (2021), Does Interaction Improve Bayesian Reasoning with Visualization? (arXiv:2103.01701v2 preprint; not checked against the published version, DOI 10.1145/3411764.3445176; supplementary materials not read).
• Hundhausen, Douglas & Stasko (2002), A Meta-Study of Algorithm Visualization Effectiveness (published version on a coauthor's personal page; the DOI printed on the first page is 10.1006/S1045-926X(02)00028-9).
• Kim, Reinecke & Hullman (2017), Explaining the Gap: Visualizing One's Predictions Improves Recall and Comprehension of Data (ACM published version on a coauthor's lab site; supplementary materials on GitHub not read).
• Conlen, Kale & Heer (2019), Capture & Analysis of Active Reading Behaviors for Interactive Articles on the Web (publisher's layout on the authors' lab site; supplementary materials and raw logs not read).
• Hohman, Conlen, Heer & Chau (2020), Communicating with Interactive Articles (published in Distill; the article list in Table 1 not read).
• Tse (2016), Why We Are Doing Fewer Interactives (slides the author posted on GitHub; slide text only, no speaker notes).
• Aisch (2017), In Defense of Interactive Graphics (original on the author's personal site; images not viewed).
• Distill Editorial Team (2021), Distill Hiatus (original announcement on the Distill site).
• Red Blob Games (Amit Patel): Making of: Introduction to A*, Making of: Line drawing tutorial, Diagram structure (original pages on the author's site; the interactive figures themselves not viewed).
• Posts on this site: Writing the Examples and Openings of a Teaching Article, When a Teaching Article Should Use a Diagram, Pairing a Teaching Article with Video, When a Reader Wants the Video Mid-Article, Writing Questions and Wording in a Teaching Article.