How Can Statistics and ML Evaluation Be Rerun to Reach the Same Conclusion?

中文版

All the formulas, tests, models, and simulations in this series share one requirement: someone should be able to trace how the result was produced. If a report only shows a figure or an accuracy value, without data version, cleaning rules, seed, code, and environment, a later rerun may differ and nobody will know why.

This post is about reproducible workflow. Exams may not always ask about engineering details directly, but statistical reports, experimental design, and ML/AI evaluation all need this ability: data, methods, computation, and conclusions must be rerunnable, inspectable, and open to challenge.

What Problem This Solves

Reproducibility protects the evidence chain.

A statistical conclusion usually follows this path:

raw data -> cleaning -> feature engineering -> analysis/model -> metrics -> tables/figures -> conclusion

If any step is missing, the result becomes a verbal impression. You may remember what you did, but a reader, instructor, colleague, or future version of yourself cannot inspect it.

ML/AI evaluation has even more moving parts:

dataset version
train/test split
prompt or config
model version
random seed
metric definition
scoring script
external API time

The goal is that every result can be traced back to its inputs and code.

Core Intuition

Reproducibility is not saving the final screenshot. A screenshot only proves that a result was once visible. It does not explain how the result was produced.

A better workflow treats the report as an output artifact, not a manually assembled object. Data cleaning, analysis, figures, and metrics are generated by code. Every number in the report can be traced to a script and a dataset.

A minimal checklist is:

data: raw data, cleaned data, data version
code: cleaning code, analysis code, evaluation code
config: parameters, model, prompt, threshold
randomness: seed, sampling rule, split rule
environment: package versions, runtime environment
outputs: tables, figures, metrics, logs
README: how to rerun

These items are not ceremony. They are tools for locating errors.

Formula and Mechanism

A simple evaluation can be described with a manifest:

dataset:
  name: customer-support-eval
  version: 2026-08-29
  split: test-v3
model:
  name: model-b
  version: 2026-08-20
evaluation:
  metric: task_success_rate
  scorer: rubric-v2
  seed: 42
outputs:
  report: reports/eval-2026-08-29.md
  raw_results: results/eval-2026-08-29.jsonl

Statistically, metric definition is critical. Accuracy, precision, recall, F1, pass rate, win rate, and human preference answer different questions. If the metric changes, the numbers cannot be compared directly.

Data splits must also be fixed. If one run uses a random split and the next run redraws the split, the score difference may come from a changed test set. At minimum, record:

split rule
random seed
train/validation/test identifiers
excluded cases and reasons

If an external model API is used, record model name, provider version information when available, date, temperature, top_p, and tool settings. These can affect outputs.

Worked Example: A Model Comparison That Cannot Be Diagnosed

Suppose you see this model comparison:

At first glance, B is two percentage points higher. But the report omits dataset version, split rule, prompt, scoring code, and seed.

One week later, the rerun says:

Now you cannot tell what changed:

• Did the dataset change?
• Was the test set resampled?
• Did the metric definition change?
• Did the model version update?
• Was the scoring prompt different?
• Did random generation create variation?

If the first run had a manifest and raw results, you could debug step by step. Check whether test IDs are identical. Check the scoring script hash. Check model version and parameters. That is the value of a reproducible workflow: it does not guarantee identical results every time, but it tells you where differences come from.

An exam or report answer can write:

The result should include data version, split rule, preprocessing pipeline, metric definition, model version, random seed, and raw item-level results. Otherwise, we cannot tell whether the score difference came from model improvement, data changes, or evaluation changes.

Where This Shows Up in ML/AI

In ML/AI teams, reproducibility becomes several engineering practices:

• eval pipeline: fixes dataset, model, metric, scorer, and output format.
• experiment tracking: records each run's parameters, results, and artifacts.
• model registry: stores model versions, training data, and deployment state.
• dataset registry: stores data versions, labeling rules, and split rules.
• CI evaluation: reruns a fixed test set when a model or prompt changes.

For LLMs and agents, raw traces matter even more. A pass/fail result is not enough because failures can come from retrieval, tool calls, planning, parsing, external APIs, or the final answer. Saving step-by-step traces connects statistical outcomes to system behavior.

How Sources Are Used

• Official past-paper PDFs are used only for years, subject names, and problem statements; two years of questions are not treated as the full future scope.
• The grad-exam-prep pages are used for study-route, question-index, and practice-flow alignment, not as official solutions.
• OpenIntro, OpenStax, and other open textbooks are used to verify formulas, definitions, assumptions, and common derivations.
• Stanford CS109 and scikit-learn documentation are used to connect each topic to ML/AI training, evaluation, experiments, and uncertainty reporting.

Problem Recognition Hints

• If the problem asks about reproducibility, answer with data, code, environment, seed, and output artifacts.
• If the problem gives benchmark scores, ask for dataset version and metric definition.
• If the setting uses external models or APIs, record model version, parameters, and execution time.
• If a rerun differs, debug data, code, randomness, environment, and external services separately.

Common Mistakes

• Saving only the final report without the data and code that produced it.
• Comparing two scores when the train/test split was not fixed.
• Keeping the same metric name while silently changing the definition or threshold.
• Reporting only averages and not saving item-level results.
• Running LLM evaluation without recording model version, temperature, prompt, and tool settings.

Practice

1. List the minimum items needed for a reproducible statistical analysis.
2. Why does saving only the final chart, but not the code that produced it, make analysis hard to audit?
3. Write the steps of an ML evaluation pipeline: data split, model version, metric calculation, report output.
4. If the same model reruns with accuracy moving from 0.86 to 0.83, what sources would you inspect?
5. In LLM agent evaluation, why are raw traces more diagnostic than a single pass rate?

What Comes Next

A reproducible workflow preserves the evidence chain. The next post turns that evidence into an ML/AI evaluation report: not just scores, but decision-ready conclusions.

Section-Level Source Map

• OpenIntro, OpenStax, and Stanford CS109 support inspectable statistical workflows around sampling, estimation, and analysis.
• scikit-learn supports data splitting, model training, and evaluation workflows; this post turns those workflows into reporting and versioning requirements.
• This post connects reproducibility to eval pipelines, experiment tracking, dataset versioning, model registries, and raw traces.

References

• Reproducible statistical workflows, dataset versioning, random seeds, metric definitions, and evaluation pipelines: OpenIntro Statistics
• NTU Library Past Exam System: Institute of Information Management (in Mandarin)
• NTU IM statistics prep page (in Mandarin)
• NTU IM past-papers page (in Mandarin)
• OpenIntro Statistics
• OpenStax Introductory Statistics 2e
• Stanford CS109
• scikit-learn Model Evaluation