When to Train an LLM From Scratch: A Decision Tree and the Full Cost Ladder

中文版

Every route this series walked can be seen through one lens: MiniMind at $0.40, nanochat at $100, the Chinese community's small models, YuLan-Mini squeezing value from every token, and the industrially fully-open OLMo 3 and LLM360 — plus the national, efficiency, and framework routes that followed. This post collapses them into one question: which route should you take — or none at all.

The cost ladder

"Training from scratch" isn't one action; it's a spectrum spanning five orders of magnitude:

How to read this table matters more than the numbers: each step up swaps the bottleneck for a different one. At 3 RMB the gate is reading the code; at $100 it's operating a multi-GPU node; at YuLan-Mini it's data curriculum design; at OLMo scale it's systems engineering plus budget.

The decision tree

Ask what you actually want:

Understand how training works → MiniMind or nanochat. The former runs on one GPU in the Chinese ecosystem and covers through Agentic RL; the latter is a full-stack strong baseline in the English ecosystem, closer to industrial practice. Run both and your understanding of LLMs will surpass most people who've only fine-tuned. For the theory behind it, our Stanford CS336 course guides are the companion material.

Teach a model your domain data → almost never from scratch. If LoRA, full fine-tuning, or RAG can solve it, pretraining from zero trades tens of thousands of dollars for a worse result. The only exception: you hold 10B+ clean tokens no open model has seen (rare languages, proprietary domains, closed-industry corpora). In that case, study YuLan-Mini's data-efficiency recipe first — it proves data quality and curriculum design can save an order of magnitude of compute.

Research requiring transparency → OLMo 3 or LLM360. Weights, data, every checkpoint, training logs, all open. If you need to trace a model output back to its training data, this is the only route that can.

Train production models → use mature frameworks like verl and Llama-Factory, building on public recipes from OLMo and Marin. Don't rewrite a trainer.

When not to train

The honest list:

• "The model should know my documents" → RAG, one afternoon
• "The model should answer in my format" → fine-tuning
• "The model needs fresh knowledge" → RAG or re-fine-tuning
• "I want my own model" but the use case is unclear → clarify the use case first; the answer is usually an API

From-scratch pretraining in 2026 resembles building your own PC: enormous learning value, usually negative economic value. But "not economical" doesn't mean "not worth it" — that's the entire reason this series exists.

The takeaway

Recapping the spectrum: MiniMind pushed the entry bar down to where anyone can run it; Karpathy's nanochat shows the industrial-grade floor one person can maintain; the Chinese community projects prove the data ecosystem matters more than the architecture; YuLan-Mini takes data efficiency to its limit; OLMo and LLM360 define the ceiling of "fully open." These routes don't replace each other — they answer different questions. The only mistake to avoid is running a flagship with toy-project expectations, or dismissing educational projects because they can't do flagship work.

References

• MiniMind GitHub
• nanochat GitHub
• nanochat release discussion
• YuLan-Mini GitHub
• YuLan-Mini paper
• OLMo 3 — Allen AI Blog
• LLM360 K2 paper
• SmolLM3 — Hugging Face
• Self-Training a Small LLM From Scratch (2026 Guide)
• Series posts: order 0 intro, order 2 Karpathy lineage, order 3 Chinese community routes, order 4 YuLan-Mini, order 5 fully-open flagships, order 7 national teams, order 8 efficiency/edge, order 9 training framework