Training an LLM from Scratch: A Project Spectrum from $0.4 to 65B

🌏 中文版

Calling an LLM through an API and training one from scratch are the two ends of the same map. Everyone has tried the former; a few years ago the latter was a "big-lab-only" activity — billion-dollar budgets and thousands of GPUs. MiniMind pulled that line down to about $0.4 (the cost of renting one RTX 3090 for two hours), while at the other end LLM360 K2 shows what "fully reproducible" looks like at industrial scale: 65B parameters, 1.4T tokens. This is the introduction to the "Training an LLM from Scratch" series (11 articles): first we define what we selected and why, then lay out the cost spectrum, the coverage matrix, and the bias annotations, and finally the learning path.

What this series covers

The population definition for "training an LLM from scratch": projects that publish the complete training pipeline — code plus data or a full recipe — where the model is pretrained from random initialization. We exclude three kinds of projects commonly mistaken for "from scratch": weight-only releases (models like Llama), fine-tuning only (Llama-Factory and the like), and pure inference frameworks. In other words, this series only includes projects where "you get every button" — training hyperparameters, the data recipe, and checkpoints at each stage.

Why train from scratch: the learning-vs-product tradeoff

The same motivation, "I want to understand LLMs," grows into two very different shapes at either end of the spectrum. This is the most important classification axis of the series:

• Learning-oriented, typified by MiniMind: 64M parameters, about $0.4 and 2 hours, 56.8k stars, with the goal that "everyone can read every line of code." Its value is not the model itself — 64M is useless for any real task — but the decision chain it lays open: why 8 layers, why a 6,400-token vocabulary, why GRPO is more stable than PPO.
• Product-oriented, typified by LLM360 K2 (65B / 1.4T tokens): the training goal is to compete with commercial models on real tasks, and "full reproducibility" is its biggest asset — for any training decision you want to verify, you can find the original code, data, and intermediate checkpoints.

In between there is a whole pricing ladder: learn algorithms from Karpathy's minimal codebases, learn data efficiency from YuLan-Mini, learn industrial transparency from the OLMo family. The decisive question is one sentence: is your question "why is this written this way," or "can this be reproduced"? — pick the cheap tutorial project for the former, and for the latter pick the most open project, not the cheapest one.

The cost spectrum

The same goal can cost six orders of magnitude differently: budget, parameter scale, and openness all shift together.

A few numbers worth remembering: Karpathy spent roughly $43,000 training GPT-2 in 2019; nanochat's "time to GPT-2" speedrun has squeezed that to the ~$100 level in under 4 hours (the early-2026 official record is ~$48 / 2 hours; spot pricing varies). YuLan-Mini reaches parity or better than 7–18T-token baselines on math and code with 1.08T tokens. SmolLM3 shows a 3B model can ingest 11.2T tokens without saturating — TinyLlama's 1.1B eating 3T without saturation was the precedent.

Coverage matrix

Framed as a matrix, you can see which dimensions this series has a dedicated part for and which only get a passing mention:

Retired cases: we mention Pythia (EleutherAI's 154-checkpoint research suite) and TinyLlama (1.1B, 3T tokens), not because they broke, but because they finished their job and retired gracefully — Pythia's value lies in the intermediate checkpoints it left for research; TinyLlama proved "small models can also eat tokens to saturation." They are "past-tense tools still in use," used here only as reference points.

Bias note: what this map omits

• GitHub-star-selected population: selection leans heavily toward English and Chinese circles (GitHub's star mechanism inherently favors large language markets), so projects from non-English, non-mainstream communities are easily missed — LLM-jp has a complete Japanese ecosystem and high openness but did not make the cut on stars and mindshare alone. This is a known gap in this survey. (Update): this gap has been closed by part 7 — the two national-scale cases, Apertus and LLM-jp, are now part of the series.
• Retired cases are only research-oriented Pythia/TinyLlama: "retired" here means "finished the job and got overtaken," not failed. Pythia's 154 checkpoints remain tools for studying learning dynamics; TinyLlama delivered the experiment "a 1B model trained on 3T tokens." Including them shows that obsolescence in this category usually means someone did something to completion.
• Architecturally one-sided: every selection is a Transformer. RWKV and Mamba-style non-attention sequence models were once hot alternative paths but aren't part of the "fully open pipeline" selection here (or the main toolchains didn't follow) — for alternatives, CS336: Attention, MoE, and Mamba on this site is an existing match.

The arc (11 posts)

$0.4 ─────────────────────────────→ 65B (hundreds of thousands of dollars)
teaching / readability ────────→ product / reproducibility
English → Chinese → academic data-efficiency ──→ industrial transparency

How to read this

• Hands-on readers: parts 1 (MiniMind) → 2 (Karpathy) → 3 (Chinese community). Run the README once and you will have first-hand experience with training "your own" model from scratch.
• Research readers: parts 4 → 5. For a solid theoretical base, first read the CS336 overview — the reference course of this series — then come back for the cost and openness reality.
• Decision-makers / budget holders: read part 6's decision framework first, then backfill the cases you need.
• Policy / compliance angle: part 7. How non-English communities trade national-scale collaboration and data governance for sovereignty and legitimacy, with RWKV as the architecture contrast.
• Efficiency / deployment angle: part 8. If you want to train from scratch under tight parameter budgets or for multimodal edge scenarios, OpenELM and MiniCPM are the loadout.
• Framework angle: part 9. If you want mature tooling for a from-scratch rewrite instead of hand-writing every line, LitGPT is the entry point.

Each part is self-contained, but the arc's order follows the graduation of cost magnitude and data complexity; following it is the least tiring path.

Bottom line

The two ends of this spectrum answer different questions: MiniMind proves "cheap can teach," OLMo 3 / LLM360 prove "expensive can verify" — the common denominator is openness, just with different objects (code vs data + logs). The first published part is MiniMind; the rest climb the cost ladder, and the last part re-examines "when not to spend this money at all." The site's CS336 series and OLMo: full openness give you the theoretical base and the previous generation's full picture; Marin's 535B hero run shows another viable route for an individual pushing the limit. Any order works, but we suggest starting with MiniMind — it's the only one you can start on tonight.

References

• MiniMind GitHub
• karpathy/nanochat GitHub
• karpathy/llm.c
• Llama-Factory
• YuLan-Mini GitHub (RUC-GSAI)
• OLMo GitHub (AI2)
• Olmo-3-1125-32B model card (Hugging Face)
• LLM360 K2 training code (k2-train)
• SmolLM3-3B model card (Hugging Face)
• TinyLlama GitHub (jzhang38)
• Pythia GitHub (EleutherAI)
• baby-llama2-chinese GitHub
• ChatLM-mini-Chinese GitHub
• Steel-LLM GitHub