Modern Transformers: What Position Rotation and KV Sharing Change | CMU 10-423 Lecture 4
Attention needs both order information and a manageable memory footprint during generation.
#pre-training
16 postsReading Transformers, T5, LLaMA, and GPT-3 Like a Systems Engineer | CMU 11-868 Lectures 6–7
11-868 spends only two lectures on the model itself. L06 breaks the Transformer into embeddings, multi-head attention, FFN, LayerNorm, and residuals; L07 uses T5, LLaMA, and GPT-3 to show what modern LLMs changed. For a systems engineer the point is to remember the shapes: GPT-3 175B has 96 layers, d_model 12288, a 2048-token context, and trained on 300B tokens; LLaMA 65B has 80 layers, d_model 8192, and trained on 1.4T tokens. Those numbers set the workload for every acceleration, parallelism, and serving lecture that follows.
NTHU NLP Guide 8: ELMo, BERT, T5, BART, GPT and the Three Roads to Pretraining | NTHU Hung-Yu Kao Natural Language Processing Topic Guide
A guide to the BERT and its Family unit of Prof. Hung-Yu Kao's NLP course at NTHU (Fall 2025). It starts from "I record the record": Word2Vec and GloVe give both records the same vector. ELMo fixes this with a bidirectional LSTM language model. With Transformers, pretraining splits into three roads: encoders (BERT: MLM plus NSP, strong at understanding, weak at generation), encoder-decoders (T5's span corruption, BART's five noise types), and decoders (GPT: pure next-token prediction). The last part covers GPT-3's in-context learning and scaling laws, and why decoders became today's dominant backbone.
The Bill for Training an LLM, and Where Pretraining, SFT, and LoRA Spend It | Stanford CME295 Lecture 4
CME295 Lecture 4 splits LLM training into two stages: pretraining on trillions of tokens (Llama 3 used 15 trillion), then SFT on thousands to millions of demonstrations so the model stops continuing text and starts answering. In between sits a map of memory savers (ZeRO, FlashAttention, mixed precision); the lecture closes with LoRA and QLoRA, which let people without big GPUs finetune, with QLoRA cutting VRAM by about 16x on a 65B model.
What GPT, BERT, RoBERTa, ELECTRA, T5, BART, and Distillation Each Change in Pretraining | Stanford CS224U Topic Guide
CS224U Spring 2023 tells the story of the Transformer families through BERT's four known limitations. RoBERTa addresses the first (optimization was only partly explored). ELECTRA addresses the second and third (the [MASK] mismatch, and only about 15% of tokens giving a learning signal per batch). XLNet addresses the fourth (the assumption that masked tokens are independent of each other). GPT changes the objective and the mask, T5 and BART change the architecture and how inputs are corrupted, and distillation changes model size. The course's 2023 view: autoregressive architectures have taken over, but bidirectional models may still have the edge for representation.
Training Small LLMs From Scratch in the Chinese Community: Corpus, Tokenizers, and Three Open-Source Projects
A close look at three open-source projects training LLMs from scratch in the Chinese community — baby-llama2-chinese (218M, 63.4B tokens), ChatLM-mini-Chinese (0.2B T5, 10.23M dialogues), and Steel-LLM (1.12B, 1T tokens, 8 months) — comparing corpus strategy, tokenizer decisions, and community ecosystem. Honest evaluation included: baby-llama2 scored a bottom-ranking 21 in MiniMind's side-by-side test; ChatLM has the strongest knowledge (62) but weak coding.
How to Spend Every Parameter: OpenELM's Layer-wise Scaling and MiniCPM's Three-stage Unfreezing
OpenELM uses layer-wise scaling to shift parameters toward layers near the output; with 1.08B parameters and 1.5T tokens it beats OLMo 1.2B (+2.36% on the LLM360 average) despite OLMo training on 3T tokens. MiniCPM trains multimodal small models from scratch with a three-stage unfreezing recipe (Resampler first, vision encoder next, everything unfrozen last); MiniCPM-V 4.5 reaches sub-30B SOTA on VideoMME with only 8B parameters, and 4-bit quantization squeezes fp16's 16–17GB memory footprint down to about 5GB for phones.
Karpathy's nano lineage: nanoGPT, llm.c, and nanochat
A tour of Karpathy's three teaching repos: nanoGPT (2022, ~300 lines each to reproduce GPT-2 124M), llm.c (pure C/CUDA training), and nanochat (2025-10, one speedrun.sh from tokenizer to WebUI). $100 and 4 hours on 8×H100 buys a chatty model; GPT-2-grade capability is now down to about 2 hours and $48.
The framework for training from scratch: LitGPT's no-abstraction rewrites, pretrain flow, and the TinyLlama track record
LitGPT (Lightning AI, ~13,600 stars, Apache 2.0) rewrites 20+ mainstream LLMs — Llama 3, Qwen2.5, Phi 4 — from scratch as single-file, no-abstraction implementations, with a full pretrain / finetune / evaluate / serve CLI. TinyLlama (1.1B parameters, 3T tokens) was trained on this codebase. This post breaks down how it differs from MiniMind, how to actually use it, and where it stops.
MiniMind: Train an LLM From Scratch for $0.40
MiniMind is an open-source project for training LLMs from scratch: a 64M Dense model and a 198M-A64M MoE model that run the entire chain — Pretrain → SFT → LoRA → DPO → PPO/GRPO/CISPO → Agentic RL — in ~2 hours on a single RTX 3090 at roughly 3 RMB (~$0.40). Every core algorithm is implemented natively in PyTorch with no high-level wrappers.
National-Team LLM Training: Apertus' Compliance Route and LLM-jp's Japanese Ecosystem
Switzerland's Apertus (8B/70B, 15T tokens, 1,000+ languages) filters opt-outs and personal data before training to satisfy the EU AI Act; Japan's LLM-jp consortium shipped LLM-jp-4 (12T tokens) in April 2026, claiming wins over GPT-4o and Qwen3-8B on standard benchmarks. Both prove that from-scratch training outside the English sphere is a data-governance problem, not a technical one — plus a note on RWKV-7 as the non-Transformer alternative.
How fully transparent LLMs are built: OLMo 3's model flow and LLM360 K2's 360-degree openness
"Open-source LLM" is a spectrum: weights-only (Llama), weights plus data (most fully open projects), or data order, intermediate checkpoints, and training logs all released (LLM360 K2, OLMo 3's model flow). This piece unpacks the two projects that pushed transparency furthest: OLMo 3 shipped the first fully open 32B thinking model in November 2025, and K2 is the first 65B-class model whose checkpoints even include optimizer states.
Training an LLM from Scratch: A Project Spectrum from $0.4 to 65B
Open-source projects have pushed the cost of training an LLM from scratch absurdly low: MiniMind runs the full PreTrain-to-RL pipeline for about $0.4 (2 hours on a single RTX 3090), while at the other end OLMo 3 and LLM360 K2 publish everything — data, code, and stage-by-stage checkpoints of 65B models. This series walks the whole project spectrum from $0.4 to 65B in 11 articles, and flags where the map is biased.
When to Train an LLM From Scratch: A Decision Tree and the Full Cost Ladder
Training from scratch only makes sense in three cases: you want to learn how training works, you have 10B+ clean tokens no open model has seen, or you need a fully transparent training process for research. Otherwise fine-tuning or RAG is almost always cheaper. This post collapses the series' main routes into one cost ladder and a decision tree.
YuLan-Mini: Squeezing a 2.4B Flagship-Scale Small Model Out of 1.08T Tokens
YuLan-Mini is a 2.4B open-source model from Renmin University's AI Box lab, trained on 48 A800 GPUs with only 1.08T tokens — scoring 37.8 on MATH-500 and 64.0 on HumanEval, beating Qwen2/Qwen2.5 peer models trained on 7T–18T tokens at math and code. What's public isn't a slogan: per-phase data mixes, pre-annealing optimizer states, and even W&B logs of the ablation studies.
Pre-training, SFT, RLHF: Three Stages That Turn a Text Predictor into a Useful Assistant
Every LLM goes through three training stages: pre-training reads the internet to learn language, SFT uses example conversations to learn the format, and RLHF uses human preferences to learn what a good answer looks like. The gap between a base model and a chat model is what the last two stages do.