The first two lectures of the Scaling Up unit in CMU 10-423 Spring 2026 answer two questions. The second half of L15 covers scaling laws: Kaplan 2020 says 8x more parameters needs only about 5x more data, Chinchilla says scale both equally, and the Phi models and data-filtering scaling laws add data quality as a third axis. L16 covers MoE: feed-forward layers hold most of GPT-3's parameters, so split them into experts and send each token through only the top k. Memory follows total parameters, compute follows active parameters, and the price is load balancing and training stability. No programming homework covers this half of the course; quizzes, practice exam question 13, and the final project do.
CMU 11-868 (Spring 2026) spends two lectures on models too big for one GPU. L16 covers pipeline parallelism, which splits layers (GPipe micro-batches, 1F1B, interleaved stages), and tensor parallelism, which splits matrices (Megatron-LM's cuts for FFN, attention, and embeddings). The rule of thumb: TP inside a node, PP across nodes, DP on top. L17 treats MoE as a third way to split: each GPU holds different experts and replicates everything else. The price is all-to-all communication and load balancing, shown through GShard, DeepSpeed-MoE, and DeepSeek-V3.
The ADL Fall 2025 course page schedules seven TA recitations: Dev Infra (PyTorch, debugging) → NLP project lifecycle → the underlying logic of NLP projects → LLM LoRA training → LLM basics, architecture, and MoE → LLM inference and evaluation → LLM deployment. All ten videos are older recordings by Yen-Ting Lin from 2023 and 2024, reused in Fall 2025. The course page's five slide links all return 404; files with the same names still open under the Fall 2024 path, and Deployment has a video only. The first three sessions walk through the Hugging Face data → model → demo loop that HW1 needs; the last four cover training, inference, and serving LLMs.
Previous articles covered the landscape, training from scratch, long-horizon memory, and planning optimization. This one zooms out to see a complete system that threads all these insights together: Tongyi DeepResearch. Its core innovation is Agentic CPT — inserting an agentic mid-training stage between pre-training and fine-tuning, giving the model an inherent agent bias. MoE 30B parameters activating 3B, HLE 32.9 surpassing OpenAI o3.
Ling-3.0-flash-Fin (Ant Group): Released at the 2026 Inclusion·Conference on the Bund (2026-09-09), 124B total parameters with 5.1B active, 256K context (scalable to 1M), MIT open-source; built on Ling-3.0-flash's MoE architecture and long-context capabilities with continued financial-domain pretraining, tested across seven financial benchmarks including FinFIRST, FinSearchComp Verified, FinCRAFT, Finance Agent, APEX-Agents, SpreadsheetBench, and τ³-Banking; AA Intelligence Index v4.1.1 improved from 38 to 41; weights available on Hugging Face and ModelScope, OpenRouter offers a one-month free API
Laguna is Poolside's agentic coding model family: XS 2.1 packs 33B-A3B into a 36GB Mac, while S 2.1 brings 118B-A8B with 1M context to 70.2% on Terminal-Bench 2.1 and 40.4% on DeepSWE, both open under OpenMDW-1.1.
Ant Group's Ling model family deep-dive: 2025→2026 evolution timeline, Ling/Ring/Ming three-series strategy, architecture journey from Ling 1.0 to Ling 3.0, Ling-3.0-flash-Fin finance model, and an Agent developer's selection guide
Nex-N2.5 is Nex AGI's open agentic model family: mini scores 82.9 on OSWorld-G at 35B-A3B, Pro tops Claude Opus 5 with 87.4 at 397B-A17B, and Max leads the whole official table on BrowseComp with 92.6 at 1.6T, all open under Apache-2.0.
MiniMind is an open-source project for training LLMs from scratch: a 64M Dense model and a 198M-A64M MoE model that run the entire chain — Pretrain → SFT → LoRA → DPO → PPO/GRPO/CISPO → Agentic RL — in ~2 hours on a single RTX 3090 at roughly 3 RMB (~$0.40). Every core algorithm is implemented natively in PyTorch with no high-level wrappers.
Thinking Machines Lab (founded 2025 by Mira Murati, $2B seed at a $12B valuation) released Inkling in July 2026 under Apache 2.0 (975B total / 41B active params, 1M context, native multimodality, controllable thinking effort) plus a smaller Inkling-Small (276B / 12B), paired with the Tinker fine-tuning platform—turning customizability itself into the product.
Nearly every frontier open-source model in 2026 is MoE: Ornith 35B activates only 3B to beat 31B dense models, MiniMax M3 uses 456B total but 45.9B active to hit SWE-bench Pro 59%, DeepSeek V4 runs 1.6T total with 49B active. This post explains why MoE dominates coding and agentic benchmarks using four case studies.
MiniMax started as a consumer chat app company, then M2.5 scored 80.2% on SWE-bench Verified at 1/10-1/20 the cost of Claude Opus; M3 (456B total / 45.9B active) became the first open-weight model to clear 59% on SWE-bench Pro, with 1M context powered by their novel Sparse Attention mechanism.
DeepReinforce's Ornith 1.5 family, trained with self-improvement RL: the 397B flagship scores 86.0 on SWE-bench Verified, matching Claude Opus 4.8; the 35B-A3B activates only 3B parameters per token yet leads every coding benchmark in its class; the 9B runs on phones. MIT-licensed, fully open-source.
DeepSeek used MLA and MoE innovations to drive inference costs to an industry low. V4 Flash activates only 13B parameters while approaching frontier-model quality and ranks first by OpenRouter usage. This guide traces V1 through V4, the R1 reasoning branch, and how to choose each version.
GLM is Zhipu AI (Z.ai)'s open LLM family from Tsinghua's KEG Lab. GLM-5.3 (2026/08) lifts coding +50% over the previous generation, hits 84.5% on CyberGym ahead of Anthropic Mythos 5 and OpenAI GPT-5.6 Sol, and scores 60 on the Artificial Analysis Intelligence Index tied with Kimi K3 for open-source #1. The only frontier open model trained entirely on Huawei Ascend.
Grok is xAI's LLM family: founded July 2023, opened with a 314B MoE under Apache 2.0 in March 2024, and two and a half years later spans Grok 4.6 (500K, $2/$6, four reasoning levels), Grok 4 Fast (2M), Imagine for image/video, and Grok Build for terminal coding — its moat is distribution (X / grok.com / Tesla / Bedrock), not single-model supremacy. This post traces Grok 1→4.6, sub-line positioning, pricing, and licensing traps.
Kimi is Moonshot AI's LLM family, born from ultra-long context. Kimi K3 (2026/07) is the world's first open 3T-class model—2.8T params, 104B active, 1M context, scoring 60 on the Artificial Analysis Intelligence Index tied with GLM-5.3 for open-source #1. Its Kimi Delta Attention brings a 2.5× scaling efficiency gain.
Llama is Meta's open-source LLM family, with the largest enterprise deployment footprint and the most mature ecosystem. Llama 4 Scout (10M context) and Maverick (17B active / 400B total MoE) are the current open multimodal benchmarks, but Meta pivoted to closed-source Muse Spark in April 2026—Llama 4 is likely the last major open Llama, and its license is not truly open (Llama 4 Community License, separate license required above 700M MAU).
Mistral is Europe's most successful AI startup, cutting through the market with a 'smaller, faster, cheaper' strategy and European data-sovereignty positioning. Mistral Large 3 is Europe's strongest commercial LLM, Small 4 is the 24B efficiency king, and Medium 3.5 is the open Modified-MIT model optimized for agentic coding. Its moat is not technical scale but the 'European compliance' card.
Qwen is the most-downloaded model family on HuggingFace, spanning sizes from 0.8B to 2.4T. In August 2026, Alibaba open-sourced a Max-tier flagship for the first time (Qwen3.8-2.4T-A95B) — but swapped the customary Apache 2.0 license for custom terms. Meanwhile the other new release, Qwen3.8-27B, runs native vision on laptop-class hardware and is the only one shipping under Apache 2.0. This post traces the family from 2023 through generation 3.8, explains how the open line and the commercial line split apart, and helps you pick the right model at each tier.
2026 Q1 saw a full-blown open-source model explosion: on the LLM front, GLM-5, Kimi K2.5, and Qwen3.5 caught up with closed-source models; Embedding and Reranker are dominated by Qwen3 and BGE; speech has Voxtral TTS and Whisper V3; image has FLUX.2; and video has Wan 2.2 rivaling Sora. This is the complete navigation map.