Skip to content
All tags

#mixture-of-experts

3 posts
ai deep-dive

How Marin Trains 535B: Scaling Ladder, MoE Expert Parallel, Harrier Data and Live W&B

Stanford Marin pre-registers a paloma macro-loss of 2.04 with a 5-rung Scaling Ladder at 1% cost, then trains 535B-A23B on 11×GB200 in public with live W&B telemetry — 847 training buckets already show the most teachable frontier run.

#marin#mixture-of-experts#scaling-laws#llm#open-source#stanford-cs336
ai deep-dive Reading Stanford CS336

CS336 Lecture 4: Attention Has Alternatives, and MoE Does Not Scale for Free

Lecture 4 studies two kinds of sparsity: linear/recurrent attention reduces sequence-length cost, while MoE activates only part of a model for each token. Both turn saved FLOPs into routing, balancing, communication, and kernel problems.

#cs336#llm#attention#mixture-of-experts#mamba
ai deep-dive Reading Stanford CS229

Large Language Models: Tokenization, Transformers, MoE, and SFT

Chapter 17 runs from next-token loss through Transformers, KV caches, MoE, and SFT, connecting an LLM's objective and architecture to its inference costs.

#cs229#llm#transformer#mixture-of-experts#sft