Skip to content
All tags

#chinchilla

2 posts
techguideReading MIT 6.7960 Deep Learning

Scaling Laws: With Ten Times the Compute, Grow the Model or the Data? | MIT 6.7960 Lecture 20

Test loss follows an approximate power law in data, parameters and compute, but the exponents are small: Kaplan et al. measure a data exponent of about 0.095, so doubling the data lowers the loss by only 6% and halving it takes over 1,400 times more. Two papers disagree on how to split compute: Kaplan gives parameters ∝ C^0.73 and data ∝ C^0.27, Chinchilla about half each. The public recording stops at the second finding; everything after that is written here from the slides, without the lecturer's spoken explanation.

#mit-67960#deep-learning#scaling-laws#power-law#chinchilla#batch-size#fall-2024
aideep-dive認識 AI 模型

Scaling Laws: How Big Should a Model Be, and Why Bigger Isn't Always Better

Scaling laws show that loss decreases predictably with more parameters, data, and compute — following power-law relationships. The Chinchilla paper's key finding: most models were too large and undertrained. Given the same compute budget, training a smaller model on more data produces better results. This reshaped the entire industry's training strategy.

#scaling-laws#chinchilla#compute-optimal#training#ai-model#llm