Skip to content
All tags

#pretraining

2 posts
tech MIT 6.7960 導讀 (Fall 2024 OCW)

MIT 6.7960 L18: Transfer Learning — Pretraining, Feature Extraction, and Fine-Tuning Strategies

Transfer learning's core insight is 'features learned on big data are good general-purpose representations': freeze the backbone and train only a linear head when downstream data is tiny; full fine-tune when data is plentiful; reach for LoRA / adapter when compute is tight. SimCLR and MAE removed the need for upstream labels and pushed downstream quality another notch.

#mit-67960#deep-learning#transfer-learning#pretraining#fine-tuning#simclr#mae#self-supervised#fall-2024
ai deep-dive Reading Stanford CS224N

CS224N Lecture 7: Pretraining, Subwords, and In-Context Learning

Lecture 7 decomposes pretraining into scalable data, subword tokenization, three model objectives, and in-context learning. A general self-supervised objective yields reusable representations; downstream signals specify their use.

#cs224n#pretraining#language-model#tokenization#stanford