This lecture unpacks the SimCLR loss into a B×B similarity matrix: each row is one softmax classification problem whose answer is the diagonal entry. The diagonal gets pulled up and everything else gets pushed down, so the model learns that similar things have nearby vectors without any labels. The second half uses the same vectors for semantic retrieval, then connects that to RAG.
CS231N Lecture 12 asks whether we can learn good representations without huge manually labeled datasets. The answer comes in three parts. First, pretext tasks that generate labels from image transformations: predicting rotation, solving jigsaw puzzles, inpainting, colorization, and MAE with a 75% mask ratio. Second, the more general contrastive learning: the InfoNCE loss, SimCLR with its large batches, MoCo, which decouples batch size from the number of negatives with a queue, and sequence-level CPC. Third, DINO, which needs no negatives: a student predicts the output of a momentum teacher, and centering plus sharpening prevent collapse. The core evaluation is linear probing: freeze the encoder and train only a linear classifier.
A lot of data does not mean a lot of annotation. This lecture asks: with no annotations, how do you learn useful representations from data? The shared idea is to find the latent factors behind the observations. An auto-encoder compresses the input into a short code and reconstructs it. A VAE additionally requires the code to follow a Gaussian distribution, so it can generate by sampling from the distribution. Dual learning lets two tasks that run in opposite directions serve as each other's feedback. Self-supervised learning divides into self-prediction (hide a part and guess it back) and contrastive learning (pull similar ones together, push dissimilar ones apart). CLIP does contrastive learning on image-text pairs, so image classification needs no separate training; DALL·E 2 then uses its representations to generate images. The lecture ends with multimodal models: projecting images into the language model's space, and streaming interaction that listens while deciding when to speak.
Representation learning compresses raw data into a 'useful' vector: autoencoders force a meaningful latent space via reconstruction, VQ discretizes it into a codebook, and self-supervision turns 'mask-and-reconstruct' into free supervision.
Similarity-based representation learning does not reconstruct input; it directly shapes latent geometry: pull same-class representations together, push different ones apart. InfoNCE turns this into 'spot the positive among negatives', and alignment / uniformity give it interpretable metrics.
Take a net to infinite width and its random-init output becomes a Gaussian process (NN–GP); its training dynamics freeze into the Neural Tangent Kernel (NTK). This theory analyzes nets and, in reverse, guides us to design the 'right inductive bias'.
Lecture 1 uses an alien autoencoder, the scope of AI and ML, and AI history to establish the course's coordinate system: an intelligent system turns inputs into representations and decisions under uncertainty.
Lecture 11 expands a logistic unit into a multilayer network: linear layers produce z, activations produce a, and multiple neurons jointly learn a feature transform trained through a final loss.
Lecture 15 splits a pretrained model into representation g and task head h: freeze g and train only the head, or fine-tune some or all parameters at a smaller learning rate depending on data volume and source-target distance.