Skip to content

Conditional Generative Models — cGAN, cVAE, and Classifier-Free Guidance | MIT 6.7960 Lecture 16

Aug 30, 20261 min
Home›tech
TL;DRThe key to conditional generation is 'feed y into the model': cGAN concatenates y into G/D; cVAE passes y to both encoder and decoder; in diffusion, Classifier Guidance uses gradients from an external classifier to push samples toward a class, while Classifier-Free Guidance trains conditional + unconditional together and linearly combines them at inference — the latter is the standard weapon behind Stable Diffusion and Imagen.
#mit-67960#deep-learning#conditional-generative#cgan#cvae#classifier-free-guidance#diffusion#text-to-image#fall-2024
Series: Reading MIT 6.7960 Deep Learning (21 / 28)
Variational Autoencoders (VAE) — ELBO, Reparameterization Trick, and Latent Representations | MIT 6.7960 Lecture 15Out-of-Distribution Generalization — Distribution Shift, Spurious Correlations, and Three Practical Remedies | MIT 6.7960 Lecture 17

🌏 中文版

Video status: Videos included. Source details

Source: based on MIT 6.7960 Fall 2024 OCW (corresponds to OCW Lec 16). Videos, slides, and assignments are all open on MIT OCW. This lecture is taught by Phillip Isola.

From "generate" to "conditional generate"

L14–L15 were about generating x, but in practice we more often need a condition y, and generate the matching x: y might be a class label, a sentence, or another image. Conditional generative models answer: how do we estimate and sample from p(x | y)?

cGAN: stuff y into G and D

The most direct approach is conditional GAN (Mirza & Osindero, 2014): feed the condition y concatenated with z (noise) into the generator G, and feed y together with real/fake samples into the discriminator D. The training objective is unchanged:

min_G max_D  E[log D(x|y)] + E[log(1 − D(G(z|y)))]

Intuitively: the generator must "generate something that looks right given y"; the discriminator must "judge (x, y) real or fake." Conditional MNIST generation (pick the digit 0–9) is the classic toy experiment.

Minimal cGAN training skeleton:

import torch, torch.nn as nn
G = nn.Sequential(nn.Linear(z_dim+num_classes, 256), nn.ReLU(),
                  nn.Linear(256, 784), nn.Sigmoid())
D = nn.Sequential(nn.Linear(784+num_classes, 256), nn.LeakyReLU(),
                  nn.Linear(256, 1))
def embed(y, K):           # one-hot
    e = torch.zeros(y.size(0), K, device=y.device); e[range(y.size(0)), y] = 1; return e
# training: D sees (x, y); G produces x from (z, y)

cVAE: stuff y into encoder / decoder

For L15's VAE the same idea: conditional VAE treats y as an extra input to both encoder and decoder,

q(z | x, y),   p(x | z, y),   p(z)

The ELBO becomes E_q[log p(x|z,y)] − KL(q(z|x,y) ‖ p(z)). In practice y can be a class one-hot, a text embedding, or coordinates. This is the foundation for later text-to-image VAEs and inpainting variants.

Conditional control in the diffusion era

Video note: classifier guidance and classifier-free guidance do not appear in the Lec 16 transcript; the video presents the general recipe of feeding the condition (e.g. text) into the denoiser, generator or discriminator. The guidance details below are this site's supplement.

Diffusion models split generation into "iterative denoising" (covered in L14); conditional generation has two main routes:

Classifier Guidance (Dhariwal & Nichol 2021)

Train a conditional diffusion p_θ(x_t | x_{t+1}, y) and separately train a classifier p(y | x_t); during the reverse process add a gradient term that pushes the sample toward the specified class:

x_{t−1} ← x_t − γ · ∇_{x_t} log p(y | x_t) + noise

γ controls conditioning strength — larger means more obedient but less diverse. The downside: you need a separate classifier that works on noisy images.

Classifier-Free Guidance (Ho & Salimans 2022)

A cleaner solution: the same model trains both conditional and unconditional, implemented by replacing y with a special "null token" for the unconditional case. At inference, linearly combine:

ε̂ = (1 + w) ε_θ(x_t | y) − w ε_θ(x_t | ∅)

w controls how obedient the sample is. w = 0 is pure conditional, larger w pushes harder toward the condition (diversity drops). This is the standard weapon behind Stable Diffusion and Imagen.

The actual text-to-image workflow

Bringing the past few lectures together, today's text-to-image system is roughly:

  1. Text encoding (L11 / L12 representation learning): a CLIP-style text encoder turns the prompt into an embedding y.
  2. Conditional generation: latent diffusion (run diffusion in a VAE's latent space to save compute) + Classifier-Free Guidance to steer toward the prompt.
  3. Decoding: a VQ-VAE / KL-VAE decoder turns the latent back into pixels.

In other words, VAE latent + conditional diffusion + CFG is the basic recipe of current SOTA.

Trade-offs and common failure modes

  • CFG weight too high: images become oversaturated, textures distort, low diversity.
  • Prompt not specific enough: the model collapses to mean-image (a fundamental issue for the VAE family).
  • y and x misaligned: in fine-tuning, use LoRA / DreamBooth to "slot" a new concept into the model's condition space.
  • Controllability vs diversity: this is the fundamental trade-off in conditional generation; there is no silver bullet.

Course video sources

Rechecked against the live MIT OCW Fall 2024 gallery on 2026-10-10: the lecture number matches and the video is public and embeddable. Lecture numbering and topics follow OCW.

YouTube：MIT 6.7960 Fall 2024 — Lec 16. Generative Models: Conditional Models

Original videos: MIT 6.7960 Fall 2024 — Lec 16. Generative Models: Conditional Models

Course and recording entries:

Checked: 2026-10-10.

Content check: verified against the video transcript (2026-10-10): read the Lec 16 transcript (Phillip Isola, 1:21:31): the video is about conditional generation (it opens by previewing "conditional models"), covering conditional distributions and prediction problems (about 10%–36%), conditional GANs and cVAEs (about 50%–58% and 73%–77%), text-to-image and U-Net denoisers (about 71%–84%) and paired/unpaired image translation (about 89%–99%); classifier and classifier-free guidance do not appear in the transcript, so that section now carries a note marking it as this site's supplement.

Update Log

  • 2026-10-10: Added explicit video status and checked recording sources and access notes.
  • 2026-10-10: Rechecked video status. Live-checked the official course page: lecture numbers and recording links match and the videos are public, so the status is now “Videos included.”
  • 2026-10-10: Checked the video content against its transcript. Added a note to the diffusion conditioning section: classifier / classifier-free guidance are not in the video.

References

Related

Generative Models Basics — Density/Energy Models, GANs, Autoregressive, Diffusion | MIT 6.7960 Lecture 141 minHacker's Guide to Deep Learning — Practical Know-How to Make Nets Actually Obey | MIT 6.7960 Lecture 91 minApproximation Theory — Universal Approximation, Barron's Theorem, and Why Depth Matters | MIT 6.7960 Lecture 31 min