🌏 中文版
Video status: recordings require sign-in or course authorization. This guide uses public slides and assignments; numerical examples are site-authored, not a lecture transcript.
A conventional autoencoder compresses an input into a vector and reconstructs it. A VAE instead assigns each input a latent distribution, connecting reconstruction and generation within a probabilistic model. We separate the two ELBO terms, calculate Gaussian KL, and identify the reparameterization gradient. The lecture's closing zero/few-shot section is covered in the ICL guide, not presented as a VAE mechanism.
Prerequisites are Gaussian means and variances, expectations, and autograd. Rather than merely adding a KL term, begin with the reason an approximate inference distribution is needed.
Generation and inference point in different directions
The generative model samples z∼p(z), then x from pθ(x|z), often choosing p(z)=N(0,I). Training observes x and asks which latent values explain it. The exact posterior pθ(z|x) is generally difficult to evaluate, so an encoder supplies a tractable qφ(z|x), such as a diagonal Gaussian with learned mean and log variance.
This q distribution performs approximate inference. Reconstruction can encode an image and decode its sampled z; unconditional generation samples z directly from the prior. Both use the decoder, but their latent inputs have different sources.
Derive the bound
For a fixed x, log pθ(x) equals ELBO plus KL(qφ(z|x)||pθ(z|x)). KL is nonnegative, so ELBO is a lower bound. Expanding it gives E_q[log pθ(x|z)]−KL(qφ(z|x)||p(z)).
Minimizing the negative ELBO therefore combines reconstruction negative log likelihood with a penalty measuring the approximate posterior's divergence from the prior. Reconstruction asks the latent to explain x; the KL constrains its distribution. It is not an image-distance term.
Multiplying KL by an arbitrary β does not preserve this exact likelihood-bound interpretation. A β-VAE uses a modified objective. Our calculation uses the original β=1 decomposition.
Calculate two inputs completely
In this original scalar example, input A has x=1 and encoder outputs μ=1, σ=0.5. Input B has x=0 and outputs μ=0, σ=1. The decoder is p(x|z)=N(z,1), and the prior is N(0,1).
Reparameterize as z=μ+σε. Draw ε=0 for A and ε=1 for B, giving z=1 in both cases. Unit-variance Gaussian negative log likelihood is (x−z)²/2 + ln(2π)/2. Its squared-error part is zero for A and 0.5 for B; the common constant is about 0.9189.
The scalar KL is (μ²+σ²−1−ln σ²)/2. A contributes approximately 0.8181 and B contributes zero. Omitting the shared reconstruction constant, this one-draw batch estimate of negative ELBO is approximately 0.6591; including it gives 1.5780.
A single draw is not the exact expectation. Here E[(x−z)²/2]=((x−μ)²+σ²)/2, giving 0.125 for A and 0.5 for B. The exact expected batch objective without the constant is approximately 0.7216. Monte Carlo estimates can lie above or below the expectation.
Follow gradients through randomness
The random ε is independent of the encoder parameters, while z=μ+σε is differentiable in μ and σ. For A's sampled ε=0, ∂z/∂μ=1 and ∂z/∂σ=0. Its reconstruction residual is zero, so this draw provides no reconstruction gradient. KL still contributes derivative μ=1 with respect to μ and σ−1/σ=−1.5 with respect to σ.
Perfect reconstruction for one draw therefore does not minimize the entire objective. σ must remain positive; a common implementation outputs log variance and uses σ=exp(logvar/2).
Interpret collapse and discrete latents carefully
A powerful decoder can ignore z, making q resemble the prior with a small KL while the latent carries little input information. Falling KL alone does not establish useful representations. Check reconstruction, generation, and latent usage together.
The slides also discuss VQ-VAE, which quantizes continuous representations into a codebook. Quantization and codebook learning use different machinery; this Gaussian KL is not automatically the loss for discrete codec tokens. The audio representation guides revisit that route. Here the core target is continuous variational inference and its gradients.
Variations with answers
Change A's encoder to μ=0, σ=1. KL becomes zero, but expected reconstruction half-squared error becomes ((1−0)²+1)/2=1, compared with 0.125 before. Matching the prior has a reconstruction tradeoff.
For ε=2, μ=1, and σ=0.5, z=2, ∂z/∂μ=1, and ∂z/∂σ=2. Reparameterization enables differentiation through a sampled value; it does not remove randomness. Assignment 2 next checks diffusion tensors, and latent diffusion later connects compressed representations to generation.
Previous | Next | Series overview
Course Video Sources
The official recording index is on SCS Panopto. The anonymous folder view is unloaded and requests sign-in; this is an access limitation, not evidence that recordings do not exist. Original audio and subtitles require the appropriate course access.
References
Loading...