Once a network is stacked deep, each layer only has to do a little, but structure, model complexity, optimization, and computation all get harder. This lecture picks two of those, initialization and regularization: pre-train the weights layer by layer with a d—d̃—d autoencoder so that each layer first learns an information-preserving encoding; train on inputs with artificial noise added, as a form of regularization; and finally make the autoencoder linear and derive that the optimal W is the d̃ top eigenvectors of XᵀX, which becomes PCA once the mean is subtracted from the data.
The first half uses Jensen's inequality to build a lower bound on the log-likelihood, the ELBO, then picks Q to be the posterior so the bound is tight at the current parameters. From that it follows that no round of EM can lower the likelihood, and plugging in a Gaussian mixture recovers last lecture's weighted-average formulas. The second half writes PCA as maximizing uᵀΣu over unit vectors; the answer is the eigenvector of the covariance matrix with the largest eigenvalue.
HW3 has three parts. The programming part has you build an LP solver by vertex enumeration, stack branch and bound on top of it for integer programs, and formulate three word problems. The written part covers integer programming by hand, the ethics of Amazon's delivery routing, PCA via SVD, and a proof that a Laplace prior equals L1. It is due 10/1, so this guide explains structure and concepts only, with no solutions.
07-380 Lec7 frames PCA as low-rank optimization: approximate the data with a matrix of rank at most r. For a unit vector v, each point's reconstruction error equals ‖x‖² minus the squared projection length, so minimizing reconstruction error and maximizing projected variance are the same problem. Lagrange multipliers show the answer is an eigenvector of the covariance matrix, which you can also read straight off the V in the SVD. The site lists the LoRA paper as reading; its ΔW = BA applies the same low-rank idea to weight updates.
HW5 Problems 3–4 give handwritten digits to three methods that never see a label: K-means summarizes the data with 10 mean images, HAC builds a merge tree you can cut at any number of clusters, and PCA compresses images onto a few continuous directions. All three answer the same question — how much error do you pay to describe the data with a few objects — and the assignment makes you compare their objectives and reconstruction errors directly.
The final written assignment combines ensembles, clustering, representation, and recommendation to test whether you can choose a learning paradigm from problem structure.