Low Rank Optimization, PCA's Reconstruction Error, Projected Variance, and LoRA | CMU 07-380 Lecture 7
07-380 Lec7 frames PCA as low-rank optimization: approximate the data with a matrix of rank at most r. For a unit vector v, each point's reconstruction error equals ‖x‖² minus the squared projection length, so minimizing reconstruction error and maximizing projected variance are the same problem. Lagrange multipliers show the answer is an eigenvector of the covariance matrix, which you can also read straight off the V in the SVD. The site lists the LoRA paper as reading; its ΔW = BA applies the same low-rank idea to weight updates.