Deep Learning: Pre-training, Autoencoders, and PCA | NTU Hsuan-Tien Lin Machine Learning Techniques Lecture 13
Once a network is stacked deep, each layer only has to do a little, but structure, model complexity, optimization, and computation all get harder. This lecture picks two of those, initialization and regularization: pre-train the weights layer by layer with a d—d̃—d autoencoder so that each layer first learns an information-preserving encoding; train on inputs with artificial noise added, as a form of regularization; and finally make the autoencoder linear and derive that the optimal W is the d̃ top eigenvectors of XᵀX, which becomes PCA once the mean is subtracted from the data.