Hsuan-Tien Lin's ML Techniques T12–T13: Neural Networks and Deep Learning (Autoencoders, PCA)

🌏 中文版

Sources: The core material is the MOOC version of Machine Learning Techniques: 212_handout.pdf (Neural Network), 213_handout.pdf (Deep Learning), and videos 46–53 of the Techniques YouTube playlist. Textbook sections follow the LFD e-Chapter 7 sections listed on the Fall 2024 course page. Homework references come from Fall 2024 HW7. All facts were checked against the originals on 2026-09-30. The lectures are taught in Mandarin; the slides are in English. Access level: the MOOC alone is A2; adding the Fall 2024 homework PDFs brings it to A3 (minus the grading chain). There are no official solutions, and Gradescope and NTU COOL are for enrolled students only.

Series: Previous: Decision Trees, Random Forests, and Gradient Boosted Trees | Next: RBF Networks, k-Means, and Matrix Factorization | Series overview

The first part of Techniques packs many features in with kernels; the second combines predictive features with aggregation. The third part, "Distilling Implicit Features: Extraction Models," goes in a different direction. Instead of people choosing features, the model extracts hidden features from the data itself. The four models in T12–T15 (neural networks, deep learning, RBF networks, matrix factorization) are all variations on this theme.

One piece of context matters for these two lectures. The videos were uploaded in 2016, when deep learning was "gaining attention in recent years" (the slides' words). So T13 centers on layer-wise pre-training and autoencoders, not the ReLU, Adam, or Transformers common today. In Fall 2024 Lin added two modern deep learning slide decks, 302u and 303u, which this series covers in post 16.

Where these lectures sit

LFD's e-Chapters are online chapters covering topics not in the printed book; see amlbook.com for access.

T12 Neural Network

Videos: Motivation, Neural Network Hypothesis, Neural Network Learning, Optimization and Regularization

Starting from a linear combination of perceptrons

T12 continues in the language of the previous part: G(x) = sign(Σ α_t·sign(w_tᵀx)) is linear aggregation with perceptrons as the g_t. It has two layers of weights (w_t and α) and two layers of sign functions.

What boundaries can this G draw? The slides start with logic operations. G(x) = sign(−1 + g_1(x) + g_2(x)) outputs +1 only when both g_1 and g_2 are +1, which is AND; OR and NOT can be built similarly.

Then come power and limits. The slides approximate a smooth target boundary with 8 and then 16 perceptrons; with enough perceptrons you can approximate any smooth boundary, and Foundations already showed that convex-set hypotheses have infinite d_vc. But it can't build XOR(g_1, g_2), because XOR is not linearly separable in the space φ(x) = (g_1(x), g_2(x)).

The fix is one more layer of transforms: XOR(g_1, g_2) = OR(AND(−g_1, g_2), AND(g_1, −g_2)). That is the multi-layer perceptron. The slides' progression: perceptron (simple) → aggregation of perceptrons (powerful) → multi-layer perceptrons (more powerful). They also note that neural networks are bio-inspired models, and inspired is all.

The NNet hypothesis: a linear output, tanh in the hidden layers

The output layer is just a linear model, s = wᵀφ^(2)(φ^(1)(x)). Any of the three linear models from Foundations fits: sign with 0/1 error, identity with squared error, logistic with cross-entropy. From here T12 uses regression with squared error.

What transformation should the hidden layers use? All-linear makes the whole network linear and not very useful; sign is discrete and hard to optimize over w. The popular choice is tanh. It is an "analog" approximation of sign, easier to optimize, and somewhat closer to biological neurons. tanh(s) = (exp(s) − exp(−s))/(exp(s) + exp(−s)) = 2θ(2s) − 1, where θ is the logistic function from Foundations.

In a d^(0)-d^(1)-…-d^(L) network, each layer ℓ computes scores s_j^(ℓ) = Σ_i w_ij^(ℓ) x_i^(ℓ−1); hidden outputs are x_j^(ℓ) = tanh(s_j^(ℓ)), and the output layer takes s directly. Physically, each layer is a transform learned from data that checks whether x "matches" the patterns in its weight vectors. The slides sum it up as pattern extraction with layers of connection weights.

Backprop: computing gradients efficiently

The goal is to find all w_ij that minimize E_in. With a single hidden layer the network is just aggregation of perceptrons, and gradient boosting could add hidden neurons one at a time; with multiple layers that gets hard. So run (stochastic) gradient descent on each example's error e_n = (y_n − NNet(x_n))². The key is computing ∂e_n/∂w_ij^(ℓ).

The backprop algorithm has four steps: pick a random n; compute all x_i^(ℓ) forward; compute all δ_j^(ℓ) backward; update by gradient descent w_ij^(ℓ) ← w_ij^(ℓ) − η·x_i^(ℓ−1)·δ_j^(ℓ). The slides add that the first three steps are sometimes done many times (in parallel) and the update uses the average of x_i^(ℓ−1)δ_j^(ℓ). That is called mini-batch.

Optimization and regularization

Optimization is hard but works in practice. With multiple hidden layers, E_in is generally non-convex. GD/SGD with backprop only reaches a local minimum, and different initial weights lead to different local minima. Large weights saturate tanh and shrink the gradient. The slides' advice: try some random and small initial values.

VC dimension. With tanh-like transfer functions, roughly d_vc = O(VD), where V is the number of neurons and D the number of weights. With enough neurons the network can approximate "anything," but it can also overfit.

Regularization options:

• Weight decay (L2): shrinks large weights a lot and small weights a little, but never quite to zero.
• L1: can drive weights to zero and effectively lower d_vc, but isn't differentiable.
• Weight elimination: a scaled L2, Σ (w_ij)² / (1 + (w_ij)²), which gives both large and small weights a medium shrink.
• Early stopping: the more steps GD/SGD takes, the more weight combinations it visits, and the larger the effective d_vc. Stopping in the middle controls model complexity. When to stop? Use validation.

T13 Deep Learning

Videos: Deep Neural Network, Autoencoder, Denoising Autoencoder, Principal Component Analysis

Shallow versus deep

Each layer extracts pattern features, so how many neurons and how many layers? The slides' answer: subjectively, your design; objectively, maybe validation. Structural decisions are the key issue in applying NNets.

The slides' comparison:

The "meaningful" example is handwritten digit recognition: the first layer extracts strokes from pixels, and later layers combine them into parts and digits. Each layer carries less of the burden, moving from simple to complex features, which suits tasks like vision and speech where raw features are hard to use.

Four challenges of deep learning and the techniques for each:

• Difficult structural decisions: use domain knowledge, such as convolutional NNets for images.
• High model complexity: less of a worry with big enough data; also use regularization that makes the model noise-tolerant, such as dropout (tolerant when the network is corrupted) and denoising (tolerant when the input is corrupted).
• Hard optimization: careful initialization to avoid bad local minima, called pre-training.
• Huge computation: new hardware and architectures, such as mini-batch on GPUs.

Lin's personal view (the slides say IMHO) is that careful regularization and initialization are the key techniques. So the rest of T13 covers the simplest pre-training technique plus one regularization technique.

Autoencoders: learning an approximate identity function

A two-step deep learning framework: first pre-train weights layer by layer from shallow to deep (with earlier layers fixed), then fine-tune the whole network with backprop.

What should pre-training aim for? Good weights should be an information-preserving encoding: the next layer represents the same information differently, so you can decode accurately after encoding.

That is an autoencoder: a d—d̃—d network trained so that g_i(x) ≈ x_i, with w_ij as encoding weights and w_ji as decoding weights. Why learn an approximate identity function?

• For supervised learning: the hidden structure of the data can serve as a reasonable transform Φ(x), an "informative" representation.
• For unsupervised learning: where g(x) ≈ x the structure matches, which supports density estimation; points where g(x) is far from x are outliers.

A basic autoencoder uses squared error Σ_i (g_i(x) − x_i)², so backprop applies directly; it is shallow and easy to train. Usually d̃ < d, giving a compressed representation. The data are {(x_n, y_n = x_n)}, so it is often categorized as unsupervised learning. Sometimes the weights are tied, w_ij^(1) = w_ji^(2), as regularization.

Pre-training with autoencoders: layer ℓ's weights come from training a basic autoencoder with d̃ = d^(ℓ) on the previous layer's outputs x_n^(ℓ−1). The slides also note that many successful pre-training techniques use fancier autoencoders with different architectures and regularization.

Denoising autoencoders: noise as a hint

Deep networks have high model complexity and need regularization: structural constraints, weight decay or weight elimination, early stopping. These are old friends. T13 adds a new one.

Back to the causes of overfitting in Foundations L13: little data, lots of noise, and excessive model power each make overfitting worse. How do you deal with noise? The direct option is data cleaning. The slides propose a "wild" option: add noise to the data on purpose.

The idea is that a robust autoencoder should not only give g(x) ≈ x but also g(x̃) ≈ x when x̃ is slightly different from x. So a denoising autoencoder trains a basic autoencoder on {(x̃_n, y_n = x_n)}, with x̃_n = x_n + artificial noise. It often replaces the basic autoencoder in deep learning and is useful for image denoising. The slides' conclusion: artificial noise (a hint) is itself regularization, and it is practically useful for other NNets and models too.

A linear autoencoder is PCA

The last section asks: what if the autoencoder is linear? The slides' reasoning is "linear first": it may be more efficient and less prone to overfitting.

Add three conditions: drop x_0 (so input and output dimensions match), tie the weights w_ij^(1) = w_ji^(2), and require d̃ < d (so the solution isn't trivial). The hypothesis becomes h(x) = WWᵀx with W a d × d̃ matrix, and E_in(W) = (1/N) Σ_n ‖x_n − WWᵀx_n‖².

This is a fourth-order polynomial in w and hard to solve directly. The slides reframe it with linear algebra:

The full algorithm: compute the mean x̄, subtract it from every x_n, compute the top d̃ eigenvectors w_1…w_d̃ of XᵀX, and return the feature transform Φ(x) = W(x − x̄).

The slides end by separating the two. A linear autoencoder maximizes the squared magnitude after projection; principal component analysis (PCA) from statistics maximizes the variance after projection, which is what the mean-subtraction step in the algorithm is for. Both are useful for linear dimension reduction, and PCA is more popular.

Practice with the Fall 2024 homework

The problems in HW7 (released 2024-12-02, due 12-16) that map to these two lectures:

Q9 is worth thinking about alongside T12's advice to "try some random and small initial values." For Q4, a small script that enumerates every layer arrangement makes a good check. There are no official solutions. Q13 requires you to judge for yourself which step of chatGPT's argument breaks, which is a problem type the Fall 2024 homework uses deliberately. The full homework walkthrough is in Techniques homework and final project.

How to use these two lectures for self-study

1. Start with T12's Neural Network Learning. Work through the δ recursion in the collapsed derivation, then write a d-3-1 tanh network in numpy and check its gradients against numerical differentiation.
2. T13's autoencoders and pre-training were mainstream practice in 2016. Understanding why pre-training was needed then matters more than memorizing the steps. For modern training techniques, continue with post 16 of this series.
3. One thing to do tonight: on any small dataset, compute the top two principal components with numpy's np.linalg.eigh(X.T @ X) and with scikit-learn's PCA, and see how much subtracting the mean changes the result. That is the difference on the slides' last page.

Further reading

These series on this site overlap with this post, but this post stands on its own; they are links only:

• Reading CMU 11-785 Deep Learning, especially Lecture 5: Backpropagation: a full course on deep learning.
• Reading Stanford CS230: focused on deep learning project practice.
• Reading MIT 6.7960: graduate-level deep learning.
• Reading NTU Hung-yi Lee Machine Learning 2026 Spring: NTU's parallel track, leaning toward deep learning and generative AI.
• Harvard CS181 HW4: Autoencoders and VAEs and CMU 10-301 HW5: Neural Networks: homework on the same topics from other schools.

References

• Hsuan-Tien Lin > MOOCs — section titles and slides for Techniques T12 and T13
• Lecture 12: Neural Network (212_handout.pdf) — perceptron aggregation, XOR, tanh, backprop, d_vc = O(VD), weight elimination, early stopping
• Lecture 13: Deep Learning (213_handout.pdf) — shallow vs. deep, four challenges, autoencoder pre-training, denoising, linear autoencoders and PCA
• Machine Learning Techniques YouTube playlist — videos 46–53 (lectures in Mandarin)
• Machine Learning, Fall 2024 course page — W13 schedule, 212u/213u slides, and LFD e-7 sections
• Fall 2024 Homework 7 (hw7.pdf)
• Machine Learning, Fall 2026 course page — W14 final exam and W15 schedule
• Learning from Data textbook site
• On this site: Series overview