Hard-margin SVM insists on separating the data perfectly, so it fits noise. This lecture gives every point a violation ξₙ and uses the parameter C to trade margin width against total violation, which yields soft-margin SVM; its dual differs from the hard-margin dual only in that αₙ gains the upper bound C. The value of αₙ sorts the data into non-support vectors, free SVs, and bounded SVs, and b is computed from a free SV; C and γ are chosen by cross-validation, and the fraction of support vectors #SV/N is an upper bound on the leave-one-out error.
This lecture splits test error into three terms: noise σ², squared bias, and variance. It then uses the closed-form solution of ridge regression to show how regularization moves each eigenvalue from λ to λ+ρ, trading a little bias for a large cut in variance. The second half covers double descent, ImageNet-V2, and choosing hyperparameters with a dev set, k-fold cross-validation, and successive halving.
Selecting a model by E_in always favors the most complex one, and selecting by E_test needs test data you cannot get. This lecture holds out K of the N examples on hand as a validation set and obtains E_out(g_m*) ≤ E_out(g⁻_m*) ≤ E_val(g⁻_m*) + O(√(log M / K)), which explains why you retrain on all the data after selecting and why K is taken as N/5. It then pushes K down to 1 to get the almost unbiased leave-one-out estimate, and finally brings the computation back to something feasible with 5-fold or 10-fold cross validation.