🌏 中文版
Video status: recordings require login or course authorization. Sources
This guide follows CMU 10-423 Spring 2026 public slides. Worked examples are original teaching additions.
Ask what actually gets updated
A pretrained model already has language capabilities. Adapting it to sentiment classification may not require relearning every weight. Parameter-efficient fine-tuning (PEFT) searches a restricted update space while preserving the backbone. You need matrix multiplication and gradient descent; review training and decoding if weights and inputs are still easy to confuse. The goal here is to count LoRA parameters, trace a forward pass, and identify the gradients that should exist.
LoRA does not replace the original weight with a low-rank approximation. It retains W and represents its update as ΔW=(α/r)BA, giving y=Wx+(α/r)BAx. For a d_out×d_in weight, A has shape r×d_in and B has shape d_out×r. Full fine-tuning changes d_out·d_in values; LoRA trains r(d_in+d_out). The rank restricts this layer's update, not the number of answers the whole nonlinear model can express.
Work through the update
This is an original teaching example, not a homework solution. Let W be a zero 4×3 matrix, r=1, α=1, A=[1,0,−1], B=[1,2,0,−1]ᵀ, and x=[2,1,1]ᵀ. First Ax=1, then B(Ax)=[1,2,0,−1]ᵀ. The rows of BA are [1,0,−1], [2,0,−2], [0,0,0], and [−1,0,1]. All rows change along the same direction.
A full update has 12 parameters; this adapter has seven. Raising r to two would require 14, more than the original matrix. Parameter savings depend on dimensions and rank. Large projection matrices are where small ranks provide substantial savings. A forward pass can compute Ax and then B(Ax), without materializing BA. For suitable inference settings the update can be merged into W, while switching between tasks may favor keeping separate adapters.
Initialization determines the first gradients
A common initialization uses random A and zero B. The first forward pass then matches the backbone. B may receive a nonzero gradient, while A's first gradient is zero because B is zero. Once B changes, A can receive a nonzero signal. A test that demands nonzero gradients from every adapter parameter on the first step would therefore reject correct code.
Freezing W means preventing parameter updates. It does not mean wrapping the whole LoRA operation in no_grad. Earlier trainable adapters may still need gradients through later frozen layers. Disabling weight gradients and disabling the entire computation graph are different operations.
Other PEFT methods change other components
A bottleneck adapter transforms hidden states through down-projection, a nonlinearity, and up-projection. Prefix tuning adds learnable key/value prefixes. These methods all reduce trainable parameters, but a prefix changes attention normalization while LoRA changes a projection matrix. Counting a small set of trainable weights cannot replace explaining the data flow. A frozen backbone still incurs forward computation, activation storage, and potentially backward computation for gradients to its inputs; memory savings need measurement.
Check your understanding
Replace x with [1,2,1]ᵀ. Then Ax=0, so the adapter contribution vanishes. The model has not lost all capacity; this particular update direction has no effect on that input. For a 1024×1024 matrix with r=8, full tuning changes 1,048,576 values, whereas LoRA trains 16,384, a ratio of 1/64 excluding biases. If W changes, falling loss does not prove adapter-only training. Continue with the Assignment 3 testing example.
Course video sources
The official Spring 2026 course site directs recordings to SCS Panopto. On 2026-10-12, anonymous retrieval returned an unloaded folder and a sign-in prompt, without playable classroom content. This establishes an access limit, not the absence of recordings. This guide uses public slides and linked primary papers; examples are original teaching additions, not a transcript. Official recording folder.
References
Loading...