How to Train a Neural Net: Backprop Is About Not Recomputing Shared Terms | MIT 6.7960 Lecture 2
Backprop is not new math; it computes the part of the chain rule that every layer shares exactly once. For a linear layer the forward pass multiplies by W, the backward pass multiplies by W transposed, and the weight gradient is 'input times the gradient coming back'. Gradients add at a branch, so shared parameters add their gradients too. A hand-computed update on a 2-2-1 network at the end takes the loss from 0.0175 to 0.0076.