The Chain Rule, Layer by Layer

A neural network is just functions wrapped inside functions: each layer feeds its output into the next. Backprop needs the derivative of that whole stack.

The chain rule says the derivative of nested functions multiplies the derivatives of each piece. That single fact is the engine behind all of backprop. ⛓️

Say z depends on y, and y depends on x. To get how z changes with x, you multiply dz/dy by dy/dx. Each link passes its slope along.

Every layer is one link. The loss sits at the far end, the input at the near end, and gradients flow back link by link as a chain of multiplications.

Backprop begins at the loss and asks: how does loss change if this layer output nudges a bit? That answer is the first derivative in the chain.

Each layer receives the gradient from the layer above, multiplies by its own local derivative, and hands the result to the layer below.

A layer only needs to know its own local derivative: how its output changes with its input and weights. It never needs the whole network at once.

At each layer the incoming gradient also tells you how the loss changes with that layer's weights. That is exactly what the optimizer will use to update them.

You multiply derivatives in reverse order, from loss back to input. Going forward computes values; going backward computes how to improve them.

Add ten more layers and nothing changes: you just multiply ten more local derivatives into the chain. Depth is no problem for the chain rule.

Backprop is the chain rule applied layer by layer, from the loss back to the inputs, multiplying local slopes to find each weight's gradient.

Let's check the core rule.

A network is nested functions, so its derivative is a chain of multiplied local slopes. Backprop walks that chain from loss back to input. ⛓️