Gradients Point Uphill — So Step the Other Way

To walk downhill you first need to know which way is down. The gradient is the tool that tells you the slope of the loss at your current spot.

Here is the surprise: the gradient points in the direction where loss increases fastest. It shows you the steepest way up, not down.

Since the gradient aims uphill, you move in the opposite direction to go down. Subtracting the gradient takes you toward lower loss.

This one line is the heart of training. You scale the gradient by a small step size, then subtract it from each weight. Repeat and loss drops.

With many weights, the gradient is a whole vector: one slope per weight. Each weight gets nudged by its own piece of that vector.

A large gradient value means the loss is very steep in that direction, so that weight needs a bigger correction. Small values mean you are nearly flat there.

At the very bottom of a valley the slope flattens and the gradient becomes zero. With nothing to subtract, the weights stop moving. You have converged.

You almost never reach the bottom in one jump. Each step only improves things a little, so descent is a long series of tiny downhill moves.

The path you trace by always heading along the negative gradient is the route of steepest descent. It is the greedy shortcut toward lower loss.

Subtracting moves you against the increase, so loss must fall for a small enough step. That simple sign flip is what turns a slope into progress.

You will rarely compute these slopes by hand. PyTorch's autograd finds the gradient for you, but the rule stays the same: subtract it to descend.

Which direction does the gradient point?

The gradient points uphill toward higher loss, so you subtract a scaled version of it to step down. Repeat that update and the model rolls toward the valley. ⬇️