Learning Rate: Too Big, Too Small, Just Right

The gradient tells you which way to go, but how far? The learning rate is the multiplier that sets how big each downhill step is.

The learning rate is often the single most important setting in training. Get it wrong and even a perfect model will fail to learn.

A tiny learning rate takes tip-toe steps. Loss drops, but so slowly that training drags on for ages or stalls before reaching the valley. 🐢

A huge learning rate leaps so far it jumps past the valley and lands higher up the other side. Loss bounces around or shoots off to infinity.

If loss climbs or turns into NaN after a few steps, your learning rate is almost always too big. That is the clearest signal to turn it down.

A good learning rate makes loss fall steadily and quickly without wild swings. The curve glides down and levels off near the bottom.

People rarely guess blindly. A common starting learning rate is around 0.001 to 0.01, then you adjust based on how the loss curve behaves.

Plotting loss over time is your best guide. A diverging curve says go smaller; a flat, barely-moving curve says you can go a bit larger.

The right learning rate depends on other choices like batch size and optimizer. Change one and you may need to retune the rate.

You can even shrink the learning rate as training goes on: take big strides early, then small careful steps near the bottom for a clean finish.

There is no universal best value, so trying a few learning rates and comparing curves is a normal, expected part of training a model.

What happens with too large a learning rate?

The learning rate sets step size: too small crawls, too big overshoots and diverges, and just right glides smoothly down. Watch the loss curve and tune it. 🎯