Sigmoid 与 Tanh：将值压缩到指定范围

Some activations squeeze any input into a fixed range. The two classics are sigmoid and tanh, and both bend big numbers gently inward. 🤏

Sigmoid maps every value into the open interval from 0 to 1. That makes its output read naturally as a probability.

Sigmoid has a smooth S shape: near zero it changes fast, but far out it flattens. Large inputs all map to nearly the same value.

Tanh is sigmoid's cousin, but it squashes inputs into the range from minus 1 to plus 1, centered neatly on zero.

Because tanh is zero-centered, its outputs balance around zero. That often gives smoother, faster learning than sigmoid for hidden layers.

Out at the edges, both curves go almost flat. A flat region means a tiny slope, and a tiny slope means a tiny gradient.

When gradients shrink toward zero, early layers barely update. This vanishing gradient trap stalls deep networks during training. 😴

This vanishing problem is exactly why ReLU replaced sigmoid and tanh in most hidden layers. ReLU keeps a healthy gradient for positives.

Sigmoid stays useful at the output of a binary classifier, where you genuinely want a single probability between 0 and 1.

Tanh still appears inside recurrent cells like LSTMs, where its bounded, zero-centered output helps keep the hidden state stable.

Rule of thumb: avoid sigmoid and tanh in deep hidden stacks, but keep sigmoid for a single probability output. Match the tool to the job.

Recall what happens at the flat tails of these curves.

Sigmoid squashes to 0 to 1 and tanh to minus 1 to 1. Their flat tails cause vanishing gradients, so save them for outputs and special cells. 🎯