ReLU와 leaky ReLU, GELU 계열

The most popular activation is ReLU: it keeps positive values and turns every negative one into zero. Simple and fast. ⚡

ReLU is cheap to compute and its gradient is a clean 1 for positives. That keeps signals flowing and makes deep nets train quickly.

ReLU is just max(0, x). Positive inputs pass straight through; negatives flatten to zero. That single hinge is the whole trick.

If a neuron always outputs zero, its gradient is zero too, so it stops learning forever. We call this a dead neuron. 💀

Leaky ReLU lets a tiny slope through for negatives instead of a hard zero. That small leak keeps dead neurons alive.

PReLU goes further: it learns the negative slope during training instead of fixing it. The network tunes the leak itself.

GELU smooths the ReLU corner into a soft curve. It gates inputs by how likely they are to be useful, not with a hard cutoff.

Modern models like transformers favor GELU because its smooth shape gives gentler gradients. That often means steadier training. 🤖

SiLU, also called Swish, multiplies the input by its own sigmoid. Like GELU, it is smooth and frequently edges out plain ReLU.

Start with ReLU for hidden layers; it is fast and reliable. Reach for Leaky ReLU or GELU only if you see dead neurons or want extra smoothness.

You can drop these in as modules inside a model, not just as functions. That makes them easy to chain in nn.Sequential.

ReLU is the fast default, but it can let neurons die. Leaky ReLU, PReLU, and GELU smooth or leak the negatives to keep learning healthy. 🌟