Initialisation des poids : initialisations de Xavier et de He

The weights of a neural network must be initialised to non-zero values before training — but the choice of how to initialise them profoundly affects training dynamics. Poor initialisation causes vanishing gradients (weights shrink to near zero, gradients become negligible) or exploding gradients (weights grow unboundedly, gradients become NaN). Good initialisation keeps activations and gradients in a healthy range from the very first batch, enabling stable and fast training.

If all weights are initialised to the same value (including zero), every neuron in a layer computes exactly the same output and receives exactly the same gradient. All neurons learn the same feature — the hidden layer collapses to a single neuron for all practical purposes. This symmetry problem is why random initialisation is necessary: each neuron must start with a different random weight to break symmetry and learn different representations.

Initialising weights from a standard normal distribution N(0, 1) seems reasonable but causes problems in deep networks. For a layer with fan-in (number of input connections) of 1000, the weighted sum of 1000 normally distributed values has variance 1000 — causing exploding activations in deep networks. Conversely, very small random values (e.g., N(0, 0.001)) cause vanishing activations. Neither extreme allows gradients to flow through many layers.

Xavier initialisation (Glorot and Bengio, 2010) was designed for networks using tanh or sigmoid activations. The insight is to choose weights so that the variance of activations and gradients remains roughly constant across layers. Weights are drawn from a uniform or normal distribution with variance 2 / (fan_in + fan_out). This is the default initialisation for nn.Linear (uniform variant) in PyTorch.

He initialisation (He et al., 2015) was designed specifically for networks using ReLU activations. Because ReLU zeros out half of its inputs (the negatives), the effective variance after activation is halved. He init compensates by using variance 2 / fan_in — twice what Xavier uses. Using Xavier with ReLU causes gradients to vanish in deep networks; using He initialisation enables training networks with 100+ layers.

The effect of initialisation becomes visible when you track activation statistics across layers of a deep network. With Xavier init and tanh, activation variance stays near 1 through all layers. With He init and ReLU, the same holds for ReLU networks. Using the wrong combination (Xavier + ReLU, or He + sigmoid) leads to systematic activation collapse or explosion, confirming that initialisation choice must match the activation function.

You can apply a custom initialisation to an entire model using model.apply(init_fn), which recursively visits every module. The function receives each module and can apply different initialisations based on the layer type. A common pattern is to apply He init to Linear and Conv2d layers, Xavier init to embedding layers, and set biases to zero. This single call replaces PyTorch's defaults across the entire network.

Orthogonal initialisation sets weight matrices to be orthogonal (Q from QR decomposition), which preserves gradient norms during backpropagation through time. This is particularly useful for recurrent networks, where the same weight matrix is multiplied repeatedly (once per timestep). Orthogonal weights prevent gradients from exploding or vanishing as they are propagated back through many timesteps in long sequences.

PyTorch applies sensible defaults automatically: nn.Linear uses Kaiming uniform for weights and uniform distribution for biases. nn.Conv2d also uses Kaiming uniform. nn.Embedding uses standard normal N(0, 1). nn.LSTM uses uniform in [-1/sqrt(hidden), 1/sqrt(hidden)]. In many cases the defaults work well, but for very deep networks or non-standard activations, explicit initialisation with the formulas above gives better results.

A practical guide for choosing initialisation: use He (Kaiming) normal or uniform for any network using ReLU or its variants (LeakyReLU, ELU, GELU). Use Xavier normal or uniform for tanh or sigmoid activations. Use orthogonal for recurrent weights. Set biases to zero in all cases. For Transformers with GELU, N(0, 0.02) is the empirical standard used in GPT-2 and subsequent models. Trust PyTorch's defaults for standard architectures; only override when training is unstable.

After applying initialisation, verify it by checking activation statistics in the first forward pass. A healthy network should have activation standard deviations near 1.0 across all layers and gradient norms of similar magnitude across layers. Large discrepancies (e.g., std=10 in one layer, std=0.001 in another) indicate initialisation mismatch. This quick sanity check takes seconds and can save hours of debugging poor training dynamics.

In this lesson you learned: Xavier initialisation is designed for tanh/sigmoid activations using variance 2/(fan_in + fan_out), He (Kaiming) initialisation is designed for ReLU using variance 2/fan_in to compensate for ReLU zeroing half its inputs, and model.apply(init_fn) applies a custom initialisation to every layer in the network. Next up we dive into convolutional neural networks starting with convolution and filter operations.