Why do gradients vanish or explode when training deep neural networks or RNNs with backpropagation? Explain the role of the activation function, and describe at least four techniques that address the problem.
Show hints
Backpropagation multiplies many per-layer derivatives together.
Show the explanation
Model answer.
Where it comes from
Backpropagation applies the chain rule: the gradient reaching an early layer is a product of many terms — each layer's activation derivative times its weights. Multiplying many numbers smaller than 1 shrinks the product exponentially; many numbers larger than 1 make it grow exponentially.
- Vanishing gradients: early layers receive almost no signal, so they learn extremely slowly or not at all, while later layers train fine.
- Exploding gradients: updates become huge, the loss jumps around or becomes NaN.
- RNNs are especially vulnerable: backpropagation through time multiplies by the same recurrent weights at every step.
Role of the activation function
- The sigmoid's derivative is σ(x)(1 − σ(x)), which has a maximum of 0.25 (at x = 0) and approaches 0 when the unit saturates at large |x|.
- Even in the best case, ten sigmoid layers contribute a factor of at most 0.25^10 ≈ 9.5 × 10⁻⁷ from the activations alone.
- tanh is better (maximum derivative 1 at x = 0) but still saturates.
Remedies
- ReLU and variants: ReLU's derivative is exactly 1 for positive inputs, so it does not shrink gradients in the active region. Its weakness is "dying ReLU" (units stuck outputting 0); Leaky ReLU, ELU and GELU keep a non-zero gradient for negative inputs.
- Careful weight initialisation: Xavier/Glorot (suited to tanh/sigmoid) and He initialisation (suited to ReLU) scale initial weights so activation and gradient variances stay roughly constant across layers.
- Normalisation layers: batch or layer normalisation keep activations well-scaled, improving gradient flow.
- Residual (skip) connections: y = x + F(x) gives gradients an identity path back through the network, making 100+ layer ResNets trainable.
- Gated recurrent units: LSTMs and GRUs use additive cell-state updates controlled by gates, letting gradients flow across many time steps.
- Gradient clipping: rescale gradients when their norm exceeds a threshold — the standard fix for exploding gradients, especially in RNNs.
Follow-up: "How would you detect it?" Log per-layer gradient norms: early layers orders of magnitude smaller signals vanishing; sudden spikes and NaN losses signal exploding.
Original UniCircle question. Last reviewed 28 Oct 2026.