L1 vs L2 regularisation

Compare the gradient of |w| with the gradient of w² near zero.

Model answer. Regularisation adds a penalty on the size of the weights to the training loss, discouraging overly complex models and reducing overfitting.

• L2 (Ridge): Loss + λ Σ wᵢ²
• L1 (Lasso): Loss + λ Σ |wᵢ|

λ sets the strength (too large → underfitting). The intercept is usually not penalised.

Why L1 gives sparsity

• The L2 penalty's gradient is 2λw, which shrinks as w approaches zero. The pull towards zero becomes weaker and weaker, so weights become small but rarely exactly zero.
• The L1 penalty's gradient has constant magnitude λ (sign of w), however small w gets. That steady push can drive a weight exactly to zero and keep it there if the feature does not reduce the loss enough to justify a non-zero weight.
• Geometric view: minimising the loss subject to a constraint on the weights. The L1 constraint region is a diamond (in 2-D) with corners on the axes; the loss contours tend to touch it at a corner, where some weights are zero. The L2 region is a circle with no corners, so contact usually happens where all weights are non-zero.

When to use which

• L1: many features, expected to be mostly irrelevant; you want automatic feature selection and an interpretable model.
• L2: most features are expected to contribute a little; handles correlated features gracefully by spreading weight across them. L1 tends to pick one feature from a correlated group somewhat arbitrarily.
• Elastic Net: Loss + λ₁ Σ |wᵢ| + λ₂ Σ wᵢ² — combines sparsity with stability among correlated features.

Other useful points

• Scale features before regularising; otherwise features with small numeric ranges need large weights and are penalised unfairly.
• In neural networks, L2 is often implemented as weight decay. For plain SGD they are equivalent; for Adam they are not, which is why AdamW applies decoupled weight decay.

Common follow-up: "Does L2 perform feature selection?" No — it shrinks weights but does not usually set them exactly to zero.