Gradient descent and the learning rate

The gradient points in the direction of steepest increase of the loss.

Model answer. Gradient descent is an iterative method for minimising a loss function L(θ) with respect to model parameters θ.

Intuition: imagine standing on a hilly landscape in fog, trying to reach the lowest valley. You feel the slope under your feet and take a step downhill. Repeat until the ground is flat.

Update rule
θ ← θ − η · ∇L(θ)

• ∇L(θ) is the gradient — the vector of partial derivatives, pointing in the direction of steepest increase, so we move the opposite way.
• η (eta) is the learning rate, the step size.

Choosing the learning rate

• Too large: steps overshoot the minimum; the loss oscillates or even diverges to infinity/NaN.
• Too small: progress is very slow, and training may stall in flat regions.
• Practical approaches: try values on a log scale (0.1, 0.01, 0.001), use learning-rate schedules (decay, warm-up, cosine annealing), or adaptive optimisers.

Variants by how much data each step uses

• Batch gradient descent: gradient computed over the entire training set per step. Stable, but each step is expensive on large datasets.
• Stochastic gradient descent (SGD): one example per step. Cheap and fast, but noisy; the noise can help escape poor regions but makes convergence jumpy.
• Mini-batch gradient descent: a small batch (e.g. 32–512 examples) per step. The standard choice in deep learning — it balances noise and efficiency and maps well onto GPU parallelism. In practice "SGD" usually means mini-batch.

Improvements

• Momentum: accumulates a moving average of past gradients to speed up consistent directions and damp oscillations.
• Adam: combines momentum with per-parameter adaptive step sizes; a common default.

Convexity note: for convex losses (e.g. linear regression with MSE), gradient descent with a suitable learning rate reaches the global minimum. Neural network losses are non-convex, with saddle points and many minima, yet gradient descent works well in practice.

Common follow-up: "Why scale features before gradient descent?" If features have very different ranges, the loss surface becomes elongated and gradient descent zig-zags; standardising features makes convergence much faster.