Gradient Descent Complete Guide: Learning Rates, Variants, and Convergence

Gradient descent is the workhorse behind much of modern machine learning. Whether a model is fitting a straight line, recognizing an object in an image, or predicting the next token in a sentence, training usually means adjusting parameters so that a numerical measure of error becomes smaller. Gradient descent provides a practical rule for making those adjustments. The rule is simple enough to write in one line, but using it well requires understanding learning rates, noisy gradients, curved loss surfaces, convergence checks, and the behavior of popular optimizers such as momentum, RMSProp, and Adam.

Optimization Is the Engine of Machine Learning

A machine learning model maps inputs to predictions. Its parameters determine exactly how that mapping behaves. A linear regression model has coefficients and an intercept; a neural network may have millions or billions of weights and biases. Training selects parameter values that make the model's predictions useful.

To express "useful" mathematically, we define an objective function. In supervised learning, the objective often contains a loss that compares predictions with known targets. If the model parameters are collected in a vector $\boldsymbol{\theta}$, a typical empirical objective is

$$J(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^{n} \ell\left(f_{\boldsymbol{\theta}}(\mathbf{x}_i), y_i\right) + \lambda R(\boldsymbol{\theta}),$$

where $f_{\boldsymbol{\theta}}$ is the model, $\ell$ is the loss for one example, $R$ is a regularization penalty, and $\lambda$ controls the strength of regularization. Training asks for parameters that minimize $J$:

$$\boldsymbol{\theta}^{*} = \arg\min_{\boldsymbol{\theta}} J(\boldsymbol{\theta}).$$

This formulation separates the model from the training procedure. The model describes what predictions are possible. The objective says which predictions are preferred. The optimizer searches for suitable parameters.

For a tiny problem, one might solve the minimization analytically or evaluate every possible parameter value. Those approaches do not scale to high-dimensional models. Gradient descent instead uses local information: it asks how the objective changes near the current parameter vector, takes a step in a promising direction, and repeats.

Optimization is therefore not an optional final detail. It is the mechanism through which a model learns. A strong architecture with a poorly configured optimizer may never reach useful parameters, while an appropriate optimizer can make a difficult training problem manageable.

Gradients, Directional Derivatives, and Going Downhill

For a differentiable scalar function $J(\boldsymbol{\theta})$, the gradient is the vector of partial derivatives:

$$\nabla J(\boldsymbol{\theta}) = \begin{bmatrix} \frac{\partial J}{\partial \theta_1} \\ \frac{\partial J}{\partial \theta_2} \\ \vdots \\ \frac{\partial J}{\partial \theta_d} \end{bmatrix}.$$

Each component measures how sensitive the objective is to a small change in one parameter while the other parameters are held fixed. The complete vector contains more information than a list of independent slopes: it identifies the direction of steepest local increase under the ordinary Euclidean notion of distance.

To see why, choose any unit direction $\mathbf{u}$ with $\|\mathbf{u}\|_2=1$. The directional derivative is

$$D_{\mathbf{u}}J(\boldsymbol{\theta}) = \nabla J(\boldsymbol{\theta})^\top \mathbf{u}.$$

By the Cauchy-Schwarz inequality, this quantity is at most $\|\nabla J\|_2$, and the maximum occurs when $\mathbf{u}$ points in the gradient direction. The minimum occurs for

$$\mathbf{u} = -\frac{\nabla J}{\|\nabla J\|_2}.$$

Thus the negative gradient is the direction of steepest local decrease. This fact motivates the basic update

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \nabla J(\boldsymbol{\theta}_t),$$

where $\eta_t>0$ is the learning rate at iteration $t$.

The word "local" matters. A gradient describes the immediate neighborhood, not the entire landscape. A large step can leave that neighborhood and land at a point with a higher loss. A small enough step usually reduces a smooth objective, but an excessively small step wastes computation. Much of practical optimization is about choosing steps that are aggressive enough to make progress without becoming unstable.

The gradient can also be zero at several kinds of points. It is zero at a local minimum, but also at a local maximum or a saddle point. Consequently, a small gradient is evidence that movement has slowed, not proof that the globally best solution has been found.

Batch, Stochastic, and Mini-Batch Gradient Descent

When the objective is an average over training examples, the exact gradient is also an average:

$$\nabla J(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^{n} \nabla_{\boldsymbol{\theta}} \ell_i(\boldsymbol{\theta}).$$

How many terms we use in each update distinguishes the three main gradient descent regimes.

Batch gradient descent

Batch gradient descent computes the gradient using all $n$ training examples before every update. Its direction is deterministic for fixed parameters and data. Loss curves are usually smooth, and each update accurately reflects the stated training objective.

The drawback is cost. If the dataset contains ten million examples, every parameter update requires ten million forward and gradient calculations. The method may make only a few updates per hour even though each update is precise. Batch training is most attractive for small datasets, inexpensive models, or optimization problems where very accurate gradients are important.

Stochastic gradient descent

Pure stochastic gradient descent, or SGD, samples one example $i_t$ and updates with

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \nabla \ell_{i_t}(\boldsymbol{\theta}_t).$$

One example's gradient is a noisy estimate of the full gradient. If examples are sampled uniformly, this estimate is unbiased:

$$\mathbb{E}\left[\nabla \ell_{i_t}(\boldsymbol{\theta})\right] = \nabla J(\boldsymbol{\theta}).$$

Updates are cheap and begin immediately. Noise can help the optimizer leave shallow basins and saddle regions, but it also makes the path fluctuate. Near a minimum, a fixed learning rate may cause SGD to bounce indefinitely rather than settle.

Mini-batch gradient descent

Mini-batch training uses a subset $B_t$ of examples:

$$\mathbf{g}_t = \frac{1}{|B_t|} \sum_{i\in B_t} \nabla \ell_i(\boldsymbol{\theta}_t), \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t-\eta_t\mathbf{g}_t.$$

This is the standard choice for neural networks. A mini-batch uses vectorized hardware efficiently, averages away some example-level noise, and still allows many updates per pass through the data. Batch sizes such as 32, 64, 128, or 256 are common, but the best value depends on memory, architecture, data diversity, and optimization goals.

Increasing batch size reduces gradient variance, but it does not automatically reduce the number of epochs needed. Very large batches can require learning-rate adjustments and may converge to solutions with different generalization behavior. Compare experiments by both epochs and number of processed examples, and remember that an epoch means one pass through the training set.

Training data should normally be shuffled each epoch. Without shuffling, ordered labels or correlated time blocks can produce strongly biased sequences of updates. Time-series tasks are an exception when preserving temporal order is essential, though even there the sampling design should be deliberate.

The Learning Rate: Step Size Controls the Journey

The learning rate $\eta$ is often the most consequential optimizer setting. It converts a gradient, which gives a direction and sensitivity, into an actual parameter displacement.

When the learning rate is too small

A tiny learning rate produces safe but slow updates. The loss may decrease so gradually that training appears frozen. In an elongated valley, progress along the shallow direction can require an enormous number of iterations. Small steps also consume more computation and may leave a model undertrained when a fixed epoch budget expires.

A monotonically decreasing loss is not sufficient evidence that the learning rate is good. If increasing $\eta$ tenfold leads to the same or better final validation performance in one tenth of the time, the original value was inefficient.

When the learning rate is too large

A large learning rate can overshoot a minimum. On a one-dimensional quadratic $J(\theta)=\frac{1}{2}a\theta^2$, the update is

$$\theta_{t+1}=(1-\eta a)\theta_t.$$

Convergence requires $|1-\eta a|<1$, or

$$0<\eta<\frac{2}{a}.$$

If $1<\eta a<2$, the sign of $\theta$ alternates while its magnitude shrinks. If $\eta a>2$, oscillations grow and the algorithm diverges. In real training, divergence may appear as rapidly increasing loss, infinities, NaNs, or weights with exploding magnitudes.

Learning-rate schedules

A useful strategy is to start with a learning rate large enough to move quickly and then reduce it for fine adjustment. Common schedules include:

• Step decay: multiply $\eta$ by a factor at chosen epochs.
• Exponential decay: use $\eta_t=\eta_0\gamma^t$ for $0<\gamma<1$.
• Inverse-time decay: use $\eta_t=\eta_0/(1+kt)$.
• Cosine decay: smoothly reduce the rate following part of a cosine curve.
• Warmup: begin with a small rate and increase it over early iterations before applying decay.
• Reduce on plateau: lower the rate when a monitored metric stops improving.

Warmup is especially useful in large neural networks because early gradients and optimizer statistics can be unreliable. Cosine schedules are popular when the training budget is known. Plateau-based schedules are convenient when the required number of epochs is uncertain, though they add monitoring choices and may react to metric noise.

Classical convergence results for stochastic approximation often require learning rates satisfying

$$\sum_{t=1}^{\infty}\eta_t=\infty, \qquad \sum_{t=1}^{\infty}\eta_t^2<\infty.$$

These conditions make steps persistent enough to reach the solution but eventually small enough for noise to settle. Practical deep learning schedules do not always follow this theory exactly, yet the underlying idea remains useful.

Adaptive learning rates

Adaptive optimizers maintain a separate effective step size for each parameter. Parameters with consistently large squared gradients receive smaller normalized updates, while parameters with sparse or small gradients may receive relatively larger ones. This can be valuable when features have different scales or gradients are sparse.

Adaptation does not eliminate the global learning rate. Adam with $\eta=10^{-1}$ can still diverge, and Adam with $\eta=10^{-8}$ can still crawl. Treat the base learning rate as an important hyperparameter even when the optimizer adapts it.

Convex and Non-Convex Loss Landscapes

A function $J$ is convex if every line segment between two points lies above the function:

$$J(\lambda\mathbf{x}+(1-\lambda)\mathbf{y}) \leq \lambda J(\mathbf{x})+(1-\lambda)J(\mathbf{y}), \quad 0\leq\lambda\leq1.$$

For differentiable convex objectives, every local minimum is global. Linear regression with mean squared error is convex in its coefficients. Logistic regression with the usual cross-entropy objective is also convex in its parameters, assuming the model remains linear in those parameters. Gradient descent on a smooth convex objective has a comparatively clean theory.

Strong convexity provides an even more bowl-like landscape and supports faster convergence guarantees. If the Hessian eigenvalues lie between $\mu>0$ and $L$, then an appropriate fixed learning rate can produce geometric, or linear, convergence in distance to the optimum.

Neural network objectives are non-convex. Layers multiply parameters together, activation functions introduce nonlinearities, and symmetries create many equivalent parameter settings. The landscape may contain multiple basins, saddle points, ridges, and flat regions. Gradient descent can no longer promise the global minimum.

This sounds alarming, but global optimization is not always necessary. Large neural networks often have many parameter configurations with similarly low training loss, and the quality that matters is validation performance rather than the mathematical status of one minimum. Stochasticity, overparameterization, normalization, residual connections, and carefully designed initialization can make these enormous non-convex problems surprisingly trainable.

Momentum, Nesterov, RMSProp, and Adam

All four methods begin with gradient information, but they transform it differently.

Momentum

Ordinary gradient descent reacts only to the current gradient. Momentum accumulates a velocity:

$$\mathbf{v}_t = \beta\mathbf{v}_{t-1}+\mathbf{g}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t-\eta\mathbf{v}_t.$$

Some libraries place the learning rate or a factor $1-\beta$ inside the velocity definition, so formulas and hyperparameters should be interpreted in the context of the implementation.

Imagine a ball rolling down a ravine. Gradients that repeatedly point in a similar direction build velocity, accelerating progress. Gradients that alternate across the narrow walls partly cancel, reducing side-to-side oscillation. Momentum is therefore effective on poorly conditioned landscapes and often works well for convolutional networks. A typical $\beta$ is $0.9$.

Choose momentum SGD when you want a simple optimizer, can tune the learning rate and schedule, and care about strong final generalization. It may need more tuning than Adam, but it remains a robust baseline.

Nesterov accelerated gradient

Nesterov momentum evaluates the gradient after looking ahead in the velocity direction:

$$\mathbf{g}_t = \nabla J(\boldsymbol{\theta}_t-\eta\beta\mathbf{v}_{t-1}).$$

The optimizer effectively asks, "If momentum carries me forward, what will the slope be there?" It can correct the velocity before fully committing to the move. In convex optimization, Nesterov acceleration has important theoretical guarantees. In deep learning, Nesterov momentum is a modest but sometimes useful refinement of classical momentum.

Pick Nesterov when momentum SGD already works and you want anticipatory damping around curved valleys. Do not expect it to rescue a fundamentally unsuitable learning rate.

RMSProp

RMSProp tracks an exponential moving average of squared gradients:

$$\mathbf{s}_t = \rho\mathbf{s}_{t-1} +(1-\rho)\mathbf{g}_t^2,$$

where the square is elementwise. The update is

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t -\eta\frac{\mathbf{g}_t}{\sqrt{\mathbf{s}_t}+\epsilon}.$$

Dividing by the recent root-mean-square gradient normalizes coordinates with persistently different gradient magnitudes. RMSProp is useful for nonstationary objectives and has historically worked well with recurrent neural networks. The small $\epsilon$ prevents division by zero and can affect numerical behavior in low-precision training.

Choose RMSProp when per-parameter scaling is helpful but you do not need Adam's explicit first-moment estimate. It is also appropriate when an established recipe for the architecture already uses it.

Adam

Adam combines momentum-like first moments with RMSProp-like second moments:

$$\mathbf{m}_t = \beta_1\mathbf{m}_{t-1} +(1-\beta_1)\mathbf{g}_t,$$ $$\mathbf{v}_t = \beta_2\mathbf{v}_{t-1} +(1-\beta_2)\mathbf{g}_t^2.$$

Because both moving averages start at zero, Adam applies bias corrections:

$$\hat{\mathbf{m}}_t = \frac{\mathbf{m}_t}{1-\beta_1^t}, \qquad \hat{\mathbf{v}}_t = \frac{\mathbf{v}_t}{1-\beta_2^t}.$$

The update is

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t -\eta \frac{\hat{\mathbf{m}}_t} {\sqrt{\hat{\mathbf{v}}_t}+\epsilon}.$$

Defaults such as $\beta_1=0.9$, $\beta_2=0.999$, and $\epsilon=10^{-8}$ are often reasonable starting points. Adam tends to make fast initial progress and handles sparse gradients well. It is a strong default for transformers, language models, and rapid experimentation.

Adam is not automatically superior in final test performance. Momentum SGD sometimes reaches solutions that generalize better after sufficient schedule tuning. For regularized neural networks, AdamW is generally preferred over adding naive L2 regularization to Adam because AdamW decouples weight decay from the adaptive gradient update.

A practical selection rule is: start with the optimizer supported by a reliable recipe for your model family. Otherwise, use Adam or AdamW for a quick, forgiving baseline; compare with momentum SGD when final generalization matters; consider RMSProp for recurrent or nonstationary settings; and use Nesterov as an enhancement when momentum is already suitable.

Local Minima, Saddle Points, and Plateaus

A local minimum has a loss no greater than nearby points. In one dimension, a differentiable local minimum usually has zero first derivative and positive second derivative. In multiple dimensions, a positive definite Hessian indicates positive curvature in every direction.

A saddle point also has a zero gradient, but curves upward in some directions and downward in others. For example,

$$J(x,y)=x^2-y^2$$

has $\nabla J(0,0)=\mathbf{0}$, yet the origin is not a minimum. High-dimensional non-convex landscapes contain many saddle points, and exact local maxima are usually less troublesome because there are many directions in which to escape.

Near a saddle, gradients can be tiny, causing long periods of slow movement. Mini-batch noise and momentum may help by perturbing the path or carrying velocity through the region. A plateau is a broad area where gradients are small. Plateaus may result from saturated activation functions, poor initialization, excessive regularization, or genuine flatness in the objective.

Not every flat region is bad. Flat directions can reflect redundant parameterizations, and a wide basin may be compatible with robust generalization. The operational question is whether the training and validation metrics are improving, not whether every Hessian eigenvalue is comfortably positive.

Worked 1D Numerical Example

Consider

$$J(w)=(w-3)^2.$$

Its derivative is $J'(w)=2(w-3)$, and the minimum is at $w=3$. Start from $w_0=0$ with $\eta=0.2$:

$$w_{t+1}=w_t-0.2\cdot 2(w_t-3).$$

The first iterations are:

• Iteration 0: $w_0=0$, gradient $=-6$, loss $=9$.
• Iteration 1: $w_1=0-0.2(-6)=1.2$, gradient $=-3.6$, loss $=3.24$.
• Iteration 2: $w_2=1.2-0.2(-3.6)=1.92$, gradient $=-2.16$, loss $=1.1664$.
• Iteration 3: $w_3=1.92-0.2(-2.16)=2.352$, gradient $=-1.296$, loss $=0.419904$.
• Iteration 4: $w_4=2.352-0.2(-1.296)=2.6112$, gradient $=-0.7776$, loss $=0.15116544$.
• Iteration 5: $w_5=2.6112-0.2(-0.7776)=2.76672$, loss $\approx0.05442$.

Each error $w_t-3$ is multiplied by $0.6$, so convergence is geometric. If $\eta=0.5$, one update reaches $w=3$ exactly for this particular quadratic. If $\eta=1.1$, the error multiplier is $1-2.2=-1.2$; its magnitude exceeds one, so the iterates alternate sides and diverge.

Worked 2D Numerical Example

Now consider an anisotropic quadratic:

$$J(x,y)=x^2+4y^2.$$

The gradient is

$$\nabla J(x,y)= \begin{bmatrix} 2x\\ 8y \end{bmatrix}.$$

Start at $(x_0,y_0)=(4,2)$ with $\eta=0.1$. The coordinate updates are

$$x_{t+1}=0.8x_t, \qquad y_{t+1}=0.2y_t.$$

The numerical path is:

• Iteration 0: $(4,2)$, gradient $(8,16)$, loss $16+16=32$.
• Iteration 1: $(3.2,0.4)$, gradient $(6.4,3.2)$, loss $10.24+0.64=10.88$.
• Iteration 2: $(2.56,0.08)$, gradient $(5.12,0.64)$, loss $6.5536+0.0256=6.5792$.
• Iteration 3: $(2.048,0.016)$, gradient $(4.096,0.128)$, loss $4.194304+0.001024=4.195328$.
• Iteration 4: $(1.6384,0.0032)$, gradient $(3.2768,0.0256)$, loss approximately $2.684395$.

The $y$ coordinate converges much faster because its curvature is larger. The learning rate cannot be increased freely to accelerate $x$: stability in the steep $y$ direction imposes $\eta<2/8=0.25$. This tension is the essence of poor conditioning.

The same iterations can be reproduced with NumPy:

import numpy as np

theta = np.array([4.0, 2.0])
learning_rate = 0.1

for step in range(5):
    x, y = theta
    loss = x**2 + 4 * y**2
    gradient = np.array([2 * x, 8 * y])
    print(step, theta.copy(), loss, gradient)
    theta -= learning_rate * gradient

This small program also demonstrates a useful debugging habit: print or log the parameters, loss, and gradient separately. A wrong gradient can otherwise be hidden behind an apparently plausible loss value.

Feature Scaling and Hessian Conditioning

Feature scale affects optimization even when it does not change the expressive power of a model. Suppose one linear-regression feature ranges from 0 to 1 and another from 0 to 100,000. A unit change in the second coefficient has a far larger effect on predictions, so the objective curves much more sharply along that parameter direction.

The Hessian collects second derivatives:

$$\mathbf{H}_{ij} = \frac{\partial^2 J} {\partial\theta_i\partial\theta_j}.$$

Near a quadratic minimum, Hessian eigenvectors identify principal curvature directions, and eigenvalues describe how sharply the loss curves in those directions. The condition number is

$$\kappa(\mathbf{H}) = \frac{\lambda_{\max}}{\lambda_{\min}}$$

for a positive definite Hessian. A large condition number means a long, narrow valley. A learning rate small enough to remain stable across the steep direction makes slow progress along the shallow direction, producing a zigzagging path.

Standardization transforms a feature approximately to

$$x'_j=\frac{x_j-\mu_j}{\sigma_j}.$$

This often makes the curvature more balanced. Min-max scaling is another option, especially when bounded ranges are meaningful. Neural networks also benefit from normalized inputs, and normalization layers can improve intermediate activation scales.

Scaling is not merely cosmetic preprocessing. It acts like a basic form of preconditioning: it changes the coordinate system so that gradient descent can move efficiently. Adaptive optimizers partly compensate for coordinate-wise scale differences, but good input scaling still improves numerical stability and usually makes training easier.

Stopping Criteria and Loss-Curve Monitoring

An optimizer needs a rule for when to stop. Common criteria include:

• A fixed maximum number of iterations or epochs.
• A gradient norm below a threshold: $\|\nabla J\|_2<\epsilon_g$.
• A small parameter change: $\|\boldsymbol{\theta}_{t+1}-\boldsymbol{\theta}_t\|_2<\epsilon_\theta$.
• A small relative objective improvement.
• No validation improvement for a chosen patience period.
• A wall-clock or compute budget.

No single criterion is reliable in every setting. A small gradient may indicate a plateau rather than a useful minimum. Tiny parameter updates may simply reflect a tiny learning rate. Training loss can keep improving while validation loss worsens because of overfitting. In supervised neural-network training, a maximum epoch count combined with validation-based early stopping is often practical.

Monitor more than one curve. Record training loss, validation loss, the current learning rate, gradient norms, and task metrics such as accuracy or mean absolute error. If training loss falls while validation loss rises, the optimizer is still doing its job on the training objective, but generalization is deteriorating. Consider early stopping, stronger regularization, more data, or augmentation.

If both losses remain high and nearly flat, suspect an optimization issue, insufficient model capacity, or a data pipeline problem. If loss oscillates wildly, reduce the learning rate, inspect batch composition, and check gradient magnitudes. If the loss decreases smoothly but painfully slowly, raise the learning rate cautiously or improve scaling and conditioning.

Use smoothed curves for readability, but retain raw measurements. Heavy smoothing can hide spikes that precede divergence. Also compare metrics at consistent units: per mini-batch values are noisy, while epoch averages are easier to interpret.

Common Implementation Bugs

Using the wrong sign

The minimization update subtracts the gradient. Writing

$$\boldsymbol{\theta}\leftarrow \boldsymbol{\theta}+\eta\nabla J$$

performs gradient ascent and usually increases the loss. The sign can become confusing when maximizing a reward or log-likelihood. State explicitly whether the objective is minimized or maximized, and test the update on a simple quadratic with a known minimum.

Forgetting to normalize the loss or gradient

If a batch loss is summed rather than averaged, doubling the batch size doubles the gradient magnitude. The same nominal learning rate then behaves differently. Summed gradients are valid if the learning rate is designed for them, but accidental inconsistency creates hard-to-explain instability.

Training on unnormalized data

Features with radically different magnitudes create poor conditioning and can overflow intermediate calculations. Standardize numerical inputs based only on training-set statistics, then apply the same transformation to validation and test data. Do not recompute separate normalization statistics on each split.

Choosing an exploding learning rate

An excessive rate may work for several updates before entering a steep region and diverging. Watch for sudden loss spikes, non-finite gradients, and rapidly growing parameter norms. Lower the rate and consider gradient clipping when rare large gradients are expected, especially in recurrent networks.

Failing to clear accumulated gradients

Some automatic differentiation frameworks accumulate gradients by default. If gradients are not reset between steps, the optimizer unintentionally uses a sum across multiple batches. This can mimic a growing learning rate.

Updating parameters before measuring diagnostics

Logging a loss from before the update with parameters from after the update makes records inconsistent. Adopt a precise order and label values by step. This matters when validating a custom optimizer against hand calculations.

Data leakage and train/evaluation mode errors

Optimization can appear excellent when preprocessing has leaked validation information into training. Likewise, dropout and batch normalization behave differently in training and evaluation modes. Incorrect mode switching can make validation curves noisy or misleading even if parameter updates are correct.

Silent broadcasting or shape mistakes

Array libraries may broadcast incompatible-looking tensors without raising an error. The code runs, but the objective is not the intended one. Assert key shapes, compare analytical gradients with finite differences on a tiny problem, and write tests for one manually computed update.

Linear Regression as Gradient Descent

For a design matrix $\mathbf{X}$, targets $\mathbf{y}$, and coefficients $\mathbf{w}$, mean squared error can be written

$$J(\mathbf{w}) = \frac{1}{2n} \|\mathbf{X}\mathbf{w}-\mathbf{y}\|_2^2.$$

Its gradient is

$$\nabla J(\mathbf{w}) = \frac{1}{n} \mathbf{X}^\top (\mathbf{X}\mathbf{w}-\mathbf{y}).$$

Gradient descent therefore updates

$$\mathbf{w}_{t+1} = \mathbf{w}_t -\frac{\eta}{n} \mathbf{X}^\top (\mathbf{X}\mathbf{w}_t-\mathbf{y}).$$

Linear regression also has a closed-form normal-equation solution under suitable conditions. Why use gradient descent? Matrix inversion can be expensive or numerically undesirable for very high-dimensional data, while mini-batch methods can process datasets too large for memory. The linear-regression case is also ideal for learning because the objective is convex and the gradient can be checked exactly.

The Hessian is $\mathbf{X}^\top\mathbf{X}/n$, directly connecting feature correlations and scaling to conditioning. Highly correlated features create shallow directions, while large scale disparities create very different curvature magnitudes.

Neural Network Training as Gradient Descent

A neural network composes many parameterized functions. Backpropagation applies the chain rule efficiently to calculate gradients of the loss with respect to every weight and bias. Backpropagation is not itself an optimizer: it computes gradients. SGD, Adam, or another optimizer decides how to use those gradients.

A typical training step performs these operations:

1. Select a mini-batch.
2. Run a forward pass to obtain predictions.
3. Compute the loss.
4. Run backpropagation to obtain gradients.
5. Optionally clip or otherwise transform gradients.
6. Apply the optimizer update.
7. Update the learning-rate schedule if required.

Deep networks add complications such as vanishing and exploding gradients. Initialization schemes, ReLU-like activations, normalization, residual connections, and gradient clipping all help gradients remain informative. The optimizer cannot repair a completely broken gradient path; architecture and optimization must work together.

Regularization also interacts with training. Dropout changes the stochastic objective seen by each mini-batch. Weight decay pulls parameters toward zero. Data augmentation changes sampled inputs. Early stopping limits how long the optimizer fits training-specific details. These techniques influence both the landscape and the eventual solution.

Practice with Solver360

Reading equations becomes much more useful when paired with experimentation. Open the free Gradient Descent Calculator and begin with a one-dimensional quadratic. Change the starting point while keeping the learning rate fixed. Then keep the starting point fixed and gradually increase the learning rate.

Look for three regimes: slow monotonic progress, convergent oscillation, and divergent oscillation. Relate each observed path to the quadratic multiplier $1-\eta a$. Next, compare a circular two-dimensional bowl with an elongated one. The elongated contours should reveal why a single global learning rate zigzags when curvature differs by direction.

Try these focused exercises:

1. Find the largest stable learning rate for a selected quadratic and compare it with $2/\lambda_{\max}$.
2. Add momentum and observe whether oscillation across a narrow valley decreases.
3. Move the initial point near a saddle and compare deterministic and noisy updates.
4. Rescale one variable and inspect how the path changes without changing the location of the optimum.
5. Compare fixed, decaying, and adaptive learning rates using the same update budget.
6. Record loss by iteration and decide which stopping rule would terminate at a sensible point.

The goal is not to declare one optimizer universally best. It is to connect visible behavior to gradient direction, curvature, noise, and step size. That mental model transfers directly to real model training.

Frequently Asked Questions

Does gradient descent always find the global minimum?

No. For a differentiable convex objective, appropriate gradient descent settings can converge to a global minimum because every local minimum is global. Stronger assumptions determine the convergence rate. In non-convex problems such as neural networks, the optimizer may reach a local minimum, a nearly flat region, or a point that is merely good enough. It can also fail through an unsuitable learning rate or poor numerical behavior.

The absence of a global guarantee does not make gradient descent ineffective. Modern models often have many low-loss solutions, and exact global optimality on training loss is less important than performance on unseen data. Initialization, stochastic mini-batches, architecture, schedules, and regularization all shape which solution is reached. Judge success with training diagnostics and held-out evaluation rather than assuming a mathematical guarantee that does not apply.

How should I choose the initial learning rate?

Begin with a value recommended for the optimizer and model family, then test orders of magnitude. For Adam, values near $10^{-3}$ are common starting points; for momentum SGD, the useful value depends strongly on batch size, normalization, and architecture. These are starting heuristics, not laws.

A learning-rate range test can be informative: start very small, increase the rate over a short run, and identify where loss begins decreasing rapidly and where it becomes unstable. Choose a value below the unstable region, then verify it in a full training run. Compare multiple seeds when stochastic variation is meaningful. If batch size changes substantially, retune the rate rather than assuming the original value transfers unchanged.

Why does my loss increase occasionally even when training is healthy?

Mini-batch gradients optimize sampled estimates of the objective, not the exact full-dataset loss. One update can improve the current batch while worsening another, and difficult batches can report higher loss than easy ones. Data augmentation and dropout add further randomness. Occasional increases are therefore normal.

Focus on the trend of epoch-level training loss and validation metrics. Sharp, increasingly frequent spikes may indicate an excessive learning rate, numerical instability, corrupted examples, or a problematic batch. Logging gradient norms and batch identifiers can distinguish ordinary noise from systematic failures. Avoid reacting to every fluctuation by changing hyperparameters; use a sufficiently long observation window.

What is the difference between gradient clipping and lowering the learning rate?

Lowering the learning rate scales every update. Gradient clipping changes only gradients that exceed a threshold. Norm clipping commonly replaces $\mathbf{g}$ by

$$\mathbf{g}_{\text{clip}} = \mathbf{g} \min\left(1,\frac{c}{\|\mathbf{g}\|_2}\right),$$

where $c$ is the maximum norm. This protects training from rare extreme gradients while leaving ordinary updates unchanged. Lowering the learning rate may unnecessarily slow all steps.

Clipping is especially useful for exploding gradients in recurrent models or occasional outliers. It should not be used to conceal persistent instability. If almost every gradient is clipped, investigate the learning rate, data scale, initialization, loss reduction, and architecture. Also log the unclipped norm so the clipping operation does not hide the underlying issue.

Is Adam always better than SGD?

No. Adam often reaches a useful solution quickly and requires less initial tuning, especially with sparse gradients or transformer-like architectures. Momentum SGD can sometimes achieve better final generalization, particularly in computer vision recipes that use carefully tuned schedules and weight decay.

The comparison must be fair. Giving Adam a tuned schedule while leaving SGD at a default learning rate says little about the optimizers themselves. Compare validation performance under similar compute budgets and competent settings. Established model recipes are valuable because optimizer, schedule, batch size, normalization, and regularization are interdependent. When uncertain, AdamW is a practical first baseline and momentum SGD is a valuable second experiment.

How do batch size and learning rate interact?

A larger batch averages more examples, reducing gradient variance and often permitting a larger learning rate. Some recipes scale the learning rate linearly with batch size, while others use square-root scaling. Neither rule is universal. Very large batches may need warmup because a high rate is dangerous during unstable early training.

Batch size also changes the number of updates per epoch. Doubling it halves the number of updates in one pass through the dataset, so comparing only by epochs can be misleading. Hardware throughput may improve with larger batches until memory or communication limits dominate. Tune batch size for both statistical behavior and system efficiency, then retune the learning rate and schedule.

What does it mean if the gradient norm is nearly zero but the loss is high?

Several explanations are possible. The optimizer may be near a saddle point or plateau. Saturating activations can suppress gradients. A parameter may be disconnected from the computational graph. The loss may include a nondifferentiable or incorrectly implemented operation. Poor initialization can also place the network in a region where signals vanish.

Check gradients layer by layer rather than only as one global norm. Verify the loss on a tiny batch, inspect activation distributions, and confirm that parameters require gradients and participate in the forward pass. Try overfitting a handful of examples; failure to do so often reveals an implementation or optimization problem. Raising the learning rate is not a reliable fix when the gradients are structurally broken.

Should I stop when training loss stops decreasing?

Not necessarily. A temporary plateau may end after a scheduled learning-rate change, and noisy mini-batch loss can conceal slow improvement. Conversely, training loss may continue decreasing after validation performance has begun to worsen. Stopping should reflect the actual goal.

For supervised learning, monitor a validation metric with patience and preserve the best checkpoint. Patience should be long enough to tolerate noise and scheduled transitions. Also retain a maximum epoch or compute limit. For convex deterministic optimization, gradient norm or relative objective change may be more appropriate. Whatever rule you choose, define it before interpreting the curve so that stopping is not based on wishful reading of random fluctuations.

Why does feature scaling matter if the model could learn compensating weights?

The final predictive function may indeed be expressible with compensating weights, but the path used to find those weights depends on the coordinate system. Unequal feature scales create unequal curvature. A stable learning rate for the steep direction then moves too slowly along a shallow direction.

Scaling improves conditioning, making a single learning rate more effective across parameters. It also reduces floating-point problems and makes regularization more comparable across coefficients. Adaptive optimizers help but do not remove every benefit. Fit scaling statistics on training data only, store them with the model, and apply exactly the same transformation during validation and deployment.

Summary and Next Reading

Gradient descent turns learning into a sequence of informed parameter updates. The negative gradient is the steepest local descent direction, while the learning rate determines whether steps are efficient, hesitant, oscillatory, or unstable. Batch gradients provide accuracy, stochastic gradients provide cheap noisy progress, and mini-batches offer the practical balance used in most neural-network systems.

Curvature explains much of optimizer behavior. Convex objectives offer global guarantees, while non-convex landscapes introduce saddles, plateaus, and many basins. Feature scaling improves Hessian conditioning. Momentum builds velocity, Nesterov looks ahead, RMSProp normalizes with recent squared gradients, and Adam combines first- and second-moment estimates. None is universally best; model family, data, compute budget, and validation performance should guide the choice.

Reliable training requires more than selecting an optimizer name. Monitor loss, validation metrics, learning rate, and gradient norms. Use coherent stopping criteria. Test gradients on small known problems, normalize data, check signs and reduction conventions, and treat non-finite values as evidence to investigate rather than something to ignore.

For next reading, study backpropagation to understand how neural networks compute gradients, convex optimization for formal convergence results, Hessian eigenvalues and preconditioning for a deeper account of curvature, and regularization for the distinction between minimizing training error and generalizing well. Then return to the Gradient Descent Calculator and use controlled experiments to connect those ideas to visible optimization paths.