Loss Functions in Machine Learning: MSE, MAE, Likelihood, and Cross-Entropy

Machine learning turns a practical goal into a numerical optimization problem. A model produces predictions, a loss function assigns a cost to those predictions, and a training algorithm adjusts the model parameters to reduce that cost. This compact description hides an important design decision: different losses reward different behavior. Squared error emphasizes large regression mistakes, absolute error treats every additional unit of error equally, and cross-entropy evaluates the probability assigned to the observed class. Understanding those choices is essential because a model learns the objective it is given, not the intention its designer had in mind.

Loss Functions in Machine Learning: MSE, MAE, Likelihood, and Cross-Entropy

A supervised learning dataset contains input-target pairs

$$\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^{n}.$$

A model $f_{\boldsymbol{\theta}}$ maps an input $\mathbf{x}_i$ to a prediction $\hat y_i$, where $\boldsymbol{\theta}$ denotes all trainable parameters. A loss function $\ell(y_i,\hat y_i)$ measures the cost of one prediction. Training usually minimizes the empirical average of those per-example losses:

$$\mathcal{L}(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^{n} \ell\left(y_i,f_{\boldsymbol{\theta}}(\mathbf{x}_i)\right).$$

This framework applies to a straight line with two coefficients, a tree ensemble with thousands of splits, and a neural network with millions of weights. What changes is the prediction rule, the form of the loss, and the algorithm used to search for parameters.

The notation also distinguishes several ideas that are often blurred together. A per-example loss evaluates one prediction. An empirical risk aggregates losses over a sample. An objective function is the complete quantity optimized during training and may include regularization or auxiliary terms. An evaluation metric summarizes behavior that matters after training. These quantities can coincide, but they need not.

A Loss Is a Training Objective, Not a Report Card

A training loss is an instruction to the learning algorithm. It tells the optimizer which parameter changes count as improvements. It is not automatically a complete judgment of whether the fitted system is useful, safe, fair, calibrated, fast, or appropriate for deployment.

Suppose a medical classifier minimizes cross-entropy. A lower loss usually means that it assigns more probability to the correct labels in the training distribution. That fact does not answer several operational questions:

• How many actual cases does the model miss at the chosen decision threshold?
• Does sensitivity remain acceptable for important patient subgroups?
• Are probability estimates calibrated at a new hospital?
• Is the model robust to changes in measurement equipment?
• Is the cost of a false negative much larger than the cost of a false positive?

The optimizer cannot infer these priorities unless they are represented in the training design. Even when they are represented, they should be checked independently on validation and test data.

The phrase “minimize the loss” can also be misleading because there are several relevant losses. Training loss measures fit on examples used for parameter updates. Validation loss helps choose settings such as model size, regularization strength, and stopping time. Test loss estimates generalization only if the test set remains untouched until final evaluation. A model with an extremely low training loss may simply have memorized its sample. A widening gap between training and validation loss is a classic sign of overfitting.

Loss values are meaningful only in context. An MSE of $25$ has different implications when predicting temperatures in degrees Celsius, annual revenue in millions of dollars, or a standardized target with unit variance. Cross-entropy also depends on the number of classes, label uncertainty, and the difficulty of the data. Comparing raw losses across unrelated tasks is rarely informative.

Finally, optimization quality is not identical to model quality. If two training runs use the same data, model, and objective, the lower validation objective is usually preferable. But changing the objective changes the meaning of “lower.” A model optimized for MAE estimates a different conditional quantity than one optimized for MSE. Loss selection is therefore part of statistical modeling, not merely a numerical implementation detail.

Regression Losses

Regression predicts a numeric target such as demand, duration, concentration, or price. Let the residual for observation $i$ be

$$e_i=y_i-\hat y_i.$$

The sign tells whether the prediction is above or below the observation. A regression loss converts this signed residual into a nonnegative penalty.

Mean squared error

Mean squared error, or MSE, averages squared residuals:

$$\operatorname{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat y_i)^2.$$

Squaring prevents positive and negative residuals from canceling. More importantly, it makes the penalty grow quadratically. A residual of $4$ contributes $16$, whereas a residual of $2$ contributes only $4$. The first residual is twice as large but has four times the loss.

For one prediction, the squared-error derivative with respect to $\hat y$ is

$$\frac{\partial}{\partial \hat y}(y-\hat y)^2 =2(\hat y-y).$$

Large residuals therefore produce proportionally large gradients. This property can help an optimizer correct major mistakes quickly, but it also makes fitting sensitive to extreme observations. One corrupted target can dominate many ordinary examples.

MSE has a central statistical interpretation. Among all constant predictions $c$, the value minimizing expected squared error is the conditional mean:

$$\arg\min_c E[(Y-c)^2\mid X=\mathbf{x}] =E[Y\mid X=\mathbf{x}].$$

Thus a sufficiently flexible model trained with MSE aims at the conditional mean of the response. This is appropriate when the mean is the desired summary and large errors deserve extra attention.

Root mean squared error is

$$\operatorname{RMSE}=\sqrt{\operatorname{MSE}}.$$

RMSE has the same units as the target, which makes it easier to communicate. Because the square root is monotonic, minimizing RMSE and minimizing MSE over the same fixed dataset yield the same optimum. Their gradient scaling and reporting interpretation differ, however, so implementations commonly train with MSE and report RMSE.

Mean absolute error

Mean absolute error, or MAE, averages absolute residuals:

$$\operatorname{MAE} = \frac{1}{n}\sum_{i=1}^{n}|y_i-\hat y_i|.$$

The absolute penalty grows linearly. Residuals of $2$ and $4$ contribute $2$ and $4$, respectively. Each additional unit of error has the same marginal cost, so an extreme residual receives less influence than it would under MSE.

For $y\ne\hat y$, a subgradient with respect to the prediction is

$$\frac{\partial}{\partial\hat y}|y-\hat y| = \begin{cases} -1, & \hat y<y,\\ +1, & \hat y>y. \end{cases}$$

The loss has a corner at zero, where the ordinary derivative is undefined, but optimization libraries handle this with a subgradient convention. The constant gradient magnitude makes MAE robust to large residuals, although it can also produce slower or less smooth optimization near an optimum.

Among constant predictions, expected absolute error is minimized by a conditional median:

$$\arg\min_c E[|Y-c|\mid X=\mathbf{x}] \in \operatorname{Median}(Y\mid X=\mathbf{x}).$$

This distinction from MSE matters for skewed targets. If most delivery times are near 20 minutes but a small fraction are several hours because of severe disruptions, the conditional mean may lie above the experience of a typical order. The conditional median describes the middle outcome and is less affected by the long right tail.

Huber loss

Huber loss combines quadratic behavior for small residuals with linear behavior for large residuals. For residual $e=y-\hat y$ and threshold $\delta>0$,

$$\ell_{\delta}(e)= \begin{cases} \frac{1}{2}e^2, & |e|\leq\delta,\\ \delta\left(|e|-\frac{1}{2}\delta\right), & |e|>\delta. \end{cases}$$

Near zero, Huber loss behaves like squared error and provides a smooth gradient that shrinks as the fit improves. Beyond $\delta$, it behaves like a scaled absolute error, limiting the influence of extreme residuals. The pieces meet continuously at $|e|=\delta$.

The threshold determines what counts as an outlier in residual space. A large $\delta$ makes Huber loss similar to MSE; a small $\delta$ makes it more like MAE. Because $\delta$ uses the target’s units, target scaling affects its meaning. Standardizing the response or selecting $\delta$ from a robust scale estimate can make tuning more interpretable.

Huber loss does not make a model immune to bad data. It reduces the leverage of large residuals, but systematic label errors, influential feature outliers, and distribution shift still require investigation. Robust loss is one layer of protection, not a substitute for data validation.

The three principal regression losses can be summarized as follows:

Worked Numeric Example: MSE

Consider three observed targets and predictions:

$$\mathbf y=[3,\ 5,\ 10], \qquad \hat{\mathbf y}=[2,\ 7,\ 6].$$

The residuals are

$$\mathbf e=\mathbf y-\hat{\mathbf y} =[1,\ -2,\ 4].$$

Squaring each residual gives

$$\mathbf e^2=[1,\ 4,\ 16].$$

The sum of squared errors is $1+4+16=21$, so

$$\operatorname{MSE}=\frac{21}{3}=7.$$

The corresponding RMSE is

$$\operatorname{RMSE}=\sqrt{7}\approx2.646.$$

The third observation contributes $16/21$, or about $76.2\%$, of the total squared error. This illustrates the emphasis MSE places on large mistakes. For comparison, the absolute errors are $[1,2,4]$, giving

$$\operatorname{MAE}=\frac{1+2+4}{3}=\frac{7}{3}\approx2.333.$$

Under MAE, the third observation contributes $4/7$, or about $57.1\%$, of the total. It remains important, but it is less dominant.

Now imagine the final observed target was entered incorrectly as $100$ while its prediction remained $6$. The final residual would become $94$, contributing $8{,}836$ to squared error but only $94$ to absolute error. Both losses would worsen, yet MSE would redirect training much more aggressively toward that one record. Whether this is desirable depends on whether the value is a meaningful rare event or a data error.

Why Gaussian Likelihood Leads to MSE

Loss functions often arise from probability models. Suppose a regression model states that the target equals a deterministic mean plus independent Gaussian noise:

$$Y_i=f_{\boldsymbol{\theta}}(\mathbf{x}_i)+\varepsilon_i, \qquad \varepsilon_i\sim\mathcal{N}(0,\sigma^2).$$

Conditioned on $\mathbf{x}_i$, the density of $y_i$ is

$$p(y_i\mid\mathbf{x}_i,\boldsymbol{\theta},\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left( -\frac{(y_i-f_{\boldsymbol{\theta}}(\mathbf{x}_i))^2}{2\sigma^2} \right).$$

Assuming observations are conditionally independent, the likelihood is the product of these densities. Products can be numerically awkward, so maximum-likelihood estimation works with the log-likelihood:

$$\log p(\mathbf y\mid\mathbf X,\boldsymbol{\theta},\sigma^2) = -\frac{n}{2}\log(2\pi\sigma^2) -\frac{1}{2\sigma^2} \sum_{i=1}^{n} (y_i-f_{\boldsymbol{\theta}}(\mathbf{x}_i))^2.$$

If $\sigma^2$ is fixed, the first term and the positive scale $1/(2\sigma^2)$ do not change which $\boldsymbol{\theta}$ maximizes the expression. Therefore,

$$\arg\max_{\boldsymbol{\theta}}\log p(\mathbf y\mid\mathbf X,\boldsymbol{\theta},\sigma^2) = \arg\min_{\boldsymbol{\theta}} \sum_{i=1}^{n}(y_i-\hat y_i)^2.$$

Minimizing MSE is thus equivalent to maximum-likelihood estimation under independent, constant-variance Gaussian noise. The factor $1/n$ changes the numerical scale, not the minimizer.

This connection explains both the usefulness and assumptions of squared error. If residual variance grows with the target, if the residual distribution has very heavy tails, or if errors are correlated across time, the simple Gaussian model is questionable. MSE can still be used as a pragmatic objective, but its likelihood interpretation no longer fits exactly.

Other noise models lead naturally to other losses. Laplace noise produces an absolute-error negative log-likelihood. A Poisson model leads to a count-data deviance. Predicting both a mean and a variance yields a heteroscedastic Gaussian negative log-likelihood in which uncertain examples are treated differently. Loss selection can therefore be understood as choosing a probabilistic story about observations.

Classification: Why 0–1 Loss Needs a Surrogate

In classification, a model ultimately chooses among categories. The most direct error measure is 0–1 loss:

$$\ell_{0\text{-}1}(y,\hat y) = \begin{cases} 0, & y=\hat y,\\ 1, & y\ne\hat y. \end{cases}$$

Its average is the misclassification rate, and one minus that rate is accuracy. This is intuitive for reporting, but it is usually unsuitable for gradient-based training.

Consider binary classification with a score $z$ and predicted class $\mathbb{1}[z\geq0]$. Moving $z$ from $0.2$ to $5$ leaves the class correct and the 0–1 loss at zero. Moving it from $-5$ to $-0.1$ leaves the class incorrect and the loss at one. Almost every small parameter change produces no change in loss. At the decision boundary, the loss jumps discontinuously. The gradient is therefore zero where it exists and undefined at the jump, giving an optimizer little useful direction.

A surrogate loss replaces 0–1 loss with a smoother function that rewards useful movement. Logistic loss, cross-entropy, hinge loss, and exponential loss are examples. A good surrogate is computationally manageable and aligned with classification quality. It may penalize a confidently wrong prediction much more than an uncertain wrong prediction, even though both count as one error under 0–1 loss.

The surrogate also allows a model to learn probability estimates or margins rather than only hard labels. Those richer outputs support threshold selection, ranking, uncertainty-aware decisions, and cost-sensitive policies. Hard predictions can be produced later by applying an appropriate decision rule.

Binary Cross-Entropy and Log Loss

For binary targets $y\in\{0,1\}$, let a model predict $p=P(Y=1\mid\mathbf x)$. Binary cross-entropy is

$$\ell_{\mathrm{BCE}}(y,p) = -\left[y\log p+(1-y)\log(1-p)\right].$$

If $y=1$, the second term disappears and the loss is $-\log p$. If $y=0$, the first term disappears and the loss is $-\log(1-p)$. In either case, the loss is the negative logarithm of the probability assigned to the observed outcome:

$$\ell_{\mathrm{BCE}}=-\log P(Y=y\mid\mathbf x).$$

This is why binary cross-entropy is also called log loss or negative log-likelihood. Minimizing its average is equivalent to maximizing the Bernoulli likelihood of the observed labels.

The logarithm creates desirable behavior. Assigning high probability to the true label gives a small loss. Assigning probability $1$ gives an idealized loss of $0$. Assigning tiny probability to the true label produces a very large penalty. A confident error is worse than a cautious error because it represents a badly contradicted probability statement.

Logistic regression usually computes a real-valued logit

$$z=\mathbf w^\top\mathbf x+b$$

and converts it to a probability with the sigmoid:

$$p=\sigma(z)=\frac{1}{1+e^{-z}}.$$

Combining sigmoid with binary cross-entropy yields the particularly simple derivative

$$\frac{\partial\ell}{\partial z}=p-y.$$

This clean error signal is one reason the combination is so widely used. In software, a “binary cross-entropy with logits” operation is preferable to manually computing sigmoid and then taking logarithms. The fused implementation uses stable algebra and avoids evaluating $\log(0)$ when finite-precision arithmetic rounds a probability to exactly zero or one.

Worked numeric example: binary log loss

Suppose the true labels and predicted probabilities for the positive class are

$$\mathbf y=[1,\ 0,\ 1], \qquad \mathbf p=[0.8,\ 0.3,\ 0.1].$$

For the first observation, the true class is $1$, so

$$\ell_1=-\log(0.8)\approx0.2231.$$

For the second, the true class is $0$. The model assigned probability $1-0.3=0.7$ to that observed outcome:

$$\ell_2=-\log(0.7)\approx0.3567.$$

For the third, the true class is $1$, but the model assigned it only $0.1$ probability:

$$\ell_3=-\log(0.1)\approx2.3026.$$

The mean binary log loss is therefore

$$\frac{0.2231+0.3567+2.3026}{3} \approx0.9608.$$

All three observations are not equally informative under log loss. Thresholding at $0.5$ makes the first two predictions correct and the third incorrect, for an accuracy of $2/3$. Yet the third example dominates the log loss because the model was confidently wrong. If its predicted probability increased from $0.1$ to $0.4$, it would still be misclassified at the same threshold, so accuracy would remain $2/3$, but its loss would fall from about $2.3026$ to $-\log(0.4)\approx0.9163$. The surrogate recognizes meaningful progress before the hard class changes.

Multiclass Cross-Entropy

For $K$ mutually exclusive classes, a model outputs a probability vector

$$\mathbf p=(p_1,p_2,\ldots,p_K), \qquad p_k\geq0, \qquad \sum_{k=1}^{K}p_k=1.$$

With one-hot target vector $\mathbf y$, multiclass cross-entropy is

$$\ell_{\mathrm{CE}}(\mathbf y,\mathbf p) = -\sum_{k=1}^{K}y_k\log p_k.$$

Only the entry for the observed class $c$ has $y_c=1$, so the expression reduces to

$$\ell_{\mathrm{CE}}=-\log p_c.$$

The model commonly produces logits $\mathbf z$, which softmax converts into probabilities:

$$p_k=\frac{e^{z_k}}{\sum_{j=1}^{K}e^{z_j}}.$$

As in binary classification, production code should pass logits to a numerically stable cross-entropy function. Directly exponentiating large logits can overflow, while very negative logits can underflow. Stable implementations subtract the largest logit or use the log-sum-exp identity without changing the resulting probabilities.

For example, suppose a three-class model predicts probabilities $(0.70,0.20,0.10)$. If the observed class is the first, loss is $-\log(0.70)\approx0.3567$. If the observed class is the third, loss is $-\log(0.10)\approx2.3026$. The predicted top class and its confidence both matter.

Cross-entropy is a proper scoring rule: in expectation, it is minimized by reporting the true conditional class probabilities. This gives it a stronger interpretation than merely finding a separating boundary. However, finite data, model misspecification, regularization, distribution shift, and optimization limitations can still produce poorly calibrated probabilities. Calibration should be measured rather than assumed.

Multiclass softmax is appropriate when exactly one class is correct. A multi-label task, such as assigning several topics to one document, instead uses one binary output per label and commonly sums binary cross-entropies. Treating mutually exclusive and independently co-occurring labels as the same structure is a modeling error.

Regularization Adds Terms to the Objective

The data loss rewards agreement with training examples. A flexible model can sometimes obtain excellent agreement by learning unstable patterns or memorizing noise. Regularization modifies the training objective to express a preference for certain parameter values or behaviors:

$$J(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^{n} \ell\left(y_i,f_{\boldsymbol{\theta}}(\mathbf{x}_i)\right) +\lambda R(\boldsymbol{\theta}).$$

Here $R(\boldsymbol{\theta})$ is a penalty and $\lambda\geq0$ controls its influence. The complete optimized quantity is now data loss plus regularization loss.

L2 regularization uses

$$R(\boldsymbol{\theta})=\frac{1}{2}\|\boldsymbol{\theta}\|_2^2.$$

It discourages large weights smoothly and is closely related to weight decay in many optimizers. L1 regularization uses

$$R(\boldsymbol{\theta})=\|\boldsymbol{\theta}\|_1,$$

which can drive some coefficients exactly to zero and thereby encourage sparse models. The intercept is often excluded from these penalties.

Regularization extends beyond weight magnitude. A model may be penalized for rough predictions, unfair disparities, constraint violations, excessive confidence, or disagreement between augmented versions of the same input. Neural networks may also combine a primary task loss with auxiliary losses from intermediate outputs.

These extra terms introduce trade-offs. The numerical value of total loss cannot be interpreted as prediction error alone. A model with slightly higher training data loss may generalize better because its parameters are less extreme. When monitoring experiments, it is useful to record data loss, each regularization component, and total objective separately.

Feature scaling matters here. An L1 or L2 penalty acts on coefficient size, but coefficient size depends on feature units. If one predictor is measured in meters and another in millimeters, equivalent effects require very different coefficients. Standardizing features makes regularization more comparable across predictors.

Matching the Loss to the Task and Its Outliers

Choosing a loss begins with the target structure. Continuous outcomes suggest a regression loss; mutually exclusive classes suggest multiclass cross-entropy; independent labels suggest binary cross-entropy per label. But target type alone is not enough.

For regression, ask which conditional summary the application needs. MSE targets a conditional mean and strongly emphasizes large residuals. MAE targets a conditional median and provides greater resistance to heavy tails. Huber loss offers a compromise when ordinary residuals should receive a smooth squared penalty but rare extremes should have bounded influence.

The word “outlier” should be used carefully. A data-entry mistake, a failed sensor, and a legitimate rare event can all appear extreme, but they require different responses. Data errors should be corrected or removed through a documented rule. Failed sensors may justify robust methods and a quality indicator. Legitimate rare events may be the most important cases in the application; suppressing their effect with a robust loss could be harmful.

Asymmetric costs call for asymmetric objectives. Underpredicting inventory demand may cause shortages, while overpredicting causes storage expense. Quantile loss, also called pinball loss, can estimate a chosen conditional quantile:

$$\ell_{\tau}(y,\hat y) = \begin{cases} \tau(y-\hat y), & y\geq\hat y,\\ (1-\tau)(\hat y-y), & y<\hat y. \end{cases}$$

Choosing $\tau=0.9$ estimates the conditional 90th percentile under suitable conditions, deliberately producing a high forecast. This is often more principled than taking a mean prediction and adding an arbitrary buffer.

Class imbalance also requires care. Accuracy can look excellent when a rare class is ignored. Weighted cross-entropy increases the contribution of selected classes, while focal loss reduces emphasis on already easy examples. These modifications change the effective training target and may affect probability calibration. They should be chosen according to decision costs and validated with metrics appropriate to the rare class, not applied automatically whenever labels are imbalanced.

When uncertain, compare plausible losses through cross-validation, inspect residual distributions, and evaluate application-level consequences. The winning loss is not necessarily the one with the smallest numerical value because values from different formulas are not directly comparable. Train separate candidate models, then compare them on common held-out metrics.

Evaluation Metrics Versus Training Losses

A loss must provide an optimization signal. An evaluation metric must communicate behavior relevant to model selection or deployment. One quantity can serve both roles, but the requirements differ.

MSE is smooth and convenient for regression training, while MAE may be easier for stakeholders to interpret. A model trained with cross-entropy may be evaluated using accuracy, precision, recall, F1 score, area under a receiver operating characteristic curve, area under a precision-recall curve, calibration error, and inference latency. None of these views alone is complete.

Some metrics are difficult to optimize directly. F1 depends on counts after thresholding and changes in discrete jumps. Ranking metrics depend on relationships among many examples. Business utility may involve capacity constraints or delayed consequences. A differentiable surrogate makes training practical, while validation selects thresholds and compares the measures that matter operationally.

Dataset splitting preserves the distinction. The optimizer uses training loss. Hyperparameters and thresholds are selected on validation data. Final metrics are computed once on a held-out test set or through a carefully designed nested evaluation. Repeatedly choosing changes based on test performance turns the test set into another validation set and makes its reported estimate optimistic.

Metrics should also be aggregated at the correct unit. Frame-level accuracy may not reflect video-level performance. Per-transaction error may hide failures concentrated among customers. A time-series average may conceal poor performance during peak periods. The loss is computed over mathematical examples; the report card should reflect the real decision unit.

A Small Python Example

The formulas are short enough to implement directly. The following sample computes MSE, MAE, binary cross-entropy, and multiclass cross-entropy with NumPy. Clipping probabilities is useful in a demonstration that accepts probabilities directly; training libraries should generally use stable loss functions that accept logits.

import numpy as np


def mse(y_true, y_pred):
    residuals = np.asarray(y_true) - np.asarray(y_pred)
    return np.mean(residuals**2)


def mae(y_true, y_pred):
    residuals = np.asarray(y_true) - np.asarray(y_pred)
    return np.mean(np.abs(residuals))


def binary_cross_entropy(y_true, p_positive):
    y = np.asarray(y_true, dtype=float)
    p = np.asarray(p_positive, dtype=float)
    eps = np.finfo(float).eps
    p = np.clip(p, eps, 1.0 - eps)
    return -np.mean(y * np.log(p) + (1.0 - y) * np.log(1.0 - p))


def multiclass_cross_entropy(class_indices, probabilities):
    classes = np.asarray(class_indices, dtype=int)
    p = np.asarray(probabilities, dtype=float)
    eps = np.finfo(float).eps
    true_class_probabilities = p[np.arange(len(classes)), classes]
    return -np.mean(np.log(np.clip(true_class_probabilities, eps, 1.0)))


y_reg = np.array([3.0, 5.0, 10.0])
y_hat = np.array([2.0, 7.0, 6.0])
print("MSE:", mse(y_reg, y_hat))
print("MAE:", mae(y_reg, y_hat))

y_binary = np.array([1, 0, 1])
p_binary = np.array([0.8, 0.3, 0.1])
print("Binary log loss:", binary_cross_entropy(y_binary, p_binary))

y_multiclass = np.array([0, 2])
p_multiclass = np.array([
    [0.70, 0.20, 0.10],
    [0.15, 0.25, 0.60],
])
print("Multiclass log loss:", multiclass_cross_entropy(y_multiclass, p_multiclass))

This implementation is educational rather than a replacement for a machine-learning framework. Mature libraries handle batching, sample weights, ignored labels, mixed precision, distributed reduction, and numerically stable logit transformations.

Common Mistakes

Choosing a loss by habit

MSE for every numeric target and cross-entropy for every categorical target are useful defaults, not universal laws. The desired conditional statistic, tail behavior, class structure, decision costs, and probability requirements should determine the choice.

Comparing incomparable loss values

An MAE of $2$ is not necessarily better than an MSE of $3$, because the quantities use different units and scales. Even cross-entropy values from tasks with different class counts are not simple quality rankings. Compare models using the same definition on the same held-out examples.

Passing probabilities to a logits-based function

Loss APIs differ. Some expect raw logits and internally apply sigmoid or softmax; others expect probabilities. Applying sigmoid twice distorts predictions, while passing unconstrained logits to a probability-based logarithm can produce invalid values. Read the function contract and check tensor shapes.

Taking logarithms without numerical protection

Directly computing $\log(p)$ fails when finite-precision calculations produce $p=0$. Arbitrary clipping can also alter gradients if used carelessly. Prefer stable cross-entropy-with-logits implementations based on log-sum-exp identities.

Averaging over the wrong dimensions

Loss may be reduced over examples, classes, pixels, tokens, or time steps. Padding tokens often need masks. Image segmentation may require per-pixel averaging before per-image aggregation. An unintended reduction changes how examples are weighted.

Ignoring target and feature scale

MSE changes quadratically when target units change. Huber’s threshold uses target units. Regularization depends on feature scaling. A model can appear to need a different optimizer when the actual problem is inconsistent scaling.

Treating all extreme residuals as disposable

An extreme case may reveal a data error, a new regime, or the most consequential event in the dataset. Automatically removing it or selecting a robust loss can hide a model’s inability to handle important cases. Investigate provenance and domain meaning first.

Using class weights without checking calibration

Class weighting can improve attention to rare classes, but weighted probabilities may not estimate natural class frequencies directly. If downstream decisions require calibrated risk, assess calibration on representative data and consider a separate calibration procedure.

Reporting only training loss

Training loss measures optimization on seen data. It cannot establish generalization. Always monitor validation behavior, and preserve an independent test protocol for final claims.

Assuming lower surrogate loss guarantees every metric improves

Cross-entropy can decrease while accuracy at a fixed threshold remains unchanged. Accuracy can improve while calibration worsens. Model selection should use a coherent collection of metrics tied to the application.

Practice with Solver360

Loss functions become clearer when formulas are connected to actual model behavior. Solver360 calculators provide that bridge.

Use the Linear Regression Calculator to fit numeric outcomes and inspect predictions, residuals, and squared-error behavior. Change one target value to an extreme number and observe how strongly the fitted line and error respond. Then ask whether that point represents noise, an error, or a valid event that the model should learn.

Use the Logistic Regression Calculator to connect logits, sigmoid probabilities, and binary decisions. Compare two predictions that lie on the same side of the classification threshold but have different confidence. Their 0–1 losses are equal, while their log losses can differ substantially.

Use the Neural Network Calculator to examine how an objective guides many parameter updates through backpropagation. The network architecture determines what functions can be represented, but the loss determines which represented behavior training prefers. Watch how learning rate, initialization, and loss interact rather than treating any one component in isolation.

A productive practice sequence is:

1. Calculate a small loss by hand.
2. Confirm it with a calculator or short program.
3. Change one prediction while holding the target fixed.
4. Explain the direction and size of the loss change.
5. Predict how the gradient should respond.
6. Compare the training objective with a separate evaluation metric.

This process builds intuition that transfers across models. The arithmetic of one small batch reveals the same principles used in large-scale training.

Frequently Asked Questions

Is a cost function the same as a loss function?

Terminology varies. “Loss” often means the penalty for one example, while “cost” or “empirical risk” means an aggregate over a dataset. “Objective” usually means the entire optimized expression, including regularization. Many libraries and textbooks use the words interchangeably, so the formula matters more than the label.

Why divide MSE by $n$ instead of using the sum of squared errors?

For a fixed dataset, dividing by $n$ does not change the minimizing parameters. It makes the value comparable across batches or datasets of different sizes and keeps gradient scale more stable as sample count changes. Some derivations also include a factor of $1/2$ to cancel the $2$ produced by differentiation.

Is RMSE better than MSE?

Neither is universally better. They produce the same minimizer on a fixed dataset because the square root is increasing. MSE is convenient for algebra and optimization; RMSE returns to the target’s units and is often easier to interpret. Both emphasize large residuals.

When should I prefer MAE over MSE?

MAE is attractive when the conditional median is the desired prediction, the target has heavy tails, or extreme residuals should not dominate fitting. MSE is attractive when the conditional mean is desired and large misses deserve rapidly increasing penalties. Validation and domain costs should decide between them.

Does cross-entropy automatically produce calibrated probabilities?

Cross-entropy encourages truthful probabilities in the ideal population setting because it is a proper scoring rule. Real models can still be miscalibrated because of limited data, model mismatch, regularization, selection bias, overfitting, or distribution shift. Reliability diagrams, Brier score, and calibration error can provide additional evidence.

Why can cross-entropy decrease while accuracy stays constant?

Accuracy observes only the final class label. Cross-entropy observes the probability assigned to the true class. Moving a true-class probability from $0.55$ to $0.85$ improves cross-entropy without changing a correct binary decision. Moving it from $0.10$ to $0.40$ also improves cross-entropy even though the prediction remains incorrect at a $0.5$ threshold.

What happens if predicted probability is exactly zero for the true class?

The mathematical log loss is infinite because $-\log(0)=+\infty$. A stable logits-based implementation avoids explicitly taking the logarithm of a rounded zero. This severe penalty reflects that the model declared the observed event impossible.

Should regularization be included when reporting validation loss?

It depends on the purpose of the report. The validation data loss directly measures predictive fit under the selected loss, while the total objective reflects the training preference. Recording both avoids ambiguity. Final application metrics should usually be reported separately from the parameter penalty.

Can one model use several losses?

Yes. Multi-task models often sum losses for several outputs, and regularization adds further terms. The general form is

$$J=\sum_{m=1}^{M}\alpha_m\mathcal{L}_m+\lambda R,$$

where coefficients $\alpha_m$ balance tasks. Those coefficients matter because losses may have very different scales and gradient magnitudes. A nominally included task can be ignored if its weighted gradient is too small.

Is the lowest possible training loss always zero?

No. Zero may be unattainable because observations contain noise, identical inputs have conflicting labels, the model class is restricted, or regularization adds a positive penalty. With soft labels, entropy in the target distribution can also imply a nonzero optimum. A zero training loss, when attainable, does not guarantee low test loss.

What to Learn Next

Loss functions sit at the intersection of optimization, probability, and decision-making. Two next topics deepen the mathematical foundation from complementary directions.

Next: Calculus for Machine Learning. Derivatives explain how a change in a prediction affects loss and how backpropagation carries that signal through model parameters. Gradients, chain rules, partial derivatives, and curvature turn an objective formula into a training procedure.

Next: Information Theory for Machine Learning. Entropy, cross-entropy, and Kullback–Leibler divergence explain why log loss measures probability quality and how coding, uncertainty, and likelihood are connected. This perspective makes cross-entropy more than a formula to memorize.

The central lesson is that a loss function encodes a preference. MSE prefers conditional means and strongly resists large residuals; MAE prefers conditional medians and limits outlier influence; Huber loss blends those behaviors; cross-entropy rewards probability assigned to observed classes and sharply penalizes confident contradictions. Regularization adds further preferences about model complexity or behavior. Choosing and interpreting these quantities carefully makes training objectives useful instruments rather than misleading report cards.