Information Theory for Deep Learning: Entropy, Cross-Entropy, and KL Divergence

Information Theory for Deep Learning: Entropy, Cross-Entropy, and KL Divergence connects the language of uncertainty to the practical objectives used to train classifiers. A neural network does not merely choose a label; it usually produces a probability distribution over possible labels. Information theory supplies the mathematics for evaluating that distribution, comparing it with reality, and deciding how strongly an error should affect learning. Entropy describes uncertainty, cross-entropy scores predictions against a target distribution, and Kullback–Leibler divergence measures the mismatch between two distributions. These ideas also explain decision-tree splits, softmax outputs, mutual information, and the unusually severe penalty assigned to a confident but incorrect prediction.

Information as Surprise

In everyday language, information is often treated as a quantity of facts. In information theory, information has a more precise meaning: an event is informative to the extent that it was difficult to predict. Learning that the sun rose this morning provides little information because its probability was already extremely high. Learning that an ordinary coin landed on its edge would provide much more information because that outcome was highly improbable.

For an event $x$ with probability $p(x)$, its self-information, also called surprisal, is

$$I(x)=-\log p(x)=\log\frac{1}{p(x)}.$$

This definition has three useful properties. First, probable events carry little surprise. As $p(x)$ approaches one, $I(x)$ approaches zero. Second, rare events carry substantial surprise. As $p(x)$ approaches zero, $I(x)$ grows without bound. Third, the information in independent events adds. If $x$ and $y$ are independent, then $p(x,y)=p(x)p(y)$, so

$$I(x,y)=-\log[p(x)p(y)]=-\log p(x)-\log p(y)=I(x)+I(y).$$

The logarithm is therefore not an arbitrary decoration. It converts multiplication of independent probabilities into addition of information.

The logarithm's base determines the unit. Base-2 logarithms measure information in bits. Natural logarithms measure it in nats. Base-10 logarithms produce hartleys. Machine-learning libraries normally use natural logarithms because they fit naturally with exponentials, calculus, and the softmax function. Changing the base rescales all values by a constant and does not change which model has the lower loss.

Consider three weather forecasts. An event assigned probability $0.5$ has surprisal $-\log_2(0.5)=1$ bit when it occurs. An event assigned probability $0.125$ has surprisal $3$ bits. An event assigned probability $0.99$ has surprisal only about $0.0145$ bits. The information is not an intrinsic property of the event alone; it depends on the probability model used before observing the outcome. Rain can be surprising under one forecast and unsurprising under another.

This observation is central to deep learning. A classifier announces probabilities before seeing the target. Once the true class is revealed, the loss measures the surprise that the model assigned to that class. Training then adjusts the model so that future observed labels become less surprising.

Shannon Entropy for Discrete Distributions

Self-information concerns one realized event. Shannon entropy concerns the average uncertainty of an entire random variable. Let a discrete random variable $X$ take values $x_1,\ldots,x_K$ with probabilities $p_1,\ldots,p_K$. Its entropy is

$$H(X)=H(p)=-\sum_{k=1}^{K}p_k\log p_k.$$

Entropy is the expected surprisal:

$$H(X)=\mathbb{E}_{x\sim p}[-\log p(x)].$$

The convention $0\log 0=0$ is justified by the limit $\lim_{p\to0^+}p\log p=0$. An impossible outcome never occurs, so it contributes nothing to the expected surprise even though its individual surprisal would be infinite if it somehow occurred.

Entropy is lowest when a distribution is certain. If one class has probability one and all others have probability zero, then $H(p)=0$. There is no uncertainty because the outcome is already known. For $K$ possible classes, entropy is highest at the uniform distribution:

$$p_k=\frac{1}{K} \quad\Longrightarrow\quad H(p)=\log K.$$

In base 2, a fair coin has entropy one bit, a fair four-sided outcome has entropy two bits, and a uniform distribution over eight outcomes has entropy three bits. This gives entropy an operational interpretation: under ideal coding assumptions, it is the minimum expected number of bits needed to encode outcomes generated by the distribution. Frequent outcomes can receive short codes, while rare outcomes receive longer codes.

Entropy is a property of a distribution, not a property of a model's correctness. A low-entropy prediction is concentrated and confident, but it may be confidently wrong. A high-entropy prediction is uncertain, but that uncertainty may be appropriate when the input is genuinely ambiguous. To assess correctness, the predicted distribution must be compared with a target distribution; this leads to cross-entropy.

It is also important not to confuse the entropy of class labels with the entropy of every hidden representation. A dataset may have balanced labels and therefore high label entropy, yet contain features that make the labels easy to predict. Conversely, a highly imbalanced label distribution has low marginal entropy, but rare-class detection can remain difficult and operationally important.

Conditional entropy

Conditional entropy measures the uncertainty remaining in one variable after another is known:

$$H(Y\mid X) =-\sum_x p(x)\sum_y p(y\mid x)\log p(y\mid x).$$

If $X$ completely determines $Y$, then $H(Y\mid X)=0$. If knowing $X$ tells us nothing about $Y$, then $H(Y\mid X)=H(Y)$. Supervised learning can be viewed as searching for a function of the input that reduces uncertainty about the target. Even an optimal predictor cannot eliminate uncertainty caused by label noise, incomplete features, or genuinely stochastic outcomes.

Binary Entropy and Decision-Tree Splits

For a binary random variable with probability $p$ for class 1 and $1-p$ for class 0, Shannon entropy becomes the binary entropy function:

$$H_b(p)=-p\log_2p-(1-p)\log_2(1-p).$$

The function is symmetric around $p=0.5$. It equals zero at $p=0$ and $p=1$, where the label is certain, and reaches one bit at $p=0.5$, where the two labels are equally likely. This shape makes binary entropy a natural impurity measure for classification trees.

Suppose a tree node contains 10 examples: 6 positive and 4 negative. Its entropy is

$$H_{\text{parent}} =-0.6\log_2(0.6)-0.4\log_2(0.4) \approx0.971\text{ bits}.$$

Now consider a candidate split that produces a left child with four positive examples and no negatives, and a right child with two positives and four negatives. The left child has entropy zero. The right child's positive proportion is $2/6$, giving entropy about $0.918$ bits. The weighted post-split entropy is

$$H_{\text{children}} =\frac{4}{10}(0)+\frac{6}{10}(0.918) \approx0.551.$$

The information gain is

$$\operatorname{IG} =H_{\text{parent}}-H_{\text{children}} \approx0.971-0.551 =0.420\text{ bits}.$$

The split is useful because knowing which branch an example enters reduces uncertainty about its label by an average of $0.420$ bits. A tree-training algorithm compares many features and thresholds, then typically selects a split with large impurity reduction. You can explore this process interactively with the Decision Tree Calculator.

Information gain must be interpreted carefully. A feature with many distinct values can create tiny, nearly pure groups and appear highly informative on training data while generalizing poorly. Practical tree algorithms constrain depth, minimum leaf size, and other complexity controls. Entropy quantifies purity, but it does not remove the need for validation and regularization.

Cross-Entropy as a Classification Loss

Let $p$ be the true data distribution and $q$ be a model's predicted distribution. Their cross-entropy is

$$H(p,q)=-\sum_{k=1}^{K}p_k\log q_k =\mathbb{E}_{y\sim p}[-\log q(y)].$$

The first distribution supplies the weights, while the second appears inside the logarithm. This asymmetry matters. Cross-entropy asks: if outcomes actually follow $p$, how surprising are they under the model $q$?

In ordinary multiclass classification, each training label is represented by a one-hot vector. If the true class is $c$, then $p_c=1$ and $p_k=0$ for $k\ne c$. The per-example cross-entropy simplifies to

$$\mathcal{L}=-\log q_c.$$

Only the probability assigned to the observed class appears explicitly. The other probabilities still matter indirectly because they must sum to one; increasing probability on wrong classes leaves less probability for the correct class.

For a dataset of $N$ independently labeled examples, empirical cross-entropy is commonly averaged:

$$\mathcal{L}_{\text{dataset}} =-\frac{1}{N}\sum_{i=1}^{N}\log q_\theta(y_i\mid x_i).$$

Minimizing this expression is equivalent to maximizing the likelihood of the observed labels:

$$\arg\min_\theta\left[-\sum_i\log q_\theta(y_i\mid x_i)\right] =\arg\max_\theta\prod_i q_\theta(y_i\mid x_i).$$

Thus cross-entropy is not merely a convenient heuristic. Under the categorical probability model, it is the negative log-likelihood objective implied by maximum likelihood estimation.

For binary classification, if $y\in\{0,1\}$ and the model predicts $\hat p=P(Y=1\mid x)$, binary cross-entropy is

$$\mathcal{L}_{\text{BCE}} =-\left[y\log\hat p+(1-y)\log(1-\hat p)\right].$$

When $y=1$, this reduces to $-\log\hat p$. When $y=0$, it reduces to $-\log(1-\hat p)$. The formula therefore always evaluates the probability assigned to the observed outcome.

KL Divergence: Measuring Distribution Mismatch

Kullback–Leibler divergence compares a reference distribution $p$ with an approximating distribution $q$:

$$D_{\mathrm{KL}}(p\parallel q) =\sum_{k=1}^{K}p_k\log\frac{p_k}{q_k}.$$

It can be interpreted as the expected extra information, or coding cost, incurred when data generated from $p$ is represented using a code optimized for $q$. If $p=q$ everywhere, the ratio is one, every logarithm is zero, and the divergence is zero.

KL divergence is always nonnegative:

$$D_{\mathrm{KL}}(p\parallel q)\ge0,$$

with equality when the distributions agree almost everywhere. This is Gibbs' inequality. Despite this distance-like behavior, KL divergence is not a mathematical distance metric. In general,

$$D_{\mathrm{KL}}(p\parallel q) \ne D_{\mathrm{KL}}(q\parallel p),$$

and it does not satisfy the triangle inequality.

The direction has practical consequences. In $D_{\mathrm{KL}}(p\parallel q)$, outcomes that have substantial probability under $p$ strongly penalize a $q$ that assigns them little probability. Outcomes impossible under $p$ make no contribution, even if $q$ assigns them probability. Reversing the direction changes those priorities. This distinction appears in variational inference, knowledge distillation, and methods that approximate complex probability distributions.

There is also a support condition. If $p_k>0$ but $q_k=0$, then the term $p_k\log(p_k/q_k)$ is infinite. The model declared an outcome impossible even though the reference distribution says it can occur. Neural-network softmax outputs are mathematically positive, but finite-precision implementations can still underflow if probabilities are computed naively. Stable loss functions work directly with logits and log-sum-exp operations.

KL divergence can compare soft target distributions, not only one-hot labels. In knowledge distillation, a student network may learn from a teacher's probability distribution. In variational autoencoders, a KL term encourages an approximate posterior to remain near a chosen prior. In distribution shift analysis, KL can quantify a change in class proportions or feature distributions, provided probabilities are estimated reliably and support is handled carefully.

The Relationship Among Entropy, Cross-Entropy, and KL Divergence

The three central quantities are related by a simple identity:

$$H(p,q)=H(p)+D_{\mathrm{KL}}(p\parallel q).$$

To derive it, expand the logarithmic ratio:

$$\begin{aligned} D_{\mathrm{KL}}(p\parallel q) &=\sum_k p_k\log\frac{p_k}{q_k}\\ &=\sum_k p_k\log p_k-\sum_k p_k\log q_k\\ &=-H(p)+H(p,q). \end{aligned}$$

Rearranging yields the identity. The interpretation is equally important:

• $H(p)$ is the irreducible uncertainty inherent in the true distribution.
• $D_{\mathrm{KL}}(p\parallel q)$ is the additional cost caused by using the wrong distribution.
• $H(p,q)$ is the total expected surprise under the model.

During standard supervised training, the target distribution $p$ is fixed by the data, so $H(p)$ does not depend on the model parameters. Minimizing cross-entropy with respect to the model is therefore equivalent to minimizing $D_{\mathrm{KL}}(p\parallel q_\theta)$. The two objectives differ by a constant with respect to $\theta$.

The following summary distinguishes their roles.

For one-hot targets, the target entropy is zero. Cross-entropy and KL divergence then have the same numeric value for each example. For soft targets, target entropy is generally positive, so the values differ even though minimizing one still minimizes the other when the target is fixed.

Why Softmax and Cross-Entropy Belong Together

A multiclass neural classifier often produces one real-valued logit $z_k$ for each class. Logits are unconstrained: they can be negative, positive, and need not sum to anything. Softmax converts them into probabilities:

$$q_k=\operatorname{softmax}(\mathbf z)_k =\frac{e^{z_k}}{\sum_{j=1}^{K}e^{z_j}}.$$

Each $q_k$ is positive and the probabilities sum to one. Softmax also preserves ordering: a larger logit produces a larger probability. Adding the same constant to every logit does not change the result, because that common factor cancels from numerator and denominator.

Combining softmax with one-hot cross-entropy gives

$$\mathcal{L} =-\log\frac{e^{z_c}}{\sum_j e^{z_j}} =-z_c+\log\sum_j e^{z_j},$$

where $c$ is the true class. This expression says that training should raise the true-class logit relative to the aggregate of all logits.

The pairing has an especially clean derivative:

$$\frac{\partial\mathcal{L}}{\partial z_k}=q_k-y_k.$$

For the true class, the gradient is $q_c-1$, which is negative unless the model already assigns probability one; gradient descent therefore raises the true logit. For a wrong class, the gradient is $q_k$, so gradient descent lowers that logit in proportion to its predicted probability. The resulting signal is simple, informative, and well scaled.

Software libraries normally combine softmax and cross-entropy in a single operation. In PyTorch, for example, CrossEntropyLoss expects raw logits, not probabilities. The combined implementation evaluates the log-softmax stably by subtracting the largest logit and using the log-sum-exp identity. Applying softmax manually before such a loss is both redundant and potentially less stable.

For binary classification, the analogous pair is one logit passed conceptually through a sigmoid and evaluated with binary cross-entropy. Stable implementations such as binary cross-entropy with logits combine those steps. For multilabel classification, where several labels can be true independently, use one sigmoid per label rather than a single softmax across labels. Softmax represents mutually exclusive alternatives; sigmoids represent independent Bernoulli outcomes.

A Worked Numeric Example with Two Classes

Consider a tiny binary dataset with two examples:

• Example A has true class 1, and the model predicts $(q_0,q_1)=(0.10,0.90)$.
• Example B has true class 0, and the model predicts $(q_0,q_1)=(0.60,0.40)$.

Using natural logarithms, the loss for Example A is the negative log-probability of class 1:

$$\mathcal{L}_A=-\ln(0.90)\approx0.1054.$$

The loss for Example B is the negative log-probability of class 0:

$$\mathcal{L}_B=-\ln(0.60)\approx0.5108.$$

The mean cross-entropy is

$$\bar{\mathcal{L}} =\frac{0.1054+0.5108}{2} \approx0.3081\text{ nats}.$$

Both predictions choose the correct class, so accuracy is 100%. Cross-entropy nevertheless distinguishes their quality. Example A is confidently correct and receives a small loss. Example B is only moderately confident and receives a larger loss. Accuracy cannot express this difference because it reduces each probability distribution to one winning label.

Now suppose the second prediction changes to $(0.01,0.99)$ while its true class remains 0. The model is confidently wrong, and the second loss becomes

$$\mathcal{L}_B'=-\ln(0.01)\approx4.6052.$$

The new dataset average is approximately

$$\frac{0.1054+4.6052}{2}=2.3553.$$

One confident mistake dominates the mean. This is intentional: a calibrated probability of $0.01$ should correspond to an event that occurs only about once in one hundred comparable cases. If such events occur often, the model's probabilities are badly misleading.

The same computation can be reproduced with a numerically stable implementation:

import numpy as np

logits = np.array([
    [-1.09861229, 1.09861229],  # softmax -> [0.10, 0.90]
    [ 0.20273255, -0.20273255], # softmax -> [0.60, 0.40]
])
targets = np.array([1, 0])

shifted = logits - logits.max(axis=1, keepdims=True)
log_probs = shifted - np.log(np.exp(shifted).sum(axis=1, keepdims=True))
losses = -log_probs[np.arange(len(targets)), targets]

print("Per-example loss:", losses)
print("Mean cross-entropy:", losses.mean())

Subtracting each row's maximum does not alter softmax probabilities, but it prevents exponentials of large positive logits from overflowing. Production code should generally use the framework's fused cross-entropy function rather than implementing this operation manually.

Why Log-Loss Punishes Confident Wrong Answers

The logarithmic loss for the true-class probability $q_c$ is $-\log q_c$. Its behavior near the extremes explains its training effect:

$$-\log(0.9)\approx0.105,\qquad -\log(0.5)\approx0.693,\qquad -\log(0.1)\approx2.303,\qquad -\log(0.001)\approx6.908.$$

As $q_c$ approaches zero, the loss approaches infinity. The derivative with respect to the probability is

$$\frac{d}{dq_c}[-\log q_c]=-\frac{1}{q_c}.$$

Consequently, reducing the true-class probability from $0.2$ to $0.1$ matters far more than reducing it from $0.9$ to $0.8$. The first change moves toward declaring reality impossible; the second reflects a moderate reduction in confidence.

This strong penalty is not arbitrary harshness. Log-loss is a strictly proper scoring rule: in expectation, a forecaster minimizes it by reporting its genuine probability beliefs. If an event truly occurs 70% of the time, consistently reporting 70% achieves lower expected log-loss than consistently reporting 60% or 90%. This property encourages honest probability estimates rather than only correct rankings.

However, noisy labels can make the unbounded penalty problematic. If a mislabeled example is repeatedly treated as certain truth, the model may spend excessive capacity trying to fit it. Label smoothing, robust data cleaning, regularization, early stopping, and loss modifications can help. These techniques alter training behavior, but they should be chosen with an understanding of what probabilistic objective is being changed.

Cross-entropy also does not guarantee calibration. A flexible model can overfit and become overconfident, especially when evaluated outside its training distribution. Validation loss, reliability diagrams, expected calibration error, temperature scaling, and out-of-distribution testing provide complementary evidence. Low training cross-entropy alone is not proof that predicted probabilities are trustworthy.

Mutual Information Intuition

Mutual information measures how much knowing one variable reduces uncertainty about another. For random variables $X$ and $Y$,

$$I(X;Y)=H(Y)-H(Y\mid X).$$

By symmetry, it can also be written as

$$I(X;Y)=H(X)-H(X\mid Y).$$

If $X$ and $Y$ are independent, knowing $X$ does not reduce uncertainty about $Y$, so $I(X;Y)=0$. If $Y$ is completely determined by $X$, then $H(Y\mid X)=0$ and $I(X;Y)=H(Y)$. Mutual information therefore captures general statistical dependence, including nonlinear relationships that correlation may miss.

Another expression reveals a KL interpretation:

$$I(X;Y) =D_{\mathrm{KL}}\!\left(p(x,y)\parallel p(x)p(y)\right).$$

Mutual information is the mismatch between the actual joint distribution and the joint distribution that would exist if the variables were independent. It is always nonnegative.

In deep learning, mutual information provides a language for discussing useful representations. A hidden representation $Z$ should retain information about a target $Y$ that supports prediction. At the same time, it may be desirable for $Z$ to discard irrelevant details of the input $X$, producing a compact or invariant representation. This intuition motivates the information bottleneck perspective, contrastive objectives, representation learning, and feature-selection methods.

The concept is powerful, but estimation is difficult in high-dimensional continuous spaces. Neural mutual-information estimators can have substantial bias or variance, and deterministic networks create technical complications for some continuous definitions. Mutual-information claims about hidden layers should therefore be treated as model-dependent analyses, not as automatically measured facts.

Decision-tree information gain is closely related. A split variable $S$ indicates which child branch receives an example. The reduction $H(Y)-H(Y\mid S)$ is the mutual information between the branch and the label at that node. A good split is one whose branch assignment reveals something about the target.

Common Mistakes and Misconceptions

Using mean-squared error for ordinary classification

Mean-squared error is ideal when a Gaussian observation model and numeric deviations are appropriate. It can technically train a classifier, especially if outputs are constrained, but it is usually a poor default for categorical targets. Cross-entropy matches the Bernoulli or categorical likelihood, gives a direct probability interpretation, and typically provides stronger gradients when a sigmoid or softmax unit is confidently wrong.

With sigmoid output $\hat p=\sigma(z)$ and squared error $\frac{1}{2}(\hat p-y)^2$, the logit derivative contains the extra factor $\hat p(1-\hat p)$. When the sigmoid saturates near zero or one, this factor becomes tiny, even if the answer is wrong. Binary cross-entropy with a sigmoid yields the cleaner derivative $\hat p-y$, avoiding that additional saturation factor.

MSE is not universally forbidden. It can be appropriate for regression, probability distillation under a deliberately chosen objective, or specialized tasks. The mistake is using it automatically for mutually exclusive classification without considering the implied statistical model and gradient behavior.

Passing probabilities to a logits-based loss

Many APIs named “cross entropy” expect raw logits and internally apply log-softmax. Passing already normalized probabilities causes an unintended second transformation. Likewise, binary-cross-entropy-with-logits expects raw binary logits, whereas plain binary cross-entropy expects probabilities. Always confirm the interface contract.

Taking logarithms without numerical safeguards

Directly computing log(softmax(logits)) can overflow in the exponential or underflow to zero before the logarithm. Use a fused loss or a stable log-softmax. Adding a small epsilon to probabilities can prevent a crash in custom analysis code, but it changes the objective and is not a substitute for a stable formulation during training.

Mixing logarithm bases

Bits and nats differ by a constant factor:

$$\log_2 x=\frac{\ln x}{\ln2}.$$

Either unit is valid, but comparing values computed with different bases is misleading. Deep-learning frameworks typically report nats because they use natural logarithms.

Treating KL divergence as symmetric

The order of arguments is part of the definition. $D_{\mathrm{KL}}(p\parallel q)$ and $D_{\mathrm{KL}}(q\parallel p)$ answer different questions and can have dramatically different values. If a symmetric comparison is required, alternatives such as Jensen–Shannon divergence may be more suitable.

Confusing low entropy with high accuracy

A model can have low-entropy predictions because it is highly confident, yet have poor accuracy because those predictions are wrong. Prediction entropy measures concentration, not correctness. Cross-entropy against observed labels evaluates both confidence and correctness.

Ignoring reduction and class weighting

Loss functions may return a sum, mean, or per-example vector. Changing batch size alters the scale of a summed loss but not an ordinary mean. Class weights, sample weights, ignored labels, and label smoothing also change the objective. Report these settings when comparing experiments.

Assuming lower training loss means a better model

An overparameterized network may drive training cross-entropy nearly to zero while validation loss rises. Generalization requires evaluation on unseen data. Accuracy, precision, recall, calibration, robustness, and task-specific costs may all matter alongside cross-entropy.

Practice with Solver360

The formulas become more intuitive when their inputs can be changed and their consequences observed. Start with a two-class distribution and compute entropy at $p=0$, $0.1$, $0.5$, $0.9$, and $1$. Notice the symmetry and the maximum at equal probability. Then construct a small decision-tree node, try several candidate partitions, and compare their weighted child entropy with the parent entropy in the Decision Tree Calculator.

For neural classification, use the Neural Network Calculator to connect logits, activations, predictions, and loss. A useful experiment is to hold the correct-class logit fixed while increasing one incorrect-class logit. Softmax probability moves away from the true class, cross-entropy rises, and the gradient pushes against the competing logit. Then add the same constant to every logit and verify that the softmax probabilities do not change.

Another exercise is to compare models with identical accuracy but different confidence. Create four binary examples and arrange for both models to classify three correctly. Let one model use moderate probabilities such as $0.65$, while the other uses $0.99$ on every decision. The second model receives tiny losses on correct cases but an enormous loss on its mistake. Which model has lower average loss depends on the full set of probabilities, demonstrating why accuracy and cross-entropy measure different qualities.

Finally, explore label smoothing. Replace a one-hot target such as $(1,0,0)$ with a softened target such as $(0.9,0.05,0.05)$. Compute the target entropy, cross-entropy, and KL divergence. Their values no longer coincide, but the identity $H(p,q)=H(p)+D_{\mathrm{KL}}(p\parallel q)$ remains exact.

FAQ

Is entropy always measured in bits?

No. Entropy uses bits when the logarithm has base 2 and nats when it uses base $e$. Most deep-learning software uses natural logarithms and therefore reports losses in nats. Model rankings are unchanged when every value is converted consistently.

Can cross-entropy be zero?

With a one-hot target, cross-entropy approaches zero as the model assigns probability one to the correct class. A finite softmax generated from finite logits produces probabilities strictly between zero and one, so the loss is positive, though it can be extremely small. With a genuinely soft target having positive entropy, the minimum cross-entropy is $H(p)$ rather than zero.

Why can KL divergence be infinite?

If the reference distribution assigns positive probability to an event but the approximating distribution assigns exactly zero, observing that event would have infinite surprisal under the approximation. This creates an infinite KL term. Smoothing or support-aware modeling may be needed when estimated distributions contain zeros.

What is the difference between binary and categorical cross-entropy?

Binary cross-entropy models a Bernoulli outcome and is also used independently across labels in multilabel classification. Categorical cross-entropy models one outcome among mutually exclusive classes and is typically paired with softmax. A two-class softmax model and a one-logit sigmoid model can represent equivalent probabilities, though their parameterizations differ.

Does cross-entropy handle class imbalance automatically?

No. Standard cross-entropy averages the observed examples, so frequent classes contribute more terms. Class weighting, resampling, focal loss, threshold selection, and suitable evaluation metrics may be needed. The right choice depends on prevalence, error costs, and whether well-calibrated probabilities are required.

Why not optimize accuracy directly?

Accuracy depends on a discrete argmax or threshold and is piecewise constant with respect to small parameter changes. It supplies no useful gradient across most of parameter space. Cross-entropy is differentiable with respect to logits and provides a graded signal indicating both direction and confidence.

Is perplexity related to cross-entropy?

Yes. Perplexity is the exponential of average cross-entropy when natural logarithms are used:

$$\operatorname{Perplexity}=\exp(H(p,q)).$$

With base-2 cross-entropy, perplexity is $2^{H(p,q)}$. In language modeling, it can be interpreted loosely as the effective number of equally plausible next-token choices, although that interpretation should not be taken too literally across different tokenizations.

Does a lower cross-entropy guarantee better calibration?

Not by itself on a finite sample. Expected log-loss rewards calibrated, sharp probabilities, but a model can overfit training cross-entropy or be miscalibrated under distribution shift. Calibration should be checked on held-out, representative data using reliability analysis and proper scoring rules.

What does label smoothing change?

Label smoothing replaces exact one-hot targets with distributions that reserve some probability for non-target classes. It discourages infinitely separated logits and can improve generalization in some settings. It also changes the target distribution, the minimum attainable cross-entropy, and the interpretation of output confidence.

Are entropy and thermodynamic entropy the same?

They share a closely related mathematical form, but their contexts and units differ. Shannon entropy quantifies uncertainty in probability distributions and coding systems. Thermodynamic entropy describes physical macrostates and includes Boltzmann's constant. The conceptual connection is real, but the quantities should not be used interchangeably without specifying the model.

What Comes Next

The next natural topic is a dedicated softmax article. It can develop logit geometry, temperature, numerical stability, Jacobians, calibration, and the difference between multiclass and multilabel outputs in greater depth. Softmax explains how arbitrary neural scores become a categorical distribution; cross-entropy explains how that distribution is trained.

A subsequent Bayes article can connect prior beliefs, likelihoods, evidence, and posterior distributions. Bayesian reasoning complements the information-theoretic view: Bayes' rule updates a distribution after evidence arrives, while entropy and KL divergence quantify uncertainty and change between distributions. Together, these subjects provide a foundation for probabilistic deep learning, uncertainty estimation, variational inference, and principled decision-making.

The central lesson is compact. Information is surprise, entropy is average surprise within a distribution, cross-entropy is average surprise when one distribution predicts another, and KL divergence is the excess surprise caused by mismatch. Deep-learning classifiers turn logits into probabilities, then use these quantities to translate probabilistic error into an optimization signal. Once that chain is understood, the loss function stops looking like a black box and becomes a precise statement about what the model is being trained to believe.