Sigmoid, Softmax, and the Mathematics of Classification

Classification begins with a simple question: given measured features, which category is most plausible? The mathematical answer is less simple than selecting the largest raw model output. A classifier usually produces unrestricted real-valued scores, yet probabilities must obey strict rules: they must lie between zero and one, and the probabilities assigned to mutually exclusive classes must sum to one. Sigmoid and softmax functions create the bridge between those two worlds. Understanding that bridge clarifies logistic regression, neural-network output layers, decision boundaries, numerical stability, calibration, and the loss functions used to train modern classifiers.

Sigmoid, Softmax, and the Mathematics of Classification

The central pattern in classification is score first, normalize second. A model forms one or more scores from the input. An output transformation then converts those scores into quantities that can be interpreted as probabilities under the model. For binary classification, the logistic sigmoid converts one score into one probability. For multiclass classification, softmax converts a vector of scores into a probability distribution.

These transformations do more than make outputs look convenient. They encode assumptions about the target, define how scores compete, preserve useful decision boundaries, and interact directly with likelihood-based training. A careful treatment therefore begins before sigmoid or softmax is applied: with the raw score itself.

Why Linear Scores Are Not Probabilities

Suppose a model receives a feature vector $\mathbf{x}$ and computes a linear score

$$z=\mathbf{w}^{T}\mathbf{x}+b.$$

The score $z$ is often called a logit, although that term is most precise when $z$ is interpreted as log-odds. Nothing in the linear equation restricts $z$. Depending on the input and parameters, it can be $-20$, $0.7$, $14$, or any other real number.

A probability $p$, by contrast, must satisfy

$$0\leq p\leq 1.$$

If the classes are mutually exclusive and exhaustive, their probabilities must also satisfy

$$\sum_{k=1}^{K}p_k=1.$$

It would therefore be incorrect to call a raw score such as $z=3.8$ a probability. Clipping the score into the interval $[0,1]$ is not a satisfactory solution either. Clipping maps every score above one to exactly one and every score below zero to exactly zero, destroying information about relative confidence. It is also flat outside the interval, so its derivative is zero there. A gradient-based learning algorithm would receive no useful adjustment signal for many incorrect predictions.

Dividing several scores by their sum is also unreliable. Scores can be negative, their sum can be zero, and adding the same constant to every score changes the resulting ratios even though it should not change the classes' relative evidence. A sound probability mapping should handle every real score smoothly and preserve meaningful order.

The output transformation solves these problems. Sigmoid maps the real line smoothly to $(0,1)$. Softmax maps any real vector to the interior of the probability simplex: every component is positive and all components sum to one. Neither function proves that a model is correct or well calibrated. Instead, each gives a mathematically coherent probabilistic form to the model's scores.

The Logistic Sigmoid

Formula and range

The logistic sigmoid function is

$$\sigma(z)=\frac{1}{1+e^{-z}}.$$

Because $e^{-z}$ is always positive, the denominator is greater than one. Consequently,

$$0<\sigma(z)<1$$

for every finite $z$. The limiting values are

$$\lim_{z\to-\infty}\sigma(z)=0, \qquad \lim_{z\to+\infty}\sigma(z)=1.$$

At the origin,

$$\sigma(0)=\frac{1}{2}.$$

The function is strictly increasing, so larger scores always produce larger probabilities. It is also symmetric in the useful sense

$$\sigma(-z)=1-\sigma(z).$$

Thus a score of $-2$ assigns the positive class the same probability that a score of $2$ assigns the negative class.

The sigmoid's S-shaped curve has three qualitative regions. Around zero, probability changes substantially when the score changes. At large positive scores, the curve approaches one. At large negative scores, it approaches zero. These outer regions are called saturated regions because increasingly large score changes produce very small probability changes.

Derivative

The derivative has an especially compact form:

$$\frac{d\sigma(z)}{dz} =\sigma(z)\left(1-\sigma(z)\right).$$

To see why, write $\sigma(z)=(1+e^{-z})^{-1}$ and differentiate:

$$\sigma'(z) =\frac{e^{-z}}{(1+e^{-z})^2}.$$

Since

$$1-\sigma(z)=\frac{e^{-z}}{1+e^{-z}},$$

the product $\sigma(z)(1-\sigma(z))$ gives the same result.

The derivative reaches its maximum at $\sigma(z)=0.5$, which occurs at $z=0$:

$$\sigma'(0)=0.5(1-0.5)=0.25.$$

As $|z|$ grows, the derivative approaches zero. This property matters in optimization. A sigmoid used repeatedly in deep hidden layers can contribute to vanishing gradients because saturated units transmit very small derivatives. At a binary output layer paired with a suitable cross-entropy loss, however, the algebra simplifies and sigmoid remains a natural choice.

Binary probability representation

For a binary target $Y\in\{0,1\}$, define

$$P(Y=1\mid\mathbf{x})=\sigma(z),$$

where $z=\mathbf{w}^{T}\mathbf{x}+b$. The other class probability does not require a second independent output:

$$P(Y=0\mid\mathbf{x})=1-\sigma(z).$$

The two values automatically sum to one. A threshold of $0.5$ corresponds to predicting class one whenever $z\geq 0$, because $\sigma(0)=0.5$ and sigmoid is increasing. Other thresholds may be preferable when false positives and false negatives have different costs.

Logit, Odds, and Log-Odds

Probability is one way to express uncertainty. Odds are another. For an event with probability $p$, the odds in favor of the event are

$$\operatorname{odds}(p)=\frac{p}{1-p}.$$

If $p=0.75$, the odds are $0.75/0.25=3$, often stated as three to one. If $p=0.2$, the odds are $0.2/0.8=0.25$, or one to four. Probability $0.5$ corresponds to odds of one.

The logit function takes the natural logarithm of the odds:

$$\operatorname{logit}(p) =\log\left(\frac{p}{1-p}\right).$$

Probabilities in $(0,1)$ map to every real number. Probabilities below $0.5$ have negative logits, $p=0.5$ has logit zero, and probabilities above $0.5$ have positive logits. Sigmoid is the inverse of the logit function:

$$p=\sigma(z) \quad\Longleftrightarrow\quad z=\log\left(\frac{p}{1-p}\right).$$

This inverse relationship explains why a linear score can sensibly model a binary outcome. Logistic regression does not claim that probability itself is linear in the features. It claims that the log-odds are linear:

$$\log\left( \frac{P(Y=1\mid\mathbf{x})} {P(Y=0\mid\mathbf{x})} \right) =\mathbf{w}^{T}\mathbf{x}+b.$$

The coefficients have a multiplicative interpretation on the odds scale. Holding other features fixed, increasing feature $x_j$ by one unit increases the log-odds by $w_j$ and multiplies the odds by $e^{w_j}$. If $w_j=0.7$, the odds multiplier is approximately $e^{0.7}\approx2.01$. This does not mean the probability doubles; the same odds change produces different probability changes depending on the starting probability.

The distinction among a score, odds, log-odds, and probability prevents many interpretation errors. In a properly specified binary logistic model, the linear score is the log-odds, sigmoid converts it to probability, and exponentiation converts it to odds.

Softmax as the Multiclass Generalization

For $K$ mutually exclusive classes, a model produces a score vector

$$\mathbf{z}=(z_1,z_2,\ldots,z_K).$$

The softmax function defines the probability of class $k$ as

$$\operatorname{softmax}(\mathbf{z})_k =\frac{e^{z_k}}{\sum_{j=1}^{K}e^{z_j}}.$$

Every exponential is positive, so every resulting probability is positive. Summing over classes gives

$$\sum_{k=1}^{K} \frac{e^{z_k}}{\sum_{j=1}^{K}e^{z_j}} =1.$$

Softmax therefore turns arbitrary real scores into a valid categorical distribution. It also preserves rank: if $z_a>z_b$, then $p_a>p_b$. The class with the largest score remains the class with the largest softmax probability.

The word “soft” contrasts this operation with a hard maximum. A hard maximum could assign probability one to the largest score and zero to every other class. Softmax instead gives all classes some probability while favoring larger scores exponentially. As score differences grow, its output approaches a one-hot distribution.

Relative scores matter

Softmax probabilities depend on score differences, not absolute score levels. Adding the same constant $c$ to every score changes nothing:

$$\frac{e^{z_k+c}}{\sum_j e^{z_j+c}} = \frac{e^c e^{z_k}}{e^c\sum_j e^{z_j}} = \frac{e^{z_k}}{\sum_j e^{z_j}}.$$

This shift invariance is both conceptually important and computationally useful. A vector $(2,1,0)$ yields the same probabilities as $(102,101,100)$ and $(-8,-9,-10)$. Softmax interprets how each class compares with the alternatives.

The probability ratio between classes $a$ and $b$ is

$$\frac{p_a}{p_b}=e^{z_a-z_b}.$$

Taking logarithms yields

$$\log\left(\frac{p_a}{p_b}\right)=z_a-z_b.$$

Thus differences between multiclass logits represent pairwise log-odds. A one-unit advantage in score means an odds ratio of $e\approx2.718$ between those two classes, independent of the other scores.

Sigmoid as a two-class softmax

For two classes with logits $z_0$ and $z_1$, softmax assigns

$$p_1=\frac{e^{z_1}}{e^{z_0}+e^{z_1}}.$$

Divide numerator and denominator by $e^{z_1}$:

$$p_1 =\frac{1}{e^{z_0-z_1}+1} =\sigma(z_1-z_0).$$

A two-logit softmax is therefore equivalent to applying sigmoid to the difference between the logits. Binary systems commonly use one sigmoid logit because it avoids a redundant degree of freedom. Both representations are valid when implemented consistently with the target encoding and loss.

Sigmoid and Softmax at a Glance

The two functions solve related but distinct prediction problems.

The multilabel distinction is essential. An image may contain both a bicycle and a person, so those labels are not mutually exclusive. Separate sigmoid outputs allow both probabilities to be high. If exactly one label must be selected—cat, dog, or bird—softmax expresses the required competition.

Temperature and Probability Peakiness

Temperature introduces a positive scale parameter $T$:

$$p_k(T) =\frac{e^{z_k/T}}{\sum_j e^{z_j/T}}, \qquad T>0.$$

When $T=1$, this is ordinary softmax. If $0<T<1$, dividing by $T$ enlarges score differences. The distribution becomes sharper or more peaked, placing more mass on the largest logit. As $T\to0^+$, softmax approaches a hard argmax when there is a unique largest score.

If $T>1$, score differences shrink. The distribution becomes softer and more uniform. As $T\to\infty$, every finite score divided by $T$ approaches zero, and the output approaches

$$p_k=\frac{1}{K}.$$

Temperature is used in several settings. In knowledge distillation, a higher temperature exposes similarities among nonwinning classes in a teacher model's output. In probabilistic sampling, temperature adjusts diversity. In post-hoc calibration, a temperature parameter can be fitted on validation data to reduce overconfidence or underconfidence without changing the ranking of logits.

Temperature does not add information. A low value can make a weak preference look nearly certain, while a high value can obscure a strong preference. It should be selected for a defined objective rather than treated as a cosmetic confidence control. For calibration, it must be learned on data separate from the training set and evaluated on unseen examples.

A corresponding temperature-scaled sigmoid can be written as

$$\sigma\left(\frac{z}{T}\right).$$

The same intuition applies: lower temperature makes the transition around zero steeper, while higher temperature flattens it.

Numerical Stability and Log-Sum-Exp

The formulas for sigmoid and softmax are exact on paper, but computers represent numbers with finite ranges. Directly evaluating $e^{1000}$ overflows in ordinary floating-point arithmetic. Evaluating $e^{-1000}$ underflows to zero. A robust implementation must preserve the mathematics while avoiding dangerous intermediate values.

Stable softmax

Softmax shift invariance supplies the standard remedy. Let

$$m=\max_j z_j.$$

Subtract $m$ from every score:

$$p_k =\frac{e^{z_k-m}}{\sum_j e^{z_j-m}}.$$

The result is mathematically unchanged. Computationally, however, the largest exponent is now $e^0=1$, and every other exponent lies between zero and one. This prevents positive overflow.

The log-sum-exp identity

Many losses require the logarithm of a sum of exponentials:

$$\operatorname{LSE}(\mathbf{z}) =\log\left(\sum_j e^{z_j}\right).$$

The stable form is

$$\operatorname{LSE}(\mathbf{z}) =m+\log\left(\sum_j e^{z_j-m}\right), \qquad m=\max_j z_j.$$

This identity is known as the log-sum-exp trick. It appears directly in multiclass negative log-likelihood. If the true class is $y$, then

$$-\log p_y =-\log\left(\frac{e^{z_y}}{\sum_j e^{z_j}}\right) =-z_y+\operatorname{LSE}(\mathbf{z}).$$

Computing the loss from logits with a fused cross-entropy routine is usually more stable than first calculating probabilities and then taking their logarithms. Mature machine-learning libraries provide operations such as “cross entropy with logits” for exactly this reason.

Stable sigmoid

The direct sigmoid formula can overflow when a large negative $z$ makes $e^{-z}$ enormous. A stable piecewise evaluation is

$$\sigma(z)= \begin{cases} \dfrac{1}{1+e^{-z}}, & z\geq0,\\[6pt] \dfrac{e^z}{1+e^z}, & z<0. \end{cases}$$

For binary cross-entropy, stable software again works directly with logits rather than manually computing $\log(\sigma(z))$ or $\log(1-\sigma(z))$ near the endpoints.

The following compact Python example demonstrates stable sigmoid, softmax, and log-sum-exp calculations without relying on a machine-learning framework:

import math

def stable_sigmoid(z):
    if z >= 0:
        return 1.0 / (1.0 + math.exp(-z))
    exp_z = math.exp(z)
    return exp_z / (1.0 + exp_z)

def stable_softmax(logits, temperature=1.0):
    if temperature <= 0:
        raise ValueError("temperature must be positive")
    scaled = [z / temperature for z in logits]
    maximum = max(scaled)
    exponents = [math.exp(z - maximum) for z in scaled]
    total = sum(exponents)
    return [value / total for value in exponents]

def log_sum_exp(logits):
    maximum = max(logits)
    return maximum + math.log(
        sum(math.exp(z - maximum) for z in logits)
    )

binary_probability = stable_sigmoid(1.2)
class_probabilities = stable_softmax([2.0, 1.0, -0.5])

Decision Boundaries

A probability transformation changes the scale of a score, but it does not necessarily make the decision boundary nonlinear. Consider binary logistic regression:

$$p(\mathbf{x})=\sigma(\mathbf{w}^{T}\mathbf{x}+b).$$

With a threshold of $0.5$, the model predicts class one when

$$\sigma(\mathbf{w}^{T}\mathbf{x}+b)\geq0.5.$$

Because sigmoid is monotonic and $\sigma(0)=0.5$, this is equivalent to

$$\mathbf{w}^{T}\mathbf{x}+b\geq0.$$

The decision boundary is therefore

$$\mathbf{w}^{T}\mathbf{x}+b=0,$$

a line in two dimensions, a plane in three, and a hyperplane in higher dimensions. Sigmoid makes the output probabilistic, but it does not curve a boundary formed from a linear score.

If the classification threshold is some value $\tau$ other than $0.5$, then the boundary becomes

$$\mathbf{w}^{T}\mathbf{x}+b =\log\left(\frac{\tau}{1-\tau}\right).$$

Changing the threshold shifts the boundary parallel to itself; it does not change its orientation or shape.

For multiclass linear classification, each class can have a score

$$z_k=\mathbf{w}_k^{T}\mathbf{x}+b_k.$$

The predicted class is $\arg\max_k z_k$, which is also $\arg\max_k p_k$. The boundary between classes $a$ and $b$ occurs where their scores are equal:

$$\mathbf{w}_a^{T}\mathbf{x}+b_a =\mathbf{w}_b^{T}\mathbf{x}+b_b.$$

Rearranging gives

$$(\mathbf{w}_a-\mathbf{w}_b)^{T}\mathbf{x} +(b_a-b_b)=0.$$

This is again a hyperplane. A neural network can create nonlinear decision boundaries because its hidden layers transform $\mathbf{x}$ nonlinearly before the final linear logits are formed. The last sigmoid or softmax then converts those logits to probabilities; it is not by itself the source of the network's nonlinear geometry.

Worked Binary Example

Suppose a binary classifier predicts whether a student will pass an assessment from two standardized features:

• $x_1$: hours of focused practice,
• $x_2$: score on a prerequisite quiz.

Let the model be

$$z=0.8x_1+1.2x_2-1.0.$$

For a student with $x_1=1.5$ and $x_2=0.5$,

$$\begin{aligned} z &=0.8(1.5)+1.2(0.5)-1.0\\ &=1.2+0.6-1.0\\ &=0.8. \end{aligned}$$

The estimated probability of passing is

$$\begin{aligned} p &=\sigma(0.8)\\ &=\frac{1}{1+e^{-0.8}}\\ &\approx\frac{1}{1+0.4493}\\ &\approx0.6900. \end{aligned}$$

The model therefore assigns approximately $69.0\%$ probability to passing and $31.0\%$ probability to not passing. The odds are

$$\frac{p}{1-p} \approx\frac{0.6900}{0.3100} \approx2.2255.$$

The same result follows from exponentiating the logit:

$$e^{0.8}\approx2.2255.$$

At the usual threshold of $0.5$, the predicted class is “pass” because $z>0$. If an intervention program wants to flag any student whose pass probability is below $0.75$, the student would be flagged despite the positive-class prediction. This illustrates why a class label and an operational decision need not use the same threshold.

The $0.75$ probability threshold corresponds to a logit threshold of

$$\operatorname{logit}(0.75) =\log\left(\frac{0.75}{0.25}\right) =\log(3) \approx1.0986.$$

The current logit $0.8$ is below $1.0986$, so the two calculations agree.

Worked Three-Class Example

Now suppose an image classifier distinguishes among a circle, square, and triangle. Its logits for one image are

$$\mathbf{z}=(2.0,\ 1.0,\ -0.5).$$

The largest logit belongs to “circle,” but probabilities require normalization. For stable computation, subtract the maximum value $2.0$:

$$\mathbf{z}-2.0=(0,\ -1.0,\ -2.5).$$

Exponentiating gives approximately

$$(e^0,e^{-1},e^{-2.5}) \approx(1,\ 0.3679,\ 0.0821).$$

Their sum is

$$1+0.3679+0.0821=1.4500.$$

The softmax probabilities are therefore

$$\begin{aligned} P(\text{circle})&\approx\frac{1}{1.4500}=0.6897,\\ P(\text{square})&\approx\frac{0.3679}{1.4500}=0.2537,\\ P(\text{triangle})&\approx\frac{0.0821}{1.4500}=0.0566. \end{aligned}$$

The values sum to one, apart from rounding. Notice that a one-unit logit difference between circle and square produces the probability ratio

$$\frac{0.6897}{0.2537}\approx2.718\approx e^1.$$

The difference between circle and triangle is $2.5$, producing a ratio near $e^{2.5}\approx12.182$.

To see temperature in action, take $T=2$. The scaled logits are

$$(1.0,\ 0.5,\ -0.25).$$

After subtracting the maximum, exponentiating, and normalizing, the distribution is approximately

$$(0.5283,\ 0.3204,\ 0.1513).$$

The winning class is unchanged, but confidence is spread more broadly. With $T=0.5$, the logits double before softmax, producing a much sharper distribution of approximately

$$(0.8756,\ 0.1185,\ 0.0059).$$

Again, no new evidence has entered the model. Temperature has only changed how strongly existing score differences are expressed.

Connection to Logistic Regression

Logistic regression combines a linear predictor with the sigmoid:

$$P(Y=1\mid\mathbf{x}) =\sigma(\mathbf{w}^{T}\mathbf{x}+b).$$

Despite its name, it is a classification model. “Regression” refers to modeling the log-odds as a weighted sum. Training usually chooses parameters by maximizing the conditional likelihood of the observed labels, equivalently minimizing binary cross-entropy:

$$\mathcal{L} =-\sum_{i=1}^{n} \left[ y_i\log p_i+(1-y_i)\log(1-p_i) \right].$$

This objective strongly penalizes confident wrong predictions. It is also a proper scoring rule: under appropriate conditions, expected loss is minimized by reporting the true conditional probability.

Multinomial logistic regression uses one linear score per class and applies softmax:

$$P(Y=k\mid\mathbf{x}) = \frac{\exp(\mathbf{w}_k^{T}\mathbf{x}+b_k)} {\sum_j\exp(\mathbf{w}_j^{T}\mathbf{x}+b_j)}.$$

Its usual categorical cross-entropy loss for one-hot targets is

$$\mathcal{L} =-\sum_{i=1}^{n}\sum_{k=1}^{K} y_{ik}\log p_{ik}.$$

Because only the true class has $y_{ik}=1$, each observation contributes the negative logarithm of the probability assigned to its correct class.

The Logistic Regression Calculator can help connect coefficients, logits, and probabilities. When checking an individual exponential, logarithm, odds conversion, or rounded intermediate value, the Scientific Calculator is useful for reproducing the arithmetic step by step.

Connection to Neural-Network Output Layers

A neural network's hidden layers learn a representation $\mathbf{h}(\mathbf{x})$. The final affine layer commonly computes

$$\mathbf{z}=W\mathbf{h}+\mathbf{b}.$$

For binary classification, one output logit and a sigmoid are typical. For mutually exclusive multiclass classification, $K$ logits and softmax are typical. For multilabel classification, $K$ logits are each passed through an independent sigmoid.

This separation clarifies the roles of network components:

• Hidden layers learn nonlinear features.
• The last affine layer converts those features into class scores.
• Sigmoid or softmax converts scores into a probabilistic form.
• The loss compares predictions with targets and supplies a training signal.

During training, many libraries expect raw logits rather than already transformed probabilities. A binary-cross-entropy-with-logits operation internally combines sigmoid and binary cross-entropy. A categorical-cross-entropy operation may internally combine log-softmax and negative log-likelihood. These fused forms are stable and avoid redundant computation.

At inference time, the application may require explicit probabilities for display, thresholding, ranking, or decision analysis. For a simple top-class prediction, applying softmax is mathematically unnecessary because $\arg\max$ of logits and probabilities is identical. It may still be necessary if downstream code needs a probability distribution.

Care is required with output dimension and label encoding. A single sigmoid output pairs naturally with binary labels such as zero and one. Two softmax outputs pair naturally with a two-class categorical target. For three mutually exclusive classes, three softmax outputs are required. For three independent labels, three sigmoid outputs are required.

Calibration: Probabilities Are Claims

A classifier is calibrated when its stated probabilities match empirical frequencies. Among predictions assigned probability near $0.8$, approximately $80\%$ should be correct. Calibration is different from accuracy and ranking. A model can rank examples extremely well but be systematically overconfident. Another model can be calibrated while having weak discrimination because it assigns nearly the same base-rate probability to everyone.

Sigmoid and softmax guarantee valid numerical ranges, not calibration. A softmax output of $0.99$ means the model's mathematical output is $0.99$; it does not guarantee a $99\%$ real-world success frequency. Miscalibration can arise from overfitting, distribution shift, class imbalance, label noise, model misspecification, aggressive regularization, or training objectives that do not match deployment conditions.

Calibration should be evaluated on held-out data. Reliability diagrams group predictions into probability bins and compare mean confidence with observed frequency. The Brier score measures squared probability error. Negative log-likelihood is sensitive to confident mistakes. Expected calibration error offers a summary, although its value depends on binning and should not be treated as a complete diagnosis.

Post-hoc methods include temperature scaling, Platt scaling, and isotonic regression. Temperature scaling is simple and often effective for multiclass neural networks, but it mainly corrects global confidence. Isotonic regression can learn a more flexible monotonic mapping but may overfit small calibration sets. No calibration procedure protects indefinitely against a changed population. Monitoring remains necessary after deployment.

Decision thresholds should also be selected from costs and constraints, not from calibration alone. A calibrated disease-screening model may still use a low threshold to favor sensitivity. A spam filter may use a high threshold when false positives are expensive. Probability estimation and decision policy are connected but distinct stages.

Common Mistakes

Applying softmax to already normalized probabilities

Softmax is designed for unrestricted logits, not probabilities that already sum to one. If probabilities $(0.8,0.1,0.1)$ are passed through softmax, the result is approximately $(0.502,0.249,0.249)$, not the original distribution. Exponentiation and renormalization compress the differences because numbers in $[0,1]$ occupy a narrow range. Applying softmax twice therefore changes the prediction and usually makes it less decisive.

The correct question is always: “Does this function receive logits or probabilities?” Variable names, library documentation, and model architecture should make that distinction explicit.

Treating independent labels as competing classes

Softmax is inappropriate when several labels can be true simultaneously. In a medical record, hypertension and diabetes may both be present. A softmax layer forces their probabilities to compete and sum to one. Independent sigmoid outputs represent the task more faithfully.

Using independent sigmoids for one-of-many classes

The reverse mistake is also common. If exactly one class is correct, independent sigmoids do not enforce a sum of one and can assign high probability to several classes. This can be useful in specially designed systems, but standard mutually exclusive classification calls for softmax.

Confusing logits with probabilities

Raw values such as $(-1.4,2.1,0.3)$ are not invalid model outputs; they are valid logits. They become invalid only when mislabeled as probabilities. Thresholding a binary logit at $0.5$ is another version of this error. A $0.5$ probability threshold corresponds to a logit threshold of zero.

Computing unstable exponentials manually

Directly exponentiating large logits may overflow, and taking the logarithm of a rounded zero probability may produce an infinite loss. Subtract the maximum for softmax, use log-sum-exp, and prefer fused loss functions that consume logits.

Interpreting the largest probability as certainty

Softmax always distributes all mass among the listed classes, even if the input is unlike anything seen during training. If a model knows only “cat,” “dog,” and “bird,” an image of a tractor still receives probabilities summing to one across those three labels. The maximum may be high without indicating that the input belongs to any known class. Out-of-distribution detection and abstention require additional design.

Assuming a nonlinear activation creates a nonlinear boundary

Sigmoid and softmax are nonlinear functions, but when applied to linear logits they preserve linear class boundaries. Nonlinear boundaries require nonlinear features, kernels, trees, hidden layers, or other transformations before the output mapping.

Rounding too early

Rounding exponentials or probabilities during intermediate calculations can noticeably alter the final distribution, especially when values are close. Keep full precision during computation and round only for presentation.

Comparing logits across unrelated models

Logit scales can differ among models because of training, regularization, architecture, and temperature. A logit of four from one model is not inherently more trustworthy than a logit of two from another. Compare models through validated probabilities, metrics, and calibration behavior on the same data.

Practice with Solver360

The most effective practice moves back and forth among four representations: score, probability, odds, and decision. Begin with a simple binary logit such as $z=-1.5$. Compute $\sigma(z)$, convert the result to odds, take the log-odds, and verify that the original score returns. Repeat with $z=0$ and a positive score. The symmetry $\sigma(-z)=1-\sigma(z)$ provides a useful check.

Next, choose a two-feature logistic model and calculate its score for several points. Plot or reason about the line $\mathbf{w}^{T}\mathbf{x}+b=0$. Confirm that points on one side have probabilities above $0.5$ and points on the other have probabilities below $0.5$. Then replace the threshold with $0.8$ and calculate the corresponding logit. Observe that the boundary shifts without rotating.

For multiclass practice, start with three logits such as $(3,2,1)$. Subtract the maximum, exponentiate, normalize, and verify that the probabilities sum to one. Add $100$ to every logit and confirm that the result is unchanged. Multiply the logits by two and observe that this is not shift invariance: scaling changes their differences and makes the distribution sharper.

Use the Scientific Calculator to check values such as $e^{-1}$, $\log(3)$, and probability ratios. Use the Logistic Regression Calculator to investigate how feature values and coefficients combine into binary predictions. Do not merely record the displayed probability. Predict first whether each coefficient change should raise or lower the logit, then verify the numeric result.

A useful self-study sequence is:

1. Calculate sigmoid values from logits by hand.
2. Recover logits from probabilities using log-odds.
3. Compare two-class softmax with a sigmoid of the logit difference.
4. Calculate a stable three-class softmax.
5. Apply two temperatures and explain the change.
6. Derive a binary boundary for two features.
7. Distinguish multiclass from multilabel output design.
8. Explain why valid probabilities can still be miscalibrated.

The goal is not memorizing isolated formulas. It is learning which mathematical object exists at each stage of a classifier and which operation legitimately transforms it into the next.

Frequently Asked Questions

Is sigmoid the same as logistic regression?

No. Sigmoid is a mathematical function that maps a real number to $(0,1)$. Logistic regression is a statistical model that computes a linear score from features and then applies sigmoid. Sigmoid can also appear in neural-network outputs, hidden units, gating mechanisms, and other models.

Why are raw model outputs called logits?

In binary logistic regression, the raw score equals the log-odds, or logit, of the positive-class probability. In multiclass models, individual scores are commonly called logits because pairwise score differences equal pairwise log-odds. In practice, “logit” often means any pre-sigmoid or pre-softmax score.

Do softmax probabilities always sum to one?

Yes, apart from tiny floating-point rounding errors, because every exponential is divided by the sum of all exponentials. That guarantee applies to the mathematical output of softmax. It does not establish that the classes are complete, mutually exclusive in reality, or probabilistically calibrated.

Can sigmoid be used for multiclass classification?

It can be used when the task is multilabel, meaning several classes may be true at once. Use one sigmoid per label. For ordinary multiclass classification with exactly one correct class, softmax is usually the appropriate output because it models competition and enforces a total probability of one.

Does softmax change the predicted class?

Not when prediction is based on the largest component. Exponentiation is strictly increasing and the denominator is shared, so the largest logit has the largest softmax probability. Softmax changes the scale and provides normalized probabilities, but $\arg\max$ remains the same.

Why subtract the maximum logit?

Subtracting the maximum leaves softmax unchanged because adding or subtracting one common constant cancels between numerator and denominator. It prevents very large positive exponents from overflowing and makes the computation reliable.

What does temperature do to softmax?

A temperature below one sharpens the distribution; a temperature above one softens it. Positive temperature scaling preserves class order, so it does not change the top class. It changes confidence and can be fitted as a post-hoc calibration parameter.

Is a softmax value of 0.9 a true 90 percent chance?

It is the probability implied by the model's current logits, but its real-world interpretation depends on calibration and data relevance. If the model is overconfident or the deployment distribution has shifted, outcomes assigned $0.9$ may occur much less than $90\%$ of the time.

Should cross-entropy receive logits or probabilities?

That depends on the software interface. Many library functions named “cross entropy” or “with logits” expect raw logits and internally apply a stable log-softmax or sigmoid calculation. Other functions expect probabilities or log-probabilities. Passing the wrong representation can apply an activation twice or produce incorrect losses. Always check the documented contract.

Why not simply normalize logits by dividing by their sum?

Logits can be negative and can sum to zero. Simple division can therefore produce negative values, values outside $[0,1]$, or undefined results. Softmax first exponentiates scores to make them positive and then normalizes them, while preserving useful relationships through log-odds differences.

How are decision thresholds chosen?

The default binary threshold of $0.5$ is convenient, not universal. A threshold should reflect false-positive and false-negative costs, class prevalence, capacity constraints, safety requirements, and the intended metric. It should be selected on validation data and rechecked after deployment.

Does a higher logit always mean a better calibrated model?

No. Within one model, a higher binary logit means a higher predicted positive-class probability, and a higher class logit tends to increase that class's softmax share relative to others. Across different models, logit magnitudes are not directly comparable. Calibration must be evaluated empirically.

What Comes Next: Loss Functions and Neural Networks

Sigmoid and softmax explain how scores become probabilities, but they do not explain how the scores are learned. That requires a loss function. Binary cross-entropy pairs naturally with sigmoid, while categorical cross-entropy pairs naturally with softmax. Their gradients reward increasing the correct class's logit and decreasing competing logits, with the size of the adjustment depending on the prediction error.

The next mathematical step is to derive those losses from maximum likelihood and inspect their gradients. For softmax with cross-entropy, a particularly elegant result appears: the derivative of the loss with respect to logit $z_k$ is

$$\frac{\partial\mathcal{L}}{\partial z_k}=p_k-y_k.$$

For binary sigmoid with cross-entropy, the corresponding derivative is also prediction minus target:

$$\frac{\partial\mathcal{L}}{\partial z}=p-y.$$

These compact errors are propagated backward through a neural network by the chain rule. Hidden layers then adjust the representation that produced the logits. Regularization, optimization algorithms, batch construction, class weighting, and validation all influence the resulting classifier.

The larger lesson is that classification is a connected mathematical system. Linear or neural computations create logits. Sigmoid or softmax gives those logits probabilistic structure. Cross-entropy evaluates the assigned probability of observed outcomes. Gradients move parameters toward better predictions. Calibration checks whether reported confidence deserves a real-world probability interpretation. Once these links are understood, output layers stop being arbitrary implementation choices and become the natural final step in a coherent model of classification.