Logistic Regression Explained: Sigmoid Function, Odds, and Binary Classification

Logistic regression is the standard first model for binary classification. It predicts a probability that an example belongs to the positive class, then converts that probability into a yes/no decision with a threshold. The method is closely related to linear regression, but it is built for labels such as spam/not spam, default/no default, or disease/no disease rather than for a continuous number.

This logistic regression tutorial develops the sigmoid function, log-odds, training with cross-entropy, evaluation metrics, and the mistakes that make a “95% accurate” model useless.

What logistic regression actually predicts

A linear model for a binary label $y\in\{0,1\}$ can output any real number. That is a problem: a probability must lie in $(0,1)$. Logistic regression therefore passes a linear score through the sigmoid function:

$$p(y=1\mid\mathbf{x})=\sigma(z)=\frac{1}{1+e^{-z}},\qquad z=\beta_0+\boldsymbol{\beta}^\top\mathbf{x}.$$

The quantity $z$ is a logit. Large positive logits become probabilities near 1; large negative logits become probabilities near 0. At $z=0$, $\sigma(0)=0.5$.

The complementary probability is automatic:

$$p(y=0\mid\mathbf{x})=1-\sigma(z).$$

Unlike k-nearest neighbors, logistic regression is a parametric model: after training you keep only the coefficients. Prediction is a dot product plus a sigmoid — cheap enough for real-time scoring.

From odds to log-odds

Odds are the ratio of success probability to failure probability:

$$\mathrm{odds}=\frac{p}{1-p}.$$

If $p=0.8$, the odds are $4:1$. Taking a logarithm produces log-odds, which logistic regression models as a linear function of the features:

$$\log\frac{p}{1-p}=\beta_0+\boldsymbol{\beta}^\top\mathbf{x}.$$

That identity is why the method is called logistic regression: it is linear in log-odds, not in the probability itself. A coefficient $\beta_j=0.7$ means a one-unit increase in feature $j$ multiplies the odds by $e^{0.7}\approx 2.01$, holding other features fixed. Interpret coefficients on the odds scale, not as “+0.7 probability.”

How the model is trained

Maximum likelihood for Bernoulli labels is equivalent to minimizing binary cross-entropy (log loss):

$$L(\boldsymbol{\beta})=-\frac{1}{n}\sum_{i=1}^{n}\Big[y_i\log p_i+(1-y_i)\log(1-p_i)\Big].$$

There is no OLS-style closed form. Gradient descent (or Newton/IRLS) updates coefficients so that predicted probabilities move toward the observed labels. L2 regularization (Ridge-style) is common:

$$L_{\lambda}(\boldsymbol{\beta})=L(\boldsymbol{\beta})+\frac{\lambda}{2}\|\boldsymbol{\beta}\|_2^2.$$

Regularization shrinks coefficients, which helps when features are correlated or when $p$ is large relative to $n$.

Try a fit in the Logistic Regression Calculator: change the decision threshold and watch precision, recall, and the confusion matrix move even though the underlying sigmoid is unchanged.

Decision thresholds are not always 0.5

The model outputs a probability. The class label is a separate policy:

$$\hat y=\mathbf{1}\{p\ge \tau\}.$$

The default $\tau=0.5$ is only reasonable when false positives and false negatives cost about the same and the class balance is not extreme. Fraud detection often uses a much smaller $\tau$ so that rare events are not missed. Medical screening may do the opposite or the same depending on the cost of follow-up tests.

Always report metrics at the threshold you would actually use, or report a curve (ROC or precision-recall) that summarizes many thresholds.

How to evaluate a logistic regression model

Accuracy is a weak headline. If 98% of transactions are legitimate, a model that always predicts “not fraud” is 98% accurate and worthless.

Use a set of complementary metrics:

• Confusion matrix: true positives, false positives, true negatives, false negatives.
• Precision: of predicted positives, how many were correct.
• Recall (sensitivity): of actual positives, how many were found.
• F1 score: harmonic mean of precision and recall.
• ROC AUC: ranking quality across thresholds.
• PR AUC: often more informative on imbalanced data than ROC AUC.
• Log loss / Brier score: quality of the probabilities themselves, not just the labels.

Calibration matters. A predicted $0.2$ should occur about 20% of the time in that bin. A well-ranked but poorly calibrated model can still be useful if you only need ordering; it is not enough if you need honest probabilities.

Assumptions and failure modes

Logistic regression assumes a linear log-odds relationship. If the true boundary is a ring or a XOR pattern, a linear logit cannot represent it unless you add features (polynomials, interactions, or embeddings).

It also assumes observations are independent. Time series and clustered patients violate that; standard errors and p-values then mislead even if predictions remain usable.

Multicollinearity inflates coefficient variance, just as in linear regression. For prediction that may be tolerable; for “which feature matters?” it is not.

Class imbalance does not break the math, but it changes the operating point. Class weights, resampling, and threshold tuning are policy tools, not magic.

Logistic regression versus related methods

Linear regression on a 0/1 label can produce predictions outside $[0,1]$ and uses a squared-error loss that is a poor match to classification. Use logistic regression instead.

Naive Bayes also produces class probabilities but assumes feature independence given the class. Logistic regression models $p(y\mid\mathbf{x})$ directly (a discriminative model).

SVMs maximize a margin and do not natively output calibrated probabilities. They can outperform logistic regression on some kernels; they are harder to interpret on the odds scale.

Neural networks with a sigmoid or softmax last layer are logistic regression stacked on learned features. Understanding the single-layer case is the right way to understand that last layer.

A compact worked picture

Suppose $z= -1.2 + 0.8\cdot \text{income} - 2.1\cdot \text{late\_payments}$. For a customer with income $=2$ (scaled) and one late payment, $z=-1.2+1.6-2.1=-1.7$, so

$$p=\sigma(-1.7)\approx 0.15.$$

If $\tau=0.5$, the prediction is negative. If the business costs make missing a default expensive, $\tau=0.12$ might still flag the account. The coefficients did not change; the decision policy did.

Frequently asked questions

Is logistic regression a regression or a classification algorithm?

It is trained as a probability model (regression on the log-odds) and used as a classifier once you apply a threshold. In machine-learning libraries it lives in the classification API.

Can logistic regression handle more than two classes?

Yes. Multinomial logistic regression (softmax regression) generalizes the sigmoid to $K$ classes. Each class has a linear score; softmax converts the scores into a probability vector.

Do I need to scale features?

Gradient-based training is more stable when features live on similar scales. Coefficient interpretation also becomes cleaner after standardization, with the caveat that “one standard deviation” must be explained to stakeholders.

Why is my accuracy high but recall terrible?

The threshold or the class prior is fighting you. Inspect the confusion matrix and precision-recall curve. Accuracy can hide a model that never predicts the rare class.

Next steps

Fit the same dataset with a linear logit and with a few interaction terms. Compare log loss and ROC AUC on a held-out set, not on the training set. Then open the Logistic Regression Calculator to see the sigmoid, threshold, and metrics update together. When you need nonlinear boundaries, move to SVMs or a small neural network — both still end with a logistic-style probability layer.