Support Vector Machines Explained: Margins, Kernels, and Decision Boundaries

A support vector machine (SVM) is a classification algorithm that finds a decision boundary with the largest possible margin between classes. The idea is geometric: among all hyperplanes that separate the training points, prefer the one that stays farthest from the nearest examples. Those nearest examples are the support vectors — they alone determine the boundary.

This guide explains hard and soft margins, the kernel trick, RBF versus linear kernels, and when an SVM is a better choice than logistic regression.

The maximum-margin hyperplane

In two dimensions a linear classifier is a line; in $d$ dimensions it is a hyperplane

$$\mathbf{w}^\top\mathbf{x}+b=0.$$

The signed distance from a point to the hyperplane is $(\mathbf{w}^\top\mathbf{x}+b)/\|\mathbf{w}\|$. For labels $y_i\in\{-1,+1\}$, a hard-margin SVM requires

$$y_i(\mathbf{w}^\top\mathbf{x}_i+b)\ge 1$$

and minimizes $\|\mathbf{w}\|^2/2$. Minimizing the weight norm maximizes the margin $2/\|\mathbf{w}\|$.

Only points that sit on the margin constraints matter. Move a point that is already correctly classified and far from the boundary, and the solution does not change. That sparsity is why the method is named for support vectors.

Soft margins: allowing mistakes

Real data overlap. A hard margin then has no solution, or it overfits a noisy point. Soft-margin SVM introduces slack variables $\xi_i\ge 0$:

$$\min_{\mathbf{w},b,\boldsymbol{\xi}}\ \frac{1}{2}\|\mathbf{w}\|^2+C\sum_i\xi_i \quad\text{s.t.}\quad y_i(\mathbf{w}^\top\mathbf{x}_i+b)\ge 1-\xi_i.$$

The hyperparameter $C$ is the trade-off:

• Large $C$: few training errors, smaller margin, higher overfitting risk.
• Small $C$: wider margin, more training violations, often better generalization.

Think of $C$ as the opposite of a regularization strength: it penalizes slack rather than shrinking $\mathbf{w}$ directly, but the effect is similar.

The kernel trick

A linear boundary in the original features may be too weak. One response is to map $\mathbf{x}$ into a richer space $\phi(\mathbf{x})$ and separate there. The kernel trick computes inner products in that space without building $\phi$ explicitly:

$$K(\mathbf{x},\mathbf{x}')=\phi(\mathbf{x})^\top\phi(\mathbf{x}').$$

Common kernels:

The RBF kernel is a similarity: nearby points get $K\approx 1$, distant points get $K\approx 0$. The length-scale $\gamma$ controls how local that similarity is. Too large $\gamma$ memorizes islands around each point; too small $\gamma$ behaves almost linearly.

The SVM Calculator lets you switch linear, polynomial, and RBF kernels and see the decision boundary change — the fastest way to build intuition for $\gamma$ and $C$.

Dual form and why support vectors appear

The optimization is usually solved in the dual, with coefficients $\alpha_i\ge 0$ on each training point. The decision function becomes

$$\hat y(\mathbf{x})=\mathrm{sign}\Big(\sum_{i\in SV}\alpha_i y_i K(\mathbf{x}_i,\mathbf{x})+b\Big).$$

Most $\alpha_i$ are zero. The nonzero $\alpha_i$ mark support vectors. Prediction cost therefore grows with the number of support vectors, not with the full training set — still slower than a logistic regression with a handful of coefficients if many points sit on the margin.

SVM versus logistic regression

Both can produce linear decision boundaries. Differences that matter in practice:

• Objective: SVM uses hinge loss (margin); logistic regression uses log loss (likelihood).
• Probabilities: logistic regression outputs calibrated-ish probabilities; SVM scores need extra calibration (Platt scaling) if you need $p(y=1)$.
• Outliers: a huge $C$ lets a single mislabeled point yank an SVM; logistic regression degrades more smoothly.
• Kernels: SVMs made kernel methods mainstream. Logistic regression can use the same kernels but is less commonly tuned that way.
• Scaling: both want standardized features, especially with RBF.

If you need odds ratios for a paper or a regulator, start with logistic regression. If you need a flexible nonlinear boundary and have a few thousand points, try an RBF SVM.

Practical tuning checklist

1. Scale features. RBF distance is meaningless if one column is in meters and another is in millions of dollars.
2. Start with a linear kernel. If it is already strong, you may not need RBF.
3. Search $C$ on a log grid ($10^{-2}$ to $10^{3}$).
4. For RBF, search $\gamma$ on a log grid as well; use cross-validation, not the training accuracy.
5. Class imbalance: use class_weight or resampling; a wide margin on the majority class is a common silent failure.
6. Multiclass: libraries use one-vs-rest or one-vs-one. Report the scheme.

Frequently asked questions

What is a support vector in simple terms?

A training example that sits on or inside the margin (or is misclassified). Remove a non-support-vector and the boundary stays put; remove a support vector and it usually moves.

Is SVM good for huge datasets?

Linear SVMs (or logistic regression / linear models) scale to large $n$. Kernel SVMs struggle when $n$ is hundreds of thousands because the kernel matrix is $n\times n$ in spirit. For large nonlinear problems, gradient-boosted trees or neural nets are usually easier.

Does SVM work for regression?

Yes: SVR (support vector regression) uses an $\varepsilon$-insensitive tube instead of a classification margin. The kernel idea is the same.

Why is my RBF SVM 100% on train and poor on test?

$\gamma$ is too large or $C$ is too large. The model built a tight bubble around every training point. Increase regularization (lower $C$), lower $\gamma$, and validate with k-fold CV.

Next steps

Sketch two blobs that are linearly separable, then two rings. The first wants a linear SVM; the second wants RBF. Reproduce both in the SVM Calculator. Then compare the same data with logistic regression and with k-NN to see how parametric, kernel, and memory-based classifiers disagree near the boundary.