Vector Norms and Distance Metrics in AI: Euclidean, Manhattan, Cosine, and Beyond

Machine-learning systems constantly compare objects: a new patient with previous patients, a document with a search query, an image embedding with known examples, or a customer with a cluster center. Yet “close,” “similar,” and “different” are not properties that data possess automatically. They are conclusions produced by a mathematical rule. Choosing that rule determines which observations become neighbors, which clusters appear compact, which coefficients are considered small, and sometimes which model wins. This article develops the most important vector norms and distance metrics from first principles, connects their geometry to common AI algorithms, and shows how preprocessing can change the answer even when the raw observations stay exactly the same.

Vector Norms and Distance Metrics in AI: Euclidean, Manhattan, Cosine, and Beyond

Similarity is a mathematical choice

Suppose a recommendation system represents each user by three numbers: weekly viewing hours, fraction of content that is comedy, and average rating. Two users can be compared in several reasonable ways. We might add the absolute feature differences, measure the straight-line distance between them, compare only the direction of their preference vectors, or give some features more importance than others. Each method encodes a different meaning of similarity.

This point is easy to overlook because software libraries expose a metric parameter that looks like an implementation detail. It is not. A metric is part of the model. If a k-nearest neighbors classifier changes from Euclidean to Manhattan distance, its decision boundary may change. If document retrieval changes from Euclidean distance to cosine similarity, the ranking may change. If features are rescaled, the nearest observation may change even though no record was added or removed.

A vector is an ordered list of numbers,

$$\mathbf{x}=(x_1,x_2,\ldots,x_d)\in\mathbb{R}^d,$$

where each coordinate usually represents one feature. A norm maps a vector to a nonnegative number that represents its magnitude:

$$\lVert\mathbf{x}\rVert \geq 0.$$

A norm must satisfy three properties. First, $\lVert\mathbf{x}\rVert=0$ only when $\mathbf{x}=\mathbf{0}$. Second, scaling the vector scales its norm by the absolute scaling factor:

$$\lVert a\mathbf{x}\rVert=|a|\lVert\mathbf{x}\rVert.$$

Third, the triangle inequality holds:

$$\lVert\mathbf{x}+\mathbf{y}\rVert \leq \lVert\mathbf{x}\rVert+\lVert\mathbf{y}\rVert.$$

A norm induces a distance by applying the norm to the difference between two vectors:

$$d(\mathbf{x},\mathbf{y})=\lVert\mathbf{x}-\mathbf{y}\rVert.$$

A true metric must be nonnegative, equal zero only for identical objects, symmetric, and satisfy the triangle inequality. These conditions make “distance” behave consistently. Some useful dissimilarity scores do not satisfy every metric axiom. For example, squared Euclidean distance is extremely useful in optimization, but it fails the triangle inequality. The distinction matters in algorithms that rely on metric properties for indexing or pruning, even though casual discussion often calls every dissimilarity a distance.

The best metric is therefore not the one that sounds most familiar. It is the one whose assumptions match the representation and the task. Physical coordinates may support ordinary geometric distance. Word-frequency vectors may be better compared by angle. Binary strings may call for Hamming distance. Mixed medical records may require a domain-specific combination. Similarity is designed, not discovered in a vacuum.

The L1, L2, and L-infinity norms

The family of $L_p$ norms provides a unified way to measure vector magnitude. For a finite-dimensional real vector and $p\geq1$,

$$\lVert\mathbf{x}\rVert_p =\left(\sum_{j=1}^{d}|x_j|^p\right)^{1/p}.$$

Three members are especially important in AI.

The L1 norm

The $L_1$ norm adds absolute coordinate values:

$$\lVert\mathbf{x}\rVert_1 =\sum_{j=1}^{d}|x_j|.$$

For $\mathbf{x}=(3,-4,2)$, the result is

$$\lVert\mathbf{x}\rVert_1=|3|+|-4|+|2|=9.$$

The corresponding distance between two points adds their absolute coordinate-wise differences:

$$d_1(\mathbf{x},\mathbf{y}) =\sum_{j=1}^{d}|x_j-y_j|.$$

This is called Manhattan distance, city-block distance, or taxicab distance. The names evoke travel on a rectangular street grid: moving three blocks east and four blocks north requires seven blocks of travel, not the five-unit diagonal available to a bird.

$L_1$ is less dominated by a single large coordinate difference than squared Euclidean objectives are. It is also associated with sparsity in optimization. An $L_1$ penalty can drive some learned coefficients exactly to zero, which makes it important in Lasso regression and feature selection.

The L2 norm

The $L_2$ norm is the familiar Euclidean length:

$$\lVert\mathbf{x}\rVert_2 =\sqrt{\sum_{j=1}^{d}x_j^2}.$$

For the same vector,

$$\lVert(3,-4,2)\rVert_2 =\sqrt{3^2+(-4)^2+2^2} =\sqrt{29}.$$

The induced Euclidean distance is

$$d_2(\mathbf{x},\mathbf{y}) =\sqrt{\sum_{j=1}^{d}(x_j-y_j)^2}.$$

It is the straight-line distance between points. Euclidean geometry is rotation invariant: rotating the coordinate axes without distorting the space leaves distances unchanged. This property is attractive when every direction should be treated equally.

The squared Euclidean distance,

$$d_2^2(\mathbf{x},\mathbf{y}) =\sum_{j=1}^{d}(x_j-y_j)^2,$$

preserves nearest-neighbor rankings because squaring is monotonic for nonnegative numbers. However, the squared quantity is not itself a metric. It is widely used because eliminating the square root simplifies derivatives and because means minimize sums of squared deviations.

The L-infinity norm

The $L_\infty$ norm keeps only the coordinate with the largest absolute magnitude:

$$\lVert\mathbf{x}\rVert_\infty =\max_j |x_j|.$$

Thus,

$$\lVert(3,-4,2)\rVert_\infty=4.$$

The induced distance is

$$d_\infty(\mathbf{x},\mathbf{y}) =\max_j|x_j-y_j|.$$

It is also called Chebyshev distance. Imagine a process where all coordinates can change simultaneously at the same maximum rate. The required time is governed by the largest coordinate difference. On a chessboard, a king can move diagonally, so the number of moves between squares is the maximum of the horizontal and vertical displacements.

$L_\infty$ is useful when the worst individual deviation determines acceptability. A manufacturing part may fail tolerance if any dimension is too far from its target. An adversarial-robustness study may constrain every pixel perturbation to remain below a maximum. Unlike $L_1$, which aggregates all deviations, $L_\infty$ focuses entirely on the largest.

Euclidean and Manhattan geometry

The difference between Euclidean and Manhattan distance is more than a formula. Each norm creates its own geometry. One way to see that geometry is to examine all points one unit from the origin.

Under $L_2$,

$$x_1^2+x_2^2=1,$$

so the unit boundary in two dimensions is a circle. Under $L_1$,

$$|x_1|+|x_2|=1,$$

so the boundary is a diamond whose corners lie on the coordinate axes. Under $L_\infty$,

$$\max(|x_1|,|x_2|)=1,$$

so the boundary is an axis-aligned square.

These shapes affect which points count as neighbors. Consider a query at the origin. A point at $(0.9,0.9)$ has Euclidean distance about $1.273$ and Manhattan distance $1.8$. A point at $(1.3,0)$ has Euclidean and Manhattan distances both equal to $1.3$. Euclidean distance considers the diagonal point slightly closer, while Manhattan distance considers the axis point much closer. A nearest-neighbor model can therefore make a different prediction under the two metrics.

Euclidean distance penalizes distributed coordinate changes through squaring before summation. Manhattan distance charges each unit of coordinate change at a constant rate. In noisy data with occasional large feature deviations, Manhattan distance can be less sensitive to those deviations. That does not make it automatically robust: an extreme coordinate still contributes a large absolute difference, and badly scaled features still dominate.

Correlated features present another issue. If height in centimeters and height in inches both appear, ordinary Euclidean distance effectively counts the same underlying characteristic twice. More generally, Euclidean and Manhattan distances assume that the supplied coordinate axes provide an appropriate representation. Mahalanobis distance addresses scale and correlation by using a covariance matrix:

$$d_M(\mathbf{x},\mathbf{y}) =\sqrt{(\mathbf{x}-\mathbf{y})^\top \mathbf{S}^{-1}(\mathbf{x}-\mathbf{y})},$$

where $\mathbf{S}$ is an estimated covariance matrix. Differences along high-variance directions are discounted, while differences along low-variance directions receive more weight. This can be valuable, but covariance estimation becomes unstable with limited data, redundant features, or very high dimension. Regularization may be necessary.

Minkowski distance as a general family

Minkowski distance unifies Manhattan and Euclidean distance:

$$d_p(\mathbf{x},\mathbf{y}) =\left(\sum_{j=1}^{d}|x_j-y_j|^p\right)^{1/p}, \qquad p\geq1.$$

When $p=1$, it is Manhattan distance. When $p=2$, it is Euclidean distance. As $p$ approaches infinity, it approaches Chebyshev distance:

$$\lim_{p\to\infty}d_p(\mathbf{x},\mathbf{y}) =d_\infty(\mathbf{x},\mathbf{y}).$$

Increasing $p$ gives the largest coordinate differences progressively more influence. Suppose a difference vector is $(1,4)$. Its $L_1$ magnitude is $5$, its $L_2$ magnitude is $\sqrt{17}\approx4.123$, and its $L_\infty$ magnitude is $4$. Although the numeric values decrease as $p$ grows in this example, the relative importance of the coordinate with difference $4$ increases.

Values $0<p<1$ sometimes appear in sparse modeling, but the resulting expression is not a norm and does not satisfy the triangle inequality. It is better described as a quasi-norm. Optimization with such penalties can be nonconvex and more difficult. Meanwhile, a weighted Minkowski distance can express feature importance:

$$d_{p,\mathbf{w}}(\mathbf{x},\mathbf{y}) =\left(\sum_{j=1}^{d}w_j|x_j-y_j|^p\right)^{1/p}, \qquad w_j\geq0.$$

Weights should come from domain knowledge, careful validation, or a learned metric—not from arbitrary tuning on the final test set.

Cosine similarity and Euclidean distance

Cosine similarity compares vector direction rather than raw separation:

$$\operatorname{cos\_sim}(\mathbf{x},\mathbf{y}) =\frac{\mathbf{x}^\top\mathbf{y}} {\lVert\mathbf{x}\rVert_2\lVert\mathbf{y}\rVert_2}.$$

For nonzero vectors, the value is the cosine of the angle $\theta$ between them:

$$\operatorname{cos\_sim}(\mathbf{x},\mathbf{y})=\cos\theta.$$

It ranges from $-1$ to $1$ for real-valued vectors. A value of $1$ means the vectors point in the same direction, $0$ means they are orthogonal, and $-1$ means they point in opposite directions. For vectors with only nonnegative components, values usually lie between $0$ and $1$.

This directional view is useful when magnitude is a nuisance. In a bag-of-words representation, a long document may contain ten times as many occurrences of every word as a short document while expressing nearly the same topic proportions. Their Euclidean distance can be large, but their cosine similarity is $1$ if one vector is exactly a positive multiple of the other.

Cosine distance is often defined as

$$d_{\cos}(\mathbf{x},\mathbf{y}) =1-\operatorname{cos\_sim}(\mathbf{x},\mathbf{y}).$$

Despite its name, this form does not satisfy all metric axioms in general, especially the triangle inequality. Angular distance, based directly on the angle, can provide a proper metric:

$$d_{\text{angle}}(\mathbf{x},\mathbf{y}) =\frac{\arccos(\operatorname{cos\_sim}(\mathbf{x},\mathbf{y}))}{\pi}.$$

There is a precise connection between cosine similarity and Euclidean distance. If both vectors are normalized to unit $L_2$ length, then

$$\lVert\mathbf{x}-\mathbf{y}\rVert_2^2 =\lVert\mathbf{x}\rVert_2^2+\lVert\mathbf{y}\rVert_2^2 -2\mathbf{x}^\top\mathbf{y} =2-2\operatorname{cos\_sim}(\mathbf{x},\mathbf{y}).$$

Therefore, ranking unit-normalized vectors by smallest Euclidean distance is equivalent to ranking them by largest cosine similarity. Without normalization, the equivalence disappears because Euclidean distance responds to magnitude.

Cosine similarity is not universally preferable for embeddings. Magnitude can contain useful information, such as confidence, frequency, popularity, or distance from a learned origin. Normalizing every vector discards that information. The decision should follow the semantics of the representation and empirical validation on the real retrieval or prediction objective.

Why feature scaling changes nearest neighbors

Distance calculations operate on numbers, not on the human meanings attached to units. Suppose a dataset contains age in years and annual income in dollars. A ten-year age difference contributes $10$ to Manhattan distance or $100$ to squared Euclidean distance. A $10{,}000$ difference in income contributes $10{,}000$ or $100{,}000{,}000$, respectively. Income will dominate even if age is more relevant to the target.

Standardization transforms feature $j$ according to

$$z_{ij}=\frac{x_{ij}-\mu_j}{\sigma_j},$$

where $\mu_j$ and $\sigma_j$ are the training mean and standard deviation. Afterward, a one-unit difference represents roughly one training standard deviation. Min-max scaling instead maps a feature to a chosen range, commonly $[0,1]$:

$$x'_{ij} =\frac{x_{ij}-\min(x_j)} {\max(x_j)-\min(x_j)}.$$

Robust scaling uses a median and interquartile range, reducing the effect of extreme values:

$$x'_{ij} =\frac{x_{ij}-\operatorname{median}(x_j)} {\operatorname{IQR}(x_j)}.$$

No scaling method is neutral. Standardization treats variation relative to standard deviation as meaningful. Min-max scaling is sensitive to extremes and future values beyond the observed range. Robust scaling changes the influence of tails. Domain-based scaling, such as dividing a temperature error by an allowed tolerance, may be more defensible when practical thresholds are known.

To see how a neighbor can reverse, consider query $Q=(30,50{,}000)$, candidate $A=(40,50{,}100)$, and candidate $B=(31,60{,}000)$, with coordinates representing age and income. Raw Euclidean distance strongly prefers $A$:

$$d_2(Q,A)=\sqrt{10^2+100^2}\approx100.50,$$ $$d_2(Q,B)=\sqrt{1^2+10{,}000^2}\approx10{,}000.$$

Now suppose age has training standard deviation $10$ years and income has standard deviation $20{,}000$. In standardized units, the difference to $A$ is $(1,0.005)$ and the difference to $B$ is $(0.1,0.5)$. The distances become approximately $1.000$ and $0.510$, so $B$ is now closer. Scaling did not merely improve numerical stability; it changed the model’s concept of resemblance.

Preprocessing must be fitted only on the training data. Computing means, standard deviations, minima, maxima, or feature weights from the full dataset leaks information from validation and test records into training. In production, the saved training transformation must be applied unchanged to new observations. The Feature Engineering Calculator is useful for exploring how transformations and feature choices alter downstream geometry.

Distances inside common AI methods

k-nearest neighbors

k-nearest neighbors, or k-NN, predicts from the labels or values of nearby training observations. For classification, a basic rule is

$$\hat y(\mathbf{x}) =\operatorname{mode}\{y_i:i\in N_k(\mathbf{x})\},$$

where $N_k(\mathbf{x})$ contains the $k$ nearest training vectors under the selected metric. A distance-weighted version gives closer neighbors more influence, often using a weight such as

$$w_i=\frac{1}{d(\mathbf{x},\mathbf{x}_i)+\varepsilon}.$$

The metric determines the neighborhood, while $k$ controls how local the prediction is. Small $k$ can follow noise; large $k$ can blur real local structure. Scaling, categorical encoding, missing-value handling, and irrelevant features can matter as much as $k$. Use validation within a preprocessing pipeline to choose the complete configuration. The KNN Calculator helps make neighbor selection and voting visible rather than treating them as hidden library operations.

k-means clustering

Standard k-means minimizes within-cluster squared Euclidean distance:

$$J=\sum_{r=1}^{k}\sum_{\mathbf{x}_i\in C_r} \lVert\mathbf{x}_i-\boldsymbol{\mu}_r\rVert_2^2.$$

The arithmetic mean is the optimal cluster representative specifically for this squared Euclidean objective. Replacing Euclidean distance with Manhattan distance while continuing to update centers by the mean is not ordinary k-means and does not preserve its optimization logic. Under an $L_1$ objective, coordinate-wise medians are appropriate representatives, leading toward k-medians. When representatives must be actual observations and arbitrary dissimilarities are allowed, k-medoids is often suitable.

This distinction prevents a common misconception: a clustering algorithm is not just “assign to the nearest center” with any interchangeable distance. Its assignment rule, center update, and objective must agree. Experiment with centroid assignments and inertia in the K-Means Calculator, then compare the behavior with alternative metrics conceptually.

L1 and L2 regularization

Norms also measure model parameters rather than distances between observations. A regularized learning objective combines data error with a penalty. Ridge regression uses a squared $L_2$ penalty:

$$\min_{\boldsymbol{\beta}} \left[ \sum_{i=1}^{n}(y_i-\mathbf{x}_i^\top\boldsymbol{\beta})^2 +\lambda\lVert\boldsymbol{\beta}\rVert_2^2 \right].$$

Lasso regression uses an $L_1$ penalty:

$$\min_{\boldsymbol{\beta}} \left[ \sum_{i=1}^{n}(y_i-\mathbf{x}_i^\top\boldsymbol{\beta})^2 +\lambda\lVert\boldsymbol{\beta}\rVert_1 \right].$$

$L_2$ regularization smoothly shrinks coefficients toward zero and tends to share weight across correlated predictors. $L_1$ regularization has corners in its constraint geometry, making exact zero coefficients more likely. Elastic net combines both penalties.

Feature scaling is crucial here too. A coefficient’s numeric magnitude depends on its feature’s units. Without scaling, penalizing coefficients equally does not penalize the real influence of features equally. Intercepts are typically excluded from the penalty. Also, “L2 loss” and “L2 regularization” are different concepts: one measures residuals, while the other constrains parameters.

A worked example on the same three points

Let the query point and two candidate neighbors be

$$Q=(1,1),\qquad A=(4,1),\qquad B=(3,3).$$

Their difference vectors from $Q$ are

$$A-Q=(3,0),\qquad B-Q=(2,2).$$

The metrics do not agree about which candidate is closer. The complete comparison is:

For Manhattan distance,

$$d_1(Q,A)=|1-4|+|1-1|=3,$$ $$d_1(Q,B)=|1-3|+|1-3|=4.$$

Manhattan distance therefore selects $A$. The path to $A$ requires three units of movement along one coordinate, whereas reaching $B$ requires two units along each coordinate, totaling four.

For Euclidean distance,

$$d_2(Q,A)=\sqrt{(1-4)^2+(1-1)^2}=3,$$ $$d_2(Q,B)=\sqrt{(1-3)^2+(1-3)^2} =\sqrt{8}\approx2.828.$$

Euclidean distance selects $B$. Its diagonal displacement is slightly shorter than the three-unit horizontal displacement to $A$.

For Chebyshev distance,

$$d_\infty(Q,A)=\max(3,0)=3,$$ $$d_\infty(Q,B)=\max(2,2)=2.$$

Chebyshev distance also selects $B$, because neither coordinate needs to change by more than two.

Cosine similarity gives another perspective. Since $B=3Q$, $Q$ and $B$ point in exactly the same direction, so

$$\operatorname{cos\_sim}(Q,B)=1.$$

For $A$,

$$\operatorname{cos\_sim}(Q,A) =\frac{1\cdot4+1\cdot1} {\sqrt{1^2+1^2}\sqrt{4^2+1^2}} =\frac{5}{\sqrt{34}} \approx0.857.$$

Cosine similarity strongly prefers $B$, even though $B$ is farther under Manhattan distance. This is not a contradiction. Manhattan distance asks how much coordinate-wise travel separates the points. Cosine similarity asks how closely their directions align.

The same calculations can be reproduced in Python:

import numpy as np

q = np.array([1.0, 1.0])
a = np.array([4.0, 1.0])
b = np.array([3.0, 3.0])

def cosine_similarity(x, y):
    return np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))

for name, point in {"A": a, "B": b}.items():
    delta = q - point
    print(
        name,
        "L1:", np.linalg.norm(delta, ord=1),
        "L2:", np.linalg.norm(delta, ord=2),
        "L-infinity:", np.linalg.norm(delta, ord=np.inf),
        "cosine:", cosine_similarity(q, point),
    )

The lesson is not that one result is correct and the others are wrong. The lesson is that a metric supplies the question to which “nearest” is the answer.

Distance concentration in high dimensions

Distance-based intuition is largely formed in two or three dimensions, but AI representations may have hundreds, thousands, or millions of coordinates. As dimensionality grows, several surprising effects appear.

Consider points whose coordinates are sampled independently from similar bounded distributions. A Euclidean squared distance is a sum over coordinates:

$$d_2^2(\mathbf{x},\mathbf{y}) =\sum_{j=1}^{d}(x_j-y_j)^2.$$

As $d$ increases, many coordinate contributions accumulate. The absolute distances generally grow, but their relative spread can shrink. Informally, the nearest and farthest points may become similar in distance compared with the overall scale. A common summary is

$$\frac{d_{\max}-d_{\min}}{d_{\min}}\longrightarrow0$$

under particular distributional assumptions as dimension increases. This phenomenon is called distance concentration.

The intuition comes from averaging. In high dimensions, each pairwise distance combines many small random contributions. Unless the data have strong structure, those sums become relatively predictable. A point can be somewhat closer on some coordinates and farther on others, with differences averaging out. The space also becomes sparse: an enormous volume exists, but finite datasets occupy very little of it.

For nearest-neighbor methods, concentration can make the identity of the “nearest” point unstable and the contrast between useful and irrelevant neighbors weak. Adding irrelevant features introduces extra differences that do not help predict the target. This is one aspect of the curse of dimensionality. More data are needed to cover the space, local neighborhoods become less local, and density estimates become difficult.

The effect does not imply that all high-dimensional machine learning is impossible. Real data often lie near lower-dimensional manifolds, contain sparse structure, or use embeddings trained so that meaningful relationships are preserved. Feature selection, principal component analysis, representation learning, metric learning, and regularization can improve the geometry. Cosine similarity can be effective for sparse text, although it does not abolish every high-dimensional problem. Approximate nearest-neighbor indexes accelerate search but cannot repair an uninformative representation.

The practical response is empirical. Compare neighbor-distance distributions, evaluate against labeled relevance or downstream performance, remove irrelevant dimensions, and test stability. Do not assume that a metric successful on a two-dimensional visualization remains informative in a 1,536-dimensional embedding space.

Hamming and other discrete distances

Continuous vector norms are not natural for every data type. Hamming distance counts positions at which two equal-length sequences differ:

$$d_H(\mathbf{x},\mathbf{y}) =\sum_{j=1}^{d}\mathbb{1}(x_j\neq y_j).$$

For binary strings $101101$ and $100111$, two positions differ, so their Hamming distance is $2$. A normalized version divides by sequence length. Hamming distance is useful for binary codes, categorical attributes encoded position by position, error-correcting codes, and comparing equal-length symbolic sequences.

Jaccard similarity compares sets:

$$J(A,B)=\frac{|A\cap B|}{|A\cup B|},$$

with Jaccard distance $1-J(A,B)$. It is especially useful for sparse binary features when shared absences should not count as evidence of similarity. Two customers who both did not buy thousands of products should not appear highly similar merely because of those joint zeros.

Edit distance, or Levenshtein distance, counts the minimum insertions, deletions, and substitutions required to transform one string into another. It handles sequences of different lengths, unlike ordinary Hamming distance. Dynamic time warping compares time series while allowing nonlinear alignment in time. Earth mover’s distance, closely related to Wasserstein distance, measures the work required to transform one distribution into another.

Mixed tabular data can use Gower distance, which combines normalized numeric differences with categorical matches and can accommodate missingness carefully. Graphs, probability distributions, geographic coordinates, and rotations each have specialized notions of distance. Latitude and longitude, for example, should use spherical or geodesic calculations over large geographic regions rather than naive Euclidean distance in degrees.

Choosing a metric systematically

A defensible metric choice begins with the data-generating meaning of each feature. Ask what transformations should leave similarity unchanged. If doubling every count in a document should preserve its meaning, a directional measure may be appropriate. If a one-unit error has constant cost regardless of current magnitude, Manhattan distance may fit. If large deviations should receive increasing emphasis, Euclidean or squared Euclidean objectives may fit. If any single tolerance violation is decisive, $L_\infty$ may fit.

Next, inspect representation quality. Remove identifiers and leakage variables. Decide how missing values should behave. Encode categorical variables in a way compatible with the metric. One-hot encoding plus Euclidean distance can make categories contribute a fixed geometric cost, but a high-cardinality categorical field can dominate by occupying many columns. Ordinal encodings can falsely imply ordered spacing. Domain-specific similarity may be needed.

Then place preprocessing and metric selection inside validation. For supervised learning, compare candidate pipelines using cross-validation and a task-relevant score. For retrieval, use judged query-result pairs and metrics such as recall at $k$ or normalized discounted cumulative gain. For clustering, combine internal criteria with stability, external labels when legitimate, and expert interpretation. Never select a distance solely because it creates the most visually pleasing two-dimensional projection.

Finally, consider computational requirements. Some indexing structures depend on the triangle inequality. Sparse data favor operations that avoid densification. Mahalanobis distance needs a covariance inverse. Learned metrics require enough representative training data. Approximate search systems may support dot product, Euclidean distance, and cosine similarity differently; normalization can sometimes convert one supported operation into an equivalent ranking.

Common mistakes

Treating units as harmless labels

Changing kilometers to meters multiplies a coordinate difference by $1{,}000$. Unless scaling or weighting compensates for the conversion, a distance-based model changes. Units are part of the model.

Scaling before the train-test split

Fitting a scaler on all observations leaks test-distribution information. Split first, fit preprocessing on training data, and apply the learned transformation to validation, test, and production inputs.

Calling squared Euclidean distance a metric

It is a useful dissimilarity and produces the same nearest-neighbor ordering as Euclidean distance, but it fails the triangle inequality. That distinction can affect metric-tree algorithms and theoretical claims.

Using cosine similarity with zero vectors

The cosine formula divides by both vector norms. A zero vector has no direction, so its cosine similarity is undefined. Decide explicitly whether to remove zero vectors, assign a convention, or redesign the representation.

Assuming cosine removes every scale issue

Cosine similarity ignores global vector magnitude, not coordinate-wise feature scales. Multiplying one feature column by $100$ changes angles and can alter rankings. Feature construction and weighting still matter.

Swapping the k-means metric without changing updates

Means minimize squared Euclidean deviation. Manhattan assignments paired with arithmetic-mean updates do not optimize the standard k-means objective. Use an algorithm whose representative and update rule match its dissimilarity.

Mixing numeric and categorical codes naively

Encoding colors as red $=1$, green $=2$, blue $=3$ makes red twice as far from blue as from green under Manhattan distance. The numbers create an artificial order. Use appropriate categorical handling.

Keeping many irrelevant dimensions

Every irrelevant feature adds noise to distance. In high dimensions, this can overwhelm useful neighborhood structure. Feature selection and learned representations are often more important than fine-tuning $p$.

Interpreting “nearest” as “similar in every sense”

A neighbor is near only under the chosen representation, preprocessing, and metric. It may be demographically near but behaviorally distant, or textually similar but factually contradictory. State the scope of similarity.

Selecting the metric on the test set

Trying many metrics and reporting the one with the best test performance turns the test set into a validation set. Choose using training and validation procedures, then evaluate once on held-out data.

Practice with Solver360

The most effective way to learn metric behavior is to calculate a small case by hand, predict what will happen, and then test it.

1. Open the KNN Calculator and create a two-feature classification dataset. Place one candidate diagonally from a query and another along one axis. Compare the neighbors you expect under Manhattan and Euclidean geometry.
2. Multiply one feature by $100$ without changing its meaning. Observe how the neighborhood would change, then standardize the features and compare again.
3. Use the K-Means Calculator to follow centroid assignment and update steps. Connect the mean update to the squared Euclidean objective rather than viewing it as an arbitrary averaging step.
4. Open the Feature Engineering Calculator and explore standardization, normalization, and transformed features. Ask what notion of similarity each transformation creates.
5. Reproduce the three-point example from this article. Before calculating, write down which point you expect each metric to prefer and why.
6. Construct two proportional vectors, such as $(1,2,3)$ and $(10,20,30)$. Compare their Euclidean distance and cosine similarity. Then perturb one coordinate and observe how the angle changes.

Good practice focuses on explanation, not just arithmetic. For every result, complete the sentence: “These points are similar because the metric values...” If the sentence does not correspond to the real problem, the metric probably does not either.

Frequently asked questions

Is Euclidean distance always the default?

It is a common baseline for continuous, appropriately scaled features, but it is not a universal default. Euclidean distance assumes coordinate differences are meaningful, combines them through squared contributions, and treats directions symmetrically. Sparse text, binary attributes, geographic positions, correlated measurements, and mixed data may require other choices.

Should I use Manhattan or Euclidean distance for k-NN?

Validate both when each is semantically plausible. Manhattan distance can behave better when effects add coordinate by coordinate or when occasional large deviations should have less relative influence. Euclidean distance fits straight-line geometry and is rotation invariant. Scaling and feature relevance often matter more than the difference between the two.

Is cosine similarity the same as Euclidean distance after normalization?

For $L_2$-normalized vectors, they produce equivalent rankings because

$$\lVert\mathbf{x}-\mathbf{y}\rVert_2^2 =2-2\operatorname{cos\_sim}(\mathbf{x},\mathbf{y}).$$

The numeric scores are not identical, but one is a monotonic transformation of the other. The equivalence does not hold for unnormalized vectors.

Why does standardization sometimes reduce model accuracy?

Standardization changes the relative importance of features. It helps when raw units are arbitrary and comparable variation should receive comparable weight. It can hurt when original scale carries valid domain importance or when low-variance noise is amplified. Treat scaling as a model choice and validate it within the training pipeline.

Can distance be negative?

A proper distance metric cannot be negative. Cosine similarity can be negative because it is a similarity, not a distance. Dot products can also be negative. If a software API reports a negative “distance,” inspect its definition; it may return a negated similarity for sorting convenience.

Why does L1 regularization create zero coefficients?

The $L_1$ penalty has a geometry with sharp corners on coordinate axes. When an error contour first touches an $L_1$ constraint region, it often touches at a corner where one or more coefficients are exactly zero. The $L_2$ region is smooth, so Ridge usually shrinks coefficients without setting them exactly to zero.

Does a smaller distance always mean a more confident prediction?

No. A small nearest-neighbor distance may indicate local support, but confidence also depends on class agreement, sampling density, noise, calibration, and whether the query resembles training data in a meaningful representation. Distances from different datasets or preprocessing pipelines are not automatically comparable.

Which metric works best in high dimensions?

There is no universal winner. Cosine similarity is often effective for sparse directional data, and lower-order Minkowski distances sometimes show more contrast than higher-order alternatives, but representation quality dominates. Evaluate on the downstream task, reduce irrelevant dimensions, and inspect whether nearest neighbors are stable and meaningful.

Can I learn a distance metric from data?

Yes. Metric-learning methods learn transformations or similarity functions so that related examples move closer and unrelated examples move farther apart. Siamese networks, contrastive losses, triplet losses, and learned Mahalanobis transformations are examples. Learned metrics can capture task-specific structure, but they inherit biases and coverage limitations from their training pairs.

What is the difference between normalization and standardization?

Terminology varies, but normalization often means scaling each vector to unit length or mapping feature ranges to a fixed interval. Standardization usually means centering a feature and dividing by its standard deviation. Unit-vector normalization changes magnitudes while preserving directions; feature standardization changes the coordinate system itself.

Where to go next

Vector norms connect geometry, optimization, and statistical modeling. A strong next step is linear algebra: vectors, dot products, projections, matrices, eigenvectors, and positive-definite quadratic forms explain why Euclidean, cosine, and Mahalanobis measures behave as they do. Linear algebra also clarifies how dimensionality reduction transforms a space before distances are calculated.

Then study the full k-means article to see one particular geometry turned into an optimization algorithm. Pay close attention to the squared Euclidean objective, the arithmetic-mean update, initialization, scaling, and cluster-shape assumptions. Together, linear algebra and k-means show the central lesson of this guide in action: an AI system does not merely find similarity. It computes similarity according to choices encoded in its features, transformations, norms, and objectives.