Eigenvalues, SVD, and the Mathematics of PCA

Principal component analysis is often introduced as a button that turns many columns into a few new columns. That description is operationally correct but mathematically incomplete. PCA is a precise answer to a geometric optimization problem: among all lower-dimensional linear coordinate systems, find directions that preserve as much variation in centered data as possible. Eigenvalues explain how much variation lies along each direction, eigenvectors specify those directions, and singular value decomposition provides a stable computational route to the same result. Understanding these connections turns PCA from a preprocessing ritual into an interpretable linear-algebra method.

Eigenvalues, SVD, and the Mathematics of PCA

PCA sits at the intersection of statistics and geometry. Statistically, it studies covariance: how variables vary individually and together. Geometrically, it rotates a coordinate system so that the first new axis follows the longest direction of the data cloud, the second follows the longest remaining orthogonal direction, and so on. Algebraically, those axes are eigenvectors of a covariance matrix or right singular vectors of a centered data matrix.

These are not three competing interpretations. They are the same construction viewed through different mathematical objects. This article develops that construction carefully, including the role of centering, variance explained, reconstruction, whitening, and scaling. A complete numerical example shows every important quantity in two dimensions, while a NumPy implementation connects the formulas to practical computation.

Dimensionality as a Mathematical Problem

Suppose a dataset has $n$ observations and $p$ numeric features. Arrange it in a matrix

$$X\in\mathbb{R}^{n\times p},$$

where each row is an observation and each column is a feature. A row can be viewed as a point in $\mathbb{R}^p$. If $p=2$, the points can be drawn on a plane; if $p=3$, they occupy ordinary three-dimensional space. When $p$ is larger, the same geometric ideas remain valid even though direct visualization is impossible.

High dimensionality is not automatically a problem. It becomes a problem when many coordinates are redundant, noisy, difficult to visualize, or expensive for a later algorithm. Measurements of height in centimeters and height in inches, for example, contain essentially one dimension of information despite occupying two columns. Less extreme correlations also create directions in feature space with much more variation than others.

Dimensionality reduction seeks a map from $\mathbb{R}^p$ to $\mathbb{R}^k$, where $k<p$. PCA restricts attention to linear maps. For one component, it chooses a unit vector $w\in\mathbb{R}^p$ and replaces every centered observation $x_i$ with the scalar projection

$$z_i=x_i^\top w, \qquad w^\top w=1.$$

The unit-length condition matters. Without it, multiplying $w$ by an arbitrarily large constant would make projected values arbitrarily large, so maximizing their variance would have no meaningful solution.

There are two equivalent ways to state the PCA objective. The first asks for the projection with maximum variance. The second asks for the $k$-dimensional linear reconstruction with minimum total squared error. Equivalence is a consequence of the Pythagorean theorem: for an orthogonal projection, total centered energy separates into retained energy plus discarded energy.

If $W_k$ contains $k$ orthonormal directions as columns, then

$$W_k^\top W_k=I_k, \qquad Z=X_cW_k,$$

where $X_c$ is the centered data matrix. The rows of $Z$ are the reduced coordinates, commonly called principal component scores. Reconstruction in the original feature space is

$$\widehat X_c=ZW_k^\top=X_cW_kW_k^\top.$$

PCA chooses $W_k$ to minimize $\lVert X_c-\widehat X_c\rVert_F^2$, where the Frobenius norm sums the squares of every matrix entry. This formulation makes clear that PCA is a compression method based on squared Euclidean error.

Centering Data and Why It Matters

For feature $j$, let

$$\bar x_j=\frac{1}{n}\sum_{i=1}^n x_{ij}.$$

The centered entry is

$$(X_c)_{ij}=x_{ij}-\bar x_j.$$

Thus every column of $X_c$ has mean zero. Centering translates the data cloud so that its centroid lies at the origin. It does not rotate the cloud, change pairwise distances, or alter covariance. It establishes the correct origin from which variation should be measured.

Why is this step essential? PCA is intended to model variation around the mean, not distance from the arbitrary coordinate origin. Imagine adult heights measured in centimeters. Values may cluster around 170, but the scientifically relevant variation is a range of perhaps tens of centimeters around that mean. An uncentered decomposition treats the large offset from zero as dominant structure. The first component may then point toward the mean vector rather than along the direction in which observations differ.

With centered data, projected component scores also have mean zero:

$$\frac{1}{n}\mathbf{1}^\top X_cw=0.$$

When reconstructed values are needed in original coordinates, the mean must be added back:

$$\widehat X=\widehat X_c+\mathbf{1}\bar x^\top.$$

This training mean must also be reused for future observations. A production pipeline should fit means, scaling parameters, and PCA directions on training data only. Centering the complete dataset before a train/test split leaks information about the held-out distribution. For each new row $x_{\text{new}}$, calculate $x_{\text{new}}-\bar x_{\text{train}}$, project with the training directions, and use the same training mean during inverse transformation.

Centering is distinct from scaling. Centering removes location; scaling changes the relative units of the axes. PCA always needs deliberate treatment of location, while the decision to standardize depends on what feature magnitudes mean.

Covariance Matrices

The sample covariance matrix of centered data is

$$S=\frac{1}{n-1}X_c^\top X_c.$$

It is a $p\times p$ matrix. Its diagonal entries are sample variances:

$$S_{jj}=\frac{1}{n-1}\sum_{i=1}^n(x_{ij}-\bar x_j)^2.$$

Its off-diagonal entries are sample covariances:

$$S_{jk}=\frac{1}{n-1}\sum_{i=1}^n (x_{ij}-\bar x_j)(x_{ik}-\bar x_k).$$

A positive covariance means the two features tend to move above and below their means together. A negative covariance means one tends to be above its mean when the other is below. A covariance near zero indicates little linear co-movement, though nonlinear dependence may still exist.

The matrix is symmetric because $S_{jk}=S_{kj}$. It is also positive semidefinite. For any vector $w$,

$$w^\top Sw =\frac{1}{n-1}w^\top X_c^\top X_cw =\frac{1}{n-1}\lVert X_cw\rVert_2^2 \geq 0.$$

This expression has a direct statistical meaning: $w^\top Sw$ is the sample variance of observations after projection onto $w$. Therefore, finding the one-dimensional projection with maximum variance means solving

$$\max_{w}\;w^\top Sw \quad\text{subject to}\quad w^\top w=1.$$

Introduce a Lagrange multiplier $\lambda$:

$$\mathcal{L}(w,\lambda)=w^\top Sw-\lambda(w^\top w-1).$$

Differentiating with respect to $w$ gives

$$2Sw-2\lambda w=0,$$

and therefore

$$Sw=\lambda w.$$

The maximizing direction must be an eigenvector of $S$. Multiplying on the left by $w^\top$ and using $w^\top w=1$ shows

$$w^\top Sw=\lambda.$$

The eigenvalue is exactly the variance of the component. The maximum is attained by the eigenvector with the largest eigenvalue.

Eigenvalues and Eigenvectors of Symmetric Matrices

For a square matrix $A$, a nonzero vector $v$ is an eigenvector if

$$Av=\lambda v$$

for some scalar eigenvalue $\lambda$. The transformation $A$ can stretch, shrink, or reverse $v$, but it does not turn it away from its line.

Covariance matrices have unusually helpful structure. The spectral theorem states that every real symmetric matrix can be diagonalized by an orthogonal matrix:

$$S=V\Lambda V^\top.$$

The columns $v_1,\ldots,v_p$ of $V$ are orthonormal eigenvectors, so

$$V^\top V=VV^\top=I.$$

The diagonal matrix $\Lambda$ contains real eigenvalues. For a covariance matrix they are nonnegative:

$$\Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_p), \qquad \lambda_j\geq0.$$

PCA orders them so that

$$\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p.$$

The first principal axis is $v_1$, the second is $v_2$, and so forth. Orthogonality ensures that later components do not reuse a direction already selected. Component scores are uncorrelated because

$$\operatorname{Cov}(X_cV) =V^\top SV =\Lambda.$$

The diagonal covariance of the scores is one of PCA's defining results. It does not imply statistical independence except under special distributional assumptions, such as a multivariate Gaussian model. Uncorrelated variables can still have strong nonlinear relationships.

Eigenvector signs are not unique. If $v$ is an eigenvector, then $-v$ is also an eigenvector with the same eigenvalue. Consequently, different software may report opposite signs for component loadings and scores while representing exactly the same axes and reconstructions. This is not an error.

Repeated eigenvalues create a broader non-uniqueness. If $\lambda_j=\lambda_{j+1}$, any orthonormal rotation within their shared eigenspace is valid. The subspace is well defined, but individual basis vectors inside it may vary between algorithms or tiny data perturbations.

Variance Explained

The total sample variance across original features is the trace of the covariance matrix:

$$\operatorname{tr}(S)=\sum_{j=1}^pS_{jj}.$$

Trace is preserved by orthogonal diagonalization, so

$$\operatorname{tr}(S)=\sum_{j=1}^p\lambda_j.$$

The explained variance ratio of component $j$ is

$$r_j=\frac{\lambda_j}{\sum_{\ell=1}^p\lambda_\ell}.$$

The cumulative explained variance of the first $k$ components is

$$R_k=\frac{\sum_{j=1}^k\lambda_j} {\sum_{\ell=1}^p\lambda_\ell}.$$

If $R_2=0.92$, the first two principal directions contain 92% of the total sample variance under the chosen preprocessing. This does not mean that they contain 92% of every kind of useful information, produce 92% predictive accuracy, or preserve 92% of a target signal. PCA does not inspect a supervised target. A low-variance direction can be crucial for classification or scientific interpretation.

Loadings also require precise language. Depending on convention, “loading” may refer to the eigenvector coefficient $v_{jk}$ or to a correlation-scaled quantity involving $\sqrt{\lambda_k}v_{jk}$. Software documentation should be checked before comparing reported values. Eigenvector coefficients describe how standardized or unstandardized original coordinates combine to form a component:

$$z_{ik}=\sum_{j=1}^p (X_c)_{ij}v_{jk}.$$

Large absolute coefficients identify variables strongly represented in that axis, but interpretation depends on scaling, correlation, and sign convention.

SVD and Its Relation to PCA

The singular value decomposition of the centered data matrix is

$$X_c=U\Sigma V^\top.$$

For the compact SVD, $U$ has orthonormal columns in observation space, $V$ has orthonormal columns in feature space, and $\Sigma$ contains nonnegative singular values

$$\sigma_1\geq\sigma_2\geq\cdots\geq0.$$

Substitute the SVD into the covariance matrix:

$$S =\frac{1}{n-1}X_c^\top X_c =\frac{1}{n-1} V\Sigma^\top U^\top U\Sigma V^\top.$$

Since $U^\top U=I$,

$$S=V\frac{\Sigma^2}{n-1}V^\top.$$

Therefore the right singular vectors of $X_c$ are the covariance eigenvectors, and

$$\lambda_j=\frac{\sigma_j^2}{n-1}.$$

The PCA scores can be written in either of two equivalent ways:

$$Z=X_cV=U\Sigma.$$

This identity connects all major objects. $V$ supplies axes in feature space, $\Sigma$ measures the strength of each axis, and $U\Sigma$ locates observations along those axes.

In computation, direct SVD of the centered data matrix is often preferable to explicitly forming $X_c^\top X_c$. Forming the cross-product squares the condition number and can lose numerical precision. SVD is robust, handles rank deficiency naturally, and supports efficient truncated algorithms when only the first few components are required.

Which decomposition is cheaper depends on matrix shape and implementation. If $p$ is modest, eigendecomposition of a $p\times p$ covariance matrix may be convenient. If $p$ and $n$ are large, randomized or iterative truncated SVD can avoid computing all components. Importantly, applying generic truncated SVD to an uncentered matrix is not ordinary PCA; centering must still be handled unless the method explicitly incorporates it.

Choosing the Number of Components

Selecting $k$ is a modeling decision rather than a theorem. Several criteria can contribute evidence.

Cumulative explained variance

A common rule chooses the smallest $k$ for which $R_k$ exceeds a threshold such as 0.90, 0.95, or 0.99. This is easy to communicate but arbitrary. The appropriate loss depends on the application. Compression for visualization may tolerate more loss than preprocessing for a sensitive measurement system.

The scree plot

A scree plot places component index on the horizontal axis and eigenvalue or explained variance on the vertical axis. The name refers to rubble at the base of a cliff. Analysts look for an “elbow” separating steep, informative components from a flatter tail.

The scree plot is useful because it exposes the full decay pattern rather than hiding it behind one threshold. Yet many datasets have no sharp elbow, and two viewers may choose different bends. The plot is a diagnostic, not an objective selector.

Reconstruction and downstream validation

When compression quality matters, evaluate reconstruction error on held-out observations. When PCA feeds a classifier, regressor, or clustering algorithm, choose $k$ inside a validation pipeline using the downstream metric. Every fold must fit its own centering, scaling, and PCA transformation. Fitting PCA before cross-validation leaks information from validation folds even though PCA does not use labels.

Domain and interpretability constraints

A component may be retained because it represents a physically meaningful mode, an operationally important rare variation, or a required tolerance. Conversely, retaining hundreds of components to reach a conventional percentage may defeat the purpose of reduction. Effective selection combines mathematical summaries with the intended use.

A Worked Two-Dimensional Numeric Example

Consider four observations with two features:

The feature means are

$$\bar x_1=\frac{1+2+4+5}{4}=3, \qquad \bar x_2=\frac{2+1+5+4}{4}=3.$$

Subtracting $(3,3)$ gives

$$X_c= \begin{bmatrix} -2&-1\\ -1&-2\\ 1&2\\ 2&1 \end{bmatrix}.$$

Compute the cross-product:

$$X_c^\top X_c= \begin{bmatrix} 10&8\\ 8&10 \end{bmatrix}.$$

Because $n=4$, the sample covariance matrix is

$$S=\frac{1}{3} \begin{bmatrix} 10&8\\ 8&10 \end{bmatrix} = \begin{bmatrix} 10/3&8/3\\ 8/3&10/3 \end{bmatrix}.$$

For a symmetric matrix with equal diagonal values $a$ and equal off-diagonal values $b$, the directions $(1,1)$ and $(1,-1)$ are eigenvectors. After normalization,

$$v_1=\frac{1}{\sqrt2} \begin{bmatrix}1\\1\end{bmatrix}, \qquad v_2=\frac{1}{\sqrt2} \begin{bmatrix}1\\-1\end{bmatrix}.$$

Apply $S$ to $v_1$:

$$Sv_1 =\frac{1}{\sqrt2} \begin{bmatrix} 10/3+8/3\\ 8/3+10/3 \end{bmatrix} =6v_1.$$

Thus $\lambda_1=6$. Similarly,

$$Sv_2 =\frac{1}{\sqrt2} \begin{bmatrix} 10/3-8/3\\ 8/3-10/3 \end{bmatrix} =\frac{2}{3}v_2,$$

so $\lambda_2=2/3$. Total variance is

$$\lambda_1+\lambda_2 =6+\frac23 =\frac{20}{3},$$

which agrees with $\operatorname{tr}(S)=10/3+10/3$.

The explained variance ratios are

$$r_1=\frac{6}{20/3}=0.9, \qquad r_2=\frac{2/3}{20/3}=0.1.$$

One component therefore retains 90% of total sample variance.

Project each centered observation onto $v_1$:

$$Z_1=X_cv_1 =\frac{1}{\sqrt2} \begin{bmatrix} -3\\-3\\3\\3 \end{bmatrix}.$$

The scores are $-3/\sqrt2,-3/\sqrt2,3/\sqrt2,3/\sqrt2$. Their sample variance is

$$\frac{1}{3}\left( \frac92+\frac92+\frac92+\frac92 \right)=6=\lambda_1.$$

Using only the first component, centered reconstruction is $Z_1v_1^\top$. Observation A has centered coordinates $(-2,-1)$ and score $-3/\sqrt2$, so its reconstruction is

$$\widehat x_{c,A} =-\frac{3}{\sqrt2} \frac{1}{\sqrt2}(1,1) =\left(-\frac32,-\frac32\right).$$

Adding back the mean gives

$$\widehat x_A =\left(\frac32,\frac32\right).$$

The original point $(1,2)$ has been projected orthogonally onto the diagonal line through $(3,3)$. Its residual is

$$(1,2)-\left(\frac32,\frac32\right) =\left(-\frac12,\frac12\right),$$

whose squared length is $1/2$. Every observation has the same squared residual here, so total squared reconstruction error is $2$. This equals

$$(n-1)\lambda_2=3\left(\frac23\right)=2,$$

exactly as PCA theory predicts.

The corresponding singular values are

$$\sigma_1=\sqrt{(n-1)\lambda_1}=\sqrt{18}=3\sqrt2, \qquad \sigma_2=\sqrt{(n-1)\lambda_2}=\sqrt2.$$

This small example displays the complete equivalence among covariance eigenvalues, singular values, projected variance, and reconstruction loss.

A NumPy Implementation

The following sample performs PCA by SVD, computes explained variance, reconstructs with one component, and verifies the error identity. NumPy may choose opposite eigenvector signs from the hand calculation; reconstructions and explained variances remain unchanged.

import numpy as np

X = np.array([
    [1.0, 2.0],
    [2.0, 1.0],
    [4.0, 5.0],
    [5.0, 4.0],
])

# Fit centering on the observed training matrix.
mean = X.mean(axis=0)
X_centered = X - mean

# X_centered = U @ diag(singular_values) @ Vt
U, singular_values, Vt = np.linalg.svd(X_centered, full_matrices=False)

n = X.shape[0]
explained_variance = singular_values**2 / (n - 1)
explained_ratio = explained_variance / explained_variance.sum()

# Keep the first principal direction.
k = 1
components = Vt[:k].T
scores = X_centered @ components
X_reconstructed = scores @ components.T + mean

squared_error = np.sum((X - X_reconstructed) ** 2)
discarded_error = (n - 1) * explained_variance[k:].sum()

print("mean:", mean)
print("principal directions:\n", Vt.T)
print("explained variance:", explained_variance)
print("explained ratio:", explained_ratio)
print("one-component scores:\n", scores)
print("reconstruction:\n", X_reconstructed)
print("squared reconstruction error:", squared_error)
print("error from discarded eigenvalues:", discarded_error)

For a real workflow, wrap centering, optional standardization, PCA, and any downstream estimator in a pipeline. This prevents preprocessing leakage and records the exact transformation applied to future data.

Reconstruction Error and Optimality

The rank-$k$ truncated SVD is

$$X_{c,k}=U_k\Sigma_kV_k^\top.$$

The Eckart–Young–Mirsky theorem states that this is a best rank-$k$ approximation to $X_c$ under both the Frobenius norm and spectral norm. For squared Frobenius error,

$$\lVert X_c-X_{c,k}\rVert_F^2 =\sum_{j=k+1}^{r}\sigma_j^2,$$

where $r$ is the rank. Using $\sigma_j^2=(n-1)\lambda_j$,

$$\lVert X_c-X_{c,k}\rVert_F^2 =(n-1)\sum_{j=k+1}^{r}\lambda_j.$$

Thus every discarded eigenvalue contributes directly to total squared reconstruction error. Retaining the largest eigenvalues is not merely intuitive; it is the globally optimal linear rank-$k$ choice for this loss.

The result has boundaries. “Optimal” refers to approximation of the same centered matrix under a squared Euclidean criterion. It does not promise optimal class separation, causal discovery, robustness to outliers, or preservation of nonlinear manifolds. Change the objective, and another representation may be preferable.

Relative average reconstruction loss can be expressed through omitted variance. Because

$$\lVert X_c\rVert_F^2=(n-1)\sum_j\lambda_j,$$

the retained fraction of squared centered energy is $R_k$, while the discarded fraction is $1-R_k$. Individual observations can have very different reconstruction errors even when the overall retained fraction is high. Inspecting row-level errors can reveal unusual cases, but PCA itself is sensitive to those cases during fitting.

Whitening

Ordinary PCA produces uncorrelated components whose variances are $\lambda_1,\ldots,\lambda_k$. Whitening additionally rescales retained scores to unit variance:

$$Z_{\text{white}} =X_cV_k\Lambda_k^{-1/2}.$$

Then, in the fitted sample,

$$\operatorname{Cov}(Z_{\text{white}})=I_k.$$

In SVD form, since $\Lambda_k^{-1/2}$ has entries $\sqrt{n-1}/\sigma_j$,

$$Z_{\text{white}}=U_k\sqrt{n-1}.$$

Whitening can be useful when a later method expects isotropic input or when unequal component variances would dominate Euclidean distances. It appears in signal processing, independent component analysis preprocessing, and some optimization pipelines.

However, whitening removes the very variance hierarchy that PCA discovers. A weak, potentially noisy component is amplified until its variance equals that of a dominant component. If $\lambda_j$ is extremely small, division by $\sqrt{\lambda_j}$ can magnify numerical noise severely. Practical implementations may drop small components or regularize the denominator.

Whitening is also not the same as standardizing original features. Standardization rescales coordinate axes before PCA; whitening rotates into principal axes and rescales after PCA. Nor does whitening guarantee independence. It creates zero sample covariance and unit variance, which are second-order properties.

To invert a retained whitened representation, reverse the scaling and rotation:

$$\widehat X =Z_{\text{white}}\Lambda_k^{1/2}V_k^\top +\mathbf{1}\bar x^\top.$$

If all nonzero components are retained, this reconstructs the centered data up to numerical precision. If components are discarded, the same rank-reduction loss remains; whitening does not recover omitted information.

Limitations: Linearity, Scaling, and Data Quality

PCA is linear

PCA represents data with a flat subspace. It works well when a cloud lies near a line, plane, or higher-dimensional hyperplane. It can fail on curved structure. Points arranged around a circle require two linear components for exact representation even though their position can be described by one angular coordinate. Kernel PCA, autoencoders, Isomap, or other manifold methods may capture nonlinear structure, but each introduces new assumptions and tuning choices.

Variance is not synonymous with importance

PCA preserves directions with large variance. Measurement noise can have large variance, while a subtle low-variance signal can determine an outcome. In supervised tasks, evaluate component choices against held-out target performance rather than assuming high explained variance preserves predictive value.

Scaling controls the answer

If one feature is measured in dollars and another in proportions, the dollar feature may dominate the covariance matrix merely because of units. Standardizing each feature,

$$z_{ij}=\frac{x_{ij}-\bar x_j}{s_j},$$

makes PCA operate on the correlation matrix. This gives every nonconstant feature unit sample variance before rotation.

Standardization is often appropriate when units are incomparable and equal relative variation is intended. It is not universally correct. In a set of sensor channels measured in the same calibrated unit, larger variance may be physically meaningful. Scaling all channels equally could amplify a nearly constant noisy sensor. Domain knowledge must determine whether raw covariance, correlation, robust scaling, or custom weights express the relevant geometry.

Outliers and heavy tails

Means, covariances, and squared reconstruction error are all sensitive to extreme observations. A few unusual points can rotate principal axes and inflate eigenvalues. Verify data quality, visualize robust summaries, compare fits with sensitivity analyses, and consider robust covariance or robust PCA methods when contamination is plausible.

Interpretation can be unstable

When eigenvalues are close, small sampling changes can rotate the associated eigenvectors substantially. A component interpreted as a specific latent factor may not reproduce in another sample. Bootstrap analyses and subspace comparisons can reveal instability. PCA discovers mathematical directions, not guaranteed real-world causes.

Missing and mixed data

Basic PCA requires a complete numeric matrix. Mean imputation underestimates variability and can distort covariance. More principled options include iterative imputation, probabilistic PCA, or methods designed for missingness. Nominal categories do not naturally support subtraction, averaging, and Euclidean projection. One-hot encoding is possible but changes geometry and may overweight high-cardinality variables.

Common Mistakes

1. Skipping centering. Uncentered decomposition can model distance from zero rather than variation around the sample mean.
2. Standardizing automatically. Unit variance is an assumption about relative importance, not a harmless default in every domain.
3. Fitting preprocessing before splitting. Means, scales, and components must be learned inside each training fold.
4. Reading explained variance as accuracy. It measures retained unsupervised sample variance, not predictive correctness.
5. Assuming components are independent. PCA decorrelates components; independence requires stronger conditions.
6. Treating sign changes as disagreement. $v$ and $-v$ define the same principal axis.
7. Interpreting every component causally. Loadings are algebraic combinations, not proof of hidden causal factors.
8. Keeping components only by a fixed threshold. A 95% rule may discard a useful low-variance signal or retain too many noisy directions.
9. Using PCA to guarantee better models. Reduction can remove signal, reduce interpretability, and add pipeline complexity.
10. Computing an explicit covariance matrix unnecessarily. Direct SVD is frequently more stable and efficient.
11. Ignoring reconstruction units. Squared errors on unscaled features are dominated by large-unit variables.
12. Confusing PCA with feature selection. Components combine features; they do not usually select a subset of original columns.
13. Forgetting the inverse mean shift. Reconstructed centered values must have the training mean added back.
14. Whitening tiny eigenvalues. Near-zero variances create extreme scale factors and amplify noise.
15. Overinterpreting a two-dimensional plot. A clear projection can hide overlap, outliers, or structure in omitted dimensions.

A sound PCA report states whether data were centered and scaled, which convention was used for sample variance, how $k$ was chosen, what validation was performed, and how much reconstruction error remains. Reproducibility requires these decisions, not merely a list of eigenvalues.

Practice with Solver360

Use the PCA Calculator to connect the algebra to a visible rotation. Begin with the four-point worked example. Verify the mean $(3,3)$, covariance matrix, principal directions, eigenvalues $6$ and $2/3$, and 90% first-component explained variance. Follow each observation from original coordinates to centered coordinates, component scores, and reconstructed coordinates.

Then change one point at a time. Move observation D far away and observe how covariance and the first axis react. Multiply the first feature by 100, compare the raw PCA result with a standardized result, and explain why the directions differ. Create points near a horizontal line, then near a circle, to contrast linear and curved structure. If the calculator provides a scree plot, look for how exact collinearity creates one nonzero eigenvalue while isotropic data creates similar eigenvalues.

The Matrix Operations Calculator is useful for checking the intermediate linear algebra. Enter the centered matrix, calculate $X_c^\top X_c$, divide by $n-1$, and verify matrix-vector products such as $Sv_1=\lambda_1v_1$. Confirm that $V^\top V=I$ and that $V\Lambda V^\top$ reconstructs the covariance matrix.

A disciplined practice sequence is:

1. Compute feature means and center a tiny dataset by hand.
2. Form its covariance matrix.
3. Predict the dominant geometric direction before calculating it.
4. Compute eigenpairs and sort them by descending eigenvalue.
5. Project observations and verify score variances.
6. Reconstruct with fewer components and calculate squared error.
7. Compare omitted eigenvalues with reconstruction loss.
8. Repeat after scaling to see how geometry changes.

This workflow makes every software output accountable to a formula.

Frequently Asked Questions

What is the shortest mathematical definition of PCA?

PCA finds orthonormal directions that maximize projected variance in centered data. Equivalently, its first $k$ directions define the rank-$k$ linear projection that minimizes total squared reconstruction error. They are the eigenvectors associated with the largest eigenvalues of the sample covariance matrix, or the leading right singular vectors of the centered data matrix.

Why are covariance eigenvalues never negative?

For every vector $w$,

$$w^\top Sw=\frac{1}{n-1}\lVert X_cw\rVert_2^2\geq0.$$

A symmetric matrix with this property is positive semidefinite, and all its eigenvalues are nonnegative. Tiny negative values occasionally reported by numerical software usually come from floating-point rounding and should be near zero.

Is PCA calculated from the covariance matrix or with SVD?

Both routes describe the same ordinary PCA when applied consistently to centered data. Eigendecomposition diagonalizes $S=X_c^\top X_c/(n-1)$. SVD factors $X_c=U\Sigma V^\top$. The covariance eigenvectors are $V$, and eigenvalues are $\sigma_j^2/(n-1)$. Direct SVD is often numerically preferable because it avoids explicitly forming the cross-product.

Must data always be standardized before PCA?

No. Data should ordinarily be centered, but standardization is a modeling choice. Standardize when feature units are incomparable and one standard deviation should have comparable importance across variables. Retain original scaling when common units and absolute variation have meaningful interpretation. Always state the choice because it can substantially change the components.

How much explained variance is enough?

There is no universal threshold. Visualization may work with two or three components even when they retain substantially less than 95%. Compression or measurement reconstruction may need 99% or a direct error tolerance. A supervised pipeline should select the number through validation of downstream performance. Scree plots, stability, domain requirements, and computational constraints should supplement cumulative variance.

Why do two programs return component signs that are reversed?

Eigenvectors and singular vectors are sign-indeterminate. If $v$ is valid, $-v$ is equally valid. Reversing a component direction also reverses its scores, so their product in reconstruction is unchanged. Compare absolute direction or reconstructed results rather than requiring identical signs.

Are principal components independent?

They are uncorrelated in the sample used to fit PCA because their covariance matrix is diagonal. Independence is stronger: it requires the joint distribution to factorize. For multivariate Gaussian data, uncorrelated linear components are independent, but this does not hold generally. Independent component analysis pursues a different objective aimed at non-Gaussian independence.

What happens if there are more features than observations?

After centering, the rank of $X_c$ is at most $\min(p,n-1)$. Therefore, at most $n-1$ covariance eigenvalues can be positive when $p\geq n$. The remaining feature-space directions have zero sample variance. SVD handles this rank deficiency naturally, and truncated implementations can avoid constructing a very large $p\times p$ covariance matrix.

Does PCA remove multicollinearity?

Retained principal component scores are mutually uncorrelated, so using them in a regression removes exact linear dependence among those scores. However, PCA does not make the original variables conceptually independent, and unsupervised variance ranking may discard directions important for predicting the target. Ridge regression or supervised dimension-reduction methods may be preferable when prediction is the primary goal.

Can reconstruction error detect anomalies?

Large error can flag an observation that lies far from the retained principal subspace. This can be useful when normal data are expected to occupy a stable low-dimensional linear region. It is not a complete anomaly detector: outliers influence the fitted subspace, anomalies can lie within that subspace, and different subgroups may have different normal error scales. Fit on representative normal training data and validate thresholds against realistic cases.

What is the difference between PCA and whitening?

PCA rotates centered data into uncorrelated axes and usually retains their natural variances. Whitening performs that rotation and divides each retained score by the square root of its eigenvalue, producing unit sample variance. Whitening can help later algorithms but erases variance magnitudes and may amplify noisy low-variance directions.

Can PCA recover nonlinear structure?

Ordinary PCA cannot unfold a curved manifold because it uses a linear subspace. It may still provide a useful approximation if curvature is mild or if local regions are analyzed separately. Kernel PCA and nonlinear representation methods can model curves, but they are harder to invert and interpret and require additional choices. A nonlinear method should be validated rather than assumed superior.

From PCA to Linear Algebra and K-Means

PCA becomes much easier to remember once its equations are treated as one chain. Centering defines variation around a meaningful origin. The covariance matrix records variation in every direction. Symmetry guarantees orthonormal eigenvectors and real, nonnegative eigenvalues. The leading eigenvectors maximize projected variance. SVD reaches the same axes directly from the data matrix, with squared singular values determining explained variance. Truncation preserves the strongest directions and minimizes squared linear reconstruction error.

The next natural topic is a broader linear algebra article covering vector spaces, rank, bases, orthogonality, projections, matrix factorizations, and condition numbers. Those ideas explain not only PCA but also least squares, optimization, neural network layers, and numerical stability.

PCA also leads directly to k-means. Clustering in a carefully selected principal-component space can reduce noise and computational cost, and two-dimensional component scores can help visualize candidate groups. But PCA optimizes variance, while k-means optimizes within-cluster squared distance. A component with modest variance may separate clusters, and a colorful projected plot can hide overlap in omitted dimensions. Treat PCA and k-means as distinct objective functions in a shared geometric toolkit, then validate the combined pipeline against the structure and purpose of the data.