Statistics for Machine Learning: Expectation, Variance, Bias, and Evaluation

Machine learning is often presented as a collection of algorithms, but every useful model is also a statistical argument. A training set is a limited sample from a larger process; a fitted parameter is an estimate rather than an eternal constant; and an evaluation score is an uncertain measurement of future performance. Statistics supplies the language for reasoning about those facts. It explains what averages summarize, why fitted models vary across datasets, how uncertainty should be reported, and when an apparent improvement is too small or unstable to support a conclusion.

Statistics for Machine Learning: Expectation, Variance, Bias, and Evaluation

The central statistical question in machine learning is not merely, “How well did this model fit the data?” It is, “What can this finite dataset tell us about behavior on observations we have not yet seen?” Answering that question requires several connected ideas: populations and samples, expectation and variance, estimation and uncertainty, dependence between variables, resampling, and careful evaluation.

This chapter develops those ideas as a practical foundation. The goal is not to replace a full course in mathematical statistics. It is to establish enough theory and intuition to recognize what a machine-learning result does and does not justify.

Statistics, probability, and machine learning

Probability begins with a model of uncertainty and asks what outcomes that model can produce. If a fair coin is tossed ten times, probability can describe the chance of observing exactly six heads. The probability model and its parameters are treated as known, and possible data are the unknown quantities.

Statistics reverses the direction. It begins with observed data and asks what underlying process could plausibly have produced them. After observing six heads in ten tosses, a statistician may estimate the coin's probability of heads, quantify uncertainty in that estimate, or test whether fairness is consistent with the evidence. The data are known, while the process and its parameters are partly unknown.

Machine learning uses both directions. During training, it resembles statistics: the learner uses a sample to estimate a prediction rule. During prediction, it resembles probability: the fitted rule assigns outcomes, scores, or distributions to new inputs. Machine learning adds a strong computational and predictive emphasis. It often tolerates models whose parameters are not individually interpretable, provided that their predictions generalize reliably.

These disciplines therefore overlap without being identical:

• Probability formalizes randomness and uncertainty.
• Statistics draws conclusions about a population or process from sampled data.
• Machine learning constructs data-driven systems that predict, classify, rank, generate, or decide.

A probability distribution is a mathematical description such as $Y\sim\mathcal{N}(\mu,\sigma^2)$. A statistic is any function computed from observed data, such as the sample mean. A model parameter, such as $\mu$, belongs to the assumed population model; an estimate, such as $\bar{x}$, is calculated from the sample. Keeping these terms separate prevents many common errors.

Random variables, expectation, and observed values

A random variable represents a quantity whose value is uncertain before observation. Let $X$ denote the response time of a future user. Its expectation,

$$\mathbb{E}[X]=\sum_x xP(X=x)$$

for a discrete variable, or

$$\mathbb{E}[X]=\int_{-\infty}^{\infty}x f(x)\,dx$$

for a continuous variable, is the distribution's probability-weighted center. Expectation is a property of the random process, not necessarily a value that can actually occur. The expected result of a fair six-sided die is $3.5$, although no roll can equal $3.5$.

An observed value $x_i$ is one realization of $X$. A dataset $x_1,\ldots,x_n$ contains realizations, while formulas containing $X$ describe the uncertain mechanism. Machine-learning notation sometimes blurs this distinction, but statistical reasoning depends on it.

Populations, samples, and descriptive statistics

The population is the full set of cases about which a conclusion is intended. It might mean all transactions produced by a payment system next month, all images captured by a particular sensor, or all patients satisfying clinical criteria. The population need not be a physically available list. It is often conceptual: the stream of observations generated under specified conditions.

A sample is the subset actually observed. If the sample is not representative of the target population, even flawless calculations can support the wrong conclusion. A model trained on daytime photographs may fail at night; a churn model built only from long-standing customers may misrepresent new users. More data do not automatically repair selection bias when the added observations come from the same distorted source.

Mean, median, and mode

For observations $x_1,\ldots,x_n$, the sample mean is

$$\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i.$$

The mean uses every value and has important algebraic properties. It is the value that minimizes the sum of squared deviations $\sum_i(x_i-c)^2$. It is also sensitive to extreme observations. One exceptionally large transaction can move the mean substantially.

The median is the middle ordered value, or the average of the two middle values when $n$ is even. It minimizes the sum of absolute deviations $\sum_i|x_i-c|$ and is more resistant to outliers. For skewed quantities such as income, latency, or transaction size, the median may better represent a typical observation.

The mode is the most frequent value. It is immediately useful for categories and discrete values. For continuous measurements, an exact mode may be unstable because repeated values are uncommon; a histogram-based mode also depends on bin widths. A distribution can have one mode, several modes, or no uniquely useful mode.

No center is universally best. The relevant choice depends on the distribution and the question. A mean latency describes total resource demand, while a median latency describes the midpoint user experience. A high percentile may be more relevant than either when service-level objectives concern slow requests.

Variance and standard deviation

Center alone does not describe a distribution. The population variance is

$$\operatorname{Var}(X) =\mathbb{E}\left[(X-\mu)^2\right] =\mathbb{E}[X^2]-\mu^2,$$

where $\mu=\mathbb{E}[X]$. Variance is nonnegative and gives more weight to large deviations because deviations are squared.

The usual sample variance is

$$s^2=\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2.$$

The denominator $n-1$, rather than $n$, corrects the systematic downward bias that appears when the same sample is used to estimate the mean and dispersion. Once $\bar{x}$ has been calculated, the deviations must sum to zero, leaving only $n-1$ independent degrees of freedom.

The standard deviation is $s=\sqrt{s^2}$. It is expressed in the original units, making it easier to interpret than variance. If heights have a standard deviation of 7 cm, the scale is immediately meaningful; a variance of $49\text{ cm}^2$ is mathematically convenient but less intuitive.

The following summary distinguishes common descriptive quantities and their machine-learning uses.

Standard deviation should not be confused with standard error. Standard deviation describes variation among individual observations. Standard error describes variation in an estimator across hypothetical repeated samples.

Bias, variance, and prediction error

The words bias and variance have precise meanings in statistical learning. Suppose a training procedure is repeatedly applied to new datasets drawn from the same population. At an input $x$, it produces a fitted prediction $\hat f(x)$ that changes from sample to sample.

The bias of the prediction procedure at $x$ is

$$\operatorname{Bias}[\hat f(x)] =\mathbb{E}[\hat f(x)]-f(x),$$

where $f(x)=\mathbb{E}[Y\mid X=x]$ is the true conditional mean. High bias means that even the average fitted model systematically misses the target. A straight line fitted to a sharply curved relationship is a typical example.

The variance is

$$\operatorname{Var}[\hat f(x)] =\mathbb{E}\left[ \left(\hat f(x)-\mathbb{E}[\hat f(x)]\right)^2 \right].$$

High variance means that modest changes in the training sample produce substantially different fitted predictions. A deep unpruned decision tree can have this behavior: it adapts closely to individual training observations and changes its structure when those observations change.

The bias-variance decomposition

Assume the outcome follows

$$Y=f(X)+\varepsilon, \qquad \mathbb{E}[\varepsilon\mid X]=0, \qquad \operatorname{Var}(\varepsilon\mid X)=\sigma_\varepsilon^2.$$

For squared prediction error at a fixed input $x$, the expected test error decomposes as

$$\mathbb{E}\left[(Y-\hat f(x))^2\right] =\sigma_\varepsilon^2 +\operatorname{Bias}[\hat f(x)]^2 +\operatorname{Var}[\hat f(x)].$$

The first term is irreducible noise: variation in $Y$ that cannot be eliminated by predicting its conditional mean. The second reflects systematic model error. The third reflects sensitivity to the training sample.

Increasing model flexibility often lowers bias but raises variance. A constant predictor has strong assumptions and usually high bias, but it changes little across samples. A highly flexible predictor can approximate complex functions, yet may chase accidental sample details. The bias-variance tradeoff is the search for a useful balance, not a rule that one component must always rise exactly when the other falls.

Regularization, feature selection, early stopping, pruning, and data augmentation can reduce variance. Richer features or more flexible model classes can reduce bias. More representative training data often reduce estimator variance without requiring a simpler model. The right intervention depends on evidence: similar and poor training and validation scores suggest underfitting, while excellent training performance and much worse validation performance suggest overfitting.

Bias here is not the same as social or measurement bias, though both matter. Statistical bias concerns the expected value of an estimator. Dataset bias concerns systematic mismatch or distortion in data collection. Fairness concerns the distribution of consequences across groups. A model may have low statistical bias in estimating an overall mean while still producing unfair group-level outcomes.

Covariance and correlation

Machine learning depends on relationships among variables. Covariance measures whether two numeric variables tend to move together:

$$\operatorname{Cov}(X,Y) =\mathbb{E}\left[(X-\mu_X)(Y-\mu_Y)\right].$$

Its sample estimate is

$$s_{XY} =\frac{1}{n-1} \sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}).$$

A positive covariance indicates that above-average values of $X$ tend to accompany above-average values of $Y$. A negative covariance indicates that above-average values of one tend to accompany below-average values of the other. A covariance near zero indicates little linear co-movement, but it does not rule out nonlinear dependence.

Covariance depends on units. Measuring height in meters rather than centimeters changes covariance numerically. Pearson correlation standardizes covariance:

$$r_{XY}=\frac{s_{XY}}{s_Xs_Y}.$$

It lies between $-1$ and $1$. Values near $1$ indicate a strong positive linear relationship, values near $-1$ a strong negative linear relationship, and values near $0$ a weak linear relationship.

Correlation has several limitations. It does not establish causation. A third variable may influence both observed variables, the direction of influence may be reversed, or selection may create an association. Correlation can also conceal nonlinear structure: if $Y=X^2$ and $X$ is symmetric around zero, Pearson correlation can be near zero despite exact dependence. Outliers can dominate it, restricted ranges can weaken it, and aggregating groups can reverse an apparent relationship.

In machine learning, covariance matrices appear in principal component analysis, Gaussian models, and uncertainty calculations. Highly correlated predictors can make individual linear-regression coefficients unstable even when predictions remain adequate. Correlation analysis is therefore diagnostic, not a mechanical rule for deleting features. Two correlated variables may carry distinct causal or operational meaning.

Estimation and sampling uncertainty

An estimator is a rule that maps a sample to an estimate. The formula $\bar{X}=n^{-1}\sum_iX_i$ is an estimator; the value $12.4$ calculated from a particular dataset is an estimate. Good estimators are assessed by properties such as bias, variance, consistency, robustness, and mean squared error.

If $X_1,\ldots,X_n$ are independent observations with mean $\mu$ and variance $\sigma^2$, then

$$\mathbb{E}[\bar X]=\mu, \qquad \operatorname{Var}(\bar X)=\frac{\sigma^2}{n}.$$

The standard deviation of the sampling distribution of $\bar X$ is the standard error:

$$\operatorname{SE}(\bar X)=\frac{\sigma}{\sqrt n}.$$

Because $\sigma$ is normally unknown, it is commonly estimated by $s$, giving $s/\sqrt n$. The square-root relationship is important: reducing standard error by half requires approximately four times as many independent observations. Merely duplicating records does not add information, and correlated observations usually provide less information than the nominal sample size suggests.

Confidence intervals as procedures

A confidence interval combines an estimate with its sampling uncertainty. Under standard conditions, an approximate interval for a mean is

$$\bar x \pm t_{1-\alpha/2,n-1}\frac{s}{\sqrt n},$$

where $t_{1-\alpha/2,n-1}$ is a critical value from a Student $t$ distribution. For large samples and a 95% interval, the multiplier is often near $1.96$.

The careful interpretation is procedural: if the same sampling and interval-construction procedure were repeated many times, approximately 95% of the resulting 95% intervals would contain the fixed population mean. After one interval has been computed, classical frequentist theory does not assign a 95% probability to the fixed parameter being inside that particular interval.

In practice, the interval communicates a range of parameter values reasonably compatible with the data and assumptions. Its validity depends on how the sample was collected, whether observations are sufficiently independent, whether the standard-error formula fits the design, and whether the approximation is appropriate. A very narrow interval around a biased estimate is precise but wrong.

Confidence intervals can also be constructed for differences in model metrics, not only means. Paired evaluation is often valuable: calculate each model's performance on the same folds or test cases and estimate uncertainty in their difference. Shared cases remove some irrelevant variation and focus analysis on comparative performance.

Hypothesis testing and p-values

A hypothesis test begins with a null hypothesis $H_0$, such as “the population mean difference between two procedures is zero.” A test statistic summarizes how far the observed data depart from what $H_0$ predicts. The p-value is the probability, assuming the null hypothesis and the test assumptions are correct, of obtaining a result at least as incompatible with $H_0$ as the one observed.

The p-value is not:

• the probability that the null hypothesis is true;
• the probability that the result occurred “by chance”;
• the size or practical importance of an effect;
• a guarantee that the finding will replicate;
• evidence that all modeling assumptions are valid.

A small p-value indicates tension between the observed statistic and the null model. It does not identify which assumption failed. A large p-value does not prove equality; the test may have low power because the sample is small or noisy.

Thresholding at $p<0.05$ can create false certainty. A result just below 0.05 is not categorically different from one just above it. Repeatedly testing features, subgroups, metrics, and random seeds also raises the chance of finding an apparently significant result. Confirmatory studies should specify hypotheses and analyses before inspecting outcomes, while exploratory analyses should be labeled as such and validated on fresh data.

For machine learning, effect sizes and uncertainty intervals are usually more informative than isolated p-values. If a new classifier improves accuracy by 0.1 percentage points with a tiny p-value on ten million examples, the effect may still be operationally irrelevant. Conversely, a potentially valuable improvement with a wide interval may justify collecting more evidence rather than being dismissed.

Train/test evaluation as statistical estimation

The training set serves two roles: it provides examples from which parameters are estimated, and it influences choices about preprocessing, architecture, and hyperparameters. Evaluating on the same observations produces an optimistically biased estimate of future performance because the fitted procedure was selected to perform well there.

A test set approximates new draws from the target population. If $(X_i,Y_i)$ in the test set are independent of model fitting, the average test loss

$$\widehat R_{\text{test}} =\frac{1}{m}\sum_{i=1}^{m} \ell\left(Y_i,\hat f(X_i)\right)$$

estimates the model's expected generalization risk. It remains an estimate. A different test sample would produce a different score, and a distribution shift can make the entire estimate irrelevant to deployment.

Overfitting is a selection phenomenon

Overfitting is often described as memorization, but its statistical form is broader. A model overfits whenever it adapts to sample-specific noise or accidental structure that does not persist in the target population. Flexible parameters can overfit, but so can repeated human decisions. If researchers try many feature sets and retain the one with the highest test score, the test set has become part of training even if no gradient was computed from it.

This is why a validation set is commonly separated from the final test set. Training data fit parameters. Validation data guide model and hyperparameter choices. Test data support a final, relatively unbiased evaluation. Once test results influence further choices, they should no longer be presented as untouched confirmation.

Data splitting must respect the observation-generating process. Random row splits are inappropriate when rows from the same person, device, document, or future time period can appear on both sides. Grouped splits prevent entity leakage. Time-ordered splits prevent future information from predicting the past. Preprocessing transformations, including scaling and imputation, must be fitted only on training portions and then applied to held-out data.

Predictive metrics and statistical significance

A predictive metric answers a performance question. Mean absolute error measures the average absolute numeric miss. Log loss evaluates probabilistic classification and strongly penalizes confident mistakes. Precision and recall emphasize different error types. A ranking metric may be appropriate when ordering matters more than calibrated probabilities.

Statistical significance answers a different question: whether observed evidence is difficult to reconcile with a specified null model. Neither concept substitutes for the other.

When comparing models, ask at least four questions:

1. Is the metric aligned with the actual cost or goal? Accuracy can be misleading under class imbalance.
2. How large is the improvement? Report the absolute difference and, when meaningful, a relative difference.
3. How uncertain is that difference? Use confidence intervals, repeated resampling, or a suitable paired test.
4. Will the difference matter in deployment? Consider latency, calibration, maintenance, fairness, and economic value.

Suppose model A has accuracy $0.912$ and model B has accuracy $0.914$. Those decimals alone do not establish superiority. The scores may differ because of test-sample variation, random initialization, or hyperparameter search. Even if the difference is statistically distinguishable, two additional correct predictions per thousand may not offset greater computation or lower interpretability.

Metrics also have distributions. Accuracy on $m$ approximately independent test cases is a sample proportion, while cross-validation generates correlated fold scores because training sets overlap. Treating fold scores as independent observations in a simple test can underestimate uncertainty. The resampling design must be considered when interpreting variability.

Worked example with a small dataset

Consider six observed validation errors, measured in percentage points:

$$4,\ 6,\ 6,\ 8,\ 10,\ 14.$$

This dataset is deliberately small enough to calculate by hand. The sample mean is

$$\bar x =\frac{4+6+6+8+10+14}{6} =\frac{48}{6} =8.$$

The ordered values are already shown. With six observations, the median is the average of the third and fourth values:

$$\operatorname{median} =\frac{6+8}{2} =7.$$

The mode is $6$, because it occurs twice and every other value occurs once. Notice how the three centers tell different stories. The high value $14$ pulls the mean above the median, while the mode identifies the most frequently observed value.

To calculate sample variance, subtract the mean and square each deviation:

$$\begin{aligned} \sum_{i=1}^{6}(x_i-\bar x)^2 &=(4-8)^2+(6-8)^2+(6-8)^2\\ &\quad +(8-8)^2+(10-8)^2+(14-8)^2\\ &=16+4+4+0+4+36\\ &=64. \end{aligned}$$

Therefore,

$$s^2=\frac{64}{6-1}=12.8, \qquad s=\sqrt{12.8}\approx3.578.$$

The estimated standard error of the mean is

$$\operatorname{SE}(\bar x) =\frac{s}{\sqrt n} =\frac{3.578}{\sqrt6} \approx1.461.$$

For a 95% confidence interval with $n-1=5$ degrees of freedom, the $t$ critical value is approximately $2.571$. The interval is

$$8\pm2.571(1.461) \approx8\pm3.756,$$

or approximately $(4.24,11.76)$. The interval is wide because only six observations are available and the data are dispersed. Reporting the mean as exactly 8 without this uncertainty would imply more knowledge than the sample provides.

Now pair the errors with model-complexity scores

$$1,\ 2,\ 3,\ 4,\ 5,\ 6.$$

The complexity mean is $3.5$. The cross-deviation sum is

$$\sum_i(c_i-\bar c)(x_i-\bar x)=32,$$

so the sample covariance is $32/5=6.4$. The complexity standard deviation is approximately $1.871$, and the error standard deviation is approximately $3.578$. Thus,

$$r =\frac{6.4}{(1.871)(3.578)} \approx0.956.$$

This small sample shows a strong positive linear association between complexity and validation error. It does not prove that complexity causes error, nor that the relationship will persist. Perhaps later experiments were conducted under harder conditions, or perhaps the sample is simply too small. The calculation is descriptive evidence that motivates further investigation.

The same values can be verified programmatically:

from math import sqrt
from statistics import mean, median, mode, stdev, variance

errors = [4, 6, 6, 8, 10, 14]
complexity = [1, 2, 3, 4, 5, 6]

error_mean = mean(errors)
error_sd = stdev(errors)
standard_error = error_sd / sqrt(len(errors))

covariance = sum(
    (c - mean(complexity)) * (e - error_mean)
    for c, e in zip(complexity, errors)
) / (len(errors) - 1)
correlation = covariance / (stdev(complexity) * error_sd)

print("mean:", error_mean)
print("median:", median(errors))
print("mode:", mode(errors))
print("sample variance:", variance(errors))
print("sample standard deviation:", error_sd)
print("standard error:", standard_error)
print("covariance:", covariance)
print("correlation:", correlation)

The hand calculation remains important. Software can produce numbers without establishing that the sample is representative, the observations are independent, or the selected summary answers the intended question.

Resampling: bootstrap and cross-validation

Resampling methods reuse observed data to approximate what repeated sampling would reveal. They are valuable when analytic formulas are unavailable, unreliable, or difficult to derive.

The bootstrap

The nonparametric bootstrap repeatedly draws samples of size $n$ with replacement from the observed dataset. For each bootstrap sample, the statistic of interest is recalculated. The resulting empirical distribution approximates the estimator's sampling distribution.

For example, to estimate uncertainty in a median:

1. Draw $n$ observations with replacement from the original $n$ observations.
2. Compute the bootstrap sample's median.
3. Repeat the process many times.
4. Examine the distribution of bootstrap medians and construct an appropriate interval.

Because sampling uses replacement, a bootstrap sample typically contains some original observations multiple times and omits others. The bootstrap treats the empirical distribution of the observed data as an approximation to the population distribution.

Bootstrap methods can estimate standard errors, bias, and confidence intervals for complicated statistics. They are not magic. If the original sample is biased, the bootstrap reproduces that bias. A basic independent-observation bootstrap is unsuitable for strong time dependence, clusters, or grouped data; block and cluster bootstraps are designed for those structures. Very small samples and extreme-tail statistics can also yield unstable bootstrap results.

Cross-validation

In $k$-fold cross-validation, data are divided into $k$ folds. The model is trained on $k-1$ folds and evaluated on the remaining fold. Each fold serves as validation data once, and the fold results are aggregated.

Cross-validation estimates the performance of a training procedure, not a single already-fitted parameter vector. Each fold fits a different model. It is especially useful for comparing hyperparameters and using limited data more efficiently than a single validation split.

The value of $k$ controls a computational and statistical tradeoff. Larger $k$ trains each model on a greater fraction of the data but requires more fits and can produce highly correlated estimates. Five- or ten-fold cross-validation is common, but the split strategy matters more than choosing between those conventional values.

Classification folds are often stratified to maintain approximate class proportions. Group cross-validation keeps related observations together. Time-series validation preserves temporal order. Nested cross-validation uses an inner loop for model selection and an outer loop for evaluation, reducing the optimism caused by tuning and evaluating on the same folds.

Explore fold construction and validation behavior with the Cross-Validation Calculator. The calculator is most useful when paired with a clear definition of the deployment population and the unit that must remain independent.

Bootstrap and cross-validation have different primary purposes. Bootstrap commonly quantifies estimator uncertainty; cross-validation commonly estimates predictive performance and supports model selection. They can be combined, but they should not be treated as interchangeable labels for “repeat the analysis.”

Common statistical mistakes in machine learning

Treating a dataset as the population

A large dataset is still a sample from particular times, places, users, devices, and collection rules. Sampling error may become small while systematic selection bias remains large. Define the target population before interpreting precision.

Confusing standard deviation with standard error

Standard deviation measures variation among observations. Standard error measures variation in an estimate across repeated samples. Dividing the standard deviation by $\sqrt n$ is justified for the mean under suitable conditions, not as a universal uncertainty formula.

Computing before splitting

Feature selection, scaling, imputation, dimensionality reduction, and target encoding can leak information if they use the full dataset before train/test separation. Put learned preprocessing inside the training pipeline and refit it within every resampling fold.

Selecting on the test set

Repeatedly checking the test metric and modifying the model turns test data into validation data. Preserve a final untouched set or obtain fresh confirmation after development.

Assuming rows are independent

Multiple records from one user, nearby time points, augmented copies of an image, or measurements from one device are correlated. Random row splitting can place nearly identical information in training and evaluation data, producing falsely optimistic scores.

Equating correlation with causation

Predictive association can be useful without being causal, but causal language requires stronger design and assumptions. Intervening on a correlated feature may not produce the predicted outcome change.

Optimizing a convenient metric

An easy metric may not represent the deployment cost. Class imbalance, asymmetric errors, probability calibration, ranking requirements, and subgroup behavior all affect metric choice.

Ignoring multiplicity and researcher choices

Trying many models, seeds, subsets, and metrics creates opportunities for accidental wins. Record the search process, distinguish exploration from confirmation, and validate selected results on independent evidence.

Reporting only an average

An average can conceal instability across folds, time periods, classes, or subgroups. Report dispersion, uncertainty, and relevant slices. Examine the distribution of errors, not only one aggregate.

Believing more decimal places mean more certainty

A score of $0.873421$ is not known to six meaningful decimal places merely because software printed it that way. Match numerical precision to sampling uncertainty and measurement quality.

Practice statistics with Solver360

The Statistics Calculator provides a structured environment for checking descriptive calculations and connecting formulas to data. Begin with a short dataset and calculate the mean, median, mode, variance, and standard deviation by hand. Then use the calculator to verify the results. Change one central observation and observe which summaries move. Next, introduce one extreme value and compare the sensitivity of the mean, median, variance, and standard deviation.

A productive sequence is:

1. Create a symmetric dataset and compare its mean and median.
2. Add an outlier and recalculate both centers.
3. Duplicate every observation and notice that the descriptive distribution is unchanged, even though a naive standard-error calculation appears smaller if duplicates are falsely treated as independent data.
4. Construct two variables with a positive linear relationship and inspect covariance and correlation.
5. Replace the relationship with a U-shaped pattern and observe why Pearson correlation can miss dependence.
6. Compare two small sets of model scores using both their mean difference and their variability.

Then use the Cross-Validation Calculator to examine how model evaluation changes across folds. Compare a single split with repeated folds, and consider whether stratified, grouped, or time-ordered splitting matches the data-generating process. The purpose of practice is not only to obtain the correct number. It is to learn which assumptions make that number relevant.

A disciplined workflow for statistical evaluation

A defensible machine-learning evaluation can be organized around a few explicit decisions.

First, define the observation unit and target population. State what one row represents and where future cases will come from. Second, specify the outcome and metric before comparing models. Third, reserve independent evaluation data using a split that respects groups, time, and collection structure. Fourth, fit every learned preprocessing step using training data only.

Fifth, report a baseline and the candidate model under the same evaluation design. Sixth, quantify uncertainty in their difference, preferably using paired observations or an appropriate resampling design. Seventh, inspect error distributions and operationally important subgroups. Eighth, separate exploratory tuning from confirmatory evaluation. Finally, document assumptions, exclusions, random seeds, and known distribution shifts.

This workflow does not eliminate uncertainty. It makes uncertainty visible and prevents ordinary sample variation from being mistaken for a durable discovery.

Frequently asked questions

What is the difference between expectation and sample mean?

Expectation is the theoretical probability-weighted average of a random variable under a distribution. The sample mean is an estimator calculated from observed data. Across repeated representative samples, the sample mean may center on the expectation, but any particular sample mean can differ from it.

Why does sample variance use $n-1$?

The sample mean is estimated from the same observations used to calculate deviations. Those deviations must sum to zero, so only $n-1$ can vary freely. Dividing by $n-1$ makes the usual sample variance unbiased for population variance under independent, identically distributed sampling.

Is a larger sample always better?

A larger representative sample generally reduces sampling uncertainty, but it does not automatically reduce measurement error, leakage, confounding, or selection bias. A million systematically unrepresentative records can give a very precise estimate of the wrong population quantity.

Does zero correlation mean two variables are independent?

No. Independence implies zero covariance when the relevant moments exist, but zero covariance or zero Pearson correlation does not generally imply independence. Nonlinear relationships can have zero correlation. Under special distributional assumptions, such as a jointly Gaussian model, zero correlation does imply independence.

What does a 95% confidence interval mean?

It describes the long-run coverage of an interval-building procedure: approximately 95% of intervals constructed by that procedure would contain the fixed true parameter under the assumptions. It is often informally understood as a range of values compatible with the data, but it is not a statement that the fixed parameter has a 95% frequentist probability of lying in the observed interval.

Is $p<0.05$ proof that an effect is real?

No. It indicates that the observed result would be relatively unusual under the null hypothesis and test assumptions. It does not measure effect size, practical value, replication probability, or the probability that the hypothesis is true. Multiple testing and flexible analysis can also make small p-values easier to obtain.

How many cross-validation folds should I use?

Five or ten folds are common practical choices, but there is no universally best number. Dataset size, training cost, estimator stability, and dependence structure matter. Selecting grouped, stratified, or time-aware folds correctly is generally more important than a small change in $k$.

When should I use the bootstrap instead of cross-validation?

Use bootstrap methods primarily when estimating the uncertainty, bias, or sampling distribution of a statistic. Use cross-validation primarily when estimating out-of-sample predictive performance or selecting model settings. The appropriate resampling unit must preserve clusters, temporal dependence, and other structure in either case.

Can a statistically significant model improvement be useless?

Yes. With enough observations, an extremely small performance difference can be statistically detectable while having negligible economic or operational value. Evaluate effect size, uncertainty, decision costs, latency, maintainability, and subgroup consequences alongside significance.

Why can test performance change after deployment?

The test set may be small, unrepresentative, or collected under different conditions. User behavior, sensors, policies, competitors, and prevalence can change over time. Test performance estimates future behavior only to the extent that the test distribution represents deployment and the system itself does not alter that distribution.

Where to go next

Statistics provides the bridge from observed data to general claims, but two neighboring subjects deepen the picture. Continue with the linear regression article to see how expectation, variance, covariance, residuals, and uncertainty come together in a concrete predictive model. Then study the probability article for a fuller treatment of random variables, conditional probability, Bayes' rule, and common distributions—the mathematical language underlying statistical inference.

As you continue, retain one central principle: every reported model result is conditional on data, assumptions, and an evaluation design. Strong machine learning does not hide those conditions behind a single score. It states them clearly, measures relevant uncertainty, and treats conclusions as evidence about a population rather than facts guaranteed by a sample.