K-Fold Cross Validation Explained: Stratified Splits, Bias, and Model Selection

K-fold cross-validation estimates how a model will behave on unseen data by rotating which slice of the training set is held out. A single 80/20 split is noisy: one unlucky test slice can make a good model look bad, or hide overfitting. Cross-validation averages several such slices so that model choices — algorithm, features, hyperparameters — are based on more than one draw of luck.

This guide explains k-fold and stratified k-fold, nested CV for tuning, time-series splits, and the bias-variance trade-off in the choice of $k$.

Train, validation, and test

Keep three conceptual roles:

• Training data: fit parameters.
• Validation data: choose models and hyperparameters.
• Test data: a single final report, not used for decisions.

Cross-validation is a way to manufacture validation scores when you cannot afford a huge dedicated validation set. You still want a true test set that you touch once.

How k-fold works

Partition the training set into $k$ folds of roughly equal size. For each $i=1,\ldots,k$:

1. Train on all folds except $i$.
2. Evaluate on fold $i$.

Average the $k$ scores. Every example is predicted exactly once as a validation point. Typical $k$ is 5 or 10.

$$\mathrm{CV}=\frac{1}{k}\sum_{i=1}^k L\big(h^{-i}, D_i\big).$$

Larger $k$ means more training data per fold (lower bias of the estimate) and more compute. $k=n$ is leave-one-out CV (LOOCV): nearly unbiased, high variance, expensive.

The Cross Validation Calculator is useful for seeing how fold assignments change estimated error.

Stratified k-fold

In classification, a random fold can accidentally contain almost no minority-class examples. Stratified k-fold preserves class proportions in each fold. Always prefer it for imbalanced labels.

For regression, stratification on binned $y$ is sometimes used; it is less standard.

Grouped and time-aware splits

If rows are not independent — several visits per patient, several clicks per user — random k-fold leaks. Use GroupKFold so that all rows of a group sit in the same fold.

If rows are ordered in time, shuffling leaks the future. Use TimeSeriesSplit (walk-forward): train on the past, validate on the next block, roll forward. A random 5-fold on stock bars will flatter any model that peeked.

Nested cross-validation

The most common CV mistake is this: search hyperparameters with CV, then report the best CV score as generalization. That score is optimistically biased because the search peeked at all folds.

Nested CV uses an outer loop to estimate generalization and an inner loop to tune:

• Outer fold $i$ held out.
• Inner CV on the remaining data selects $C$, depth, $k$, …
• Refit the winner on the outer training data, score the outer test fold.

It is slower and honest. If you cannot afford it, keep a locked test set and only CV on the rest.

Hyperparameter search belongs with nested or at least inner validation — see hyperparameter tuning.

Bias, variance, and the meaning of the CV number

CV estimates the performance of the fitting procedure, not of one frozen model, unless you refit on all data afterward (the usual deployment step). The number you quote should name the metric (log loss, MAE, AUC), the split scheme, and $k$.

Comparing two models with CV: use the same folds (paired comparison). Otherwise fold luck confounds the difference.

Frequently asked questions

Is 10-fold always better than 5-fold?

Not always. 10-fold is a bit less biased and about twice the training work. For large $n$, 5-fold is standard. For tiny $n$, 10-fold or repeated CV reduces luck.

Can I use CV to select features?

Yes, but selection must sit inside the loop. Selecting features on the full dataset, then CV’ing the model, leaks. Pipelines exist so that feature selection is refit per fold.

Why is my CV score much better than production?

Leakage, non-iid data, label shift, or reporting the best of many searches. Nested CV and a time-based holdout catch most of this.

Does cross-validation replace a test set?

It replaces a large validation set. A final test set (or a later production A/B test) is still the only number you should market.

Next steps

Run logistic regression with a single split and with stratified 5-fold on the same table; watch the score move. Put scaling inside the CV pipeline, not outside. Then open the Cross Validation Calculator and pair it with feature engineering and hyperparameter tuning so evaluation, features, and search stay honest.