Hyperparameter Tuning Explained: Grid Search, Random Search, and Bayesian Optimization

Hyperparameter tuning is the search for settings that are not learned by gradient descent or least squares: the learning rate, tree depth, $k$ in k-NN, $C$ in an SVM, dropout, number of boosting rounds. Parameters (weights, splits) are fit given those choices. If you pick hyperparameters on the test set, the test set is no longer a test.

This guide distinguishes parameters from hyperparameters, compares grid search, random search, and Bayesian optimization, and explains nested validation so the search does not leak.

Parameters versus hyperparameters

A linear model’s $\boldsymbol{\beta}$ is a parameter. Its L2 penalty $\lambda$ is a hyperparameter. XGBoost leaf weights are parameters; max_depth is a hyperparameter.

The Hyperparameter Tuning Calculator is a sandbox for grid versus random search thinking before you burn CPU on a large job.

Grid search

Grid search evaluates the Cartesian product of listed values:

$$C\in\{0.1,1,10\},\quad \gamma\in\{0.01,0.1,1\} \quad\Rightarrow\quad 9\text{ models}.$$

It is simple and reproducible. It wastes time when some hyperparameters barely matter: you still pay for every combination. In high-dimensional search spaces, most grid points are redundant.

Use grids on log scales for $C$, $\lambda$, $\eta$, $\gamma$. Linear grids from 1 to 100 miss the interesting region near $10^{-3}$.

Random search

Random search draws combinations from distributions (log-uniform is typical for scale parameters). Bergstra and Bengio showed that if only a few hyperparameters actually matter, random search finds good regions faster than a grid of the same budget.

Budget $N$ trials. You can stop early. You can add extra samples later without redesigning a grid.

Random is the right default when you have more than two or three knobs.

Bayesian optimization

Bayesian optimization (Gaussian processes, TPE, SMAC) fits a cheap surrogate of the validation score and picks the next trial by an acquisition function (expected improvement, UCB). It shines when each trial is expensive (big neural nets) and the space is continuous-ish.

It can get stuck, it has its own knobs, and it is overkill for a 30-second logistic regression. For XGBoost on a medium table, Optuna/TPE is often a good middle ground.

The evaluation protocol matters more than the optimizer

No search method can save a leaked pipeline.

Rules:

1. Split off a final test set.
2. Tune only with cross-validation (or a dedicated validation set) on the remainder.
3. Put preprocessing inside the search pipeline.
4. Prefer nested CV if you will report the tuned CV score.
5. Use early stopping inside each trial for boosting/neural nets, with a inner validation fold — do not peek at the outer test fold.
6. Fix random seeds for reproducibility, but do not tune the seed.

If you search 200 configurations and quote the best CV AUC as “the” performance, you have multiple-comparison bias. Nested CV or a locked test set is the correction.

What to search first (practical order)

For logistic regression / linear SVM: $C$ (log grid).
For RBF SVM: $C$ and $\gamma$.
For k-NN: $k$ and the metric; scaling on/off is a pipeline choice.
For XGBoost: $\eta$, max_depth, min_child_weight, subsample; n_estimators via early stopping.
For neural nets: learning rate first, then weight decay, then architecture.

Do not start by tuning everything at once. Lock a sensible default, search the one or two knobs that usually dominate, then expand.

Frequently asked questions

Is grid search outdated?

No. It is fine for one or two log-spaced hyperparameters. It becomes inefficient as dimension grows. Random or Bayesian methods then win on a fixed trial budget.

Can I tune on accuracy?

You can, but accuracy is a poor objective on imbalanced data. Tune on log loss, AUC, F1, or a business cost. The search will obey whatever you optimize — including a silly metric.

How many trials do I need?

Enough that the validation curve flattens. For two log parameters, a 5×5 grid (25) may suffice. For eight parameters, 30–80 random/TPE trials often beat a coarse grid. More data and more compute allow more.

Should I include architecture in the search?

Treat layer size as a hyperparameter only if you can afford it. Search learning rate on a fixed architecture first. Neural architecture search is a different, heavier sport.

Next steps

Pick one model you already understand — logistic regression or XGBoost. Search a single hyperparameter with 5-fold CV, then add a second. Compare grid versus random at the same number of fits. Use the Hyperparameter Tuning Calculator and keep k-fold CV as the scoring engine so ranking improvements are real.