Feature Engineering Explained: Scaling, Encoding, and Polynomial Features

Feature engineering is the work of turning raw columns into inputs a model can actually use: scaling numbers, encoding categories, building interactions, handling missingness, and avoiding leakage. On tabular problems it often moves metrics more than switching from logistic regression to XGBoost. Algorithms assume geometry and types; your job is to give them a geometry that matches the task.

This feature engineering guide covers standardization versus normalization, one-hot and ordinal encoding, polynomial features, binning, and the leakage mistakes that inflate test scores.

Scaling: standardization versus normalization

Many models care about units. k-NN, SVM with RBF, k-means, PCA, and gradient descent all behave badly when one feature is in millions and another is in units.

Standardization (z-score):

$$z=\frac{x-\mu}{\sigma}.$$

Use it as the default for linear models, SVMs, k-NN, and neural nets.

Min-max normalization maps to $[0,1]$:

$$x'=\frac{x-x_{\min}}{x_{\max}-x_{\min}}.$$

It preserves sparsity better for some non-negative data but is sensitive to outliers. Robust scaling uses the median and IQR when outliers dominate.

Tree ensembles (random forest, XGBoost) are invariant to monotone scaling of a single feature. You can still scale for mixed pipelines, interpretability, or regularization that depends on coefficient size.

Fit scalers on the training fold only, then apply to validation and test. Fitting on all rows leaks distribution information. The Feature Engineering Calculator is built to show these transforms before you paste sklearn code into a pipeline.

Encoding categorical variables

One-hot encoding creates a binary column per level (usually dropping one level in linear models to avoid the dummy trap). It is the safe default when cardinality is low.

Ordinal encoding assigns integers. Use it only when the order is real (education levels, Likert scales). Encoding “Paris, Tokyo, Nairobi” as 1, 2, 3 tells k-NN that Tokyo is between the others.

Target encoding replaces a category with a smoothed mean of $y$. It is powerful and leakage-prone. Use nested cross-validation or library implementations with proper smoothing (CatBoost-style ordered encoding).

High-cardinality IDs (user_id, zip with thousands of rare levels) often should be dropped, hashed, or grouped as “other,” not one-hot into a 50,000-column matrix.

Polynomial features and interactions

Linear models become curved if you expand $x$ into $x,x^2,x^3$ or add products $x_1x_2$. That is still linear in coefficients — see the linear regression article. Degree 2 on 20 features explodes the column count. Prefer explicit domain interactions (price × discount) over a blind polynomial explosion, then regularize.

Missing values are a feature

Mean-impute only after understanding why values are missing. Create a missingness indicator when “blank” is informative (skipped credit field). Impute inside each CV fold. Trees can split on missing; linear models cannot without a fill rule.

Leakage: the silent accuracy killer

Leakage means the model saw information that would not be available at prediction time.

Examples:

• Scaling or PCA fit on train+test together.
• Target encoding computed on the full dataset.
• Using “days until churn” as a feature to predict churn.
• Random CV on time series instead of a forward split.
• Duplicate customers in both train and test.

If a metric looks too good, assume leakage first. The cross-validation guide is the companion to this article.

A minimal pipeline

1. Split data (grouped or time-aware if needed).
2. Define preprocessing in a pipeline: impute → encode → scale → model.
3. Fit the pipeline on training folds only.
4. Evaluate on a held-out test set once.
5. Inspect transformed columns for impossible values and exploded dimensionality.

sklearn’s ColumnTransformer exists so you do not scale one-hot dummies by accident — or forget to scale the numeric block that k-NN needs.

Frequently asked questions

Do I always need feature scaling?

No. Pure tree methods usually do not. Distance-based and gradient-based methods usually do. When in doubt, scale inside a pipeline; it rarely hurts trees and often saves SVMs.

One-hot or label encoding for XGBoost?

XGBoost can split on integer-coded categories, but that imposes a fake order. Native categorical support (or CatBoost) is cleaner. One-hot is fine at low cardinality.

Are polynomial features the same as a neural net?

No. Polynomials are a fixed expansion. A neural net learns nonlinear features from data. Polynomials are interpretable and easy to regularize on small $n$.

Why did scaling make my coefficients change?

They are now in standard-deviation units. That is usually what you want for comparing feature strength in a linear model, with the usual causal caveats.

Next steps

Take a table with age, income, and a city column. Build a pipeline: median impute, one-hot city, standardize numerics, logistic regression. Then skip scaling and watch k-NN collapse. Use the Feature Engineering Calculator to compare scaling and encoding, then lock the process down with k-fold CV.