Controlling Model Complexity for Generalization

David I. Inouye

Better training fit can mean worse prediction on new data

We have learned how to minimize a training objective.

The ML goal is still performance on new examples.

Regularization limits how freely a model can fit the training data, trading some flexibility for potentially better generalization.

Recall KNN: a tiny neighborhood can follow sample-specific fluctuations.

Connect back to Arc 3 rather than introduce regularization as an optimizer trick. Regularization is broader than adding a penalty. It can improve prediction when the restriction suppresses irrelevant variation, but excessive restriction underfits. No universal guarantee of improved test performance.

Larger KNN neighborhoods and coefficient penalties both control flexibility

Mechanism	Weaker restriction	Stronger restriction
KNN neighborhood size \(k\)	Small \(k\): follow nearby examples closely	Large \(k\): average over more examples
Coefficient penalty strength \(\lambda\)	Small \(\lambda\): allow larger coefficients	Large \(\lambda\): discourage large coefficients

For a linear model, one common choice is L2 regularization:

\[J_\lambda(\boldsymbol{w},b)=J_{\mathrm{train}}(\boldsymbol{w},b)+\frac\lambda2\|\boldsymbol{w}\|^2.\]

Different mechanisms, same purpose: control sensitivity to the training sample.

Too much restriction can hide useful signal. \(k\) and \(\lambda\) have no numerical equivalence.

source:boyd-vandenberghe-convex-optimization. Here the intercept is unpenalized and features must be scaled sensibly. KNN is implicit smoothing; L2 is an explicit penalty. Neither changes the underlying objective of ML: useful held-out prediction. The penalty changes the fitted solution, unlike learning rate alpha, which controls the numerical route to a fixed objective. Penalties can also affect conditioning or ensure finite solutions, but these are secondary effects here.

Both mechanisms can improve held-out prediction

Ames KNN using living area only — validation RMSE:

\(k=1\)	\(k=100\)	\(k=1758\) (all training sales)
$75,300	$57,700	$80,200

Ames linear models using 226 encoded features — test RMSE:

No penalty: OLS	L2 penalty: validation-selected ridge
$26,300	$25,600

Compare within each row: these use different representations and evaluation splits.

The restriction can help; the strongest restriction need not be best.

source:ames-housing-data; source:ames-housing-documentation. Existing fixed seed570, 1758/586/586 split. KNN numbers from ames-knn-complexity.json: 75304.58, 57676.68, 80158.28; k=100 was selected from the original candidate grid using validation. This row illustrates tuning, not an unbiased final test estimate. Ridge versus OLS from ames-regression.json: test26279.26 versus25566.97, validation-selected sklearn Ridge alpha10 with train-only preprocessing. Library alpha uses a sum-loss convention and is not numerically the lambda in our mean-loss equation. No across-row ranking or universal superiority claim. The real Cornell classifier also used an L2 penalty; today’s unregularized examples isolated optimization behavior.

Choose the restriction with validation; assess the choice on test data

1. Fit: train each candidate using only training observations.
2. Choose: compare candidates on validation data.
3. Assess: evaluate the selected procedure on untouched test data.

Search positive \(\lambda\) values on a log scale: multiply, rather than add.

Search	Example \(\lambda\) values
Broad search: factors of 10	\(10^{-4},10^{-3},10^{-2},10^{-1},1,10,100\)
Refine near a promising \(\lambda=1\)	\(0.1,0.3,1,3,10\)

Common mistake: \(0,1,2,3,\ldots\) misses the entire range \(0<\lambda<1\).

Include \(\lambda=0\) separately as the no-penalty baseline; zero has no logarithm.

Revisit the existing split protocol briefly. Budget about three minutes, including the tuning example. Positive penalty strengths often need exploration across orders of magnitude: equal ratios cover relative scales that equal increments can miss. The displayed grids are illustrative, not universal bounds. Standardize features using training statistics; the appropriate lambda range depends on feature scaling and the loss/penalty normalization. If the best validation result is at a grid boundary, expand in that direction; otherwise refine around the promising region. The second grid is approximately logarithmically spaced (factors near three) and is an example of refinement, not evidence that lambda=1 is optimal. Check lambda=0 separately when an unregularized fit is well-defined. Choose the entire search using validation data and inspect test performance only after selection.

Learning rate controls how we approach a chosen objective; regularization changes which fitted solution we prefer. Optimization should be sufficiently accurate for a fair comparison; an under-converged candidate can confuse an optimization failure with a poor regularization choice. Validation-guided early stopping can itself act as regularization, but defer its details. Improving test performance is a measured outcome, not something every penalty guarantees.