5. Hyperparameter Optimization and Nested Cross-validation

Chapter 4 gave us dials to control model complexity, and then showed that turning them by looking at cross-validated performance quietly inflates the result.

This chapter closes the loop. The answer, both times, is the same move: if a decision must not be influenced by the data you will be judged on, make that decision inside a cross-validation of its own.

By the end of it you will have a complete, honest recipe for building and reporting a predictive model.

In this chapter:

• Cross-validation Revisited — feature selection inside the loop, and pipelines as leakage prevention by construction.

• Nested Cross-validation — training, validation and test sets; GridSearchCV; how much the nesting actually changes; and the recipe.