4. Fighting Overfitting: the Advent of Machine Learning#
Chapter 3 taught us to measure predictive performance honestly, and produced a picture to aim at: a valley, with oversimplification on one side and memorization on the other. This chapter is about steering into it.
The tool is the hyperparameter: a setting that is not learned from the data during fitting, but chosen beforehand, and that controls how much freedom the model has. Adding such a dial to a linear model is, in essence, what turns it into a machine learning model.
And then, in the last section, the second villain arrives.
In this chapter:
Reducing Complexity: a Brief Theory — feature selection, PCA, PLS, Ridge and LASSO, and why standardization is not optional.
Reducing Complexity in Action — all five applied to the IXI data, and scikit-learn pipelines.
The Second Villain: Leakage — how information about the test data reaches your model without you noticing, and what it does to your results.