Generalization

What it is

Generalization is how well a model does on new data it has never seen, not on the examples it trained on. A model that only memorises its training data is overfitting. Why very large networks still generalize well is not fully understood (Zhang et al., 2016).

What it is for

It is the real goal of machine learning: a model is useful only if it works on tomorrow's data. Checking it is the first thing to do before trusting any result.

How to do it

The process, step by step. Libraries and tools change; these steps stay.

1. Split your data into training, validation and test sets before doing anything else.
2. Train on the training set. Use the validation set to choose settings and to decide when to stop.
3. Keep the test set for one final measurement.
4. If the training score is high and the test score is low, you are overfitting: get more or more varied data, simplify the model, or add regularization (dropout, weight decay, augmentation).
5. If both scores are low, the model is too simple or the data does not contain the signal.