What it is
Ensemble learning combines many models so that together they make better predictions than any one of them. Bagging trains many models on different random samples of the data and averages them (random forests do this with decision trees). Boosting trains models one after another, each one concentrating on the mistakes of the previous ones.
What it is for
Tables of data (rows and columns, like spreadsheets), where forests and boosted trees are often among the strongest methods and need little tuning. Also a way to get more stable predictions when a single model is too sensitive to the exact data it saw.
How to do it
- Start with a table of features and a label, and a single decision tree as a baseline.
- Try a random forest: it trains many trees on random subsets of rows and features and averages their votes.
- Try a boosted-trees library (for example gradient boosting in scikit-learn) and compare on the same validation data.
- Tune only a few settings: number of trees, tree depth, learning rate for boosting.
- Check the result on the untouched test set, and look at which features the ensemble relied on to catch suspicious shortcuts.
- Remember the cost: an ensemble is harder to explain and slower than one tree, so keep the simple model if it is good enough.