What it is
Statistical learning treats learning as reasoning under uncertainty: the model is a description of how the data could have been produced, with unknown numbers (parameters) that we estimate from examples. When some information is hidden, such as the group each example belongs to, the Expectation-Maximisation (EM) algorithm alternates between guessing the hidden part and improving the parameters.
What it is for
When you want probabilities and not just answers, when data is scarce, or when some variables are never observed: grouping data into overlapping clusters, filling in missing values, and many classic models of speech and text. It also explains where common training objectives come from.
How to do it
- Write down a story of how the data is generated, for example "each point comes from one of three bell-shaped groups".
- List the unknown numbers in the story (the centres and widths of the groups, how common each group is).
- Pick starting guesses, then repeat EM: estimate how likely each point is to belong to each group, then update the numbers using those estimates.
- Stop when the fit stops improving, and try several starting points because EM can get stuck.
- Check that the story fits: compare what the model would generate with the real data.
- Use the fitted model to give probabilities for new cases, and keep the uncertainty in the decision.