What it is
Scaling laws are regular patterns found by measuring many training runs: the error of a language model falls smoothly, following a power law, as you increase the model size, the amount of data and the computing used for training. Kaplan and colleagues showed this for language models, which let labs predict how good a bigger model would be before building it.
What it is for
Planning how to spend a training budget (bigger model or more data), explaining why labs keep building larger models, and judging claims about "the next model". They describe average loss, not every skill, and they say nothing about whether a skill will be safe or useful.
How to do it
- Pick a family of models that differ only in size, and train small versions on different amounts of data.
- Measure the loss on held-out text for each run and plot it against size, data and compute on logarithmic axes.
- Fit a straight line (a power law) to the points; the slope tells how fast it improves.
- Use the line to predict the loss of a bigger run, and decide how to split the budget between model size and data.
- Check the prediction with a bigger run, and watch for where the pattern bends or breaks.
- When reading news about a new model, ask what was scaled, what was measured, and whether the claim is about the loss or about a real task.