Pretraining & transfer learning

What it is

Pretraining means training a large model first on a huge general collection of text, with a simple goal such as predicting the next word or filling in hidden words, so that it picks up grammar, facts and patterns of reasoning. The result can then be reused for many specific tasks, which is transfer learning applied to language.

What it is for

It is why one model can translate, summarise and answer questions without being built separately for each. For you it means you almost never start from nothing: you take a pre-trained model and adapt it, which needs far less data and computing power.

How to do it

The process, step by step. Libraries and tools change; these steps stay.

1. Understand the two stages: expensive pretraining on general text, done once by a lab, then cheap adaptation to your task, done by you.
2. Choose a pre-trained model whose language, size and licence fit your project.
3. Try it as it is, with a good prompt, on a sample of your task and measure the result.
4. If that is not enough, collect examples of the task and fine-tune the model on them, or add retrieval of your documents.
5. Hold out test examples the model never saw, and check that your test material was not already in its pretraining text.
6. Keep in mind what pretraining cannot give: knowledge after its training cut-off date and facts it never saw.