What it is
Attention lets a model decide, for each part of its output, which parts of the input to look at most. Instead of squeezing a whole sentence into one fixed-size summary, it keeps every part and gives each one a weight.
What it is for
Translation, summarising, answering questions, image models: anywhere the useful part of the input changes from step to step. It is the core of the transformer, and so of modern language models.
How to do it
- Turn every input item into a vector.
- For the item you are working on, compare it with all the others to get a relevance score for each.
- Turn the scores into weights that add up to 1 (a softmax).
- Take the weighted average of the items: that is what the model attends to.
- Pass the result to the next layer. Transformers repeat this in parallel in several heads, so the model can follow different relations at once.