What it is
The transformer is a neural network architecture built around attention. Instead of reading a text word by word, each position looks at all the other positions at once and decides which ones matter for it. Stacking many such layers gives a network that can be trained in parallel on huge amounts of text. It is the basis of almost all modern language models.
What it is for
Language models and chat assistants, translation, code generation, and, with small changes, images, audio and protein structures. If you work with modern AI you will meet it almost everywhere, so it pays to know its parts: tokens, embeddings, attention, feed-forward layers.
How to do it
- Cut the input into tokens and turn each token into a vector (an embedding), adding information about its position.
- Pass the vectors through a stack of identical blocks; in each block attention mixes information between positions and a small feed-forward network processes each position.
- For text generation, mask the future so each position only sees what came before it, and predict the next token.
- Train on a very large amount of text, which needs many specialised chips; this is why most people reuse an existing pre-trained model.
- Adapt the pre-trained model to your task with prompting or fine-tuning (see Pretraining and Fine-tuning).
- To experiment, load a small open model with a standard library and inspect its attention and outputs.