Sequence-to-sequence models

What it is

A sequence-to-sequence model reads one sequence and writes another, possibly of a different length. An encoder reads the input and squeezes it into a representation; a decoder then writes the output one piece at a time from that representation. Translation is the classic example: a sentence in Italian in, a sentence in English out.

What it is for

Any task that maps text to text: translation, summarising, rewriting, answering questions, turning speech into text. It is also the shape that led directly to attention and the transformer, which fixed its weak point, the single squeezed representation of a long input.

How to do it

The process, step by step. Libraries and tools change; these steps stay.

1. Define the pair: what sequence goes in, what sequence should come out.
2. Collect many example pairs (for translation, sentences with their translations) and split them for training and testing.
3. Pick an encoder-decoder model, today normally a transformer, rather than building a recurrent one from scratch.
4. Train it to produce the correct output piece by piece, or start from a pre-trained model and fine-tune it on your pairs.
5. Generate: encode the input, then let the decoder write the output, choosing at each step between the most likely piece and more varied options.
6. Evaluate with a metric suited to the task (for translation, BLEU-like scores) and with a human reading, because scores miss meaning.