What it is
A multimodal model works with more than one kind of data, such as text and images (and sometimes audio and video), in a shared space. CLIP is the classic example: it learns from pairs of images and captions so that a picture and a sentence describing it end up close together, which lets it match images to words it was never specifically trained on.
What it is for
Describing a photo, answering questions about a chart or a screenshot, searching images with a text query, reading documents with figures, and giving an assistant eyes and ears. It is the reason a chat assistant can accept a photo of your homework or a diagram.
How to do it
- Pick a model that accepts the kind of input you have (image, audio, video) and check its limits.
- Give the input together with a precise question: "what is the trend in this chart?" works better than "describe this".
- For search, use an image-text embedding model: embed your images once, embed the text query, return the closest images.
- Check the answer against the input yourself, since models can invent details that are not in the image or misread small text.
- Mind privacy: photos can reveal faces, places and documents.
- For critical uses, such as medical or legal images, treat the model as a helper and keep a qualified human responsible.