Capabilities
Multimodal AI
Also known as: multimodal model
AI that can work with more than one type of content at once, for example understanding text, images, audio and video together.
Multimodal AI is AI that can handle several kinds of input and output, not just text. You can show it a photo and ask a question about it, have it describe a chart, listen to speech, or generate an image from a description. Where earlier AI dealt with one thing at a time, a multimodal model can mix them, much like people combine what they see, hear and read.
"Modality" means a type of data: text, image, audio, video. A multimodal model can take in and/or produce more than one modality. Modern assistants are increasingly multimodal: you can paste a screenshot and ask about it, or ask for a picture from a written prompt.
Common combinations
- Vision + language: describe an image, read a document photo, answer questions about a diagram.
- Speech: talk to the assistant and hear it reply.
- Generation: text-to-image, text-to-speech, text-to-video.
The trick that makes this possible is representing different data types in a shared "meaning space" of embeddings, so the model can relate a picture of a dog to the word "dog".
Multimodal models learn a joint representation across modalities, typically by encoding each modality into a shared embedding space or by projecting non-text inputs into the token stream of a transformer. Contrastive pretraining on paired data (e.g. image-caption pairs in CLIP; Radford et al. 2021) aligns modalities so semantically related items are geometrically close; generative multimodal systems extend this with decoders for images (diffusion), audio or video.
Architecturally, many current "multimodal LLMs" attach modality-specific encoders (a vision encoder, an audio encoder) whose outputs are adapted into the LLM's embedding space, letting one autoregressive backbone reason over interleaved text and non-text tokens. Key challenges are cross-modal alignment quality, the scarcity and bias of paired training data, tokenising continuous signals efficiently, and evaluation, since faithfulness must be judged across modalities, not just in text.