# Multimodal (/academy/multimodal)



For a long time the chat box only accepted prose. You described the broken layout in words. You transcribed the whiteboard. You hoped the model imagined the right screenshot.

**Multimodal** means the same model can take more than one kind of input — text, images, sometimes audio or video — and sometimes produce more than text back. "Modal" is just mode. One mode was typing. Add pixels or sound and you are multi.

Under the hood it is still next-[token](/academy/token) prediction. Some of those tokens now stand for a patch of the screenshot, a slice of the waveform, a frame. [Inference](/academy/inference) still runs. The bill still meters it. A dense screenshot can cost more than the paragraph you would have typed.

That is the practical unlock at a hackathon. The judge's bug is a toast you cannot spell from memory. The design note is a photo of a napkin. The invoice is a phone picture. You stop writing a novel about the image and you attach the image. An [agent](/academy/agent) that can *see* the error state wastes fewer turns asking you to paste the stack trace you already have on screen.

[Computer use](/academy/computer-use) is the next step: not only seeing the desktop, but clicking it. Seeing is still cheaper and safer than a hand on the mouse. Prefer a real [API](/academy/api) when one exists. Hands are for software that never shipped a window.

It is still a pattern machine. A screenshot of a bug is not a repro. A photo of a whiteboard is not a spec. The model will describe what it thinks it sees, including text it half-reads. Verify the number on the receipt. Verify the hex on the button.

**What this unlocks**

You design the product around the artifacts people actually have — dumps, clips, photos — not around a clean markdown file they will never write. Text is still the cheap channel. Pictures are the honest one.