Multimodal

For a long time the chat box only accepted prose. You described the broken layout in words. You transcribed the whiteboard. You hoped the model imagined the right screenshot.

Multimodal means the same model can take more than one kind of input — text, images, sometimes audio or video — and sometimes produce more than text back. "Modal" is just mode. One mode was typing. Add pixels or sound and you are multi.

Under the hood it is still next-token prediction. Some of those tokens now stand for a patch of the screenshot, a slice of the waveform, a frame. Inference still runs. The bill still meters it. A dense screenshot can cost more than the paragraph you would have typed.

That is the practical unlock at a hackathon. The judge's bug is a toast you cannot spell from memory. The design note is a photo of a napkin. The invoice is a phone picture. You stop writing a novel about the image and you attach the image. An agent that can see the error state wastes fewer turns asking you to paste the stack trace you already have on screen.

Computer use is the next step: not only seeing the desktop, but clicking it. Seeing is still cheaper and safer than a hand on the mouse. Prefer a real API when one exists. Hands are for software that never shipped a window.

It is still a pattern machine. A screenshot of a bug is not a repro. A photo of a whiteboard is not a spec. The model will describe what it thinks it sees, including text it half-reads. Verify the number on the receipt. Verify the hex on the button.

What this unlocks

You design the product around the artifacts people actually have — dumps, clips, photos — not around a clean markdown file they will never write. Text is still the cheap channel. Pictures are the honest one.