Inference

Training already happened. You were not in the room.

Someone spent weeks and a warehouse of chips teaching a model to predict the next token. That bill is not on your card. Inference is the live pass: you send a prompt, GPUs walk those tokens, new tokens come back. Hit send in Cursor. Call an API. Let an agent take a step. That is inference. You pay for it. You wait for it. "The model is down" means nobody is running that pass.

This is why the words stream. You are watching tokens land one at a time, not a paragraph appearing from a drawer. A longer answer is a longer pass. A fatter prompt is a fatter pass. That is also why latency feels like "thinking" even when the model is not doing extra reasoning — it is just cooking tonight's plate.

Once you see the live pass, a few product choices stop being vibes:

• Loops have a price. "Call this a thousand times" is a thousand plates. Demo-night agents that retry forever are a surprise invoice.
• Prompt caching is leftover prep. The stable prefix gets paid once, then reused.
• A rate limit is the kitchen capping how many tickets it will take this minute.
• Reasoning models spend extra inference on a private draft before the sentence you asked for. Same kitchen, longer ticket.

The runtime on your laptop is not the same bill. Local models still do inference — they just burn your fan instead of your API key.

What this unlocks

You budget the thing you actually buy. Not "AI" as a lump. Turns, tokens, retries, tool calls. A hackathon agent that looks smart in one chat can go broke in a loop. Design the loop like it has a meter.