News
Model tiering, caching and loop caps: keeping the bill predictable
The same model can cost a hundred dollars a session or under a dollar. The difference is four engineering decisions.
Production notes
Frontal Designs
Where the money goes
Input tokens dominate the bill, not output. An agent resends its context on every loop, and the context is usually far bigger than it needs to be. Uber exhausted its annual AI budget by April this year. Most of that was context, not answers.
Context size: load the two relevant paragraphs, not the whole policy manual.
Model tiering: the easy 80% on a small cheap model, escalate the rest.
Loop caps: an agent that can retry forever will, at your expense.
Model tiering in practice
The three lists tell you which tier to use. Unattended actions with high confidence run on the cheap model. Propose-only actions can afford the expensive one because a person is about to read the output. This alone usually halves the bill and nobody notices the difference in quality, because the evaluation set proves there is none.
Caching and caps
Cache the system prompt and the tool definitions. Cap the number of steps per task and the number of retries per step. Log cost per task from day one so you can see which process is expensive before the invoice tells you.
Why this matters for the business case
A well-scoped agent at hundreds to low thousands of tasks a day should run for 500 to 3,000 dollars a month including tokens. If the estimate you have been given is far above that, the build is loading too much context or looping too freely. If it is far below, someone has forgotten the person who owns the exception queue.
We put a cost per task on the weekly report next to the two error rates. Three numbers, every week, is enough to run an agent for years.
Blog & Insight
Read More Notes
89% of AI agent pilots never reach production. What the other 11% did differently.
