Cost, speed, and quality.

Key terms

Input tokens

Everything the model reads for one call: instructions, tool descriptions, the conversation so far, added facts, and tool results.

Output tokens

The tokens the model writes in reply. Usually priced higher per token than input.

Prompt caching

A provider feature that stores the start of a prompt so later calls that begin with exactly the same text are processed faster and billed at a lower rate.

Batch processing

Sending many non-urgent requests together, to be processed later instead of right away, usually at a discount. At Anthropic, most batches finish within an hour, and any request not done within 24 hours expires.

Latency

The time between sending a prompt and receiving the model’s output.

Time to first token (TTFT)

The time from sending the prompt until the model produces the first token of its answer.

AI gateway

A layer between your apps and the models they call, where access, routing, limits, caching, logs, and spending are managed in one place.

The idea

When a feature is new, nobody looks at the bill. Then it works, usage grows, and someone in finance asks what one conversation costs. It’s a hard question to answer if nobody planned for it. Or the answers are great, but shoppers leave before the first word appears.

This lesson gives you the numbers to have that conversation early. You’ll see where the cost of an answer comes from, which waits customers feel, and how to trade cost and speed against quality without guessing. Then you’ll meet the AI gateway. It’s not glamorous work. It’s often what decides whether a good AI feature can stay live.

The lesson, in seven parts

1. 1What one answer really costs.

   Lesson 1 said every word in front of the model, and every word it writes, has a price. Providers usually bill per token, and they bill input and output separately. Input is everything the model reads. Output is what it writes.

   Output usually costs more per token. At the time of writing (October 2026), Anthropic’s pricing page lists output at five times the input price for each of its models. Other providers set their own rates.

   Here’s what surprises many people new to agents. An agent doesn’t make one call per question. As lesson 2 showed, it works in a loop, and each turn is a separate model call. Anthropic’s Messages API is stateless: every call sends the full conversation again. So the model re-reads its instructions, its tool descriptions, and everything so far, every time. And Anthropic’s pricing page notes that tool descriptions and the results tools return all count as input tokens.

   Anthropic engineers Erik Schluntz and Barry Zhang put it plainly: “Agentic systems often trade latency and cost for better task performance.” That’s not a reason to avoid agents. It’s a reason to know what each question costs before launch. Subagents (lesson 4) multiply this again, since each runs its own loop.

2. 2Estimate the cost of one conversation.

   You don’t need an engineer for a first estimate. You need four numbers: calls per question, tokens in and out on each call, questions per conversation, and your provider’s current prices. Later, your engineers can pull real token counts from the logs (lesson 7). Here’s the method, with made-up numbers.

   A shopper asks the made-up furniture store’s assistant: “Will the Harbor sofa fit through a 30-inch door, and can it arrive by Saturday?” The instructions and tool descriptions come to 2,900 tokens. To keep the math simple, use made-up prices in “credits”: 1 credit per 1,000 input tokens and 5 credits per 1,000 output tokens.

   Call	What the model does	Input tokens	Output tokens
   1	Reads the question, asks for the doorway-size check	3,000 (2,900 instructions + the question)	100
   2	Reads the size result, asks for the delivery date check	3,500 (call 1 again + its request + the result)	100
   3	Reads the delivery result, writes the answer	3,900 (call 2 again + its request + the result)	300
   Total	One question	10,400 = 10.4 credits	500 = 2.5 credits

   Made-up example. The store, token counts, and credit prices are invented. Swap in your provider’s current prices and your own token counts.

   • One question costs about 12.9 credits. A team that estimated one call per question (instructions, question, and answer) would have guessed about 4.5, a third of the real cost.
   • Input is about four-fifths of the bill, though each output token costs five times as much. Most of it is the same 2,900 tokens of instructions, read three times.
   • With three questions per conversation, each re-reading the ones before, a conversation costs a bit over 40 credits. Multiply by monthly conversations for a budget line finance can check.

3. 3Prompt caching: stop paying full price to re-read the same instructions.

   Look at that table again. The model read the same 2,900 tokens three times, at full price. Prompt caching is built for exactly this. The provider stores the start of your prompt, and later calls that begin with exactly the same text read it from the cache, faster and cheaper.

   The discount depends on the provider and the model, so a rule of thumb like “a cache hit costs about 10%” is too loose to plan with. At the time of writing (October 2026):

   Provider	Reading from the cache	Writing to the cache	Worth knowing
   Anthropic	0.1 times the normal input price on most models; 0.05 or 0.025 times on some of its newest	1.25 times the input price for a 5-minute cache; 2 times for a 1-hour cache	The cache lasts 5 minutes by default and refreshes each time it’s used. A 5-minute cache pays for itself after one read.
   OpenAI	Depends on the model; OpenAI says cached input is discounted by up to 95%	No extra charge on older models; 1.25 times the input price on its newest	Caching is on by default for supported models.

   Ratios from each provider’s pricing and caching pages, October 2026. They change, so check the current pages before you plan.

   Back to the made-up store, with Anthropic’s standard ratios. On a busy day the cache stays warm, so all three calls read the 2,900 instructions at a tenth of the price. Input drops from 10.4 credits to about 2.6, and the question costs about 5.1 credits instead of 12.9. (Made-up numbers; a real system also pays for cache writes now and then.)

   • Order matters. A hit needs the start of the prompt to be exactly the same, so what never changes (instructions, tool descriptions) goes first and the question goes last. Even a timestamp in the cached part means no hits.
   • There’s a minimum. Each provider only caches a prompt start of at least a set length, and it differs by model: at the time of writing (October 2026), anywhere from 512 to 4,096 tokens at Anthropic. Below it, nothing is cached and no error tells you. Check the number for your model.
   • It doesn’t change the answer. Anthropic says the response is identical to what you’d get without caching.
   • It can make answers start sooner: Anthropic says you’ll generally see improved time to first token for long documents.

4. 4Four more levers, and what each one costs you.

   Caching is often the easiest saving. Here are the other main levers. Each has a catch, and the catch is usually quality or speed, which is why the next parts matter.

   Lever	What it means	At the made-up store	The catch
   Choose the model per step	Send simple steps to a smaller, cheaper, faster model	A small model picks which check to run; a larger one writes the answer	The small model may pick wrong. Test each step with evals
   Batch non-urgent work	Send many requests together and collect results later. Anthropic’s Batch API costs 50% less; most batches finish within an hour, and anything unfinished after 24 hours expires	Rewriting 5,000 product descriptions overnight	Never for a shopper who’s waiting
   Shorter prompts and answers	Cut instructions nobody needs; ask for concise answers	Answer in three sentences, not ten	Cut too much and quality drops. Rerun your evals
   Fewer steps	Skip tools a question doesn’t need, and stop the loop when the job is done	Answer “What colors does the Harbor come in?” without the delivery check	Skipping a needed check makes answers wrong

   Made-up examples. Batch figures from Anthropic’s batch processing page, October 2026.

   A close cousin appears in Anthropic’s guide to agents as routing: Schluntz and Zhang describe sending easy, common questions to smaller, cost-efficient models and hard or unusual ones to more capable models. Choosing a model per step applies the same thinking inside one question. Your question as a product manager: which steps are simple enough for a cheaper model, and how will we know?

5. 5Speed: the wait customers actually feel.

   Speed has more than one number. Anthropic’s glossary and its guide to reducing latency define the two you’ll hear most. Latency is the time from sending a prompt to getting the output. Time to first token (TTFT) is the time until the first piece of the answer appears.

   Streaming connects them to the customer. Without it, the shopper sees nothing until the whole answer is done. With it, the answer appears word by word. Anthropic says this can significantly improve perceived responsiveness. Note the word perceived. Streaming doesn’t shorten the total time; it makes the wait feel shorter.

   • In an agent, tool calls happen before the answer starts, so the shopper’s wait for the first word includes every check, not just the model’s time to first token on the final call. A short progress line, like “Checking delivery dates…”, shows something is happening.
   • Longer answers take longer to finish. Anthropic’s first suggestions for speed are choosing the right model and keeping prompts and outputs short.
   • Set targets in customer terms (first words within, full answer within), and measure both after launch (lesson 7).

6. 6The three-way trade-off, decided with evals.

   Cost, speed, and quality pull against each other. A bigger model is often better and slower. A shorter prompt is cheaper and may drop a rule. Teams get into trouble when opinion settles it: “the big model feels smarter”, or “finance wants it cheaper”.

   Settle it with evals instead (lesson 5). Anthropic’s latency guide gives a sensible order: “It’s always better to first engineer a prompt that works well without model or prompt constraints, and then try latency reduction strategies afterward.” Here’s one way to work:

   • Set the quality floor: the eval results that must pass. At the made-up store, every measurement and delivery date is correct and sourced.
   • Set budgets: a cost per conversation, and a wait in customer terms.
   • Get a version that clears the floor, ignoring cost and speed for now.
   • Change one lever at a time, and rerun the same evals after each change.
   • Pick the cheapest, fastest version that still clears the floor. Write down why.

   Here’s how that might look at the made-up store, on a set of 200 real shopper questions:

   Version	Eval result	Cost per question	First words appear
   A: large model for every step	Passes the floor	12.9 credits	About 4 seconds
   B: A plus prompt caching	Passes (caching doesn’t change how it answers)	About 5.1 credits	A little sooner
   C: small model for every step	Fails: wrong doorway math on 14 questions	Much lower	About 2 seconds
   D: small model picks the checks, large model answers, with caching	Passes the floor	Lower than B	About 3 seconds

   Made-up example. Every number and result is invented to show the method.

   Version C is the trap: cheapest, fastest, and wrong about whether a sofa fits. Version D wins, but only because the evals proved it held the line. Rerun the comparison when your provider releases a new model or changes prices. The best answer moves.

7. 7The AI gateway: one control point for every model call.

   Picture the store a year from now. The shopping assistant, a product copy tool, and a support summary tool all call models, sometimes from two providers. Each team keeps its own keys, retry code, and cost tracking. Nobody can say what the company spends on AI this month.

   An AI gateway puts one layer between your apps and the models, so every request passes through one place. Microsoft’s Azure Architecture Center describes it as a general pattern that takes complexity away from each app and adds authentication, rate limiting, monitoring across models, routing, failover to healthy models, and caching. Gateway products differ, but the jobs are similar.

   Here are the jobs, and why each one matters to you:

   Job	What it does	Why a product manager cares
   Keys and access	Apps sign in to the gateway; provider keys stay in one place	Keys aren’t copied into every app, and you know which app made each call
   Routing and fallback	Sends each request to the right model, or to a backup when one is down	The assistant keeps answering in an outage, if the backup passed your evals
   Rate limits	Caps requests or tokens per app per minute	One busy tool can’t starve the shopping assistant
   Caching	Can return a stored answer to a repeated or similar question	Fast and cheap, but possibly out of date
   Logs and costs	Records requests, responses, tokens, and cost, by app	Real cost per conversation, and data for lesson 7
   Spend limits	Usage caps per app or team per hour, day, or month	A looping bug hits a cap, not your budget

   Generic jobs; what you get depends on what your team builds or buys. Don’t confuse the gateway’s response cache, which can skip the model and return an earlier answer (fine for “What’s your return policy?”, risky for “Is the Harbor in stock?”), with prompt caching (part 3), which still runs the model.

   Ask

   • Does every model call from our apps go through the gateway?
   • If our main model is down, what does the assistant use, and has that backup passed our evals?
   • What are the rate and spend limits for each app, and who’s alerted when one is hit?
   • Which answers may come from a response cache, and how stale can they be?
   • What does the gateway store, for how long, and who can read it?

   Avoid

   • “Let’s cache everything.” (A cached stock or delivery answer can be wrong by the time a shopper reads it.)
   • “Fallback to any model is fine.” (A model your evals never tested is an untested product.)
   • “We’ll add limits after launch.” (A looping agent can run up a large bill before anyone looks.)

   A gateway has costs too. Microsoft’s guide warns it can become a single point of failure, sees raw requests and responses (so privacy rules cover it), and adds running costs, and it says not to add one if it puts your service targets at risk. Weigh those against the control it gives you. Lesson 9 puts it together with every other part.

With AI: Use an AI assistant to check your arithmetic, not to supply your prices. Give it your token counts and your provider’s current rates, ask it to show each step, then check the steps yourself. Never trust a price an assistant remembers. Prices change often, and the source is your provider’s current pricing page.

COST, SPEED, AND QUALITY PLAN

Feature:
Provider and model(s):
Prices checked on (date), from (page):
  Input:        Output:        Cache read/write:        Batch:

One request, call by call:
  Call 1:  does:            In:        Out:
  Call 2:  does:            In:        Out:
  Call 3:  does:            In:        Out:

Cost per request:
Requests per conversation:
Cost per conversation:
Monthly cost (estimate):

Quality floor (evals that must pass):
Wait targets: first words within ___ / full answer within ___

Levers, one at a time (rerun evals after each):
  [ ] Prompt caching   [ ] Smaller model for simple steps
  [ ] Batch            [ ] Shorter prompt/answers   [ ] Fewer steps

Gateway: spend limit ___  alert goes to ___  backup model (passed evals?) ___

Compare estimate with real costs on (date):

Ask AI about this lesson

All lessons