Observability: see what your AI actually did.

Key terms

Observability

The ability to understand what a system did, and why, from the data it records, including problems nobody predicted.

Trace

The full record of one request from start to finish. For an AI assistant, usually one shopper question and everything that happened to answer it.

Span

One unit of work inside a trace, such as a single model call or a single tool call. Spans can contain smaller spans.

Log

A timestamped record of one event, such as “delivery date check returned October 14”.

Metric

A number measured over time, such as answers per hour, average wait, or cost per conversation.

Error analysis

Reading real conversations, writing down what went wrong, sorting those notes into categories, and counting each category.

Failure category

A named, recurring kind of problem found in error analysis, such as “stale delivery date” or “missed handoff to a person”.

The idea

When a normal web page breaks, it usually breaks loudly. A button stops working, an error page appears, and an alarm goes off. AI features fail differently. The assistant answers, politely and on time, and the answer is wrong. Nothing crashes. Nobody gets paged. The first sign is often an unhappy customer, days later.

Evals (lesson 5) test your feature before it ships, on questions you chose. Guardrails (lesson 6) block what must never happen. Observability is how you see what happens after launch, on questions you didn’t think of. It isn’t a dashboard you buy. It’s a record of what happened, plus the habit of reading it.

The lesson, in seven parts

1. 1Why AI features need observability.

   OpenTelemetry, an open-source toolkit for recording what software does (traces, metrics, and logs), describes observability as the ability to understand a system from the outside by asking questions about it, without knowing its inner workings. It says observability helps you answer one question in particular: “Why is this happening?”

   Its docs also make a point that fits AI features exactly. A shopping site could be up 100% of the time, they note, and still be unreliable if clicking “Add to Cart” on a black pair of shoes doesn’t always add black shoes. Up isn’t the same as right.

   AI features make that gap wider, for three reasons:

   • Answers vary. As lesson 1 showed, the same question can get different answers, so one good test doesn’t prove the next answer will be good.
   • Failures are quiet. A wrong answer usually arrives on time, with no error code. Every system check can pass.
   • Failures are plausible. A wrong delivery date reads exactly like a right one. Customers believe it, and so do you, unless you check.

   Anthropic’s engineers hit this with a research feature built from several AI agents. Users reported the agents “not finding obvious information”, and the team couldn’t see why. Their June 2025 write-up says that adding full production tracing let them diagnose why agents failed and fix issues systematically. You can’t fix what you can’t see.

2. 2Traces and spans, in plain words.

   OpenTelemetry calls a trace “the path of a request through your application”. For your assistant, the request is one shopper question. The trace is everything that happened between the question and the answer.

   Each step inside it is a span, which OpenTelemetry defines as a unit of work or operation. Spans nest. The first span, the root, covers the whole question from start to finish. Inside it sit child spans: each model call, each tool call, each lookup. All of them share one trace ID, so you can pull up the whole story at once.

   Here’s one trace from the made-up furniture store, for the delivery question in the hook:

   • Each span can carry attributes: details stored as name and value pairs. For a tool call, the useful ones are what went in, what came back, and how long it took.
   • Each span also has a status: Unset (the default: it finished without an error), Error, or Ok (someone explicitly marked it successful). Status tells you whether the code failed, not whether the answer was right. That’s why “no errors in the logs” can sit next to a broken promise.
   • OpenTelemetry is writing shared names for AI work, the generative AI semantic conventions: for example, “chat” for a model call and “execute_tool” for a tool call. At the time of writing (October 2026) they’re marked as in development, so expect changes. Shared names help your traces read the same way in different tools.

3. 3Logs and metrics: what to record and count.

   Traces show one question in depth. Two other signals fill in the rest. A log, in OpenTelemetry’s words, is “a timestamped text record”: one event, written down when it happened. A metric is a measurement taken while the system runs, and you usually look at metrics added up over time.

   An AI feature needs a few more numbers than a normal web page. Here’s a starting set:

   Metric family	What to count	The question it answers
   Quality	Share of sampled conversations that pass your checks; counts per failure category	Are answers right and useful?
   Cost	Tokens in and out, model calls and tool calls per conversation, cost per conversation	Is each answer affordable?
   Speed	Time to the first word of the answer, time to the full answer, slowest tool calls	How long does a shopper wait?
   Errors	Tool calls that failed or timed out, model calls that failed, retries	Is something technically broken?
   Refusals	Answers where the assistant declined or said it couldn’t help	Is it saying no too often, or not often enough?

   Made-up furniture store. OpenTelemetry’s generative AI conventions include standard metrics for some of these, such as how long an operation took, time to first token, and input and output tokens.

   Two warnings. Quality is the hardest number to get and the most important: cost, speed, and errors come from your systems for free, but quality only exists if someone defines “good” and checks for it (lesson 5). And numbers tell you that something changed, rarely why. For why, you read traces.

4. 4Customer signals: where to look first.

   Your customers are already telling you which answers failed. A few say it directly. Most say it through what they do next. Treat these signals as pointers to conversations worth reading.

   Here’s how the common ones read at the made-up furniture store:

   Signal	What it may mean	Watch out
   Thumbs down	The shopper didn’t like the answer	Many people never click either thumb, and the ones who do may not be typical. A thumbs up can also mean “polite”, not “correct”.
   Follow-up or rephrased question	The first answer missed: “No, I mean the two-piece one”	Some follow-ups are just the next step in a good conversation.
   Handoff to a person	The assistant couldn’t help, or the shopper didn’t trust it	A handoff can be the right outcome. A missing handoff can be the real failure.
   Conversation ends, no next step	The shopper gave up, or got what they needed	You can’t tell which without reading it.
   Contact to support or a return later	A promise in the chat didn’t hold	Link the support ticket to the conversation, or you’ll never connect them.

   Made-up example. Which signals you can collect depends on your product and your privacy rules.

   Don’t only read the conversations customers complained about. Hamel Husain and Shreya Shankar, who teach AI evals, call user feedback a good way to find problem conversations, and also recommend reviewing a random sample. The shopper in the hook didn’t complain in the chat. She believed the answer.

5. 5Read real conversations: error analysis.

   Here’s the habit that matters most, and the one teams skip most often. In a 2025 field guide, Hamel Husain calls error analysis “the single most valuable activity in AI development”. He also warns against the “tools trap”: the belief that the right tools or generic metrics will solve your AI problems. He’s seen teams celebrate a better “helpfulness score” while their users still struggled with basic tasks.

   Husain and Shankar describe the process in their evals FAQ. In plain steps:

   • Gather a set of real conversations (traces). They recommend reviewing at least 100.
   • Read each one and write a short, open-ended note on what went wrong. They call this open coding. Note the first thing that went wrong, because early mistakes cause later ones.
   • Write down every problem that makes the product less useful, even when it isn’t the model’s fault.
   • Group similar notes into categories (they call this axial coding), then count how many conversations fall into each.
   • Keep reading until new conversations stop showing you new kinds of failure.
   • This is product work, not only engineering work. Husain and Shankar write that engineers catch technical issues like tool errors, while product managers catch product failures like unmet expectations. Their example of the right question to ask: “Has an appointment been made?” not “Did the tool call succeed?”
   • Start from the data, not from a list of generic labels. Husain’s field guide notes that starting with labels like “hallucination” often misses the problems specific to your product. “Stale delivery date” is a category you’d only find by reading.
   • Make reading easy. In a 2024 post, Husain writes: “You must remove all friction from the process of looking at data.”

6. 6Worked example: the wrong delivery date.

   Back to the hook. Here’s how a product manager and an engineer at the made-up furniture store would trace it, step by step:

   • Find the conversation. Support links her ticket to the chat, and the chat to its trace.
   • Read what the shopper saw first. The answer says, “Your Harbor sofa can arrive October 14.” Clear, confident, and wrong.
   • Open the trace. The first model call planned the steps: check stock, then check delivery. Fine.
   • Check each tool call. The stock check returned “in stock”. Correct. The delivery date check returned “earliest: October 14”, with status Ok.
   • Look at that tool call’s attributes. The schedule it read was dated October 4, three days old. The nightly job that copies it had stopped, and nothing raised an alarm.
   • Check the last model call. It repeated the tool’s date faithfully. The model did its job. The data was wrong.
   • Count how far it spread. Search the traces for delivery date checks that read a schedule more than a day old.

   Layer	Fix	Who owns it
   Data	Alert when the delivery schedule is more than a day old	Engineering
   Tool	Return the schedule’s date with every answer, so freshness is visible	Engineering, with the product manager
   Instructions	If the schedule is old, give the date as an estimate and offer to confirm	Product manager
   Evals (lesson 5)	Add test cases where the tool returns an old schedule; the answer passes only if it doesn’t promise a firm date	Product manager, with engineering
   Dashboard	Add “age of delivery data” next to the other metrics	Product manager

   Made-up example. The dates, the nightly job, and the fixes are invented for this lesson.

   The template

   New eval case Input: [the shopper’s question] Setup: [what the tools return, including the bad data] Pass if: [what a good answer does] Fail if: [what this failure looked like] Found in: [failure category, and the week you found it]

   Filled in, for the made-up furniture store

   Input: “When can the Harbor sofa arrive at 60614?” Setup: the delivery date check returns October 14 from a schedule dated three days ago. Pass if: it says the date is an estimate and offers to confirm. Fail if: it promises October 14 as a firm date. Found in: stale delivery date, week of October 5. (Made-up example.)

   That last search is why traces matter. Without them, you have one angry shopper. With them, you know which conversations got a date from old data, and can decide whether to contact those shoppers. Then turn the finding into fixes, one for each layer that let it through:

7. 7What you record, and what you review each week.

   Recording conversations is how you find failures. It’s also a privacy decision. Shoppers type addresses, phone numbers, and sometimes things they shouldn’t. OpenTelemetry’s generative AI conventions treat instructions, user messages, and model answers as sensitive: tools following them shouldn’t record that content by default, and should let teams opt in. For production systems, the conventions suggest storing conversation content separately from the rest of the trace, so it can have its own access controls.

   OpenTelemetry’s guidance on sensitive data adds data minimization: only collect data that serves an observability purpose, and regularly review what you collect. The team behind Anthropic’s Research feature watched patterns in how its agents made decisions, without monitoring the contents of individual conversations, to protect user privacy. Your team will pick its own balance. The decisions you’ll help make:

   Ask

   • Which conversations are recorded in full, and which only as counts and timings?
   • Which personal details are removed or masked before anyone reads a conversation?
   • Who can read full conversations, and is that access logged?
   • How long is conversation content kept before it’s deleted?
   • What do customers agree to, and does the privacy notice say so? (Ask your legal and privacy team.)

   Avoid

   • “Record everything, we’ll sort it out later.” (You’ll keep data you don’t need, and you may not be allowed to.)
   • “Record nothing, it’s safer.” (Then nobody can do error analysis, and failures go unfixed.)
   • “Anyone on the team can browse conversations.” (Limit and log who reads them.)

   Then make it a ritual. Once a week, check the numbers for what changed, read a sample of conversations, update your failure categories, and pick the top fix. Husain and Shankar suggest a deeper error analysis after big changes, like a new model or new instructions, and in between, reading 10 to 20 conversations a week, focused on unusual ones: very long chats, many retries, or ones your checks flagged. The dashboard tells you where to look. Reading tells you what to fix.

With AI: AI can help with the reading, but not replace it. Husain and Shankar suggest letting a model propose groupings once you’ve written your own notes on the first 30 to 50 conversations, and they say the first reading of raw conversations should never be handed to a model. Try it: write your notes on 20 conversations, ask an assistant to group them, then check every grouping against the conversations yourself.

WEEKLY AI FEATURE REVIEW

Week of:                Reviewer:

1. NUMBERS (what changed since last week?)
  Quality (share of sampled conversations that pass):
  Cost per conversation:
  Speed (first word / full answer):
  Errors (failed or timed-out tool and model calls):
  Refusals:
  Customer signals (thumbs down, rephrases, handoffs):
  Anything that moved, and when it started:

2. READ (10 to 20 conversations)
  How picked (random / flagged / complaints / outliers):
  #   Note on the first thing that went wrong
  1.
  2.
  3.

3. SORT AND COUNT
  Failure category            Count   New this week?
  -
  -
  -

4. TOP FIX
  Category:
  Which span or layer failed (data / tool / instructions / model):
  Fix, and owner:
  How many conversations it affected:

5. NEW EVAL CASES
  Input:
  Setup:
  Pass if:
  Fail if:

6. PRIVACY CHECK
  Personal details masked before reading?  Y / N
  Access to full conversations limited and logged?  Y / N

Open questions for engineering:

Ask AI about this lesson

All lessons