Find the root cause.

Key terms

Root cause

The underlying reason a problem happened. Fix it, and the same problem is much less likely to come back. Compare a symptom, which is only what you see.

Segment

A slice of your users or data, such as Android users, new customers, or one country.

Funnel step

One stage on the path to a goal, like product page, cart, checkout, and payment.

Hypothesis

A specific, testable guess about the cause: “If X caused it, we’d expect to see Y in the data.”

Five whys

Asking “why?” again and again, each time about the previous answer, until you reach a cause you can act on.

Fishbone diagram

A cause-and-effect diagram, also called an Ishikawa diagram, that groups possible causes of one problem into categories.

The idea

When a number drops, everyone has a theory within minutes. The new design. The competitor’s sale. The weather. Many of those theories turn out to be wrong, and acting on the wrong one costs time and trust.

Root cause analysis is a calm, ordered way to go from “a number moved” to “this is why, and here’s what we’ll do.” It isn’t a flash of insight. It’s narrowing down, one check at a time. This lesson teaches the order, with a worked case from the made-up furniture store.

The lesson, in seven parts

1. 1Check that the change is real.

   Before you look for a cause in the product, look for one in the measurement. A “drop” is often a broken tag, a late data load, or a change in how something is counted. Google’s HEART paper names some of the challenges in keeping log-based numbers accurate, such as making sure “all of the important user actions are being logged” and filtering out traffic from automated sources such as crawlers.

   Then ask whether the change is bigger than normal. Most metrics wobble. Compare with the same day last week and the same period last year, and look at how much the number usually moves. A Monday that’s lower than Friday may simply be a Monday.

   Ask

   • Did anything change in tracking or reporting?
   • Is the data complete, or still loading?
   • How does this compare with the same day last week, and last year?
   • Is this bigger than the usual ups and downs?

   Avoid

   • Starting with “Which team broke it?”
   • Comparing a holiday week with a normal week
   • Trusting one chart without checking the raw counts
   • Announcing a cause before checking the data

2. 2Define the change exactly.

   “Orders are down” isn’t precise enough to investigate. Pin down four things: which metric (and its formula), how much it moved, since when, and compared with what. Then look at the parts of the formula. A rate has a top and a bottom, and either one can move.

   Say conversion fell. Did fewer people buy, or did many more people visit who were never going to buy? A big ad campaign can bring in visitors who are only browsing. Conversion falls, but nothing is broken. That’s a different story from the same number of visitors buying less.

   Question	In the made-up case
   Which metric, exactly?	Orders per day, and order conversion rate (visits, or sessions, that ended in an order ÷ all sessions)
   How big?	Orders down 15% on the same day last week; conversion down by a similar amount
   Since when?	Wednesday. Sudden, not gradual.
   Compared with what?	Same days last week and the last eight weeks. No holiday in either.
   Top or bottom of the rate?	Sessions are flat. Fewer of them end in an order.

   Sudden and gradual changes point to different causes. As a rule of thumb, a sudden step often means something changed on a specific day: a release, an outage, a price change. A slow slide is more often about customers, competitors, or the season.

3. 3Split it until you find where it lives.

   A total hides its parts. Break the metric down along a few dimensions, one at a time, and look for the slice where the change is concentrated:

   • Time: which day, which hour did it start?
   • Platform: desktop, iPhone app, Android app, mobile web; browser and app version.
   • Place: country, region, city.
   • Customer type: new or returning, signed in or guest.
   • Traffic source: search, ads, email, direct.
   • Funnel step: product page, cart, checkout, payment, order.

   If every slice fell by about the same amount, look for something that touches everyone: prices, a sitewide outage, the season. If one slice fell and the rest held steady, you’ve narrowed the search a lot. One more trap: the total can move just because the mix changed. If more of your traffic comes from a slice that always converts less (mobile web, say), the overall rate falls even though no slice got worse. This is called a mix shift.

4. 4Inside or outside?

   Next, list what changed around the time the number moved, on both sides of the company wall. Ask the people who’d know: engineering for releases, marketing for campaigns, support for complaints, operations for delivery problems.

   Inside the company	Outside the company
   App or website releases, and experiments	Seasons and holidays
   Price or delivery-fee changes	A competitor’s sale or launch
   Marketing campaigns starting or stopping	News, weather, or the economy
   Outages, slow pages, payment errors	Changes by app stores, browsers, or payment providers
   Tracking or reporting changes	A change in search results that sends you less traffic

   Line each change up against the timeline. A cause has to come before its effect, and it should affect the slice you found. A competitor’s sale wouldn’t hit only Android users. An Android app update might.

5. 5Write hypotheses, then test the likeliest first.

   Turn each suspect into a hypothesis you can check. The useful form is: “If this caused it, we’d expect to see that.” A good hypothesis predicts something specific you can look for, and could turn out to be wrong.

   Rank them by how well they fit what you’ve already found, and by how quick they are to check. Then test one at a time. Look at the data for the predicted pattern. Try to reproduce the problem yourself, on the device or path in question. If a release is the suspect, undoing it and watching the number recover is strong evidence. On a website, you can roll the change back or switch it off for some users. In an app, people who already updated keep that version, so you usually pause the rollout and ship a fix.

   The template

   If [suspected cause] caused the drop, we’d expect to see [specific pattern] in [data or test]. Check by [how]. If we don’t see it, we can probably rule it out.

   Filled in, for the made-up furniture store

   If Tuesday’s Android app update caused the drop, we’d expect it only on the new app version, starting as the update spread, and concentrated at the payment step. Check by splitting Android orders by app version and funnel step, and by trying to pay on an older Android phone.

   Watch for things that only happen at the same time. Two numbers moving together doesn’t mean one caused the other. That’s why the prediction should be specific enough that a coincidence would fail it.

6. 6Dig to the root: five whys and a fishbone.

   Finding the broken thing isn’t the end. Ask why it broke, and why that happened, until you reach something you can fix so it doesn’t happen again. This is the five whys. The Institute for Healthcare Improvement (IHI) describes it simply: ask “Why?” five times until you reach the root cause. IHI attributes it to Taiichi Ohno, who helped build the Toyota Production System. Five is a guide, not a rule: the point is to keep asking until you reach the root cause. IHI also notes there may be more than one. Sometimes the useful question changes along the way, from “why did it break?” to “why wasn’t it caught?”

   When you’re stuck listing causes, a fishbone diagram helps you think wider. It’s a cause-and-effect diagram, also called an Ishikawa diagram after Kaoru Ishikawa, the Japanese quality expert best known for it. The problem goes at the head. Each bone is a category of possible causes. IHI lists the classic categories as Materials, Methods, Equipment, Environment, and People. For a digital product, pick categories that fit, like the ones below.

7. 7Decide, explain, and follow up.

   Root cause analysis ends with a decision, not a slide. You usually have four options: fix it now (or roll it back), wait and watch if the evidence is weak, dig further if two causes still fit, or accept it if it’s outside your control and plan around it.

   Then tell people, briefly, in this order: what happened, what caused it, how sure you are, what you’re doing, and when you’ll update them. Say what you don’t know yet. A clear “we’re fairly sure, here’s why, and here’s how we’ll confirm it” usually builds more trust than false certainty.

   Afterwards, check that the number actually recovered. Then fix the process from your last “why”, so the same kind of problem is caught earlier next time.

With AI: AI tools are getting good at the slicing: you can ask in plain words for a metric split by platform, version, and funnel step, and get the charts in seconds. Some tools flag unusual changes on their own. That speeds up the search. It doesn’t replace the reasoning: whether the data is trustworthy, which hypothesis actually predicts what you see, and which fix addresses the cause rather than the symptom. Check an AI’s suggested cause the same way you’d check a colleague’s.

THE CHANGE
  Metric and formula:
  How much:
  Since when (sudden or gradual):
  Compared with:

1. IS IT REAL?
  Tracking or reporting changes:
  Data complete:
  Bigger than the usual ups and downs:

2. TOP OR BOTTOM OF THE RATE?

3. SPLIT IT (where is the change concentrated?)
  Time:
  Platform / version:
  Place:
  Customer type:
  Traffic source:
  Funnel step:

4. WHAT CHANGED AROUND THEN?
  Inside:
  Outside:

5. HYPOTHESES (If X caused it, we'd expect Y. Check by Z.)
  -
  -

6. FIVE WHYS
  Why?
  Why?
  Why?
  Why?
  Why?

7. DECISION (fix / roll back / wait and watch / dig further / accept)
  How sure are we:
  Next update:
  How we'll confirm the number recovered:

Ask AI about this lesson

All lessons