System design for product managers.

Key terms

Client and server

The client is the device and software the shopper uses, usually a browser or an app. A server is a computer that stores the site or app and answers the client’s requests.

API

Application programming interface: a set of rules that lets one piece of software ask another for something. A contract: send this, get that back.

Database

An organized, electronically stored collection of data (products, prices, orders, accounts), managed by software that stores, finds, and changes it.

Cache

A stored copy of an answer, reused for later requests instead of working it out again. Faster, but it can be out of date.

Queue

A waiting line for work. One part of the system leaves a message; another picks it up and does the work later, so nobody waits for it.

Latency

The delay between asking for something and getting it. Low latency means fast.

Availability

The share of time (or of requests) in which a service works as agreed. Often written in “nines”: 99.9% is three nines.

The idea

“System design” sounds like a topic for engineers only. Partly, it is. You won’t pick the database or decide how the data is organized, and nobody expects you to.

But almost every system decision is also a product decision. How fresh must the delivery date be? What should a shopper see if the warehouse system is down? Is it fine if the confirmation email arrives a minute late? Engineers can’t answer those alone. This lesson gives you enough vocabulary to answer them together, and a step-by-step way to talk through any design. One made-up example runs through it: an online furniture store and its delivery-date estimate.

The lesson, in seven parts

1. 1What happens when a shopper opens a page.

   Start with one tap. A shopper opens the page for a gray sofa. Their phone’s browser, the client, sends a request across the internet to the store’s servers. A server gathers what the page needs and sends back a response. The browser draws the page. On the web, that request and response travel over HTTP, which MDN Web Docs describes as a client-server protocol (a shared set of rules for the conversation): the browser starts the request, and the server answers it.

   That one page usually needs several things at once, and each comes from a different piece:

   Piece	What it does	In the made-up furniture store
   Client	The shopper’s browser or app; sends requests and shows the result	The phone showing the gray sofa page
   Server	Receives requests, runs the store’s code, sends responses	Builds the sofa page
   Database	Stores data so it can be found and changed	Products, prices, stock, orders
   Cache	Keeps recent answers close by, to reuse them	Sofa descriptions that rarely change
   CDN (content delivery network)	Servers in many places that keep copies of files near users	Sofa photos load from a server near the shopper
   Other services, through APIs	Separate systems that each do one job	Delivery dates, payments, the warehouse
   Queue	Holds work to be done a little later	New orders waiting for the warehouse

   A made-up example. Real systems have more pieces; these are the ones you’ll hear about most. The CDN description follows MDN Web Docs.

   You don’t need to know how each piece works inside. You need to know what each one is for, so that when an engineer says “that’s a database read on every page view”, you understand why they’re frowning.

2. 2APIs: the contracts between the pieces.

   MDN Web Docs defines an API, short for application programming interface, as “a set of features and rules that exist inside a software program” that let other software work with it, and says it can be seen as “a simple contract” between the program that offers it and whatever uses it. In plain words: send this, get that back.

   Engineers design how an API keeps its promise. You help decide what it promises, because that’s the part shoppers feel. Here’s the delivery-date API at the made-up store, described the way a product manager might:

   • Send: the sofa’s product number and the shopper’s ZIP code.
   • Get back: the earliest delivery window, like “Thu, Oct 15 to Sat, Oct 17”, and whether in-home delivery (carried to the room the shopper picks) is available.
   • Promise: an answer in under half a second for 99 requests in 100 (part 5 explains why it’s said that way).
   • If it can’t answer: say so clearly, so the page can show something else.

   That last line is a product decision, and so is the speed. Questions worth asking about any API your feature depends on:

   • What do we send, and what do we get back?
   • How fast does it answer, usually and at its slowest?
   • Who owns it, and who do we call when it breaks?
   • Is there a limit on how often we can call it?
   • What does it return when it fails, and what will our page show then?

3. 3Databases and caches: where data lives, and how fresh it is.

   A database is where the store keeps what it knows. AWS describes a database as “an electronically stored, systematic collection of data”, managed by software that stores, retrieves, and edits it. Products, prices, stock counts, orders, and accounts all live in databases.

   Asking a database the same question thousands of times a minute is slow and costly. So teams add a cache: a copy of a recent answer, kept close by and reused. MDN explains that the HTTP cache “stores a response associated with a request and reuses the stored response for subsequent requests.” It also names the catch. A stored response is either fresh or stale, and a stale one has expired.

   So every cache comes with a product question: how out of date is acceptable? The answer is different for each kind of data:

   Data on the sofa page	How fresh must it be?	Cache it?
   Description and photos	Hours old is fine	Yes, for a long time
   Price	Must match what the shopper pays	Briefly, and check again at checkout
   “Only 2 left”	Close to live, or it misleads	Briefly, or not at all
   Delivery date estimate	Minutes old is fine here; exact at checkout	Yes, for about 15 minutes
   The shopper’s cart	Always exact	No. Each cart is read fresh, never from a copy shared between shoppers

   A made-up example. Your engineers know the real limits; your job is to say what the shopper needs.

   That’s the engineer’s line from the start of this lesson. Caching delivery dates for 15 minutes trades a little freshness for a lot of speed, and it takes load off the warehouse system. Checking the date again at checkout keeps the promise honest.

4. 4Queues: work that doesn’t need to happen right now.

   When a shopper places an order, a lot has to happen. Charge the card. Save the order. Tell the warehouse. Book a delivery slot. Send a confirmation email. If the shopper had to wait for all of it, checkout would crawl, and one slow system could hold up every order.

   A queue fixes that. AWS describes a message queue as asynchronous communication between services: messages wait on the queue until they’re processed, and queues can “smooth spiky workloads.” Asynchronous means the sender doesn’t wait for the work to finish. The order is saved, the shopper sees “Thanks, your order is in”, and the warehouse and email systems pick up their messages moments later.

   For a product manager, the useful question is short: does the customer need to wait for this?

   Step after “Place order”	Must the shopper wait?	Why
   Confirm the payment went through	Yes	They need to know the order is real
   Save the order	Yes	Nothing else can happen without it
   Book the delivery slot	It depends	If you promise a date at checkout, hold it now; the final booking can follow
   Send the order to the warehouse	No: queue it	Seconds later is fine
   Send the confirmation email	No: queue it	A minute later is fine
   Update the “you might also like” suggestions	No: queue it	Hours later is fine

   A made-up example.

   A queue has a cost too: things happen soon, not instantly. So you also decide what the shopper sees in between, like “Your confirmation email is on its way.”

5. 5Speed and scale: latency, percentiles, and more servers.

   Latency is the delay between asking and getting an answer. MDN defines it as the network time it takes for a requested resource to reach its destination, and puts it simply: low latency is good, high latency is bad. Google’s Site Reliability Engineering book uses the wider idea teams track for a whole request: how long it takes to return a response. Shoppers feel it as a page that hangs.

   Here’s what many people miss. Don’t judge speed by the average alone. Google’s Site Reliability Engineering book shows a service where a typical request takes about 50 milliseconds, yet 5% of requests are 20 times slower, and an average hides it. So teams look at percentiles: the 99th percentile is the time that 99% of requests beat, which the book calls “a plausible worst-case value”. If your sofa page is slow at the 99th percentile, one page view in a hundred is at least that slow. A shopper opens many pages, so far more than one shopper in a hundred feels it. On a busy day, that’s a lot of people.

   Scale is how the system copes as more people use it. Microsoft’s Azure architecture guidance describes two basic moves. Vertical scaling increases the capacity of a single resource, like a bigger server. Horizontal scaling adds more copies of the server to share the work. Adding servers usually means adding a load balancer, which, in AWS’s words, distributes traffic “across a pool of resources”.

   Move	What it means	Helps with	Watch out for
   Scale up (vertical)	One bigger server	Parts that are hard to split	Servers only come so big, and one machine can still fail
   Scale out (horizontal)	More servers sharing the work, behind a load balancer	Traffic spikes, like a holiday sale	The software must be built to run on many machines

   Scale up and scale out follow Microsoft’s Azure Well-Architected guidance. The trade-offs are typical, not universal: your engineers know your system’s limits.

   Your part is to bring the numbers behind these choices: how many people, at what peak, expecting what speed. Part 7 shows how.

6. 6Reliability: what happens when a piece fails.

   Every piece fails sometimes. AWS defines availability as “the percentage of time that a workload is available for use”, and teams describe it in nines. Each extra nine cuts the allowed downtime to about a tenth:

   Availability	Downtime allowed per year, at most
   99% (two nines)	3 days 15 hours
   99.9% (three nines)	8 hours 45 minutes
   99.95%	4 hours 22 minutes
   99.99% (four nines)	52 minutes
   99.999% (five nines)	5 minutes

   From the AWS Well-Architected Framework, Reliability Pillar, which lists online commerce among its examples at the 99.95% level.

   Two ideas from the same AWS guide matter most to a product manager. First, pieces that depend on each other multiply. If a page can’t work without two other services, and all three are designed for 99.99%, the page can reach only about 99.97% (99.99% × 99.99% × 99.99%). AWS calls that a hard dependency. Second, the fix is often a product decision: make it a soft dependency, where the page still works when the other piece fails. If the delivery-date service is down, the sofa page shouldn’t break. It can say “See delivery dates at checkout” and let the shopper keep going.

7. 7Talk through a design, step by step.

   When engineers sketch a design, or an interviewer says “design a delivery-date feature”, it helps to follow a steady order. Alex Xu’s book System Design Interview uses four steps: understand the problem and establish the design scope, propose a high-level design and get buy-in, design deep dive, and wrap up. This lesson breaks that into six moves for product managers:

   • Clarify the requirements. What must it do, for whom, on which devices? And how well: how fast, how fresh, how available?
   • Estimate the scale. A rough count of users, requests, and peaks, done on the back of an envelope. Round numbers are fine.
   • Sketch the components. Boxes for the client, servers, data, caches, queues, and other services.
   • Follow the data. Walk one request through the boxes, start to finish.
   • Find the bottlenecks. Which piece breaks first at peak? Which failure hurts shoppers most?
   • Name the trade-offs. Speed or freshness, cost or availability, now or later. Decide them out loud, with reasons.

   The template

   We’re designing [feature] for [users]. It must [what it does], answer within [time], be no more than [how old] out of date, and stay available [target], or fall back to [what the shopper sees]. At peak we expect about [requests per second]. The main pieces are [components]. A request flows [path]. The likely bottleneck is [piece], so we’re thinking about [cache, queue, scale out, or fall back], and engineering decides how. We’re trading [this] for [that], because [reason that matters to users].

   Filled in, for the made-up furniture store

   We’re designing a delivery-date estimate for shoppers on sofa pages. It must show the earliest delivery window for their ZIP code, answer within half a second, be no more than 15 minutes out of date on the product page (and exact at checkout), and never block the page: if it can’t answer, the page says “See delivery dates at checkout”. At peak we expect about 25 requests a second. Here’s the rough math, with made-up numbers: 200,000 page views on a busy day, spread over the 86,400 seconds in a day, is about 2 a second, and a sale might bring ten times that. A request goes from the page to the delivery-date service, which answers from its cache or, if it has no fresh answer, asks the warehouse and carrier schedules. The likely bottleneck is the warehouse system, so we’re thinking about caching answers by product and ZIP-code area, with the fallback if it’s slow, and engineering decides how. We’re trading a little freshness for speed, because a fast date that’s confirmed at checkout helps shoppers more than an exact date on a page that hangs.

   Notice how much of that is yours: who it’s for, how fresh, how fast, what the shopper sees when something fails. The boxes are the engineers’ craft. The reasons are shared.

With AI: AI tools can now draft an architecture diagram, explain an unfamiliar term, or list the failure cases you forgot, in seconds. Use them to learn faster and to prepare better questions. Don’t use them to overrule your engineers: an AI doesn’t know your systems, your traffic, or your past outages, and it sounds just as confident either way. AI features also bring system questions of their own, like slower responses and answers that vary, so the habit of asking “how fast, how fresh, and what if it fails?” matters more, not less.

FEATURE:
WHO USES IT, ON WHAT DEVICES:

1. REQUIREMENTS
   What it must do:
   How fast (usual, and slowest acceptable):
   How fresh must each piece of data be:
   Availability target, and what shoppers see if it fails:

2. SCALE (round numbers)
   Users per day:          Requests per second, average:
   Peak (sale, holiday):   Data stored:

3. COMPONENTS
   Client | Servers | Database | Cache | Queue | Other services (APIs)

4. DATA FLOW
   One request, start to finish:

5. BOTTLENECKS
   What breaks first at peak:
   Which failure hurts users most:
   Hard or soft dependency? Fallback:

6. TRADE-OFFS
   We're trading ___ for ___, because ___.

QUESTIONS FOR ENGINEERING:
   What does it return when it fails?
   What's slow at the 95th and 99th percentile?
   What would it take to cache this, and for how long?

Ask AI about this lesson

All lessons