System Design

Why big websites don't fall over

Same request. Now a million people send it at once.

The whole field, in three boxes

Everything else is an optimization

Browser → HTTP request → Server → query → Database
Browser ← HTML / JSON  ← Server ← rows  ← Database

The browser handles display and interaction. The server validates requests and applies rules. The database stores durable data.

Scaling

One server, then too many people

Vertical scaling gives one server more CPU or memory. Horizontal scaling adds more server copies. Either way, first identify the bottleneck: extra web servers cannot fix every slow database query.

Buying a bigger computer works — until there isn't a bigger computer.

Load balancing

Run many copies, share the work

A load balancer distributes requests among healthy instances. Health checks help it stop routing new requests to a copy that is no longer responding.

One machine dying becomes a non-event instead of an outage.

Callback to Session 1

Statelessness is what makes that legal

Any copy can handle a request if it receives the necessary credentials and can access shared data. Keeping essential session information only in one instance's memory would make switching instances unreliable.

Neither copy remembers you — so it does not matter which one you get.

Caching

Don't redo work you already did

Instead of recalculating a popular leaderboard on every request, save the result and reuse it for a chosen interval. Decide when the saved answer must expire or be invalidated.

A cache is a saved answer, kept close. That is the whole idea.

Caching

Four places to keep the answer

Cache	Where it lives	Work it can save
Browser	On the user's device	Downloading a resource again
CDN	At distributed edge locations	A trip to the origin server
Server cache	Near the application	Repeated calculations or database queries
Database cache	Within the database system	Reading the same data from storage again

Distance

You cannot buy your way out of physics

A Content Delivery Network (CDN) serves cached copies from locations nearer to users. Shorter network journeys can reduce latency even when the original server is already fast.

So don't make the trip. Keep a copy nearby. That is a CDN.

SQL

You say what you want, not how

PYTHON — you write the how

results = []
for row in submissions:
    if row.user_id == 42:
        results.append(row)
results.sort(key=lambda r: r.created_at)
top = results[:10]

SQL — you write the what

SELECT *
FROM submissions
WHERE user_id = 42
ORDER BY created_at
LIMIT 10;

Same answer. In SQL the database picks the how — and it picks better than you would.

SQL

A table is a spreadsheet with rules

A table contains rows and typed columns. A primary key identifies a row; constraints can require values to be present or unique. A foreign key can connect a submission's user_id to a user row.

SQL

Four verbs, again

SQL statement	Job
SELECT	Read rows
INSERT	Add rows
UPDATE	Change rows
DELETE	Remove rows

These resemble the create/read/update/delete operations exposed by many APIs, though HTTP methods and SQL statements do not map one-to-one.

The same shape as HTTP’s GET, POST, PUT, DELETE. That is not a coincidence.

SQL

One query, read left to right

In the earlier query, SELECT * asks for all columns, FROM submissions chooses the table, WHERE filters rows, ORDER BY sorts, and LIMIT 10 caps the result. This is how to read the query, not a claim about the database's execution order.

SQL

JOIN: two tables, one answer

SELECT users.name, submissions.exercise
FROM submissions
JOIN users ON users.id = submissions.user_id;

The ON condition matches each submission to its user so the result can include columns from both tables.

Names live in one table, work lives in another. A join is how you ask about both at once.

SQL · the payoff

The leaderboard is one query

SELECT users.id, users.name, SUM(submissions.points) AS total_points
FROM users
JOIN submissions ON submissions.user_id = users.id
GROUP BY users.id, users.name
ORDER BY total_points DESC, users.id
LIMIT 10;

This example assumes each submission has a points value. The database groups, totals, and sorts before sending the top ten rows.

Ten rows come back instead of 500,000. The database did the work, not the browser.

The database · you already know this one

An index is binary search

An index organizes keys so the database can narrow a lookup instead of scanning every row. Many database indexes use balanced trees such as B-trees, rather than literally running binary search over a plain table. The connection is the same search principle: discard large parts of the search space.

This is where binary search and BSTs went. Every database query, forever.

The same table, two ways

500,000 steps, or 19

For intuition, halving 500,000 possibilities takes about 19 steps. A real index's cost depends on its structure, matching rows, and storage access. An appropriate index can help a query; maintaining indexes also costs storage and write work.

A thousand times more data costs ten more steps.

Latency

If reading memory took one second

The useful lesson is relative scale: accessing memory is generally much faster than reading storage, and a distant network trip can cost much more again. Reuse nearby data when freshness requirements allow it.

Fast systems are not clever. They avoid the trip.

Queues

Answer now, do the slow part later

Save the submission, enqueue an email job, then respond to the browser. A worker sends the email separately. A queued job is pending work, so the interface should distinguish acceptance from completion when that matters.

Sending email, encoding video, generating a report — nobody should watch a spinner for that.

Message queues

A to-do list two programs share

A producer adds a job to the queue. A worker receives it, performs the work, and acknowledges completion. The programs can run at different times and at different speeds.

The queue is the only thing they share. That is the whole trick.

Message queues

It absorbs the spike

When jobs arrive faster than workers can finish them, the backlog grows. The queue lets workers catch up later, provided there is enough capacity and storage; it does not make the work disappear.

Message queues · in depth

What happens when a job fails

In a queue configured for acknowledgement and retries, a job that is not acknowledged can become available again after a timeout. Retry limits and dead-letter handling keep repeatedly failing jobs from cycling forever. Durability depends on the queue's configuration.

Nothing is lost when a worker crashes. The job was never removed.

Message queues · in depth

The same job can arrive twice

With at-least-once delivery, a crash after completing work but before acknowledgement can cause another delivery. Use a stable job ID and a durable deduplication check, or make the operation idempotent: repeating it has the same effect as doing it once.

Queues promise at least once, not exactly once. Plan for the duplicate.

Message queues · in depth

One event, several listeners

A submission-created event might trigger grading, an email, and analytics. Publish/subscribe can give each listener its own copy. A work queue with competing workers instead assigns a job to one worker at a time; those are different delivery patterns.

Adding a fourth listener changes nothing about the code that fired the event.

Message queues

When not to use one

For a quick operation whose result the user needs immediately, handling it directly may be simpler. A queue adds infrastructure, retries, monitoring, and a period when work is accepted but unfinished.

A queue buys you time, and charges you a second moving part. Only pay when you need the time.

The catch

Every cache lies a little

Choose a freshness policy for the data: a five-minute-old leaderboard may be acceptable, while a purchase must use authoritative inventory and payment state. A cache's speed benefit comes with expiration and invalidation decisions.

Speed is bought with freshness. Somebody has to choose the price.

15 minutes · whiteboard, no code

Now design the leaderboard

1. Define the browser's request and the minimum fields in the response.
2. Choose the server's permission checks and the database query.
3. Identify useful indexes and where a cached answer could live.
4. Choose how stale the ranking may be after a new submission.
5. Compare ten students with ten million: what gets slow first, and which change addresses it?

What does the browser ask for? What does the server do? What if 10 students — then 10 million?

Keep these four

If everything else fades

1. Follow the request: names, addresses, transport, HTTP, then the response.
2. Share work across interchangeable servers and keep needed state accessible to them.
3. Avoid repeated work and long trips with queries, indexes, and caches.
4. Move slow work to queues when it can finish later, and plan for retries and duplicates.