Vertical and horizontal scaling
Vertical scaling (scale up) means a bigger machine: more CPU, memory, faster disks. It's simple, needs no code changes, and is often the right first step. But there's a ceiling, big machines get expensive fast, and one machine is still a single point of failure.
Horizontal scaling (scale out) means more machines behind a load balancer. It has no hard ceiling and survives individual failures, but the application must be designed for it: any server must be able to handle any request.
| Vertical | Horizontal | |
|---|---|---|
| Limit | Largest machine you can buy | Practically none |
| Failure | Single point of failure | Survives losing a node |
| Complexity | Low | Load balancing, shared state, coordination |
| Cost curve | Grows faster than linearly | Roughly linear with commodity machines |
Stateless services
To add servers freely, web and application servers should be stateless: they keep nothing between requests that another server would need. Session data, uploads and caches move to shared stores (a database, Redis, object storage), so the load balancer can send each request anywhere.
- Sticky sessions (pinning a user to one server) work, but break when that server dies and make load uneven.
- Store sessions in a shared cache, or use signed tokens (such as JWTs) that carry the session with each request.
- Put user files in object storage (like S3), not on a server's disk.
From one server to many
- One server runs the app and the database.
- Move the database to its own machine, so each can be sized separately.
- Put a load balancer in front of several stateless app servers.
- Add a cache for hot reads and a CDN for static files.
- Add database read replicas; send reads to replicas, writes to the primary.
- Move slow work (emails, thumbnails) to a message queue and background workers.
- When one database can't take the writes or the data, shard it.
Each step fixes the current bottleneck. In an interview, describe which bottleneck you're addressing rather than adding every box at once.
Latency, throughput and percentiles
Latency is how long one request takes; throughput is how many requests the system handles per second. They're related but different: batching raises throughput and can raise latency.
Report latency as percentiles, not averages. A p99 of 800 ms means 1 in 100 requests takes at least 800 ms. A page that makes 50 backend calls will hit a p99 call on most loads, so tail latency matters more than the mean.
See it animated
Interview questions
Try answering out loud first, then open the model answer.
Q1When would you scale vertically rather than horizontally?beginner
Early on, or for components that are hard to distribute (a relational database primary). It's cheaper in engineering time and keeps the system simple. Switch to horizontal when you approach the hardware ceiling, need fault tolerance, or the cost curve turns against you.
Q2Why should application servers be stateless?beginner
So any server can handle any request: the load balancer can spread load evenly, you can add or remove servers at any time, and a crash loses no user state. State lives in shared stores built for it.
Q3What is p99 latency and why does it matter?intermediate
The latency that 99% of requests beat. Averages hide slow outliers, and when one user action fans out to many backend calls the chance of hitting at least one slow call is high, so the tail becomes the experience. Systems are usually tuned against p95/p99 targets.
Q4What is a single point of failure and how do you remove it?beginner
A component whose failure takes down the whole system: one database, one load balancer. Remove it with redundancy: multiple instances, replicas with automatic failover, a pair of load balancers with a floating IP, and deployment across availability zones.
Quick check
0 of 3 answered