Availability in numbers
| Availability | Downtime per year | Downtime per month |
|---|---|---|
| 99% (two nines) | ~3.65 days | ~7.2 hours |
| 99.9% | ~8.8 hours | ~43 minutes |
| 99.99% | ~53 minutes | ~4.3 minutes |
| 99.999% | ~5 minutes | ~26 seconds |
Components in series multiply: two 99.9% services that both must work give 99.8%. Components in parallel (redundant copies) fail only if all fail: two independent 99% copies give 1 − 0.01² = 99.99%.
An SLI is a measurement (fraction of requests under 300 ms), an SLO is the target for it (99.9% over 30 days), and an SLA is the contract with consequences if you miss it. The gap between 100% and the SLO is your error budget: spend it on releases and experiments.
Resilience patterns
- Timeouts on every network call. Without them, one slow dependency ties up all your threads.
- Retries with exponential backoff and jitter, only for idempotent operations, with a cap. Synchronized retries can turn a blip into an outage.
- Circuit breaker: after repeated failures, stop calling the dependency for a while and fail fast; probe occasionally (half-open) and close when it recovers.
- Bulkheads: separate thread pools or connection pools per dependency, so one failure can't exhaust everything.
- Graceful degradation: serve a cached or simplified response (hide recommendations) instead of failing the page.
- Load shedding: reject excess requests early with 503 rather than slowing down for everyone.
// retry with exponential backoff and full jitterfor (int attempt = 0; attempt < 5; attempt++) {try { return client.call(); }catch (TransientException e) {long cap = Math.min(10_000, 100L << attempt); // 100, 200, 400 … msThread.sleep(ThreadLocalRandom.current().nextLong(cap)); // random in [0, cap)}}throw new UnavailableException();
Observability
- Metrics: numbers over time. The four golden signals: latency, traffic, errors, saturation.
- Logs: structured events with request ids, for investigating specific failures.
- Traces: one request's path across services with timings, to find where latency comes from.
- Alerts on symptoms users feel (error rate, latency SLO burn), not on every CPU spike.
Interview questions
Try answering out loud first, then open the model answer.
Q1What's the availability of three services in series, each 99.9%?intermediate
Multiply: 0.999³ ≈ 0.997, about 99.7%, or roughly 26 hours of downtime a year. Every hard dependency lowers availability, which is why critical paths are kept short and redundant.
Q2How does a circuit breaker work?intermediate
It wraps calls to a dependency. Closed: calls pass through and failures are counted. When failures cross a threshold it opens: calls fail immediately for a cool-down period, giving the dependency room to recover and freeing your threads. Then it goes half-open, lets a few trial calls through, and closes if they succeed.
Q3Why add jitter to retries?intermediate
Without it, every client that failed at the same moment retries at the same moments, producing synchronized waves that keep the service overloaded. Randomizing the delay spreads the retries out.
Q4What's the difference between an SLO and an SLA?beginner
An SLO is an internal reliability target for a measured indicator (99.9% of requests succeed). An SLA is an external promise to customers, usually looser than the SLO, with penalties such as credits if it's broken.
Q5How would you find why one API got slow?advanced
Check dashboards for when it started and whether it correlates with a deploy, traffic change or dependency. Use distributed traces of slow requests to see which span grew. Then drill into that component's metrics (saturation, GC, connection pools, slow queries) and logs.
Quick check
0 of 3 answered