High Availability & Fault Tolerance

High availability means the system keeps serving requests even when components fail. The target is expressed in nines: 99.9% uptime allows ~8.7 hours of downtime per year; 99.99% allows ~52 minutes. Achieving higher nines requires eliminating every single point of failure — each one becomes the ceiling on your overall availability.

Circuit breakers stop cascading failures. When a downstream service starts timing out, the circuit breaker opens and returns an error or fallback immediately instead of waiting and holding threads. After a cooldown it probes again and closes if the service recovers. Retries with exponential backoff and jitter handle transient failures without thundering-herd storms. Together these patterns isolate component failures so they stay local rather than cascading into system-wide outages.

Before

No resilience — one slow service takes down the API

1async function getRecommendations(userId: string) {2  // If recommendation service hangs for 30s,3  // this holds a thread for 30s.4  // Under load: all threads exhausted → entire API down.5  return recommendationService.get(userId);6}

After

Circuit breaker sketch with fallback (pseudo-code)

1const breaker = new CircuitBreaker(recommendationService.get, {2  timeout: 3000,                 // fail fast after 3s3  errorThresholdPercentage: 50,  // open after 50% errors4  resetTimeout: 10000,           // probe again after 10s5});6 7async function getRecommendations(userId: string) {8  try {9    return await breaker.fire(userId);10  } catch {11    // Fallback: return popular items — always available12    return redis.get('popular:recommendations');13  }14}

Exercise

Pick one non-critical dependency. Write a circuit-breaker policy (timeout, error threshold, reset) and a fallback response. State what the user sees when the circuit is open.