4. Reliability Patterns

Patterns that keep a system up and fair when traffic spikes or parts fail. Back to Fundamentals; pieces used here are in Building Blocks.

Rate limiting

A rate limiter caps how many requests a client can make in a period. It protects against abuse and bots, stops one noisy user starving the rest, and keeps cost down on paid third-party calls.

flowchart LR
    C[Client] --> RL{Rate limiter middleware}
    RL -- under limit --> API[API servers]
    RL -- over limit --> R429["429 Too Many Requests Retry-After: 30"]
    RL <--> RD[(Redis counters)]

• Do it server-side or in an API gateway, never only in the client (clients can be forged).
• Counters live in Redis (in-memory, fast): INCR to count, EXPIRE to reset.
• Reply 429 with headers: X-Ratelimit-Limit, X-Ratelimit-Remaining, Retry-After.
• Rules are config, e.g. login: 5 per minute per IP, marketing messages: 5 per day per user.
• One rule can have different keys: per user, per IP, per endpoint, or one global bucket.

Algorithms compared

flowchart LR
    R[Refiller +2 tokens / s] --> B(["Bucket capacity 4"])
    Q[Request] --> B
    B -- token available --> OK[Allowed]
    B -- empty --> NO[Dropped]

Token bucket in code

import time

class TokenBucket:
    def __init__(self, capacity, refill_per_sec):
        self.capacity, self.rate = capacity, refill_per_sec
        self.tokens, self.last = capacity, time.monotonic()

    def allow(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

Sliding window counter worked example (limit 7 per minute)

$ previous minute = 5 requests, current minute = 3, we are 30% into the current minute
$ estimate = 3 + 5 x (1 - 0.3) = 6.5   -> rounds to 6, under 7, allow

Fixed window problem: limit 5 per minute, 5 requests at 2:00:50 and 5 more at 2:01:10. Both windows are fine, yet 10 requests landed in 20 seconds.

Distributed gotchas

• Race condition: two servers read the counter, both see 4, both write 5. Fix with an atomic Redis command or a Lua script, not read-then-write.
• Many limiter nodes: keep counters in one shared Redis (or sync between nodes) instead of per-server memory.

$ redis-cli INCR rl:user42:2026-10-29T10:15
$ redis-cli EXPIRE rl:user42:2026-10-29T10:15 60

Retries and backoff

• Retry only idempotent operations (GET, or a POST carrying an idempotency key), otherwise a retry can charge a card twice.
• Exponential backoff with jitter: wait 1s, 2s, 4s... plus a random amount, so clients do not all retry together (a retry storm).

delay = min(cap, base * 2 ** attempt)
time.sleep(random.uniform(0, delay))     # "full jitter"

Circuit breaker

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failures exceed threshold
    Open --> HalfOpen: after timeout
    HalfOpen --> Closed: trial request succeeds
    HalfOpen --> Open: trial request fails

• Closed: calls pass through. Open: fail immediately, giving the broken service room to recover. Half-open: let one trial call through.
• Prevents cascading failure, where one slow dependency ties up every thread in its callers.

Other safeguards