Redis fully down
e.g. crash, OOM, network partition |
Rails' built-in cache error handler swallows the Redis error and returns nil → Rails.cache.write(..., unless_exist: true) returns nil → LockAndFetchRate falls into the follower branch → poll_for_result sees nil for 6s → raises LockAndFetchRate::TimeoutError |
504 "Request timed out waiting for upstream result" |
0 calls — no one acquires the lock, so upstream is never reached |
Service recovers immediately when Redis comes back online (verified by the Fail-closed when Redis dies test in Load & Chaos Tests) |
Redis restart
e.g. deploy, OOM, container restart |
AOF replays everything back — cache, lock, counter — with their original expiry times. Stale lock self-expires within 7s (same as leader crash); cache stays inside the 5-min freshness window; counter resumes counting. Up to ~1s of writes may be lost (appendfsync everysec default). |
Brief unavailability during restart (a few seconds); reads resume normally after AOF replay |
0 calls — cache and counter both survive; nothing extra to fetch |
Self-healing via AOF + persistent volume. The 1000/day cap holds across restarts, not just within a single uptime. |
Leader crashes mid-fetch
e.g. kill -9, OOM, container restart |
ensure never runs → lock stays held until 7s auto-expire → followers wait then time out at 6s. Each leader holds a unique UUID (owner id, not a fencing token), so a slow leader waking up after expiry won't accidentally delete the new leader's lock. |
504 "Request timed out waiting for upstream result" |
≤1 call — 0 if leader crashed before calling upstream, 1 if it crashed during or after |
Wait up to 7s for the lock to auto-expire (that's the lock TTL). Then the next request takes over as new leader. |
Upstream timeout
5s HTTPParty timeout (~7%) |
RateApiClient catches Net::ReadTimeout, raises TimeoutError → lock released |
504 "Upstream timed out" |
1 call — quota consumed for the failed attempt |
Next request becomes new leader and retries upstream. |
Upstream HTTP 5xx
~7% rate observed in live testing |
RateApiClient sees non-200 status, raises UpstreamError → lock released, cache not written |
502 "Upstream error" |
1 call — quota consumed for the failed attempt |
Next request becomes new leader and retries upstream. |
Upstream returns 200 with error body
~7% rate observed in live testing |
RateApiClient parses body, sees no rate field, raises UpstreamError → lock released, cache not written |
502 "Upstream error" |
1 call burned — but we deliberately don't cache the bad response. Next request retries, no stale garbage stuck in cache. |
Next request becomes new leader and retries upstream. |
Upstream returns garbage values
unknown hotel, zero/negative rate, decimal strings, missing fields |
RateApiClient runs each record through a dry-schema + positive-numeric check. Bad records are logged and skipped; the rest still get cached. |
Service still returns the requested rate if it survived validation, otherwise 502 |
1 call — partial response still better than zero |
One bad row doesn't deny cache to 35 good rows. If every record is invalid the whole response is rejected. |
Upstream 401 (auth failure)
e.g. token rotation needed |
RateApiClient raises UnauthorizedError → PricingService catches → refunds the quota unit (the call was rejected before processing) |
502 "Upstream authentication failed" |
1 call but quota refunded — QuotaGuard.refund_quota! ensures bad tokens don't burn budget |
Operator must rotate token. Until then, every cache miss costs 1 quota then refunds — budget stays intact. |
Upstream HTTP 429 (their rate limit)
~0.1% observed — upstream's daily limit hit |
RateApiClient raises QuotaExhaustedError → PricingService catches |
429 + Retry-After header |
1 call — rejected call still counts upstream-side |
Wait until upstream's window resets (assumed UTC midnight, see D4). Our QuotaGuard catches the same condition locally before the call — this row is the rare race where our counter and upstream's diverge. |
Quota counter at 1000
budget exhausted for the day |
QuotaGuard's atomic INCR returns 1001 → raises ExhaustedError before RateApiClient is called |
429 + Retry-After header (seconds until UTC midnight) |
0 calls — the deterministic guarantee that 1000 is never exceeded (verified by the Quota hard cap test in Load & Chaos Tests) |
Counter auto-resets at UTC midnight via EXPIREAT. Users hitting cache still get their rate — only fresh fetches get blocked. |