17 min read · Days 76–90 · Notion
Goal: Think and design at staff-engineer level. Build complete production systems from scratch. Understand observability, reliability mathematics, and multi-region architecture. This is the phase where you become dangerous.
At this level, the job is no longer "write code that works." It's "design systems that are correct under failure, observable when they degrade, and scalable to 10x current load without a rewrite." Every decision carries a latency cost, an operational burden, and a blast radius. You must be able to articulate all three.
traceparent header across service boundariesuser_id as a metric label kills Prometheuspprof: CPU profile, memory profile, goroutine dump, block profilego test -bench, wrk, vegeta. Measure under realistic load, not microbenchmarks.htm (method), htu (URL), iat (timestamp), jti (unique ID)Stack: OpenTelemetry Go SDK, Jaeger, Prometheus, Loki, Grafana
Instrument your Phase 2 Go service end-to-end. Every HTTP handler and gRPC method emits: (1) a trace span with custom attributes, (2) a request duration histogram, (3) a structured log with trace ID. Export to Jaeger (traces), Prometheus (metrics), Loki (logs). Build a Grafana dashboard: request rate, p99 latency, error rate, active goroutines. Write an SLO alert: page if error budget burn rate exceeds 5x for 1 hour.
Deliverable: A single Grafana dashboard where you can click a spike in the error rate chart → drill into the trace → see the exact log line that caused the error. Zero context switching between tools.
Stack: Go, Redis, JWT, circuit breaker (custom implementation)
Reverse proxy that handles: (1) JWT validation on every request — reject 401 before forwarding, (2) rate limiting per user ID in Redis (sliding window), (3) request logging with trace IDs injected as headers, (4) circuit breaker per upstream: track error rate in a 10-second window, open circuit at 50% error rate, half-open after 30 seconds. Route config loaded from a YAML file without restart (file watcher).
Deliverable: Gateway handles 5,000 req/sec on a single core. Circuit breaker opens when you intentionally kill an upstream. Dashboard shows the circuit state transitions.
Stack: Go, WebSocket (gorilla/websocket), Kafka, Redis pub/sub
Architecture: Producer publishes events to Kafka. Consumer reads from Kafka and publishes to Redis pub/sub channels per user. WebSocket server subscribes to Redis channel for each connected client. Handle: graceful reconnect (client sends last received event ID, server replays missed events from Kafka), backpressure (drop messages to slow clients after buffer fills), presence (Redis sorted set with heartbeat TTL).
Deliverable: 1,000 concurrent WebSocket connections. Kill a WebSocket server instance — clients reconnect to another instance and receive all missed messages with no duplicates.
❌ Using average latency as your SLI.
A service with average 50ms response time can have a p99 of 4 seconds if 1% of requests hit a slow path. 1 in 100 users is experiencing a 4-second load time. The average hides this completely.
✅ Mental model: Always measure and alert on percentiles: p95 for “typical” bad experience, p99 for “tail” bad experience, p999 for “worse case”. Averages are for capacity planning only, never for reliability signals.
❌ Active-active multi-region without a conflict resolution strategy.
Two regions accepting writes to the same entity simultaneously means write conflicts. Without explicit conflict resolution, you'll silently lose data or serve inconsistent state.
✅ Mental model: Define your conflict resolution model before designing active-active. Options: last-writer-wins (simple, loses data), vector clocks (tracks causality), CRDTs (merge-friendly data structures), or avoid it entirely with region-affinity routing (route each user to their home region).
❌ Treating observability as logging only.
Logs answer “what happened.” They don’t tell you why your p99 spiked, which downstream call is slow, or how a request flowed through 8 microservices. You need all three pillars.
✅ Mental model: Use traces to find WHERE latency is. Use metrics to see HOW OFTEN it happens. Use logs to understand WHAT specifically failed. The debugging workflow is: metrics alert → trace for context → logs for root cause.
❌ Cache invalidation via TTL only.
TTL causes thundering herd: when TTL expires for a popular key, thousands of concurrent requests all miss the cache simultaneously and all query the database. The DB collapses under the spike.
✅ Mental model: Use Probabilistic Early Recomputation (PER): each cache read has a small probability of triggering a background refresh as TTL approaches. This spreads recomputation over time. Or use stale-while-revalidate: return stale data immediately and refresh async.
Google SRE: Invented the SLO/error budget framework documented in the SRE book. Their rule is iron-clad: if the error budget is depleted, all feature work freezes until reliability is restored. This creates a shared incentive between product and infrastructure.
Cloudflare: Their Workers platform uses a distributed cache backed by Durable Objects with consistent hashing across 300+ PoPs. Cache invalidation is broadcast via the Durable Objects messaging system — not TTL-based. This is how they achieve near-instant cache purge globally.
Discord: Real-time presence system handles 500M+ events per day. They shard by guild ID — all members of a guild connect to the same process, making fan-out local instead of distributed. The lesson: smart sharding strategy can eliminate an entire class of distributed systems problems.
Uber: Their multi-region architecture for payments uses region-affinity: each driver and rider is assigned a home region. Payments involving cross-region parties go through an arbitration service. This avoids active-active write conflicts entirely by making conflict impossible through design.
After 90 days, you should be able to:
Next steps:
Core mental model: Observability is the ability to understand system behavior from external signals. Logs, metrics, and traces answer different questions and become powerful when correlated.
Logs: Record discrete events. Use structured JSON, stable field names, request IDs, trace IDs, user/org identifiers where safe, and clear error fields. Avoid high-volume noisy logs and secrets.
Metrics: Measure behavior over time. Counters only increase, gauges go up/down, histograms capture distributions. Use labels carefully because high cardinality can break metric stores.
Traces: Show request flow through services. Spans represent units of work. Parent-child relationships reveal where latency is spent.
OpenTelemetry: Vendor-neutral instrumentation standard. Apps emit traces/metrics/logs through SDKs and exporters, often via the OpenTelemetry Collector.
Sampling: Head sampling decides at request start. Tail sampling decides after seeing the whole trace and can retain errors or slow requests more intelligently.
Practice: Instrument one handler, one outbound HTTP call, and one database query. Confirm the same trace ID appears in logs and spans.
Core mental model: Reliability is a product decision expressed through measurements and error budgets.
SLI: What you measure, such as request success rate, p99 latency, or freshness.
SLO: Target for the SLI over a time window, such as 99.9% successful requests over 30 days.
SLA: Contractual promise with external consequences.
Error budget: Allowed unreliability. A 99.9% monthly availability SLO allows about 43.8 minutes of bad time per 30 days.
Burn rate: How quickly the service is consuming budget. Multi-window alerts catch both fast outages and slow degradation.
Practice: Define SLIs for an API, background worker, streaming system, and cache. Not all systems should use the same SLI.
Core mental model: Users feel tail latency. Averages hide pain. Capacity and latency are connected by queueing behavior.
Percentiles: p50 is typical, p95 is common bad experience, p99 is tail pain, and p999 reveals rare but severe failures. Alert on percentiles, not averages.
Little’s Law: L = lambda * W. In service terms, concurrency equals throughput multiplied by latency. If latency rises under fixed throughput, in-flight work grows.
Tail amplification: End-to-end reliability drops as services are chained. Ten dependencies each at 99% success can produce much worse total success.
Profiling: Use pprof CPU, heap, goroutine, mutex, and block profiles. Flame graphs show where time or allocations concentrate.
Benchmarking: Use realistic data, concurrency, payloads, and dependencies. Microbenchmarks alone do not prove production performance.
Practice: Load test an endpoint with wrk or vegeta, capture p99, then profile while the test runs.
Core mental model: Multi-region design trades latency, availability, consistency, cost, and operational complexity. Active-active is not a default; it is a commitment.
Active-passive: One primary region serves writes. Secondary waits for failover. Simpler consistency, slower recovery.
Active-active: Multiple regions serve traffic. Improves locality and availability but requires conflict resolution, routing, and careful data design.
Replication: Async replication is faster but can lose recent writes during failover. Sync replication reduces loss but increases latency.
Data locality: Store and process data near users when latency or regulation demands it. GDPR and data residency can shape architecture.
Conflict resolution: Last-writer-wins is simple but can lose data. Vector clocks track causality. CRDTs allow mergeable state. Region affinity can avoid many conflicts.
Practice: Design multi-region for profile reads, payment writes, chat messages, and analytics events. Choose active-passive or active-active deliberately.
Core mental model: An API gateway is the controlled entry point to a service graph. It should protect services without becoming an unmaintainable central bottleneck.
Responsibilities: TLS termination, auth verification, routing, rate limiting, request transformation, response transformation, tracing, logging, and circuit breaking.
Rate limiting: Local in-memory limits are fast but approximate. Redis-backed global limits are more accurate but add dependency and latency.
Circuit breakers: Track failures per upstream. Closed means normal traffic. Open means fail fast. Half-open probes recovery.
Service discovery: Gateways should use live service registry data or dynamic config rather than hardcoded endpoints.
Plugin order: Authentication should happen before expensive transformations. Rate limits may be per anonymous IP before auth and per identity after auth.
Practice: Sketch the request path through gateway plugins and define what happens when auth, rate limit, or upstream call fails.
Core mental model: A secure auth server is a key-management, token-lifecycle, replay-defense, and policy-enforcement system.
Issuance pipeline: Validate client and user, enforce grant rules, bind access token to DPoP public key, issue refresh token, log security-relevant events.
JWKS: Publish public signing keys so resource servers can validate JWTs without calling the auth server. Keep old public keys until old tokens expire.
Key rotation: Introduce new signing key, publish it, start signing new tokens, keep old keys for validation, then retire old keys after expiry windows.
DPoP validation: Resource server checks proof signature, method, URL, timestamp, unique jti, nonce if required, and key binding through the token cnf claim.
Revocation: Use short-lived access tokens, stateful refresh tokens, rotation, blocklists for emergency access-token revocation, and audit logs.
Practice: Write the validation checklist for one incoming DPoP request and identify every rejection condition.
Core mental model: Real-time systems are about fan-out, ordering, backpressure, reconnection, and replay.
Transport choice: Long polling is simple but inefficient. SSE is excellent for one-way server push. WebSockets provide full-duplex communication but require connection management and careful load balancing.
Fan-out: Push fan-out writes to each subscriber queue at publish time. Pull fan-out lets subscribers read from a shared stream. Choose based on read/write ratio and subscriber count.
Presence: Use heartbeat TTLs and last-seen timestamps. Presence is usually eventually consistent; design UI accordingly.
Backpressure: Slow clients must not block the whole server. Use bounded buffers, drop policies, disconnect thresholds, or replay-based recovery.
Reconnect and replay: Clients should send last received event ID. Server replays missed events and deduplicates on client or server side.
Practice: Design what happens when a WebSocket server dies while a user has unread messages in flight.
Core mental model: Caches trade freshness and complexity for latency and load reduction. A cache is part of correctness once users depend on it.
Consistent hashing: Maps keys to nodes so adding/removing a node moves only a fraction of keys. Virtual nodes improve distribution.
Invalidation: TTL is simple but creates stale windows and stampedes. Event-driven invalidation is fresher but more complex. Write-through and write-behind change write latency and durability risk.
Stampede control: Use request coalescing, locks, stale-while-revalidate, jittered TTLs, or probabilistic early recomputation.
Multi-tier caching: L1 in-process cache is fastest but local and inconsistent. L2 Redis is shared. L3 database is source of truth.
Practice: Design cache strategy for product details, user permissions, feature flags, and timeline feeds. Define freshness requirements for each.
pprof to find CPU, memory, goroutine, and lock issues.