16 min read · Days 56–75 · Notion
Goal: Learn to take a production Go service and deploy it reliably at scale — containers, orchestration, IaC, secret management, and zero-downtime deployments. This is where backend engineering meets systems engineering.
A service that works on your laptop is not a product. Infrastructure is the bridge between code and reliability. Every decision here — L4 vs L7, Vault vs env vars, blue-green vs canary — has a direct cost in either operational complexity, latency, or blast radius when things go wrong.
/healthz vs /readyz — liveness vs readiness. Different semantics.proxy_pass, upstream blocks, connection keepaliveRetry-After, exponential backoff, why naive retries cause thundering herdCOPY go.mod go.sum ./ then RUN go mod download before COPY . .requests for scheduling decisions, limits for enforcementlivenessProbe (restart if unhealthy), readinessProbe (remove from load balancer if unready)terraform plan before apply. Review every change.terraform refresh.@sha256:...), not tags, for securityStack: Kubernetes (minikube/kind), Helm, Docker, GitHub Actions
Write a Helm chart for your Phase 2 Go service. Configure resource requests and limits. Add liveness and readiness probes. Configure HPA to scale at 70% CPU. Write a GitHub Actions pipeline: test → build image → push to registry → helm upgrade. Test pod disruption budget: run kubectl delete pod during a load test and verify zero dropped requests.
Deliverable: Fully automated deploy from git push. kubectl get hpa shows autoscaling in action during a wrk load test.
Stack: Terraform, AWS (VPC, ALB, ECS, RDS, ElastiCache, Secrets Manager)
Provision from zero: VPC with public/private subnets across 2 AZs. NAT Gateway for private subnets. ALB in public subnet. ECS Fargate service in private subnet. RDS PostgreSQL in private subnet. ElastiCache Redis in private subnet. Secrets Manager for DB credentials. Zero manual AWS console interactions.
Deliverable: terraform apply provisions the entire stack. terraform destroy tears it down cleanly. State stored in S3 with DynamoDB locking.
Stack: GitHub Actions, Kubernetes, ArgoCD, Prometheus
Pipeline: build → push → deploy 10% canary with a separate Deployment and weighted Service. Monitor error rate via Prometheus for 10 minutes. If error rate > 1%, automatically roll back by scaling canary to 0. If healthy, promote by scaling canary to 100% and scaling down old version.
Deliverable: Introduce a deliberate bug in a new image version. Pipeline deploys canary, detects the error rate spike, and rolls back — fully automated, zero human intervention.
❌ Running stateful workloads (databases) inside Kubernetes.
StatefulSets are complex. Persistent volumes are tricky. Storage class behavior varies by cloud. A database crash in a pod means complex recovery. The operational overhead is massive.
✅ Mental model: Use managed services for state: RDS for PostgreSQL, ElastiCache for Redis, MSK for Kafka. Keep Kubernetes for stateless application workloads. Let the cloud provider handle storage durability.
❌ Passing secrets as environment variables through CI pipelines.
Env vars appear in kubectl describe pod output, process dumps, crash reports, and CI logs. They're inherited by child processes. A leaked env var is a leaked secret.
✅ Mental model: Use Vault Agent sidecar or AWS Secrets Manager + IAM roles for service accounts (IRSA). Secrets are mounted as files in a tmpfs volume, not environment variables. Rotate them automatically with short TTLs.
❌ Using ALB (L7) for gRPC services with long-lived bidirectional streams.
ALB terminates HTTP/2 connections and enforces its own timeout. Long-lived gRPC streams get cut. ALB also doesn't support gRPC load balancing at the request level — it load balances at the connection level.
✅ Mental model: Use NLB (L4) in front of your gRPC services. Or use a service mesh (Istio) which understands gRPC at the application level and can load balance individual streams.
❌ No circuit breaker on downstream service calls.
One slow database or downstream API causes request threads to pile up. Connection pool exhausts. Memory grows. Your service is now also slow. This cascades until the entire call graph is down.
✅ Mental model: Every external call needs a timeout + circuit breaker. When the downstream is degraded, fail fast (return 503) instead of waiting. This preserves your service's capacity for other requests.
HashiCorp Vault: Dynamic secrets are their killer feature — Vault generates a fresh PostgreSQL credential per service instance, auto-expired in 1 hour. No long-lived database passwords anywhere in the fleet. Breaching one service's credentials expires in 60 minutes with no action required.
AWS: ALB + NLB in tandem is how AWS internally routes at scale. NLB handles TCP at line rate (millions of packets/sec), forwards to ALB for HTTP/2 routing decisions. This is the architecture behind API Gateway under the hood.
Netflix: Invented the canary deploy pattern and Chaos Monkey simultaneously. Their deployment pipeline automatically rolls back if error rate spikes >0.1% in the canary cohort. Every deploy is a chaos experiment.
Cloudflare: Zero-trust networking across their entire internal fleet using SPIFFE-based certificates. Every internal service-to-service call is mTLS. Network location (being inside the VPN) grants zero implicit trust.
Core mental model: Load balancers distribute traffic, but different layers understand different things. L4 sees connections and packets. L7 understands application protocols such as HTTP.
L4: Routes by IP and port. It is fast, protocol-agnostic, and useful for raw TCP, UDP, and some gRPC workloads. It usually preserves more end-to-end behavior but cannot route by HTTP path or header.
L7: Terminates or understands HTTP. It can route by host, path, header, method, cookie, or auth context. It can also terminate TLS, inject headers, and apply request-level policies.
Health checks: Liveness means the process should be restarted if broken. Readiness means the instance should receive traffic. Mixing these causes bad deploy behavior.
Connection draining: Before removing an instance, the load balancer should stop sending new requests and allow in-flight requests to finish.
Practice: Design routing for REST, gRPC unary, and long-lived gRPC streams. Decide when ALB, NLB, or service mesh is appropriate.
Core mental model: A reverse proxy forwards traffic to backends. An API gateway adds policy: auth, rate limiting, transformation, routing, and observability.
Nginx: Event-driven workers handle many connections efficiently. Common uses include TLS termination, static assets, reverse proxying, buffering, and simple routing.
Envoy: A modern data-plane proxy built for dynamic configuration. xDS lets control planes update clusters, routes, listeners, and endpoints without restarts.
Gateway responsibilities: Validate identity, enforce quotas, route traffic, normalize headers, add trace IDs, transform requests/responses, and protect downstream services.
Retries: Use bounded retries with jitter and respect Retry-After. Never retry unsafe non-idempotent operations blindly.
Circuit breaking: Fail fast when downstreams are unhealthy to preserve capacity and prevent cascading failure.
Core mental model: A VPC is your private cloud network boundary. Subnets, routes, gateways, security groups, and NACLs define how traffic moves.
Public vs private subnets: Public subnets have a route to an internet gateway. Private subnets do not accept direct inbound internet traffic and usually use NAT for outbound connections.
Security groups: Stateful instance-level firewall rules. If inbound is allowed, return traffic is automatically allowed.
NACLs: Stateless subnet-level rules. You must handle inbound and outbound explicitly.
NAT Gateway: Allows private resources to initiate outbound internet connections, such as package downloads or external API calls.
PrivateLink: Exposes a service privately across VPC boundaries without full network peering.
Practice: Draw a two-AZ VPC with public ALB, private app subnets, private database subnets, NAT gateways, and route tables.
Core mental model: Containers are isolated processes, not lightweight virtual machines. Isolation comes from kernel features.
Namespaces: Separate views of process IDs, mounts, networking, hostnames, and IPC.
cgroups: Limit and account for CPU, memory, and I/O usage. Kubernetes resource limits eventually map to cgroups.
Images and layers: Images are layered filesystems. Each Dockerfile instruction can add a layer. Good Dockerfiles maximize cache reuse and minimize final image size.
Multi-stage builds: Compile in a builder image, then copy only the final binary into a smaller runtime image.
Distroless: Reduces attack surface by excluding shells and package managers. Debugging requires good logs and external tools.
Practice: Build a Go service image with a multi-stage Dockerfile and compare image sizes between full, alpine, and distroless variants.
Core mental model: Kubernetes continuously reconciles desired state against actual state. You declare what should exist; controllers work to make it true.
Pod: Smallest deployable unit. Containers in a pod share network namespace and can communicate over localhost.
Deployment: Manages ReplicaSets and rolling updates for stateless workloads.
Service: Stable virtual endpoint for pods. ClusterIP is internal, NodePort exposes a node port, LoadBalancer asks the cloud for an external load balancer.
Ingress: HTTP routing layer implemented by an ingress controller.
ConfigMap vs Secret: ConfigMaps hold non-sensitive config. Kubernetes Secrets are base64 by default and need encryption-at-rest plus careful access controls.
Requests and limits: Requests influence scheduling. Limits enforce maximum usage and can cause throttling or OOM kills.
Practice: Deploy a service with readiness/liveness probes, resource requests, and HPA. Break readiness and observe traffic removal.
Core mental model: Infrastructure as Code gives repeatability, reviewability, and drift detection, but state becomes critical infrastructure.
State: Terraform state maps config to real resources. Store it remotely with locking. Treat it as sensitive because it can contain secrets.
Plan/apply: plan shows intended changes. Read it carefully, especially destroys, replacements, IAM changes, and networking changes.
Modules: Reusable building blocks. Version modules and providers so changes are intentional.
Drift: Manual console edits create drift. Terraform may undo them or fail unexpectedly.
Import: Existing resources can be brought under Terraform, but imported state still needs matching configuration.
Practice: Create a VPC module and consume it from dev/staging/prod with separate state.
Core mental model: CI proves a change is safe enough to merge. CD moves a verified artifact through environments with controlled risk.
Pipeline stages: Test, lint, build, scan, publish artifact, deploy to dev, promote to staging, approve/progress to prod.
Artifact immutability: Promote the same image digest through environments. Do not rebuild separately for prod.
Secrets: Avoid long-lived cloud keys in CI. Prefer OIDC federation to cloud IAM where possible.
Caching: Cache dependencies and Docker layers to reduce build time, but invalidate safely.
Rollback: A rollback plan should be part of the pipeline, not an emergency improvisation.
Core mental model: Deploy strategy controls blast radius. Releasing safely is about progressive exposure and fast rollback.
Rolling: Replaces instances gradually. Efficient and common, but rollback may take time.
Blue-green: Two full environments. Switch traffic quickly. Easier rollback, higher cost.
Canary: Send a small percentage of traffic to the new version, measure health, then increase traffic gradually.
Feature flags: Separate deploy from release. They also need ownership, cleanup, and auditability.
Database migrations: Use expand-contract: add backward-compatible schema first, deploy code that uses it, then remove old schema later.
Practice: Write a deploy plan for a breaking database column rename without downtime.
Core mental model: Secrets should be short-lived, auditable, access-controlled, and rotated. Static secrets copied into environments create long-lived blast radius.
Dynamic secrets: Vault can create database credentials on demand with TTLs. Leaked credentials expire automatically.
Transit engine: Applications send plaintext to Vault and receive ciphertext, without ever owning encryption keys directly.
PKI engine: Vault can issue short-lived certificates for internal TLS and mTLS.
Auth methods: AppRole, Kubernetes auth, and cloud IAM auth let workloads authenticate without embedding human credentials.
Vault Agent: Sidecar or daemon that renews tokens and writes secrets to files for the app.
Practice: Design secret flow for a Kubernetes Go service accessing PostgreSQL with rotating credentials.
Core mental model: mTLS authenticates both ends of a connection. It changes internal networking from location-based trust to identity-based trust.
Certificates: Each workload presents a client certificate and validates the server certificate. Trust depends on CA roots and certificate identity fields.
SPIFFE/SPIRE: Provides workload identity in a standard format such as spiffe://trust-domain/ns/default/sa/api.
Service mesh: Sidecar proxies can handle mTLS transparently, but add operational complexity and latency.
Rotation: Short-lived certs reduce compromise window but require automated issuance and renewal.
Practice: Explain why security groups alone do not prove which workload made a request.