Progress · 0/10 sections
- 01 — L4 vs L7 Load Balancing (Days 56–57)
- 02 — Reverse Proxies & API Gateways (Days 58–59)
- 03 — VPC & Cloud Networking (Days 60–61)
- 04 — Docker Internals (Days 62–63)
- 05 — Kubernetes Core Concepts (Days 64–65)
- 06 — Terraform & IaC (Days 66–67)
- 07 — CI/CD Pipelines (Days 68–69)
- 08 — Deployment Strategies (Days 70–71)
- 09 — Vault & Secret Management (Days 72–73)
- 10 — mTLS & Production Security (Days 74–75)
09 — Vault & Secret Management (Days 72–73)
9 min read · Days 72–73
09 — Vault & Secret Management (Days 72–73)
Core Mental Model: Secrets infrastructure mein hain, code mein nahi. Dynamic secrets static secrets se better hain — auto-expire hoti hain, per-service hoti hain, compromise hone pe blast radius limited hota hai. Vault = central secret authority.
The Problem With Static Secrets
Environment variable approach (❌ BAD):
DB_PASSWORD=super_secret_password
Problems:
1. Rotation pain: change karo → redeploy all services
2. Over-sharing: 50 microservices → all share SAME DB password
One service compromised → all DBs exposed
3. Drift: password last rotate kab hua? Unknown.
4. Audit trail: koi service ne password use kiya → no visibility
5. Env vars leak: process list, logs, crash dumps mein appear ho sakte hain
Vault dynamic secrets approach (✅ GOOD):
Service A requests DB credentials → Vault creates unique creds for A
Service B requests DB credentials → Vault creates unique creds for B
Creds expire in 1 hour automatically
Service A compromised → revoke A's creds only → Service B unaffected
Audit log: Service A ne creds request ki 14:32 pe, use ki 14:33 pe
Vault Architecture
┌─────────────────────────────────────────────────────────────┐
│ HashiCorp Vault │
│ │
│ Auth Methods Secret Engines Audit Logs │
│ ┌──────────┐ ┌─────────────┐ ┌──────────┐ │
│ │ AppRole │ │ KV v2 │ │ File │ │
│ │ K8s │ │ (static) │ │ Syslog │ │
│ │ AWS IAM │ ├─────────────┤ └──────────┘ │
│ │ OIDC │ │ Database │ │
│ └──────────┘ │ (dynamic) │ Policies │
│ ├─────────────┤ ┌──────────┐ │
│ Storage Backend │ Transit │ │ HCL ACLs │ │
│ ┌──────────┐ │ (encryption)│ └──────────┘ │
│ │ Raft │ ├─────────────┤ │
│ │ (HA) │ │ PKI │ │
│ └──────────┘ │ (certs) │ │
│ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
Auth Methods: "Who are you?" Verify karta hai
Secret Engines: "Kya chahiye?" Secret create/retrieve karta hai
Policies: "Kya access hai?" Control karta hai
Audit: "Kya hua?" Log karta hai
KV v2 — Static Secrets
# Simple key-value secret storage
# Version history, soft delete support
# Write
vault kv put secret/user-service/production \
db_url="postgres://user:pass@rds.amazonaws.com:5432/users" \
jwt_secret="$(openssl rand -hex 32)"
# Read
vault kv get secret/user-service/production
# Get specific field
vault kv get -field=db_url secret/user-service/production
# Version history
vault kv get -version=2 secret/user-service/production
# Policy (HCL)
path "secret/data/user-service/production" {
capabilities = ["read"]
}
# user-service sirf apna secret read kar sakti hai, dusre ki nahiDynamic Secrets — Database Engine
How it works:
1. Vault se pehle karo: DB admin credentials configure
2. Service → Vault: "Mujhe user-service ke liye DB creds chahiye"
3. Vault → DB: "CREATE USER vault-user-service-abc123 WITH PASSWORD 'random'"
"GRANT SELECT, INSERT, UPDATE ON users_schema TO vault-user-service-abc123"
4. Vault → Service: username=vault-user-service-abc123, password=..., TTL=1h
5. Service: DB connection karo with these creds
6. After 1h: Vault → DB: "DROP USER vault-user-service-abc123"
(or on service explicit revoke)
# Vault setup (admin karta hai once)
vault secrets enable database
vault write database/config/users-db \
plugin_name=postgresql-database-plugin \
allowed_roles="user-service,order-service" \
connection_url="postgresql://{{username}}:{{password}}@rds.amazonaws.com:5432/users" \
username="vault-admin" \
password="admin-password"
# Role define karo (kya creds banaenge)
vault write database/roles/user-service \
db_name=users-db \
creation_statements="
CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";
" \
default_ttl="1h" \
max_ttl="24h"
# Service se dynamic creds request karo
vault read database/creds/user-service
# Output:
# Key Value
# --- -----
# lease_id database/creds/user-service/xyz123
# lease_duration 1h
# username v-approl-user-ser-RANDOM
# password A1a-RANDOM-RANDOM
# Revoke before expiry (service stop/compromise)
vault lease revoke database/creds/user-service/xyz123Transit Engine — Encryption as a Service
App ko encryption keys manage nahi karni padtein.
App Vault ko plaintext bhejta hai, ciphertext wapas milta hai.
Keys kabhi app ke paas nahi aatein.
Use cases:
- PII encryption (Aadhaar, phone numbers, email IDs)
- Payment card data encryption before DB store
- Sensitive config fields encryption
Mental model:
App → Vault: "Encrypt this: 9876543210"
Vault: AES-256-GCM se encrypt karta hai (key app ko nahi pata)
Vault → App: "vault:v1:ENCRYPTED_BASE64_DATA"
App: stores in DB: users.phone = "vault:v1:ENCRYPTED_BASE64_DATA"
Decrypt:
App → Vault: "Decrypt: vault:v1:ENCRYPTED_BASE64_DATA"
Vault → App: "9876543210"
Key rotation:
Vault admin: vault write transit/keys/user-phone/rotate
New ciphertext uses new key version (v2)
Old data (v1) still decryptable until re-encrypted
Vault admin: vault write transit/keys/user-phone/rewrap_minimum_version=2
App: re-encrypt old v1 data (gradually)
// Go Transit engine usage
package encryption
import (
"encoding/base64"
vault "github.com/hashicorp/vault/api"
)
type VaultEncryptor struct {
client *vault.Client
keyName string
}
func (e *VaultEncryptor) Encrypt(plaintext string) (string, error) {
encoded := base64.StdEncoding.EncodeToString([]byte(plaintext))
secret, err := e.client.Logical().Write(
"transit/encrypt/"+e.keyName,
map[string]interface{}{
"plaintext": encoded,
},
)
if err != nil {
return "", fmt.Errorf("vault encrypt: %w", err)
}
ciphertext, ok := secret.Data["ciphertext"].(string)
if !ok {
return "", errors.New("vault: invalid ciphertext response")
}
return ciphertext, nil // "vault:v1:ABC123..."
}
func (e *VaultEncryptor) Decrypt(ciphertext string) (string, error) {
secret, err := e.client.Logical().Write(
"transit/decrypt/"+e.keyName,
map[string]interface{}{
"ciphertext": ciphertext,
},
)
if err != nil {
return "", fmt.Errorf("vault decrypt: %w", err)
}
encoded := secret.Data["plaintext"].(string)
decoded, err := base64.StdEncoding.DecodeString(encoded)
if err != nil {
return "", fmt.Errorf("base64 decode: %w", err)
}
return string(decoded), nil
}PKI Engine — Internal Certificate Authority
Problem: Internal services mein TLS certificates kaise manage karein?
Public CA (Let's Encrypt) = internet-facing only
Manual cert management = painful, error-prone, rotation forgotten
Vault PKI = Internal CA
Vault → issues certs for internal services
Short-lived (24h TTL) → auto-rotation → no "forgot to renew" incidents
SPIFFE-compatible → workload identity (next file mein)
Benefits:
- On-demand cert issuance (no waiting, no tickets)
- Short TTL → compromise window small
- Automatic rotation via Vault Agent
- Audit log: kab koi cert issue hua
# PKI engine setup
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki
# Root CA generate karo
vault write pki/root/generate/internal \
common_name="company.internal" \
ttl=87600h # 10 years (root CA)
# Intermediate CA (better practice)
vault secrets enable -path=pki_int pki
vault write pki_int/intermediate/generate/internal \
common_name="company Intermediate CA"
# Certificate role
vault write pki_int/roles/user-service \
allowed_domains="user-service.default.svc.cluster.local" \
allow_subdomains=true \
max_ttl="24h" # certificates valid for 24 hours only
# Issue certificate (service karta hai)
vault write pki_int/issue/user-service \
common_name="user-service.default.svc.cluster.local" \
ttl=24h
# Returns: certificate, private_key, issuing_caAppRole — Service Authentication
Question: Service Vault se authenticate kaise kare?
"Chicken and egg: secret access ke liye secret chahiye"
AppRole:
Two components:
1. RoleID: public (like username), static, per-application
2. SecretID: private (like password), short-lived, rotate frequently
Both together → Vault Token → Secret access
SecretID delivery (secure):
CI/CD pipeline: deploy karte waqt SecretID inject karo (one-time use)
Vault Agent: handle karta hai automatically in K8s
Trusted orchestrator (K8s SA): K8s auth better option
# Setup AppRole for user-service
vault auth enable approle
vault write auth/approle/role/user-service \
token_policies="user-service-policy" \
token_ttl=1h \
token_max_ttl=4h \
secret_id_ttl=24h \ # SecretID expires in 24h
secret_id_num_uses=0 # Unlimited uses within TTL
# Get RoleID (permanent, non-secret)
vault read auth/approle/role/user-service/role-id
# role_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# Generate SecretID (per deployment)
vault write -f auth/approle/role/user-service/secret-id
# secret_id: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy
# secret_id_ttl: 24h
# Login
vault write auth/approle/login \
role_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
secret_id="yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy"
# Returns: token (1h valid)Vault Agent — K8s Sidecar Pattern
Problem: App code mein Vault logic nahi hona chahiye.
Vault token renewal, secret rotation, file updates = boilerplate.
Vault Agent:
- Sidecar container in K8s Pod
- App se pehle start hota hai (init container style)
- Vault se authenticate karta hai (K8s Service Account se)
- Secrets tmpfs volume pe write karta hai (memory-only, no disk)
- Token auto-renewal karta hai
- Secrets change hone pe template re-render karta hai
App: simply file read karta hai /vault/secrets/config.env
Vault logic zero in app code!
# K8s Pod with Vault Agent sidecar
apiVersion: v1
kind: ServiceAccount
metadata:
name: user-service
annotations:
vault.hashicorp.com/role: "user-service" # Vault role name
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-service
spec:
template:
metadata:
annotations:
# Vault Agent Injector automatically injects sidecar
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "user-service"
# Which secret, as what file
vault.hashicorp.com/agent-inject-secret-config.env: "secret/data/user-service/production"
# Template: secret ko env file format mein
vault.hashicorp.com/agent-inject-template-config.env: |
{{- with secret "secret/data/user-service/production" -}}
export DB_URL="{{ .Data.data.db_url }}"
export JWT_SECRET="{{ .Data.data.jwt_secret }}"
export REDIS_URL="{{ .Data.data.redis_url }}"
{{- end }}
spec:
serviceAccountName: user-service
containers:
- name: user-service
image: user-service:abc1234
command: ["sh", "-c", "source /vault/secrets/config.env && exec /server"]
# OR: read file programmatically in Go// Go: read secrets from Vault Agent rendered file
func loadSecrets() (*Config, error) {
// Vault Agent ne /vault/secrets/db-creds.json write kiya hai
data, err := os.ReadFile("/vault/secrets/db-creds.json")
if err != nil {
return nil, fmt.Errorf("read vault secrets: %w", err)
}
var creds struct {
Username string `json:"username"`
Password string `json:"password"`
}
if err := json.Unmarshal(data, &creds); err != nil {
return nil, fmt.Errorf("parse vault secrets: %w", err)
}
return &Config{
DBUser: creds.Username,
DBPassword: creds.Password,
}, nil
}Why Environment Variables for Secrets Are Wrong
❌ Never do this:
DB_PASSWORD=super_secret
JWT_KEY=another_secret
Reasons:
1. Process listing:
ps aux → shows env vars in old Linux kernels / some debugging tools
/proc/<pid>/environ → readable by root (and some exploits)
2. Logging libraries: accidentally log request context (which contains env vars)
structuredlog.Info("request", "env", os.Environ()) → oops
3. Core dumps: crash dump contains env vars
4. Child processes: env vars inherited by all child processes
One child logs env → leaked
5. Container inspection:
docker inspect container_name → shows env vars!
kubectl describe pod user-service → env vars visible to K8s admins
✅ Correct approach:
- Vault Agent: secrets as files in tmpfs (/vault/secrets/)
- External Secrets Operator: K8s Secret (with etcd encryption)
- AWS SDK: IAM role → Secrets Manager direct API call at startup
Application reads from file or API, not env vars for sensitive data.