Project 04: Kubernetes Autoscaling & the EFK Observability Stack

The mission: stop paying for idle pods and stop debugging production by kubectl logs-ing one pod at a time. Two builds, one outcome: a cluster that scales itself and explains itself.

#Architecture

          ┌─────────────── K8S CLUSTER ───────────────┐
 traffic ─▶ ingress ─▶ [ pods ×N ] ◀── HPA (CPU/RPS)  │
          │               │  ▲                        │
          │               │  └────── VPA (requests)   │
          │          stdout/stderr                    │
          │               ▼                           │
          │        [ fluentd DaemonSet ]              │
          └───────────────┼───────────────────────────┘
                          ▼
                 [ elasticsearch ] ──▶ [ kibana ]
                   (retention ILM)      (dashboards
                                         + alerts)

#Step 1 — HPA on a Signal Users Feel

CPU-only autoscaling lies for I/O-bound services. Scale on requests-per-second alongside CPU:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 40
  metrics:
    - type: Pods
      pods:
        metric: { name: http_requests_per_second }
        target: { type: AverageValue, averageValue: "80" }
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 65 }
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # no flapping

#Step 2 — VPA in Recommendation Mode First

VPA in Auto mode restarts pods to resize them. Run it in Off (recommend-only) for two weeks, then set requests from its data:

1
2
3
4
5
6

kubectl get vpa api-vpa -o jsonpath='{.status.recommendation.containerRecommendations[0]}' | jq
# {
#   "target":     { "cpu": "210m", "memory": "340Mi" },
#   "upperBound": { "cpu": "540m", "memory": "512Mi" }
# }
# Old request: cpu 1000m / mem 1Gi  →  we were paying 4x for nothing

#Step 3 — EFK: One Query Instead of Forty Pods

Fluentd runs as a DaemonSet, tails every container, enriches with pod metadata, ships to Elasticsearch:

fluentd:
  configMapConfigs:
    - fluentd-prometheus-conf
  fileConfigs:
    01_sources.conf: |
      <source>
        @type tail
        path /var/log/containers/*.log
        <parse> @type cri </parse>
      </source>
    02_enrich.conf: |
      <filter kubernetes.**>
        @type kubernetes_metadata   # namespace, pod, labels
      </filter>

Retention is an ILM policy — hot 7 days, warm 30, delete at 90. Log storage stopped being a surprise line item.

#Results