Observability di Distributed Systems

Di monolith, debugging = buka satu log file. Di distributed system, request melintasi 10+ service. Observability adalah kemampuan memahami internal state sistem dari external output.

Three Pillars + Context

• Metrics — Angka aggregated: error rate, latency p99, throughput. "Apakah ada masalah?"
• Logs — Event records dengan context. "Apa yang terjadi?"
• Traces — Request flow across services. "Di mana masalahnya?"
• + Correlation — Menghubungkan ketiganya via trace ID

RED Method (untuk request-driven services)

Rate    — Requests per second
Errors  — Error rate (% of failed requests)
Duration — Latency distribution (p50, p95, p99)

// Alert jika:
// - Error rate > 1% selama 5 menit
// - P99 latency > 2s selama 5 menit
// - Request rate drop > 50% dari normal

USE Method (untuk resources)

Utilization — % resource being used (CPU 80%)
Saturation  — Queue depth (10 pending connections)
Errors      — Error count (5 disk errors)

// Untuk setiap resource: CPU, memory, disk, network

Service Level Objectives

// Define SLOs per service
Order Service:
  - Availability: 99.95% (22 menit downtime/bulan)
  - Latency: p95 < 200ms, p99 < 500ms

Payment Service:
  - Availability: 99.99% (4.3 menit downtime/bulan)
  - Latency: p95 < 1s
  - Correctness: 100% (payment tidak boleh salah)

Dashboard Hierarchy

• Level 1: Business — Revenue per menit, conversion rate, active users
• Level 2: Service — RED metrics per service, error breakdown
• Level 3: Infrastructure — CPU, memory, disk, network per node
• Level 4: Debug — Trace view, individual request deep-dive

🎭 Analogi sehari-hari

Dokter periksa pasien. Metrics = tanda vital (suhu, tensi, denyut) — angka cepat lihat ada masalah. Logs = catatan dokter ("pasien batuk sejak hari Jumat") — context detail. Traces = MRI scan — lihat di organ mana masalahnya. Tiga-tiganya wajib untuk diagnosis lengkap. Cuma metrics? Tahu sakit tapi tidak tahu di mana. Cuma logs? Banyak text tapi gak tahu severity. Cuma traces? Lihat satu organ tapi miss systemic issue.

💡 Observability bukan Monitoring

• Monitoring = predefined dashboards untuk known problems. "CPU naik? Alert."
• Observability = ability ask new questions tentang system tanpa deploy ulang. "Kenapa user X di region Y dapat error tapi user Z gak?"

Modern systems perlu observability karena failure modes tidak predictable. High-cardinality data (user_id, session_id, region) = critical untuk debug.

⚠️ Jebakan klasik

• Cuma alert pada CPU/memory = miss business issue (revenue drop, conversion turun)
• SLO terlalu ketat (99.999%) = chase ghost, dev resources habis untuk last 0.1%
• Log everything tanpa structure = haystack tanpa needle. Pakai structured JSON logs
• Sampling 100% di production = storage cost meledak. Sample 1-10%, head-based atau tail-based
• No correlation ID = log dari 10 service tidak bisa di-link untuk satu request
• Alert fatigue = ratusan alert harian = team mute semua = miss real incident

🎯 Observability stack default

1. Tracing — OpenTelemetry SDK + Jaeger / Tempo / Honeycomb
2. Metrics — Prometheus + Grafana
3. Logs — Loki / ELK / CloudWatch with structured JSON
4. Correlation — trace ID propagation via W3C Trace Context header
5. SLO + error budget — define SLO, alert kalau burn rate tinggi (Google SRE book)
6. Runbook per alert — apa yang harus dilakukan saat alert fire

TL;DR: Observability = ability ask new questions tentang system production. Three pillars: metrics (apakah ada masalah), logs (apa yang terjadi), traces (di mana masalahnya). Wajib correlation ID. RED untuk service, USE untuk infrastructure, SLO untuk reliability target. OpenTelemetry standard. Cardinality matters.