Alerting: Reaksi Otomatis terhadap Masalah

Monitoring tanpa alerting hanya dashboard yang tidak pernah dilihat. Alerting memberi notifikasi saat metric melewati threshold tertentu.

Aturan Alert yang Baik

• Actionable — Setiap alert harus punya aksi yang jelas
• Tidak terlalu sensitif — Jangan alert pada spike sesaat
• Tidak terlalu longgar — Jangan sampai masalah besar tidak terdeteksi
• Prioritas jelas — Bedakan critical vs warning

Contoh Alert Rules

# Grafana Alert / Prometheus Alertmanager

# Critical: Error rate > 5% selama 5 menit
- alert: HighErrorRate
  expr: rate(http_requests_total{status=~"5.."}[5m]) /
        rate(http_requests_total[5m]) > 0.05
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Error rate above 5%"

# Warning: Response time p95 > 2 detik
- alert: SlowResponses
  expr: histogram_quantile(0.95, rate(http_response_time_bucket[5m])) > 2
  for: 10m
  labels:
    severity: warning

On-Call & Incident Response

Alert Triggered → PagerDuty/Opsgenie → On-Call Engineer
                                              │
                                    1. Acknowledge alert
                                    2. Assess severity
                                    3. Mitigate (rollback?)
                                    4. Root cause analysis
                                    5. Post-mortem document

Severity Levels

• SEV1 (Critical) — Service down, semua user terdampak. Page immediately.
• SEV2 (Major) — Fitur utama rusak, sebagian user terdampak. Page during business hours.
• SEV3 (Minor) — Bug yang tidak blocking. Fix in next sprint.
• SEV4 (Low) — Cosmetic issue. Fix when convenient.