Monitoring & Observability

Core Workflow: Observability Implementation

Use this decision tree to determine your starting point:

Are you setting up monitoring from scratch?

├─ YES → Start with "1. Design Metrics Strategy"

└─ NO → Do you have an existing issue?

    ├─ YES → Go to "9. Troubleshooting & Analysis"

    └─ NO → Are you improving existing monitoring?

        ├─ Alerts → Go to "3. Alert Design"

        ├─ Dashboards → Go to "4. Dashboard & Visualization"

        ├─ SLOs → Go to "5. SLO & Error Budgets"

        ├─ Tool selection → Read references/tool_comparison.md

        └─ Using Datadog? High costs? → Go to "7. Datadog Cost Optimization & Migration"

1. Design Metrics Strategy

Start with The Four Golden Signals

Every service should monitor: Latency (p50/p95/p99), Traffic (req/s), Errors (failure rate), Saturation (resource utilization).

RED Method (request-driven): Rate, Errors, Duration | USE Method (infrastructure): Utilization, Saturation, Errors

Quick Start - Web Application Example:

# Rate (requests/sec)

sum(rate(http_requests_total[5m]))

# Errors (error rate %)

sum(rate(http_requests_total{status=~"5.."}[5m]))

  /

sum(rate(http_requests_total[5m])) * 100

# Duration (p95 latency)

histogram_quantile(0.95,

  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)

)

Deep dive: references/metrics_design.md — metric types, cardinality, naming conventions, dashboard design

Querying Metrics Directly

Query Prometheus and CloudWatch metrics using CLI/curl:

# Query Prometheus instant value

curl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total[5m])' | jq .

# Query Prometheus over a time range (last 1h, 15s step)

curl -s 'http://localhost:9090/api/v1/query_range?query=rate(http_requests_total[5m])&start='$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)'&end='$(date -u +%Y-%m-%dT%H:%M:%SZ)'&step=15s' | jq .

# Query CloudWatch metrics (e.g., EC2 CPU over last 1 hour)

aws cloudwatch get-metric-statistics \

  --namespace AWS/EC2 \

  --metric-name CPUUtilization \

  --dimensions Name=InstanceId,Value=i-1234567890abcdef0 \

  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ) \

  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \

  --period 300 \

  --statistics Average Maximum

2. Log Aggregation & Analysis

Structured Logging Checklist

Every log entry should include: timestamp (ISO 8601), log level, message, service name, request ID (for tracing).

Example structured log (JSON):

{

  "timestamp": "2024-10-28T14:32:15Z",

  "level": "error",

  "message": "Payment processing failed",

  "service": "payment-service",

  "request_id": "550e8400-e29b-41d4-a716-446655440000",

  "user_id": "user123",

  "order_id": "ORD-456",

  "error_type": "GatewayTimeout",

  "duration_ms": 5000

}

Log Analysis

Analyze logs for errors, patterns, and anomalies:

# Analyze log file for patterns

python3 scripts/log_analyzer.py application.log

# Show error lines with context

python3 scripts/log_analyzer.py application.log --show-errors

# Extract stack traces

python3 scripts/log_analyzer.py application.log --show-traces

→ Script: scripts/log_analyzer.py

Deep dive: references/logging_guide.md — structured logging, aggregation patterns (ELK, Loki, CloudWatch), PII redaction, sampling

3. Alert Design

Alert Design Principles

• Actionable - If you can't act on it, don't alert

• Symptom-based - Alert on user experience, not components

• SLO-tied - Connect to business impact

• Low noise - Only page for critical issues

Alert Severity Levels

Multi-Window Burn Rate Alerting

Alert when error budget is consumed too quickly:

# Fast burn (1h window) - Critical

- alert: ErrorBudgetFastBurn

  expr: |

    (error_rate / 0.001) > 14.4  # 99.9% SLO

  for: 2m

  labels:

    severity: critical

# Slow burn (6h window) - Warning

- alert: ErrorBudgetSlowBurn

  expr: |

    (error_rate / 0.001) > 6  # 99.9% SLO

  for: 30m

  labels:

    severity: warning

Alert Quality Checker

Audit your alert rules against best practices:

# Check single file

python3 scripts/alert_quality_checker.py alerts.yml

# Check all rules in directory

python3 scripts/alert_quality_checker.py /path/to/prometheus/rules/

Checks: naming conventions, required labels/annotations, PromQL quality, 'for' clause to prevent flapping.

→ Script: scripts/alert_quality_checker.py

Alert Templates

Production-ready alert rule templates:

→ Templates:

• assets/templates/prometheus-alerts/webapp-alerts.yml - Web application alerts

• assets/templates/prometheus-alerts/kubernetes-alerts.yml - Kubernetes alerts

Deep dive: references/alerting_best_practices.md — alert design patterns, routing, inhibition rules, runbook structure, on-call practices

Runbook Template

Create comprehensive runbooks for your alerts:

→ Template: assets/templates/runbooks/incident-runbook-template.md

4. Dashboard & Visualization

Dashboard Design Principles

• Top-down layout: Most important metrics first

• Color coding: Red (critical), yellow (warning), green (healthy)

• Consistent time windows: All panels use same time range

• Limit panels: 8-12 panels per dashboard maximum

• Include context: Show related metrics together

Recommended layout: Overall Health (single stats) -> Request Rate & Errors -> Latency Distribution -> Resource Usage

Generate Grafana Dashboards

Automatically generate dashboards from templates:

# Web application dashboard

python3 scripts/dashboard_generator.py webapp \

  --title "My API Dashboard" \

  --service my_api \

  --output dashboard.json

# Kubernetes dashboard

python3 scripts/dashboard_generator.py kubernetes \

  --title "K8s Production" \

  --namespace production \

  --output k8s-dashboard.json

# Database dashboard

python3 scripts/dashboard_generator.py database \

  --title "PostgreSQL" \

  --db-type postgres \

  --instance db.example.com:5432 \

  --output db-dashboard.json

Supports: Web applications, Kubernetes, Databases (PostgreSQL, MySQL).

→ Script: scripts/dashboard_generator.py

5. SLO & Error Budgets

SLO Fundamentals

SLI (measurement), SLO (target), Error Budget (allowed failure = 100% - SLO). See references/slo_sla_guide.md for full definitions.

Common SLO Targets

SLO Calculator

Calculate compliance, error budgets, and burn rates:

# Show SLO reference table

python3 scripts/slo_calculator.py --table

# Calculate availability SLO

python3 scripts/slo_calculator.py availability \

  --slo 99.9 \

  --total-requests 1000000 \

  --failed-requests 1500 \

  --period-days 30

# Calculate burn rate

python3 scripts/slo_calculator.py burn-rate \

  --slo 99.9 \

  --errors 50 \

  --requests 10000 \

  --window-hours 1

→ Script: scripts/slo_calculator.py

Deep dive: references/slo_sla_guide.md — choosing SLIs, setting targets, error budget policies, burn rate alerting, SLA contracts

6. Distributed Tracing

When to Use Tracing

Use distributed tracing when you need to:

• Debug performance issues across services

• Understand request flow through microservices

• Identify bottlenecks in distributed systems

• Find N+1 query problems

OpenTelemetry Implementation

Python example:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@tracer.start_as_current_span("process_order")

def process_order(order_id):

    span = trace.get_current_span()

    span.set_attribute("order.id", order_id)

    try:

        result = payment_service.charge(order_id)

        span.set_attribute("payment.status", "success")

        return result

    except Exception as e:

        span.set_status(trace.Status(trace.StatusCode.ERROR))

        span.record_exception(e)

        raise

Sampling Strategies

• Development: 100% (ALWAYS_ON)

• Staging: 50-100%

• Production: 1-10% (or error-based sampling)

Error-based sampling (always sample errors, 1% of successes):

class ErrorSampler(Sampler):

    def should_sample(self, parent_context, trace_id, name, **kwargs):

        attributes = kwargs.get('attributes', {})

        if attributes.get('error', False):

            return Decision.RECORD_AND_SAMPLE

        if trace_id & 0xFF < 3:  # ~1%

            return Decision.RECORD_AND_SAMPLE

        return Decision.DROP

OTel Collector Configuration

→ Template: assets/templates/otel-config/collector-config.yaml — OTLP/Prometheus/host metrics receivers, batching, tail sampling, multiple exporters (Tempo, Jaeger, Loki, Prometheus, CloudWatch, Datadog)

Deep dive: references/tracing_guide.md — OTel instrumentation (Python, Node.js, Go, Java), context propagation, backend comparison, analysis patterns

7. Datadog Cost Optimization & Migration

Scenario 1: I'm Using Datadog and Costs Are Too High

Check usage directly in the Datadog UI at Plan & Usage > Usage Summary, or query the API:

# Get Datadog usage summary for the current month (requires DD_API_KEY and DD_APP_KEY)

curl -s -X GET "https://api.datadoghq.com/api/v1/usage/summary?start_month=$(date -u +%Y-%m)" \

  -H "DD-API-KEY: ${DD_API_KEY}" \

  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" | jq .

# Get hourly usage for hosts (identify peak usage)

curl -s -X GET "https://api.datadoghq.com/api/v1/usage/hosts?start_hr=$(date -u -v-24H +%Y-%m-%dT%H)" \

  -H "DD-API-KEY: ${DD_API_KEY}" \

  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" | jq .

#### Common Cost Optimization Strategies

• Custom Metrics (20-40% savings): Remove high-cardinality tags, delete unused metrics, aggregate before sending

• Log Management (30-50% savings): Sample high-volume services, exclude debug logs in prod, archive to S3 after 7 days

• APM (15-25% savings): Reduce trace sampling (10% to 5%), remove APM from non-critical services

• Infrastructure (10-20% savings): Use container-based pricing, remove agents from ephemeral instances, consolidate staging

Scenario 2: Migrating Away from Datadog

Migration to open-source stack (Prometheus + Grafana, Loki, Tempo/Jaeger, Alertmanager). Estimated savings: 60-77%.

#### Migration Strategy

• Phase 1 (Month 1-2): Deploy open-source stack in parallel, migrate metrics first, validate accuracy

• Phase 2 (Month 2-3): Convert dashboards to Grafana, translate alert rules (DQL to PromQL), train team

• Phase 3 (Month 3-4): Set up Loki for logs, deploy Tempo/Jaeger for traces, update instrumentation

• Phase 4 (Month 4-5): Confirm all functionality migrated, decommission Datadog

Full migration guide: references/datadog_migration.md | Query translation: references/dql_promql_translation.md

8. Tool Selection & Comparison

Deep dive: references/tool_comparison.md — full comparison of metrics, logging, tracing, and full-stack platforms

9. Troubleshooting & Analysis

Health Check Validation

Validate health check endpoints using curl:

# Check endpoint returns 200 and measure response time

curl -sf -o /dev/null -w "status:%{http_code} time:%{time_total}s\n" https://api.example.com/health

# Get full response body (check for JSON 'status' field, version info)

curl -sf https://api.example.com/health | jq .

# Check multiple endpoints in sequence

for ep in /health /readiness /liveness; do

  printf "%-20s " "$ep"

  curl -sf -o /dev/null -w "status:%{http_code} time:%{time_total}s\n" "https://api.example.com${ep}"

done

What to verify: 200 status, response time < 1s, JSON format with status field, version/build info, dependency checks, no caching headers (Cache-Control: no-cache).

Common Troubleshooting Workflows

High Latency: Check dashboards for spike -> query traces for slow ops -> check DB slow queries -> check external APIs -> review deployments -> check resource utilization

High Error Rate: Check error logs -> identify affected endpoints -> check dependency health -> review deployments -> check resource limits -> verify configuration

Service Down: Check pods/instances running -> check health endpoint -> review deployments -> check resource availability -> check network -> review startup logs

Quick Reference Commands

Prometheus Queries

# Request rate

sum(rate(http_requests_total[5m]))

# Error rate

sum(rate(http_requests_total{status=~"5.."}[5m]))

  /

sum(rate(http_requests_total[5m])) * 100

# P95 latency

histogram_quantile(0.95,

  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)

)

# CPU usage

100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage

(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

See references/quick_commands.md for Kubernetes, Elasticsearch, Loki, and CloudWatch query references.

Resources Summary

Scripts: alert_quality_checker.py | dashboard_generator.py | log_analyzer.py | slo_calculator.py

References: metrics_design.md | alerting_best_practices.md | logging_guide.md | tracing_guide.md | slo_sla_guide.md | tool_comparison.md | datadog_migration.md | dql_promql_translation.md

Templates: prometheus-alerts/webapp-alerts.yml | prometheus-alerts/kubernetes-alerts.yml | otel-config/collector-config.yaml | runbooks/incident-runbook-template.md