SKILL.md

APM Service Health

Name: observability-service-health
Author: elastic

Assess APM service health using Observability APIs,

ES|QL against APM indices, Elasticsearch APIs, and (for correlation and APM-specific logic) the Kibana repo. Use

SLOs, firing alerts, ML anomalies, throughput, latency (avg/p95/p99), error rate, and dependency health.

Where to look

Observability APIs (Observability APIs): Use the

SLOs API (Stack |

Serverless) to get SLO definitions, status, burn

rate, and error budget. Use the Alerting API

(Stack |

Serverless) to list and manage alerting

rules and their alerts for the service. Use APM annotations API to create or search annotations when needed.

ES|QL and Elasticsearch: Query traces*apm*,traces*otel* and metrics*apm*,metrics*otel* with ES|QL (see

[Using ES|QL for APM metrics](#using-esql-for-apm-metrics)) for throughput, latency, error rate, and dependency-style

aggregations. Use Elasticsearch APIs (e.g. POST _query for ES|QL, or Query DSL) as documented in the Elasticsearch

repo for indices and search.

APM Correlations: Run the apm-correlations script to get attributes that correlate with high-latency or failed

transactions for a given service. It tries the Kibana internal APM correlations API first, then falls back to

Elasticsearch significant_terms on traces*apm*,traces*otel*. See

[APM Correlations script](#apm-correlations-script).

Infrastructure: Correlate via resource attributes (e.g. k8s.pod.name, container.id, host.name) in

traces; query infrastructure or metrics indices with ES|QL/Elasticsearch for CPU and memory. OOM and **CPU

throttling** directly impact APM health.

Logs: Use ES|QL or Elasticsearch search on log indices filtered by service.name or trace.id to explain

behavior and root cause.

Observability Labs: Observability Labs and

APM tag for patterns and troubleshooting.

Health criteria

Synthesize health from all of the following when available:

Signal

What to check

SLOs

Burn rate, status (healthy/degrading/violated), error budget.

Firing alerts

Open or recently fired alerts for the service or dependencies.

ML anomalies

Anomaly jobs; score and severity for latency, throughput, or error rate.

Throughput

Request rate; compare to baseline or previous period.

Latency

Avg, p95, p99; compare to SLO targets or history.

Error rate

Failed/total requests; spikes or sustained elevation.

Dependency health

Downstream latency, error rate, availability (ES|QL, APIs, Kibana repo).

Infrastructure

CPU usage, memory; OOM and CPU throttling on pods/containers/hosts.

Logs

App logs filtered by service or trace ID for context and root cause.

Treat a service as unhealthy if SLOs are violated, critical alerts are firing, or ML anomalies indicate severe

degradation. Correlate with infrastructure (OOM, CPU throttling), dependencies, and logs (service/trace context) to

explain why and suggest next steps.

Using ES|QL for APM metrics

When querying APM data from Elasticsearch (traces*apm*,traces*otel*, metrics*apm*,metrics*otel*), use **ES|QL by

default** where available.

Availability: ES|QL is available in Elasticsearch 8.11+ (technical preview; GA in 8.14). It is **always

available** in

Elastic Observability Serverless Complete tier.

Scoping to a service: Always filter by service.name (and service.environment when relevant). Combine with a

time range on @timestamp:

WHERE service.name == "my-service-name" AND service.environment == "production"

  AND @timestamp >= "2025-03-01T00:00:00Z" AND @timestamp <= "2025-03-01T23:59:59Z"

Example patterns: Throughput, latency, and error rate over time: see Kibana trace_charts_definition.ts

(getThroughputChart, getLatencyChart, getErrorRateChart). Use from(index) → where(...) → stats(...) /

evaluate(...) with BUCKET(@timestamp, ...) and WHERE service.name == "<service_name>".

Performance: Add LIMIT n to cap rows and token usage. Prefer coarser BUCKET(@timestamp, ...) (e.g. 1 hour)

when only trends are needed; finer buckets increase work and result size.

APM Correlations script

When only a subpopulation of transactions has high latency or failures, run the apm-correlations script to list

attributes that correlate with those transactions (e.g. host, service version, pod, region). The script tries the Kibana

internal APM correlations API first; if unavailable (e.g. 404), it falls back to Elasticsearch significant_terms on

traces*apm*,traces*otel*.

# Latency correlations (attributes over-represented in slow transactions)

node skills/observability/service-health/scripts/apm-correlations.js latency-correlations --service-name <name> [--start <iso>] [--end <iso>] [--last-minutes 60] [--transaction-type <t>] [--transaction-name <n>] [--space <id>] [--json]

# Failed transaction correlations

node skills/observability/service-health/scripts/apm-correlations.js failed-correlations --service-name <name> [--start <iso>] [--end <iso>] [--last-minutes 60] [--transaction-type <t>] [--transaction-name <n>] [--space <id>] [--json]

# Test Kibana connection

node skills/observability/service-health/scripts/apm-correlations.js test [--space <id>]

Environment: KIBANA_URL and KIBANA_API_KEY (or KIBANA_USERNAME/KIBANA_PASSWORD) for Kibana; for fallback,

ELASTICSEARCH_URL and ELASTICSEARCH_API_KEY. Use the same time range as the investigation.

Workflow

Service health progress:

- [ ] Step 1: Identify the service (and time range)

- [ ] Step 2: Check SLOs and firing alerts

- [ ] Step 3: Check ML anomalies (if configured)

- [ ] Step 4: Review throughput, latency (avg/p95/p99), error rate

- [ ] Step 5: Assess dependency health (ES|QL/APIs / Kibana repo)

- [ ] Step 6: Correlate with infrastructure and logs

- [ ] Step 7: Summarize health and recommend actions

Step 1: Identify the service

Confirm service name and time range. Resolve the service from the request; if multiple are in scope, target the most

relevant. Use ES|QL on traces*apm*,traces*otel* or metrics*apm*,metrics*otel* (e.g.

WHERE service.name == "<name>") or Kibana repo APM routes to obtain service-level data. If the user has not provided

the time range, assume last hour.

Step 2: Check SLOs and firing alerts

SLOs: Call the SLOs API to get SLO definitions and status for the service (latency, availability),

healthy/degrading/violated, burn rate, error budget. Alerts: For active APM alerts, call

/api/alerting/rules/_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active.

When checking one service, include both rules where params.serviceName matches the service and rules where

params.serviceName is absent (all-services rules). Do not query .alerts* indices for active-state checks. Correlate

with SLO violations or metric changes.

Step 3: Check ML anomalies

If ML anomaly detection is used, query ML job results or anomaly records (via Elasticsearch ML APIs or indices) for the

service and time range. Note high-severity anomalies (latency, throughput, error rate); use anomaly time windows to

narrow Steps 4–5.

Step 4: Review throughput, latency, and error rate

Use ES|QL against traces*apm*,traces*otel* or metrics*apm*,metrics*otel* for the service and time range to get

throughput (e.g. req/min), latency (avg, p95, p99), error rate (failed/total or 5xx/total). Example:

FROM traces*apm*,traces*otel* | WHERE service.name == "<service_name>" AND @timestamp >= ... AND @timestamp <= ... | STATS ....

Compare to prior period or SLO targets. See [Using ES|QL for APM metrics](#using-esql-for-apm-metrics).

Step 5: Assess dependency health

Obtain dependency and service-map data via ES|QL on traces*apm*,traces*otel*/metrics*apm*,metrics*otel* (e.g.

downstream service/span aggregations) or via APM route handlers in the Kibana repo that expose

dependency/service-map data. For the service and time range, note downstream latency and error rate; flag slow or

failing dependencies as likely causes.

Step 6: Correlate with infrastructure and logs

APM Correlations (when only a subpopulation is affected): Run

node skills/observability/service-health/scripts/apm-correlations.js latency-correlations|failed-correlations --service-name <name> [--start ...] [--end ...]

to get correlated attributes. Filter by those attributes and fetch trace samples or errors to confirm root cause. See

[APM Correlations script](#apm-correlations-script).

Infrastructure: Use resource attributes from traces (e.g. k8s.pod.name, container.id, host.name) and

query infrastructure/metrics indices with ES|QL or Elasticsearch for CPU and memory. OOM and **CPU

throttling** directly impact APM health; correlate their time windows with APM degradation.

Logs: Use ES|QL or Elasticsearch on log indices with service.name == "<service_name>" or

trace.id == "<trace_id>" to explain behavior and root cause (exceptions, timeouts, restarts).

Step 7: Summarize and recommend

State health (healthy / degraded / unhealthy) with reasons; list concrete next steps.

Examples

Example: ES|QL for a specific service

Scope with WHERE service.name == "<service_name>" and time range. Throughput and error rate (1-hour buckets; LIMIT

caps rows and tokens):

FROM traces*apm*,traces*otel*

| WHERE service.name == "api-gateway"

  AND @timestamp >= "2025-03-01T00:00:00Z" AND @timestamp <= "2025-03-01T23:59:59Z"

| STATS request_count = COUNT(*), failures = COUNT(*) WHERE event.outcome == "failure" BY BUCKET(@timestamp, 1 hour)

| EVAL error_rate = failures / request_count

| SORT @timestamp

| LIMIT 500

Latency percentiles and exact field names: see Kibana trace_charts_definition.ts.

Example: "Is service X healthy?"

Resolve service X and time range. Call SLOs API and Alerting API; run ES|QL on

traces*apm*,traces*otel*/metrics*apm*,metrics*otel* for throughput, latency, error rate; query

dependency/service-map data (ES|QL or Kibana repo).

Evaluate SLO status (violated/degrading?), firing rules, ML anomalies, and dependency health.

Answer: Healthy / Degraded / Unhealthy with reasons and next steps (e.g.

Observability Labs).

Example: "Why is service Y slow?"

Service Y and slowness time range. Call SLOs API and Alerting API; run ES|QL for Y and dependencies;

query ML anomaly results.

Compare latency (avg/p95/p99) to prior period via ES|QL; from dependency data identify high-latency or failing deps.

Summarize (e.g. p99 up; dependency Z elevated) and recommend (investigate Z; Observability Labs for latency).

Example: Correlate service to infrastructure (OpenTelemetry)

Use resource attributes on spans/traces to get the runtimes (pods, containers, hosts) for the service. Then check

CPU and memory for those resources in the same time window as the APM issue:

From the service’s traces or metrics, read resource attributes such as k8s.pod.name, k8s.namespace.name,

container.id, or host.name.

Run ES|QL or Elasticsearch search on infrastructure/metrics indices filtered by those resource values and the

incident time range. Check CPU usage and memory consumption (e.g. system.cpu.total.norm.pct); look for

OOMKilled events, CPU throttling, or sustained high CPU/memory that align with APM latency or error spikes.

Example: Filter logs by service or trace ID

To understand behavior for a specific service or a single trace, filter logs accordingly:

By service: Run ES|QL or Elasticsearch search on log indices with service.name == "<service_name>" and time

range to get application logs (errors, warnings, restarts) in the service context.

By trace ID: When investigating a specific request, take the trace.id from the APM trace and filter logs by

trace.id == "<trace_id>" (or equivalent field in your log schema). Logs with that trace ID show the full request

path and help explain failures or latency.

Guidelines

Use Observability APIs (SLOs API,

Alerting API) and ES|QL on

traces*apm*,traces*otel*/metrics*apm*,metrics*otel* (8.11+ or Serverless), filtering by service.name (and

service.environment when relevant). For active APM alerts, call

/api/alerting/rules/_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active.

When checking one service, evaluate both rule types: rules where params.serviceName matches the target service, and

rules where params.serviceName is absent (all-services rules). Treat either as applicable to the service before

declaring health. Do not query .alerts* indices when determining currently active alerts; use the Alerting API

response above as the source of truth. For APM correlations, run the apm-correlations script (see

[APM Correlations script](#apm-correlations-script)); for dependency/service-map data, use ES|QL or Kibana repo route

handlers. For Elasticsearch index and search behavior, see the Elasticsearch APIs in the Elasticsearch repo.

Always use the user's time range; avoid assuming "last 1 hour" if the issue is historical.

When SLOs exist, anchor the health summary to SLO status and burn rate; when they do not, rely on alerts, anomalies,

throughput, latency, error rate, and dependencies.

When analyzing only application metrics ingested via OpenTelemetry, use the ES|QL TS (time series) command for

efficient metrics queries. The TS command is available in Elasticsearch 9.3+ and is always available in

Elastic Observability Serverless.

Summary: one short health verdict plus bullet points for evidence and next steps.

observability-service-health