opentelemetry

>

INSTALLATION
npx skills add https://github.com/grafana/skills --skill opentelemetry
Run in your project or agent environment. Adjust flags if your CLI version differs.

SKILL.md

OpenTelemetry with Grafana

Overview

OpenTelemetry (OTel) is a vendor-neutral framework for collecting observability data (metrics, logs,

traces, profiles). Grafana Labs integrates it as a core strategy, offering a full stack to collect,

ingest, store, analyze, and visualize telemetry data.

Four-Step Implementation Model

  • Instrument - Add telemetry using Grafana SDKs, Beyla (eBPF), or upstream OTel SDKs
  • Pipeline - Build processing infrastructure with Grafana Alloy or OTel Collector
  • Ingest - Route data to Grafana Cloud OTLP endpoint or self-managed backends
  • Analyze - Dashboards, alerts, Application Observability, Drilldown apps

Grafana Backends

Signal

Backend

Metrics

Grafana Mimir

Logs

Grafana Loki

Traces

Grafana Tempo

Profiles

Grafana Pyroscope

OTLP Endpoint and Authentication

Grafana Cloud OTLP Endpoint

Grafana Cloud exposes a managed OTLP gateway endpoint:

https://otlp-gateway-<region>.grafana.net/otlp

Example regions: prod-us-east-0, prod-eu-west-0, prod-ap-southeast-0

Full example:

https://otlp-gateway-prod-us-east-0.grafana.net/otlp

Authentication - Basic Auth

Grafana Cloud OTLP uses HTTP Basic Auth:

  • Username: Grafana Cloud Instance ID (numeric, e.g. 123456)
  • Password: Grafana Cloud API token (with MetricsPublisher, LogsPublisher, TracesPublisher roles)

#### Via environment variable (recommended)

# Base64-encode "instanceID:apiToken"

export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n '123456:glc_eyJ...' | base64)"

#### Via Alloy environment variables

export GRAFANA_CLOUD_INSTANCE_ID=123456

export GRAFANA_CLOUD_API_KEY=glc_eyJ...

export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp

Direct Send (no collector) - Environment Variables

export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp

export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64(instanceID:apiToken)>"

export OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=production"

Instrumentation by Language

Go

Requirements: Go 1.22+

Install packages:

go get "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" \

  "go.opentelemetry.io/contrib/instrumentation/runtime" \

  "go.opentelemetry.io/otel" \

  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" \

  "go.opentelemetry.io/otel/exporters/otlp/otlptrace" \

  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" \

  "go.opentelemetry.io/otel/sdk" \

  "go.opentelemetry.io/otel/sdk/metric"

Run with environment variables:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \

OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \

go run .

See references/instrumentation.md for full Go code example.

Java (Grafana Distribution - JVM Agent)

Requirements: JDK 8+

Download: grafana-opentelemetry-java.jar from https://github.com/grafana/grafana-opentelemetry-java/releases

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=shoppingcart,service.namespace=ecommerce,deployment.environment=production" \

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \

OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \

OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \

java -javaagent:/path/to/grafana-opentelemetry-java.jar -jar myapp.jar

Optional: Data saver mode (reduces metric cardinality):

export GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=true

Debug:

export OTEL_JAVAAGENT_DEBUG=true

# Enable console output alongside OTLP

export OTEL_TRACES_EXPORTER=otlp,console

export OTEL_METRICS_EXPORTER=otlp,console

export OTEL_LOGS_EXPORTER=otlp,console

Node.js

Install:

npm install --save @opentelemetry/api

npm install --save @opentelemetry/auto-instrumentations-node

Run:

OTEL_TRACES_EXPORTER="otlp" \

OTEL_METRICS_EXPORTER="otlp" \

OTEL_LOGS_EXPORTER="otlp" \

OTEL_NODE_RESOURCE_DETECTORS="env,host,os" \

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \

OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \

NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" \

node app.js

Warning: Bundlers like @vercel/ncc can break auto-instrumentation hooks.

See references/instrumentation.md for manual SDK setup example.

Python

Install:

pip install "opentelemetry-distro[otlp]"

opentelemetry-bootstrap -a install

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \

OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \

OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \

opentelemetry-instrument python app.py

Multi-process servers (Gunicorn, uWSGI): implement post-fork hooks to reinitialize OTel providers per worker.

.NET (Grafana Distribution)

Install NuGet:

dotnet add package Grafana.OpenTelemetry

ASP.NET Core setup:

using Grafana.OpenTelemetry;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()

    .WithTracing(configure => configure.UseGrafana())

    .WithMetrics(configure => configure.UseGrafana());

builder.Logging.AddOpenTelemetry(options => options.UseGrafana());

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \

OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \

OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \

dotnet run

Requirements: .NET 6+ or .NET Framework 4.6.2+

See references/instrumentation.md for full .NET examples.

Beyla (eBPF - Language Agnostic)

Grafana Beyla instruments at the network layer - no code changes required, works with any language.

# Docker

docker run --rm -it \

  --privileged \

  -e BEYLA_SERVICE_NAME=myapp \

  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \

  -v /sys/kernel/security:/sys/kernel/security \

  grafana/beyla

Verify with: curl http://localhost:9090/metrics

Full docs: https://grafana.com/docs/beyla/

Grafana Alloy Collector

Grafana Alloy is the recommended OTel Collector distribution. It combines upstream OTel Collector

components with Prometheus exporters for infrastructure + application observability correlation.

Why Use a Collector?

  • Cost control: Aggregate, sample, and drop data before sending
  • Reliability: Buffer and retry on connection failures
  • Enrichment: Add resource attributes, transform, redact, and route data

Alloy Ports

Port

Protocol

Purpose

4317

gRPC

OTLP gRPC receiver

4318

HTTP

OTLP HTTP/protobuf receiver

Application -> Alloy -> Grafana Cloud

Application env vars (point to local Alloy):

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Alloy config env vars (Alloy -> Grafana Cloud):

export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp

export GRAFANA_CLOUD_INSTANCE_ID=123456

export GRAFANA_CLOUD_API_KEY=glc_eyJ...

See references/collector-config.md for full Alloy configuration.

Kubernetes Setup

Option 1: Grafana Kubernetes Monitoring Helm Chart (recommended)

The Grafana Kubernetes Monitoring Helm chart deploys Alloy with OTLP receivers pre-configured.

  • Enable "OTLP Receivers" in the Cluster Configuration tab
  • Get gRPC/HTTP endpoints from "Configure Application Instrumentation" section
  • Point apps to the in-cluster Alloy endpoint:
export OTEL_EXPORTER_OTLP_ENDPOINT=<GRPC_ENDPOINT_FROM_HELM>

export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Option 2: OpenTelemetry Operator

Install via official docs, then use Instrumentation CR for auto-injection:

apiVersion: opentelemetry.io/v1alpha1

kind: Instrumentation

metadata:

  name: my-instrumentation

spec:

  exporter:

    endpoint: http://otelcol:4317

  propagators:

    - tracecontext

    - baggage

  java:

    # Use Grafana distribution image

    image: us-docker.pkg.dev/grafanalabs-global/docker-grafana-opentelemetry-java-prod/grafana-opentelemetry-java:2.3.0-beta.1

  nodejs: {}

  python: {}

Inject into pods with annotation:

metadata:

  annotations:

    instrumentation.opentelemetry.io/inject-java: "true"

    # or: inject-nodejs, inject-python, inject-dotnet

See references/collector-config.md for Kubernetes Alloy Helm values and OTel Collector YAML.

Sampling Strategies

Head-Based Sampling

Decision made at trace start - low overhead, may miss rare errors.

Environment variable (probability sampler):

export OTEL_TRACES_SAMPLER=parentbased_traceidratio

export OTEL_TRACES_SAMPLER_ARG=0.1   # 10% of traces

Alloy head sampling config:

otelcol.processor.probabilistic_sampler "default" {

  sampling_percentage = 10

  output {

    traces = [otelcol.exporter.otlphttp.grafana_cloud.input]

  }

}

Tail-Based Sampling

Decision made after all spans collected - can sample based on outcome (e.g. keep all errors).

Alloy tail sampling config:

otelcol.processor.tail_sampling "default" {

  decision_wait            = "10s"

  num_traces               = 100000

  expected_new_traces_per_sec = 10

  policy {

    name = "keep-errors"

    type = "status_code"

    status_code {

      status_codes = ["ERROR"]

    }

  }

  policy {

    name = "probabilistic-sample"

    type = "probabilistic"

    probabilistic {

      sampling_percentage = 10

    }

  }

  output {

    traces = [otelcol.exporter.otlphttp.grafana_cloud.input]

  }

}

Key Environment Variables Reference

Variable

Description

Example

OTEL_EXPORTER_OTLP_ENDPOINT

OTLP receiver URL

https://otlp-gateway-prod-us-east-0.grafana.net/otlp

OTEL_EXPORTER_OTLP_PROTOCOL

Transport protocol

grpc or http/protobuf

OTEL_EXPORTER_OTLP_HEADERS

Auth headers

Authorization=Basic <base64>

OTEL_RESOURCE_ATTRIBUTES

Service metadata

service.name=myapp,service.namespace=team,deployment.environment=prod

OTEL_TRACES_EXPORTER

Trace exporter type

otlp

OTEL_METRICS_EXPORTER

Metrics exporter type

otlp

OTEL_LOGS_EXPORTER

Logs exporter type

otlp

OTEL_SERVICE_NAME

Service name (shorthand)

myapp

OTEL_TRACES_SAMPLER

Sampler type

parentbased_traceidratio

OTEL_TRACES_SAMPLER_ARG

Sampler argument

0.1 (10%)

Key Resource Attributes

Attribute

Purpose

Example

service.name

Service identifier

shoppingcart

service.namespace

Groups related services

ecommerce

deployment.environment

Environment tier

production, staging

service.version

App version

1.2.3

Useful Links

BrowserAct

Let your agent run on any real-world website

Bypass CAPTCHA & anti-bot for free. Start local, scale to cloud.

Explore BrowserAct Skills →

Stop writing automation&scrapers

Install the CLI. Run your first Skill in 30 seconds. Scale when you're ready.

Start free
free · no credit card