Grafana dashboards and the local stack

Grafana is a visualization platform: it stores no telemetry. Metrics need a Prometheus-compatible store (Prometheus, Grafana Mimir), traces a trace store (Tempo, Jaeger), logs a log store (Loki). The local stack runs all of them.

#Run it

# Option A: Prometheus scrapes the app (no Collector)
docker compose -f observability/docker-compose.yml --profile prometheus up --build

# Option B: OTLP → Collector → Prometheus, Tempo, Loki
docker compose -f observability/docker-compose.yml --profile otlp up --build

URL
http://localhost:3000	the demo API (`examples/observability-demo`)
http://localhost:3001	Grafana (anonymous admin — local use only), folder Kerniq
http://localhost:9090	Prometheus

A load generator (observability/scripts/loadgen.sh) sends payments, loans, logins and streaming generations, including abandoned streams, so every panel has data within a minute.

#Dashboards

Dashboard	Shows
Kerniq / Overview	request rate, error ratio, p95, active requests, latency percentiles, CPU, memory, top and slowest routes, telemetry pipeline health
Kerniq / HTTP	per route: traffic, errors by `error_type`, 4xx, latency, time to first byte, duration heatmap, payload sizes
Kerniq / Runtime	Tokio workers, alive tasks, global queue depth, process CPU, RSS, virtual memory, file descriptors, application queues
Kerniq / AI inference	requests by outcome, errors, time to first token, queue wait, total duration, token rates, per-request throughput, stream lifetime vs. first byte, abandoned streams

All dashboards filter by service_name, which both architectures produce (Option A through PrometheusConfig::resource_labels, Option B through the Collector's resource_to_telemetry_conversion), so the same dashboards work for either.

#Dashboards use real metric names

The dashboards are generated by observability/scripts/generate-dashboards.py and checked by observability/scripts/check-dashboards.py against a real scrape: every metric name and label in every query must exist.

cargo run -p example-observability-demo &
observability/scripts/smoke.sh localhost:3000
curl -s localhost:3000/metrics > /tmp/scrape.txt
python3 observability/scripts/check-dashboards.py /tmp/scrape.txt

When this documentation was written, all 50 panel queries returned data in both the Option A and Option B stacks.

#Provisioning

observability/grafana/provisioning/ configures:

datasources: Prometheus (uid: prometheus), Tempo (uid: tempo, linked to Loki by trace id) and Loki (uid: loki, trace_id linked to Tempo),
a file provider loading observability/grafana/dashboards/*.json into the Kerniq folder.

To use the dashboards elsewhere, import the JSON files and select your Prometheus datasource in the datasource variable; panels only reference it through that variable.

#Pinned versions

Component	Image
Prometheus	`prom/prometheus:v3.5.0`
Grafana	`grafana/grafana:12.1.0`
OpenTelemetry Collector	`otel/opentelemetry-collector-contrib:0.130.0`
Tempo	`grafana/tempo:2.8.1`
Loki	`grafana/loki:3.5.3`

These are development settings: anonymous admin access, local storage, no retention planning. See Configuration for production.