Custom metrics

Application metrics use the OpenTelemetry Meter API directly. Kerniq does not wrap it; Observability::meter hands you one.

use kerniq_server::observability::prelude::*;

let meter = observability.meter("payment-service");

let payment_counter = meter
    .u64_counter("payments.processed")
    .with_unit("{payment}")
    .with_description("Payments processed, by type and outcome.")
    .build();

payment_counter.add(1, &[
    KeyValue::new("payment.type", "transfer"),
    KeyValue::new("outcome", "accepted"),
]);

When metrics are disabled, meter returns a no-op meter, so instrumentation code never has to check.

#Instruments

#Rules

1. Create instruments once (at startup, in your state) and clone them. Creating an instrument per request is slow and defeats aggregation.

2. Keep attributes bounded. Every distinct attribute combination is a separate series kept in memory. Allowed: route templates, enumerations (payment type, outcome, currency, model family). Never: user, customer or account ids, transaction or request ids, e-mail addresses, raw URLs, error messages, prompts, free-form model names. Put those on spans and log events instead.

3. Use BoundedLabel to map untrusted input onto an allow-list:

   rust

   use kerniq_server::observability::BoundedLabel;
   
   static CURRENCY: BoundedLabel = BoundedLabel::new(&["USD", "EUR", "GBP", "KES"], "other");
   assert_eq!(CURRENCY.get("XYZ"), "other");

4. Use &'static str values where possible (KeyValue::new("outcome", "accepted")); they do not allocate.

5. Record amounts in histograms, not labels.

As a safety net the SDK caps each instrument at 2 000 attribute sets (MetricsConfig::cardinality_limit to change it); excess measurements are folded into one otel.metric.overflow series. In tests, TestTelemetry::assert_bounded_metric_labels fails on forbidden keys.

#Examples by domain

All of these are implemented in examples/observability-demo/src/business.rs.

Prefer OpenTelemetry semantic-convention names (db.client.*, http.client.*, messaging.*) where one exists.

#AI inference

kerniq_telemetry::inference provides the instruments and a per-request guard for long-running generation:

use kerniq_server::observability::telemetry::inference::InferenceMetrics;
use kerniq_server::observability::{BoundedLabel, KeyValue};

static MODEL: BoundedLabel = BoundedLabel::new(&["chat-small", "chat-large"], "other");
let metrics = InferenceMetrics::new(&observability.meter("inference"));

let mut request = metrics.start([KeyValue::new("inference.model", MODEL.get("chat-small"))]);
request.input_tokens(512);
request.execution_started();          // left the queue → inference.queue.duration
for _token in 0..128 {
    request.tokens_generated(1);      // first call → inference.time_to_first_token
}
request.finish();                     // or .fail("timeout"); dropping it = cancelled

Per-token cost is an integer add: tokens are counted locally and recorded once at the end, and nothing is logged per token. If the client disconnects, Hyper drops the response stream, the guard is dropped with it, and the request is recorded as cancelled. Kerniq does not run models; these are conventions for applications that do. They map onto the OpenTelemetry GenAI conventions (gen_ai.server.time_to_first_token, gen_ai.server.request.duration, gen_ai.client.token.usage) for teams standardizing on those.

#Testing metrics

use kerniq_observability_testing::TestTelemetry;
use kerniq_server::observability::KeyValue;

let telemetry = TestTelemetry::new().await;
let counter = telemetry.observability().meter("t").u64_counter("payments.processed").build();
counter.add(1, &[KeyValue::new("payment.type", "transfer")]);

let metrics = telemetry.metrics();
assert_eq!(metrics.total("payments.processed", &[("payment.type", "transfer")]), 1.0);
telemetry.assert_bounded_metric_labels();