LLM observability: what to capture, trace and alert on in production

TL;DR

• LLM observability means any bad answer, latency spike, error burst or cost jump can be traced to a specific request, user, feature and model within minutes.
• Capture one fixed record per call, success or failure: identity, routing, tokens, cost, total latency and time to first token, normalized errors, attribution and gated content.
• OpenTelemetry’s GenAI semantic conventions are the shared vocabulary, but they are still in Development status, moved to their own repository in 2026, and have renamed attributes and metrics more than once. Pin a version.
• Keep metrics at 100 percent and low-cardinality; sample content, but keep every error, slow request, fallback and negative-feedback trace. Redact before data leaves the process or gateway.
• The open-source llm-observability-stack agent skill packages the record, dashboards, alert rules and a production eval job for Claude Code, Codex and Cursor.

LLM observability is the practice of recording every model call in production so that any bad response, latency spike, error burst or cost jump can be traced back to the exact prompt, model, provider, user and feature behind it. It borrows from application performance monitoring but differs in four ways that shape every decision: the payload is the debugging signal and often contains personal data, cost varies per request, streaming gives latency two meanings, and an HTTP 200 says nothing about whether the answer was any good. This explainer covers what to capture, how OpenTelemetry names it, how to trace agents, sample and redact, which alerts to set, and how to connect traces to feedback and evaluation. It is about the practice; for choosing a product, The Frontier Wire’s comparison of agent observability tools covers the options.

LLM monitoring, tracing and observability

LLM monitoring watches aggregate signals and alerts on thresholds; LLM tracing records each call and step as spans in a tree; LLM observability is the combination, plus the content and quality data needed to explain why a signal moved. Teams need all three, built on one consistent per-call record.

The three terms are often used interchangeably, but they answer different questions:

The joining is what makes it work. A cost alert is only actionable if it can be broken down by feature, and a thumbs-down is only useful if it resolves to one stored record with the prompt, model and output.

The per-call record

The per-call record is a fixed set of fields written for every LLM call, success or failure, so that every later question has data to answer it. If a gateway writes the record, the application only adds attribution fields as headers; otherwise a shared client wrapper must write it in every service.

The llm-observability-stack skill’s reference on what to capture groups the minimum into seven sets:

Two rules govern where each field goes. Metric labels must stay low-cardinality: provider, model, feature, environment, team, error type. The skill’s rule of thumb is that “if a label can take more than a few hundred values, it does not belong on a metric.” User, session and request IDs go on spans and logs only. And cost should be computed at capture time from the price table in effect, including cached-input discounts, because recomputing later loses which price applied. The sibling LLM cost optimization playbook builds its attribution on this record.

Latency, TTFT and streaming usage

Time to first token (TTFT) is the delay from sending a request to receiving the first content chunk, and it drives perceived speed in chat interfaces. Total latency drives agent step time and timeouts. Inter-token latency shows provider throughput. Each needs its own percentile; mixing them makes p95 meaningless.

Three practices follow from that split:

• Never mix streaming and non-streaming traffic in one percentile. A non-streamed summary and a streamed chat reply measure different things.
• Normalize by output tokens when comparing models or prompts, since a longer answer is slower by construction rather than a regression.
• Request usage on streams. For OpenAI-compatible streaming, the skill’s fix is stream_options={"include_usage": True}; without it, token counts and cost are missing for every streamed call, and dashboards undercount spend.

The OpenTelemetry client inference conventions define a gen_ai.response.time_to_first_chunk span attribute, “measured from when the client issues the generation request to when the first chunk is received,” and a matching gen_ai.client.inference.time_to_first_chunk metric. Agents complicate the definition: with tool calls or reasoning models, the first chunk may be a tool-call delta or reasoning output, not visible text. The skill advises deciding whether TTFT means “first visible token” or “first byte” and recording it consistently.

A gateway measures TTFT at its own edge. Timing it in the client path too captures what the user actually waits through, including the application’s own server and network.

Errors, fallbacks and silent failures

LLM error handling for observability means classifying every failure by whose fault it is and recording retries and fallbacks as data. A provider 429 calls for more capacity; a 429 from an internal rate limit means the limit is too tight. Alerting on raw status codes conflates the two.

The skill’s fault domains are caller (bad request, unknown model), policy (an organization’s own budgets and rate limits), provider (429, 5xx, timeout, auth), and the gateway or application itself. Bifrost’s Prometheus documentation uses the same split in its error_type label, with values prefixed caller_, policy_, provider_ and bifrost_, and describes the set as closed, so an unclassified _OTHER value should stay at zero.

Three kinds of failure are easy to miss:

• Successful fallbacks. A request that fails over to a second provider still succeeds, but its quality, cost and latency profile changed. The skill treats a sustained fallback share as an incident signal. The Frontier Wire’s guide to LLM failover and load balancing covers the routing side.
• Silent 200s. finish_reason values of length and content_filter, empty outputs and structured outputs that fail to parse all return HTTP 200. They should count as errors in metrics.
• Swallowed exceptions. The skill’s grep audit hunts for except ...: pass and empty catch blocks around LLM calls, and its rule is to return a typed failure and record it.

OpenTelemetry’s GenAI spans document adds that when a request is retried automatically, the span “SHOULD cover the duration of the logical operation with all retries,” so retry counts belong in attributes rather than in separate sibling spans.

OpenTelemetry GenAI semantic conventions

The OpenTelemetry GenAI semantic conventions are a shared vocabulary of span names, attributes and metrics for model calls, tools and agents, so that any backend can read telemetry from any instrumentation. They are the right default, with one caveat: they are still in Development status and their names keep changing.

The status changed shape in 2026. The GenAI page on opentelemetry.io, under semantic conventions 1.44.0, now reads “Moved: Generative AI semantic conventions” and points to a dedicated semantic-conventions-genai repository, created in May 2026. Every document there carries “Status: Development.” Within it, inference spans and metrics live in a client-inference document, agent spans in their own file, and token metrics in a third.

The core names an LLM span carries:

Name drift is real. Older instrumentation emits a single token histogram, gen_ai.client.token.usage, and the OpenTelemetry Python GenAI utility README still documents that histogram. The current client inference document defines gen_ai.client.inference.usage.input_tokens and siblings instead, which is what the skill’s reference now lists. The same README says content is not captured by default, that OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental enables experimental features, and that OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT accepts NO_CONTENT, SPAN_ONLY, EVENT_ONLY or SPAN_AND_EVENT. The skill’s advice holds: pin the version emitted, check names on a real span before writing queries, and put custom fields such as cost and prompt version in an application namespace like app.cost_usd.

Trace structure for multi-step agents

An agent trace is one tree per run or per user turn: a root span for the run, child spans for each model call, tool execution, retrieval and sub-agent. Loose log lines cannot show that step seven repeated step six; a trace makes it visible at a glance.

A layout adapted from the skill’s reference follows the conventions’ agent span names, where the span name “SHOULD be invoke_agent {gen_ai.agent.name}”:

invoke_agent support-agent        (root: session.id, user hash, feature, step_count, exit reason)
├─ retrieval kb.search            (top_k, doc IDs, latency)
├─ chat gpt-6.1-sol               (step 1, tokens, cost, finish_reason=tool_calls)
├─ execute_tool lookup_order      (tool.call.id, args gated, result size, error)
├─ invoke_agent billing-subagent  (nested run, same trace)
└─ chat gpt-6.1-sol               (step N, finish_reason=stop)

The root span records total tokens and cost, step count, which tools ran, and the loop’s exit reason: final_answer, max_steps, error or user_cancelled. Runaway loops then show up as traces with abnormal span counts, and the skill recommends alerting on step counts above a ceiling and on max_steps exits.

Two mechanics hold the tree together. W3C traceparent headers must be propagated into every outbound LLM and tool HTTP call, so gateway and provider spans join the application’s trace; the skill’s Python example instruments httpx, which the OpenAI and Anthropic SDKs use. And every span in a multi-turn conversation should carry a stable session.id or gen_ai.conversation.id. The skill’s default is one trace per turn grouped by session, which is safer than one very long trace per session. When calls go through a gateway that exports spans, the application should not create a duplicate span per LLM call; it propagates context and lets the gateway’s span nest under the agent step.

Sampling and PII redaction

Sampling decides which telemetry is kept; redaction decides what personal data it may contain. Metadata is safe to keep at 100 percent. Content (prompts, outputs, tool payloads) is where personal data lives, so it is sampled, redacted at the edge, and retained briefly.

The skill’s sampling policy:

Keeping every error while sampling the rest requires tail-based sampling, which decides after the whole trace is seen. The OpenTelemetry Collector’s tail sampling processor, rated beta for traces, offers status_code, latency and probabilistic policies among others. Head-sampling at 1 percent loses every rare failure.

For redaction, the order of preference is not capturing content on regulated routes, redacting in process or at the gateway before content reaches any sink, then restricting backend access with short retention. The skill ships regex redactors for emails, API keys, card numbers, phones and IPv4 addresses, and calls regex “a floor, not a guarantee” that misses names and addresses; it suggests an NER-based detector such as Presidio for free text. Prompts and outputs both need redaction, because models echo input.

The most common leak is a second sink. Bifrost’s content logging documentation states that every observability connector “has an independent disable_content_logging” that is “not inherited from the client setting,” so turning content off for the log store does not turn it off for an OpenTelemetry exporter. Dimension headers and captured request headers are exported regardless of content flags and must never carry personal data.

Prometheus dashboards and alert thresholds

An LLM dashboard puts health first (request rate, error rate excluding caller and policy errors, p95 latency, p95 TTFT, cost per hour) and then breaks each down by provider, model and feature. Alerts fire on rates and ratios over windows long enough to ride out one-minute provider blips.

The skill’s dashboard reference lays out six rows: health, errors by type, latency and TTFT by model, spend by feature and team, resilience (fallbacks, retries, key health) and quality from the eval store. Its PromQL targets Bifrost’s metrics, with a note that teams emitting OpenTelemetry GenAI metrics from app code should map the names. The alert that matters most excludes errors that are not the provider’s:

sum by (provider) (rate(bifrost_error_requests_total{error_type!~"(caller|policy)_.*"}[5m]))
  / sum by (provider) (rate(bifrost_upstream_requests_total[5m])) > 0.05

Starting thresholds from the skill, to be tuned against each team’s SLOs:

The input-tokens alert catches prompt bloat, runaway agent context and over-eager retrieval. The skill pairs each alert with a runbook row (symptom, first query, likely cause, action) and recommends Grafana data links from panels to the request log. Provider-side limits behind 429 alerts are covered in The Frontier Wire’s rate limiting guide.

User feedback and production evals

Feedback linkage joins thumbs, edits and regenerate clicks to the exact stored call by request ID, never by timestamp. Production evaluation then samples real traffic daily, scores it, and turns failures into regression tests, because prompt and model drift happens after release.

The feedback flow has five steps in the skill: generate a request ID before the call and send it upstream as x-request-id; return it to the client and store it with the message; write feedback events with that ID; join feedback to records and traces by ID; and on negative feedback, force-retain the full trace and queue it for review. Implicit signals are often more plentiful than thumbs: regenerate clicks, copy events, a user rephrasing within 30 seconds, or abandoning the conversation after an answer.

The eval job runs daily. It samples yesterday’s successful records stratified by feature and model, including every negative-feedback and fallback record; scores them with cheap deterministic checks first (schema validity, length, required citations), then an LLM judge with a task-specific rubric; stores scores by request ID with the judge model and rubric version; tracks mean score per feature and prompt version; and promotes reviewed failures into the offline regression set. The skill’s judge hygiene is specific: calibrate on 50 to 100 human-labeled examples and report agreement, version the rubric, and tag the judge’s own traffic with a separate feature dimension so it does not distort product dashboards. The Frontier Wire’s guide to comparing LLM evaluation frameworks covers the scoring side in depth.

Gateway-level capture

A gateway is the cheapest way to get one consistent record for every call, because every service in every language passes through it by changing a base URL. It covers the model calls; retrieval, tool logic and business outcomes still need app-side spans.

The llm-observability-stack skill ranks options by coverage per unit of effort: a gateway in front of all providers, then one shared client wrapper per language, then per-SDK auto-instrumentation, whose attribute names “drift across languages.” Its gateway examples use Bifrost, whose built-in observability logs content, tokens, cost and latency per request, exposes Prometheus metrics at /metrics including bifrost_stream_first_token_latency_seconds, and exports OpenTelemetry traces with trace_type: "genai_extension". Per-request dimension headers such as x-bf-dim-feature reach logs, spans and declared Prometheus labels, and response headers report the provider that actually served the request and whether it was a fallback. Requests sharing an x-bf-session-id header carry session.id on their root span, and the group_traces_by_session option folds a whole agent run into one trace. The Frontier Wire’s piece on running an AI gateway in front of every model call covers the architecture.

The llm-observability-stack skill

llm-observability-stack is an MIT-licensed agent skill that walks a coding agent through building this record, gateway-first, with app-side spans for what a gateway cannot see. It covers capture, tracing, redaction, dashboards, alerts and production evals.

What it contains. The repository follows the Agent Skills format: SKILL.md plus three references. what-to-capture.md holds the per-call record, cardinality rules, OpenTelemetry attribute tables, agent span code in Python and TypeScript, TTFT measurement, redactors, the sampling table, feedback linkage and the production eval job. dashboards-and-alerts.md holds PromQL panels, a Prometheus alert-rule file, the threshold table and the runbook. bifrost-telemetry.md covers log store, content controls, dimensions, metrics, OTel export and headers. SKILL.md opens with a six-part grep audit for LLM call sites, existing instrumentation, prompts written to plain logs, streams without include_usage, swallowed errors and missing user or session identity, and ends with a verification checklist.

What triggers it. Its description lists phrases such as “why was this response bad”, “TTFT is slow”, “silent 429s”, “trace my agent run”, “our logs contain PII” and “connect thumbs-down feedback to traces.”

How to install it. The skill is on skills.sh and installs with the skills CLI:

# Preview the repository's skills without installing
npx skills add build-with-ai-hq/llm-observability-stack --list

# Install for Claude Code, Codex and Cursor in the current project
npx skills add build-with-ai-hq/llm-observability-stack -a claude-code -a codex -a cursor

# Global install, no prompts; then list and update
npx skills add build-with-ai-hq/llm-observability-stack -g -y
npx skills list
npx skills update

In a test install on October 28, 2026, The Frontier Wire ran npx -y skills@1.7.4 add build-with-ai-hq/llm-observability-stack -a claude-code -a codex -a cursor -y in an empty repository. The CLI reported “Installed 1 skill,” placed files in .agents/skills/llm-observability-stack (read directly by Codex and Cursor, symlinked for Claude Code), recorded a computed hash in skills-lock.json, and printed “Review skills before use; they run with full agent permissions.” SKILL.md runs to 2,646 words, and the lint_skills.py linter from the sibling agent-skills-for-teams skill raised no errors or warnings on it. The npx skills and skills.sh guide explains the CLI.

How to use it. Example prompts drawn from its triggers:

• “We have no visibility into our LLM calls. Add OpenTelemetry to every SDK call.”
• “TTFT is slow on the chat endpoint. Find out whether it is the provider or the prompt.”
• “Connect thumbs-down feedback to traces and evaluate production traffic.”

For cost questions, the sibling llm-cost-reduction skill builds budgets on the same attribution.

Limits of this explainer

This piece describes practices and the skill’s files; it did not instrument a production application or measure overhead, and its thresholds are starting points from the skill, not benchmarks. The OpenTelemetry names quoted were current on October 28, 2026, and will change. Instrumentation libraries lag the conventions, which is why the skill tells teams to check a real span. Both the skill and its organization are new; the build-with-ai-hq GitHub organization was created on October 25, 2026, and like any skill it runs with the agent’s full permissions, so reading it before installing is worth the few minutes.

LLM observability is less about tools than about one discipline: every call writes the same record, every record carries a request ID, and every signal can be followed down to the request that caused it. Teams that get that right can answer “why was this response bad” with a query instead of a meeting. The llm-observability-stack skill is a practical way to have a coding agent audit a repository against that standard, starting with npx skills add build-with-ai-hq/llm-observability-stack.