LLM Observability in Production: Traces, Evals, Cost, and Feedback

LLM observability explains why a model-driven request produced its outcome. A normal application trace may say an endpoint returned 200 in 1.8 seconds. An LLM trace needs to reveal that retrieval missed the authoritative document, the model called a tool twice, a fallback changed providers, the parser repaired malformed output, and the final answer still failed the task.

Production observability therefore combines three disciplines:

• Tracing — the complete execution path for one request.
• Evaluation — whether the output was correct, grounded, useful, safe, or otherwise fit for purpose.
• Operations — latency, cost, errors, saturation, alerts, access, and retention.

Any one alone leaves a blind spot. A quality score without a trace cannot explain a regression. A trace without evaluation shows what happened but not whether it was good. A dashboard without data governance can become a second copy of every sensitive conversation.

Start with the user outcome

Define observability around tasks, not model calls. “Chat completion succeeded” is infrastructure health; “support question answered correctly without escalation” is product health.

For each important task, define:

• Success: the user-visible or downstream outcome
• Quality: accuracy, faithfulness, completion, format validity, or another task metric
• Latency: time to first useful output and total completion time
• Cost: tokens and provider spend per successful task
• Reliability: error, timeout, fallback, retry, and abandonment rates

These measures create the questions the instrumentation must answer. Tools come afterward.

Trace the whole request tree

Use one correlation or trace ID from the incoming request through every LLM-specific step:

request
├── input policy / redaction
├── retrieval
│   ├── query rewrite
│   ├── vector search
│   └── rerank
├── model call
│   ├── prompt render
│   ├── provider request
│   └── streamed response
├── tool call
│   ├── argument validation
│   └── external API
├── output parser / guardrail
└── retry, escalation, or fallback

Each span should capture its duration, status, input and output sizes, errors, and the identifiers needed to compare behavior across deployments. For model spans, record tokens, cost, cache usage, finish reason, and provider request ID when available. For retrieval, record query, filters, document IDs, scores, and selected chunks. For tools, record the tool name, validated arguments or a safe digest, result status, and side-effect classification.

Prefer portable telemetry standards such as OpenTelemetry and LLM-oriented semantic conventions when they fit the stack. Backend portability matters less than consistent instrumentation, but open spans reduce the cost of changing the storage and analysis layer later.

Version every behavior-changing input

Many regressions are impossible to explain because the trace says only “model X.” Attach:

• application and Git revision
• prompt template and system-instruction version
• model, provider, and relevant inference configuration
• retrieval index, embedding model, and reranker version
• tool schema version
• routing and fallback policy version
• evaluation rubric and judge version
• experiment or feature-flag cohort

Store stable IDs rather than pasting whole configuration objects into every span. A trace should let you answer “what changed between the healthy and unhealthy cohort?” without reconstructing deployment history from memory.

Monitor four pillars together

Quality

Use deterministic checks for schema validity, citations, and business rules. Apply a calibrated LLM-as-judge to a sampled subset for semantic criteria. Combine those signals with explicit user feedback, edits, retries, abandonment, and downstream task completion.

No single score is truth. A thumbs-up rate is selection-biased; a judge inherits model bias; task completion can be delayed. Triangulate and slice.

Latency

Track end-to-end p50, p95, and p99 plus span-level contribution. Separate time to first token from total duration for streaming interfaces. Watch queueing, retrieval, tools, retries, and fallback rather than blaming the model for the entire request.

Cost

Measure input, cached input, output, tool, embedding, reranking, and judge cost. Report cost per successful task, not only per request. A cheap model that retries and escalates may cost more than a stronger one that finishes once.

Reliability

Track provider errors, timeouts, malformed outputs, guardrail blocks, tool failures, retry counts, route changes, and user-visible failure. Agent systems also need step count and loop detection: a technically successful 40-step run can be an operational failure.

Slice before averaging

Global averages conceal the failures users feel. Break dashboards down by:

• task and endpoint
• prompt, model, and route version
• language and input length
• tenant or customer tier, with privacy-safe aggregation
• tool or retrieval path
• experiment cohort
• success, retry, escalation, and fallback outcome

Compare releases on matched slices. If traffic mix shifted toward easier queries, a better aggregate quality score may say nothing about the new prompt.

Make payload capture a policy decision

Prompts, retrieved documents, and tool output can contain personal data, credentials, source code, customer secrets, or regulated information. Decide before production:

• which fields may be stored raw
• which must be redacted, hashed, tokenized, or omitted
• whether content capture is sampled or metadata-only
• who can query raw traces
• how data is encrypted and isolated
• how long each class is retained
• how deletion and tenant boundaries are enforced

Redact before export when possible. A dashboard access control does not undo sensitive data already copied into logs. Test redaction with adversarial payloads, not only well-formed examples.

Alert on user impact

Avoid alerting on every noisy model score. Useful alerts combine magnitude, duration, sample size, and a task-level slice:

• schema-valid rate below the release floor for 15 minutes
• p95 latency above the user budget with enough traffic
• cost per successful task rising beyond a configured band
• fallback rate spiking for one provider or route
• faithfulness falling on a sustained sampled window
• tool-loop count exceeding a safe limit

Link alerts directly to filtered traces and the deployment or configuration change most likely responsible. An alert that begins with ten minutes of dashboard archaeology is incomplete.

Turn failures into eval cases

The observability loop closes when a confirmed production failure becomes reproducible:

1. Locate the trace and identify the failing span.
2. Sanitize and minimize the input while preserving the failure.
3. Label the expected behavior and failure category.
4. Add it to the offline eval dataset.
5. Reproduce the current failure.
6. Fix the prompt, retrieval, tool, routing, or code.
7. Gate future releases on the new regression case.

This is the LLM equivalent of turning an incident into a unit test. Over time, the eval dataset becomes a compressed history of what production has taught the team.

The LLM Observability Engineer packages this rollout as an implementation workflow; Langfuse, Arize Phoenix, and LangSmith are cataloged backends for storing and exploring the resulting traces and evaluations.