Observability
LLM Observability for Local and Cloud Models
What to trace across local and hosted LLM calls: route, latency, cost, retries, prompt versions, privacy mode, and answer quality.
A successful HTTP response is not enough
An LLM call can return 200 and still be slow, expensive, routed somewhere unexpected, based on the wrong prompt version, or unacceptable to the user. Production observability connects transport success with execution and outcome evidence.
For a hybrid system, the route is part of the result. Operators need to know whether the request ran locally, used another machine, or crossed to a hosted provider.
The minimum useful interaction record
A trace should identify the application, session or thread when supplied, agent and prompt version when relevant, requested and actual model, route, start and finish times, token usage, error shape, and a stable trace ID.
- Latency and time to first token, not only total duration.
- Every provider attempt and whether fallback recovered the call.
- The model and execution location that actually produced the answer.
- Known cost or an explicitly labeled estimate.
- User feedback, evaluation score, or another quality signal.
Content capture must be a policy choice
Full-content traces are useful for debugging and evaluation but may be inappropriate for sensitive inputs. Metadata-only mode can retain route, timing, tokens, errors, and identifiers while excluding prompt and response content from long-term history.
Metadata-only storage does not change where request content transits during a managed call. Data-boundary documentation must distinguish transit, execution, and retention.
Trace local and hosted paths with the same vocabulary
If local calls and hosted calls use unrelated dashboards, cost and reliability comparisons become guesswork. Normalize core fields while retaining provider-specific detail. The same status model should distinguish succeeded, failed, running, recovered, and reconciliation-required calls.
Optimaiz records both paths in one interaction model so teams can answer a practical question: which workloads are genuinely better on available local capacity?
Turn traces into routing decisions carefully
Observability can reveal a candidate for local execution, but routing should not change solely because a chart suggests savings. Validate model quality, capacity, failure behavior, privacy policy, and the meaning of the estimated cost first.
The goal is a feedback loop: observe, move one bounded workload, compare, and expand only when the evidence remains good.
Frequently asked questions
What is the difference between LLM monitoring and LLM observability?+
Monitoring reports known metrics and alerts. Observability keeps enough linked context to explain route, model, prompt version, attempts, cost, latency, and outcome for an individual interaction.
Can I trace an LLM call without storing the prompt?+
Yes. Metadata-only tracing can retain operational fields while excluding prompt and response content from long-term interaction history.
Why record the actual execution route?+
In a hybrid system, route determines cost, privacy boundary, available model, and recovery behavior. The requested route and actual route may differ when fallback is allowed.