Map the complete operation
A single user request may involve access checks, retrieval, ranking, several model calls and a business tool. Represent these as spans in one trace. Latency can then be attributed to a provider, index or tool rather than to a vague statement that AI is slow.
Record service and operation names, model and version, generation settings, input and output token counts, status and error code. Pseudonymize user and project identifiers.
Keep content out by default
Prompts, responses and retrieved documents may contain trade secrets, personal data or credentials. Default telemetry should retain sizes, hashes, classifications and protected references instead of complete text. Content capture is enabled only for controlled test sets or investigations with separate access and retention.
Filtering must happen before export to the collector. Redaction after text reaches a shared log does not undo the disclosure.
Join reliability, quality and cost
The technical baseline includes time to first token, total duration, failures, retries, queue depth and GPU saturation. Application signals add retrieval success, citation validity, policy refusal, approval outcome and the final business result.
Allocate cost by request, user, team and use case. An average hides the long tail, so monitor quantiles and anomalies such as agent loops, unusual context length or a surge in tool calls.
Make telemetry part of acceptance
Release a new model or provider only after comparing latency, quality and cost on the same evaluation set. Dashboards should connect application, prompt, index and model versions so a regression has a testable cause.
Alerts must lead to an action: disable a tool, change model routing, reduce concurrency or restore the previous configuration. Observability creates value when it contains an incident, not when it merely produces another log store.