Traces, Spans, and Logs: The Anatomy of LLM Observability

What Is a Trace

A trace is the complete record of what happened during a single request to your AI application. When a user sends a message to a chatbot, triggers a document search, or asks an AI agent to complete a task, the system performs a sequence of operations: it might embed the query, search a vector database, assemble a prompt from a template and retrieved context, call a language model, parse the response, execute tool calls, and format the final output. A trace captures all of these operations as a connected tree, with a unique trace ID that ties them together. The trace is the atomic unit of debuggability, because it contains everything you need to understand why a specific request produced the result it did.

Each trace has a root span that represents the overall request and child spans that represent individual operations. The tree structure reflects the causal relationships between operations: the retrieval span is a child of the root because it was triggered by the request, and the generation span is a sibling of the retrieval span because it happens after retrieval completes. If the model makes a tool call, the tool-call span is a child of the generation span because the generation produced it. This hierarchical structure is what lets you see not just what happened, but the order and causal chain in which it happened.

A well-constructed trace answers every debugging question for a specific request. Why was the answer wrong? Open the trace and check what context was retrieved, because the retrieval span shows the documents that were injected into the prompt. Why was the response slow? Look at the duration of each span to find the bottleneck. Why was this request so expensive? Check the token counts on the generation span. Why did the agent take an unexpected action? Read the tool-call spans to see what arguments the model generated and what the tool returned. Without traces, each of these questions requires reproduction and guesswork. With traces, each is answered by reading.

What Is a Span

A span represents a single operation within a trace. Every span has a span ID, a reference to its parent span (except the root span), a name that describes the operation, a start timestamp, an end timestamp, a status (OK or ERROR), and a set of attributes that carry the operation-specific data. The duration of a span is the difference between its start and end timestamps. The total request latency is the duration of the root span. The time spent on each step is the duration of that step's span.

Spans in LLM observability are typed, meaning different kinds of operations carry different attributes. The OpenTelemetry semantic conventions for generative AI define the standard types and their attributes, and most LLM observability platforms either follow these conventions directly or map to them. The main span types you will encounter are generation spans, retrieval spans, embedding spans, and tool-call spans, each described below.

Generation Spans

A generation span captures a single call to a language model. Its attributes include the model name and version (for example, gpt-4o-2024-08-06 or claude-sonnet-4-20250514), the full prompt as a list of messages (system, user, assistant, tool results), the full completion text, the input token count, the output token count, the cached token count if prompt caching is enabled, the generation parameters (temperature, top-p, max output tokens, stop sequences), the finish reason (stop, length, content_filter, tool_calls), and the latency broken into time-to-first-token and total generation time. Some platforms also record the cost in dollars derived from the model's per-token pricing.

The prompt and completion content is the most valuable attribute for debugging and the most sensitive for privacy. A generation span without the prompt content tells you how long the call took and how many tokens it consumed, but it cannot tell you why the output was wrong. A generation span with the prompt content lets you see the exact instructions and context the model received, which is usually where the root cause lives. Teams balance this by capturing full content for sampled traffic and metadata only for the rest, or by applying redaction rules that mask PII while preserving the prompt's structure.

Retrieval Spans

A retrieval span captures a search operation against a vector database, knowledge graph, or any retrieval backend. Its attributes include the query text (or query embedding), the collection or index searched, the number of results requested (top-k), the number of results returned, the similarity or relevance scores for each result, the content of the retrieved documents (optionally, for debugging), and any filters applied to the search (metadata filters, time range, tenant ID). For hybrid search that combines vector similarity with keyword matching, the span may also record the keyword query and the fusion method used.

Retrieval spans are critical for diagnosing RAG failures, because the most common cause of a wrong answer in a RAG system is wrong retrieval, not wrong generation. If the retrieval span shows that none of the returned documents contain the information needed to answer the question, the generation model cannot be blamed for hallucinating, because it had no correct context to draw from. Conversely, if the retrieval span shows that the correct document was retrieved but the answer is still wrong, the problem is in the prompt template or the model's instruction-following, not in retrieval. This diagnostic separation is only possible when retrieval is captured as its own span with its own attributes.

Embedding Spans

An embedding span captures a call to an embedding model that converts text into a vector representation. Its attributes include the text being embedded (or a hash of it), the embedding model name and version, the vector dimensions, the token count of the input text, and the latency. Embedding spans are less commonly inspected during debugging than generation or retrieval spans, but they matter for two reasons. First, embedding latency contributes to the total retrieval latency, and for high-volume systems with many embedding calls per request, it can be a significant component. Second, embedding model changes (upgrading to a newer model version or switching providers) can cause retrieval quality regressions if the index was built with a different model, and embedding spans make it possible to detect when the embedding model version changed relative to the index.

Tool-Call Spans

A tool-call span captures an external action triggered by the language model through function calling or tool use. Its attributes include the tool name, the arguments the model generated (as structured data), the tool's response, whether the call succeeded or failed, the latency, and any error message if it failed. In agentic systems where the model decides which tools to call and in what order, the sequence of tool-call spans under a generation span is the agent's trajectory, the record of what it decided to do and what happened when it did it.

Tool-call spans are especially important for safety and auditability. If an agent has access to tools that modify state (sending emails, updating databases, making purchases), the tool-call spans provide an audit trail of what was changed, with what arguments, and what the tool reported. For agents with destructive capabilities, these spans are not just debugging aids but compliance requirements, and many teams add assertions that verify tool-call arguments against safety constraints before execution.

Logs vs Traces vs Metrics

Logs, traces, and metrics are three distinct signal types that serve different purposes in observability, and understanding the distinction clarifies how LLM observability fits into a broader monitoring architecture.

Logs are unstructured or semi-structured text records emitted at specific points in the code. They are good for recording events ("model call started", "retrieval returned 5 results", "tool call failed with timeout") but poor for showing relationships between events. Searching through logs to reconstruct what happened during a specific request requires correlating timestamps and request IDs across multiple log entries, which is tedious and error-prone. Logs are the oldest observability signal and remain useful for recording details that do not fit neatly into spans, such as configuration values at startup, deprecation warnings, or retry decisions.

Traces are structured records that model a request as a tree of operations, as described above. They are good for debugging specific requests, understanding pipeline structure, and capturing the causal relationships between operations. They are poor for answering aggregate questions like "what was the average latency this week?" without first being aggregated into metrics. Traces are the primary data source for LLM observability because they capture the content and context that makes debugging possible.

Metrics are numeric aggregations computed from traces, logs, or direct measurements. They are good for dashboards, alerting, and trend analysis: average cost per request, p95 generation latency, quality score distribution, error rate by model. They are poor for debugging individual requests because they discard the detail. Metrics are what you look at to understand the system's behavior in aggregate, and when a metric moves in a concerning direction, you drill into the underlying traces to find the specific requests responsible.

The standard practice in LLM observability is to emit traces as the primary data source, derive metrics from the traces for dashboards and alerting, and use logs for supplementary event recording. This approach gives you the best of all three: traces for debugging, metrics for monitoring, and logs for context.

OpenTelemetry and Standard Conventions

OpenTelemetry is an open-source observability framework that provides APIs, SDKs, and semantic conventions for generating traces, metrics, and logs in a vendor-neutral way. In 2025 and 2026, the OpenTelemetry community published semantic conventions specifically for generative AI, defining standard span names, attribute keys, and event names for LLM operations. These conventions mean that instrumentation code written against the OpenTelemetry standard works with any compatible backend, whether that is an LLM-specific platform like Langfuse, a general-purpose backend like Jaeger, or a commercial APM like Datadog.

The key conventions include span names like gen_ai.chat for a chat completion call, span attributes like gen_ai.system (the model provider), gen_ai.request.model (the model name), gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons. Auto-instrumentation libraries for popular frameworks (LangChain, LlamaIndex, the OpenAI SDK, the Anthropic SDK) implement these conventions, so adopting them is often as simple as installing the right package and initializing the exporter.

The practical benefit of following OpenTelemetry conventions is portability. If you start with one observability backend and later need to switch, the instrumentation code stays the same, because it produces standard spans that any compatible backend can ingest. This avoids the vendor lock-in that comes from using a platform's proprietary SDK, and it means the observability investment you make today is durable regardless of which tools you use in the future.

Putting It Together: Reading a Trace

To make this concrete, consider a RAG chatbot that receives the question "What is the refund policy for annual plans?" Here is what the trace might look like as a span tree:

The root span (chat_request) captures the user's message, the session ID, and the total request duration of 2.3 seconds. Its first child is an embedding span (embed_query) that converts the question into a vector in 45 milliseconds using text-embedding-3-small. The second child is a retrieval span (vector_search) that searches the "policies" collection, returns 5 documents with similarity scores ranging from 0.89 to 0.72, and takes 120 milliseconds. The third child is a generation span (chat_completion) using gpt-4o-mini that receives a system prompt, the 5 retrieved documents, and the user's question as input (2,800 input tokens), generates a 340-token response in 1.9 seconds (180 milliseconds to first token), and finishes with reason "stop".

If the response incorrectly states that annual plans are non-refundable when in fact the policy allows refunds within 30 days, the debugging process is: open the retrieval span and check whether the correct policy document was retrieved. If the document about annual refunds is missing from the results, the problem is retrieval (perhaps the embedding model changed, the document was not indexed, or the similarity threshold filtered it out). If the correct document is present in the retrieved results but the model still gave the wrong answer, the problem is generation (perhaps the system prompt instructions are ambiguous, or the model failed to follow them). This diagnostic process takes seconds with a trace. Without a trace, it would take a reproduction attempt, manual document checking, and prompt iteration.