How do you set up observability for a RAG application in 2027?

By Kory White · Updated Nov 2026 · 6 min read

A domed observatory on a grassy plain under a dense Milky Way with snowy mountains behind

Direct answer

Make one trace per user request, with one span for each RAG step: the query embedding, the retrieval, the model call and the answer. OpenTelemetry's GenAI conventions define spans for embeddings, retrieval and chat inference, so you can use them instead of inventing your own names. Then attach scores to those traces: retrieval metrics such as context precision and context recall, and answer metrics such as faithfulness and response relevancy.

The sources are product and standards documentation, not a benchmark. The OpenTelemetry GenAI conventions are marked Development, meaning they can still change. We label our own reading as judgment. For a comparison of tools, see our Top 10 of AI observability tools for RAG pipelines; this page is about what to instrument.

Start with the shape: one trace, one span per step

A RAG request is a chain: embed the question, retrieve documents, build a prompt, call the model. Observability means you can open one request and see each of those steps with its inputs, timing and outcome. Arize's Phoenix docs describe tracing as capturing "document retrieval, embedding generation, language model invocation, and response generation", and say it can show the score and order of retrieved documents and the embedding text used for retrieval.

OpenTelemetry's GenAI conventions give those steps standard shapes. Embeddings and inference spans are named from the operation and the model, and are CLIENT spans. The retrieval convention describes a retrieval operation as "an execution of a framework retrieval pipeline that returns documents relevant to a query", and lists LangChain, LlamaIndex and Haystack retrievers as examples.

Judgment: put the whole request under one parent span, so a slow or wrong answer can be traced to the step that caused it. The conventions define the child spans; the parent is up to your application.

What to record at each step

Embedding the query. Span name is the operation plus the model (per the embeddings convention, operation embeddings, kind CLIENT). The model name is part of the span name, so you can see which embedding model ran.
Retrieval. Span name is retrieval, kind INTERNAL, with gen_ai.operation.name set to retrieval. Record gen_ai.retrieval.top_k (the maximum number of documents asked for) and, if you choose, gen_ai.retrieval.documents with each document's id and score. Set error.type when it fails.
Model call. Chat inference spans are named from the operation and model too (inference convention). Recommended attributes include gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, the counts the provider reports.
Answer. The prompt and completion are Opt-In attributes (gen_ai.input.messages and gen_ai.output.messages), so you decide whether to store them. Evaluation scores for the answer attach to this trace, covered below.

The retrieval convention also defines a gen_ai.retrieval.duration metric, a histogram in seconds, so retrieval latency can be tracked as a metric as well as per trace. Both it and the span are marked Development.

Judgment: the highest-value record is the retrieved document ids and scores next to the final answer. Without them you cannot tell a retrieval miss from a model failure.

Logging: opt in on purpose

The conventions treat content as opt-in. The query text attribute carries the warning "This attribute may contain sensitive information", and the retrieved documents attribute is Opt-In as well. The inference convention adds reference variants, such as gen_ai.input.messages_ref, defined as an absolute URI identifying the messages in external storage, so large or sensitive content can live outside the trace.

That gives three options for each field: keep it off, keep it in the trace, or keep a pointer in the trace and the content in a store you control.

Judgment: log document ids and scores by default and treat query text and full document bodies as sensitive until you have decided retention and access. Ids are enough to find a miss; bodies are needed only when you debug a specific case.

Scoring quality: retrieval metrics and answer metrics

Traces tell you what happened. Scores tell you whether it was good. Ragas lists a Retrieval Augmented Generation group of metrics, including Context Precision, Context Recall, Noise Sensitivity, Response Relevancy and Faithfulness, and says LLM-based metrics may use one or more LLM calls to produce a score.

Context Precision. Per Ragas, it evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones. The ContextPrecision variant needs a reference answer; ContextUtilization is for when you have the generated response but no reference.
Context Recall. Measures how many relevant pieces of information were retrieved, and per its page always requires a reference.
Faithfulness. Measures how factually consistent a response is with the retrieved context, from 0 to 1 (Ragas). Its example takes the user input, the response and the retrieved contexts, with no reference answer.
Response Relevancy. Measures how relevant a response is to the user input without evaluating factual accuracy (Ragas). Its example takes only the input and response.

Judgment: the metrics that need no reference answer (faithfulness, response relevancy, context utilization) are the ones you can run on production traces. The reference-based ones (context recall, context precision with a reference) belong in an offline test set with known answers.

Offline set and online sampling

Run reference-based metrics against a fixed question set whenever you change the chunking, embedding model or prompt. For live traffic, LangSmith's online evaluation docs describe running evaluators on production traces with filters (for example runs with user feedback, specific tool calls or certain metadata) and a sampling rate; their example of 0.1 applies the evaluator to 10% of traces to help control costs. They describe LLM-as-a-judge evaluators as a scalable substitute for human-like judgment.

Because LLM-based judges make their own model calls, sampling is a cost control, not only a performance one. Sampling decides which traces get scored; it should not decide which traces get stored if you need every failure.

Judgment: score a small sample of everything, and score all traces that carry a negative user signal, a retrieval error or an empty result.

Sending it somewhere: OpenTelemetry is the common thread

Phoenix says it accepts traces over OTLP. Langfuse documents an /api/public/otel endpoint, supports OTLP over HTTP (JSON and protobuf) but not gRPC yet, and says it aims to be compliant with the OpenTelemetry GenAI semantic conventions, which it notes are still evolving, mapping gen_ai.* attributes to prompts, completions, usage, cost and model parameters.

That is the practical reason to use the standard attributes: instrument once, and the backend can change. The tool-by-tool comparison is on our Top 10 page; the setup above does not depend on which one you pick.

Not settled: the GenAI conventions are in Development status, and the older opentelemetry.io page says they have moved to a separate repository. Pin your instrumentation version and expect attribute names to change.

A practical order of work, labeled as judgment

What follows is our synthesis, not a procedure any one source gives. First, get one parent span with embedding, retrieval and model-call children, and record retrieval top_k, document ids and scores, and token usage. Second, decide what content you will log and where. Third, build a small question set with reference answers and run context precision, context recall and faithfulness on it. Fourth, turn on sampled scoring that needs no reference answer on live traces and add the failure cases you find to the test set.

What this page does not claim: it does not say what scores are good, how much sampling is enough, or that any one tool is best. Those depend on your data, and the sources do not say.

Questions people ask

How do you set up observability for a RAG application?

Create one trace per request with a span for each step (query embedding, retrieval, model call), record retrieved document ids and scores, and attach evaluation scores. OpenTelemetry's GenAI conventions define embeddings, retrieval and inference spans, though they are marked Development.

What should I trace in the retrieval step?

Per the OpenTelemetry retrieval convention: a span named retrieval, the top_k requested, and optionally the retrieved documents with id and score. The query text is opt-in and flagged as possibly sensitive.

Which metrics evaluate a RAG pipeline?

Ragas lists Context Precision, Context Recall, Noise Sensitivity, Response Relevancy and Faithfulness, among others. Retrieval is judged by the first two, and the answer by faithfulness and response relevancy.

Do I need reference answers?

Some metrics do. Ragas says context recall always requires a reference. Its faithfulness example needs only the input, response and retrieved contexts, and its Response Relevancy example needs only the input and response.

Should I score every production request?

LangSmith's docs describe a sampling rate for online evaluators, with 0.1 giving 10% of traces as an example, to control cost. Our judgment is to also score every trace with a negative signal or retrieval error.

Is OpenTelemetry enough on its own?

It defines the spans and attributes. You still need a backend that accepts OTLP; Phoenix and Langfuse both document OTLP ingestion. See our Top 10 page for the tools.

Sources

OpenTelemetry GenAI conventions: Retrieval · Retrieval span, attributes, opt-in content, duration metric, Development status
OpenTelemetry GenAI conventions: Client Embeddings · Embeddings span name and kind
OpenTelemetry GenAI conventions: Client Inference · Inference span name, token usage, opt-in message attributes and _ref variants
OpenTelemetry GenAI conventions: Spans overview · Development status, index of embeddings, retrieval and inference conventions
Ragas: Faithfulness · Definition, 0 to 1 range, inputs
Ragas: Context Precision · Definition, reference versus no-reference variants
Ragas: Context Recall · Definition, reference required
Ragas: Response Relevancy · Definition, inputs
Ragas: Available metrics · List of RAG metrics
Arize Phoenix docs: LLM traces · OTLP ingestion, what RAG-style tracing captures
Langfuse docs: OpenTelemetry · OTLP endpoint, protocols, GenAI attribute mapping
LangSmith docs: Online evaluations · Sampling rate, filters, LLM-as-a-judge evaluators
Hero photo: "Assy-Turgen Observatory against the backdrop of the center of the Milky Way" by Astrobond, CC BY-SA 4.0, Wikimedia Commons · Credit for the photo at the top of this page

Facts were read from the linked pages on November 7, 2026. These are standards and product documentation pages, and the OpenTelemetry GenAI conventions are in Development status. Sections marked as judgment are our reading, not a source claim.

Top photo: Astrobond / CC BY-SA 4.0 via Wikimedia Commons