How Do You Architect a RAG Pipeline for Low Latency in 2027?

By Kory White · Updated Oct 2026 · 8 min read

The Qdrant console, a vector database dashboard where a collection is created with a vector size and cosine distance

Direct answer

You architect a RAG pipeline for low latency by treating it as a chain of stages, measuring each one, and cutting the slowest first: embed the query, search the vector index, optionally rerank, then generate the answer. For most pipelines the language model call is the biggest lever, because OpenAI's latency guide says cutting 50% of output tokens may cut about 50% of latency, while cutting 50% of the prompt may only improve latency by 1-5%.

On the retrieval side, pick an approximate index (HNSW is the usual speed-recall choice per pgvector's docs) and tune its search width, compress vectors with quantization where the recall cost is acceptable, run keyword and vector search in one request, keep the reranking candidate set small, cache what repeats, and stream the answer so users see text early.

This page gives no end-to-end latency figure, because none exists that fits every stack: the numbers below are what each vendor or paper states about its own component and setup. Which settings are right for you is a recommendation to test against your own data and recall target.

Where the time goes in a RAG request

A RAG request is a pipeline, so its latency is the sum of its stages plus the network hops between them: embedding the question, searching the vector index (with any metadata filter), an optional reranking pass, then the model reading the retrieved text and writing the answer. OpenAI's latency guide adds that each request adds round-trip latency, which is why every extra service call between your app, your embedding model, your database and your reranker costs time.

Recommendation: time each stage separately in your own system before changing anything, using real questions and the recall you need. The right fix depends on which stage dominates, and the sources below describe components, not your stack.

Looking for products rather than design? Our list of retrieval and search infrastructure tools covers the vendors; this page is about how to wire them for speed.

Choose the ANN index, then tune its search width

Exact nearest-neighbor search compares the query with every vector, so production systems use approximate (ANN) indexes. HNSW, from Malkov and Yashunin's paper, builds a multi-layer graph, starts the search at the sparse upper layers, and the authors report roughly logarithmic scaling.

pgvector states the tradeoff plainly for its two index types:

HNSW: better query performance than IVFFlat in terms of speed-recall tradeoff, but slower build times and more memory.
IVFFlat: faster build times and less memory, but lower query performance in terms of speed-recall tradeoff.

The main query-time dial is the search width. In pgvector, hnsw.ef_search is 40 by default and ivfflat.probes is 1 by default. Weaviate explains the direction: a higher ef gives a more extensive search with better accuracy but potentially slower queries, and a lower ef is faster but may lose accuracy. For IVFFlat, pgvector suggests starting with rows / 1000 lists up to 1M rows and sqrt(rows) beyond that, with probes starting near sqrt(lists). Qdrant's indexing docs show a sample HNSW setup with m of 16 and ef_construct of 100.

Memory matters for latency too: Weaviate notes HNSW is an in-memory index where nodes and edges are stored in memory, so an index that no longer fits in RAM is a latency risk. Recommendation: pick the lowest search width that still meets your recall target on your own queries, rather than copying a default.

pgvector SQL example creating a vector column and ordering rows by distance to a query vector with LIMIT

For published comparisons, ANN-Benchmarks plots recall against queries per second. Read its setup before trusting a chart: the project enforces that only one CPU is saturated, with no multi-threading, its results used an r6i.16xlarge AWS machine, they are dated April 2025, and the repository says it is no longer actively maintained. Treat it as a guide to index families, not a ranking of today's products.

Shrink the vectors: dimensions and quantization

Smaller vectors mean less memory to scan. Two levers are documented. First, some embedding models let you shorten the vector: OpenAI's embeddings guide lists default lengths of 1536 for text-embedding-3-small and 3072 for text-embedding-3-large, and recommends its dimensions parameter for shortening.

Second, quantization. Qdrant's guide states:

Scalar quantizationMemory per vector reduced by a factor of 4; also speeds up search; Qdrant says the error it introduced in its experiments was usually less than 1%
Binary quantizationMemory footprint reduced by a factor of 32; up to a 40x speedup compared to the original vectors; only efficient for high-dimensional vectors; rescoring recommended
Product quantizationUp to 64x compression; slower than scalar quantization; recommended only for high-dimensional vectors

Those are Qdrant's own figures for its engine, not guarantees for yours. The common pattern is to search the compressed index, then rescore a few extra candidates with the original vectors. Qdrant's oversampling setting controls how many extra vectors are pre-selected and re-scored, and it warns rescoring can lower speed when the original vectors are stored on disk. Recommendation: measure recall with and without quantization on your data before adopting the largest compression.

Use hybrid search and filters without extra trips

Keyword (BM25) and vector search catch different matches. Elastic says hybrid search runs full-text and vector search in one request, and recommends reciprocal rank fusion, which merges the rankings and gives more weight to documents that score well in either. Weaviate runs the two searches in parallel, offers relativeScoreFusion (default since v1.24) and rankedFusion, and sets the weighting with alpha from 0 (keyword only) to 1 (vector only), with a server default of 0.75. One request to one engine avoids an extra network hop compared with stitching two systems together in application code.

Filters need care. pgvector warns that with approximate indexes, filtering is applied after the index is scanned, so with the default ef_search of 40, a condition matching 10% of rows leaves only 4 rows on average, and it offers iterative index scans to keep scanning until enough results are found. Qdrant advises creating payload indexes before ingesting data and indexing only the fields you filter on, since they use extra memory and disk.

Rerank a small candidate set

Reranking improves quality but adds a model call. Pinecone describes it as part of a two-stage retrieval process, and its examples retrieve a small set and keep fewer (a top_k of 4 and top_n of 2). The number you retrieve cannot exceed the reranking model's document limit, which Pinecone lists as 250 documents for cohere-rerank-4-fast and 100 for pinecone-rerank-v0, the latter at 512 tokens per query-document pair.

The research reason a reranker stays small is cost per query. The ColBERT paper says it pre-computes document representations offline, which considerably speeds up query processing, and reports effectiveness competitive with existing BERT-based models while executing two orders of magnitude faster and requiring four orders of magnitude fewer FLOPs per query. That is a result for that model against the BERT rerankers it compared, not a general claim about all rerankers. Recommendation: rerank only as many candidates as lift your answer quality in a test, and skip the stage if it does not.

Cut generation time: model choice, output length, prompt caching, streaming

The generation step is often the longest. OpenAI's latency guide says smaller models usually run faster (and cheaper), that cutting 50% of output tokens may cut about 50% of latency, and that cutting 50% of the prompt may only give a 1-5% improvement. For RAG, that argues for asking for concise answers, capping output with max_tokens and shortening any structured output syntax. It also says to run independent steps as parallel calls and to combine sequential steps to make fewer requests.

When many requests share a long, identical prefix (system instructions, tool definitions, a fixed document), prompt caching helps. Anthropic's prompt caching docs say it significantly reduces processing time and cost for prompts with consistent elements, that you generally see improved time-to-first-token for long documents, that it has no effect on output token generation, and that the default cache lifetime is 5 minutes with a 1-hour option at additional cost. Place the stable content first and the retrieved, per-question chunks last.

Finally, stream. OpenAI calls streaming the single most effective approach to making users wait less, because the first words arrive while the rest is still being generated.

Cache what repeats, and keep stages close together

The fastest retrieval is the one you skip. OpenAI's guide lists traditional caching among the ways not to default to a model call. A semantic cache answers a new question from a stored answer to a similar earlier one, at the risk of returning a stale or slightly wrong answer; we cover how that works and what it can save in What Is a Semantic Cache and How Much Can It Cut LLM Inference Costs, so it is not repeated here.

Recommendation, not a sourced figure: because each request adds round-trip latency per OpenAI's guide, run the app, vector database, reranker and model endpoint in the same region where you can, and avoid chaining extra network calls between them. Check each vendor's region options for your setup.

A practical order of work

Instrument first. Log time per stage: embedding, search, rerank, time to first token, total.
Fix generation. Smaller model if quality holds, shorter outputs, stable prompt prefix for caching, streaming on.
Tune the index. Lower the search width until recall on your own test set just meets your target.
Compress. Try shortened embeddings or scalar quantization with rescoring; check recall.
Trim stages. Rerank fewer candidates, merge sequential calls, parallelize independent ones, cache repeats.

This ordering is our recommendation, built from the cited vendor guidance. It is not a benchmark, and the right order depends on where your own measurements show the time going.

Questions people ask

What is the biggest source of latency in a RAG pipeline?

It depends on your stack, so measure each stage. OpenAI's latency guide says cutting 50% of output tokens may cut about 50% of latency, while cutting 50% of the prompt may only improve latency by 1-5%, which points at generation length before retrieval tweaks.

HNSW or IVF for low latency?

pgvector says HNSW has better query performance than IVFFlat in terms of speed-recall tradeoff, but slower build times and more memory. IVFFlat builds faster and uses less memory but has lower query performance.

Does quantization hurt accuracy?

Usually somewhat. Qdrant says the error introduced by scalar quantization was usually less than 1% in its experiments, notes binary quantization loses accuracy that rescoring recovers, and recommends it only for high-dimensional vectors. Test on your own data.

Is hybrid search slower than vector search alone?

The sources used here do not publish a latency comparison. Elastic says hybrid search runs full-text and vector search in one request, and Weaviate runs both in parallel, so it need not add a separate network round trip.

Should I always add a reranker?

No. Pinecone describes reranking as a second stage to improve result quality, and it adds a model call. Add it if your tests show better answers, and keep the candidate set small.

How does prompt caching help RAG?

Anthropic says it reduces processing time for prompts with consistent elements and generally improves time-to-first-token for long documents, but has no effect on output token generation.

Sources

Malkov and Yashunin: HNSW (arXiv) · Original HNSW paper, layered graph search
pgvector README · HNSW vs IVFFlat, ef_search, probes, filtering
Qdrant: Quantization guide · Scalar, binary and product quantization figures
Qdrant: Indexing · HNSW parameters, payload indexes
Weaviate: Vector index · ef speed and accuracy, in-memory HNSW
Weaviate: Hybrid search · Fusion methods and alpha
Elastic: Hybrid search · One-request hybrid search, reciprocal rank fusion
Pinecone: Rerank results · Two-stage retrieval and reranker limits
ColBERT (arXiv) · Offline document representations, speed claim
OpenAI: Latency optimization · Output vs input token impact, streaming, fewer requests
OpenAI: Embeddings guide · Default dimensions and shortening
Anthropic: Prompt caching · Time-to-first-token, cache lifetime
ANN-Benchmarks · Recall vs queries per second, setup and maintenance status

Figures were read from the linked pages on October 30, 2026. Each number is the source's statement about its own product, paper or setup, not a result for your system. Design advice marked as a recommendation is our judgement.