What Infrastructure Does Retrieval-Augmented Generation Require in 2027?
Retrieval-augmented generation (RAG) needs eight pieces of infrastructure: an ingestion and chunking pipeline, an embedding model, a vector store or search index, a retriever with a reranker, a large language model (LLM), orchestration code, evaluation, and access control with monitoring. AWS describes the core loop plainly: convert data to text, "split it into manageable pieces," turn the chunks into embeddings, write them to a vector index, then at question time embed the query, find similar chunks and add them to the prompt.
You can assemble these yourself or rent them. Amazon Bedrock Knowledge Bases, Azure AI Search and Pinecone each cover part of the stack, and pgvector turns an existing Postgres database into the vector store.
Prices below are the ones each vendor publishes. Where a vendor publishes no price or limit, this page says so. Sizing advice is marked as judgment, and no figure here is a recommendation for your workload.
The eight components, in the order data flows
Think of RAG as two paths. The ingestion path runs ahead of time and prepares your content. The query path runs on every question.
- Ingestion and chunking. Connectors pull documents in, parse them to text and split them. Bedrock's default chunking "splits content into text chunks of approximately 300 tokens" and also offers fixed-size, hierarchical, semantic and no chunking.
- Embedding model. Turns each chunk and each query into a vector so the two can be compared.
- Vector store or search index. Holds the vectors, the chunk text and metadata, and answers nearest-neighbor queries.
- Retriever and reranker. The retriever fetches candidates. A reranker reorders them by relevance so "you can retrieve fewer, but more relevant, results," in AWS's words.
- The LLM. Receives the question plus the retrieved chunks and writes the answer.
- Orchestration. The code or managed service that sequences the steps, builds the prompt and handles retries. Azure describes a classic pattern, where your application sends one query and handles the handoff to the LLM, and an agentic pattern, where an LLM plans the sub-queries.
- Evaluation. Measures whether retrieval and answers are any good, before and after changes.
- Access control and monitoring. Makes sure people only retrieve what they may see, and shows you what the system is doing.
Ingestion, chunking and the embedding model
Chunking is a quality decision, not a plumbing detail. Azure AI Search says RAG quality "depends on how you prepare content for retrieval" and lists automatic chunking, more than 50 language analyzers, and indexers that pull from more than 10 Azure data sources.
For embeddings, OpenAI's guide lists text-embedding-3-small, text-embedding-3-large and text-embedding-ada-002, each with an 8192-token input limit. Default vector lengths are 1536 for the small model and 3072 for the large one. OpenAI's pricing page lists the standard rate as $0.02 per million input tokens for the small model and $0.13 for the large one.
- Embedding price (OpenAI, per million input tokens)text-embedding-3-small $0.02; text-embedding-3-large $0.13
- Vector length (OpenAI defaults)1536 (small), 3072 (large)
- Bedrock managed knowledge base embeddings$0 for the managed model, per AWS's pricing page
The vector store or search index
This is the part most people think of first. You have three families of choice.
- Your existing database. pgvector is "open-source vector similarity search for Postgres." Vectors can have up to 16,000 dimensions, though the plain vector type can be indexed only up to 2,000. It offers two approximate index types, HNSW and IVFFlat, and by default it "performs exact nearest neighbor search, which provides perfect recall."
- A search engine you may already run. Elastic offers textual, vector, hybrid and semantic search and document-level security. Its vector database page claims "up to 95% memory savings" with Better Binary Quantization and a roughly 32x smaller index, but states no datasets or test setup for those figures, so treat them as marketing claims until you test them.
- A dedicated managed vector service. Pinecone and Google Cloud's Vector Search are examples. Google says Vector Search uses the ScaNN algorithm, supports dense, sparse and hybrid search, and that a minimal setup under $100 per month "can accommodate high throughput for moderate-sized use cases." It publishes no latency or throughput numbers on that page.
One catch with filtering applies to Postgres-based stores. pgvector says that with approximate indexes "filtering is applied after the index is scanned." AWS repeats the warning for Aurora: without iterative scans, selective metadata filters "can return fewer results than expected."
Bedrock Knowledge Bases can sit on top of Amazon OpenSearch Serverless, OpenSearch Service managed clusters, Amazon S3 Vectors, Aurora, Neptune Analytics, Pinecone, Redis Enterprise Cloud or MongoDB Atlas. Only the OpenSearch stores support binary vectors. For a ranked comparison see our list of real-time vector search engines for AI agents and RAG.
Retriever and reranker: where quality is won
Vector search alone is rarely the whole retriever. Azure's classic pattern combines keyword and vector search ("hybrid queries combine keyword and vector search for better recall") and then applies a semantic ranker that, in Microsoft's words, "identifies the top 50 most relevant results."
- BenchmarkMicrosoft's own Azure AI Search test, NDCG@3 on four customer datasets, text-embedding-ada-002 embeddings and cosine similarity
- Keyword only40.6
- Vector only43.8
- Hybrid (keyword + vector)48.4
- Hybrid + semantic ranker60.1
Read this with its conditions. It is a vendor benchmark on customer datasets Microsoft does not publish, using an older embedding model, and the post itself carries a note that the team has since released newer learnings on RAG quality. The direction is the useful part: adding keyword search and a reranker scored higher than vector search alone on that test. Your own data may differ.
Rerankers are also sold per query. AWS lists the Cohere Rerank 3.5 model at $2.00 per 1,000 queries, where one query can hold up to 100 document chunks. Bedrock's managed knowledge base lists re-ranking with its managed reranker at $0.
The LLM, orchestration and managed alternatives
The LLM is the component you are most likely to rent. Because it receives only the retrieved chunks, Azure notes that "your retrieval system must return highly relevant, concise results," since models accept limited input. Retrieval quality therefore drives both answer quality and token cost.
Orchestration is either code you write or a service. Bedrock Knowledge Bases wraps the whole path. Azure offers agentic retrieval, and says its LLM-based query planning is in preview. If you want full control, keep orchestration in your own application and call each service separately.
- Bedrock managed knowledge base: index storage$5.00 per GB of raw data per month
- Bedrock managed knowledge base: standard retrieval$1.00 per 1,000 API calls
- Bedrock agentic retrieval (managed LLM planning)$4.00 per 1,000 Agentic Retrieve API calls, plus other charges
- Self-managed Bedrock knowledge base pricesNot published on the pricing page read
- Azure AI Search prices and limitsNot stated on the RAG overview page
What a managed vector service charges
Pinecone publishes the most detail of the dedicated services. Its Standard plan has a "$50/month minimum usage" and Enterprise a $500 per month minimum with a 99.95% uptime SLA. Bring Your Own Cloud has no published price.
- PlansStarter free; Builder $20 per month; Standard $50 per month minimum; Enterprise $500 per month minimum
- Storage$0.33 per GB per month on Standard and Enterprise
- Read unitsStandard $16-$18 per million; Enterprise $24-$27 per million
- Write unitsStandard $4-$4.50 per million; Enterprise $6-$6.75 per million
- Starter allowanceUp to 2 GB storage, 2M write units and 1M read units per month
Pinecone says prices vary by cloud and region, and that inference, Assistant and the initial data import are billed separately. The page publishes no total monthly cost for a typical RAG app, so none is given here.
Evaluation, access control and monitoring
Evaluation: Bedrock offers two RAG evaluation job types, "Retrieve only" and "Retrieve and generate," so you can judge retrieval separately from answer writing. Whatever tool you use, keep a fixed set of questions and rerun it after every change to chunking, the embedding model or the reranker.
Access control: Azure lists document-level security trimming, Microsoft Entra ID permission metadata for content indexed from Azure Storage, and network isolation through private endpoints. Elastic lists document-level security. If your store has no per-document permission model, the filter has to live in your own retrieval code.
Monitoring: Pinecone's Builder plan adds Prometheus and Datadog monitoring. The other vendor pages read for this article describe monitoring only in general terms, so check each one for what is logged and what it costs.
How to size it (judgment, not vendor data)
- Start with what you already run. If your data is in Postgres, try pgvector first. If you already run Elastic or Azure AI Search, start there. Fewer systems means fewer failure points.
- Add the reranker before adding hardware. The Microsoft benchmark above suggests ranking changes can matter more than the vector store itself, though that is one vendor's test.
- Budget the embedding step once and the retrieval step forever. Embedding is a one-time cost per document plus updates. Retrieval, reranking and the LLM are charged on every question.
- Plan access control on day one. Retrofitting permissions onto a finished index is harder than designing the metadata at ingestion.
- Also read: how to architect a RAG pipeline for low latency and the best retrieval and search infrastructure tools for AI.
Questions people ask
What are the main components of a RAG system?
Ingestion and chunking, an embedding model, a vector store or search index, a retriever with a reranker, an LLM, orchestration, evaluation, and access control with monitoring.
Do I need a dedicated vector database?
Not always. pgvector adds vector search to Postgres, Elastic and Azure AI Search combine keyword and vector search, and Bedrock Knowledge Bases can sit on several stores. A dedicated service such as Pinecone is one option among these.
How much does a managed vector database cost?
Pinecone publishes a $50 per month minimum on Standard, $0.33 per GB per month for storage, and read units from $16 to $18 per million. Prices vary by cloud and region.
How much does embedding cost?
OpenAI lists text-embedding-3-small at $0.02 and text-embedding-3-large at $0.13 per million input tokens on its standard tier. Bedrock's managed knowledge base lists its managed embedding model at $0.
Does a reranker help?
In Microsoft's own Azure AI Search test, hybrid search scored 48.4 NDCG@3 and hybrid plus the semantic ranker 60.1 on customer datasets. It is a vendor benchmark, so test it on your data.
What is not published?
Azure AI Search prices were not on the overview page read, Bedrock self-managed knowledge base prices are not listed on its pricing page, and Google gives no latency numbers on its Vector Search overview.
Sources
- AWS: How Bedrock Knowledge Bases work · Ingestion and runtime flow
- AWS: Content chunking · Chunking strategies
- AWS: Vector store setup · Supported vector stores
- AWS: Rerank models · Reranking
- AWS: RAG evaluation · Evaluation job types
- AWS: Bedrock pricing · Managed knowledge base and reranker prices
- pgvector on GitHub · Dimensions, indexes, filtering
- Pinecone pricing · Plans, units, limits
- OpenAI: Embeddings guide · Models, vector lengths, input limit
- OpenAI: API pricing · Embedding prices
- Microsoft Learn: RAG and Azure AI Search · Classic and agentic RAG, security
- Microsoft: Outperforming vector search with hybrid retrieval · Vendor benchmark
- Elastic: What is RAG · RAG definition, document-level security
- Elastic: Vector database · Quantization and reranker claims
- Google Cloud: Vector Search overview · ScaNN, hybrid search, cost note
- Wikimedia Commons: Pleiades supercomputer racks 2 · Hero photo, Oleg Alexandrov, CC BY-SA 4.0
- Wikimedia Commons: Google Servers · Server closeup, Wesley Nitsckie, CC BY-SA 2.0
RAG infrastructure is a chain: prepare the content, embed it, store and retrieve it, rank it, generate from it, and check and protect the result. Use each vendor's own pages for current prices and limits, since they change. This page is general information, not advice for your specific business.
Top photo: Oleg Alexandrov / CC BY-SA 4.0 via Wikimedia Commons