What Is a Semantic Cache and How Much Can It Cut LLM Inference Costs in 2027?
A semantic cache stores earlier prompts and the responses they got, as embeddings in a vector store. When a new prompt arrives, it is embedded and compared with the stored ones; if one is similar enough, the saved response is returned and the model is never called.
How much it saves is a simple product: hit rate x the cost of the call you avoided, minus what the lookup itself costs. No neutral benchmark publishes a savings number that applies to everyone. Vendors do publish claims, and this page labels each one with whose it is: Redis advertises "Save 90% on API costs," and one Redis customer reports a 70% cache hit rate. The GPTCache README claims 10x lower costs without showing a benchmark. No source publishes a figure specific to 2027.
How a semantic cache works
The GPTCache README describes the idea this way: semantic caching "identifies and stores similar or related queries, thereby increasing cache hit probability." It splits the job into an embedding generator, a vector store that finds the K most similar earlier requests, and a similarity evaluator that decides whether a match is close enough (GPTCache README).
- Embed. The incoming prompt is turned into a vector by an embedding model. Redis's own example uses a 768-dimension model with COSINE distance (RedisVL guide).
- Search. The vector store returns the nearest stored prompts.
- Threshold. If the best match is within the similarity threshold, return its stored response. Otherwise call the model and store the new pair.
- Expire. Entries get a time to live so old answers age out. The RedisVL guide shows TTL support.
The threshold is the main control. In RedisVL it is a distance from 0 to 2, where 0 means identical and lower is stricter. Redis's guide says the right threshold "is not a fixed quantity" and depends on the embedding model, the queries and the business use.
Semantic cache vs exact-match caching
An exact-match cache returns a stored answer only when the whole request is identical. Cloudflare AI Gateway says its caching "is based on exact match of the entire request," so any change in messages or parameters is a separate entry. That is safe and cheap, but it misses "What is the capital of France?" versus "What actually is the capital of France?", which a semantic cache can match.
The price of that flexibility is a lookup on every request (an embedding call plus a vector search) and the chance of a wrong match.
Semantic cache vs provider prompt caching
Provider prompt caching is a different thing. It does not reuse answers; it reuses the processed prefix of your prompt, so the model still runs and still writes a fresh response. You pay a reduced rate on the repeated input tokens only. Published terms from the three big providers:
- AnthropicCache reads cost 0.1 times the base input price, and 0.05 times for Claude Opus 5.5 and Claude Sonnet 5.5. Writes cost 1.25 times (5-minute cache) or 2 times (1-hour cache). Default lifetime is 5 minutes. Source: Anthropic docs.
- OpenAICached input is "discounted up to 95%." For GPT-5.6 and later, reads cost 0.1 times the rate on most models, writes cost 1.25 times, and a prefix stays eligible for 30 minutes after its last use. Minimum prompt is 1,024 tokens. Source: OpenAI docs.
- Google GeminiImplicit caching is on by default for Gemini 2.5 and newer, and Google says it will "automatically pass on cost savings." The caching page does not state a discount percentage. Source: Gemini API docs.
The two stack. Prompt caching cheapens the calls you still make; a semantic cache removes some calls entirely. A semantic cache only helps if many users ask near-duplicate questions. Prompt caching helps whenever you resend a long shared prefix such as a system prompt or document.
Tools you can use
Each description below comes from that tool's own documentation.
- GPTCache (Zilliz). An open-source library built for semantic caching of LLM queries, with the embedding, vector store and similarity modules above (README, paper).
- Redis and RedisVL. RedisVL's SemanticCache stores prompt, response and metadata, with an adjustable distance threshold, TTL and tag filters such as a user id (RedisVL guide). Redis also sells a managed product, LangCache.
- LangChain. Caches are registered globally with set_llm_cache. The Redis integration offers RedisCache for exact match and RedisSemanticCache, whose examples use a distance threshold of 0.2 and a stricter 0.1 (LangChain docs).
- Azure API Management. The llm-semantic-cache-lookup policy returns a cached response based on vector proximity and a score threshold from 0.0 to 1.0 (Microsoft Learn).
- Cloudflare AI Gateway. Caching is off by default and exact-match only, so it is a response cache but not a semantic one (Cloudflare docs).
What savings have been published
Treat all of these as vendor or author claims, not independent measurements.
- Redis LangCache page: "Save 90% on API costs and shorten response times." No method is given.
- A Redis customer (Mangoes.ai): "a 70% cache hit rate, which saves 70% of our LLM spend," and 4X faster, in the customer's own words on the Redis page. The period and method are not stated.
- GPTCache README: "Slash Your LLM API Costs by 10x" and "Boost Speed by 100x." The README gives no benchmark data or methodology.
- GPTCache paper: Integrating GPTCache with OpenAI's GPT service "can increase response speed 2-10 times when the cache is hit." The abstract gives no hit-ratio figure and no cost figure (paper).
- RedisVL guide: A one-question timing test showed 97.26% of time saved. That is latency, not cost.
- Azure and Cloudflare: Their pages describe lower cost and latency but publish no measured hit rate or savings figure.
Risks: wrong answers, thresholds and privacy
- Wrong or stale answers. Microsoft's policy page warns that because semantic caching "returns responses based on similarity (not exact match), it can surface responses that are incorrect, outdated, or unsafe for the current request."
- Threshold tuning. Microsoft suggests starting at a low value such as 0.05 and adjusting, and says a score threshold above 0.2 "may lead to cache mismatch." Redis's guide shows a widened threshold of 0.5 still matching a loosely related question.
- Privacy. A shared cache can return one user's answer to another. Both Azure (the vary-by setting) and RedisVL (tag filters such as user_id) provide ways to partition entries per user or group.
- Freshness. Use a TTL for anything that changes. Neither vendor publishes a recommended TTL.
How to estimate your own savings
Use this formula: net savings = calls x hit rate x cost per call - lookup cost - cost of bad answers. The lookup cost is the embedding call and vector search made on every request, hit or miss.
Worked example with a published price. Anthropic's price table lists Claude Opus 5.5 at $4 per million input tokens and $20 per million output tokens. Assume a call with 2,000 input tokens and 500 output tokens (an assumption for this example).
- Cost per call2,000 input tokens x $4 per million = $0.008, plus 500 output tokens x $20 per million = $0.010, so $0.018 per call
- Assumed traffic100,000 calls a month at an assumed 30% hit rate (not a published figure)
- Gross savings100,000 x 0.30 x $0.018 = $540 a month, before lookup cost and any bad answers
Measure the real hit rate first: log every request with its nearest-neighbor distance for a few weeks without serving from the cache, then see what share would have been hits at your chosen threshold, and spot-check those matches for correctness.
Questions people ask
What is a semantic cache?
A cache that stores earlier prompts as embeddings with their responses, and returns a stored response when a new prompt is similar enough, instead of calling the model again.
How much can a semantic cache cut inference costs?
Savings equal hit rate times the cost of each avoided call, minus lookup overhead. Vendors claim large numbers, such as Redis's "Save 90% on API costs" and a customer's reported 70% hit rate, but these are not independent benchmarks and no 2027 figure is published.
Is a semantic cache the same as prompt caching?
No. Prompt caching, offered by Anthropic, OpenAI and Google, discounts repeated input tokens but still runs the model. A semantic cache skips the model call.
Does Cloudflare AI Gateway do semantic caching?
Its documentation describes caching as exact match of the entire request, so no. See the Cloudflare docs.
What similarity threshold should I use?
There is no universal value. Microsoft suggests starting at 0.05 on its 0.0 to 1.0 scale and warns about values above 0.2; Redis says the right threshold depends on your model and use case. Tune it on your own traffic.
Is it safe for private data?
Only if entries are partitioned per user or group, for example with Azure's vary-by setting or RedisVL tag filters. Otherwise a similar prompt from another user could return someone else's answer.
Sources
- GPTCache README (Zilliz) · What GPTCache is, its modules and the 10x and 100x claims
- GPTCache paper (Fu Bang) · The 2-10 times faster-on-hit figure
- RedisVL LLM caching guide · SemanticCache, thresholds, TTL, filters
- Redis LangCache · The 90% claim and the customer's 70% hit rate
- LangChain Redis cache integration · RedisCache and RedisSemanticCache
- Azure API Management llm-semantic-cache-lookup · Score threshold, warnings, vary-by
- Cloudflare AI Gateway caching · Exact-match caching
- Anthropic prompt caching · Cache multipliers and the Claude Opus 5.5 price table
- OpenAI prompt caching · Cached input discount and retention
- Google Gemini API caching · Implicit caching
This page is general information, not financial or engineering advice. Vendor documentation changes often, so check the linked pages before you budget around any figure.
Top photo: Carl Lender / CC BY 2.0 via Wikimedia Commons