The 10 Best Semantic Caching Tools for LLM Apps in 2027
By Kory White · Updated Oct 2026 · 6 min read
Direct answer
GPTCache is the best semantic caching tool for LLM apps: free under the MIT License and built only for semantic caching. Redis LangCache is #2 as the managed service, and LiteLLM is #3 with Qdrant, Valkey and Redis backends. Upstash is the cheap Node.js pick.
A library for creating a semantic cache for LLM queries, described on GitHub as a semantic cache for LLMs fully integrated with LangChain and llama_index
Self-hosted teams that want a free, swappable embedding, vector store and similarity evaluator
A Node.js package that caches natural text by semantic similarity on top of an Upstash Vector database
Node.js and serverless apps that want a tiny cache on a hosted vector database
Pick by what you want
Best overall
GPTCache
Best managed
Redis LangCache
Best inside an LLM proxy
LiteLLM
Best open-source gateway
Bifrost
Best for Azure
Azure API Management Semantic Caching
Best for Node.js
Upstash Semantic Cache
The ranking
1
GPTCache
Answrs VerdictThe original open-source semantic cache, still free and MIT licensed.
What it is: A library for creating a semantic cache for LLM queries, described on GitHub as a semantic cache for LLMs fully integrated with LangChain and llama_indexGitHub stars: 8,216 stars as read on GitHub on Oct 28License: MIT LicensePrice: Free to use under the MIT License; you pay only for your own hosting and embedding callsCaveat: The README says the project no longer adds support for new APIs or models and recommends the get and set API
GPTCache is a library for creating a semantic cache for LLM queries, with 8,216 GitHub stars when we read the page on Oct 28 and an MIT License. It is free, so you pay only for your own hosting and embedding calls. One caveat from its README: the project no longer adds support for new APIs or models and points users to its get and set API.
Go ifSelf-hosted teams that want a free, swappable embedding, vector store and similarity evaluator
Skip ifYou need a managed service or support for the newest model APIs
Why #1 over #2: It is free and built only for semantic caching, where LangCache is a managed service with no published price.
2
Redis LangCache
Answrs VerdictThe managed option, run by the company behind Redis.
What it is: A semantic caching service, available fully managed on Redis Cloud or self-managed on your own infrastructureEmbeddings: Handles embeddings automatically, with no embedding model to manageControls: Configurable similarity thresholds, TTLs and eviction policiesIntegration: Two API calls to integrate: search before the LLM, store after; Python and JavaScript SDKs are on PyPI and npmPrice: Not published on the LangCache docs page
Redis LangCache is a semantic caching service, available fully managed on Redis Cloud or self-managed on your own infrastructure. It handles embeddings automatically and lets you configure similarity thresholds, TTLs and eviction policies. Integration takes two API calls, search before the LLM and store after, with Python and JavaScript SDKs. Redis does not publish a price on the docs page.
Go ifTeams that want a managed semantic cache without running embeddings themselves
Skip ifYou need a published price before you start
Why #2 over #3: It removes the embedding model work that LiteLLM leaves to you.
3
LiteLLM
Answrs VerdictSemantic caching inside the most-starred LLM proxy here.
Backends: Semantic caching supports three backends: Qdrant, Valkey and RedisHeader: A response served from a semantic cache carries an x-litellm-semantic-similarity headerWarning: The docs say semantic caching is designed for single-shot prompts and will replay stale responses on multi-turn or agentic trafficGitHub stars: 62,250 stars on the LiteLLM repository as read on GitHub on Oct 28Price: Not published on the semantic caching docs page
LiteLLM’s semantic cache supports three backends: Qdrant, Valkey and Redis, and marks served responses with an x-litellm-semantic-similarity header. Its docs warn that semantic caching is designed for single-shot prompts and will replay stale responses on multi-turn or agentic traffic. The LiteLLM repository had 62,250 stars on Oct 28.
Go ifTeams already routing many providers through one OpenAI-format proxy
Skip ifYour traffic is multi-turn or agentic and you cannot tolerate replayed answers
Why #3 over #4: It sits inside a far more widely used proxy than Bifrost, though Bifrost documents its defaults more clearly.
4
Bifrost
Answrs VerdictAn open-source gateway with clearly documented cache defaults.
Modes: Two modes: direct exact-match hashing and semantic similarity search; direct runs first and semantic runs on a direct missDefaults: Default similarity threshold of 0.8 and default TTL of 5m (300s)Multi-turn: Skips caching when a conversation has more than 3 messages by defaultGitHub stars: 9,181 stars as read on GitHub on Oct 28Price: Not published on the semantic caching docs page
Bifrost runs two modes, direct exact-match hashing and semantic similarity search, with direct first and semantic on a miss. The defaults are a similarity threshold of 0.8 and a TTL of 5m (300s), and it skips caching when a conversation has more than 3 messages. The repository had 9,181 stars on Oct 28.
Go ifTeams that want an open-source gateway with direct and semantic caching side by side
Skip ifYou want semantic mode without an embedding provider
Why #4 over #5: It publishes cache defaults in detail, which RedisVL’s guide does not.
5
RedisVL SemanticCache
Answrs VerdictThe library route to semantic caching on Redis.
What it is: A SemanticCache class in the RedisVL Python library that caches LLM responses by semantic similarityThreshold: The docs example sets distance_threshold to 0.1 on Redis COSINE, where lower is stricterTTL: Supports optional TTL policies so cache entries expireGitHub stars: 432 stars as read on GitHub on Oct 28License: MIT
RedisVL’s SemanticCache class caches LLM responses by semantic similarity in a few lines of Python. The docs example sets a distance_threshold of 0.1 on Redis COSINE, where lower is stricter, and supports optional TTL policies. The library is MIT licensed with 432 stars on Oct 28.
Go ifPython teams already running Redis who want a cache in a few lines of code
Skip ifYou are not on Redis or not using Python
Why #5 over #6: It is a library you control in code, while Apache APISIX needs a gateway deployment.
6
Apache APISIX ai-cache
Answrs VerdictA gateway plugin with strict defaults and an Apache license.
Layers: Two cache layers: exact (L1) and semantic (L2); L2 is disabled by default and enabled by adding "semantic" to layersDefaults: Default semantic similarity_threshold of 0.95 and semantic TTL of 86400 secondsRequirement: Semantic caching needs Redis Stack with the RediSearch moduleGitHub stars: 17,261 stars on the Apache APISIX repository as read on GitHub on Oct 28License: Apache-2.0
Apache APISIX’s ai-cache plugin has an exact layer (L1) and a semantic layer (L2) that is off by default. Semantic defaults are a similarity_threshold of 0.95 and a TTL of 86400 seconds, and it requires Redis Stack with RediSearch. The project is Apache-2.0 licensed with 17,261 stars on Oct 28.
Go ifTeams that run Apache APISIX as their API gateway
Skip ifYou cannot run Redis Stack
Why #6 over #7: It is fully open source, unlike Azure’s policy and Kong’s plugin.
7
Azure API Management Semantic Caching
Answrs VerdictBuilt-in semantic caching for Azure-hosted LLM APIs.
Policy: The llm-semantic-cache-lookup policy looks up responses from an external cache by vector proximity to previous prompts and a score thresholdTuning: Microsoft suggests starting with a score-threshold of 0.05 and warns that a threshold above 0.2 may lead to cache mismatchTiers: Applies to all API Management tiersCaveat: Microsoft warns semantic caching can surface responses that are incorrect, outdated or unsafe for the current requestPrice: Not published on the policy page
Azure API Management’s llm-semantic-cache-lookup policy returns cached LLM responses based on vector proximity and a score threshold. Microsoft suggests starting at 0.05 and warns that above 0.2 may cause cache mismatch, and that semantic caching can surface incorrect, outdated or unsafe responses. It applies to all API Management tiers. No price appears on the policy page.
Go ifAzure teams that already front their LLM APIs with API Management
Skip ifYou are not on Azure
Why #7 over #8: It is built into Azure, where Kong’s plugin needs an enterprise license.
8
Kong AI Semantic Cache
Answrs VerdictSolid for Kong customers, but enterprise only.
What it is: A plugin that stores user requests to an LLM in a vector database based on semantic meaning and retrieves them when a similar query is madeCache store: Supports Redis as a cache, including Redis with Redis Vector Search and Redis CloudAvailability: Only available as part of Kong AI Gateway EnterprisePrice: Not published on the plugin page
Kong’s AI Semantic Cache plugin stores LLM requests in a vector database by meaning and retrieves them for similar queries, using Redis, including Redis Vector Search and Redis Cloud. It is only available as part of Kong AI Gateway Enterprise, and Kong does not publish a price on the plugin page.
Go ifKong Gateway shops with an AI Gateway Enterprise license
Skip ifYou want an open-source option
Why #8 over #9: It supports Redis Cloud as a store, while Portkey requires its own vector database setup on an Enterprise plan.
9
Portkey
Answrs VerdictSimple cache for everyone, semantic cache for Enterprise.
Modes: Simple caching is an exact match on input; semantic caching matches semantically similar requestsAvailability: Simple caching is available for all plans; semantic caching requires a vector database and is only available on select Enterprise plansDefaults: Semantic hits need cosine similarity above the default threshold of 0.95, and work with requests under 8,191 tokens and 4 messages or fewerGitHub stars: 13,289 stars on the Portkey gateway repository as read on GitHub on Oct 28Price: Not published on the caching docs page
Portkey offers simple caching, an exact match on input, on all plans. Semantic caching requires a vector database and is only on select Enterprise plans. Hits need cosine similarity above the 0.95 default and work with requests under 8,191 tokens and 4 messages or fewer. The Portkey gateway repository had 13,289 stars on Oct 28.
Go ifTeams on Portkey Enterprise that want caching with the rest of the gateway
Skip ifYou need semantic caching on a non-Enterprise plan
Why #9 over #10: It documents hit rules clearly, but Upstash’s package is usable without a sales call.
10
Upstash Semantic Cache
Answrs VerdictTiny and cheap, but quiet on GitHub.
What it is: A Node.js package that caches natural text by semantic similarity on top of an Upstash Vector databaseThreshold: minProximity ranges from 0 to 1 and the README example uses 0.95GitHub stars: 304 stars as read on GitHub on Oct 28; the last push was in NovemberPrice: Upstash Vector lists a free plan at $0 with a 10K daily query and update limit and pay as you go at $0.4 per 100K requestsLicense: MIT
Upstash’s Semantic Cache is a Node.js package on top of an Upstash Vector database, with a minProximity setting from 0 to 1 and a 0.95 example. Upstash Vector lists a free plan at $0 with a 10K daily query and update limit and pay as you go at $0.4 per 100K requests. The package is MIT licensed with 304 stars on Oct 28, and its last push was in November.
Go ifNode.js and serverless apps that want a tiny cache on a hosted vector database
Skip ifYou need a library under active development
How we ranked these
We ranked tools whose own docs describe similarity-based caching of LLM calls, favoring clear thresholds, TTLs and requirements, and options you can use without an enterprise contract. The old list also had Milvus, MongoDB Atlas, Elasticsearch, LangChain, Helicone and Cloudflare AI Gateway. We dropped Helicone and Cloudflare because their docs describe caching for identical requests only, and the vector databases because they are storage rather than caching tools, and replaced them with Bifrost, RedisVL, Apache APISIX, Azure API Management and Kong. Facts are taken from each maker's own published information; check the official site before you buy.
Questions people ask
What is the best semantic caching tool for LLM apps?
GPTCache is our top pick: a free, MIT-licensed library built only for semantic caching of LLM queries, with swappable embedding, vector store and similarity components. Redis LangCache is #2 as the managed option and LiteLLM is #3 for teams that already use it as an LLM proxy.
Do Cloudflare AI Gateway and Helicone do semantic caching?
Not according to their own docs. Cloudflare says AI Gateway caching applies only to identical requests and that it plans to add semantic search in the future. Helicone’s caching docs describe caching of identical requests. We left both off for that reason.
Which semantic caches are free?
GPTCache, RedisVL and Upstash’s package are MIT licensed, and Apache APISIX is Apache-2.0. Upstash Vector, which its cache needs, lists a free plan at $0 with a 10K daily query and update limit. Kong’s plugin requires AI Gateway Enterprise and Portkey’s semantic cache is only on select Enterprise plans.
What similarity threshold should I use?
It depends on the tool. Portkey and Apache APISIX default to 0.95 cosine similarity and Bifrost defaults to 0.8. Azure API Management uses a score-threshold where lower is stricter; Microsoft suggests starting at 0.05 and warns that above 0.2 may cause cache mismatch.
GitHub stars on each project’s repository, as read on Oct 28 (Redis LangCache, Azure and Kong have no public repository for the cache) at a glance
Upstash Semantic Cache304 stars
RedisVL SemanticCache432 stars
GPTCache8,216 stars
Bifrost9,181 stars
Portkey13,289 stars
Apache APISIX ai-cache17,261 stars
LiteLLM62,250 stars
GitHub stars on each project’s repository, as read on Oct 28 (Redis LangCache, Azure and Kong have no public repository for the cache), as published by each maker. Picks without a published figure are left off.
