Amazon Web Services (AWS) has launched a new distributed KV caching agent skill designed to significantly cut costs and improve inference speed for self-hosted large language models (LLMs). The new capability aims to address the challenge of redundant computation and slow response times encountered when LLMs, particularly those used in AI agents and chat applications, need to reprocess large amounts of contextual data.
The Chenab Times has learned that this distributed KV caching solution, integrated into the Agent Toolkit for AWS, leverages Amazon ElastiCache to create a shared cache tier that supplements GPU-local caches. This allows any server within a GPU fleet to access and reuse Key-Value (KV) attention states across multiple sessions, even in the event of routing shifts or server failures, potentially reducing self-hosted LLM inference costs by up to 40% and decreasing time to first token (TTFT) by as much as 80%.
Self-hosting LLMs is increasingly popular among enterprises seeking greater control over cost, accuracy, and latency. However, the memory limitations of individual GPU servers often lead to inefficiencies. When an AI agent or application needs to access past context, such as extensive legal case history or complex code snippets, the GPU must recompute the KV attention states—a process known as prefill—if that history is not stored locally. This recomputation can result in response delays of several minutes.
The distributed KV caching skill aims to automate the evaluation and implementation of this solution. It guides coding agents through assessing workload suitability, setting up a cache cluster for testing or production, and troubleshooting inference caching. The skill is compatible with various coding agents, including Kiro, Claude Code, and OpenAI Codex, that support Agent Skills.
To facilitate adoption, the skill employs a structured approach to identify benefits and configure the system. Before any infrastructure is provisioned, it uses a tool called KVTrace to estimate potential savings based on specific workload characteristics such as LLM models used, user traffic patterns, and prompt routing. This evaluation is performed at no additional charge.
Assessing Suitability and Estimating Savings
The agent skill first performs a “fit check” by querying the GPU fleet to count Elastic Fabric Adapter (EFA) interfaces. It then requests essential details like the engine and model in use, tensor-parallel size, KV data type, server price, and prompt repetition frequency. The system will halt and provide an explanation if the workload does not appear to be a good fit for distributed KV caching.
A critical factor in determining the efficacy of KV caching is the prompt length relative to the recomputation time. For short prompts, recomputing KV state might be faster than retrieving it from a cache. However, beyond a certain “cutover point,” cache retrieval becomes more advantageous. The agent estimates potential savings by analyzing traffic patterns and prompt repetition to ensure the benefits outweigh the cache’s cost. This analysis is transparent, with each number clearly sourced.
For example, a legal AI assistant accumulating tens of thousands of tokens in a conversation might experience significant delays when a user returns after a break. Distributed KV caching can retain this context, allowing subsequent requests to access the precomputed states instantaneously, thereby drastically reducing the TTFT.
Configuration and Deployment
The agent skill manages cache configuration across inference servers using the existing technology stack. It integrates the LMCache connector with engines like vLLM or SGLang, ensuring that routing and engine settings remain intact. Servers in the inference fleet are configured with a unified KV ID mechanism and data type to prevent key mismatches, where different servers might generate incompatible KV states.
Before production deployment, the skill implements best practices, including emptying local tiers and verifying that two cache-loaded runs produce identical outputs, thus preventing errors caused by layout mismatches. For high-impact operations such as creating, modifying, or deleting resources, the skill requires explicit user approval.
Prerequisites for Implementation
To utilize the distributed KV caching skill, users will need:
- A coding agent supporting Agent Skills (e.g., Kiro, Claude Code).
- GPU instances with EFA RDMA write support, such as AWS’s p5 instances.
- LMCache version 0.5.8rc1 or later, and vLLM version 0.11.1 or later (or SGLang 0.5.13 or later).
- For setting up the cache, Amazon ElastiCache for Valkey version 9.1 or later, utilizing cache.i8ge.48xlarge nodes.
Troubleshooting and Support
The skill also includes a troubleshooting guide to address common issues. For instance, if an LMCache server connects successfully to the ElastiCache cluster and a basic test like BLOB.RDMA_HELLO succeeds, but actual data transfers fail with increasing rdma_write_errors, the agent can diagnose this as a potential security group misconfiguration blocking EFA traffic between instances. It provides commands to verify and rectify such security rules.
Users can access the distributed KV caching skill and the AWS MCP Server through the Agent Toolkit for AWS. Detailed information on distributed KV caching with ElastiCache and performance benchmarks can be found on the AWS blog, while setup instructions are available in the Amazon ElastiCache documentation. The KVTrace tool’s repository is hosted on GitHub.

The Chenab Times News Desk

