Maximize Inference ROI: Get 6.2x More Tokens/Sec with S3/TCP and KV$

AI cloud providers face the persistent challenge of maintaining SLAs for users running agentic AI workloads with extensive context windows. While performance is critical, architectural decisions are often driven by the need for cost efficiency to accommodate varied budgetary and performance needs.

When a LLM encounters a massive prompt (think a 50,000-token document) the GPU must invest significant compute time to calculate the KV Cache before token generation can begin. Under high concurrency, this prefill bottleneck can cause Time-to-First-Token (TTFT) to spike from milliseconds into minutes.

To avoid redundant recomputation, the KV Cache can be stored on high-performance storage and streamed to the GPU upon request. While GPUDirect Storage (GDS) over RDMA/NVMe-oF enables maximum throughput and minimizes TTFT, this blog explores a cost-effective yet highly performant alternative.

By utilizing S3/TCP as a backend, VAST Data engineers have validated an architecture that significantly improves efficiency. Our results show: 

• A 3.9x reduction in end-to-end response latency

• 620% more tokens as the average decode speed increases by 6.2x 

We did this achieving a peak throughput of 9.79 GB/s with only two RTX 6000 GPUs, reaching approximately 80% of line rate.

vLLM + LMCache + NIXL + S3

Moving your KV cache backend to S3 usually triggers immediate anxiety about latency. Standard cloud object storage is notoriously slow, introduces massive TTFT penalties, and turns what should be a speed boost into an architectural bottleneck.

While flash storage and GPU memory are architecturally distinct, VAST AI OS support for S3 provides consistent, low-latency performance needed to offload KV cache from expensive, limited GPU memory. This effectively transforms object storage into a hot, highly scalable, and cost-efficient extension of your compute infrastructure.

By layering LMCache and the NIXL storage backend over a VAST enterprise storage cluster, organizations get the absolute best of both worlds: the infinite, cost-efficient, and elastic scaling of an S3 bucket combined with sub-millisecond, line-rate NVMe performance. Setting it up requires zero complex cluster state management - just a clean config file, a standard vllm serve launch, and lmcache bench to watch the performance scale.

Here are the steps we took and the detailed results we achieved

Step1: Bucket Creation 

Follow the following guide to create a S3 bucket on VAST via VMS

https://kb.vastdata.com/documentation/docs/creating-s3-buckets-via-vms-6

For example: create a bucket named kvc. 

Assign an owner, group, and identity policy, then generate access and secret keys for the owner.

Configure the access and secret keys in ~/.aws/credentials

[default]
endpoint_url = http://vast3115-var.VastENG.lab
aws_access_key_id = xxx
aws_secret_access_key = yyy
response_checksum_validation = when_required
preferred_transfer_client = crt

In the example above, the S3 endpoint is configured to use the VAST DNS endpoint defined in the backend configuration.

Verify S3 connectivity from the GPU server using the AWS CLI. The bucket should initially be empty. For example:

kvc@rtx6k02-var:~$ aws s3 ls
2026-05-14 15:20:59 kvc

Step 2: Create a Python Virtual Environment and Install LMCache with vLLM

Note: LMCache now has a new deployment model called “MP Mode” for Multi Process mode, which comes with quite a few advantages in terms of performance, maintenance and fault tolerance. We’ll check it out soon. If you want to check it out for yourself, follow these steps instead.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv lmcache_vast  --python 3.12 --seed
source ~/lmcache_vast/bin/activate
uv pip install vllm nixl lmcache

Step 3: Configure LMCache Backend with NIXL + S3

LMCache supports multiple storage backends for offloading and sharing KV cache data across processes and nodes.

In this example, we configure the NIXL dynamic mode backend using an S3 bucket as the remote storage layer.

The NIXL storage backend supports a dynamic mode, which creates NIXL storage descriptors on demand instead of preallocating them during initialization. To enable dynamic mode, set: extra_config: nixl_pool_size: 0

This configuration is recommended for:

• dynamic workloads

• large-scale KV sharing

• reducing initialization overhead

• improving storage flexibility

Configure the NIXL + S3 backend YAML as follows (e.g /path/to/nixl_s3.yaml):

chunk_size: 1024
local_cpu: False
save_unfull_chunk: False
enable_async_loading: False
nixl_buffer_size: 4294967296
nixl_buffer_device: cpu
extra_config:
  s3_num_io_threads: 64
  enable_nixl_storage: true
  nixl_backend: OBJ
  nixl_pool_size: 0
  nixl_presence_cache: False
  nixl_async_put: False
  save_chunk_meta: true
  nixl_backend_params:
    access_key: xxx
    secret_key: yyy
    bucket: kvc
    region: us-east-1
    endpoint_override: http://vast3115-var.VastENG.lab

Step 4: Export the Required Environment Variables

Configure the following environment variables before starting the vLLM server:

export LMCACHE_CONFIG_FILE=/home/kvc/kvc/lmcache/s3/nixl_s3.yaml
export CUDA_VISIBLE_DEVICES=0,1
export PYTHONHASHSEED=0

Variable breakdown:

• LMCACHE_CONFIG_FILE Points to the LMCache backend configuration file. In this example, the configuration enables the NIXL + S3 storage backend.

• CUDA_VISIBLE_DEVICES=0,1 Restricts the server to GPUs 0 and 1 only. vLLM will use these GPUs for tensor parallel execution.

• PYTHONHASHSEED=0 Enables deterministic Python hash generation, helping ensure stable cache key generation and consistent prefix hashing behavior across workers and restarts.

• VLLM_ATTENTION_BACKEND=FLASH_ATTN Configures vLLM to use the FlashAttention backend for optimized attention computation, improving throughput and reducing memory overhead during inference.

Step 5: Serve vLLM

model=/mnt/kvcache/models/Llama-3.1-8B-Instruct/
vllm serve $model \
 --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}' --gpu-memory-utilization 0.85 --enable-prefix-caching --tensor-parallel-size 2

Step 6: LMCache Benchmarking

Create the following YAML configuration file named long-doc-qa.yaml for the LMCache Bench workload test:

{
  "workload": "long-doc-qa",
  "tokens_per_gb_kvcache": 16384,
  "model": "/mnt/kvcache/models/Llama-3.1-8B-Instruct/",
  "kv_cache_volume": 100.0,
  "ldqa_document_length": 40000,
  "ldqa_query_per_document": 2,
  "ldqa_shuffle_policy": "tile",
  "ldqa_num_inflight_requests": 12
} 

Run the LMCache benchmark using the long-doc-qa workload.

This workload simulates repeated question-and-answer requests over large synthetic documents:

• An initial warmup phase populates the KV cache for each document.

• The benchmark phase then issues additional queries against the cached prefixes to measure KV reuse efficiency.

The total number of generated documents is automatically derived from:

• kv_cache_volume

• tokens_per_gb_kvcache

• ldqa_document_length

rather than being configured explicitly.

In this example:

• kv_cache_volume: 100 represents a logical 100GB KV cache working set.

• tokens_per_gb_kvcache: 16384 estimates how many tokens fit into 1GB of KV cache for this model. ("tokens_per_gb_kvcache": 16384 = GPU KV cache size tokens: 928,128 / Available KV cache memory: 56.65 GiB)

• With 40k-token documents, the benchmark automatically generates approximately 40 documents and issues 2 queries per document.

Results

The latency overlay clearly demonstrates the impact of KV cache reuse with LMCache. Without caching, request latency continuously grows under large 40k-token prefills, quickly saturating the system leading to longer wait times for end user and agent queries. With LMCache and the VAST AI OS high performance S3 backend enabled, repeated prefixes are served directly from cache, collapsing latency from ~37s average down to ~9s and maintaining consistently fast response times. 

Once the cache is warm, the workload shifts from expensive prefill computation to lightweight decode execution, delivering dramatically higher throughput and near-interactive responsiveness.

We are using a 1MB block size for the S3-over-TCP workload and achieving a peak throughput of 9.79 GB/s with only two RTX 6000 GPUs, reaching approximately 80% of line rate.

Key Takeaways from the S3 + NIXL Architecture

• S3 Latency is Dead: Hitting a 0.79 minimum TTFT proves that VAST's high-performance object storage acts like local memory, completely bypassing the typical HTTP/S3 handshake and retrieval penalties.

• Massive Concurrency Protection: Under heavy prefill pressure, dropping peak TTFT from 36.78 down to 8.85 prevents the engine from cascading into an unrecoverable request queue backlog.

• 620% more tokens at full throttle - Because the engine skips the heavy lifting of prefix evaluation, the GPUs get a 6.2x decode boost that immediately magnifies token output.

During warmup write bandwidth holds steady at about 1 GB/s as the system populates the cache. Each document's KV state is computed once and persisted to S3. Once the cache is warm, the workload flips to read-dominated: as queries hit the cached prefixes, KV state is pulled back from S3 in a series of high-throughput bursts, peaking at 9.79 GB/s (9,794 MB/s) read bandwidth. 

At that peak, writes have dropped to under 1 MB/s, as the system is doing almost pure reads. With a 1 MB transfer block size over S3-over-TCP, this reaches roughly 80% of 100 GbE line rate using only two RTX 6000 GPUs.

This read/write asymmetry is exactly the profile a KV cache should have - written once, read many times - and the bursty read pattern maps directly to batches of queries arriving against the cache. Between bursts, the GPUs are decoding rather than waiting on storage. 

The headline takeaway is efficiency: a single modest two-GPU node drives the storage and network layer to near line rate - meaning the compute, not the storage backend, sets the ceiling. As GPUs and NICs are added, the same architecture has obvious headroom to scale.

A similar data structure to this will be created:

...
2026-05-18 15:24:23   64.0 MiB _mnt_kvcache_models_Llama-3.1-8B-Instruct__2_1_f59976a188d5671_bfloat16
2026-05-18 15:26:38   64.0 MiB _mnt_kvcache_models_Llama-3.1-8B-Instruct__2_1_f7f22757bdd1700_bfloat16
2026-05-18 15:26:25   64.0 MiB _mnt_kvcache_models_Llama-3.1-8B-Instruct__2_1_f91e3cd732b8302_bfloat16
2026-05-18 15:26:53   64.0 MiB _mnt_kvcache_models_Llama-3.1-8B-Instruct__2_1_fa70bc4b8db540b_bfloat16
2026-05-18 15:25:43   64.0 MiB _mnt_kvcache_models_Llama-3.1-8B-Instruct__2_1_fb9217af340d519_bfloat16
2026-05-18 15:25:47   64.0 MiB _mnt_kvcache_models_Llama-3.1-8B-Instruct__2_1_fdf85844cf436b1_bfloat16

Total Objects: 2934
   Total Size: 183.4 GiB

And can be removed with parallel cmd:

S3cmd ls s3://kvc | awk '{print $4}' | parallel -j 96 s3cmd del {}

Why It Matters

At inference scale, latency and GPU count are two sides of the same coin. Every second a GPU spends recomputing a prefix it has already seen is a second it isn't serving new tokens.

Redundant prefill doesn't just slow individual requests, it inflates the number of GPUs you need to hold a given latency SLA. By serving shared prefixes from cache instead of recomputing them, LMCache on the VAST AI OS S3 backend cut average request latency from ~37s to ~9s - a roughly 3.9x improvement. 

At a fixed SLA, that reclaimed GPU time translates fairly directly into capacity: the same node sustains far more concurrent requests, or the same workload runs on a fraction of the GPUs. For RAG, agentic, and multi-turn workloads - where prefix reuse is the rule rather than the exception - that's the difference between scaling out hardware and scaling up efficiency.

In summary:

These results demonstrate how combining vLLM with LMCache and the VAST AI OS on a high-performance S3 backend can significantly accelerate inference performance for long-context workloads. By leveraging LMCache for KV-cache acceleration over cost-optimized S3/TCP, the system avoids redundant GPU prefill computation, reducing GPU time per request with 3.9x reduction in end-to-end response latency and increasing average decode speed by 6.2X. 

For long-context, high-reuse inference patterns - common in RAG systems, agentic workflows, and multi-turn question answering - this architecture frees up GPU capacity allowing AI cloud providers and large enterprise AI infrastructure to host more concurrent user sessions per GPU, shorten the wait time for end users, and ultimately provides a compelling approach to improving both performance and infrastructure efficiency for large inference workloads. 

Performance and cost efficiency are critical priorities for large scale production inference providers. VAST provides options to optimize both, but we also go beyond lowering TTFT and maximizing infrastructure efficiency. The VAST AI OS provides KV cache data enterprise data services such as life cycle management, encryption, and data reduction. These features aren’t turned off to speed up performance; they are always on and always providing critical data services to our customers.

What’s Next

Keep an eye on our upcoming blogs for further insights and practical guides on optimizing KV Cache performance with LMCache, GPUDirect Storage, and other emerging architectural solutions in the market. 

VAST is set to introduce support for S3/RDMA later this year, providing an additional high-performance path for accelerated KV Cache offloading. 

Tensormesh, the company founded by the creators of LMCache, is working on many new features in LMCache and on a more advanced product named Tensormesh Operator that will potentially benefit any VAST customers.  We’ll keep you informed once they release their product in the coming months.