A tiered, persistent KV cache that keeps your agents’ context in host DRAM, NVMe and storage, and hands it back to the GPU on the next turn instead of recomputing it. Up to 3.6x the throughput and 3x the tokens per joule on the hardware you already own.
Every turn, an agent sends its whole session back to the model: the task, every tool call, every result. Turn 25 carries 24 turns of context with it. The KV cache for that context lives in GPU memory, and GPU memory is the most expensive memory you own.
Once 128 or 256 sessions no longer fit, the inference engine evicts them. The next turn recomputes the session from its first token. Time to first token climbs into the minutes, throughput falls, and the GPU burns full power doing work nobody sees.
The fix is not more GPUs. It is keeping the context somewhere cheaper than HBM and handing it back fast enough that the GPU never redoes the work.
LifeboatCache is a tiered, persistent KV cache for vLLM, SGLang and Lifeboat. When the engine runs out of GPU memory, LifeboatCache takes the evicted KV blocks into host DRAM. When DRAM fills, they move to local NVMe. From there they can go to S3-compatible object storage or NetApp ONTAP. On the next turn, the engine asks for the session and LifeboatCache streams it back from whichever tier holds it.
DRAM, then NVMe, then object or network storage. You buy as much fast memory as the workload needs and let cheaper storage absorb the rest. In our tests, 64 GB of DRAM still served 85% of prompts from cache because NVMe caught the overflow.
Under 90 minutes of sustained load at 256 agents, LifeboatCache stayed at 91–94% of prompts served from cache and finished the run. The alternative we tested slid from 91% to 47% and never completed.
OSCAR compresses KV to 2 bits in the offload tiers, halving stored bytes at the same speed. EpiCache evicts by topic rather than recency, so a long session keeps what it still needs.
At 256 agents under sustained load, LifeboatCache served 91–94% of prompts from cache and finished in 80 minutes. LMCache slid from 91% to 47% and never completed.
A live dashboard and Prometheus metrics show you hit rates, tier occupancy and bytes moved.
LifeboatCache uses vLLM’s native KV offloading interface. Start your server the way you already do, point one environment variable at the cache, and the engine begins offloading evicted blocks and pulling them back on demand.
LifeboatCache can sit underneath LMCache as its storage tier, so you keep your LMCache configuration and gain the persistent NVMe and storage tiers.
For SGLang, LifeboatCache is a HiCache storage backend: the same install, the same tiers. Lifeboat, Iterate.ai’s inference platform, embeds SGLang, so LifeboatCache is available there as an optional component with no extra setup.
If you run vLLM today, keep running vLLM. If you run SGLang, keep running SGLang. LifeboatCache is the cache; it is not asking you to move.
We measured three configurations on the same servers, with the same host DRAM budget, fresh caches for every run, and public agent traces replayed turn by turn.
| Plain vLLM | vLLM + LMCache | vLLM + LifeboatCache | |
|---|---|---|---|
| Where KV lives | GPU only | DRAM, then NVMe | DRAM, then NVMe, then S3 or ONTAP |
| Survives a restart | No | No | Yes |
| 4x RTX PRO 6000, 235B model, 128 agents | 70 tok/s | 87 tok/s | 208 tok/s |
| Prompts served from cache, same run | 1% | 24% | 79% |
| Time to first token, same run | 137.5 s | 113.5 s | 42.7 s |
| Output tokens per joule, same run | 0.053 | 0.065 | 0.159 |
| 90 minutes at 256 agents | — | Hit rate fell 91% → 47%, did not complete | Held 91–94%, completed in 80 min |
| AMD MI210, 7B model, 256 agents | 41 tok/s | 78 tok/s | 145 tok/s |
LMCache is a good offload layer and we support it. The difference shows up in two places: when the DRAM tier fills and the cache has to lean on NVMe, and when the run goes on long enough for eviction to start competing with loads. LifeboatCache is built for both.
LifeboatCache turns the memory you have into throughput you can serve.
3.0x on NVIDIA with a 235B model, 3.6x on AMD. Same capex, three times the sessions.
3x the output tokens per joule of GPU energy. The same 1,280 requests finished in 11.8 minutes instead of 34.8.
Give the cache 64 GB and it works. Give it 256 GB and throughput rises another 12%. You choose the point on the curve.
A returning user on an Apple M4 Max got a first token in 1.1 seconds instead of 49.8. After a server restart, 0.88 seconds.
| Engine | NVIDIA | AMD (ROCm) | Apple silicon | CPU servers |
|---|---|---|---|---|
| vLLM 0.32 | Measured | Measured MI210 | Measured CPU backend | Supported |
| vLLM + LMCache | Measured | Measured MI210 | — | — |
| SGLang 0.5.19+ | Measured | Measured MI210 | Measured MLX | — |
| Lifeboat 2.2 | Measured | Measured MI210 | — | — |
LifeboatCache also runs on data center servers such as H100, H200, B300 and others at scale.
LifeboatCache ships with Lifeboat, our confidential-compute inference engine. Turn it on and Lifeboat gets the DRAM, NVMe and storage tiers with no extra install.
Explore Lifeboat →If you already have vLLM or SGLang in production, LifeboatCache is a plugin for the engine you run today. No migration, no fork, no new hardware.
LifeboatCacheTest replays public agent traces against your server and reports time to first token, throughput, cache hit rate and GPU energy for plain vLLM, LMCache and LifeboatCache side by side. We will help you set it up and read the results.
All figures measured by Iterate.ai with LifeboatCache 0.1.23–0.1.28 against the same engine without offload, with the same host DRAM for every offload option and a fresh server for every run. Setups: 4x NVIDIA RTX PRO 6000 (Qwen3-235B-A22B-FP8), 1x RTX PRO 6000 (gpt-oss-20b), AMD Instinct MI210 (Qwen2.5-7B-Instruct), Apple M4 Max (Qwen3-0.6B). Method and raw results: docs.iterate.ai/lifeboatcache/benchmarks.
