Between your question and the answer sits an engine. How good it is decides what private AI costs — and whether it runs on one small box or three hundred.
A chatbot asks the model one question and gets one answer. An agent does not. One user request can trigger five to ten model calls — planning, fetching, checking, retrying — and they run for minutes, holding the whole conversation in memory the whole time.
Most engines were built for the chatbot. Point agents at them and they buckle. Standard software can collapse at four or five long jobs on one GPU.
Lifeboat is Iterate.ai's proprietary inference engine — the layer that runs the model on your own hardware. It is built for the agent, and handles 25 or more long jobs on that same GPU.
Measured July 2026. One NVIDIA RTX PRO 6000 Blackwell GPU, Qwen 30B, long-context workloads simulating agents reading large documents in parallel. This work is continuous — the gap has widened at every round of testing so far.
Concurrent sessions at 100% request success.
Throughput at 2,048 concurrent sessions — 76% faster.
The story starts with a box you can carry, so people assume that is the limit. It is not. The same engine runs a departmental AIPod Mini and a full enterprise cluster. You change the hardware, not the software.
RTX-class GPUs in one box, in an office or a closet.
Tensor parallelism across 1 to 8 GPUs in a single machine.
300+ node clustering across a data center.
One Docker container · 168 model architectures · OpenAI-compatible API · air-gapped deployment.
Agents re-read the same conversation constantly. Compressing it frees 40% of the GPU.
Quick requests stay responsive while long agent jobs run, instead of queueing behind the slowest.
Tuned pipelines for mixture-of-experts models, so a capable model fits smaller hardware.
Two production workloads, neither on our hardware. The model stayed as it was in both, and gains on this scale have been consistent.
We did not replace the models. We made them run on less hardware, in less memory, for less money.
Private AI is an economics problem before it is a privacy one. Get twice the work from one GPU, and owning your inference stops being the expensive choice.
Full paper, 12 pages — iterate.ai/partners/netapp/papers. Benchmarks: iterate.ai/lifeboat.
A joint educational series on private AI and the AIPod Mini. NetApp — the governed data-control layer. Iterate.ai — the private intelligence layer.
