The world's first confidential compute inference engine, built for the agentic era. Lifeboat handles the repeated model calls, tool chaining, and long-running context that overwhelm traditional inference engines — while protecting data even while it's being processed, not just at rest or in transit. Run more concurrent agents on existing GPUs while maintaining full model quality and keeping data inside your perimeter.
Lifeboat is the world's first confidential compute inference engine — an LLM inference engine purpose-built for the concurrency and long-running context that agentic AI workloads demand, with confidential compute built directly into the engine to protect data in use. It runs on infrastructure enterprises already own, delivering 2-6x more concurrent capacity on the same GPU hardware while eliminating per-token cloud costs and keeping data inside the enterprise perimeter.
A single chatbot question is one model call. A single agent task can be dozens — planning, tool calls, MCP roundtrips, retries, and reflection, all riding on a conversation that keeps growing.
Lifeboat is purpose-built for the way agents actually consume inference. Instead of treating every request like an isolated chatbot prompt, it manages memory, scheduling, and model precision for fleets of long-running, tool-using agents on shared hardware. Four things make that possible:
The payoff: 2x or more concurrent agent sessions on the same GPU, full model quality intact, and performance that holds steady as load climbs.
Lifeboat isn't just a faster engine. It's a managed control plane for inference at scale, with a built-in cluster manager that orchestrates Lifeboat across nodes — and manages your other inference engines alongside it. One place to deploy, route, monitor, and govern the GPUs serving your agents.
Your data stays where it belongs — inside your perimeter, governed by your policies, and never exposed to shared AI services.
Use, tune, and control the models that power your workflows. Own what you build instead of depending on opaque external systems.
Run inference on infrastructure you control — on-premise or cloud VPC — and edge deployments.
Benchmarks used a Qwen 30B A3B model and long-context workloads that simulate real enterprise agents processing large documents in parallel.
Performance is only part of the challenge. Production AI requires governance, orchestration, security, and operational controls that raw inference engines don't provide. Lifeboat combines high-performance inference with the infrastructure required to run AI at scale.
Lifeboat is the world's first confidential compute inference engine — an
enterprise-grade, self-hosted LLM inference platform that combines high-performance
model serving with a complete operational control plane. Confidential compute is
built directly into the engine, protecting data even while it's being processed.
It delivers roughly 2x concurrent user capacity per GPU without quantizing model
weights, while providing a web dashboard, role-based access control, audit logging,
security capsules for sandboxed session isolation, and multi-node cluster
orchestration—all behind an OpenAI-compatible API.
Lifeboat uses patent-pending optimizations including FP8 KV-cache compression, TurboQuant vector quantization, adaptive memory management, and fairness-aware scheduling to roughly double concurrent-session capacity on the same GPU hardware. These optimizations compress the cache while keeping model weights at full BF16 precision, preserving output quality.
Lifeboat deploys as a hardened Docker container or via a bare-metal installer. It supports air-gapped, on-premise, or cloud VPC environments with no external dependencies beyond NVIDIA or AMD GPUs and HuggingFace Hub for model downloads. All state is managed via an embedded SQLite database, enabling deployment in restricted and regulated environments.
Yes. Lifeboat exposes an OpenAI-compatible API, making it a drop-in replacement for existing OpenAI client code. It also supports Anthropic and Ollama protocol compatibility, working seamlessly with LangChain, LlamaIndex, LiteLLM, and other standard LLM client libraries.
Lifeboat provides four-tier role-based access control (RBAC), API key management
with lifecycle controls, encryption at rest for secrets, comprehensive audit
logging with CSV export, container hardening (non-root user, dropped capabilities,
read-only filesystem), and optional TLS/HTTPS support. Every session runs inside a
security capsule for secure guardrailing and sandboxed execution, and confidential
compute protects data while it's actively being processed. It's designed for
self-hosted deployment — on-premise or cloud VPC — to keep private AI data within
your environment.
Lifeboat supports 168+ model architectures through the underlying SGLang runtime. Because it's built as a non-invasive overlay touching only three upstream files, new model architectures from SGLang are absorbed automatically with a single upgrade command, ensuring broad and current model support.
Yes. Lifeboat includes multi-node cluster orchestration with a control-plane node and node agents on each GPU host. It features token-aware weighted load balancing, capacity gating, and intelligent routing across cluster nodes, all managed through a unified gateway and web dashboard.
Lifeboat provides a comprehensive web dashboard covering real-time metrics, server lifecycle management, a searchable model catalog with capability probing, cluster orchestration, user and API key management, configuration presets, and a full audit trail—turning inference operations into a point-and-click experience with no terminal required.
Yes — Lifeboat ships a native ROCm build for AMD Instinct alongside the CUDA one. It is the same control plane, dashboard and OpenAI-compatible API, validated end-to-end on an Instinct MI210. Pull iterateai/lifeboat:latest-rocm instead of :latest — a container image is selected by CPU architecture rather than GPU vendor, so one tag cannot serve both. AMD hosts need no container toolkit; the GPU is passed through as the /dev/kfd and /dev/dri device nodes. Kernels are compiled per CDNA generation, so there is one image for MI210/MI250, one for MI300X/MI325X and one for MI350X/MI355X. The only feature that is NVIDIA-only is GPU confidential computing, which has no AMD equivalent; on AMD it reports as unsupported and inference runs normally.