Definition: Inference latency is the time between sending a request to an AI model and receiving its response. For language models, it is often measured as time to first token and total generation time.Why It Matters: Latency shapes user experience and what applications are possible. Slow responses frustrate users, and in agentic workflows, delays add up across many calls. Reducing latency often involves trade-offs with cost, quality and throughput.Key Characteristics: Latency depends on model size, input and output length, hardware, batching, network distance and queueing. Common metrics are time to first token, tokens per second, and percentile latencies such as p95.
A request travels over the network to the endpoint, waits in a queue if capacity is busy, and is processed. For language models, the input is processed first, which determines time to first token, and the output is then generated token by token. Total time grows with output length. Techniques to reduce latency include smaller or quantized models, prompt caching, streaming responses, batching tuned for latency, faster hardware and running models closer to users. Teams measure percentiles rather than averages, since slow outliers affect users most.ProsLower latency improves user experience and enables interactive and multi-step applications.ConsOptimizations can reduce quality or raise cost, and tail latency is hard to control.
Chat Assistants: A team streams tokens so users see an answer begin within a second.Agent Workflows: An agent making many calls uses a smaller model for simple steps to cut total time.Edge Deployment: A retailer runs models near stores to avoid network delay.
Batch Prediction (2010s): Many models ran offline, so latency mattered less.Real-Time Serving (2015-2020): Online recommendation and vision systems required low latency.LLM Generation (2022-2023): Token-by-token generation made streaming and time to first token key metrics.Current Practice and Trends (2023-Present): Optimized inference engines, caching and specialized hardware reduce latency.
When to Use: Focus on latency for interactive applications and multi-step agents.Designing for Reliability: Set latency targets by percentile and test under load.Operating at Scale: Monitor percentiles, autoscale and route simple tasks to smaller models.Governance and Risk: Keep optimizations within data and region rules, and verify quality after changes.
