Latency (AI Context)

Why It Matters

Latency determines whether your AI feature feels instant or sluggish. Research consistently shows users expect responses within 100-200ms for interactive features, yet LLM inference typically takes seconds. Managing this gap is a core challenge for AI engineers.

The perception of latency matters as much as actual latency. Streaming tokens to users as they’re generated makes a 3-second response feel fast, while waiting for the complete response makes the same 3 seconds feel slow. Streaming is now standard practice for any user-facing LLM application.

For AI engineers, latency tradeoffs appear constantly. Larger models produce better outputs but are slower. More context improves answers but increases processing time. RAG adds retrieval latency before generation even starts. Every architectural decision involves latency considerations.

Implementation Basics

Latency in LLM systems breaks down into distinct components:

1. Network Latency Round-trip time to the inference server. Minimize with edge deployment, connection pooling, and request compression. For cloud APIs, geographic proximity matters, so choose regions close to users.

2. Time-to-First-Token (TTFT) Time from request receipt to first generated token. Dominated by prompt processing (prefill phase). Longer prompts mean longer TTFT. Optimize with prompt caching, smaller context windows, and efficient batching.

3. Time-per-Output-Token (TPOT) Time between subsequent tokens during generation. Determines how fast text appears during streaming. Affected by model size, batch load, and hardware. Users notice when TPOT exceeds ~50ms per token.

4. Total Response Time TTFT + (TPOT × output_tokens). For a 500-token response with 500ms TTFT and 30ms TPOT, total time is 15.5 seconds. Streaming makes this feel much faster.

Optimization strategies:

Streaming - Return tokens as generated. Users see progress immediately, dramatically improving perceived latency even when total time is unchanged.

Model Selection - Smaller models are faster. Use GPT-5 for complex tasks, o4-mini for simple ones. Route requests based on complexity.

Caching - Cache common responses. Semantic caching with embeddings can match similar queries. Even partial caching (system prompt KV cache) helps.

Async Processing - For non-interactive workloads, use async patterns with webhooks or polling. Don’t make users wait for long-running tasks.

Measure latency end-to-end from the user’s perspective, not just inference time. Include network, preprocessing, and post-processing in your metrics.