Latency Optimization for LLM Applications

Hey everyone, welcome to the thirty-sixth issue of The Main Thread.

If your LLM app takes three seconds to load, users are gonna abandon using it. They click the button, see a spinner, count to three under breath (well, not literally), and switch tabs. By the time response arrives, they are reading email. The app technically worked but the user experience failed miserably.

LLM latency is unlike any other latency you have optimized for. A typical web request finishes in 50-200ms; a slow one is 500ms; anything over a second is a bug. With LLMs, your fast path is 800ms to first token, and a multi-step agent flow can easily run 10-30 seconds end to end. The model doesn’t give a damn that your users are impatient.

What makes this tractable is that perceived latency and actual latency are different problems. You can’t make GPT/Claude/Grok generate tokens faster than the inference stack allows, but you can make a 10 seconds response feel like 1 second. You can also avoid generating tokens at all when you don’t need to. Good engineers that ship fast LLM products understand both.

This is a tour of the techniques that actually move the needle in production. No micro-optimizations, no magic. Just the patterns that consistently turn slow LLM apps into ones that feel snappy.

What “Latency” Actually Means for LLMs

Before optimizing anything, we need to measure the right thing. LLM latency decomposes into:

TTFT (time to first token). From requests sent to first token of the response received. This is dominated by the network, queueing at the inference provider, and prompt processing time (which scales with prompt length).
TPOT (time per output token). Steady-state generation speed. Roughly inverse to model size: small models stream at 100+ tokens/second, frontier models at 30-80 tokens/second.
Total latency. TTFT + (output_tokens * TPOT). For a 500-token response on a frontier model, this is easily 8-10 seconds.

These three numbers behave very differently and respond to very different optimizations. Caching reduces TTFT. Switching to a smaller model improves both TTFT and TPOT. Streaming doesn’t change any of them but it changes the perceived latency dramatically because users see output starting at TTFT instead of total latency.

If we only have one number, we are optimizing blind. We need to instrument all three from day 1.

import time

class LLMTiming:
    def __init__(self):
        self.start = time.monotonic()
        self.first_token_at = None
        self.last_token_at = None
        self.output_tokens = 0

    def on_token(self):
        now = time.monotonic()
        if self.first_token_at is None:
            self.first_token_at = now
        self.last_token_at = now
        self.output_tokens += 1

    def report(self):
        ttft = self.first_token_at - self.start
        generation_time = self.last_token_at - self.first_token_at
        tpot = generation_time / max(self.output_tokens - 1, 1)
        return {
            "ttft_ms": ttft * 1000,
            "tpot_ms": tpot * 1000,
            "total_ms": (self.last_token_at - self.start) * 1000,
            "output_tokens": self.output_tokens,
        }

We should log this for every request. The aggregate distribution (p50, p95, p99) is what tells us where to send optimization effort.

Streaming: The Single Biggest Win

If we do nothing else, we must stream.

A non-streaming request makes the user wait for the entire response. A streaming request shows the response as it generates. The total latency is identical but the perceived latency is dropped by 5-10x.

This works because users don’t measure latency with a stopwatch. They measure it by when something starts happening. A response that begins in 800ms and takes 8s to complete feels faster than a response that takes 4s to appear all at once.

Every major LLM provider exposes streaming over Server-Sent Events:

from anthropic import Anthropic

client = Anthropic()

with client.messages.stream(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain TCP slow start."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

Server-side, we forward the stream to the browser as SSE:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

@app.post("/chat")
async def chat(request: ChatRequest):
    async def event_generator():
        async with client.messages.stream(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            messages=request.messages,
        ) as stream:
            async for text in stream.text_stream:
                yield f"data: {json.dumps({'delta': text})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(
        event_generator(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",  # Disable nginx buffering
        },
    )

There are two operational landmines:

Buffering Kills Streaming

If anything between the app and the browser buffers the response (nginx, a CDN, a load balancer), the user sees nothing until the whole response is flushed. The X-Accel-Buffering: no header disables nginx’s buffer. CDNs need to be explicitly configured for streaming responses; some don’t support it all. We need to test the full path from production.

Streaming Doesn’t Help if the Model is Silent

If the prompt is huge (long context, large RAG payload), TTFT can be several seconds. Streaming starts the clock at TTFT, so users still see a long blank wait. To fix this, we need to send a placeholder immediately (“Thinking…”, “Searching for relevant docs…“) so the UI is responsive even before the model produces tokens.

For agents and tool-use flows, stream an intermediate state too. Show “Calling search tool…” and “Reading 3 documents…” before the final answer. Users tolerate long runs when they can see what’s happening; they abandon them when the spinner doesn’t change.

Parallelism: Stop Waiting in Series

Most LLM apps do too much sequentially. Look at any agent flow and we will find independent operations stacked on top of each other when they could run side by side.

Few common offenders:

Embedding the user query, the querying the vector store, then calling the LLM. The embedding and (in some flows) a metadata DB lookup can happen concurrently.
Running a moderation check, then the main LLM call. Run them in parallel, cancel the LLM call if moderation fails.
Calling multiple tools sequentially when they have no data dependency on each other.
Generating multiple completions for an A/B comparison one after another instead of concurrently.

The fix is asyncio.gather (Python), Promise.all() (Node), or whatever your runtime calls it.

import asyncio

async def handle_query(user_query: str):
    # All three are independent — run concurrently
    embedding_task = asyncio.create_task(embed(user_query))
    moderation_task = asyncio.create_task(moderate(user_query))
    user_context_task = asyncio.create_task(load_user_context())

    embedding, moderation, user_ctx = await asyncio.gather(
        embedding_task, moderation_task, user_context_task
    )

    if not moderation.allowed:
        return {"error": "Content blocked"}

    docs = await vector_store.search(embedding, top_k=5)

    return await llm.complete(
        system=build_system_prompt(user_ctx, docs),
        user=user_query,
    )

Total latency drops from embed + moderate + load + search + llm to max(embed, moderate, load) + search + llm. On real workloads this is often a 30-50% reduction in TTFT.

For agent flows that call multiple tools, the model itself can request parallel tool calls - both Anthropic’s and OpenAI’s function-calling APIs return multiple tool_use blocks in a single response. We must honor that parallelism in our executor:

async def execute_tool_calls(tool_uses):
    results = await asyncio.gather(*[
        execute_tool(tool_use) for tool_use in tool_uses
    ])
    return results

If you agent loop runs tools serially when the model returned them in parallel, we have thrown away the model’s intent and slowed every multi-tool turn by Nx.

The hidden cost: Parallel calls multiply the rate-limit pressure and concurrent connections. Provider rate limits are typically requests/minute and tokens/minute. A naive fan-out can blow through both. We need to use a Semaphore to cap concurrency, and implement backoff with jitter on 429 responses.

Model Selection: A Tradeoff Framework

The single most overused phrase in LLM engineering is “we use GPT-5.5“ for everything”. It’s almost always wrong.

Different model tiers have order-of-magnitude differences in latency, cost, and capability. A frontier model at 50 tokens/s and $15/M output tokens is not the same product as a small model at 200 tokens/s and $0.25/M tokens. Using one when the other suffices is wasted under user time and wasted budget.

A pragmatic framework for picking models:

Workload	Recommended tier	Why
Classification, routing, intent detection	Smallest reliable model	Output is short; quality plateau is reached early
Extraction with a clear schema	Small or mid model	Structured output, low ambiguity
Summarization of moderate complexity	Mid-tier	Quality matters but not at frontier-level
Open-ended reasoning, code generation, planning	Frontier	Capability gap is real and visible
Final user-facing response in a multi-step flow	One tier above the intermediate steps	Users notice quality on the output, not the scaffolding

The pattern that consistently works: use cheap, fast models for the scaffolding and frontier models only for the steps that actually need them. A typical retrieval flow might use a small model for query rewriting and intent classification, a mid-tier model for re-ranking candidates, and a frontier model only for the final response generation. The user-perceived latency is dominated by the final step, but the cumulative latency of the scaffolding is the wall we hit when we scale.

Concretely, with Claude's current lineup: route classification or simple extraction to Haiku 4.5; reach for Sonnet 4.6 for most production work; reserve Opus 4.7 for genuinely hard reasoning. Mixing tiers within a single user request is normal, not exotic.

There's a measurement trap here too. People benchmark models on capability and cost, but rarely on latency under their actual prompt sizes. A model with 50 tok/s steady-state and 300ms TTFT is faster end-to-end than one with 80 tok/s and 1200ms TTFT for short responses, and slower for long ones. We must measure with the real prompt distribution.

Caching, Properly Layered

Caching is the highest-leverage latency optimization in LLM apps because the cheapest token is the one we never generate. There are three meaningfully different caching layers, and the best apps use all of them.

Layer 1: Exact-match response cache

If the exact same input produces the exact same (deterministic) output, we should cache the entire response. This is the standard HTTP-style cache, keyed by a hash of the canonical request

import hashlib
import json

def cache_key(model: str, messages: list, temperature: float) -> str:
    canonical = json.dumps(
        {
         "model": model, 
         "messages": messages, 
         "temperature": temperature
       },
        sort_keys=True,
    )
    return hashlib.sha256(canonical.encode()).hexdigest()

async def cached_complete(model, messages, temperature=0.0):
    if temperature > 0:
        # Cache only deterministic requests
        return await llm.complete(model, messages, temperature)

    key = cache_key(model, messages, temperature)
    cached = await redis.get(f"llm:{key}")
    if cached:
        return json.loads(cached)

    result = await llm.complete(model, messages, temperature)
    await redis.set(f"llm:{key}", json.dumps(result), ex=86400)
    return result

Caveats:

Only safe for temperature=0 or (near-zero). Higher temperatures intentionally produce different outputs each time.
Don’t cache outputs that depend on time, user state, or anything not in cache key. If the prompt includes “current date: 2026-06-01” only via the system, we may need to include that in the key explicitly.
For RAG, the retrieved documents are part of the input. Cache the full prompt, not just the user query.

Layer 2: Semantic Cache

When users ask the same question with different wording, exact-match misses. Semantic caching embeds the query and looks for prior requests with high cosine similarity, returning the cached answer if the match is found above some threshold.

async def semantic_cached_complete(query: str, threshold: float = 0.95):
    query_embedding = await embed(query)

    matches = await vector_store.search(
        query_embedding,
        top_k=1,
        namespace="semantic_cache",
    )

    if matches and matches[0].score >= threshold:
        return matches[0].metadata["response"]

    response = await llm.complete(query)

    await vector_store.upsert(
        id=hashlib.sha256(query.encode()).hexdigest(),
        embedding=query_embedding,
        metadata={"query": query, "response": response},
        namespace="semantic_cache",
    )

    return response

Semantic caching is powerful and dangerous. The threshold is the whole game. If we set it too low, the system will return wrong answers to subtly different questions (“How do I cancel my subscription?“ ≠ “How do I pause my subscription?“). If we set it too high, we might as well use exact-match.

We should use semantic caching where:

Wrong-but-plausible answers are tolerable (FAQ chatbots, doc search assistants).
Queries cluster around a finite set of intents.
You can fall back to a real LLM call cheaply when the user signals dissatisfaction.

Avoid it where:

Stakes are high (anything financial, medical, legal).
Each user has different context that affects the right answer.
The query space is genuinely open-ended.

Layer 3: KV Cache and Prompt Caching

This is the LLM specific one and the one most apps underuse.

When a transformer processes a prompt, it computes attention key/value tensors for every token. For a prompt of N tokens, this O(N ^ 2) work and is what dominates TTFT for long prompts. The clever observation is that if the next request shares a prefix with a previous one, the KV cache for the prefix can be reused.

Both Anthropic and OpenAI exposes this as “prompt caching”. Anthropic requires explicit cache control breakpoints; OpenAI caches automatically for prompts above a certain size. The savings are large: reused cached tokens are billed at a fraction of normal input price (around 10% of the input rate at Anthropic, 50% at OpenAI), and TTFT for the cached portion drops dramatically.

This matters most for:

Long system prompts that are identical across requests. Mark them as cacheable; we stop paying full processing cost on every request.
RAG with stable corpora. If the same set of document is included in many requests, a cached prefix is enormous.
Multi-turn conversations. Each turn shares the prefix of all prior turns. With prompt caching, the marginal cost of an additional turn is roughly the cost of just the new turn’s tokens.

# Anthropic: explicit cache breakpoint after a long system prompt
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT,  # 5,000 tokens of policy/context
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=conversation_history,
)

The cache has a TTL (5 minutes for Anthropic’s ephemeral tier, with a longer-lived option available). Design the prompt structure so the cacheable parts are at the front and stable across requests. A typical order is: system prompt, then tool definitions, then RAG context, then the per-request user message at the end. If we mutate the system prompt per request, we lose the cache. We should be deliberate about where the variable parts go.

The combined effect of these three layers: exact-match catches the obvious repeats; semantic catches the paraphrased repeats; KV/prompt caching makes the requests we do send cheaper and faster.

Speculative Execution and Pre-Computation

This is where we start anticipating the user instead of just reacting to them.

Speculative execution at the application layer

Many LLM apps have a moment where we can predict, with high confidence, what the user is about to do. Start the work before they ask.

Examples:

Auto-suggest in a search bar. Pre-fetch results for the top suggestion as the user types.
"Continue" or "regenerate" buttons. Pre-compute the regeneration in the background as soon as the first response renders.
Multi-step wizards. Pre-fetch the next step's data based on the most likely answer.
Email or document drafting. Pre-generate the next paragraph while the user reads the current one.

async def handle_chat_with_speculation(user_message, history):
    # Stream the primary response
    primary = stream_response(user_message, history)

    # In parallel, speculatively pre-compute likely follow-ups
    speculation = asyncio.create_task(
        speculate_follow_ups(user_message, history)
    )

    yield from primary

    # Cache the speculative results for instant follow-up
    follow_ups = await speculation
    await cache.set(f"speculation:{session_id}", follow_ups, ex=300)

The cost of this is the wasted compute if we guessed wrong so we must track our speculation hit rate. If it’s below 30%, we are paying for nothing so we need to either fix our predictor or stop speculating

Speculative Decoding (inside the model)

A different technique with the same name: speculative decoding uses a small “draft” model to generate several tokens ahead, which the larger model then verifies in a single forward pass. When the draft is right (which it usually is for easy tokens), we get multiple tokens for the cost of one. This is how providers serve large models at higher tokens/second than naive autoregressive generation would allow.

We don’t implement it ourselves unless we are running our own inference. But it’s worth knowing it exists. It’s why hosted models often stream faster than we’d expect from raw model size, and why latency improvements appear without changing anything.

Pre-computation of stable artifacts

Anything that doesn’t depend on the live request can be pre-computed. This sounds obvious but it is routinely missed:

Embeddings of the document corpus. Compute once at ingestion time, never at query time.
Summaries of long documents that get included in prompts. Generate offline; reference the summary, not the original.
System prompts assembled from many sources. Build them at deploy or config-change time, not per request.
User profile contexts. Maintain a denormalized "context blob" that’s updated when the user's data changes, instead of rebuilding it on every chat.

The pattern is: move work from the request path to the background job. Every millisecond we delete from the request path is a millisecond every user benefits from.

Edge Deployment: What Helps and What Doesn’t

Edge deployment (Cloudflare Workers, Vercel Edge, Fastly Compute, etc.) is often pitched as a latency optimization. For LLM apps, it’s more nuanced then the marketing suggests.

What Edge Deployment Helps With:

Network roundtrip to the gateway. A user in Sydney hitting and edge POP is Sydney saves ~150ms versus a US-East origin. This is real, especially for streaming where every millisecond before TTFT counts.
Authentication and routing. Doing JWT validation, rate limiting, and request routing at the edge keeps the cold path short.
Caching. Edge KV stores (Cloudflare KV, Vercel Edge Config) are great for storing exact-match LLM caches near the user.

What Edge Deployment Doesn’t Help With:

The LLM call itself. The LLM provider has a finite set of regions. The edge worker still has to reach across the internet to OpenAI/Anthropic. Putting the gateway in 200 cities doesn’t change that.
Long-running compute. Edge runtimes have CPU time limits and request duration caps. A 30-second agent flow probably doesn't fit.
Stateful workloads. Edge runtimes have limited memory and constrained APIs. If the app has heavy in-process state, the edge model fights us.

The pattern that works in practice: edge for the front door, regional for the agent loop, provider region for LLM call. The edge worker terminates the connection, validation auth, checks the cache, and either returns cached content immediately or proxies the request to a regional backend that runs actual LLM logic. The regional backend should be deployed in or near the LLM provider’s region.

A thing that's easy to get wrong: edge worker cold starts. V8 isolates start fast (single-digit ms), but they still cost something on the first request to a region. For low-traffic geos, the cold-start tax can swamp the latency win. We should measure end-to-end from real user devices in real geos before committing.

Checklist

Before you ship an LLM-powered feature:

Measurement:

You log TTFT, TPOT, and total latency per request
You track p50, p95, p99: not just averages
You can break down latency by model, prompt template, and user segment

Streaming:

All user-facing endpoints stream
No buffering layer between server and browser silently breaks streaming
Long-prompt requests show a placeholder before TTFT
Agent intermediate states are streamed to the UI

Parallelism:

Independent operations run concurrently, not sequentially
Multi-tool calls from the model execute in parallel
Concurrency is capped to respect provider rate limits
Backoff with jitter on 429 responses

Model selection:

Each LLM call uses the smallest model that achieves required quality
Routing/classification uses a small model
Frontier models reserved for the steps that need them

Caching:

Exact-match cache for deterministic requests
Semantic cache where wrong-but-plausible is tolerable
Prompt caching configured for stable prefixes
Prompt order: stable cacheable parts first, variable parts last

Speculation and pre-computation:

High-probability follow-ups pre-computed in the background
All artifacts derivable from non-request data are computed offline
Speculation hit rate is measured and above 30%

Edge:

Edge handles auth, routing, and cache lookup
Heavy/stateful logic runs in the region nearest the LLM provider
Cold-start tax measured per geo, not assumed

Takeaway

LLM latency is a product problem disguised as a systems problem. The model's speed is largely outside your control; everything else is yours.

Every fast LLM product does four things consistently.

They stream every user-facing response, even when the underlying call is fast, because perceived latency is what users feel.
They run independent work in parallel and never let the structure of their code force operations into a series.
They use the smallest model that does the job and reserve the frontier model for the moments where the quality gap is real.
They cache aggressively at every layer where the input distribution allows it.

Everything else, speculative execution, edge deployment, pre-computation, is a multiplier on top of those four. They are worth doing, but they don't substitute for the basics. If your app feels slow and you haven't done the four basics yet, do them first.

Three seconds will still feel like forever to your users. Make sure they never have to wait that long staring at a spinner.

What's the most surprising latency win you've shipped in an LLM app? I am collecting examples: the optimizations that turned out to matter way more than expected, and the ones that didn't move the needle at all. The patterns are educational.

Hit reply. I read everything.

If this clicked, forward it to your team. Latency optimization is a discipline, not a one-time fix. And if you want more deep dives like this, subscribe to The Main Thread: practical engineering for AI and distributed systems, one essay per week.