Hey everyone, welcome to the thirty-fourth issue of The Main Thread.
Let me start with a blunt statement: your LLM API invoice is a lie. It may not be a dishonest one but it definitely is an incomplete one.
It shows you what you paid for tokens but it doesn’t show you what you paid for:
the retries on failed requests
the evaluation runs you ran to trust those requests
the vector database that pre-selects context before the model even sees it
the engineer-hours lost debugging a prompt that costs $0.003 per call but produces subtly wrong outputs at a 15% rate
It doesn’t show you the cost of the caching layer you haven’t built yet - the one that would reduce 40% of your API calls. The gap between your invoice and your true cost is a tricky spot where most LLM apps bleed money.
This piece is about closing that gap by making full cost visible, so you can make deliberate tradeoffs instead of accidental ones.
TL;DR
Output tokens cost 3-5x input tokens: This is not arbitrary pricing, but a reflection of the compute architecture underneath
LLM applications have no natural economies of scale: The marginal cost per call is constant unless we engineer for it deliberately
The cost iceberg: retries, evaluation overhead, and embedding regeneration are often 30-50% of total API spend
Prompt caching has the highest ROI of any optimization, often immediate break-even, 80–90% savings on repeated contexts
Goodhart's Law applies to cost optimization: optimizing token count without regard for output quality can increase total cost by forcing more calls
Self-hosting breaks even at roughly 10–20M tokens/day for most use cases, below that, API pricing wins on total cost of ownership
Architecture of Token Pricing
Input tokens and output tokens are priced differently on every major LLM API. On Claude Opus 4.7, output tokens cost 5x input tokens. On GPT-5.5, this ratio is 6x. This isn’t arbitrary margin extraction, but it reflects a real asymmetry in how inference works.
Processing input tokens is a parallel operation. The transformer attends over all input tokens simultaneously in a single forward pass. Processing is fast, and the compute cost scales mildly with input length due to attention mechanisms quadratic complexity on the input.
Output tokens are generated sequentially. Each output token requires its own forward pass through the entire model, conditioned on all previous tokens. We can’t parallelize this because token N requires token N-1 to exist first. Ten output token means ten serial forward passes. This is why autoregressive generation is slow and why output tokens are expensive.
The practical implication of this is that output length is the highest-leverage cost variable in our application. A prompt that reliably generates 50-token answers instead of 200-token answers reduces cost by 50% on the output side. This is not about being stingy but about matching output length to task requirements.
# Structurally constraining output length
system_prompt = """You are a classification assistant.
Respond with EXACTLY one word: "positive", "negative", or "neutral". Do not explain your reasoning. Do not add punctuation."""
# With this prompt, output is always 1 token — not 50
# On Sonnet 4.6 at $15/MTok output: 50 calls with 50-token output = $0.0375
# Same 50 calls with 1-token output = $0.00075
# 50x cost difference from output length aloneStructured outputs and JSON mode compound this: forcing the model to produce parseable JSON adds tokens (the JSON scaffolding) but eliminates follow-up calls to fixed malformed responses. Often cheaper in total.
The KV Cache Asymmetry
Every LLM provider maintains a KV (key-value) cache of attention states of previously processed tokens. When we send the context twice, the provider doesn’t recompute the attention for the cached portion. This is why prompt caching dramatically changes the economics of repeated system prompts.
Anthropic’s prompt caching prices the write at $6.25/MTok and the read at $0.50/MTok (75% discount and write and 90% discount on read). This is not a minor optimization but a structural cost change that pays for itself in the first hour after deployment. Any application with a substantial, repeated system prompt that isn’t using prompt caching is overpaying by an order of magnitude.
The Absence of Economies of Scale
In classical software economics, marginal cost approaches zero as volume increases. A DB query that costs $0.01 in infra at 1K queries/day costs roughly $0.00001 at 100L queries/day. The fixed costs are amortized, caching comes into picture, and efficiency compounds.
LLM pricing doesn’t work that way. The marginal cost per API call is constant. 1000 calls/day at $0.01 per call → $10/day. 1M calls/day = $10k/day. This invoice scales linearly with usage unless we deliberately engineer away that linearity.
This is a fundamentally different cost structure than traditions SaaS infrastructure.
The economies of scale that do exist in LLM apps are not inherent but they must be built:
Caching turns repeated work into constant cost (exact-match and semantic caching)
Batching amortizes per-request overhead across multiple items
Model routing uses cheap models for easy tasks and reserves expensive ones for hard ones
Prompt optimization reduces tokens per call across the entire volume
None of these happen automatically. They require deliberate engineering investment. Without them, LLM bill grows linearly with user base.
The Cost Iceberg
The API invoice is just the visible portion. The iceberg runs deeper.
Retry and Cost Amplification
Every retry on a failed or insufficient response consumes real tokens. The cost becomes the sum of original failed call, retry, and disambiguation prompts.
The probability math matters here. If the prompt fails to produce usable response at rate f and it retries up to k times, the expected number of calls per successful response is:
E[calls] = 1 + f + f² + f³ + ... + f^(k-1)
= (1 - f^k) / (1 - f) for f < 1At a 5% failure rate with 3 retries: E[calls] ≈ 1.05: a 5% cost premium, negligible.
At a 20% failure rate: E[calls] ≈ 1.25: a 25% cost premium that silently inflates every line item.
At a 40% failure rate (common with complex extraction tasks on poorly optimized prompts): E[calls] ≈ 1.56: we are paying 56% more than we think per successful result.
This compounds with volume. An app making 100K calls/day at a 20% retry rate is paying for 125K calls. The 25K phantom calls cost real money and show nowhere in the cost attribution because they are invisible in usage dashboards unless we instrument for them explicitly.
We must fix the prompt before scaling the application.
Evaluation Overhead
Rigorous evaluation of LLM outputs cost real tokens. Running an LLM-as-judge on the outputs means every production call generates a second evaluation call, often against an equally expensive model.
For a system processing 50K calls/day with LLM-as-judge evaluation at 100% sampling:
Production calls: 50K × $0.01 = $500/day
Evaluation calls: 50K × $0.008 = $400/day
Total: $900/day vs. the $500/day that appears to be the cost
I am not saying we should skip evaluation. Without evaluation, we are shipping only vibes, which is worse than the cost. The correct response is to sample strategically: we can run evaluation on 5-10% of production traffic and supplement it with deterministic checks on 100%, and reserve LLM-as-judge for regression testing and quality audits rather than continuous production monitoring.
Embedding Generation Cost
Embeddings are cheap per call but generate a large hidden cost at scale because they are often generated unnecessarily. A document that’s embedded once and stored should never need re-embedding unless its content changes. In practice, many systems re-embed on every application restart, on schema migrations, or whenever the embedding model version changes.
A text-embedding-3-small pricing ($0.02/MTok) with 1M documents with 500 average tokens:
One-time embedding cost: 1M docs × 500 tokens × $0.02/MTok = $10
If re-embedded weekly: $10 × 52 = $520/year
If re-embedded on every deploy (10/week): $10 × 520 = $5,200/yearThe actual hidden cost here is regeneration frequency. Content-addressed storage (hash the document, skip re-embedding if matches) eliminates most of this:
import hashlib
def embed_with_cache(text: str, cache: dict) -> list[float]:
content_hash = hashlib.sha256(text.encode()).hexdigest()
if content_hash in cache:
return cache[content_hash] # Free: already embedded
embedding = embedding_model.encode(text)
cache[content_hash] = embedding # Store for future
return embeddingInfrastructure: The Build vs. Buy Crossover
The most significant infrastructure cost decision for a scaling LLM application is API pricing vs. self-hosting. This is a classic make-or-buy decision with a quantifiable crossover point.
API pricing has zero fixed cost and linear variable cost. We pay per token, we get managed infrastructure, and we carry no operational burden.
Self-hosting has high fixed cost (GPU hardware or rental) and near-zero variable cost. The marginal cost of an additional token on a rented GPU is effectively the idle cost to be paid anyway.
The break-even is roughly where API variable cost equals self-hosting fixed cost:
API model (e.g., Llama 3.1 70B via API): ~$0.35/MTok input, ~$0.40/MTok output
Self-hosting (A100 80GB on cloud): ~$3.00/hr, handles ~15 MTok/day at sustained load
Break-even: ($3.00/hr × 24hr) / ($0.40/MTok) ≈ 180 MTok/day output ≈ 10-15M tokens/day at realistic input:output ratiosBelow 10M tokens/day, API pricing wins on total cost of ownership - we are paying for GPUs we wouldn’t fully utilize. Above that threshold, self-hosting generates positive unit economics. The crossover is raw compute plus engineering time for model serving infra + upgrade costs + latency improvements from dedicated hardware.
Vector Database Costs at Scale
Vector databases present a similar build-vs-buy decision, but the cost structure is different: we pay per stored vector, not per query.
At 1 million 1536-dimension vectors (float32 = 6GB storage):
pgvector on a $50/month RDS instance: $50/month + operational time
Pinecone s1 pod: ~$70/month, managed
The pricing is competitive at small scale; the real cost advantage of self-hosted pgvector appears only when we have a dedicated infrastructure already in use for other purposes. For greenfield deployments under 10M vectors, managed vector databases typically win on total cost once engineering time is factored in.
Goodhart’s Law and the Cost Optimization Trap
Goodhart’s law, originally from monetary economics says:
When a measure becomes a target, it ceases to be a good measure.
In LLM cost optimization, it manifests as: when we optimize aggressively for token count, we often increase the total cost.
The failure mode: we shorten our system prompt to reduce input tokens. The model, with less instruction context, produces ambiguous or incorrect outputs more frequently. The failure rate rises from 5% to 20%. We have reduced per-call cost by 15% and increased effective cost per successful result by 20%.
Or: we switch from Claude Sonnet to Haiku to cut costs by 80%. Our classification task now needs re-prompt for ambiguous cases 30% of the time. $0.002 × 1.3 = $0.0026 vs. $0.010 × 1.02 = $0.0102. The cheaper model is genuinely cheaper here. But for a complex reasoning task, that 30% re-prompt rate might become 60%, and Haiku’s answers on re-prompt might still be worse and require human review that costs more than Sonnet would have.
The correct optimization is cost per correct output, not per API call. These are related, but not identical, and the gap between them widens as tasks become more complex.
def measure_effective_cost(
model: str,
prompt: str,
test_cases: list[dict],
cost_per_success: float, # Cost per correct result target
) -> dict:
total_api_cost = 0
total_calls = 0
successes = 0
for case in test_cases:
attempts = 0
while attempts < 3:
response = call_llm(model, prompt, case["input"])
total_api_cost += response.cost
total_calls += 1
attempts += 1
if is_correct(response.output, case["expected"]):
successes += 1
break
return {
"cost_per_api_call": total_api_cost / total_calls,
"cost_per_correct_output": total_api_cost / successes if successes > 0 else float("inf"),
"success_rate": successes / len(test_cases),
}We should run this measurement before and after cost optimization. If cost per API call drops, but per correct output rises, the optimization made things worse.
A Framework for Total Cost Accounting
The complete cost of an LLM feature has seven components:
Total Cost = API Costs
+ Infrastructure Costs
+ Hidden Costs
+ Operational Costs
API Costs:
1. Inference tokens (input + output, with retry multiplier)
2. Embedding tokens (generation + regeneration)
3. Evaluation tokens (sampling rate × eval model cost)
Infrastructure Costs:
4. Vector database (storage + query)
5. Caching layer (Redis/semantic cache)
6. Observability (logging, tracing, storage)
Hidden Costs:
7. Engineering time for prompt maintenance, evaluation, and debuggingMost teams account for 1-2. Few account for 3-6. Almost none account for 7, which is frequently the largest line item.
A practical accounting template:
@dataclass
class LLMFeatureCost:
# Monthly volume assumptions
daily_requests: int
avg_input_tokens: int
avg_output_tokens: int
failure_rate: float # Fraction requiring retry
eval_sample_rate: float # Fraction evaluated by LLM judge
# Pricing (per million tokens)
input_price_per_mtok: float
output_price_per_mtok: float
eval_model_price_per_mtok: float = 10.0
def monthly_api_cost(self) -> float:
monthly_requests = self.daily_requests * 30
# Expected calls per success
retry_multiplier = 1 / (1 - self.failure_rate)
input_cost = (monthly_requests * retry_multiplier
* self.avg_input_tokens
* self.input_price_per_mtok / 1_000_000)
output_cost = (monthly_requests * retry_multiplier
* self.avg_output_tokens
* self.output_price_per_mtok / 1_000_000)
eval_cost = (monthly_requests * self.eval_sample_rate
* (self.avg_input_tokens
+ self.avg_output_tokens)
* self.eval_model_price_per_mtok
/ 1_000_000)
return input_cost + output_cost + eval_cost
# Example: Customer support classification feature
feature = LLMFeatureCost(
daily_requests=50_000,
avg_input_tokens=800,
avg_output_tokens=50,
failure_rate=0.08,
eval_sample_rate=0.05,
input_price_per_mtok=3.0,
output_price_per_mtok=15.0,
)
print(f"Monthly API cost: ${feature.monthly_api_cost():,.2f}")
# Without accounting for retries and eval: ~$750/month
# With full accounting: ~$870/month: 16% higherThe gap between naive and full accounting grows with failure rate and evaluation intensity. At 20% failure rate with 10% eval sampling, the gap is 40%.
Cost Optimization That Actually Works
In priority order, by ROI:
1. Prompt caching (immediate ROI). If the system prompt is longer than 1000 tokens that's sent with every request, we should enable prompt caching today. This is the single highest-leverage optimization available. This works on most major providers. Expected savings: 80-90% on the cached portion.
2. Fix failure rate before scaling. A prompt with a 20% failure rate at 1000 requests/day becomes $2000/month of wasted spend at 100,000 requests/day. Evaluate the failure rate on a representative sample of production traffic. If it is above 5%, fixing it yields better ROI than any infrastructure optimization.
3. Model routing by task complexity. We should classify incoming requests and route simple tasks (classification, extraction with clear schema, yes/no questions) to smaller, cheaper models. Route complex tasks (multi-step reasoning, creative generation, ambiguous judgment calls) to capable models. A simple classifier (cheap to run) pays for itself in the first day.
def route_request(request: str, complexity_classifier) -> str:
complexity = complexity_classifier.predict(request)
if complexity == "simple":
# $0.25/MTok input — 12x cheaper than Sonnet
return "claude-haiku-4-5"
elif complexity == "medium":
# $3/MTok input
return "claude-sonnet-4-6"
else:
# $15/MTok input: for tasks that need it
return "claude-opus-4-6"
4. Semantic caching for repeated intent. Exact-match caching handles identical prompts. Semantic caching handles prompts that are different in wording but identical in intent. We should store embeddings of past prompts; retrieve and return cached responses for queries above a cosine similarity threshold. Hit rates of 20–40% are common for FAQ-style applications.
5. Output length control. Explicit instructions constraining output format reduce output tokens, which are the expensive half. JSON schema, length constraints, enumerated choices, and chain-of-thought suppression (for tasks where reasoning isn't needed in the output) all reduce output length with minimal quality impact on appropriate tasks.
6. Batch embedding calls. Most embedding APIs charge the same per-token regardless of batch size, but per-request overhead is amortized across the batch. Batching 100 strings into one API call instead of 100 individual calls reduces latency and API connection overhead, though it has minimal impact on token cost itself. More relevant for managing rate limits and request throughput.
7. Right-size the context. The context window piece established that 128K context doesn't mean we should use 128K context. Injecting 50 RAG chunks when 5 would suffice multiplies your input cost by 10x while degrading output quality due to the lost-in-the-middle effect. Retrieve fewer, more relevant chunks.
Mental Model: Opportunity Cost Accounting
Economists think about cost as opportunity cost, not just what we paid, but what we gave up. In LLM applications, the opportunity cost of not optimizing early is compounding overspend at scale.
A feature that costs $0.01/call and is called 10K times/day costs $3000/month. The same feature, optimized to $0.004/call through prompt caching and model routing, costs $1200/month. That $1,800/month difference is not savings: it's $21.6K/year that would have been spent had we not acted. At 10x growth, it's $216K/year.
But the inverse also matters: premature optimization at 100 calls/day optimizes a $9/month problem while spending $5,000 in engineering time. The break-even horizon matters.
The right time to optimize is when the feature shows traction but before it scales. After product-market fit, before growth mode. This is when optimization ROI is highest: the feature's volume is predictable, the call patterns are stable, and the savings compound over the feature's full lifetime rather than amortizing a late engineering investment.
Measure total cost. Make the full iceberg visible. Then optimize - in that order.
I hope you enjoyed this post and will develop the mental model to save tons of money. Let me know your thoughts in comments.
Namaste!
— Anirudh
