Hey everyone, welcome to the thirty-ninth issue of The Main Thread. This issue is all about adding observability to the LLM applications.

A traditional service is observable by construction. Requests have well-defined shapes, responses have predictable structures, errors throw exceptions you can count, and latency distributes around a stable mean. When something goes wrong, we read the logs, find the bad request, and trace it through our system.

I wrote a tweet a few days ago on “how staff engineers debug in prod”, you must take a look at it:

LLM applications break almost every assumption that makes traditional observability work. The "request” is a free-form prompt. The “response” is a generated string whose correctness we can’t easily verify. Errors are often silent: the model returns a valid-looking answer that is wrong. Latency varies by prompt size, model load, and provider region in ways that defeat naive aggregation. Cost is a per-request variable instead of a fixed line item. Whether the system is “healthy” is no longer a question with binary answer.

After working for almost two years with LLM apps, it is a common occurrence that no one is able to answer basic questions: How many tokens did we burn yesterday? Which feature is driving most of our spend? Which user requests are getting bad responses? When the answer to all three is “I don’t know”, we just have an experiment running on a our customers, not a product.

This is the missing manual for instrumenting LLM systems. Most of the tooling is two years old or less, the patterns are still being worked out, and the engineers who do this well borrow heavily from the existing observability tradition while adding the things that are genuinely new. Let’s get into the details.

What to Log: Start With the Right Primitives

Before tools, before dashboards, we must decide what we are capturing per LLM call. The minimum useful payload looks like this:

Request identity

  • Request ID (correlation ID across the stack)

  • User ID, session ID, tenant ID

  • Feature / endpoint that originated the call

  • Trace ID and parent span ID

Model invocation

  • Provider, model name, model version

  • Temperature, top_p, max_tokens, other sampling params

  • Tools / functions provided (names, not full schemas)

  • Whether streaming was used

Inputs and outputs

  • Full prompt (with privacy considerations — see below)

  • Full completion

  • Stop reason (end_turn, max_tokens, tool_use, stop_sequence)

  • Tool calls returned, if any

Performance

  • TTFT (time to first token)

  • TPOT (time per output token)

  • Total wall-clock latency

  • Provider-side latency vs network time, if separable

Economics

  • Input tokens (prompt)

  • Output tokens (completion)

  • Cached input tokens (prompt-cache hits)

  • Cost in dollars, computed from current pricing

Outcome

  • HTTP status, error type, retry count

  • Downstream evaluation result, if any (more on this later)

  • User feedback signal (thumbs up/down, edited, ignored), if available

This list looks long but it’s the actual minimum. Cutting any of these will block a question we will need to answer later. The cost of capturing them is trivial compared to the cost of not having them when we need them.

A working envelope:

@dataclass
class LLMCallRecord:
    # Identity
    request_id: str
    trace_id: str
    parent_span_id: str | None
    user_id: str | None
    tenant_id: str | None
    feature: str

    # Invocation
    provider: str
    model: str
    params: dict
    tool_names: list[str]
    streaming: bool

    # Payload (subject to redaction policy)
    prompt: str
    completion: str
    stop_reason: str
    tool_calls: list[dict]

    # Performance
    ttft_ms: float | None
    tpot_ms: float | None
    total_ms: float

    # Economics
    input_tokens: int
    output_tokens: int
    cached_input_tokens: int
    cost_usd: float

    # Outcome
    error: str | None
    retries: int
    feedback: str | None
    eval_score: float | None

We push this through a single endpoint - wrap the LLM client so every call goes through one path that emits one record. We mustn’t sprinkle logging across every call site. We might forget to update half of them when the schema evolves.

class InstrumentedLLMClient:
    def __init__(self, client, sink):
        self.client = client
        self.sink = sink

    async def complete(self, model, messages, feature, ctx, **params):
        record = LLMCallRecord(
            request_id=ctx.request_id,
            trace_id=ctx.trace_id,
            parent_span_id=ctx.span_id,
            user_id=ctx.user_id,
            tenant_id=ctx.tenant_id,
            feature=feature,
            provider="anthropic",
            model=model,
            params=params,
            ...
        )
        start = time.monotonic()
        try:
            response = await self.client.messages.create(
                model=model, messages=messages, **params
            )
            record.completion = extract_text(response)
            record.input_tokens = response.usage.input_tokens
            record.output_tokens = response.usage.output_tokens
            record.cached_input_tokens = response.usage.cache_read_input_tokens or 0
            record.stop_reason = response.stop_reason
            record.cost_usd = compute_cost(model, response.usage)
        except Exception as e:
            record.error = type(e).__name__
            raise
        finally:
            record.total_ms = (time.monotonic() - start) * 1000
            await self.sink.emit(record)
        return response

The sink can fanout to whatever destinations we need: a structured log file, a tracing backend, a data warehouse for offline analysis, a real-time dashboard. The point is that every call produced one canonical record.

Tracing: Multi-Step Chains Are Where The Value Is

A single LLM call is interesting. A chain of LLM calls, tool invocations, vector store queries, and intermediate computations is what we actually have in production, and it’s where flat per-call logs fail.

Consider an agent flow:

Typical agent flow

Five LLM calls, two tool invocations, one embedding, one retrieval. Total latency: 8 seconds. Where did the time go? Which step failed when the response was wrong? Which call exploded the cost? We cannot answer any of these questions from per-call logs. We need a trace.

OpenTelemetry has formalized this with the GenAI semantic conventions (`gen_ai.* attributes), and the major providers and frameworks are converging on it. The mental model is identical to distributed tracing for microservices: spans nested inside spans, with timing, attributes, and events on each.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def handle_question(question, ctx):
    with tracer.start_as_current_span("handle_question") as span:
        span.set_attribute("user.id", ctx.user_id)

        with tracer.start_as_current_span("classify_intent") as s:
            s.set_attribute("gen_ai.system", "anthropic")
            s.set_attribute("gen_ai.request.model", "claude-haiku-4-5")
            intent = await llm.complete(...)
            s.set_attribute("gen_ai.usage.input_tokens", intent.usage.input_tokens)
            s.set_attribute("gen_ai.usage.output_tokens", intent.usage.output_tokens)

        with tracer.start_as_current_span("retrieve") as s:
            embedding = await embed(question)
            docs = await vector_store.query(embedding)
            s.set_attribute("retrieval.doc_count", len(docs))

        with tracer.start_as_current_span("synthesize") as s:
            answer = await llm.complete(
                model="claude-sonnet-4-6",
                messages=build_messages(question, docs),
            )

        return answer

What this gives us in a trace viewer is the gantt chart we need: each span as a horizontal bar, nested under its parent, with timing and attributes visible at every level. We can see at a glance that “synthesize” took 6 of the 8 seconds, that retrieval returned 50 documents when it should have returned 5, that the tool-call inside the synthesis hit a 2-second timeout.

A few rules that separate useful traces from noisy ones:

Span boundaries should match the conceptual step, not code structure

A span per LLM call, per tool invocation, per retrieval. Not a span per function. The trace should read like a story of what the system did, not a stack trace.

Attach LLM-specific attributes consistently

Use the OpenTelemetry GenAI conventions: gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens. The whole point of convention is that downstream tooling understands them. Inventing our own attribute names locks us into custom dashboard forever.

Capture enough payload to reproduce the call

When we find a failed span, we should be able to copy the prompt and re-run it locally. Truncating the prompt to the first 200 characters to “save space” is the false economy we will regret first.

Sample if must, but whole traces

If we sample 10% of traces to control storage costs, we should sample at the trace level - keep all spans of a trace together. A trace with 80% of its spans missing tells us nothing.

Cost Attribution: Where the Money Actually Lives

A traditional service has a flat infrastructure cost: we pay for the boxes regardless of what runs on them. For LLM apps, cost is per-call, varies widely per request, and accumulates faster than most of us realize.

Cost attribution is a discipline of mapping every dollar spent on LLM APIs back to a user, feature, team, or use case. Without it, we have a single line in our invoice labeled “Anthropic” and no idea what’s driving it. With it, we can answer questions like:

  • Which feature accounts for 40% of our LLM spend?

  • Which 10 users account for 30% of token consumption?

  • Did the new prompt template increase per-request cost?

  • Is the prompt cache actually paying off?

The mechanics are simple if we captured the right primitives. Every LLM call record has a feature, user_id, tenant_id, and cost_usd. Roll those up:

-- Cost by feature, last 7 days
SELECT
    feature,
    SUM(cost_usd) AS total_cost,
    COUNT(*) AS call_count,
    SUM(input_tokens) AS total_input_tokens,
    SUM(output_tokens) AS total_output_tokens,
    SUM(cached_input_tokens) AS total_cached_tokens,
    AVG(cost_usd) AS avg_cost_per_call
FROM llm_calls
WHERE created_at >= NOW() - INTERVAL '7 days'
GROUP BY feature
ORDER BY total_cost DESC;

Following are three patterns we will find on first look at a real cost breakdown:

A small number of features dominate spend

Power-law distribution, every time. The expensive features are usually the ones with long context windows (large RAG payloads) or high call frequency (per-keystroke completions). Knowing which is which directs our optimization effort.

A small number of users dominate per-feature spend

Some users are running scripts. Some are stress-testing. Some are using the product in ways we didn’t anticipate. Per-user cost views are how we find them. We may decide to rate-limit, charge differently, or build a different product for them, but we can’t decide anything if we can’t see them.

Cached input tokens save more than we’d think

If cached_input_tokens is 0 across the board, we are not actually using prompt caching, regardless of what our provider’s pricing page says. Verify it from the data, not from the config.

For multi-tenant or B2B systems, cost attribution has a stronger purpose: it is the basis for chargeback or fair-use enforcement. Customers who consume disproportionate compute should be on a metered plan, not a flat fee. The data we need to build that pricing model lives in our call records.

A budget alerting layer on top is essentially free once we have the data:

async def check_budget(tenant_id: str, daily_budget_usd: float):
    today_spend = await db.fetch_one("""
        SELECT COALESCE(SUM(cost_usd), 0) AS spent
        FROM llm_calls
        WHERE tenant_id = $1
          AND created_at >= CURRENT_DATE
    """, tenant_id)

    if today_spend["spent"] > daily_budget_usd * 0.8:
        await alert(f"Tenant {tenant_id} at {today_spend['spent']/daily_budget_usd:.0%} of daily budget")
    if today_spend["spent"] > daily_budget_usd:
        raise BudgetExceeded(tenant_id)

Far better to refuse a request than to wake up to a $40,000 surprise.

Quality Monitoring: The Hard Problem

Traditional observability tells us whether the system responded. LLM observability has to tell us whether the system responded well, and that is qualitatively harder.

A model returning a valid HTTP 200 with a coherent-sounding paragraph is not the same as the model returning a correct paragraph. There is no exception to count, no error rate to alert on. The system can be quietly degrading for weeks before anyone notices, because every individual response looks plausible.

There are four practical approaches, and mature systems use them in combination.

Structural validation

The cheapest signal: validate that the response has the structure it should. If we asked for JSON, did it return parseable JSON? If we asked for a function call, did it use a valid tool with valid arguments? If we asked for a numeric answer, did it return a number?

 def validate_extraction(response: str, schema: dict) -> tuple[bool, str]:
    try:
        data = json.loads(response)
    except json.JSONDecodeError as e:
        return False, f"invalid_json: {e}"

    try:
        jsonschema.validate(data, schema)
    except jsonschema.ValidationError as e:
        return False, f"schema_violation: {e.message}"

    return True, "ok"

We should track the structural failure rate per feature. A spike in failures usually points to a prompt regression, a model behaviour change, or a malformed input class we didn’t handle. Cheap, high-signal, runs on every request.

Reference-based evaluation

When we have ground truth (known correct answers), we must score the model’s output against it. This works well for classification, extraction, and tasks with deterministic answers. It works poorly for open-ended generation.

For a subset of production traffic, we can spot-check correctness against gold labels we have curated. The pattern: stratify-sample requests, hand-label the correct outputs once, and run an automated comparison continuously. Alert when accuracy on this canary set degrades.

LLM-as-judge

For tasks without ground truth, we can use a stronger model to evaluate the output of our production model. We provide both the prompt and the response to the judge model and ask it to score along defined dimensions: relevance, correctness, tone, faithfulness to provided sources.

async def judge(
    prompt: str, 
    response: str, 
    criteria: list[str]
) -> dict:
    judge_prompt = f"""
    Evaluate the following AI response against these criteria: 
    {criteria}. For each criterion, give a score 1-5 and a 
    brief justification.
    
    Return JSON.

    Original prompt:
    {prompt}

    Response:
    {response}
    """
    return await stronger_model.complete(
        judge_prompt, 
        response_format="json"
   )

Caveats are also real: the judge has its own biases, and the judge and production model can fail in correlated ways, and the cost of judging every production response is prohibitive. The pragmatic version is to judge a sample (say, 1-5%) and trend the scores over time. A statistically significant drop in average judge score means something has changed - even if we can’t yet say what.

User Signal

The cheapest, noisiest, most ground-truth-adjacent signal: did the user accept the response? Thumbs up/down, copy-to-clipboard, click-through, whether they edited it before using it, whether they immediately rephrased and asked again.

We should capture every such signal and tie it back to the originating LLM call via the trace ID. Aggregate over time; alert on trends. A 10% drop in thumbs-up rate this week is a real-world quality regression even if every other metric looks fine.

The four together give us a layered defence. Structural validation catches outright breakage. Reference evaluation catches drift on the things we can measure. LLM-as-judge catches degradation on the things we can’t. User signal catches everything we missed. None of them alone is sufficient. All four together is realistic for production systems.

Tools: Honest Comparisons

The tooling space has consolidated rapidly. The following categories matter:

LangSmith

This is tightly integrated with LangChain (and increasingly framework-agnostic). It has strong trace visualization, dataset and eval workflows, prompt management. It’s a good fit if we are already in the LangChain ecosystem; good standalone if we can absorb the SaaS dependency. It can either be managed or self-hosted for enterprise.

Langfuse

It is open-source, self-hostable, trace-and-eval focused. It is lighter than LangSmith but covers most of the same ground. Often the right choice if data residency or self-hosting matters.

Helicone

This is proxy-based; we point our LLM SDK at Helicone’s gateway and it captures everything transparently. It has lower implementation friction; the cost is that we have added an inference-path dependency. Strong cost analytics.

Arize Phoenix

This is open-source observability with strong roots in ML monitoring. More analytical/data-scientist tilted. It is good for embedding-space drift detection and other ML-adjacent concerns alongside LLM tracing.

Datadog LLM Observability

(and similar from Honeycomb, New Relic, etc.). The incumbents have added LLM-specific products to their existing observability platforms. Right answer if the org is already standardized on one of them and we don't want a new vendor surface for one workload.

Custom on top of OpenTelemetry

Here, we use the GenAI semantic conventions, send spans to whatever backend we already use. Works well if the team has observability discipline and the LLM workload is one part of a larger system. Bad fit if we need LLM-specific UI affordances (prompt diffs, eval result browsers, dataset management).

The decision matrix that works:

  • Small team, moving fast, no compliance constraints: start with hosted Langfuse or LangSmith. Time to instrumentation matters more than fit.

  • Strict data residency or PII constraints: self-hosted Langfuse or Phoenix. Don't ship prompts to a third party you can't audit.

  • Already on a major observability platform: use their LLM product if it exists. Avoid the second pane of glass.

  • Mature engineering org with strong OTel investment: custom on OTel + GenAI conventions. Maximum flexibility, maximum maintenance.

A rule that holds across all of these: whichever tool we pick, capture the canonical record ourselves first, in our own data store. Tools change. Vendors get acquired. The eval workflows want SQL access to the data. The tool is the UI; the source of truth should be ours.

Privacy: Logging Prompts is Logging User Data

LLM prompts contain user data by definition: the user’s questions, the documents they uploaded, the personal details they shared. Logging prompts the way we’d log API request bodies is, in many jurisdictions, a compliance violation. In all jurisdictions, it is a risk we should be deliberate about.

The wrong answer is “log nothing”. Without prompts, debugging is impossible and quality monitoring is half-blind. The right answer is a layered policy.

Classify what’s logged

Not every field of every prompt has the same sensitivity. The system prompt is your IP and is fine to log fully. The user message contains user data and needs handling. RAG-injected documents may contain other users’ data and need stricter handling.

Redact PII before storage

Run prompts through a PII detector (Presidio, AWS Comprehend, custom regex for known patterns) and replace detected entities with tokens. Email addresses become <EMAIL>, phone numbers become <PHONE>, names become <PERSON>. The redacted version is what’s stored long-term

def redact(text: str) -> str:
    text = re.sub(r'\b[\w.-]+@[\w.-]+\.\w+\b', '<EMAIL>', text)
    text = re.sub(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b', '<PHONE>', text)
    text = re.sub(r'\b(?:\d[ -]*?){13,16}\b', '<CARD>', text)
    return text

Redaction is necessarily imperfect. Treat it as risk reduction, not elimination.

Sample full payloads, retain redacted summaries

Keep the full unredacted prompt for a small percentage of requests (say, 1%) in a short-retention secure store, for debugging. Keep redacted versions of all requests for longer-term analysis. The full data is available when we need it; the bulk store has dramatically less risk.

Honor deletion requests

When a user invokes their right to be forgotten, we need a way to actually delete their LLM call records. This means user IDs in records (so we can find them) and deletion tooling that scans our storage layers. We must build it from the start - retrofitting it is painful.

Set retention windows by data class

Full prompt: days. Redacted prompts and metadata: months. Aggregated metrics: indefinitely. The longer we keep raw content, the larger the breach blast radius if we have one.

Be honest in the privacy policy

If we log prompts, we must say so. If we use logged prompts to improve the product, we must say that as well. The cost of accurate disclosure is small. The cost of being caught logging things we said we didn’t is large.

The Checklist

Before we ship an LLM-backed feature, check following items:

Logging:

  • One canonical call record per LLM invocation, captured at one chokepoint

  • Records include identity, params, tokens, cost, latency, and outcome

  • Cost computed per call, not estimated post-hoc

Tracing:

  • Multi-step flows produce nested spans matching conceptual steps

  • Spans use OpenTelemetry GenAI semantic conventions

  • Sampling preserves whole traces, not random spans

  • You can copy a prompt from a failed span and reproduce it locally

Cost attribution:

  • Every call is tagged with feature, user, and tenant

  • You can break down spend by all three dimensions in under 60 seconds

  • Per-tenant budget alerts fire before invoice surprises

Quality monitoring:

  • Structural validation runs on every response where applicable

  • A reference eval set runs on every model/prompt change and on a continuous canary

  • LLM-as-judge runs on a sample, with trended scores

  • User feedback signals are captured and tied back to the originating call

Privacy:

  • PII is redacted before long-term storage

  • Full payloads are sampled and retained briefly, in a separate store

  • Deletion-by-user is implemented, not just promised

  • Your privacy policy reflects what you actually log

Operational:

  • We can answer "how much did we spend yesterday and on what" in real time

  • We have at least one quality alert that has fired and was meaningful

  • Tool choice does not create a single point of failure for instrumentation

Takeaways

Observability for traditional services tells us whether the system worked. Observability for LLM applications has to tell us whether it worked, what it cost, and whether the answer was actually any good. None of these are observable for free; all of them are observable cheaply once we have decided to be deliberate about it.

The teams that operate LLM systems well do four things consistently. They capture a canonical record per call at a single chokepoint, with all the dimensions they will later need to slice on. They trace multi-step flows so the gantt chart of an agent run is one click away from any incident. They attribute cost to features and users so spend is a managed budget, not an end-of-month surprise. And they triangulate quality from structural validation, reference evals, judge models, and user signals because no single source is sufficient.

The stack for building all of this with is more mature than it was two years ago. OpenTelemetry's GenAI conventions are real. Tools like Langfuse, LangSmith, Helicone, and the major incumbents cover most of the surface area. The remaining problems are about discipline. Decide what to log. Stand up the pipeline. Make the dashboards. Look at them.

The black box opens when we decide to open it. Most teams don't decide. The ones that do operate a different kind of system; one where you can answer questions about it, optimize it deliberately, and trust it in production. That's the difference between an LLM feature and an LLM product.

What's the most surprising thing you found when you finally instrumented your LLM app? I am collecting stories: runaway features, pathological users, prompt regressions that snuck through, costs that turned out to be 5x what people guessed. The patterns repeat.

Hit reply. I read everything.

Namaste!

If this clicked, forward it to your team. Most LLM apps are flying blind, and most teams don't know how blind until they look. And if you want more deep dives like this, subscribe to The Main Thread - practical engineering for AI and distributed systems, one essay per week.

Reply

Avatar

or to participate

Keep Reading

© 2026 The Main Thread.
beehiivPowered by beehiiv