Hey everyone, welcome to the forty-ninth issue of The Main Thread.
To this day, many LLM features have just one system instruction: “You are a helpful assistant”. This instruction leaves three production decisions unspecified: what the model may say, what data it may use, and what the application can safely parse.
Assume that a support assistant has only that instruction. A customer asks about a refund. The help centre doesn’t cover their case, so the model fills the gap by inventing a policy. In another request, the application expects a JSON object. But the model sends a paragraph before the JSON, and the parser fails.
What do people usually do? They keep adding instructions to the system prompt as they encounter incidents; this causes the prompt to bloat. But this is the less evil problem. The more evil problems are that this is a “reactive” approach (you fix it once it fails), and adding one instruction can weaken the other.
This is avoidable. Prompt engineering has a credibility problem because the name suggests that it has one specific phrase. LinkedIn is full of those people who sell these phrases. In real production, it is interface design (a prompt is a contract with the model), economics (section order changes what we pay), and engineering discipline (version it, test it, measure it). A useful pattern is the one with the backing of published numbers and API features.
This field changes rapidly; vendor APIs and model defaults will keep changing. The principles in this issue are designed in such a way that they can survive the test of time: use structure, measure behaviour, and keep every model-specific choice behind an eval.
We will cover system-prompt order, few-shot example selection, reasoning effort, constrained JSON, sampling controls, and prompt versioning.
A System Prompt is a Contract
A system prompt defines three jobs:
It says who the model is (role)
What it must and mustn’t do (constraints)
What the output should look like (format)
Now, a prompt that mixes these three might work in a demo or a simple app, but it will definitely decay in production. Why? Because every edit to the prompt endangers all three at once. We should separate them with labelled instructions.
<role>
We are the support assistant for Acme, a payroll product.
Audience: customers, non-technical.
</role>
<constraints>
- Answer only from the provided help articles.
- If the articles don't cover it, say so and offer a support ticket.
- Never state prices, dates, or policy details from memory.
</constraints>
<format>
Reply in plain text, 3 sentences maximum, no markdown.
</format>
<articles>
{retrieved_help_articles}
</articles>Structured sections make the contract legible and reduce additional mixing of instructions with data. They don’t make untrusted content safe: prompt-injection defences still need separate controls around data access and tool use.
Section order can also affect the cost. A provider who supports prefix caching reuses a processed prompt prefix when later requests match it exactly. The provider’s cache controls, lifetime, thresholds, and prices will change, but the underlying mechanism won’t. A timestamp, username, or request ID near the top makes the remaining prefix different and can cause a cache miss.
So, the rule of thumb is: stable content must go first, and volatile content must go last. Role, constraints, format, and fixed examples belong in a byte-identical block at the beginning. The user’s question, retrieved documents, and today’s date belong at the end.
The bigger problem with failures (more than their presence) is that they are usually silent, and that makes them expensive. The following are a few failures that destroy the benefits of having a cache:
A timestamp or “current date” interpolated at the top.
A request ID or UUID anywhere in the fixed section.
A dictionary serialized without sorted keys, so the same data renders in a different order per request.
A tool or example list that varies per user.
None of the above throws an error, but the cost remains high. This is the reason cache-read usage should be depicted in a dashboard.
The math behind this is not difficult to understand. If the fixed prefix contains S tokens, receives R requests, and costs P per input token, the uncached daily cost is S x R x P. Then check 3 numbers for the model in use: the cost to write the cache, the cost to read from it, and how long the cache stays alive. Those numbers show whether caching saves money for this endpoint.
Long prompts have another problem: models can miss information present in the middle. The 2023 paper Lost in the Middle measured this. Keep instructions at the start and the user's question near the end, then test that order on the deployed model.
Few-Shot Examples
The original GPT-3 paper in 2020 suggested the idea of few-shot prompting, where we give worked input-output examples before the actual input. This remains worth testing on formatting-heavy and judgement-heavy tasks.
The following two research findings explain why it works and overturn common intuition:
Finding 1
Examples show the model the exact input and output format. In a 2022 study, changing the labels in examples had little effect on 12 tested models. For those tasks, the format did much of the work. The result may not hold for another task, so test it.
Finding 2
Changing only the order of examples can change the answer quality. The same order may not work for every model. Run the eval with several orders, then keep the best one.
A production few-shot block for a support ticket classifier might look like this:
Classify each ticket as billing, bug, or how_to.
Ticket: "I was charged twice for March."
Category: billing
Ticket: "The export button does nothing when I click it."
Category: bug
Ticket: "Can I add a second admin to my account?"
Category: how_to
Ticket: "My invoice is wrong because the app double-counted my seats."
Category: billing
Ticket: "{user_ticket}"
Category:The above examples cover all three categories and use the same format. The fourth ticket could be a billing problem or a bug. It is labelled billing to show the model how to handle the case. Thus, the examples define these edge cases.
These findings suggest practical advice, not universal laws. They came from particular tasks and model families, so the eval set decides whether they apply to a given endpoint:
Use the same format in every example. Include at least one example for every category. For a yes/no task, show both a yes and a no. Add difficult cases, such as an unclear ticket or empty input, so the model sees what to do with them.
Test several example orders on the eval set. Keep the order with the best result.
Do not spend days finding the perfect example. Covering the categories and edge cases matters more.
If incoming requests are similar, you should use the same examples every time, and choose that set with the eval results.
If requests vary a lot, choose examples that look like the incoming request. Store labelled examples in a library, find the closest ones for each request, and add them to the prompt. This can improve answers, but it also adds search time, token cost, and cache misses. Test both approaches on real traffic.
Changing examples on every request also changes the prompt. Prefix caching only works when the start of the prompt stays the same. Put the instructions that never change first. Put the selected examples and the user's request after them. Then check the cache-hit numbers.
Reasoning
Models can improve their answers by writing down the steps before the answer.
In 2022, Google researchers (Wei and coauthors) showed math examples in this form: question → working → answer. This drastically improved the correctness of the model.
Later that year, another study added “let’s think step by step” to a prompt, with no examples. Accuracy on the same test rose from 10.4% to 40.7% for the tested model.
Written steps give the model more to work with. Each step becomes the input for the next step, like writing down a long-division problem instead of doing it all in your head.
We can use this for tasks that need several steps: math, logic, planning, or combining facts from several documents. Do not add it by default to a ticket classifier or a lookup. Extra reasoning produces more tokens, which makes the response slower and more expensive. So, test whether it improves the endpoint before keeping it.
For offline jobs, one option is to ask the same question many times and choose the answer that appears most often. This is called self-consistency. It can improve accuracy, but every extra answer costs another model call. It usually suits batch jobs better than chat.
Some models do their own internal reasoning or offer a setting for how much reasoning to use. Test 3 versions on the deployed model: a direct answer, a prompt that asks for steps, and the provider's reasoning setting. Keep the version that provides enough quality within the latency and cost budget.
Structured Output
An application may expect JSON such as {"category": "billing"}. A normal model may add a sentence before it, wrap it in a code block, or return invalid JSON. If this happens, then the parser fails.
There are 3 ways to ask for JSON:
1. Prompting
"Return valid JSON only." The model can still ignore this.
2. JSON Mode
The API returns valid JSON. It can still use the wrong field names or leave out a required field.
3. Structured Output
Give the API a schema that lists the fields and values allowed. When the provider supports it, the model can only return JSON that follows that schema. Use this for any response that code must read.
In Python, a Pydantic class can define the output:
from typing import Literal
from pydantic import BaseModel
import anthropic
class Ticket(BaseModel):
category: Literal["billing", "bug", "how_to", "other"]
summary: str
needs_human: bool
client = anthropic.Anthropic()
response = client.messages.parse(
model="<pinned model version>",
max_tokens=1024,
messages=[{"role": "user", "content": ticket_text}],
output_format=Ticket,
)
ticket = response.parsed_outputSee how structured output solves the problem: the response has the fields the program expects. The model cannot return a paragraph when the program expects a Ticket.
However, it does not solve 2 other problems:
The API request can still fail or stop halfway through. We must handle those errors as usual.
The model can still choose the wrong value.
{"category": "billing"}has the right format, even when the ticket is not about billing. So, we must check important decisions after parsing.
We should also give the model a other category. A phishing-email ticket is not billing, a bug, or a how-to question. Without the other, the model must choose a wrong category. Use null for a value it cannot know.
Use the provider's documented structured-output feature and test it. Do not rely on tricks such as starting the reply with {.
Temperature and top_p
The model chooses its answer one word at a time. At each word, it has several possible choices. For example: after “your”, it may choose “invoice”, “bill”, or “statement”.
The parameter temperature controls how strongly the model prefers the most likely choice.
Low temperature: it usually picks the most likely word. The answers are more similar each time.
High temperature: it gives less likely words a better chance. The answers vary more.
Temperature 0: it picks the top choice each time. The result can still differ between API requests, so do not rely on identical output.
The parameter top_p removes unlikely choices before the model picks a word. At top_p=0.9, it keeps the most likely choices until their total chance reaches 90%, then removes the rest.
For classification, extraction, or another task with a correct answer, test low randomness. For brainstorming or copywriting, test higher randomness. Change one setting at a time and compare the results on the same eval set.
Version and Test Prompts Like the Code They Are
A prompt changes what customers see. Keep it in source control, along with the model name and settings that run it.
There are 4 steps:
Save the prompt, model, and settings together. The same prompt can behave differently on another model or with different settings.
A change now has a history, a review, and a way to undo it. Treat a model upgrade the same way as a prompt change.
Test every change before release. Keep a set of real inputs and their expected outputs. Run both the old and new prompt on that set. Release the new one only when it is as good or better.
This is the prompt's test suite. Run it before release, like any other test in CI.
Record the prompt version with every request. When a bad answer appears, the engineering team can see which prompt produced it.
Test the new prompt on a small part of real traffic. Compare task completion, bad-answer rate, latency, and cost with the old prompt. Do not choose a winner after a few requests. Wait until there is enough traffic to trust the result.
A dedicated prompt tool can help, but it is not mandatory. Source control, a test command in CI, a feature flag, and request logs are enough.
The Takeaway
Start with the production prompt that receives the most traffic. Put its template, model, SDK version, and parameters in version control. Add 10 golden test cases.
If the provider supports prefix caching, move the fixed prefix above dynamic request data and inspect cache-read usage for a week. This gives the prompt a reproducible baseline before the next model or product change arrives.
Namaste!
If this clicked, forward it to your team. Prompt engineering patterns are one of those topics that don’t get enough debate and techniques until you run them in production. And if you want more deep dives like this, subscribe to The Main Thread — practical AI engineering, one essay per week.
