Hello everyone, welcome to the twenty-seventh issue of The Main Thread. After spending last two article talking about distributed systems, we will shift our focus back to AI engineering in this article - after all, learning how to apply distributed systems principles in AI engineering is what makes an engineer different from others.
In this issue, we will learn about problems and techniques on getting structured data from LLMs. It is harder than it looks.
We ask for JSON, we get JSON with a conversational preamble. We ask for specific fields but we get fields with creative names. We ask for an array, but we get an object. We ask for a number but we get a string containing a number, sometimes with a dollar sign.
One of the common issues I have seen systems crash because a model decided to return ”price”: “varies“ instead of ”price”: 29.99”. Having a markdown code fence around the JSON is one of the major problems in broken pipelines. No one validated the output schema, and the application failed silently.
Let me put your mind at ease. Structured output from LLMs is a solved problem (if you know the patterns). This article talks about those patterns, approaches that work, validation strategies that catch failures, and fallbacks that keep our system running when the model gets creative.
The Three Approaches
There are three ways to get structured output from modern LLMs. Each has different reliability characteristics.
1. JSON Mode
This is the simplest approach in which we tell the model to output JSON, and enable JSON mode to guarantee syntactically valid JSON.
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
response_format={"type": "json_object"}, # Guarantees valid JSON
messages=[
{
"role": "system",
"content": "Extract product info. Return JSON with fields: name, price, currency."
},
{
"role": "user",
"content": "The Nike Air Max 90 is currently $129.99"
}
]
)
data = json.loads(response.choices[0].message.content)
# {"name": "Nike Air Max 90", "price": 129.99, "currency": "USD"}This mode guarantees:
Output will be a valid JSON (parseable, no syntax errors)
No markdown code fences or conversational text
This model doesn’t guarantee:
Correct schema (might have wrong fields)
Correct types (might return
”129.99"instead of129.99).All required fields present
JSON mode is necessary but not sufficient. We will still get valid JSON that doesn’t match our schema.
2. Function Calling/Tool Use
This is a more structured approach where we define a function schema, and the model returns arguments that match the schema.
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "user", "content": "The Nike Air Max 90 is currently $129.99"}
],
tools=[
{
"type": "function",
"function": {
"name": "extract_product",
"description": "Extract product information from text",
"parameters": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Product name"
},
"price": {
"type": "number",
"description": "Price as a decimal number"
},
"currency": {
"type": "string",
"enum": ["USD", "EUR", "GBP"],
"description": "Currency code"
}
},
"required": ["name", "price", "currency"]
}
}
}
],
tool_choice={"type": "function", "function": {"name": "extract_product"}}
)
# Extract the structured arguments
tool_call = response.choices[0].message.tool_calls[0]
data = json.loads(tool_call.function.arguments)The function/tool calling adds:
Schema is explicitly defined with JSON schema
Model is “guided“ toward the schema
Required fields are most likely to be present
Enum constraints are usually respected
The function/tool calling doesn’t guarantee:
Types are still sometimes wrong
Complex nested schema can confuse the model
The model might still hallucinate invalid enum values
3. Structured Outputs (Schema Enforcement)
This is the newest and most reliable approach where the model’s output is constrained to exactly match a JSON schema.
from openai import OpenAI
from pydantic import BaseModel
class Product(BaseModel):
name: str
price: float
currency: str
client = OpenAI()
response = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[
{"role": "user", "content": "The Nike Air Max 90 is currently $129.99"}
],
response_format=Product # Pydantic model defines exact schema
)
product = response.choices[0].message.parsed
# Product(name='Nike Air Max 90', price=129.99, currency='USD')The structured outputs guarantee:
Output matches the schema exactly
Types are correct
Required fields are present
Enum values are valid (if defined)
The structured outputs doesn’t guarantee:
Some schema features aren’t supported (recursive schema, some formats)
Slightly higher latency due to constrained decoding
Recommendation: Use structured outputs when available. Fallback to function calling with validation when not.
Schema Design for Reliability
The schema we provide dramatically affects the output quality. Here’s what works.
Keep It Flat When Possible
Deep nesting confuses models and makes validation harder
