LLM Structured Outputs: JSON Schema, Pydantic, Zod, Instructor

JSON mode makes the model emit syntactically valid JSON. OpenAI's guide is explicit that it does not ensure schema adherence: fields can be missing, extra, or of the wrong type. It is a fallback for models or providers without native schema support.

Native structured outputs take a JSON Schema and constrain generation to it. OpenAI says Structured Outputs ensures the model always generates responses that adhere to the supplied schema; Anthropic says its structured outputs guarantee schema-compliant responses through constrained decoding, in most cases. Both document exceptions (next sections), and both support only a subset of JSON Schema.

Validate-and-retry runs your own validation (a Pydantic model or Zod schema, including rules no JSON Schema subset can express) and, on failure, sends the error back to the model and asks again. Instructor packages this pattern across providers.

They combine. Constrained decoding gets the shape right in one call; your validator checks the meaning: ranges, formats beyond what the provider supports, and rules across fields, such as an end date after a start date.

A language model produces text one token at a time by sampling from a probability distribution over its vocabulary. Constrained decoding compiles your schema into a grammar and, at each step, masks out every token that would make the output impossible to complete validly. If the schema says the next thing must be the key "email" or a closing brace, no other token can be sampled. The model still chooses the values; the grammar only removes invalid paths.

Two practical consequences follow from that mechanism:

• Compilation has a cost. Both providers note extra latency on the first request with a new schema while it is processed. Anthropic caches compiled grammars for 24 hours from last use; changing the schema's structure or the set of tools recompiles, while changing only name or description fields does not.
• Not every schema can be compiled. Grammars cannot cheaply express some JSON Schema features, so providers support a subset and reject the rest with a 400 error.

Anthropic also notes that changing the output format invalidates the prompt cache for that conversation, so keep the schema stable within a thread.

OpenAI (Responses API): pass the schema under text.format with type: "json_schema" and strict: true. The official SDKs accept Pydantic models and Zod objects and parse the result:

1from openai import OpenAI2from pydantic import BaseModel3 4class Invoice(BaseModel):5    vendor: str6    total_cents: int7    currency: str8    due_date: str | None9 10client = OpenAI()11resp = client.responses.parse(12    model="<model>",13    input=[{"role": "user", "content": invoice_text}],14    text_format=Invoice,15)16invoice = resp.output_parsed

Claude (Messages API): pass the schema under output_config.format with type: "json_schema" (no beta header is needed; the older output_format parameter is deprecated). Set strict: true on tool definitions to constrain tool inputs as well. The Python SDK's messages.parse() takes a Pydantic model and returns parsed_output; in TypeScript, zodOutputFormat() with messages.parse() does the same for Zod:

1import Anthropic from "@anthropic-ai/sdk";2import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";3import { z } from "zod";4 5const Invoice = z.object({6  vendor: z.string(),7  total_cents: z.number().int(),8  currency: z.string(),9  due_date: z.string().nullable(),10});11 12const client = new Anthropic();13const msg = await client.messages.parse({14  model: "<model>",15  max_tokens: 1024,16  messages: [{ role: "user", content: invoiceText }],17  output_config: { format: zodOutputFormat(Invoice) },18});19const invoice = msg.parsed_output;

Unsupported features fail loudly with a 400 error, which is the better failure. The quieter ones are the documented exceptions where a successful response does not match your schema:

• Refusals. If the model refuses for safety reasons, the refusal takes precedence over the schema. OpenAI adds a refusal field; Claude returns stop_reason: "refusal", still with a 200 and billed tokens.
• Truncation. If the response hits the output token limit, it is incomplete and will not parse. OpenAI marks the response status: "incomplete" with reason max_output_tokens; Claude returns stop_reason: "max_tokens".
• Details. Claude does not guarantee the capitalisation of string enum and const values (compare case-insensitively), and puts required properties before optional ones.

So the parsing code always checks the stop reason or status first, and validates numeric ranges, string lengths, and cross-field rules in code, because the provider's subset cannot express them.

Instructor wraps provider clients so that a call returns an instance of your Pydantic model, and automatically re-asks the model when validation fails. Its current recommended entry point is a provider-agnostic client:

1import instructor2from pydantic import BaseModel, field_validator3 4class Invoice(BaseModel):5    vendor: str6    total_cents: int7    currency: str8 9    @field_validator("total_cents")10    @classmethod11    def positive(cls, v: int) -> int:12        if v <= 0:13            raise ValueError("total must be positive")14        return v15 16client = instructor.from_provider("openai/gpt-4o-mini")17invoice = client.create(18    response_model=Invoice,19    messages=[{"role": "user", "content": invoice_text}],20    max_retries=3,21)

When validation fails, here because a validator rejects a non-positive total, Instructor sends the validation error back to the model and tries again, up to max_retries. That covers rules no provider schema subset supports. The trade-off is cost and latency: each retry is another full model call, and a model that cannot satisfy a rule will burn every retry before failing. Instructor also has a TypeScript version built on Zod.

Use ten real or realistic invoice texts, including two that lack a due date and one that is not an invoice at all.

1. Ask for JSON in the prompt only, and count parse failures and missing fields.
2. Switch to native structured outputs with the Invoice model (OpenAI responses.parse or Claude messages.parse). Count failures again.
3. Add a validator for a rule the schema cannot express (total positive, currency in a list), via Pydantic or Instructor with max_retries=2.
4. Set max_tokens very low on one request and confirm your code detects truncation instead of crashing.
5. For the non-invoice text, see what the model puts in required fields.

Pass criterion: zero unhandled parse errors, truncation detected by stop reason, the business rule enforced in code, and a decision about how the schema should let the model say "this is not an invoice".