How to Get Structured JSON Output from Any LLM

Why Structured Output Matters

In a research or chat context, free-form text is fine. In a production application, it breaks things. When your application calls a model to extract data from a document, classify a support ticket, generate metadata, or produce a recommendation, it needs the output in a consistent structure that code can parse. If the model sometimes returns JSON and sometimes returns prose, if it sometimes includes a field and sometimes omits it, if it sometimes wraps strings in quotes and sometimes does not, your downstream code breaks on every inconsistency.

The cost of inconsistent output is higher than it appears. Every format inconsistency requires either a fallback parser (which adds complexity), a retry (which adds latency and cost), or a human to fix the output (which defeats the purpose of automation). For an application handling 10,000 requests per day, even a 5% format failure rate means 500 failures per day that need handling. Bringing that failure rate from 5% to 0.1% is the practical goal of structured output engineering.

Structured output also improves the quality of the content within the structure. When a model knows it needs to fill specific fields (a severity rating from 1 to 5, a category from a defined list, a boolean for whether escalation is needed), it makes more deliberate decisions for each field than it would in free-form text. The structure forces the model to address each required data point explicitly, which reduces omissions and vague answers.

Provider-Level Structured Output

The most reliable approach to structured output in 2026 is using the model provider's native structured output features. These constrain the model's token generation at the decoding level, meaning the model literally cannot produce tokens that would violate the schema. This is fundamentally more reliable than prompt-based techniques that rely on the model choosing to follow your format instructions.

OpenAI offers the response_format parameter with type "json_schema" that accepts a full JSON schema. The model's output is guaranteed to be valid JSON conforming to your schema. Fields, types, enum values, and required properties are all enforced. This works with GPT-4o and later models. For simpler cases, type "json_object" guarantees valid JSON without a specific schema. The schema approach is strongly preferred for production because it eliminates the entire class of "valid JSON but wrong structure" failures.

Anthropic achieves structured output through tool use. You define a tool with a JSON schema describing its parameters, and the model "calls" that tool with the structured data as its arguments. The output conforms to the tool's schema. This is slightly less direct than OpenAI's response_format approach, but it works reliably and has the advantage of fitting naturally into workflows that already use tool calling. Anthropic also supports direct JSON output through prompt instructions, which works well with Claude models' strong instruction-following capabilities.

Google Gemini offers response_schema in its generation configuration, which accepts a JSON schema and constrains the output to match. The behavior is similar to OpenAI's approach. Gemini also supports enum constraints, which are useful for classification tasks where the output must be one of a predefined set of values.

If your application uses multiple providers, design your code to use each provider's native structured output feature rather than relying on prompt-based techniques for cross-provider consistency. The provider-specific code paths are worth the maintenance cost because of the dramatically higher reliability.

Prompt-Based Structured Output

When provider-level structured output is not available (older model versions, open-source models running on your own infrastructure, providers without native schema support), prompt-based techniques are the fallback. These are less reliable than provider-level features but can achieve 95%+ format compliance with careful design.

The most effective prompt-based technique is to include both a schema description and a complete example. "Respond in JSON with the following schema: {name: string, category: one of [bug, feature, question], priority: integer 1-5, summary: string max 100 characters}. Example output: {\"name\": \"Login button unresponsive\", \"category\": \"bug\", \"priority\": 4, \"summary\": \"Login button does not respond to clicks on mobile Safari\"}." The schema tells the model what structure to use. The example shows it exactly what a correct response looks like. Together, they are much more effective than either alone.

For complex schemas with nested objects, arrays, or optional fields, the example is even more important because natural language descriptions of nested JSON are ambiguous. Show a complete example with all nesting levels populated, including arrays with multiple elements and optional fields both present and absent (in different examples). This removes all ambiguity about the expected structure.

Wrapping the output instruction is a useful technique. "Your entire response must be valid JSON. Do not include any text before or after the JSON. Do not include markdown code fences. Start with { and end with }." This prevents common failure modes where the model adds explanatory text before or after the JSON, or wraps the JSON in markdown backticks that break JSON parsers.

Validation and Retry Patterns

Even with the best prompting techniques, structured output can fail. A robust production system includes validation and retry logic.

The validation layer parses the model's response and checks it against your expected schema. In Python, libraries like Pydantic handle this elegantly: define a model class with typed fields, parse the JSON into the model, and Pydantic raises a validation error with specific field-level details if the output does not conform. In TypeScript, Zod provides the same capability. The validation error messages are specific enough to include in a retry prompt.

The retry pattern takes the validation error and sends it back to the model. "Your previous response had the following validation errors: [list of errors]. Please produce a corrected response that fixes these issues." This works because the model can see its previous attempt and the specific errors, which gives it enough information to self-correct. One retry fixes the issue in the vast majority of cases. Setting a maximum of two retries prevents infinite loops for fundamentally broken cases.

Libraries like Instructor (Python) and the Vercel AI SDK (TypeScript) wrap this entire pattern, schema definition, model call, validation, and retry, into a single function call. For production applications, these libraries save significant engineering effort and handle edge cases that manual implementations often miss.

Structured Output for Data Extraction

One of the most common applications of structured output is extracting structured data from unstructured text: pulling entities from documents, extracting fields from emails, converting natural language descriptions into database records. This combines the model's language understanding with the reliability requirement of structured output.

For extraction tasks, pair structured output with chain-of-thought by having the model reason about the content before filling in the schema. Use a two-part output: a "reasoning" field where the model analyzes the text, and a "result" field with the extracted structured data. The reasoning step improves extraction accuracy because the model explicitly considers what information is present, what is ambiguous, and what is missing before committing to values in the schema.

Handle missing data explicitly. Define whether optional fields should be null, omitted, or filled with a default value. Provide instructions for ambiguous cases: "If the document does not explicitly state the date, set date to null rather than guessing." Without these instructions, models tend to hallucinate plausible values for missing fields rather than admitting the information is not available, which corrupts your data.

Cost Considerations

Structured output adds tokens to the response (JSON syntax characters, field names, quotation marks) compared to a free-form text answer conveying the same information. The overhead is typically 20 to 40% more output tokens for a simple schema, and higher for schemas with long field names or deeply nested structures. This cost increase is almost always justified by the elimination of downstream processing, human review, and retry costs that inconsistent output would require.

Schema design affects cost. Short field names ("cat" vs "category") save tokens but hurt readability. Flat schemas save tokens compared to deeply nested ones. For high-volume applications, optimizing your schema structure for token efficiency is worth the effort. But never sacrifice schema clarity for a few tokens, because the cost of misinterpreted field semantics far exceeds the token savings.