Structured Output Enforcement: Ensuring LLMs Return Valid Data

The Problem with Unstructured LLM Output

LLMs generate text token by token, optimizing for the most probable next token given the context. They have no inherent concept of JSON syntax, schema validity, or data types. When you ask a model to return JSON, it generates characters that look like JSON based on patterns in its training data, but it has no mechanism to guarantee that the result is valid JSON, let alone JSON that matches your specific schema.

The failure modes are numerous and common. The model generates JSON with a trailing comma after the last element, which is valid JavaScript but invalid JSON. The model generates a string value where an integer is expected. The model omits a required field. The model generates nested objects where a flat structure is expected. The model wraps the JSON in markdown code blocks or adds explanatory text before and after the JSON. The model generates JSON that is syntactically valid but contains values that are semantically wrong: a negative price, an email address in a phone number field, or a date in the future where a past date is required.

Without enforcement, these failures crash downstream systems, corrupt data, and require manual intervention. A JSON parsing error that happens 0.5% of the time seems negligible until you are processing 100,000 requests per day and dealing with 500 failures. At production scale, even rare format errors are operational burdens that structured output enforcement eliminates.

Native Structured Output Modes

The major LLM providers now offer built-in structured output support that constrains the model's generation at the token level. OpenAI's Structured Outputs feature accepts a JSON schema and guarantees that the model's output conforms to that schema by modifying the token sampling process to only allow tokens that maintain schema validity. Anthropic's tool use system returns structured arguments that conform to the tool's defined schema. Google's Gemini supports response schema constraints that enforce structure during generation.

Native structured output modes are the strongest guarantee available because they operate at the generation level rather than the validation level. The model physically cannot generate invalid output because invalid tokens are excluded from the sampling process. This eliminates entire categories of failures: no more missing closing braces, no more wrong data types, no more missing required fields. The constraint is enforced with the same reliability as any other software constraint, not probabilistically but deterministically.

The trade-offs of native modes are limited. Schema support varies by provider: some support full JSON Schema including conditional logic and pattern matching, while others support only a subset. Response quality can decrease for complex schemas because the generation constraints limit the model's flexibility, occasionally producing technically valid but less natural or less accurate content. Latency may increase slightly because the constrained generation process is computationally more expensive than unconstrained generation. For most production use cases, these trade-offs are well worth the guarantee of valid output.

Not all models support native structured output. Open-source models, older API versions, and some specialized models lack this capability. When native modes are not available, you need post-generation validation, which is the traditional guardrail approach.

Post-Generation Validation with Pydantic

Pydantic has become the standard tool for validating LLM output in Python applications. You define a Pydantic model that describes the expected schema, parse the model's output as JSON, and validate it against the Pydantic model. If validation fails, Pydantic provides detailed error messages that identify exactly which field failed and why, which you can use for targeted retry logic.

The advantage of Pydantic validation over raw JSON schema validation is that Pydantic supports custom validators, type coercion, and semantic validation in addition to structural validation. A Pydantic validator can check not only that a field is a string, but that it matches a regex pattern, falls within a length range, or exists in a set of allowed values. It can check cross-field relationships: if the "type" field is "refund," then the "amount" field must be negative. These semantic validations catch errors that structural validation alone misses.

Define strict and lenient validation modes. Strict mode rejects any response that does not exactly match the schema. Lenient mode attempts to fix common issues before validating: stripping markdown code fences, removing trailing commas, coercing numeric strings to integers, and truncating over-long string fields. Lenient mode reduces the retry rate by automatically fixing trivial format issues, but it must be implemented carefully to avoid silently corrupting data. A string "42" coerced to an integer 42 is safe. A string "forty-two" coerced to an integer is not, and should be rejected rather than guessed.

Libraries like Instructor (built on top of Pydantic and the major LLM APIs) automate the entire validation-retry loop. You define a Pydantic model, pass it to the LLM call, and Instructor handles parsing, validation, and retrying with error feedback automatically. The library adds the schema to the system prompt, validates the response, and if validation fails, sends the validation errors back to the model with a retry prompt that tells the model exactly what was wrong and asks it to fix it. This approach typically achieves 99%+ success rates within 1-2 retries.

JSON Schema Enforcement

JSON Schema provides a language-agnostic way to define the expected structure of JSON output. While Pydantic is Python-specific, JSON Schema works across any language and can be used with any LLM API. The schema defines required fields, data types, string patterns, numeric ranges, array lengths, enum values, and conditional requirements.

Embed the JSON schema in the system prompt so the model understands the expected structure during generation, and then validate the output against the same schema after generation. This two-layer approach, schema as generation guidance plus schema as validation enforcement, produces better results than either layer alone. The model generates output that approximates the schema because it is in the prompt, and the validator catches any deviations.

For complex schemas, provide one or two examples of valid output alongside the schema definition. Models follow examples more reliably than abstract schema definitions, especially for nuanced formatting requirements like date formats, phone number patterns, and nested object structures. An example of a valid response is worth more than a paragraph of schema documentation in terms of guiding the model's output.

Handle partial compliance gracefully. A response that is valid JSON, contains all required fields, but has one field with an unexpected value type is much closer to correct than a response that is not valid JSON at all. Your validation pipeline should distinguish between structural failures (invalid JSON, missing required fields) that require a full retry and value-level failures (wrong data type, out-of-range value) that can potentially be fixed by coercion or targeted reprompting.

Retry Strategies for Failed Validation

When structured output validation fails, the retry strategy determines how efficiently the system recovers. The simplest approach is blind retry: send the same prompt again and hope the model generates valid output on the second attempt. Blind retries work surprisingly well for intermittent format errors because the model's generation is stochastic, and a different random seed often produces a valid response. However, blind retries are inefficient for systematic errors where the model consistently makes the same mistake.

Error-feedback retry is more effective. When validation fails, include the specific validation errors in the retry prompt: "Your previous response had these errors: field 'age' must be an integer but you returned a string. Please correct these issues and return the updated JSON." This approach works because the model can usually fix specific, well-described errors. Error-feedback retries have a higher success rate than blind retries, typically resolving the issue in a single retry for most error types.

Set a maximum retry count. Two retries (three total attempts) is the practical limit for most applications. If the model cannot produce valid output in three attempts, the prompt or schema has a fundamental issue that retries will not solve. After the maximum retries, return a structured error response to the caller rather than continuing to retry in a loop that wastes tokens and adds unbounded latency.

Track retry rates by schema and by error type. A schema that requires retries on 20% of requests suggests that the schema is too complex for the model to follow reliably, or that the prompt needs better examples. An error type that appears across many schemas suggests a systematic model weakness (like difficulty with specific date formats or numeric precision) that can be addressed by preprocessing or postprocessing rather than retries.

Beyond Structural Validation: Semantic Checks

Valid JSON that matches your schema is necessary but not sufficient. The values in the JSON also need to make semantic sense. A product recommendation with a price of $0.00, a user profile with an age of 250, or a shipping estimate of -3 days are all structurally valid but semantically wrong.

Add semantic validators after structural validation. These check business rules that the schema cannot express: prices must be positive, dates must be in the past for historical records, quantities must be integers, email addresses must be syntactically valid, URLs must resolve, and referenced IDs must exist in your database. Semantic validators turn the output from "valid data" into "correct data."

Semantic validation overlaps with the business logic validators described in custom output validation. The key distinction is scope: structural validators check the container (is this valid JSON with the right fields?), while semantic validators check the contents (are the values in those fields actually correct?). Both layers are necessary. A system that validates structure but not semantics ships syntactically perfect nonsense. A system that validates semantics but not structure crashes on the parsing step before it can check the values.