How to Stream Structured Output from LLMs
Streaming and structured output might seem like conflicting features. Structured output guarantees a complete, valid JSON response, while streaming delivers partial, incomplete data. In practice, they work together because constrained decoding operates at the token level: each token is individually validated against the schema during generation, regardless of whether the response is streamed or returned as a whole. The stream is a sequence of tokens that will, when complete, form a valid schema-compliant JSON response, and at every point during the stream, the tokens generated so far are a valid prefix of such a response.
Enable Streaming with Your Schema
Enabling streaming with structured output is straightforward: add stream=True (or the equivalent parameter for your provider) to the API call while keeping the structured output schema in place. The provider handles the rest, streaming tokens that are constrained to the schema exactly as they would be in a non-streaming call.
With OpenAI, you set both response_format (with your json_schema) and stream=True on the chat completion call. The response is a stream of server-sent events (SSE), each containing a chunk of the JSON being generated. The final event contains the complete response, which matches the schema exactly as it would in a non-streaming call.
With Anthropic, you set output_config with your schema and use the streaming API (either the stream parameter or the Messages stream helper in the SDK). Claude streams the structured response token by token, with each content block delta containing the next piece of the JSON.
With Gemini, you set response_schema in the generation config and call the streaming generation method. The behavior is analogous: tokens arrive progressively while the schema constraint remains active throughout.
Accumulate and Parse Partial JSON
As tokens stream in, you accumulate them into a buffer. At any given point during the stream, your buffer contains a partial JSON document: the opening brace, some completed fields, and potentially a field that is still being generated. The challenge is extracting useful data from this partial JSON without waiting for the complete response.
There are two approaches to partial JSON parsing. The simple approach is to wait for complete field values: watch for patterns like "field_name": "complete value", (with the closing comma or brace) and extract completed fields as they appear. This is reliable and easy to implement but only works for top-level fields in flat schemas.
The more sophisticated approach uses a partial JSON parser that can handle incomplete JSON. Libraries exist for this in both Python and JavaScript. They return a partially populated object where completed fields have their final values and incomplete fields are omitted or marked as in-progress. The Instructor library's streaming mode uses this approach, returning a partial Pydantic model that updates as new tokens arrive, with completed fields available immediately.
For nested schemas, partial parsing becomes more valuable because inner objects or array items may complete before the outer object is finished. If your schema is an array of extracted entities, each entity might complete as a fully parsed object while the model is still generating subsequent entities. You can process each entity as it completes rather than waiting for the entire array.
Process Completed Fields Progressively
Progressive processing is the main reason to stream structured output. Common patterns include:
UI updates: In a user-facing application, display each extracted field as it becomes available. If the model is extracting contact information from a document, show the name as soon as it is generated, then the email, then the phone number. This gives the user immediate feedback rather than a loading spinner followed by all data at once.
Early termination: If a critical field reveals that the response will not be useful, cancel the stream early to save tokens. For example, if you are classifying documents and the first field is a relevance score of 0, you can cancel the stream without waiting for the model to generate the full analysis. This saves both time and cost.
Pipeline feeding: In a data pipeline, start processing the first extracted record while the model is still generating subsequent records. If you are extracting 50 entities from a long document, you can begin enrichment, deduplication, or storage for the first entity as soon as it completes, rather than waiting for all 50.
Progress indication: Track the number of completed fields versus total fields in the schema to show a progress percentage. This is more meaningful than a generic loading indicator because it tells the user how much of the extraction is complete.
Validate the Complete Response
Streaming does not change the validation requirements. After the stream completes, you should run the complete JSON through your validation layer (Pydantic, Zod, or custom validators) exactly as you would with a non-streaming response. The constrained decoding guarantee ensures the structural validation will pass, but semantic validation, range checks, format checks, and cross-field consistency, still needs to run on the complete response.
If you processed fields progressively during the stream, you may need to reconcile your progressive processing with the final validation result. A field that looked complete during streaming might fail cross-field validation when the complete response is available. Design your progressive processing to be tentative, showing partial results to the user or pipeline but not committing them as final until validation passes.
In practice, this means treating streamed partial results as a preview and the validated complete response as the source of truth. For UI applications, this is natural: update the display progressively, then finalize when validation passes or show an error if it fails. For pipeline applications, hold partial results in a staging area and commit them only after final validation.
When Streaming Adds Value
Streaming structured output adds the most value in three scenarios. First, when the response is large (complex schemas with many fields or arrays of objects) and latency to the first useful data point matters more than latency to the complete response. Second, when you want to show progress to a user, because streamed fields give a natural progress indicator. Third, when early termination can save significant cost, because canceling a stream after the first few fields is cheaper than generating the entire response and discarding it.
Streaming adds little value when the response is small (a simple classification with 2-3 fields), when latency is not user-facing (batch processing where all results are consumed together), or when your validation requires the complete response before any processing can begin. In these cases, the simpler non-streaming approach is appropriate.
Streaming with Instructor
The Instructor library supports streaming through its create_partial method, which returns a stream of partially populated Pydantic models. As tokens arrive, completed fields are populated in the model, and your application receives updated model instances. This is the most ergonomic way to consume streaming structured output in Python, because you work with typed Pydantic objects throughout, never with raw JSON strings or partial parsing logic.
The trade-off is that Instructor's automatic retry on validation failure only works on the complete response, not on partial data during streaming. If the complete response fails validation, Instructor retries the entire request, not just the failed field. This means streaming with Instructor gives you progressive display but does not reduce the total latency if a retry is needed.
Streaming structured output gives you progressive access to schema-compliant data as it generates. Use it when response size, user-facing latency, or early termination opportunities justify the additional parsing complexity. The schema guarantee holds for every token in the stream, so you can trust partial data as valid prefixes of the final response. Always validate the complete response after the stream ends.