How to Build a Custom Output Validator for LLMs

Step 1: Define Your Validation Rules

Start by cataloging every rule your LLM output must follow. Pull these from three sources: your product requirements, your compliance obligations, and your incident history (every time the model produced wrong output that reached a user, that failure implies a missing validation rule).

Organize rules into four categories based on how they are checked:

Format rules verify the structure of the output. Does the response contain valid JSON when JSON is expected? Does it stay within the maximum length? Does it include required sections? Format rules are the cheapest to check because they use parsing and pattern matching rather than semantic analysis. A JSON schema validator runs in microseconds. A regex that checks for required section headers runs equally fast.

Content policy rules verify that the output follows your organization's policies. Does the response avoid mentioning competitors? Does it refrain from making guarantees that your legal team has not approved? Does it stick to the topics within the bot's designated scope? Content rules typically require keyword matching for simple cases and classifier models for nuanced cases. Checking whether the response mentions a competitor name is a string match. Checking whether the response implies a guarantee without using the word "guarantee" requires a trained classifier.

Business logic rules verify that factual claims in the output align with your data. Does the quoted price match the price in your database? Does the product availability claim reflect current inventory? Does the shipping estimate align with the actual shipping calculator? Business logic rules require data lookups: extracting the claim from the model's response and comparing it against a source of truth.

Factual accuracy rules verify that general knowledge claims are correct. These are the hardest validators to build because they require either a comprehensive knowledge base or a secondary model call to verify claims. Most teams start without factual accuracy validators and add them only for high-risk domains where wrong facts create liability.

For each rule, define three properties: the check logic (how to detect a violation), the severity (informational, warning, or critical), and the failure action (log, retry, block, or escalate). Critical rules that trigger a block must have a fallback response ready so the user does not see an empty or error response.

Step 2: Build the Validator Interface

Every validator in your system should implement the same interface so that they can be composed into pipelines, swapped, and tested independently. The interface needs exactly three things: a validate method that accepts the LLM output string (and optionally the original input for context), a result object that reports whether validation passed or failed along with the reason and severity, and a name string for logging and observability.

Keep the result object structured rather than boolean. A simple pass/fail boolean loses the information you need for debugging, monitoring, and tuning. The result should include the validator name, the pass/fail status, the severity level if it failed, a human-readable reason string, and optionally the specific substring or value that triggered the failure. This metadata becomes invaluable when you are analyzing thousands of validation results to understand patterns in model behavior.

The validate method should be stateless. It receives the output, checks it, and returns a result. It should not modify the output, cache state between calls, or have side effects. Statelessness makes validators testable, composable, and safe to run concurrently. If you need a validator that modifies output (like a PII redactor), build it as a separate transformer step in the pipeline rather than mixing transformation logic into validation logic.

Define a consistent error taxonomy. Every failure should map to a category: FORMAT_ERROR, POLICY_VIOLATION, DATA_MISMATCH, FACTUAL_ERROR, SAFETY_CONCERN. Categories let you aggregate metrics meaningfully. Knowing that "12% of responses failed validation" is less useful than knowing that "8% failed on FORMAT_ERROR and 4% failed on POLICY_VIOLATION," because the two categories have different root causes and different fixes.

Step 3: Implement Rule-Specific Validators

Regex and pattern validators handle format checks and simple content rules. These run in microseconds and should always be the first layer in your pipeline. Build validators for JSON structure (use a JSON parser, not regex, for actual JSON validation), response length limits, required field presence, prohibited patterns (competitor names, banned phrases, internal jargon that should not be user-facing), and format consistency (dates in the right format, currency values with proper symbols).

Lookup validators handle business logic rules by extracting claims from the output and comparing them against your data. The extraction step is the hard part. If your model returns structured output (JSON with a "price" field), extraction is trivial. If your model returns free text that mentions a price ("The subscription costs $29/month"), you need a small extraction step, either a regex for well-formatted values or a lightweight NER model for more complex extractions. Once you have the extracted value, the comparison against your data source is straightforward database or API query.

Classifier validators handle nuanced content rules that pattern matching cannot express. Train or fine-tune a small classifier model (BERT-base or DistilBERT is sufficient for most classification tasks) on examples of compliant and non-compliant responses specific to your domain. A classifier that detects whether a response makes an unauthorized guarantee, goes off-topic, or adopts an inappropriate tone requires training data from your actual production traffic. Start with 500-1000 labeled examples, train a classifier, deploy it as a guardrail, and use the production traffic it processes to generate more training data for iterative improvement.

LLM-based validators handle checks that require genuine language understanding. Use a fast, cheap model (Claude Haiku or GPT-4o Mini) with a specific evaluation prompt. The prompt should describe the rule, provide 2-3 examples of passing and failing responses, and ask the evaluator model to return a structured judgment. LLM-based validators are the most flexible but the most expensive; reserve them for rules where simpler approaches demonstrably fail. A common pattern is to start with an LLM-based validator for a new rule, collect its judgments as training data, then train a classifier to replace it once you have enough labeled examples.

Step 4: Chain Validators in a Pipeline

A validation pipeline runs multiple validators against the same output in a defined order. The order matters for both performance and correctness. Arrange validators from cheapest to most expensive, and from most critical to least critical within the same cost tier.

The pipeline should support two execution modes. Fail-fast mode stops at the first critical failure and immediately triggers the failure action. This is the right mode for production traffic where latency matters and a single critical violation is enough to block the response. Full-scan mode runs every validator regardless of failures and collects all results. This is the right mode for testing, debugging, and batch evaluation where you want a complete picture of all violations in a single response.

Implement short-circuit logic based on severity. If a cheap regex validator detects a critical violation (the response contains a credit card number), skip the expensive LLM-based validators entirely because the response is already going to be blocked. If a format validator detects that the response is not valid JSON, skip the business logic validators that would parse that JSON. Short-circuiting prevents wasting compute on responses that have already failed.

Make the pipeline configurable rather than hardcoded. Store the list of active validators, their order, and their severity thresholds in configuration rather than code. This lets you enable or disable validators, adjust thresholds, and reorder the pipeline without deploying new code. Configuration-driven pipelines also let you run different validator sets for different use cases (a customer-facing chat might have stricter content policy validators than an internal tool).

Step 5: Handle Validation Failures

What happens when a response fails validation is as important as detecting the failure. The wrong failure handling creates a worse user experience than the original violation. A system that blocks 5% of responses with a generic "I cannot help with that" message frustrates users who asked perfectly reasonable questions that happened to trigger an overly aggressive validator.

Retry with modified prompt is the best first action for soft failures. If the response failed a format check (wrong JSON structure), retry the same request with an additional instruction in the prompt emphasizing the required format. If the response went slightly off-topic, retry with a more constrained system prompt. Limit retries to 1-2 attempts to avoid latency spirals. Track retry rates by validator; a validator that triggers retries on more than 10% of requests is either too aggressive or the prompt needs improvement.

Return a safe fallback for hard failures where the violation is clear and the response should not reach the user. The fallback should be specific to the type of failure rather than a generic error. If the model quoted the wrong price, return a response that directs the user to the pricing page rather than providing the wrong number. If the model went off-topic, return a response that redirects to the supported topics rather than blocking silently.

Escalate to human review for critical failures in high-stakes domains. If the model provided medical advice that a validator flagged as potentially dangerous, route the conversation to a human rather than either blocking the response or letting it through. Escalation preserves the user experience while keeping a human in the loop for decisions that code cannot safely make alone.

Log and pass through for informational validators. Not every validation failure needs to block the response. Some validators exist to collect data: tracking how often the model mentions competitors, how frequently responses exceed target length, how often the tone drifts from the desired style. These validators log their findings without interrupting the response flow, and the data they collect informs prompt improvements and future validator development.

Step 6: Add Observability and Iterate

Every validation result should be logged with the validator name, the pass/fail status, the severity, the reason, the response time of the validator, and a hash or ID linking back to the original request. This data is the foundation for improving both your validators and your prompts.

Build dashboards that track four key metrics: the overall validation pass rate (what percentage of responses pass all validators), the per-validator failure rate (which validators trigger most often), the false positive rate (how often validators block good responses, estimated by sampling and human review), and the latency impact (how much time the validation pipeline adds to each request).

Review false positives weekly. Every false positive is a validator that is too aggressive, hurting user experience by blocking legitimate responses. Tune the validator's threshold, add exceptions for specific patterns, or retrain the underlying model with corrected labels. The goal is not zero failures but zero false positives at an acceptable false negative rate. In practice, this means accepting that some bad responses will occasionally pass through (and handling them through feedback mechanisms) while ensuring that good responses are never incorrectly blocked.

Use validation data to improve prompts. If a specific validator fires frequently, the underlying prompt is not constraining the model effectively. Rather than making the validator more aggressive (which increases false positives), improve the prompt to reduce the frequency of the violation in the first place. The ideal state is validators that rarely fire because the prompts are well-designed, but catch the occasional edge case when they do.