How to Build Guardrails That Catch Hallucinations

Step 1: Extract Individual Claims from the Response

The first step in hallucination detection is decomposing the model's response into individual, verifiable claims. A response like "The company was founded in 2019 by Sarah Chen, has 500 employees, and raised $50M in Series B funding" contains four distinct claims, each of which can be independently true or false. Checking the response as a single unit is insufficient because a response might be mostly accurate with one fabricated detail, and whole-response verification would either miss the detail or falsely reject the entire response.

Claim extraction can be done with a secondary LLM call or with a trained extraction model. The LLM approach prompts a fast model (Claude 3.5 Haiku, GPT-4o Mini) with instructions like: "Extract every factual claim from the following text. List each claim as a separate statement. Include only claims that assert specific facts, not opinions, hypotheticals, or general statements." This produces a list of atomic claims that the verification step processes individually.

The trained model approach uses a fine-tuned sequence labeling model that identifies claim boundaries in the text. This is faster than an LLM call (10-50 milliseconds vs 200-500 milliseconds) but requires a trained model and may be less accurate on complex or unusual text structures. For most applications, the LLM approach provides the best accuracy and is worth the additional latency because it runs only once per response.

Filter extracted claims to focus verification on claims that matter. Not every statement in a response needs fact-checking. Subjective statements ("this is a popular approach"), hedged statements ("this might be worth considering"), and definitional statements ("machine learning is a subset of AI") rarely need verification. Focus verification resources on specific factual assertions: named entities, numerical values, dates, relationship claims, and causal statements. This filtering reduces the number of claims to verify, which directly reduces the latency and cost of the verification step.

Step 2: Check Claims Against Grounding Sources

Each extracted claim is verified against the grounding sources that were provided to the model during generation. The grounding sources include any documents retrieved by the RAG pipeline, entries from the knowledge graph, memories from the persistent memory system, and structured data from databases or APIs. The key principle is that a claim is only considered "supported" if it can be traced to a specific source.

Entailment verification uses a natural language inference (NLI) model to determine whether each claim is entailed by (logically follows from) the grounding sources. Given a claim and a passage from the grounding context, the NLI model classifies the relationship as entailment (the passage supports the claim), contradiction (the passage contradicts the claim), or neutral (the passage neither supports nor contradicts the claim). Cross-encoder models fine-tuned for NLI, such as DeBERTa-v3-large-mnli, achieve 90%+ accuracy on this task with 50-100 milliseconds per claim-source pair.

Semantic similarity checking is a lighter alternative to entailment verification. Compute the embedding similarity between each claim and the grounding sources. Claims with high similarity to at least one source passage are likely supported. Claims with low similarity to all source passages are potentially hallucinated. Semantic similarity is faster than entailment verification (it uses pre-computed embeddings rather than cross-encoder inference) but less accurate because high semantic similarity does not guarantee factual consistency, two passages can be about the same topic but assert different facts.

Knowledge graph verification checks entity and relationship claims against structured knowledge. If the claim "the company was founded in 2019" involves an entity (the company) and an attribute (founding year) that exist in the knowledge graph, the graph provides a definitive answer: either the graph records the founding year as 2019 (supported) or as a different year (contradicted) or does not have a founding year recorded (unverifiable). Knowledge graph checks are the most precise verification method for the claims they can address, but they can only verify claims about entities and relationships that are represented in the graph.

Persistent memory verification extends grounding beyond the static knowledge base. Adaptive Recall's memory system stores facts that the system has observed, discussed, and verified in the context of specific users, projects, or organizations. When the model claims "your project uses PostgreSQL," the memory system can confirm or deny this based on previous interactions. Memory-based verification catches hallucinations about user-specific details that neither the general knowledge base nor the knowledge graph would cover, because those details exist only in the interaction history.

For each claim, run verification against all available sources and aggregate the results. A claim supported by the knowledge graph and corroborated by persistent memory is high-confidence supported. A claim with moderate semantic similarity to a retrieved passage but no knowledge graph entry is lower-confidence. A claim that contradicts the knowledge graph, regardless of what the retrieved passages say, is a detected hallucination. The aggregation logic determines the overall verification verdict for each claim.

Step 3: Score and Classify Each Claim

After verification, classify each claim into one of three categories with an associated confidence score.

Supported claims are those where at least one grounding source entails or directly states the claim. The confidence score reflects the strength of the support: a claim stated explicitly in a retrieved document with high entailment score gets high confidence, while a claim that is loosely implied by a passage with moderate similarity gets lower confidence. Supported claims pass through the guardrail unchanged.

Contradicted claims are those where at least one grounding source directly contradicts the claim. The model said the founding year was 2019 but the knowledge graph says 2021. The model said the API supports JSON output but the documentation says it returns XML only. Contradicted claims are the most certain type of hallucination detection because the guardrail has positive evidence that the claim is wrong, not just absence of support.

Unverifiable claims are those where no grounding source supports or contradicts the claim. The model made a specific assertion that the grounding context simply does not address. Unverifiable claims are the trickiest category because they might be accurate (the model might be using valid parametric knowledge that the grounding sources do not happen to cover) or they might be hallucinated (the model might be fabricating a claim that sounds plausible). The treatment of unverifiable claims depends on your application's risk tolerance.

Assign numerical confidence scores to each claim based on the aggregated verification results. A claim supported by multiple sources with high entailment scores gets a confidence above 0.9. A claim supported by a single source with moderate similarity gets a confidence of 0.5-0.7. A contradicted claim gets a confidence below 0.1 regardless of how many sources contradict it. An unverifiable claim gets a confidence in the 0.3-0.5 range, reflecting the uncertainty inherent in the absence of evidence.

Step 4: Handle Detected Hallucinations

How you handle detected hallucinations depends on the severity, the application context, and the specific claim category.

For contradicted claims (the guardrail has evidence the claim is wrong), the strongest response is correction. Replace the hallucinated claim with the correct information from the grounding source. "The company was founded in 2019" becomes "The company was founded in 2021 (Source: Company Profile)." If automatic correction is not feasible (the grounding source is ambiguous or the correction would disrupt the response's coherence), remove the claim and note the removal: "I removed a claim about the founding year because I could not verify it against our records."

For unverifiable claims in high-stakes applications (medical, legal, financial), treat unverifiable claims as potentially hallucinated and either remove them or add explicit caveats. "According to available sources, [supported claims]. Note: I could not verify [specific claim] against our knowledge base, so please check this independently." This transparency preserves the value of the supported claims while protecting users from acting on unverified information.

For unverifiable claims in lower-stakes applications, you can deliver them with softer hedging. Add qualifiers like "I believe" or "based on general knowledge" to claims that are not grounded in specific sources. This signals to the user that these claims come from the model's parametric knowledge rather than verified sources, without the heavy-handed intervention of removal or blocking.

Regeneration is an option when the response contains too many unsupported claims. If more than 30-40% of the claims in a response fail verification, the response as a whole is unreliable. In this case, regenerate with an enhanced prompt that explicitly instructs the model to answer only from the provided context and to say "I don't have enough information" for questions the context does not address. The regenerated response typically has a much higher verification rate because the model received explicit grounding instructions.

Logging every hallucination detection event is essential for system improvement. Log the hallucinated claim, the grounding sources that contradicted or failed to support it, the action taken (correction, removal, hedging, regeneration), and the user query that triggered the response. Over time, these logs reveal patterns: specific topics that consistently produce hallucinations (indicating a gap in the knowledge base), specific claim types that the model fabricates most often (indicating a prompt engineering opportunity), and specific source types that provide the best grounding (indicating where to invest in better retrieval).