How Do You Prevent AI from Learning Wrong Things

Where Wrong Learning Comes From

Wrong learning enters through three channels. User misinformation is the most common: a user states something incorrect, and the system stores it as knowledge. This can be accidental (the user is genuinely confused) or deliberate (an attacker trying to poison the knowledge base). Noisy feedback is the second channel: a user provides positive feedback for a response that contained incorrect information, perhaps because they did not notice the error or because the response was helpful despite the inaccuracy. The system interprets the positive feedback as validation of the underlying memories, including the incorrect one. Stale knowledge is the third channel: information that was correct when stored becomes incorrect as the world changes. A system that learned "the API supports v2 and v3" continues serving that information after v2 is deprecated unless something corrects it.

Evidence Gating

The primary defense against wrong learning is evidence gating. New information enters the system at a base confidence level (typically 5.0 on a 10-point scale). To increase above moderate confidence (7.0), the information must be corroborated by at least one independent source. To reach high confidence (8.0+), it needs two or more independent corroborations or verification against an authoritative external source. The gate prevents a single incorrect input from becoming trusted knowledge because incorrect claims rarely receive independent corroboration.

The evidence gate operates asymmetrically: increasing confidence is harder than decreasing it. A single credible contradiction is enough to reduce confidence, but a single confirmation is not enough to increase it significantly. This asymmetry is deliberate. The cost of maintaining false information (it gets served to users who rely on it) is higher than the cost of temporarily underrating true information (it still appears in results, just ranked lower). The system is designed to be skeptical by default and trusting only when evidence warrants it.

Source Independence

Source independence prevents gaming the evidence gate through repetition. If one user states the same incorrect fact in five different sessions, the system records five observations but counts them as one source. The independence check traces each piece of evidence back to its origin: the user account, the data source, or the system process that generated it. Only evidence from genuinely different origins contributes to the corroboration count.

System-derived information is treated carefully. If the consolidation process generates a summary from three memories that all originated from the same user, the summary inherits the parent origin rather than being treated as an independent source. This prevents a single input from being laundered through system processes into apparent corroboration.

Contradiction Detection

When new information contradicts existing knowledge, both pieces of information are preserved with their respective evidence chains. The contradiction is flagged for the consolidation process to review. The process compares the evidence supporting each side: how many independent sources, how reliable are those sources, how recent is the evidence. The side with stronger evidence retains higher confidence while the other side is demoted. Neither is deleted, because the weaker claim might turn out to be correct in a different context or at a different time.

Contradiction detection also catches stale knowledge. When a user provides information that contradicts a high-confidence memory, the system does not dismiss the new information. It flags the contradiction and checks whether the high-confidence memory's last verification is recent. If the memory has not been verified in months, the new contradicting information might reflect a genuine change rather than an error. The system responds by flagging the older memory for re-verification rather than simply trusting its historical confidence.

Confidence Thresholds in Retrieval

Even if incorrect information enters the system, confidence thresholds prevent it from dominating retrieval results. Memories below a retrieval threshold (configurable, typically 3.0 to 4.0) do not appear in standard recall results. New, unverified information stays at base confidence (5.0) and appears in results but is ranked below high-confidence, well-corroborated knowledge. For applications that require high accuracy, the retrieval threshold can be raised to only surface memories above 7.0, which requires at least one independent corroboration.

This tiered approach means that wrong information can exist in the system without causing harm. A low-confidence memory with incorrect information is present but rarely surfaces because higher-confidence correct memories rank above it. Over time, if the incorrect memory is never corroborated and never accessed, it fades through the natural decay process. The system self-corrects without requiring manual intervention for most cases of wrong learning.

Periodic Verification

For critical domains where wrong information has serious consequences, add periodic automated verification. Run a scheduled process that checks high-confidence memories against authoritative sources (official documentation, verified databases, known-good reference data). If a high-confidence memory fails verification, reduce its confidence to the base level and flag it for review. This catches both stale knowledge (the fact changed after it was stored) and errors that somehow passed through the evidence gate (rare but possible).