Evidence-Gated Learning: Proof Before Updating

The Problem It Solves

Standard reinforcement learning updates behavior after every interaction. This creates a vulnerability: a few coincidental observations can push the system toward a suboptimal policy. If a memory happens to be useful in one unusual query, basic RL boosts it. If a ranking strategy happens to fail on a few atypical queries, basic RL demotes it. The system chases noise instead of learning signal.

This is especially dangerous in memory systems where the stakes are high. A memory that gets incorrectly boosted might inject outdated or wrong information into future conversations. A ranking strategy that gets incorrectly demoted might miss genuinely relevant context. The cost of learning wrong things can exceed the cost of not learning at all.

How Evidence Gating Works

Evidence gating adds a filter between the feedback signal and the policy update. The filter requires that a pattern be observed multiple times, across independent contexts, before it is treated as real. This is analogous to the scientific method: a single experiment is a data point, but you do not change your theory until the result is replicated.

For memory systems, evidence gating means a memory's confidence score increases only when independent interactions corroborate its value. A fact mentioned once in one conversation gets baseline confidence. The same fact mentioned independently in a second conversation (not a continuation of the first) gets increased confidence. After five independent confirmations, the fact reaches high confidence and is protected from decay.

For ranking systems, evidence gating means a ranking parameter change is applied only when the same directional signal is observed across multiple queries from multiple users. If recency should be weighted more heavily, that signal needs to appear consistently across diverse query types, not just in a few queries from a single user during a single session.

Gating Thresholds

The gating threshold determines how many independent confirmations are required before an update is applied. A low threshold (2-3 confirmations) makes the system responsive but still vulnerable to coincidence. A high threshold (10+ confirmations) makes the system very conservative but slow to learn.

The right threshold depends on the cost of being wrong. For safety-critical applications (medical information, financial advice), high thresholds are appropriate because incorrect learning could cause real harm. For low-stakes applications (entertainment recommendations, casual search), lower thresholds are fine because the cost of a bad recommendation is minimal.

Adaptive Recall uses a graduated gating approach through its confidence scoring system. New memories start with baseline confidence (1.0). Each independent corroboration increases confidence by a decreasing increment (diminishing returns). Memories above 8.0 confidence are considered well-established and are protected from lifecycle decay. This creates a natural evidence gate: only memories that have been validated across multiple independent interactions achieve the confidence levels that protect them long-term.

Independence Is Key

The evidence must be independent. Three mentions of the same fact in a single conversation are not three independent confirmations; they are one data point. The same user asking similar questions in rapid succession is not independent evidence; it is one session. Independence requires different sessions, different query contexts, or different users encountering the same pattern.

Checking for independence requires tracking the provenance of each piece of evidence. When a memory receives corroboration, the system checks whether the current interaction is genuinely independent from previous corroborations. Same user, same day, same topic cluster: not independent. Different user, different week, different topic but same entity: independent.

Benefits Beyond Noise Reduction

Evidence gating provides benefits beyond noise reduction. It creates an audit trail of why the system believes what it believes. Each high-confidence memory has a history of corroborations that explain how it achieved that status. This transparency is valuable for debugging, for compliance, and for user trust.

It also enables graceful contradiction handling. When new evidence contradicts existing high-confidence knowledge, the system does not immediately flip. It reduces the confidence of the old memory and accumulates evidence for the new claim. Only when the new claim reaches a similar evidence threshold does it replace the old one. This prevents flip-flopping on contested facts and gives the system stability even when information is evolving.