RLHF vs RLVR: Two Paths to Self-Improvement

How RLHF Works

Reinforcement learning from human feedback was popularized by OpenAI's work on InstructGPT and later ChatGPT. The core idea is to train a reward model on human preference data and then use that reward model to guide the AI system's behavior through reinforcement learning. Humans compare pairs of AI outputs and indicate which they prefer. These preferences train a reward model that predicts which outputs humans would rate more highly. The AI system then optimizes against this reward model, learning to produce outputs that score well.

In the context of LLM training, RLHF operates on the model weights: the reward signal drives gradient updates that adjust how the model generates text. For memory systems, the same principle applies at the retrieval layer. Human feedback (thumbs up, thumbs down, ratings) creates preference data about which retrievals are good and which are bad. This preference data guides confidence updates, ranking adjustments, and knowledge graph evolution without touching the underlying LLM weights.

The strength of RLHF is that it captures quality dimensions that cannot be automatically measured. Was the response helpful? Was the tone appropriate? Did the information address the user's actual need rather than their literal question? These are subjective judgments that require human evaluation. RLHF allows these judgments to influence the system's behavior over time.

The weakness of RLHF is that human preferences are noisy, inconsistent, and sometimes wrong. Different humans prefer different things. The same human may prefer different things on different days. Some humans provide feedback carelessly. Some provide feedback strategically (rating a response poorly not because it was bad but because they want a different outcome). A system that optimizes purely for human preference can learn to produce outputs that sound good rather than outputs that are accurate, a phenomenon known as reward hacking or sycophancy.

How RLVR Works

Reinforcement learning with verifiable rewards replaces subjective human judgment with objective, automatically checkable outcomes. Instead of asking "did the human prefer this output?" RLVR asks "did this output lead to a verifiably correct result?" The reward comes from the outcome itself rather than from a human evaluation of the outcome.

For a coding assistant, a verifiable reward might be: did the suggested code compile and pass tests? For a math reasoning system, it might be: does the answer match the known correct answer? For a retrieval system, it might be: was the retrieved information factually accurate when checked against a ground truth source? The key property of verifiable rewards is that they can be evaluated automatically, without human intervention, at scale.

RLVR gained prominence through DeepSeek's R1 model (2025), which demonstrated that reinforcement learning with verifiable rewards on math and reasoning tasks could produce emergent reasoning capabilities without the expensive human labeling process that RLHF requires. The system learned to solve problems correctly by receiving reward signals based on whether its answers matched known solutions, without any human preference data.

The strength of RLVR is precision. The reward signal is objective and unambiguous: the answer is either right or wrong, the code either compiles or it does not, the retrieved fact either matches the source or it does not. There is no noise from subjective interpretation and no risk of reward hacking because the system cannot produce a wrong answer that "sounds right" and get rewarded for it.

The weakness of RLVR is limited scope. Many important aspects of AI quality are not objectively verifiable. Whether a response was helpful, whether the level of detail was appropriate, whether the phrasing was clear, whether the system addressed the user's real concern rather than their surface question, these are all quality dimensions that verifiable rewards cannot capture. A system that optimizes purely for verifiable correctness might produce technically accurate but unhelpful responses.

Comparison for Memory Systems

Applied to memory and retrieval systems, RLHF and RLVR have complementary roles. RLVR handles factual accuracy: when the system retrieves a memory that claims a specific fact, you can verify that fact against authoritative sources. If the memory says "the API rate limit is 1000 requests per minute" and the actual documentation says 500, that is a verifiably wrong retrieval. The confidence of that memory should decrease, and the system should learn to prefer memories that are factually verified.

RLHF handles relevance and usefulness: when the system retrieves a factually correct memory that does not actually address the user's question, only human feedback can indicate that the retrieval was unhelpful. The memory might be accurate, but it was the wrong information for this context. RLHF captures this distinction by learning which memories users find useful in which contexts, even when all the retrieved memories are factually correct.

In practice, most retrieval events involve both dimensions. The system retrieves several memories, some of which are factually accurate and relevant (high RLVR reward and high RLHF reward), some of which are accurate but irrelevant (high RLVR reward but low RLHF reward), and some of which are relevant but potentially outdated (uncertain RLVR status but high RLHF reward because the user still found the dated information useful). A system that uses only RLVR would miss the relevance signal. A system that uses only RLHF would be vulnerable to popular-but-wrong information that users rate positively because it sounds authoritative.

Combining Both Approaches

The most effective strategy for self-improving memory systems combines both reward types. Use verifiable rewards as the primary gate for factual accuracy: no memory should increase above a moderate confidence threshold unless its factual claims can be verified. Use human feedback as the secondary signal for relevance and usefulness: among the memories that pass the factual verification gate, prefer the ones that users consistently find helpful.

This layered approach prevents the failure modes of each method in isolation. RLVR prevents the system from learning to prefer wrong-but-popular information (the RLHF failure mode). RLHF prevents the system from being technically correct but unhelpful (the RLVR failure mode). Together, they push the system toward information that is both accurate and useful, which is what users actually need.

The weighting between the two signals should depend on the consequence of errors. For medical information, legal advice, or financial data, RLVR should dominate because factual accuracy matters more than subjective helpfulness. For creative assistance, brainstorming, or exploratory research, RLHF should have more weight because the "right" answer depends on what the user finds useful. For most production use cases, an equal weighting is a reasonable starting point that can be adjusted based on the specific domain's error tolerance.

Implementation Complexity

RLHF is simpler to implement for memory systems because the feedback collection infrastructure (thumbs up, thumbs down, user ratings) is straightforward and domain-independent. The challenge is feedback sparsity: most users do not provide feedback, so the signal is thin and learning is slow.

RLVR is more complex to implement because the verification mechanism is domain-specific. Verifying that a code suggestion compiles requires a build system. Verifying that a factual claim is correct requires access to authoritative sources. Verifying that a retrieved document is relevant requires a ground truth relevance label that may not exist. Each verification mechanism must be built and maintained separately. The advantage is that once built, verification is automatic and scales without human involvement.