How Long Before RL Improves Retrieval Quality

The Learning Timeline

Days 1-3 (0-50 interactions). The system operates on its baseline scoring: vector similarity with default weights for recency, frequency, and confidence. New memories are stored and begin accumulating access history. The system is functional but not yet adaptive. Retrieval quality is equivalent to a static system because there is not enough usage data to differentiate memories by value. During this phase, the system is essentially indexing, building the raw material that later learning will refine.

Week 1-2 (50-500 interactions). Access patterns begin to differentiate memories. Frequently retrieved memories gain higher activation scores than those retrieved once or never. Memories that are relevant to common query types rise in the rankings, while memories about rarely discussed topics settle into lower positions. The system starts surfacing the most-used memories more readily, which aligns with what is likely to be useful. Users may notice that the system is "getting better" at finding relevant context, though the improvement is subtle at this stage.

Month 1 (500-2,000 interactions). The entity graph has enough data to provide meaningful spreading activation. Querying one topic surfaces related memories through entity connections, not just text similarity. Confidence scores stabilize based on corroboration patterns: facts confirmed across multiple independent sessions reach high confidence, while facts from a single interaction remain at baseline. The system produces noticeably different rankings than it did on day one, with measurably better precision and recall.

Months 2-3 (2,000-10,000 interactions). The consolidation system has had time to merge redundant memories, clean up contradictions, and establish high-confidence core knowledge. The active memory set is lean and well-ranked. Retrieval quality reaches a steady state where further improvement is incremental rather than dramatic. The system continues to improve, but the rate of improvement slows as the most impactful learning has already occurred.

Factors That Affect Speed

Traffic volume. More interactions mean faster learning. A system serving 1,000 queries per day accumulates enough feedback for meaningful learning within a week. A system serving 10 queries per day needs months to accumulate the same signal. If your application has low traffic, consider backfilling the memory store from historical data (conversation logs, CRM records) to give the system a head start on corroboration and confidence building.

Query diversity. Diverse queries improve learning because they exercise different parts of the memory store and different ranking strategies. If all queries are about the same topic, the system learns to rank well for that topic but may not generalize to other topics. Applications with diverse user bases and varied query patterns converge to better overall ranking quality than applications with homogeneous usage.

Feedback quality. Explicit feedback (ratings, corrections, "that's wrong" statements) is more informative than implicit feedback (access patterns). A single explicit correction provides as much learning signal as dozens of implicit access events. Systems that capture explicit feedback alongside implicit patterns learn faster and converge to higher quality levels.

Memory store size. Larger stores take longer to converge because there are more memories whose activation scores need to stabilize. A store with 100 memories converges in days because each memory gets tested by many queries. A store with 100,000 memories may take weeks because most individual memories are rarely queried and thus slow to accumulate meaningful access histories.

How to Measure Improvement

Track retrieval quality metrics over time to verify the system is actually improving. The most actionable metrics are:

Mean reciprocal rank (MRR). The average of 1/rank for the first relevant result. If the most useful memory is consistently the first result, MRR approaches 1.0. If it is buried at position 5, MRR is 0.2. A learning system should show MRR increasing over the first few weeks.

Memory utilization rate. What percentage of injected memories does the model actually reference in its response? If the system injects five memories and the model uses three, utilization is 60%. A learning system should show utilization increasing as the retrieval becomes more precise, injecting context the model actually needs rather than tangentially related content.

Reformulation rate. How often does the user rephrase their question after receiving a response? High reformulation rates suggest the system is not retrieving the right context. A learning system should show decreasing reformulation rates as it converges on effective retrieval strategies.

Confidence distribution. Track the average confidence of retrieved memories over time. A learning system with good consolidation should show the average confidence increasing as facts are corroborated across multiple sessions and low-confidence noise is filtered out or decayed.

What Convergence Looks Like

A system that has converged shows stable, flat metrics rather than continuing improvement. MRR plateaus at a level that reflects the inherent difficulty of the retrieval task. Memory utilization stabilizes at a rate that reflects how well the system matches injected context to model needs. Confidence distributions settle into a bimodal pattern: high-confidence well-established facts and lower-confidence recent observations, with little in between.

Convergence does not mean the system stops learning. It means the rate of improvement has slowed to the point where changes are incremental. The system continues to adapt to new content, changing user patterns, and evolving contexts, but the rankings are no longer swinging dramatically with each batch of interactions. This stability is desirable: it means the system has found an effective strategy and is maintaining it.

If you never see convergence (metrics keep oscillating), the system may have insufficient feedback data, conflicting signals from different user groups, or reward function issues. Investigate by segmenting metrics by user type, query type, and time period to identify the source of instability.