AI Memory Architecture: A Decision Framework

Why You Need a Framework

Memory architecture decisions are interdependent. Your choice of storage backend constrains your retrieval strategy options. Your retrieval strategy constrains your latency characteristics. Your latency requirements constrain your scaling options. Making these decisions independently, choosing a storage backend without considering retrieval strategy, or choosing a retrieval strategy without considering latency, leads to architectures with internal contradictions that surface as production problems.

The framework presented here orders decisions by dependency: each decision takes the output of previous decisions as input, so by the time you reach the final decision, the architecture is internally consistent. It also separates "what do I need" decisions (driven by your application requirements) from "how do I build it" decisions (driven by available technologies), which prevents technology enthusiasm from overriding practical requirements.

The Seven Decision Points

Decision 1: Memory Content Classification

Before choosing any technology, classify the information your memory system will handle. Every memory falls into one of four categories based on two dimensions: structure (whether the information has a predictable format) and stability (how frequently the information changes).

Structured and stable: account details, configuration settings, factual attributes. These have known fields and change infrequently. They are best served by traditional database storage with strong consistency guarantees. Retrieval is primarily by key lookup or attribute filtering.

Structured and volatile: metrics, counters, access patterns, activation levels. These have known fields but change frequently. They need storage optimized for high-frequency updates with eventual consistency acceptable. Caching layers work well here.

Unstructured and stable: consolidated knowledge, verified facts, established relationships. These are free-form text or content that has been validated and does not change frequently. They benefit from semantic indexing (vector embeddings) and knowledge graph representation.

Unstructured and volatile: conversation context, recent observations, preliminary extractions. These are free-form content that may be updated, corrected, or superseded quickly. They need fast write and read operations with relaxed consistency, and should be treated as candidates for consolidation into stable knowledge.

Most applications have memories in multiple categories. The distribution across categories drives your storage architecture: if 80% of your memories are unstructured and stable, a vector database with graph capabilities is your primary backend. If 80% are structured and stable, a traditional database with a vector extension handles the semantic search needs without a separate vector store.

Decision 2: Retrieval Pattern Matrix

For each memory category, define the query patterns your application uses. Map each pattern to one of five retrieval strategies: semantic search (find memories similar in meaning to a query), entity traversal (find memories connected to a specific entity through relationships), temporal queries (find memories within a time range or ordered by recency), attribute filtering (find memories matching specific metadata criteria), and full-text keyword search (find memories containing specific terms).

Build a matrix with memory categories as rows and retrieval strategies as columns. Mark each cell as primary (this is the main way this category is queried), secondary (used occasionally for this category), or unused. The strategies marked as primary for any category are requirements for your storage architecture. Strategies marked as secondary are desirable but can be approximated. Strategies marked as unused everywhere can be ignored entirely.

This matrix prevents over-engineering. If no memory category uses graph traversal as a primary retrieval pattern, you do not need a graph database, regardless of how appealing the technology is. If every category uses semantic search as primary, a vector-capable backend is non-negotiable.

Decision 3: Performance Envelope

Define the performance requirements for your primary retrieval patterns. For each primary pattern, specify: the latency budget (maximum acceptable response time at p95), the throughput requirement (queries per second at peak load), the freshness requirement (how quickly after a memory is written must it be available for retrieval), and the result quality requirement (minimum acceptable precision and recall).

These four dimensions create a performance envelope that your architecture must operate within. Trade-offs between dimensions are explicit: you can achieve lower latency by accepting lower freshness (use caches that are slightly stale), or higher quality by accepting higher latency (run more expensive re-ranking), or higher throughput by accepting lower quality (reduce the number of retrieval strategies per query).

Be specific and realistic. "As fast as possible" is not a latency budget; "under 300ms at p95 with 50 concurrent users" is. Specific requirements lead to specific architectural choices. Vague requirements lead to architectures optimized for nothing in particular.

Decision 4: Storage Architecture

With your retrieval patterns and performance envelope defined, the storage architecture follows logically. If you need only semantic search within a latency budget: a single vector database or a relational database with vector extension is sufficient. If you need semantic search plus entity traversal: you need a hybrid architecture with vector search and graph capabilities, either through separate backends or through a service that integrates both (Adaptive Recall provides both through its API). If you need all five retrieval strategies at low latency: you need a multi-backend architecture with a caching layer, vector store, graph store, and metadata index, with a query coordinator that routes to the appropriate backend.

The decision tree is: how many primary retrieval strategies do you have? One strategy: single backend optimized for that strategy. Two strategies: hybrid backend or managed service that covers both. Three or more strategies: multi-backend architecture with query routing. At each tier, prefer managed services over self-operated infrastructure unless you have specific requirements (data residency, customization, cost at extreme scale) that managed services cannot satisfy.

Decision 5: Lifecycle Model

Define how memories evolve over time. The lifecycle model has three components: promotion rules (when does information move from one layer or confidence level to another), consolidation rules (when and how are related memories merged), and retirement rules (when are memories archived or deleted).

The simplest lifecycle model is "store everything forever, never consolidate." This works for applications with small memory volumes (under 10,000 memories total) and short time horizons (months, not years). As volume grows, you need active lifecycle management.

A moderate lifecycle model adds consolidation (merge related memories after a threshold count) and archival (move untouched memories to cold storage after a time period). This handles most production applications with moderate volume.

An advanced lifecycle model adds confidence tracking (memories gain confidence through corroboration and lose it through contradiction), evidence-gated promotion (memories only move to higher confidence tiers when corroborated by independent evidence), and adaptive retention (retention periods adjust based on access patterns and confidence). This is the model Adaptive Recall uses, and it is appropriate for applications where memory quality directly affects user experience.

Decision 6: Isolation and Multi-Tenancy

Define your tenant model and isolation requirements. The key question is: what is the blast radius of a data breach? If a bug exposes one tenant's memories to another tenant, what is the consequence?

For applications where tenant data is low-sensitivity (public information, non-personal data), logical isolation with namespace separation is sufficient and cost-effective. For applications where tenant data is moderate-sensitivity (personal preferences, interaction history), logical isolation with enforced query scoping and audit logging is appropriate. For applications where tenant data is high-sensitivity (health records, financial data, legal communications), physical isolation with separate storage instances per tenant is necessary, despite the higher operational cost.

The isolation decision also affects cost structure. Physical isolation means each tenant has a minimum infrastructure cost regardless of usage. Logical isolation allows cost sharing across tenants, reducing per-tenant costs but increasing the complexity of capacity planning and resource contention management.

Decision 7: Operational Model

Define who is responsible for operating the memory system and what level of operational investment you are committing to. Three models exist. Fully managed (use a service like Adaptive Recall that handles storage, retrieval, lifecycle, monitoring, and scaling): lowest operational burden, fastest time to production, limited customization. Self-managed with managed components (run your own retrieval logic but use managed databases, managed vector stores, managed caches): moderate operational burden, good customization, requires integration expertise. Fully self-hosted (operate everything on your own infrastructure): highest operational burden, full control, requires dedicated infrastructure team.

Match the operational model to your team's capabilities and priorities. If memory infrastructure is not your core product, a fully managed approach lets you ship faster and focus on your application. If you need deep customization or have strict data residency requirements, self-managed or self-hosted models give you the control you need at the cost of ongoing operational investment.

Applying the Framework

Work through the seven decisions in sequence, documenting each choice and the rationale. When you finish, you should have a document that specifies: what types of memories you store, how you query each type, what performance you require, which storage backends you need, how memories evolve over time, how tenants are isolated, and who operates the infrastructure. This document is your architecture specification. It can be reviewed by stakeholders, validated against requirements, and used as the blueprint for implementation.

The framework is also useful for evaluating existing architectures. Walk through the seven decisions for your current system and see where the decisions were implicit rather than explicit. Implicit decisions are where architectural problems hide, because nobody chose them deliberately and nobody is monitoring whether they are still appropriate.