Metadata Filtering in RAG: Beyond Vector Similarity

The Problem with Pure Semantic Search

Vector similarity finds chunks that are semantically close to the query, but "semantically close" and "correct answer" are not the same thing. A user asking "what is the refund policy?" should get the current refund policy, not the one from 2023 that was superseded 18 months ago. Both versions are equally semantically similar to the query because they contain the same vocabulary and concepts. Without a date filter or version tag, the vector search has no way to prefer the current version.

This problem multiplies in enterprise environments. A company with regional offices has different policies for different regions. A product with multiple tiers has different feature sets for each tier. A regulated industry has different compliance requirements depending on jurisdiction. The user's question might be perfectly clear, but the answer depends on structured attributes (region, tier, jurisdiction, effective date) that vector similarity cannot evaluate because they are not semantic concepts, they are metadata.

The result of ignoring metadata is retrieval that feels randomly wrong. The system returns accurate-sounding answers based on outdated documents, irrelevant departments, or wrong product versions. The user cannot tell the answer is wrong because it reads correctly, and the system cannot detect the error because the retrieved chunk was genuinely semantically relevant, just contextually wrong. Metadata filtering prevents this category of failure entirely.

Pre-Filtering vs Post-Filtering

There are two fundamental approaches to combining metadata filters with vector search, and the choice has significant implications for both accuracy and performance.

Pre-Filtering

Pre-filtering applies metadata constraints before the vector search. The vector database first narrows the candidate set to only chunks matching the filter (for example, documents from the last 12 months in the engineering department), then performs vector similarity search within that filtered subset. The result is guaranteed to satisfy both the metadata constraint and semantic relevance.

The advantage is correctness: every returned result matches the metadata criteria. The disadvantage is that aggressive pre-filtering can leave too few candidates for meaningful semantic search. If the filter narrows the candidate set from 1 million chunks to 200, the vector search is only comparing against 200 chunks, and the most semantically relevant result in the unfiltered index might not be in those 200. This is the "filter then search" precision-recall trade-off: tight filters increase precision (every result is correct by metadata) but can reduce recall (the best semantic match might be excluded).

Pre-filtering also has a performance consideration. Vector databases like Pinecone, Qdrant, and Weaviate implement pre-filtering by building filtered HNSW sub-graphs or using partition-based indexing. When the filter is selective (matches less than 1% of the index), the search can actually be faster because the traversal space is smaller. When the filter is broad (matches 50%+ of the index), the overhead of applying the filter is minimal. The performance problem occurs in the middle range (1% to 10% selectivity) where the database must traverse a filtered sub-graph that is small enough to miss good neighbors but large enough to be slow. Most vector databases handle this well in practice, but you should benchmark with realistic filter selectivity to confirm.

Post-Filtering

Post-filtering runs the vector search first without any metadata constraints, retrieves a larger candidate set (50 to 200 chunks), then filters the results by metadata criteria. The advantage is that semantic search operates on the full index, maximizing the chance of finding the best semantic match. The disadvantage is that the filter may discard most of the candidates, leaving few or no results. If 45 of the top 50 semantically similar chunks are from the wrong department, post-filtering leaves you with 5 results that may or may not be the most relevant.

Post-filtering works well when the metadata filter is broad (for example, "documents in English" in a multilingual index where 70% of documents are English). It fails when the filter is selective (for example, "documents from the Zurich office" in a global index where the Zurich office represents 2% of content). In the selective case, you would need to retrieve hundreds of candidates to get a reasonable number that pass the filter, which increases latency and cost.

Which to Use

Use pre-filtering when the metadata constraint is a hard requirement (access control, compliance, version currency) where returning a result that violates the filter is worse than returning no result at all. Use post-filtering when the metadata constraint is a preference (newer documents preferred but older ones acceptable) and the filter applies to a large fraction of the index. Most production systems default to pre-filtering because metadata constraints in enterprise RAG are usually hard requirements, not preferences.

Designing a Metadata Schema

The metadata you attach to each chunk during ingestion determines what filters you can apply at query time. A well-designed schema anticipates the filtering needs of your users without over-indexing on attributes that will never be used.

Essential fields for any RAG system:

source_document: The title or ID of the document the chunk came from. Enables filtering by document and linking back to the source for citation.

section_heading: The heading or section title that contains the chunk. Provides contextual information that helps the LLM understand where the information came from within the document.

document_date: The creation or last-modified date of the source document. Enables time-based filtering (last 6 months, current fiscal year) and helps the LLM prioritize recent information.

content_type: Whether the chunk is prose, a table, a code snippet, a FAQ entry, or an image description. Enables modality-based filtering and helps the retrieval system match the query type to the content type.

Additional fields for enterprise deployments:

department or team: The organizational unit that owns the document. Essential for multi-tenant RAG systems where different teams have different knowledge bases.

access_level: The permission level required to view the document (public, internal, confidential, restricted). This is the minimum metadata field for any system where not all users should see all content. Pre-filtering by access level is a security requirement, not an optimization.

product or version: The product name and version number that the document applies to. Prevents the system from answering questions about version 4.0 with documentation from version 3.2.

language: The language of the content. Critical for multilingual knowledge bases to prevent cross-language retrieval noise.

document_status: Whether the document is draft, published, deprecated, or archived. Prevents retrieval of draft content or superseded documents.

Store metadata as typed fields (strings, integers, dates, arrays) rather than free-text fields. Typed fields enable efficient filtering in vector databases because they can use indexes, while free-text fields require substring matching that is slower and less reliable. Most vector databases (Pinecone, Qdrant, Weaviate, Milvus) support typed metadata fields with indexed filtering out of the box.

Auto-Extracting Metadata During Ingestion

Manually tagging every document with metadata is unsustainable for large knowledge bases. Production systems automate metadata extraction during ingestion using a combination of rules and LLM classification.

Rule-based extraction handles structured metadata that is embedded in the document or its file system path. The file name, directory path, URL, and file properties (creation date, author, file type) can be extracted without any AI processing. Documents in a folder named "engineering/api-docs/v4/" automatically get department=engineering, content_type=api-docs, version=4.

Header and structure parsing extracts section headings, table captions, and document hierarchy from the parsed document. A Markdown file with "## Authentication" followed by content produces chunks with section_heading="Authentication". This is reliable for well-structured documents but fails on unstructured text.

LLM-based classification handles metadata that requires understanding the content. Pass each chunk (or a representative sample) to a fast LLM (GPT-4o mini, Claude 3.5 Haiku) with a classification prompt: "Classify this text by department (engineering, sales, legal, hr, finance), content type (policy, tutorial, reference, meeting notes), and sensitivity (public, internal, confidential)." This costs $0.01 to $0.05 per 1,000 chunks and produces metadata that would require human reviewers to assign manually.

Validate extracted metadata by sampling 100 chunks and checking that the auto-assigned metadata is correct. If accuracy is below 90% for any field, the extraction rule or classification prompt needs tuning. Incorrect metadata is worse than missing metadata because it causes invisible retrieval failures: the chunk exists in the index but the wrong metadata tag makes it unfindable by the filter.

Automatic Filter Extraction from Queries

The hardest part of metadata filtering is not the filtering itself but determining what filter to apply for each query. Users rarely specify metadata constraints explicitly ("show me the Q2 2026 engineering policy on code review"). They ask natural questions ("what is the code review process?") and expect the system to apply the right context automatically.

The standard approach is to use an LLM to extract implicit filter criteria from the query and the user's context. A system prompt instructs the model: "Given the user's query, extract any implicit metadata filters. Consider the user's department (engineering), the current date (2026-07-12), and the products they work on (Platform v4). Output a JSON object with applicable filters." The model interprets "what is the code review process?" as {department: "engineering", document_status: "published", version: "4"} based on the user's context.

This approach adds one LLM call (50 to 200 milliseconds with a fast model) but dramatically improves retrieval precision. Without it, the system searches the entire index and might return the sales team's code review process or an outdated version 3 policy. With it, the search is scoped to the right context before it starts.

Default filters based on the user's profile can reduce the need for per-query extraction. If the system knows the user's department, role, and access level, these can be applied as default pre-filters on every query, with the LLM extraction adding query-specific filters (date range, product version, specific document type) on top. This handles the 80% case (user is searching within their own domain) efficiently and uses LLM extraction only for the 20% of queries that need additional filtering.

Combining Metadata Filtering with Hybrid Search

Metadata filtering composes naturally with hybrid search. The typical production pipeline applies metadata pre-filters, then runs both vector search and keyword search within the filtered subset, fuses the results using reciprocal rank fusion, and optionally reranks the fused results. Each step narrows and refines the candidate set: metadata filtering ensures contextual correctness, hybrid search ensures both semantic and keyword relevance, and reranking ensures the most relevant result is at the top.

The order matters. Metadata filtering must happen first (as a pre-filter) because it is a hard constraint. Hybrid search happens within the filtered set. Reranking happens on the hybrid search results. Reversing the order (search first, filter later) risks discarding the best results during post-filtering and wasting compute on irrelevant chunks during reranking.