How to Build a Multimodal RAG Pipeline

Step 1: Choose Your Architecture

There are three main architectural approaches, and the right choice depends on your content mix.

The unified embedding approach uses a single multimodal embedding model (Cohere Embed 4, Voyage Multimodal 3) to embed all content, whether text chunks, images, or full document pages, into the same vector space. This is the simplest architecture and works well for mixed-content knowledge bases. A single vector index holds everything, and retrieval returns the most relevant items regardless of their modality.

The VisRAG approach treats every document page as an image, bypassing text extraction entirely. Each page is rendered at sufficient resolution, embedded with a multimodal model, and stored as a visual embedding. At query time, the system retrieves the most relevant page images and passes them to a VLM. This eliminates all PDF parsing issues but increases embedding and storage costs significantly.

The dual pipeline approach maintains separate retrieval paths for text and visual content. Text chunks are embedded with a text model and stored in one index. Images and document pages are embedded with a multimodal model and stored in another. At query time, both indices are searched, and the results are merged using reciprocal rank fusion or a similar combination strategy. This adds complexity but lets you use the best embedding model for each modality.

For most teams starting out, the unified embedding approach is the right choice. It has the simplest infrastructure, a single embedding model and a single vector index, and the current generation of multimodal embedding models handles mixed content well enough for production use.

Step 2: Set Up Document Ingestion

The ingestion pipeline converts your source documents into embeddable units. For multimodal RAG, this means you need to handle both text extraction and image rendering.

For PDF documents, render each page as an image at 150 to 300 DPI. Higher DPI preserves more detail for the embedding model and VLM but increases token costs. 200 DPI is a reasonable default that balances quality and cost. Use a library like pdf2image (Python) or Poppler to render pages. Store the rendered images in object storage (S3, GCS) with metadata linking them back to the source document and page number.

For text-heavy PDFs where the visual layout is not critical, also extract the text content and chunk it normally. This gives you both the visual representation (for layout-dependent queries) and the text representation (for cheaper text-based retrieval). You can embed both and store them in the same index with metadata indicating whether each item is a text chunk or a page image.

For standalone images (photos, diagrams, screenshots), store them directly. If the images have associated captions or alt text, store those as text metadata. Some teams also generate text descriptions of images using a VLM during ingestion, storing the description alongside the image embedding. This provides a text-searchable representation of visual content at the cost of additional VLM calls during ingestion.

For web content, you can use tools like Firecrawl to scrape pages and capture both the text content and screenshots of the rendered pages. This captures visual elements like charts and embedded images that text-only scraping misses.

Step 3: Generate Multimodal Embeddings

With your content prepared, generate embeddings using your chosen multimodal model. The process is similar to text embedding but with additional considerations.

For Cohere Embed 4, you can send interleaved text and images in a single embedding request. A document page with surrounding text context (the page title, section heading, or a brief description) produces a better embedding than the image alone, because the text provides semantic anchoring. Use the input_type="search_document" parameter for indexing and input_type="search_query" for queries.

For Voyage Multimodal 3, send each image or document page as a separate embedding request. The model handles the visual content natively without needing text context, though adding context can improve retrieval quality for ambiguous images.

Batch your embedding requests to reduce API overhead. Most providers support batch sizes of 10 to 100 items per request. For large corpora (tens of thousands of pages), the embedding phase can take hours and cost hundreds of dollars, so plan for this in your budget and timeline.

Store the resulting vectors with rich metadata: source document ID, page number, modality type (text/image/page), a text snippet or description for display purposes, and the storage path to the original image file. This metadata is essential for the retrieval and generation steps.

Step 4: Index in a Vector Database

Store your multimodal embeddings in a vector database. The indexing process is identical to text-only RAG: insert vectors with metadata, and the database handles the ANN (approximate nearest neighbor) index construction.

Any major vector database works for multimodal embeddings because the vectors have the same format as text embeddings. Pinecone, Weaviate, Qdrant, Milvus, and pgvector all support the standard embedding dimensions (768, 1024, 1536) used by multimodal models.

When designing your index, consider whether you want a single collection for all modalities or separate collections. A single collection is simpler and returns mixed-modality results naturally. Separate collections let you weight modalities differently at query time (for example, boosting text results for text-heavy queries and image results for visual queries).

Add metadata filters to support scoped search. A filter on source_document lets users search within a specific document. A filter on modality lets users restrict results to images only or text only. A filter on page_number lets the system retrieve pages near a relevant result for additional context.

Step 5: Build the Retrieval Layer

The retrieval layer processes incoming queries, searches the vector index, and returns the most relevant items for generation.

Query embedding uses the same multimodal model you used for indexing, with the query mode parameter. Most queries will be text, but some systems support image queries (reverse image search within the knowledge base) or mixed queries (text question with a reference image).

Retrieve the top-k results (typically 5 to 20 items) and apply any post-retrieval filtering or reranking. For multimodal results, reranking is particularly valuable because the initial vector similarity can produce false matches between modalities. A text chunk about "bar charts" and an actual bar chart image both relate to bar charts, but one might be much more useful for the specific query. A cross-encoder reranker or LLM-based reranker can assess the actual relevance of each retrieved item in context.

Deduplicate results from the same document page. If you embedded both the text and the image of the same page, both might appear in the top results. Keep the highest-ranked version and remove the duplicate to avoid sending redundant context to the generator.

Step 6: Connect to a Vision Language Model

The generation step passes the retrieved context to a VLM for response generation. This is where multimodal RAG differs most from text-only RAG, because the prompt to the VLM includes images alongside text.

Construct the VLM prompt with the user's question, the retrieved text chunks as text context, and the retrieved images as image inputs. Most VLM APIs accept interleaved text and image content in the message body. For example, with the Claude API, you create a message with content blocks that alternate between text and image types.

Instruct the VLM to reference the provided images and text when answering. A system prompt like "Answer the user's question based on the provided documents and images. Cite specific visual elements when relevant. If the answer is not found in the provided context, say so" keeps the model grounded in the retrieved evidence.

For responses that reference visual content, consider including a reference notation so the user can identify which image or page the answer came from. This builds trust and allows users to verify the VLM's interpretation of the visual content. Something like "[Page 3, Figure 2]" alongside the relevant part of the answer works well.

Monitor the total token count of your generation prompt. Each image adds 1,000 to 4,000 tokens. If you retrieve 5 images plus text context, your prompt can easily reach 15,000 to 25,000 tokens. Factor this into your cost planning and consider limiting the number of images passed to the generator based on your quality/cost tradeoff.