How to Build a Multimodal RAG Pipeline

Why Text-Only RAG Leaves Knowledge on the Table

Most production RAG systems treat documents as pure text. PDFs get parsed into strings, images get skipped, and tables get flattened into pipe-delimited rows that lose their visual structure. This works when your knowledge base is articles, documentation, and plain-text notes. It fails when the answer to a user's question lives in a bar chart, a workflow diagram, a photograph of a product label, or a table where the spatial relationships between cells matter more than the text within them.

The scale of the problem is significant. Deloitte estimates that 80% of enterprise data is unstructured, and within that unstructured data, a large portion is visual: presentations, scanned forms, engineering drawings, medical imaging reports, and dashboards. A RAG system that only searches text is effectively blind to most of the organization's knowledge. The user asks "what were Q2 revenue trends by region?" and the system retrieves paragraphs that mention revenue but misses the chart on slide 14 that directly answers the question.

Multimodal RAG solves this by treating images, tables, and other visual content as first-class retrievable objects alongside text. The system can retrieve a chart and pass it to a vision-capable LLM (GPT-4o, Claude 3.5, Gemini 1.5) that can interpret the visual content and synthesize it with text-based context to produce a complete answer. The engineering challenge is making non-text content searchable and assembling it into prompts that the LLM can process.

Three Embedding Strategies for Multimodal Content

The core architectural decision in multimodal RAG is how to embed non-text content so it can be retrieved alongside text using a unified search. There are three approaches, each with distinct trade-offs.

Strategy 1: Unified Multimodal Embeddings

Models like CLIP (OpenAI), SigLIP (Google), and Nomic's nomic-embed-vision produce embeddings where text and images occupy the same vector space. A text query like "revenue by region chart" produces a vector that is geometrically close to the vector of an image containing a revenue-by-region chart, because the model was trained on image-text pairs and learned to align their representations. This means you can store text chunks and images in the same vector index and search them with a single query embedding.

The advantage is simplicity: one index, one search, one ranking. The disadvantage is that current multimodal embedding models have lower retrieval precision than text-only models for text-to-text matching. CLIP was trained on short image captions, not long document paragraphs, so it understands broad semantic concepts ("a chart showing revenue") but struggles with fine-grained text retrieval ("the specific clause about indemnification in section 4.2"). For knowledge bases that are 90% text, using a multimodal embedding model for everything degrades text retrieval quality to gain image retrieval capability.

Strategy 2: Separate Embeddings with Late Fusion

This approach uses the best embedding model for each modality: a text embedding model (like text-embedding-3-large or Cohere embed-v4) for text chunks, and a vision embedding model (like CLIP or SigLIP) for images and diagrams. Each modality gets its own vector index. At query time, the query is embedded with both models, searched against both indexes in parallel, and the results are merged using reciprocal rank fusion or a learned fusion model.

The advantage is that each modality gets optimal retrieval quality because the embedding model is specialized. The disadvantage is operational complexity: two indexes, two embedding pipelines, and a fusion step that needs tuning. The fusion weights matter, because a naive 50/50 merge will over-retrieve images for text-heavy queries and over-retrieve text for visual queries. Production systems typically weight text results higher by default (70/30 or 80/20) and adjust dynamically based on query classification.

Strategy 3: Text-Description Bridging

The most practical approach for teams starting with multimodal RAG is to convert all non-text content into text descriptions and embed those descriptions with a standard text embedding model. Images get captioned by a vision LLM ("This is a bar chart showing quarterly revenue for North America, Europe, and Asia Pacific from Q1 2024 through Q2 2026. North America leads at $4.2M in Q2 2026..."). Tables get converted to natural language summaries. Diagrams get described step by step.

The advantage is that the entire pipeline remains text-based, so you keep the full retrieval quality of your existing text embedding model and do not need a separate index or fusion logic. The disadvantage is that the text description is a lossy representation of the visual content. A caption describes what the chart shows but cannot capture every data point, visual pattern, or spatial relationship. When the system retrieves a description and passes it to the LLM, the LLM is working from the description, not the original image, so subtle visual details may be lost.

The hybrid solution that works best in practice: use text-description bridging for retrieval (so the search step finds the right images) but pass the original image to the vision LLM for generation (so the model can see the actual visual content, not just the description). This gives you the retrieval quality of text embeddings and the generation quality of direct visual understanding.

Step 1: Build the Multimodal Ingestion Pipeline

The ingestion pipeline for multimodal RAG extends the standard text pipeline with modality-specific extraction and processing for each content type.

Images and Diagrams

Extract images from source documents using a layout-aware parser like Unstructured.io, Docling, or PyMuPDF (for PDFs). Each extracted image needs metadata: the page number it came from, the surrounding text (captions, headings, paragraphs that reference it), and the source document. Generate a text description using a vision LLM (GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Flash for cost efficiency). The prompt should instruct the model to describe the image in detail, including all text visible in the image, all data values in charts, and the relationships depicted in diagrams. Store the original image (in S3, GCS, or your blob store), the text description (for embedding and retrieval), and the metadata together as one retrievable unit.

Tables

Tables are the content type that text-only RAG handles worst and multimodal RAG handles best. Extract tables from PDFs using Camelot, Tabula, or Amazon Textract (which handles complex multi-header tables that rule-based extractors miss). Convert each table to both a structured format (JSON or CSV with column headers preserved) and a natural language summary that describes what the table contains, the key values, and any trends. Embed the natural language summary for retrieval. When the table is retrieved, include both the structured format and the original table image in the LLM prompt so the model can reference exact values.

Audio and Video

Audio content (podcasts, meeting recordings, customer calls) gets transcribed using Whisper, Deepgram, or AssemblyAI. The transcript becomes text that follows the standard text chunking pipeline. For video, extract key frames at scene changes or at fixed intervals (every 30 to 60 seconds), transcribe the audio track, and create paired text-image chunks where each chunk contains a key frame image and the corresponding transcript segment. This allows retrieval based on either spoken content or visual content.

Scanned Documents and OCR

Scanned documents, faxes, and photographed text require OCR before any processing. Tesseract handles simple layouts, but production systems use Amazon Textract, Google Document AI, or Azure AI Document Intelligence for complex layouts with forms, tables, handwriting, and multi-column text. The OCR output should preserve layout structure (reading order, table boundaries, form field labels) rather than producing a flat text stream. Post-OCR, the text follows the standard chunking pipeline, but you should also store the original page image so it can be passed to the LLM when high-fidelity reading is needed.

Step 2: Configure Retrieval and Prompt Assembly

Multimodal retrieval introduces a prompt assembly challenge that text-only RAG does not have: the LLM prompt must include both text and images, and the model must understand which images correspond to which parts of the text context.

For vision-capable LLMs (GPT-4o, Claude 3.5, Gemini 1.5), images are passed as base64-encoded data or as URLs within the message content array. Each image should be accompanied by its metadata (source document, page number, caption) so the model can reference it by source when generating the answer. The prompt structure for a multimodal RAG response looks like: system instruction (answer from provided context, cite sources, decline when context is insufficient), text chunks with source labels, images with source labels and descriptions, then the user's question.

Token budget management becomes more important with multimodal prompts. Images consume tokens based on resolution: GPT-4o charges 85 tokens for a low-resolution image and up to 1,105 tokens for a high-resolution image. Claude charges based on image dimensions. A prompt with 5 retrieved text chunks and 3 images can easily reach 5,000 to 8,000 tokens of input context. Resize images to the minimum resolution that preserves readability (typically 768x768 or 1024x1024 for charts and diagrams) to control costs without losing interpretability.

When using the text-description bridging approach for retrieval, the prompt assembly step should swap the text description for the original image. The retrieval used the description to find the right image, but the generation should use the actual image so the LLM can see details that the description may have missed. This swap is a simple lookup: the description record in your database includes a reference to the original image file, and the prompt builder fetches and includes it.

Step 3: Evaluate Multimodal Retrieval Quality

Evaluating multimodal RAG requires evaluation sets that specifically test cross-modal retrieval. A text-only evaluation set will not reveal whether the system can find the right chart, table, or diagram when the answer lives in visual content.

Build evaluation queries in three categories. First, text-answerable queries where the answer exists in text chunks (these test that multimodal indexing has not degraded text retrieval). Second, image-answerable queries where the answer requires interpreting a specific chart, diagram, or image ("What was the failure rate trend in Q3?" where the answer is in a line chart). Third, cross-modal queries where the answer requires combining text and visual context ("How does the architecture diagram relate to the performance numbers in section 4?").

Measure retrieval recall separately for each category. If text recall drops significantly compared to a text-only baseline, your multimodal embedding or fusion strategy is hurting text retrieval and needs adjustment. If image recall is below 60% on image-answerable queries, the image descriptions or multimodal embeddings are not capturing the visual content well enough. Target recall@5 above 80% for text queries and above 70% for image queries as a reasonable starting benchmark.

For generation quality, evaluate whether the LLM correctly interprets visual content in its answers. This requires human review because automated metrics cannot reliably judge whether a chart was read correctly. Sample 20 to 30 image-dependent answers and verify that the model extracted the right values, identified the right trends, and did not hallucinate data points that do not appear in the image.

Production Considerations

Image storage and serving adds infrastructure cost. Each extracted image needs to be stored, versioned, and served with low latency at generation time. Using a CDN-backed object store (S3 with CloudFront, GCS with Cloud CDN) keeps image serving fast without overloading your application servers. Budget 2 to 10 MB per document page for images at usable resolution, which adds up quickly for large knowledge bases.

The vision LLM captioning step during ingestion is the most expensive part of multimodal ingestion. GPT-4o charges approximately $0.0075 per image at high resolution, and Gemini 1.5 Flash is roughly $0.0003 per image. For a knowledge base with 50,000 images, GPT-4o captioning costs $375, while Gemini 1.5 Flash costs $15. Use the cheaper model for initial captioning and reserve the expensive model for images where the cheap model's description fails quality checks.

Latency increases because multimodal prompts are larger and image encoding adds processing time. Expect multimodal RAG queries to take 2 to 4 seconds end-to-end compared to 1 to 2 seconds for text-only RAG, primarily due to the larger prompt and vision model processing. Caching frequently retrieved images in memory (rather than fetching from object storage on every request) and pre-encoding images to base64 during ingestion rather than at query time reduce the latency gap.