How Multimodal Embeddings Work: Shared Vector Spaces for Text, Images, and Documents

The Core Idea: One Vector Space for Everything

In a traditional text embedding system, you embed text chunks into vectors and search with text queries. The embedding model understands text, and both the indexed items and the queries are text. This works well when your knowledge base is entirely text, but it breaks down when your content includes images, diagrams, charts, screenshots, or document pages with complex layouts.

Multimodal embeddings solve this by training a model to produce vectors where semantically similar items, regardless of their modality, end up close together. A photo of a golden retriever and the text "a golden retriever dog" should have embedding vectors that are nearly identical in direction, even though one is an image and the other is text. This shared representation enables cross-modal retrieval: searching for images with text, finding text with images, or mixing modalities freely.

The training process for multimodal embedding models uses contrastive learning on paired data. The model is shown millions of image-text pairs (an image and its caption), and the training objective pushes matching pairs closer together in the vector space while pushing non-matching pairs apart. After training, the model generalizes to produce useful embeddings for images and text it has never seen before.

From CLIP to Modern Multimodal Embeddings

CLIP (Contrastive Language-Image Pre-training), released by OpenAI in 2021, was the model that proved cross-modal embeddings work at scale. Trained on 400 million image-text pairs scraped from the internet, CLIP could match images to text descriptions with surprising accuracy. It used a dual-encoder architecture: a ViT-based image encoder and a text transformer encoder, trained to produce vectors where matching image-text pairs had high cosine similarity.

CLIP was effective for its time but had significant limitations for production retrieval systems. It was designed for image-text matching, not for the fine-grained retrieval needed in RAG pipelines. It struggled with text-heavy images (like document pages), could not handle multi-page documents, and produced a single 512-dimensional vector per image regardless of the image's complexity.

SigLIP (Sigmoid Loss for Language-Image Pre-training) improved on CLIP's training methodology by replacing the softmax-based contrastive loss with a pairwise sigmoid loss. This produced better calibrated similarity scores and allowed more efficient training on larger batches. SigLIP 2, released by Google, added better multilingual support and improved localization capabilities. Many current VLMs use SigLIP 2 as their vision encoder component.

The current generation of production multimodal embedding models represents a substantial step forward from CLIP. These models are designed specifically for retrieval use cases, with better handling of document-style content, support for longer text inputs, and optimized vector representations.

Current Multimodal Embedding Models

Cohere Embed 4 is arguably the most capable multimodal embedding model available in 2026. It processes interleaved text and images in a single embedding call, meaning you can embed a document page that contains both text paragraphs and inline figures as a single coherent vector. It handles raw PDF pages without any parsing pipeline, the model sees the page as rendered image and understands the content. The 128K context window means a single embedding call can cover roughly 200 pages of a document. Cohere also supports Matryoshka dimensions (256, 512, 1024, and 1536), which lets you trim vector dimensions to reduce storage cost without regenerating embeddings.

Voyage Multimodal 3 was one of the first major single-vector multimodal models designed for retrieval rather than matching. It handles screenshots, slides, figures, and tables without requiring a text extraction pipeline. The model reports strong performance on table retrieval and screenshot search benchmarks, making it a good choice for corporate document search where presentation slides and spreadsheet screenshots are common content types.

Gemini Embedding 2 from Google provides another option for unified text-image embeddings. It benefits from Google's massive training data and integrates well with Google Cloud's Vertex AI infrastructure. For organizations already in the Google Cloud ecosystem, this is the lowest-friction choice.

Jina CLIP v2 is an open-source option that handles both text and images. While it does not match the proprietary models on complex document retrieval, it runs locally without API costs and provides acceptable quality for simpler multimodal search applications.

For most new projects, Cohere Embed 4 is the strongest default choice. If you are already using Voyage for text embeddings, Voyage Multimodal 3 provides a natural upgrade path with consistent API patterns. If you need fully local processing without API dependencies, Jina CLIP v2 or an open-source alternative is the way to go.

Single-Vector vs Multi-Vector: ColPali and Beyond

The models described above produce a single embedding vector per input, whether that input is a text passage, an image, or a document page. This single-vector approach is operationally simple: you store one vector per item, and retrieval is a standard nearest-neighbor search in your vector database.

ColPali introduced an alternative approach: multi-vector representations. Instead of collapsing an entire document page into a single vector, ColPali produces one vector per image patch (similar to how a VLM processes patches internally). At query time, the system computes a late interaction score by finding the best-matching patch vector for each query token vector and summing the maximum similarities. This is the MaxSim operation from the ColBERT retrieval model, applied to visual content.

The multi-vector approach preserves spatial and structural information that a single vector must compress away. For a complex document page with a table, a chart, and three paragraphs of text, a single 1024-dimensional vector has to somehow represent all of that content in a compressed form. Multi-vector representations maintain separate embeddings for each region, allowing the retrieval system to match specific visual elements to specific parts of the query.

The ViDoRe V2 benchmark (released May 2025) tests retrieval quality on figure-heavy, multilingual documents. ColPali-style models outperform single-vector models on this benchmark, particularly for queries that target specific visual elements within complex pages. However, the gap has narrowed as single-vector models have improved.

The tradeoff is operational complexity. Multi-vector representations require 100x to 1000x more storage than single vectors (hundreds of vectors per page versus one). Retrieval requires specialized late interaction scoring rather than standard ANN (approximate nearest neighbor) search, which limits your vector database options. And indexing cost scales with the number of patches per page, which can make large-scale ingestion expensive.

For most production systems, single-vector models provide the best balance of retrieval quality and operational simplicity. ColPali is worth the added complexity for specialized use cases: research paper search where finding specific figures matters, engineering document search where diagrams are the primary content, or compliance applications where missing a specific table cell could have legal consequences.

Practical Considerations for Multimodal Embeddings

Embedding cost is the first practical concern. Multimodal embedding models charge per image or per page, and the cost is typically 10x to 50x higher than text-only embeddings. Cohere Embed 4 prices images as token-equivalent inputs, with a typical document page consuming several thousand tokens worth of embedding cost. For a corpus of 100,000 pages, expect embedding costs of $500 to $2,000 depending on the model and image complexity. Compare this to $10 to $50 for text-only embeddings of the same content.

Embedding dimension and storage are the same as text embeddings in terms of format. Multimodal models produce vectors of standard dimensions (768, 1024, 1536), and these vectors are stored and indexed identically to text vectors. You can mix multimodal and text-only embeddings in the same vector index, as long as they come from the same model and have the same dimensionality.

When to embed as image vs text is a key design decision. Content that is primarily text (articles, documentation, plain-text emails) should still be embedded with a text embedding model because it is cheaper and often more accurate for text retrieval. Content where layout matters (invoices, forms, slides, diagrams) benefits from image-based embedding because the visual structure carries information that text extraction loses. For mixed content (web pages, reports with both text and figures), consider embedding the text portions separately and the visual portions as images, then indexing both in the same vector space.

Re-embedding when switching models is a concern because multimodal embedding models produce vectors in different spaces. If you switch from Cohere Embed 4 to Voyage Multimodal 3, you need to re-embed your entire corpus because the vector spaces are not compatible. This is the same issue as with text embedding models, but the re-embedding cost is much higher for visual content. Choose your multimodal embedding model carefully, because switching is expensive.

Building a Multimodal Vector Index

A multimodal vector index is structurally identical to a text-only vector index, just populated with vectors from different input types. Your vector database (Pinecone, Weaviate, Qdrant, pgvector, or any other) stores the embedding vectors and metadata, and retrieval returns the nearest neighbors regardless of what modality produced the vector.

The metadata you store alongside each vector should include the modality type (text, image, page), the source document identifier, a page number or chunk index for ordering, and a reference to the original content (text string, image URL, or file path). During retrieval, the search returns a ranked list of results, and your application can display text results as text, image results as images, and page results as rendered document pages.

For the generation step in a multimodal RAG pipeline, the retrieved content is passed to a VLM. If the retrieved items include images, those images are included in the VLM prompt alongside text context. The VLM can then reference the visual content in its generated response, producing answers that are grounded in both textual and visual evidence.