Scaling Data Ingestion to Millions of Documents

A prototype ingestion pipeline that processes a few thousand documents in minutes breaks down at millions of documents. Processing time extends from minutes to days. Memory runs out as the pipeline tries to hold too many documents in flight. Embedding API calls become the dominant cost. Storage systems slow under the weight of millions of small writes. This guide covers the architecture changes, infrastructure decisions, and optimization techniques that let an ingestion pipeline scale from thousands of documents to millions while keeping costs manageable and processing times reasonable.

Where Pipelines Break at Scale

Understanding why pipelines fail at scale helps prioritize which optimizations to implement first. The bottlenecks are predictable and well understood.

Parsing is the first bottleneck. PDF parsing with layout analysis (using tools like Unstructured.io or Docling) takes 2 to 10 seconds per page. A corpus of one million documents with an average of 5 pages each means 5 million pages, which takes 115 to 578 days to process on a single core. Even simple text extraction with PyPDF2 takes 0.1 to 0.5 seconds per page, making a single threaded approach infeasible for corpora above roughly 100,000 pages.

Embedding is the second bottleneck. Embedding API calls (OpenAI, Cohere, or similar) process text at a rate limited by the API's tokens per minute quota and the network round trip time. At a typical rate of 1 million tokens per minute, embedding a corpus of 100 million tokens takes 100 minutes even under optimal conditions. Self hosted embedding models eliminate the rate limit but introduce GPU compute costs and memory management challenges.

Storage writes are the third bottleneck. Writing millions of documents to a vector database or search index involves both the data transfer and the index building. Vector databases like Pinecone, Weaviate, and Qdrant can ingest thousands of vectors per second, but at millions of vectors, index building becomes a significant operation that affects both write throughput and query latency during the ingestion window.

Memory is the fourth bottleneck. A naive pipeline that loads all documents into memory before processing runs out of RAM at scale. A pipeline that holds many partially processed documents in flight (parsed but not yet embedded, embedded but not yet stored) accumulates memory proportional to the number of in-flight documents. At scale, the pipeline must process documents in bounded batches or streaming fashion to keep memory usage constant regardless of corpus size.

Distributed Workers

The solution to the parsing bottleneck is horizontal scaling: running many workers in parallel, each processing a portion of the corpus independently. A task queue (Celery with Redis, Amazon SQS, or Google Cloud Tasks) distributes documents across workers and handles retry logic for failed tasks.

The architecture has three components. A coordinator process reads document identifiers from the source connectors and publishes them as tasks to the queue. Worker processes consume tasks from the queue, run the full ingestion pipeline (parse, clean, validate, enrich) for each document, and write results to shared storage. A monitor process tracks progress, detects stalled workers, and reports metrics.

Worker count depends on the bottleneck resource. For CPU bound parsing (layout analysis, OCR), workers should equal the number of available CPU cores. For I/O bound fetching (web scraping, API calls), workers can exceed the CPU count because most time is spent waiting for network responses. For GPU bound operations (OCR with GPU acceleration, local embedding models), workers should match the number of available GPUs.

Cloud compute makes horizontal scaling elastic. Spin up workers when a batch run starts and terminate them when it finishes. Spot instances or preemptible VMs reduce cost by 60% to 90% for batch workloads that can tolerate interruptions. The task queue handles worker interruptions naturally: when a spot instance is terminated, its in-progress task returns to the queue and another worker picks it up. For teams that need GPU compute at scale, services like Vast.ai provide access to GPU instances at significantly lower cost than the major cloud providers, which can reduce embedding and OCR costs substantially for large corpus processing.

Container orchestration (Kubernetes, ECS, Cloud Run) automates worker lifecycle management. Define a worker container image that includes all parsing dependencies (Tesseract for OCR, Playwright for JavaScript rendering, layout analysis models), deploy it as a scalable service, and let the orchestrator handle provisioning, health checks, and scaling based on queue depth.

Parallel Embedding

Embedding is the most expensive operation in the ingestion pipeline at scale because it involves either API calls with per-token pricing or GPU compute for self-hosted models. Optimizing embedding throughput and cost is essential for large corpora.

For API based embedding, batching is the primary optimization. Embedding APIs accept multiple texts per request (OpenAI accepts up to 2048 texts per batch call). Sending individual texts wastes network round trips and does not take advantage of server-side parallelism. Group texts into batches that approach the API's per-request token limit, send batches in parallel up to the rate limit, and handle partial batch failures by retrying only the failed texts.

Token budget management prevents cost surprises. Before starting a large ingestion run, estimate the total token count by sampling the corpus: measure the average token count per document on a representative sample, multiply by the total document count, and multiply by the per-token embedding price. This estimate lets you approve the cost before committing to the run. For very large corpora, consider embedding a high value subset first (documents most likely to be queried) and adding the remainder incrementally.

Self-hosted embedding models (sentence-transformers, instructor-xl, e5-large) eliminate per-token costs and rate limits at the expense of GPU infrastructure costs. A single A100 GPU can embed approximately 5,000 to 10,000 short texts per second depending on the model and text length. For corpora where the total embedding cost exceeds the cost of running a GPU instance for the processing duration, self-hosted models are more economical. The crossover point depends on your corpus size, embedding model choice, and cloud GPU pricing, but typically falls around 10 to 50 million tokens.

Incremental embedding avoids re-embedding unchanged content. Store the content hash alongside each embedding vector. When a document is re-ingested and its content hash matches the stored hash, skip the embedding step entirely. This optimization is particularly important for large corpora where most documents are unchanged between ingestion runs. The incremental ingestion guide covers the content hashing pattern.

Storage Partitioning

Writing millions of documents to a single storage target creates contention that slows both writes and reads. Partitioning distributes the data across multiple storage units, increasing write throughput and enabling parallel queries.

For vector databases, the partitioning strategy depends on the database. Managed services like Pinecone handle partitioning internally through sharding. Self-hosted databases like Qdrant and Milvus support explicit collection partitioning by a metadata field (source type, date range, department). Partitioning by date range is particularly useful for corpora where recent documents are queried more frequently than old ones, because the retrieval system can search only the recent partition for time-sensitive queries and fall back to the full corpus when needed.

For file based intermediate storage (JSON documents between pipeline stages), partition by document batch or source connector. Write each batch to a separate directory or object storage prefix. This prevents directory listings from becoming slow (a directory with millions of files has O(n) listing performance in most file systems) and enables parallel reading by downstream stages that can process partitions independently.

For relational databases used for sync state and metadata, index the fields used for incremental sync queries (source_id, timestamp, content_hash). Without these indexes, the queries that identify new and modified documents degrade from milliseconds to minutes as the sync state table grows. A sync state table with 10 million rows and no index on timestamp will full-scan on every incremental sync query, creating an artificial bottleneck that makes the pipeline appear slow when the real problem is a missing index.

Checkpointing and Recovery

A pipeline that processes millions of documents will inevitably fail partway through. Network errors, API outages, out-of-memory crashes, and infrastructure failures are statistical certainties at scale. Without checkpointing, a failure after processing 900,000 of 1,000,000 documents means starting over from document one.

Checkpointing records progress so that the pipeline can resume from the last successful point rather than restarting. The simplest checkpoint is the task queue itself: tasks that complete successfully are acknowledged and removed from the queue, tasks that fail are returned to the queue for retry, and tasks that were never started remain in the queue. When the pipeline restarts after a failure, it consumes tasks from the queue and automatically skips the work that was already completed.

For pipelines that do not use a task queue, explicit checkpointing writes a progress marker (last successfully processed document ID, current page number, batch number) to persistent storage at regular intervals. After a failure, the pipeline reads the checkpoint and skips documents that were processed before the checkpoint. The checkpoint interval determines the maximum wasted work: a checkpoint every 1,000 documents means at most 999 documents need to be reprocessed after a failure.

Idempotent processing ensures that reprocessing a document (due to retry or checkpoint overlap) does not create duplicate entries in the knowledge base. Use upsert operations (insert or update) keyed on the document's source_id when writing to the vector database. This guarantees that processing the same document twice produces the same result as processing it once, making retries and restarts safe.

Cost Optimization

At scale, the costs of parsing, embedding, and storage become significant budget items. Several optimization strategies reduce costs without reducing quality.

Deduplication before embedding prevents paying to embed the same content multiple times. At the document level, content hashing catches exact duplicates cheaply. At the near-duplicate level, MinHash signatures (as described in the cleaning guide) catch documents that differ only in boilerplate, headers, or formatting. Deduplicating a corpus of 1 million documents typically eliminates 5% to 30% of content, with corresponding savings on embedding costs.

Tiered parsing applies expensive parsing only where necessary. Simple text extraction (PyPDF2) costs essentially nothing in compute. Layout-aware parsing (Unstructured.io with models) costs 10x to 100x more per page. Route documents to the appropriate parser based on their complexity: single column text documents use simple extraction, complex layouts use layout-aware parsing, scanned documents use OCR. This routing requires either metadata about the document type (available from the source connector) or a quick complexity check (attempt simple extraction first, fall back to layout-aware parsing if the output fails quality validation).

Embedding model selection affects both cost and quality. Larger embedding models (3072 dimensions) produce slightly more accurate embeddings but cost more to compute, store, and search. Smaller models (384 to 768 dimensions) are often sufficient for practical retrieval quality and cost significantly less at every stage. Benchmark your retrieval accuracy with different embedding dimensions on a representative sample before committing to the largest model for a million document corpus.

Storage tiering keeps frequently accessed data on fast, expensive storage and moves infrequently accessed data to slower, cheaper storage. Recent documents that are queried frequently stay in the primary vector index. Older documents that are rarely queried can be moved to a secondary index with higher latency but lower cost. The real time vs batch guide discusses freshness requirements that inform which documents should be in the fast tier.

Monitoring at Scale

A pipeline processing millions of documents needs operational monitoring that goes beyond simple success/failure logging. At scale, partial failures, performance degradation, and cost overruns are the problems that need attention, not total outages.

Track throughput (documents per second, pages per second, tokens per second) across each pipeline stage. When throughput drops below expected levels, the monitoring system should identify which stage is the bottleneck: is parsing slower than usual (perhaps the current batch contains unusually complex documents), is the embedding API throttling requests (rate limit exceeded), or is the vector database falling behind on writes (index building contention)?

Track error rates per source connector, per document type, and per pipeline stage. A spike in parsing errors for PDF documents might indicate a new PDF variant that the parser does not handle. A spike in embedding API errors might indicate a service degradation on the provider's side. A spike in storage write errors might indicate capacity limits being reached.

Track costs in real time: embedding API spend, compute costs for workers, storage costs for the knowledge base. Set budget alerts that pause the pipeline if costs exceed expected levels, preventing runaway spending from a bug that causes infinite retries or unnecessary re-embedding. Cost monitoring is especially important during the initial large scale ingestion when the full corpus is processed for the first time and actual costs may diverge from estimates.

The quality validation guide covers the quality specific metrics (rejection rates, quality score distributions) that should be monitored alongside the operational metrics described here.