Incremental Ingestion: Keeping Your AI Knowledge Base Fresh

Incremental ingestion processes only the data that has changed since the last ingestion run, rather than reprocessing the entire corpus every time. For a knowledge base with 100,000 documents where 500 change per day, incremental ingestion processes 500 documents instead of 100,000, reducing processing time by 99.5% and cutting embedding API costs proportionally. This guide covers the patterns for detecting changes, managing sync state, handling deletions, recovering from failures, and maintaining the consistency guarantees that keep your AI system working with accurate, current information.

Why Full Reprocessing Does Not Scale

A full ingestion run that reprocesses the entire corpus works fine when the corpus is small. Processing 1,000 documents takes minutes and costs a few dollars in embedding API calls. But the cost of full reprocessing grows linearly with corpus size while the amount of new information grows much more slowly. A corpus of 500,000 documents where 1,000 change daily means that full reprocessing spends 99.8% of its time and budget on unchanged content.

The costs compound across multiple dimensions. Parsing costs grow with document count, especially for layout-aware parsers that use GPU compute. Embedding costs grow with total token count, and re-embedding unchanged documents wastes money on identical vectors. Vector database write costs grow with the number of upserted records. Network transfer costs grow with the volume of data moved. Processing time grows with all of the above, extending the window during which the knowledge base contains a mix of old and new data.

Incremental ingestion solves these scaling problems by detecting which documents are new, modified, or deleted, and processing only the changes. The pipeline still performs the same operations (parse, clean, validate, embed, store) but applies them to a much smaller set of documents on each run. This makes daily or hourly ingestion practical for corpora that would take days to reprocess fully.

Change Detection with Timestamps

Timestamp-based change detection is the simplest and most common approach. Each source provides a modification timestamp for each document: the file system provides the file's last modified time, web servers provide the Last-Modified header or sitemap lastmod date, APIs return updated_at fields, and databases have timestamp columns.

The pipeline records the timestamp of the most recently modified document from each run. On the next run, it queries the source for documents modified after that timestamp. Only those documents are processed through the full pipeline. This approach is efficient because the source handles the filtering, returning only the changed documents rather than the full list.

The timestamp approach has several edge cases that must be handled. Clock skew between the source system and the ingestion pipeline can cause documents to be missed (if the source clock is behind) or re-processed (if the source clock is ahead). Using the source system's timestamp rather than the pipeline's wall clock avoids most clock skew issues. Backdated modifications (documents modified with a timestamp in the past, common in data migrations) will be invisible to timestamp-based detection. Bulk updates that modify thousands of documents with the same timestamp create a large batch that may exceed the pipeline's per-run capacity.

For file system sources, the inotify mechanism (on Linux) or FSEvents (on macOS) provides real-time notification of file changes, eliminating the need to scan the file system for modification times. This is more efficient and lower latency than timestamp scanning but requires a persistent watcher process. For web sources, HTTP conditional requests (If-Modified-Since header) let the server report whether a page has changed without transferring the full content, saving bandwidth when most pages are unchanged.

Change Detection with Content Hashing

Content hashing detects changes by comparing the content itself rather than relying on metadata timestamps. The pipeline computes a hash (SHA-256 is standard) of each document's cleaned content and compares it against the stored hash from the previous ingestion run. If the hashes match, the content is unchanged and can be skipped. If they differ, the document has changed and should be reprocessed.

Content hashing is more reliable than timestamps because it detects actual content changes regardless of metadata accuracy. A document whose file was touched (updating the modification time) but whose content is unchanged will not be reprocessed. A document whose content changed but whose timestamp was not updated (due to a bug or a direct database edit) will be detected and reprocessed.

The tradeoff is that content hashing requires reading the full document to compute the hash, which means the pipeline must fetch every document from the source on every run, even if most are unchanged. For file system sources, reading file content is fast. For web sources, it means making an HTTP request for every URL. For API sources, it means paginating through all records. The fetching overhead is the cost of reliable change detection.

A hybrid approach combines timestamps and content hashing. The pipeline first filters by timestamp to identify candidate documents that might have changed. It then computes content hashes for only the candidates and compares against stored hashes. Documents whose content hash is unchanged despite a newer timestamp are skipped (the modification was metadata-only or formatting-only, not a content change). This hybrid approach reduces both false positives (timestamp changed but content did not) and false negatives (content changed but timestamp did not) while keeping the fetching overhead manageable.

Sync State Management

The sync state is the record of what has been ingested. It maps each source document (identified by source_id) to its last-known state: the modification timestamp, the content hash, and the ingestion timestamp (when the pipeline last processed this document). The sync state enables the pipeline to answer three questions on each run: which documents are new (source_id not in the sync state), which are modified (source_id in the sync state with a different timestamp or hash), and which are deleted (source_id in the sync state but not in the current source listing).

The simplest sync state implementation is a JSON file that maps source IDs to their state records. This works for small corpora (under 50,000 documents) but becomes slow to read and write as the corpus grows. A SQLite database scales better, supporting indexed lookups by source_id and efficient range queries on timestamps. For distributed pipelines with multiple workers, a shared database (PostgreSQL, DynamoDB) provides concurrent access with proper locking.

Sync state updates must be atomic with respect to the knowledge base updates. If the pipeline processes a document, updates the knowledge base, but crashes before updating the sync state, the next run will reprocess the document (because the sync state still shows the old version). This is wasteful but safe because the processing is idempotent (re-processing produces the same result). The reverse scenario is worse: if the pipeline updates the sync state but crashes before updating the knowledge base, the next run will skip the document (because the sync state shows it was already processed) but the knowledge base does not contain it. This creates a silent gap.

The safe pattern is to update the sync state only after the knowledge base write is confirmed. This means that a crash between the knowledge base write and the sync state update causes reprocessing on the next run, but never causes a gap. For pipelines that process documents in batches, update the sync state after each batch rather than after the entire run, so that a crash midway through a long run loses at most one batch worth of progress rather than the entire run.

Handling Deletions

Documents that are deleted from the source should be removed from the knowledge base to prevent the AI from citing information that no longer exists. Detecting deletions requires comparing the current set of source documents against the set recorded in the sync state.

For sources that provide a complete listing (file system directory listing, API endpoint that returns all records, sitemap with all URLs), deletion detection is straightforward. Source IDs in the sync state that are not in the current listing have been deleted. The pipeline removes the corresponding documents from the knowledge base and removes the entries from the sync state.

For sources that do not provide a complete listing (web pages discovered through link-following, API endpoints that only return recently modified records), deletion detection is harder. The pipeline cannot distinguish between a document that was deleted and a document that simply was not returned by the current query. Options include periodic full scans (crawl all known URLs and verify they still exist), source-provided deletion events (webhooks or API endpoints that report deletions), and time-based expiration (remove documents that have not been re-confirmed after a configurable number of ingestion runs).

Soft deletion is safer than hard deletion for knowledge bases where accuracy matters. Instead of immediately removing a deleted document from the knowledge base, mark it as deleted with a timestamp. Retrieval queries exclude documents marked as deleted. After a retention period (30 to 90 days), the soft-deleted documents are permanently removed. This approach protects against false deletion: if a document disappears temporarily (website outage, API pagination bug) and reappears on the next run, the soft delete is reversed rather than losing the document's embeddings and requiring re-processing.

Failure Recovery

Incremental ingestion runs regularly, and any run can fail due to network errors, API outages, resource exhaustion, or bugs in connector code. The failure recovery strategy determines how much work is lost and how quickly the pipeline returns to normal operation after a failure.

The checkpoint pattern records progress within a run so that a failed run can be resumed from the last checkpoint rather than restarted from the beginning. For a run processing 5,000 changed documents, checkpointing after every 500 documents means a crash at document 3,700 loses only 200 documents of work (documents 3,501 through 3,700) rather than 3,700.

The implementation is straightforward: maintain a "last processed" marker that is updated after each batch. When the pipeline starts, check for an existing marker. If one exists, resume from that position. If not, start from the beginning. Store the marker in the same sync state store used for document tracking. Clear the marker when the run completes successfully.

Idempotent processing ensures that reprocessing a document after a failure does not create duplicates or inconsistencies in the knowledge base. Use upsert operations (insert or update, keyed on source_id) when writing to the knowledge base. This guarantees that processing the same document twice produces the same final state as processing it once. Without idempotent writes, a retry after a partial failure can create duplicate entries for documents that were written to the knowledge base before the crash.

Error isolation prevents one problematic document from blocking the entire run. If a document fails to parse, clean, or embed, log the error with the document's source_id and details, skip the document, and continue with the remaining documents. The failed document will be retried on the next run (because its sync state was not updated). If it fails repeatedly (the same document fails across multiple runs), flag it for manual investigation and stop retrying after a configurable limit.

Semantic Change Detection

Not all content changes are meaningful for AI applications. A web page might update its copyright year, change a navigation link, or rephrase a sentence without altering the information it conveys. Re-embedding and re-indexing these cosmetic changes wastes compute without improving the knowledge base.

Semantic change detection distinguishes meaningful changes from cosmetic ones by comparing the semantic similarity between the old and new versions of a document. After computing the content hash and determining that the document has changed, the pipeline compares the embeddings of the old and new versions. If the cosine similarity between the embeddings is above a threshold (typically 0.95 to 0.99), the change is cosmetic and the re-embedding can be skipped. If the similarity is below the threshold, the change is meaningful and the document should be re-embedded.

This optimization requires storing the embedding vector alongside the content hash in the sync state, which increases storage requirements. The tradeoff is worth it for large corpora with frequent cosmetic changes: a documentation site that regenerates every page on each build (updating timestamps, build numbers, and navigation links) would trigger full re-embedding on every crawl without semantic change detection. With it, only pages with actual content changes are re-embedded.

The threshold should be calibrated on your specific corpus. Too high (0.999) and meaningful changes are missed. Too low (0.90) and cosmetic changes trigger unnecessary re-embedding. Test with a sample of known cosmetic changes and known meaningful changes to find the threshold that separates them for your content.

Scheduling and Frequency

The right ingestion frequency depends on how quickly source data changes and how stale the knowledge base can be before users notice. The real time vs batch guide covers this decision in depth; here we cover the practical scheduling considerations.

For sources with predictable update patterns (documentation that updates weekly, databases that bulk-load nightly), schedule ingestion to run shortly after the expected update window. This ensures the knowledge base reflects the latest changes within a predictable delay. Running the pipeline more frequently than the update pattern wastes compute without improving freshness.

For sources with unpredictable update patterns (user-generated content, support tickets, web pages that change at any time), more frequent ingestion reduces the average staleness. Running hourly means the average document staleness is 30 minutes (half the interval). Running every 15 minutes reduces average staleness to 7.5 minutes. The cost of more frequent runs is proportional: twice the runs means roughly twice the compute cost, though incremental processing minimizes the cost per run when few documents have changed.

Monitor the change volume per run to validate your scheduling. If most runs process zero or very few documents, the schedule is more frequent than necessary and can be relaxed to save costs. If most runs process a large number of documents, the schedule may not be frequent enough, and users may be experiencing stale data between runs.

The pipeline building guide covers the operational aspects of scheduling, including cron configuration, Airflow DAGs, and monitoring pipeline health across runs.