How to Clean Data Before Feeding It to an LLM

Why Cleaning Matters for AI Specifically

Data cleaning has always mattered for any data processing system, but AI applications add two specific reasons to care more about it than traditional pipelines.

First, embedding models are sensitive to noise. An embedding model converts text into a vector that represents its semantic meaning. When the text contains boilerplate ("Copyright 2024 Acme Corp. All rights reserved."), encoding artifacts ("don’t" instead of "don't"), or extraneous content (navigation menus mixed with article text), the embedding vector is pulled toward the noise rather than toward the actual content. This means a query about the document's topic will produce a weaker similarity match because the embedding represents a blend of content and noise rather than the content alone. The practical impact is measurable: cleaning text before embedding consistently improves retrieval recall@10 by 5% to 15% on real-world corpora.

Second, LLMs faithfully incorporate whatever they see in their context window. If retrieved text contains garbled characters, the LLM may reproduce them in its answer or, worse, misinterpret the garbled text as meaningful information and generate a confidently wrong answer based on it. If retrieved text contains boilerplate from every document ("For more information, visit our website"), the LLM may incorporate this repetitive noise into its response. If the same fact appears three times because of ingested duplicates, the LLM treats it as three separate sources, which may inappropriately increase its confidence in the information.

Encoding Normalization

Character encoding issues are the most common and most insidious cleaning problem. They are common because documents originate from diverse systems that use different encodings (UTF-8, Latin-1, Windows-1252, Shift-JIS). They are insidious because the resulting text looks almost correct: a single garbled character in an otherwise readable sentence is easy to miss during manual inspection but produces measurably worse embeddings and occasionally misleading LLM output.

The standard approach is to detect the encoding of each input document and convert everything to UTF-8. The chardet library (or its faster alternative, charset-normalizer) analyzes the byte patterns in the input to detect the encoding with reasonable accuracy. Once detected, the text is decoded from the detected encoding and re-encoded as UTF-8. This catches the most common case: documents encoded in Latin-1 or Windows-1252 that are interpreted as UTF-8, producing mojibake like "don’t" instead of "don't" or "résumé" instead of "resume".

After encoding normalization, apply Unicode normalization (NFC form is the standard choice) to ensure that characters with multiple representations are consistently represented. The character "e" can be encoded as a single codepoint (U+00E9, precomposed) or as two codepoints (U+0065 + U+0301, decomposed). These look identical to humans but are different bytes, which means string comparison, hashing, and embedding will treat them differently. NFC normalization ensures all such characters use the precomposed form.

A practical check: after encoding normalization, scan the output for sequences that should not appear in your target language. In English text, characters like "’" (the UTF-8 bytes of a Windows-1252 right single quote misinterpreted as UTF-8), "Â" (a padding byte), or sequences of multiple consecutive non-ASCII characters in otherwise ASCII text are almost always encoding errors. Flagging these for review catches problems that automated encoding detection misses.

Character Normalization

Beyond encoding, text contains typographic characters that create matching problems. Smart quotes (curly quotes), typographic dashes (em dash, en dash), ligatures (fi, fl), and other special characters are used by word processors and publishers but are not consistently handled by embedding models and can cause retrieval failures when a user's query uses the plain version but the document contains the typographic version.

The standard normalizations are: convert smart quotes (single and double) to straight quotes. Convert em dashes and en dashes to plain hyphens. Expand ligatures to their component characters. Convert non-breaking spaces to regular spaces. Convert ellipsis characters to three periods. Convert bullet characters to plain hyphens or asterisks. Convert fraction characters to their spelled-out equivalents or slash notation (1/2 instead of the fraction glyph).

These normalizations are deliberately lossy: they remove typographic richness in exchange for consistency. This is the right trade-off for AI applications because the semantic meaning is preserved (a straight quote and a smart quote mean the same thing) while the surface form is standardized (every quote character is the same bytes, enabling consistent embedding and matching).

Whitespace Handling

Extracted text often contains whitespace artifacts from the parsing process. PDF extraction produces extra spaces where text boxes are adjacent. HTML extraction preserves formatting whitespace that is invisible in the browser. OCR output includes spacing artifacts from character segmentation errors. These artifacts waste tokens (LLMs are charged by token count, and excessive whitespace creates unnecessary tokens), degrade embedding quality (whitespace dilutes the semantic signal), and look unprofessional in retrieved context shown to users.

The cleaning operations for whitespace are: collapse multiple consecutive spaces to a single space. Collapse multiple consecutive blank lines to a single blank line (or two blank lines as a paragraph separator). Remove leading and trailing whitespace from each line. Remove trailing whitespace at the end of the document. Convert tabs to spaces (unless tab structure is semantically meaningful, as in code). Remove zero-width characters (zero-width space, zero-width non-joiner, zero-width joiner) that are invisible but present.

Be careful with line breaks. In some document formats, each line of a paragraph ends with a line break (hard-wrapped text), and the paragraph boundary is indicated by a blank line. In others, only paragraph boundaries have line breaks. Normalizing line breaks requires understanding the source format: hard-wrapped text should have its intra-paragraph line breaks removed to produce flowing paragraphs, while paragraph-delimited text should keep its line breaks as paragraph separators.

Boilerplate Removal

Boilerplate is text that appears identically or nearly identically across many documents and carries no unique information. In PDFs, this includes headers ("CONFIDENTIAL"), footers ("Page 3 of 12"), and watermarks. In web pages, this includes navigation text that the content extractor missed, cookie consent language, and "subscribe to our newsletter" prompts. In emails, this includes signature blocks, legal disclaimers, and "this email is confidential" notices. In corporate documents, this includes standard disclaimers, copyright notices, and revision history boilerplate.

Frequency-based detection is the most reliable approach for large corpora. Process a representative sample of documents and compute the frequency of each text segment (sentences, paragraphs, or fixed-length text windows). Segments that appear in more than a threshold percentage of documents (typically 15% to 30%) are likely boilerplate. Build a set of boilerplate patterns from these frequent segments and remove matching text from all documents. This approach adapts to your specific corpus without requiring manual pattern creation.

Pattern-based detection handles known boilerplate types that may not appear frequently enough for frequency-based detection. Regular expressions can match page numbers ("Page \d+ of \d+"), copyright notices ("Copyright \d{4}.*All rights reserved"), email disclaimers ("This email and any attachments are confidential"), and document classification markers ("CONFIDENTIAL", "DRAFT", "FOR INTERNAL USE ONLY"). These patterns are specific to your document types and should be developed iteratively as you encounter new boilerplate patterns in your corpus.

The important principle is to remove boilerplate after extraction but before chunking. If boilerplate text makes it into chunks, it will be embedded and retrieved, wasting context window space when those chunks are returned as retrieval results. A chunk that is half product documentation and half repeated disclaimer text will embed as a blend of both topics, reducing the precision of retrieval for the actual content.

Deduplication

Deduplication detects and removes documents or passages that appear multiple times in the corpus. Duplicates are common when ingesting from multiple sources: the same FAQ might appear on the website, in a help center article, and in a product PDF. A press release might be published on the company's news page, sent as an email, and posted to multiple distribution sites. Without deduplication, the same information appears multiple times in the knowledge base, consuming storage and embedding budget unnecessarily and returning redundant retrieval results.

Exact deduplication uses content hashing. Compute a hash (SHA-256 is standard) of each document's cleaned text. Documents with identical hashes are exact duplicates. Keep one copy and discard the rest. This catches byte-for-byte identical content efficiently but misses near-duplicates where the same content has minor formatting differences, different headers, or small editorial changes.

Near-duplicate detection catches documents that are substantially similar but not identical. MinHash is the standard algorithm: it computes a compact signature (typically 128 to 256 hash values) from the set of word shingles (overlapping word n-grams, typically 5-grams) in each document. The Jaccard similarity between two documents can be estimated from their MinHash signatures without comparing the full text. Documents with estimated similarity above a threshold (typically 0.8 to 0.9) are near-duplicates. This scales to millions of documents because MinHash signatures are small and comparison is fast.

The deduplication decision (which copy to keep) should favor the most complete, most recent, and most authoritative version. If the same content appears on the company's website and in a cached version from a third-party site, keep the company's version. If the same document appears with and without formatting, keep the formatted version because it preserves structural information useful for chunking.

PII Redaction

Documents ingested into an AI knowledge base may contain personally identifiable information (PII) that should not be stored, embedded, or returned in generated answers. Email addresses, phone numbers, social security numbers, credit card numbers, home addresses, and medical record numbers are common PII types that appear in support tickets, customer correspondence, and internal documents.

Automated PII detection uses a combination of pattern matching (regular expressions for structured PII like phone numbers, SSNs, and credit card numbers) and named entity recognition (NER models that identify names, organizations, and locations in free text). Microsoft Presidio is an open-source PII detection and anonymization framework that combines both approaches and supports customizable detection rules for domain-specific PII types.

The redaction approach depends on the use case. Full redaction replaces PII with placeholder tokens ("[REDACTED]" or "[EMAIL]"), preventing the information from being embedded or retrieved. Pseudonymization replaces real values with realistic but fake values ("John Smith" becomes "Alex Johnson"), preserving the document's readability and semantic structure while removing real PII. Tokenization replaces values with reversible tokens that can be de-tokenized by authorized systems, enabling PII-free retrieval with PII-inclusive display when the user has access rights.

PII redaction should be applied after text extraction and encoding normalization (so that the PII patterns are in a consistent format that the detectors can match) but before deduplication (so that documents are compared based on their redacted content, preventing false duplicate rejection when the same document appears with different PII).

Cleaning Pipeline Order

The order of cleaning operations matters because some operations depend on the output of previous ones. The recommended order is:

1. Encoding normalization first, because all subsequent operations assume UTF-8 encoded text.

2. Unicode normalization (NFC), to ensure consistent character representation.

3. Character normalization (smart quotes, dashes, ligatures), to standardize typography.

4. Whitespace normalization, to collapse artifacts and standardize line breaks.

5. Boilerplate removal, to strip repeated non-content text.

6. PII redaction, to remove sensitive information.

7. Deduplication, to eliminate redundant documents.

8. Quality validation, to verify the cleaned output meets minimum standards.

Each step produces a cleaner version of the text that the next step can process more reliably. Encoding normalization before character normalization ensures that typographic characters are correctly identified rather than garbled. Boilerplate removal before deduplication prevents false negatives where documents that differ only in boilerplate are not recognized as duplicates. Quality validation last catches any issues introduced by the cleaning process itself.

The full quality validation step is covered in the dedicated guide on data quality validation. For the cleaning pipeline specifically, the validation checks should verify that the output is non-empty (catching cases where cleaning removed all content), that the output is significantly shorter than the input (confirming that cleaning actually removed something), and that the output does not contain known error patterns (encoding artifacts, boilerplate fragments, PII that escaped redaction).