Process Documents with Vision Models Instead of OCR

Why Text Extraction Fails on Complex Documents

The traditional document processing pipeline has several stages: OCR (or text extraction for digital PDFs), layout analysis, table detection, structure reconstruction, and entity extraction. Each stage introduces errors that compound through the pipeline.

OCR accuracy on clean, printed text is above 99%, which sounds excellent until you realize that a single-page invoice with 500 characters will have 5 errors on average, any of which could be a digit in a total amount or a character in an account number. On lower-quality scans, handwritten text, or unusual fonts, accuracy drops further.

Layout analysis is where most pipelines struggle. A two-column layout might get merged into a single text stream. A table might lose its column alignment, turning structured data into an unreadable sequence of values. Headers and footers get mixed into the body text. Footnotes lose their connection to the text they reference. These errors are not random; they are systematic failures of trying to reconstruct 2D visual structure from 1D text output.

Table extraction deserves special attention because it is the single most common failure mode. Most business documents contain tables, and most text extraction pipelines handle tables poorly. A complex table with merged cells, nested headers, or spanning rows is extremely difficult to reconstruct from extracted text. The resulting data is often garbled, with values in the wrong columns or rows missing entirely.

Contrast this with how a human reads a document. A human sees the page as a visual object: the table is a table, the heading is visually distinct from body text, the signature block is at the bottom. A VLM processes the page the same way, using the visual structure rather than trying to reconstruct it from extracted text.

How VLM Document Processing Works

VLM-based document processing is conceptually simple: render the document page as an image, send it to a vision language model with a prompt describing what you want to extract, and parse the model's structured response.

For a typical invoice processing workflow, you render the invoice page at 200 to 300 DPI, send the image to a VLM with a prompt like "Extract the following fields from this invoice as JSON: vendor_name, invoice_number, invoice_date, due_date, line_items (array of description, quantity, unit_price, amount), subtotal, tax, total", and receive a structured JSON response. The model reads the text on the invoice, understands the table layout visually, and maps the values to the requested fields.

The accuracy of this approach on modern VLMs is impressive. Claude Sonnet and GPT-4o both extract invoice data with over 95% field-level accuracy on standard business invoices, and over 90% on complex invoices with unusual layouts. This is competitive with or better than commercial IDP platforms that cost significantly more per document.

For more complex documents like contracts, the VLM can be prompted to extract specific clauses, identify parties and dates, summarize sections, or flag unusual terms. The model understands the document structure visually and can navigate multi-page documents when pages are processed sequentially with context carried forward.

VLM vs Traditional OCR: When Each Wins

VLM-based processing is not universally better than traditional OCR. Understanding when each approach wins helps you make the right architectural choice.

VLMs win on complex layouts. Any document with tables, multi-column text, mixed content (text and images), forms with varied layouts, or non-standard formatting is better handled by a VLM. The model sees the layout and does not need to reconstruct it.

VLMs win on flexibility. When document formats vary, meaning you receive invoices from hundreds of different vendors each with their own layout, a VLM handles the variation naturally. A traditional pipeline requires template configuration or layout training for each new format.

VLMs win on development speed. Building a VLM extraction pipeline takes hours. Building a reliable traditional extraction pipeline with table handling, layout analysis, and template matching takes weeks or months.

Traditional OCR wins on pure text extraction. If your documents are text-heavy with simple layouts (like text-only PDFs, articles, or plain correspondence), text extraction with a library like PyMuPDF or pdfplumber is faster, cheaper, and equally accurate. There is no reason to run a VLM on a document that is just paragraphs of text.

Traditional OCR wins on cost at extreme scale. Processing a document page through a VLM API costs $0.01 to $0.05. Traditional OCR (Tesseract is free, cloud OCR is $0.001 to $0.01 per page) is cheaper per page. At millions of pages per month, this difference adds up. However, when you factor in the engineering cost of building and maintaining the traditional pipeline, the total cost often favors VLMs for all but the highest-volume workloads.

Traditional OCR wins for offline processing. If documents cannot leave your infrastructure and you cannot use cloud VLM APIs, traditional OCR (Tesseract, PaddleOCR) runs entirely locally. Open-source VLMs can also run locally but require GPU hardware, which adds cost and operational complexity.

Practical Implementation Patterns

Single-page extraction is the simplest pattern. Render the page, send to VLM, get structured output. Use this for invoices, receipts, forms, and single-page documents where all the relevant information is on one page. Include a JSON schema in your prompt to get consistently structured output that your application can parse programmatically.

Multi-page document processing requires a sequential approach because most VLMs cannot process dozens of pages in a single call efficiently (even if the context window is large enough, the cost scales linearly with the number of image tokens). Process each page individually, extract key information, then use a text-only LLM to synthesize the per-page extractions into a complete document analysis. For a 20-page contract, this means 20 VLM calls for page-level extraction plus one text LLM call for synthesis.

Classification-then-extraction adds a classification step before extraction. Send the first page of a document to a VLM with a prompt like "What type of document is this? Options: invoice, contract, form, letter, report, other." Based on the classification, route to the appropriate extraction prompt with field definitions specific to that document type. This handles mixed document queues where you receive different document types in the same pipeline.

Human-in-the-loop verification is important for high-stakes documents. Extract data with the VLM, present the extracted fields alongside the original document image, and let a human reviewer confirm or correct the extraction. The VLM handles the heavy lifting (reading the document and mapping fields), and the human handles the quality assurance. This typically reduces human review time by 70% to 90% compared to fully manual data entry.

Optimizing Accuracy

Several techniques improve VLM extraction accuracy beyond naive prompting.

Resolution matters. Send documents at 200 to 300 DPI. Lower resolutions lose small text details. Higher resolutions increase token costs without proportional quality gains. For documents with very small text (like fine print in contracts), 300 DPI is worth the extra cost.

Structured output prompting. Use structured output techniques to get consistent JSON responses. Define the exact schema you expect, include field descriptions, and specify data types. Models like Claude and GPT-4o support JSON mode that constrains the output to valid JSON, eliminating parsing failures.

Confidence scoring. Ask the model to include a confidence score for each extracted field. "For each field, also provide a confidence level: high, medium, or low." Fields marked as low confidence can be flagged for human review, focusing reviewer attention where it is most needed.

Cross-validation. For critical extractions (financial amounts, dates, identifiers), extract the same field twice with different prompting approaches and compare the results. If both extractions agree, confidence is high. If they disagree, flag for review. This doubles the VLM cost but significantly reduces errors on high-value fields.

Image preprocessing. For scanned documents, basic preprocessing like deskewing, contrast adjustment, and noise removal can improve extraction accuracy. These operations are fast and cheap, and they give the VLM a cleaner input to work with.

Cost Analysis

A realistic cost comparison helps justify the VLM approach to stakeholders.

For a mid-volume use case of 10,000 documents per month with an average of 3 pages each (30,000 page images):

VLM approach using Claude Sonnet: roughly 2,000 tokens per page image, plus 500 tokens per extraction prompt, plus 500 tokens per response. Total tokens: roughly 90 million. At $3/$15 per million input/output, the cost is approximately $270 for input and $225 for output, totaling about $500 per month.

Commercial IDP platform: $1 to $5 per document, totaling $10,000 to $50,000 per month. Plus setup fees, template configuration, and ongoing maintenance.

Traditional OCR pipeline (self-built): OCR costs near zero (Tesseract), but engineering time to build and maintain the pipeline, handle edge cases, update templates, and fix extraction errors easily consumes 40 to 80 engineering hours per month. At $100 to $200 per hour for a capable engineer, that is $4,000 to $16,000 per month in labor.

The VLM approach is typically the cheapest option when you include engineering maintenance costs, and it handles layout variations without template configuration.