Phase 12: Multimodal AI

Document and Diagram Understanding

Type: Build

Languages: Python (stdlib, layout-aware document parser skeleton)

Prerequisites: Phase 12 · 05 (LLaVA), Phase 5 (NLP)

Time: ~180 minutes

The Problem

"Understand this PDF" is deceptively hard. The information sits in:

• Text content (90% of the signal).
• Layout (headers, footnotes, sidebars, two-column format).
• Tables (rows, columns, merged cells).
• Figures and diagrams.
• Handwritten annotations.
• Fonts and typography (title vs body).

Raw OCR dumps the text and loses the rest. A system that cares about invoices needs to know "Total: $1,245" came from the bottom-right, not from a footnote.

The Concept

Era 1 — OCR pipeline (pre-2021)

The classic stack:

1. PDF → image per page.
2. Tesseract (or commercial OCR) extracts text with per-word bounding boxes.
3. Layout analyzer identifies blocks (header, table, paragraph).
4. Table structure recognizer parses tables.
5. Domain rules + regex extract fields.

Works for clean printed text. Breaks on handwriting, skewed scans, complex tables, non-English scripts. Every failure mode requires a custom exception path.

TrOCR (2021)

TrOCR (Li et al., arXiv:2109.10282) replaced Tesseract's classic CNN-CTC with a transformer encoder-decoder trained on synthetic + real text images. Clean win on handwritten and multilingual text. Still a pipeline (detector then TrOCR then layout), but the OCR step improved dramatically.

Era 2 — OCR-free (2022-2023)

The first OCR-free models said: skip detection entirely, map image pixels to structured output directly.

Donut (Kim et al., arXiv:2111.15664):

• Encoder-decoder transformer, encoder is Swin-B.
• Output is JSON for form understanding, markdown for summarization, or any task-specific schema.
• No OCR, no layout, no detection.

Nougat (Blecher et al., arXiv:2308.13418):

• Trained specifically on scientific papers.
• Output is LaTeX / markdown.
• Handles equations, multi-column layout, figures.
• The model every arXiv-parser calls.

These are specialists, not generalists. Donut on a scientific paper fails; Nougat on an invoice fails.

LayoutLMv3 (2022)

A different track. LayoutLMv3 (Huang et al., arXiv:2204.08387) keeps OCR but adds layout understanding:

• Three input streams: OCR text tokens, per-token 2D bounding boxes, image patches.
• Masked training objective across all three modalities (masked text, masked patches, masked layout).
• Downstream: classification, entity extraction, table QA.

LayoutLMv3 is the peak of OCR-based document understanding. Strong on forms and invoices. Requires OCR upstream. Best pre-VLM accuracy on standardized document benchmarks.

DocLLM (2023)

DocLLM (Wang et al., arXiv:2401.00908) is LayoutLM's generative sibling. Generates free-form answers conditioned on layout tokens. Better for QA on documents; still depends on OCR input.

Era 3 — VLM-native (2024+)

2024 VLMs became good enough to replace the pipeline entirely. Feed the full page image at high resolution to a VLM, ask the question, get an answer.

• LLaVA-NeXT 336-tile AnyRes works for small documents.
• Qwen2.5-VL dynamic-resolution handles 2048+ pixels natively.
• Claude Opus 4.7 supports 2576px documents.
• PaliGemma 2 (April 2025) trains specifically for documents + handwriting.

The gap between VLM-native and OCR-pipeline closed rapidly. By 2026, VLM-native wins on:

• Scene text (hand-written + printed, mixed scripts).
• Complex tables with merged cells.
• Math equations embedded in text.
• Figures with text annotations.

OCR pipelines still win on:

• Pure-scan workloads at massive scale where per-page latency matters.
• Pipeline reliability (deterministic failures vs VLM hallucinations).
• Regulated environments requiring auditable OCR output.

The Claude 4.7 / GPT-5 frontier

At 2576-pixel native input, frontier VLMs do document understanding at near-human accuracy. The benchmark numbers from early 2026:

• DocVQA: Claude 4.7 ~95.1, PaliGemma 2 ~88.4, Nougat ~77.3, pipelined LayoutLMv3 ~83.
• ChartQA: Claude 4.7 ~92.2, GPT-4V ~78.
• VisualMRC: Claude 4.7 ~94.

The closed-model gap is mostly resolution and base-LLM scale. Open models at 7B are a few points behind but catching up.

Math equations and LaTeX output

Scientific papers need exact LaTeX output for equations. Nougat was trained on this. VLMs trained with LaTeX targets (Qwen2.5-VL-Math, Nougat derivatives) produce usable LaTeX. Without explicit LaTeX training, VLMs produce readable but imprecise transcriptions.

For scientific-paper pipelines in 2026: chain Nougat on the PDF, then a VLM on tricky pages.

Handwriting

Still the hardest sub-task. Mixed printed + handwritten (doctors' notes, filled forms) is where OCR pipelines still beat VLMs for cost. Handwritten-only VLMs are improving (Claude 4.7, PaliGemma 2).

2026 recipe

For a new document-AI project:

• Pure-printed invoices at scale: LayoutLMv3 + rules, cost-efficient.
• Mixed documents (scientific + handwritten + forms): VLM-native (PaliGemma 2 or Qwen2.5-VL).
• Full arXiv ingestion: Nougat for math, VLM for figures.
• Regulatory: OCR pipeline + VLM validator for cross-check.

Use It

code/main.py:

• A toy layout-aware tokenizer: given (text, bbox) pairs, produces the LayoutLMv3-style input.
• A Donut-style task schema generator: JSON template for forms.
• A comparison of token budgets per page across OCR-pipeline, Donut, Nougat, and VLM-native.

Ship It

This lesson produces outputs/skill-document-ai-stack-picker.md. Given a document-AI project (domain, scale, quality, regulatory), picks between OCR pipeline, OCR-free specialist, and VLM-native.

Exercises

1. Your project is 10M invoices per day. Which stack minimizes cost-per-page without losing accuracy?

2. Why does LayoutLMv3 outperform pure-CLIP-VLMs on form QA but underperform at scene-text? What does the bbox stream give up?

3. Nougat generates LaTeX. Propose a test case where VLM-native output beats Nougat on LaTeX fidelity, and a case where Nougat wins.

4. Read PaliGemma 2 paper (Google, 2024). What was the key training-data addition that lifted document accuracy vs PaliGemma 1?

5. Design a regulatory-safe hybrid: OCR pipeline as primary, VLM as secondary cross-check. How do you resolve disagreement?

Key Terms

Further Reading

• Li et al. — TrOCR (arXiv:2109.10282)
• Blecher et al. — Nougat (arXiv:2308.13418)
• Huang et al. — LayoutLMv3 (arXiv:2204.08387)
• Kim et al. — Donut (arXiv:2111.15664)
• Wang et al. — DocLLM (arXiv:2401.00908)

This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.

Browse the complete course catalog or open this lesson on GitHub.