Document and Diagram Understanding
Documents are not photos. A PDF, scientific paper, invoice, or handwritten form has layout, tables, diagrams, footnotes, headers, and semantic structure that plain image understanding cannot capture. The pre-VLM stack was a pipeline: Tesseract OCR + LayoutLMv3 + table-extraction heuristics. The VLM wave replaced that with OCR-free models — Donut (2022), Nougat (2023), DocLLM (2023) — that emit structured markup directly. By 2026 the frontier is just "feed the page image to Claude Opus 4.7 at 2576px native," and the structured-markup output comes for free. This lesson reads the three-era arc of document AI. Explain the three eras of document AI: OCR pipeline, OCR-free, VLM-native. Describe LayoutLMv3's three input streams: text, layout (bbox), image patches, with unified masking. Compare Donut (OCR-free, image → markup), Nougat (scientific paper → LaTeX), DocLLM (layout-aware generative), PaliGemma 2 (VLM-native). Pick a document model for a new task (invoices, scientific papers, handwritten forms, Chinese receipts). "Understand this PDF" is deceptively hard. The information sits in: Text content (90% of the signal). Layout (headers, footnotes, sidebars, two-column format). Tables (rows, columns, merged cells). Figures and diagrams. Handwritten annotations. Fonts and typography (title vs body). Raw OCR dumps the text and loses the rest. A system that cares about invoices needs to know "Total: $1,245" came from the bottom-right, not from a footnote. The classic stack: PDF…
Document and Diagram Understanding: Documents are not photos. A PDF, scientific paper, invoice, or handwritten form has layout, tables, diagrams, footnotes,…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.