OCR and Document Understanding

Train CTC OCR, pick a document stack, retrieve pages with MaxSim and approve fields against exact source spans.

Concepts and terms
Tasks and commands
Annotated code
Diagrams
Defaults and numbers
Troubleshooting
Questions and tips

Classical OCR pipeline#

Tasks in code#

Task	Code
CTC loss	`ctc_loss(log_probs, targets, ...)`
Greedy CTC decode	`greedy_ctc_decode(log_probs, blank=0)`
Line recogniser	`TinyCRNN(vocab_size=40, hidden=128, feat=32)`
Production OCR	`PaddleOCR(lang="en").ocr(image_path)`
LayoutLMv3 input	`layoutlm_input(tokens)`
Donut JSON schema	`donut_schema("invoice")`
Top-k pages by MaxSim	`retrieve(query, pages, k=3)`

From Lesson 04.19, 04.19 code/main.py, 12.22 code/main.py, 12.23 code/main.py, 19.04 code/main.py

Train a CRNN with CTC#

vocab = ["_"] + list("0123456789abcdefghijklmnopqrstuvwxyz")blank at index 0

model = TinyCRNN(vocab_size=len(vocab))

opt = torch.optim.Adam(model.parameters(), lr=1e-3)

imgs, targets, target_lens = build_batch(strings, vocab)height 32

log_probs = model(imgs)(W', N, vocab)

input_lens = torch.full((8,), log_probs.size(0), dtype=torch.long)

loss = F.ctc_loss(log_probs, targets, input_lens, target_lens,

blank=0, reduction="mean", zero_infinity=True)

opt.zero_grad(); loss.backward(); opt.step()~3 to ~0.2 in 200 steps

From Lesson 04.19, 04.19 code/main.py

MaxSim late interaction#

def maxsim(query_tokens, patches):

s = 0.0

for q in query_tokens:one vector per query token

best = max(cosine(q, p) for p in patches)best page patch

s += bestsum over query tokens

return s

scored = [(p.doc_id, maxsim(query.tokens, p.patches)) for p in pages]

scored.sort(key=lambda x: -x[1])top-k pages

idx = build_index(prune=True)DocPruner on, 50% patches

hits = idx.retrieve(q, k=3)

From 12.23 code/main.py, 19.04 code/main.py

Commands#

Command	What it does
`python3 cli.py sample.txt --schema schema.json --output output/review.html`	Builds the HTML review desk and a JSON report
`--proposals <file>`	Loads external OCR or model candidates
`--approve <file>`	Applies decisions downloaded from the HTML desk
`python3 scripts/project_test.py document-extraction-desk --all --solution --strict`	Grades the reference solution
`python main.py`	Runs the capstone MaxSim and DocPruner scaffold
`npm start`	Runs the viewer self-check once and exits 0
`npm run serve`	Starts the viewer HTTP server on 127.0.0.1

From document-extraction-desk/README.md, solution/cli.py, 19.04 code/ts/README.md, 19.04 code/main.py

Abbreviations#

CTC: Connectionist Temporal Classification. Trains without per-timestep labels.
CRNN: conv features + BiLSTM + CTC. The 2015 baseline.
VLM-OCR: a vision-language model trained or prompted for OCR.
DocVQA: the document VQA standard. The most-cited score.
ViDoRe: Visual Document Retrieval Evaluation. Metric is nDCG@5.
PQ: product quantization. Shrinks storage ~8x.

From Lesson 04.19, Lesson 12.22, Lesson 12.23

Vision-first document QA#

From Lesson 19.04

Formulas and numbers#

CTC collapse: merge repeats, then remove blanks. "h h h _ _ e e l l _ l l o _ _" gives "hello".
CER: Levenshtein distance divided by the reference length. WER is the same per word.
MaxSim: $inline$ over query tokens $inline$ and page patches $inline$ .
ColPali storage: 50 pages x 729 patches x 128 dims x 4 bytes = ~18 MB raw, ~4 MB after PQ.
Text-RAG storage: 50 chunks x 768 dims x 4 bytes = ~150 kB. ColPali is ~30x more.
ViDoRe: ColPali-v1 scores ~80% nDCG@5. Text-RAG scores ~50-60%.

From Lesson 04.19, Lesson 12.23

Defaults and limits#

Setting	Default	Note
CER target	< 2%	On clean scans
CRNN input height	32	CNN pools height to 1. Width is time
Page render	180 DPI	PyMuPDF in the capstone
DocPruner	50%	Under 0.5% accuracy loss
Schema fields	1 to 30	Lowercase names, unique labels
Document size	100000	Characters at most
Candidates	300	Proposals at most per review

From Lesson 04.19, Lesson 19.04, solution/main.py

Pick a document stack#

Printed invoices at scale: LayoutLMv3 + rules. Cost-efficient.
Mixed, handwritten, forms: VLM-native (PaliGemma 2 or Qwen2.5-VL).
arXiv papers: Nougat for math, a VLM for figures.
Regulated: OCR pipeline + VLM validator as a cross-check.
Receipts with fixed structure: fine-tune Donut.
Pure-text documents: text-RAG. It is simpler and uses less storage.
Visually rich PDFs: ColPali for quality, VisRAG for scale.

From Lesson 12.22, Lesson 12.23, Lesson 04.19

Troubleshooting#

Symptom	Cause	Fix
"Approval belongs to another document version"	Text changed, `sourceSha256` differs	Review the new version again
"Proposal does not match its exact source span"	Quote is not at `start`, `end`	Fix the offsets of the proposal
"Use an ISO date: YYYY-MM-DD"	Date field text is not ISO	Correct the source text
Field state is ambiguous	Two Seats quotes	Choose one span with `--approve`
Donut fails on a paper	OCR-free models are specialists	Use Nougat for papers
Chart data missing in RAG	OCR drops chart data	Embed page images with ColPali

From solution/main.py, solution/cli.py, docs/en.md, Lesson 12.22, Lesson 12.23

Questions#

Why CTC?: It trains without character-level alignment.
When do OCR pipelines still win?: Massive scan scale, deterministic failures, auditable output.
Why MaxSim over a mean?: Each query token picks its best patch. Irrelevant patches do not wash it out.
Do offsets prove the value is right?: No. They prove provenance. A reviewer confirms the field.

From Lesson 04.19, Lesson 12.22, Lesson 12.23, document-extraction-desk/README.md