Multimodal and Vision-Native RAG

Embed page images, score with MaxSim, fuse modalities, cite evidence regions. Python and TypeScript from the lessons.

Run the lessons and the project#

Command	What it does
`python main.py`	Capstone MaxSim scaffold plus a pruning ablation
`npm start`	Viewer self-check pass with `--demo`. Exits 0
`npm run serve`	HTTP viewer on 127.0.0.1. Free port when `PORT` is unset
`npm test`	Fixture, HTML escape and route tests
`python3 scripts/project_test.py visual-evidence-library --init my-visual-evidence-library`	Creates the incomplete workspace
`python3 scripts/project_test.py visual-evidence-library --stage 1 --path my-visual-evidence-library`	Grades stage 1
`python3 main.py --manifest ./fixtures/manifest.json --query "return dry seeds" --out ./visual-output`	Writes index.html and evidence.json
`python3 demo.py`	Runs the search on the original fixtures

def max_sim_score(query_tokens, doc_patches):

total = 0.0

for q in query_tokens:one vector per query token

best = -1e9

for p in doc_patches:every patch on the page

s = dot(q, p)

if s > best:

best = s

total += bestsum of the best match per token

return total

idx = build_index(prune=True)DocPruner keeps 50% of patches

hits = idx.retrieve(q, k=3)list of (Page, score)

Factor	ColPali	VisRAG	Text-RAG
Page vectors	$inline$ patch vectors	One pooled vector	One vector per chunk
Scoring	MaxSim	Bi-encoder	Cosine on chunks
OCR step	None	None	OCR / pdftotext
Storage	~30x text-RAG	Smaller	~150 kB per 50 chunks
Recall	Strong	Weaker	Loses charts and tables
Use for	Quality	Scale	Pure-text documents

Symptom	Cause	Fix
Asset IDs must be nonempty and unique	Empty or repeated `id`	Give each asset a unique id
Image dimensions must be positive integers at most 20000	Bad `width` or `height`	Declare integer pixel sizes
Asset must be an image inside the manifest directory	Path escapes root or wrong suffix	Use .svg, .png, .jpg or .jpeg under root
Asset exceeds 5 MB	Large file	Shrink the image
Bounding box lies outside the declared image dimensions	x + w or y + h too large	Fix the `bbox`
Model-proposed text needs explicit review before indexing	`reviewed` is not true	Check the image, then set reviewed=true
Vision extraction requires --provider-url, --width and --height	Missing flags	Pass all three
Image bytes do not match the extension	PNG or JPEG magic bytes differ	Use the correct extension

Task	Code
Score one page	`maxsim(query_tokens, patches)`
Rank toy pages	`retrieve(query, pages, k=3)`
Prune patches	`doc_prune(patches, keep_fraction=0.5)`
Index with pruning	`build_index(prune=True)`
Fuse three modalities	`fuse([t, i, a], [0.3, 0.4, 0.3])`
Retry on low confidence	`agentic_loop(query, confidence_floor=0.8)`
Check a manifest	`validate_manifest(manifest, root)`
Search text regions	`search(assets, query, limit=20)`

Use ColPali or ColQwen2 for financial reports, papers, contracts and medical records.
Use VisRAG when indexing speed and storage matter more than recall.
Use text-RAG for wiki articles, chat logs and multi-million-page archives.
Use text-RAG when regulation demands extractable OCR text.
Use CLIP or SigLIP 2 for text and image. Use CLAP for text and audio.
Use score fusion first. Upgrade to MoE fusion when A/B shows clear wins.

Setting	Default	Note
`EMB_DIM`	16	Fake patch vectors in the capstone
`keep_fraction`	0.5	`doc_prune` keeps at least 1 patch
Retrieve k	5	Top pages by MaxSim
`confidence_floor`	0.8	`main` passes 0.7
`search` limit	20	Must be between 1 and 100
`--query`	seed return	Evidence library CLI
`--model`	local-vision-model	Used with `--vision-image`
`--out`	visual-output	Holds index.html and evidence.json

Why MaxSim, not mean similarity?: Irrelevant patches wash out the mean.
How do you cite mixed sources?: Text [1], image [img 3], audio [audio 2 at 0:34].
How do you grade cross-modal RAG?: Recall@k per modality and fused top-k accuracy.
Does the evidence library do OCR?: No. It indexes supplied text rectangles.
What is query coverage?: Matched terms divided by query terms. It is not confidence.