Literature Retrieval
A hypothesis is cheap. Knowing whether someone already proved it is the expensive part. Build the retrieval layer that answers that question before the runner spins up a sandbox. Model a small paper record with the fields the loop will read downstream. Build a BM25 index over abstracts with stdlib data structures only. Walk a citation graph to surface papers the lexical search missed. Deduplicate hits across the lexical and graph passes by stable paper id. Wrap two mock external APIs behind a single client so the upstream call site stays the same when real endpoints land. A keyword search over abstracts returns papers that share vocabulary with the query. That covers most of the surface. It misses two cases. The first is when the foundational paper uses different vocabulary; for example a query for "sparse attention" misses a paper titled "block selection in transformer routing." The second is when the relevant paper is a follow up that cites a known anchor; it is more efficient to find the anchor and walk forward than to brute force the abstract pool. The lesson builds both passes. BM25 over abstracts catches the lexical hits. A citation graph traversal expands a seed set forward and backward by one or two hops. The union is deduplicated by paper id and ranked by a small…
Literature Retrieval: A hypothesis is cheap. Knowing whether someone already proved it is the expensive part. Build the retrieval layer that answers that…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.