Word Embeddings — Word2Vec from Scratch
A word is the company it keeps. Train a shallow net on that idea and geometry falls out. TF-IDF knows dog and puppy are different words. It does not know they mean nearly the same thing. A classifier trained on dog cannot generalize to a review about puppy. You can paper over this by listing synonyms, but that fails on rare terms, domain jargon, and every language you did not anticipate. You want a representation where dog and puppy land close together in space. Where king - man + woman lands near queen. Where a model trained on dog transfers some signal to puppy for free. Word2Vec gave us that space. Two layer neural network, trillion-token training runs, published in 2013. The architecture is almost embarrassingly simple. The results reshaped NLP for a decade. Distributional hypothesis (Firth, 1957): "You shall know a word by the company it keeps." If two words appear in similar contexts, they probably mean similar things. Word2Vec comes in two flavors, both exploiting that idea. Skip-gram. Given a center word, predict the surrounding words. cat -> (the, sat, on) with window size 2. CBOW (continuous bag of words). Given surrounding words, predict the center. (the, sat, on) -> cat. Skip-gram is slower to train but handles rare words better. It became the default. The network has…
Word Embeddings — Word2Vec from Scratch: A word is the company it keeps. Train a shallow net on that idea and geometry falls out.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.