CNNs and RNNs for Text
Convolutions learn n-grams. Recurrences remember. Both are superseded by attention. Both still matter on constrained hardware. TF-IDF and Word2Vec produced flat vectors that ignored word order. A classifier built on them could not tell dog bites man from man bites dog. Word order sometimes carries the signal. Two families of architectures filled that gap before transformers arrived. Convolutional nets for text (TextCNN). Apply 1D convolutions over sequences of word embeddings. A filter of width 3 is a learnable trigram detector: it spans three words and outputs a score. Stack different widths (2, 3, 4, 5) to detect multi-scale patterns. Max-pool to a fixed-size representation. Flat, parallel, fast. Recurrent nets (RNN, LSTM, GRU). Process tokens one at a time, maintaining a hidden state that carries information forward. Sequential, memory-bearing, flexible input lengths. Dominated sequence modeling from 2014 to 2017, then attention happened. This lesson builds both, then names the failure that motivated attention. TextCNN (Kim, 2014). Tokens get embedded. A width-k 1D convolution slides a filter over consecutive k-grams of embeddings, producing a feature map. Global max-pooling over that map picks the strongest activation. Concatenate max-pooled outputs from several filter widths. Feed to a classifier head. Why it works. A filter is a learnable n-gram. Max-pooling is position-invariant, so "not good" fires the same feature at the start or middle of…
CNNs and RNNs for Text: Convolutions learn n-grams. Recurrences remember. Both are superseded by attention. Both still matter on constrained hardware.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.