Text Processing — Tokenization, Stemming, Lemmatization
Language is continuous. Models are discrete. Preprocessing is the bridge. A model cannot read "The cats were running." It reads integers. Every NLP system opens with the same three questions. Where does a word start. What is the root of the word. How do we treat "run", "running", "ran" as the same thing when it helps, and as different things when it doesn't. Get tokenization wrong and the model learns from garbage. If your tokenizer treats don't as one token but do n't as two, the training distribution splits. If your stemmer collapses organization and organ to the same stem, topic modeling dies. If your lemmatizer needs part-of-speech context but you don't pass it, verbs get treated as nouns. This lesson builds the three preprocessing steps from scratch, then shows how NLTK and spaCy do the same work so you can see the tradeoffs. Three operations. Each has a job and a failure mode. Tokenization splits a string into tokens. "Token" is deliberately vague because the right granularity depends on the task. Word-level for classical NLP. Subword for transformers. Character for languages without whitespace. Stemming chops suffixes with rules. Fast, aggressive, dumb. running -> run. organization -> organ. That second one is the failure mode. Lemmatization reduces a word to its dictionary form using grammar knowledge. Slower, accurate, needs a…
Text Processing — Tokenization, Stemming, Lemmatization: Language is continuous. Models are discrete. Preprocessing is the bridge.
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.