Cheat sheets

Classic Text Processing and TF-IDF

Tokenize, stem, lemmatize, count, weight with TF-IDF, classify sentiment and find topics with LDA or BERTopic.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Classic text pipeline#

1Tokenizesplit the string into word tokens. Keeppunctuation as its own tokens.2Stem or lemmatizestem for speed. Lemmatize with a POS tag whenmeaning matters.3Negation scopeprefix tokens after a negation word with NOT_ upto punctuation.4Countbag of words. One row per document, one columnper vocabulary word.5WeightTF-IDF scales down common words. L2 rows makecosine a dot product.6ClassifyNaive Bayes for a baseline, logistic regressionfor production.7Evaluateper-class precision and recall, macro-F1,confusion matrix.

From Lesson 05.01, Lesson 05.02, Lesson 05.05

Tasks in code#

TaskCode
Tokenize, stem and lemmatizepreprocess(text, pos_tagger=demo_pos_tagger)
Porter step 1astem_step_1a(word)
Table lemmatizer with fallbacklemmatize(word, pos)
Count matrixbag_of_words(docs, build_vocab(docs))
TF-IDF rows on the unit spherel2_normalize(tfidf(bow))
Scope negationapply_negation(tokens)
Naive Bayes baselinetrain_nb(docs_by_class, vocab, alpha=1.0)
LDA by collapsed Gibbs samplingcollapsed_gibbs_lda(docs, n_topics=3, n_iters=300, seed=42)

From 05.01 code/main.py, 05.02 code/main.py, Lesson 05.05, 05.15 code/main.py

NLTK: tokens, stems, lemmas#

nltk.download("punkt_tab")
nltk.download("wordnet")
nltk.download("averaged_perceptron_tagger_eng")
tokens = word_tokenize(text)contractions, Unicode
stems = [PorterStemmer().stem(t) for t in tokens] # all five phases
tagged = pos_tag(tokens)Penn Treebank tags
lemmatizer = WordNetLemmatizer()
lemmas = [lemmatizer.lemmatize(t, nltk_pos_to_wordnet(tag)) for t, tag in tagged]V>v J>a R>r else n

From Lesson 05.01

Sentiment baseline in scikit-learn#

pipe = Pipeline([
("tfidf", TfidfVectorizer(
ngram_range=(1, 2),bigrams, so not_good is a feature
min_df=2,
sublinear_tf=True,dampens repeated words
stop_words=None)),keeps negations
("clf", LogisticRegression(C=1.0, max_iter=1000)),
])
pipe.fit(X_train, y_train)
print(pipe.score(X_test, y_test))75% to 85% on SST-2 with the 3 flags

From Lesson 05.05

Vectorizer knobs#

CommandWhat it does
CountVectorizer()Tokenization, vocabulary and BoW in one call
TfidfVectorizer()Adds IDF weighting and L2 normalization
ngram_range=(1, 2)Includes bigrams. Usually boosts classification
min_df=2Drops words in fewer than 2 docs
max_df=0.95Drops words in more than 95% of docs
stop_words="english"Builtin stopword list. Not for sentiment
sublinear_tf=TrueUses 1 + log(tf) instead of raw tf
get_feature_names_out()Lists the vocabulary in column order

From Lesson 05.02

Use X when#

Stem or lemmatize?
Stem for search indexing. Lemmatize for text the user reads.
NLTK or spaCy?
NLTK to teach and swap parts. spaCy for production speed.
TF-IDF or embeddings?
TF-IDF for spam, logs, low data and explainable models.
When a transformer?
Sarcasm, aspects, sentiment shifts and low-resource languages.
LDA or BERTopic?
LDA for long docs with topic mixtures. BERTopic for short text.
No GPU?
Use LDA or NMF.

From Lesson 05.01, Lesson 05.02, Lesson 05.05, Lesson 05.15

Tips#

  • Stay sparse. Dense TF-IDF for 100k documents does not fit in memory.
  • Read 5 wrong predictions per class before you trust a metric.
  • Regenerate the baseline on your own pipeline, not the paper's number.

From Lesson 05.02, Lesson 05.05, Lesson 05.15

Topic models#

cv = CountVectorizer(max_features=max_features, stop_words="english",
min_df=2, max_df=0.9)LDA needs raw counts
X = cv.fit_transform(documents)
lda = LatentDirichletAllocation(n_components=n_topics, random_state=42,
max_iter=50, learning_method="online")
doc_topic = lda.fit_transform(X)(n_docs, n_topics), rows sum to 1
topic_model = BERTopic(
embedding_model="sentence-transformers/all-MiniLM-L6-v2",
min_topic_size=15)library default is 10
topics, probs = topic_model.fit_transform(documents)
info = topic_model.get_topic_info()Topic -1 is the outlier bucket

From Lesson 05.15

Formulas#

TF-IDF
\frac{\mathrm{count}(w \text{ in } d)}{|d|} \cdot \log\frac{N}{\mathrm{df}(w)}
Smoothed IDF
\log\frac{n + 1}{d + 1} + 1. A word in every doc keeps IDF 1.
L2 row
divide each row by its norm. Long docs stop dominating.
Naive Bayes
log prior plus the sum of log P(\text{word} \mid \text{class}). Take the max.
Laplace
(\mathrm{count} + \alpha) / (\mathrm{total} + \alpha |V|). No zero probabilities.
Logistic regression
sigmoid of Xw + b. Predict 1 at 0.5 or more.
Macro-F1
mean of per-class F1, equally weighted.
c_v coherence
NPMI of top-word pairs, compared by cosine. Higher is better.

From Lesson 05.02, Lesson 05.05, Lesson 05.15

Defaults#

SettingDefaultNote
NB alpha1.0Teaching default. 0.01 is common in practice
LR l20.01Start here and tune. Sparse features overfit
LR epochs, lr500Learning rate 0.05
LR C1.0With max_iter=1000 in scikit-learn
BERTopic min_topic_size10Use 50 or 100 over 10,000 docs
Gibbs alpha, beta0.1Beta is 0.01. Both must be positive
Gibbs n_iters200The demo runs 300
Topic countmax(5, round(sqrt(n_docs)))Clamp to 200 under 40,000 docs

From Lesson 05.05, Lesson 05.15, 05.15 code/main.py

Troubleshooting#

SymptomCauseFix
Accuracy drops after deployTraining and inference preprocess differShip one preprocess function in the model
Tokens change after an upgradeLibrary version driftPin versions. Freeze 20 sample sentences
ponies stems to poniStep 1a ies rule onlyUse full Porter with step 1b
watched stays watchedLemma table has no ed ruleUse WordNet or spaCy
"not good" reads positiveStopword list drops negationsstop_words=None plus bigrams
High score, bad minority classMicro-F1 or accuracy on imbalanceReport macro-F1, AUPRC over 95-5
LDA junk topicsStopwords and frequent termsSet min_df and max_df
Many docs in topic -1HDBSCAN outlier bucketCheck min_topic_size, read the -1 docs

From Lesson 05.01, Lesson 05.02, Lesson 05.05, Lesson 05.15