Cheat sheets

Word2Vec, GloVe and FastText

Count words, weight them with TF-IDF, train dense vectors and split rare words into subwords. Python from the lessons.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Skip-gram with negative sampling#

lookupdot productu_pos, u_negsclientone-hot(center)vocab sizestoreWcenter table,keptprocesshiddend-dim, nononlinearitystoreW'context tablemodelsigmoid scores1 positive, knegativeslookupdot productu_pos, u_negsclientone-hot(center)vocab sizestoreWcenter table, keptprocesshiddend-dim, no nonlinearitystoreW'context tablemodelsigmoid scores1 positive, k negatives

From Lesson 05.03

One training step: train_pair#

1Pairsskipgram_pairs emits every (center, context)pair inside window.2Negativesdraw k_neg random ids. Drop the center and thecontext id.3Scoressigmoid(v_c @ u_pos) should be near 1.sigmoid(u_negs @ v_c) near 0.4Errorspos_err is the positive score minus 1.0.neg_errs are the negative scores.5Update W'subtract lr times error times v_c for eachcontext row.6Update Wsubtract lr * grad_center from the center row.

From 05.03 code/main.py, Lesson 05.03

Tasks in code#

TaskCode
Count matrixbag_of_words(docs, vocab)
Unit-length TF-IDF rowsl2_normalize(tfidf(bow))
Train skip-gram from scratchtrain(docs, dim=16, window=2, k_neg=5, epochs=100, lr=0.05, seed=0)
Analogy queryanalogy(vocab, W, "man", "king", "woman")
Co-occurrence countsbuild_cooccurrence(docs, window=5)
Train GloVeglove_train(vocab, pair_counts, dim=16, epochs=100, lr=0.05, x_max=100, alpha=0.75, seed=0)
FastText n-gramschar_ngrams(word, n_min=3, n_max=6)
Learn and apply BPEapply_bpe("lowest", learn_bpe(corpus, k_merges=10))

From 05.02 code/main.py, Lesson 05.03, Lesson 05.04

Libraries in production#

tfidf = TfidfVectorizer().fit_transform(docs)IDF weights plus L2 norm
model = Word2Vec(sentences, vector_size=100, window=5,
min_count=1, sg=1, negative=5, workers=4, epochs=30)
model.wv.most_similar("cat", topn=3)nearest neighbours
fasttext.util.download_model("en", if_exists="ignore")
ft = fasttext.load_model("cc.en.300.bin")
ft.get_word_vector("zoomerapproved")OOV word still gets a vector
tok = AutoTokenizer.from_pretrained("gpt2")
tok.tokenize("unbelievably tokenized")Ġ marks a word boundary

From Lesson 05.02, Lesson 05.03, Lesson 05.04

Formulas#

TF-IDF
\frac{\mathrm{count}(w \text{ in } d)}{|d|} \cdot \log\frac{N}{\mathrm{df}(w)}
Smoothed IDF
\log\frac{n+1}{d+1} + 1. A word in every doc keeps IDF 1.
Cosine
dot product of L2-normalized rows. 1 is identical, 0 is orthogonal.
GloVe fit
v_i \cdot v_j + b_i + b_j \approx \log X_{ij}
GloVe weight
f(x) = (x / x_{max})^{\alpha} below x_max, else 1.0.
GloVe output
W + W_tilde. The sum beats one table alone.
FastText vector
the sum of the vectors of the word's character n-grams.
BPE vocabulary
k + 256 tokens after k merges.

From Lesson 05.02, Lesson 05.04

Vectorizer knobs#

CommandWhat it does
ngram_range=(1, 2)Adds bigrams. Usually helps classification
min_df=2Drops words in fewer than 2 docs
max_df=0.95Drops words in more than 95% of docs
stop_words="english"Built-in stopword list. Not for sentiment
sublinear_tf=TrueUses 1 + log(tf) for repeated terms
get_feature_names_outLists the vocabulary columns

From Lesson 05.02

Tips#

  • Under 500 labeled examples, start with a TF-IDF baseline.
  • Below about 50k labels, try TF-IDF weighted embeddings.
  • Do not judge embeddings on analogy accuracy alone. Add a downstream test.
  • Download pretrained vectors. You almost never train Word2Vec yourself.
  • GPT, BERT and T5 tokenizers learn 30k-100k merges.

From Lesson 05.02, Lesson 05.03, Lesson 05.04

Defaults and numbers#

SettingDefaultNote
gensim vector_size100Dense vectors are typically 100-300
gensim window5The from-scratch code uses 2
negative, k_neg5Negatives for each positive pair
sg1Skip-gram. Better for rare words
Init std0.1rng.normal(0, 0.1, size=(vocab_size, dim))
GloVe x_max100Pairs at or above it get weight 1.0
GloVe alpha0.75Downweights frequent pairs
FastText n-gram length3 to 6Characters, with < and > marks

From Lesson 05.03, Lesson 05.04

Troubleshooting#

SymptomCauseFix
"not good" and "excellent" look alikeBoW drops word orderAdd bigrams or use embeddings
Zoomer-approved has no vectorOOV in BoW and Word2VecUse FastText or BPE
bank has one vector for two sensesStatic embeddingUse contextual embeddings
Long docs dominate similarityRows not normalizedApply l2_normalize
Negation signal lostStopword removalKeep stopwords for sentiment
Norm below 0.01 or above 100Training bugFlag the model
Silent bad production outputTokenizer and model mismatchUse the tokenizer the model shipped with
Dense 100k-doc matrix does not fitDense conversion too earlyStay sparse

From Lesson 05.02, Lesson 05.03, Lesson 05.04

Abbreviations#

BoW
Bag of Words. Counts of vocabulary words. Order is lost.
TF / DF
term frequency in a doc, and the count of docs with the word.
IDF
inverse document frequency. Downweights words that occur everywhere.
CBOW
Continuous Bag of Words. Predict the center from its context.
Skip-gram
predict the context from the center. The default.
GloVe
Global Vectors. Factorizes the co-occurrence matrix.
OOV
Out of vocabulary. Word2Vec and GloVe fail on it.
BPE
Byte-Pair Encoding. Merge the most frequent adjacent pair, repeat.

From Lesson 05.02, Lesson 05.03, Lesson 05.04

Questions#

When does TF-IDF still win?
Spam, topics, logs, low data, tight latency, explainable output.
Why negative sampling?
Softmax over 100k words is too slow. A binary task replaces it.
Why does an analogy work?
(king - man) holds a "royal" direction. Add it to woman.
GloVe or FastText?
FastText for misspellings, new words and rich morphology.
Train a tokenizer for fine-tuning?
No. Use the tokenizer of the pretrained model.

From Lesson 05.02, Lesson 05.03, Lesson 05.04