Cheat sheets

NER, POS Tagging and Coreference

BIO tags, CRF and HMM taggers, Viterbi, dependency parses, coreference clusters and entity linking to a KB.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Extraction pipeline#

tokensmentionseach mentioncandidatestop-1clientDocumentmodelNERBIO token labels to spansmodelCorefmention clustersstoreCandidate generationalias index, 10-30 permentionprocessDisambiguationprior + context orembeddingsstoreKB id per entityor NILtokensmentionseach mentioncandidatestop-1clientDocumentmodelNERBIO token labelsto spansmodelCorefmention clustersstoreCandidategenerationalias index, 10-30per mentionprocessDisambiguationprior + context orembeddingsstoreKB id per entityor NIL

From Lesson 05.24, Lesson 05.25, Lesson 05.06

Core ideas#

BIO
B-X begins, I-X continues, O is outside. New B-GPE, York I-GPE.
BILOU
adds L-X (last) and U-X (unit) for cleaner boundaries.
HMM
P = \prod_i P(t_i \mid t_{i-1}) \, P(w_i \mid t_i). Transition times emission.
CRF
discriminative. Mixes word shape, case and neighbor features. Enforces valid tag sequences.
Viterbi
dynamic programming for the best tag sequence over the tag lattice.
Dependency parse
each word has one head and a relation label.
Mention types
named entity, nominal, pronominal, appositive.
Prior + context
P(entity | mention) × context-similarity(entity, text).

From Lesson 05.06, Lesson 05.07, Lesson 05.24, Lesson 05.25

Rule-based pronoun resolver#

1extract_mentionscapitalized spans, pronouns by lookup, definite"the X" phrases.2resolvefor each pronoun, take the earlier named andnominal mentions.3agreement_score+1 for number match. Gender clash costs 2.4recency_scorecloser candidates score higher.5resolvelink the candidate with the highest sum.6clustersunion-find over the links gives one cluster perentity.

From Lesson 05.24, 05.24 code/main.py

Tasks in code#

TaskCode
Spans to BIO labelsspans_to_bio(tokens, spans)
BIO labels back to spansbio_to_spans(tokens, labels)
Capitalization shapeword_shape(word)
Gazetteer baselinerule_based_ner(tokens)
Most-frequent-tag baselinepredict_mft(tokens, word_best, default_tag)
Train the bigram HMMtrain_hmm(examples, alpha=0.01)
Decode tagsviterbi(tokens, transitions, emissions, tags, vocab, alpha=0.01)
Link a mentiondisambiguate(mention, context, use_prior=True)

From 05.06 code/main.py, 05.07 code/main.py, 05.25 code/main.py

CRF and transformer NER#

crf = sklearn_crfsuite.CRF(algorithm="lbfgs",
c1=0.1, c2=0.1,L1 and L2 regularization
max_iterations=100,
all_possible_transitions=True)learns I-ORG after O is unlikely
X_train = [to_features(s) for s in sentences_tokenized]
crf.fit(X_train, bio_labels_train)~84 F1 on CoNLL-2003
ner = pipeline("ner", model="dslim/bert-base-NER",
aggregation_strategy="simple")merges B-X, I-X into spans
print(ner("Apple sued Google over its iPhone in the US."))

From Lesson 05.06

spaCy: tags, parse, entities, clusters#

nlp = spacy.load("en_core_web_sm")
doc = nlp("The cats were running at 3pm.")
for token in doc:
print(token.tag_, token.pos_, token.dep_, token.head.text)Penn, UD, relation
for ent in doc.ents:
print(f"{ent.text:20s} {ent.label_}")iPhone comes out ORG in sm
nlp = spacy.load("en_coreference_web_trf")experimental model
doc = nlp("<text>")rerun with the coref model
for cluster in doc._.coref_clusters:
print(cluster, "->", [m.text for m in cluster])

From Lesson 05.07, Lesson 05.06, Lesson 05.24

Numbers to expect#

SettingDefaultNote
Most-frequent-tag on Brown~85%The floor for any tagger
Bigram HMM on Brown~93%Gain comes from transitions
CRF with features~97%BiLSTM-CRF or transformer ~98%+
Annotator agreement, PTB97%Past 98% is likely overfitting
Tagsets36PTB has 36 tags. UD has 17
Coref SOTA, CoNLL-2012~83 F1Mean of MUC, B³, CEAF
Candidates per mention10-30From the alias index
Classical NER latency50msUnder it, classical NER wins

From Lesson 05.07, Lesson 05.24, Lesson 05.25, Lesson 05.06

Troubleshooting#

SymptomCauseFix
CoNLL model fails on contractsDomain shiftFine-tune on your domain
ORG and FACILITY overlapBIO cannot nest spansSpan-based or multi-pass NER
Long entity split in twoToken-level labelsaggregation_strategy or post-process
F1 looks too highToken-level F1Entity-level F1 with seqeval
Every mention its own clusterSingleton explosionCheck MUC, B³ and CEAF together
Coref drops on long docs~15 F1 loss over 2,000 tokensSliding window plus merge
Wrong famous entity pickedPopularity biasDomain fine-tune, check NIL
EL recall dropsNER cut the span shortFix mention boundaries upstream

From Lesson 05.06, Lesson 05.24, Lesson 05.25, Lesson 05.07

Use X when#

Classical NER or an LLM?
Classical under 50ms, at 98%+ F1, or on-prem. Else start with an LLM zero-shot baseline.
PTB or UD?
PTB for English-only legacy pipelines. UD for multilingual work.
Which parser?
spaCy for most production, stanza for multilingual, trankit for top UD accuracy.
Which EL method?
BLINK or REL for English Wikipedia. Exact-match prior for the lowest latency.
LLM coref?
Easy cases only. It over-merges and drops mentions in long docs.

From Lesson 05.06, Lesson 05.07, Lesson 05.24, Lesson 05.25

Abbreviations#

NER
named entity recognition. Label token spans with types.
CRF
Conditional Random Field. Models label transitions.
PTB / UD
Penn Treebank and Universal Dependencies tagsets.
EL
entity linking. Map a mention to one KB entry.
NIL
no KB entry matches the mention.
CoNLL F1
mean of MUC, B³ and CEAF-φ4 F1.
Cataphora
the pronoun comes before its referent.

From Lesson 05.06, Lesson 05.07, Lesson 05.24, Lesson 05.25

Tips#

  • Report mention recall and disambiguation accuracy. 99% on 80% recall is an 80% pipeline.
  • Merge coref clusters into NER entities. Emit one KB id per entity, not per mention.
  • Constrain LLM entity linking to valid KB ids.
  • Test noun-verb ambiguity (saw, book, can) on 20 sampled outputs.

From Lesson 05.25, Lesson 05.24, Lesson 05.07