Summarization, QA, NLI and Multilingual NLP
Summarize, answer, check entailment and transfer across languages. Models, metrics, defaults and failures.
- Concepts and terms
- Tasks and commands
- Annotated code
- Diagrams
- Defaults and numbers
- Troubleshooting
- Questions and tips
Grounded QA with an NLI gate#
From Lesson 05.13, Lesson 05.21
Tasks in code#
| Task | Code |
|---|---|
| Extractive summary | textrank(text, |
| Abstractive summary | pipeline("summarization", |
| Generate the summary | summarizer(article, |
| ROUGE with stemming | rouge_scorer.RougeScorer(["rouge1", |
| Extractive QA | pipeline("question-answering", |
| Allow an empty answer | pass handle_impossible_answer=True to the pipeline call |
| SQuAD metrics | exact_match(pred, gold) and token_f1(pred, gold) |
| Zero-shot topic labels | zs(text, |
From Lesson 05.12, Lesson 05.13, 05.13 code/main.py, Lesson 05.21
SQuAD normalization#
def normalize(text): text = text.lower() text = re.sub(r"\b(a|an|the)\b", " ", text)remove articles text = re.sub(r"[^\w\s]", " ", text)remove punctuation return " ".join(text.split())exact_match("June 29th, 2007", "June 29, 2007")0.0token_f1("June 29th, 2007", "June 29, 2007")0.67token_f1("29 June 2007", "June 29, 2007")1.00, EM is 0refusal_from_score(top_retrieval_score, threshold=0.3)True means refuseFrom 05.13 code/main.py
TextRank, step by step#
Metrics#
- ROUGE-1, ROUGE-2
- unigram and bigram overlap with the reference.
- ROUGE-L
- longest common subsequence. 40 is good. 50 is exceptional.
- EM
- exact match after normalization. Else the score is 0.
- Token F1
- F_1 = \frac{2PR}{P + R} over token overlap. It gives partial credit.
- BERTScore
- contextual embedding similarity. Report it next to ROUGE.
- G-Eval
- LLM judge with chain-of-thought. Matches humans ~80% of the time.
- Entity-level F1
- entities that occur only in the summary are suspect.
- RAGAS
- faithfulness, answer relevance, context precision, context recall.
Questions#
- Extractive or abstractive?
- Extractive for news, medical, legal and financial text. Abstractive needs a factuality check.
- How does zero-shot classification work?
- Each label becomes a hypothesis. Pick the label with the highest entailment.
- Which source language for transfer?
- Try a close high-resource relative first. Hindi for Indic targets.
- Why per-language accuracy?
- The aggregate hides the long tail.
- Does RAGAS need gold answers?
- No. It scores four dimensions without references.
Tips#
- Always use stemming in ROUGE. Else "running" and "run" do not match.
- Report ROUGE-L, BERTScore and G-Eval. Calibrate on 50-100 human-labeled summaries.
- Tell the model to use only the context and to say "I don't know."
- Always check the
scorefield of the QA pipeline. - Run a 100-example labeled sanity check before you ship zero-shot classification.
- For input over the context window, use chunked map-reduce, not truncation.
From Lesson 05.12, Lesson 05.13, Lesson 05.21
NLI and faithfulness#
nli = pipeline("text-classification", model="facebook/bart-large-mnli", top_k=None)return all three labelsresult = nli({"text": premise, "text_pair": hypothesis})[0]zs = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")result = zs(text, candidate_labels=labels)labels sorted by scoredef is_faithful(answer, context, threshold=0.5): result = nli({"text": context, "text_pair": answer})[0] entail = next(s for s in result if s["label"] == "entailment") return entail["score"] > thresholdcontext supports the claimFrom Lesson 05.21
Troubleshooting#
| Symptom | Cause | Fix |
|---|---|---|
| Summary says "John Brown" | Entity swap in abstractive output | Flag entities not in the source |
| "25,000" becomes "25 million" | Number drift | Use extractive or a factuality gate |
| Span returned for an unanswerable question | Null score ignored by default | Set handle_impossible_answer=True |
| Reader looks bad | Retriever missed the passage | Measure retrieval recall in top-k |
| ~60% from the hypothesis alone | "not", "never" leak the label | Hypothesis-only baseline, ANLI |
| 20+ F1 drop on long premises | Sentence-level NLI model | Use a DocNLI-trained model |
| Zero-shot swings 10+ points | Template sensitivity | Tune hypothesis_template |
| Hindi output subtly wrong | 3-5x tokens per word | Check fertility, byte_fallback=True |
Defaults and numbers#
| Setting | Default | Note |
|---|---|---|
TextRank damping | 0.85 | PageRank default |
TextRank iterations | 50 | Stops early below epsilon 1e-4 |
BART max_length | 120 | With min_length=60 |
Refusal threshold | 0.3 | Top retrieval score below it: refuse |
is_faithful threshold | 0.5 | Entailment score must be above it |
| Few-shot examples | 100-500 | Target language. 95-98% of English |
learning_rate | 2e-5 | Higher rates collapse the alignment |
num_train_epochs | 5 | Safe for 100-500 examples |
From Lesson 05.12, 05.13 code/main.py, Lesson 05.18, Lesson 05.21
Abbreviations#
- ROUGE
- Recall-Oriented Understudy for Gisting Evaluation. N-gram and LCS overlap.
- SQuAD
- Stanford Question Answering Dataset. EM and F1 metrics.
- NLI
- Natural Language Inference. Entailment, contradiction or neutral.
- RTE
- Recognizing Textual Entailment. The older name for NLI.
- MultiNLI
- 433k pairs across 10 genres. The standard NLI training corpus.
- ANLI
- Adversarial NLI. Humans wrote pairs to break models.
- XLM-R
- 100-language RoBERTa on CommonCrawl. Base 270M, Large 550M.
- NLLB
- No Language Left Behind. 200 languages, 55 low-resource.
Pick a model#
| Command | What it does |
|---|---|
facebook/ | News summary of 3 to 5 sentences, English |
google/ | Summaries of scientific papers |
philschmid/ | Dialog summaries |
deepset/ | Answer span from a given passage |
facebook/ | Lightweight zero-shot classification |
cross-encoder/ | Fast NLI for the edge |
joeddav/ | Zero-shot classification in many languages |
facebook/ | Translation, 200 languages |