PEGASUS: A State-of-the-Art Model for Abstractive Text Summarization

Abstraction: Google model using gap-sentence pre-training for abstractive summarization

Key points:

• Pre-training objective: mask entire important sentences (chosen by ROUGE similarity to rest of document) and train the model to reconstruct them — closely mirrors the summarization task
• Fine-tuned on 12 diverse downstream datasets (news, patents, legal, email, stories) achieving state-of-the-art results at only 5% of T5's parameter count
• With only 1000 fine-tuning examples, PEGASUS outperforms fully-supervised Transformer baselines that used orders-of-magnitude more data
• Human evaluators did not consistently prefer human-written summaries over PEGASUS outputs on XSum and CNN/DailyMail datasets
• Model demonstrates emergent counting: correctly abstracts 2–5 ships as "X Royal Navy frigates" but miscounts 6 as "seven", showing limited symbolic reasoning
• Code and model checkpoints released on GitHub to support reproducibility

Connections: Google · Google Research · Pegasus · Text Summarization · Self Supervised Learning · Transformers

Source: https://ai.googleblog.com/2020/06/pegasus-state-of-art-model-for.html?m=1