Statistical Natural Language Processing in Python

Abstraction: Peter Norvig's Python NLP notebook covering probabilistic language models and tasks

Key points:

• Covers bag-of-words model, unigram and bigram language models using Python Counter; 1M-word corpus from Project Gutenberg; Zipf's Law demonstrated on log-log word frequency plot
• Spelling corrector uses edit distance (0, 1, 2 edits: deletions, transpositions, replacements, insertions) with ~24K candidates at edit distance 2; selects correction maximizing unigram count
• Word segmentation via dynamic programming with memoization: splits unseparated text by maximizing Pwords over all possible first-word / rest splits; segment2 uses bigram model and outperforms unigram version
• Smoothing techniques covered: Laplace additive smoothing, Good-Turing smoothing (re-estimates zero-count probabilities from one-count word length distribution), and interpolation smoothing
• Kneser-Ney smoothing concept introduced: words appearing in more diverse bigram contexts get higher probability as second word in unseen bigrams
• Caesar/rotation cipher decoding via exhaustive search over 26 candidates, combined with word segmentation to handle space-stripped ciphertext

Connections: Peter Norvig · Natural Language Processing · Statistical Language Models · Spelling Correction

Source: http://nbviewer.ipython.org/url/norvig.com/ipython/How%20to%20Do%20Things%20with%20Words.ipynb