Natural Language Processing in the kitchen

Abstraction: Using NLTK Maximum Entropy classifier to parse recipe text from newspaper archives

Key points:

• LA Times Data Desk used NLTK to extract structured recipes from thousands of unstructured plain-text library archive records
• MaxEnt (Maximum Entropy) classifier chosen for its accuracy; initial bag-of-words approach achieved ~90% accuracy
• Improved to ~98% accuracy by adding part-of-speech tags, bigrams, and trigrams as features alongside individual words
• MegaM external optimizer used with NLTK to speed up MaxEnt training substantially
• Active learning loop: parse ~10-20 records by hand, build Django review app, correct errors, retrain periodically
• Only about 1 in 20 records had structural problems requiring human intervention after automated parsing

Connections: Nltk · Los Angeles Times · Natural Language Processing · Text Classification · Maximum Entropy

Source: http://datadesk.latimes.com/posts/2013/12/natural-language-processing-in-the-kitchen/