Out-of-core classification of text documents

Abstraction: Scikit-learn online learning for text classification without loading full dataset

Key points:

• Uses partial_fit method to train on mini-batches (1000 documents at a time) without loading the full Reuters dataset into memory
• HashingVectorizer with 2^18 features maps text to a fixed-size feature space across all batches, essential when new words appear in later batches
• Classifiers supporting partial_fit: SGDClassifier, Perceptron, MultinomialNB, Passive-Aggressive (SGDClassifier with hinge loss)
• Binary classification task: "acq" category vs. all others from Reuters SGML files
• Validation set of 1000 held-out documents used to track accuracy over training; learning curve plotted as accuracy vs. number of examples and vs. runtime
• Total script runtime approximately 7.6 seconds

Connections: Scikit Learn · Out Of Core Learning · Text Classification · Online Learning

Source: http://scikit-learn.org/stable/auto_examples/applications/plot_out_of_core_classification.html#example-applications-plot-out-of-core-classification-py