Analyzing Documents with TF-IDF

Abstraction: TF-IDF weighting to identify distinctively frequent document terms

Key points:

• TF-IDF originated in Karen Spärck Jones' 1972 paper "term specificity"; weights term frequency against inverse document frequency to surface distinctive — not just common — terms
• Formula (Scikit-Learn variant): idf = log((N+1)/df) + 1; tf-idf = tf × idf; normalization prevents scores below raw term count
• Applied to 366 NYT obituaries: top raw-frequency terms are stopwords; top TF-IDF terms (e.g., "bly", "nellie", "seaman") reveal subject-specific signal
• TfidfVectorizer parameters — stopwords, min_df, max_df, max_features, norm, sublinear_tf — significantly alter output; no single correct setting exists
• TF-IDF useful as (1) exploratory/visualization tool, (2) textual similarity and feature sets, (3) pre-processing for supervised ML; less suitable before topic modeling
• Alternative NLP measures include keyness (chi-square, p-value), topic models (LDA), and TextRank graph-based summarization

Connections: Scikit Learn · Programming Historian · Tf Idf · Text Mining · Information Retrieval

Source: https://programminghistorian.org/en/lessons/analyzing-documents-with-tfidf