Gensim: topic modelling for humans

Abstraction: Gensim document similarity queries using LSI and cosine measure

Key points:

• Workflow: build corpus with corpora.Dictionary and doc2bow, transform to LSI space with models.LsiModel, index with similarities.MatrixSimilarity
• LSI (Latent Semantic Indexing) enables semantic retrieval: documents without shared keywords can still match on latent topics
• Cosine similarity scores range from -1 to 1; similarities.MatrixSimilarity holds entire index in RAM (requires ~2GB per million docs in 256-dim space)
• For large corpora, use similarities.Similarity instead — operates in fixed memory using on-disk shards
• Index persistence via standard .save() / .load() methods on all similarity index classes
• Gensim's stated mission is large-scale topic modelling for NLP practitioners, not an all-encompassing ML framework

Connections: Gensim · Topic Modeling · Information Retrieval · Vector Space Model

Source: https://radimrehurek.com/gensim/auto_examples/core/run_similarity_queries.html#sphx-glr-auto-examples-core-run-similarity-queries-py