SciPy Hierarchical Clustering and Dendrogram Tutorial

Abstraction: SciPy hierarchical clustering with dendrogram visualization and cut-off selection

Key points:

• scipy.cluster.hierarchy.linkage(X, 'ward') produces linkage matrix Z; Ward method minimizes intra-cluster variance; alternatives: single, complete, average with various distance metrics
• Cophenetic Correlation Coefficient (cophenet) validates clustering quality — values close to 1 indicate good preservation of original distances
• Dendrogram: horizontal lines are cluster merges; height on y-axis is the merge distance; large gaps indicate natural cluster boundaries
• Cut-off selection: manual inspection of dendrogram gaps is preferred; automated methods (inconsistency, elbow) are unreliable — inconsistency depends on depth parameter, elbow assumes monotone branch distances
• fcluster(Z, max_d, criterion='distance') retrieves cluster assignments given a max distance cut-off, or criterion='maxclust' for fixed k
• Clusters with very different within-cluster variances can mislead both inconsistency and elbow methods

Connections: Scipy · Hierarchical Clustering · Clustering · Dendrogram

Source: https://joernhees.de/blog/2015/08/26/scipy-hierarchical-clustering-and-dendrogram-tutorial/