Evaluation of Hierarchical Clustering Algorithms for Document Datasets

Abstract

Fast and high-quality document clustering algorithms play animportant role in providing intuitive navigation and browsing mechanisms by organizing large amounts of information into a small number of meaningful clusters. In particular, hierarchical clustering solutions provide a view of the data at different levels of granularity, making them ideal for people to visualize and interactively explore large document collections. The focus of this paper is to evaluate different hierarchical clustering algorithms and toward this goal we compared variouspartitional and agglomerative approaches. Our experimentalevaluation showed that partitional algorithms always lead tobetter clustering solutions than agglomerative algorithms, which suggests that partitional clustering algorithms are well-suited for clustering large document datasets due to not only their relatively low computational requirements, but also comparable or even better clustering performance.We also present a new class of clustering algorithms called {em constrained agglomerative algorithms} that combine the features of both partitional and agglomerative algorithms. Our experimental results showed that they consistently lead to better hierarchical solutions than agglomerative or partitional algorithms alone.