Incremental PDDP for the Clustering of Large Data Sets

Abstract

Principal Direction Divisive Partitioning (PDDP) is an unsupervised method for partitioning data into clusters. The original method was designed to be applied to the entire data set at once, and for good performance required the entire data set be present in core memory. This paper introduces a variant of PDDP which allows a PDDP tree representing the entire data set to be built in sections. This permits the construction of PDDP trees on large data sets even with limited memory. The performance of the resulting Incremental PDDP tree is comparable to a basic PDDP tree.