This article presents an automatic k-means clustering solution targeting the Sunway TaihuLight supercomputer. We first introduce a multilevel parallel partition approach that not only partitions by dataflow and centroid, but also by dimension, which unlocks the potential of the hierarchical parallelism in the heterogeneous many-core processor and the system architecture of the supercomputer. The parallel design is able to process large-scale clustering problems with up to 196,608 dimensions and over 160,000 targeting centroids, while maintaining high performance and high scalability. Furthermore, we propose an automatic hyper-parameter determination process for k-means clustering, by automatically generating and executing the clustering tasks with a set of candidate hyper-parameter, and then determining the optimal hyper-parameter using a proposed evaluation method. The proposed autoclustering solution can not only achieve high performance and scalability for problems with massive high-dimensional data, but also support clustering without sufficient prior knowledge for the number of targeted clusters, which can potentially increase the scope of k-means algorithm to new application areas.
|Number of pages||12|
|Journal||IEEE Transactions on Parallel and Distributed Systems|
|Early online date||25 Nov 2019|
|Publication status||Published - 1 May 2020|
- Many-core Processor
- Parallel Computing
Yu, T., Zhao, W., Liu, P., Janjic, V., Yan, X., Wang, S., Fu, H., Yang, G., & Thomson, J. (2020). Large-Scale Automatic K-Means Clustering for Heterogeneous Many-Core Supercomputer. IEEE Transactions on Parallel and Distributed Systems, 31(5), 997-1008. https://doi.org/10.1109/TPDS.2019.2955467