Statistical Comparative Analysis and Evaluation of Validation Indices for Clustering Optimization
Clustering is a relevant exploratory tool for a broad range of machine learning applications as it aids identification of meaningful subgroups. For a given clustering algorithm, multiple partitions can be obtained on the same data set by varying algorithmic parameters. Internal validation indices provide a means to objectively evaluate how well groupings obtained from a clustering configuration partitions the data, since there is no prior labeled data. This work presents a rigorous statistical evaluation framework that analyzes performance of internal validation indices based on correlation with external indices. A synthetic data generator that captures a wide range of complexity is proposed. Evaluation is conducted on a varied set of synthetic data types and real data sets to investigate performance of the indices.
clustering, statistical analysis, validation indices
Nguyen, Thy, Jason Viehman, Dacosta Yeboah, Gayla R. Olbricht, and Tayo Obafemi-Ajayi. "Statistical Comparative Analysis and Evaluation of Validation Indices for Clustering Optimization." In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 3081-3090. IEEE, 2020.
2020 IEEE Symposium Series on Computational Intelligence, SSCI 2020