Statistical significance of clustering for high-dimension, low–sample size data

Y Liu, DN Hayes, A Nobel, JS Marron - Journal of the American …, 2008 - Taylor & Francis
Journal of the American Statistical Association, 2008Taylor & Francis
Clustering methods provide a powerful tool for the exploratory analysis of high-dimension,
low–sample size (HDLSS) data sets, such as gene expression microarray data. A
fundamental statistical issue in clustering is which clusters are “really there,” as opposed to
being artifacts of the natural sampling variation. We propose SigClust as a simple and
natural approach to this fundamental statistical problem. In particular, we define a cluster as
data coming from a single Gaussian distribution and formulate the problem of assessing …
Clustering methods provide a powerful tool for the exploratory analysis of high-dimension, low–sample size (HDLSS) data sets, such as gene expression microarray data. A fundamental statistical issue in clustering is which clusters are “really there,” as opposed to being artifacts of the natural sampling variation. We propose SigClust as a simple and natural approach to this fundamental statistical problem. In particular, we define a cluster as data coming from a single Gaussian distribution and formulate the problem of assessing statistical significance of clustering as a testing procedure. This Gaussian null assumption allows direct formulation of p values that effectively quantify the significance of a given clustering. HDLSS covariance estimation for SigClust is achieved by a combination of invariance principles, together with a factor analysis model. The properties of SigClust are studied. Simulated examples, as well as an application to a real cancer microarray data set, show that the proposed method works remarkably well for assessing significance of clustering. Some theoretical results also are obtained.
Taylor & Francis Online