# Finding a Better k: A psychophysical investigation of clustering

- Joshua M. Lewis,
*University of California, San Diego*

## Abstract

Finding the number of groups in a data set, k, is an important
problem in the field of unsupervised machine learning with applications across
many scientific domains. The problem is difficult however, because it is
ambiguous and hierarchical, and current techniques for finding k often produce
unsatisfying results. Humans are adept at navigating ambiguous and hierarchical
situations, and this paper measures human performance on the problem of finding k
across a wide variety of data sets. We find that humans employ multiple
strategies for choosing k, often simultaneously, and the number of possible
interpretations of even simple data sets with very few (N < 20) samples can be
quite high. In addition, two leading machine learning algorithms are compared to
the human results and methods for improving these techniques are discussed.

Back to Thursday Papers