Zoé van Havre ((Paris Dauphine & QUT): Overfitting mixture models and Hidden Markov models with an unknown number of states

Finite mixture models and Hidden Markov Models (HMMs) are commonly used to capture the underlying structure of today’s increasingly complex datasets, but a complicated problem is posed when the number of unobserved groups (or states) is unknown. Recent advances in the asymptotic theory of overfitted Bayesian mixture models and HMMs have motivated the development of a new class of methods for this problem, based on deliberately overfitting models given some restrictions on the prior. We investigated the impact the prior on the weights in overfitted Gaussian mixture models has on the number of components in the posterior, given the theoretical results by Rousseau and Mengersen (2011). We developed a method using a dynamic hyperparameter in the prior as part of an overfitted Gibbs sampling scheme which results in unnecessary components having weights near zero, while automatically resolving the label switching problem. We extended our scope to HMMs, where the same method cannot be directly applied due to lack of underpinning theory. To overfit in this case, we modified our use of a prior for the HMM transition matrix in order to encourage merging of extra states, and combined it with a repulsive prior on the component parameters. We present our methods and results from overfitted mixtures and HMMs using extensive simulation experiments and real-world applications.

Keywords: overfitting, finite mixture model, hidden Markov model