# Item

ITEM ACTIONSEXPORT

Released

Conference Paper

#### How biased are maximum entropy models?

##### MPS-Authors

There are no MPG-Authors in the publication available

##### External Resource

https://papers.nips.cc/paper/4357-how-biased-are-maximum-entropy-models.pdf

(Publisher version)

##### Fulltext (public)

There are no public fulltexts stored in PuRe

##### Supplementary Material (public)

There is no public supplementary material available

##### Citation

Macke, J., Murray, I., & Latham, P. (2012). How biased are maximum entropy models?
In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, & K. Weinberger (*Advances in Neural Information Processing Systems 24* (pp. 2034-2042). Red Hook, NY, USA:
Curran.

Cite as: http://hdl.handle.net/11858/00-001M-0000-0013-B87A-5

##### Abstract

Maximum entropy models have become popular statistical models in neuroscience and other areas in biology, and can be useful tools for obtaining estimates of mutual
information in biological systems. However, maximum entropy models fit to small data sets can be subject to sampling bias; i.e. the true entropy of the data can be severely underestimated. Here we study the sampling properties of estimates of the entropy obtained from maximum entropy models. We show that if the data is generated by a distribution that lies in the model class, the bias is equal to the number of parameters divided by twice the number of observations. However, in practice, the true distribution is usually outside the model class, and we show here that this misspecification can lead to much larger bias. We provide a perturbative approximation of the maximally expected bias when the true model is out of
model class, and we illustrate our results using numerical simulations of an Ising model; i.e. the second-order maximum entropy distribution on binary data.