Analyzing the Effect of Bin-width on the Computed Entropy

Sri Purwani, Sudradjat Supian, Carole Twining


The Shannon entropy is a mathematical expression for quantifying the amount of randomness which can be used to measure information content. It is used in objective function. Mutual Information (MI) uses Shannon entropy in order to determine shared information content of two images. The Shannon entropy, which was originally derived by Shannon in the context of lossless encoding of messages, is also used to define an optimum message length used in the Minimum Description Length (MDL) principle for groupwise registration. We first derived the Shannon entropy from the integral of probability density function (pdf), and thenfound that Gaussian has maximum entropy over all possible distribution. We also show that the entropy of the flat distribution is less than the entropy of the Gaussian distribution with the same variance.We then investigated the effect of bin-width on the computed entropy. Weanalyzed the relationship between the computed entropy and the integral entropy when we vary bin-width, but fix variance and the number of samples. We then found that the value of the computed entropy lies within the theoretical predictions at small and large bin-widths. Wealso show two types of bias in entropy estimators.


Entropy; Objective function; Gaussian distribution; Flat distribution

Full Text:



L.G. Brown, A survey of image registration techniques, ACM Computing Surveys 24 (1992), 325 – 376.

C.C. Brun, N. Leporé, X. Pennec, Y. Chou, A.D. Lee, G.D. Zubicaray, M.J. Wright, J.C. Gee, P.M. Thompson and et al., A non-conservative lagrangian framework for statistical fluid registration - SAFIRA, IEEE Transactions on Medical Imaging 30(2) (2011), 184 – 202.

A.G. Carlton, On the bias of information estimates, Psychological Bulletin 71(2) (1969), 108 – 109.

J.H.C. Lisman and M.C.A. van Zuylen, Note on the generation of most probable frequency distributions, Statistica Neerlandica 26(1) (1972), 19 – 23.

R. Moddemeijer, On estimation of entropy and mutual information of continuous distributions, Signal Processing 16(3) (1989), 233 – 248.

L. Paninski, Estimation of entropy and mutual information, Neural Computation 15(6) (2003), 1191 – 1253.

J.P.W. Pluim, J.B.A. Maintz and M.A. Viergever, Interpolation artefacts in mutual informationbased image registration, Computer Vision and Image Understanding 77 (2000), 211 – 232.

J.P.W. Pluim, J.B.A. Maintz and M.A. Viergever, Mutual-information based registration of medical images: a survey, IEEE Transactions on Medical Imaging 22(8) (2003), 986 – 1004.

M.R. Sabuncu and P. Ramadge, Using spanning graphs for effcient image registration, IEEE Transactions on Image Processing 17(5) (2008), 788 – 797.

C.E. Shannon, A mathematical theory of communication, The Bell System Technical Journal 27(1948), 379 – 423.

C. Studholme, D.L.G. Hill and D.J. Hawkes, Multiresolution voxel similarity measures for MR-PET registration, in Information Processing in Medical Imaging, Vol. 3 of Computational Imaging and Vision (1995), 287 – 298, Kluwer.

C.J. Twining and C.J. Taylor, Specificity: a graph-based estimator of divergence, IEEE Transactions on Pattern Analysis and Machine Intelligence 33(12) (2011), 2492 – 2505.

P.A. Viola, Alignment by Maximization of Mutual Information, PhD thesis, Massachusetts Institute of Technology (1995).

P. Viola and W.M. Wells III, Alignment by maximization of mutual information, International Journal of Computer Vision 24(2) (1997), 137 – 154.


eISSN 0975-5748; pISSN 0974-875X