Stein's density approach and information inequalities

Christophe Ley (Université Libre de Bruxelles)
Yvik Swan (Université du Luxembourg)

Abstract


We provide a new perspective on Stein's so-called density approach by introducing a new operator and characterizing class which are valid for a much wider family of probability distributions on the real line. We prove an elementary factorization property of this operator and propose a new Stein identity which we use to derive information inequalities in terms of what we call the "generalized Fisher information distance". We provide explicit bounds on the constants appearing in these inequalities for several important cases. We conclude with a comparison between our results and known results in the Gaussian case, hereby improving on several known inequalities from the literature.

Full Text: Download PDF | View PDF online (requires PDF plugin)

Pages: 1-14

Publication Date: January 27, 2013

DOI: 10.1214/ECP.v18-2578

References

  • Afendras, G.; Papadatos, N.; Papathanasiou, V. : An extended Stein-type covariance identity for the Pearson family with applications to lower variance bounds. Bernoulli 17 (2011), no. 2, 507--529. MR2787602
  • Barbour, A. D. : Stein's method for diffusion approximations. Probab. Theory Related Fields 84 (1990), no. 3, 297--322. MR1035659
  • Barbour, A. D., and Chen, L. H. Y.: An introduction to Stein's method, vol. 4. World Scientific, 2005. MR2205339
  • Barbour, A. D., and Chen, L. H. Y.: Stein's method and applications, vol. 5. World Scientific, 2005. MR2205339
  • Barbour, A. D.; Johnson, O.; Kontoyiannis, I.; Madiman, M. : Compound Poisson approximation via information functionals. Electron. J. Probab. 15 (2010), 1344--1368. MR2721049
  • Barron, Andrew R. : Entropy and the central limit theorem. Ann. Probab. 14 (1986), no. 1, 336--342. MR0815975
  • Brown, Lawrence D. : A proof of the central limit theorem motivated by the Cramér-Rao inequality. Statistics and probability: essays in honor of C. R. Rao, pp. 141--148, North-Holland, Amsterdam-New York, 1982. MR0659464
  • Cacoullos, T.; Papathanasiou, V. : Characterizations of distributions by variance bounds. Statist. Probab. Lett. 7 (1989), no. 5, 351--356. MR1001133
  • Cacoullos, T.; Papathanasiou, V.; Utev, S. A. : Variational inequalities with examples and an application to the central limit theorem. Ann. Probab. 22 (1994), no. 3, 1607--1618. MR1303658
  • Chatterjee, Sourav; Fulman, Jason; Röllin, Adrian : Exponential approximation by Stein's method and spectral graph theory. ALEA Lat. Am. J. Probab. Math. Stat. 8 (2011), 197--223. MR2802856
  • Chatterjee, Sourav; Shao, Qi-Man : Nonnormal approximation by Stein's method of exchangeable pairs with application to the Curie-Weiss model. Ann. Appl. Probab. 21 (2011), no. 2, 464--483. MR2807964
  • Chen, Louis H. Y.; Goldstein, Larry; Shao, Qi-Man : Normal approximation by Stein's method. Probability and its Applications (New York). Springer, Heidelberg, 2011. xii+405 pp. ISBN: 978-3-642-15006-7 MR2732624
  • Chen, Louis H. Y.; Shao, Qi-Man : Stein's method for normal approximation. An introduction to Stein's method, 1--59, Lect. Notes Ser. Inst. Math. Sci. Natl. Univ. Singap., 4, Singapore Univ. Press, Singapore, 2005. MR2235448
  • Cover, Thomas M.; Thomas, Joy A. : Elements of information theory. Second edition. Wiley-Interscience [John Wiley & Sons], Hoboken, NJ, 2006. xxiv+748 pp. ISBN: 978-0-471-24195-9; 0-471-24195-4 MR2239987
  • Döbler, C.: Stein's method of exchangeable pairs for absolutely continuous, univariate distributions with applications to the polya urn model. arXiv:1207.0533, July 2012.
  • Gibbs, A. L., and Su, F. E.: On choosing and bounding probability metrics. International Statistical Review / Revue Internationale de Statistique, 70 (2002), 419--435.
  • Goldstein, Larry; Reinert, Gesine : Distributional transformations, orthogonal polynomials, and Stein characterizations. J. Theoret. Probab. 18 (2005), no. 1, 237--260. MR2132278
  • Goldstein, Larry; Reinert, Gesine : Stein's method and the beta distribution. arXiv:1207.1460, July 2012.
  • Götze, F. : On the rate of convergence in the multivariate CLT. Ann. Probab. 19 (1991), no. 2, 724--739. MR1106283
  • Johnson, Oliver : Information theory and the central limit theorem. Imperial College Press, London, 2004. xiv+209 pp. ISBN: 1-86094-473-6 MR2109042
  • Johnson, Oliver; Barron, Andrew : Fisher information inequalities and the central limit theorem. Probab. Theory Related Fields 129 (2004), no. 3, 391--409. MR2128239
  • Kontoyiannis, Ioannis; Harremoës, Peter; Johnson, Oliver : Entropy and the law of small numbers. IEEE Trans. Inform. Theory 51 (2005), no. 2, 466--472. MR2236061
  • Ley, C., and Swan, Y.: Stein's density approach for discrete distributions and information inequalities. arXiv:1211.3668v1, November 2012.
  • Linnik, Ju. V. : An information-theoretic proof of the central limit theorem with Lindeberg conditions. (Russian) Theor. Probability Appl. 4 1959 288--299. MR0124081
  • Luk, H. M.: Stein's method for the gamma distribution and related statistical applications. PhD thesis, University of Southern California, 1994.
  • Mayer-Wolf, Eduardo : The Cramér-Rao functional and limiting laws. Ann. Probab. 18 (1990), no. 2, 840--850. MR1055436
  • Nourdin, Ivan; Peccati, Giovanni : Stein's method meets Malliavin calculus: a short survey with new estimates. Recent development in stochastic dynamics and stochastic analysis, 207--236, Interdiscip. Math. Sci., 8, World Sci. Publ., Hackensack, NJ, 2010. MR2807823
  • Nourdin, Ivan; Peccati, Giovanni : Normal approximations with Malliavin calculus. From Stein's method to universality. Cambridge Tracts in Mathematics, 192. Cambridge University Press, Cambridge, 2012. xiv+239 pp. ISBN: 978-1-107-01777-1 MR2962301
  • Picket, A.: Rates of convergence of χ^2 approximations via Stein's method. PhD thesis, Lincoln College, University of Oxford, 2004.
  • Röllin, A.: On the optimality of stein factors. phProbability Approximations and Beyond/ (2012), ~61--72.
  • Ross, Nathan : Fundamentals of Stein's method. Probab. Surv. 8 (2011), 210--293. MR2861132
  • Sason, I.: An information-theoretic perspective of the poisson approximation via the chen-stein method. pharXiv:1206.6811/, June 2012.
  • Sason, I.: On the entropy of sums of bernoulli random variables via the chen-stein method. pharXiv:1207.0436/, July 2012.
  • Schoutens, Wim: Orthogonal polynomials in Stein's method. J. Math. Anal. Appl. 253 (2001), no. 2, 515--531. MR1808151
  • Shimizu, R.: On fisher's amount of information for location family. In phStatistical Distributions in Scientific Work/ (1975), G. P. et~al., Ed., vol. 3, ~305--312.
  • Shimizu, Ryoichi: Error bounds for asymptotic expansion of the scale mixtures of the normal distribution. Ann. Inst. Statist. Math. 39 (1987), no. 3, 611--622. MR0930533
  • Stein, Charles: A bound for the error in the normal approximation to the distribution of a sum of dependent random variables. Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability (Univ. California, Berkeley, Calif., 1970/1971), Vol. II: Probability theory, pp. 583--602. Univ. California Press, Berkeley, Calif., 1972. MR0402873
  • Stein, Charles: Approximate computation of expectations. Institute of Mathematical Statistics Lecture Notes—Monograph Series, 7. Institute of Mathematical Statistics, Hayward, CA, 1986. iv+164 pp. ISBN: 0-940600-08-0 MR0882007
  • Stein, Charles; Diaconis, Persi; Holmes, Susan; Reinert, Gesine: Use of exchangeable pairs in the analysis of simulations. Stein's method: expository lectures and applications, 1--26, IMS Lecture Notes Monogr. Ser., 46, Inst. Math. Statist., Beachwood, OH, 2004. MR2118600
  • Stein, Charles M.: Estimation of the mean of a multivariate normal distribution. Ann. Statist. 9 (1981), no. 6, 1135--1151. MR0630098
Bookmark and Share


Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.