Skip to main content
Top
Published in: Annals of Data Science 2/2015

01-06-2015

Entropy Estimation Using Numerical Methods

Author: Hadi Alizadeh Noughabi

Published in: Annals of Data Science | Issue 2/2015

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Direct integration of the Riemann–Stieltjes integral has been used to computing convolution integrals. This approach has been established to be simple and accurate with good convergence property. In this paper, we used some numerical methods to estimation of entropy of a continuous random variable and then some estimators are introduced. Bounds on the error terms are derived for some direct Riemann–Stieltjes integration methods. Consistency of estimators is proved and by simulation, the proposed estimators are compared with some prominent estimators, namely Correa (Commun Stat Theory Methods 24:2439–2449, 1995), Ebrahimi et al. (Stat Probab Lett 20:225–234, 1994), van Es (Scand J Stat 19:61–72, 1992) and Vasicek (J R Stat Soc B 38:54–59, 1976). The results indicate that the proposed estimators have smaller mean squared error than other estimators.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Alizadeh Noughabi H, Arghami NR (2013) General treatment of goodness-of-fit tests based on Kullback–Leibler information. J Stat Comput Simul 83:1556–1569MathSciNetCrossRef Alizadeh Noughabi H, Arghami NR (2013) General treatment of goodness-of-fit tests based on Kullback–Leibler information. J Stat Comput Simul 83:1556–1569MathSciNetCrossRef
2.
go back to reference Balakrishnan N, Habibi Rad A, Arghami NR (2007) Testing exponentiality based on Kullback–Leibler information with progressively type-II censored data. IEEE Trans Reliab 56:301–307CrossRef Balakrishnan N, Habibi Rad A, Arghami NR (2007) Testing exponentiality based on Kullback–Leibler information with progressively type-II censored data. IEEE Trans Reliab 56:301–307CrossRef
3.
go back to reference Boehme TK, Preuss W, van der Wall V (1991) On a simple numerical method for computing Stieltjes integrals in reliability theory. Probab Eng Inf Sci 5:113–128MATHCrossRef Boehme TK, Preuss W, van der Wall V (1991) On a simple numerical method for computing Stieltjes integrals in reliability theory. Probab Eng Inf Sci 5:113–128MATHCrossRef
4.
go back to reference Chaudhry ML (1995) On computations of the mean and variance of the number of renewals—a unified approach. J Oper Res Soc 46:1352–1364MATHCrossRef Chaudhry ML (1995) On computations of the mean and variance of the number of renewals—a unified approach. J Oper Res Soc 46:1352–1364MATHCrossRef
6.
go back to reference Deligonul ZS, Bilgen S (1984) Solution of the Volterra equation of renewal theory with the Galerkin technique using cubic splines. J Stat Comput Simul 20:37–45MATHMathSciNetCrossRef Deligonul ZS, Bilgen S (1984) Solution of the Volterra equation of renewal theory with the Galerkin technique using cubic splines. J Stat Comput Simul 20:37–45MATHMathSciNetCrossRef
7.
go back to reference Den Iseger PW, Smith MAJ, Dekker R (1997) Computing compound distribution faster!. Insur Math Econ 20:23–34MATHCrossRef Den Iseger PW, Smith MAJ, Dekker R (1997) Computing compound distribution faster!. Insur Math Econ 20:23–34MATHCrossRef
8.
go back to reference Dobrushin RL (1958) Simplified method of experimental estimate of entropy of stationary sequence. Theory Probab Appl 3:462–464MATH Dobrushin RL (1958) Simplified method of experimental estimate of entropy of stationary sequence. Theory Probab Appl 3:462–464MATH
10.
go back to reference Esteban MD, Castellanos ME, Morales D, Vajda I (2001) Monte Carlo comparison of four normality tests using different entropy estimates. Commun Stat Simul Comput 30:761–785MATHMathSciNetCrossRef Esteban MD, Castellanos ME, Morales D, Vajda I (2001) Monte Carlo comparison of four normality tests using different entropy estimates. Commun Stat Simul Comput 30:761–785MATHMathSciNetCrossRef
11.
go back to reference Goria MN, Leonenko NN, Mergel VV, Novi Inverardi PL (2005) A new class of random vector entropy estimators and its applications in testing statistical hypotheses. Nonparametr Stat 17:277–297MATHMathSciNetCrossRef Goria MN, Leonenko NN, Mergel VV, Novi Inverardi PL (2005) A new class of random vector entropy estimators and its applications in testing statistical hypotheses. Nonparametr Stat 17:277–297MATHMathSciNetCrossRef
12.
go back to reference Grzegorzewski P, Wieczorkowski R (1999) Entropy-based goodness-of-fit test for exponentiality. Commun Stat Theory Methods 28:1183–1202MATHMathSciNetCrossRef Grzegorzewski P, Wieczorkowski R (1999) Entropy-based goodness-of-fit test for exponentiality. Commun Stat Theory Methods 28:1183–1202MATHMathSciNetCrossRef
13.
go back to reference Habibi Rad A, Yousefzadeh F, Balakrishnan N (2011) Goodness-of-fit test based on Kullback–Leibler information for progressively Type-II censored data. IEEE Trans Reliab 60:570–579CrossRef Habibi Rad A, Yousefzadeh F, Balakrishnan N (2011) Goodness-of-fit test based on Kullback–Leibler information for progressively Type-II censored data. IEEE Trans Reliab 60:570–579CrossRef
15.
go back to reference Hutcheson K, Shenton LR (1974) Some moments of an estimate of Shannon’s measure of information. Commun Stat 3:89–94MATHMathSciNet Hutcheson K, Shenton LR (1974) Some moments of an estimate of Shannon’s measure of information. Commun Stat 3:89–94MATHMathSciNet
17.
go back to reference Kao EPC (1997) An introduction to stochastic processes. Duxbury Press, Belmont Kao EPC (1997) An introduction to stochastic processes. Duxbury Press, Belmont
18.
go back to reference Mack SP (1988) A comparative study of entropy estimators and entropy based goodness-of-fit tests. PhD Dissertation, University of California, Riverside Mack SP (1988) A comparative study of entropy estimators and entropy based goodness-of-fit tests. PhD Dissertation, University of California, Riverside
19.
go back to reference McConalogue DJ (1981) Numerical treatment of convolution integrals involving distributions with densities having singularities at the origin. Commun Stat Simul Comput 10:265–280MathSciNetCrossRef McConalogue DJ (1981) Numerical treatment of convolution integrals involving distributions with densities having singularities at the origin. Commun Stat Simul Comput 10:265–280MathSciNetCrossRef
20.
go back to reference Pakyari R, Balakrishnan N (2012) A general purpose approximate goodness-of-fit test for progressively type-II censored data. IEEE Trans Reliab 61:238–244CrossRef Pakyari R, Balakrishnan N (2012) A general purpose approximate goodness-of-fit test for progressively type-II censored data. IEEE Trans Reliab 61:238–244CrossRef
21.
go back to reference Pakyari R, Balakrishnan N (2013) Goodness-of-fit tests for progressively type-II censored data from location-scale distributions. J Stat Comput Simul 83:167–178MATHMathSciNetCrossRef Pakyari R, Balakrishnan N (2013) Goodness-of-fit tests for progressively type-II censored data from location-scale distributions. J Stat Comput Simul 83:167–178MATHMathSciNetCrossRef
22.
go back to reference Rudin W (1976) Principles of mathematical analysis, 3rd edn. McGraw-Hill, New YorkMATH Rudin W (1976) Principles of mathematical analysis, 3rd edn. McGraw-Hill, New YorkMATH
23.
go back to reference Shannon CE (1948) A mathematical theory of communications. Bell Syst Tech J 27:379–423; 623–656 Shannon CE (1948) A mathematical theory of communications. Bell Syst Tech J 27:379–423; 623–656
24.
go back to reference Silverman BW (1986) Density estimation for statistics and data analysis. Chapman and Hall, LondonMATHCrossRef Silverman BW (1986) Density estimation for statistics and data analysis. Chapman and Hall, LondonMATHCrossRef
25.
go back to reference Tijms HC (1994) Stochastic models: an algorithmic approach. Wiley, New YorkMATH Tijms HC (1994) Stochastic models: an algorithmic approach. Wiley, New YorkMATH
26.
go back to reference van der Wall V, Preuss W (1989) A simple numerical procedure for solving Volterra–Stieltjes integral equations. In: Proceedings of the conference on complex analysis, Sofia, pp 424–430 van der Wall V, Preuss W (1989) A simple numerical procedure for solving Volterra–Stieltjes integral equations. In: Proceedings of the conference on complex analysis, Sofia, pp 424–430
27.
go back to reference van Es B (1992) Estimating functionals related to a density by class of statistics based on spacings. Scand J Stat 19:61–72MATH van Es B (1992) Estimating functionals related to a density by class of statistics based on spacings. Scand J Stat 19:61–72MATH
28.
29.
go back to reference Vatutin VA, Michailov VG (1995) Statistical estimation of entropy of discrete random variables with large numbers of results. Russ Math Surv 50:963–976MATHCrossRef Vatutin VA, Michailov VG (1995) Statistical estimation of entropy of discrete random variables with large numbers of results. Russ Math Surv 50:963–976MATHCrossRef
30.
go back to reference Xie M (1989) On the solution of renewal-type integral equations. Commun Stat Simul Comput 18:281–293MATHCrossRef Xie M (1989) On the solution of renewal-type integral equations. Commun Stat Simul Comput 18:281–293MATHCrossRef
31.
go back to reference Xie M, Preuss W, Cui L (2003) Error analysis of some integration procedures for renewal equation and convolution integrals. J Stat Comput Simul 73:59–70MATHMathSciNetCrossRef Xie M, Preuss W, Cui L (2003) Error analysis of some integration procedures for renewal equation and convolution integrals. J Stat Comput Simul 73:59–70MATHMathSciNetCrossRef
Metadata
Title
Entropy Estimation Using Numerical Methods
Author
Hadi Alizadeh Noughabi
Publication date
01-06-2015
Publisher
Springer Berlin Heidelberg
Published in
Annals of Data Science / Issue 2/2015
Print ISSN: 2198-5804
Electronic ISSN: 2198-5812
DOI
https://doi.org/10.1007/s40745-015-0045-9

Other articles of this Issue 2/2015

Annals of Data Science 2/2015 Go to the issue