Information Criteria and Statistical Modeling

verfasst von: Sadanori Konishi, Genshiro Kitagawa

Verlag: Springer New York

Buchreihe : Springer Series in Statistics

Enthalten in: Springer Professional "Wirtschaft+Technik" , Springer Professional "Technik" , Springer Professional "Wirtschaft"

Einloggen, um Zugang zu erhalten

Über dieses Buch

The Akaike information criterion (AIC) derived as an estimator of the Kullback-Leibler information discrepancy provides a useful tool for evaluating statistical models, and numerous successful applications of the AIC have been reported in various fields of natural sciences, social sciences and engineering.

One of the main objectives of this book is to provide comprehensive explanations of the concepts and derivations of the AIC and related criteria, including Schwarz’s Bayesian information criterion (BIC), together with a wide range of practical examples of model selection and evaluation criteria. A secondary objective is to provide a theoretical basis for the analysis and extension of information criteria via a statistical functional approach. A generalized information criterion (GIC) and a bootstrap information criterion are presented, which provide unified tools for modeling and model evaluation for a diverse range of models, including various types of nonlinear models and model estimation procedures such as robust estimation, the maximum penalized likelihood method and a Bayesian approach.

Inhaltsverzeichnis

Frontmatter

1. Concept of Statistical Modeling

Statistical modeling is a crucial issue in scientific data analysis. Models are used to represent stochastic structures, predict future behavior, and extract useful information from data. In this chapter, we discuss statistical models and modeling methodologies such as parameter estimation, model selection, regularization method, and hierarchical Bayesian modeling. Finally, the organization of this book is described.

2. Statistical Models

In this chapter, we describe probability distributions, which provide fundamental tools for statistical models, and show that conditional distributions are used to acquire various types of information in the model-building process. By using regression and time series models as specific examples, we also discuss why evaluation of statistical models is necessary.

3. Information Criterion

In this chapter, we discuss using Kullback–Leibler information as a criterion for evaluating statistical models that approximate the true probability distribution of the data and its properties. We also explain how this criterion for evaluating statistical models leads to the concept of the information criterion, AIC. To this end, we explain the basic framework of model evaluation and the derivation of AIC by adopting a unified approach.

4. Statistical Modeling by AIC

The majority of the problems in statistical inference can be considered to be problems related to statistical modeling. They are typically formulated as comparisons of several statistical models. In this chapter, we consider using the AIC for various statistical inference problems such as checking the equality of distributions, determining the bin size of a histogram, selecting the order for regression models, detecting structural changes, determining the shape of a distribution, and selecting the Box-Cox transformation.

5. Generalized Information Criterion (GIC)

In this chapter, we describe a general framework for constructing information criteria in the context of functional statistics and introduce a generalized information criterion, GIC [Konishi and Kitagawa (1996)]. The GIC can be applied to evaluate statistical models constructed by various types of estimation procedures including the robust estimation procedure and the maximum penalized likelihood procedure. Section 5.1 describes the fundamentals of a functional approach using a probability model having one parameter. In Section 5.2 and subsequent sections, we introduce the generalized information criterion for evaluating statistical models constructed in various ways. We also discuss the relationship among the AIC, TIC, and GIC. Various applications of the GIC to statistical modeling are shown in Chapter 6. Chapter 7 gives the derivation of information criteria and investigates their asymptotic properties with theoretical and numerical improvements.

6. Statistical Modeling by GIC

The current wide availability of fast and inexpensive computers enables us to construct various types of nonlinear models for analyzing data having a complex structure. Crucial issues associated with nonlinear modeling are the choice of adjusted parameters including the smoothing parameter, the number of basis functions in splines and B-splines, and the number of hidden units in neural networks. Selection of these parameters in the modeling process can be viewed as a model selection and evaluation problem. This chapter addresses these issues as a model selection and evaluation problem and provides criteria for evaluating various types of statistical models.

7. Theoretical Development and Asymptotic Properties of the GIC

Information criteria have been constructed as estimators of the Kullback–Leibler information discrepancy between two probability distributions or, equivalently, the expected log-likelihood of a statistical model for prediction.

In this chapter, we introduce a general framework for constructing information criteria in the context of functional statistics and give technical arguments and a detailed derivation of the generalized information criterion (GIC) defined in (5.64).We also investigate the asymptotic properties of information criteria in the estimation of the expected log-likelihood of a statistical model.

8. Bootstrap Information Criterion

Advances in computing now allow numerical methods to be used for modeling complex systems, instead of analytic methods. Complex Bayesian models can now be used for practical applications by using numerical methods such as the Markov chain Monte Carlo (MCMC) technique. Also, when the maximum likelihood estimator cannot be obtained analytically, it is possible to obtain it by a numerical optimization method. In conjunction with the development of numerical methods, model evaluation must now deal with extremely complex and increasingly diverse models. The bootstrap information criterion [Efron (1983), Wong (1983), Konishi and Kitagawa (1996), Ishiguro et al. (1997), Cavanaugh and Shumway (1997), and Shibata (1997)], obtained by applying the bootstrap methods originally proposed by Efron (1979), permits the evaluation of models estimated through complex processes.

9. Bayesian Information Criteria

This chapter considers model selection and evaluation criteria from a Bayesian point of view. A general framework for constructing the Bayesian information criterion (BIC) is described. The BIC is also extended such that it can be applied to the evaluation of models estimated by regularization. Section 9.2 presents Akaike’s Bayesian information criterion (ABIC) developed for the evaluation of Bayesian models having prior distributions with hyperparameters. In the latter half of this chapter, we consider information criteria for the evaluation of predictive distributions of Bayesian models. In particular, Section 9.3 gives examples of analytical evaluations of bias correction for linear Gaussian Bayes models. Section 9.4 describes, for general Bayesian models, how to estimate the asymptotic biases and how to perform the second-order bias correction by means of Laplace’s method for integrals.

10. Various Model Evaluation Criteria

So far in this book, we have considered model selection and evaluation criteria from both an information-theoretic point of view and a Bayesian approach. The AIC-type criteria were constructed as estimators of the Kullback–Leibler information between a statistical model and the true distribution generating the data or equivalently the expected log-likelihood of a statistical model. In contrast, the Bayes approach for selecting a model was to choose the model with the largest posterior probability among a set of candidate models.

There are other model evaluation criteria based on various different points of view. This chapter describes cross-validation, generalized cross-validation, final predictive error (FPE), Mallows’ C_p, the Hannan–Quinn criterion, and ICOMP. Cross-validation also provides an alternative approach to estimate the Kullback–Leibler information. We show that the cross-validation estimate is asymptotically equivalent to AIC-type criteria in a general setting.

Backmatter

Titel: Information Criteria and Statistical Modeling
verfasst von: Sadanori Konishi
Genshiro Kitagawa
Verlag: Springer New York
Electronic ISBN: 978-0-387-71887-3
Print ISBN: 978-0-387-71886-6
DOI: https://doi.org/10.1007/978-0-387-71887-3

Springer Professional