Introductory Statistical Inference with the Likelihood Function

Frontmatter

Chapter 1. Introduction

Abstract

I have a list of 100 individuals, numbered 1–100. Associated with each is their disease status, diseased, d, or not diseased, \(\overline{d}\).

Charles A. Rohde

Chapter 2. The Statistical Approach

Abstract

Assume that we have observed data D = x which was the result of a random experiment X (or can be approximated as such). The data are then modelled using

1.

A sample space, \(\mathcal{X}\) for the observed value of x

 

2.

A probability density function for X at x, f(x; θ)

 

3.

A parameter space for θ, \(\Theta \)

 

Charles A. Rohde

Chapter 3. Estimation

Abstract

As previously indicated the major problems of inference are estimation, testing, and interval estimation.

1.

Of these estimation is the easiest to understand and in certain respects the most important.

 

2.

Intuitively, a point estimate is simply a guess as to the value of a parameter of interest.

 

3.

Along with this guess it is customary to provide some measure of reliability.

 

Charles A. Rohde

Chapter 4. Interval Estimation

Abstract

In this chapter we describe methods for finding interval or set estimators mainly illustrated using random samples from the normal distribution.

Charles A. Rohde

Chapter 5. Hypothesis Testing

Abstract

Suppose that hypothesis f specifies that the probability that the random variable X takes on the value x is f(x) and hypothesis g specifies that the probability that the random variable X takes on the value x is g(x).

Charles A. Rohde

Chapter 6. Standard Practice of Statistics

Abstract

Most commonly used statistical methods, interval estimation (confidence intervals), estimation, hypothesis tests, and significance tests have as justification their properties under repeated sampling. This is the frequency interpretation of statistical methods and is the basis for much (most) of the statistical methods commonly applied to data in research and practice. This chapter is based on Table 2.1 in Royall. What we have shown in Chaps. 3–5 is that many of the standard statistical results are misleading.

Charles A. Rohde

Chapter 7. Maximum Likelihood: Basic Results

Abstract

As we have seen once we have an estimator and its sampling distribution we can easily obtain confidence intervals and tests regarding the parameter. We now develop the theory of estimation focusing on the method of maximum likelihood, which for parametric models is the most widely used method. This will also supply us with a collection of statistical methods for important problems.

Charles A. Rohde

Chapter 8. Linear Models

Abstract

There is no doubt that the linear model is one of the most important and useful models in statistics. In this chapter we discuss the estimation problem in linear models and discuss interpretations of standard results.

Charles A. Rohde

Chapter 9. Other Estimation Methods

Abstract

Suppose that we have sample data x ₁, x ₂, …, x _n assumed to be observed values of independent random variables each having the same distribution function F where

$$\displaystyle{F(x) = \mathbb{P}(X \leq x)}$$

Define new random variables Z _i as the indicator functions of the interval (−∞, x], i.e.,

$$\displaystyle{Z_{i}(x) = \left \{\begin{array}{rl} 1&X_{i} \leq x\\ 0 &\mbox{ otherwise} \end{array} \right.}$$

Charles A. Rohde

Chapter 10. Decision Theory

Abstract

• In this chapter we introduce some of the ideas and notation of decision theory.
• At one time it was thought that all statistical problems could be cast in the decision theoretic framework and statistics could thus be reduced to a study of optimization techniques.
• This subsided, partly due to the complexity of real-world problems and partly due to the realization that inference was more subtle than optimization.
• Nevertheless some knowledge of the basic concepts is useful for consolidation of ideas and as an introduction to Bayesian ideas.

Charles A. Rohde

Chapter 11. Sufficiency

Abstract

We consider a set of observations x thought to be a realization of some random variable X whose probability distribution belongs to a set of distributions

$$\displaystyle{\mbox{ $\boldsymbol{\mathcal{F}}$} =\{ f(\cdot \:;\:\theta )\::\:\theta \in \Theta \}}$$

The distributions in \(\mbox{ $\boldsymbol{\mathcal{F}}$}\) are indexed by a parameter θ, i.e., the parameter θ determines which of the distributions is used to assign probabilities to X. The set \(\Theta \) is called the parameter space and \(\mbox{ $\boldsymbol{\mathcal{F}}$}\) is called the family of distributions. \(\mbox{ $\boldsymbol{\mathcal{F}}$}\) along with X constitutes the probability model for the observed data.

Charles A. Rohde

Chapter 12. Conditionality

Abstract

Conditioning arguments are at the center of many disputes regarding the foundations of statistical inference. We present here only some simple arguments and examples.

Charles A. Rohde

Chapter 13. Statistical Principles

Abstract

A number of principles for evaluating the evidence (information) provided by data have been formulated:

• The repeated sampling principle: evidence (information) is evaluated using hypothetical repeated sampling.
• The sufficiency principle: evidence (information) should depend only on the value of a sufficient statistic.
• The conditionality principle: evidence (information) should depend only on the experiment actually performed.
• The likelihood principle: evidence (information) resulting from observations with proportional likelihoods should be the same.
• The Bayesian coherency principle: evidence (information) from data is used to obtain (beliefs) using Bayes theorem which requires consistent (coherent) betting behavior.
• Birnbaum’s confidence concept: a concept of statistical evidence is not plausible unless it finds “strong evidence” for H ₂ as against H ₁ with small probability (α) when H ₁ is true and with much larger probability (1 −β) when H ₂ is true.

Charles A. Rohde

Chapter 14. Bayesian Inference

Abstract

In the frequentist approach to parametric statistical inference:

1.

Probability models are based on the relative frequency interpretation of probabilities.

 

2.

Parameters of the resulting probability models are assumed to be fixed, unknown constants.

 

3.

Observations on random variables with a probability model depending on the parameters are used to construct statistics. These are used to make inferential statements about the parameters.

 

4.

Inferences are evaluated and interpreted on the basis of the sampling distribution of the statistics used for the inference. Thus an interval which claims to be a 95 % confidence interval for θ has the property that it contains θ 95 % of the time in repeated use.

 

5.

In all cases inferences are evaluated on the basis of data not observed.

 

Charles A. Rohde

Chapter 15. Bayesian Statistics: Computation

Abstract

• By Bayes theorem the posterior density of θ is given by

  $$\displaystyle{p(\theta \vert x) = \frac{f(x;\theta )p(\theta )} {f(x)} }$$

  where

  $$\displaystyle{f(x) =\int _{\Theta }f(x;\theta )p(\theta )dm(\theta )}$$
• The calculation of the posterior thus requires calculation of an integral of the likelihood weighted by the prior.
• Usually this integral can only be determined in closed form for conjugate priors.

Charles A. Rohde

Chapter 16. Bayesian Inference: Miscellaneous

Abstract

Suppose you have obtained a posterior distribution for θ based on data y ₁. At a later date you are given data y ₂ whose distribution depends on the same parameter and is independent of the previous data. Then we have that

$$\displaystyle\begin{array}{rcl} p(\theta \vert y_{1},y_{2})& =& \frac{f(y_{1},y_{2};\theta )g(\theta )} {\int _{\Theta }f(y_{1},y_{2};\theta )g(\theta )d\theta } {}\\ & =& \frac{f(y_{2};\theta )f(y_{1};\theta )g(\theta )} {\int _{\Theta }f(y_{2};\theta )f(y_{1};\theta )g(\theta )d\theta } {}\\ & =& \frac{f(y_{2};\theta )\frac{f(y_{1};\theta )g(\theta )} {f(y_{1}} } {\int _{\Theta }f(y_{2};\theta )\frac{f(y_{1};\theta )g(\theta )} {f(y_{1})} d\theta } {}\\ & =& \frac{f(y_{2};\theta )p(\theta \vert y_{1})} {\int _{\Theta }f(y_{2};\theta )p(\theta \vert y_{1})d\theta )} {}\\ \end{array}$$

i.e., “yesterday’s posterior becomes today’s prior.”

Charles A. Rohde

Chapter 17. Pure Likelihood Methods

Abstract

As we have seen in previous chapters use of the likelihood is important in frequentist methods and in Bayesian methods. In this chapter we explore the use of the likelihood function in another context, that of providing a self-contained method of statistical inference. Richard Royall in his book, Statistical Evidence: A Likelihood Paradigm, carefully developed the foundation for this method building on the work of Ian Hacking and Anthony Edwards. Royall lists three questions of interest to statisticians and scientists after having observed some data

1.

What do I do?

 

2.

What do I believe?

 

3.

What evidence do I now have?

 

Charles A. Rohde

Chapter 18. Pure Likelihood Methods and Nuisance Parameters

Abstract

In most, if not all, statistical problems we have not one parameter but many. However, we are often interested in inference or statements on just one of the parameters. Suppose then that the parameter of interest is θ and that the remaining parameters, called nuisance parameters, are denoted by γ.

Charles A. Rohde

Chapter 19. Other Inference Methods and Concepts

Abstract

R.A. Fisher introduced the concept of fiducial probability and used in to develop fiducial inference. Counterexamples though the years have lead to its lack of use in statistics.

Charles A. Rohde

Chapter 20. Finite Population Sampling

Abstract

It is fair to say that most of the information we know about contemporary society is obtained as the result of sample surveys. Real populations are finite and the branch of statistics which treats sampling of such populations is called survey sampling. For many years survey sampling remained the province of “survey samplers” with very little input from statisticians involved in the more traditional aspects of the subject. The decades of the 1970s, 1980s, and 1990s saw somewhat successful mergers of the two areas using new approaches to finite population sampling theory based on prediction theory and population based models.

Charles A. Rohde

Chapter 21. Appendix: Probability and Mathematical Concepts

Abstract

If E ₁, E ₂, … is a denumerable collection of sets in \(\mathcal{W}\) then \(\cup _{i=1}^{\infty }E_{i} \in \mathcal{W}\).

Charles A. Rohde

Backmatter