Matrix Tricks for Linear Statistical Models

Frontmatter

Chapter 0. Introduction

Abstract

Let us start by considering the three scatter plots of Figures 0.1 and 0.2. In Figure 0.1a we have three data points \( \left(^{x_1}_{y_1}\right), \left(^{x_2}_{y_2}\right), \left(^{x_3}_{y_3}\right),\) while in Figure 0.2 there are 1000 data points.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 1. Easy Column Space Tricks

Abstract

The column space of an n × m matrix A,

$$\mathcal{C} (A_{n \times m}) = \{{\bf y} \in \mathbb{R}^n: \text{there exists}\ {\bf x} \in \mathbb{R}^m \text{such that}\ {\bf y} = {\bf Ax}\}, $$

(1.1)

and, correspondingly, the null space of A,

$$\mathcal{N} (A_{n \times m}) = \{{\bf x} \in \mathbb{R}^m: {\bf Ax} = 0\}, $$

(1.2)

are in every-day use throughout this book. In this chapter we take a good look at some of their properties, most of them rather elementary. Our experience is that decent steps in linear models are slow to take unless a reasonable set of column space tricks is in the immediate access. Some of the results that we go through are already likely to be in the reader’s toolbox. However, there cannot be harm in repeating these helpful rules and going through their proofs.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 2. Easy Projector Tricks

Abstract

In this chapter we go through some basic properties of the orthogonal projectors. We are great fans of projectors and believe that their heavy use simplifies many considerations and helps to understand what is going on particularly in various minimization problems. We first consider the orthogonal projections under the standard inner product, and then under the more general cases. We believe that it is instructive to proceed in this order instead of first going to the general case.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 3. Easy Correlation Tricks

Abstract

In this chapter we remind the reader about one fundamental fact: the geometric interpretation of the sample correlation coefficient. It’s hardly news for the reader but because of its importance it is worth its own chapter.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 4. Generalized Inverses in a Nutshell

Abstract

Let A be a given n × m matrix and y a given n × 1 vector. Consider the linear equation

$${\bf Ab} = \bf y $$

(1.4)

.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 5. Rank of the Partitioned Matrix and the Matrix Product

Abstract

There are numerous situations in the world of linear models and multivariate analysis when we need to find some appropriate expressions for the rank of the matrix product AB, or of the partitioned matrix (A : B), for conformable matrices A and B. Our favourite expressions are represented in the following theorem.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 6. Rank Cancellation Rule

Abstract

If \( a \in \mathbb{R}\) and \( y \in \mathbb{R}\) have property \( ay \neq 0,\) then trivially

$$lay = may \Longrightarrow la = ma,$$

(6.1)

that is, we can cancel y from (6.1) (as well as a). For matrices, the corresponding cancellation does not work. However, there is a very handy trick, the rank cancellation rule, which allows cancellations for matrices in the style of (6.1). It seems, according to our experience, that this simple rule has not received so much appreciation in statistical literature as it earns.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 7. Sum of Orthogonal Projectors

Abstract

In this chapter we give a well-known result regarding the sum of two orthogonal projectors. It is a short chapter, but nevertheless, we believe that it is worthwhile to keep this result in the gallery of active tools when dealing with matrices in statistics. This chapter is strongly related to the next chapter on a decomposition of orthogonal projector, beginning on page 155.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 8. A Decomposition of the Orthogonal Projector

Abstract

In this chapter we consider a decomposition of the orthogonal projector onto the column space of the partitioned matrix (A : B) and demonstrate its usefulness through several examples. The decomposition introduced is a consequence of Theorem 7 (p. 151).

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 9. Minimizing cov(y - Fx)

Abstract

In this chapter we consider the problem of finding the minimum—in the Löwner sense—for the covariance matrix of y - Fx where y and x are given random vectors and the matrix F is free to vary. This is a fundamental task in linear models and multivariate analysis and the solution, utilized in several places in this book, is very much worth remembering.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 10. BLUE

Abstract

In this chapter we focus on the BLUE-related matters, so to say. The most important thing is the fundamental BLUE equation (10.4) (p. 216).

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 11. General Solution to AYB = C

Abstract

In almost every chapter of this book we meet linear equations whose explicit solutions we wish to write up. This is what generalized inverses make nicely possible. In this chapter we represent the basic result which shows how to express the general solution for Y satisfying the equation AYB = C.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 12. Invariance with Respect to the Choice of Generalized Inverse

Abstract

Because we are heavily using generalized inverses, it is very important to know something about the invariance of the matrix expressions with respect to the choice of generalized inverses appearing in the expression. Our main attention in this chapter is focused on the matrix product AB ⁻C, but one can make corresponding questions concerning, for example, the invariance of the column space

$$\mathcal{C} ({\bf AB}^-{\bf C}),$$

the rank of AB ⁻C, and the eigenvalues of AB ⁻C.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 13. Block-Diagonalization and the Schur Complement

Abstract

In this chapter we present a block-diagonalization result for a symmetric nonnegative definite matrix. We may emphasize that the block-diagonalization result, sometimes called the Aitken block-diagonalization formula, due to Aitken (1939, Ch. 3, §29), is mathematically indeed quite simple just as it is. However, it is exceptionally handy and powerful tool for various situations arising in linear models and multivariate analysis, see, e.g., the derivation of the conditional multinormal distribution in Anderson (2003, §2.5); cf. also (9.21)–(9.22) (p. 193). We also consider the Schur complements whose usefulness in linear models and related areas can hardly be overestimated.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 14. Nonnegative Definiteness of a Partitioned Matrix

Abstract

The nonnegative definiteness of a symmetric partitioned matrix can be characterized in an interesting way in terms of submatrices. Our experience is that this characterization is a very handy tool in various problems related to the Löwner partial ordering.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 15. The Matrix $$\dot{\rm M}$$

Abstract

It is well known that if V is a symmetric positive definite n × n matrix, and (X : Z) is a partitioned orthogonal n × n matrix, then

$$({\bf X}^{\prime}{\bf V}^{-1}{\bf X})^{-1} = {\bf X}^{\prime}{\bf V}{\bf X} - {\bf X}^{\prime}{\bf V}{\bf Z}({\bf Z}^{\prime}{\bf V}{\bf Z})^{-1}{\bf Z}^{\prime}{\bf V}{\bf X}.$$

.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 16. Disjointness of Column Spaces

Abstract

In this chapter we collect together various equivalent characterizations for the disjointness of the column spaces \(\mathcal{C}(\bf A)\) and \(\mathcal{C}(\bf B),\) by which we mean that \(\mathcal{C}(\bf A)\) and \(\mathcal{C}(\bf B)\)have only the null vector in common; here A and B are n×p and n×q matrices. There are numerous situations in linear models and multivariate analysis when we meet the problem of disjointness.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 17. Full Rank Decomposition

Abstract

This chapter shows how helpful it is to express a matrix A as a product \({\bf UV}^{\prime}\) where both U and V have full column ranks.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 18. Eigenvalue Decomposition

Abstract

There is no way to survive in the middle of statistical considerations without being pretty well aware of the main properties of the eigenvalues and eigenvectors. This chapter provides a summary of some central results.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 19. Singular Value Decomposition

Abstract

While the eigenvalue decomposition \({\bf A} = \bf T{\bf \Lambda}T^{\prime},\) say, concerns only symmetric matrices, the singular value decomposition (SVD) \({\bf A} = \bf U{\bf \Delta}V^{\prime},\) say, concerns any n × m matrix. In this chapter we illustrate the usefulness of the SVD, particularly from the statistical point of view. Surprisingly many statistical methods have connections to the SVD.

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Chapter 20. The Cauchy–Schwarz Inequality

Abstract

As Steele (2004, p. 1) says, there is no doubt that the Cauchy–Schwarz inequality is one of the most widely and most important inequalities in all of mathematics. This chapter gives some examples of its use in statistics; further examples appear in several places in this book. The Cauchy–Schwarz inequality is also known as the Cauchy–Bouniakowsky–Schwarz inequality and is named after Augustin-Louis Cauchy (1789–1857) (see also Philatelic Item 12.1, p. 290), Viktor Yakovlevich Bouniakowsky [Buniakovskii, Bunyakovsky] (1804–1899), and [Karl] Hermann Amandus Schwarz (1843–1921); see Cauchy (1821)1 Bouniakowsky (1859, pp. 3–4), and Schwarz (1888, pp. 343–345), and the book by Steele (2004, Ch. 1).

Simo Puntanen, George P. H. Styan, Jarkko Isotalo

Backmatter