The failure of models that predict failure: Distance, incentives, and defaults

https://doi.org/10.1016/j.jfineco.2014.09.012Get rights and content

Abstract

Statistical default models, widely used to assess default risk, fail to account for a change in the relations between different variables resulting from an underlying change in agent behavior. We demonstrate this phenomenon using data on securitized subprime mortgages issued in the period 1997–2006. As the level of securitization increases, lenders have an incentive to originate loans that rate high based on characteristics that are reported to investors, even if other unreported variables imply a lower borrower quality. Consistent with this behavior, we find that over time lenders set interest rates only on the basis of variables that are reported to investors, ignoring other credit-relevant information. As a result, among borrowers with similar reported characteristics, over time the set that receives loans becomes worse along the unreported information dimension. This change in lender behavior alters the data generating process by transforming the mapping from observables to loan defaults. To illustrate this effect, we show that the interest rate on a loan becomes a worse predictor of default as securitization increases. Moreover, a statistical default model estimated in a low securitization period breaks down in a high securitization period in a systematic manner: it underpredicts defaults among borrowers for whom soft information is more valuable. Regulations that rely on such models to assess default risk could, therefore, be undermined by the actions of market participants.

Introduction

Statistical predictive models are extensively used in the marketplace by policy makers, regulators, and practitioners to infer the true quality of a loan. Such models are used by regulators to determine capital requirements for banks based on the riskiness of loans issued, rating agencies to predict default rates on underlying collateral, and banks to decide what information they should collect to assess the creditworthiness of borrowers. In each case, the true quality of the loan might not be known for years, so participants in current transactions must rely on some observable features about the loan to assess the quality. For example, a bank regulator could consider the credit scores of borrowers and a collateralized debt obligation (CDO) investor could consider the interest rates on the underlying loans.

These statistical models have come under much scrutiny in the context of the subprime mortgage market, where they were extensively used to forecast the default likelihood of borrowers and of collateral. There has been a public outcry over the failure of rating agency models that estimate the quality of CDO tranches (see Faltin-Traeger, Johnson, and Mayer, 2010, and Griffin and Tang, 2012). In addition, statistical scoring models such as FICO credit scores that assess a subprime borrower׳s default probability and guide lender screening have come under scrutiny.1 Why did statistical default models fare so poorly in the build-up to the subprime crisis? A common answer to this question is that they were undermined by unanticipated movements in the house prices (see, e.g., Brunnermeier, 2009). We argue that this is far from the complete story. Our central thesis is that a primary reason for the poor performance of these predictive models is that they are subject to the classic Lucas critique (Lucas, 1976): They fail to account for a change in the relations between variables when the behavior of agents that influence these relations changes.

We analyze this phenomenon in the context of subprime mortgage loans issued in the US over the period 1997–2006. A notable feature of this period is a progressive increase in the proportion of loans that are securitized. Securitization changes the nature of lending from “originate and hold” to “originate and distribute,” and it increases the distance between a homeowner and the ultimate investor. A loan sale to an investor results in information loss: some characteristics of the borrower that are potentially observable by the originating lender are not transmitted to the final investor.2 Because the price paid by the investors depends only on verifiable information transmitted by the lender, this introduces a moral hazard problem: The lender originates loans that rate high based on the characteristics that affect its compensation, even if the unreported information implies a lower quality. The same tension exists in the multitasking framework of Holmström and Milgrom (1990): An agent compensated for specific tasks ignores other tasks that also affect the payoff of the principal.

In general, the quality of a mortgage loan is a function of both hard and soft information that the lender can obtain about the borrower (see Stein, 2002). Hard information, such as a borrower׳s FICO credit score, is easy to verify; conversely, soft information, such as the borrower׳s future job prospects, is costly to verify (see, e.g., Agarwal and Hauswald, 2010, Liberti and Mian, 2009 on the role of soft information in the context of business lending). In the absence of securitization, a lender internalizes the benefits and costs of acquiring both kinds of information and adequately invests in both tasks. With securitization, hard information is reported to investors; soft information, which is difficult to verify and transmit, remains unreported. Investors, therefore, rely only on hard information to judge the quality of loans. This eliminates the lender׳s incentives to produce soft information.3 Consequently, after a securitization boom, among borrowers with similar hard information characteristics, over time the set that receives loans becomes worse along the soft information dimension. That is, securitization changes the incentives of lenders, and hence their behavior. The result is a change in the relation between the hard information variables (such as the FICO score) and the quality of the loan (such as the likelihood of default). This implies a breakdown in the quality of predictions from default models that use parameters estimated using data from the pre-boom period.

We provide evidence for our thesis by demonstrating three main effects of increasing securitization over time. First, due to the greater distance between originators and investors, the interest rate on new loans depends increasingly on hard information reported to the investor. Second, due to the loss of soft information, the interest rate on a loan becomes an increasingly poor predictor of the likelihood of default on a loan. Third, because the change in lender behavior modifies the relation between observed characteristics of loans and their quality, a statistical model fitted on past data underestimates defaults in a predictable manner—precisely for those borrowers on whom soft information not reported to investors is likely to be important.

Our first result is that the mapping between borrower and loan characteristics and the interest rate on a loan changes with the degree of securitization. In setting the interest rate on a loan, the lender ceases to use information that is not reported to the final investor. Using a large database on securitized subprime loans across different US lenders, we find that over time the interest rate on new loans relies increasingly on a small set of variables. Specifically, the R2 of a regression of interest rates on borrower FICO credit scores and loan-to-value (LTV) ratios increases from 9% for loans issued in the period 1997–2000 to 46% for 2006 loans. Further confirmation comes from the dispersion of interest rates: Conditioning on the FICO score, the standard deviation of interest rates on new loans shrinks over time. Finally, using data from a single large subprime lender, we demonstrate the converse: As securitization increases, interest rates depend less on information observed by the lender but unreported to investors.

Second, we show that with increased securitization the interest rate becomes a worse predictor of default likelihood on a loan. With securitization, there is an information loss, because the lender offers the same interest rate to both good and bad types of borrowers (see Rajan, Seru, and Vig, 2010). As a result, in a high securitization regime, the interest rate becomes a noisier predictor of default for the loan pool. To demonstrate this, we regress actual loan defaults on the interest rate for loans in our main sample, where default is a binary variable considered in a two-year window from the issue date. We find that the pseudo-R2 of this logit regression declines with securitization, confirming that the interest rate loses some of its ability to predict loan defaults.

Third, we show that the change in lender behavior as securitization increases alters the data generating process by transforming the mapping from all observables to loan defaults. We expect that reliance on past data will lead to underprediction of defaults in a high securitization regime, with the underprediction being more severe on borrowers for whom the unreported (or lost) information is more important. These borrowers include those with low FICO scores and high LTV ratios. To illustrate this effect, we estimate a baseline statistical model of default for loans issued in a period with a low degree of securitization (1997–2000), using information reported by the lender to the investor. We show that the model underpredicts defaults on loans issued in a regime with high securitization (2001 onward). The degree of underprediction is progressively more severe as securitization increases, indicating that, for the same observables, the set of borrowers receiving loans worsens over time. Further, we find a systematic variation in the prediction errors, which increase as the borrower׳s FICO score falls and the LTV ratio increases. As a placebo test, we estimate a default model for low-documentation loans over a subset of the low securitization era, and examine its out-of-sample predictions on loans issued in 1999 and 2000 (also a low securitization period). The statistical model performs significantly better than in our main test, and in particular yields prediction errors that are approximately zero on average.

We perform several cross-sectional tests to confirm our results. First, as a direct test of our information channel, we separately consider loans with full documentation and loans with low documentation. More information about a borrower is reported to investors on a full-documentation loan, including information on the borrower׳s income and assets. As a result, we expect that the prediction errors from the default model in the high securitization era should be lower for such loans. This is borne out in the data. Accounting for observables, the prediction errors on low-documentation loans are almost twice those on full-documentation loans during the high securitization regime.

Second, we perform two tests to rule out the concern that our findings on the performance of a statistical default model could be influenced by other macro factors that have changed over time with securitization. In the first test we compare loans securitized in states with foreclosure procedures that are more friendly to lenders with those issued in states with less lender-friendly procedures. Following Pence (2006) and Mian, Sufi, and Trebbi (2011), we compare loans in zip codes that border states with different foreclosure laws to account for both observable and unobservable differences across states. We postulate that lender-friendly foreclosures facilitate the securitization of loans, and we empirically confirm that the number of securitized loans (scaled by households) increases in lender-friendly states over time. Therefore, our expectation is that a statistical default model fitted to historical data should suffer a larger breakdown for loans in such states. This is confirmed by the data. The prediction errors from the default model are greater for loans made in lender-friendly states. Our second test has a similar flavor. We compare low-documentation loans whose borrowers have FICO scores just above 620 (which are easier to securitize; see Keys, Mukherjee, Seru, and Vig, 2010; Keys, Seru, and Vig, 2012) with those whose borrowers have FICO scores just below 620 (which are more difficult to securitize). We find that default prediction errors are higher for loans that are easier to securitize. Overall, these cross-sectional tests strongly corroborate our earlier findings.

Our baseline default model does not include the effects of changes in house prices, so one concern could be that a fall in house prices could lead to high defaults and explain most of the prediction errors in our analysis. It is important to note that several of our empirical strategies suggest otherwise. First, our cross-sectional tests compare loans in the same time period and with similar exposure to house prices. In addition, in the time series, we find that the default model underpredicts errors even in a period in which house prices were increasing (i.e., for loans issued in 2001–2004). Nevertheless, we also consider a stringent specification that both estimates the baseline model over a rolling window and explicitly accounts for the effects of changing house prices. We determine the statewide change in house prices for two years after the loan has been issued and include it as an explanatory variable in the default model (i.e., we assume perfect foresight on the part of regulators estimating the default model). Approximately 50% of the prediction error survives the new specification, and the qualitative results remain: A default model estimated in a low securitization regime continues to systematically underpredict defaults in a high securitization regime.

As long as soft information cannot be contracted upon, a securitizing lender has no incentive to collect it. This statement remains true even if rational investors anticipate higher default rates going forward and price loans accordingly. If investors are boundedly rational and underestimate future defaults, the moral hazard problem with respect to soft information collection is exacerbated. We examine the subordination levels of AAA-rated CDO tranches backed by subprime mortgage loans, and find essentially no relation between the mean prediction errors on defaults and subordination levels. This finding is consistent with rating agencies either being unaware of or choosing to ignore the adverse effects of securitization on the quality of the loan pool over time.

Our work directly implies that regulations based on statistical models can be undermined by the actions of market participants. For instance, the Basel II guidelines assign risk to asset classes relying in part on probability of default models.4 We highlight the role of incentives in determining the riskiness of loans and, in turn, affecting the performance of models used to determine capital requirements. Our findings suggest that a blind reliance on statistical default models results in a failure to assess and regulate risks taken by financial institutions. Indeed, the regulation itself must be flexible enough for regulators to be able to adapt it to changing market circumstances (see Brunnermeier, Crockett, Goddhart, Persaud, and Shin, 2009 for another argument for flexible regulation).

More broadly, we identify a dimension of model risk (i.e., the risk of having an incorrect model) that cannot be corrected by mere application of statistical technique. The term “model risk” is often understood to refer to an incomplete set of data or conceptual errors in a model, or both. The focus in the literature has thus been on testing the consistency and robustness of inputs that go into statistical models. Collecting more historical data, possibly on extreme (and rare) events, is a key corrections that is frequently suggested. However, when incentive effects lead to a change in the underlying regime, the coefficients from a statistical model estimated on past data have no validity going forward, regardless of how sophisticated the model is or how well it fits the prior data. Indeed, aggregating data from different regimes may exacerbate the problem.

Although a naïve regulator might not understand that the lending regime has changed, we expect that rational investors will price loans accurately in either regime. Our hypotheses do not depend in any way on investors being boundedly rational.5 However, if investors too are naïve, prices of loans or CDO tranches will fail to suitably reflect the default risk in a given loan pool. If anything, this exacerbates the tendency of lenders to stop screening borrowers on unreported information, leading to even greater underprediction of defaults. Misestimation of default risk by either regulators or investors could, in turn, lead to a misallocation of capital and a loss of welfare.

Section snippets

Hypothesis development

We start by examining how securitization changes the decision-making process of an originating lender, and thus affects the manner in which the interest rate evolves in our data. A lender has an imperfect screening technology that can generate two sets of observables, Xit and Zit, on loan application i at time t. Here, observation i is a borrower–property pair; that is, the lender can acquire information both about a borrower and the property. Securitization entails the sale of the loan to an

Data

We use two sets of data in our analysis. Here, we describe the primary data set, which comes from LoanPerformance and is used in the bulk of the paper. A second data set consisting of loans from a single lender, New Century Financial Corporation (NCFC), is described in Section 4.3.

Our primary data set contains loan-level information on securitized non-agency mortgage loans. The data include information on issuers, broker dealers, deal underwriters, servicers, master servicers, bond and trust

Evolution of interest rate process: increased reliance on reported information

Our first prediction is that under high securitization interest rates will depend to a greater extent on variables that are reported to the investor. To test this prediction, we examine the evolution of the interest rate process over time. In Section 4.1, we consider our main sample. First, we directly regress the interest rate on a loan on the LTV ratio and the FICO score of the borrower. We predict that the explanatory power of the right-hand-side variables (i.e., the R2 of the regression)

Evolution of default process

We now consider the effect of securitization on mortgage defaults. Following the arguments in Section 2, we have two predictions on the default rates of loans. First, the ability of the interest rate to predict defaults should fall over time as information not being reported to the investor is no longer collected by the lender. Thus, in a year-by-year regression of default rates on interest rates, the R2 should decrease over time. To test this prediction, we directly consider the evolution of

Cross-sectional tests

We now describe several cross-sectional tests that both confirm our findings and alleviate the concern that some of our results on prediction errors could be due to macro factors other than securitization levels that also changed over time.

Alternative hypotheses

An important alternative hypothesis is that our finding of a positive prediction error in default models is driven by falling house prices. We provide three pieces of evidence to rule this out. First, in each of our cross-sectional tests, the two sets of loans being compared are subject to the same effects of changing house prices. Therefore, changing house prices cannot explain the differences across the loans being compared in each case. Second, in Fig. 1, we show positive prediction errors

Role of investors

The boom in securitization of subprime mortgage loans over our sample period was possible only because investors showed a continued and increasing willingness to buy these loans. Our analysis shows that a naïve regulator relying only on past data would underestimate loan defaults. Were investors better able to forecast defaults?

Our analysis is largely agnostic on whether investors were also fooled by the change in the lending regime. Importantly, our predictions obtain even when both lenders

Conclusion

Establishing a liquid market for a complicated security requires standardization of not just the terms of the security, but also of the fundamental valuation model for the security, both of which help investors to better understand the security. Inevitably, the process of constructing and validating a model includes testing it against previous data. We argue in this paper that the growth of the secondary market for a security can have an important incentive effect that affects the quality of

References (47)

  • M. Brunnermeier

    Deciphering the liquidity and credit crunch 2007–2008

    Journal of Economic Perspectives

    (2009)
  • M. Brunnermeier et al.

    Geneva Reports on the World Economy 11: The Fundamental Principles of Financial Regulation

    (2009)
  • Calomiris, C., 2009. The debasement of ratings: What׳s wrong and how we can fix it. Unpublished working paper. Columbia...
  • R. Cole et al.

    Cookie-cutter versus characterthe micro structure of small business lending by large and small banks

    Journal of Financial and Quantitative Analysis

    (2004)
  • J. Coval et al.

    Economic catastrophe bonds

    American Economic Review

    (2009)
  • Y. Demyanyk et al.

    Understanding the subprime mortgage crisis

    Review of Financial Studies

    (2011)
  • O. Faltin-Traeger et al.

    Issuer credit quality and the price of asset-backed securities

    American Economic Review

    (2010)
  • H. Fishelson-Holstein

    Credit scoring role in increasing homeownership for underserved populations

  • K. Gerardi et al.

    Making sense of the subprime crisis

    Brookings Papers on Economic Activity

    (2008)
  • E. Gramlich

    Subprime Mortgages: America׳s Latest Boom and Bust

    (2007)
  • Greenspan, A., 2008. Testimony before House Committee of Government Oversight and Reform. October...
  • J. Griffin et al.

    Did subjectivity play a role in CDO credit ratings?

    Journal of Finance

    (2012)
  • J. Heckman

    Varieties of selection bias

    American Economic Review

    (1980)
  • Cited by (230)

    • Climate change concerns and mortgage lending

      2024, Journal of Empirical Finance
    View all citing articles on Scopus

    For helpful comments and discussions, we thank numerous individuals, including two anonymous referees, as well as participants at seminars at Bank of London, University of California at Berkeley, Federal Reserve Board of Governors, Brigham Young University, Federal Reserve Bank of Chicago, Columbia University, Harvard University, Houston, London School of Economics, University of Michigan, Michigan State University, MIT Sloan School of Management, New York University Stern School of Business, University Naples Federico II, Federal Reserve Bank of Philadelphia, Stanford University, University of California at Los Angeles, University of Utah and at the American Economic Association, American Law and Economics Association, Rothschild Caesarea Center, European Finance Association, Financial Intermediation Research Society, Freiburg, Indian School of Business, London Business School and London School of Economics Credit Risk, National Bureau of Economic Research (NBER) Behavioral, NBER Summer Institute, Southwind Finance and Western Finance Association conferences. We are also indebted to Tanmoy Mukherjee for extensive discussions. All errors are our responsibility. Amit Seru thanks the Initiative on Global Markets at the University of Chicago for financial support. Vikrant Vig acknowledges the support provided by the Research and Materials Development grant at the London Business School. Part of this work was undertaken when Amit Seru was a visiting scholar at Sorin Capital Management.

    View full text