What makes a critic tick? Connected authors and the determinants of book reviews

https://doi.org/10.1016/j.jebo.2013.08.008Get rights and content

Highlights

  • We use a novel data set to test the incentives of media outlets.

  • An author's connections are correlated with review outcomes.

  • Media outlets cater to the tastes of readers.

  • Experts and consumers tend to agree in aggregate about the quality of a book.

  • However, professional critics are less favorable to first-time authors.

Abstract

This paper investigates the determinants of expert reviews in the book industry. Reviews are determined not only by the quality of the product but also by the incentives of the media outlet providing the review. For example, a media outlet may have the incentive to provide favorable coverage to certain authors or to slant reviews toward the horizontal preferences of certain readers. Empirically, we find that an author's connection to the media outlet is related to the outcome of the review decision. When a book's author also writes for a media outlet, that outlet is 25% more likely to review the book relative to other media outlets, and the resulting ratings are roughly 5% higher. Prima facie, it is unclear whether media outlets are favoring their own authors because these are the authors that their readers prefer or simply because they are trying to collude. We provide a test to distinguish between these two potential mechanisms, and present evidence that this is because of tastes rather than collusion – the effect of connections is present both for authors who began writing for a media outlet before and after the book release. We then investigate other determinants of expert reviews. Both consumers and expert reviewers seem to favor authors who have won book prizes. Yet relative to consumer reviews, professional critics are less favorable to first time authors and more favorable to authors who have garnered other attention in the press (as measured by number of media mentions outside of the review).

Introduction

Expert reviews published in media outlets pervade our everyday lives. For decisions ranging from choosing a restaurant or a book to planning a vacation or buying a car, media outlets provide much of the information we use. A growing empirical literature has shown that these reviews are influential in a variety of markets (Reinstein and Snyder, 2005, Boatwright et al., 2007, Berger et al., 2010). Given this widespread influence, it is critical to understand the content of such reviews; yet little empirical evidence exists. This paper addresses this issue by providing empirical evidence on the determinants of expert reviews in the book industry.

In practice, expert reviews are determined not only by the quality of the product, or even by the tastes of the reviewer, but also by the incentives of the media outlet providing the review. A media outlet may not always have the incentive to provide objective reviews, such as in cases when there is a relationship between the reviewer and the maker of the product. For example, professional critics often write reviews for books of authors to whom they have a connection, such as a colleague at that newspaper.

Connections between a media outlet and an author can undermine the editorial integrity of the process by giving rise to collusion and favorable treatment. Theoretically, collusive incentives may be reduced through careful mechanism design (Laffont and Tirole, 1991, Laffont and Martimort, 1997, Motta, 2012). Even in the absence of collusion, these types of connections can give rise to favorable treatment. An experimental literature has shown that subjects tend to give more in dictator games when they are connected to the recipient (Charness and Gneezy, 2008). Hence, in equilibrium, the impact of connections becomes an empirical question.

In this paper, we investigate the role of these connections between authors and book reviewers. Specifically, we consider the fact that many non-fiction authors have also served as journalists or opinion writers for newspapers. We present robust evidence that these connections matter. For this analysis, we construct a new data set consisting of the 100 highest rated non-fiction books1 on Metacritic.com between 2004 and 2007. To identify a fixed set of recognized expert reviewers, we use the same set of media outlets (magazines and newspapers) that Metacritic uses to identify experts. To identify affiliated authors who are connected with these periodicals, we use a variety of sources (including author biographies in each of these 100 books) to identify which media outlets (if any) each author has worked for.

Our analysis shows that a media outlet is more likely to review a book written by a connected author, and the resulting review is more favorable on average. We begin by regressing a book's likelihood of being reviewed by a given periodical on an indicator for whether the author is connected to that periodical. This may be problematic if connected authors write better books. To control for this, we include book specific fixed effects. Now, the regression is measuring whether an author is more likely to be reviewed by periodicals to which the author is connected. To clarify our identification strategy, consider two authors – one works for the New York Times while the other works for the Wall Street Journal. Our evidence is essentially that the New York Times author tends to receive more favorable reviews from the New York Times, while the Wall Street Journal author tends to receive more favorable reviews from the Wall Street Journal. Overall, we find that being connected to a media outlet increases an author's likelihood of being reviewed by roughly 25%.

Should managers and policymakers be concerned that the connection between an evaluator and the creator of the product is so important? They should be concerned if this is driven by collusion. Yet there are other reasons why connections might matter. Specifically, different media outlets may have different tastes in books. In our setting, the New York Times has a very different audience than the Village Voice. Further, a book by a New York Times journalist may be preferred by New York Times readers relative to readers of the Village Voice. Hence, even if the authors are reviewed more often by media outlets to which they are connected, it need not be collusion.

We propose a test to distinguish between mechanisms, by exploiting the timing of the connection. Some of the authors wrote the book before writing for a given media outlet, while others wrote the book after becoming connected to a certain media outlet. If the link between being reviewed and writing for a media outlet was due to an effort by the paper to help its authors, then only authors who wrote for the media outlet before the book came out should benefit. However, we find that the likelihood of being reviewed by a given media outlet is statistically indistinguishable between the two groups. While we cannot truly reject that there is collusion, the data suggest that horizontal differentiation plays an important role in the expert review process.

In addition to tastes and bias, there is a third potential interaction between evaluations and connections. Specifically, the New York Times may know more about the work of their authors than about other authors, and therefore have a more precise signal of the quality of their books. For instance, in the NIH grant setting, Li (2012) finds that evaluators are better able to assess the quality of work of connected applicants. We test for this, but do not find conclusive evidence that this is the case.

The policy implications of these alternative mechanisms are very different. A collusion-driven preferential treatment is tantamount to providing readers with misleading information, something that policymakers should be concerned about. Just how concerning collusion is will depend on readers’ access to multiple media outlets, and their ability to rationally anticipate collusion and adjust their beliefs accordingly. On the other hand, if the differential treatment reflects horizontal differentiation, with media outlets optimally providing information that caters to the preferences of their respective audiences, then there is no inefficiency. Finally, the third mechanism has yet different policy implications. If critics are better able to assess the quality of work of connected authors, oversampling these authors could be welfare improving from the readers’ standpoint. Given how sensitive policy implications are to the presence of alternative mechanisms, it is crucial to determine which ones are active. Ours is an exploratory attempt to disentangle these channels and analyze their relative importance.

Finally, we also consider the relationship between expert and consumer reviews. Both expert and consumer reviews are ultimately written to inform consumers, and hence one might expect the two to be very similar. At the same time, these two types of information are generated in very different ways. Expert reviews are horizontally differentiated (as shown in this paper) and are written by professional critics who may be able to more accurately assess the quality of a book. Using their training and experience, they can assign an instant judgment on a book's quality and place it in context (with respect to the past and current trends) for the readers. On the other hand, there is virtually no quality assurance in consumer reviews. For instance, these reviews can be gamed by the producers or the competitors of a product by submitting fake reviews (Luca and Zervas, 2013, Mayzlin et al., 2012). Further, consumers who choose to leave a review may not reflect the underlying population. Hence, the link between expert and consumer reviews is a priori unclear.

Empirically, we find that there is no correlation between the likelihood of being reviewed by an expert and a book's Amazon rating. Yet, we find that expert ratings are correlated with Amazon ratings, and so, conditional on leaving a review, experts and consumers agree in aggregate about the quality of a book.

Expert reviews and consumer reviews also systematically diverge at times. Relative to the consumer opinion as measured by Amazon reviews, we find that professional critics are less favorable to first time authors and more favorable to authors who have garnered other attention in the press (as measured by number of media mentions outside of the review). However, both consumers and expert reviewers seem to favor authors who have won book prizes.

Overall, our paper contributes to and connects two different literatures. First, there is a related literature on conflict of interest in the media. Reuter and Zitzewitz (2006) investigate the link between advertisements in media outlets and editorial coverage, and show that some publications provide favorable editorial coverage to advertisers. In work that is concurrent to but independent from ours, DellaVigna and Kennedy (2011) present evidence that media outlets tend to accord more favorable treatment to movies produced by affiliated (connected) studios. They also decompose the possible bias by reviewer and find evidence of reviewer bias, but no evidence of explicit editorial bias (i.e., dismissal of reviewers who are too independent, and assignment of movies to different reviewers). Their work is thus complementary to ours – both show that connections matter. However, their definition of bias is slightly different, something that is largely due to the different type of connections considered. In particular, what we refer to as horizontal differentiation, they simply refer to as bias that is “shared with readers.” Despite the differing definitions, our results are largely consistent. Using a smaller and earlier data set, Rossman (2011) presents evidence that media outlets are not biased toward movies produced by their parent companies.

Our paper also contributes to the literature on reviews. For example, in a recent paper, Sun (2012) argues that the patterns of consumer reviews on Amazon.com and BarnesAndNoble.com are consistent with a model where variations in reviews are driven by horizontal preferences. Ours is among the first to investigate the determinants of reviews.

Section snippets

The review process

Newspapers and magazines are the primary outlet for expert book reviews. A typical newspaper can review several books per week, or a few hundred per year. With more than 85,000 books published per year in the United States alone,2 it is clear that there are many more books than can possibly be reviewed.

The book reviewing process varies between publications, but typically any author can submit his or her book for review. At any given

Books

A book is characterized by vertical quality q and a set of attributes A = {a1, …, an}. These attributes capture a variety of features including whether a book is well-written, funny, or socially relevant.

Media outlets

Media outlet j has preferences over attributes A, represented by weights Λj = {α1j, …, αnj}, with k=1nαkj=1. The set Λj can be thought of as reviewer tastes, or as the reviewer's preference to cater to a certain audience. Essentially, this allows some reviewers to prefer biographies, while others

Determinants of reviews

We investigate the determinants of expert reviews in two different ways. First, we investigate the variation of reviews across different media outlets. Second, we investigate the differences between expert and consumer reviews.

The estimating equation isReviewij=Bookcharcteristicsi+Authorcharcteristicsi+Mediaconnectionsij,for all reviews of book i at media outlet j.

All specifications include media outlet fixed effects and some specifications also include book fixed effects. We use logit

Discussion

In this paper we look at the determinants of expert reviews in the book industry. We find that an author's connection to the media outlet is related to the outcome of the review decision. When a book author also writes for a media outlet, that outlet is 25% more likely to review the book relative to other media outlets, and the resulting ratings are roughly 5% higher. We investigate a number of alternative mechanisms that could explain this bias. Media outlets might favor their own authors

Acknowledgements

We would like to thank two anonymous referees for very helpful comments and suggestions. We are also grateful to Dimitris Christelis, Jerry Green, Dara Lee, Dilip Mookherjee, Daniele Paserman, Pian Shu, and Jonathan Smith for useful comments and suggestions. All errors remain our own.

References (27)

  • G. Charness et al.

    What's in a name? Anonymity and social distance in dictator and ultimatum games

    Journal of Economic Behavior & Organization

    (2008)
  • J. Berger et al.

    Positive effects of negative publicity: when negative reviews increase sales

    Marketing Science

    (2010)
  • P. Boatwright et al.

    Reviewing the reviewers: the impact of individual film critics on box office performance

    Quantitative Market Economics

    (2007)
  • B. Calame

    The Book Review: Who Critiques Whom and Why?

  • C. Cameron et al.

    Robust inference with multiway clustering

    Journal of Business and Economic Statistics

    (2011)
  • DellaVigna, S., Kennedy, A., 2011. Does media concentration lead to biased coverage? Evidence from movie reviews....
  • Fowder, L., Kadiyali, V., Prince, J., 2011. Racial bias in quality assessment: a study of newspaper movie reviews....
  • J. Heckman

    Shadow prices, market wages, and labor supply

    Econometrica

    (1974)
  • J. Heckman

    Sample selection bias as a specification error

    Econometrica

    (1979)
  • M.B. Holbrook

    Popular appeal versus expert judgments of motion pictures

    Journal of Consumer Research

    (1999)
  • J. Krames

    When is the Best Time to Publish a Business Book?

  • J.J. Laffont et al.

    Collusion under asymmetric information

    Econometrica

    (1997)
  • J.J. Laffont et al.

    The politics of government decision-making: a theory of regulatory capture

    Quarterly Journal of Economics

    (1991)
  • Cited by (17)

    • Bias in expert product reviews

      2022, Journal of Economic Behavior and Organization
      Citation Excerpt :

      For reviews of video games in a major games magazine owned by a game retailer, Hamami and Bailey (2021) find evidence of bias. Dobrescu et al. (2013) find that the tendency of media outlets to review books written by authors that also write for the outlet is likely to be due to taste rather than collusion. Further exceptions are Ravid et al. (2006) who find that movie critics are biased towards certain production companies, and Chossat and Gergaud (2003) who find that some gourmet restaurants heavily invest in luxurious surroundings to manipulate their rating in gastronomic guides.

    • Biases in fiscal multiplier estimates

      2020, European Journal of Political Economy
      Citation Excerpt :

      Their results suggest the absence of any bias. With a similar focus but arriving at the opposite conclusion, Dobrescu et al. (2013) test the independence of book reviews when the author is connected to a media outlet. Li (2017) scrutinizes the interdependency of experts' conflict of interest and the quality of their judgment in the context of peer review at the National Institute of Health.

    • Product success in cultural markets: The mediating role of familiarity, peers, and experts

      2015, Poetics
      Citation Excerpt :

      Also, differences of review scores are minimal and insignificant for both newcomers and incumbents. Still, one might expect filtering decisions to vary with book characteristics resulting, most plausibly, in more frequent reviews for promising books (Dobrescu et al., 2013; Reinstein and Snyder, 2005). Indeed, covered books hold Top 50 positions some 1.5-ranks higher on average than non-reviewed books in the week before a review's publication (p = .087).

    • Digital trust and cooperation with an integrative digital social contract

      2022, Business and the Ethical Implications of Technology
    • Effects of COVID-19 on Critics’ Rating Behavior

      2022, Lecture Notes in Business Information Processing
    View all citing articles on Scopus
    View full text