Measuring employment during COVID-19: challenges and opportunities

Abstract

Measuring the economy during COVID-19 has created a set of hurdles and prospects for the U.S. statistical agencies, the private sector, and academia. This included how to deal with unprecedented movements in economic variables while ensuring data quality. At the same time, new entrants released data that are sometimes at odds with gold standard releases. Given the multitude of data, this article focuses on both existing and new ways to track the employment statistics. The results leave more questions than answers.

Measuring the economy during COVID-19 has been fraught with both challenges and opportunities for U.S. statistical agencies, researchers, and new players—such as Homebase—to provide accurate and valuable low- and high-frequency economic data. The statistical agencies dealt with lower than normal response rates, delays in processing, adding new measures, questions about whether responses were being classified correctly, and whether seasonal adjustments were appropriate. Meanwhile, new players who had never before provided macroeconomic data released statistics. Unfortunately, in the vast majority of cases, the time series is extremely limited—often beginning in January 2020—which means the value of the data cannot be assessed. At this point, it is too early to know if any of these statistics reflect economic realities. We will only know over the next several years when the full extent of the recession and recovery is correctly measured based on full tax and unemployment insurance data.

Given the host of data additions from the public and private sectors as well as academia, this paper focuses on regularly released statistics that are constructed to provide timely insight into the monthly employment report, and issues related to that report itself.¹ This paper is organized as follows. First, it analyzes the impact of COVID on traditional labor market statistics: jobless claims and the employment report. Second, it studies the newly developed Household Pulse Survey, Dallas Fed Real-Time Employment Survey, and Homebase timesheet tracking data. The paper concludes by offering some caution about interpreting both traditional and new data sources.

1 Jobless Claims

The March 19 jobless claims report likely provided the first hard data illustrating the impact of COVID-19 on the U.S. labor market. At that time, jobless claims increased 70,000 to what now seems a mere 281,000. This was the highest level of claims since September 2017. At the top of the release (https://oui.doleta.gov/press/2020/031920.pdf) the Department of Labor (DoL) noted in a box the impact of COVID on jobless claims for the week ending March 14. The focus of the box was to discuss the reason for the jump and affected industries (in particular accommodation and food services).

As COVID-related claims soared into unprecedented territory—the 3.3 million initial claims in the week ending March 21 far surpassed the previous record of 695,000 from 1982, both on a level basis, and adjusted for the size of the labor force—questions arose about the impact of the seasonal factors used by the DoL. March through June is a seasonally low period of claims; thus, seasonal factors tend to amplify seasonally adjusted claims. Figure 1 illustrates how seasonally adjusted initial jobless claims (black bars) were meaningfully higher than not seasonally adjusted data (gray bars) during the spring. That gap reached 851,000 in the week ending March 28, when seasonally adjusted claims hit a record 6.9 million. Put another way, the boost from seasonal factors in just that week surpassed the 1982 record in claims. As a result, some analysts focused on not seasonally adjusted claims, which were still at unprecedented levels.

Fig. 1

Initial claims, state programs (Millions)

Traditionally, DoL used multiplicative seasonal factors, which assume the seasonal effect is proportional to the level of the series. This means that a sudden large increase in the level of the series will be accompanied by a proportionally large seasonal effect. Normally, these seasonal impacts should wash out over the course of the year. However, the jump in claims in the spring was not normal, raising the question of whether claims would remain high enough during the seasonally high period of claims—early summer and November through February—to offset the boost in the spring.

On August 27, DoL acknowledged that their multiplicative factors may not be appropriate at this time (https://oui.doleta.gov/press/2020/082720.pdf). As a result, DoL has switched to additive seasonal factors, which are not affected by the level of claims. Unfortunately, as of this writing, DoL is only using additive factors, starting with September 3, 2020 release. This further complicates matters because it created a break in the seasonally adjusted initial claims series between the weeks of August 22 and August 29.² This has led even more analysts to focus on the not seasonally adjusted data.

Haver Analytics has calculated a full history of claims using additive seasonal factors. This calculation indicates that instead of adding 851,000 in the week ending March 28th, additive claims would’ve added 30,000 to the initial claimant count (Fig. 2, vertical line is the late August break for DoL data).³ DoL stated that “in accordance with the usual practice, the seasonal adjustment models and factors will be reviewed at the beginning of each calendar year, when prior years of seasonally adjusted estimates will be subject to revision.” Thus, in the fullness of time, the DoL seasonally adjusted history may be revised meaningfully.

Fig. 2

State initial claims, alternative seasonal adjustments (Millions)

Not only did DoL face the reporting challenges of regular/state claims, where states were already overwhelmed by filers, but DoL had to handle adding two new very large programs after the CARES act was passed. Pandemic Unemployment Assistance (PUA) covers individuals such as the self-employed who are not qualified for regular/state unemployment insurance. Pandemic Emergency Unemployment Compensation (PEUC) covers people who were unemployed before COVID but exhausted their state benefits and are now eligible to receive an additional 13 weeks of unemployment insurance, up to a total of 39 weeks. While PUA initial claims (black line in Fig. 3) did not reach the peak levels of regular/state claims, they quickly rose to over 1 million per week and are currently almost the same level as regular/state claims (gray line).⁴

Fig. 3

Pandemic unemployment assistance and state initial claims (NSA, Millions)

Unfortunately, because of problems with states handling and verifying PUA initial claims filings, the first reported data for those claims do not reflect the reality as we currently see it.⁵ Moreover, it’s not clear that the weekly pattern in those statistics today reflect actual layoffs. For example, the state of New Jersey currently exhibits the following pattern for initial PUA claims: 44,300 in the week ending April 18, zero in the next week, and 76,100 in the week ending May 2. Compounding this problem even further were some communications issues between state and federal labor departments. On May 14, the Connecticut Department of Labor tweeted that the federal DoL incorrectly reported 298,680 initial regular/state claims for Connecticut when the correct number was 29,846 (https://twitter.com/CTDOL/status/1260962610729619456). DoL decided not to immediately revise the data, but waited until the next release on May 21 when they revised down total U.S. initial claims by 294,000, the vast majority of which was accounted for by Connecticut.

While the jump in PUA initial claims was less dramatic than initial claims, the type of work embodied by those different programs (gig versus more standard jobs) has likely led to different recall dynamics. Thus, regular/state continuing claims have fallen substantially more than PUA claims (Fig. 4). And at this point (late September) the number of PUA continuing claims is roughly the same as state claims, despite the fact that the number of gig workers is substantially smaller than the number of workers eligible for state claims.⁶

Fig. 4

Continuing claims (NSA, Millions)

Unfortunately, because of similar issues of ramp-up with initial claims, there has been some evidence that as a result of the backlog of claims, some states have at times reported the number of weeks of continuing claims paid rather than the number of people receiving claims.⁷ For example, continuing PUA claims jumped 29% in Michigan in the week ending May 18 and then more than reversed that decline the next week. Some, but not all states have shown a similar pattern around mid-May. Finally, there are reports of fraud substantially overstating the number of PUA claims, with some citing the jump in claims in California in August from 3 to 7 million (Casselman et al. 2020). While much of the August increase was reversed, suggesting that some of the increase was the backlog issue discussed above, California ultimately took a two week pause in its processing of initial claims to help the state tackle its backlog, and implement fraud prevention technology (https://oui.doleta.gov/press/2020/100120.pdf).

2 Employment report

Since the release of the March employment report the Bureau of Labor Statistics (BLS) has released a supplemental report on the impact of COVID-19 on both the Establishment (payroll) and Household (unemployment) surveys. The most notable, though not surprising, impact of COVID has been a change in data collection—from telephone to web-based forms in the case of the establishment survey, and from some in-person to all telephone interviews for the household survey (https://www.bls.gov/covid19/employment-situation-covid19-faq-august-2020.htm). Collection rates for the first release of the establishment survey were meaningfully lower. In March, May, and June collection rates were roughly 65%, but have now rebounded to around 77%, which is above the 75% average in the 12 months ending in February 2020 (https://www.bls.gov/web/empsit/cesregrec.htm). Interestingly, the third/final release collection rates for March, May and June were much closer to normal.⁸ Despite this, even in June, the worst month, BLS stated that “although the collection rates were adversely affected by pandemic-related issues, BLS was still able to obtain estimates that met our standards for accuracy and reliability.”

Unfortunately, the response rate for the household survey also declined meaningfully during the COVID period. From March through August the total response rate was 10 to 18 percentage points below the 83% 12-month average in the period ending February 2020. The response rate for households entering the sample for their first month—which is typically an in-person interview—was down 20 to 33 percentage points from the normal 80%. BLS noted that a low response rate in the first month has a direct impact on the response rate for the next seven months that households are in the sample. Thus, response rates are likely to remain well below average for the foreseeable future. The last few months have shown some improvement in response rates and August—the latest report available at this writing—had a total response rate of 70%, up from the 65% nadir in June.⁹

Even during the lowest response month of June, BLS stated the “although the response rate was adversely affected by pandemic-related issues, BLS was still able to obtain estimates that met our standards for accuracy and reliability.” Despite this, some analysts have questioned the results, especially the very surprising drop in the unemployment rate to 8.4% in August from 10.2% in July. While the decline in the unemployment rate is for the ‘right’ reasons—job creation, a decline in the ranks of the unemployed to 13.6 million, and increased participation in the labor force—the labor force is still down an unprecedented 3.7 million since February.

Figure 5 depicts the relative movement of the labor force in the twelve months following the peak of each of the last eight economic cycles. It shows that with the exception of the current recession (peak in February 2020—solid black line) the labor force has not meaningfully declined during previous post-war recessions.¹⁰ This suggests the ranks of the unemployed may be higher than has been reported by BLS. Further, research such as Ahn and Hamilton (2020), statistics such as the 29.2 million continuing jobless claims reported by the DoL during the August household survey week, and other measures of employment such as the Dallas Fed’s Real-Time Population unemployment measure, which will be discussed below, suggests a meaningfully higher unemployment rate.

Fig. 5

Labor force (Peak = 100)

Another significant challenge faced by the BLS was a potential misclassification of workers as employed but not working for ‘other reasons,’ rather than unemployed. The household survey classifies people as ‘unemployed on temporary layoff’ if they are furloughed/temporarily laid off and expecting recall. That group numbered 6.2 million in August, down from 9.4 million in July and a peak of 18.1 million in April. Workers who are on vacation, ill, or absent for ‘other reasons’ whether paid or unpaid, are classified as ‘employed but absent from work’. There were 2.0 million workers in August who were considered employed but absent from work for ‘other reasons.’ That is down from 2.5 million in July and a peak of 8.1 million in April.

However, even in normal circumstances there are employed workers who absent from work for ‘other reasons.’ In the previous four years, that number averaged 931,000 in August. Taking the difference between the current number of ‘other reasons’ and the four-year average yields the BLS estimate of 1.1 million misclassified workers. This is down from 1.3 million in July and a peak of 7.5 million in April. Adjusting the standard (U-3) unemployment rate for the potential misclassification yields an adjusted unemployment rate of 9.1% in August, which is meaningfully lower than 11.0% reading in July and a peak of 19.5% in April (black line in Fig. 6).

Fig. 6

Unemployment rate (SA, %)

As Fig. 6 illustrates, the potential misclassification rate has fallen substantially since April. This is partly because of a decrease in the overall level of unemployed. However, the decline is also the result of efforts put in place by the Census Bureau, which conducts the household survey, to better identify workers who were unemployed. While Census provided special guidance prior to the March survey, continued high rates of misclassification in the spring led to increased training, culminating in newly developed computer-based training prior to July data collection (https://www.bls.gov/covid19/effects-of-covid-19-pandemic-and-response-on-the-employment-situation-news-release.htm#ques6). The Bureau’s review of survey responses suggests that the degree of misclassification in July and August is substantially lower than previous months, which is illustrated by the narrowing gap in Fig. 6.

Some analysts argued that the BLS should have changed the survey responses to correct for the misclassification. However, the BLS has stated “according to usual practice, the data from the household survey are accepted as recorded. To maintain data integrity, no ad hoc actions are taken to reassign survey responses." Also, the BLS notes that the exact extent of the misclassification is unknown; however, the current 1.1 million represents the upper bound of the estimate of misclassification.¹¹

In addition to dealing with response-related issues, like many statistical agencies, since the May survey the BLS has added five specific questions to the household survey regarding the impact of COVID on employment. The responses to these questions revealed that in August 24.3% of people teleworked because of the disease, down from 26.4% in July; 24.2 million workers’ employment status or hours were affected by COVID, versus 31.3 million in July; and 5.2 million people were prevented from looking for a job because of COVID, down from 6.5 million in July (https://www.bls.gov/cps/effects-of-the-coronavirus-covid-19-pandemic.htm).

3 Household pulse survey

In April, the Census Bureau, in partnership with the BLS, the National Center for Health Statistics, the United States Department of Agriculture’s Economic Research Service, the National Center for Education Statistics, and the Department of Housing and Urban Development, initiated the Household Pulse Survey (HPS) to “quickly and efficiently deploy data collected on how people’s lives have been impacted by the coronavirus.” HPS asks individuals about their experiences regarding employment status, food security, housing, physical and mental health, access to health care, and educational disruption.

The HPS utilizes the Census Bureau’s Master Address File—which is the gold standard frame for U.S. statistics—as the source to select a very large sample. Census noted it is sufficient in size to accommodate anticipated low response rates and still produce estimates at the state level as well as for 15 Metropolitan Statistical Areas (MSAs). To enable rapid deployment, households were contacted by email and/or text. After a testing period starting on March 23, formal weekly data collection for Phase 1 of the Household Pulse Survey began on April 23, 2020. The data are released eight days after the end of the survey period. Thus the HPS for the July 9–14 week, which overlaps with the household employment survey, was available on July 22, 16 days before the July employment report was released.¹²

Unfortunately, Phase 1 of the HPS ended on July 21, 2020 when initial funding ran out. Phase 2 carries over many of these questions and includes additional content regarding receipt of benefits, spending patterns, availability of financial resources, post-secondary education disruptions, capacity to telework, and travel practices with data released biweekly starting in August 19, and is currently funded through October 2020.¹³

The line in Fig. 7 depicts the HPS estimate of the number of people employed in the last seven days. The bars are civilian employment for 18 year olds and over as measured by the household survey, which reflects estimates for the week including the 12th of the month, generally corresponding to the second observation of the month for the HPS.¹⁴ While the level of employment indicated by the HPS is meaningfully lower than that found in the household survey—likely reflecting differences in survey methodology—HPS seems to be informative about trends in employment, though it suggests more month-to-month volatility than the standard household survey.¹⁵

Fig. 7

Employment: 18 years and over (NSA, Millions)

4 Dallas fed real-time population survey

The Real-Time Population Survey (RPS) is a biweekly online survey of adults, which is meant to provide an early read on the household survey. The RPS was developed by Blick and Blandin (2020) and was initially conducted in late March and early April 2020. The authors partnered with the Federal Reserve Bank of Dallas on the survey beginning in May 2020. Data are released six days after the survey week. Thus, the RPS for the week of July 12–19, which overlaps with the July household survey, was available on July 24, two weeks before the employment report was released (https://www.dallasfed.org/research/rps.aspx).¹⁶

The core of the RPS closely follows the main labor market questions in the household survey. In addition to employment status, RPS asks several more questions of employed workers, including type of employer, industry, and hours of work. The survey is run by online by Qualtrics, with a sample—which is selected to be representative of the U.S. population—of between 1600 and 2000, who receive a modest compensation. Figure 8 depicts the RPS unemployment rate (solid black line) with the 95% confidence levels (dashed and dotted black lines) as well as the standard (U-3) unemployment rate (solid gray line) and the misclassification-adjusted rate (dashed gray line) discussed earlier. While the decline in the unemployment rate is generally matched by the RPS, the gap between the series has not narrowed, and in fact widened following the August unemployment data. Blick and Blandin believe this gap is related to the measurement challenges discussed in Ahn and Hamiltion (2020), that may lead to a sizeable downward bias in the official unemployment rate.

Fig. 8

Alternative measures of the unemployment rate (%)

5 Homebase

Homebase is a free scheduling and time tracking tool used by over 100,000 local businesses and their hourly employees. Homebase customers in the U.S.primarily consist of restaurants, food and beverage vendors, retailers, as well as other services such as healthcare and fitness. They are largely individually owned/operator-managed businesses. In late March Homebase started providing daily data statistics on the volume of hours worked by employees, the number of businesses open and the number of employees working (https://joinhomebase.com/data/). Each statistic was based on use of Homebase timecards, with business openings defined as having on at least one employee clocking in. Each of the statistics compared the measure of hours worked on that day to the average for that day of the week for the period January 4–31, 2020. Thus, these might show the extent to which COVID-19 has impacted Homebase users.¹⁷ However, since January is a seasonally low period in employment and other measures of economic activity, that choice of baseline is potentially problematic over the medium term.

Hombase data are available on a national level—depicted below—as well as by MSA and industry. The solid line in Fig. 9 illustrates that Homebase hours data have intra-week seasonality, and most notably, strong seasonality associated with holidays, with big dips and recoveries around Easter, Memorial Day, July 4th and Labor Day.¹⁸ Not surprisingly, Homebase Food and Drink and, in particular, Leisure and Entertainment measures, show much smaller holiday declines. Figure 9 also indicates there is a relationship between Homebase hours worked and the total index of aggregate weekly hours in the payroll employment report (dashed line) and more strikingly the hours worked in the Leisure and Hospitality Sector (dotted line).¹⁹

Fig. 9

Hours worked (Percentage Point Difference from January)

6 More questions than answers

The U.S. statistical agencies, researchers, and the private sector have supplied a plethora of data to facilitate real-time analysis of the economic impact of COVID-19. This survey just scratches the surface of the data available and the challenges faced by both the statistical agencies and new data sources. Many more questions than answers remain.

For example, how could it be that the ranks of unemployed dropped from 23.1 million in April to 13.6 million in August, while at the same time the total number of continuing jobless claims in all programs rose from 19.0 to 29.2 million? How much were continuing claims undercounted in April because of reporting challenges, and how much were later readings overstated by fraud? Has civilian employment been on a steady increase or has it been more volatile, as suggested by the Household Pulse or the Real-Time Population surveys? Have overall hours worked fallen more than the BLS numbers indicate, as suggested by the Homebase data? And, as with all data, how much weight should we be putting on a particular set of observations, especially when the data has been untested or has potential flaws? Many of these questions will be answered in time, but that does not help business economists or policy makers make decisions in close to real time.