Analyzing complex survey data: the Kauffman Firm Survey

The Kauffman Firm Survey (KFS) was a panel study of new businesses that employed a complex sample design to collect key data about the dynamics of high-technology, medium-technology, and female-owned business entities. Complex sample designs of the type employed in the KFS typically have multi-frame sampling, stratification, non-response adjustment, and over-sampling components. Each of these design elements has been proven to enhance the efficiency with which researchers analyze and draw inferences from the available data. However, there is also a risk that a complex sample design approach can make data analysis more complicated due to non-independent selections and selection with varying probabilities. In this technical overview of the KFS, we describe the sampling method that was utilized in the panel survey. We examine how failing to take into account the probability-based weights impact the parameter estimates and the resulting standard errors. Through adopting an empirical approach, we show why it is important to take account of stratification and weighting. This paper demonstrates the importance of taking the features of a complex survey design into account during the data analysis process.