How does qualitative data collection modality affect disclosure of sensitive information and participant experience? Findings from a quasi-experimental study

Prior to the COVID-19 pandemic, online focus groups (FGs) and individual interviews (IDIs) were being used with increasing frequency across multiple disciplines (Braun et al. 2017; Salmons 2016), a trend that the pandemic has accelerated and mainstreamed, at least for the near-term (Saarijärvi and Bratt 2021; Rahman et al. 2021). A key advantage of online data collection has always been that a researcher can collect data remotely and from individuals across multiple locations (Comley and Beaumont 2011). Eliminating the need to travel or meet in person also extends data collection to marginalized populations (Reisner et al. 2018) or to those for whom travel might be difficult (Tates et al. 2009). Online modalities have been shown to work well with populations facing unique health challenges such as traumatic brain injury (Egan et al. 2006), autism (Benford and Standen 2011), multiple sclerosis (Synnot et al. 2014), and chronic conditions (Nicholas et al. 2010).

Online qualitative data collection modalities can be categorized along two dimensions. One dimension refers to the nature of communication medium—text or video. In a text-based modality, questions and responses are typed via computer. Video-based modalities use online video (with audio) technology and questions/responses are spoken. The other dimension pertains to temporality—synchronous or asynchronous. Synchronous methods are conducted in real-time, typically through text- or video-conferencing (Fox et al. 2007). Conversely, asynchronous methods occur in a back-and-forth manner over a period of time, and are typically conducted through discussion boards or listservs (Rezabek 2000). Synchronous methods tend to be relatively fast-paced with rapid communication flow, whereas asynchronous methods allow participants more time to consider and respond to questions. The latter are purported to generate richer and deeper data (Fox et al. 2007; Oringderff 2004).

As online qualitative data collection becomes more common, it is important to understand what the trade-offs of moving from in-person to remote data collection might be. Recent empirical studies designed to compare features of qualitative datasets generated in-person and online have found some differences in the amount of data provided by participants, but very little difference in the thematic content of responses (Krouwel et al. 2019; Namey et al. 2020). In terms of the effects of data collection modality on specific aspects of the data, the limited empirical research on disclosure of sensitive topics, expression of dissenting opinions, and participant experiences of different modes for focus group research are summarized in Table 1. The findings of these studies, along with similar work on comparisons of quantitative surveys, suggests that individuals are more likely to express socially unsanctioned or highly personal (“sensitive”) opinions or behavior in online settings (Campbell et al. 2001; Walston and Lissitz 2000; Turner et al. 1998; Spector 2004), a phenomenon described as the “online disinhibition effect” (Suler 2004). Researchers have found similar results of greater online openness when comparing focus group participants’ willingness to express dissenting or contrary opinions (Massey and Clapper 1995; Underhill and Olmsted 2003; Reid and Reid 2005; Graffigna and Bosio 2006). Of the studies that compared participant satisfaction between or among modes of data collection, there was no consensus finding on how participants preferred to connect (Reid and Reid 2005; Nicholas et al. 2010; Massey and Clapper 1995; Underhill and Olmsted 2003; Walston and Lissitz 2000).

Nearly all of these studies comparing online and face-to-face modes of data collection are limited to focus groups, however, and few employed an experimental design. Many did not control for instrument or interviewer variation, had very small sample sizes, featured only one type of online data collection, and/or lacked systematic and transparent analytic procedures. And, despite many articles reflecting on advantages and disadvantages and lessons learned about online qualitative research before and since the pandemic (Weller 2015; 2017; Janghorban et al. 2014; Corti and Fielding 2016; Vindrola-Padros et al. 2020; Lobe et al. 2020), few actually compare data generated or participant experiences with the different modalities, as Weller (2015) notes.

In early 2016 we designed a quasi-experimental design to address these gaps in the literature. Participants were randomized to either an IDI or FG arm and systematically assigned to one of four modes of data collection: (1) face-to-face, (2) video-based, (3) synchronous text-based, or (4) asynchronous text-based. The analyses we present in this article focus specifically on comparing the amount of sensitive data, dissenting opinions in focus groups, and participant satisfaction with face-to-face and remote modes of data collection. Findings on thematic content generated across modalities and comparative costs are reported elsewhere (Namey et al. 2020).

This study addressed questions about how changes in the modality of data collection might affect participants’ willingness to disclose sensitive personal experiences, dissenting opinions, and their overall comfort with the data collection process.

We found mixed evidence of an “online disinhibition effect” (Suler 2004) with regard to participants’ sharing of sensitive personal information across modalities. In terms of unsolicited sensitive personal disclosures, we found no statistically significant differences across modalities for either interviews or focus groups. It is difficult to determine the “usual” rate of sensitive disclosures in a qualitative data collection event, but we consider the range of themes and their presence across IDIs and FGs (and modalities) as an indication that these were salient issues. The relative frequency of mention of alcohol use during pregnancy may suggest that it is not a “sensitive” issue; however, in response to an earlier question that asked about “things a pregnant woman shouldn’t do to have a safe pregnancy”, drinking alcohol was mentioned by nearly everyone, so we considered personal disclosure of that behavior as going against social norms and therefore potentially sensitive. The more frequent disclosure of sensitive information in the focus group context aligns with earlier findings using a similar methodology (Guest et al. 2017b), though is subject to the same issues of different numbers of people represented in the unit of analysis (i.e., individual versus group of 7–8).

Evidence of an online disinhibition effect was observed in terms of focus group participants presenting dissenting opinions, which aligns with previous findings of more disagreement/willingness to dissent when visual cues are removed (Montoya-Weiss et al. 1998). Data on participant perceptions of the data collection modalities, which showed that the online video focus groups, in particular, were viewed as less comfortable and less of a safe space than the other methods, furthered this point. By contrast, greater proportions of women in the online text-based FG modalities, particularly the message board option, reported that the mode of data collection made them more willing to share, consistent with findings from Graffigna and Bosio (2006), who postulated there was also less social desirability bias evident in responses in their forum-based groups. Yet the data on abstentions—the number of participants in a group who chose to remain silent on a question—suggests that the anonymity of the online text-based modes of data collection provided participants more cover to “pass” on a question as well.

Findings on participant experiences of, and satisfaction with, different data collection modalities may reflect general use and familiarity with some modes of communication over others. The face-to-face IDIs and FGs both scored consistently high, with participants remarking on the conversational tone and ease of interpersonal exchange, even when discussing a sensitive topic. Given the similarity between face-to-face data collection and online video-based forms—spoken conversation, visual connection and ability to read non-verbal cues—it is perhaps surprising that online video FGs consistently received the lowest proportion of women reporting strong feelings of rapport, comfort, and safety. As participant feedback indicated, some of this relative discomfort was associated with technology issues that could interrupt the flow of conversation (freezing video, connection challenges), but women also seemed to “warm up” to each other less when connected via video versus in person. The pre-focus group small talk that happens in a conference room did not occur in online video contexts, and it may be that this helps participants to get a “read” on one another, identify similarities and connections, and set a more relaxed tone. Further, seeing oneself live on a computer screen among five to six others was cited as uncomfortable or distracting by a few women, in line with arguments put forth by Bailenson (2021) that “nonverbal overload” contributes to the phenomenon of Zoom fatigue. Like Archibald et al. (2019), we did not find a similar issue with video-based individual interviews.

Conversely, the relatively anonymous online discussion post modality scored high for the FGs in terms of rapport and comfort, perhaps reflecting women’s use of and familiarity with social media and posting platforms in other areas of their lives, and consistent with earlier conceptualizations of how people relate online (Montoya‐Weiss et al. 1998). We did not consider telephone (audio-only) data collection as part of this study given that telephone-based focus groups are uncommon, but the literature suggests the benefits of geographic reach and privacy, paired with speech, may make them a suitable option in some cases (Allen 2013; Koskan et al. 2014).

As with any research, our findings come with limitations. First, our study had relatively low statistical power to detect small to medium-sized differences between modalities, particularly when comparing focus groups. Our a priori definition of sensitive information may have differed from what individual women would consider sensitive, and the restriction of our analysis of dissenting opinions to one question provides a limited breadth of data. Also, our sample was relatively homogenous, limiting the generalizability of our findings. However, given our wider methodological objectives (Namey et al. 2020) and stratified participant assignment to data collection modality, sample homogeneity is also a strength in that it reduced a potential confounder in our comparative analysis. Relatedly, we recognize that the nature of the study and the experimental design necessitated eligibility criteria related to computer access and typing skills that may also introduce bias into the study vis-a-vis generalizability—that women who have home computers and can type are not representative of the general population of women who could benefit from remote data collection. Interpretations should be made with these limitations in mind.

The Covid-19 pandemic motivated researchers across disciplines to use online qualitative data collection techniques, often for the first time. While many decisions were born of necessity, going forward, researchers will have the chance to be more intentional about the selection and use of online modes of data collection to supplement or replace in-person sessions. We have contributed empirical data points for consideration of the relative strengths and trade-offs of four modes of data collection for eliciting sensitive information and dissenting opinions, and in terms of participants’ experiences of the process. Though unsatisfying, there is no clear “right” choice of data collection modality, as it will always depend on research objectives, topic, and population. But knowing, for instance, that there may be an advantageous disinhibition effect for non-visual online data collection modalities can provide a starting point for assessing the options. Comparative research with more diverse populations—in terms of socio-economic status, race, computer/literacy, and mobility— and on topics of varying sensitivity would provide a broader base of empirical data to help guide study designs. And in the meantime, early engagement with individuals from the research population can provide a complement to existing literature by generating input on logistical, technological, or comfort/capacity concerns that might indicate a “better fit” option(s) for data collection.