Advances in Quantitative Ethnography

Regular expression (regex) based automated qualitative coding helps reduce researchers’ effort in manually coding text data, without sacrificing transparency of the coding process. However, researchers using regex based approaches struggle with low recall or high false negative rate during classifier development. Advanced natural language processing techniques, such as topic modeling, latent semantic analysis and neural network classification models help solve this problem in various ways. The latest advance in this direction is the discovery of the so called “negative reversion set (NRS)”, in which false negative items appear more frequently than in the negative set. This helps regex classifier developers more quickly identify missing items and thus improve classification recall. This paper simulates the use of NRS in real coding scenarios and compares the required manual coding items between NRS sampling and random sampling in the process of classifier refinement. The result using one data set with 50,818 items and six associated qualitative codes shows that, on average, using NRS sampling, the required manual coding size could be reduced by 50% to 63%, comparing with random sampling.

In quantitative ethnography (QE) studies which often involve large datasets that cannot be entirely hand-coded by human raters, researchers have used supervised machine learning approaches to develop automated classifiers. However, QE researchers are rightly concerned with the amount of human coding that may be required to develop classifiers that achieve the high levels of accuracy that QE studies typically require. In this study, we compare a neural network, a powerful traditional supervised learning approach, with nCoder, an active learning technique commonly used in QE studies, to determine which technique requires the least human coding to produce a sufficiently accurate classifier. To do this, we constructed multiple training sets from a large dataset used in prior QE studies and designed a Monte Carlo simulation to test the performance of the two techniques systematically. Our results show that nCoder can achieve high predictive accuracy with significantly less human-coded data than a neural network.

The use of data and algorithms in the social sciences allows for exciting progress, but also poses epistemological challenges. Operations that appear innocent and purely technical may profoundly influence final results. Researchers working with data can make their process less arbitrary and more accountable by making theoretically grounded methodological choices.

We apply this approach to the problem of reducing networks representing ethnographic corpora. Their nodes represent ethnographic codes, and their edges the co-occurrence of codes in a corpus. We introduce and discuss four techniques to reduce such networks and facilitate visual analysis. We show how the mathematical characteristics of each one are aligned with a specific approach in sociology or anthropology: structuralism and post-structuralism; identifying the central concepts in a discourse; and discovering hegemonic and counter-hegemonic clusters of meaning.

The task of succinctly and insightfully discussing themes in the differences between several (three or more) groups in naturalistic, ethnographic research faces a number of constraints. The number of all possible pairs is a quadratic function of the number of groups, and prior order and stand-out subsets may not exist to narrow that number down. We define and compare methods for guiding this task during Epistemic Network Analysis.

Data simulations are powerful analytic tools that give researchers a great degree of control over data collection and experimental design. Despite these advantages, data simulations have not yet received the same amount of use as other techniques within the context of quantitative ethnography. In this paper, we explore the reasons for this and use examples of recent work to argue that data simulations can—and already do—play an important role in quantitative ethnography.

Collaborative Problem Solving (CPS) is a socio-cognitive process that is interactive, interdependent, and temporal. As individuals interact with each other, information is added to the common ground, or the current state of a group’s shared understanding, which in turn influences individuals’ subsequent responses to the common ground. Therefore, to model CPS processes, especially in a context where the order of events is hypothesized to be meaningful, it is important to account for the ordered aspect. In this study, we present Ordered Network Analysis (ONA), a method that can not only model the ordered aspect of CPS, but also supports visual and statistical comparison of ONA networks. To demonstrate the analytical affordances and interpretable visualizations of ONA, we analyzed the collaborative discourse data of air defense warfare teams. We found that ONA was able to capture the qualitative differences between the control and experimental condition that cannot be captured using unordered models, and also tested that such differences were statistically different.

In this paper, I argue that elements of Epistemic Network Analysis (ENA), a quantitative ethnography tool, can be adapted to facilitate co-construction of knowledge in an interview setting. ENA needs to be further explored in collaborative contexts with researchers and participants to generate nuanced examinations of its affordances and limitations. To respond to this challenge and continue the efforts of participatory QE, in this paper, I explain the outcomes of joint researcher-participant discussions of constructed discourse networks using ENA. For these discussions, I developed a simple tool using Google Slides, which I describe in detail. This tool allowed research participants and researcher to come together and delve into the themes and puzzles emerging from the creation and discussion of the networks. The outcomes of these network discussions shed light on the intricacies of joint cognition in discourse when data visualizations such as ENA are used to create a reflective and collaborative space for researchers and participants.

Depending on analytical goals and techniques, qualitative data may be coded and segmented to investigate code or code co-occurrence frequencies. As codes are relevant aspects of data vis-à-vis the topic of inquiry, segments are meaningful divisions of those data. To explore various modes of segmentation, their underlying assumptions, and effects on potential models, the framework and terminology of Epistemic Network Analysis was employed as an analytical tool where coding and segmentation both contribute to data visualization. Three operationalizations of segmentation are elaborated: moving, infinite, and whole conversation stanza windows and demonstrated through instances where each of these may be applicable to data.

Segmentation is a crucial step in analyses of qualitative data where code or code co-occurrence frequencies are of interest. Decisions about how best to segment are inextricably connected to coding decisions, as well as wider analytical goals and research questions. These decisions directly affect resulting models and the interpretations derived from them. However, while there is a wealth of frameworks guiding code development and application, far fewer guidelines exist for segmentation. This paper reports on the development of an initial set of heuristics for the segmentation of monologic data. Using the framework of Epistemic Network Analysis, we demonstrate various approaches to segmentation and show how these segmentation decisions affect models and subsequent interpretations. We argue that segmentation should be aligned with research questions and developed in conjunction with coding, and we offer considerations and techniques for doing so.

Research has robustly documented the long-term life impacts of student loan debt on borrowers while in college and post-graduation. During the pandemic, repayment policies attempting to alleviate the debt burden were instituted to account for changes in income during government lockdowns. Since these reforms were implemented, what is needed is a more nuanced examination of the differences in post-graduation attitudes towards student loan debt in both pre-COVID and during the pandemic (after supportive government policy). This study utilizes Epistemic Network Analysis to identify and illustrate the connections in attitudes about debt between pre-COVID-19 and COVID-19 conversations. The results from this study illustrate statistically significant changes in attitudes towards debt, with pre-COVID-19 group members discussing how their negative feelings towards their debt drive their life choices while members of the COVID-19 conversations focused on repayment and hopeful feelings. Thus, this work positions itself to contribute to our understanding of the potential impact of debt-relief policies.

This study examines the patterns of feedback among early-career elementary mathematics teachers participating in an online inquiry group focused on the practice of number talk routines. Number Talk Routines are instructional practices designed to help facilitate students’ computational fluency in ways that promote flexible number sense. Teachers facilitate these discussions using responsive teaching practices that elicit student thinking and highlight how students’ strategies relate to each other and to key mathematical concepts. Data for this study come from time-stamped feedback comments posted by members of the inquiry group to correspond with specific moments during each participant’s number talk routine. Epistemic network analysis was used to examine the patterns in the form and content of feedback over time. The results suggest that early-career teachers became more reflective in their feedback, connecting their own practices to the work of others and focusing more on teachers’ decision making that supported enactment of responsive teaching practices.

The broader education research community has adopted co-design, or participatory design, as a method to increase adoption of innovations in classrooms and to support professional learning of teachers. However, it can be challenging, due to co-design’s dynamic nature, to closely investigate how the co-process played out over time, and how it led to changes in teachers’ perceptions, beliefs, and/or practices. Applying Quantitative Ethnography, we investigate how teachers and researchers collaboratively designed assessment metrics and data visualizations for an educational math game; we discuss the interactions among the co-design activities, teachers’ learning, and qualities of the dashboard created as the output of the process.

This study introduces and investigates the capabilities of three different text mining approaches, namely Latent Semantic Analysis, Latent Dirichlet Analysis, and Clustering Word Vectors, for automating code extraction from a relatively small discussion board dataset. We compare the outputs of each algorithm with a previous dataset that was manually coded by two human raters. The results show that even with a relatively small dataset, automated approaches can be an asset to course instructors by extracting some of the discussion codes, which can be used in Epistemic Network Analysis.

In this paper, we consider how we could use two different frameworks, our own Robust Understandings of proportions plus the Knowledge Quartet, to better understand the relationship between mathematics teachers’ knowledge and their teaching practices. We present both frameworks, then describe each of two teachers by describing their classroom, considering an ENA graph of their understanding of proportional reasoning and key patterns that emerged through use of the Knowledge Quartet. We end by discussing how we have been able to use these two frameworks together and why this research is important in ongoing efforts to make sense of the relationships between teachers’ knowledge and practice.

Online discussion forums are often used as a point of contact between students and their instructors for college courses. While asynchronous discourse has proven to be effective for learning, it remains unclear whether the student interactions manifest in socially constructive ways in addition to the cognitive benefits. In this paper, we consider the social dimension of student interactions within a Canvas course discussion forum. In particular, we examine the influence of instructional contexts to shape the mapping of different indicators that constitute social presence within the Community of Inquiry framework. For the analysis, data was collected from two instances of the same course: one taught in a hybrid format and the other in a remote format. The results of epistemic network analysis reveal that elements of social presence manifest differently in hybrid and fully remote modalities. The remote modality yielded more interconnected, balanced networks than their hybrid counterparts. The findings suggest that discourse from online discussions is conducive to collaborative inquiry through the mediation of social presence when pedagogical decisions work with the different instruction modalities to support student-to-student interaction.

The education community continues to struggle to support students to make meaningful connections between disciplinary learning at schools with their everyday life experiences. Even when the students engage in meaningful science learning experiences, recognizing the connections and relations that students make through their engagement is methodologically challenging, especially through the analysis of qualitative data. The purpose of this study was to explore the patterns of connections that students generated through their participation in a co-designed physics unit. We analyzed 76 high school students written and pictorial responses to performance assessment tasks designed to engage students in physics learning. We used quantitative ethnographic techniques and a tool named Epistemic Network Analysis (ENA), to visualize the structure of connections between physics concepts and real-life experiences in students’ assessment tasks. The ENA results revealed patterns of connections that students generated between physics concepts they learned at school and their everyday experiences. Notably, the analyses showed differences in patterns of connections between male and female participants and between written and pictorial preferences in momentum and impulse unit assessment tasks. The implications for curriculum design and performance assessment in science are discussed.

In this paper, we study the relationship between the content and goals of curriculum revisions toward the integration of computational thinking (CT) in compulsory schools in Denmark, Sweden, and England. Our analyses build on data consisting of a combination of official documents such as new curricula, white papers, and implementation strategies and interviews of experts who are either highly knowledgeable about or were involved in developing the curriculum revisions in these three countries. Our study found that there are strong connections between the CT content data practices and goal competitiveness in England. In Sweden, we found that the relationship between data practices and the goal of competitiveness is strongest. In Denmark, we found that the CT content codes related to data practices, modeling and simulation practices, and computational problem solving practices were all strongly represented, but all were weakly related to policy goals.

Understanding indicators in self-regulated learning (SRL) that affect mathematical success using quantitative techniques such as epistemic networks hold potential for providing effective scaffolds that draw directly from the learner’s perspective. Tied to learning success, SRL provides a range of frameworks for identifying students' affective, cognitive, and metacognitive performance in a computer-based learning environment. This research can investigate how ENA can contribute as a visualization device to understanding of the metacognitive aspect of math learning. With the aim, we collected text responses from an online math problem-solving environment that encouraged reflections on self-regulated learning patterns that differ by the rate of correctness and familiarity with the educational tool. Student responses consisted of their explanations of strategies and solutions after the scaffolding instructions. Our team deductively designed detectors reflecting on assembling and translating operations (Winne’s SMART model) to examine differences in the learner’s self-regulated learning behaviors. We then leveraged Epistemic Network Analysis (ENA) using these detected indicators as codes to compare the results within two categories: performance on correctness and familiarity developed over time. Models show stronger co-occurrence between numerical representation and contextual representation and highlight the critical impact of outcome orientation on learner success. When the final answer is correct, or learners are more familiar with the educational tool, there is a strong outcome orientation connected to contextual representation within SRL operations.

This study examines experiences in the museum workplace shared by the Change the Museum Instagram account from June–December 2020. These posts were recorded and subsequently hand coded, then put into an Epistemic Network Analysis (ENA) model. Networks were analyzed by month and by construct, in this case looking specifically at BIPOC. Results showed a statistically significant difference between the months of June and December. Main constructs for June were Microaggression, Ignorance, and Senior (Leadership), compared to December with Employment and Wages. For BIPOC networks, the strongest connections throughout all months were linked to White, including Employment and Wages, Senior (Leadership), Microaggression, and Peers/Colleagues. Using these results can help inform meaningful change within museum culture.

This study analyzes early episodes of the Ukraine War Diaries podcast to examine the initial lived experiences of those who stayed in Ukraine after Russia’s invasion on February 24, 2022. Aspects of Mezirow’s transformational learning theory provided a lens to code the data, as it posits various stages of psychological adaptation after a catalyzing crisis. ENA was used to model the discourse patterns of two residents in the city of Kyiv to identify the most relevant connected constructs that emerged. The most prominent connections were between Self-Examination and Relating Discontent to Others, and Relating Discontent to Others with Disorienting Dilemma. Other strong connections were mostly tied to Self-Examination. These thoughts are consistent with realizations that have not shifted towards change and action, which is expected given the invasion is still ongoing at the time of the episodes, with no resolution in sight. This analysis seeks to document initial experiences of those living in Ukraine through the Russian invasion.

This paper extends an analytic framework for political discourse that takes place over digital social media. First proposed in 2020, the framework applies principles of epistemic frame theory and quantitative ethnography to classify and investigate relationships in political discourse patterns, to situate and visualize broad discourse patterns, and to facilitate ethnographic analysis that incorporates emotion as paramount to explaining these patterns. The paper also reviews the constructs of discursive transactions and emotional grammars to scaffold the framework’s explanatory value. This research is meant to use quantitative ethnography and its tools to contribute to a broader dialog on the nature and cost of dysfunctional political discourse patterns, to help researchers articulate both the spiraling nature of dysfunctional political discourse, and the profound damage it inflicts on social goals of fairness, well-being, and prosperity. Commentary threads following political articles from the Washington Post and the Wall Street Journal are modeled with the Epistemic Network Analysis (ENA) software tool to illustrate the viability of a political discourse coding system for the proposed framework.

Stack Overflow provides a popular and practical community for software developers to ask and answer questions related to coding. These answers are ranked by users to evaluate their quality. For newcomers, participating in answering questions can be challenging, as they must learn what the expectations for answers in this online community are. In this paper, using epistemic networks, we analyze the content structure of the answers posted to Stack Overflow’s most highly ranked question with the goal of understanding characteristics of answers valued by the Stack Overflow community. Network models show that answer content is qualitatively different between high and low ranked answers, with high ranked answers including general explanations and code examples to contextualize question-specific code and explanations. We discuss how these findings could be used to better support and scaffold new participants in crafting their answers.

This study examines the design activities of engineers and product designers from the perspective of knowledge building. The practice of knowledge building has been studied for more than 30 years. However, in recent years, analytical methods have been developed to analyze it from two directions—idea improvement and epistemic frames—and these methods are currently being enhanced. Nevertheless, studies that have analyzed idea improvement and epistemic frames have focused on practices in the classroom rather than discussing the activities of engineers and designers, who are also knowledge building models. Therefore, this study analyzed the co-design process of a product design team and an engineering team that engaged in creative activities for their work from the perspectives of idea improvement using socio-semantic network analysis (SSNA) and the epistemic frame by epistemic network analysis (ENA). Moreover, this study discussed defining meaning segments using SSNA as a computational approach for quantitative ethnography (QE). As a result, both teams showed good knowledge building characteristics in that they continuously improved their ideas. Furthermore, the engineering team worked under various epistemic actions, while the product designers worked under a limited epistemic frame. We also confirmed that the analysis method of this study was able to extract the characteristic discourse of each team. These results support future knowledge building practices, as they illustrate that designers and engineers engage in the same continuous idea improvement under different epistemic actions. Furthermore, this study contributes to future QE research because the results show the qualitative differences between designers and engineers using determining meaning segments as a computational approach.

In healthcare settings, poor communication between physicians and nurses is one of the most common causes of adverse events. This study used Epistemic Network Analysis to help identify communication patterns in physician-nurse dyad interactions. We used existing video data where physicians made patient care rounds on two oncology patient units at a large academic medical center, and video recordings captured conversations physicians had with nurses on the plan of care. All data was transcribed, segmented and annotated using the Verbal Response Mode (VRM) taxonomy. The results showed that the relationship between Edification and Disclosure was strongest for the dyads that reached a shared understanding, suggesting the importance of these two modes to reaching shared understanding during patient care rounds. Reflection and Interpretation were the least used VRM codes, and this might be one possible area for intervention development. This pilot study provided new insight into how to improve communication between physicians and nurses using ENA coupled with VRM taxonomy.

Analysis of policy ecosystems can be challenging due to the volume of documentary and ethnographic data and the complexity of the interactions that define the ecology of such a system. This paper uses climate change adaptation policy as a case study with which to explore the potential for QE methods to model policy ecosystems. Specifically, it analyzes policies and draft policies constructed by three different categories of governmental entity—nations, state and local governments, and tribal governments or Indigenous communities—as well as guidance for policy makers produced by the United Nations Intergovernmental Panel on Climate Change and other international agencies, as a first step toward mapping the ecology of climate change adaptation policy. This case study is then used to reflect on the strengths of QE methods for analyzing policy ecosystems and areas of opportunity for further theoretical and methodological development.