Advances in Quantitative Ethnography

To adapt to the needs of a 21st century context, educational researchers and practitioners could benefit from leveraging the potential of virtual learning environments such as games and the participatory cultures that surround them to support learning as a transformational and intentional process of identity exploration. This research offers a theoretically-comprehensive look into how a participant in an online game community forum engaged in identity exploration processes. Publicly-available longitudinal data was downloaded from Kerbal Space Program (KSP) players, which informed the development of an illustrative case study selected to elucidate how individual processes of identity exploration manifest. Lines of player data were deductively coded as representative of identity exploration and visualized using Epistemic Network Analysis to represent shifts in integration of identity constructs over time. Findings suggest that player participation in the community forum can support statistically significant identity change and highlight future areas of research in this field.

Diagnostic argumentation can be decomposed referring to the dimensions of content (see Toulmin 2003) and explicated strategy use indicated by epistemic activities (see Fischer et al. 2014). We propose a conceptual framework to analyze these two dimensions within diagnostic argumentation and explore its use within initial applications using the method of Epistemic Network Analysis (Shaffer 2017). The results indicate that both approaches of solely analyzing the dimension of content and solely analyzing the dimension of epistemic activities offer less insights into diagnostic argumentations than an analysis that includes both dimensions.

This paper seeks to contribute to the emerging field of Quantitative Ethnography (QE) by demonstrating its utility to solve a complex challenge in Learning Analytics: the provision of timely feedback to collocated teams and their coaches. We define two requirements that extend the QE concept in order to operationalise it such a design process, namely, the use of co-design methodologies, and the availability of automated analytics workflow to close the feedback loop. We introduce the Multimodal Matrix as a data modelling approach that can integrate theoretical concepts about teamwork with contextual insights about specific work practices, enabling the analyst to map between higher order codes and low-level sensor data, with the option add the results of manually performed analyses. This is implemented in software as a workflow for rapid data modelling, analysis and interactive visualisation, demonstrated in the context of nursing teamwork simulations. We propose that this exemplifies how a QE methodology can underpin collocated activity analytics, at scale, with in-principle applications to embodied, collocated activities beyond our case study.

Coding is a process of assigning meaning to a given piece of evidence. Evidence may be found in a variety of data types, including documents, research interviews, posts from social media, conversations from learning platforms, or any source of data that may provide insights for the questions under qualitative study. In this study, we focus on text data and consider coding as a process of identifying words or phrases and categorizing them into codes to facilitate data analysis. There are a number of different approaches to generating qualitative codes, such as grounded coding, a priori coding, or using both in an iterative process. However, both qualitative and quantitative analysts face the same coding problem: when the data size is large, manually coding becomes impractical. nCoder is a tool that helps researchers to discover and code key concepts in text data with minimum human judgements. Once reliability and validity are established, nCoder automatically applies the coding scheme to the dataset. However, for concepts that occur infrequently, even with an acceptable reliability, the classifier may still result in too many false negatives. This paper explores these problems within the current nCoder and proposes adding a semantic component to the nCoder. A tool called “nCoder+” is presented with real data to demonstrate the usefulness of the semantic component. The possible ways of integrating this component and other natural language processing techniques into nCoder are discussed.

Passive behavior in collaborative group settings is often associated with negative or no contributions to the group (social loafing). This paper examines low and high participation levels of students in a virtual collaborative group setting within a global, STEM-focused digital makerspace community. The results of using epistemic network analysis show that both high and low participation levels contributed to the overall balance of the group discourse, overcoming social loafing behavior. High participation level students provided social aspects that contributed to the development of a safe social space for sharing, while low level participation provided content focused dialogue for the group.

This paper investigates how different configuration of epistemic network analysis parameters influence the examination of student interactions in asynchronous discussions in online learning environments. Specifically, the paper investigates strategies for dealing by unintended consequences of a dominant node in epistemic network analysis (ENA). In particular, the paper reports on a study that explored the effects of two different strategies including (i) the use of different dimensions calculated with singular value decomposition (SVD), and (ii) exclusion of a dominant code. Our results showed that the use of different SVDs did not change the influence of a dominant code in the graph. On the other hand, the exclusion of the dominant code led to an entirely different configuration in ENA. The practical implications of the results are further discussed.

This paper is situated in a 5-year NSF CAREER project awarded to test and refine Projective Reflection (PR) as a theoretical and methodological framework for facilitating learning as identity exploration in play-based environments. 54 high school students engaged in Virtual City Planning, an iteratively refined course that provided systematic and personally-relevant opportunities for play, curricular, reflection and discussion activities in Philadelphia Land Science, a virtual learning environment, and in an associated curriculum enacted in a STEM museum-classroom. In-game logged data and in-class student data were examined using Epistemic Network Analysis. An illustrative case study revealed visual and interpretive patterns in students’ identity exploration. The change was reflected in their knowledge, interest and valuing, self-organization and self-control, and self-perception and self-definition (KIVSSSS) in relation to the roles explored from the start of the intervention (Starting Self), during (Exploring role-specific Possible Selves), and the end (New Self).

African-, European-, Mexican-, and Native-American adolescents (N  =  270) described times they had responded to a peer’s victimization with efforts to repair relationships or avenge the aggression. They provided ratings of self-evaluative emotions and judgements (e.g.., pride, shame, helpfulness) and explanations for each rating. Explanations were coded as exemplifying one of eight goals, and whether the action described in each condition promoted or threatened the desired goal. Complementary analyses utilizing the general linear model and epistemic network examined the rates of each type of goal and the connections between goals, and between goals and outcomes. Benevolence was the most frequently cited goal, and third-party reparative efforts were viewed as promoting benevolence and competence. Benevolence goals were both promoted and threatened by third-party revenge. Self-directed growth was cited most often following revenge, as an example of goal threat and often in conjunction with shame. Relationships between revenge and reparative efforts are explored.

Understanding why students quit a level in a learning game could inform the design of appropriate and timely interventions to keep students motivated to persevere. In this paper, we study student quitting behavior in Physics Playground (PP) – a Physics game for secondary school students. We focus on student cognition that can be inferred from their interaction with the game. PP logs meaningful and crucial student behaviors relevant to physics learning in real time. The automatically generated events in the interaction log are used as codes for quantitative ethnography analysis. We study epistemic networks from five levels to study how the temporal interconnections between the events are different for students who quit the game and those who did not. Our analysis revealed that students who quit over-rely on nudge actions and tend to settle on a solution more quickly than students who successfully complete a level, often failing to identify the correct agent and supporting objects to solve the level.

Using automated classifiers to code discourse data enables researchers to carry out analyses on large datasets. This paper presents a detailed example of applying training, validation and test sets frequently utilized in machine learning to develop automated classifiers for use in quantitative ethnography research. The method was applied to two dispositional constructs. Within one cycle of the process, reliable and valid automated classifiers were developed for Social Disposition. However, the automated coding scheme for Inclusive Disposition was rejected during the validation stage due to issues of overfitting. Nonetheless, the results demonstrate the beneficial potential of using preclassified datasets in enhancing the efficiency and effectiveness of the automation process.

Summer melt is a phenomenon when college-intending students fail to enroll in the fall after high school graduation. Previous research on summer melt utilized surveys, typically consisting of Likert scale questions and open-ended response questions. Open-ended responses can elicit more information from students, but they have not been fully analyzed due to the cost, time, and complexity of theme extraction with manual coding. In the present study, we applied the topic modeling approach to extract topics and relevant themes, and evaluated model performance by comparing model-generated topics and categories with the human-identified topics and themes. Results showed that the topic model allows for extracting similar topics as the survey questions that were investigated, but only extracted part of the themes classified by the human. Discussion and implications focus on potential improvements in automated topic and theme classification from open-ended survey responses.

In this paper, we describe a way of using multi-modal learning analytics to augment qualitative data. We extract facial expressions that may indicate particular emotions from videos of dyads playing an interactive table-top game built for a museum. From this data, we explore the correlation between students’ understanding of the biological and complex systems concepts showcased in the learning environment and their facial expressions. First, we show how information retrieval techniques can be used on facial expression features to investigate emotional variation during key moments of the interaction. Second, we connect these features to moments of learning identified by traditional qualitative methods. Finally, we present an initial pilot using these methods in concert to identify key moments in multiple modalities. We end with a discussion of our preliminary findings on interweaving machine and human analytical approaches.

Learning Sciences research often concerns the analysis of data from individual or collaborative learning processes. For the analysis of such data, various methods have been proposed, including Process Mining (PM) and Epistemic Network Analysis (ENA). Both methods have advantages and disadvantages when analyzing learning processes. We argue that a concerted use of both techniques may provide valuable information that would be obscured when using only one of these methods. We demonstrate this by applying PM and ENA on data from a study that investigated how students regulate collaborative learning when faced with either motivational or comprehension-related problems. While PM showed that collaborative learners are more incoherent (i.e. more heterogeneous in their chosen activities) when regulating motivational problems than comprehension-related problems at the beginning, ENA revealed that in later stages of their learning process, they focus on fewer activities when being confronted with motivational than with comprehension-related problems. Thus, a combination of the two approaches seems to be warranted.

In this paper, we aim at uncovering collaborative problem-solving patterns associated with students’ successful learning of social sciences research methods in a Productive Failure (PF) setting. We report an epistemic network analysis (ENA) of PF students’ conversations. Conversations are compared between PF groups that generated high quality solution ideas (HQ groups) and groups that developed low quality solution ideas (LQ groups). The ENA results demonstrate significantly different patterns. The collaborative problem solving of four HQ triads in a PF setting is characterized by debates and elaborations related to canonical contents of the targeted learning concept. The collaborative problem solving of four LQ triads is featured by task-pursuance actions and elaborations related to the instructions and contents stated in the worksheet. We also compared the eight groups based on their learning outcome (i.e., performance on a knowledge test). The comparison of four groups with a high learning outcome and of four groups with a low learning outcome revealed similar ENA results as the comparison of the HQ and LQ groups. These findings offer empirical evidence for the often hypothesized but rarely supported notion of certain collaborative problem-solving processes being important for the effectiveness of PF. The potential relevance of the collaborative problem-solving patterns of HQ groups for learning in a PF setting is discussed in light of mechanisms hypothesized to underlie the PF effect.

The knowledge required by teachers has long been a focus of public and academic attention. Following a period of intense research interest in teachers’ knowledge in the 1980s and 1990s, many researchers have adopted Shulman’s suggestion that expert teaching practice is based on seven forms of knowledge which collectively are referred to as a knowledge base for teaching. Shulman’s work also offered a decision-making framework known as pedagogical reasoning and action, which allows teachers to use their seven forms of knowledge to make effective pedagogical decisions. Despite the widespread acceptance of these ideas, no empirical evidence exploring the connections between knowledge and decision-making is evident in the research literature. This paper reports on a pilot study in which the connections between knowledge and decisions in science, mathematics and information technology teachers’ lesson plans are quantified and represented using epistemic network analysis. Findings reveal and levels of complexity that have been intimated but, until now, not supported with empirical evidence.

This paper falls in the field of playing analytics. It deals with an empirical work dedicated to explore the potential of data sonification (i.e. the conversion of data into sound that reflects their objective properties or relations). Data sonification is proposed as an alternative to data visualization. We applied data sonification for the analysis of gameplays and players’ strategies during a session dedicated to game-based learning. The data of our study (digital traces) was collected from 200 pre-service teachers who played Tamagocours, an online collaborative multiplayer game dedicated to learn the rules (i.e. copyright) that comply with the policies for the use of digital resources in an educational context. For one typical individual (parangon) for each of the 5 categories of players, the collected digital traces were converted into an audio format so that the actions that they performed become listenable. A specific software, SOnification of DAta for Learning Analytics (SODA4LA), was developed for this purpose. The first results show that different features of the data can be recognized from data listening. These results also enable for the identification of different parameters that should be taken into account for the sonification of diachronic data. We consider that this study open new perspectives for playing analytics. Thus we advocate for new research aiming at exploring the potential of data sonification for the analysis of complex and diachronic datasets in the field of educational sciences.

This paper is situated in an NSF CAREER project awarded to test and refine Projective Reflection (PR) as a theoretical and methodological framework for facilitating learning as identity exploration in virtual learning environments. PR structured the design, implementation, and refinement of Virtual City Planning, a play-based course that included identity exploration experiences mediated by a virtual learning environment (Philadelphia Land Science), and classroom experiences designed to augment the virtual learning experience. In this paper, Quantitative Ethnography techniques were applied to visualize and interpret changes at the group level (N = 20) for the first of three iterations of Virtual City Planning, as a result of exploring role-possible selves of an environmental scientist and urban planner. Changes were reflected in students’ knowledge, interest and valuing, patterns of self-organization and self-control, and self-perceptions and self-definitions (KIVSSSS) in relation to the roles explored from the start of Virtual City Planning (starting self), during (exploring role-specific possible selves), and at the end of the play-based learning experience (new self).

Simulators have been widely adopted to help surgical trainees learn procedural rules and acquire basic psychomotor skills, and research indicates that this learning transfers to clinical practice. However, few studies have explored the use of simulators to help more advanced learners improve their understanding of operative practices. To model how surgeons with different levels of experience use procedural simulators, we conducted a quantitative ethnographic analysis of small-group conversations in a continuing medical education short course on laparoscopic hernia repair. Our research shows that surgeons who had less experience with laparoscopic surgery tended to use the simulators to learn and rehearse the basic procedures, while more experienced surgeons used the simulators as a platform for exploring a range of hernia presentations and operative approaches based on their experiences. Thus simple, inexpensive simulators may be effective with both novice and more experienced learners.

The Next Generation Science Standards propose an integrated and holistic view of science education that teaches science through three-dimensional learning. In this vision of science, content and practices are interconnected and inseparable. While the NGSS has influenced K-12 education standards in 40 states, there has not been a systematic analysis of the standards themselves. In this study, we investigate three-dimensional learning in order to identify new insights into underlying relationships between science concepts as well as make comparisons between different science disciplines. We used Epistemic Network Analysis to measure and models the structure of connections among crosscutting concepts and practices within and across disciplines. Results show systematic differences between how Physical and Life Sciences use and describe cause and effect relationships in which Physical Sciences predominantly focuses on the generation of causal relationships while Life Sciences focuses on the explanation of causal relationships.

This study investigates how puppetry-based tabletop microteaching systems can contribute to student teacher training compared with normal microteaching. The study analyzes student teachers’ discourse using a puppetry-based microteaching system called “EduceBoard” introduced to a university class. The analysis included an epistemic network analysis to identify the specific features that influence changes and clarify particular discourse patterns that were found and a qualitative analysis of the discourse data. Results indicate that the puppetry-based microteaching and improvisational dialogs that it elicited enhanced student teachers’ practical insights and gave them the opportunity to develop their students’ learning and run the class smoothly.

Care transitions are important to patient safety, but we lack consensus on what outcomes of transitions to evaluate. We interviewed 28 physicians and nurses who participate in transitions of adult and pediatric trauma patients from the operating room to the intensive care unit. The handoff (i.e., communication about patient information) in the pediatric care transition was done together as a team while the other handoff was separated by profession. In this study, we identify nine care transition outcomes: (1) communication sufficiency, completeness and accuracy, (2) handoff timing, (3) patient outcomes, (4) change in workload, (5) individual situation awareness, (6) team situation awareness, (7) organization awareness, (8) team experience and (9) timing of feedback. These outcomes could be positive and negative (i.e., good or bad). This study also investigates relationships between outcomes in the two groups using epistemic network analysis (ENA). While we found the no difference between the outcomes in the team and separate handoff when comparing frequency counts, relationships between outcomes did differ when using ENA. Interviewees with the team handoff described more relationships between care team level outcomes – team situation awareness and team experience – and other outcomes, while interviewees with the separate handoffs focused on the relationship between communication and patient outcome. Future work should investigate differences in relationships between positive and negative valences of the outcomes.

Facilitating reflection of pre-service teacher is becoming a more and more important topic in teacher education. There are a number of social media tools which can support teacher professional development. It also enables us to examine the development of individuals’ reflective process and group dynamics. In this study, 50 pre-service teachers were involved to write scripts collaboratively using wikis and they were encouraged to reflect upon their written texts and script-writing strategies during the online collaborative writing process. In particular, epistemic network analysis is adopted in order to characterize learners’ reflection dynamics during the two phases of collaborative script writing. The research results show that the characteristics of reflection type in different phases are different. Also, teachers tend to reflect on the content and methods of the group in the first phase; while in the second phase, they tend to reflect on the group methods and personal gains. Using content analysis and epistemic network analysis, this paper characterize the development of reflection during collaborative writing activities and provides reference for the cultivation of reflection among pre-service teachers.

Applying Quantitative Ethnography (QE) techniques to continuous narratives in an inquiry where manual segmentation with a multitude of codes is preferred poses several challenges. In order to address these issues, we developed the Reproducible Open Coding Kit – convention, open source software, and interface – that eases manual coding, enables researchers to reproduce the coding process, compare results, and collaborate. The ROCK can also be employed to prepare data for Epistemic Network Analysis software. Our paper elaborates the challenges we encountered and the insights we gained while conducting a research project on decision-making regarding therapy choice among patients in Budapest, Hungary. Our aim is to broaden the usage of QE, while facilitating Open Science principles and transparency.

We propose a novel method called Quantitative Multimodal Interaction Analysis to understand the meaning of interactions from a set of multimodal observable behavior. We apply this method for the measurement of collaborative problem-solving skills in a dyadic online game specially designed for this purpose. We outline our assumptions and describe the machine learning approach that help us tag multimodal behaviors connecting the theoretical construct with the empirical evidence.

Inductive content analysis is a research task in which a researcher manually reads text and identifies categories or themes that emerge from a document corpus. Inductive content analysis is usually performed as part of a formal qualitative research methodology such as Grounded Theory. Topic modeling algorithms discover the latent topics in a document corpus. There has been a general assumption, that topic modeling is a suitable algorithmic aid for inductive content analysis. In this short paper, the findings from a between-subjects experiment to evaluate the differences between topics identified by manual coders and topic modeling algorithms is discussed. The findings show that the topic modeling algorithm was only comparable to the human coders for broad topics and that topic modeling algorithms would require additional domain knowledge in order to identify more fine-grained topics. The paper also reports issues that impede the use of topic modeling within the quantitative ethnography process such as topic interpretation and topic size quantification.

The qualitative content analysis process has traditionally been reliant on human researchers to read and code data, with limited use of automation. However, recent advances in Natural Language Processing (NLP) offer new techniques to improve the reliability and usefulness of content analysis, especially in the area of quantitative ethnography. In this paper we propose a new qualitative content analysis workflow that utilizes techniques such as contextual word embeddings and semantic search. Each of the design principles that inform this workflow are outlined and potential NLP solutions are discussed. This is followed by the description of a new prototype, currently in development, that implements elements of the workflow. The paper concludes with an outline of two proposed research studies to evaluate the effectiveness of the workflow and prototype as well as directions for future research.

Grounded in child/robot interaction and inclusive education, this research has designed a small socio-technical community of a robot and two children where children play and learn equitably together while they help the robot learn. This designed community was implemented in a school media lab twice a week over three weeks, each session taking about 20 min. We ethnographically observed and video recorded children’s participation in the triadic interaction naturally. The phenomena of interest include friendship development, collaborative communication, and engagement with the community. Data collection is still ongoing, and analysis will occur over this summer. This paper presents the theoretical frameworks and data analytic scheme. We expect to report the findings at the ICQE conference in October.

Perspective-taking of a wide variety of pupils or students is fundamental in designing a dialogic classroom. As a vehicle of perspective-taking, a tangible puppetry CSCL can create a learning environment that reduces the participants’ anxiety or apprehension toward evaluation and elicits various types of pupils or students, allowing them to learn various perspectives. The CSCL also provides a 3D animation that records the puppetry for prompting perspective-taking of a variety of pupils in mutual feedback discussions. A comparative experiment, which comprised of a self-performed, a puppetry, and a second self-performed microteachings, showed a relatively stable impact of the puppetry microteaching in the mutual feedback discussions on the second self-performed. This paper discusses the potential effectiveness of puppetry as a catalyst of perspective-taking to learn a variety of pupils’ viewpoints through their possible reactions in undergraduate teacher education.

This study proposes a new temporal Socio-Semantic Network Analysis (SSNA) of discourse by using the network lifetime and the moving stanza window method to analyze idea improvement in learning as knowledge-creation. The procedure of our proposed method has four steps. The first step entails making a discourse analysis unit. One discourse analysis unit is composed of discourses depending on the set numbers at a size of the moving stanza window method. The second step is calculating the total value of degree centrality for each discourse analysis unit with periods of the network lifetime by using SSNA. The third step involves calculating the difference value between discourse analysis units to define the candidates for the pivotal points. The last step is tracing the discourse back from the candidates for the pivotal points to identify segments for in-depth dialogical discourse analysis. To evaluate the proposed method, we analyzed discourse data in collaborative learning using different methods with and without the network lifetime and moving stanza window. As a result, new pivotal points were detected by implementing both the network lifetime and the moving stanza window method. An in-depth dialogical discourse analysis of a new pivotal discourse segment confirmed the appropriateness of the detection. Based on the results, it is concluded that our proposed method is better in detecting pivotal points of learning as knowledge-creation compared to the previous approach.

Recently, there have been growing calls to make research data more widely available. While the potential benefits of sharing research data are many, there are also many challenges, including the interpretability, attendability, and complexity of the data. These challenges are particularly salient for research data associated with quantitative ethnographic analyses, which often use relatively novel and sophisticated techniques. In this paper, we explore design considerations for an interface for sharing research data that attempts to address these challenges for quantitative ethnographic analyses. These considerations include: (a) maintaining the consistency of the interpretive space, (b) simplifying model details, (c) including example results and interpretations, and (d) highlighting key affordances in the user interface. To explore these considerations, we describe the design of an interactive visualization of the thematic networks present in the HBO television series, Game of Thrones.

Applied researchers are often forced to test an uninteresting (and unrealistic) hypothesis: that the mean difference between groups is zero in some imagined population. Misinterpretation of these common null hypothesis tests often obscure actual findings, and the testing process itself can result in inflated estimates over time. In this paper, we demonstrate the use of freely available software to conduct Bayesian hypothesis tests on ENA findings, in addition to traditional null hypothesis testing.

With the rapid development of information technology, teaching assistant software has been constantly appearing in classroom learning. How to effectively apply this technical resource in classroom learning has become one of the focus of educational research and practice. In this study, the epistemic network analysis method was used to process the interview text of students using teaching AIDS, and the effect of the application of teaching AIDS in classroom learning was discussed. The results show that there is a significant difference between high-score students and low-score students in using instructional software in classroom learning, especially with regards to their learning motivation towards it. Additionally, the use of epistemic network analysis technology could improve the accuracy of decision-making reference for evaluating the effect of software use and implementing accurate teaching.