Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience

The Generalized Intelligent Framework for Tutoring (GIFT) project is partially an effort to standardize the systems and processes of intelligent tutoring systems. In addition to these efforts, there is emerging research in agent-driven systems. Agent-based systems obey software and messaging communication protocols and accomplish objectives to the original system, but have different architectural structure. This paper describes the upcoming research changes for GIFT, from a module-driven system to an agent-driven system, the reasons for wanting to do so, the advantages of the change, some initial technical approaches which encapsulate current functionality, and the types of research that this change will enable in the future.

Extending the application of intelligent tutoring beyond the desktop and into the physical world is a sought after capability. If implemented correctly, Artificial Intelligence (AI) tools and methods can be applied to support personalized and adaptive on-the-job training experiences as well as assist in the development of knowledge, skills and abilities (KSAs) across athletics and psychomotor domain spaces. While intelligent tutoring in a physical world is not a traditional application of such technologies, it still operates in much the same fashion as all Intelligent Tutoring Systems (ITS) in existence. It takes raw system interaction data and applies modeling techniques to infer performance and competency while a learner executes tasks within a scenario or defined problem set. While a traditional ITS observes learner interaction and performance to infer cognitive understanding of a concept and procedure, a physical ITS will observe interaction and performance to infer additional components of behavioral understanding and technique. A question the authors address in this paper is how physical interactions can be captured in an ITS friendly format and what technologies currently exist to monitor learner physiological signals and free-form behaviors? Answering the question involves a breakdown of the current state-of-the-art across technologies spanning wearable sensors, computer vision, and motion tracking that can be applied to model physical world components. The breakdown will include the pros and cons of each technology, an example of a domain model the data provided can inform, and the implications the derived models have on pedagogical decisions for coaching and reflection.

Augmented reality (AR) and virtual reality (VR) training systems provide an opportunity to place learners in high stress conditions that are impossible in real life due to safety risks or the associated costs. Using physiological classifiers it is possible to continually measure the stress levels of learners within AR and VR training environments to adapt training based on their responses. This paper reviews stress measurement approaches, outlines an adaptive stress training model that can be applied to augment training and describes key characteristics and future research that is critical to realizing adaptive VR and AR training platforms that take into account learner stress levels.

Gauging the impact of simulation-based training (SBT) technology has been straightforward in the past when applied to domains such as pilot training and ground vehicle operator training. In the dismounted infantry soldier skills domain, the low hanging fruit for effective use of (SBT) are weapons and equipment operations training. However, the complexities of the operational environment are often too difficult to replicate in current virtual environments to represent an accurate or effective training for the skills requiring identification of enemy activity or reacting to enemy contact. This paper discusses the need for an alternate method of performance assessment when comparing traditional training means to SBT.

This paper discusses our research efforts aimed at improving the training effectiveness and efficiency of the U.S. Army’s gunnery and rifle marksmanship curriculum. Soldier assessments, typically in the form of final qualifications scores, are insufficient to conduct experimental comparisons required to evaluate training effectiveness. More importantly, this level of performance assessment does not speak to the root cause of the errors soldiers make during training. Using the Experience API (xAPI) specification, learning experiences are represented in terms of activity statements and can be used to track learning that happens both inside and outside of the classroom, enabling the development of robust, persistent student models. Importantly, xAPI data are interoperable across training systems, allowing a student’s performance to be tracked across multiple platforms. Our research demonstrates the utility of xAPI to improve the effectiveness of Army simulation-based training through improved performance assessment capabilities.

Intelligent Tutoring Systems (ITS) are not yet widely implemented in learning, despite the general prevalence of digital resources in educational and training environments. ITS have been demonstrated to be effective for learners, but ITS development is not yet efficient for authors. Creating an ITS requires time, resources, and multidisciplinary skills. Authoring tools are intended to reduce the time and skill required to create an ITS, but the current state of those tools is categorized as a series of design tradeoffs between functionality, generalizability, and usability. In practice, the former two factors matter little if potential authors disregard the ITS in favor of other solutions. In this sense, authors, not learners, are the primary users of an ITS; the user experience of authors is critical to greater ITS adoption at an organizational level. With those challenges in mind, ongoing work and lessons learned on the design of authoring tools are described for a specific ITS platform, the Generalized Framework for Intelligent Tutoring (GIFT). User-centered design considerations are examined through the lens of authors’ goals, mental models for authoring, and the definition of authoring sub-roles. Recommendations for authoring tool design and future research directions for design research in authoring tools are discussed.

The goal of the Squad Overmatch (SOvM) for Tactical Combat Casualty Care (TC3) study was to introduce and assess an integrated training approach (ITA) for producing adaptable, high performing infantry squads. The challenge is to create the conditions and encode learning experiences for re-use in combat situations. Effective performance embedded in force-on-force actions are unscripted and required unpacking to understand and use as performance feedback. This paper describes the development of a prototype team performance observation tool developed to support the assessment of mission critical tasks during the simulation and live training phases of the ITA. The tool was constructed based on tactical use cases developed with subject matter experts. Discrete TC3 tasks were defined so that observers could recognize and record squad member performance, and that could be traceable to understanding underlying cognitions of team members during an after action review. Lessons learned on usability and reliability of the tool are discussed.

Research has examined the benefits and retractors of immersing the learner in an environment. Immersive computer-based training environments are costly to construct and may not always lead to significant learning or transfer benefits over other methods. The current paper presents a brief review of presence and immersion research in computer-based learning and adaptive tutoring. The Generalized Intelligent Framework for Tutoring (GIFT) is an open source domain-independent framework for creating intelligent tutoring systems (ITS). GIFT offers flexibility, and can be interfaced with training applications ranging from highly immersive computer-based learning environments (e.g., TC3Sim, VBS2) to less immersive mediums such as PowerPoint. The capabilities of GIFT that can be used to create immersive adaptive tutoring are discussed. Additionally, the use of GIFT to run and generate experimental studies to examine the impact of immersion is highlighted. Finally, recommendations are given on how to provide more opportunities to integrate immersive environments into GIFT.

This paper discusses critical elements of adaptive instruction in support of training and education. Modeling and assessing learners and teams, optimizing adaptive instructional methods, applying domain modeling outside of traditional training and educational domains, automating authoring processes, and assessing the learning effect of instruction are among the challenges reviewed.

This paper discusses tools and methods which are needed to support adaptive instruction for individual learners within the Generalized Intelligent Framework for Tutoring (GIFT), an open source architecture for authoring, adapting instruction, and evaluating intelligent tutoring systems. Specifically, this paper reviews the learning effect model (LEM) which drives adaptive instruction within GIFT-based tutors. The original LEM was developed in 2012 and has been enhanced over time to represent a full range of function encompassing both the learner’s and the tutor’s interactions and decisions. This paper proposes a set of 10 functions to enhance the scope and functionality of the LEM and to extend it to be a career-long model of adaptive instruction and competency.

The Flip-Flop instructional methodology involves students in creating quizzes synchronized with video recordings of lectures. While students create questions, which involves generating right and wrong answers, feedback for the answers, hints and links leading to relevant resources, they get deeply involved with the content presented in the lecture screencasts. We propose to conduct a wide range of experiments testing the effectiveness of this approach – from simple surveys, evaluations of time spent creating quizzes and assessment of their quality to extensive longitudinal studies of the students’ emotional responses and cognitive load using a electroencephalography (EEG), electrodermal activity (EDA), heart rate variability (HRV) and facial electromyography (EMG).

Inferring the cognitive state of an individual in real time during task performance allows for implementation of corrective measures prior to the occurrence of an error. Current technology allows for real time cognitive state assessment based on objective physiological data though techniques such as neuroimaging and eye tracking. Although early results indicate effective construction of classifiers that distinguish between cognitive states in real time is a possibility in some settings, implementation of these classifiers into real world settings poses a number of challenges. Cognitive states of interest must be sufficiently distinct to allow for continuous discrimination in the operational environment using technology that is currently available as well as practical to implement.

We present an empirical study using eye tracking equipment to understand how novices read source code in the context of two introductory programming classes. Our main goal is to begin to understand how novices read source code and to determine if we see any improvement in program comprehension as the course progresses. The results indicate that novices put in more effort and had more difficulty reading source code as they progress through the course. However, they are able to partially comprehend code at a later point in the course. The relationship between fixation counts and durations is linear but shows more clusters later in the course, indicating groups of students that learned at the same pace. The results also show that we did not see any significant shift in learning (indicated by the eye tracking metrics) during the course, indicating that there might be more than one course that needs to be taken over the course of a few years to realize the significance of the shift. We call for more studies over a student’s undergraduate years to further learn about this shift.

Designing and testing products for high-risk emergencies is a challenging task, especially due to the inhibitive cost of building testing environments that recreate the psychological pressures of the field. The chaotic nature of emergency environments makes gathering accurate data amidst the chaos of such environments difficult, while ethical and practical considerations limit prototype deployment in potentially life-threatening situations. These environments pose serious risk to physical and mental well-being. This paper provides a case study to examine the benefits and drawbacks of a Virtual Reality (VR) environment to test prototypes of a tool for firefighters. The VR simulated environment out performs a physical simulation because it is cheaper and safer, generates more reliable data, and provides greater control and flexibility of prototypes, allowing designers to test prototypes more rapidly than in a physical environment. This paper summarizes a 9-month Draper-sponsored capstone project with 5 HCII students.

In this paper we describe current and historic permutations of control room technology and describe a new set of design principles for digitally displaying process control parameters. The design principles focus on helping operators effectively monitor changes during process control. The change detection approach is called RevealFlow and is illustrated in the context of the Computerized Operator Support System currently being developed for nuclear power plant control rooms.

Calibrated trust in an automation is a key factor supporting full integration of the human user into human automation integrated systems. True integration is a requirement if system performance is to meet expectations. Trust in automation (TiA) has been studied using surveys, but thus far no valid, objective indicators of TiA exist. Further, these studies have been conducted in tightly controlled laboratory environments and therefore do not necessarily translate into real world applications that might improve joint system performance. Through a literature review, constraints on an operational paradigm aimed at developing indicators of TiA were established. Our goal in this paper was to develop an operational paradigm designed to develop valid TiA indicators using methods from human factors and cognitive neuroscience. The operational environment chosen was driving automation because most adults are familiar with the task and its consequent structure and therefore required little training. Initial behavioral and survey data confirm that the design constraints were met. We therefore believe that our paradigm provides a valid means of performing operational experiments aimed at further understanding TiA and its psychophysiological underpinnings.

The goal of this study is to explore how ocular behavior is different in groups that possessed varying levels of performance in dynamic control tasks with complex visual components. Twenty two university students participated in this study by operating a human-in-the-loop (HITL) simulator. The participants were asked to identify unknown air track(s) and take proper actions to defend a battleship. During the experiment, a head-mounted eye-tracking device was used continuously to record participants’ visual attention span regarding the normalized coordinates of their gaze points. In the current study, fixation duration was the main eye-tracking metrics. Air track identification accuracy and the NASA Task Load Index (NASA-TLX) were also used to measure participants’ task performance and overall subjective mental workload.

Operations in cyberspace are enabled by a digitized battlefield. The ability to control operations in cyberspace has become a central goal for defence forces. As a result, terms like cyber power, cyberspace operations and cyber deterrence have begun to emerge in military literature in an effort to describe and highlight the importance of related activities. Future military personnel, in all branches, will encounter the raised complexity of joint military operations with cyber as the key enabler. The constant change and complexity raises the demands for the structure and content of education and training. This interdisciplinary contribution discusses the need for a better understanding of the relationships between cyberspace and the physical domain, the cognitive challenges this represents, and proposes a theoretical framework - the Hybrid Space - allowing for the application of psychological concepts in assessment, training and action.

The general public’s understanding of “secure” passwords, and how they are generated is investigated. Habits that tend to foster the creation of more secure passwords are suggested. Empirical data collected by survey participants is shown to present solid evidence that “secure” passwords created by the participants who could recall them later contained substantial substrings of simpler password chosen earlier by the participants. In contrast, those who encounter difficulty in recalling the passwords are seen to have created complex passwords substantially different from simpler ones created earlier. Some user-coping methods for the complexity-memorability dilemma are addressed, Companies are urged to adopt a salting approach before encryption, and consider new hashing mechanisms to ensure the security of user passwords. Given the limitations of human memory, it is recommended that two-factor authentication be used.

Remotely piloted aircraft systems (RPAS) are steadily increasing in their presence and role in the Military’s overall strategic operational picture. The benefits of RPAS are apparent, ranging from saving time, money, and lives. Yet, the utilization of RPAS is still very challenging in many different aspects. Teams have become a central focus of RPAS due to their many benefits. Yet, teamwork is challenging and the RPAS community must continue to attempt to understand how to support it. A specific aspect of teamwork that has proven over the years to be of paramount importance is team cognition. In this paper, we discuss how team cognition needs to be considered during the development of collaborative and proactive RPAS decision support. We highlight the concept of team cognition accounting for multiple perspectives, outline an integrative perspective of team cognition for the RPAS domain, and conclude by outlining multiple design objectives for utilizing team cognition as a mechanism for RPAS decision support.

This paper discusses decision making challenges involved in the management of multiple unmanned vehicles within a dynamic mission environment. Given the increased likelihood of this new supervisory control paradigm, the authors developed the Supervisory Control Operations User Testbed (SCOUT). A brief overview of SCOUT will be provided, followed by a summary of initial research conducted within the testbed which demonstrates how eye tracking measurements can be utilized to assess workload and predict situation awareness. Subsequent discussion will address challenges associated with dynamic decision making under uncertainty, with respect to multiple asset allocation. Techniques for measuring the accuracy of these decisions as well as assessing operator risk throughout the mission will also be presented. The paper concludes with discussion of how these new decision making metrics can be used to drive decision aids and compares decision making performance and risk bias under varying levels of task load.

‘Big data’ is a phrase that has gained much traction recently. It has been defined as ‘a broad term for data sets so large or complex that traditional data processing applications are inadequate and there are challenges with analysis, searching and visualization’ [1]. Many domains struggle with providing experts accurate visualizations of massive data sets so that the experts can understand and make decisions about the data e.g., [2, 3, 4, 5].Abductive reasoning is the process of forming a conclusion that best explains observed facts and this type of reasoning plays an important role in process and product engineering. Throughout a production lifecycle, engineers will test subsystems for critical functions and use the test results to diagnose and improve production processes.This paper describes a value-driven evaluation study [7] for expert analyst interactions with big data for a complex visual abductive reasoning task. Participants were asked to perform different tasks using a new tool, while eye tracking data of their interactions with the tool was collected. The participants were also asked to give their feedback and assessments regarding the usability of the tool. The results showed that the interactive nature of the new tool allowed the participants to gain new insights into their data sets, and all participants indicated that they would begin using the tool in its current state.

Present aircraft are highly automated systems. In general, automation improved aviation safety significantly. However, automation exhibits itself in many forms of adverse behaviors related to human factors problems. A major finding is that insufficient support of partnership between the pilot crew and the aircraft automation can result in conflicting intentions. The European project A-PiMod (Applying Pilot Models for Safer Aircraft) addresses issues of conventional automation in the aviation domain. The overall objective of the project is to foster pilot crew-automation partnership on the basis of a novel architecture for cooperative automation. An essential part of the architecture is an intention recognition module. The intention recognition module employs a Hidden Markov Model (HMM) to infer the most probable current intention of the human pilots. The HMM is trained and evaluated with data containing interactions of human pilots with the aircraft cockpit systems. The data was obtained during experiments with human pilots in a flight simulator.

Despite the shift in scholarly and managerial attention from the effects of objective distance to psychological distance between team members, the questions of how virtual worker psychological distance is formed and how it can be measured have not been fully answered. In search of answers, this study develops and tests a model that examines the antecedents and the consequence of psychological distance, drawing on construal level theory. The results reveal that psychological distance is a multi-dimensional construct consisting of responsiveness, subjective proximity, and accessibility. The results also demonstrate that the different dimensions of objective distance increase psychological distance in a nonlinear fashion; interaction between different dimensions of objective distance produces greater psychological distance than does their combination. Our findings contribute to the IS literature by conceptualizing psychological distance, showing that it can be directly modeled, and highlighting that it should be carefully managed to improve virtual workers’ job performance.

Tactical medical situations require squads to coordinate achieving tactical mission objectives while providing competent medical treatment. A tactical situation may require foregoing all but the most essential point-of-wounding care until tactical dangers are suppressed (effective shooting stops) and security allows for more definitive treatment. Core knowledge and skills, within the content areas of advanced situational awareness, resilience, tactical combat casualty care, and team performance can help teams coordinate medical and tactical team decisions and tasks. The objective of the Squad Overmatch Tactical Combat Casualty Care (SOvM TC3) project was to improve individual and team performance within the context of tactical medical care. To do this, the team utilized the Team Dimensional Training (TDT) model to integrate and train the above skills through guided team self-correction [3]. The empirically derived expert model of teamwork (TDT) has been found to be effective in a variety of team settings. Smith-Jentsch, Cannon-Bowers, Tannenbaum, and Salas (2008) demonstrated that teams who participated in facilitator-led guided self-correction developed more accurate mental models of teamwork, demonstrated superior teamwork processes, and achieved more effective performance outcomes than did those briefed and debriefed using a traditional method. This effort extended the TDT model to the core skills within an integrated training curriculum for tactical medical skills. This paper discusses team members’ reported efficacy of the TDT approach in fostering individual and team process skills.

We present here an example of how a large, multi-dimensional unstructured data set, namely aircraft trajectories over the United States, can be analyzed using relatively straightforward unsupervised learning techniques. We begin by adding a rough structure to the trajectory data using the notion of distance geometry. This provides a very generic structure to the data that allows it to be indexed as an n-dimensional vector. We then do a clustering based on the HDBSCAN algorithm to both group flights with similar shapes and find outliers that have a relatively unique shape. Next, we expand the notion of geometric features to more specialized features and demonstrate the power of these features to solve specific problems. Finally, we highlight not just the power of the technique but also the speed and simplicity of the implementation by demonstrating them on very large data sets.

The execution of sequential tasks tightly bound to the daily lives of people. Being possible to identify it during the keyboard typing, creating sequences of actions on the steering wheel or even playing a musical instrument. Researches shows is possible use the sequence learning as an executive function training. Which is considered essential skills for fiscal and mental health, life and scholar success, in addiction of the cognitive, social and psychological development. Other possible way to train executive functions is the use of digital games. In this context, in this work was developed a prototype of a digital game that permits a player to train the executive function working memory. The game permits the player to interact with a serial sequence, while his reaction time are collected for the progress evaluate during a match.

Research provides ample evidence of the impact web page design has on comprehension; and that Generation Y users are impatient and dislike reading text. Yet there has been little research that focuses on content, in particular to examine the impact of text simplification on younger users’ processing of textual information. To address this need, we report the initial steps of a larger research effort that focuses on developing a set of guidelines for designing simple and effective text passages. Specifically, we compiled a set of existing plain language rules and tested its effectiveness of conveying information to Generation Y users. The results suggest the compiled set of rules can serve as an appropriate tool for designing textual passages to reduce cognitive effort and improve readability of textual content for Generation Y users. Also, the results show that eye tracking serves as an excellent objective measurement for examining the effectiveness of text simplification.

Autonomous agents, both software and robotic, are becoming increasingly common. They are being used to supplement human operators in accomplishing complex tasks, often acting as collaborators or teammates. Agents can be designed to keep their human operators ‘in the loop’ by reporting information concerning their internal decision making process. This transparency can be expressed in a number of ways, including the communication of the human and agent’s respective responsibilities. Agents can communicate information supporting transparency to human operators using visual, auditory, or a combination of both modalities. Based on this information, we suggest an approach to exploring the utility of the teamwork model of transparency. We propose some considerations for future research into feedback supporting teamwork transparency, including multimodal communication methods, human-like feedback, and the use of multiple forms of automation transparency.

People often become disoriented and frustrated when navigating complex, multi-level buildings. We argue that the principle reason underlying these challenges is insufficient access to the requisite information needed for developing an accurate mental representation, called a multi-level cognitive map. We postulate that increasing access to global landmarks (i.e., those visible from multiple locations/floors of a building) will aid spatial integration between floors and the development of these representations. This prediction was investigated in three experiments, using either direct perception or Augmented Reality (AR) visualizations. Results of Experiment 1 demonstrated that increasing visual access to a global landmark promoted multi-level cognitive map development, supporting our hypothesis. Experiment 2 revealed no reliable performance benefits of using two minimalist (icon-based and wire-frame) visualization techniques. Experiment 3, using a third X-ray visualization, showed reliably better performance for not only a no-visualization control but also the gold standard of direct window access. These results demonstrate that improving information access through principled visualizations benefit multi-level cognitive map development.

Organizing multivariate time series data for presentation to an analyst is a challenging task. Typically, a dataset contains hundreds or thousands of datapoints, and each datapoint consists of dozens of time series measurements. Analysts are interested in how the datapoints are related, which measurements drive trends and/or produce clusters, and how the clusters are related to available metadata. In addition, interest in particular time series measurements will change depending on what the analyst is trying to understand about the dataset.Rather than providing a monolithic single use machine learning solution, we have developed a system that encourages analyst interaction. This system, Dial-A-Cluster (DAC), uses multidimensional scaling to provide a visualization of the datapoints depending on distance measures provided for each time series. The analyst can interactively adjust (dial) the relative influence of each time series to change the visualization (and resulting clusters). Additional computations are provided which optimize the visualization according to metadata of interest and rank time series measurements according to their influence on analyst selected clusters.The DAC system is a plug-in for Slycat (slycat.readthedocs.org), a framework which provides a web server, database, and Python infrastructure. The DAC web application allows an analyst to keep track of multiple datasets and interact with each as described above. It requires no installation, runs on any platform, and enables analyst collaboration. We anticipate an open source release in the near future.

The National Alliance on Mental Illness reports that one in four adults experience mental health issues in a given year. Stigmas surrounding mental health issues often leave those afflicted reluctance to seek treatment. Those individuals that do decide to pursue treatment are often denied because of cost and lack of health care coverage or simply do not know where to find it. Assisted technologies can bridge these gaps, providing not only information on how to manage symptoms, but viable treatment options (e.g., adaptive management plans and training, physiological sensing, and alerts for physical symptom onset). The pairing of wearable technology, smart applications, and blended learning techniques can teach patients and caregivers the skills needed for lifetime management. The present theoretical paper provides a literature review of current technology platforms that can be utilized by the mental health domain and explores viable mental health technology options for the next five years.

Although haptic cueing is well researched, its effects on performance accuracy and workload are mixed (Hancock et al. 2013). As such, there is still a need to further develop our understanding of the effects of haptic cueing on performance and workload. The objective of this effort is to develop a cost-effective and non-invasive experimental methodology to investigate the effects of haptic cueing on unmanned aerial vehicle operator performance and workload utilizing commercial off-the-shelf products, specifically, Unity 3D™ - Game Engine and an Xbox 360™ controller.

The aim of this study was to determine if dimensions and sub-categories of a previously-tested obtrusiveness framework were represented in interviews conducted with community-dwelling older adults at three- and six-month study visits during an in-home sensor study. Secondary analysis of interviews was performed using a codebook based on an obtrusiveness framework. Eight community-dwelling older adults aged 79–86 participated in 15 interviews. One participant died between the three- and six-month interviews. Some elements of the obtrusiveness framework were present at three months but not at six months, indicating that perceptions of obtrusiveness of in-home sensors may decline over time. Findings highlight the importance of privacy issues and perceived usefulness for sensor technology use and adoption. There is a need to develop an obtrusiveness assessment instrument that enables nuanced measurements based on specific contexts and types of technologies.

Human-machine teaming is becoming an ever present aspect of executing modern military missions. In this paper, we discuss an extensive line of research currently being conducted at the Air Force Institute of Technology focused specifically on using simulation in the design of automated systems in order to improve human-automation interactions. This research includes efforts to predict operator performance, mental workload, situation awareness, trust, and fatigue. This research explores using simulation to design interfaces, perform trade studies, create adaptive systems, and make task allocation decisions.

In this paper we discuss various navigational aids for people who have a visual impairment. Navigational technologies are classified according to the mode of accommodation and the type of sensor utilized to collect environmental information. Notable examples of navigational aids are discussed, along with the advantages and disadvantages of each. Operational and design considerations for navigational aids are suggested. We conclude with a discussion of how multimodal interaction benefits people who use technology as an accommodation and can benefit everyone.

Effective ways of presenting digital data are needed to augment a user’s experience in the real world without distracting or overloading them. We propose a system of systems approach for the design, development, and evaluation of information presentation devices, particularly augmented reality devices. We developed an evaluation environment that enables the synchronized presentation of multimodal stimuli and collection of user responses in an immersive environment. We leveraged visual, audio, thermal, and tactile information presentation modalities during a navigation and threat identification task. Twelve participants completed the task while response time and accuracy data were collected. Results indicated variability among devices and pairs of devices, and suggested that information presented by some pairs of devices was more effective and easily acted upon than that presented by others. The results of this work provided important guidance regarding future design decisions and suggest the utility of our system of systems approach. Implications and future directions are discussed.

Applications in Augmented Cognition can be hampered by obstacles to the effective instrumentation of the data space, making the collection of informative feature data difficult. These obstacles usually arise from technical limitations, but can also be present due to methodological and legal considerations. We address a specific instance of the difficulty of characterizing a complex behavior space under legally constrained data collection: the instrumentation of social media platforms, where privacy, policy, and marketing considerations can severely hamper 3rd-party data collection activities. This paper documents our constrained empirical analysis and characterization of the behaviors of Twitter account-holders from their account metadata alone. The characterization is performed by coding user account data as feature vectors in a low-dimensional Euclidean space, then applying parametric and non-parametric methods to the resulting empirical distribution. Suggestions for future work are offered.

In this paper we describe a cognitive model for provoking gestural dialogue with humans, embodied in an interactive marionette. The cognitive model is a framework for the design and implementation of a gesture to gesture interaction. The marionette perceives gestures of humans using a Microsoft Kinect, reasons about perceived gestures to determine a response, and then performs the selected response gesture. This simple cognitive model: perceive-reason-perform, operates in a social context where humans interact with the marionette. The marionette was built as a 3D replica of a human body. The marionette’s responses were designed using interaction design techniques such as body storming, gesture elicitation, and the “Wizard of Oz” method to provoke an emotional response from humans. Several user studies were conducted during and after the design process to guide the design goal of achieving an engaging and provocative interaction. These studies showed that participants were encouraged to engage in a gesture-based dialogue with the marionette, and that they perceived the system to possess a kind of intelligence.

Analysts across national security domains are required to sift through large amounts of data to find and compile relevant information in a form that enables decision makers to take action in high-consequence scenarios. However, even the most experienced analysts are unable to be 100 % consistent and accurate based on the entire dataset, unbiased towards familiar documentation, and are unable to synthesize and process large amounts of information in a small amount of time. Sandia National Laboratories has attempted to solve this problem by developing an intelligent web crawler called Huntsman. Huntsman acts as a personal research assistant by browsing the internet or offline datasets in a way similar to the human search process, only much faster (millions of documents per day), by submitting queries to search engines and assessing the usefulness of page results through analysis of full-page content with a suite of text analytics. This paper will discuss Huntsman’s capability to both mirror and enhance human analysts using intelligent web crawling with analysts-in-the-loop. The goal is to demonstrate how weaknesses in human cognitive processing can be compensated for by fusing human processes with text analytics and web crawling systems, which ultimately reduces analysts’ cognitive burden and increases mission effectiveness.

During materiel development, limitations of soldiers and their interactions with tasks and equipment are often inadequately considered until after product development. This can result in poor requirements generation and thus inadequate specifications [1]. These flaws have produced the largest cost driver in acquisition programs: performance requirement changes [2]. The Army has begun work to incorporate the human dimension into future materiel development of both equipment and training systems. Modeling and Simulation (M&S) have been viewed as ways to train soldiers and to predict performance before money has been invested in creating and fielding new products. The success of early M&S in reducing cost hinges on understanding how the human, task, and equipment work together and impact each other. In addition, their relationship must be linked to cognitive aspects of performance, especially under high arousal conditions. The Army currently lacks a way to describe these relationships. The goal of this project is to create a methodology to define the data needed to describe the relationship between levels of stress or arousal and soldier performance using a live training event. The methodology should provide the training and modeling communities with information on gaps in their technologies that prevent effective training or accurate predictive analysis through modeling efforts. The methodology will also help define measures of performance needed to assess training and correctly model performance.

Research, the manufacture of knowledge, is currently practiced largely as an “art,” not a “science.” Just as science (understanding) and technology (tools) have revolutionized the manufacture of other goods and services, it is natural, perhaps inevitable, that they will ultimately also be applied to the manufacture of knowledge. In this article, we present an emerging perspective on opportunities for such application, at three different levels of the research enterprise. At the cognitive science level of the individual researcher, opportunities include: overcoming idea fixation and sloppy thinking, and balancing divergent and convergent thinking. At the social network level of the research team, opportunities include: overcoming strong links and groupthink, and optimally distributing divergent and convergent thinking between individuals and teams. At the research ecosystem level of the research institution and the larger national and international community of researchers, opportunities include: overcoming performance fixation, overcoming narrow measures of research impact, and overcoming (or harnessing) existential/social stress.