Adaptive Instructional Systems

Sensor-based, mobile behavioral analytics have much potential for adaptive human-machine interactivity in team- and multiteam-based, live simulation training. This paper will explore a human-technology adaptive system where real-time data is generated from multiple sensor systems to inform multiteam-based training. Examples from first responder law enforcement training contexts will be discussed as well as the future potential of these sensor-based technologies to iteratively and adaptively inform both the smart technology system and the human system in a reciprocal learning cycle.

Despite recent advances in technology, personalized learning to address diverse needs of students remains difficult to achieve at scale. With the availability and affordability of smart devices in the era of the Internet of Things, learners, parents, and educators are more “connected” than ever before. Education stakeholders and technology developers can leverage these advances to collect data about, inform, deliver, and improve education for all learners. In this paper, we review the core components of a Smart Learning framework and describe a personalized mastery-based learning system that leverages the framework to deliver personalized learning at scale. In the context of Smart Learning in the Internet of Things, we propose an Ambient and Pervasive Personalized Learning Ecosystem (APPLE), a learner-centered approach that uses Bloom’s Four Agents of Change in the Internet of Things ecosystem to provide learners a comprehensive and personalized learning experience. This ecosystem uses people, processes, data, things, and networked connections to create new capabilities, richer learning experiences, and unprecedented educational opportunities for learners, educators, and families. We further discuss the challenges surrounding the implementation of such an ecosystem, specifically calling for applications of learning engineering approaches, the need of interoperability across systems and components, and the importance of ethical considerations.

This paper serves to connect the papers between the AIS conceptual modeling group and the architectural interchange group by deriving requirements for the required components and the information that they need to exchange. It serves as an update to the original work on the subject, prior to the establishment of the conceptual modeling subgroup.

This paper proposes an expansion of the classical ethical foundations as laid out by the IEEE’s Global Initiative on the Ethics of Autonomous and Intelligent Systems that are of particular relevance for developers and interested parties concerned with establishing standards to inform the design and implementation of adaptive instructional systems. Ethically Aligned Design [1] argues for the value of integrating the following ethical traditions into either autonomous and intelligent systems public awareness campaigns or engineering or science education programs: virtue ethics, deontological ethics, utilitarian ethics, and ethics of care. Though these traditions cover a broad spectrum of important considerations, they lack specificity for adaptive instructional systems. We argue that an alternative, more manageable and particularly relevant framework should be considered: Dewey’s notion of the ethics of moral principles and deliberation. Following from this framework, we also argue for the need to explore education of ethical thinking and related skills through the medium of adaptive instructional systems.

Providing experiential training in complex tasks on an any-time anywhere basis—whether for individual or team tasks—often requires simulating interaction with non-player characters (NPCs): co-workers, superiors, subordinates, opponents, subjects, stakeholders, consultants, tutors, peers etc.Simulating all aspects of human behavior is overwhelmingly complex. Pursuing full human simulation is also needlessly costly and distracts from the task at hand, which is providing a learner with prompts and reactions supporting experiences that promote mastery of learning objectives and appropriate transfer. The question then is, What techniques can be used to create relevantly realistic NPC agents to support desired learning outcomes?Rather than advance a one-size-fits-all silver bullet for instructional system NPC modeling, we advocate a flexibly configurable bag-of-tools approach. Using example systems that the authors have worked on, we discuss several different approaches to building NPCs for pedagogical effect. Choices of technologies to employ should be based on application requirements, considering issues such as: (1) content/authoring costs—both for achieving short term capability and for longer-term maintenance and scalability; (2) pedagogical approaches; and (3) relevant aspects of realism in behavior and interaction methods.

The current paper contains elements relevant for the conceptual modelling and interoperability of measurements and assessments across adaptive instructional systems (AIS). After an introduction, the first section presents a generic use case, where knowledge acquisition is supported by a sequence of training simulators of increasing complexity, and the need to capture regularity and variations in measurements and assessments across instructional systems. The second section briefly discusses the role of measurements and assessments as a core of functions of adaptive instructional systems. The section indicates that an adaptive instructional system needs minimally to capture references to learners’ performance, knowledge components, learning tasks, and learning attempts. The third section maps the main Generalized Intelligent Framework for Tutoring (GIFT) components to a generic feedback control system model. The mapping makes explicit the dual interpretation of measurements and assessments as both results (learning traces) and functions (computation templates). The fourth section examines some non-proprietary frameworks in terms of their capability to support the interoperability of measurements and assessments across adaptive instructional systems. The section briefly discusses xAPI, the Competency And Skill System (CASS), the Evidence Trace File (ETF), the Training Objective Package (TOP), and the Human Performance Markup Language (HPML). The main contributions of the paper are: 1) an AIS conceptual model based on a feedback control system model, and 2) a brief review of learning data frameworks as they relate to measurements and assessments as both learning traces and computation templates.

Developing context-based adaptive applications that are compliant to the law require experts from different domains, e.g. software designer, developers, legal professionals, providers and users, and a tight collaboration between them. Software providers must be assured that the applications they serve to users run compliant to the law. Additionally, it is important to support users to act according to the law, e.g. when they upload content which may concern the Copyright Law. This aspect is also relevant to providers, because they are responsible for users’ legal breaches. To support users to act compliant to the law when using an application, it is important that they understand the current situation and the related consequences to the usage of the system. This paper presents a development and provisioning process for domain-specific context-based adaptive software applications. We define roles and responsibilities as well as artifacts that need to be specified in the different lifecycle phases of the process. Using the process, related stakeholders will be able to integrate legal regulations into context-based adaptive systems. We use a sample scenario, where Copyright Law and personalized explanations get relevant, to describe how legal experts support other stakeholders in the development process and how they can configure legal requirements and provide explanations. Our approach adds flexibility to the development and provisioning process, because only relevant regulations for a specific application have to be considered.

Learners often have metacognitive deficits that limit their ability to select material at appropriate levels in independent studying situations. The increasing prevalence of intelligent recommender systems can assume this role, while also fostering a kind of experiential meta-instruction. The creation of hybrid tutors (federated systems of both adaptive and static learning resources with a single interface and learning record store) provides an opportunity to test this experiential instruction of metacognitive strategies. As a test case, we examine the hybrid tutor ElectronixTutor, which has two distinct intelligent recommender engines corresponding to distinct use cases. Each of these constitutes a method of providing scaffolding to learners so that they can internalize the principled, theoretically informed reasons for the order of their progression through learning content. However, the learning described is speculative and requires evaluation. By examining expected efficacy, perceived efficacy, actual efficacy, and especially the relationships among these three concepts, actionable insights should arise pertaining to adaptive instructional system design, learning science generally, and other areas.

To meet modern organizational job performance needs, effective training solutions must accelerate student learning while simultaneously maintaining exceptionally high standards for learner knowledge and skill proficiency. As organizations seek to revitalize traditional training programs to fit the needs of next-generation learners, innovation incubators allow for the exploration of new technologies to facilitate adaptive learning solutions. Transitioning traditional training programs through an innovation phase and toward cohesive, results-based adaptive learning solutions is a complex undertaking. This transformation is neither a simple nor painless process, and as such it presents many complicated and unexpected challenges. The key to a successful implementation of an adaptive learning program is a structure that allows for extensibility and reproducibility. These characteristics are supported by the adoption of a well-defined process maturity model and a deliberate instructional systems design framework based in human performance improvement and learning science and ensure the deliberate application of learning technologies. To mitigate challenges and ensure quality for future adaptive learning design efforts, SAIC presents a phased roadmap that describes programmatic structure required for success.

Buried explosives, such as Improvised Explosive Devices (IEDs), are a threat to operations in the military. This challenge is compounded by limits in training the military to detect IEDs using a handheld detector called the Minehound. Thus, a call for improved IED detector training is answered through testing Virtual Reality (VR) and Augmented Reality (AR) Minehound trainers: these trainers are subjected to a usability investigation. Further, the VR and AR developments are framed within a Systems Engineering Process Model. Following traditional Minehound instruction, a data collection event occurred over a two-day period, where ten Marines were asked to use the VR and AR Minehound trainers. Following the Marines’ interaction with the trainers, the Marines completed a usability questionnaire (i.e., agreement with the usefulness, ease of use, ease of learning, satisfaction, and effectiveness of the trainers; and responses to open-ended questions). Ratings indicated future iterations should not emphasize aspects of ease of use and ease of learning, such as for user interfaces, but emphasize challenging aspects, such as helping users accomplish training tasks. A lower mean score in the usefulness subscale may be linked to breaks in fidelity (e.g., lag issues, weight issues, and a non-standard Marine sweep technique). Primarily, considerations for usefulness, satisfaction, and effectiveness aspects should be highlighted in the future as per an iterative design process. A cost-benefit analysis is given to compare the traditional and experimental forms of training. Limits of the study include experimental, environmental, and technical issues.

The past 60 years has seen tremendous development in adaptive instructional systems such as simulation-based intelligent tutoring systems and media-based adaptive interactive multimedia instruction (IMI). Solid empirical findings have repeatedly demonstrated the power of these technologies. Unfortunately, there are also well-known impediments that prevent the broad adoption of these powerful tools. Among these impediments, the cost and timelines associated with development and sustainment are prominent. To address these impediments, various researchers have worked to develop authoring tools for conventional and adaptive instructional systems. These tools speed the development and maintenance of these systems, reduce the associated costs, and allow less skilled developers to create high-quality courseware. This paper will address the development of authoring tools for adaptive IMI. In it, we will trace the development of these tools over time and explore the mismatch between the current development trajectory and the needs of consumers. Using this discordance as a point of departure, we will explore the possibility of a different approach.

This paper examines the functionality of artificially-intelligent agents as a methodology for supporting automated decisions in adaptive instructional systems (AISs). AISs are artificially-intelligent, computer-based systems that guide learning experiences by tailoring instruction and recommendations based on the goals, needs, preferences, and interests of each individual learner or team in the context of domain learning objectives. AISs are a class of instructional technologies that include intelligent tutoring systems (ITSs), intelligent mentors or recommender systems, and intelligent instructional media. This paper explores various agent-based methods to gauge their impact on four automated decisions within the Learning Effect Model (LEM): 1) determining current and predicting future learner states, 2) making recommendations for new experiences (e.g., courses or problem selection), 3) selecting high level instructional strategies to influence long-term learning, and 4) selecting low level instructional tactics to influence near-term learning.

This paper examines the relationships of various functional elements within a class of instructional technologies called adaptive instructional systems (AISs) which include intelligent tutoring systems (ITSs), intelligent mentors or recommender systems, and intelligent instructional media. AISs are artificially-intelligent, computer-based systems that guide learning experiences by tailoring instruction and recommendations based on the goals, needs, and preferences of each individual learner or team in the context of domain learning objectives. Under Project 2247.1, The Institute for Electrical and Electronic Engineers (IEEE) is developing standards and guidance for the modeling of AIS to characterize what is and is not an AIS. This paper was composed to document recommendations and generate discussion about the four models that have been proposed as core to the concept of AISs: learner models, adaptive models, domain models and interface models.

Modern black-box artificial intelligence algorithms are computationally powerful yet fallible in unpredictable ways. While much research has gone into developing techniques to interpret these algorithms, less have also integrated the requirement to understand the algorithm as a function of their training data. In addition, few have examined the human requirements for explainability, so these interpretations provide the right quantity and quality of information to each user. We argue that Explainable Artificial Intelligence (XAI) frameworks need to account the expertise and goals of the user in order to gain widespread adoptance. We describe the Knowledge-to-Information Translation Training (KITT) framework, an approach to XAI that considers a number of possible explanatory models that can be used to facilitate users’ understanding of artificial intelligence. Following a review of algorithms, we provide a taxonomy of explanation types and outline how adaptive instructional systems can facilitate knowledge translation between developers and users. Finally, we describe limitations of our approach and paths for future research opportunities.

Adaptive autonomous intelligent systems (A/IS) may satisfy design functionality and user experiential requirements but prior to deployment an assessment must be made of their impact on user rights. A/IS systems may assist rather than replace humans but it is unclear where the line is drawn between supplementing human endeavour and knowledge, on the one hand, and gradual erosion of human cognitive abilities on the other. This paper makes the case for development of ethical standards for user awareness of A/IS in operation, taking account of rights under the EU General Data Protection Regulation (GDPR) and the Council of Europe Modernised Convention for the Protection of Individuals with Regard to Automatic Processing of Personal Data (Convention 108+). It sets out three main user awareness stages (pre-use, during-use, and post-use) along with consideration of commensurate rights. In the pre-use stage potential users will need to be aware that an A/IS is either fully or partially in operation, and consent to such an operation or have the option to opt out. During A/IS use if there is a part of the A/IS operation which involves a “black box” scenario, that is, it is difficult for a human to discern what the system is doing and why, then appropriate risk-based parameters need to be set for the systems use. Post-use requires users to be aware of how their data and information shared with the A/IS will be used by the system and any third parties.

With constant technological advances, there are new and evolving methods for assisting learners master content. This paper conceptualizes how an adaptive application could deliver metacognitive prompts within an online learning environment to help learners achieve their goals, shift mindsets about learning, and develop motivation to learn. The nature of the adaptive application will be described and literature on the methods for delivery and metacognitive topics reviewed. Combining adaptive technology with metacognitive learning strategies could provide a level of learning support not yet realized in current online learning environments. This paper proposes that this style of prompt-based adaptive application delivering carefully crafted metacognitive prompts on goals, mindsets, and motivation could help foster better learning habits and outcomes.

Requirements of human-machine systems change over long product lifecycles. Anticipating those changes during the initial design is challenging. Once deployed the changing requirements demands a systematic evaluation of the human-machine system, otherwise inefficiencies and accidents can happen. The interaction design concept Iteration by Design introduces a fifth design phase to the user-centred design process of the ISO 9241-210 norm to close the open evaluation loop of safety critical systems with long lifecycles. The implications of Iteration by Design are discussed in the context of resilience engineering. A higher order adaptation regulation loop copes with the complexity of the human-machine system interaction design. The resilience principles drift reaction and human in the loop are utilized to adapt the system to the changing requirements. A design assistance system is proposed to inform a design team about required changes. Finally, adaptive instructional training could be interwoven with the design assistance system to sustain the adaptability of the human-machine system. The value of Iteration by Design as an extension for user-centred design is illustrated by the accident of the USS John S. McCain from 2017.

The reasoning process about the level of student’s knowledge can be challenging even for experienced human tutors. The Bayesian networks are a formalism for reasoning under uncertainty, which has been successfully used for various artificial intelligence applications, including student modeling. While Bayesian networks are a highly flexible graphical and probabilistic modeling framework, its main challenges are related to the structural design and the definition of “a priori” and conditional probabilities. Since the AC&NL Tutor’s authoring tool automatically generates tutoring elements of different linguistic complexity, the generated sentences and questions fall into three difficulty levels. Based on these levels, the probability-based Bayesian student model is proposed for mastery-based learning in intelligent tutoring system. The Bayesian network structure is defined by generated questions related to the node representing knowledge in a sentence. Also, there are relations between inverse questions at the same difficulty level. After the structure is defined, the process of assigning “a priori” and conditional probabilities is automated using several heuristic expert-based rules.

Internet of Things (IoT) enables the creation of sensing and computing machines to enhance the level of continuous adaptation and support provided by intelligent systems to humans. Nevertheless, these systems still depend on human intervention, for example, in maintenance and (re)configuration tasks. To this measure, the development of an Adaptive Instructional System (AIS) in the context of IoT allows for the creation of new, improved learning and training environments. One can test new approaches to improve the training and perception efficiency of humans. Examples are the use of virtual and augmented reality, the inclusion of nature inspired metaphors based on biophilic design and calm computing principles and the design of technology that aims at changing the users’ behaviour through persuasion and social influence. In this work, we specifically propose a nature inspired visual representation concept, BioIoT, to communicate sensor information. Our results show that this new representation contributes to the users’ well-being and performance while remaining as easy to understand as traditional data representations (based on an experiment with twelve participants over two weeks). We present a use case under which we apply the BioIoT concept. It serves the purpose of demonstrating the BioIoT benefits in a AR setting, when applied in households and workplaces scenarios. Furthermore, by leveraging our previous experience in the development of adaptive and supportive systems based on eye-tracking, we discuss the application of this new sensing technology to the support of users in machine intervention by using the user attention, i.e., eye-gaze, on different machine parts as a way to infer the user’s needs and adapt the system accordingly. In this way, a new level of continuous support can be provided to the users depending on their skill level and individual needs in the form of contextualized instructions and action recommendations based on user attention.

The USAF has established a new approach to Specialized Undergraduate Pilot Training (SUPT) called Pilot Training Next (PTN) that integrates traditional flying sorties with VR-enabled ground-based training devices and data-driven proficiency tracking to achieve training efficiencies, improve readiness, and increase throughput. Eduworks and USC’s Institute for Creative Technologies are developing machine learning (ML) models that can measure user engagement during any computer-mediated training (simulation, courseware) and offer recommendations for restoring lapses in engagement. We are currently developing and testing this approach, called the Observational Motivation and Engagement Generalized Appliance (OMEGA) in a PTN context. Two factors motivate this work. First, one goal of PTN is for an instructor pilot (IP) to simultaneously monitor multiple simulator rides. Being alerted to distraction, attention and engagement can help an IP manage multiple students at the same time, with recommendations for restoring engagement providing further instructional support. Second, the virtual environment provides a rich source of raw data that machine learning models can use to associate user activity with user engagement. We have created a testbed for data capture in order to construct the ML models, based on theoretical foundations we developed previously. We are running pilots through multiple PTN scenarios and collecting formative data from instructors to evaluate the utility of the recommendations OMEGA generates regarding how lapsed engagement can be restored. We anticipate findings that validate the use of ML models for learning to detect engagement from the rich data sources characteristic of virtual environments. These findings will be applicable across a broad range of conventional and VR training applications.

After more than two decades of large scale use of the IntelliMetric™ Artificial Intelligence for automated marking, research into the application of AI has begun to take on a decidedly different focus. Inquiry into IntelliMetric now centers on questions of how to leverage the highly reliable and immediately available information provided by IntelliMetric into an adaptive learning environment creating a customized learning pathway. IntelliMetric’s feedback engine, MY Tutor™, provides a patented process of selective writing feedback, specific to the individual needs of the user. MY Tutor feedback incorporates comments and prompts provided to the user which are generated in response to assessment of the student’s written text. Skill level and developmentally appropriate comments on various genre specific domains are provided in real-time and on-demand modalities as selected by the user.Through the data presented in this paper we demonstrate the positive impact of providing adaptive, machine generated, instant feedback in a quantifiable manner for students in grades 3 through 8. We take a deeper dive into the adaptive feedback created by MY Tutor and discuss the analysis conducted by IntelliMetric needed to feed the MY Tutor feedback.

Automatic knowledge acquisition is a rather complex and challenging task. This paper focuses on the description and evaluation of a semi-automatic authoring tool (SAAT) that has been developed as a part of the Adaptive Courseware based on Natural Language AC&NL Tutor project. The SAAT analyzes a natural language text and, as a result of the declarative knowledge extraction process, it generates domain knowledge that is presented in a form of natural language sentences, questions and domain knowledge graphs. Generated domain knowledge presents expert knowledge in the intelligent tutoring system Tutomat. The natural language processing techniques are applied and the tool’s functionalities are thoroughly explained. This tool is, to our knowledge, the only one that enables natural language question and sentence generation of different levels of complexity. Using an unstructured and unprocessed Wikipedia text in computer science, evaluation of domain knowledge extraction algorithm, i.e. the correctness of extraction outcomes and the effectiveness of extraction methods, was performed. The SAAT outputs were compared with the gold standard, manually developed by two experts. The results showed that 68.7% of detected errors referred to the performance of the integrated linguistic resources, such as CoreNLP, Senna, WordNet, whereas 31.3% of errors referred to the proposed extraction algorithms.

Human-swarm interaction (HSI) as a research discipline can be seen as a combination of swarm robotics and human factors engineering. In this work, we combine perspectives from cognitive systems engineering and systems science to discuss the importance of operator training in HSI and how training of swarm dynamics may be implemented. The concept of neglect benevolence, i.e. temporal sensitivity in swarm control, is described as a case example for operator training in HSI. We propose the application of adaptive instructional systems to (1) optimize mission performance, (2) increase understanding of swarm dynamics and (3) allocate mental workload to different work demands over a variety of complexities. Future empirical investigations must show the utility of the proposed concepts.

This paper investigates whether mental workload can be classified in an operator setting using unobtrusive psychophysiological measures. Having reliable predictions of workload using unobtrusive sensors can be useful for adaptive instructional systems, as knowledge of a trainee’s workload can then be used to provide appropriate training level (not too hard, not too easy). Previous work has investigated automatic mental workload prediction using biophysical measures and machine learning, however less attention has been given to the level of physical obtrusiveness of the used measures. We therefore explore the use of color-, and infrared-spectrum cameras for remote photoplethysmography (rPPG) as physically unobtrusive measures. Sixteen expert train traffic operators participated in a railway human-in-the-loop simulator. We used two machine learning models (AdaBoost and Random Forests) to predict low-, medium- and high-mental workload levels based on heart rate features in a leave-one-out cross-validated design. Results show above chance classification for low- and high-mental workload states. Based on infrared-spectrum rPPG derived features, the AdaBoost machine learning model yielded the highest classification performance.

This paper reviews current research on deep knowledge tracing (DKT) and discusses the benefits of using DKT in adaptive instructional systems (AIS). Namely, DKT allows for accurate measurement of ability levels across a set of attributes in a content domain and this information can be leveraged to deliver personalized content to the learner. DKT uses a recurrent neural network with long short-term memory units (RNN-LSTM), which is difficult to interpret, although provides higher prediction accuracy than Bayesian knowledge tracing (BKT) or item response theory (IRT) measurement approaches. This makes DKT ideal for learner-focused or formative assessment systems, in which the measurement of attribute proficiencies and the delivery of relevant content to promote learning is valued above the understanding of the measurement process itself. The paper focuses on practical considerations for preparing and deploying an RNN-LSTM in a production system. Namely, data demands for training the network are explored through an analysis on real data from an adaptive tutoring program, and novel methods for training the network when no data are available and for measuring learning trajectories are proposed. Finally, strategies around monitoring production prediction services are discussed, as well as tips for approaching latency, stability, and security issues in production environments. These discussions are meant to provide a researcher or data scientist with enough information to effectively collaborate with technical teams on the production implementation of RNNs, with the goal of making cutting-edge advances in DKT available to real learners.

The training curriculum of air force cadets is currently identical for all, not taking trainees’ individual differences in skill acquisition into account. A model of physiological arousal conceptualized as “ease in flight” is proposed as an objective metric for individualization. Considering that a significant part of air force cadets training takes place on a flight simulator, the metrics used to provide cursus recommendation should be valid both in flight and in a simulator. This work concerns the validation of “ease in flight” as a metric for training individualization in a simulated task environment. Eight participants performed two consecutive flights on a low fidelity aircraft simulator, whilst wearing a chest strap to measure the electrical activity of the heart and respiratory activity. Results show that declared ease in flight and declared stress are strongly negatively correlated. In addition, measured ease in flight increased significantly from first to second flight. Together, these results suggest that the ease in flight model previously defined using data from experts in-flight generalizes to simulated flight, both from a perceived and objective point of view. Finally, the potential of the model for providing adaptive cursus recommendation through the individualized analysis of measured ease in flight across different required skills is discussed.

We conducted an experiment to evaluate the viability of using heart rate parameters for real-time adaptation of applications to the mental state of human operators. The experiment consisted of a fast-jet flying task with secondary tasks in our simulator. We created five mission segments to induce differing levels of workload. During the experiment, heart rate data and subjective workload ratings were collected. The subjective workload ratings show different workload levels for each mission segment. However, from the considered heart rate parameters, we were only able to reproduce two of the known correlations from the literature; namely, average heart rate and high frequency activity of the heart rate variability. Additionally, we encountered the opposite of the expected relationships for the RMSSD of the heart rate as well as the standard deviation across the principal axis of the Poincaré plot. We suppose that the short time-frame, which we deemed necessary for real-time applications, is a possible explanation for our surprising results. Finally, we conclude that heart rate variability parameters may not be robust enough for real-time applications, especially as each measured parameter had participants who showed converse reactions to the average.

In real life settings, human operators work in cooperation to optimize both safety and performance. The goal of this study is to assess teammates’ cooperation level using cerebral measures and machine learning techniques. We designed an experimental protocol with a modified version of the NASA MATB-II that was performed in 8 five-minute blocks. Each participant was either Pilot Flying (PF) or Pilot Monitoring (PM) with specific sub-tasks to attend to. In half the blocks they were instructed to cooperate by helping the other with one of his/her sub-tasks. Five teams of two healthy volunteers were recruited among the students of the ISAE-SUPAERO engineering school. In addition to behavioral data, their electroencephalogram (EEG) was recorded. The cooperation level of the participants was estimated using a brain-computer interface pipeline with a classification step applied on basic connectivity features, i.e. covariance matrices computed between participants’ EEG sensors. Behavioral results revealed a significant impact of cooperative instructions. Also, the implemented estimation pipeline allowed to estimate cooperative states using covariance matrices with an average accuracy of 66.6% using the signal filtered in the theta band, 64.5% for the alpha band and 65.3% for the low beta band. These preliminary estimation results are above the adjusted chance level and pave the way towards adaptive training tools based on hyperscanning for aeronautical settings.

Many real-world questions that professionals face occur in complex, dynamic environments where information is often sparse, e.g., clinical decision-making, cyber security, stock market prediction. In many cases, problems are open-ended without a single or optimal solution. Providing effective training in these ill-defined environments presents an important challenge for educators. Using the healthcare professions as a case study, this chapter outlines a framework for knowledge acquisition in the professions. It argues that dynamic, adaptive criteria must be identified based on educational theory, psychometric techniques, and properties of expert performance. From this approach, educators must develop assessment criteria that satisfice, framing problems in terms of an order of difficulty relative to the learner’s current level of comprehension. This reflects a quantitative approach to the zone-of-proximal development (ZPD), that removes the upper-bound for knowledge acquisition. In health professions education, this approach can be used to create a competency profile. Finally, given that professional practices often focus on the efficient use of resources, I argue that measures of the speed-accuracy trade-off should be used to assess expert performance.

Many professional operations require employees with complex skills. Once these skills have been taught, it is important that the skills are (a) retained, and (b) retrained when the skills start to decay. The ability to determine the precise moment when a skill needs to be retrained will have a positive effect on the productivity of the individual, and therefore also on the cost-effectiveness of scheduled training courses. However, modelling the retention of complex skills remains challenging as it is difficult to gather enough data. In this paper, we present an online adaptive instructional system that serves two purposes: (1) to gather performance data on a complex video game called Space Fortress, so that the skill retention can be modelled, and (2) to apply the newly built model directly to the participants, so that its effectiveness can be analysed. We expect that the lessons learned by building and applying the model in the context of Space Fortress will transfer to complex real-world skills.

Learner models maintain representations of students’ cognitive, metacognitive, affective, personality, social and perceptual skills. This information can be used to adapt the adaptive instructional system’s interactions with the student. Our work on caring assessments has provided us with an opportunity to explore learner modelling issues applied to assessment. This paper elaborates on issues such as the nature of the learner model, types of student emotions in assessment and opportunities for adaptations, and the role of individual differences in student characteristics that could inform an expanded learner model to support fine-tuned adjustments to assessment tasks. Other issues discussed include using cognitive and affective information to implement adaptations, as well as implications for reporting systems and open learner models, supporting student access to these systems, and data privacy and data security challenges.

Adaptive instructional systems (AIS) hold great promise for improving the efficiency and availability of quality learning experiences for lifelong learners. Some have speculated that the rich, authentic, and personalized evidence available from AIS in support of a learner’s competency may indeed become so good that the need for summative assessment will diminish. This paper suggests that to prepare for this eventuality, a number of actions may be taken to move from good intent to well understood and research best practices. Specifically, this paper examines aspects of fairness, equity of access, and security as they relate to the use of AIS evidence for eventual summative decision-making. It makes a case for minimally instrumenting AIS to allow for collection of contextual metadata in addition to learning evidence in support of robust research agendas in these areas and in preparation for proper evaluation of suitability of purpose of individual data elements during formative and summative decision-making processes.

The value and validity within highly personalized programs can either be strengthened or diminished by variability in implementation, environment, and interpretation. Developers of adaptive programs have an opportunity – a responsibility – to provide transparency and insights on how variation plays a part in the value and validity of the program. This paper describes the impact variability can have on value perceptions and the resulting degradation of trust in results from an adaptive assessment used as part of an instructional program. Described is the process used to redesign score reports that considers the context of the administration and resulting interpretations. The paper then presents a theoretical challenge to the adaptive nature of instructionally situated assessments intended to reflect ultimate use.

This paper is an exploratory theoretical study of the role of time in learning. We present a computer simulation based on Carroll’s model of school learning. Our aim is to probe some key theoretical questions in educational research: Can all students learn well? If so, under what conditions? What is time’s role in learning achievement? How does time relate to other instructional variables such as student aptitude, student perseverance, and quality of instruction? In our approach we regard learning as a causal system in which a few variables predict and explain different levels of learning. While the simulation is not a causal analysis in the strict sense, it lays some of the groundwork for a fuller causal approach. Our main result confirms the Carroll-Bloom hypothesis that time, as opportunity to learn, is a central variable in learning achievement and also key to closing the achievement gap. We also demonstrate that time, as learner perseverance, accelerates achievement, especially for less prepared students. However, perseverance becomes effective only when the instructional environment surpasses a basic quality threshold. We conclude by considering some implications for designing alternative learning environments, particularly adaptive instructional systems.

A training ecosystem leverages multiple complementary instructional resources to target competency and skill development. In this paper, we introduce work that is integrating assessment functions in the Generalized Intelligent Framework for Tutoring (GIFT) with core components in the Total Learning Architecture (TLA) to support persistent performance tracking and reporting in dynamic simulation-based environments. This capability creates a data strategy to translate multi-modal raw data into contextualized statements of performance for use in a long-term readiness monitoring strategy. In this paper we discuss the integration activities, what this new extended architecture supports, and provide a high-level use case associated with infantry squad level competency sets.

Although many praise the positive benefits of game-based training to increase learner engagement and performance, there has been little empirical research to support these claims. The goal of this experiment was to establish whether adding game features has a positive impact on performance during training and leads to better learning outcomes. Specifically, we explored whether the presence of game features (i.e., performance gauges) and competition features (i.e., leaderboard) affected motivation and learning outcomes within the Periscope Operator Adaptive Trainer (POAT). We conducted an experiment with 49 Submarine Officer Basic Course students who were assigned randomly to either training with a version of POAT with game features (Game Features condition) or one without game features (Control condition). Analyses revealed no differences between the two conditions on learning gains or reported motivation. The results did show that students in both conditions improved significantly on the accuracy (i.e., angle on the bow and range) and timeliness of their periscope calls from pre-test to post-test, providing additional support for the benefits of adaptive training but not game features.

This concept paper presents an initial exploration of measuring Situation Awareness (SA) dynamics in team settings. SA dynamics refer to the evaluation of SA’s temporal evolution of one or more teammates. We discuss why current methods are inherently limited by the subjective nature of SA and why it is important to identify measures for the temporal evolution of SA. Most current approaches focus on measuring the accuracy of SA (i.e., What? is known)and the similarity in the context of shared SA, often resulting in rather qualitative assessments. However, quantitative assessments are important to address temporal aspects (i.e., When? and ultimately For how long?). Thus, we propose, as a complementary approach to accuracy and similarity of SA, to consider the concept of SA synchrony as a quantitative metric of SA dynamics in teams. Specifically, we highlight the existence of three latencies with high relevance to shared SA dynamics and discuss options for their assessment.

This paper presents the results of student engagement with eBRAVO, an Intelligent Tutoring System designed to support students’ development of reading comprehension strategies. The eBRAVO curriculum is a personalized experience based on the students’ previous engagement with the tool as well as their demonstration of deep comprehension of the current materials. This personalization may include support in the form of video lessons that target the comprehension strategy with which the reader has recently struggled embedded within the chapter context the reader was currently working. This paper outlines the results from a deployment during a summer program supporting students reading ecology content, and shows that students are clearly distinguishable into categories that denote their patterns of engagement with these videos. It also discusses how these results connect to comprehension assessment results within the system and at a unit level and the implications these results have for the design of future classroom intervention systems.

Previous research has shown that adaptive training (AT) is an effective tool for improving training outcomes relative to non-adaptive approaches. Taking a value-added perspective, in this study we sought to determine whether the presence of an embedded non-player character (NPC) served to improve performance outcomes relative to AT alone. To support this research, we utilized the Submarine EW Adaptive Trainer (SEW-AT) as our testbed, which is a scenario-based AT system that simulates a trip to periscope depth. The submarine EW operator’s role is to monitor the radio frequency signals in the environment and submit reports of the contact picture at prescribed intervals (i.e., scheduled reports) and irregular intervals as the environment changes (i.e., unscheduled reports) to the Officer of the Deck (OOD). Sixty-eight U.S. Navy EW operators completed training with one of two versions of SEW-AT: one with an NPC OOD and one without. The NPC OOD was designed carefully to provide realistic immediate feedback to the trainee while also minimizing distraction from the task. In general, all EW operators improved their performance using SEW-AT, but those using SEW-AT with the NPC OOD displayed significantly greater improvement in scheduled report timeliness than those who used SEW-AT without the NPC OOD. These results suggest that the addition of the NPC OOD to provide immediate feedback added an overall benefit over AT alone.

The purpose of this paper is to explain the learning methodologies behind the adaptive activities within Acrobatiq’s courseware, and to investigate the impact of these adaptive activities on learning estimates and summative assessment scores using real course data. The adaptive activities used for this analysis were part of a Probability and Statistics course, which was delivered to college students at a public four-year institution as part of an educational grant. The data were analyzed to identify if the adaptive activities had an impact on learning estimates as well as on summative assessment scores. Results showed that the adaptive activities had a net positive effect on learning estimates. Results also showed that not only did learning estimate states correlate to mean summative assessment scores, but improving learning estimates after completing the adaptive activity practice yielded higher mean summative assessment scores. The implications of this analysis and future research are discussed.

Students often use flashcards to study but they do not always use them effectively. In this experiment, we explored different methods of dropping flashcards to inform the development of an adaptive flashcard-based trainer. Forty-seven U.S. Marine Corps students were randomly assigned to one of three groups in an armored vehicle training task. In the Mastery Drop condition, cards were dropped from training based on objective criteria (i.e., accuracy and reaction time). In the Learner Drop condition, cards were dropped based on the learner’s choice. In the No Drop condition, cards were not dropped during training, which served as a control group. Using a pre-test post-test design, results showed that the Learner Drop condition had the lowest learning gains on the immediate post-test and the delayed post-test (two days after training), perhaps because participants were unsuccessful at self-regulating their learning and completed training too quickly. Although the No Drop condition had the highest learning gains on the immediate post-test, the gains significantly decreased on the delayed post-test. In contrast, the Mastery Drop condition maintained consistent learning gains from immediate to delayed post-test. Although the No Drop condition completed more training trials than the Mastery Drop condition, this additional practice did not significantly aid long-term retention. Finally, the No Drop condition had the highest immediate transfer test scores, which involved identifying images of real-world vehicles, but there were no group differences on the delayed transfer test. These results suggest that adaptive flashcard training should incorporate mastery criteria, rather than learner-driven decisions about when to drop flashcards from the deck.