Interactive Collaborative Information Systems

Dynamic programming (DP) and reinforcement learning (RL) can be used to address problems from a variety of fields, including automatic control, artificial intelligence, operations research, and economy. Many problems in these fields are described by continuous variables, whereas DP and RL can find exact solutions only in the discrete case. Therefore, approximation is essential in practical DP and RL. This chapter provides an in-depth review of the literature on approximate DP and RL in large or continuous-space, infinite-horizon problems. Value iteration, policy iteration, and policy search approaches are presented in turn. Model-based (DP) as well as online and batch model-free (RL) algorithms are discussed. We review theoretical guarantees on the approximate solutions produced by these algorithms. Numerical examples illustrate the behavior of several representative algorithms in practice. Techniques to automatically derive value function approximators are discussed, and a comparison between value iteration, policy iteration, and policy search is provided. The chapter closes with a discussion of open issues and promising research directions in approximate DP and RL.

Relational reinforcement learning is a promising direction within reinforcement learning research. It upgrades reinforcement learning techniques using relational representations for states, actions, and learned value functions or policies to allow natural representations and abstractions of complex tasks. Multi-agent systems are characterized by their relational structure and present a good example of a complex task. In this article, we show how relational reinforcement learning could be a useful tool for learning in multi-agent systems. We study this approach in more detail on one important aspect of multi-agent systems, i.e., on learning a communication policy for cooperative systems (e.g., resource distribution). Communication between agents in realistic multi-agent systems can be assumed costly, limited, and unreliable. We perform a number of experiments that highlight the conditions in which relational representations can be beneficial when taking the constraints mentioned above into account.

This chapter gives an overview of the state of the art in decision-theoretic models to describe cooperation between multiple agents in a dynamic environment. Making (near-) optimal decisions in such settings gets harder when the number of agents grows or the uncertainty about the environment increases. It is essential to have compact models, because otherwise just representing the decision problem becomes intractable. Several such model descriptions and approximate solution methods, studied in the Interactive Collaborative Information Systems project, are presented and illustrated in the context of crisis management.

This chapter discusses decision making under uncertainty. More specifically, it offers an overview of efficient Bayesian and distribution-free algorithms for making near-optimal sequential decisions under uncertainty about the environment. Due to the uncertainty, such algorithms must not only learn from their interaction with the environment but also perform as well as possible while learning is taking place.

Ant colony optimization (ACO) is an optimization heuristic for solving combinatorial optimization problems and is inspired by the swarming behavior of foraging ants. ACO has been successfully applied in various domains, such as routing and scheduling. In particular, the agents, called ants here, are very efficient at sampling the problem space and quickly finding good solutions. Motivated by the advantages of ACO in combinatorial optimization, we develop a novel framework for finding optimal control policies that we call Ant Colony Learning (ACL). In ACL, the ants all work together to collectively learn optimal control policies for any given control problem for a system with nonlinear dynamics. In this chapter, we discuss the ACL framework and its implementation with crisp and fuzzy partitioning of the state space. We demonstrate the use of both versions in the control problem of two-dimensional navigation in an environment with variable damping and discuss their performance.

Teamwork is important in many organizations. When a task is assigned to a team rather than to an individual, there are several benefits, hence the appealing maxim “two heads are better than one.” Sometimes the job requires bringing people together who are dispersed across different geographical locations to work on a common task using information and communication technologies. These teams are often called virtual teams [36].

Partially Observable Markov Decision Processes (POMDPs) have been demonstrated empirically to be good models for robust spoken dialogue design. This chapter shows that such models are also very appropriate for designing affective dialogue systems. We describe how to model affective dialogue systems using POMDPs and propose a novel approach to develop an affective dialogue model using factored POMDPs. We apply this model for a single-slot route navigation dialogue problem as a proof of concept. The experimental results demonstrate that integrating user’s affect into a POMDP-based dialogue manager is not only a nice idea but is also helpful for improving the dialogue manager performance given that the user’s affect influences their behavior. Further, our practical findings and experiments on the model tractability are expected to be helpful for designers and researchers who are interested in practical implementation of dialogue systems using the state-of-the-art POMDP techniques.

Crisis response and management involve the collaboration of many people. To perform and coordinate their activities, they must rely on detailed and accurate information about the crisis, the environment, and many more factors. To ensure collaboration of emergency services and high-quality care for victims, the ability to supply dynamic and contextually correlated information is necessary. However, current approaches to construct globally consistent views of crises suffer from problems identified in [60]: (a) the setting of events is constantly changing, (b) the information is distributed across geographically distant locations, and (c) the complexity of the crisis management organization makes it difficult and time consuming to collaborate and verify obtained information.

Recent advances in interactive pen-aware systems, pattern recognition technologies, and human–computer interaction have provided new opportunities for pen-based communication between human users and intelligent computer systems. Using interactive maps, users can annotate pictorial or cartographic information by means of pen gestures and handwriting. Interactive maps may provide an efficient means of communication, in particular in the envisaged contexts of crisis management scenarios, which require robust and effective exchange of information. This information contains, e.g., the location of objects, the kind of incidents, or the indication of route alternatives. When considering human interactions in these contexts, various pen input modes are involved, like handwriting, drawing, and sketching. How to design the required technology for grasping the intentions of the user based on these pen inputs remains an elusive challenge, which is discussed in this chapter. Aspects like the design of a suitable set of pen gestures, data collection in the context of the envisaged scenarios, and the development of distinguishing features and pattern recognition technologies for robustly recognizing pen input from varying modes are described. These aspects are illustrated by presenting our recent results on the development of interactive maps within the framework of the ICIS project on crisis management systems.

This chapter concerns user responses toward adaptive and autonomous system behavior. The work consists of a review of the relevant literature off-set with the findings from three studies that investigated the way people respond to adaptive and autonomous agents. The systems evaluated in this chapter make decisions on behalf of the user, behave autonomously and need user feedback or compliance. Apart from the need for systems to competently perform their tasks, people will see adaptive and autonomous system behavior as social actions. Factors from humans’ social interaction will therefore also play a role in the experience users will have when using the systems. The user needs to trust, understand and control the system’s autonomous actions. In some cases the users need to invest effort in training the system so that it can learn. This indicates a complex relationship between the user, the adaptive and autonomous system and the specific context in which the system is used. This chapter specifically evaluates the way people trust and understand a system as well as the effects of system transparency and autonomy.

For human pose recovery, the presence of occlusions due to objects or other persons in the scene remains a difficult problem to cope with. However, recent advances in the area of human detection allow for simultaneous segmentation of humans and the prediction of occluded regions. In this chapter, we present an example-based pose recovery approach where this information is used. We effectively used the grid-based nature of histograms of oriented gradients descriptors to ignore part of the image observation space. This allowed us to recover poses directly, even in the presence of significant occlusions. We evaluated our approach on the HumanEva-I dataset, where we simulated different occlusion conditions. Without occlusion, we obtained relative 3D errors of approximately 69 mm. Our results showed approximately 10% increase in error when 20% of the observation is occluded. When 33% of the observation is occluded, the error is on average 15% higher compared to the observations without occlusions. These results showed that poses can be recovered from partially occluded observations, with a moderate increase in error. To the best of our knowledge, our approach is the first to investigate the effect of partial occlusions in a direct matching approach. Future work is aimed at combining our work with human detection.

In any multi-actor environment, there is an inevitable trade-off between achieving global coordination of activities and respecting the autonomy of the actors involved. Agile and resilient behavior demands dynamic coordination capabilities, but task and resource allocation quickly becomes challenging because of individual constraints and demands. In this study, we present research on adaptive autonomy in multi-agent organizations. We have studied the relationship between autonomy and coordination, and developed an agent reasoning model not only that enables collaborative task coordination, but also guarantees individual autonomy—the capability to self-manage behavior. We define autonomy as the amount of influence other agents have on one’s decision-making process. We have given the agent options to adapt its openness to external influences, so it can change its own level of autonomy. This allows agents to select the level of autonomy that best fits the circumstances, given a certain tasking, individual policies and organizational structure. We have incorporated this concept in a practical model and added heuristics for environmental events, information relevance and organizational rules. Our approach addresses fundamental collaborative challenges in dynamic environments, and may bring about new perspectives on autonomy in collaborative environments.

In this chapter, we discuss the design of adaptive hierarchical organizations for multi-agent systems (MAS). Hierarchical organizations have a number of advantages such as their ability to handle complex problems and their scalability to large organizations. By introducing adaptivity in the structure of hierarchical MAS organizations, we enable agents to balance resources in their organization. We will first provide a number of generic principles for the design of hierarchical MAS organizations. We show how these principles are used to design three different hierarchical organizations for a search and rescue task in the RoboCupRescue simulation environment. The first two of these organizations are static, and the third is able to adapt its structure. An empirical study on the performance of these three organizations shows that the dynamic organization performs better than the two static organizations.

Advances in network technologies enable distributed systems, operating in complex physical environments, to co-ordinate their activities over larger areas within shorter time intervals. Some envisioned application domains for such systems are defence, crisis management, traffic management and public safety. In these systems, humans and intelligent machines will, in close interaction, be able to adapt their behavior under changing conditions and situations to reach their common goals. Various architecture models are proposed for such Networking Adaptive Interactive Hybrid Systems (NAIHS) from different research areas such as sensor web technology, sensor fusion, command and control, cognitive science, ergonomics, agent technology, multi-agent systems and robotics. However, most of these models only cover part of the system are too much focussed on specific research areas and use different design philosophies for intelligent systems. In this article a top-down design methodology is proposed that may combine the merits of the various approaches and pave the way to efficient design and effective operation of such systems.

In this chapter, we explore four information processing challenges commonly experienced in crisis situations, which form the basis of the design of information systems that should support actors in these situations. When we explore the difference between Sensemaking and decision making, two activities that are undertaken to cope with information processing challenges, we can understand the two types of information systems support that are needed. The first type—decision support systems—supports actors in dealing with information-related problems of uncertainty and complexity, and is the traditional focus of information systems design. The second type—sensemaking support systems—should support actors in dealing with problems of frames of reference, ambiguity, and equivocality, but is not commonplace yet. We conducted three case studies in different crisis situations to explore these information processing challenges: A case study of the sudden crisis of an airplane crash in the Barents Rescue Exercise, a case study of the yearly recurring forest fires crises in Portugal, and a case study of the post-conflict European Union Police Mission in Bosnia and Herzegovina. We discuss design premises for crisis management information systems and compare these to our findings, and observe that systems designed accordingly will provide for the necessary Sensemaking support.

This chapter introduces a system for early detection of gaseous substances and coarse source localization by using heterogeneous sensor measurements and human reports. The system is based on Distributed Perception Networks, a Multi-agent system framework implementing distributed Bayesian reasoning. Causal probabilistic models are exploited in several complementary ways. They support uniform and efficient integration of very heterogeneous information sources, such as different static and mobile sensors as well as human reports. In principle, modular Bayesian networks allow creation of complex probabilistic observation models which adapt to changing constellations of information sources at runtime. On the other hand, Bayesian networks are used also for coarse modeling of transitions in the gas propagation processes. By combining dynamic models of gas propagation processes with the observation models, we obtain adaptive Bayesian systems which correspond to Hidden Markov Models. The resulting systems facilitate seamless combination of prior domain knowledge and heterogeneous observations.

Traffic light control is one of the main means of controlling road traffic. Improving traffic control is important because it can lead to higher traffic throughput and reduced traffic congestion. This chapter describes multiagent reinforcement learning techniques for automatic optimization of traffic light controllers. Such techniques are attractive because they can automatically discover efficient control strategies for complex tasks, such as traffic control, for which it is hard or impossible to compute optimal solutions directly and hard to develop hand-coded solutions. First, the general multi-agent reinforcement learning framework is described, which is used to control traffic lights in this work. In this framework, multiple local controllers (agents) are each responsible for the optimization of traffic lights around a single traffic junction, making use of locally perceived traffic state information (sensed cars on the road), a learned probabilistic model of car behavior, and a learned value function which indicates how traffic light decisions affect long-term utility, in terms of the average waiting time of cars. Next, three extensions are described which improve upon the basic framework in various ways: agents (traffic junction controllers) taking into account congestion information from neighboring agents; handling partial observability of traffic states; and coordinating the behavior of multiple agents by coordination graphs and the max-plus algorithm.

Fusing traffic data from a variety of traffic sensors into a coherent, consistent, and reliable picture of the prevailing traffic conditions (e.g. densities, speeds, flows) is a critical and challenging task in any off- or online traffic management or information system which use these data. Recursive Kalman filter-based approaches provide an intuitive and powerful solution for traffic state estimation and data fusion, however, in case the data cannot be straightforwardly aligned over space and time, the equations become unwieldy and computationally expensive. This chapter discusses three alternative data fusion approaches which solve this alignment problem and are tailored to fuse such semantically different traffic sensor data. The so-called PISCIT and FlowResTD methods both fuse spatial data (individual travel times and low-resolution floating car data, respectively) with a prior speed map obtained from either raw data or another estimation method. Both PISCIT and FlowResTD are robust to structural bias in those a priori speeds, which is critically important due to the fact that many real-world local sensors use (arithmetic) time averaging, which induces a significant bias. The extended and generalized Treiber–Helbing filter (EGTF) in turn is able to fuse multiple data sources, as long as for each of these it is possible to estimate under which traffic regime (congested, free flowing) the data were collected. The algorithms are designed such that they can be used in a cascaded setting, each fusing an increasingly accurate posterior speed map with new data, which in the end could be used as input for a model-based/Kalman filter approach for traffic state estimation and prediction.

Bayesian networks are widely accepted as models for reasoning with uncertainty. In this chapter, we focus on models that are created using domain expertise only. After a short review of Bayesian network models and common Bayesian network modeling approaches, we will discuss in more detail three applications of Bayesian networks.With these applications, we aim to illustrate the modeling power and flexibility of the Bayesian networks, which go beyond the standard textbook applications. The first network is applied in a system for medical diagnostic decision support. A distinguishing feature of this network is the large amount of variables in the model. The second one involves an application for petrophysical decision support to determine the mineral content of a well, based on borehole measurements. This model differs from standard Bayesian networks in terms of its continuous variables and nonlinear relations. Finally, we will discuss an application for victim identification by kinship analysis based on DNA profiles. The distinguishing feature in this application is that Bayesian networks are generated and computed on-the-fly based on case information.