Handbook of Ambient Intelligence and Smart Environments

Advances in the miniaturization of electronics is allowing computing devices with various capabilities and interfaces to become part of our daily life. Sensors, actuators, and processing units can now be purchased at very affordable prices. This technology can be networked and used with the coordination of highly intelligent software to understand the events and relevant context of a specific environment and to take sensible decisions in real-time or

a posteriori

.

Over the past twenty years, the computer vision community has made great strides in the automatic solution to such problems as camera localization and visual tracking. Many algorithms have been made tractable by the rapid increases in computational speed and memory size now available to a single computer. However, the world of visual sensor networks poses several challenges to the direct application of traditional computer vision algorithms. First, visual sensor networks are assumed to contain tens to hundreds of cameras- many more than are considered in many vision applications. Second, these cameras are likely to be spread over a wide geographical area- much larger than the typical computer lab. Third, the cameras are likely to have modest local processors with no ability to communicate beyond a short range.

Vision-based human pose tracking promises to be a key enabling technology for myriad applications, including the analysis of human activities for perceptive environments and novel man-machine interfaces. While progress toward that goal has been exciting, and limited applications have been demonstrated, the recovery of human pose from video in unconstrained settings remains challenging. One of the key challenges stems from the complexity of the human kinematic structure itself. The sheer number and variety of joints in the human body (the nature of which is an active area of biomechanics research) entails the estimation of many parameters. The estimation problem is also challenging because muscles and other body tissues obscure the skeletal structure, making it impossible to directly observe the pose of the skeleton.

One subarea in the context of ambient intelligence concerns the support of moving objects, i. e. to monitor the course of events while an object crosses a smart environment and to intervene if the environment could provide assistance. For this purpose, the smart environment has to employ methods of knowledge representation and spatiotemporal reasoning. This enables the support of such diverse tasks as wayfinding, spatial search, and collaborative spatial work.

Heterogeneous sensor networks (HSNs) are gaining popularity in diverse fields, such as military surveillance, equipment monitoring, and target tracking yarvis:2005:infocom. They are natural steps in the evolution of wireless sensor networks wireless sensor network (WSNs) driven by several factors. Increasingly, WSNs will need to support multiple, although not necessarily concurrent, applications. Different applications may require different resources. Some applications can make use of nodes with different capabilities. As the technology matures, new types of nodes will become available and existing deployments will be refreshed. Diverse nodes will need to coexist and support old and new applications.

There is an ever increasing demand for security monitoring systems in the modern world. Visual surveillance is one of the most promising areas in security monitoring for several reasons. It is easy to install, easy to repair, and the initial setup cost is inexpensive when compared with other sensor based monitoring systems, such as audio sensors, motion detection systems, thermal sensors etc.

The idea of supporting collaboration with computers goes back to the work done by Douglas C. Engelbart. In his seminal demonstration in 1968 [1], he introduced and demonstrated remote collaboration between two persons through sharing computer screens and using audio-visual communication channels over a network.

Pervasive computing envisions applications that provide intuitive, seamless and distraction-free task support for their users. To do this, the applications combine and leverage the distinct functionality of a number of devices. Many of these devices are invisibly integrated into the environment. The devices are equipped with various sensors that enable them to perceive the state of the physical world. By means of wireless communication, the devices can share their perceptions and they can combine them to accurate and expressive models of their surroundings. The resulting models enable applications to reason about past, present and future states of their context and empower them to behave according to the expectations of the user.

In ambient intelligence environments, computers and sensors are embedded in our daily lives massively[1]. This should not be taken in the narrow-minded sense of “a computer on every desk”. Rather, computers will be embedded in everyday objects augmenting them with information processing capabilities. This kind of embedding would be discreet and unobtrusive: The computers would disappear from our perception, leaving us free to concentrate on the task at hand—unlike today, where a majority of users perceives computers as getting in the way of their work.

The next wave of mobile applications is at hand. Mobile phones, PDAs, cameras, music players, and gaming gadgets are creating a connected mobile ecosystem where it is possible to implement systems with significant embedded intelligence. Such advances will make it possible to move many functions of the current PC-centric applications to the mobile domain. Since the inherent difficulties that come with mobility—limited UIs, short attention spans, power dependency, intermittent connectivity, to name but a few—are still not going away, new solutions are needed to make mobile computing satisfactory. We are facing the paradox of cramming ever more functions into our ever more portable devices, while seeking to achieve radically better usablility and semi-usable automated intelligence.

The manifesto of ubiquitous computing is traditionally considered to be the justly famous 1991 visionary article written for

Scientific American

by the late Mark Weiser of Xerox PARC [64]. But the true birth date of this revolution, perhaps hard to pinpoint precisely, precedes that publication by at least a few years: Weiser himself first spoke of “ubiquitous computing” around 1988 and other researchers around the world had also been focusing their efforts in that direction during the late Eighties. Indeed, one of the images in Weiser’s article depicts an Active Badge, an infrared-emitting tag worn by research scientists to locate their colleagues when they were not in their office (in the days before mobile phones were commonplace) and to enable audio and video phone call rerouting and other follow-me applications.

An important characteristic of ℌAmbient Intelligenceℍ (AmI) environments is the merging of physical and digital space (i.e. tangible objects and physical environments are acquiring a digital representation). As the computer disappears in the environments surrounding our activities, the objects therein are transformed to artifacts, that is, they are augmented with Information and Communication Technology (ICT) components (i.e. sensors, actuators, processor, memory, wire-less communication modules) and can receive, store, process and transmit information. Artifacts may be totally new objects or improved versions of existing objects, which by using the ambient technology, allow people to carry out novel or traditional tasks in unobtrusive and effective ways.

Computing is at one of itsmost excitingmoments in history, playing an essential role in supporting many important human activities. The explosion in the availability of information in various media forms and through multiple sensors and devices means, on one hand, that the amount of data we can collect will continue to increase dramatically, and, on the other hand, that we need to develop new paradigms to search, organize, and integrate such information to support all human activities.

One of the striking aspects of world-wide-web is how it has empowered ordinary non-technical people to participate in a digital revolution by transforming the way services such as shopping, education and entertainment are offered and consumed. The proliferation of networked appliances, sensors and actuators, such as those found in digital homes heralds a similar ‘sea change’ in the capabilities of ordinary people to customise and utilise the electronic spaces they inhabit. By coordinating the actions of networked devices or services, it is possible for the environment to behave in a holistic and reactive manner to satisfy the occupants needs; creating an intelligent environment. Further, by deconstructing traditional home appliances into sets of more elemental network accessible services, it is possible to reconstruct either the original appliance or to create new user defined appliances by combining basic network services in novel ways; creating a so called virtual appliance. This principle can be extended to decompose and re-compose software applications allowing users to create their own bespoke applications. Collectively, such user created entities are referred to as Meta – appliances or – applications, more generally abbreviated to MAps.

Severe motion impairments can result from non-progressive disorders, such as cerebral palsy, or degenerative neurological diseases, such as Amyotrophic Lateral Sclerosis (ALS), Multiple Sclerosis (MS), or muscular dystrophy (MD). They can be due to traumatic brain injuries, for example, due to a traffic accident, or to brainstem strokes [9, 84]. Worldwide, these disorders affect millions of individuals of all races and ethnic backgrounds [4, 75, 52]. Because disease onset of MS and ALS typically occurs in adulthood, afflicted people are usually computer literate. Intelligent interfaces can immensely improve their daily lives by allowing them to communicate and participate in the information society, for example, by browsing the web, posting messages, or emailing friends. However, people with advanced ALS, MS, or MD may reach a point when they cannot control the keyboard and mouse anymore and also cannot rely on automated voice recognition because their speech has become slurred.

Among the many possibilities of automation enabled by multi-sensor environments - several of which are discussed in this Handbook - one particularly relevant is the analysis of social interaction in the workplace, and more specifically, of conversational group interaction. Group conversations are ubiquitous, and represent a fundamental means through which ideas are discussed, progress is reported, and knowledge is created and disseminated.

Traditionally smart environments have been understood to represent those (often physical) spaces where computation is embedded into the users’ surrounding infrastructure, buildings, homes, and workplaces. Users of this “smartness” move in and out of these spaces. Ambient intelligence assumes that users are automatically and seamlessly provided with context-aware, adaptive information, applications and even sensing – though this remains a significant challenge even when limited to these specialized, instrumented locales. Since not all environments are “smart” the experience is not a pervasive one; rather, users move between these intelligent islands of computationally enhanced space while we still aspire to achieve a more ideal anytime, anywhere experience. Two key technological trends are helping to bridge the gap between these smart environments and make the associated experience more persistent and pervasive. Smaller and more computationally sophisticated mobile devices allow sensing, communication, and services to be more directly and continuously experienced by user. Improved infrastructure and the availability of uninterrupted data streams, for instance location-based data, enable new services and applications to persist across environments.

Facial expressions, resulting from movements of the facial muscles, are the face changes in response to a person’s internal emotional states, intentions, or social communications. There is a considerable history associated with the study on facial expressions. Darwin [22] was the first to describe in details the specific facial expressions associated with emotions in animals and humans, who argued that all mammals show emotions reliably in their faces. Since that, facial expression analysis has been a area of great research interest for behavioral scientists [27]. Psychological studies [48, 3] suggest that facial expressions, as the main mode for nonverbal communication, play a vital role in human face-to-face communication. For illustration, we show some examples of facial expressions in Fig. 1.

Once upon a time… Stories used to be the only way to pass a message. The story teller would take his audience through the events by mere oration. Here and there he would hesitate, whisper, or gesticulate to emphasise his story or induce the right emotions in his audience. No doubt troubadourswere loved, they both brought news of the world as well as entertainment.

As this book clearly demonstrates, there are many ways to create smart environments and to realize the vision of ambient intelligence. But whatever constitutes this smartness or intelligence, has to manifest itself to the human user through the human senses. Interaction with the environment can only take place through phenomena which can be perceived through these senses and through physical actions xecuted by the human. Therefore, the devices which create these phenomena (e.g., light, sound, force, …) or sense these actions are the user’s contact point with the underlying smartness or intelligence.

Ambient Intelligence (AmI) and Smart Environments (SmE) are based on three foundations: ubiquitous computing, ubiquitous communication and intelligent adaptive interfaces [41]. This type of systems consists of a series of interconnected computing and sensing devices which surround the user pervasively in his environment and are invisible to him, providing a service that is dynamically adapted to the interaction context, so that users can naturally interact with the system and thus perceive it as intelligent.

Infrastructures are the backbone of today’s economies. Physical Infrastructures such as transport and energy networks are vital for ensuring the functioning of a society. They are strained by the ever increasing demand for capacity, the need for stronger integration with other infrastructures, and the relentless push for higher cost efficiency.

Smart homes provide many research challenges, but some of the most interesting ones are in dealing with data that monitors human behaviour and that is inherently both spatial and temporal in nature. This means that context becomes all important: a person lying down in front of the fireplace could be perfectly normal behaviour if it was cold and the fire was on, but otherwise it is unusual. In this example, the context can include temporal resolution on various scales (it is winter and therefore probably cold, it is not nighttime when the person would be expected to be in bed rather than the living room) as well as spatial (the person is lying in front of the fireplace) and extra information such as whether or not the fire is lit. It could also include information about how they reached their current situation: if they went from standing to lying very suddenly there would be rather more cause for concern than if they first knelt down and then lowered themselves onto the floor. Representing all of these different temporal and spatial aspects together is a major challenge for smart home research. In this chapter we will provide an overview of some of the methodologies that can be used to deal with these problems. We will also outline our own research agenda in the Massey University Smart Environments (MUSE) group.

During the past few years, starting from the two mainstream fields of Ambient Intelligence [2] and Robotics [17], several authors recognized the benefits of the socalled

Ubiquitous Robotics

paradigm. According to this perspective, mobile robots are no longer autonomous, physically situated and embodied entities adapting themselves to a world taliored for humans: on the contrary, they are able to interact with devices distributed throughout the environment and get across heterogeneous information by means of communication technologies. Information exchange, coupled with simple actuation capabilities, is meant to replace physical interaction between robots and their environment. Two benefits are evident: (i) smart environments overcome inherent limitations of mobile platforms, whereas (ii) mobile robots offer a mobility dimension unknown to smart environments.

The application need for distributed artificial intelligence (AI) systems for behavior analysis and prediction is a requirement today versus a luxury of the past. The advent of distributed AI systems with large numbers of sensors and sensor types and unobtainable network bandwidth is also a key driving force. Additionally, the requirement to fuse a large number of sensor types and inputs is required and can now be implemented and automated in the AI hierarchy, and therefore, this will not require human power to observer, fuse, and interpret.

While ambient intelligence and smart environments (AISE) technologies are expected to provide large impacts to human lives and social activities, it is generally difficult to show utilities and effects of these technologies on societies. AISE technologies are not only methods to improve performance and functionality of existing services in the society, but also frameworks to introduce new systems and services to the society. For example, no one expected beforehand what Internet or mobile phone brought into out social activities and services, although they changes our social system and patterns of behaviors drastically and emerge new services (and risks, unfortunately). The main reason of this difficulty is that actual effects of IT systems appear when enough number of people in the society use the technologies.

Multi-agent modeling in ambient intelligence (AmI) is concerned with the following task [19]: How can external observations of multi-agent systems in the ambient be used to analyze, model, and direct agent behavior? The main purpose is to obtain knowledge about acts in the environment thus enabling proper actions of the AmI systems [1]. Analysis of such systems must thus capture complex world state representation and asynchronous agent activities. Instead of studying basic numerical data, researchers often use more complex data structures, such as rules and decision trees. Some methods are extremely useful when characterizing state space, but lack the ability to clearly represent temporal state changes occurred by agent actions. To comprehend simultaneous agent actions and complex changes of state space, most often a combination of graphical and symbolical representation performs better in terms of human understanding and performance.

With the introduction of more complex intelligent environment systems, the possibilities for customizing system behavior have increased dramatically. Significant headway has been made in tracking individuals through spaces using wireless devices [1, 18, 26] and in recognizing activities within the space based on video data (see chapter by Brubaker et al. and [6, 8, 23]), motion sensor data [9, 25], wearable sensors [13] or other sources of information [14, 15, 22]. However, much of the theory and most of the algorithms are designed to handle one individual in the space at a time. Resident tracking, activity recognition, event prediction, and behavior automation becomes significantly more difficult for multi-agent situations, when there are multiple residents in the environment.

Mobile agents are autonomous programs that can travel from computer to computer in a network, at times and to places of their own choosing. The state of the running program is saved, by being transmitted to the destination. The program is resumed at the destination continuing its processing with the saved state. They can provide a convenient, efficient, and robust framework for implementing distributed applications and smart environments for several reasons, including improvements to the latency and bandwidth of client-server applications and reducing vulnerability to network disconnection. In fact, mobile agents have several advantages in the development of various services in smart environments in addition to distributed applications.

In the current technological landscape colored by environmental and security concerns the logic of replacing traveling by technical means of communications is undisputable. For example, consider a comparison between a normal family car and a video conference system with two laptop computers connected over the Internet. The power consumption of the car is approximately 25 kW while the two computers and their share of the power consumption in the intermediate routers in total is in the range of 50 W. Therefore, to meet a person using a car at an one hour driving distance is equivalent to 1000 hours of video conference. The difference in the costs is also increasing. An estimate on the same cost difference between travel and video conference twenty years ago gave only three days of continuous video conference for the same situation [29]. The cost of video conference depends on the duration of the session while traveling depends only on the distance. However, in a strict economical and environmental sense even a five minute trip by a car in 2008 becomes more economical than a video conference only when the meeting lasts more than three and half days.

The term smart environment refers to a physical space enriched with sensors and computational entities that are seamlessly and invisibly interwoven. A challenge in smart environments is to identify the location of users and physical objects. A smart environment provides location-dependent services by utilizing obtained locations. In many cases, estimating location depends on received signal strength or the relative location of other sensors in the environment. Although devices employed for location detection are evolving, identification of location is still not accurate. Therefore, n addition to devices or utilized physical phenomena, algorithms that enhance the accuracy of location are important. Furthermore, other aspects of utilizing location information need to be considered: who is going to name important places and how are the name ontologies used.

Nowadays computing technology research is focused on the development of Smart Environments. Following that line of thought several Smart Rooms projects were developed and their appliances are very diversified. The appliances include projects in the context of workplace or everyday living, entertainment, play and education. These appliances envisage to acquire and apply knowledge about the environment state in order to reason about it so as to define a desired state for its inhabitants and perform adaptation adaptation to these desires and therefore improving their involvement and satisfaction with that environment.

In recent years, distance learning has increasingly become one of themost important applications on the internet and is being discussed and studied by various universities, institutes and companies. The Web/Internet provides relatively easy ways to publish hyper-linked multimedia content for more audiences. Yet, we find that most of the courseware are simply shifted from textbook to HTML files. However, in ost cases the teacher’s live instruction is very important for catching the attention and interest of the students. That’s why Real-Time Interactive Virtual Classroom (RTIVC) always plays an indispensable role in distance learning, where teachers nd students located in different places can take part in the class synchronously through certain multimedia communication systems and obtain real-time and mediarich interactions using Pervasive Computing technologies [1]. The Classroom 2000 project [2] at GIT has been devoted to the automated capturing of the classroom experience. Likewise, the Smart Classroom project [3] at our institute is focused on Tele-education. Most currently deployed real-time Tele-education systems are desktop-based, in which the teacher’s experience is totally different from teaching in a real classroom.

Ambient intelligence has the potential for improving urban life specifically and the commonwealth in general. As artists and architects working in and with ambient intelligence, our hopes and anxieties towards ambient intelligence are not primarily in the technical domain. Our interest lies in re-making urbanity with pervasive technologies as a means to invigorate urban life. For this we need to take a break, after almost two decades of ambient intelligence related research, and recalibrate all instruments.

Since the early 1990s, and particularly with the popularization of the Internet and the World Wide Web, a wave of experiments and initiatives has emerged, aiming at acilitating city functions such as community activities, local economies and municipal ervices. This chapter reviews advancements of worldwide activities focused on the creation of regional information spaces. In the US and Canada, a large number f community networks using the city metaphor appeared in the early 1990s. In Europe, more than one hundred similar initiatives have been tried out, often supported by large governmental project funds. Asian countries are rapidly adopting the latest information and communication technologies for actively interacting real-time city information and creating civic communication channels.

Recent improvements in technology have facilitated the use of robots and virtual humans not only in entertainment and engineering but also in the military (Hill et al., 2003), healthcare (Pollack et al., 2002), and education domains (Johnson, Rickel, & Lester, 2000). As active partners of humans, such machine assistants can take the form of a robot or a graphical representation and serve the role of a financial assistant, a health manager, or even a social partner. As a result, interactive technologies are becoming an integral component of people’s everyday lives.

An elderly woman wanders about aimlessly in a home for assisted living. Suddenly, she collapses on the floor of a lonesome hallway. Usually it can take over two hours until a night nurse passes this spot on her next inspection round. But in this case she is already on site after two minutes, ready to help. She has received an alert message on her beeper: “Inhabitant fallen in hallway 2b”. The source: the SmartSurv distributed network of smart cameras for automated and privacy respecting video analysis.Welcome to the future of smart surveillance Although this scenario is not yet daily practice, it shall make clear how such systems will impact the safety of the elderly without the privacy intrusion of traditional video surveillance systems.

User modeling (UM) has traditionally been concerned with analyzing a user’s interaction with a system and with developing cognitive models that aid in the design of user interfaces and interaction mechanisms. Elements of a user model may include representation of goals, plans, preferences, tasks, and/or abilities about one or more types of users, classification of a user into subgroups or stereotypes, the formation of assumptions about the user based on the interaction history, and the generalization of the interaction histories of many users into groups, among many others.

At the turn of the century, a distinguished group of researchers identified the potential devastating effects of rapid technological developments, as described by the generalized Moore’s law, for the balanced relationship between humans and technology (Aarts et al., 2001). Whilst not ignoring the threads and risks of so called technology push, the Ambient Intelligence Ambient Intelligence (AmI) vision was introduced to emphasize the positive contribution these technologies could bring to our daily lives. Within the AmI vision human needs are positioned centrally and technology is seen as a means to enrich our life. In course terms Ambient Intelligence refers to

the embedding of technologies into electronic environments that are sensitive and responsive to the presence of people

.

It is a common experience in our modern world, for us humans to be overwhelmed by the complexities of technological artifacts around us, and by the attention they demand. While technology provides wonderful support and helpful assistance, it also causes an increased preoccupation with technology itself and a related fragmentation of attention. But as humans, we would rather attend to a meaningful dialog and interaction with other humans, than to control the operations of machines that serve us. The cause for such complexity and distraction, however, is a natural consequence of the flexibility and choice of functions and features that technology has to offer. Thus flexibility of choice and the availability of desirable functions are in conflict with ease of use and our very ability to enjoy their benefits.

environmental control is the control, operation, and monitoring of an environment via intermediary technology such as a computer. Typically this means control of a domestic home.Within the scope of COGAIN, this environmental control concerns the control of the personal environment of a person (with or without a disability). This defines environmental control as the control of a home or domestic setting and those objects that are within that setting. Thus, we may say that environmental control systems enable anyone to operate a wide range of domestic appliances and other vital functions in the home by remote control. In recent years the problem of self-sufficiency for older people and people with a disability has attracted increasing attention and resources. The search for new solutions that can guarantee greater autonomy and a better quality of life has begun to exploit easily available state-of-the-art technology. Personal environmental control can be considered to be a comprehensive and effective aid, adaptable to the functional possibilities of the user and to their desired actions.

With computing and communication capabilities now embedded in most physical objects of the surrounding environment and most users carrying wireless computing devices, the Ambient Intelligence (AmI) / pervasive computing vision [28] pioneered by Mark Weiser [32] is becoming a reality. Devices carried by nomadic users can seamlessly network with a variety of devices, both stationary and mobile, both nearby and remote, providing a wide range of functional capabilities, from base sensing and actuating to rich applications (e.g., smart spaces). This then allows the dynamic deployment of

pervasive applications

, which dynamically compose functional capabilities accessible in the

pervasive network

at the given time and place of an application request.

The project PERSONA aims at advancing the paradigm of Ambient Intelligence through the harmonization of Ambient Assisted Living (AAL) Technologies and concepts for the development of sustainable and affordable solutions for the independent living of senior citizens. PERSONA is one of the integrated projects funded by the European Commission within the 6th Framework Program for IST (Information Society Technologies) on AAL for the Aging Society. It involves the participation of 21 partners, from Italy, Spain, Germany, Greece, Norway and Denmark, with a total budget of around 12 million Euros.

Like Aladdin in the medieval oriental folk-tale, the assistive lighting system developed by ALADIN (Ambient Lighting Assistance for an Ageing Population), a research project co-financed by the European Commission, is expected to bring enchantment to people’s lives. But this will not be achieved by magic and genies, but by exploiting our knowledge about the impact of lighting. adaptive lighting can contribute considerably to sound sleep and a regular sleep-wake cycle regulated by people’s ’inner clock’. This tends to deteriorate with ageing, but is essential to preserve and enhance comfort and wellbeing. And this is the main goal of the assistive ALADIN lighting system.

Recently, the advent of sophisticated technologies has stimulated ambient paradigms that may include high-performance CPU, compact real-time operating systems, a variety of devices/sensors, low power and high-speed radio communications, and in particular, third generation mobile phones. In addition, due to the spread of broadband ccess networks, various ubiquitous terminals and sensors can be connected closely.

This chapter consists of four sections. Each section introduces a major project related with ubiquitous computing technology which has been conducted by Korean government. In the second section, we introduce UCN (Ubiquitous Computing Network) project in which service convergence solutions have been developed to design and manage human-centered composite services. In the third section, we introduce a project on intelligent service robots in ubiquitous environment. The fourth describes several projects related with ubiquitous health. Finally, we introduce a projectwhich focuses on the architecture of USN (Ubiquitous Sensor Network) technology for nation-wide monitoring.