Automatic agent generation for IoT-based smart house simulator

doi:10.1016/j.neucom.2015.04.130

Neurocomputing

Volume 209, 12 October 2016, Pages 14-24

https://doi.org/10.1016/j.neucom.2015.04.130 Get rights and content

Abstract

In order to evaluate the quality of Internet of Things (IoT) environments in smart houses, large datasets containing interactions between people and ubiquitous environments are essential for hardware and software testing. Both testing and simulation require a substantial amount of time and volunteer resources. Consequently, the ability to simulate these ubiquitous environments has recently increased in importance. In order to create an easy-to-use simulator for designing ubiquitous environments, we propose a simulator and autonomous agent generator that simulates human activity in smart houses. The simulator provides a three-dimensional (3D) graphical user interface (GUI) that enables spatial configuration, along with virtual sensors that simulate actual sensors. In addition, the simulator provides an artificial intelligence agent that automatically interacts with virtual smart houses using a motivation-driven behavior planning method. The virtual sensors are designed to detect the states of the smart house and its living agents. The sensed datasets simulate long-term interaction results for ubiquitous computing researchers, reducing the testing costs associated with smart house architecture evaluation.

Introduction

A smart house provides an intelligent home management interface and a comfortable living environment. Smart houses have recently become important research topics in the Internet of Things (IoT) [1], [2], [3]. A variety of IoT-based sensors connected by wireless networks are installed in smart houses, to enhance the life of the home's residents [4], [5], [6], [7], [8]. Smart house architectural engineers desire the ability to intuitively configure their smart houses and meet user needs before construction begins. Therefore, reliable, low-cost test beds are required in order to examine the architecture design.

In addition, smart houses must monitor interaction between users and house components in order to provide appropriate services [9]. Sensors detect various kinds of environmental datasets [10]. However, individual sensors work independently and report simple information. In order to apply reliable ubiquitous computing and detect the living situation, multiple sensors must be mounted [11]. However, hardware reconstruction for this type of interface installation substantially increases the cost.

Therefore, in order to provide a low-cost, effective test bed, we present a simulator that creates a virtual smart house and simulates action recognition in the virtual environment. In the simulator, a virtual smart house is created with independent virtual sensors, which record environmental information, including user location and house temperature [12]. The simulator provides designers with a graphical user interface (GUI) in order to help them arrange house components and sensors. The smart house situation and user state are then estimated using these sensed datasets [13].

In addition, an autonomous agent generator is provided for defining virtual agent behaviors and action effects. Virtual simulation must mimic the real world. Therefore, in order to simulate an actual person within the smart house, we create an intelligent agent, which autonomously executes behavior planning based on various motivations [14]. The interactions between virtual agents and the house are recorded and visualized. After a long-term test, the recorded information is provided and used as a reference by smart house researchers in order to ensure convenient, desirable services.

The rest of this paper is organized as follows. In Section 2, we discuss works related to smart house simulations and behavior planning. Section 3 describes the simulator's structure, sensor-based simulation techniques, and the autonomous agent generator. The proposed simulator's performance is analyzed in Section 4, and Section 5 presents our conclusions.

Section snippets

Related work

Smart house simulation research aims at generating long-term testing data in order to verify a self-adaptive house architecture. Helal et al. [15] proposed an event-driven simulator, which presents the common elements of a smart house, including house components, sensors, agents, and their interactions. Given the sensing datasets, agents enact behavior planning and interact with the smart house using an event list. Park et al. [16] designed a context-aware simulation system, which allows smart

Simulation system

In this section, we describe the architecture of the proposed simulator, the applied sensor-based simulation techniques, the developed sensor models, and the autonomous agent generator.

Experiments and analysis

In this section, we analyze the performance of the multiple-sensor operation and the automatic scenario generator in a developed virtual smart house. The proposed methods were implemented with Unity3D engine on a computer which had an Intel i7 CPU, 3 GB RAM, and an NVidia GTX 275 GPU. The methods provide real-time processing performance at a minimum of 59.7 fps (frames per second), or 65.1 fps on average.

Conclusions

In this paper, we presented a simulator that provides a 3D smart house configuration tool and autonomous agent generator. Using the configuration tool and intuitive GUI operations, smart house designers arrange house components and sensors. In order to monitor environmental information and a virtual agent's states, we proposed multiple virtual sensors, which report information similar to that sensed in an actual environment. The entire simulation process was monitored, and the simulator was

Acknowledgments

This research was supported by the MSIP (Ministry of Science, ICT and Future Planning), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2016-H8501-16-1014) supervised by the IITP (Institute for Information & communications Technology Promotion).

Wonsik Lee received his B. Eng. Degree in Multimedia Engineering in 2010, his M. Eng. Degree in Multimedia Engineering in 2012, respectively, all from Dongguk University, Seoul, Republic of Korea. He has done many works mainly associated with 3D simulation applications in the military and artificial intelligence areas, such as remote terrain visualization and sensor simulation, and so on. He is currently affiliated to LG Electronics Inc.

References (29)

W.J. Clancey
Simulating activities: relating motives, deliberation, and attentive coordination
Cogn. Syst. Res.
(2002)
J. Lertlakkhanakul et al.
Building data model and simulation platform for spatial interaction management in smart home
Automat. Constr
(2008)
J.C. Augusto et al.
Intelligent Environments: a manifesto
Human-centric Comput. Inf. Sci.
(2013)
J. Vanus et al.
Development and testing of a visualization application software, implemented with wireless control system in smart home care
Human-centric Comput. Inf. Sci.
(2014)
E.-J. Lee et al.
An intelligent green service in internet of things
J. Converg.
(2014)
S.-K. Bae
Power consumption analysis of prominent time synchronization protocols for wireless sensor networks
J. Inf. Process. Syst.
(2014)
Q. Zhu, R. Wang, Q. Chen, Y. Liu, W. Qin, IOT Gateway: BridgingWireless Sensor Networks into Internet of Things,...
S.D.T. Kelly et al.
Towards the implementation of IoT for environmental condition monitoring in homes
Sens. J. IEEE
(2013)
M.V. Moreno, J. Santa, M.A. Zamora, A.F. Skarmeta, A holistic IoT-based management platform for smart environments,...
T. Perumal, M.N. Sulaiman, N. Mustapha, A. Shahi, R. Thinaharan, Proactive architecture for Internet of Things (IoT)s)...

K. Bouchard et al.

SIMACT: A 3D Open source smart home simulator for activity recognition

Lecture Notes Comput. Sci.

(2010)

P. Nurmi, P. Floréen, M. Przybilski, G. Lindén, A Framework for Distributed Activity Recognition in Ubiquitous Systems,...

H. Lieberman et al.

Out of context: Computer systems that adapt to, and learn from, context

IBM Syst. J.

(2000)

S. Jang et al.

(2005)

Cited by (38)

A survey of smart home energy conservation techniques
2023, Expert Systems with Applications
Citation Excerpt :
In order to give users full control over the activities and the resulting datasets, interactive techniques strongly emphasize user engagement with simulation software (Alshammari, Alshammari, Sedky, Champion, & Bauer, 2017; Synnott et al., 2015). Alsalemi, Ramadan et al. (2019), Alshammari et al. (2017), Lee et al. (2016), Weng, Wu, Lu, Yeh, and Fu (2012) develop multiple smart home data simulators which can generate several types of energy consumption datasets. Accordingly, dissimilar smart homes can be created as per requirements and simulated to get the required dataset.
Smart homes are equipped with easy-to-interact interfaces, providing a more comfortable living environment and less energy consumption. There are currently satisfactory approaches proposed to deliver adequate comfort and ease to smart home inhabitants through infrared sensors, motion sensors, and other similar technologies. However, the goal of reducing energy consumption is always a significant concern for smart home stakeholders. A detailed discussion about energy management techniques might open new leads for advanced research and even introduce more ways to improve existing methods since a summary of effective energy conservation techniques are helpful to get a quick overview of the state-of-the-art techniques. This review study aims to provide an overview of previously proposed techniques for energy conservation and energy-saving recommendations. We identify various critical features in energy conservation techniques, i.e., user energy profiling, appliance energy profiling, and off-peak load scheduling to perform a comparative analysis among different techniques. Then, we explain various energy conservation techniques, describe common and rare evaluation metrics, identify several techniques for realizing synthetic smart home energy consumption datasets, and provide a statistical analysis of the existing literature. The survey finally points out possible research directions which might lead to new inquiries in energy conservation research.
Secure and energy-efficient smart building architecture with emerging technology IoT
2021, Computer Communications
With the advent of the Internet-of-Things (IoT), it is considered to be one of the latest innovations that offer interesting opportunities for different vertical industries. One of the most relevant IoT technology areas is smart construction. IoT operates in several sectors on a daily basis; implementation includes smart building, smart grids, smart cities, smart houses, physical defense, e-health, asset, and transportation management, but it is not restricted to this. Support from smart IoT buildings is an IoT-level, connected, and cost-effective system. Commercial space has major requirements in terms of comfort, accessibility, security, and energy management. Such requirements can be served organically by IoT-based systems. As the supply of energy has been exhausted and energy demand has risen, there has been a growing focus on energy usage and the maintenance of buildings. With the use of evolving IoT technology, we present a secure and energy-efficient smart building architecture. Every device is known by its unique address, and one of the key web transfer protocols is the Constrained Application Protocol (CoAP). It is an application layer protocol that does not use protected channels for data transfer. Automatic key management, confidentiality, authentication, and data integrity are all features of the Datagram Transport Layer Protection (DTLS). To achieve energy efficiency, we propose a smart construction architecture that, through IoT, manages the performance of all technological systems. The results of the simulation show that the energy consumption is lowered by about 30.86% with the use of the CoAP in the smart building, which is less than the Message Queuing Telemetry Transport case (MQTT). This paper also aims to observe how to integrate the DTLS protocol with the Secure Hash Algorithm (SHA-256) using optimizations from the Certificate Authority (CA) to improve security.
SISG4HEI_Alpha: Alpha version of simulated indoor scenario generator for houses with elderly individuals
2021, Journal of Building Engineering
Citation Excerpt :
The indoor spatial data impact the travel pattern data, because the starts and destinations of the walking trajectories are near doors or furniture, whose locations are determined by the indoor spatial data. The activity schedule also determines the time attribute of each walking path, because the resident is moving while he/she is in-between activities. [25–27] show methods of travel pattern generation.
Developers improve and test the sensor networks and robots of smart houses by conducting simulations. However, they need to build scenarios in the simulators manually, which takes a lot of time. Manually built scenarios are also few in number. Moreover, developers have to repeat this procedure whenever they conduct a simulation with the same scenario on another simulator. To address these issues, we have developed a simulation tool (alpha version), called SISG4HEI_Alpha, which produces diverse simulated indoor scenarios of houses with elderly individuals for multiple simulators. It consists of a GUI, database, three generators, and interfaces. The spatial attribute generator produces virtual spatial data necessary for building the models of the house and furniture. The activity schedule and travel pattern generator generate resident's information to control the virtual agent within a simulator. One interface converts the generated data into simulated PIR sensor records, which are simulation results of the OPENSHS simulator. The two other interfaces convert data into scenarios of the Stage and Gazebo simulators. SISG4HEI_Alpha is modular and extensible; it is easy to improve the generators and/or add new interfaces.
Cross-environment activity recognition using word embeddings for sensor and activity representation
2020, Neurocomputing
Citation Excerpt :
A clear way to improve our systems is to explore more powerful architectures such as bidirectional LSTMs and transformers. Simulators to generate meaningful synthetic data: as having more training data improves the results of the machine learning systems, it would be interesting to research how current smart home simulators could help [32–34]. The idea is to generate synthetic sensor activations for several activities and check whether training on them helps to recognize activities in real datasets.
Cross-environment activity recognition in smart homes is a very challenging problem, specially for data-driven approaches. Currently, systems developed to work for a certain environment degrade substantially when applied to a new environment, where not only sensors, but also the monitored activities may be different. Some systems require manual labeling and mapping of the new sensor names and activities using an ontology. Ideally, given a new smart home, we would like to be able to deploy the system, which has been trained on other sources, with minimal manual effort and with acceptable performance. In this paper, we propose the use of neural word embeddings to represent sensor activations and activities, which comes with several advantages: (i) the representation of the semantic information of sensor and activity names, and (ii) automatically mapping sensors and activities of different environments into the same semantic space. Based on this novel representation approach, we propose two data-driven activity recognition systems: the first one is a completely unsupervised system based on embedding similarities, while the second one adds a supervised learning regressor on top of them. We compare our approaches with some baselines using four public datasets, showing that data-driven cross-environment activity recognition obtains good results even when sensors and activity labels significantly differ. Our results show promise for reducing manual effort, and are complementary to other efforts using ontologies.
A performance-aware dynamic scheduling algorithm for cloud-based IoT applications
2020, Computer Communications
Citation Excerpt :
Smart homes have recently become essential study areas in IoT. An assortment of IoT grounded sensors associated with wireless networks is mounted in smart homes, to arouse the life of users in the home [3,4]. Generally, an intellectual home computerization method comprises of groups of sensors accumulating dissimilar types of records concerning the users and utility depletion at home.
Cloud computing has been employed for supporting storage and handling of Internet of Things (IoT) data. There is an increasing demand for IoT framework to provide services with fast processing time and less delay to offer latency sensitivity real-time applications like disaster management in smart homes. IoT process is mostly comprised of scheduling techniques that makes it hard to self-adapt, self-configure to respond with performance aware on environment changes. Existing scheduling techniques of IoT applications are not based on allocating tasks through sleep modes, which unavoidably lead to more power consumption and longer time delays. Consenting sensor devices and applying separate queueing to a sensor device that varies differently in their capabilities are increasingly significant. In this work, a dynamic management framework for IoT devices in cloud (DMFIC) algorithm is proposed to evaluate and schedule requests and sensor data, which allow coordinating huge data with high time-based resolution in a cost-effectual manner through anticipating various queues for sending an appropriate notification to users. A smart home application was used to demonstrate the proposed framework. The experimental result shows that the DMFIC algorithm gives an average of 5% higher processing time and 0.2% less delay compared to other IoT services and can efficiently manage sensor data in cloud.
Automatic spatial attribute and travel pattern generation for simulating living spaces for elderly individuals living alone
2020, Building and Environment
Citation Excerpt :
Conducting experiments on a test bed is the most straightforward way to collect data; however, building one is expensive and time-consuming. As alternatives, smart-home and robot simulators [11–14] have been proposed as ways to generate virtual-sensor data for testing, verifying, and improving smart-home systems. The simulations often follow four steps: (1) users manually build home and resident body models or use example models provided by the simulator; (2) they define the activity schedules and movement routes of the virtual resident or control the virtual resident manually; (3) they place virtual sensors, which could be fixed or on moveable robots, that generate virtual sensor records; (4) with these records, they verify the performance of the smart-home system and/or modify its design, such as by changing the placements of the sensors and the operation policies of the robots.
Simulators that generate records of virtual sensors are necessary for verifying and testing smart-home systems. However, their users usually need to build home models and body models of residents manually. They also have to define resident activity schedules and movement routes manually. This study proposes a framework for building a virtual database and using it to make diverse home and body models, activity schedules, and travel patterns for multiple simulation tools. We show that this framework is feasible by making two generators and two interfaces as a demonstration. The spatial attribute generator generates indoor spatial data on studio apartments by following basic design principles and rules for designing homes for elderly individuals. The travel pattern generator generates travel patterns of elderly individuals with dementia on the basis of a realistic-path model. Two interfaces are used to convert the generated spatial data into the home models of a simulator and convert travel patterns into the records of virtual sensors.

View all citing articles on Scopus

Seoungjae Cho received his B. Eng. Degree in Multimedia Engineering in 2012 from Dongguk University, Seoul, Republic of Korea. Since March 2012, he is in the M. Eng. – Ph.D. Eng. integrated course at the Department of Multimedia Engineering, Dongguk University, Seoul, Republic of Korea. He has done many works mainly associated with 3D simulation applications in the military and human computer interaction, artificial intelligence areas, such as remote terrain visualization and sensor simulation, brain computer interface games, robot learning, and so on. His current research interests are focused on the areas of sensor simulation applications using 3D technology and NUI (Natural User Interface) utilizing various NUI devices.

Phuong Chu received his Bachelor Degree in Information Technology in 2011 from Le Quy Don Technical University, Hanoi, Vietnam. After that, he worked in Institute of Simulation Technology in the same university. Since September 2014, he is in the M. Eng. course at the Department of Multimedia Engineering, Dongguk University, Seoul, Republic of Korea. He has done many works mainly associated with 3D simulation applications in artificial intelligence areas and human computer interaction, such as Q-learning for virtual robot.

Hoang Vu received his Bachelor Degree in Mathematics-Applied Informatics in 2005 from Vietnam National University, Hanoi, Vietnam. After that, he worked in Institute of Simulation Technology in the Le Quy Don Technical University, Hanoi, Vietnam. Since September 2014, he is in the M. Eng. course at the Department of Multimedia Engineering, Dongguk University, Seoul, Republic of Korea. He has done many works mainly associated with 3D simulation applications in artificial intelligence areas.

Sumi Helal is a Professor at the Computer and Information Science and Engineering Department (CISE) at the University of Florida (UF), USA, and a Finland Distinguished Professor (FiDiPro) at Aalto University and the EIT ICT Labs, Finland. He is a pioneer and a recognized leader in the fields of Mobile, Pervasive and Ubiquitous Computing. He is well known for his interdisciplinary research on smart spaces and Health Telematics in support of Health Care and Aging, Disabilities and Independence (ADI). He directs the Mobile and Pervasive Computing Laboratory in the CISE department at UF. He is co-founder and director of the Gator Tech Smart House, an experimental facility for applied research development and validation in the domains of elder care and health telematics.

Wei Song is a full lecturer at the Department of Digital Media at the North China University of Technology (NCUT), Beijing, China, since July. 2013. He received his B. Eng. Degree in Software Engineering from Northeastern University, Shenyang, China, in 2005, and his M. Eng. and Dr. Eng. Degrees in Multimedia (Major of Computer Game Production) from Dongguk University, Seoul, Korea, in 2008 and 2013, respectively. Since September 2013, he has been the director of the interactive studio of NCUT. His current research interests are focused on the areas of mixed reality, NUI, interactive information visualization, Image processing, computer graphics, pattern recognition, 3D reconstruction, mobile robot, network game, and other multimedia technologies.

Young-Sik Jeong is a professor in the Department of Multimedia Engineering at Dongguk University in Korea. His research interests include multimedia cloud computing, mobile computing, ubiquitous sensor network (USN), and USN middleware. He received his B.S. degree in Mathematics and his M.S. and Ph.D. degrees in Computer Science and Engineering from Korea University in Seoul, Korea in 1987, 1989, and 1993, respectively.

Kyungeun Cho is a full professor at the Department of Multimedia Engineering at the Dongguk University in Seoul, Republic of Korea since Sept. 2003. She received her B. Eng. Degree in Computer Science in 1993, her M. Eng. and her Dr. Eng. Degrees in Computer Engineering in 1995 and 2001, respectively, all from Dongguk University, Seoul, Korea.During 1997–1998 she was a research assistant at the Institute for Social Medicine at the Regensburg University, Germany, and a visiting researcher at the FORWISS Institute at TU-Muenchen University, Germany. Her current research interests are focused on the areas of intelligence of robot and virtual characters and real-time computer graphics technologies. She has led a number of projects on robotics and game engines and also has published many technical papers in these areas.

View full text

Automatic agent generation for IoT-based smart house simulator

Abstract

Introduction

Section snippets

Related work

Simulation system

Experiments and analysis

Conclusions

Acknowledgments

Cogn. Syst. Res.

Automat. Constr

Intelligent Environments: a manifesto

Human-centric Comput. Inf. Sci.

Development and testing of a visualization application software, implemented with wireless control system in smart home care

Human-centric Comput. Inf. Sci.

An intelligent green service in internet of things

J. Converg.

Power consumption analysis of prominent time synchronization protocols for wireless sensor networks

J. Inf. Process. Syst.

Towards the implementation of IoT for environmental condition monitoring in homes

Sens. J. IEEE

SIMACT: A 3D Open source smart home simulator for activity recognition

Lecture Notes Comput. Sci.

Out of context: Computer systems that adapt to, and learn from, context

IBM Syst. J.