Concept proposal and experimental verification of a sidewalk supporting system utilizing a smartphone

Along with the spread of smartphones and cloud technology in recent years, an environment of collecting and accumulating records of individuals’ activities or so-called life logs through networks, has been prepared. Life logs include internet operation histories, application portfolios in place, and GPS information, that occur during daily activities. As an example of its application, a system has been proposed to estimate individuals’ preferences and life patterns from the accumulated data to provide a personalized information service [1]. In addition, wearable devices [2] have generally been spread in a rapid manner, and a wide variety of services using sensors and network terminals owned by users have been considered.

To date, robotic rooms [3, 4] that assist users by managing a wide variety of sensor information installed indoors and studies that assume robot services in the street or in multiple dwellings [5‐7] have been proposed as a framework for managing and utilizing a plurality of sensor information. Authors have built a network robot platform for sharing information with robots that deals with many types of sensor information with a plurality of guide systems [8‐10]. Here we introduce abstracted information from sensors and robots in the form of 4W (who/where/when/what) user information or robots based on the XML format. Furthermore, while expanding on this, we have built a system to perform operations such as the physical transportation of goods in addition to the guidance of only providing information, and we have linked the information from a wide variety of sensors and robots [11, 12]. An approach has been adopted that streamlines the system by handling only the information which has been abstracted to a limited form. Through these methods, it is possible to accumulate information from multiple sensors designed and arranged for the purpose of measuring the specific information in the particular experimental environment and to capture it as an actual example of its application. It can also be said that these are examples where the data collector side built systems to collect data primarily with sensors whose specifications were defined in advance.

Among the wide variety of sensors and robots that are researched and developed, research and development of middleware and operating system from the viewpoint of software development include RT-Middleware [13] as a common platform that uses a network-distributed component technology. In RT-Middleware, devices such as servo motors, sensors, and cameras and the functional elements of robots such as various processing programs, are processed into RT components, which are then linked to develop a system. Moreover, in ROS [14], an environment is provided in which devices such as actuators and sensors and various processing programs are modularized or processed into components to develop software. They enhance the reusability of resources by processing various functional elements into software components to aim at efficient software development. Furthermore, in SensorML [15], where sensor information is managed by utilizing XML technology and focus is on the discovery of sensors and the acquisition of positional information, there are examples of utilization for the purpose of collecting information on the population and states of pollution by dividing the area into specific regions of widely distributed sensors.

In parallel with the evolution of demonstration experiments at laboratories or specific facilities and the preparation of system development environments as described above, services using sensors built in smartphones carried by users and information from connected sensors are thought to increase along with the rapid spread of smartphones in the future. If we look at such a point of view, in the demonstration of robotic room [3, 4] and multiple robot cooperation [5, 12], a particular device is determined in advance, it is also fixed function sharing the server and the client. Also the research and development of middleware [13‐15] related development efficiency software is improved. But the overall system design is dependent on the developer. Here, the efficient and usable systems are considered to be important in the process from the early experiment stage to commercialization. It is considered to be important to change the flexible function sharing in consideration of the client’s capabilities and performance in the experimental stage. In addition, responses to the diversity of smartphone models and changes in their performances as well as the standpoint of dealing with the subjective evaluation values of users, which cannot be automatically acquired by sensors, are also important.

The authors have conducted studies on understanding the health status in those who suffer from rheumatoid arthritis as an application in the medical field using smartphone sensors, based on a platform [8‐10] for liaising information to the robot as described above. The joint destruction of limbs caused by rheumatoid arthritis lasts for many years, and has become a social problem leading to severe motility disturbance and dysfunction in daily life activities [16]. It is necessary to accurately grasp the daily health conditions of patients undergoing this treatment, and thus objective, quantitative, and temporal evaluation methods are sought. For these situations, the authors performed phased experiments to build a system for estimating the disease activity score 28 (DAS28) without using blood test values from the information on gait that was obtained from the acceleration information of patients’ smartphones. This was done while patients walked and self-reported information that they entered in their smartphones, verifying its effectiveness [17‐19].

In this study, we propose technology for efficient information gathering as to situations involving sidewalks based on sensor information gathered from the smartphones of sidewalk-users. A sidewalk supporting system is used as the basis of the service for properly managing the collected data and results and extracting useful information such as that on traffic availability. This is then provided to sidewalk users, local governments, and others with the goal of expansion into daily life, including the aforementioned application to the medical field.

As conventional studies for detecting the states of road surfaces, a method of detection has been proposed which uses a smartphone’s acceleration sensor fixed to the vehicle against the differences in level that become roadway traffic obstacles [20]. With regard to sidewalks, there is a test example of detection of surface damage using mobile robot equipped with laser scanner [21]. And data specifications for organizing sidewalk information have been proposed [22], and appropriate information presentation has been demanded as a social need for people who find it difficult to pass along streets, such those who use wheelchairs or strollers. Technical points needed to build services systematically against such problems could include information gathering technologies through the carrier network from smartphone sensors with the aim of collecting sidewalk movement logs, which are less burdensome for users.

In this paper, we propose a concept for a system that accumulates information from strollers, mobility scooters, bicycles, etc. through sidewalk users’ smartphones, sensors and other such technology to provide valid information for the purpose of building a system to support social infrastructure where users can move safely and securely. Then we show a proposed basic system configuration utilizing data collection technologies in robotics and gait analysis for responding to the diversity of sensors and the characteristics of smartphones. By using this system, it can be handled physical data of a stroller, a bicycle and a wheelchair, the user’s movement as well as the user’s subjectivity data and vital data. Furthermore, we will introduce a method for estimating the conditions of sidewalks utilizing the subjectivity of sidewalks passers-by with regard to the acquisition of training data to analyze data effectively. As a specific application case, we evaluate the estimation of the gradient and the unevenness of sidewalks through machine learning from the sensor information and users’ subjective information during the passage of strollers to show the effectiveness of this technique.

Cross-validation of the obtained data was conducted by machine learning. The SVM was used for machine learning, and the kernlab library ksvm function of the open source free software in the R language was used for statistical analysis. The feature amounts used for learning were the mean and variance of the pitch angle of the angle sensor and the variance of the roll angle with respect to the road surface shapes. The mean and variance of the pitch angle of the angular velocity and the mean and variance of the roll angular velocity with respect to the road surface properties were used. Figures 6 and 7 indicate the distribution of feature values of labeled points. Figure 6 denotes the pitch and roll angle with respect to the road surface shapes. The variance of pitch and roll angle of composite gradient point tends to become large. And the mean of pitch angle of slope point trends to become large. Figure 7 denotes the pitch and roll angular velocity with respect to the road surface properties. The variance of pitch and roll angular velocity of unevenness point tends to become large. And the mean of pitch angular velocity of unevenness point tends to become small. Table 1 shows the conditions of experiments. The frequency of cross-validation is 450 times for the road surface shapes and 1000 times for the road surface properties. In other words, it is considered that one sensor datum is not labelled for road surface shapes, and the estimation of road surface conditions was repeated 450 times as the total number in the estimation model which was constructed from the sensor data corresponding to 449 labels. One set of data was estimated in the model which was constructed from 999 sets of data for road surface properties as well. Results of estimation in the cross-validation of road surface shapes and road surface properties are shown in Tables 2 and 3 respectively.

The figure in each column in Tables 2, 3 represents the number of data whose respective labels were estimated for the sensor data considered to be unlabeled. Italics numbers represent the number of data whose estimated results are correct. In addition, the recall ratio represents the percentage of the data correctly estimated for labels. The precision factor represents the ratio of correct answers for the estimated results.

Looking at the results, angle sensors and angular velocity sensors were effective in determining road surface shapes and properties, respectively. However, the relevance factor of the flatness for the composite gradient: 83.4 %, and the recall ratio of the composite gradient: 88 %, are lower values as compared to others in the estimated results for the road surface shapes. The estimation of composite gradients is likely to be less sufficient with only angle sensors. In addition, composite gradient points involve the difficulty or reluctance to press for the travelling of strollers as uneven points. In order to improve estimation accuracy, we plan to proceed with the efficient collection of sensor data and user-subjective training data based on the proposed system, considering that if such subjective experiences in the difficulties in passage affect the acceleration and the angular velocity, there is a possibility to improve the estimation accuracy of the composite gradients using their feature amounts.

Figure 8 denotes pictures of the surface judged to be uneven at a part besides the data used for learning. The shape of the road surface changed and these places were a part where a pedestrian feels the resistance more than usual to push a stroller. These results confirm the effect of the machine learning. Measuring error of GPS is several meters. Therefore the uneven spot strictly can not be specified, but we think that the outline of the shape of the sidewalk can be useful information. It was validity confirmation by the limited condition by an experiment of this first step. As the future works, we consider expansion of input sensor information, introduction of vital information, a variation of strollers and a variation of movement. And we examine the scalability of the proposed system.

In this report, we proposed a system to be the basis for efficient information gathering on sidewalk situations and services for extracting useful information and providing it to sidewalk users and local governments for the purpose of building a system to support social infrastructures through which users can move safety and securely. Specifically, we proposed a concept for a sidewalk supporting system that accumulates information from strollers, mobility scooters, bicycles, etc. through sidewalk users’ smartphones, sensors, and other such devices to provide valid information. Next, we showed a proposed basic system configuration utilizing data collection technologies that have been developed in robots and walking appearance analysis with regard to responses to the diversity of sensors and the characteristics of smartphones. Furthermore, we introduced techniques for estimating sidewalk conditions using sidewalk passers’ subjectivity with regard to the acquisition of training data to analyse data effectively. As a specific case of application, we evaluated the estimation of the gradient and unevenness of sidewalks by machine learning from the sensor data during the passage of strollers, verifying the effectiveness of this technique.