2016 International Symposium on Experimental Robotics

The ability to transfer knowledge gained in previous tasks into new contexts is one of the most important mechanisms of human learning. Despite this, adapting autonomous behavior to be reused in partially similar settings is still an open problem in current robotics research. In this paper, we take a small step in this direction and propose a generic framework for learning transferable motion policies. Our goal is to solve a learning problem in a target domain by utilizing the training data in a different but related source domain. We present this in the context of an autonomous MAV flight using monocular reactive control, and demonstrate the efficacy of our proposed approach through extensive real-world flight experiments in outdoor cluttered environments.

Prescribed fires have many benefits, but existing ignition methods are dangerous, costly, or inefficient. This paper presents the design and evaluation of a micro-UAS that can start a prescribed fire from the air, while being operated from a safe distance and without the costs associated with aerial ignition from a manned aircraft. We evaluate the performance of the system in extensive controlled tests indoors. We verify the capabilities of the system to perform interior ignitions, a normally dangerous task, through the ignition of two prescribed fires alongside wildland firefighters.

Infrastructure as bridges and dams requires periodic inspection. And it is necessary to record a change over the years. It is required to establish a maintenance cycle. Inspection there is a need to be observed by eyes from a short distance. But, it would need to cost and effort. In fact often cases of a telescope. Difference of the inspection method, there also are differences in the test results. In this study, to develop a testing apparatus of Unmanned Aerial Vehicle for the purpose of labor saving of inspection. It’s a hammering test equipment to be mounted to medium-size UAV. Safely to realize the hammering test in UAV hovering state.

The paper reports on a new data-driven methodology for uncertainty quantification in centimeter-scale robots. We employ tools from functional expansion-based methods, the Karhunen-Loeve (KL) decomposition in particular, to identify appropriate reduced-order models of robotic systems through empirical observations, and discover underlying dominant dynamical behaviors of a system in the presence of uncertainty. The approach is applied to a quadrotor aerial vehicle tasked to hover at various heights from the ground. Several experimental data sets are collected to extract dominant modes. First-order modes correctly capture expected behaviors of the system, while higher-order modes quantify the degree of uncertainty at different hovering conditions. The information provided by this model can be used to develop robust controllers in the face of aerodynamic disturbances and unmodeled nonlinearities.

This paper demonstrates how a heterogeneous fleet of unmanned aerial vehicles (UAVs) can support human operators in search and rescue (SaR) scenarios. We describe a fully autonomous delegation framework that interprets the top-level commands of the rescue team and converts them into actions of the UAVs. In particular, the UAVs are requested to autonomously scan a search area and to provide the operator with a consistent georeferenced 3D reconstruction of the environment to increase the environmental awareness and to support critical decision-making. The mission is executed based on the individual platform and sensor capabilities of rotary- and fixed-wing UAVs (RW-UAV and FW-UAV respectively): With the aid of an optical camera, the FW-UAV can generate a sparse point-cloud of a large area in a short amount of time. A LiDAR mounted on the autonomous helicopter is used to refine the visual point-cloud by generating denser point-clouds of specific areas of interest. In this context, we evaluate the performance of point-cloud registration methods to align two maps that were obtained by different sensors. In our validation, we compare classical point-cloud alignment methods to a novel probabilistic data association approach that specifically takes the individual point-cloud densities into consideration.

This paper describes fundamental principles for two-dimensional pattern design of folded robots, specifically mobile robots consisting of closed-loop kinematic linkage mechanisms. Three fundamental methods for designing closed-chain folded four-bar linkages – the basic building block of these devices – are introduced. Modular connection strategies are also introduced as a method to overcome the challenges of designing assemblies of linkages from a two-dimensional sheet. The result is a design process that explores the tradeoffs between the complexity of linkage fabrication and also allows the designer combine multiple functions or modes of locomotion. A redesigned modular robot capable of multi-modal locomotion and grasping is presented to embody these design principles.

The mobile sensor platform Moball uses an array of sliding magnets and solenoids inside a spherical shell to both harvest energy and displace its center of mass or barycenter from its center of rotation in order to control the path along which it rolls. Previous simulations of the harvesting potential for the complete system are validated experimentally, and certain phenomena that restrict effective operating conditions for energy harvesting are investigated. Tracking of characteristic trajectories for a single mass control element is used to assess the performance of the solenoids as actuators, and the ability of the system to generate a control torque during motion is demonstrated.

This study presents methods for understanding, modeling and control of tele-operated pneumatic actuators for rehabilitation in Magnetic Resonance Imaging (MRI). Pneumatic actuators have excellent MRI-compatibility as opposed to conventional electro-mechanical systems; however, the actuator and the system drivers cannot be co-located due to the MRI-compatibility requirements. The actuators are driven via long transmission lines, which affect the system dynamics significantly. Methods provided in this work produced accurate pressure estimation and control by accounting for the pressure dynamics in the lines, which has been neglected by previous work in this area. The effectiveness of the presented modeling and control methods were demonstrated on tele-operation test setups. This study also includes the design of necessary system components for the developed algorithms. An MRI-compatible optical sensor was developed for force feedback and its design was analyzed for high precision.

The ability to traverse uneven terrain is one of the key advantages of legged robots. However, their effectiveness relies on selecting appropriate gait parameters, such as stride height and leg stiffness. The optimal parameters highly depend on the characteristics of the terrain. This work presents a novel stereo vision based terrain sensing method for a hexapod robot with 30 degrees of freedom. The terrain in front of the robot is analyzed by extracting a set of features which enable the system to characterize a large number of terrain types. Gait parameters and leg stiffness for impedance control are adapted based on this terrain characterization. Experiments show that adaptive impedance control leads to efficient locomotion in terms of energy consumption, mission success and body stability.

Current and previous single-legged hopping robots are energetically tethered and lack portability. Here, we present the design and control of an untethered, energetically autonomous single-legged hopping robot. The thrust-producing mechanism of the robot’s leg is an actuated prismatic joint, called a linear elastic actuator in parallel (LEAP). The LEAP mechanism comprises a voice coil actuator in parallel with two compression springs, which gives our robot passive compliance. An actuated gimbal hip joint is realized by two standard servomotors. To control the robot, we adapt Raibert’s hopping controller, and find we can maintain balance roughly in-place for up to approx. 7 s (19 hops) while continuously hopping.

Legged locomotion is characterised by a repetitive appearance of impulsive ground collisions which are strongly influencing the locomotion behaviour. The collisions depend on the shape of the contacting foot, but little is known on how the foot needs to be shaped to assist stable and fast locomotion. This paper investigates discrepancies in locomotion dynamics caused by a discrete foot shape change. A curved foot, open-loop controlled hopping robot which can be switched between two foot shape states was built and tested for the experimental investigations. The results indicate that the right timing of foot shape change can induce a variety of locomotion gaits and increase maximal speed by up to 40%, without the shape change doing any positive work on the robot. Three distinct take off cases were identified which depend on the robot’s state and foot shape. The switching between the cases in consecutive hops can explain the observed behaviour qualitatively as presented in this paper.

In this work, a supervised learning strategy has been applied in conjunction with a control strategy to provide anthropomorphic hand-arm systems with autonomous grasping capabilities. Both learning and control algorithms have been developed in a synergy-based framework in order to address issues related to high dimension of the configuration space, that typically characterizes robotic hands and arms with human-like kinematics. An experimental setup has been built to learn hand-arm motion from humans during reaching and grasping tasks. Then, a Neural Network (NN) has been realized to generalize the grasps learned by imitation. Since the NN approximates the relationship between the object characteristics and the grasp configuration of the hand-arm system, a synergy-based control strategy has been applied to overcome planning errors. The reach-to-grasp strategy has been tested on a setup constituted by the KUKA LWR 4+ Arm and the SCHUNK 5-Finger Hand.

The perception of affordances in unknown environments is an essential prerequisite for autonomous humanoid robots. In our previous work we developed a perceptual pipeline for the extraction of affordances for loco-manipulation actions based on a simplified representation of the environment starting from RGB-D camera images. The feasibility of this approach has been demonstrated in various examples in simulation as well as on real robotic platforms. The overall goal of the perceptual pipeline is to provide a robust and reliable perceptual mechanism for affordance-based action execution.In this work we evaluate the performance of the perceptual pipeline in combination with sensor systems other than RGB-D cameras, in order to utilize redundant sensor equipment of humanoid robots. This is particularly important when considering challenging scenarios where particular sensors are not applicable, e.g. due to intense sunlight or reflective surfaces. In this work we focus on stereo cameras and LIDAR laser scanners.

For direct-drive legged robots operating in unstructured environments, workspace volume and force generation are competing, scarce resources. In this paper we demonstrate that introducing geared core actuation (i.e., proximal to rather than distal from the mass center) increases workspace volume and can provide a disproportionate amount of work-producing-force to the mass center without affecting leg linkage transparency. These effects are analytically quantifiable up to modest assumptions, and are demonstrated empirically on a spined quadruped performing a leap both on level ground and from an isolated foothold (an archetypal feature of unstructured terrain).

Robotic grasping has attracted considerable interest, but it still remains a challenging task. The data-driven approach is a promising solution to the robotic grasping problem; this approach leverages a grasp dataset and generalizes grasps for various objects. However, these methods often depend on the quality of the given datasets, which are not trivial to obtain with sufficient quality. Although reinforcement learning approaches have been recently used to achieve autonomous collection of grasp datasets, the existing algorithms are often limited to specific grasp types. In this paper, we present a framework for hierarchical reinforcement learning of grasping policies. In our framework, the lower-level hierarchy learns multiple grasp types, and the upper-level hierarchy learns a policy to select from the learned grasp types according to a point cloud of a new object. Through experiments, we validate that our approach learns grasping by constructing the grasp dataset autonomously. The experimental results show that our approach learns multiple grasping policies and generalizes the learned grasps by using local point cloud information.

We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. To train our network, we collected over 800,000 grasp attempts over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and hardware. Our experimental evaluation demonstrates that our method achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing.

We demonstrate how low-cost in-hand proximity and dynamic tactile sensing can dramatically improve the reliability of basic manipulation tasks. We use an array of infrared proximity sensors embedded in a transparent elastic polymer and an accelerometer in the robot’s wrist to extract proximity and dynamic tactile information that is inspired by the mechanoreceptors in the human skin. We break the manipulation task down into eight distinct phases and show (1) how proximity information can be used to improve reliability of picking and placing objects, and (2) how dynamic tactile information can be used to discern different phases of grasping. We present experimental results using a Baxter robot involved in a tower construction task.

Narrow passages and orientation constraints are very common in manipulation tasks and sampling-based planning methods can be quite time-consuming in such scenarios. We propose a method that can learn object orientation constraints and guiding constraints, represented as Task Space Regions, from a single human demonstrations by analyzing the geometry around the demonstrated trajectory. The key idea of our method is to explore the area around the demonstration trajectory through sampling in task space, and to learn constraints by segmenting and analyzing the feasible samples. Our method is tested on a tire-changing scenario which includes four sub-tasks and on a cup-retrieving task. Our results show that our method can produce plans for all these tasks in less than 3 min with 50 / 50 successful trials for all tasks, while baseline methods only succeed 1 out of 50 times in 30 min for one of the tasks. The results also show that our method can perform similar tasks with additional obstacles, transfer to similar tasks with different start and/or goal poses, and be used for real-world tasks with a PR2 robot.

Manipulation skills need to adapt to the geometric features of the objects that they are manipulating, e.g. the position or length of an action-relevant part of an object. However, only a sparse set of the objects’ features will be relevant for generalizing the manipulation skill between different scenarios and objects. Rather than relying on a human to select the relevant features, our work focuses on incorporating feature selection into the skill learning process. An informative prior over the features’ relevance can guide the robot’s feature selection process. This prior is computed using a meta-level prior, which is learned from previous skills. The meta-level prior is transferred to new skills using meta features. Our robot experiments show that using a meta-level prior results in better generalization performance and more efficient skill learning.

In this paper, we introduce a method to build 3D object models from RGB-D images automatically by interleaving model building with robotic manipulation. Using a fixed RGB-D camera and starting from the first view of the object, our approach gradually builds and extends a partial model (based on what has been visible) into a complete object model. In the process, the partial model is also used to guide a robot manipulator to change the pose of the object to make more surfaces visible for continued model building. The alternation of perception-based model building and pose changing continues until a complete object model is built with all object surfaces covered. The method is implemented, and experimental results show the effectiveness and robustness of this approach.

In most robotic applications, tactile sensors are modeled as rigid matrices of adjacent pressure sensing elements so that the geometric deformations of their surfaces are neglected. This paper proposes to define these deformations as rotational and translational compliances that are later used as new tactile features for tactile servoing tasks. In fact, a novel two-layered inverse tactile Jacobian matrix is developed in order to map errors of these features into cartesian errors required for touch-driven exploration and manipulation robotic tasks. The performance of this new tactile servoing approach is demonstrated in several real experiments with a 6$$\,\times \,$$×14 tactile array mounted on a 7-dof robotic manipulator.

Consumer electronic devices are made by the millions, and automating their production is a key manufacturing challenge. Fastening machine screws is among the most difficult components of this challenge. To accomplish this task with sufficient robustness for industry, detecting and recovering from failure is essential. We have built a robotic screwdriving system to collect data on this process. Using it, we collected data on 1862 screwdriving runs, each consisting of force, torque, motor current and speed, and video. Each run is also hand-labeled with the stages of screwdriving and the result of the run. We identify several distinct stages through which the system transitions and relate sequences of stages to characteristic failure modes. In addition, we explore several techniques for automatic result classification, including standard maximum angle/torque methods and machine learning time series techniques.

Household manipulation presents a challenge to robots because it requires perceiving a variety of objects, planning multi-step motions, and recovering from failure. This paper presents practical techniques that improve performance in these areas by considering the complete system in the context of this specific domain. We validate these techniques on a table-clearing task that involves loading objects into a tray and transporting it. The results show that these techniques improve success rate and task completion time by incorporating expected real-world performance into the system design.

This paper presents an application of a robot manipulator to perform Cardiopulmonary resuscitation(CPR) in emergency situations. CPR is one of the most important treatments which serves to save patients in cardiac arrest. The proposed robot CPR system attempts to overcome the limitations of current CPR methods in two aspects. First, it can provide much more consistent CPR than humans in terms of strength and timing. Second, biological data of a patient can be used to determine the best compression point during CPR. The feasibility of the proposed system is demonstrated through experiments: one simulation on a mannequin and two animal tests. It is also expected that this robotic CPR system can be a good platform to investigate many aspects of CPR methods and guidelines with accurate measurements and actions.

Here, we present insights into human contact-control strategies by defining conditions to determine whether a human controls a contact state, empirically analyzing object-to-environment contact geometry data obtained from human demonstrations in a haptic simulation environment, and testing hypothesess about underlying human contact-control strategies. Using haptic demonstration data from eleven subjects who inserted non-convex objects into occluded holes, we tested the following human contact-control hypotheses: (h1) humans follow a task trajectory that tracks pre-planned contact-state waypoints organized in a contact-state graph (contact-waypoint hypothesis); (h2) humans traverse the contact-state graph, explicitly controlling some contact states or subsets of contact states, in addition to the pre-determined initial and final goal states (controlled subgraph hypothesis); (h3) humans use a control policy where the only controlled states are the starting state for the task and the goal state (state policy hypothesis). Notably, we found that humans tend to visit a select few contact states once they enter each state’s vicinity in the graph, which is evidence against h3. Yet humans do not always visit said states (visit probability $$<40\%$$<40%), which is, in addition, evidence against h1 provided different humans adopt similar strategies. We show that a classifier to determine when humans control their trajectories to visit specific contact states, when parameterized correctly, is invariant to graph aggregation operations across the false-positive to false-negative tradeoff spectrum. This indicates our results are robust given the data we obtained and suggests that efforts to characterize human motion should focus on h2.

We present a Human-Robot-Collaboration (HRC) framework consisting of a hybrid human motion prediction approach together with a game theoretical action selection. In essence, the robot is required to predict the motions of the human co-worker, and to proactively decide on its actions. For our prediction framework, model-based human motion trajectories are learned by data-driven methods for efficient trajectory rollouts in which obstacles are also considered. We provide the reliability analysis of human trajectory predictions within a human-robot collaboration experimental setup. The HRC scenario is modeled as an iterative game to select the actions for the Human-Robot-Team (HRT) by finding the Nash Equilibrium of the game. Experimental evaluation shows how the proposed prediction approach can be successfully integrated into a game theory based action selection framework.

We present a new, lightweight prototype of the Supernumerary Robotic Limbs (SRL), a wearable robot that augments its user by providing two extra robotic legs. We then showcase the robot’s assistive capabilities by developing and implementing a control strategy that supports the user during sitting and standing motions. The reduced mass and volume of the robot are enabled by innovative design choices including advanced materials, efficient joint structure, and high-performance pneumatic actuation. The assistive control strategy is tailored to each individual based on their motion preferences, and allows the SRL to support users without getting in the way of their movements. The proposed assistive strategy is implemented and validated in experiments with the physical SRL prototype.

With recent advances in robotics technologies and autonomous systems, the idea of human-robot teams is gaining ever-increasing attention. In this context, our research focuses on developing an intelligent robot that can autonomously perform non-trivial, but specific tasks conveyed through natural language. Toward this goal, a consortium of researchers develop and integrate various types of intelligence into mobile robot platforms, including cognitive abilities to reason about high-level missions, perception to classify regions and detect relevant objects in an environment, and linguistic abilities to associate instructions with the robot’s world model and to communicate with human teammates in a natural way. This paper describes the resulting system with integrated intelligence and reports on the latest assessment.

Our research concerns the mixed-initiative coordination of air and underwater vehicles interacting over inter-operated radio and underwater communication networks for novel oceanographic field studies. In such an environment, operating multiple vehicles to observe dynamic oceanographic events such as fronts, plumes, blooms and cetaceans has required that we design, implement and operate software, methods and processes which can support ephemeral and unpredictable observations (including those of moving animals) in real-world settings with substantial constraints. We articulate an approach for coordinated measurements using such platforms, which relate directly to task outcomes. We show the use and operational value of a new Artificial Intelligence (AI) based mixed-initiative system, EUROPtus, for handling multiple platforms from a recent field experiment in open waters of the mid-Atlantic.

We focus on selecting handover configurations that result in low human ergonomic cost not only at the time of handover, but also when the human is achieving a goal with the object after that handover. People take objects using whatever grasping configuration is most comfortable to them. When the human has a goal pose they’d like to place the object at, however, the most comfortable grasping configuration at the handover might be cumbersome overall, requiring regrasping or the use of an uncomfortable configuration to reach the goal. We enable robots to purposefully influence the choices available to the person when taking the object, implicitly helping the person avoid suboptimal solutions and account for the goal. We introduce a probabilistic model of how humans select grasping configurations, and use this model to optimize expected cost. We present results in simulation, as well as from a user study, showing that the robot successfully influences people’s grasping configurations for the better.

We present experimental data for an ankle exoskeleton that provides a metabolic benefit during running. Intuitively, there is an optimal level of power that any particular human can accept and use to benefit walking or running, which is a function of the particular human, the selected gait, and speed. We provide and discuss modeling optimization results to complement our recent data for the device, toward modifying future designs and understanding theoretical performance limits.

To increase the speed of automatic transport vehicles in underground narrow pathways, we have developed a differential four-wheel vehicle that can keep contact with the wall using roller bumpers. In wall-contacting driving, the vehicle may be damaged by the collision with convexity and concavity of the wall. In this research, a preliminary experiment highlights what kind of convexity and concavity greatly affect the vehicle. Based on those results, this paper proposes a convexity and concavity detection method using geometric feature extraction of wall roughness. To evaluate the performance of the method, experiments have been conducted by using a scale model. The experimental results clarify the feasibility of the detection and the collision avoidance of wall convexity and concavity using the proposed feature values extracted from the distance sensor data.

We present our work on autonomous vehicles in an urban environment to provide mobility-on-demand as a solution to the first and last mile problem. The software architecture for our vehicles is reviewed with focus on new developments of speed and steering control algorithms to ensure robust performance for autonomous driving. For speed control, a brake/throttle switching controller based on velocity error and desired acceleration is implemented to achieve fast speed response without excessive switching. An iterative learning algorithm is used to train feedforward signals which are then used to compensate the repeated disturbances over a fixed route. For steering control, a revised pure pursuit steering control algorithm is designed to improve path tracking performance. The methods are validated though on-road experiments which demonstrate a speed control that is robust against changing road grade and a steering control that has smaller cross-track errors.

For robots navigating in unknown environments, naïvely following the shortest path toward the goal often leads to poor visibility of free space, limiting navigation speed, or even preventing forward progress altogether. In this work, we train a guidance function to give the robot greater visibility into unknown parts of the environment. Unlike exploration techniques that aim to observe as much map as possible for its own sake, we reason about the value of future observations directly in terms of expected cost-to-goal. We show significant improvements in navigation speed and success rate for narrow field-of-view sensors such as popular RGBD cameras. However, contrary to our expectations, we show that our strategy makes little difference for sensors with fields-of-view greater than 80$$^{\circ }$$∘, and we discuss why the naïve strategy is hard to beat.

This paper introduces a method to form global patterns of 10+ autonomous millirobots, the Tiny Terrestrial Robot Platforms (TinyTeRPs), with only local information available to each robot. The TinyTeRPs are equipped with light sensors that measure a globally projected dynamic light field, and radios for RSSI inter-robot distance estimation. An agent-based simulation is used to extrapolate the experimentally observed behavior in swarms of up to 225 robots. In concert, a diffusion-inspired analytic model is presented to create closed-form equations for estimating the time it will take to form a pattern. This allows the behavior to be characterized using an algorithmic complexity framework. With this approach, forming patterns can be done in time scaling linearly with the number of robots in the pattern. Furthermore, the models predict that no matter how large swarm size grows, patterns can be formed in constant time as long as only a fixed fraction of the swarm needs to succeed. Insights into the reasons for this behavior, characterizations of the sensors and their interactions, and suggestions to build on this work are included.

In the absence of reliable and accurate GPS, visual odometry (VO) has emerged as an effective means of estimating the egomotion of robotic vehicles. Like any dead-reckoning technique, VO suffers from unbounded accumulation of drift error over time, but this accumulation can be limited by incorporating absolute orientation information from, for example, a sun sensor. In this paper, we leverage recent work on visual outdoor illumination estimation to show that estimation error in a stereo VO pipeline can be reduced by inferring the sun position from the same image stream used to compute VO, thereby gaining the benefits of sun sensing without requiring a dedicated sun sensor or the sun to be visible to the camera. We compare sun estimation methods based on hand-crafted visual cues and Convolutional Neural Networks (CNNs) and demonstrate our approach on a combined 7.8 Km of urban driving from the popular KITTI dataset, achieving up to a 43 % reduction in translational average root mean squared error (ARMSE) and a 59 % reduction in final translational drift error compared to pure VO alone.

This paper provides experimental evidence in support of the hypothesis that the Switching Four-bar Mechanism (SFM) model may serve as a template for miniature legged systems in quasi-static operation. The evidence suggests that the SFM captures salient motion behaviors of morphologically distinct centimeter-scale legged robots. Captured behaviors are then used for planning and control at small scales, thus demonstrating the practical utility of the SFM in navigation tasks.

Automatic harvesting consists of two main sub-steps: target recognition and picking/detachment of recognized targets. Target fruit recognition is a machine vision task that has been the subject of much research ever since the automatic harvesting was first introduced in the early 1960s [1]. The methods used for recognition largely depend on the properties of the fruit being harvested. Fruits, such as strawberries and tomatoes, can be relatively easily detected by a simple RGB color segmenting as the color of a ripe fruit differs significantly from both unripe fruits and the surrounding foliage, while fruits, such as green apples and green peppers, might require spectral analysis to distinguish them from the surrounding foliage [2]. In all cases, however, the recognition is greatly complicated by issues such as changing illumination conditions, shadows, occlusions of fruits by surrounding leaves and other fruits, color and shape variations and reflectance.

We have proposed a novel haptic display that can generate localized shearing pattern on human fingertip, resulting in the partial slip perception. The device comprises of a bundle of stiff pins resting on a specially designed concave base whose planar movement is controlled by a two-DOF (degree of freedom) linear stage. These pins’ free ends, when making contact with human fingertip surface, can displace horizontally and partially. The novel characteristic of this design is that the pattern of localized slippage could be generated by altering the geometric shape of the supporting base and the number of haptic pins. We introduced a dynamic model for investigation of mechanical response of stress or strain on human fingertip under operation of the proposed haptic device. By variation of the device’s design, it is possible to study the sense of partial slippage on human fingertip under experimental conditions such as applied force, sliding velocity, direction on slip perception of volunteering subjects, using VAS (visual analog scale). The results presented in this paper may help assess human slip perception for the development of haptic display.

This paper explores the idea of predicting the likely performance of a robot’s perception system based on past experience in the same workspace. In particular, we propose to build a place-specific model of perception performance from observations gathered over time. We evaluate our method in a classical decision making scenario in which the robot must choose when and where to drive autonomously in 60 km of driving data from an urban environment. We demonstrate that leveraging visual appearance within a state-of-the-art navigation framework increases the accuracy of our performance predictions.

Semantic scene understanding of unstructured environments is a highly challenging task for robots operating in the real world. Deep Convolutional Neural Network architectures define the state of the art in various segmentation tasks. So far, researchers have focused on segmentation with RGB data. In this paper, we study the use of multispectral and multimodal images for semantic segmentation and develop fusion architectures that learn from RGB, Near-InfraRed channels, and depth data. We introduce a first-of-its-kind multispectral segmentation benchmark that contains 15, 000 images and 366 pixel-wise ground truth annotations of unstructured forest environments. We identify new data augmentation strategies that enable training of very deep models using relatively small datasets. We show that our UpNet architecture exceeds the state of the art both qualitatively and quantitatively on our benchmark. In addition, we present experimental results for segmentation under challenging real-world conditions. Benchmark and demo are publicly available at http://deepscene.cs.uni-freiburg.de.

We present a novel method for yield estimation in apple orchards. Our method takes segmented and registered images of apple clusters as input. It outputs number and location of individual apples in each cluster. Our primary technical contributions are a representation based on a mixture of Gaussians, and a novel selection criterion to choose the number of components in the mixture. The method is experimentally verified on four different datasets using images acquired by a vision platform mounted on an aerial robot, a ground vehicle and a hand-held device. The accuracy of the counting algorithm itself is $$91\%$$91%. It achieves 81–85% accuracy coupled with segmentation and registration which is significantly higher than existing image based methods.

Recent advances in AI and robotics have claimed many incredible results with deep learning, yet no work to date has applied deep learning to the problem of liquid perception and reasoning. In this paper, we apply fully-convolutional deep neural networks to the tasks of detecting and tracking liquids. We evaluate three models: a single-frame network, multi-frame network, and a LSTM recurrent network. Our results show that the best liquid detection results are achieved when aggregating data over multiple frames and that the LSTM network outperforms the other two in both tasks. This suggests that LSTM-based neural networks have the potential to be a key component for enabling robots to handle liquids using robust, closed-loop controllers.

Obstacle avoidance is an essential capability for micro air vehicles. Prior approaches have mainly been either purely reactive, mapping low-level visual features directly to headings, or deliberative methods that use onboard 3-D sensors to create a 3-D, voxel-based world model, then generate 3-D trajectories and check them for potential collisions with the world model. Onboard 3-D sensor suites have had limited fields of view. We use forward-looking stereo vision and lateral structure from motion to give a very wide horizontal and vertical field of regard. We fuse depth maps from these sources in a novel robot-centered, cylindrical, inverse range map we call an egocylinder. Configuration space expansion directly on the egocylinder gives a very compact representation of visible freespace. This supports very efficient motion planning and collision-checking with better performance guarantees than standard reactive methods. We show the feasibility of this approach experimentally in a challenging outdoor environment.

In this paper, we introduce a novel type of the multirotor aerial vehicle with two-dimensional multilinks which enables the stable aerial transformation for high mobility in three-dimensional environments. Our goal is to hold and carry object by using the whole-body manipulation in the air. The research involved three steps. First, we developed the modeling of the link modules that compose a multirotor with two-dimensional multilinks and conducted a quadrotor prototype. Second, we derived a stable flight control method for aerial transformation on the basis of linear-quadratic-integral optimal control. Third, we investigated the whole-body aerial manipulation based on the enveloping grasping method for the four-link type which takes the additional inertial parameters and joint torque into account. Successful aerial transformation and manipulation with the quadrotor prototype were demonstrated, confirming the feasibility of our proposed transformable multirotor for aerial maneuvering.

In this work, we present a novel vision-based solution for operating a vehicle under Gaussian Distribution Temporal Logic (GDTL) constraints without global positioning infrastructure. We first present the mapping component that builds a high-resolution map of the environment by flying a team of two aerial vehicles in formation with sensor information provided by their onboard cameras. The control policy for the ground robot is synthesized under temporal and uncertainty constraints given the semantically labeled map. Finally, the ground robot executes the control policy given pose estimates from a dedicated aerial robot that tracks and localizes the ground robot. The proposed method is validated using a two-wheeled ground robot and a quadrotor with a camera for ten successful experimental trials.

In this paper, we present a novel resource-allocation problem formulation for vision-aided inertial navigation systems (VINS) for efficiently localizing micro aerial vehicles equipped with two cameras pointing at different directions. Specifically, based on the quadrotor’s current speed and median distances to the features, the proposed algorithm efficiently distributes processing resources between the two cameras by maximizing the expected information gain from their observations. Experiments confirm that our resource-allocation scheme outperforms alternative naive approaches in achieving significantly higher VINS positioning accuracy when tested onboard quadrotors with severely limited processing resources.

We present a method allowing a quadrotor equipped with only onboard cameras and an IMU to catch a flying ball. Our system runs without any external infrastructure and with reasonable computational complexity. Central to our approach is an online monocular vision-based ball trajectory estimator that recovers and predicts the 3-D motion of a flying ball using only noisy 2-D observations. Our method eliminates the need for direct range sensing via stereo correspondences, making it robust against noisy or erroneous measurements. Our system is made by a simple 2-D visual ball tracker, a UKF-based state estimator that fuses optical flow and inertial data, and a nonlinear tracking controller. We perform extensive analysis on system performance by studying both the system dynamics and ball trajectory estimation accuracy. Through online experiments, we show the first mid-air interception of a flying ball with an aerial robot using only onboard sensors.

This work presents an experiment-driven aerodynamic disturbance modeling technique that leverages experiences from past flights to construct a predictive model of the exogenous forces acting on an aerial robot. Specifically, we consider operation in turbulent air stemming from the interaction between wind and nearby surfaces. To construct the disturbance model, we employ Locally Weighted Projection Regression and relate the aerial robot’s state and an experimentally learned freestream wind model to the disturbance forces estimated during flight through the environment. The approach is experimentally validated through a set of flight tests in an indoor environment with artificially generated turbulent airflow that illustrate the computation of this disturbance model, its generalizability across flow conditions, and its utility for disturbance-aware motion planning.

We consider the problem of performing rapid training of a terrain classifier in the context of a collaborative robotic search and rescue system. Our system uses a vision-based flying robot to guide a ground robot through unknown terrain to a goal location by building a map of terrain class and elevation. However, due to the unknown environments present in search and rescue scenarios, our system requires a terrain classifier that can be trained and deployed quickly, based on data collected on the spot. We investigate the relationship of training set size and complexity on training time and accuracy, for both feature-based and convolutional neural network classifiers in this scenario. Our goal is to minimize the deployment time of the classifier in our terrain mapping system within acceptable classification accuracy tolerances. So we are not concerned with training a classifier that generalizes well, only one that works well for this particular environment. We demonstrate that we can launch our aerial robot, gather data, train a classifier, and begin building a terrain map after only 60 s of flight.Multimedia Material: This paper is accompanied by a video illustrating the approach, available at: http://rpg.ifi.uzh.ch.

In this work, we solve the labeled multi-robot planning problem. Most proposed algorithms to date have modeled robots as kinematic or kinodynamic agents in planar environments, making them impractical for real-world systems. Here, we present experiments to validate a centralized multi-robot planning and trajectory generation method that explicitly accounts for robots with higher-order dynamics. First, we demonstrate successful execution of solution trajectories. Next, we verify the robustness of the robots’ trajectory tracking to unmodeled external disturbances, in particular, the aerodynamic interactions between co-planar neighbors. Finally, we apply our algorithm to navigating quadrotors away from the downwash of their neighbors to improve safety in three-dimensional workspaces.

In this paper, we present soft hands guided by an RGB-D object perception algorithm which is capable of localizing the pose of an object before and after grasping. The soft hands can perform manipulation operations such as grasping and connecting two parts. The flexible soft grippers grasp objects reliably under high uncertainty but the poses of the objects after grasping are subject to high uncertainty. Visual sensing ameliorates the increased uncertainty by means of in-hand object localization. The combination of soft hands and visual object perception enables our Baxter robot, augmented with soft hands, to perform object assembly tasks which require high precision. The effectiveness of our approach is validated by comparing it to the Baxter’s original hard hands with and without the in-hand object localization.

In our previous study we have developed the seven axes multi joint gripper (MJG) having a variable stiffness mechanism and have showed it achieves some dexterous grasping. In this paper, we discuss a hand having a number of mluti-joint fingers that was subsequently designed on the basis of the former MJG. The mechanism mainly consists of a serially connected differential gear systems controlled by only two actuators: one for driving all of the joints simultaneously and the other for adjusting stiffness of every joints all together. The hand succeeded envelope grasping of various shape objects with no sensory feedback. The experiments also revealed that the hand with three multi-joint finger successfully achieves transition from pinching to envelope grasping. It also discuss how joint stiffness should be set according to handling modes of the hand.

We present a method for learning a general regrasping behavior by using supervised policy learning. First, we use reinforcement learning to learn linear regrasping policies, with a small number of parameters, for single objects. Next, a general high-dimensional regrasping policy is learned in a supervised manner by using the outputs of the individual policies. In our experiments with multiple objects, we show that learning low-dimensional policies makes the reinforcement learning feasible with a small amount of data. Our experiments indicate that the general high-dimensional policy learned using our method is able to outperform the respective linear policies on each of the single objects that they were trained on. Moreover, the general policy is able to generalize to a novel object that was not present during training.

This paper evaluates state-of-the-art contact models at predicting the motions and forces involved in simple in-hand robotic manipulations. In particular it focuses on three primitive actions—linear sliding, pivoting, and rolling—that involve contacts between a gripper, a rigid object, and their environment. The evaluation is done through thousands of controlled experiments designed to capture the motion of object and gripper, and all contact forces and torques at 250 Hz. We demonstrate that a contact modeling approach based on Coulomb’s friction law and maximum energy principle is effective at reasoning about interaction to first order, but limited for making accurate predictions. We attribute the major limitations to (1) the non-uniqueness of force resolution inherent to grasps with multiple hard contacts of complex geometries, (2) unmodeled dynamics due to contact compliance, and (3) unmodeled geometries due to manufacturing defects.

This paper proposes a iterative visual recognition system for learning based randomized bin-picking. Since the configuration on randomly stacked objects while executing the current picking trial is just partially different from the configuration while executing the previous picking trial, we consider detecting the poses of objects just by using a part of visual image taken at the current picking trial where it is different from the visual image taken at the previous picking trial. By using this method, we do not need to try to detect the poses of all objects included in the pile at every picking trial.Assuming the 3D vision sensor attached at the wrist of a manipulator, we first explain a method to determine the pose of a 3D vision sensor maximizing the visibility of randomly stacked objects. Then, we explain a method for detecting the poses of randomly stacked objects. Effectiveness of our proposed approach is confirmed by experiments using a dual-arm manipulator where a 3D vision sensor and the two-fingered hand attached at the right and the left wrists, respectively.

Field robots are gaining attentions that they can perform the tasks where human cannot reach. Improvement on physical performance of robots is a fundamental issue. Backdrivability is a key mechanical feature of actuators that enables robust and stable interaction of robots with environment. We developed a humanoid robot HYDRA with backdrivable hydraulic actuators. Parallel mechanism is used extensively in multi-DOF joints to efficiently use actuator force. In this paper, mechanical structure of Hydra with backdrivability and mechanical strength is treated. Also, realtime control of hydraulic actuators that utilize backdrivability, and realtime control of robot joint systems with parallel kinematic chains are explained.

Soft robots present a new opportunity for designing robots that can be produced quickly (on the order of hours), are capable of a variety of motions and behaviors, and are able to address a wide range of environments and tasks. The large design space of soft actuators can be leveraged to rapidly generate libraries of robotic components that can be used to compose modular soft robotic systems. To take full advantage of the large design space, we must have techniques for automatically synthesizing soft robot motions and behaviors. In this work, we develop a method for synthesizing gaits for walking soft robots, and show experimental results demonstrating synthesized gaits.

Reasoning jointly on perception and action requires to interpret the scene in terms of the agent’s own potential capabilities. We propose a Bayesian architecture for learning sensorimotor representations from the interaction between perception, action, and salient changes generated by robot actions. This connects these three elements in a common representation: affordances. In this paper, we are working towards a richer representation and formalization of affordances. Current experimental analysis shows the qualitative and quantitative aspects of affordances. In addition, our formalization motivates several experiments for exploring hypothetical operations between learned affordances. In particular, we infer affordances of composite objects, based on prior knowledge on the affordances of elementary objects.

Conditional planning enables planning for the sensing actions and their possible outcomes in addition to actuation actions, and allows for addressing uncertainties due to partial observability at the time of offline planning. Therefore, the plans (called conditional plans) computed by conditional planners can be viewed as trees of (deterministic) actuation actions and (nondeterministic) sensing actions. Hybrid conditional planning extends conditional planning further by integrating low-level feasibility checks into executability conditions of actuation actions in conditional plans. We introduce a novel hybrid conditional planning method, which extends hybrid sequential planning with nondeterministic sensing actions and utilizes this extension to compute branches of a conditional plan in parallel. We evaluate this method in a service robotics domain, by means of a set of experiments over dynamic simulations, from the perspectives of computational efficiency and plan quality.

Model-based reinforcement learning (MBRL) plays an important role in developing control strategies for robotic systems. However, when dealing with complex platforms, it is difficult to model systems dynamics with analytic models. While data-driven tools offer an alternative to tackle this problem, collecting data on physical systems is non-trivial. Hence, smart solutions are required to effectively learn dynamics models with small amount of examples. In this paper we present an extension to Data As Demonstrator for handling controlled dynamics in order to improve the multiple-step prediction capabilities of the learned dynamics models. Results show the efficacy of our algorithm in developing LQR, iLQR, and open-loop trajectory-based control strategies on simulated benchmarks as well as physical robot platforms.

This paper presents experimental insights from the deployment of an ensemble of heterogeneous autonomous sensor systems over a shallow coral reef. Visual, inertial, GPS, and ultrasonic data collected are compared and correlated to produce a comprehensive view of the health of the coral reef. Coverage strategies are discussed with a focus on the use of informed decisions to maximize the information collected during a fixed period of time.

We propose a multi robot SLAM approach that uses 3D objects as landmarks for localization and mapping. The approach is fully distributed in that the robots only communicate during rendezvous and there is no centralized server gathering the data. Moreover, it leverages local computation at each robot (e.g., object detection and object pose estimation) to reduce the communication burden. We show that object-based representations reduce the memory requirements and information exchange among robots, compared to point-cloud-based representations; this enables operation in severely bandwidth-constrained scenarios. We test the approach in simulations and field tests, demonstrating its advantages over related techniques: our approach is as accurate as a centralized method, scales well to large teams, and is resistant to noise.

Occupancy grid maps have been widely used for robot localization. Despite the popularity, this representation has some limitations, such as requirement of discretization of the environment, assumption of independence between grid cells and necessity of dense sensor data. Suppressing these limitations can improve the localization performance, but requires a different representation of the environment. Gaussian process occupancy map (GPOM) is a novel representation based on Gaussian Process that enables the construction of continuous maps (i.e. without discretization) using few laser measurements. This paper addresses a new localization method that uses GPOM to estimate the robot pose in areas not directly observed during mapping and generally provides higher accuracy compared to occupancy grid maps localization. Specifically, we devised a novel likelihood model based on the multivariate normal probability density function and adapted the particle filter localization method to work with GPOM. Experiments showed localization errors more than three times lower in comparison with particle filter localization using occupancy grid maps.

We propose an experimental method for studying mobility and surface operations of microgravity robots on zero-gravity parabolic flights—a test bed traditionally used for experiments requiring strictly zero gravity. By strategically exploiting turbulence-induced “gravity fluctuations,” our technique enables a new experimental approach for testing surface interactions of robotic systems in micro- to milli-gravity environments. This strategy is used to evaluate the performance of internally-actuated hopping rovers designed for controlled surface mobility on small Solar System bodies. In experiments, these rovers demonstrated a range of maneuvers on various surfaces, including both rigid and granular. Results are compared with analytical predictions and numerical simulations, yielding new insights into the dynamics and control of hopping rovers.

We present a solution for constant-time self-calibration and change detection of multiple sensor intrinsic and extrinsic calibration parameters without any prior knowledge of the initial system state or the need of a calibration target or special initialization sequence. This system is capable of continuously self-calibrating multiple sensors in an online setting, while seamlessly solving the online SLAM problem in real-time. We focus on the camera-IMU extrinsic calibration, essential for accurate long-term vision-aided inertial navigation. An initialization strategy and method for continuously estimating and detecting changes to the maximum likelihood camera-IMU transform are presented. A conditioning approach is used, avoiding problems associated with early linearization. Experimental data is presented to evaluate the proposed system and compare it with artifact-based offline calibration developed by our group.

The problem of state estimation using primarily visual data has received a lot of attention in the last decade. Several open source packages have appeared addressing the problem, each supported by impressive demonstrations. Applying any of these packages on a new dataset however, has been proven extremely challenging. Suboptimal performance, loss of localization, and challenges in customization have not produced a clear winner. Several other research groups have presented superb performance without releasing the code, sometimes materializing as commercial products. In this paper, ten of the most promising open source packages are evaluated, by cross validating them on the datasets provided for each package and by testing them on eight different datasets collected over the years in our laboratory. Indoor and outdoor, terrestrial and flying vehicles, in addition to underwater robots, cameras, and buoys were used to collect data. An analysis on the motions required for the different approaches and an evaluation of their performance is presented.

Accurate human pose estimation is of vital importance for a variety of human-robot interaction applications, including cooperative task execution, imitation learning and robot-assisted rehabilitation. As robots move from controlled indoor environments to unstructured and outdoor environments, the ability to accurately measure human pose without fixed sensors becomes increasingly important. In this paper, we present a general framework for accurately estimating human pose from a variety of sensors, including body-worn inertial measurement units and cameras, that can be used in indoor and outdoor environments to accurately estimate human pose during arbitrary 3D movements. Using a kinematic model of the human body, the sensor data is fused to estimate the body joint angles and velocities using a constrained Extended Kalman Filter which automatically incorporates feasible joint limits. For periodic movement such as gait, performance can be further improved via online learning of the gait model, individualized to the user. The proposed approach can deal with intermittent data availability and measurement errors during highly dynamic movements.

The purpose of this study is to establish if emotional motions are important for the human perception of robots using proxemics as a tool. In this Human-Robot Interaction (HRI) experiment, participants were given instructions from a robot that was conveying either a sad, happy or neutral emotion. The emotional motions were generated as a low priority task using the robot Jacobian null-space. Participants were guided by the robot to sit at a desk to fill in a questionnaire and then to approach the robot to a distance that made them feel comfortable. A significant difference was found between the distance taken towards the robot in the Happy and the Sad conditions confirming our hypothesis that emotions conveyed by the robots influence how it is perceived.

In this paper, we propose physically meaningful energy related safety indicators for robots sharing their workspace with humans. Based on these indicators, safety criteria are introduced as constraints in the control algorithm. The first constraint is placed on the kinetic energy of the robotic system to limit the amount of dissipated energy in case of collision. This constraint depends on the distance between the robot and the human operator. The distance is computed with a point cloud based algorithm acquired using a set of depth sensors (Kinects). The second constraint is on the amount of potential energy that is allowed to be generated within the human-robot system during physical contact. It is used to modulate the contact forces. The control algorithm is formulated as an optimization problem and computes every time step the actuation torques for a KUKA LWR4 manipulator given some task to be performed, the introduced constraints and the physical limitations of the system to respect. The overall framework allows a human operator to safely enter the robot’s workspace and physically interact with it.

This paper describes the influence of the appearance and synchronizing operation of android to the extension of “personal space” from an operator to the android. We proposed a hypothesis that the extension of personal space from an operator to the android is affected by their agency and sympathy to the android. To verify this hypothesis, we conducted an experiment (1) in which an experimenter approached the android robots and unhuman-like robot to the intimate distance after male/female participant became familiar with the synchronized operation. The SCR (Skin Conductance Response) signal of the operator was measured as an indication of psychological or physiological arousal of the operator. The fluctuation of SCR was observed as reactions to stimuli of another person closely approaching the same gender android that is controlled to be synchronizing with the operator. As a result, the SCR in the android synchronous motion condition was significantly larger than the android/unhuman-like robot no motion condition.

Algorithms for robotic swarms often involve programming each robot with simple rules that cause complex group behavior to emerge out of many individual interactions. We study an algorithm with emergent behavior that transforms a robotic swarm into a single unified computational meta-entity that can be programmed at runtime. In particular, a swarm-spanning artificial neural network emerges as wireless neural links between robots self-organize. The resulting artificial group-mind is trained to differentiate between spatially heterogeneous light patterns it observes by using the swarm’s distributed light sensors like cells in a retina. It then orchestrates different coordinated heterogeneous swarm responses depending on which pattern it observes. Experiments on real robot swarms containing up to 316 robots demonstrate that this enables collective decision making based on distributed sensor data, and facilitates human-swarm interaction.

Human communication is highly multimodal, including speech, gesture, gaze, facial expressions, and body language. Robots serving as human teammates must act on such multimodal communicative inputs from humans, even when the message may not be clear from any single modality. In this paper, we explore a method for achieving increased understanding of complex, situated communications by leveraging coordinated natural language, gesture, and context. These three problems have largely been treated separately, but unified consideration of them can yield gains in comprehension [1, 12].