Robotic Grasping and Manipulation Competition: Future Tasks to Support the Development of Assembly Robotics

The Robot Grasping and Manipulation Competition, held during the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) in Daejeon, South Korea was sponsored by the IEEE Robotic and Automation Society (RAS) Technical Committee (TC) on Robotic Hands Grasping and Manipulation (RHGM) [1]. This competition was the first of a planned series of grasping and manipulation-themed events of increasing difficulty that are intended to spur technological developments and advance test methods and benchmarks so that they can be formalized for use by the community. The coupling of standardized performance testing with robot competitions will promote the use of unbiased evaluation methods to assess how well a robot system performs in a particular application space. A strategy is presented for a series of grasping and manipulation competitions that facilitate objective performance benchmarking of robotic assembly solutions. This strategy is based on test methods that can be used for more rigorous assessments and comparison of systems and components outside of the competition regime. While competitions have proven to be useful mechanisms for assessing the relative performance of robotic systems with measures of success, they often lack a methodical measurement science foundation. Consequently, scientifically sound and statistically significant metrics, measurement, and evaluation methods to quantify performance are missing. Using performance measurement methods in a condensed format will accommodate competition time limits while introducing the methods to the community as tools for benchmarking performance in the developmental and deployment phases of a robot system. The particular evaluation methods presented here are focused on the mechanical assembly process, an application space that is expected to accelerate with the new robot technologies coming to market.