Top

Journal of Intelligent Manufacturing

Published in:

24-06-2022

A new differentiable architecture search method for optimizing convolutional neural networks in the digital twin of intelligent robotic grasping

Authors: Weifei Hu, Jinyi Shao, Qing Jiao, Chuxuan Wang, Jin Cheng, Zhenyu Liu, Jianrong Tan

Published in: Journal of Intelligent Manufacturing | Issue 7/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Convolutional neural networks (CNNs) have been widely used for object recognition and grasping posture planning in intelligent robotic grasping (IRG). Compared with the traditional usage of CNNs in image recognition, IRGs require high recognition accuracy and computational efficiency. However, the existing methodologies for CNN architecture design often rely on human experience and numerous trial-and-error attempts, which make it a very challenging task to obtain an optimal CNN for IRGs. To tackle this challenge, this paper develops a new differentiable architecture search (DARTS) method considering the floating-point operations (FLOPs) of CNNs, named the DARTS-F method, which converts the discrete CNN architecture search to a gradient-based continuous optimization problem and considers both the prediction accuracy and the computational cost of the CNN during the optimization. To efficiently identify the optimal neural network, this paper adopts a bilevel optimization, which first trains the neural network weights in the inner level and then optimizes the neural network architecture by fine-tuning the operational variables in the outer level. In addition, a new digital twin (DT) of IRG is developed considering the physics of realistic robotic grasping in the DT’s virtual space, which could not only improve the IRG accuracy but also avoid the expensive training time. In the experiments, the proposed DARTS-F method could generate an optimized CNN with higher prediction accuracy and lower FLOPs than those obtained by the original DARTS method. The DT framework improves the accuracy of real robotic grasping from 61 to 71%.

previous article Cascaded foreign object detection in manufacturing processes using convolutional neural networks and synthetic data generation methodology

next article The integrated process planning and scheduling of flexible job-shop-type remanufacturing systems using improved artificial bee colony algorithm

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Akinola, I., Angelova, A., Lu, Y., Chebotar, Y., Kalashnikov, D., Varley, J., Ibarz, J., & Ryoo, M. S. (2021). Visionary: Vision architecture discovery for robot learning. In Proceedings of 2021 IEEE international conference on robotics and automation (ICRA), Xi'an, China. https://doi.org/10.1109/ICRA48506.2021.9561998

Andrychowicz, O. M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., & Ray, A. (2020). Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1), 3–20. https://doi.org/10.1177/0278364919887447CrossRef

Bicchi, A. (1994). On the problem of decomposing grasp and manipulation forces in multiple whole-limb manipulation. Robotics and Autonomous Systems, 13(2), 127–147. https://doi.org/10.1016/0921-8890(94)90055-8CrossRef

Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., Downs, L., Ibarz, J., Pastor, P., & Konolige, K. (2018). Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In Proceedings of 2018 IEEE international conference on robotics and automation (ICRA), Brisbane, QLD, Australia. https://doi.org/10.1109/ICRA.2018.8460875

Chen, X., Xie, L., Wu, J., & Tian, Q. (2019). Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of 2019 IEEE/CVF international conference on computer vision (ICCV), Seoul, South Korea. https://doi.org/10.1109/ICCV.2019.00138

Colson, B., Marcotte, P., & Savard, G. (2007). An overview of bilevel optimization. Annals of Operations Research, 153(1), 235–256. https://doi.org/10.1007/s10479-007-0176-2CrossRef

Duan, J.-G., Ma, T.-Y., Zhang, Q.-L., Liu, Z., & Qin, J.-Y. (2021). Design and application of digital twin system for the blade-rotor test rig. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-021-01824-wCrossRef

Elsken, T., Metzen, J. H., & Hutter, F. (2019). Efficient multi-objective neural architecture search via Lamarckian evolution. In International conference on learning representations, New Orleans, LA, USA. https://doi.org/10.48550/arXiv.1804.09081

Erez, T., Tassa, Y., & Todorov, E. (2015). Simulation tools for model-based robotics: Comparison of bullet, Havok, MuJoCo, ODE and PhysX. In Proceedings of 2015 IEEE international conference on robotics and automation (ICRA), Seattle, WA, USA. https://doi.org/10.1109/ICRA.2015.7139807

Haarnoja, T., Pong, V., Zhou, A., Dalal, M., Abbeel, P., & Levine, S. (2018). Composable deep reinforcement learning for robotic manipulation. In Proceedings of 2018 IEEE international conference on robotics and automation (ICRA), Brisbane, QLD, Australia. https://doi.org/10.1109/ICRA.2018.8460756

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of 2016 IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, USA. https://doi.org/10.1109/CVPR.2016.90

Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of 2017 IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, USA. https://doi.org/10.1109/CVPR.2017.243

Hunt, K., & Torfason, L. (1987). A three-fingered pantograph manipulator. A kinematic study. Journal of Mechanisms, Transmissions, and Automation in Design, 109(2), 171–177. https://doi.org/10.1115/1.3267432CrossRef

Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021). How to train your robot with deep reinforcement learning: Lessons we have learned. The International Journal of Robotics Research, 40(4–5), 698–721. https://doi.org/10.1177/0278364920987859CrossRef

James, S., Wohlhart, P., Kalakrishnan, M., Kalashnikov, D., Irpan, A., Ibarz, J., Levine, S., Hadsell, R., & Bousmalis, K. (2019). Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. In Proceedings of the 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA. https://doi.org/10.1109/CVPR.2019.01291

Johns, E., Leutenegger, S., & Davison, A. J. (2016). Deep learning a grasp function for grasping under gripper pose uncertainty. In Proceedings of 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), Stockholm, Sweden. https://doi.org/10.1109/IROS.2016.7759657

Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., & Vanhoucke, V. (2018). Scalable deep reinforcement learning for vision-based robotic manipulation. In Proceedings of conference on robot learning (CoRL 2018), Zürich, Switzerland.

Kandasamy, K., Neiswanger, W., Schneider, J., Poczos, B., & Xing, E. P. (2018). Neural architecture search with Bayesian optimisation and optimal transport. In Advances in neural information processing systems (NeurIPS 2018), Montréal, Canada. https://doi.org/10.5555/3326943.3327130

Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In 3rd International conference on learning representations, San Diego, CA, USA.

Kumra, S., & Kanan, C. (2017). Robotic grasp detection using deep convolutional neural networks. In Proceedings of 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), Vancouver, BC, Canada. https://doi.org/10.1109/IROS.2017.8202237

Lenz, I., Lee, H., & Saxena, A. (2015). Deep learning for detecting robotic grasps. The International Journal of Robotics Research, 34(4–5), 705–724. https://doi.org/10.1177/0278364914549607CrossRef

Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018). Learning hand–eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4–5), 421–436. https://doi.org/10.1177/0278364917710318CrossRef

Liang, H., Zhang, S., Sun, J., He, X., Huang, W., Zhuang, K., & Li, Z. (2019). DARTS+: Improved differentiable architecture search with early stopping. Computing Research Repository. https://doi.org/10.48550/arXiv.1909.06035CrossRef

Lim, K. Y. H., Zheng, P., & Chen, C.-H. (2020). A state-of-the-art survey of digital twin: Techniques, engineering product lifecycle management and business innovation perspectives. Journal of Intelligent Manufacturing, 31(6), 1313–1337. https://doi.org/10.1007/s10845-019-01512-wCrossRef

Liu, H., Simonyan, K., & Yang, Y. (2019). DARTS: Differentiable architecture search. In International conference on learning representations, New Orleans, LA, USA. https://doi.org/10.48550/arXiv.1806.09055

Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Ojea, J. A., & Goldberg, K. (2017). Dex-Net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. In Robotics: Science and systems, Cambridge, Massachusetts. https://doi.org/10.15607/RSS.2017.XIII.058

Mahler, J., Matl, M., Satish, V., Danielczuk, M., DeRose, B., McKinley, S., & Goldberg, K. (2019). Learning ambidextrous robot grasping policies. Science Robotics. https://doi.org/10.1126/scirobotics.aau4984CrossRef

Mamou, K., Lengyel, E., & Peters, A. (2016). Volumetric hierarchical approximate convex decomposition. In Game engine gems 3 (bls. 141–158). AK Peters.

Matas, J., James, S., & Davison, A. J. (2018). Sim-to-real reinforcement learning for deformable object manipulation. In Proceedings of conference on robot learning (CoRL 2018), Zürich, Switzerland.

Morrison, D., Corke, P., & Leitner, J. (2020). Learning robust, real-time, reactive robotic grasping. The International Journal of Robotics Research, 39(2–3), 183–201. https://doi.org/10.1177/0278364919859066CrossRef

Ohwovoriole, E. (1987). Kinematics and friction in grasping by robotic hands. Journal of Mechanisms, Transmissions, and Automation in Design, 109(3), 398–404. https://doi.org/10.1115/1.3258809CrossRef

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., & Antiga, L. (2019). PyTorch: An imperative style, high-performance deep learning library. In Proceeding of advances in neural information processing systems, Vancouver, Canada.

Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., & Dean, J. (2018). Efficient neural architecture search via parameter sharing. In International conference on machine learning, Stockholm, Sweden.

Pinto, L., & Gupta, A. (2016). Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In Proceedings of 2016 IEEE international conference on robotics and automation (ICRA), Stockholm, Sweden. https://doi.org/10.1109/ICRA.2016.7487517

Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019). Regularized evolution for image classifier architecture search. In The AAAI conference on artificial intelligence, Honolulu, Hawaii, USA. https://doi.org/10.1609/aaai.v33i01.33014780

Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y. L., Tan, J., Le, Q., & Kurakin, A. (2017). Large-scale evolution of image classifiers. In International conference on machine learning, Sydney, NSW, Australia.

Redelinghuys, A., Basson, A. H., & Kruger, K. (2020). A six-layer architecture for the digital twin: A manufacturing case study implementation. Journal of Intelligent Manufacturing, 31(6), 1383–1402. https://doi.org/10.1007/s10845-019-01516-6CrossRef

Redmon, J., & Angelova, A. (2015). Real-time grasp detection using convolutional neural networks. In Proceedings of 2015 IEEE international conference on robotics and automation (ICRA), Seattle, WA, USA. https://doi.org/10.1109/ICRA.2015.7139361

Ruder, S. (2016). An overview of gradient descent optimization algorithms. Computing Research Repository. https://doi.org/10.48550/arXiv.1609.04747CrossRef

Sadeghi, F., Toshev, A., Jang, E., & Levine, S. (2018). Sim2Real viewpoint invariant visual servoing by recurrent control. In Proceedings of 2018 IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, USA. https://doi.org/10.1109/CVPR.2018.00493

Song, Y., Gao, L., Li, X., & Shen, W. (2020). A novel robotic grasp detection method based on region proposal networks. Robotics Computer-Integrated Manufacturing, 65, 101963. https://doi.org/10.1016/j.rcim.2020.101963CrossRef

Tao, F., Zhang, H., Liu, A., & Nee, A. Y. (2018). Digital twin in industry: State-of-the-art. IEEE Transactions on Industrial Informatics, 15(4), 2405–2415. https://doi.org/10.1109/TII.2018.2873186CrossRef

Tateno, K., Tombari, F., Laina, I., & Navab, N. (2017). CNN-SLAM: Real-time dense monocular slam with learned depth prediction. In Proceedings of 2017 IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, USA. https://doi.org/10.1109/CVPR.2017.695

Wöhlke, G. (1992). Automatic grasp planning for multifingered robot hands. Journal of Intelligent Manufacturing, 3(5), 297–316. https://doi.org/10.1007/BF01577271CrossRef

Wohlkinger, W., Aldoma, A., Rusu, R. B., & Vincze, M. (2012). 3DNet: Large-scale object class recognition from CAD models. In Proceedings of 2012 IEEE international conference on robotics and automation, Saint Paul, MN, USA. https://doi.org/10.1109/ICRA.2012.6225116

Xu, Y., Xie, L., Zhang, X., Chen, X., Qi, G.-J., Tian, Q., & Xiong, H. (2020). PC-DARTS: Partial channel connections for memory-efficient differentiable architecture search. In International conference on learning representations, Addis Ababa, Ethiopia. https://doi.org/10.48550/arXiv.1907.05737

Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018). Learning transferable architectures for scalable image recognition. In Proceedings of 2018 IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, USA. https://doi.org/10.1109/CVPR.2018.00907

Title: A new differentiable architecture search method for optimizing convolutional neural networks in the digital twin of intelligent robotic grasping
Authors: Weifei Hu
Jinyi Shao
Qing Jiao
Chuxuan Wang
Jin Cheng
Zhenyu Liu
Jianrong Tan
Publication date: 24-06-2022
Publisher: Springer US
Published in: Journal of Intelligent Manufacturing / Issue 7/2023
Print ISSN: 0956-5515
Electronic ISSN: 1572-8145
DOI: https://doi.org/10.1007/s10845-022-01971-8

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Other articles of this Issue 7/2023

Combining human guidance and structured task execution during physical human–robot collaboration

Knowledge-embedded machine learning and its applications in smart manufacturing

Evaluation of the reliability of resistance spot welding control via on-line monitoring of dynamic resistance

Deep generative model with time series-image encoding for manufacturing fault detection in die casting process

Cascaded foreign object detection in manufacturing processes using convolutional neural networks and synthetic data generation methodology

Imbalanced fault diagnosis based on semi-supervised ensemble learning

Premium Partners