nach oben

International Journal of Computer Vision

Erschienen in:

27.02.2021

An Exploration of Embodied Visual Exploration

verfasst von: Santhosh K. Ramakrishnan, Dinesh Jayaraman, Kristen Grauman

Erschienen in: International Journal of Computer Vision | Ausgabe 5/2021

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Embodied computer vision considers perception for robots in novel, unstructured environments. Of particular importance is the embodied visual exploration problem: how might a robot equipped with a camera scope out a new environment? Despite the progress thus far, many basic questions pertinent to this problem remain unanswered: (i) What does it mean for an agent to explore its environment well? (ii) Which methods work well, and under which assumptions and environmental settings? (iii) Where do current approaches fall short, and where might future work seek to improve? Seeking answers to these questions, we first present a taxonomy for existing visual exploration algorithms and create a standard framework for benchmarking them. We then perform a thorough empirical study of the four state-of-the-art paradigms using the proposed framework with two photorealistic simulated 3D environments, a state-of-the-art exploration architecture, and diverse evaluation metrics. Our experimental results offer insights and suggest new performance metrics and baselines for future work in visual exploration. Code, models and data are publicly available.

Vorheriger Artikel Enhanced 3D Human Pose Estimation from Videos by Using Attention-Based Neural Network with Dilated Convolutions

Nächster Artikel Context-Enhanced Representation Learning for Single Image Deraining

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Nur mit Berechtigung zugänglich

See Appendix G for more details on how landmarks are mined.

We select area visited because it is a simple metric that does not require additional semantic annotations.

The NRC values may be larger than 1.0 for learned methods. This is due to domain differences between the inferred occupancy used in training and the GT occupancy used in testing.

RANSAC implementation: https://github.com/facebookresearch/exploring_exploration/blob/master/exploring_exploration/utils/pose_estimation.py

See https://github.com/s-gupta/map-plan-baseline for a well-tuned implementation.

Examples of AVD object instances: https://www.cs.unc.edu/~ammirato/active_vision_dataset_website/get_data.html

Aloimonos, J., Weiss, I., & Bandyopadhyay, A. (1988). Active vision. International Journal of Computer Vision, 1, 333–356.CrossRef

Ammirato, P., Poirson, P., Park, E., Kosecka, J. & Berg, A. (2016). A dataset for developing and benchmarking active vision. In: ICRA.

Anderson, P., Chang, A., Chaplot, D. S., Dosovitskiy, A., Gupta, S., Koltun, V., Kosecka, J., Malik, J., Mottaghi, R., Savva, M. & Zamir, A. R. (2018a). On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757.

Anderson, P., Chang, A., Chaplot, D. S., Dosovitskiy, A., Gupta, S., Koltun, V., Kosecka, J., Malik, J., Mottaghi, R. & Savva, M., et al. (2018b). On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757.

Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S. & van den Hengel, A. (2018c). Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Armeni, I., Sax, A., Zamir, A. R. & Savarese, S. (2017). Joint 2D-3D-Semantic Data for Indoor Scene Understanding. ArXiv e-prints.

Bajcsy, R. (1988). Active perception. Proceedings of the IEEE.

Ballard, D. H. (1991). Animate vision. Artificial intelligence.

Batra, D., Gokaslan, A., Kembhavi, A., Maksymets, O., Mottaghi, R., Savva, M., Toshev, A. & Wijmans, E. (2020). Objectnav revisited: On evaluation of embodied agents navigating to objects.

Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D. & Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. In: Advances in Neural Information Processing Systems.

Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016). End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316.

Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T. & Efros, A. A. (2018a). Large-scale study of curiosity-driven learning. In: arXiv:1808.04355.

Burda, Y., Edwards, H., Storkey, A. & Klimov, O. (2018b) Exploration by random network distillation. arXiv preprint arXiv:1810.12894.

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A. & Zagoruyko, S. (2020). End-to-end object detection with transformers. arXiv preprint arXiv:2005.12872.

Cassandra, A. R., Kaelbling, L. P. & Kurien, J. A. (1996). Acting under uncertainty: Discrete bayesian models for mobile-robot navigation. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS’96, vol. 2, pp. 963–972. IEEE.

Chang, A., Dai, A., Funkhouser, T., , Nießner, M., Savva, M., Song, S., Zeng, A. & Zhang, Y. (2017). Matterport3d: Learning from rgb-d data in indoor environments. In: Proceedings of the International Conference on 3D Vision (3DV). MatterPort3D dataset license available at: http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf.

Chaplot, D. S., Gandhi, D., Gupta, S., Gupta, A., & Salakhutdinov, R. (2019). Learning To Explore Using Active Neural SLAM. In International Conference on Learning Representations.

Chen, T., Gupta, S. & Gupta, A. (2019). Learning exploration policies for navigation. In: International Conference on Learning Representations. https://openreview.net/pdf?id=SyMWn05F7.

Chung, J., Gulcehre, C., Cho, K. & Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.

Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D. & Batra, D. (2018a). Embodied Question Answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Das, A., Gkioxari, G., Lee, S., Parikh, D. & Batra, D. (2018b). Neural modular control for embodied question answering. In: Conference on Robot Learning, pp. 53–62.

Devlin, J., Chang, M.W., Lee, K. & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Duan, Y., Chen, X., Houthooft, R., Schulman, J. & Abbeel, P. (2016). Benchmarking deep reinforcement learning for continuous control. In: International Conference on Machine Learning, pp. 1329–1338.

Fang, K., Toshev, A., Fei-Fei, L. & Savarese, S. (2019). Scene memory transformer for embodied agents in long-horizon tasks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 538–547.

Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395.MathSciNetCrossRef

Giusti, A., Guzzi, J., Cireşan, D. C., He, F. L., Rodríguez, J. P., Fontana, F., Faessler, M., Forster, C., Schmidhuber, J., Di Caro, G., et al. (2016). A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters.

Goyal, P., Mahajan, D., Gupta, A. & Misra, I. (2019). Scaling and benchmarking self-supervised visual representation learning. arXiv preprint arXiv:1905.01235.

Gupta, S., Davidson, J., Levine, S., Sukthankar, R. & Malik, J. (2017a). Cognitive mapping and planning for visual navigation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2616–2625.

Gupta, S., Fouhey, D., Levine, S. & Malik, J. (2017b). Unifying map and landmark based representations for visual navigation. arXiv preprint arXiv:1712.08125.

Haber, N., Mrowca, D., Fei-Fei, L. & Yamins, D. L. (2018). Learning to play with intrinsically-motivated self-aware agents. arXiv preprint arXiv:1802.07442.

Hart, P. E., Nilsson, N. J., & Raphael, B. (1968). A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2), 100–107.CrossRef

He, K., Zhang, X., Ren, S. & Sun, J. (2016). Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.

Henriques, J. F. & Vedaldi, A. (2018). Mapnet: An allocentric spatial memory for mapping environments. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8476–8484.

Isola, P., Zhu, J. Y., Zhou, T. & Efros, A. A. (2016). Image-to-image translation with conditional adversarial networks. arxiv.

Jayaraman, D., & Grauman, K. (2018a). End-to-end policy learning for active visual categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7), 1601–1614.CrossRef

Jayaraman, D. & Grauman, K. (2018b). Learning to look around: Intelligently exploring unseen environments for unknown tasks. In: Computer Vision and Pattern Recognition, 2018 IEEE Conference on.

Kadian, A., Truong, J., Gokaslan, A., Clegg, A., Wijmans, E., Lee, S., Savva, M., Chernova, S. & Batra, D. (2019). Are we making real progress in simulated environments? measuring the sim2real gap in embodied visual navigation. arXiv preprint arXiv:1912.06321.

Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017). The kinetics human action video dataset. arXiv preprint arXiv:1705.06950.

Kingma, D. P. & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Gordon, D., Zhu, Y., Gupta, A. & Farhadi, A. (2017). AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv.

Kostrikov, I. (2018). Pytorch implementations of reinforcement learning algorithms. https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail.

Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P. & Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In: European Conference on Computer Vision.

Lopes, M., Lang, T., Toussaint, M. & Oudeyer, P. Y. (2012). Exploration in model-based reinforcement learning by empirically estimating learning progress. In: Advances in neural information processing systems, pp. 206–214.

Lovejoy, W. S. (1991). A survey of algorithmic methods for partially observed markov decision processes. Annals of Operations Research, 28(1), 47–65.MathSciNetCrossRefMATH

Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W. & Bergstra, J. (2018). Benchmarking reinforcement learning algorithms on real-world robots. In: Conference on Robot Learning, pp. 561–591.

Malmir, M., Sikka, K., Forster, D., Movellan, J. & Cottrell, G. W. (2015). Deep Q-learning for active recognition of GERMS. In: BMVC.

Savva, Manolis, Kadian, Abhishek, Maksymets, Oleksandr, Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., Parikh, D. & Batra, D. (2019). Habitat: A Platform for Embodied AI Research. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).

Mishkin, D., Dosovitskiy, A. & Koltun, V. (2019). Benchmarking classic and learned navigation in complex 3d environments. arXiv preprint arXiv:1901.10915.

Narasimhan, M., Wijmans, E., Chen, X., Darrell, T., Batra, D., Parikh, D. & Singh, A. (2020). Seeing the un-scene: Learning amodal semantic maps for room navigation. arXiv preprint arXiv:2007.09841.

Ostrovski, G., Bellemare, M. G., van den Oord, A. & Munos, R. (2017). Count-based exploration with neural density models. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2721–2730. JMLR. org.

Oudeyer, P. Y., Kaplan, F., & Hafner, V. V. (2007). Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation, 11(2), 265–286.CrossRef

Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L. & Lerer, A. (2017). Automatic differentiation in PyTorch. In: NIPS Autodiff Workshop.

Pathak, D., Agrawal, P., Efros, A. A. & Darrell, T. (2017). Curiosity-driven exploration by self-supervised prediction. In: International Conference on Machine Learning.

Pathak, D., Gandhi, D. & Gupta, A. (2018). Beyond games: Bringing exploration to robots in real-world.

Pathak, D., Gandhi, D. & Gupta, A. (2019) Self-supervised exploration via disagreement. arXiv preprint arXiv:1906.04161.

Qi, W., Mullapudi, R. T., Gupta, S. & Ramanan, D. (2020) Learning to move with affordance maps. arXiv preprint arXiv:2001.02364.

Ramakrishnan, S. K. & Grauman, K. (2018). Sidekick policy learning for active visual exploration. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 413–430.

Ramakrishnan, S. K., Jayaraman, D. & Grauman, K. (2019). Emergence of exploratory look-around behaviors through active observation completion. Science Robotics 4(30). https://doi.org/10.1126/scirobotics.aaw6326. https://robotics.sciencemag.org/content/4/30/eaaw6326.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., et al. (2015). ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 211–252.MathSciNetCrossRef

Savinov, N., Dosovitskiy, A. & Koltun, V. (2018a). Semi-parametric topological memory for navigation. arXiv preprint arXiv:1803.00653.

Savinov, N., Raichuk, A., Marinier, R., Vincent, D., Pollefeys, M., Lillicrap, T. & Gelly, S. (2018b). Episodic curiosity through reachability. arXiv preprint arXiv:1810.02274.

Savva, M., Chang, A. X., Dosovitskiy, A., Funkhouser, T. & Koltun, V. (2017). Minos: Multimodal indoor simulator for navigation in complex environments. arXiv preprint arXiv:1712.03931.

Schmidhuber, J. (1991). Curious model-building control systems. In: Proc. international joint conference on neural networks, pp. 1458–1463.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

Seifi, S. & Tuytelaars, T. (2019). Where to look next: Unsupervised active visual exploration on 360 \(\{\backslash \)deg\(\}\) input. arXiv preprint arXiv:1909.10304.

Soomro, K., Zamir, A. R. & Shah, M. (2012). Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402.

Stachniss, C., Grisetti, G., & Burgard, W. (2005). Information gain-based exploration using rao black-wellized particle filters. Robotics Science and Systems, 2, 101.

Straub, J., Whelan, T., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J. J., Mur-Artal, R., Ren, C., Verma, S., Clarkson, A., Yan, M., Budge, B., Yan, Y., Pan, X., Yon, J., Zou, Y., Leon, K., Carter, N., Briales, J., Gillingham, T., Mueggler, E., Pesqueira, L., Savva, M., Batra, D., Strasdat, H. M., Nardi, R. D., Goesele, M., Lovegrove, S. & Newcombe, R. (2019). The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797.

Strehl, A. L., & Littman, M. L. (2008). An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8), 1309–1331.MathSciNetCrossRefMATH

Sun, Y., Gomez, F. & Schmidhuber, J. (2011). Planning to be surprised: Optimal bayesian exploration in dynamic environments. In: International Conference on Artificial General Intelligence, pp. 41–51. Springer.

Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. Cambridge: MIT press.MATH

Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, O. X., Duan, Y., Schulman, J., DeTurck, F. & Abbeel, P. (2017). # exploration: A study of count-based exploration for deep reinforcement learning. In: Advances in neural information processing systems, pp. 2753–2762.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. & Polosukhin, I. (2017). Attention is all you need. In: Advances in neural information processing systems, pp. 5998–6008.

Wijmans, E., Datta, S., Maksymets, O., Das, A., Gkioxari, G., Lee, S., Essa, I., Parikh, D. & Batra, D. (2019). Embodied question answering in photorealistic environments with point cloud perception. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6659–6668.

Wilkes, D. & Tsotsos, J. K. (1992). Active object recognition. In: Computer Vision and Pattern Recognition, 1992. IEEE Computer Society Conference on.

Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J. & Savarese, S. (2018). Gibson env: Real-world perception for embodied agents. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9068–9079.

Yamauchi, B. (1997). A frontier-based approach for autonomous exploration.

Yang, J., Ren, Z., Xu, M., Chen, X., Crandall, D., Parikh, D. & Batra, D. (2019a). Embodied visual recognition. arXiv preprint arXiv:1904.04404.

Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R. & Le, Q. V. (2019b). Xlnet: Generalized autoregressive pretraining for language understanding. In: Advances in neural information processing systems, pp. 5753–5763.

Zamir, A. R., Wekel, T., Agrawal, P., Wei, C., Malik, J. & Savarese, S. (2016). Generic 3D representation via pose estimation and matching. In: European Conference on Computer Vision, pp. 535–553. Springer.

Zhu, Y., Gordon, D., Kolve, E., Fox, D., Fei-Fei, L., Gupta, A., Mottaghi, R. & Farhadi, A. (2017). Visual Semantic Planning using Deep Successor Representations. In: Computer Vision, 2017 IEEE International Conference on.

Titel: An Exploration of Embodied Visual Exploration
verfasst von: Santhosh K. Ramakrishnan
Dinesh Jayaraman
Kristen Grauman
Publikationsdatum: 27.02.2021
Verlag: Springer US
Erschienen in: International Journal of Computer Vision / Ausgabe 5/2021
Print ISSN: 0920-5691
Elektronische ISSN: 1573-1405
DOI: https://doi.org/10.1007/s11263-021-01437-z

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Weitere Artikel der Ausgabe 5/2021

Context-Enhanced Representation Learning for Single Image Deraining

Evaluating Visual Properties via Robust HodgeRank

Intra-Camera Supervised Person Re-Identification

Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis

Exposing Semantic Segmentation Failures via Maximum Discrepancy Competition

EfficientPS: Efficient Panoptic Segmentation