Skip to main content
Top
Published in: Autonomous Robots 2/2020

03-08-2019

Attention-based active visual search for mobile robots

Authors: Amir Rasouli, Pablo Lanillos, Gordon Cheng, John K. Tsotsos

Published in: Autonomous Robots | Issue 2/2020

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

We present an active visual search model for finding objects in unknown environments. The proposed algorithm guides the robot towards the sought object using the relevant stimuli provided by the visual sensors. Existing search strategies are either purely reactive or use simplified sensor models that do not exploit all the visual information available. In this paper, we propose a new model that actively extracts visual information via visual attention techniques and, in conjunction with a non-myopic decision-making algorithm, leads the robot to search more relevant areas of the environment. The attention module couples both top-down and bottom-up attention models enabling the robot to search regions with higher importance first. The proposed algorithm is evaluated on a mobile robot platform in a 3D simulated environment. The results indicate that the use of visual attention significantly improves search, but the degree of improvement depends on the nature of the task and the complexity of the environment. In our experiments, we found that performance enhancements of up to 42% in structured and 38% in highly unstructured cluttered environments can be achieved using visual attention mechanisms.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Footnotes
1
The robot has restrictions in the movement due to its kinematics (Eagle 1984).
 
3
Although the optimization can be computed using a gradient-based approach due to the properties of the belief, with this algorithm we can also tackle some degenerate cases where non-linearities appear.
 
Literature
go back to reference Aydemir, A., Pronobis, A., Göbelbecker, M., & Jensfelt, P. (2013). Active visual object search in unknown environments using uncertain semantics. Transactions on Robotics, 29(4), 986–1002.CrossRef Aydemir, A., Pronobis, A., Göbelbecker, M., & Jensfelt, P. (2013). Active visual object search in unknown environments using uncertain semantics. Transactions on Robotics, 29(4), 986–1002.CrossRef
go back to reference Bajcsy, R. (1988). Active perception. The Proceedings of IEEE, 76(8), 966–1005.CrossRef Bajcsy, R. (1988). Active perception. The Proceedings of IEEE, 76(8), 966–1005.CrossRef
go back to reference Bajcsy, R., Aloimonos, Y., & Tsotsos, J. K. (2016). Revisiting active perception. Autonomous Robots, 42, 1–20. Bajcsy, R., Aloimonos, Y., & Tsotsos, J. K. (2016). Revisiting active perception. Autonomous Robots, 42, 1–20.
go back to reference Borji, A., Sihite, D. N., & Itti, L. (2013). Quantitative analysis of humanmodel agreement in visual saliency modeling: A comparative study. Transactions on Image Processing, 22(1), 55–69.MathSciNetCrossRef Borji, A., Sihite, D. N., & Itti, L. (2013). Quantitative analysis of humanmodel agreement in visual saliency modeling: A comparative study. Transactions on Image Processing, 22(1), 55–69.MathSciNetCrossRef
go back to reference Bourgault, F., Furukawa, T., & Durrant-Whyte, H. F. (2003). Optimal search for a lost target in a bayesian world. In FSR (pp. 209–222). Bourgault, F., Furukawa, T., & Durrant-Whyte, H. F. (2003). Optimal search for a lost target in a bayesian world. In FSR (pp. 209–222).
go back to reference Bruce, N., & Tsotsos, J. K. (2007). Attention based on information maximization. Journal of Vision, 7(9), 950.CrossRef Bruce, N., & Tsotsos, J. K. (2007). Attention based on information maximization. Journal of Vision, 7(9), 950.CrossRef
go back to reference Bruce, N. D., & Tsotsos, J. K. (2009). Saliency, attention, and visual search: An information theoretic approach. Journal of Vision, 9(3), 5.CrossRef Bruce, N. D., & Tsotsos, J. K. (2009). Saliency, attention, and visual search: An information theoretic approach. Journal of Vision, 9(3), 5.CrossRef
go back to reference Butko, N. J., Zhang, L., Cottrell, G. W., & Movellan, J. R. (2008). Visual saliency model for robot cameras. In International conference on virtual rehabilitation (pp. 2398–2403). Butko, N. J., Zhang, L., Cottrell, G. W., & Movellan, J. R. (2008). Visual saliency model for robot cameras. In International conference on virtual rehabilitation (pp. 2398–2403).
go back to reference Bylinskii, Z., DeGennaro, E., Rajalingham, R., Ruda, H., Zhang, J., & Tsotsos, J. (2015). Towards the quantitative evaluation of visual attention models. Vision Research, 116, 258–268.CrossRef Bylinskii, Z., DeGennaro, E., Rajalingham, R., Ruda, H., Zhang, J., & Tsotsos, J. (2015). Towards the quantitative evaluation of visual attention models. Vision Research, 116, 258–268.CrossRef
go back to reference Cave, R. K. (1999). The featuregate model of visual selection. Psychological Research, 62(2–3), 182–194.CrossRef Cave, R. K. (1999). The featuregate model of visual selection. Psychological Research, 62(2–3), 182–194.CrossRef
go back to reference Chang, K., Liu, T., Chen, H., & Lai, S. (2011). Fusing generic objectness and visual saliency for salient object detection. In ICCV (pp. 914–921). Chang, K., Liu, T., Chen, H., & Lai, S. (2011). Fusing generic objectness and visual saliency for salient object detection. In ICCV (pp. 914–921).
go back to reference Chen, X., & Lee, S. (2013). Visual search of an object in cluttered environments for robotic errand service. In International conference on systems, man, and cybernetics (pp. 4060–4065). Chen, X., & Lee, S. (2013). Visual search of an object in cluttered environments for robotic errand service. In International conference on systems, man, and cybernetics (pp. 4060–4065).
go back to reference Chung, T. H., & Burdick, J. W. (2007). A decision-making framework for control strategies in probabilistic search. In ICRA (pp. 4386–4393). Chung, T. H., & Burdick, J. W. (2007). A decision-making framework for control strategies in probabilistic search. In ICRA (pp. 4386–4393).
go back to reference Eagle, J. N. (1984). The optimal search for a moving target when the search path is constrained. Operations Research, 32(5), 1107–1115.MathSciNetCrossRef Eagle, J. N. (1984). The optimal search for a moving target when the search path is constrained. Operations Research, 32(5), 1107–1115.MathSciNetCrossRef
go back to reference Eckstein, M. P. (2011). Visual search: A retrospective. Journal of Vision, 11(5), 14–14.CrossRef Eckstein, M. P. (2011). Visual search: A retrospective. Journal of Vision, 11(5), 14–14.CrossRef
go back to reference Ferreira, J. F., & Dias, J. (2014). Attentional mechanisms for socially interactive robots-a survey. Autonomous Mental Development, 6(2), 110–125.CrossRef Ferreira, J. F., & Dias, J. (2014). Attentional mechanisms for socially interactive robots-a survey. Autonomous Mental Development, 6(2), 110–125.CrossRef
go back to reference Forssén, P.-E., Meger, D., Lai, K., Helmer, S., Little, J. J., & Lowe, D. G. (2008). Informed visual search: Combining attention and object recognition. In ICRA (pp. 935–942). Forssén, P.-E., Meger, D., Lai, K., Helmer, S., Little, J. J., & Lowe, D. G. (2008). Informed visual search: Combining attention and object recognition. In ICRA (pp. 935–942).
go back to reference Frintrop, S. (2006). VOCUS: A visual attention system for object detection and goal-directed search (Vol. 3899). Berlin: Springer.CrossRef Frintrop, S. (2006). VOCUS: A visual attention system for object detection and goal-directed search (Vol. 3899). Berlin: Springer.CrossRef
go back to reference Gan, S. K., & Sukkarieh, S. (2010). Multi-UAV target search using explicit decentralized gradient-based negotiation. In ICRA (pp. 751–756). Gan, S. K., & Sukkarieh, S. (2010). Multi-UAV target search using explicit decentralized gradient-based negotiation. In ICRA (pp. 751–756).
go back to reference Goferman, S., Zelnik-Manor, L., & Tal, A. (2012). Context-aware saliency detection. PAMI, 34(10), 1915–1926.CrossRef Goferman, S., Zelnik-Manor, L., & Tal, A. (2012). Context-aware saliency detection. PAMI, 34(10), 1915–1926.CrossRef
go back to reference He, S., Lau, R., & Yang, Q. (2016). Exemplar-driven top-down saliency detection via deep association. In CVPR (pp. 5723–5732). He, S., Lau, R., & Yang, Q. (2016). Exemplar-driven top-down saliency detection via deep association. In CVPR (pp. 5723–5732).
go back to reference Hou, X., & Zhang, L. (2008). Dynamic visual attention: Searching for coding length increments. In NIPS (pp. 681–688). Hou, X., & Zhang, L. (2008). Dynamic visual attention: Searching for coding length increments. In NIPS (pp. 681–688).
go back to reference Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visua attention for rapid scene analysis. PAMI, 20(11), 1254–1259.CrossRef Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visua attention for rapid scene analysis. PAMI, 20(11), 1254–1259.CrossRef
go back to reference Jiang, H., Wang, J., Yuan, Z., Liu, T., Zheng, N., & Li, S. (2011). Automatic salient object segmentation based on context and shape prior. BMVC, 110(1–110), 12. Jiang, H., Wang, J., Yuan, Z., Liu, T., Zheng, N., & Li, S. (2011). Automatic salient object segmentation based on context and shape prior. BMVC, 110(1–110), 12.
go back to reference Kaboli, M., Feng, D., Yao, K., Lanillos, P., & Cheng, G. (2017). A tactile-based framework for active object learning and discrimination using multimodal robotic skin. Robotics and Automation Letters, 2(4), 2143–2150.CrossRef Kaboli, M., Feng, D., Yao, K., Lanillos, P., & Cheng, G. (2017). A tactile-based framework for active object learning and discrimination using multimodal robotic skin. Robotics and Automation Letters, 2(4), 2143–2150.CrossRef
go back to reference Kim, A., & Eustice, R. M. (2013). Real-time visual slam for autonomous underwater hull inspection using visual saliency. Transactions on Robotics, 29(3), 719–733.CrossRef Kim, A., & Eustice, R. M. (2013). Real-time visual slam for autonomous underwater hull inspection using visual saliency. Transactions on Robotics, 29(3), 719–733.CrossRef
go back to reference Koenig, N., & Howard, A. (2004). Design and use paradigms for gazebo, an open-source multi-robot simulator. In IROS (pp. 2149–2154). Koenig, N., & Howard, A. (2004). Design and use paradigms for gazebo, an open-source multi-robot simulator. In IROS (pp. 2149–2154).
go back to reference Kragic, D., Björkman, M., Christensen, H. I., & Eklundh, J.-O. (2005). Vision for robotic object manipulation in domestic settings. Robotics and Autonomous Systems, 52(1), 85–100.CrossRef Kragic, D., Björkman, M., Christensen, H. I., & Eklundh, J.-O. (2005). Vision for robotic object manipulation in domestic settings. Robotics and Autonomous Systems, 52(1), 85–100.CrossRef
go back to reference Langli, D., Chartier, S., & Gosselin, D. (2010). An introduction to independent component analysis: Infomax and fastica algorithms. Tutorials in Quantitative Methods for Psychology, 6(1), 31–38.CrossRef Langli, D., Chartier, S., & Gosselin, D. (2010). An introduction to independent component analysis: Infomax and fastica algorithms. Tutorials in Quantitative Methods for Psychology, 6(1), 31–38.CrossRef
go back to reference Lanillos, P. (2013). Minimum time search of moving targets in uncertain environments. PhD thesis. Lanillos, P. (2013). Minimum time search of moving targets in uncertain environments. PhD thesis.
go back to reference Lanillos, P., Besada-Portas, E., Lopez-Orozco, J. A., & de la Cruz, J. M. (2014a). Minimum time search in uncertain dynamic domains with complex sensorial platforms. Sensors, 14(8), 14131–14179.CrossRef Lanillos, P., Besada-Portas, E., Lopez-Orozco, J. A., & de la Cruz, J. M. (2014a). Minimum time search in uncertain dynamic domains with complex sensorial platforms. Sensors, 14(8), 14131–14179.CrossRef
go back to reference Lanillos, P., Ferreira, J. F., & Dias, J. (2015). Designing an artificial attention system for social robots. In IROS (pp. 4171–4178). Lanillos, P., Ferreira, J. F., & Dias, J. (2015). Designing an artificial attention system for social robots. In IROS (pp. 4171–4178).
go back to reference Lanillos, P., Gan, S. K., Besada-Portas, E., Pajares, G., & Sukkarieh, S. (2014b). Multi-uav target search using decentralized gradient-based negotiation with expected observation. Information Sciences, 282, 92–110.MathSciNetCrossRef Lanillos, P., Gan, S. K., Besada-Portas, E., Pajares, G., & Sukkarieh, S. (2014b). Multi-uav target search using decentralized gradient-based negotiation with expected observation. Information Sciences, 282, 92–110.MathSciNetCrossRef
go back to reference Li, G., & Yu, Y. (2015). Visual saliency based on multiscale deep features. In CVPR (pp. 5455–5463). Li, G., & Yu, Y. (2015). Visual saliency based on multiscale deep features. In CVPR (pp. 5455–5463).
go back to reference Moren, J., UDE, A., Koene, A., & Cheng, G. (2008). Biologically based top-down attention modulation for humanoid interactions. International Journal of Humanoid Robotics, 5(1), 3–24.CrossRef Moren, J., UDE, A., Koene, A., & Cheng, G. (2008). Biologically based top-down attention modulation for humanoid interactions. International Journal of Humanoid Robotics, 5(1), 3–24.CrossRef
go back to reference Orabona, F., Metta, G., & Sandini, G. (2005). Object-based visual attention: A model for a behaving robot. In CVPRW (pp. 89–89). Orabona, F., Metta, G., & Sandini, G. (2005). Object-based visual attention: A model for a behaving robot. In CVPRW (pp. 89–89).
go back to reference Rasouli, A., & Tsotsos, J.K. (2014a). Attention in autonomous robotic visual search. In i-SAIRAS. Rasouli, A., & Tsotsos, J.K. (2014a). Attention in autonomous robotic visual search. In i-SAIRAS.
go back to reference Rasouli, A., & Tsotsos, J. K. (2014b). Visual saliency improves autonomous visual search. In The conference on computer and robotic vision (CRV) (pp. 111–118). Rasouli, A., & Tsotsos, J. K. (2014b). Visual saliency improves autonomous visual search. In The conference on computer and robotic vision (CRV) (pp. 111–118).
go back to reference Rasouli, A., & Tsotsos, J. K. (2017). The effect of color space selection on detectability and discriminability of colored objects. arXiv:1702.05421. Rasouli, A., & Tsotsos, J. K. (2017). The effect of color space selection on detectability and discriminability of colored objects. arXiv:​1702.​05421.
go back to reference Roberts, R., Ta, D.N., Straub, J., Ok, K., & Dellaert, F. (2012). Saliency detection and model-based tracking: A two part vision system for small robot navigation in forested environments. In SPIE. Roberts, R., Ta, D.N., Straub, J., Ok, K., & Dellaert, F. (2012). Saliency detection and model-based tracking: A two part vision system for small robot navigation in forested environments. In SPIE.
go back to reference Shon, A. P., Grimes, D. B., Baker, C. L., Hoffman, M., Zhou, S., & Rao, R. P. N. (2005). Probabilistic gaze imitation and saliency learning in a robotic head. In ICRA (pp. 2865–2870). Shon, A. P., Grimes, D. B., Baker, C. L., Hoffman, M., Zhou, S., & Rao, R. P. N. (2005). Probabilistic gaze imitation and saliency learning in a robotic head. In ICRA (pp. 2865–2870).
go back to reference Shubina, K., & Tsotsos, J. K. (2010). Visual search for an object in a 3d environment using a mobile robot. CVIU, 114(5), 535–547. Shubina, K., & Tsotsos, J. K. (2010). Visual search for an object in a 3d environment using a mobile robot. CVIU, 114(5), 535–547.
go back to reference Sjöö, K., Aydemir, A., & Jensfelt, P. (2012). Topological spatial relations for active visual search. Robotics and Autonomous Systems, 60(9), 1093–1107.CrossRef Sjöö, K., Aydemir, A., & Jensfelt, P. (2012). Topological spatial relations for active visual search. Robotics and Autonomous Systems, 60(9), 1093–1107.CrossRef
go back to reference Stone, L. D. (1975). Theory of optimal search. New York: Academic Press.MATH Stone, L. D. (1975). Theory of optimal search. New York: Academic Press.MATH
go back to reference Swain, M. J., & Ballard, D. H. (1991). Color indexing. IJCV, 7(1), 11–32.CrossRef Swain, M. J., & Ballard, D. H. (1991). Color indexing. IJCV, 7(1), 11–32.CrossRef
go back to reference Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136.CrossRef Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136.CrossRef
go back to reference Trummel, K. E., & Weisinger, J. R. (1986). The complexity of the optimal searcher path problem. Operations Research, 34(2), 324–327.MathSciNetCrossRef Trummel, K. E., & Weisinger, J. R. (1986). The complexity of the optimal searcher path problem. Operations Research, 34(2), 324–327.MathSciNetCrossRef
go back to reference Tseng, K.-S., & Mettler, B. (2015). Near-optimal probabilistic search via submodularity and sparse regression. Autonomous Robots, 41, 1–25. Tseng, K.-S., & Mettler, B. (2015). Near-optimal probabilistic search via submodularity and sparse regression. Autonomous Robots, 41, 1–25.
go back to reference Tsotsos, J. K. (1989). The complexity of perceptual search tasks. IJCAI, 89, 1571–1577.MATH Tsotsos, J. K. (1989). The complexity of perceptual search tasks. IJCAI, 89, 1571–1577.MATH
go back to reference Tsotsos, J. K. (1990). Analyzing vision at the complexity level. Behavioral and Brain Sciences, 13(03), 423–445.CrossRef Tsotsos, J. K. (1990). Analyzing vision at the complexity level. Behavioral and Brain Sciences, 13(03), 423–445.CrossRef
go back to reference Tsotsos, J. K. (1992). On the relative complexity of active vs. passive visual search. IJCV, 7(2), 127–141.CrossRef Tsotsos, J. K. (1992). On the relative complexity of active vs. passive visual search. IJCV, 7(2), 127–141.CrossRef
go back to reference Ude, A., Wyart, V., Lin, L.H., & Cheng, G. (2005). Distributed visual attention on a humanoid robot. In International conference on humanoid robots, IEEE (pp. 381–386). Ude, A., Wyart, V., Lin, L.H., & Cheng, G. (2005). Distributed visual attention on a humanoid robot. In International conference on humanoid robots, IEEE (pp. 381–386).
go back to reference Vergassola, M., Villermaux, E., & Shraiman, B. I. (2007). ‘infotaxis’ as a strategy for searching without gradients. Nature, 445(7126), 406–409.CrossRef Vergassola, M., Villermaux, E., & Shraiman, B. I. (2007). ‘infotaxis’ as a strategy for searching without gradients. Nature, 445(7126), 406–409.CrossRef
go back to reference Wolfe, J. M. (2007). Guided search 4.0. Integrated Models of Cognitive Systems, 15, 99–119.CrossRef Wolfe, J. M. (2007). Guided search 4.0. Integrated Models of Cognitive Systems, 15, 99–119.CrossRef
go back to reference Xu, T., Kuhnlenz, K., & Buss, M. (2010). Autonomous behavior-based switched top-down and bottom-up visual attention for mobile robots. Transactions on Robotics, 26(5), 947–954.CrossRef Xu, T., Kuhnlenz, K., & Buss, M. (2010). Autonomous behavior-based switched top-down and bottom-up visual attention for mobile robots. Transactions on Robotics, 26(5), 947–954.CrossRef
go back to reference Yang, J., & Yangm, M. (2012). Top-down visual saliency via joint crf and dictionary learning. In CVPR (pp. 576–588). Yang, J., & Yangm, M. (2012). Top-down visual saliency via joint crf and dictionary learning. In CVPR (pp. 576–588).
go back to reference Ye, Y., & Tsotsos, J. K. (1999). Sensor planning for 3d object search. CVIU, 73(2), 145–168. Ye, Y., & Tsotsos, J. K. (1999). Sensor planning for 3d object search. CVIU, 73(2), 145–168.
go back to reference Ye, Y., & Tsotsos, J. K. (2001). A complexity-level analysis of the sensor planning task for object search. Computational Intelligence, 17(4), 605–620.MathSciNetCrossRef Ye, Y., & Tsotsos, J. K. (2001). A complexity-level analysis of the sensor planning task for object search. Computational Intelligence, 17(4), 605–620.MathSciNetCrossRef
go back to reference Zhao, R., Ouyang, W., Li, H., & Wang, X. (2015). Saliency detection by multi-context deep learning. In CVPR (pp. 1265–1274). Zhao, R., Ouyang, W., Li, H., & Wang, X. (2015). Saliency detection by multi-context deep learning. In CVPR (pp. 1265–1274).
go back to reference Zhu, J., Qiu, Y., Zhang, R., & Huang, J. (2014). Top-down saliency detection via contextual pooling. Journal of Signal Processing Systems, 74(1), 33–46.CrossRef Zhu, J., Qiu, Y., Zhang, R., & Huang, J. (2014). Top-down saliency detection via contextual pooling. Journal of Signal Processing Systems, 74(1), 33–46.CrossRef
Metadata
Title
Attention-based active visual search for mobile robots
Authors
Amir Rasouli
Pablo Lanillos
Gordon Cheng
John K. Tsotsos
Publication date
03-08-2019
Publisher
Springer US
Published in
Autonomous Robots / Issue 2/2020
Print ISSN: 0929-5593
Electronic ISSN: 1573-7527
DOI
https://doi.org/10.1007/s10514-019-09882-z

Other articles of this Issue 2/2020

Autonomous Robots 2/2020 Go to the issue