Top

Published in:

2025 | OriginalPaper | Chapter

Advancing Robotic Perception with Perceived-Entity Linking

Authors : Mark Adamik, Romana Pernisch, Ilaria Tiddi, Stefan Schlobach

Published in: The Semantic Web – ISWC 2024

Publisher: Springer Nature Switzerland

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The capabilities of current robotic applications are significantly constrained by their limited ability to perceive and understand their surroundings. The Semantic Web aims to offer general, machine-readable knowledge about the world and could be a potential solution to address the information needs of robotic agents. We introduce the Perceived-Entity Linking (PEL) problem as the task of recognizing entities and linking the sensory data of an autonomous agent to a unique identifier in a target knowledge graph. We provide a formal definition of PEL, and propose a PEL baseline based on the YOLO object detection algorithm and a conventional entity linking method as an initial attempt to solve the task. The baseline is evaluated by linking the concepts contained in MS COCO and VisualGenome datasets to WikiData, DBpedia and YAGO as target knowledge graphs. This study makes a first step in allowing robotic agents to leverage the extensive knowledge contained in general-purpose knowledge graphs.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Understanding SPARQL Queries: Are We Already There? Multilingual Natural Language Generation Based on SPARQL Queries and Large Language Models

next chapter PreAdapter: Pre-training Language Models on Knowledge Graphs

The results of the experiment, as well as the implementation of the architecture can be openly accessed online https://github.com/Dorteel/pel.

http://spotlight.dbpedia.org/.

http://www.mpi-inf.mpg.de/yago-naga/aida/.

Aguado, E., Sanz, R.: Using ontologies in autonomous robots engineering. Robot. Softw. Des. Eng. 71 (2021)

Alarcos, A.O., et al.: A review and comparison of ontology-based approaches to robot autonomy. Knowl. Eng. Rev. 34, e29 (2019). https://doi.org/10.1017/S0269888919000237

Bálint-Benczédi, F., et al.: Robosherlock: cognition-enabled robot perception for everyday manipulation tasks. CoRR abs/1911.10079 (2019). http://arxiv.org/abs/1911.10079

Balog, K.: Entity-Oriented Search. The Information Retrieval Series, vol. 39. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93935-3

Beetz, M., Balint-Benczedi, F., Blodow, N., Nyga, D., Wiedemeyer, T., Marton, Z.: Robosherlock: unstructured information processing for robot perception. In: IEEE International Conference on Robotics and Automation. ICRA 2015, Seattle, WA, USA, May 26–30 2015, pp. 1549–1556. IEEE (2015). https://doi.org/10.1109/ICRA.2015.7139395

Beetz, M., Beßler, D., Haidu, A., Pomarlan, M., Bozcuoglu, A.K., Bartels, G.: Know rob 2.0 - a 2nd generation knowledge processing framework for cognition-enabled robotic agents. In: 2018 IEEE International Conference on Robotics and Automation. ICRA 2018, Brisbane, Australia, 21–25 May 2018, pp. 512–519. IEEE (2018). https://doi.org/10.1109/ICRA.2018.8460964

Berners-Lee, T., Hendler, J.A., Lassila, O.: The semantic web: a new form of web content that is meaningful to computers will unleash a revolution of new possibilities. In: Seneviratne, O., Hendler, J.A. (eds.) Linking the World’s Information - Essays on Tim Berners-Lee’s Invention of the World Wide Web, ACM Books, vol. 52, pp. 91–103. ACM (2023). https://doi.org/10.1145/3591366.3591376

Besl, P.J., Jain, R.C.: Three-dimensional object recognition. ACM Comput. Surv. 17(1), 75–145 (1985). https://doi.org/10.1145/4078.4081

Bochkovskiy, A., Wang, C., Liao, H.M.: YOLOv4: optimal speed and accuracy of object detection. CoRR abs/2004.10934 (2020). https://arxiv.org/abs/2004.10934

10.

Brachman, R.J., Levesque, H.J.: Toward a new science of common sense. In: Thirty-Sixth AAAI Conference on Artificial Intelligence. AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelfth Symposium on Educational Advances in Artificial Intelligence. EAAI 2022 Virtual Event, 22 February–1 March 2022, pp. 12245–12249. AAAI Press (2022). https://doi.org/10.1609/AAAI.V36I11.21485

11.

Daoutis, M., Coradeschi, S., Loutfi, A.: Grounding commonsense knowledge in intelligent systems. J. Ambient Intell. Smart Environ. 1(4), 311–321 (2009). https://doi.org/10.3233/AIS-2009-0040

12.

Dredze, M., McNamee, P., Rao, D., Gerber, A., Finin, T.: Entity disambiguation for knowledge base population. In: Huang, C., Jurafsky, D. (eds.) COLING 2010, 23rd International Conference on Computational Linguistics, Proceedings of the Conference, 23–27 August 2010, Beijing, China, pp. 277–285. Tsinghua University Press (2010). https://aclanthology.org/C10-1032/

13.

Ehrig, M.: Ontology Alignment: Bridging the Semantic Gap, Semantic Web and Beyond: Computing for Human Experience, vol. 4. Springer, Heidelberg (2007). https://doi.org/10.1007/978-0-387-36501-5

14.

Ehrlinger, L., Wöß, W.: Towards a definition of knowledge graphs. In: Martin, M., Cuquet, M., Folmer, E. (eds.) Joint Proceedings of the Posters and Demos Track of the 12th International Conference on Semantic Systems - SEMANTiCS2016 and the 1st International Workshop on Semantic Change and Evolving Semantics (SuCCESS’16) Co-located with the 12th International Conference on Semantic Systems (SEMANTiCS 2016), Leipzig, Germany, 12–15 September 2016. CEUR Workshop Proceedings, vol. 1695. CEUR-WS.org (2016). https://ceur-ws.org/Vol-1695/paper4.pdf

15.

Fischer, L., et al.: Which tool to use? Grounded reasoning in everyday environments with assistant robots. In: Steinbauer, G., Ferrein, A. (eds.) Proceedings of the 11th Cognitive Robotics Workshop 2018, co-located with 16th International Conference on Principles of Knowledge Representation and Reasoning, CogRob@KR 2018, Tempe, AZ, USA, 27 October 2018. CEUR Workshop Proceedings, vol. 2325, pp. 3–10. CEUR-WS.org (2018). https://ceur-ws.org/Vol-2325/paper-03.pdf

16.

Gan, J., Luo, J., Wang, H., Wang, S., He, W., Huang, Q.: Multimodal entity linking: a new dataset and a baseline. In: Shen, H.T., et al. (eds.) MM ’21: ACM Multimedia Conference, Virtual Event, China, 2–24 October 2021, pp. 993–1001. ACM (2021). https://doi.org/10.1145/3474085.3475400

17.

Harnad, S.: Symbol grounding problem. Scholarpedia 2(7), 2373 (2007). https://doi.org/10.4249/SCHOLARPEDIA.2373

18.

Hogan, A., et al.: Knowledge graphs. ACM Comput. Surv. 54(4), 71:1–71:37 (2022). https://doi.org/10.1145/3447772

19.

Krishna, R., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. CoRR abs/1602.07332 (2016). http://arxiv.org/abs/1602.07332

20.

Li, Y., Ouyang, W., Zhou, B., Wang, K., Wang, X.: Scene graph generation from objects, phrases and region captions. In: IEEE International Conference on Computer Vision. ICCV 2017, Venice, Italy, 22–29 October 2017, pp. 1270–1279. IEEE Computer Society (2017). https://doi.org/10.1109/ICCV.2017.142

21.

Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48CrossRef

22.

Mendes, P.N., Jakob, M., García-Silva, A., Bizer, C.: DBPedia spotlight: shedding light on the web of documents. In: Ghidini, C., Ngomo, A.N., Lindstaedt, S.N., Pellegrini, T. (eds.) Proceedings the 7th International Conference on Semantic Systems. I-SEMANTICS 2011, Graz, Austria, 7–9 September 2011, pp. 1–8. ACM International Conference Proceeding Series. ACM (2011). https://doi.org/10.1145/2063518.2063519

23.

Miller, G.A.: Wordnet: a lexical database for English. Commun. ACM 38(11), 39–41 (1995). https://doi.org/10.1145/219717.219748

24.

Mozer, M.: Object recognition: theories. In: Smelser, N.J., Baltes, P.B. (eds.) International Encyclopedia of the Social and Behavioral Sciences, pp. 10781–10785. Pergamon, Oxford (2001). https://doi.org/10.1016/B0-08-043076-7/01459-5, https://www.sciencedirect.com/science/article/pii/B0080430767014595

25.

Rossetto, L., Baumgartner, M., Ashena, N., Ruosch, F., Pernischová, R., Bernstein, A.: Lifegraph: a knowledge graph for lifelogs. In: Gurrin, C., et al. (eds.) Proceedings of the Third ACM Workshop on Lifelog Search Challenge, LSC@ICMR 2020, Dublin, Ireland, 8–11 June 2020, pp. 13–17. ACM (2020). https://doi.org/10.1145/3379172.3391717

26.

Shen, W., Wang, J., Han, J.: Entity linking with a knowledge base: issues, techniques, and solutions. IEEE Trans. Knowl. Data Eng. 27(2), 443–460 (2015). https://doi.org/10.1109/TKDE.2014.2327028

27.

Stanton, C., Williams, M.-A.: Grounding robot sensory and symbolic information using the semantic web. In: Polani, D., Browning, B., Bonarini, A., Yoshida, K. (eds.) RoboCup 2003. LNCS (LNAI), vol. 3020, pp. 757–764. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-25940-4_75CrossRef

28.

Pellissier Tanon, T., Weikum, G., Suchanek, F.: YAGO 4: a reason-able knowledge base. In: Harth, A., et al. (eds.) ESWC 2020. LNCS, vol. 12123, pp. 583–596. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49461-2_34CrossRef

29.

Tenorth, M., Beetz, M.: Knowrob: a knowledge processing infrastructure for cognition-enabled robots. Int. J. Robot. Res. 32(5), 566–590 (2013). https://doi.org/10.1177/0278364913481635,

30.

Tenorth, M., Perzylo, A.C., Lafrenz, R., Beetz, M.: Representation and exchange of knowledge about actions, objects, and environments in the roboearth framework. IEEE Trans Autom. Sci. Eng. 10(3), 643–651 (2013). https://doi.org/10.1109/TASE.2013.2244883

31.

Vrandecic, D., Krötzsch, M.: Wikidata: a free collaborative knowledgebase. Commun. ACM 57(10), 78–85 (2014). https://doi.org/10.1145/2629489

32.

Waibel, M., et al.: Roboearth. IEEE Robot. Autom. Mag. 18(2), 69–82 (2011). https://doi.org/10.1109/MRA.2011.941632

33.

Wu, Z., Xu, Y., Yang, Y., Zhang, C., Zhu, X., Ji, Y.: Towards a semantic web of things: a hybrid semantic annotation, extraction, and reasoning framework for cyber-physical system. Sensors 17(2), 403 (2017). https://doi.org/10.3390/S17020403

34.

Yosef, M.A., Hoffart, J., Bordino, I., Spaniol, M., Weikum, G.: AIDA: an online tool for accurate disambiguation of named entities in text and tables. Proc. VLDB Endow. 4(12), 1450–1453 (2011). http://www.vldb.org/pvldb/vol4/p1450-yosef.pdf

35.

Young, J., Basile, V., Kunze, L., Cabrio, E., Hawes, N.: Towards lifelong object learning by integrating situated robot perception and semantic web mining. In: Kaminka, G.A., et al. (eds.) ECAI 2016 - 22nd European Conference on Artificial Intelligence, 29 August–2 September 2016, The Hague, The Netherlands - Including Prestigious Applications of Artificial Intelligence (PAIS 2016). Frontiers in Artificial Intelligence and Applications, vol. 285, pp. 1458–1466. IOS Press (2016). https://doi.org/10.3233/978-1-61499-672-9-1458

36.

Young, J., Kunze, L., Basile, V., Cabrio, E., Hawes, N., Caputo, B.: Semantic web-mining and deep vision for lifelong object discovery. In: 2017 IEEE International Conference on Robotics and Automation. ICRA 2017, Singapore, Singapore, 29 May–3 June 2017, pp. 2774–2779. IEEE (2017). https://doi.org/10.1109/ICRA.2017.7989323

37.

Zheng, Q., Wen, H., Wang, M., Qi, G.: Visual entity linking via multi-modal learning. Data Intell. 4(1), 1–19 (2022). https://doi.org/10.1162/DINT_A_00114

38.

Zhu, G., et al.: Scene graph generation: a comprehensive survey. CoRR abs/2201.00443 (2022). https://arxiv.org/abs/2201.00443

Title: Advancing Robotic Perception with Perceived-Entity Linking
Authors: Mark Adamik
Romana Pernisch
Ilaria Tiddi
Stefan Schlobach
Publisher: Springer Nature Switzerland
Book: The Semantic Web – ISWC 2024
Print ISBN: 978-3-031-77849-0

Electronic ISBN: 978-3-031-77850-6

Copyright Year: 2025
DOI: https://doi.org/10.1007/978-3-031-77850-6_11

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"