Top

Published in:

2023 | OriginalPaper | Chapter

3. How Search Engines Capture and Process Content from the Web

Author : Dirk Lewandowski

Published in: Understanding Search Engines

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

This chapter describes the technical basis of search engines. This basis includes how the documents available on the Web are brought into the search engine and how they are made searchable, as well as how the link between a search query and the documents in the database is established. Details on the workings of the crawler, the indexer, and the searcher are given.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Ways of Searching the Web

next chapter User Interaction with Search Engines

Alpert, J., & Hajaj, N. (2008). We knew the web was big... http://googleblog.blogspot.de/2008/07/we-knew-web-was-big.html

Baeza-Yates, R., & Ribeiro-Neto, B. (2011). Modern information retrieval: The concepts and technology behind search. Addison Wesley.

Bharat, K., & Broder, A. (1998). A technique for measuring the relative size and overlap of public Web search engines. Computer Networks and ISDN Systems, 30(1–7), 379–388.CrossRef

Broder, A., Kumar, R., Maghoul, F., Raghavan, P., Rajagopalan, S., Stata, R., et al. (2000). Graph structure in the web. Computer Networks, 33(1–6), 309–320.CrossRef

Chang, Y., & Deng, H. (Eds.). (2020). Query understanding for search engines. Springer.

Croft, W. B., Metzler, D., & Strohman, T. (2009). Search engines: Information retrieval in practice. Pearson.

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL HLT 2019 – 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies – Proceedings of the Conference (pp. 4171–4186).

Google. (2017). Alles über Google News. https://web.archive.org/web/20170321083046/https://www.google.de/intl/de_de/about_google_news.html

Google Search Central. (2020). How we fought Search spam on Google – Webspam Report 2019. https://developers.google.com/search/blog/2020/06/how-we-fought-search-spam-on-google

Gulli, A., & Signorini, A. (2005). The indexable web is more than 11.5 billion pages. In 14th International Conference on World Wide Web (pp. 902–903). ACM.

Karaganis, J., & Urban, J. (2015). The rise of the robo notice. Communications of the ACM, 58(9), 28–30. https://doi.org/10.1145/2804244CrossRef

Lawrence, S., & Giles, C. L. (1999). Accessibility of information on the web. Nature, 400(8), 107–109.CrossRef

Levene, M. (2010). An introduction to search engines and web navigation. Wiley.CrossRef

Lewandowski, D. (2011). Query understanding. In D. Lewandowski (Ed.), Handbuch Internet-Suchmaschinen 2: Neue Entwicklungen in der Web-Suche (pp. 55–75). Akademische Verlagsgesellschaft AKA.

Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge University Press.CrossRefMATH

Nayak, P. (2019). Understanding searches better than ever before. https://www.blog.google/products/search/search-language-understanding-bert/

Ntoulas, A., Cho, J., & Olston, C. (2004). What’s new on the web?: The evolution of the web from a search engine perspective. In Proceedings of the 13th international conference on World Wide Web (pp. 1–12). ACM.

Risvik, K. M., & Michelsen, R. (2002). Search engines and web dynamics. Computer Networks, 39(3), 289–302.CrossRef

Sánchez, D., Martínez-Sanahuja, L., & Batet, M. (2018). Survey and evaluation of web search engine hit counts as research tools in computational linguistics. Information Systems, 73, 50–60. https://doi.org/10.1016/j.is.2017.12.007CrossRef

Schwartz, B. (2016). Google’s search knows about over 130 trillion pages. Retrieved March 2, 2017, from http://searchengineland.com/googles-search-indexes-hits-130-trillion-pages-documents-263378

Stock, W. G., & Stock, M. (2013). Handbook of information science. De Gruyter Saur.CrossRefMATH

Strzelecki, A. (2019). Website removal from search engines due to copyright violation. Aslib Journal of Information Management, 71(1), 54–71. https://doi.org/10.1108/AJIM-05-2018-0108CrossRef

Sullivan, D. (2005). End of size wars? Google says most comprehensive but drops home page count. http://searchenginewatch.com/article/2066323/End-Of-Size-Wars-Google-Says-Most-Comprehensive-But-Drops-Home-Page-Count

Tyagi, V. (2017). Content-based image retrieval: Ideas, influences, and current trends. Springer.CrossRefMATH

Uyar, A. (2009). Investigation of the accuracy of search engine hit counts. Journal of Information Science, 35(4), 469–480. https://doi.org/10.1177/0165551509103598MathSciNetCrossRef

Vaidhyanathan, S. (2011). The Googlization of everything (and why we should worry). University of California Press.CrossRef

van den Bosch, A., Bogers, T., & de Kunder, M. (2016). Estimating search engine index size variability: A 9-year longitudinal study. Scientometrics, 107(2), 839–856. https://doi.org/10.1007/s11192-016-1863-zCrossRef

Vaughan, L., & Thelwall, M. (2004). Search engine coverage bias: Evidence and possible causes. Information Processing & Management, 40, 693–707.CrossRef

Vaughan, L., & Zhang, Y. (2007). Equal representation by search engines? A comparison of websites across countries and domains. Journal of Computer-Mediated Communication, 12, 888–909.CrossRef

Vavra, A. N. (2018). The right to be forgotten: An archival perspective. The American Archivist, 81(1), 100–111. https://doi.org/10.17723/0360-9081-81.1.100CrossRef

Title: How Search Engines Capture and Process Content from the Web
Author: Dirk Lewandowski
Publisher: Springer International Publishing
Book: Understanding Search Engines
Print ISBN: 978-3-031-22788-2

Electronic ISBN: 978-3-031-22789-9

Copyright Year: 2023
DOI: https://doi.org/10.1007/978-3-031-22789-9_3

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"