Skip to main content
SpringerLink
Log in

In the wild image retrieval and clustering for 3D cultural heritage landmarks reconstruction

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

One of the main characteristics of Internet era is the free and online availability of extremely large collections of images located on distributed and heterogeneous platforms over the web. The proliferation of millions of shared photographs spurred the emergence of new image retrieval techniques based not only on images’ visual information, but on geo-location tags and camera exif data. These huge visual collections provide a unique opportunity for cultural heritage documentation and 3D reconstruction. The main difficulty, however, is that the internet image datasets are unstructured containing many outliers. For this reason, in this paper a new content-based image filtering is proposed to discard image outliers that either confuse or significantly delay the followed e-documentation tools, such as 3D reconstruction of a cultural heritage object. The presented approach exploits and fuses two unsupervised clustering techniques: DBSCAN and spectral clustering. DBSCAN algorithm is used to remove outliers from the initially retrieved dataset and spectral clustering discriminate the noise free image dataset into different categories each representing characteristic geometric views of cultural heritage objects. To discard the image outliers, we consider images as points onto a multi-dimensional manifold and the multi-dimensional scaling algorithm is adopted to relate the space of the image distances with the space of Gram matrices through which we are able to compute the image coordinates. Finally, structure from motion is utilized for 3D reconstruction of cultural heritage landmarks. Evaluation on a dataset of about 31,000 cultural heritage images being retrieved from internet collections with many outliers indicate the robustness and cost effectiveness of the proposed method towards a reliable and just-in-time 3D reconstruction than existing state-of-the-art techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. http://royal.pingdom.com/2012/01/17/internet-2011-in-numbers/

  2. http://www.4d-ch-world.eu

References

  1. Agarwal S, Snavely N, Simon I, Seitz SM, Szeliski R (2009) “Building Rome in a day,” in 2009 IEEE 12th International Conference on Computer Vision, pp. 72–79

  2. Arampatzis A, Zagoris K, Chatzichristofis SA (2013) Dynamic two-stage image retrieval from large multimedia databases. Inf Process Manag 49(1):274–285

    Article  Google Scholar 

  3. Bach FR, Jordan MI (2003) “Learning Spectral Clustering”, Computer Science Division. University of California at Berkeley, California, Berkeley

    Google Scholar 

  4. Barone S, Paoli A, Razionale AV (2012) “3D virtual reconstructions of artworks by a multiview scanning process,” in 2012 18th International Conference on Virtual Systems and Multimedia (VSMM), pp. 259–265

  5. Bay H, Tuytelaars T, Gool LV (2006) SURF: Speeded Up Robust Features. In: Leonardis A, Bischof H, Pinz A (eds) Computer Vision – ECCV 2006. Springer, Berlin, pp 404–417

    Chapter  Google Scholar 

  6. Bunsch E, Guzowska A, Sitnik R (2012) “3D scanning documentation of two different objects - The King’s Chinese Cabinet in Wilanow Palace Museum and a Roman gravestone from archeological excavations in Moesia Inferior as a part of multidisciplinary research,” in 2012 18th International Conference on Virtual Systems and Multimedia (VSMM), pp. 633–636

  7. Calonder M, Lepetit V, Strecha C, Fua P (2010) BRIEF: Binary Robust Independent Elementary Features. In: Daniilidis K, Maragos P, Paragios N (eds) Computer Vision – ECCV 2010. Springer, Berlin, pp 778–792

    Chapter  Google Scholar 

  8. Cayton L (2006) “Algorithms for manifold learning,” University of California, San Diego, Tech. Rep. CS2008-0923

  9. Chum O, Philbin J, Sivic J, Isard M, Zisserman A (2007) “Total Recall: Automatic Query Expansion with a Generative FeatureModel for Object Retrieval,” in IEEE 11th International Conference on Computer Vision, 2007. ICCV 2007, pp. 1–8

  10. Cox T, Cox M, Cox T (2000) Multidimensional Scaling, Second Edition. {Chapman & Hall/CRC}

  11. Doulamis A, Doulamis N (2004) Generalized nonlinear relevance feedback for interactive content-based retrieval and organization. IEEE Trans Circ Syst Video Technol 14(5):656–671

    Article  MATH  Google Scholar 

  12. Doulamis N, Doulamis A (2006) Evaluation of relevance feedback schemes in content-based in retrieval systems. Signal Process Image Commun 21(4):334–357

    Article  MATH  Google Scholar 

  13. Doulamis AD, Doulamis ND, Kollias SD (2000) A fuzzy video content representation for video summarization and content-based retrieval. Signal Process 80(6):1049–1067

    Article  MATH  Google Scholar 

  14. Doulamis N, Doulamis A, Varvarigou TA (2003) Adaptive algorithms for interactive multimedia. IEEE Multimed 10(4):38–47

    Article  Google Scholar 

  15. Ester M, Kriegel H, JS, Xu X (1996) “A density-based algorithm for discovering clusters in large spatial databases with noise,” pp. 226–231

  16. Fan K (1951) Maximum Properties and Inequalities for the Eigenvalues of Completely Continuous Operators. Proc Natl Acad Sci U S A 37(11):760–766

    Article  MathSciNet  MATH  Google Scholar 

  17. Halkos D, Doulamis N, Doulamis A (2009) A secure framework exploiting content guided and automated algorithms for real time video searching. Multimed Tools Appl 42(3):343–375

    Article  Google Scholar 

  18. IoannidesM, Hadjiprocopis A, Doulamis N, Doulamis A, Protopapadakis E,Makantasis K, Santos P, Fellner D, Stork A, Balet O, Julien M, Weinlinger G, Johnson PS, Klein M, Fritsch D (2013) “Online 4D Reconstruction Using Multi-Images Available Under Open Acess,” ISPRS Ann Photogramm Remote Sens Spat Inf Sci, vol. II–5/W1, pp. 169–174

  19. Karaszewski M, Sitnik R, Bunsch E (2012) On-line, collision-free positioning of a scanner during fully automated three-dimensional measurement of cultural heritage objects. Robot Auton Syst 60(9):1205–1219

    Article  Google Scholar 

  20. Kekre DHB, Sarode TK, Thepade SD, Vaishali V (2011) Improved texture feature based image retrieval using Kekre’s fast codebook generation algorithm. In: Pise SJ (ed) Thinkquest~2010. Springer, India, pp 143–149

    Chapter  Google Scholar 

  21. Kosmopoulos DI, Doulamis A, Makris A, Doulamis N, Chatzis S, Middleton SE (2009) Vision-based production of personalized video. Signal Process Image Commun 24(3):158–176

    Article  Google Scholar 

  22. Lowe DG (2004) Distinctive Image Features from Scale-Invariant Keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  23. Lv Q, Josephson W, Wang Z, Charikar M, Li K (2007) “Multi-probe LSH: Efficient Indexing for High-dimensional Similarity Search”, in Proceedings of the 33rd International Conference on Very Large Data Bases. Austria, Vienna, pp 950–961

  24. Min R, Cheng HD (2009) Effective image retrieval using dominant color descriptor and fuzzy support vector machine. Pattern Recognit 42(1):147–157

    Article  MATH  Google Scholar 

  25. Murthy VSVS, Kumar S, Rao PS (2010) “Content Based Image Retrieval using Hierarchical and K-Means Clustering Techniques,” Int. J. Eng. Sci. Technol., vol. 2

  26. Ntalianis KS, Doulamis AD, Tsapatsoulis N, Doulamis N (2010) Human action annotation, modeling and analysis based on implicit user interaction. Multimed Tools Appl 50(1):199–225

    Article  Google Scholar 

  27. Papadakis N, Doulamis A, Litke A, Doulamis N, Skoutas D, Varvarigou T (2008) MI-MERCURY:Amobile agent architecture for ubiquitous retrieval and delivery of multimedia information. Multimed Tools Appl 38(1):147–184

    Article  Google Scholar 

  28. Papadopoulos S, Zigkolis C, Kompatsiaris Y, Vakali A (2010) “Cluster-based Landmark and Event Detection on Tagged Photo Collections,” IEEE Multimed

  29. Philbin J, Chum O, Isard M, Sivic J, Zisserman A (2007) “Object retrieval with large vocabularies and fast spatialmatching,” in IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR ’07, pp. 1–8

  30. Rosin PL (1999) Measuring Corner Properties. Comput Vis Image Underst 73(2):291–307

    Article  Google Scholar 

  31. Rosten E, Drummond T (2006) Machine Learning for High-Speed Corner Detection. In: Leonardis A, Bischof H, Pinz A (eds) Computer Vision – ECCV 2006. Springer, Berlin, pp 430–443

    Chapter  Google Scholar 

  32. Rublee E, Rabaud V, Konolige K, Bradski G (2011) “ORB: An efficient alternative to SIFT or SURF,” in 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2564–2571

  33. Satopaa V, Albrecht J, Irwin D, Raghavan B (2011) “Finding a ‘Kneedle’ in a Haystack: Detecting Knee Points in System Behavior”, in Proceedings of the 2011 31st International Conference on Distributed Computing Systems Workshops. Washington, DC, USA, pp 166–171

  34. Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEETrans PatternAnalMach Intell 22(8):888–905

    Google Scholar 

  35. Simon I, Snavely N, Seitz SM (2007) “Scene Summarization for Online Image Collections”, in IEEE 11th International Conference on Computer Vision, 2007. ICCV 2007:1–8

  36. Sitnik R, Karaszewski M (2010) “Automated Processing of Data from 3D Scanning of Cultural Heritage Objects,”. In: Ioannides M, Fellner D, Georgopoulos A, Hadjimitsis DG (eds) Digital Heritage. Springer, Berlin Heidelberg, pp 28–41

    Chapter  Google Scholar 

  37. Snavely N, Seitz SM, Szeliski R (2006) Photo Tourism: exploring Photo Collections in 3D. ACM Trans Graph 25(3):835–846

    Article  Google Scholar 

  38. Wu C, Agarwal S, Curless B, Seitz SM(2011) “Multicore bundle adjustment,” in 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3057–3064

  39. Wu C, Agarwal S, Curless B, Seitz SM(2012) “Schematic surface reconstruction,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1498–1505

  40. Wu C, Frahm J-M, Pollefeys M (2011) “Repetition-based dense single-view reconstruction,” in 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3113–3120

  41. Yu SX, Shi J (2003) “Multiclass spectral clustering,” in Ninth IEEE International Conference on Computer Vision, 2003. Proceedings, pp. 313–319 vol.1

  42. Zheng Y-T, Zhao M, Song Y, Adam H, Buddemeier U, Bissacco A, Brucher F, Chua T-S, Neven H (2009) “Tour the world: Building a web-scale landmark recognition engine,” in IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 1085–1092

Download references

Acknowledgments

The research leading to these results has been supported by Marie Curie IAPP project 4D-CH-World: Four Dimensional Cultural Heritage World. Grant agreement number324523.

Author information

Authors and Affiliations

  1. Computer Vision and Decision Support Laboratory, Technical University of Crete, University Campus, Kounoupidiana, Chania, Greece, 73100

    Konstantinos Makantasis & Anastasios Doulamis

  2. Dept. of Electrical and Computer Engineering, Cyprus University of Technology, 30, Archbishop Kyprianou, Lemesos, Cyprus, 3036

    Nikolaos Doulamis & Marinos Ioannides

Authors
  1. Konstantinos Makantasis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anastasios Doulamis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nikolaos Doulamis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marinos Ioannides
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Konstantinos Makantasis.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Makantasis, K., Doulamis, A., Doulamis, N. et al. In the wild image retrieval and clustering for 3D cultural heritage landmarks reconstruction. Multimed Tools Appl 75, 3593–3629 (2016). https://doi.org/10.1007/s11042-014-2191-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-2191-z

Keywords

Search

Navigation