Lei Wu
Steven C.H. Hoi
Abstract
Automated photo tagging is an important technique for many intelligent multimedia information systems, for example, smart photo management system and intelligent digital media library. To attack the challenge, several machine learning techniques have been developed and applied for automated photo tagging. For example, supervised learning techniques have been applied to automated photo tagging by training statistical classifiers from a collection of manually labeled examples. Although the existing approaches work well for small testbeds with relatively small number of annotation words, due to the long-standing challenge of object recognition, they often perform poorly in large-scale problems. Another limitation of the existing approaches is that they require a set of high-quality labeled data, which is not only expensive to collect but also time consuming. In this article, we investigate a social image based annotation scheme by exploiting implicit side information that is available for a large number of social photos from the social web sites. The key challenge of our intelligent annotation scheme is how to learn an effective distance metric based on implicit side information (visual or textual) of social photos. To this end, we present a novel “Probabilistic Distance Metric Learning” (PDML) framework, which can learn optimized metrics by effectively exploiting the implicit side information vastly available on the social web. We apply the proposed technique to photo annotation tasks based on a large social image testbed with over 1 million tagged photos crawled from a social photo sharing portal. Encouraging results show that the proposed technique is effective and promising for social photo based annotation tasks.
- Andoni, A. and Indyk, P. 2008. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Comm. ACM 51, 1, 117--122. Google Scholar
Digital Library
- Bar-Hillel, A., Hertz, T., Shental, N., and Weinshall, D. 2005. Learning a mahalanobis metric from equivalence constraints. J. Mach. Learn. Res. 6, 937--965. Google ScholarDigital Library
- Bezdek, J. C. 1981. Pattern Recognition with Fuzzy Objective Function Algorithms. Kluwer Academic Publishers, Norwell, MA. Google ScholarDigital Library
- Bezdek, J. C. and Hathaway, R. J. 2003. Convergence of alternating optimization. Neural, Parall. Sci. Comput. 11, 4, 351--368. Google ScholarDigital Library
- Carneiro, G., Chan, A. B., Moreno, P., and Vasconcelos, N. 2006. Supervised learning of semantic classes for image annotation and retrieval. IEEE Tran. Patt. Anal. Mach. Intell. 394--410. Google ScholarDigital Library
- Carneiro, G. and Vasconcelos, N. 2005. Formulating semantic image annotation as a supervised learning problem. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'05). 163--168. Google ScholarDigital Library
- Davis, J. V., Kulis, B., Jain, P., Sra, S., and Dhillon, I. S. 2007. Information-theoretic metric learning. In Proceedings of the International Conference on Machine Learning (ICML'07). 209--216. Google ScholarDigital Library
- Duchi, J., Shalev-Shwartz, S., Singer, Y., and Chandra, T. 2008. Efficient projections onto the l1-ball for learning in high dimensions. In Proceedings of the International Conference on Machine Learning (ICML'08). ACM, New York, 272--279. Google ScholarDigital Library
- Duygulu, P., Barnard, K., de Freitas, J., and Forsyth, D. A. 2002. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In Proceedings of the 2nd European Conference on Computer Vision (ECCV'02). 97--112. Google ScholarDigital Library
- Fan, J., Gao, Y., and Luo, H. 2004. Multi-level annotation of natural scenes using dominant image components and semantic concepts. In Proceedings of the 12th Annual ACM International Conference on Multimedia. 540--547. Google ScholarDigital Library
- Fukunaga, K. 1990. Introduction to Statistical Pattern Recognition. Elsevier. Google ScholarDigital Library
- Globerson, A. and Roweis, S. 2005. Metric learning by collapsing classes. In Proceedings of the Advances in Neural Information Processing Systems.Google Scholar
- Goldberger, J., Roweis, S., Hinton, G., and Salakhutdinov, R. 2005. Neighbourhood components analysis. In Proceedings of the Advances in Neural Information Processing Systems 17, 513--520.Google Scholar
- He, X. and Zemel, R. S. 2008. Learning hybrid models for image annotation with partially labeled data. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 625--632.Google Scholar
- Hoi, C.-H. and Lyu, M. R. 2004. Web image learning for searching semantic concepts in image databases. In Proceedings of the 13th International World Wide Web Conference on Alternate Track Papers & Posters. 406--407. Google ScholarDigital Library
- Hoi, S. C., Jin, R., Zhu, J., and Lyu, M. R. 2009. Semi-supervised svm batch mode active learning with applications to image retrieval. ACM Trans. Inf. Syst. 27, 3, 1--29. Google ScholarDigital Library
- Hoi, S. C. H., Liu, W., and Chang, S.-F. 2010. Semi-supervised distance metric learning for collaborative image retrieval and clustering. ACM Trans. Multimed. Comput. Comm. Appl. 6, 3, 1--26. Google ScholarDigital Library
- Hoi, S. C. H., Liu, W., Lyu, M. R., and Ma, W.-Y. 2006a. Learning distance metrics with contextual constraints for image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'06). Google ScholarDigital Library
- Hoi, S. C. H., Lyu, M. R., and Jin, R. 2006b. A unified log-based relevance feedback scheme for image retrieval. IEEE Trans. Knowl. Data Engin. 18, 4, 509--204. Google ScholarDigital Library
- Jeon, J., Lavrenko, V., and Manmatha, R. 2003. Automatic image annotation and retrieval using cross-media relevance models. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval (SIGIR'03). 119--126. Google ScholarDigital Library
- Jin, R., Wang, S., and Zhou, Y. 2009. Regularized distance metric learning: Theory and algorithm. In Proceedings of the Conference on Advances in Neural Information Processing Systems 22. 862--870.Google Scholar
- Lew, M. S., Sebe, N., Djeraba, C., and Jain, R. 2006. Content-based multimedia information retrieval: State of the art and challenges. ACM Trans. Multimed. Comput. Comm. Appl. 2, 1, 1--19. Google ScholarDigital Library
- Li, W. and Sun, M. 2006. Semi-supervised learning for image annotation based on conditional random fields. In Proceedings of the ACM International Conference on Image and Video Retrieval (CIVR). ACM, 463--472. Google ScholarDigital Library
- Liu, J. and Ye, J. 2009. Efficient euclidean projections in linear time. In Proceedings of the International Conference on Machine Learning (ICML'09). 657--664. Google ScholarDigital Library
- Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. IJCV 60, 91--110. Google ScholarDigital Library
- Qi, G.-J., Hua, X.-S., and Zhang, H.-J. 2009. Learning semantic distance from community-tagged media collection. In Proceedings of the 17th ACM International Conference on Multimedia (MM'09). ACM, New York, 243--252. Google ScholarDigital Library
- Russell, B. C., Torralba, A., Murphy, K. P., and Freeman, W. T. 2008. Labelme: A database and web-based tool for image annotation. Int. J. Comput. Vision 77, 1-3, 157--173. Google ScholarDigital Library
- Shalev-Shwartz, S. and Singer, Y. 2006. Efficient learning of label ranking by soft projections onto polyhedra. J. Mach. Learn. Res. 7, 1567--1599. Google ScholarDigital Library
- Si, L., Jin, R., Hoi, S. C. H., and Lyu, M. R. 2006. Collaborative image retrieval via regularized metric learning. ACM Multimed. Syst. J. 12, 1, 34--44.Google ScholarDigital Library
- Sigurbjörnsson, B. and van Zwol, R. 2008. Flickr tag recommendation based on collective knowledge. In Proceeding of the 17th International Conference on World Wide Web (WWW'08). 327--336. Google ScholarDigital Library
- Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., and Jain, R. 2000. Content-based image retrieval at the end of the early years. IEEE Trans. Patt. Anal. Mach. Intell. 22, 12, 1349--1380. Google ScholarDigital Library
- Stone, Z., Zickler, T., and Darrell, T. 2008. Autotagging facebook: Social network context improves photo annotation. In Proceedings of the IEEE Workshop on Internet Vision. IEEE.Google Scholar
- Torralba, A., Weiss, Y., and Fergus, R. 2008. Small codes and large databases of images for object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- Wang, C., Zhang, L., and Zhang, H.-J. 2008a. Learning to reduce the semantic gap in web image retrieval and annotation. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'08). ACM, New York, 355--362. Google ScholarDigital Library
- Wang, M., Zhou, X., and Chua, T.-S. 2008b. Automatic image annotation via local multi-label classification. In Proceedings of the ACM International Conference on Image and Video Retrieval (CIVR). ACM, New York, 17--26. Google ScholarDigital Library
- Wang, X.-J., Zhang, L., Jing, F., and Ma, W.-Y. 2006. Annosearch: Image auto-annotation by search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'06). 1483--1490. Google ScholarDigital Library
- Weinberger, K., Blitzer, J., and Saul, L. 2006. Distance metric learning for large margin nearest neighbor classification. In Proceedings of the Conference on Advances in Neural Information Processing Systems 18, 1473--1480.Google Scholar
- Wu, L., Hoi, S. C., Zhu, J., Jin, R., and Yu, N. 2009. Distance metric learning from uncertain side information with application to automated photo tagging. In Proceedings of the Conference on ACM International Conference on Multimedia (MM'09). Google ScholarDigital Library
- Xing, E. P., Ng, A. Y., Jordan, M. I., and Russell, S. 2002. Distance metric learning with application to clustering with side-information. In Proceedings of the Neural Information Processing.Google Scholar
- Yan, R., Natsev, A., and Campbell, M. 2008. A learning-based hybrid tagging and browsing approach for efficient manual image annotation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08).Google Scholar
- Yang, L., Jin, R., Sukthankar, R., and Liu, Y. 2006. An efficient algorithm for local distance metric learning. In Proceedings of AAAI. Google ScholarDigital Library
Index Terms
- Distance metric learning from uncertain side information for automated photo tagging
Recommendations
Distance metric learning from uncertain side information with application to automated photo tagging
MM '09: Proceedings of the 17th ACM international conference on MultimediaAutomated photo tagging is essential to make massive unlabeled photos searchable by text search engines. Conventional image annotation approaches, though working reasonably well on small testbeds, are either computationally expensive or inaccurate when ...
Mining social images with distance metric learning for automated image tagging
WSDM '11: Proceedings of the fourth ACM international conference on Web search and data miningWith the popularity of various social media applications, massive social images associated with high quality tags have been made available in many social media web sites nowadays. Mining social images on the web has become an emerging important research ...
Intelligent photo clustering with user interaction and distance metric learning
Photo clustering is an effective way to organize albums and it is useful in many applications, such as photo browsing and tagging. But automatic photo clustering is not an easy task due to the large variation of photo content. In this paper, we propose ...
Comments