Skip to main content
SpringerLink
Log in

Multiview image generation for vehicle reidentification

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Vehicle re-identification (ReID) with viewpoint variations is an interesting but challenging task in computer vision. Most existing vehicle ReID approaches focus on the original single view, which requires vehicle features in varying views. However, this approach limits the models’ discriminative capabilities in realistic scenarios due to the lack of visual information in arbitrary views. In this paper, we propose a multi-view generative adversarial network (MV-GAN) that can synthesize real vehicle images conditioned on arbitrary skeleton views. MV-GAN is designed specifically for viewpoint normalization in vehicle ReID. Based on the generated images, we can infer a multi-view vehicle representation to learn distance metrics for vehicle ReID from the original images that is free of the influence of viewpoint variations. We show that the features of the generated images and the original images are complementary. We demonstrate the validity of the proposed method through extensive experiments on the VeRi, VehicleID, and VRIC datasets and show the superiority of multi-view image generation for improving vehicle ReID through comparisons with the state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

The VeRi dataset can be obtained from: https://github.com/JDAI-CV/VeRidataset

The VehicleID dataset can be obtained from: https://www.pkuml.org/resources/pku-vehicleid.html

The VRIC dataset can be obtained from: https://qmul-vric.github.io/

References

  1. Zapletal D, Herout A (2016) Vehicle re-identification for automatic video traffic surveillance. In: 2016 IEEE conference on computer vision and pattern recognition workshops (CVPRW). IEEE, Las Vegas, pp. 1568–1574

  2. Khan SD, Ullah H (2019) A survey of advances in vision-based vehicle re-identification. Comput Vis Image Underst 182:50–63. https://doi.org/10.1016/j.cviu.2019.03.001

    Article  Google Scholar 

  3. Wu CW, Liu CT, Chiang CE, Tu WC, Chien SY (2018) Vehicle re-identification with the space-time prior. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW). IEEE, Salt Lake City, pp. 121–128

  4. Guo H, Zhu K, Tang M, Wang J (2019) Two-level attention network with multi-grain ranking loss for vehicle re-identification. IEEE Trans Image Process 28:4328–4338. https://doi.org/10.1109/TIP.2019.2910408

    Article  MathSciNet  MATH  Google Scholar 

  5. Peng J, Wang H, Zhao T, Fu X (2019) Learning multi-region features for vehicle re-identification with context-based ranking method. Neurocomputing 359:427–437. https://doi.org/10.1016/j.neucom.2019.06.013

    Article  Google Scholar 

  6. Zhu J, Du Y, Hu Y, Zheng L, Cai C (2019) VRSDNet: vehicle re-identification with a shortly and densely connected convolutional neural network. Multimed Tools Appl 78:29043–29,057. https://doi.org/10.1007/s11042-018-6270-4

    Article  Google Scholar 

  7. Zhou Y, Shao L (2018) Viewpoint-aware attentive multi-view inference for vehicle re-identification. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, Salt Lake City, pp. 6489–6498

  8. Lou Y, Bai Y, Liu J, Wang S, Duan L (2019) Embedding adversarial learning for vehicle re-identification. IEEE Trans Image Process 28:3794–3807. https://doi.org/10.1109/TIP.2019.2902112

    Article  MathSciNet  MATH  Google Scholar 

  9. Xu C, Fu Y, Wen C, Pan Y, Jiang YG, Xue X (2018) Pose-guided person image synthesis in the non-iconic views. IEEE Trans Image Process 29:9060–9072. https://doi.org/10.1109/TIP.2020.3023853

    Article  Google Scholar 

  10. Zhao B, Wu X, Cheng ZQ, Liu H, Jie Z, Feng J (2018) Multi-view image generation from a single-view. In: 2018 ACM MultimediaConference (MM ‘18), October 22–26, 2018, Seoul, Republic of Korea. ACM, New York, pp. 383–391

  11. Ding W, Li S, Zhang G, Lei X, Qian H (2018) Vehicle pose and shape estimation through multiple monocular vision. In: Proceedings of the 2018 IEEE international conference on robotics and biomimetics (ROBIO). IEEE, Kuala Lumpur, pp. 709–715.

  12. Zhu JY, Zhang Z, Zhang C, Wu J, Torralb A, Tenenbaum JB, Freeman WT (2018) Visual object networks: image generation with disentangled 3D representation. In: Proceedings of the 32nd international conference on neural information processing systems. NeurIPS, Montréal, pp. 118–129

  13. Liu X, Liu W, Ma H, Fu H (2016) Large-scale vehicle re-identification in urban surveillance videos. In: 2016 IEEE international conference on multimedia and expo (ICME). IEEE, Seattle, pp. 1–6

  14. Liu X, Liu W, Mei T, Ma H (2016) A deep learning-based approach to progressive vehicle re-identification for urban surveillance. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer Vision – ECCV 2016. ECCV 2016. Lecture Notes in Computer Science. Springer, Cham, pp 869–884

    Google Scholar 

  15. Liu H, Tian Y, Wang Y, Pang L, Huang T (2016) Deep relative distance learning: tell the difference between similar vehicles. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Las Vegas, pp. 2167–2175

  16. Kanacı A, Zhu X, Gong S (2018) Vehicle re-identification in context. In: Pattern Recognition. GCPR 2018. Lecture Notes in Computer Science. Springer: Cham, pp. 377–390

  17. Liu X, Liu W, Mei T, Ma H (2018) PROVID: progressive and multimodal vehicle reidentification for large-scale urban surveillance. IEEE Trans Multimed 20:645–658. https://doi.org/10.1109/TMM.2017.2751966

    Article  Google Scholar 

  18. Zheng Z, Ruan T, Wei Y, Yang Y (2020) VehicleNet: learning robust visual representation for vehicle re-identification. IEEE Trans Multimed 1. https://doi.org/10.1109/TMM.2020.3014488

  19. Zhu J, Zeng H, Du Y, Lei Z, Zheng L, Cai C (2018) Joint feature and similarity deep learning for vehicle re-identification. IEEE Access 6:43724–43731. https://doi.org/10.1109/ACCESS.2018.2862382

    Article  Google Scholar 

  20. Marín-Reyes PA, Bergamini L, Lorenzo-Navarro J, Palazzi A, Calderara S, Cucchiara R (2018) Unsupervised vehicle re-identification using triplet networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW). IEEE, Salt Lake City, pp. 166–1665

  21. He B, Li J, Zhao Y, Tian Y (2019) Part-regularized near-duplicate vehicle re-identification. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, Long Beach, pp. 3992–4000

  22. Zhu J, Zeng H, Huang J, Liao S, Lei Z, Cai C, Zheng L (2020) Vehicle re-identification using quadruple directional deep learning features. IEEE Trans Intell Transp Syst 21:410–420. https://doi.org/10.1109/TITS.2019.2901312

    Article  Google Scholar 

  23. Wang H, Peng J, Chen D, Jiang G, Zhao T, Fu X (2020) Attribute-guided feature learning network for vehicle re-identification. IEEE Multimed 27:112–121. https://doi.org/10.1109/MMUL.2020.2999464

    Article  Google Scholar 

  24. Bai Y, Lou Y, Gao F, Wang S, Wu Y, Duan LY (2018) Group-sensitive triplet embedding for vehicle reidentification. IEEE Trans Multimed 20:2385–2399. https://doi.org/10.1109/TMM.2018.2796240

    Article  Google Scholar 

  25. Hou J, Zeng H, Zhu J, Hou J, Chen J, Ma KK (2019) Deep quadruplet appearance learning for vehicle re-identification. IEEE Trans Veh Technol 68:8512–8522. https://doi.org/10.1109/TVT.2019.2927353

    Article  Google Scholar 

  26. Wang H, Sun S, Zhou L, Guo L, Min X, Li C (2020) Local feature-aware siamese matching model for vehicle re-identification. Appl Sci-Basel 10:2474. https://doi.org/10.3390/app10072474

    Article  Google Scholar 

  27. Meng D, Li L, Liu X, Li Y, Yang S, Zha Z, Gao X, Wang S, Huang Q (2020) Parsing-based view-aware embedding network for vehicle re-identification. In: Proceedings of the 2020 IEEE/CVF conference on computer vision and pattern recognition, CVPR. IEEE, Seattle, pp. 7101–7110

  28. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. In: Proceedings of the international conference on neural information processing systems (NIPS). OpeniAI: Barcelona, pp. 2672–2680

  29. Radford A, Metz L, Chintala S (2016) Unsupervised representation learning with deep convolutional generative adversarial networks. In: Proceedings of the international conference on learning representations (ICLR). San Diego, pp. 1–16.

  30. Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P (2016) InfoGAN: interpretable representation learning by information maximizing generative adversarial nets. In: Proceedings of the neural information processing systems (NIPS). NIPS, Barcelona, pp. 2180–2188

  31. Denton E, Chintala S, Szlam A, Fergus R (2015) Deep generative image models using a laplacian pyramid of adversarial networks. In: Proceedings of the 28th international conference on neural information processing systems (NIPS). MIT Press, Cambridge, pp. 1486–1494

  32. Huang X, Li Y, Poursaeed O, Hopcroft J, Belongie S (2016) Stacked generative adversarial networks. In: Proceedings of the 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Honolulu, pp. 1866–1875

  33. Huang R, Zhang S, Li T, He R (2017) Beyond face rotation: global and local perception GAN for photorealistic and identity preserving frontal view synthesis. In: Proceedings of the IEEE international conference on computer vision (ICCV). IEEE, Venice, pp. 2458–2467

  34. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Honolulu, pp. 105–114.

  35. Nguyen TD, Le T, Vu H, Phung D (2017) Dual discriminator generative adversarial nets. In: Guyon I, Luxburg UV, Bengio S et al (eds) Proceedings of the 31st international conference on neural information processing systems (NIPS). Neural Information Processing Systems Foundation, Inc., Long Beach, pp 2667–2677

    Google Scholar 

  36. Lou Y, Bai Y, Liu J, Wang S, Duan L (2019) VERI-Wild: a large dataset and a new method for vehicle re-identification in the wild. In: Proceedings of the 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, Long Beach, pp. 3230–3238

  37. Peng J, Wang H, Xu F, Fu X (2020) Cross domain knowledge learning with dual-branch adversarial network for vehicle re-identification. Neurocomputing 401:133–144. https://doi.org/10.1016/j.neucom.2020.02.112

    Article  Google Scholar 

  38. Zhou Y, Shao L (2017) Cross-view GAN based vehicle generation for re-identification. In: Kim TK, Zafeiriou S, Brostow G, Mikolajczyk K (eds) Proceedings of the british machine vision conference (BMVC). BMVA Press, London, pp 186.1–186.12

    Google Scholar 

  39. Zhou Y, Liu L, Shao L (2018) Vehicle re-identification by deep hidden multi-view inference. IEEE Trans Image Process 27:3275–3287. https://doi.org/10.1109/TIP.2018.2819820

    Article  MathSciNet  Google Scholar 

  40. Kipf T, Pol E, Welling M (2020) Contrastive learning of structured world models. In: Proceedings of the international conference on learning representations (ICLR). Addis Ababa, pp. 1–21

  41. Wang Y, Zhou Q, Liu J, Xiong J, Gao G, Wu X, Latecki LJ (2019) Lednet: a lightweight encoder-decoder network for real-time semantic segmentation. In: Proceedings of the 2019 IEEE international conference on image processing. IEEE, Taipei, pp. 1860–1864

  42. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Las Vegas, pp. 770–778

  43. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd international conference on international conference on machine learning (ICML). ACM, Lille, pp. 448–456

  44. Wang G, Giannakis GB, Chen J (2019) Learning ReLU networks on linearly separable data: algorithm, optimality, and generalization. IEEE Trans Signal Process 67:2357–2370. https://doi.org/10.1109/TSP.2019.2904921

    Article  MathSciNet  MATH  Google Scholar 

  45. Zhang F, Yang F, Li C, Yuan G (2019) CMNet: a connect-and-merge convolutional neural network for fast vehicle detection in urban traffic surveillance. IEEE Access 7:72660–72,671. https://doi.org/10.1109/ACCESS.2019.2919103

    Article  Google Scholar 

  46. Deng W, Zheng L, Ye Q, Kang G, Yang Y, Jiao J (2018) Image-image domain adaptation with preserved self-similarity and domain-dissimilarity for person re-identification. In: Proceedings of the 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, Salt Lake City, pp. 994–1003

  47. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of the 2012 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Piscataway, pp. 3354–3361

  48. Abadi M, Barham P, Chen J et al. (2016) TensorFlow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX conference on operating systems design and implementation (OSDI). USENIX Association, Berkeley, pp. 265–283

  49. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in PyTorch. In: Proceedings of the neural information processing systems (NIPS). Long Beach, pp 1–4

  50. Wen L, Du D, Cai Z, Lei Z, Chang MC, Qi H, Lim J, Yang MH, Lyu S (2015) UA-DETRAC: a new benchmark and protocol for multi-object detection and tracking. Comput Vis Image Underst 193:102907. https://doi.org/10.1016/j.cviu.2020.102907

    Article  Google Scholar 

  51. Qian X, Fu Y, Xiang T, Wang W, Qiu J, Wu Y, Jiang YG, Xue X (2017) Pose-normalized image generation for person re-identification. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y (eds) Computer Vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science. Springer, Cham, pp 661–678

    Google Scholar 

  52. Kingma D, Ba J (2015) Adam: a method for stochastic optimization. In: Proceedings of the international conference on learning representations (ICLR). San Diego, pp. 1–15

  53. Mandt S, Hoffman MD, Blei DM (2017) Stochastic gradient descent as approximate bayesian inference. J Mach Learn Res 18:1–35

    MathSciNet  MATH  Google Scholar 

  54. Zheng L, Wang S, Zhou W, Tian Q (2014) Bayes merging of multiple vocabularies for scalable image retrieval. In: Proceedings of the 2014 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Columbus, pp. 1963–1970

  55. Liao S, Hu Y, Zhu X, Li SZ (2015) Person re-identification by local maximal occurrence representation and metric learning. In: Proceedings of the 2015 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Boston, pp. 2197–2206

  56. Zheng L, Shen L, Tian L, Wang S, Wang J, Tian Q (2015) Scalable person re-identification: a benchmark. In: Proceedings of the 2015 IEEE international conference on computer vision (ICCV). IEEE, Santiago, pp. 1116–1124

  57. Xiao T, Li H, Ouyang W, Wang X (2016) Learning deep feature representations with domain guided dropout for person re-identification. In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Las Vegas, pp. 1249–1258

  58. Zhou Y, Shao L (2018) Vehicle re-identification by adversarial bi-directional LSTM network. In: Proceedings of the 2018 IEEE winter conference on applications of computer vision (WACV). IEEE, Lake Tahoe, pp. 653–662

  59. Shen Y, Xiao T, Li H, Yi S, Wang X (2017) Learning deep neural networks for vehicle re-ID with visual-spatio-temporal path proposals. In: Proceedings of the 2017 IEEE international conference on computer vision (ICCV). IEEE, Venice, pp. 1918–1927.

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant U1904119, in part by the National Natural Science Foundation of China under Grant 71774159, and in part by the Fundamental Research Funds for the Central Universities of China under Grant 2015XKMS085.

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Henan Polytechnic University, Jiaozuo, 454000, Henan, China

    Fukai Zhang, Yongqiang Ma, Haiyan Zhang & Jianji Ren

  2. School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, 221116, China

    Guan Yuan

Authors
  1. Fukai Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongqiang Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guan Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haiyan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jianji Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

G.Y. and J.R. collected and processed the data; H.Z. designed the study and performed the experiments; F.Z. performed the experiments, analyzed the test results, and wrote the manuscript; Y.M. reviewed and improved the manuscript.

Corresponding author

Correspondence to Yongqiang Ma.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Code availability

The code in the article is based on the existing pedestrian reidentification code. The code was modified and improved according to the special characteristics of the vehicle, and good results were achieved.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, F., Ma, Y., Yuan, G. et al. Multiview image generation for vehicle reidentification. Appl Intell 51, 5665–5682 (2021). https://doi.org/10.1007/s10489-020-02171-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-02171-8

Keywords

Search

Navigation