Skip to main content
SpringerLink
Log in

On the use of deep learning for blind image quality assessment

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

In this work, we investigate the use of deep learning for distortion-generic blind image quality assessment. We report on different design choices, ranging from the use of features extracted from pre-trained convolutional neural networks (CNNs) as a generic image description, to the use of features extracted from a CNN fine-tuned for the image quality task. Our best proposal, named DeepBIQ, estimates the image quality by average-pooling the scores predicted on multiple subregions of the original image. Experimental results on the LIVE In the Wild Image Quality Challenge Database show that DeepBIQ outperforms the state-of-the-art methods compared, having a linear correlation coefficient with human subjective scores of almost 0.91. These results are further confirmed also on four benchmark databases of synthetically distorted images: LIVE, CSIQ, TID2008, and TID2013.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Alaei, A., Raveaux, R., Conte, D.: Image quality assessment based on regions of interest. Signal Image Video Process. 11(4), 673–680 (2017)

  2. Allen, E., Triantaphillidou, S., Jacobson, R.: Image quality comparison between jpeg and jpeg2000. I. Psychophysical investigation. J. Imaging Sci. Technol. 51(3), 248–258 (2007)

    Article  Google Scholar 

  3. Bengio, Y.: Deep learning of representations for unsupervised and transfer learning. In: Unsupervised and Transfer Learning Challenges in Mach. Learn. vol. 7, p. 19 (2012)

  4. Bianco, S., Ciocca, G., Marini, F., Schettini, R.: Image quality assessment by preprocessing and full reference model combination. In: IS&T/SPIE Electronic Imaging, pp. 72,420O (2009)

  5. Bovik, A.C.: Automatic prediction of perceptual image and video quality. Proc. IEEE 101(9), 2008–2024 (2013)

    Google Scholar 

  6. Ciancio, A., Da Costa, A.L.N.T., da Silva, E.A., Said, A., Samadani, R., Obrador, P.: No-reference blur assessment of digital pictures based on multifeature classifiers. IEEE Trans. Image Process. 20(1), 64–75 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  7. Ciocca, G., Corchs, S., Gasparini, F., Schettini, R.: How to assess image quality within a workflow chain: an overview. Int. J. Digit. Libr. 15(1), 1–25 (2014)

    Article  Google Scholar 

  8. Corchs, S., Gasparini, F., Schettini, R.: No reference image quality classification for jpeg-distorted images. Digit. Signal Process. 30, 86–100 (2014)

    Article  Google Scholar 

  9. Eckert, M.P., Bradley, A.P.: Perceptual quality metrics applied to still image compression. Signal Process. 70(3), 177–200 (1998)

    Article  MATH  Google Scholar 

  10. Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: Liblinear: a library for large linear classification. J. Mach. Learn. Res. 9, 1871–1874 (2008)

    MATH  Google Scholar 

  11. Ghadiyaram, D., Bovik, A.C.: Blind image quality assessment on real distorted images using deep belief nets. In: Global Conference on Signal and Information Processing (GlobalSIP), pp. 946–950. IEEE (2014)

  12. Ghadiyaram, D., Bovik, A.C.: Crowdsourced study of subjective image quality. In: Asilomar Conference on Signals, Systems and Computers (2014)

  13. Ghadiyaram, D., Bovik, A.C.: Massive online crowdsourced study of subjective and objective picture quality. IEEE Trans. Image Process. 25(1), 372–387 (2016)

    Article  MathSciNet  Google Scholar 

  14. He, L., Gao, X., Lu, W., Li, X., Tao, D.: Image quality assessment based on S-CIELAB model. Signal Image Video Process. 5(3), 283–290 (2011)

    Article  Google Scholar 

  15. Hou, W., Gao, X., Tao, D., Li, X.: Blind image quality assessment via deep learning. IEEE Trans. Neural Netw Learn. Syst. 26(6), 1275–1286 (2015)

    Article  MathSciNet  Google Scholar 

  16. Huang, Y.M., Du, S.X.: Weighted support vector machine for classification with uneven training class sizes. In: 2005 International Conference on Mach. Learn. and Cybernetics, vol. 7, pp. 4365–4369. IEEE (2005)

  17. Jia, Y., et al.: Caffe: convolutional architecture for fast feature embedding. In: ACM MM, pp. 675–678. ACM (2014)

  18. Kang, L., Ye, P., Li, Y., Doermann, D.: Convolutional neural networks for no-reference image quality assessment. In: CVPR, pp. 1733–1740 (2014)

  19. Kang, L., Ye, P., Li, Y., Doermann, D.: Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks. In: ICIP, pp. 2791–2795. IEEE (2015)

  20. Kottayil, N.K., Cheng, I., Dufaux, F., Basu, A.: A color intensity invariant low-level feature optimization framework for image quality assessment. Signal Image Video Process. 10(6), 1169–1176 (2016)

    Article  Google Scholar 

  21. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)

  22. Larson, E.C., Chandler, D.M.: Most apparent distortion: full-reference image quality assessment and the role of strategy. JEI 19(1), 011,006 (2010)

    Google Scholar 

  23. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  24. LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K-R.: Efficient Back Prop. In: Montavon, G., Orr, G.B., Müller, K-R. (eds) Neural Networks: Tricks of the Trade, 2nd edn, pp. 9–48, Springer, Berlin, Heidelberg (2012)

  25. Li, J., Yan, J., Deng, D., Shi, W., Deng, S.: No-reference image quality assessment based on hybrid model. Signal Image Video Process. 11(6), 985–992 (2017)

    Article  Google Scholar 

  26. Li, J., Zou, L., Yan, J., Deng, D., Qu, T., Xie, G.: No-reference image quality assessment using Prewitt magnitude based on convolutional neural networks. Signal Image Video Process. 10(4), 609–616 (2016)

    Article  Google Scholar 

  27. Lv, Y., Jiang, G., Yu, M., Xu, H., Shao, F., Liu, S.: Difference of Gaussian statistical features based blind image quality assessment: a deep learning approach. In: ICIP, pp. 2344–2348. IEEE (2015)

  28. Mahmoudpour, S., Kim, M.: No-reference image quality assessment in complex-shearlet domain. Signal Image Video Process. 10(8), 1465–1472 (2016)

    Article  Google Scholar 

  29. Manap, R.A., Shao, L.: Non-distortion-specific no-reference image quality assessment: a survey. Inf. Sci. 301, 141–160 (2015)

    Article  Google Scholar 

  30. Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Trans Image Process. 21(12), 4695–4708 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  31. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a completely blind image quality analyzer. SPL 20(3), 209–212 (2013)

    Google Scholar 

  32. Mittal, A., Moorthy, A.K., Bovik, A.C., Chen, C.W., Chatzimisios, P., Dagiuklas, T., Atzori, L.: No-reference approaches to image and video quality assessment. In: Multimedia Quality of Experience (QoE): Current Status and Future Requirements, vol. 99. Wiley (2015)

  33. Moorthy, A.K., Bovik, A.C.: Blind image quality assessment: from natural scene statistics to perceptual quality. IEEE Trans Image Process. 20(12), 3350–3364 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  34. Pappas, T.N., Safranek, R.J., Chen, J.: Perceptual criteria for image quality evaluation. In: Handbook of Image and Video Processing, pp. 669–684 (2000)

  35. Ponomarenko, N., Ieremeiev, O., Lukin, V., Egiazarian, K., Jin, L., Astola, J., Vozel, B., Chehdi, K., Carli, M., Battisti, F., et al.: Color image database tid2013: peculiarities and preliminary results. In: Visual Information Processing (EUVIP), 2013 4th European Workshop on, pp. 106–111. IEEE (2013)

  36. Ponomarenko, N., et al.: Tid 2008-a database for evaluation of full-reference visual quality assessment metrics. Adv. Mod. Radioelectron. 10(4), 30–45 (2009)

    MathSciNet  Google Scholar 

  37. Razavian, A., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. In: CVPR Workshops, pp. 806–813 (2014)

  38. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet Large Scale Visual Recognition Challenge. Int J Comput Vis. 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  39. Saad, M.A., Bovik, A.C., Charrier, C.: Blind image quality assessment: a natural scene statistics approach in the DCT domain. Trans. Image Process. 21(8), 3339–3352 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  40. Seshadrinathan, K., Bovik, A.C.: Automatic prediction of perceptual quality of multimedia signals—a survey. Multimed. Tools Appl. 51(1), 163–186 (2011)

    Article  Google Scholar 

  41. Sheikh, H.R., Wang, Z., Cormack, L., Bovik, A.C.: Live image quality assessment database release 2 (2005). http://live.ece.utexas.edu/research/quality/subjective.htm. Accessed 29 Aug 2017

  42. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)

  43. Soundararajan, R., Bovik, A.C.: Survey of information theory in visual quality assessment. Signal Image Video Process. 7(3), 391–401 (2013)

    Article  Google Scholar 

  44. Szegedy, C., et al.: Going deeper with convolutions. In: IEEE CVPR, pp. 1–9 (2015)

  45. Tang, H., Joshi, N., Kapoor, A.: Blind image quality assessment using semi-supervised rectifier networks. In: CVPR, pp. 2877–2884 (2014)

  46. Triantaphillidou, S., Allen, E., Jacobson, R.: Image quality comparison between jpeg and jpeg2000. II. Scene dependency, scene analysis, and classification. J. Imaging Sci. Technol. 51(3), 259–270 (2007)

    Article  Google Scholar 

  47. Vu, C.T., Phan, T.D., Chandler, D.M.: S3: a spectral and spatial measure of local perceived sharpness in natural images. Trans. Image Process. 21(3), 934–945 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  48. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  49. Xu, J., Ye, P., Li, Q., Du, H., Liu, Y., Doermann, D.: Blind image quality assessment based on high order statistics aggregation. IEEE Trans. Image Process. 25(9), 4444–4457 (2016)

    Article  MathSciNet  Google Scholar 

  50. Ye, P., Kumar, J., Kang, L., Doermann, D.: Real-time no-reference image quality assessment based on filter learning. In: CVPR, pp. 987–994 (2013)

  51. Zhang, Y., Moorthy, A.K., Chandler, D.M., Bovik, A.C.: C-DIIVINE: no-reference image quality assessment based on local magnitude and phase statistics of natural scenes. Signal Process. Image Commun. 29(7), 725–747 (2014)

    Article  Google Scholar 

  52. Zhou, Z.H., Liu, X.Y.: Training cost-sensitive neural networks with methods addressing the class imbalance problem. Trans. Knowl. Data Eng. 18(1), 63–77 (2006)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Informatics, Systems and Communication, University of Milano-Bicocca, Viale Sarca 336, 20126, Milan, Italy

    Simone Bianco, Luigi Celona, Paolo Napoletano & Raimondo Schettini

Authors
  1. Simone Bianco
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luigi Celona
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paolo Napoletano
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raimondo Schettini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paolo Napoletano.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bianco, S., Celona, L., Napoletano, P. et al. On the use of deep learning for blind image quality assessment. SIViP 12, 355–362 (2018). https://doi.org/10.1007/s11760-017-1166-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-017-1166-8

Keywords

Search

Navigation