Abstract
Real-time video transmission is one of the most popular applications that are included in the versatile video coding (VVC) standard. However, real-time applications are encountered with practical limitations, including the buffer size and available bandwidth. In these applications, the buffer overflow and underflow should be strictly prevented and also the bit-rate fluctuation should be suppressed. In this paper, a video bit-rate controller is proposed that completely conforms with the constraints of real-time applications. The proposed controller is based on a multi-layer perceptron (MLP) neural network which estimates the proper quantization parameter (QP) modification at the frame level. The buffer occupancy is directly included in the QP derivation process for robust buffer control. Experimental results show that the proposed bit-rate controller fulfils the buffering constraints and controls the bit-rate accurately. The average bit-rate error of the proposed method is 0.29% while providing a low initial buffering delay of about 0.21 s. Also, the rate-distortion analysis shows that the performance of the proposed method is close to those of the conventional algorithms.
Similar content being viewed by others
References
Humberto Ochoa Domınguez, K.R.R.: Versatile video coding latest advances in video coding standards. In: Signal, Image And Speech Processing, River, Netherlands (2019)
Sullivan, G.J.: Video coding standards progress report: joint video experts team launches the versatile video coding project. SMPTE Motion Imaging J. 127(8), 94–98 (2018)
Chen, J., Karczewicz, M., Huang, Y.-W., Choi, K., Ohm, J.-R., Sullivan, G.J.: The joint exploration model (JEM) for video compression with capability beyond HEVC. IEEE Trans. Circuits Syst. Video Technol. 30, 1208–1225 (2019)
Bross, B.: General video coding technology in responses to the joint call for proposals on video compression with capability beyond HEVC. IEEE Trans. Circuits Syst. Video Technol. 30, 1226–1240 (2019)
Joginipelly, A., Varela, A., Charalampidis, D., Schott, R., Fitzsimmons, Z.: Efficient FPGA implementation of steerable Gaussian smoothers. In: Proceedings of the 2012 44th Southeastern Symposium on System Theory (SSST) 2012, pp. 78–82. IEEE.
Joginipelly, A.K., Charalampidis, D.: Efficient separable convolution using field programmable gate arrays. Microprocess. Microsyst. 71, 102852 (2019)
Choi, H., Yoo, J., Nam, J., Sim, D., Bajić, I.V.: Pixel-wise unified rate-quantization model for multi-level rate control. IEEE J. Sel. Top. Signal Process. 7(6), 1112–1123 (2013)
Lee, B., Kim, M., Nguyen, T.Q.: A frame-level rate control scheme based on texture and nontexture rate models for high efficiency video coding. IEEE Trans. Circuits Syst. Video Technol. 24(3), 465–479 (2013)
Seo, C.-W., Moon, J.-H., Han, J.-K.: Rate control for consistent objective quality in high efficiency video coding. IEEE Trans. Image Process. 22(6), 2442–2454 (2013)
Wang, S., Ma, S., Wang, S., Zhao, D., Gao, W.: Rate-GOP based rate control for high efficiency video coding. IEEE J. Sel. Top. Signal Process. 7(6), 1101–1111 (2013)
Yan, T., Ra, I.-H., Wen, H., Weng, M.-H., Zhang, Q., Che, Y.: CTU layer rate control algorithm in scene change video for free-viewpoint video. IEEE Access 8, 24549–24560 (2020)
Li, B., Li, H., Li, L., Zhang, J.: λ-Domain rate control algorithm for high efficiency video coding. IEEE Trans. Image Process. 23(9), 3841–3854 (2014)
Li, L., Li, B., Liu, D., Li, H.: λ-Domain rate control algorithm for HEVC scalable extension. IEEE Trans. Multimed. 18(10), 2023–2039 (2016)
Li, L., Li, B., Li, H., Chen, C.W.: λ-Domain optimal bit allocation algorithm for high efficiency video coding. IEEE Trans. Circuits Syst. Video Technol. 28(1), 130–142 (2016)
Wang, M., Ngan, K.N., Li, H.: Low-delay rate control for consistent quality using distortion-based Lagrange multiplier. IEEE Trans. Image Process. 25(7), 2943–2955 (2016)
Chen, Z., Pan, X.: An optimized rate control for low-delay H. 265/HEVC. IEEE Trans. Image Process. 28(9), 4541–4552 (2019)
Guo, H., Zhu, C., Xu, M., Li, S.: Inter-block dependency-based CTU level rate control for HEVC. IEEE Trans. Broadcast. 66, 113–126 (2020)
He, J., Yang, F.: Efficient frame-level bit allocation algorithm for H. 265/HEVC. IET Image Process. 11(4), 245–257 (2017)
Liu, M., Ren, P., Xiang, Z.: Frame-level bit allocation for hierarchical coding of H. 265/HEVC considering dependent rate-distortion characteristics. Signal Image Video Process. 10(8), 1457–1463 (2016)
Li, S., Xu, M., Wang, Z., Sun, X.: Optimal bit allocation for CTU level rate control in HEVC. IEEE Trans. Circuits Syst. Video Technol. 27(11), 2409–2424 (2016)
Wang, P., Ni, C., Li, Z., Zhang, G.: Optimal CTU-level bit allocation in HEVC for low bit-rate applications. Multimed. Tools Appl. 78, 23733–23747 (2019)
Zeng, H., Yang, A., Ngan, K.N., Wang, M.: Perceptual sensitivity-based rate control method for high efficiency video coding. Multimed. Tools Appl. 75(17), 10383–10396 (2016)
Zhou, M., Wei, X., Wang, S., Kwong, S., Fong, C.-K., Wong, P.H., Yuen, W.Y., Gao, W.: SSIM-based global optimization for CTU-level rate control in HEVC. IEEE Trans. Multimed. 21(8), 1921–1933 (2019)
Lim, W., Sim, D.: A perceptual rate control algorithm based on luminance adaptation for HEVC encoders. Signal Image Video Process. 14, 887–895 (2020)
Li, S., Xu, M., Deng, X., Wang, Z.: Weight-based R-λ rate control for perceptual HEVC coding on conversational videos. Signal Process. Image Commun. 38, 127–140 (2015)
Zhu, L., Wang, G., Teng, G., Yang, Z., Zhang, L.: A deep learning based perceptual bit allocation scheme on conversational videos for HEVC λ-domain rate control. In: International Forum on Digital TV and Wireless Multimedia Communications 2017, pp. 515–524. Springer
Gao, W., Kwong, S., Jia, Y.: Joint machine learning and game theory for rate control in high efficiency video coding. IEEE Trans. Image Process. 26(12), 6074–6089 (2017)
Gao, W., Kwong, S., Zhou, Y., Yuan, H.: SSIM-based game theory approach for rate-distortion optimized intra frame CTU-level bit allocation. IEEE Trans. Multimed. 18(6), 988–999 (2016)
Gao, W., Kwong, S., Jiang, Q., Fong, C.-K., Wong, P.H., Yuen, W.Y.: Data-driven rate control for rate-distortion optimization in HEVC Based on simplified effective initial QP learning. IEEE Trans. Broadcast. 65(1), 94–108 (2018)
Zupancic, I., Naccari, M., Mrak, M., Izquierdo, E.: Two-pass rate control for improved quality of experience in UHDTV delivery. IEEE J. Sel. Top. Signal Process. 11(1), 167–179 (2016)
Wang, S., Rehman, A., Zeng, K., Wang, J., Wang, Z.: SSIM-motivated two-pass VBR coding for HEVC. IEEE Trans. Circuits Syst. Video Technol. 27(10), 2189–2203 (2016)
Wien, M.: High efficiency video coding. Coding tools and specification. Berlin (2015)
Fani, D., Rezaei, M.: Novel PID-fuzzy video rate controller for high-delay applications of the HEVC standard. IEEE Trans. Circuits Syst. Video Technol. 28(6), 1379–1389 (2017)
Shojaei, M., Rezaei, M.: FJND-based fuzzy rate control of scalable video for streaming applications. Multimed. Tools Appl. 79, 13753–13773 (2020)
Yiming Li, Z.C.: Rate control for VVC JVET-K0390(ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11) (J2018)
Y.T. Peng, A.M.T.: Closed GOP support for the random access common conditions. JVET-D0135 (of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11) (2016)
Demuth, H.B., Beale, M.H., De Jess, O., Hagan, M.T.: Neural network design. Martin Hagan, New York (2014)
Suehring, K.: JVET common test conditions and software reference configurations. JVET-B1010(ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11) (2016).
VTM-1.0 reference software. https://jvet.hhi.fraunhofer.de/svn/svn_VVCSoftware_VTM/tags/VTM-1.0/. Accessed 15 Sept 2020
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Raufmehr, F., Salehi, M.R. & Abiri, E. A frame-level MLP-based bit-rate controller for real-time video transmission using VVC standard. J Real-Time Image Proc 18, 751–763 (2021). https://doi.org/10.1007/s11554-020-01018-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11554-020-01018-2