To the best of our knowledge, there is no benchmark available in the industry that is suitable for comparison of video encoding solutions in terms of performance, power, quality, and amount of compression. However, there is a well-known academic effort carried out by Moscow State University (MSU) to compare available codecs. This academic analysis is able to rank various software-based and/or hardware-accelerated encoder implementations in terms of objective quality measures. Obviously, it is possible to tune the parameters of an encoder to achieve higher coding efficiency, higher performance, or lower power use, resulting in a different ranking.
The discussion of this Moscow effort is followed by short descriptions of common industry benchmarks, which are generally limited to power and performance evaluations and do not consider other aspects of video coding. However, it is possible that new benchmarks will be suitable for a wider ranking of video encoding. Also included in this appendix is a brief list of suggested reading materials. Although existing references do not cover tradeoff analysis methods and metrics, they have in-depth discussions of certain topics only briefly mentioned in this book.
Anzeige
MSU Codec Comparison
A codec comparison project supported by the Computer Graphics and Multimedia Laboratory at Moscow State University compares the coding efficiency of various codecs.1The goal of this project is to determine the quality of various H.264 codecs using objective measures of assessment. The annual project reports are available from 2003 to 2012.
In the most recent comparison, done in 2012, the following H.264 encoders were compared:
-
DivX H.264 software
-
Elecard H.264 software
-
Intel QuickSync H.264 encoder using Intel third-generation Core processor graphics
-
MainConcept H.264 software
-
MainConcept CUDA based H.264 encoder
-
XviD MPEG-4 Advanced Simple Profile software
-
DiscretePhoton software
-
x264 software
The contents of various complexities with resolutions ranging from 352×288 to 1920×1080 were used, including 10 standard-definition, 16 high-definition (HDTV), and five video-conferencing sequences. The PSNR, SSIM, and MS-SSIM were used as the comparison objective metrics on all the color planes Y, U, and V for all frames in the video sequences. In making the comparisons and ranking the encoders, the following facts were recognized:
-
For an encoder, the output visual quality is not the same for different frames of the same video sequence. Thus, a fair comparison would consider whether the same frames are being compressed by the various encoders. Frame mismatch can easily make a difference in quality.
-
Different encoders are tuned to different content types. In particular, the default settings of an encoder may be best suited for a certain content type or video usage model. Therefore, comparing encoders with default settings may not necessarily be fair.
-
Compression quality considerably depends on coding parameters. Setting appropriate coding parameters based on practical usage models is important in obtaining a realistic evaluation of various encoders.
To make a fair comparison, codec settings provided by the developers of each codec were used. The target application was video transcoding, mainly for personal use. The fast presets were taken to be analogous to real-time encoding on a typical home-use personal computer.
Anzeige
The 2012 report ranked the eight codecs by considering the overall average achieved bit rates for approximately the same quality, and presented the following ranking based on this measure alone, without regard to encoding speed. Table A-1 shows the ranking:
Table A-1.
MSU Codec Ranking
Rank | Codec | Overall Average Achieved Bit Rate for the Same Quality (in percentage of XviD bit rate, lower is better) |
|---|---|---|
1 | x264 | 51 |
2 | MainConcept H.264 Software | 62 |
3 | DivX H.264 | 69 |
4 | Elecard H.264 | 71 |
5 | Intel QuickSync (3rd -gen. Core) | 93 |
6 | XviD | 100 |
7 | DiscretePhoton | 121 |
8 | MainConcept CUDA | 137 |
While this comparison is useful to some extent, note that only the quality aspects are considered here, regardless of performance and power consumption tradeoffs. This is a weakness of this comparison methodology; choosing different parameters for an encoder could easily provide different coding efficiency than is used for the ranking.
The tradeoffs and methodologies discussed in this book are important for getting an understanding of the big picture. Comparison of encoders should always acknowledge and take into account the various options considered by the encoders for different usage models. An encoder implementation with default settings may work better than one for video conferencing, but may not be as good for transcoding applications. However, the encoding parameters exposed by an implementation may be tuned to obtain better quality or performance. Note that different encoders give different controls to the end-users. Knowledge of parameters for an encoder is necessary to achieve best results for particular scenarios.
Industry Benchmarks
Some common benchmarks in the industry are occasionally used by enthusiasts to compare processors and their graphics and video coding capabilities. Although these benchmarks may include some video playback tests, they are not generally suitable for comparing video encoders, owing to their limited focus. Nevertheless, a few such benchmarks are briefly described below. It is hoped that points made in this book will inspire establishment of benchmarks that overcome this shortcoming and eventually reflect a higher state of the art.
MobileMark 2012
MobileMark 2012 from BAPCo is an application-based benchmark that reflects patterns of business use in the areas of office productivity, media creation, and media consumption. In addition to battery life, MobileMark 2012 simultaneously measures performance, showing how well a system design addresses the inherent performance and power management.
Unlike synthetic benchmarks, which artificially drive components to peak capacity or deduce performance using a static simulation of application behavior, MobileMark 2012 uses real applications, user workloads, and datasets in an effort to reflect the battery life a user might experience when performing similar workloads. MobileMark is commonly used by PC OEMs, hardware and software developers, IT departments, system integrators, publishers, and testing labs, as well as information technologists and computer industry analysts.
However, MobileMark is targeted to run business applications such as Microsoft Office, and uses Adobe Premiere Pro CS5 and Adobe Flash Player 11 to perform the video processing tasks. While this provides an indication of system design and CPU capabilities, it does not take advantage of the GPU-acceleration opportunities available in modern systems. Furthermore, it does not take into account the visual quality of compressed video. Therefore, this benchmark is very limited in its scope.
PCMark and PowerMark
These benchmarks were developed by FutureMark. PCMark is a standard performance benchmarking tool for personal computers of various form factors. With five separate benchmark tests and battery life testing, it can distinguish the devices based on efficiency and performance. It allows measurement and comparison of PC performance using real-world tasks and applications. The applications are grouped into scenarios that reflect typical PC use in the home and office environments.
PowerMark is a battery life and power consumption benchmark designed for professional testing labs. It delivers accurate results from realistic productivity and entertainment scenarios.
Both PCMark and PowerMark have limitations similar to those of MobileMark, as they consider performance or power alone and do not incorporate appropriate tradeoffs. Therefore, using only these benchmarks for ranking video encoders is not sufficient.
GFXBench
GFXBench, previously known as GLBenchmark and DXBenchmark, is a unified 3D graphics performance benchmark suite developed by Kishonti Ltd., who also developed CompuBench (formerly CLBenchmark) for CPUs. It allows cross-platform and cross-API comparison of GPUs in smartphones, tablets, and laptops. GFXBench 3.0 is an OpenGL ES 3 benchmark designed for measuring graphics performance, render quality, and power consumption in a single application. It utilizes OpenGL ES 3 capabilities, such as multiple render targets for deferred rendering, geometry instancing, transform feedback, and so on. It generates relevant workloads and measurement targets on different graphic performance aspects.
However, GFXBench does not deal with natural or synthetic video playback, recording, transcoding, video coferencing, screencast, or similar workloads. Furthermore, 3D graphics such as video games are primarily concerned with real-time performance and good graphics render quality, while video encoding and transcoding online and off-line applications may benefit from faster than real-time performance and an acceptable level of playback quality. Since GFXBench does not consider compressed video or bit-rate variations in quality measurements, it is difficult to ascertain the actual cost of quality. In addition, GFXBench does not report the package power, leaving open the possibility of large variations in power consumption from use of peripheral devices, while the processor package power may have been quite stable.
Therefore, the current version of GFXBench is not sufficient for measuring video applications in terms of power, performance, and quality. Yet, it is encouraging to see some commercial tool developers starting to think in terms of performance, power, and quality; perhaps future versions of GFXBench will fill the gaps that exist today in tools and benchmarking areas.
Suggested Reading
Here are a couple of academic research efforts that may be of interest.
-
H. R. Wu and K. R. Rao, eds., Digital Video Image Quality and Perceptual Coding (Boca Raton, FL: CRC Press, 2005).
Perceptual coding techniques discard superfluous data that humans cannot process or detect. As maintaining image quality, even in bandwidth- and memory-restricted environments, is very important, many research efforts are available in the perceptual coding field. This collection of research, edited by H. R. Wu and K. R. Rao, surveys the topic from a HVS-based approach. It outlines the principles, metrics, and standards associated with perceptual coding, as well as the latest techniques and applications.
The collection is divided broadly into three parts. First, it introduces the basics of compression, HVS modeling, and coding artifacts associated with current well-known techniques. The next part focuses on picture-quality assessment criteria; subjective and objective methods and metrics, including vision model-based digital video impairment metrics; testing procedures; and international standards regarding image quality. In the final part, practical applications come into focus, including digital image and video coder designs based on the HVS, as well as post-filtering, restoration, error correction, and concealment techniques.
This collection covers the basic issues and concepts along with various compression algorithms and techniques, reviews recent research in HVS-based video and image coding, and discusses subjective and objective assessment methods, quantitative quality metrics, test criteria, and procedures; however, it does not touch on performance, power, or tradeoff analysis.
-
Ahmad and S. Ranka, eds., Handbook of Energy-Aware and Green Computing (Boca Raton, FL: CRC Press, 2012).
Some power-efficient techniques from various systems points of view, including circuit and component design, software, operating systems, networking, and so on, are presented in this book by Ahmad and Ranka. It is not specific to video applications; however, this two-volume handbook explores state-of-the-art research into various aspects of power-aware computing. Although one paper in the handbook discusses about a particular approach to mobile multimedia computing, future researchers may find some of the other optimization aspects and techniques useful in the general area of video encoding as well.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits any noncommercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this chapter or parts of it.
The images or other third party material in this chapter are included in the chapter’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.