A real-time video quality estimator for emerging wireless multimedia systems

The last few years have witnessed a phenomenal growth in the wireless industry, both in terms of multimedia mobile technology and its human-centric subscribers. The current trends and demands in wireless communications require the delivery of real-time video applications over heterogeneous wireless networks with Quality of Experience (QoE) [1, 2] support. Video streaming will provide new sources of income for network operators and content providers, since it will be a major application in future wireless systems and a key factor in ensuring their success [3, 4]. At the same time, users have been producing, sharing, and accessing thousands of video services on wireless devices.

Real-time multimedia traffic consists of one or more media streams with different spatial and temporal (motion and complexity) video activities and features. A Group of Picture (GoP) is a group of successive pictures within a coded video stream and composed of a combination of three frame types for the compressed video streams, namely I (Intra-coded), P (Predictive-coded), and B (Bidirectionally predictive-coded) frames. It is important to highlight that not all video frames are equal or have the same degree of importance from the user’s point-of-view. For instance, depending on the video motion and complexity levels (e.g., a small moving region of interest on a static background or fast-moving sports clips) and the GoP length, the impact of a packet lost in the Human Visual System (HVS) may or may not be annoying [5].

Different types of wireless technologies, such as Wireless Mesh Networks (WMNs) [6], can be used to deliver a wide range of real-time video streaming services to a large number of users. However, video streaming produces a degraded performance in wireless systems, including in WMNs, due to network/channel impairments, such as packet loss [7, 8]. Understanding and modeling the relationship of network impairments, video characteristics, and human experiences by using wireless quality level assessment schemes are key requirements for the delivery of visual content in multimedia mobile networks, such as football matches and other live multimedia events [9]. The operators that assess the QoE of real-time video services have a significant advantage by being able to strike an ideal balance between network provisioning, video codec configuration, and user’s experience.

Solutions for assessing the QoE of a video service can be organized as subjective or objective [10], where the latter is hard to implement in real-time. Objective video media quality assessment technologies are categorized into several parametric model types [9], where packet-layer schemes have been gaining attention due to their high accuracy and low processing. Packet-layer models predict the perceived video quality level based on information about the IP and video codec headers, such as frame type and packet loss rate (without decoding the video flow). The impact on user perception of video flows is influenced by the number of the edges (spatial information—complexity level) in the individual frames and by the type and direction of movement (temporal information—motion level) in a GoP. However, existing solutions have not been implemented and evaluated in multimedia wireless systems, or presented inaccurate results from the user’s experience, because, mainly, they do not consider the video motion and complexity levels in their assessment procedures.

This article extends our previous work [11] with a modular and parametric packet-layer wireless video quality estimator, called MultiQoE. MultiQoE uses IP and MPEG packet header information to predict the quality level of a variety of videos (different genres and content), which reduces the system complexity and processing. Without decoding the videos, MultiQoE estimates the quality of level live video sequences by orchestrating and mapping information on networks’ impairments, videos’ characteristics, and users’ perception into a predicted Mean Opinion Score (MOS). In contrast to existing works, MultiQoE also uses information on motion and complexity levels of the video frames in a GoP to improve the system accuracy. Simulation experiments with the assistance of real viewers were carried out to demonstrate the benefits and evaluate the efficiency of MultiQoE in a wireless mesh multimedia network. The results show that MultiQoE predicts the quality level of a set of videos closest to the human experience when compared to other widely-used QoE video quality estimator models, such as the Peak Signal-to-Noise Ratio (PSNR) [12], Video Quality Metric (VQM) [13], Structural Similarity Index Metric (SSIM) [14], and Pseudo-Subjective Quality Assessment (PSQA) [15].

The remainder of the article is structured as follows. Section 2 describes the related works. The MultiQoE proposal is explained in Sect. 3. Section 4 presents the test environment, scenario, implementations, and simulation results. Some concluding remarks are summarized in Sect. 5.

The ITU-R Recommendation BT.500 [16] has defined subjective assessment as the most reliable system of Video Quality Assessment (VQA). Subjective methods measure the overall perceived video quality, under well-defined and controlled conditions, by asking observers to evaluate videos [17]. However, subjective assessment is cumbersome, expensive, and unsuitable for in-service and real-time applications [18, 19]. The most widely-used subjective scheme for video quality evaluation is MOS [16, 20] which is recommended by the ITU-Telecommunication (ITU-T) Standardization Sector. The MOS rates the video quality on a scale of 1–5, where 5 is the best possible score. It should be noted that in the tests, observers tend to avoid scores at the extreme end of the scale (1 or 5) due to the influence of psychological factors [21]. However, MOS is not suitable for real-time video assessment approaches.

In ITU-T Study Group (SG) 12, there is a study [22] on the non-intrusive objective parametric and well-structured QoE assessment models (e.g., G.OMVAS [22], P.NAMS [23] and P.NBAMS [24] as planning, packet-layer and bit stream models, respectively) that can predict the perceptual impact of network impairments on video applications, considering the kind of impairment caused by both transmission and video compression issues [7, 25]. The prediction is based on packet header information [26, 27] and prior knowledge of the media stream [28]. However, in practice, existing solutions [22‐27] have not been implemented and validated in wireless multimedia systems, where the mapping of packet/network information into MOS is required. MultiQoE follows the ITU-SG 12 recommendations, defines its specific input video/packet/network parameters, and validates an accuracy parametric video quality estimator solution for multimedia WMNs.

The most popular objective quality inference techniques include PSNR [12], VQM [13], and SSIM [14]. Although attempts to assess coding quality have often focused on estimating the PSNR, the PSNR by itself, does not always correlate well with perceived quality of the HVS [25, 28, 29]. PSNR can only be computed once the image is received, which is not appropriate for real-time prediction systems [29‐31].

VQM provides a better indication of perceptual quality than PSNR [32]. In general, VQM’s ability to loosely classify a sequence as ‘very poor’ or ‘very good’ is accurate, but it often fails to distinguish sequences that share similar levels of degradation. The VQM evaluation results vary from 0 to 5, where 0 is the best possible score. Another well-known metric is called SSIM, which is based on a frame-to-frame measurement of three components (luminance, contrast, and structural similarity). The SSIM index is a decimal value between 0 and 1, where 0 means there is no correlation with the original image, and 1 means it has exactly the same image. However, both VQM and SSIM metrics cannot be used in real-time and perform poorly compared to MOS.

A non-intrusive QoE parametric scheme, called PSQA, has been used to predict the quality level of videos flows [15]. The authors included a Random Neural Network (RNN) model (together with its learning algorithms) to assess the quality level of videos in real-time based on a set of parameters, including frame type, frame rate, and packet loss rate. The proposed solution was originally proposed to improve the understanding of Quality of Service (QoS) factors in multimedia engineering without an in-depth understanding of actual user experience (lack of QoE support).

To assess the QoE of Multiple Description Coding (MDC) videos over multiple overlay paths, the proposal [33] was compared to subjective (e.g., MOS), objective (e.g., PSNR), and non-intrusive parametric approaches (e.g., PSQA). MDC-PSQA extends PSQA, together with QoE prediction, by also using the GoP length and the percentage losses of the I, P, and B frames in a GoP. However, it only considers one video (Foreman—no motion and complexity variation), which makes the system less accurate in assessing the quality level of videos with different characteristics, such as those expected for the Internet. Furthermore, visual quality metrics must be tested on a wide variety of visual contents and distortion types before meaningful conclusions can be drawn from their performance.

As presented in [5], the impact of the video quality level on HVS is highly influenced by the number of the edges (spatial information—complexity level) in the individual frames and by the type and direction of movement (temporal information—motion level). However, in contrast to MultiQoE, the PSQA mechanism and its extensions do not consider a set of diverse videos (they use only one video) with different levels of spatial temporal activities during their training and prediction procedures which reduce the system accuracy for measuring the quality level of (many) Internet videos.

Another proposal investigates the dependence of video quality on numbers, expressed as MOS for a given set of QoS network parameters [34]. This work investigates the impact of key frames on the quality perceived by users in wireless systems. Unlike MultiQoE, it only considers one video flow (and not videos with different motion and complexity levels) and does not take the GoP length into account, which is an important input in a QoE evaluation system.

The solution proposed by Khan et al. [35] has classified videos into groups representing different content types (using a combination of temporal and spatial levels) and extraction features, by means of cluster analysis. Based on the content type, the video quality (in terms of MOS) was predicted from the network parameters (e.g., packet error rate) and application-level parameters (e.g., transmission bit rate and frame rate) by using Principal Component Analysis (PCA). In contrast to MultiQoE, the proposed solution measures the video quality level by applying the average PSNR to all the decoded frames which performs poorly compared to MOS. Other extensions of this work [3, 36, 37] failed to provide satisfactory MOS results because PSNR was still used to correlate the video’s characteristics and network’s impairments into human scores.

Few works have analyzed the impact of the distribution of videos with different motion and complexity levels over wireless networks according to the human’s perception (e.g., 38, 39). It is clear that the accuracy and performance of a non-intrusive parametric QoE video quality estimator is largely dependent on the video characteristics, including GoP length, frame type in a GoP, motion and complexity vectors. A QoE video quality estimator also requires a good mapping technique to correlate the video content features and wireless network’s impairments in the human scores, such as a cluster-based Artificial Neural Network (ANN) approach [40].

ANN has the ability to learn complex data structures and approximate a given continuous mapping. ANNs work fast (after a training phase) even with large amounts of data [41] and approximate a continuous mapping to any arbitrary degree of accuracy as expected for QoE-aware video prediction schemes. This means that they are suited to learning the salient characteristics of human perception [41] during the video quality estimation process. Existing image quality assessment schemes have demonstrated the benefits and feasibility of ANN-based models in predicting the quality level of images (not videos) [41‐46].

Many existing works only use simple objective QoE metric to validate their proposals or fail to take into account the fact that videos have different levels of motion and complexity. Another advantage of MultiQoE is the use of a Multiple ANN (MANN) correlation approach to map video’s characteristics, human’s perceptions, and network’s impairments into a predicted MOS with results close to human scores. MANN has been successfully used for QoS/QoE assessment schemes [3, 47, 48] and yielded better results than RNN and other techniques. MultiQoE has a realistic assumption that not all packets are equal or have the same degree of importance which are key parameters to determine the extension of the video impairment, as discussed in [5, 49] (see Sect. 3.5).

MultiQoE is a modular and flexible in-service parametric approach to predict the quality level of different video sequences, where it can be configured with any wired or wireless technologies with low complexity and high accuracy. However, due to the popularity of WMNs, the remainder of this article will present the use of MultiQoE in a multimedia WMN environment.

An instance of MultiQoE can be obtained by following the procedures defined in five main components. Each of them is designed to complete single or multiple tasks for the modeling of the quality evaluation model. The components of MultiQoE are illustrated in Fig. 1 and are as follows: (i) Source Video Database, (ii) Network Transmission; (iii) Subjective Quality Assessment and Distorted Video Database; (iv) Measurement Model of Factors Affecting Quality; and (v) Correlation of Video Characteristics, Human Experience, and Network Impairments into MOS.

The Source Video Database (Component 1) classifies typical Internet videos according to their spatial and temporal characteristics. The video content characteristics taken together with the percentage of losses of I, P, and B frames of a certain GoP (to improve the system accuracy, each ANN is responsible for videos with a specific GoP length, such as 10, 20, or 30) are used by Component 4 (Measurement Model of Factors Affecting Quality) to identify the video motion and complexity levels as well as the impact of the transmission on the video frames. At the same time, it is important to keep a distorted video database composed of videos delivered (as expected to be received/viewed by humans) in real/simulated networks (such as WMNs).

The Component 2 (Network Transmission) is responsible for transmitting all videos in wireless networks (with different congestion levels, errors, and impairments), getting information on packet loss and delay of video frames. The output of this component is important to create a distorted video database, where the videos experienced different network impairments, as well as measure the percentage of losses of the I, P, and B frames in a GoP as specified in Component 4. In the Component 3 (Subjective Quality Assessment and Distorted Video Database), a panel of humans evaluates all distorted videos (following the ITU recommendations) to define/score their MOS. Finally, Component 5 uses a MANN to correlate video’s characteristics, human’ experience, and network’s impairments into a predicted MOS.

This section demonstrates the benefits of MultiQoE in a practical environment. This was achieved by employing a use case in an IEEE 802.11-based WMN system, which was implemented together with MultiQoE, to measure the quality level of real video sequences distributed in WMNs. The efficiency of MultiQoE is compared to widely-used QoE metrics such as PSNR, VQM and SSIM, as well as, PSQA and MOS collected from real observers. Our results also rely on two key estimation methods, namely Pearson Correlation Coefficient (PCC) [67] and Mean Squared Error (MSE) [31] as recommended by the VQEG [68, 69].

The simulated scenario uses the topology and characteristics of the WMN backbone installed at the UFPA campus in the Amazon/Brazil. The setting consists of several buildings interspersed with parking areas and woodland. The topology is composed of 6 mesh routers, 2 of which are gateways, as depicted in Fig. 4. In addition, a mesh client was positioned to receive the video streaming from either Gateway 1 (G1) or Gateway 2 (G2). The client experienced different packet loss rates because during the tests the user location and wireless resource conditions/impairments were changed at random (up to 90 % of network congestion—see Fig. 5). Ten widely used Internet-based videos were chosen with different patterns and characteristics (duration, content types, and GoP length) as explained in Sect. 3.1.

The experiments were carried out by using Network Simulator 2.34 [70], Evalvid Tool [71], MSU Video Quality Measurement Tool [72], and the MANN was built with the aid of Matlab. The distorted video database was split into two subsets: one for training and another one for testing the generalization performance of the trained system. With this procedure, it is possible to make sure that the set of videos of the training set and test set come from disjoint video sequences with quite different video content. Each selected video was simulated 90 times (by varying the network packet loss rate and GoP lengths—10, 20, and 30) to provide a large video database and this resulted in a total of 900 videos.¹ 810 and 90 videos were randomly selected from this database for the training and test databases, respectively. Table 3 outlines other parameters used in the experiments.

While conducting the subjective evaluation tests, we followed the ITU-T MOS recommendations (with 55 observers) to obtain accurate results. The observers included undergraduates, post-graduate students, and university staff. The test platform is the same as that described in Sect. 3.4.

Figure 9a shows the results obtained when the WMN is configured with MultiQoE. It can be observed that MultiQoE and MOS achieve similar scores, while PSQA does not correlate well with MOS (Fig. 9b). The PSQA values produce scores not close to the MOS line because the parameters used as input (losses in I, P, and B frames and GoP length) and the correlation model based on a RNN, are not enough to predict the quality level of the videos accurately. Moreover, in contrast to PSQA, MultiQoE improves the procedures for estimating video quality by using the GoP length combined with a specific ANN (and the spatial/temporal (clustered) activities of the videos) as input for its correlation component. The results reveal that videos with varying content features have a different impact on the user’s perception even when the wireless channel conditions are similar.

Figure 10a–c show the results of PSNR, SSIM, and VQM, respectively, when compared with MOS. It is evident that there is a poor correlation between the objective metrics and MOS scores and that this does not reflect the user’s perception.

Further results show the benefits of MultiQoE in predicting the quality level of videos, by measuring the MSE values of both MultiQoE and PSQA for each type of cluster and GoP length (as shown in Fig. 11a, b). This analysis is important since it reveals the performance of QoE estimation methods for video clusters in scenarios with different congestion levels and packet loss rates. It is worth noting that Fig. 11a shows that MultiQoE has the lowest error, with only 1.08 × 10⁻³, for the LST cluster, while PSQA has 4.18 × 10⁻³. It should also be pointed out that MultiQoE has the best performance for clusters HSMT and MSHT with 1.88 × 10⁻³ and 2.31 × 10⁻³ compared with PSQA (5.19 × 10⁻³ and 6.98 × 10⁻³), respectively. Distortions in foreground areas, such as human faces in high motion and complexity videos, caused lower subjective ratings, while similar artifacts in the background went unnoticed for videos with low motion and complexity levels.

Figure 11b illustrates the performance of the MultiQoE and PSQA for all GoP lengths. In the case of GoP 10, the MSE results for MultiQoE and PSQA are 1.55 × 10⁻³ and 8.91 × 10⁻³, respectively. For GoP 20, MultiQoE presents a MSE of 3.09 × 10⁻³, while PSQA only 3.77 × 10⁻³. Finally, for videos encoded with GoP 30, the MultiQoE MSE is only 0.081 × 10⁻³, while PSQA is 1.09 × 10⁻³. For GoP 30, PSQA shows a better performance than PSQA, because the drop of I frames had a great impact on the video quality level. According to the MSE results, on average, MultiQoE improves the accuracy of the system by 32.22 % when compared with PSQA.

Figure 12 illustrates the PCC values for MultiQoE, PSQA, PSNR, SSIM, and VQM, where 1 indicates a perfect match between the predicted measurements and the subjective ratings and 0 indicates no correlation. The PCC coefficient obtained by MultiQoE is 0.922 (an improvement of 7 % compared with PSQA). When the system is configured to analyze the quality level of videos based only on PSNR, SSIM, and VQM, the PCC values are 0.132, 0.331, and 0.376, respectively (Fig. 12). The results confirm that the objective metrics achieved are poor compared with MOS, PSQA, and MultiQoE.

It can be observed that MultiQoE causes a low number of errors with subjective ratings for a variety of videos ranging from low activity such as Akiyo and News to high activity sequences, such as Football and Coastguard. MultiQoE also produces good results for video sequences that combine both low-activity and high-activity scenes, such as Silent and Flower. This is because MultiQoE uses the GoP length and the spatial/temporal (clustered) activities of the videos as input for its correlation component.

On average, the PCC result obtained by MultiQoE is 0.892 for the Grandma flow. When the system is configured to analyze the quality level of videos based only on PSNR, SSIM, VQM, and PSQA, the PCC values are 0.124, 0.348, 0.416 and 0.821, respectively. The results confirm that objective metrics perform poorly compared with those of MOS, PSQA, and MultiQoE. When Grandma is included in the source database and trained off-line, the PCC result is of 0.928. This is possible because, in addition to the benefits of MANN in identifying patterns of video sequences, which they were trained to deal with (as happened with Mother and Flower), and providing an accurate prediction model, MultiQoE uses a set of feed-forward back-propagation networks that are supplied with subjective MOS scores. These parameters enable MultiQoE to measure the quality level of videos even when they have different encoding patterns, content types, and network impairments/errors/congestions.

The evolution of wireless access technologies, services, and protocols has created a plethora of new human-centric environments featuring an ever-increasing amount of wireless devices and multimedia content. QoE assessment and control solutions allow network providers to keep and attract new customers, while optimizing network resource and enlarging their portfolios. Therefore, parametric in-service QoE assessment models are needed to ensure the success of multimedia wireless networks and have attracted a lot of attention from both academia and industry. MultiQoE provides a modular and non-intrusive video quality estimator implemented over wireless mesh systems that can be easily adapted to networks with different underlying technologies. MultiQoE works without the need for any decoding which saves time and reduces processing.

Our experiments investigated the relationship between video’s characteristics, human’s experience, network’s impairments, and predicted MOS scores. On average, the results show that MultiQoE has a high correlation with MOS (0.922) and a low MSE 1.75 × 10⁻³, while PSQA produced 5.45 × 10⁻³. The benefits of MultiQoE have also been highlighted when compared with PSNR, VQM, and SSIM. MultiQoE can be used together with optimization and management schemes to improve the usage of network resources, as well as system performance in key networking areas, such as pricing, routing, or mobility.

In future works, large-scale experiments will be conducted to investigate the impact of the proposed solution in networks with many users and a large set of video flows. Thus, it will be possible to analyze the MultiQoE computation cost to predict the quality level of user and content-generated video sequences over wireless networks with different channel modules and errors. An OpenFlow prototype with Cloud Computing support is being developed to evaluate the benefit of MultiQoE in production multimedia networks.