Modelling user quality of experience from objective and subjective data sets using fuzzy logic

In this experiment, test subjects evaluated the quality of 1-h video about the solar system in a home environment. We have used entertainment-oriented content selection [14], since we assumed that, while at home, the subjects usually watch video content that interests them. An additional reason for choosing the content that can entertain the subjects was the duration of the test, i.e., the video, and the need to capture the subjects’ concentration for a full hour. We believe that if different content would have been used, that would have been perceived by the subjects as, for instance, boring, then their willingness to participate in our study would decline. Note that our test conditions were unlike controlled environment experimenting in laboratory, where the tests usually last 20–30 min, during which observers rate the quality of several short video clips. In such shorter tests, it is easier to retain the subjects’ focus on the task at hand.

The experiment was performed using longer test sequences; thereby, we have acknowledged the findings presented in [15‐17]; the authors in that study demonstrate how QoE evaluation requires an increase in the duration of the test sequences, because user perception cannot be entirely shaped when using shorter video clips. The original video used in the experiment was encoded with Advanced Video Coding (H.264/AVC) and Advanced Audio Coding (AAC). The video bitrate, audio bitrate and frame rate were 9.8 Mbps, 256 kbps and 50 fps, respectively. The video resolution was 1920 × 1080 pixels. Note that the video contained video subtitles.

To create test sequences, first, the original video was streamed six times between two computers, which were connected in a peer-to-peer connection (Fig. 1). The Network Emulator Client was installed on Computer 1. The client dropped the packets on the outgoing stream toward Computer 2. PLR varied between 0.05, 0.1, 0.5, 1, 1.5 and 2%, while the burst packet loss length was set to 1. VLC Media Player was used to stream the video between the computers. Each incoming (degraded) video signal was stored on Computer 2 in the same format as the original video. During streaming of the video between the computers, UDP was used on the transport layer. This differs from Hyper Text Transfer Protocol (HTTP)-based streaming, which uses TCP (Transport Control Protocol). Nowadays, HTTP-based adaptive streaming is the relevant scenario; however, UDP-based streaming is still used for delivering Internet Protocol Television (IPTV), in particular for those services that use set top boxes.

Second, we have imported the stored video signals into CyberLink Power Director and extracted 1, 4, 7 or 10 short video clips from a degraded video signal and inserted them into the original video signal. The duration of a single inserted clip, i.e., a single packet loss occurrence (PLO), varied between 1, 4 and 7 s. By varying the number of inserted PLOs and the duration of a single PLO, we were able to generate different total durations for all PLOs in a test sequence that equaled 1, 4, 7, 10, 16, 28, 40, 49 or 70 s. The total duration is derived by multiplying the number of inserted PLOs in a sequence (1, 4, 7 or 10) with the duration of a single PLO (1, 4 or 7 s). When selecting these particular values of the parameters, the objective was to generate a wide range of total duration of the distortions in the sequences. Additionally, we wanted to create sequences with equal total duration of distortion while containing a different number of inserted PLOs and different durations of a single PLO. For instance, two sequences can contain 28 s of total quality distortions, but one can have 7 PLOs each lasting 4 s, while the other has 4 PLOs each lasting 7 s. This enabled us to evaluate the impact of each parameter individually.

In [18], the authors revealed that when the distortions are grouped into the first few minutes of the video, the quality scores of the subjects are observed to increase. Conversely, if the distortions are grouped into the last few minutes, the scores are observed to decrease. This kind of subject reasoning is influenced by human short-term memory [19] and the psychological effect of recency [18, 20]. Thus, the PLOs were evenly distributed over the entire duration of all test sequences. However, in each test sequence, the first and last 7 min and 17 s were unaffected by the degradations, which allowed the test subjects to get involved with the content in the beginning of the session as well as critically think about the audiovisual quality toward the end.

In this study, the methodology used for the subjective evaluation of the sequences was adopted from [5]. Thus, the sequences were distributed to the subjects on a DVD. This format was chosen due to the following reasons: (a) the test sequences were easily distributed to the subjects; (b) DVD players are more available compared to Blu-ray players (this was important since the survey was conducted among a student population); (c) DVD disks are cheaper compared to, e.g., memory sticks; and (d) according to [21], while evaluating other services, test subjects often use the quality of DVDs as a reference.

During the encoding of the test sequences to the DVD format, the PAL system, MPEG-2 video encoding format and variable bitrate encoding method were used. All video encoding settings were set to maintain the best possible video quality. Furthermore, all video enhancement features of the CyberLink Power Director software were turned off and the software did not use any error concealment methods. The resulting video bitrate, audio bitrate and frame rate were equal to 9.51 Mbps, 256 kbps and 25 fps, respectively.

Although we have used the settings which allowed the best possible video quality for the conversion to the DVD format, this process downgraded the quality of the sequences, since the original video (encoded with H.264/AVC) was re-encoded into the MPEG-2 format. In general, this decline in the video quality is hard to notice in low-motion scenes but can be noticed in high-motion scenes when moving objects, appearing in a scene, can become blocky. The difference in the quality becomes even more visible to the subjects if they are provided with the original sequence for the comparison (that was not the case in our study). Since we conducted the experiment using the documentary film, most of the scenes in the film were low-motion scenes (for instance, the presenter’s monologue or dialogue with other persons appearing in the film). Thus, the quality degradation caused by the re-encoding process was not apparent. Furthermore, we need to highlight that in our test conditions, which were mimicking a lifelike viewing experience, the subjects were not focused on keeping track of the video quality, they were focused on the content instead. Thus, we believe that the decline in the video quality due to the re-encoding of the video did not impact the QoE of our test subjects.

A fuzzy-based no reference objective video quality assessment model for assessing user QoE, developed in this research, correlates the values of the three objective parameters (discussed in Sect. 2.1) with the subjects’ perception of video quality. Hence, the focus of this section is to report how the subjective data set was collected, needed for the development of the inference system of the model.

The results obtained from the survey were grouped into different categories. Specifically, we have conducted analysis of user QoE for each test sequence, tested and confirmed the IQX hypothesis [26], and examined the relationships between the level of user annoyance and PLR, number of PLOs and total duration of all PLOs in a sequence. These relationships will be later used in the fuzzification process (in Sect. 5). Furthermore, we have investigated the impact of human short-term memory and the recency effect, correlated user QoE with their level of entertainment and fatigue, and analyzed the impact of social context and video subtitles on user QoE.

The user QoE analysis indicated that the subjects’ MOS remained reasonably high (always above 4, on a scale from 0, being bad quality, to 10, being excellent quality), even for those sequences that contained the most PLOs. This confirmed the findings presented in [5, 6, 27], where different authors underlined how the results of the uncontrolled experiments suggest that the subjects are not so negatively affected by the perceived quality degradations. This has direct implications on the inference system of the model and its output. Specifically, Sect. 7 will demonstrate how the model assesses a user QoE of 4.48, even for the most degraded test sequence. Detailed analysis of user QoE showed that when there is only one PLO in a 1-h video, the PLR and the duration of a single PLO do not affect user QoE. For PLRs ≥ 1%, a quality degradation that lasts ≥ 16 s can be negatively perceived by users. Furthermore, if the video contains 7 or more PLOs and PLR increases (≥ 1.5%), the duration of a single PLO comes to the fore. The analysis also revealed that, for PLRs of ≥ 0.5%, an increase in the number of PLOs significantly influences user QoE.

After confirming the IQX hypothesis, we ranked the objective parameters by their order of importance in relation to their impact on user QoE as follows: (1) total duration of quality distortions in a video, i.e., total duration of PLOs; (2) number of PLOs; (3) PLR; and (4) duration of a single PLO.

The impact of human short-term memory [19] was tested by comparing the number of PLOs in a specific sequence with the number of perceived PLOs reported by the subjects. The analysis revealed that a considerable number of test subjects (408) failed to notice and/or memorize some or even all quality distortions inserted in the sequences. This can be related with three casualties. First, longer test sequences were used in the test. Presumably, after 1 h, some subjects forgot the degradations, which they may have noticed while watching the DVD. Second, experimenting in the home environment of the subjects encouraged them to watch the video in everyday conditions (at a known location, with or without company, at any time of day). In these lifelike viewing conditions, the subjects were not focused on noticing and memorizing the PLOs; they were focused on the content instead. Third, the subjects were uninformed about the purpose of the test. Thus, before watching and during the video, they were unaware of the degradations that would appear in a sequence. This inability to notice and/or memorize PLOs impacted the subjects, which was manifested as high MOS even for the most degraded test sequences (as previously discussed). We have also discovered how the PLOs that appeared in the middle of the video were more often unreported by the subjects compared to those PLOs that appeared toward the end of the video. This confirmed the impact of the recency effect [20] on our test subjects. Notwithstanding, since the test sequences used in this experiment lasted 1 h, we cannot neglect that these results may differ if shorter test sequences had been used (or other types of content).

It is worth mentioning that the results also disclosed how the overall user experience can be redeemed despite the perceived quality distortions if the content is entertaining to the viewer. Finally, a separate analysis was conducted to see if the video subtitles could draw viewers’ attention to the bottom of the screen, thus making the PLOs harder to notice. It was observed that the subjects who watched the video with subtitles noticed fewer PLOs and achieved higher QoE compared with the subjects that watched the video without subtitles. However, we emphasized that further investigation of the impact of video subtitles on user QoE is needed.

In terms of network-related parameters, which may impact user audiovisual perception, it can be observed that we have limited our research quest to the effect of packet loss-related issues on user QoE. The inclusion of more parameters, such as network delay, jitter, and throughput, increases the number of test sequences. We have created 72 test sequences just by combining different values of the three parameters (PLR, number and duration of PLOs in a test sequence). A larger number of test sequences would mean that we would have to reach more test subjects, which was considered unfeasible. Note that each of the 602 test subjects evaluated one test sequence by watching it once (as discussed in Sect. 2.2.2). If the subjects watched the same sequence more than once before completing the questionnaire or if they watched another sequence for the first time, they would have known the purpose of the test while watching the video. We underlined earlier that the goal was to avoid that since knowing the purpose of the test would make our test subjects more perceptive to the PLOs.

Apart from the sheer number of test subjects, it would be difficult to pinpoint the effect of packet loss on user QoE if other network-related parameters had been tested as well. Hence, it can be argued that not all the parameters were tested against the subjects’ perception, yet the impact of packet loss on a wide number of subjective parameters was tested meticulously.

When critically thinking about the test conducted in this study, it has to be taken into account that the methodology had to meet the following demands:

The methodology discussed in this chapter met all of the above demands and highly impacted the obtained results. While watching the video at their homes, in a familiar environment, possibly surrounded by known people, the test subjects were not focused on keeping track of the video quality fluctuations. We can also assume that the subjects were able to relax and were entertained by the video content (on a scale from 0, being least entertaining, boring, to 10, being very entertaining, the average level of entertainment was 7.62 with a margin of error of 0.15 and a confidence level of 95%). This home test environment made the subjects more forgiving to the perceived quality distortions, which was mirrored in the results. Hence, the test environment directly affects the inference system of the model.

We are aware that the methodology has certain disadvantages. Namely, the success of such uncontrolled experiment largely depends on the honesty of the test subjects. Moreover, the test was conducted in environments, where a number of QoE influential factors may impact the subjects’ rating (as discussed in [3]). We were not able to investigate all the factors on such a large target group. However, we invested effort in (a) clearly presenting what is expected from the test subjects in the study; (b) designing a questionnaire that returns enough information for the modelling; (c) discovering and rejecting outliers from the sample; and (d) removing those questionnaires, where the subjects’ answers indicated equipment malfunction or noisy environments that may have interfered with the viewing experience.

The statistical analysis of the collected data (conducted in [4]) and the obtained results that confirmed the findings of other authors allows us to argue that the objective of the test was achieved and its outcomes can be further used for the development of the model.

In this step, the fuzzification of the crisp values shown in Figs. 2, 3, and 4 was conducted using the Fuzzy C-Means (FCM) clustering approach [55]. The objective of using this procedure was to group the data points presented on the figures into fuzzy clusters and to find centers of those clusters. Note that the FCM method allows the clusters to overlap. Therefore, the \(i\)th data point (\({x_i}\)) can be a member of several clusters (\(j\)) with different degrees of membership (\({u_{ij}}\)). According to [55], the method requires minimizing the objective function \({J_m}\):where \({c_j}\) denotes the \(d\)-dimension center of the cluster, \(||{\text{*}}||\) is any norm expressing the similarity between any measured data and the center, and \(m\) is any real number > 1. Note that in this study, \(m=2\). Using this method for grouping the data points from the figures into fuzzy clusters and finding the centers of those clusters assumed the iterative process, where the \({u_{ij}}\) and the \({c_j}\) are updated with Eqs. 2 and 3, respectively. The iterative process finishes when the stopping criteria (\(\varepsilon\)) from Eq. 4 is met (\(k\) are the iteration steps). In our case, we used the value of \(\varepsilon\) that is set by default in MATLAB to \({10^{ - 5}}\).

For each of the three input parameters, the subjects’ responses are grouped into three fuzzy clusters. The procedure required 24, 69 and 33 iterations before the stopping criteria \(\varepsilon\) was met and the centers of the clusters were defined for the PLR, Number of PLOs and Total duration of PLOs, respectively. The results are presented in Figs. 5a, 6a, and 7a, respectively. Note that thickly printed plus, × and circle signs on the figures represent the cluster centers, and their coordinates can be found in Table 2.

The final step toward defining the boundaries of the clusters was taken when the values of the data points on the \(x\) axis were correlated with their degrees of membership \({u_{ij}}\) to the specific cluster \(~j\) (see the Appendix 2, Figs. 11, 12, 13). In his book about fuzzy logic and its engineering applications [54], Ross reports how the normal distribution corresponds better with the changes in human perception, because the transitions between different opinions and attitudes are usually happening gradually (“smoothly”). Thus, bell-shaped functions can mimic that behavior better compared with, for instance, triangular functions. For this reason, we used the normal distribution for the approximation of the membership functions. Note also that the “smooth” transitions in human perception were captured with the use of continuous 11-point numerical scales for subjective data collection. The “smoothness” would be lost if the five-level discrete scales would have been used or questions with two-alternative options (as discussed in Sects. 2.2.1 and  4).

The obtained functions are presented in Figs. 5b, 6b, and 7b. Since the fuzzy systems use linguistic variables to infer conclusions, each cluster presented on the figures is named. These linguistic variables will be used in Sect. 6.1 for defining a set of fuzzy rules of the model.

Properties of the fuzzy membership functions shown in Figs. 5b, 6b, and 7b can be found in Table 3. Note that the dashed and the dashed–dotted Gaussian functions of each input parameter had to be modified so that \({u_{ij}}\) would be equal to 1 in cases when \({x_i}<\bar {x}\) (for the dashed functions) and \({x_i}>\bar {x}\) (for the dashed-dotted functions).

The mean values (\(\bar {x}\)) of the Gaussian functions depicted in Figs. 5b, 6b, and 7b correspond with the \(x\)-coordinates of the centers of the clusters presented in Table 2 and Figs. 5a, 6a, and 7a, respectively. This means that the data points have a higher degree of membership (\({u_{ij}}\)) to a specific cluster \(~j\) if they are located closer to the center of that cluster. Therefore, the shift in the sample population mean in this case is actually impacted by the number of clusters and the location of their respective centers.

We can observe how the fuzzy membership functions are overlapping due to the dispersion of the data points around the centers of the clusters. The overlapping of the membership functions is a key feature of the FCM procedure since it enables grouping a specific data point into more than one cluster. This is important, because it is not possible to unambiguously define, for instance, whether PLR of 1% is causing imperceptible, slightly or very annoying quality distortion from a user point of view. In fact, our results show that this particular PLR is a member of all three clusters (Fig. 5b) with different degrees of membership.

The most distinctive overlapping of the membership functions is present for the first input parameter (Fig. 5b). The stretch of these functions (by a factor of \(\sigma\)) shapes their slopes, which are not so steep compared with the functions depicted on the other two figures (Figs. 6b, 7b). This is due to the previously discussed fact that certain PLRs were evaluated differently by the subjects, depending on the number of PLOs in a test sequence and their total duration. These results give us solid arguments for inclusion of the three parameters in the inference system of the model since they clearly create an affiliated effect on user perception.

We noted in Sect. 4, how the subjects evaluated the quality of the video and their viewing experience on an 11-point quality scale that also contained linguistic meanings of the specific ratings. These linguistic meanings were used to name the fuzzy clusters of the output parameter. However, to increase the accuracy of the model, instead of five clusters, the output parameter is modelled with eight clusters. While experimenting with different settings of the model, we have observed that it is better to increase the number of clusters of the output parameter, because the model became more responsive to the changes of the input values and assessed the QoE with more accuracy.

The membership functions of the eight fuzzy clusters of the output parameter are depicted in Fig. 8. Again, the Gaussian functions are used (their properties can be found in Table 4). For the clusters bad and excellent quality, the functions are modified so that \({u_{ij}}\) would be equal to 1 in cases when \({x_i}<\bar {x}\) and \({x_i}>\bar {x}\), respectively. Note that the membership function of the excellent quality cluster has \(\bar {x}\) = 8.44, so that the ratings ≥ 8 would have a higher degree of membership to this cluster compared with the cluster good quality 2.

The inference system of the model is based on a set of 24 fuzzy rules that are listed in Table 5. The linguistic values of the input and the output parameters are defined earlier, in Sects. 5.1 and 5.2, respectively. The values are linked using the IF, AND THEN logical operators.

The rules are defined after several iterations, where we experimented with different settings of the model. The number of fuzzy clusters for the input and the output parameters was changed multiple times, and different properties of the fuzzy membership functions were tested (\(\bar {x}\) and \(\sigma\)). This was done for the purpose of making the model more responsive to the changes in the input parameters, thus achieving better correlation of the model output with the MOS of the users. The conclusions of each rule were also changed and tested multiple times for the same purpose.

While observing this set of fuzzy rules, it is worth remembering how the methodology used for the subjective evaluation of the sequences impacted the results discussed in Sect. 2.3. The test was taken by the subjects in an uncontrolled environment, longer test sequences were used, the subjects were unaware of the purpose of the test, and they were mostly entertained by the content as well as the video containing subtitles. The test conditions made the PLOs harder to notice/memorize and also raised the subjects’ tolerance toward the distortions. Thus, the MOS remain reasonably high for all test sequences, and this is here reflected in the rules. For instance, when the last rule listed in Table 5 is activated (very annoying quality distortions of all three input parameters), this leads to the consequence QoE = poor quality 2 (the \(\bar {x}\) for this set equals 3.5). Thus, even the most degraded sequences are not rated rigorously.

The output of the model, the QoE rating, is calculated in MATLAB using methods that are thoroughly discussed by Ross in [54]. The mathematical descriptions of these methods (Eqs. 5, 6, and 7) are based on [54] and can be found below.