Dash Quality of Experience Open Source Evaluation Framework

The exponential growth of mobile data and video traffic has risen to many new challenges for the research community in recent years. 5G/future networks are expected to support high throughput, minimal end-to-end delay, and ultra-high-quality delivery for various purposes such as mobile broadcasting, remote surgery, and augmented reality.

Two of the main expectations of 5G networks are that they will handle ultra-high-definition (UHD) video streaming and deliver services that meet the requirements of the end user’s perceived quality by adopting quality of experience (QoE) aware network management approaches.

QoS defined as key network performance metrics; however, QoE referred to an actual individual user experience. Many questions need to be addressed for QoS to QoE mapping, for instance.

• Which are key patterns in QoS metrics that impact users’ QoE?
• Does the expected QoE hold under varying network conditions?
• During a video session, video streaming freezed often or even stopped, resulting in negative feedback from users, but which Key Performance Indicators (KPIs) caused the quality degradation?

To answer the above questions and many more, we present the Dynamic Adaptive Streaming over HTTP (DASH) Quality of Experience framework featuring a set of tools and dependencies to deliver realistic 5G network scenarios to observe QoS-to-QoE behaviours.

We present a reproducible data-driven framework to analyze state-of-art Adaptive Bitrate Streaming (ABS) algorithms by varying key network QoS parameters by replaying static and mobility 5G traces.

The workflow of the proposed framework is summarized in the figure below.

Tools installed in DASH QoE Framework

• Godash player — an ABS video player
• Mininet-wifi — a wireless emulation environment
• Caddy – a WSGI web server hosting DASH video content
• Scripts – Bash scripts to apply the 5G bandwidth values sampled from the 5G traces
• Python scripts to process per-segment QoS and QoE metrics

Godash Player

Godash is implemented in “golang” and plays HAS video content without actually decoding actual video to consume less resources.  The player supports various state-of-art ABS algorithms that provide a complete functionality to stream current HTTP adaptive streaming with two protocols: TCP and QUIC. An example output “godash” player is listed in Table 1.

DASH QoE Framework

Next, we fetch per-segment QoS features from the Pcaps collected during each experiment and merged them with “godash” logs, as shown in Table below.

Download the Framework

The framework includes a pre-configured VM (download)  with all software and dependencies installed.

Video instructions to use the framework is also available inside the VM.

Conclusions

We have released a flexible DASH QoE evaluation framework to run realistic investigations of QoS impact on QoE.

If you happen to use the DASH framework in your research, please reference the following publications:

• R. Ul Mustafa, M. Islam, C. E. Rothenberg, S. Ferlin, D. Raca, J. J. Quinlan, and others, “DASH QoE performance evaluation framework with 5G datasets,” In IEEE CNSM AnServApp Workshop, 2020.
• R. Ul Mustafa, S. Ferlin, C. E. Rothenberg, D. Raca, and J. J. Quinlan, “A supervised machine learning approach for dash video QoE prediction in 5G networks,” In Proceedings of the 16th ACM Symposium on QoS and Security for Wireless and Mobile Networks (Q2SWinet’20), 2020, p. 1–8