Many live or interactive multimedia applications expect the data to arrive within the maximum acceptable end-to-end latency (i.e., meeting deadline). If the content cannot be delivered by the deadline (i.e., missing deadline), the delivery efforts may be wasted. The data of such applications are usually transmitted in blocks (e.g., chunks in live video streaming, frames in video conferencing). Missing the deadlines of blocks can greatly affect the users’ QoE. For example, the latency of video conferencing cannot exceed about 100ms to enable natural interaction.
With a long-term goal of accelerating the optimization of new applications as well as improving the users’ QoE of delay sensitive multimedia, we hope to conduct a series of “Delay sensitive Multimedia” challenges. In ACM multimedia 2021, we host the “Meet Deadline Requirements” challenge to be the first one of this series.
We will provide a simulation platform, application datasets, network datasets that can be utilized to design and test algorithms. We hope that the platform and datasets can assist everyone in optimizing the QoE of delay-sensitive multimedia in this challenge.
We take the RTC (real-time communication) application as an example to illustrate the typical scenarios of delay-sensitive applications.
Figure 1. Typical framework of RTC application
RTC peer: User node of RTC application such as Facetime/WebRTC
Data Blocks: Data transmitted by the transport protocol such as TCP/RTP/RTCP
Generally, streamed data from RTC applications can be divided into three categories. The first category is control signal，such as predicted bandwidth, target bit rate setting, etc. Control signals must arrive timely to ensure the stability of RTC application services. The second category is audio, that is, the user's voice data after noise elimination. The third category is video recorded by the camera.
These three types of data have different priorities. Missing deadline events of control signal may cause severe decline of the QoE. In most RTC applications, audio is more important than video. Therefore, the priority weight is: control signal> audio> video.
Figure 2. Decisions need to be made
For transmission, to achieve as good QoE as possible, there are two decisions that need to be made continuously.
1. Decide which data block to send. According to the state of the current block (emergency and the priority of the data) to determine which data block should to be sent first.
2. Decide the sending rate. Determine the sending rate according to the current network situation (adapt to the available bandwidth). Excessive rate will cause high queueing delay even congestion, too low rate means waste of resources, both will harm the experience of users.