IDLab-MEDIA
Mixed-Resolution HESP

We submitted our paper titled “Mixed-Resolution HESP for More Efficient Fast Channel Switching and Packet-Loss Repair” to the Picture Coding Symposium (PCS) 2022. This work was made in collaboration with THEO Technologies.

In low-delay live streaming, it is desired to have fast channel switching and packet-loss repair capabilities. Existing methods can provide both capabilities, but with a negative impact on the stream of steady-state users.

To minimize this impact, techniques such as the High Efficiency Streaming Protocol (HESP) utilize keyframe injection. Such techniques combine compression-efficient normal streams with corresponding companion streams that are used in case of random access or packet loss. Unfortunately, there are three downsides to such strategies. Because a companion stream is needed for every normal stream, first, distribution cost over the Content Distribution Network and second, encoding complexity are considerable costs of the current strategies. Third, injecting a companion keyframe into a normal stream causes a bitrate spike.

Therefore, this paper evaluates the impact of utilizing mixed-resolution keyframe injection in the H.266/VVC standard. By providing a single companion stream for all normal streams of a bitrate ladder, the three mentioned downsides can be mitigated.

Diagram Only after a (a) random-access or packet-loss event, (b) a companion stream keyframe is requested (possibly with different resolution) and (c) the keyframe is decoded instead of the predicted frame at an equal timestamp.

We found that injecting a lower-resolution keyframe can effectively reduce the bitrate spike, at the cost of only a modest quality loss. The negative impact on the quality reduces over time with dynamic video content, because the subsequent P-frames replace the low-resolution content with new high-resolution information. An example of this is shown in the image and corresponding video below, which shows the effect of injecting a keyframe of half resolution and QP 32 in a normal stream of QP 22, of the dynamic RaceHorses sequence.

DynamicContent Example of injecting a low-resolution keyframe in dynamic video content. From left to right: normal stream, keyframe-injected stream, visualisation of difference between the two. Full video can be found here.

In static video content, the injected information remains in low resolution as it is not replaced in subsequent P-frames. An example of this is shown in the image and corresponding video below, which shows the effect of injecting a keyframe of half resolution and QP 32 in a normal stream of QP 22, of the ChinaSpeed sequence which contains a static heads-up dispay (HUD).

StaticContent Example of injecting a low-resolution keyframe in static video content. From left to right: normal stream, keyframe-injected stream, visualisation of difference between the two. Full video can be found here.

We compared the quality loss to traditional packet-loss repair using frame copy. These traditional packet-loss repair solutions can create very perceptible artifacts in dynamic content. An example of this is shown in the image and corresponding video below, which shows the effect of applying frame copy in a normal stream of QP 22, of the dynamic RaceHorses sequence. Thus, mixed-resolution keyframe injection outperforms traditional packet-loss repair.

FrameCopyComparison Example of traditional packet-loss repair using frame copy in dynamic video content. From left to right: normal stream, packet-loss-repaired stream, visualisation of difference between the two. Full video can be found here.

In conclusion, our solution can reduce the bitrate spike and computational/distribution overhead of HESP when enabling fast channel switching or packet-loss repair.