Our paper titled “P-Frame Injection for Efficient Packet-Loss Repair in Ultra-Low-Latency Video Streaming” is accepted at the IEEE International Conference on Visual Communications and Image Processing (VCIP) 2023. The paper was presented between December 4 - 7 in Jeju, Korea.
Applications providing ultra-low-latency video streaming to large audiences require fast and efficient packet-loss repair. Previous methods utilizing keyframe injection (such as the High Efficiency Streaming Protocol) have a low impact on the repaired stream quality, but at a cost of a significant bitrate spike during repair.
In this paper, we propose injecting P-frames for packet-loss repair of ultra-low-latency streaming. By leveraging P-frames as reference repair pictures, we achieve improved packet-loss recovery without compromising the streaming quality. We implemented (open-source) and evaluated our approach in both H.265/HEVC and H.266/VVC standards.
Through extensive evaluations, we demonstrate that our method significantly reduces the bitrate overhead while maintaining a similar or lower decrease in quality compared to existing packet-loss-repair techniques.
For example, let us discuss the injection a P-frame of the companion stream (CS) at position f=16 in a normal stream (NS) with QP=27, using the VVEnc encoder (in the H.266/VVC standard). The P-frame references frame 8 (i.e., 8 frames before the P-frame), and also has QP=27. The median factor of frame size increase of such P-frames is 2.0, meaning the CS P-frame is approximately double in size compared to the NS P-frame that it replaces. The corresponding median VMAF decrease is 1.3, and the median PSNR decrease is 0.8 dB. Although the median quality loss is small, it should be noted that there are outlier videos with more significant quality loss values (see the graph below).
Graph that shows the VMAF decrease over time, after injecting a P-frame at frame f=16. Although most tested video sequences have a small VMAF decrease (median VMAF decrease of 1.3), there are outliers with larger quality losses. We analyze three selected videos in the examples below.
Some video examples are shown below, using the same configuration as described in the example above.
Example of injecting a P-frame in BlowingBubbles. From left to right: normal stream, p-frame-injected stream, visualisation of difference between the two. Full video without additional YouTube compression can be found here. In terms of VMAF decrease, this is one of the worst performing examples (average VMAF decrease of 7.3), yet the drift artifacts are not very perceptible nor annoying.
Example of injecting a P-frame in BQTerrace. From left to right: normal stream, p-frame-injected stream, visualisation of difference between the two. Full video without additional YouTube compression can be found here. In terms of VMAF decrease, this is also one of the worst performing examples (average VMAF decrease of 3.3), yet the drift artifacts are not very perceptible nor annoying.
Example of injecting a P-frame in Kimono1. From left to right: normal stream, p-frame-injected stream, visualisation of difference between the two. Full video without additional YouTube compression can be found here. In terms of VMAF decrease, this is one of the better performing examples (average VMAF decrease of 0.9). The drift artifacts are imperceptible or barely perceptible.
Thus, these examples demonstrated that the effect from P-frame injection results in drift-error artifacts, but these are impcerptible or barely perceptible.
Overall, our approach offers a promising solution to ensure reliable packet-loss recovery, efficient resource utilization, and a high-quality streaming experience.
Paper download: P-Frame Injection for Efficient Packet-Loss Repair in Ultra-Low-Latency Video Streaming
The source code is available on GitHub.com.
