H.264 Basics: A Summary of SVC

SVC (Scalable Video Coding) is an extension of the H.264 (AVC) video coding standard.

The Problems SVC Aims to Solve

The H.264 video compression standard is used to compress video data to facilitate storage and transmission over networks.

In point-to-point (one-to-one) transmission, standard H.264 (AVC) generally works without issues:

  • When the receiver has high processing power and good network bandwidth, the encoder can be configured to output video with higher quality, larger resolution, and higher frame rates to provide a better viewing experience.
  • Conversely, if the receiver’s processing capabilities are limited or the network bandwidth is insufficient for high-quality compressed video, the encoder can be configured to output lower-quality video with reduced resolution or frame rate. This lowers both the required transmission bandwidth and the processing demand on the decoder.
  • In practice, during point-to-point communication, network bandwidth fluctuations can be dynamically monitored between the sender and receiver to adjust the encoder on the fly. When network quality is good, the encoder outputs video with higher bitrates and bandwidth requirements; when the network degrades, it outputs lower-bitrate video that demands less bandwidth.

However, in one-to-many video communication scenarios (most typically video conferencing, live streaming, and video-on-demand), the video captured by a single camera needs to be distributed to multiple receivers. Since each receiver’s processing capability and network conditions differ, they have diverse requirements for video quality, resolution, and bitrate. Receivers with higher processing power and better bandwidth expect higher bitrates and better quality to maximize user experience. On the other hand, receivers with limited processing power or poor bandwidth expect, at the very least, a lower-bitrate video that plays smoothly without stuttering.

Yet, for a standard H.264 (AVC) encoder, once the parameters are set, the output video’s resolution, frame rate, and quality are essentially fixed.

The most straightforward solution to this problem is: after capturing video at the camera end, compress it using the highest-quality compression parameters and upload it to a central server. The central server then transcodes this high-quality video into various combinations of resolutions, frame rates, and bitrates. Finally, the server distributes the appropriate transcoded video streams to different receivers based on their respective network bandwidths and processing capabilities.

  • Indeed, this is how modern internet video-on-demand (VOD) platforms (like YouTube, Bilibili, etc.) work. The creator’s video is compressed with high-quality encoding parameters and uploaded to a central server, which then transcodes it into multiple resolutions to support various client devices.

The drawback of this solution is that server-side transcoding can be prohibitively expensive in certain scenarios. While transcoding uploaded video on a central server makes sense when thousands of users are watching, it imposes an immense burden on the server if the video is only being viewed by a few people (most typically in video conferencing). Having to transcode and forward every single incoming stream for small groups is highly inefficient.

  • Imagine a meeting with 8 participants: 8 video streams would be sent to the central server to be simultaneously transcoded into various resolutions and qualities before being distributed. The computational load on the central server would be enormous. Spending so much computing power just to serve a small 8-person meeting makes this approach extremely cost-ineffective and inefficient for media processing.

Therefore, the core challenge is: For small-scale one-to-many or many-to-many scenarios like video conferencing, how can we deliver video with different qualities and bitrates to various receivers without relying on central server transcoding? This is precisely the problem that the H.264 SVC extension aims to solve.

How SVC Solves This Problem

The approach SVC takes to solve this problem is as follows:

  • Single Encoding on the Sender Side: The video producer (the camera end) performs a single encoding session. The resulting output bitstream is structured into multiple layers (or parts). The complete stream (the combination of all parts) represents the highest quality level with the maximum bitrate. By combining different parts of this stream—retaining some and discarding others—various sub-streams can be formed to cater to different quality levels and device types.
  • No Transcoding on the Server: The camera uploads the compressed, complete bitstream to a central server. Instead of re-transcoding the video, the server simply drops or forwards specific layers of the stream to each receiver based on their processing capabilities and bandwidth.
  • Adaptive Decoding on the Receiver Side: Receivers decode and play back the stream they receive. High-performance receivers with good bandwidth will likely receive the complete stream, yielding the best playback quality. In contrast, low-performance or bandwidth-constrained receivers may only receive a subset of the stream. Although decoding this subset produces a video with lower resolution, frame rate, or quality, it guarantees smooth playback.

The “parts” of the complete video stream mentioned in this solution are referred to as Layers in SVC. The combination of all layers yields the highest-quality video, while combining only a subset of these layers produces a video with lower characteristics and reduced bandwidth requirements.

The key to this solution lies in the video codec:

  • The Encoder: It must be capable of generating a layered video stream according to the logic described above. Standard H.264 (AVC) does not support this, as it lacks the concept of layering.
  • The Decoder: It must be able to decode both the complete video stream and various combinations of its layers, outputting video with different quality attributes.

A video codec that complies with the H.264 SVC extension can satisfy both requirements.

Technical Implementation Details of SVC

The “S” in SVC stands for “Scalable”. The SVC standard provides scalability options across three main dimensions:

  • Quality Scalability: Video compression quality
  • Temporal Scalability: Frame rate
  • Spatial Scalability: Resolution

As mentioned, SVC delivers scalability across these different dimensions through layering. Specifically, a complete SVC stream consists of a Base Layer and one or more Enhancement Layers. The Base Layer represents the lowest video quality configuration supported by the stream, while each Enhancement Layer provides incremental enhancements in a particular dimension (temporal, spatial, or quality).

Untitled.png

Consequently, for receivers with poor processing capabilities and network conditions, only the Base Layer data is transmitted. For moderately capable receivers, a combination of the Base Layer and one or more Enhancement Layers is sent. For high-end receivers with optimal configurations, the entire complete SVC stream is delivered to ensure the best possible playback experience.

The data for different layers in an SVC stream generated by the encoder can be distinguished using the NALU Header when transmitted to a central server or saved to a local file. This allows the central server to easily identify which layer a particular data packet belongs to upon receipt.

Microsoft’s UCConfig

For their Teams and Skype for Business certification ecosystems, Microsoft defined five UCConfig operating modes based on the H.264 SVC standard. These modes target different combinations of scalable SVC layers to meet the requirements of video conferencing systems:

  • UCConfig Mode 0: Non-Scalable Single Layer AVC Bitstream.
    • Does not support SVC extensions; it is essentially a standard H.264 AVC stream. This mode is the only option for H.264 encoders that do not support SVC.
    • This mode can also support transmitting multiple streams of different resolutions within the same transport stream, i.e., Simulcast.
  • UCConfig Mode 1: SVC Temporal Scalability with Hierarchical P.
    • Supports temporal (frame rate) scalability. Each independent stream contains scaling options for frame rates under the same resolution.
  • UCConfig Mode 2q: SVC Temporal Scalability + Quality/SNR Scalability.
    • Builds upon UCConfig Mode 1 by adding support for quality (SNR) scalability layers. A single independent stream contains both temporal scalability layers and quality scalability layers for the same resolution.
  • UCConfig Mode 2s: SVC Temporal Scalability + Spatial Scalability.
    • Builds upon UCConfig Mode 1 by adding support for spatial (resolution) scalability layers. A single independent stream contains both spatial scalability layers and temporal scalability layers.
  • UCConfig Mode 3: Full SVC Scalability (Temporal + SNR + Spatial).
    • The highest level of scalability supported by UCConfig based on the H.264 SVC specification. A single independent stream simultaneously contains scalability layers for resolution, frame rate, and quality.

References