Summary of Basic H.264 Concepts: SPS, PPS, and SEI
- 16 Mar, 2022
SPS
- Sequence Parameter Set (SPS): This non-VCL NALU contains information required to configure the decoder such as profile, level, resolution, frame rate, which apply to a series of consecutive coded video pictures called a coded video sequence.
SPS refers to the global parameter information for an entire coded video sequence, such as resolution, frame rate, and the Profile and Level used during H.264 encoding.
Like encoded video frame data, SPS information is encapsulated inside a NALU (Network Abstraction Layer Unit). The NALU Header Type corresponding to SPS is typically 0x67. Therefore, when parsing a received H.264 stream, if a NALU starts with 0x00 0x00 0x00 0x01 0x67, it indicates that the information encapsulated in this NALU is SPS.
PPS
- Picture Parameter Set (PPS): Similar to the SPS, this non-VCL NALU contains information on entropy coding mode, slice groups, motion prediction, quantization parameters (QP), and deblocking filters, which apply to the decoding of one or more individual pictures within a coded video sequence.
PPS refers to the parameter information required by individual video frames within a coded sequence. It primarily contains encoding-related parameters used when encoding that specific frame, such as entropy coding mode, slice group mode, QP settings, etc.
Like encoded video frame data, PPS information is also encapsulated inside a NALU. The NALU Header Type corresponding to PPS is typically 0x68. Therefore, when parsing a received H.264 stream, if a NALU starts with 0x00 0x00 0x00 0x01 0x68, it indicates that the information encapsulated in this NALU is PPS.
Placement of SPS and PPS
To ensure that a player can properly decode H.264 encoded data, the decoder must be properly initialized using the information contained in the SPS and PPS before it officially begins decoding.
For the encoder, the first output data is generally PPS, followed by SPS, and then I-frames, P-frames, B-frames, etc.
When this data is transmitted over a network as streaming media or saved locally in an H.264 file, it is generally processed in the order of SPS, PPS, I-frames, P-frames, B-frames, and so on. This ensures that the receiver or decoder can configure its operating parameters in the correct order before proceeding with video frame decoding.
Additionally, SPS and PPS may also appear in the middle of a bitstream, primarily for the following reasons:
- The decoder might start receiving data in the middle of a live stream transmission. To correctly decode the subsequent stream data, SPS and PPS information must be periodically inserted into the bitstream to ensure the decoder can be properly configured.
- In this scenario, a good practice is to prepend SPS and PPS information before every I-frame.
- The encoder changes stream parameters (such as resolution) during the encoding process. Upon receiving new SPS and PPS information, the decoder needs to apply these updated parameters.
If the media stream generated by the encoder is encapsulated and saved locally in a container file (e.g., MP4, MKV), the SPS and PPS information is usually placed in the header of the container format. This allows the decoder to extract and apply the matching SPS and PPS configuration during demuxing.
SEI
SEI (Supplemental Enhancement Information) provides the capability to synchronously transmit additional custom data within a multimedia stream.
By embedding user-defined data directly into the streaming media, SEI enables real-time synchronization between custom information and audio/video playback.
- A classic example is live streaming: while a streamer transmits audio and video streams to viewers, they may also want to send real-time text or messaging data. Although this data could be sent via a separate communication channel, synchronizing the display of messages with the video stream is often challenging. By placing the instant messaging data into SEI units and encapsulating it together with media data in NALUs, real-time synchronous presentation of both data types can be guaranteed.
Parsing and processing SEI information on the decoder side is optional. If the SEI data is not needed, NALUs containing SEI can simply be discarded.
Since SEI content is custom-defined, as long as both the encoding/sending end and the receiving/decoding end are controlled and implemented under the same logic, any arbitrary data payload can be encapsulated in SEI.
The lower 5 bits of the NALU Header Type corresponding to SEI are 0x06. Therefore, when parsing a received multimedia stream, if the lower 5 bits (nal_unit_type) of a NALU header byte equal 0x06, it indicates that the information encapsulated in this NALU is SEI data.