Two Fundamental Concepts in H.264: IDR, dts/pts

Distinction Between IDR Frames and I Frames

IDR: Instantaneous Decoding Refresh.

The distinction between I frames and IDR frames involves the video decoding process for different frame types in H.264.

  • For ordinary I frames (and IDR frames), their video decoding does not depend on other frames. Each I frame (and IDR frame) can be decoded to produce a complete image.
  • For each P frame, its decoding requires referencing preceding I frames or P frames to successfully decode a complete image.
  • B-frame decoding is more complex: it not only references preceding I frames or P frames but may also reference images decoded from subsequent P frames.

To support the above decoding processes, H.264’s decoding algorithm needs to maintain a Decoded Picture Buffer (DPB), which is essentially a common buffer space used to store the image data of each decoded frame. Thus:

  • I-frame decoding does not need to depend on the contents of the DPB. It can be decoded independently to produce a complete image, which is then placed into the DPB to serve as a reference for subsequent P and B frames.
  • P-frame and B-frame decoding, however, relies on image frames already decoded and stored in the DPB. Once decoded, they are also placed into the DPB to serve as reference frames for subsequent P and B frames.

Note: P-frame and B-frame decoding may not only depend on the immediately preceding I frame but also on earlier frames. This leads to a subtle difference between I frames and IDR frames:

An IDR frame is a special type of I frame. All P frames and B frames following an IDR frame will only reference that specific IDR frame and no frames preceding it. Therefore, when a decoder processes an IDR frame, it can directly clear all cached data stored in the DPB. This is a logical consequence, as all frames following this IDR frame will not reference any data prior to it.

In summary:

  • An IDR frame is always an I frame, but an I frame is not necessarily an IDR frame.
  • An IDR frame causes the DPB to be cleared, whereas an ordinary I frame does not, because subsequent frames might need to reference frames preceding that ordinary I frame for decoding.
  • For the same reason, when performing a seek operation within a video stream or file during playback, it is crucial to start decoding and playback from an IDR frame. Otherwise, if playback starts at an I frame, subsequent frames might depend on frames preceding that I frame for decoding, potentially leading to visual corruption (e.g., green/blocky screens).

dts and pts

The concepts of dts and pts are described below:

  • dts (Decoding Time Stamp): This timestamp indicates when the player should decode the data for a given frame.
  • pts (Presentation Time Stamp): This timestamp indicates when the player should display the data for a given frame.

In practice, the concepts of dts and pts are closely related to B frames. When an encoder/decoder only uses I frames and P frames, the video decoding and playback order are completely identical. Therefore, dts and pts are equal, and discussing them at this point has little significance; the player simply decodes and displays in the original order.

However, the introduction of B frames causes the decoding and display orders to differ. This is because:

  • B-frame decoding relies on subsequent frames. Therefore, to successfully decode a B frame, its subsequent frames must first be decoded. However, during playback, frames must be presented in their original order. This means the decoding order and the display order are inconsistent.

For example:

Consider a frame sequence with the playback order: I B1 B2 P;

  • The I frame can be decoded independently, so it is decoded first. I frame decoding order is 1, playback order is 1;
  • B1 and B2’s decoding depends on the I frame and the subsequent P frame, so the P frame must be decoded first;
  • P frame decoding order is 2, playback order is 4;
  • B1’s decoding depends on B2, so B2 is decoded first. B2’s decoding order is 3, playback order is 3;
  • B1’s decoding order is 4, playback order is 2.

In other words:

Frame:    I B1 B2 P;

dts:      1 4 3 2;

pts:      1 2 3 4;

References

  1. “Advanced Guide to Audio/Video Development”, Chapter 1;
  2. Distinction Between I Frames and IDR Frames in H264
  3. Understanding Audio/Video PTS and DTS