04-Video Coding Standards Girod | Data Compression | H.264/Mpeg 4 Avc

Overview: Video Coding Standards

Video coding standards: the applications and the common structure Relevant standards organizations ITU-T Rec. H.261 ITU-T Rec. H.263 ISO/IEC MPEG-1 ISO/IEC MPEG-2 ISO/IEC MPEG-4 Recent progress: H.264/JVT

Bernd Girod: EE398B Image Communication II

Video Coding Standards no. 1

Major Applications of Video Compression
Digital television broadcasting DVD video 2 . . . 6 Mbps (10…20 Mbps for HD) 6 . . . 8 Mbps MPEG-2

MPEG-2

Internet video streaming Videoconferencing, videotelephony Video over 3G wireless

20 . . . 200 kbps

Proprietary, similar to H.263, MPEG-4, or H.26L/JVT H.261, H.263

20 . . . 320 kbps

20 . . . 100 kbps

H.263, MPEG-4

Bernd Girod: EE398B Image Communication II

Video Coding Standards no. 2

Motion-compensated Hybrid Coding
H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.26L/JVT

Coder Control Transform/ Quantizer Decoder Deq./Inv. Transform

Control Data Quant. Transf. coeffs

-

0 Intra/Inter

MotionCompensated Predictor Motion Data

Entropy Coding

Motion Estimator

Bernd Girod: EE398B Image Communication II

Video Coding Standards no. 3

Video Standards: Hierarchical Syntax I Bernd Girod: EE398B Image Communication II Video Coding Standards no. 4 .

5 .Video Standards: Hierarchical Syntax II Bernd Girod: EE398B Image Communication II Video Coding Standards no.

International Telecommunication Union (ITU) Formed in 1934 by merger of the International Telegraph Convention of 1865 and the International Radiotelegraph Convention of 1906 Several “committees.” among them CCITT (International Telephone and Telegraph Consultative Committee) 1956-1992 CCIR (International Radio Consultative Committee) 1927-1992 Reform in 1992 CCITT -> ITU-T CCIR -> ITU-R Any recommendation must be agreed upon unanimously by all its member states Bernd Girod: EE398B Image Communication II Video Coding Standards no. 6 .

120 .323 – LAN H. V. 7 .324 – PSTN T.DATA WP3 – Coding G.320 – ISDN H.25ter Bernd Girod: EE398B Image Communication II Video Coding Standards no.Video WP1 – Modems and interface V.Multimedia WP2 –Systems H.261 – Video H.263 .7xx – Audio H.34.ITU organization with subgroups relevant for video ITU ITU-R SG1 SG2 … ITU-T ITU-D Study Group 16 .

8 . non-profit company under Swiss law Agency of the United Nations Joint ISO/IEC Technical Committee 1 (JTC 1) Jointly addresses all computer-related activities About 30% of total ISO and IEC standards Bernd Girod: EE398B Image Communication II Video Coding Standards no. non-profit company under Swiss law ISO – International Organization for Standardization Established in 1947 “to facilitate the international coordination and unification of industrial standards” Private.IEC and ISO IEC – International Electrotechnical Commission founded in 1906 to establish international standards for all electrical technologies private.

9 .ISO/IEC organization with subgroups relevant for video IEC JTC1 SC29 AG WG ISO Requirements Systems Description Video Audio WG12 SG MHEG-5 MHEG-6 WG11 SG SNHC Tests Implementation Studies Liaisons AGM RA WG1 SG JBIG JPEG Bernd Girod: EE398B Image Communication II Video Coding Standards no.

Only standardize bit-stream syntax and reference decoder. Competition: should be flexible enough to allow competition between manufacturers based on technical merit. Innovation: should perform significantly better than previous standard. 10 . Forward compatibility: should decode bit-streams from prior standard Backward compatibility: prior generation decoders should be able to partially decode new bit-streams Bernd Girod: EE398B Image Communication II Video Coding Standards no. Independence from transmission and storage media: should be flexible enough to be used for a range of applications.Requirements for a successful video coding standard Interoperability: should assure that encoders and decoders from different manufacturers work together seamlessly.

2) competition RM 5 convergence RM 6 7 8 verification optimization REC.Standard Development Process subject n x 384 (n= 1-5) RM 1 2 3 4 p x 64 (p= 1-30) m x 64 (m= 1.261 Standardization Process Bernd Girod: EE398B Image Communication II Video Coding Standards no.261 1985 1986 1987 1988 1989 1990 competition convergence verification optimization Overview of H. H. 11 .

. H. 12 .. Picture quality: for 128 kbps acceptable with limited motion in the scene Stand-alone videoconferencing system or desk-top videoconferencing system.261 International standard for ISDN picture phones and for video conferencing systems (1990) Image format: CIF (352 x 288 Y samples) or QCIF (176 * 144 Y samples). typically 128 kbps including audio. frame rate 7. integrated with PC Bernd Girod: EE398B Image Communication II Video Coding Standards no.5 .ITU-T Rec. 30 fps Bit-rate: multiple of 64 kbps (= ISDN-channel).

13 Bernd Girod: EE398B Image Communication II .Image formats sub QCIF 176 x 144 QCIF ITU-R 601 352 x 288 CIF Video Coding Standards no.

14 .H.261 macroblocks Macroblock (MB) of 16x16 pixels Sampling format: 4:2:0 An MB consists of 4 luminance and 2 chrominance blocks 16x16 luminance samples 0 1 8x8 Cbsamples 4 8x8 Crsamples 5 2 3 Bernd Girod: EE398B Image Communication II Video Coding Standards no.

15 .H. ½.261 motion compensated prediction Integer-pel accuracy One displacement vector per macroblock Maximum displacement vector range +/-16 horizontally and vertically Adaptive loop filter. ¼] Differential encoding of motion vectors Bernd Girod: EE398B Image Communication II Video Coding Standards no. separable filters in 1-D horizontal and vertical impulse response: [¼.

H.….62) for AC coefficients in intra-mode and all coefficients in inter-mode Zigzag scan Run-level coding for entropy coding (zero-run. value) symbols zero-run: the number of coefficients quantized to zero since the last nonzero coefficient value: the amplitude of the current nonzero coefficient Bernd Girod: EE398B Image Communication II Video Coding Standards no.261 residual coding 8x8 DCT Quantization A uniform quantizer (∆=8) for intra-mode DC coefficients A uniform threshold quantizer (∆=2.4. 16 .

H. 17 .261 Macroblock Types (VLC Table) Prediction Intra Intra Inter Inter Inter+MC Inter+MC Inter+MC Inter+MC+FIL Inter+MC+FIL Inter+MC+FIL X X X X X X X X X X X X X X X X X X X X MQUANT MVD CBP TCOEFF X X X X VLC 0001 0000 001 1 0000 1 0000 0000 1 0000 0001 0000 0000 01 001 01 0000 01 Bernd Girod: EE398B Image Communication II Video Coding Standards no.

frame rate usually below 10 fps Bit-rate: arbitrary.263 is also the compression core of the MPEG-4 standard Bernd Girod: EE398B Image Communication II Video Coding Standards no. typically 20 kbps for PSTN Picture quality: with new options as good as H. QCIF or Sub-QCIF.263 International standard for picture phones over analog subscriber lines (1995) Image format usually CIF.ITU-T Rec.261 (at half rate) Software-only PC video phone or TV set-top box Example: 8x8 ViaTV phone VC 105 Widely used as compression engine for Internet video streaming H. H. 18 .

263 (1995): half-pel accuracy.263 Improved motion compensation H. (last.261 vs. • Switch between 1 or 4 motion vectors per MB PB pictures (Annex G) More optional features in H.263 as of 2001) Bernd Girod: EE398B Image Communication II Video Coding Standards no. loop filter. H.261 (1990): integer-pel accuracy.H. run. no loop filter. level) Reduced overhead Support more picture formats Optional features defined in annexes Unrestricted motion vectors (Annex D) Syntax-based arithmetic coding (Annex E) Advanced prediction mode (Annex F) • Overlapped block motion compensation (OBMC). 1 motion vector per MB Improved 3-D VLC for DCT coefficients. 19 .263++. 1 motion vector per MB H. (H.

20 .261 34 32 1) 2) 4) 3) 5) PSNR [dB] 30 28 26 24 1) H. integer-pel ME 5) H.263 and H.Performance of H.261 4) H.263 2) H.263 w/o options.263 w/o options 3) H.261 w/o loop filter 32 64 128 rate [kbps] Bernd Girod: EE398B Image Communication II Video Coding Standards no.

21 .263 SAC mode 36 35 34 33 32 31 30 29 28 27 26 1) 2) 1) SAC-mode 2) w/o options 0 32 64 rate [kbps] 128 Bernd Girod: EE398B Image Communication II Video Coding Standards no.Performance of H.

H. 22 .263: overlapped block motion compensation (OBMC) remote luminance block Current luminance block (8x8) M A C R O B L O C K remote luminance block remote luminance block remote luminance block Bernd Girod: EE398B Image Communication II Video Coding Standards no.

263: OBMC weights 4 5 5 5 5 5 5 4 5 5 5 5 5 5 5 5 5 5 6 6 6 6 5 5 5 5 6 6 6 6 5 5 5 5 6 6 6 6 5 5 5 5 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 5 5 5 5 5 5 4 for MV of current luminance block 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 2 2 1 1 1 1 1 1 2 2 2 1 1 1 1 2 2 2 2 1 1 1 1 2 2 2 2 1 1 1 1 2 2 2 2 1 1 1 1 2 2 2 1 1 1 1 1 1 2 2 1 1 1 1 1 1 2 for remote MV of top/bottom luminance block for remote MV of left/right luminance block Bernd Girod: EE398B Image Communication II Video Coding Standards no.H. 23 .

263 AP mode 36 35 34 33 32 31 30 29 28 27 26 1) 2) 1) AP-mode 2) w/o options 0 32 64 rate [kbps] 128 Bernd Girod: EE398B Image Communication II Video Coding Standards no. 24 .Performance of H.

263: PB pictures (Annex G) forward prediction P bidirectional prediction B bidirectional prediction P PB picture Bernd Girod: EE398B Image Communication II Video Coding Standards no.H. 25 .

5 fps) 3) PB-mode (12.263 PB-mode 36 35 34 33 32 31 30 29 28 27 26 1) 3a) 2) PSNR [dB] 3b) 1) w/o options (6.25 fps) 2) w/o options (12.5 fps) a) P-frames b) B-frames 0 32 64 128 rate [kbps] Bernd Girod: EE398B Image Communication II Video Coding Standards no. 26 .Performance of H.

324 H. 27 .225.323/H.0 Bernd Girod: EE398B Image Communication II Video Coding Standards no.245 H.320 H.222.261 Audio G.261/2 H.261/3 H.242 Q.322 H.7xx Mux H.261/3 H.1 G.321 H.261 H.0/1 G.221 H.7xx H.310 QoS LAN Non-Qos LAN H.7xx G.221 Control H.2931 H.323 Video H.245 H.7xx/MPEG H.Visual Communication Systems: H.221 H.723.245 G.7xx G.320/H.261 H.242 H.324 Network PSTN N-ISDN B-ISDN /ATM System H.223 H.

34/V.H.223 Modem V. 28 .14. LAPM etc Control protocol SRP/LAPM procedures Modem control v.261/H.25ter System control H.324 Bernd Girod: EE398B Image Communication II Video Coding Standards no.263 Audio codec G.245 Scope of H.723 Receive path delay Mux/ demux H.8 PSTN network User data applications Data protocols V.324 Multimedia Terminals Video I/O equipment Audio I/O equipment Video codec H.

H.261 ITU-T Rec.263 ISO/IEC MPEG-1 ISO/IEC MPEG-2 ISO/IEC MPEG-4 Recent progress: H. H. 29 .Overview: Video Coding Standards Video coding standards: the applications and the common structure Relevant standards organizations ITU-T Rec.26L/JVT Bernd Girod: EE398B Image Communication II Video Coding Standards no.

etc. on CD-ROM) MPEG-2 Standard (1994) (ISO/IEC 13818) Extension for interlace. PAL. Bernd Girod: EE398B Image Communication II Video Coding Standards no. 30 .ISO/IEC MPEG MPEG-1 Standard (1991) (ISO/IEC 11172) Target bit-rate about 1. 30 fps Main application: video storage for multimedia (e. error resilience.g.. with choices of interactivity.5 Mbit/s Typical image format CIF. no interlace Frame rate 24 .. optimized for TV resolution (NTSC: 704 x 480 Pixel) Image quality similar to NTSC.8 Mbit/s HDTV at 20 Mbit/s MPEG-4 Standard (1999) (ISO/IEC 14496) Object based coding Wide-range of applications. SECAM at 4 . scalability..

31 Bernd Girod: EE398B Image Communication II . GOP structure is very flexible I-Picture P-Picture P-Picture B-Pictures time 1 3 4 2 6 7 8 5 Video Coding Standards no.MPEG-1/2: GOP Structure "Group of Pictures" = “GOP“.

32 . macroblock info.MPEG-1 Encoder Preprocessing Picture reordering - DCT Weighting Quantization VLC Video multiplex Buffer Inverse quantization Video in Inverse weighting Inverse DCT zero Picture store 1 Motion vectors. start codes Bitstream + 1/2 + Motion compensation Picture store 2 Bernd Girod: EE398B Image Communication II Video Coding Standards no.

MPEG-1: coding of I-pictures I-pictures: intraframe coded 8x8 DCT Arbitrary weighting matrix for coefficients Differential coding of DC-coefficients Uniform quantization Zig-zag-scan. run-level-coding Entropy coding Unfortunately. 33 . not quite JPEG Bernd Girod: EE398B Image Communication II Video Coding Standards no.

uniform threshold quantization.MPEG-1: coding of P-pictures Motion-compensated prediction from an encoded I-picture or P-picture (DPCM) Half-pel accuracy of motion compensation. 34 . bilinear interpolation One displacement vector per macroblock Differential coding of displacement vectors Coding of prediction error with 8x8-DCT. zig-zag-scan as in I-pictures Bernd Girod: EE398B Image Communication II Video Coding Standards no.

uniform quantization. zig-zag-scan as in I-pictures Bernd Girod: EE398B Image Communication II Video Coding Standards no.MPEG-1: coding of B-pictures Motion-compensated prediction from two consecutive P. bilinear interpolation Coding of prediction error with 8x8-DCT. 35 .or I-pictures either • only forward prediction (1 vector/macroblock) or • only backward prediction (1 vector/macroblock) or • Average of forward and backward prediction = interpolation (2 vectors/macroblock) Half-pel accuracy of motion compensation.

VLC tables Various scalability modes Bernd Girod: EE398B Image Communication II Video Coding Standards no. 36 . MPEG-1 Efficiently compress interlaced digital video at broadcast quality Field/frame pictures Chroma sampling New prediction modes Field/frame DCT Additional scan patterns for DCT coefficients Motion compensation with blocks of size 16x8 pels Improved coding efficiency by different quantization.MPEG-2 vs.

Coding of Interlaced Video (1) Frame and field picture structures Bernd Girod: EE398B Image Communication II Video Coding Standards no. 37 .

38 .Coding of Interlaced Video (2) Field prediction for field pictures Bernd Girod: EE398B Image Communication II Video Coding Standards no.

Coding of Interlaced Video (3)
Field prediction for frame pictures
16 16 8 16

8

Bernd Girod: EE398B Image Communication II

Video Coding Standards no. 39

Coding of Interlaced Video (4)
Dual prime for P pictures

Bernd Girod: EE398B Image Communication II

Video Coding Standards no. 40

Coding of Interlaced Video (5)
Field/frame DCT Alternate Scan

Bernd Girod: EE398B Image Communication II

Video Coding Standards no. 41

scalability.MPEG-4 Support highly interactive multimedia applications as well as traditional applications Advanced functionalities: interactivity. multi-viewpoint training Video Coding Standards no. 42 ‘TV/film’ AV-data ‘Computer’ Interactivity ‘Telecom’ Wireless Bernd Girod: EE398B Image Communication II . as well as graphics Enable the multiplexing of audiovisual objects and composition in a scene Video on LANs. error resilience… Coding of natural and synthetic audio and video. home movies Virtual reality games. flight simulation. Internet video Wireless video Video database Interactive home shopping Video e-mail.

MPEG-4: Scene with audiovisual objects Bernd Girod: EE398B Image Communication II Video Coding Standards no. 43 .

44 .MPEG-4: Video coding Basic video coding Definition of Video Object (VO). Video Object Plane (VOP) Improved coding efficiency vs. Vide Object Layer (VOL).263 baseline • Global motion compensation • Sprite • Quarter pixel motion compensation Bernd Girod: EE398B Image Communication II Video Coding Standards no. MPEG-1/2 • Based on H.

face and body animation Bernd Girod: EE398B Image Communication II Video Coding Standards no. 45 .MPEG-4: Video coding Object-based video coding Binary shape coding Greyscale shape coding Padding for block-based DCT of texture Shape-adaptive DCT DWT for still texture coding Mesh animation.

Shape Adaptive DCT Bernd Girod: EE398B Image Communication II Video Coding Standards no. 46 .

262 [1994] • Variable block-size motion compensation H.120 [1984] • Frame difference coding H.261 [1991] • Half-pel accurate motion compensation MPEG-1 [1993]. Rao 1974]. MPEG-4 [1999] Bernd Girod: EE398B Image Communication II Video Coding Standards no. scalar quantization H. MPEG-2/H.120 Version 2 [1988] • Motion compensation: integer-pel accurate displacements H.Video compression progress • Intraframe coding: only spatial correlation exploited DCT [Ahmed. JPEG [1992] Complexity increases • Conditional replenishment. 47 . DPCM.263 [1996]. Natarajan.

263 1996) Frame difference coding (H. 48 .261 1991) compensation 38 (H.120 1988) 36 PSNR 34 ~40 % [dB] 32 30 28 26 24 0 ~30 % ~20 % Conditional Replenishment (H.120) Intraframe DCT coding (JPEG) Foreman 10 Hz.Video compression progress Half-pel Integer-pel Variable motion motion block size compensation compensation motion 40 (MPEG-1 1993) (H. QCIF 100 frames encoded 100 200 300 400 500 Bit Rate [kbps] 600 Bernd Girod: EE398B Image Communication II Video Coding Standards no.

120) 36 Intraframe DCT coding (JPEG) Mother & Daughter 10 Hz.Video compression progress 42 40 PSNR 38 [dB] Variable block size motion compensation (H. 49 .263 1996) Conditional Replenishment (H. QCIF 100 frames encoded 34 ~60 % 32 30 28 0 50 100 150 200 ~70 % 250 300 Bit Rate [kbps] 350 Bernd Girod: EE398B Image Communication II Video Coding Standards no.

QCIF 100 frames encoded 22 0 Bit Rate [kbps] 200 400 600 800 1000 1200 1400 Bernd Girod: EE398B Image Communication II Video Coding Standards no.Video compression progress 40 38 36 PSNR 34 ~ 35 % [dB] ~ 40 Variable block size motion compensation (H. 50 .263 1996) Integer-pel motion compensation (H.261 1991) 32 30 28 26 24 % Intraframe DCT coding (JPEG) Mobile &Calendar 10 Hz.

51 .H. Wiegand] Bernd Girod: EE398B Image Communication II Video Coding Standards no.26L/JVT: Motion Compensation Accuracy Coder Control Transform/ Quantizer Decoder Deq. Transform Entropy Coding MotionCompensated Predictor Mode 1 0 Mode 2 0 1 Mode 3 0 1 Mode 5 0 1 2 3 4 5 6 7 Mode 4 0 1 2 3 Control Data Quant. coeffs - 0 Intra/Inter Motion Estimator Motion Mode Data 6 0 2 4 6 1 3 5 7 Mode 7 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1/4 (QCIF) or 1/8 (CIF) pel [courtesy T./Inv. Transf.

26L/JVT: Multiple Reference Frames Coder Control Transform/ Quantizer Decoder Deq. Wiegand] Bernd Girod: EE398B Image Communication II Video Coding Standards no. 52 . coeffs - 0 Intra/Inter Motion Estimator Multiple Reference Frames for Motion Data Motion Compensation [courtesy T.H./Inv. Transf. Transform Entropy Coding MotionCompensated Predictor Control Data Quant.

263 Annex T) Deblocking Filter (in loop) Bernd Girod: EE398B Image Communication II Video Coding Standards no.H.26L/JVT: Residual Coding Transform 4x4 Integer transform approximating a DCT Expanded to 8x8 for chroma by 2x2 DC transform Intra Coding Structure Directional spatial prediction for intra mode Expanded to 16x16 for luma intra by 4x4 DC transform Quantization Two inverse scan patterns Logarithmic step size control Smaller step size for chroma (per H. 53 .

54 . Transform Entropy Coding MotionCompensated Predictor Motion Data Control Data Quant. Transf.26L/JVT: Entropy Coding (1) Universal Variable Length Code (UVLC) Coder Control Transform/ Quantizer Decoder Deq. coeffs - 0 Intra/Inter Motion Estimator [courtesy T. Wiegand] Bernd Girod: EE398B Image Communication II Video Coding Standards no./Inv.H.

Wiegand] Bernd Girod: EE398B Image Communication II Video Coding Standards no.26L/JVT: Entropy Coding (2) Context-based adaptive binary arithmetic codes (CABAC) update probability estimation Context modeling Binarization Probability estimation Coding engine Adaptive binary arithmetic coder Chooses a model conditioned on past observations Maps non-binary symbols to a binary sequence Uses the provided model for the actual encoding and updates the model [courtesy T. 55 .H.

1/8 pel (CIF) Using CABAC entropy coding 5 reference frames in 7 of 8 cases (News: 17 / 25) Both Sequence structure IBBPBBP. QPB=QPP+2 (step size: +25%) Search range: 32x32 around 16x16 predictor Well-known D+λR optimization techniques Bernd Girod: EE398B Image Communication II Video Coding Standards no..26L: Motion Compensation: 1/4 pel (QCIF).. 56 .26L to MPEG-4 MPEG-4: Advanced Simple Profile (ASP) Motion Compensation: 1/4 pel Global Motion Compensation H.Comparison of H.

26L 112 128 Bernd Girod: EE398B Image Communication II Video Coding Standards no.RD Curves: Foreman (QCIF. 10Hz) 39 38 37 Average PSNR(Y) [dB] 36 35 34 33 32 31 30 29 28 27 26 0 16 32 48 64 80 96 Bit-rate [kbit/s] [source: ITU-T VCEG] >30% MPEG-4 H. 57 .

58 .RD Curves: Flowergarden (CIF.26L 1024 1280 1536 1792 2048 2304 Bit-rate [kbit/s] [source: ITU-T VCEG] Bernd Girod: EE398B Image Communication II Video Coding Standards no. 30Hz) 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 0 256 512 768 Average PSNR(Y) [dB] >30% MPEG-4 H.

59 .Bernd Girod: EE398B Image Communication II Video Coding Standards no.

Bernd Girod: EE398B Image Communication II Video Coding Standards no. 60 .

61 .Bernd Girod: EE398B Image Communication II Video Coding Standards no.

Sign up to vote on this title
UsefulNot useful