A
New Video Coding Scheme using Warped
Reference Pictures
[JVET-J0046]
Jung Won Kang (ETRI)
Dae Young Lee, Tea Hyun Kim, Gwang Hoon Park (KHU)
Contents
q Introduction
q Problem Statement
q Warped Picture for Reference
q Proposed Coding Scheme
q Experimental Results
q Further Study
JVET-J0046 2
Introduction
q Trend-shift of captured video sequences by users using from tripod-based to
nontripod-based cameras
v Nontripod-based cameras are easier to move and more likely to create complex movement
than tripod-based cameras
Tripod-based cameras Nontripod-based cameras
(Smartphone, drone, action cam)
q Various videos with global motion due to camera movement (example)
Tripod-based capture Nontripod-based captures Nontripod-based captures
by Dron by Action Cam
JVET-J0046 3
Problem Statement
q Global motion by camera movement
v Camera movement causes global motion
Ø With nontripod-based cameras, camera movements occur very often
Ø Tripod-based cameras can also cause camera movement by camera work
v Type of global motion
Ø Linear global motion : panning, tracking
Ø Nonlinear global motion : rotation, dolly-in(zoom-in), dolly-out(zoom-out)
Ø In the conventional method, it is difficult to obtain a high coding efficiency
when nonlinear global motion occurs
No global motion Linear global Nonlinear global motion Nonlinear global motion
motion dolly-in by car black box rotation by dron
JVET-J0046 4
Problem Statement
q Inter prediction when nonlinear motion occurs in HEVC
v Motion Vector derivation examples in HEVC
Ø The prediction region is more finely subdivided when nonlinear motion occurs
JVET-J0046 5
Warped Picture for Reference
q Warped reference picture (WRP)
v Warped reference picture made from reference picture for better prediction
v Reflects geometric movements
Ø rotation, translation, enlargement, reduction, projection
v As a model for expressing geometric transformation relations, homography is
used
Ø Gemoetric transformation relations are represented by 3x3 matrix
Ø matrix generated by homography (warping matrix)
JVET-J0046 6
Warped Picture for Reference
q Features for WRP
v Higher similarity to the current picture compared to reference picture
v Reflects global movement
Ø Linear movement : panning, translation
Ø Nonlinear movement : rotation, zoom-in, zoom-out, dally-in, dally-out, projection
Current Picture Reference Picture Warped Reference Picture
Lower Similarity
A B C
Higher Similarity
JVET-J0046 7
Proposed Coding Scheme
q Reference picture structure with WRP
v WRPs are generated from reference pictures using warping matrix
Ø Warping matrix is signalled in slice header
Ø Each warping matrix takes 256 bits (32bits/parameter)
Ø Each slice has maximum of 4 warping matrices (1,024 bits)
v Current picture can be predicted from both reference picture and WRP
v Warping matrix is only signalled when WRPs are used
for inter prediction in the picture
v Warping process of generating WRP is performed
once if there is the same reference picture in L0 and
L1
JVET-J0046 8
Proposed Coding Scheme
q Inter predictions using WRP
v Motion Compensation based on WRPs and general reference pictures
Ø WRP flag is signalled to indicate whether WRP is used for a PU
o When WRP flag is true, WRP is used for the PU
o When WRP flag is false, general reference picture is used for the PU
Ø Reference list number(L0, L1) and reference picture index of the WRP
are the same as the corresponding reference picture
JVET-J0046 9
Proposed Coding Scheme
q Inter predictions using WRP
v Merge with WRP selection
Ø When a current PU is coded as merge mode, WRP flag is also copied along with
motion information from the corresponding merge candidate
JVET-J0046 10
Experimental Results
q Performance
v Sequence groups
Test Groups Video Sequences Remarks
Group A JVET CTC test sequences
Group B JVET sequences with global motion
Group C KHU-created video sequences with global motion Can be donated to JVET
v BD rate(Y) compared to HM16.9
Ø Group A: -0.68%
Ø Group B: -11.54%
Ø Group C: -22.88%
v Group A has mostly no global motion
Ø Low coding efficiency improvement for sequences without global motion
v Group B & Group C contain various degrees of global motion
Ø Higher coding efficiency compared to Group A
JVET-J0046 11
Experimental Results
q Group A : JVET CTC test sequences
BD-rate (%) for RA
EncT DecT
Group A (Anchor : HM16.9)
(%) (%) q Group B : JVET sequences with global motion
Y Cb Cr
Class A1 -0.03 -0.04 -0.06 321 161 BD-rate (%) for RA
EncT DecT
Class A2 -3.22 -3.19 -3.11 320 174 Group B (Anchor : HM16.9)
(%) (%)
Class B -0.02 -0.41 -0.32 359 138 Y U V
Class C -0.25 -0.71 -0.81 332 137 BlueSkyPart -16.22 -17.31 -13.94 196 468
Class D -0.56 -0.85 -1.17 345 142 JetsPart -7.56 -8.71 -6.32 134 177
Class E 0.03 0.05 0.01 243 138 ShieldsPart -18.99 -12.06 -10.86 189 401
Overall -0.68 -0.88 -0.82 325 148
SpincalendarPart -15.87 -18.52 -21.76 183 230
StationKtaPart -31.71 -25.30 -28.69 200 483
q Group C : KHU-created sequences TractorPart -18.39 -16.37 -16.28 232 347
IceAerial -13.28 -11.51 -10.99 216 518
BD-rate (%) for RA IceRiver -4.75 -5.49 -5.41 227 263
EncT DecT
Group C (Anchor : HM16.9)
(%) (%) IceRock2 -3.46 -3.67 -3.38 218 236
Y U V IceRock -1.82 -2.06 -1.99 212 161
KHU_ Netflix_Aerial -15.65 -14.79 -14.46 197 326
-32.83 -27.36 -27.54 232 364
Rotate
Netflix_FoodMarket2 -3.55 -4.74 -4.26 230 366
KHU_ Netflix_FoodMarket -9.62 -6.74 -7.84 239 459
-16.91 -13.11 -13.36 230 260
Zoom_in
Netflix_RitualDance -0.76 -0.73 -0.49 280 163
KHU_ Overall -11.54 -10.57 -10.48 211 328
-15.59 -12.97 -13.32 226 296
Zoom_out
Overall -22.88 -18.77 -19.02 229 313
JVET-J0046 12
Experimental Results
q Visualized analysis
v MV information
Ø Since WRP already reflects global motion, each MV becomes small and more similar
to the neighboring PU
Ø Number of MVs generated by PU segmentation is decreased
v Merge region is in a wider area
JVET-J0046 13
Experimental Results
q Visualized analysis - SpincalendarPart POC 4 (Partial Rotation)
HM 16.9 Proposed
Motion Vector
Merge Group
JVET-J0046 14
Experimental Results
q Visualized analysis - ShieldsPart POC 50 (Partial Zoom-in)
HM 16.9 Proposed
Motion Vector
Merge Group
JVET-J0046 15
Experimental Results
q Visualized analysis - RollerCoaster POC 29 (Partial Projection)
HM 16.9 Proposed
Motion Vector
Merge Group
JVET-J0046 16
Experimental Results
q Visualized analysis - StationKtaPart POC 1 (Partial Zoom-out)
HM 16.9 Proposed
Motion Vector
Merge Group
JVET-J0046 17
Experimental Results
q Visualized analysis – KHU_Rotate_fast POC 19 (Fast Rotation)
HM 16.9 Proposed
Motion Vector
Merge Group
JVET-J0046 18
Experimental Results
q Visualized analysis – KHU_Zoom_out_fast POC 1 (Fast Zoom-out)
HM 16.9 Proposed
Motion Vector
Merge Group
JVET-J0046 19
Further Study
q Issues
v Extend block size
Ø Proposed method tends to minimize PU segmentation in video sequences with
nonlinear global motion
Ø Coding efficiency can be improved with larger CTU size
o In the proposed method, CTU size is 64x64
v Parallel processing
Ø The process for acquiring WRP at encoder can be parallelized in task-level
o It is frame-based operation for each reference frame
v At decoder, WRP can be generated on block basis instead of frame basis
Ø Only for a block using WRP as a reference picture, the Warped Reference Block can
be generated
JVET-J0046
Further Study
q Issues
v Appropriate entropy coding
Ø Need entropy coding suitable for the characteristics of the proposed method
v Warping matrix compression
Ø Need efficient coding for warping matrix
o In the proposed method, warping matrix is not compressed
o Each warping matrix requires 256 bits (32 bits/parameter) and each slice has maximum
of 4 warping matrix (1,024 bits)
Ø Bits for the warping matrix are considered to decide whether to use WRP.
If effective compression for warping matrix parameters is applied, the WRP will be
applied more.
Ratio of Ratio of Ratio of
bits for warping bits for warping bits for warping
QP QP QP
matrix over total matrix over total matrix over total
bits (%) bits (%) bits (%)
22 0.15 22 0.82 22 1.27
27 0.16 27 1.68 27 2.55
Group A 32 0.32 Group B 32 2.87 Group C 32 4.58
37 0.51 37 4.39 37 7.06
Total 0.29 Total 2.44 Total 3.86
JVET-J0046 21
JVET-J0046 22
