MPEG-4 Compliant Video Encoding: Analysis and Rate Control Strategies

Paul0 Nunes, Fernando Pereira

Instituto Superior TCcnico - Instituto de Telecomunicagbes
AV.Rovisco Pais, 1049-001Lisboa, Portugal
Phone: +3512184184 60; Fax: +35121 841 84 72
e-mail: {paulo.nunes, fernando.pereira)@lx.it.pt

Abstract and picture memories and the decoding speed. Although

the Video Buffering Verifier is essentially defined in
Any set of MPEG-4 elementary bitstreams building a
terms of decoding operation, it is a task of the encoder to
video scene can only be considered profile@level
compliant if it does not violate the MPEG-4 Video implement it in order to guarantee that it is not violated by
“shaping” the encoded data in a way that meets the
Buffering Verifier constraints for the chosen
relevant constraints. Such task is mainly dealt with by the
profile@ level. This paper analyses this mechanism,
rate control mechanism that takes into account the status
discussing its major features and drawbacks, notably in
of the several Video Buffering Verifier buffers in order to
comparison with altemative solutions. Furthermore, rate
optimalIy control the encoder.
control strategies to guarantee a compliant encoding, are
This paper analyzes the three MPEG-4 Video
proposed.
Buffering Verifier mechanism models, the Video
Reference Memory Verifier (W),the Video
Complexity Verifier (VCV), and the Video Rate Buffer
1 Introduction Verifier (VBV), highlighting its major features and
drawbacks, notably in comparison with relevant
The MPEG-4 standard, MPEG’s most recent altemative models. Additionally, this paper proposes
achievement, is aimed to define an audiovisual coding video coding rate control strategies defined in terms of the
standard to address the emerging needs of the reactions taken by the rate control module to guarantee a
communication, interactive, and broadcasting service compliant encoding, considering the status of the various
models as well as of the mixed service models resulting Video Buffering Verifier buffers.
from their technological convergence. In this context, the
MPEG-4 standard has been designed to be generic in the
sense that it is not targeted for a particular application but
2 The MPEG-4 video buffering verifier
includes many coding tools and algorithms that can be
The idea of using a Video Buffering Verifier to bound
used in a variety of applications under different situations,
the decoding complexity of a given bitstream is not new,
notably in terms of functionalities and bit rates. Since it is
and was already adopted in the previous MPEG video
not reasonable that all terminals support the whole
coding standards, MPEG-1 and MPEG-2. There, the
MPEG-4 toolbox, subsets of it have been defined, through
major purpose of the Video Buffering Verifier mechanism
the concepts of profiles and levels. A subset of the syntax
was to set some restrictions on the maximum variability of
and semantics, corresponding to a subset of tools of the
the number of bits per picture, especially in the case of
MPEG-4 visual standard [l], defines a visual profile,
constant bit rate operation, and thus on the complexity of
while sets of restrictions within each visual profile, in
the encoded‘video streams.
terms of computational resources, define the various
In MPEG-4, the several video objects composing a
levels. Profiles and levels will be at the center of MPEG-4
scene may vary in size along time and may be encoded at
deployment.
different VOP rates. To limit the decoding complexity of
In order that a set of visual bitstreams building a scene
the corresponding bitstreams it is then also necessary to
may be considered compliant with a given MPEG-4 visual
set some limits on the variability of the number of MBls
profileOleve1, allowing interoperability, it must not
and on the picture memory required to store the decoded
contain any disallowed syntactic element and additionally
data. This constitutes the major novelty of the MPEG-4
it must not violate the Video Buffering Verifier
Video Buffering Verifier mechanism, relatively to the
mechanism constraints. This mechanism, based on virtual
previous MPEG standards, since it does not only bound
buffers, allows the encoder to limit the decoding
computational resources required, notably the bitstream

0-7803-6514-3/00/$10.0002000IEEE 54

Authorized licensed use limited to: Qualcomm. Downloaded on June 25,2010 at 20:50:26 UTC from IEEE Xplore. Restrictions apply.
the bitstream buffer memory but also the MB decoding
capacity and the MB picture memory.
The MPEG-4 Video Buffering Verifier mechanism
consists of three different models, each one defining a set
of rules and limits to Verify if a specific Of de~oder
resources is Within the VdUeS allowed by the
corresponding profile and level definition:
Video Reference Memory Verifier (VMV) - This
model is used to verify if the picture memory required
at the decoder for the decoding of a given scene does
not exceed the values specified for the corresponding
profile and level. The model is defined in terms of the
VMV buffer size, which is the maximum number of
decoded MBs that the decoder can store during the
decoding process (maximum in the sense that it is there
for sure). This buffer accumulates all the decoded MBs
of all VOPs and stores them until they are no longer
needed for the prediction of other VOPs.
Video Complexity Verifier (VCV) - This model is
used to verify if the computational power, defined in
terms of MB/s, required at the decoder does not exceed
the values specified for the corresponding profile and
level. The model is defined in terms of the VCV MB/s
decoding rate and VCV buffer size and is applied to all
MBs in the scene. If arbitrarily shaped VOs exist in the
scene, an additional VCV buffer and VCV decoding
rate is also defined, to be applied only to the boundary
MBs. The ratio between the VCV buffer size and the
VCV decoding rate defines the VCV latency, L, i.e. the
time it takes to decode a full VCV, and sets the
minimum latency of the decoding process. At each
vop decoding time, the number of m Sin the vop is
added to vcv buffer, and its occupancy decreases at
constant VCV decoding rate until the VOP decoding
has finished or a new VOP is added to the VCV buffer.
3. Video Rate Buffer Verifier (VBV) - This model is
used to verify if the bitstream memory required at the
decoder does not exceed the values specified for the
corresponding profile and level definition. The model is
defined in terms of the VBV buffer size which is the
maximum amount of bits that the decoder can store in
the bitstream memory. The encoded bits for each VOP
enter the VBV at constant or variable bit rate, and are
instantaneous removed from this buffer at the VOP
decoding time.
In order that a given set of elementary streams (ESs)
building a visual scene may be considered compliant with
a given profile and level, the encoder must guarantee that
none of the above mentioned buffers overflows and,
additionally, it must also guarantee that, in certain
circumstances, the VBV never underflows. Bitstream
compliance with a given profile8level guarantees that the
resources required at the decoder do not exceed a certain
pre-defined amount.
55
Authorized licensed use limited to: Qualcomm. Downloaded on June 25,2010 at 20:50:26 UTC from IEEE Xplore. Restrictions apply.
3 MPEG-4 video buffering verifier analysis
Although the Video Buffering Verifier mechanism is
essentially described in terms of decoder operation, it is a
task of the encoder to implement it and to guarantee that it
is not violated. For this, the encoder has to “shape” the
encoded data in a way that it does not violate the
constraints imposed by this mechanism. If any of the
Video Buffering Verifier models tends to be violated, the
encoder has to take appropriate countermeasures to avoid
it. Such task is mainly dealt with by the rate control
mechanism that takes into account the scene
characteristics, the encoding results, and the status of the
several Video Buffering Verifier buffers for the best
control of the encoder [2].
3.1 Analysis of the VMV model
The VMV occupancy due to a given VOP depends on
its size in MB units and on the time interval during which
the VOP stays in the VMV buffer. It is possible to
identify, at least, two major types of picture memory
required by any receiving terminal:
Decoding memory to store the decoded VOPs and the
corresponding predictions (reference VOPs), during the
. decoding urocess.
- I
Composition memory to store the decoded vops
(composition units) and the resulting composed scene,
during the composition and presentation process.
Depending on how the two mentioned types of picture
memory are considered in the VMV model there are
several alternatives for the modeling of this capability.
ahemative VMV are
compared with the adopted MPEG-4 VMV model:
Global memory approach - In this approach, the
VMV model takes into account all the memory needed
for the decoding and for the composition of the
decoded VOPs. In this case, a given VOP consumes
picture memory from the instant it starts being decoded
until it has been presented and is no longer needed for
the prediction of other VOPs.
Decoding memory approach - In this approach, the
VMV model takes only into account the memory
needed for reconstructing the encoded VOPs. Here, a
given VOP consumes picture memory from the instant
it starts being decoded until it has been decoded and is
no longer needed for the prediction of other VOPs.
-
MPEG-4 V M V model approach When the encoder
does not use B-VOPs, the MPEG-4 VMV follows a
global memory approach since it specifies that each
VOP should stay in the VMV buffer until the following
VOP starts being composed. When B-VOPs are used,
the MPEG-4 V M V specifies that these VOPs should be
released from the VMV buffer at their decoding times
 plus the VCV latency, i.e. at the time corresponding to
the maximum decoding duration of the B-VOP. Thus,
B-VOPs are released from the VMV buffer before the
following VOP (in composition order) starts being
composed. This approach is neither a decoding memory
approach, otherwise B-VOPs should be released
immediately after their decoding has finished, or a
global memory approach since B-VOPs are released
from the VMV buffer before their presentation has
finished.
The adopted MPEG-4 VMV solution underestimates
the total picture memory needed at the decoder, notably
the memory needed for the composition and presentation
of B-VOPs, but overestimates the picture memory needed
only for the decoding process. Since VOP composition is
not normative in MPEG-4, it would cause no surprise if
the decoding memory approach had been adopted.
3.2 VMV rate control strategies
For each target encoding time instant, the encoder
verifies if the amount of picture memory required for
decoding the given VOP or set of VOPs, during its
lifetime at the decoder, does not exceed the maximum
amount of memory available for the selected profile and
level. For this, the encoder estimates the picture memory
usage during the decoding period of each VOP, which can
be given by the following expression:
vmv(t) = vmv(to) + mi& - t o ) - H , M , ]- xkAM , -u(t - 1 , )
where to and r, are, respectively, the start and end
decoding times of the given VOP, H is the VCV decoding
rate, M I is the total number of MBs in the VOP, u ( t ) is
the unit step function, A represents the set of VOPs to be
released from memory in the interval j,,.t,], and rk their
corresponding releasing time instants; vmv(ro) denotes
the VMV occupancy immediately before start decoding
VOP i; min[(f - r , ) . H , M , ] denotes the allocation of
picture memory as the decoding process progresses (while
the VCV occupancy decreases at decoding rate H, the
V M V occupancy increases at the same rate); finally the
picture memory released during the decoding period of the
given VOP is given by CkE,Mk -u(t - f k ) .
By estimating in advance the local maxima of (1) the
encoder is able to take preventive actions to avoid possible
violations of the VMV, such as:
Skip the encoding of one or more of the larger VOPs.
Avoid B-VOPs (when applicable).
0 Use VOPs with less MBs (in the bounding box), by
merging VOs or reducing its number (this solution
impacts on the ‘authoring’ of the scene).
Authorized licensed use limited to: Qualcomm. Downloaded on June 25,2010 at 20:50:26 UTC from IEEE Xplore. Restrictions apply.
0 Signal the impossibility to encode the given scene with
the amount of picture memory provided by the chosen
profile0level combination (this may lead to the choice
of a more powerful profile@level).
An overflow of the W buffer may lead to a situation
where the decoder is not able to correctly decode part or
all MBs in one or more VOPs (due to the lack of picture
memory to store the decoded VOPs) and thus to a coding
desynchronization between encoder and decoder.
3.3 Analysis of the VCV model
In MPEG-4 the load on the decoder is measured by the
occupancy of the VCV buffer. There are several ways to
measure the complexity of the encoded data, which can be
defined by any of the following parameters:
0 Number of MBs per second.
Number of MBs per MB type (opaque, boundary, or
transparent) per second.
0 Number of MBs per MB coding type (I-MB, P-MB, B-
MB, etc) per second.
Number of arithmetic instructions and memory
R e a m r i t e operations per second.
In terms of the VCV model operation, the use of these
measures is similar. At each VOP decoding time, the
adopted complexity measure for the incoming VOP is
added to the VCV buffer occupancy (e.g. #MBs) which
(1) decreases at a constant rate defined in the appropriate data
complexity units by the profile@level combination (e.g.
#MB/s), until the next VOP decoding time. The sizes of
the VCV buffers, using the appropriate complexity
measure units, define the maximum load that the decoder
can accept, while the drain rates of the VCV buffers
define the maximum decoding speed of the decoder.
There are several alternatives for the modeling of the
decoder computational capability, notably depending on
the number of buffers and decoding rates used. Below
several VCV model approaches, based on virtual buffers,
are compared the with the adopted MPEG-4 VCV model:
Single buffer with single decoding rate - This is the
simplest VCV model based on the virtual buffer
approach which consists in a single VCV buffer that
accumulates the complexity of the encoded data and a
single decoding rate that defines the speed at which the
decoder can decode this data. In this case, decoding is
performed in a first in first out W O ) basis. There are
two major possible variations for this approach:
0 Without MB weights - All MBs have the same
weight in terms of decoding complexity. This approach
has the advantage of being rather simple. Its major
drawback comes from the fact that decoders have to be
designed to deal with worst case scenarios (heaviest
MB types) in terms of decoding capacity.
56
0 With MB weights - The MBs are divided into A VCV model where the three different types of
classes, e.g. according to its coding type, and each MB MBs (transparent, opaque, and boundary) could be
class has an associated decoding complexity weight. discriminated in terms of decoding complexity would
The VCV occupancy due to a given VOP is a weighted be more adequate than the current MPEG-4 VCV
sum of its MBs. As in the preceding case, the VCV model. In this context, the single buffer with single
buffer is emptied at a fixed VCV decoding rate. The decoding rate with MB weights or the multiple buffers
main advantage of this approach is to model more with multiple decoding rates solutions appear as good
closely the real decoding complexity of a given set of candidates for the VCV modeling. In the first case, the
ESs building a visual scene. However, the definition of impact of transparent MBs could be reduced by
meaningful MB weights is not a straightforward task. assigning them a lower MB weight, while for the
second case this could be done by defining a higher
Single buffer with multiple decoding rates - This
buffer size and higher decoding rate.
approach assumes also a FIFO decoding; however, here
each MB class has its own decoding rate. This
News C I F [ 4 V O s l 6’ 15 Hz
approach is equivalent to the previous approach when
450
the MB weights are the ratios between the different MB
400
class decoding rates and a reference decoding rate.
350
Multiple buffers with multiple decoding rates - This 300
VCV model approach assumes some degree of 2 250
parallelism in the decoder which may happen in Ii 200
hardware-based decoders with dedicated hardware for 150
decoding some types of MBs, e.g. a module for 100
decoding MBs without shape information or 50
completely opaque, and another module for decoding 0
MBs with shape information (i.e. boundary MBs). The 0 50 100 150 208 256 300
Frame
VCV model consists of N buffers, one for each MB
class, and the associated decoding rates. The decoding News C I F [ 4 V O s l @ IS HZ
time is determined by the last buffer to be emptied.
VCV B u r fer Occupancy -
MPEG-4 VCV model approach - The MPEG-4VCV VMV B u f f e r Occupancy

model follows a multiple buffers, multiple decoding

rate approach with two buffers: the VCV, accumulating 100

all MBs, and the Boundary VCV, accumulating only

boundary MBs. However, these two MB classes are not
mutually exclusive since boundary MBs are counted in
both VCVs. With this approach, the MPEG-4 VCV
model tries to include in the same model the parallel
nature of a pure multiple buffer with multiple decoding
rates with the serial nature of a single buffer with
multiple decoding rates. Since the VOP MBs are
organized in two classes (boundary and all), opaque
and transparent MBs are treated exactly in the same
way. This fact has a high impact in the measured
complexity of certain scenes, where transparent MBs,
although not canying “rich content”, lead to an
overestimation of the scene complexity and thus to the
impossibility of compliantly encode scenes which were
expected to be encoded with a certain profile@level
combination. This situation is illustrated in Figure 1 for
the News sequence (4 VOs) in CIF format encoded at
15 Hz with the MPEG-4 core profile @ level 2; the
high percentage of transparent MBs in the scene causes
the VCV to overflow. In this case, the feedback
mechanism that prevents the violation of the VCV
model has been disabled, thus whenever the encoder
detects a violation of the VCV model, the bitstream is
generated but is signaled as non-compliant.
0 50 I00 150 200 250 300
Frame
-
Figure 1 Sequence ‘News’ encoded with
core profile 0 level 2
3.4 VCV rate control strategies
In order to produce valid bitstreams, the encoder must
ensure that the two VCV buffers never overflow and that
each VOP is completely decoded in time. To meet these
requirements the encoder analyzes the set of VOPs to be
encoded for each target encoding time (typically the
multiples of l/fi , where fi. denotes the temporal coding
rates for the various objects in the scene) and tests if the
above conditions can be verified for this set of VOPs.
Whenever the rate control mechanism detects a possible
violation of the VCV model, it can take one of the
following actions to decrease the occupancy of the VCV
buffers:

57

Authorized licensed use limited to: Qualcomm. Downloaded on June 25,2010 at 20:50:26 UTC from IEEE Xplore. Restrictions apply.
 Skip one or more VOPs for that time instant.
Reduce the number of MBs with shape information (in
case this is the limiting factor), e.g. by changing the
shape of the objects or merging objects (if this is an
acceptable solution for the application in question)
Decrease the size of some VOPs, e.g. by splitting an
object into two or more objects it may be possible to
reduce the total number of MBs in the resulting
bounding boxes (if this is an acceptable solution for the
application in question).
Notice that the overflow of one (or both) of the VCV
buffers may lead to the incomplete decoding of one or
more VOPs and thus to a coding desynchronization
between encoder and decoder.
3.5 Analysis of the VBV model
The encoder needs to shape the encoded data in order
to guarantee that the encoded bits for each VOP are
available for decoding at the time corresponding to the
VOP decoding time. This can be done by allocating the
available channel rate in a pre-defined or rigid way -@ed
bandwidrh allocation, or dynamically, by allocating the
available channel rate among the several VOs in the scene
according to the VO’s characteristics (e.g. size, activity,
complexity, etc.) - dynamic bandwidth allocation.
As for the VMV and VCV models, MPEG-4 Visual
does not standardize any method for achieving proper
VBV buffer control. This freedom allows the
implementers to chose the best buffer control mechanism
that guarantees proper VBV operation for each case. As
shown in 131 and [4], joint rate control strategies, where
the available bit rate is dynamically allocated among the
several VOs in the scene can achieve better performance,
when compared to independent rate control, where the
available resources are rigidly allocated.
Although the MPEG-4 VBV specification states that in
the case of a scene composed by multiple VOs, each with
one or more VOLs, the VBV model should be applied
independently to each VOL [11, the MPEG-4 VBV model
does not prevent the dynamic allocation of resources and
combined buffer control, provided that the sum of all
individual buffers does not exceed the profile@level
limits.
3.6 VBV rate control strategies
Whenever the rate control mechanism detects a
possible violation of the VBV model (underflow or
overflow), it should react with the most adequate action in
the context of the application in question. This can be
done in two distinct modes:
-
Preventive mode Whenever the VBV occupancy
reaches certain high thresholds, the encoder takes
adequate action(s) to neutralize the unwanted situation.
58
Authorized licensed use limited to: Qualcomm. Downloaded on June 25,2010 at 20:50:26 UTC from IEEE Xplore. Restrictions apply.
0 Reactive mode - Whenever the coding decisions taken
by the encoder lead to a violation of the VBV
constraints, new coding decisions are adopted to avoid
this situation.
The reactive mode is typically more effective in
achieving proper VBV buffer control, since it usually
involves choosing iteratively from multiple coding
decisions the one that, fulfilling the VBV constraints, has
the best tradeoff in terms of rate-distortion operation.
However, the reactive mode is not very suited for real-
time encoding since it may involLveextra delays that are
not acceptable for real-time conditions. Besides being less
effective, the preventive mode cannot completely
guarantee that the VBV will not be violated; the less
severe the VBV thresholds to operate the preventive mode
are, the higher is the risk that a VBV violation happens.
Typically, the encoder rate control can take one of the
following actions to prevent or avoid VBV violation:
0 Adjust the MB quantization parameters (texture
losses/distortion).
0 Adjust the shape encoding losses (distortion).
0 Skip the encoding of the incoming VOP(s).
0 Introduce stuffing bits (increase the VBV buffer
occupancy if underflow is the problem).
Both the overflow and underflow of the VBV buffer
must be avoided. Notice that the VBV buffer overflow
may lead to a loss of data and thus to the incomplete
decoding of one or more VOPs, while the VBV buffer
underflow may prevent the decoder to decode the
incoming data on time and thus both may lead to a coding
desynchronization between encoder and decoder.
4 Finalremarks
This paper analyzed the MPEG-4 Video Buffering
Verifier mechanism highlighting its major features and
drawbacks and compared the adopted MPEG-4 models
with other alternative models. This analysis showed that
the MPEG-4 models can still be improved, notably the
VMV and VCV models: the VMV to better reflect the fact
that composition is not normative in MPEG-4, while the
VCV to better reflect the real complexity of visual scenes
containing a high percentage of transparent MBs.
111 ISOmEC 14496-2:1999, Information Technology - Coding
of Audio-visual Objects - Part 2 Visual.
P I P. Nunes, F. Pereira, “Implementing the MPEG-4 Natural
Visual Profiles and Levels”, Doc. M4878, 48th MPEG
meeting, Vancouver, July 1999.
131 P. Nunes, F. Pereira, “Rate Control for Scenes with
Multiple Arbitrarily Shaped Video Objects”, PCS’97,
Berlin, September 1997.
141 P. Nunes, F. Pereira, “Object-Based Rate Control for the
MPEG-4 Visual Simple Profile”, WIAMIS’99, Berlin,
May 1999.