1

Moving/motionless foreground object detection using

fast statistical background updating

W-Y Chiu and D-M Tsai*

Department of Industrial Engineering and Management, Yuan-Ze University, Tao-Yuan, Taiwan

Abstract: In video surveillance, the detection of foreground objects in an image sequence from
a still camera is very important for object tracking, activity recognition and behaviour
understanding. The conventional background subtraction cannot respond promptly to dynamic
changes in the background, and temporal difference cannot accurately extract the object shapes
and detect motionless objects. In this paper, we propose a fast statistical process control scheme
for foreground segmentation. The proposed method can promptly calculate the exact grey-level
mean and standard deviation of individual pixels in both short- and long-term image sequences
by simply deleting the earliest one among the set of images and adding the current image scene
in the image sequence. A short-term updating process can be highly responsive to dynamic
changes of the environment, and a long-term updating process can well extract the shape of a
moving object. The detection results from both the short- and long-term processes are
incorporated to detect motionless objects and eliminate non-stationary background objects.
Experimental results have shown that the proposed scheme can be well applied to both indoor
and outdoor environments. It can effectively extract foreground objects with various moving
speeds or without motion at a high process frame rate.

Keywords: motion detection, surveillance, foreground segmentation, statistical process control

1 INTRODUCTION detection can promptly absorb the changes in the

In video surveillance, the detection of moving objects in
video sequences from a stationary camera is very
important for the success of object tracking, incident
detection, activity recognition and behaviour under-
standing. Motion detection aims to segment the fore-
ground pixels corresponding to moving objects from the
background in a scene image. A robust motion detection
algorithm has three properties: (1) high responsiveness to
dynamic environments such as gradual changes in
outdoor light, sudden on/off switching of indoor lights
and door opening/closing;(2) accurate extraction of
foreground shapes; and (3) computational efficiency for
real-time implementation. Responsive foreground
The MS was accepted for publication on 21 April 2011.
* Corresponding author: Du-Ming Tsai, Department of Industrial
Engineering and Management, Yuan-Ze University, 135 Yuan-Tung
Road, Nei-Li, Tao-Yuan, Taiwan; email: iedmtsai@saturn.yzu.
edu.tw
background and prevent the erroneous detection of
non-stationary background objects such as the move-
ment of chairs as foreground regions. Accurate extrac-
tion of the complete shape of a foreground object is very
important for object and activity recognition in appear-
ance-based methods. Fast computation in foreground
segmentation means that more time is available for a
high-level classification process.
Hu et al.1 categorised motion segmentation ap-
proaches as background subtraction, temporal differ-
ence and optical flow. Optical flow represents the
motion of objects by a velocity vector for each pixel in
two consecutive images, and was originally proposed by
Horn and Schunck.2 The method proposed by Lucas
and Kanade3 is one of the most popular versions of
optical flow methods for motion detection. Gibson and
Spann,4 Liu et al.,5 Kim and Kak,6 and Chamorro-
Martinez and Fernandez-Valdivia7 have applied optical
flow to detect moving objects in video sequences.

IMAG 91 # RPS 2011 DOI: 10.1179/1743131X11Y.0000000016 The Imaging Science Journal Vol 0
2 W-Y CHIU AND D-M TSAI
Optical flow methods for motion detection are rela-
tively computationally slow, and cannot detect motion-
less foreground objects.
Temporal difference8,9 calculates the difference of
pixel features between two consecutive scene frames in
an image sequence. It is very computationally efficient,
and well accommodates environmental changes, but it
generally can only extract partial shapes of mov-
ing objects. Since temporal difference relies on the
measure of pixel variations in consecutive image
frames, it is prone to failure in detecting motionless
foreground objects.
Background subtraction detects moving objects in
an image by evaluating the difference of pixel features
between the current scene image and the reference
background image. This approach is very computa-
tionally fast, but it is also very sensitive to environ-
mental changes. To compensate for illumination
changes, a background model updating process is
generally adopted, but that process requires addi-
tional computation time. In this paper, we propose a
fast foreground segmentation scheme that can
effectively detect moving foreground objects and
motionless humans, while being promptly adaptive
to environmental changes in the background. It is
computationally very fast and accurate in updating
both short- and long-term backgrounds characterised
by the grey-level mean and standard deviation of
each individual pixel over time using the statistical
process control technique.
Many background updating strategies have been
proposed in the literature on foreground segmenta-
tion. Piccardi10 presented a review of background
subtraction techniques with an emphasis on back-
ground modelling algorithms for detecting moving
objects from a static camera. Background model
updating methods are generally based on the analysis
of the grey-level (or colour) histogram taken by each
individual pixel over a limited number of recent
frames. Wren et al.11 developed a system called
Pfinder for person segmentation, tracking and inter-
pretation. This system modelled each pixel of the
background over time with a single Gaussian dis-
tribution. Single Gaussian updating models have also
been adopted by many researchers12–15 for back-
ground subtraction.
A more robust background modelling technique is
to represent each pixel in the background image over
time by a mixture of Gaussians, which was originally
proposed by Stauffer and Grimson.16,17 It has been
The Imaging Science Journal Vol 0
used extensively as a benchmark for the comparison
of motion detection techniques. The background
model can deal with multimodal distributions caused
by shadows, swaying branches, etc. It can handle
slow lighting changes by slowly adapting the para-
meter values of the Gaussians. Since the estimation of
Gaussian parameter values for each pixel in the image
using standard algorithms such as Expectation
Maximisation is computationally prohibitive, recur-
sive updating using a simple linear adaptive filter is
applied for real-time implementation. The background
model is generally considered as a linear combination
of the current background and the current scene image
with a specific learning rate. A slow updating of the
background model cannot respond promptly to the
background changes, whereas a fast updating might
absorb the slow-moving object into the background.
KaewTrakulPong and Bowden18 argued that the
background model of Stauffer and Grimson17 suffers
from slow learning at the beginning of the updating
process. They therefore proposed two different updat-
ing formulations to estimate the Gaussian mixture
model at the beginning phase and at the stable phase
when sufficient image samples are processed. Their
methods improved the accuracy of the initial estimate,
and allowed fast convergence on a stable background
model. Lee19 presented an adaptive learning rate of the
background model to improve the updating con-
vergence without compromising model stability.
McFarlane and Schofield,20 and Manzanera and
Richefeu21 used a recursive approximation of the
temporal median for background estimation. In their
models, the background is updated with a constant
increment/decrement of 1 based on the sign of the
grey-level difference between the current frame and the
background. The method entails only a small cost in
terms of memory consumption and computational
complexity. The variance of each pixel over time is also
increased/decreased by 1 at each frame. The statistics
of the background must be constantly updated with a
pre-determined frequency. This approach might not be
fast enough to update radical changes in the back-
ground. A moving object might not be promptly
identified during the updating.
Rather than modelling the features of each pixel with
Gaussian distributions, Elgammal et al.22 and Ianasi
et al.23 evaluated the probability of a background pixel
using kernel density estimation from very recent
historical samples in the image sequence. Elgammal
et al.24 also used Fast Gauss Transform25 to improve
IMAG 91 # RPS 2011
 MOVING/MOTIONLESS FOREGROUND OBJECT DETECTION
the computation of the Gaussian kernel density es-
timation. The kernel-based density estimation ap-
proach is computationally intensive because it requires
a large data set for accurate estimation. The adaptive
background updating methods discussed above can
well identify moving objects in a dynamically changing
environment by carefully adjusting the updating rate of
the background. They generally encounter the contra-
diction that both non-stationary background objects
and motionless foreground objects are absorbed as
parts of the background, or both motionless fore-
ground objects and non-stationary background objects
are detected as foreground regions.
As mentioned above, background subtraction
cannot promptly respond to dynamic changes in the
background and fails to absorb non-stationary back-
ground objects without a carefully-designed back-
ground updating mechanism. Temporal difference
cannot extract the complete shape of a moving object
and fails to detect motionless human figures. In this
paper, we propose a fast statistical process control
(SPC) scheme for identifying both moving and
motionless foreground objects in a dynamic environ-
ment. A motionless foreground object in this study
means that a moving human figure becomes still in
the video. In SPC, the mean and variance over a
series of observed sample data must first be calcu-
lated. They are then repeatedly updated as the new
sample data are collected. The control limits (i.e. the
thresholds) are then given by some constant multi-
plication of the standard deviation with the mean as
the centre. If the measured value falls outside the
control limits, an anomaly is declared. We assume
that the grey values of foreground and background
points are generally distinctly different.
The current existing background updating models,
either with single Gaussian or multiple Gaussians,
derive the grey-level distribution parameters using
estimation techniques. The representation of the true
current background can be degraded after a very
long-run due to the accumulated errors of back-
ground estimation in each update operation.
The proposed method can quickly calculate the
exact grey-level mean and standard deviation of
individual pixels in both short- and long-term image
sequences by simply deleting the earliest one among
the set of images and adding the current image
scene in the image sequences. It therefore involves
only two arithmetic operations to update the intensity
statistics, regardless of the number of consecutive
IMAG 91 # RPS 2011
3
image frames. The mean of the short-term image
sequence represents a short-term background, which
can be used to detect moving objects and is very
responsive to dynamic changes in the background.
Conversely, the mean of the long-term image se-
quence represents a long-term background, which
can be used to extract the complete silhouettes of the
moving/motionless foreground objects. The proposed
method then takes advantage of both the background
subtraction and temporal difference methods, yet it is
very computationally efficient.
There have also been a few studies based on short-
and long-term information combination for fore-
ground segmentation. Huerta et al.26 proposed an
adaptive dual-background modeling algorithm for
motion and motionless object detection. The back-
ground model is updated using long-term and short-
term learning rates. This method can detect a
motionless person. However, when a background
object changes its status in the scene, it is falsely
detected as a foreground object. Bayona et al.27
combined simple background subtraction and tem-
poral difference for stationary foreground detection.
This mechanism cannot distinguish a motionless
person from the background objects with status
changes. Porikli and Yin28 presented a dual-back-
ground model for temporally static region detection.
Their method can detect a static foreground object
such as abandoned luggage. However, it is very
sensitive to dynamic changes of the environment,
such as a door opening/closing. The proposed
method in this paper not only detects the stationary
foreground object such as an unconscious person on
the floor, but also responsively absorbs non-station-
ary background objects such as a moving chair and
opening/closing curtains as parts of the background.
The organisation of this paper is as follows.
Section 2 presents the statistical process control
scheme for foreground segmentation. Section 3
describes the experimental results on three sets of
video scenarios, and compares the performance of the
proposed method with that of temporal difference
and background subtraction with Gaussian mixture
modelling. Conclusions are given in Section 4.
2 STATISTICAL PROCESS CONTROL FOR
FOREGROUND SEGMENTATION
As aforementioned, background subtraction techni-
ques cannot promptly respond to changes in the
The Imaging Science Journal Vol 0
4 W-Y CHIU AND D-M TSAI
environment such as cups and newspapers placed on
a table, curtain opening/closing in a living room, and
cars on a street parking and moving. Temporal
difference techniques cannot detect motionless fore-
ground objects such as an unconscious person or an
older person falling asleep.
The proposed method uses statistical process
control to identify individual foreground pixels from
both short- and long-term image frames. Statistical
process control29 has been commonly used in ma-
nufacturing to monitor process stability. The most
important aspect of implementing SPC is control
charting. Unnatural patterns in control charts can be
associated with a specific set of assignable causes,
provided that appropriate process knowledge is
available.30 A simple form of control chart for SPC
is given by m¡Ks, where m is the mean of the
measure, s is the standard deviation of the measure
and K is a control constant.
Let ft(x, y) be the 2D spatial scene image of size
R6C at frame t, for x50, 1, 2, …, R21, y50, 1, 2,
…, C21 and t5T, T21, …, T2Nz1, where T
denotes the current time frame. It can be expected
that the grey values for each pixel at (x, y) will be
approximately the same if the pixel at (x, y) is a part
of the static background over the entire N observed
frames. Because a pixel at (x, y) in the background
region has similar grey values over the N time frames,
the grey-level mean mf will represent the background
and the grey-level standard deviation sf will be
approximately zero. Therefore, detection of fore-
ground objects in a series of video images is
equivalent to identifying the anomalies from a stable
process given by the control limits mf¡Ksf. The
statistics mf and sf of the control limits can be
evaluated from both short- and long-term processes.
By considering a large number of recent frames, the
control limits give the long-term process (back-
ground) variation. The proposed SPC scheme then
acts as background subtraction to extract the
moving/motionless foreground objects with accurate
shapes. In contrast, by considering a small number of
recent frames, the control limits then present the
short-term process variation. The proposed SPC
scheme can therefore act as a temporal difference
that responds quickly to environmental changes.
2.1 Short- and long-term background statistics
In this paper, we propose a fast update process of
grey-level mean and variance from a series of
The Imaging Science Journal Vol 0
temporal images to set up the control limits. The
multiple temporal images of the background will
present approximately the same grey value with a
small variance. The grey values in a foreground
region will be distinctly different from those of the
background, and therefore outside the control limits
given by mf¡Ksf. The grey-level mean and variance
of multiple temporal images can be quickly and
precisely calculated by deleting the earliest image in
the series of the historical image frames and adding
the current video image. The statistical process
control procedure for foreground segmentation is
described in detail as follows.
Let FN5{ft(x,y), t5T21, T22, …, T2N} be a
series of N consecutive image frames, where T denotes
the current time frame. The grey-level mean and
variance of multiple temporal images are given by
1
mT{1 ðx,yÞ~ ST{1 ðx,yÞ (1)
N
s2T{1 (x,y)~
ET{1 ½f 2 (x,y){fET{1 ½f (x,y)g2 ~
1 2
S (x,y){m2T{1 (x,y) (2)
N T{1
2
where ST21(x,y) and ST{1 ðx,yÞ are defined, respec-
tively, as the sum of ft(x,y) and sum of square of ft(x,y)
over the N image frames, i.e.
XN
ST{1 ðx,yÞ~ fT{i ðx,yÞ, Vðx,yÞ (3)
i~1
X
N
2 2
ST{1 ðx,yÞ~ fT{i ðx,yÞ, Vðx,yÞ (4)
i~1
When the grey-level mean and variance at time frame
T21 are obtained, the upper and lower control limits
for foreground-pixel detection in image frame fT(x,y)
can be given by mT21(x,y)¡KsT21(x,y), where K is a
control constant. If the grey-level fT(x,y) at pixel
coordinates (x,y) is out of the control limits, the pixel is
then considered as a foreground point. Otherwise, it is
classified as a steady background point. The detection
result is represented by a binary image BT(x,y), where
BT (x,y)~
(
0 (background), if ft (x,y) [ mt{1 (x,y)+Kst{1 (x,y)
1 (foreground), otherwise (5)
Since the grey values between foreground and back-
ground points are generally distinctly different, the
control constant K is set at 5 in this study, rather than
the commonly-used three-sigma in SPC.
IMAG 91 # RPS 2011
 MOVING/MOTIONLESS FOREGROUND OBJECT DETECTION
Once the foreground objects in image frame fT(x,y)
are segmented from the background, the grey-level
mean mT and standard deviation sT at the current
time frame T can be quickly updated from ST21(x,y)
2
and ST{1 ðx,yÞ. That is
1 of the number of image frames N. Both short- and
mT ðx,yÞ~ ST ðx,yÞ (6)
N
1 2
s2T ðx,yÞ~ S ðx,yÞ{m2T ðx,yÞ (7)
N T
where ST(x,y)5ST21(x,y)2fT2N(x,y)zfT(x,y)
ST2 ðx,yÞ~ST{1
2 2
ðx,yÞ{fT{N ðx,yÞzfT2 ðx,yÞ
Note that the grey-level mean mT(x,y) and standard
deviation sT(x,y) at the current time frame T are
2 level mean and variance at the current time frame T.
derived from ST21(x,y) and ST{1 ðx,yÞ, which are
passed along from the previous updating stage. They
are calculated precisely, rather than being estimated
from an updating filter. ST(x,y) and ST2 ðx,yÞ can be
efficiently updated by dropping the earliest image
frame fT2N(x,y) in the image series and adding the
current image frame fT(x,y) to the image series.
Therefore, the updating computation involves only
two simple arithmetic operations. A very high
processing rate of image frames is thus achieved.
Note also that the mean and variance updating
processes in equations (6) and (7) are invariant
with respect to the number of image frames N in
the series.
The updated mean mT(x,y) can be interpreted as the
background at the current time. In this paper, we do
not use image differencing that computes the
difference between the reference background and
current image frame to identify foreground objects.
Rather, the standard deviation sT of pixel intensities
over time is effectively used to set up the control
limits for discriminating foreground pixels from the
stationary background. The grey-level mean and
variance updating formulae in equations (6) and (7)
involve one parameter value, the number of image
frames N, to be determined. A small number of N
produces a fast update of the background statistics.
The mean mT(x,y) obtained from a small N can then
be interpreted as a short-term background. It can
quickly absorb non-stationary background objects
such as a moving chair and opening/closing of a
curtain as parts of the background. It is therefore
very responsive to dynamic changes in the environ-
ment. In contrast, a large number of N produces a
slow update of the background statistics. The mean
IMAG 91 # RPS 2011
5
mT(x,y) obtained from a large N can be considered as
a long-term background. It can effectively extract the
accurate silhouette of a moving object. It can also be
used to detect a motionless foreground object. As
previously noted, the updating process is independent
long-term background statistics can be simultaneously
evaluated with minimum computational effort. They
can be incorporated to extract the foreground shapes
in a dynamic environment.
Let the number of image frames N be set at Ns for
short-term updating and Nl for long-term updating,
where Nl.Ns. Furthermore, denote by msT ðx,yÞ and
 s 2
sT (x,y) the short-term grey-level mean and var-
 2
iance, and mlT ðx,yÞ and slT (x,y) the long-term grey-
In this study, the number of short-term Ns is usually
set in the range of 10–100, and the number of long-
term Nl is set between 100 and 1000, depending on
the monitoring environments such as indoor scenes
or highway traffic scenes.
The foreground detection results represented as
binary images BsT ðx,yÞ and BlT ðx,yÞ for the short- and
long-term image frames are, respectively, given by

s 0, if fT (x,y) [ msT{1 (x,y)+KssT{1 (x,y)
BT (x,y)~
1, otherwise
(8)
and
(
0, if fT ðx,yÞ [ mlT{1 ðx,yÞ+KslT{1 ðx,yÞ
BlT ðx,yÞ~
1, otherwise
(9)
Figure 1a shows the sequence of an indoor scenario,
in which a man walks into a classroom, puts a bag on
the desk, sits on a chair, falls asleep for a while, then
wakes up, and, finally, leaves the room without
taking the bag with him. The background scene of the
classroom involves the rotation of ceiling fans.
Figure 1b shows the short-term mean mst ðx,yÞ with
Ns5100. It indicates that the short-term process can
quickly respond to dynamic change in the environ-
ment, where the revolving ceiling fans, the bag and
the sleeping man are all taken as the background.
Figure 1c shows the long-term mean mlt ðx,yÞ with
Nl510000. It reveals that the long-term process can
well preserve the static background, where the bag
and the sleeping man are not absorbed as parts of the
background, but the constantly revolving ceiling fans
are still taken as the background.
The Imaging Science Journal Vol 0
6 W-Y CHIU AND D-M TSAI

1 Estimated mean on an indoor scenario: (a) discrete scene images in a video sequence taken at
15 fps; (b) mean grey valuesmst calculated from a short-term number of 100 frames; (c) mean
grey valuesmlt calculated from a long-term number of 10 000 frames (the symbol t
represents the frame number in the sequence)

The Imaging Science Journal Vol 0 IMAG 91 # RPS 2011

 MOVING/MOTIONLESS FOREGROUND OBJECT DETECTION 7

2 Estimated mean on an outdoor parking-lot scenario: (a) discrete scene images in a video
sequence taken at 15 fps; (b) means calculated from the short-term process (Ns510); (c) means
calculated from the long-term process (Nt5100) (the symbol t represents the frame number in
the sequence)

Figure 2a illustrates the sequence of an outdoor
parking lot scenario. In contrast to the slow-walking
person with a short-term number Ns5100 in Fig. 1,
the foreground object in Fig. 2 is a fast-moving
motorcycle and, therefore, a relatively small short-
term number Ns510 is used for rapid SPC statistics
updating. Figure 2b presents the short-term mean
mst ðx,yÞ with Ns510. It shows that the short-term
process can respond promptly to dynamic change in
the environment, where the stationary motorcycle is
taken as a part of the background soon after it is
parked in the lot. Figure 2c shows the long-term
mean mlt ðx,yÞ with Nl5100. It indicates that the long-
term process can well preserve the static background
without showing the stationary motorcycle as a
background region. The walking person did not
present in either the short-term or the long-term
background.
Note that the indoor scenario in Fig. 1 involves the
slow movement of a person and, therefore, relatively
larger Ns and Nl are used. Conversely, the outdoor
scenario in Fig. 2 involves the fast movement of a
vehicle and, thus, small Ns and Nl are used for the
process. The selection of proper numbers of image
frames for Ns and Nl depends on the speed of moving
foreground objects in the observed scene. The short-
term grey-level mean from a small number of the
most recent frames can avoid false detection from
non-stationary background objects such as moving
chairs, and the long-term mean can detect the
stationary foreground objects such as an unconscious
person. As previously noted, the short-term process
can be highly responsive to dynamic changes in the
environment, and the long-term process can effec-
tively extract the shape of a moving/motionless
foreground object.
2.2 Motionless foreground object detection
In order to detect the motionless foreground objects,
the detection results from both the short- and long-
term processes are incorporated as follows. Let
Bst ðx,yÞ and Blt ðx,yÞ represent, respectively, the short-
and long-term binary results at time frame t. De-
note by bst ðiÞ, i~1, 2, 3, . . . , ns the moving objects
detected in Bst ðx,yÞ, where ns is the total number of

IMAG 91 # RPS 2011 The Imaging Science Journal Vol 0

8 W-Y CHIU AND D-M TSAI

3 Combined scheme of long- and short-term processes: (a) discrete scene images in a video
sequence taken at 15 fps; (b) short-term process results Bst ðx,yÞ (Ns5100); (c) long-term pro-
cess resultsBlt ðx,yÞ (Nt510 000); (d) final detection results of the combined scheme
(the symbol t represents the frame number in the sequence)

moving objects, and blt ð j Þ, j51, 2, 3, …, nl, the
foreground objects detected in Blt ðx,yÞ, where nt is the
total number of foreground objects detected.
When the short-term BsT ðx,yÞ~0 for all (x,y), i.e.
no moving objects are detected in image fT(x,y), we
then check for long-term BlT ðx, yÞ at the current
frame T and short-term BsT{1 ðx,yÞ at the previous
frame T21. If there is at least one bsT{1 ðiÞ, i.e. a
moving object in image BsT{1 ðx,yÞ, then we find the
corresponding object blT ð j Þ that has the shortest
distance to bsT{1 ðiÞ in the long-term BlT ðx,yÞ. The
Euclidean distance between the centroid of blob
bsT{1 ðiÞ and that of blob blT ð j Þ is denoted by
 
d bsT{1 ðiÞ,blT ð j Þ . Let ROI be the effective region

The Imaging Science Journal Vol 0 IMAG 91 # RPS 2011

 MOVING/MOTIONLESS FOREGROUND OBJECT DETECTION
slightly smaller than the size of the image frame. If a
moving object falls outside the ROI, this indicates
that the foreground object walks away from the field
of view of the still camera. The motionless fore-
ground objects can then be extracted using the
following detection process
If
BsT ðx,yÞ~0,
Vðx,yÞ (i:e: no moving objects)
and there exists a bsT{1 ðiÞ[ROI,
then
 
Compute d bsT{1 (i),blT (j)
~f½xsT{1 (i){xlT (j)2 z½ysT{1 (i){ylT (j)2 g1=2
for j~1, 2, . . . , n1 (10)
 s    compare the performance of the proposed method
where xT{1 (i),ysT{1 (i) and xlT (j),ylT (j) are the
coordinates of the centre of area for blobs bsT{1 ðiÞ
and blT ð j Þ, respectively.
 
Let d bsT{1 (i),blT (j  ) ~
   
min d bsT{1 (i),blT (j) , j~1, 2, :::, nl
Superimpose blT (j  ) on the binary image BsT ðx,yÞ,
i.e.
BsT ðx,yÞ/BsT ðx,yÞ|blT ðj  Þ (11)
Endif
In equation (10), blob bsT{1 ðiÞ represents a detected
moving object in BsT{1 ðx,yÞ at frame T21, but
disappears in BsT ðx,yÞ at frame T. The blob blT (j  )
that has the shortest distance to bsT{1 (i) then
represents the object bsT{1 (i) that becomes motionless
at frame T in the long-term background. Equa-
tion (11) indicates that all final detection results of
the motionless foreground objects are superimposed
on the binary image BsT ðx,yÞ at the current frame T.
Figure 3a shows a sequence of indoor scenario
with a person opening the door and then fainting on
the floor. Figure 3b shows the short-term process
results, whereby the motionless person cannot be
detected in BsT ðx,yÞ after fainting (at frame T5201).
Figure 3c shows the long-term process results, in
which the opened door cannot be promptly absorbed
as a part of the background, but the fainted person is
well extracted in BlT ðx,yÞ. Figure 3d displays the
results of the combined scheme. It reveals that the
proposed method can effectively detect motionless
foreground objects and is highly responsive to
IMAG 91 # RPS 2011
9
dynamic environments. It requires no complicated
tracking algorithms to detect the stationary fore-
ground objects.
3 EXPERIMENTAL RESULTS
This section presents the experimental results from
three sets of image sequences that involve one
outdoor and two indoor scenarios. The proposed
algorithms were implemented using the Czz lan-
guage and run on a Pentium 4, 3.0 GHz personal
computer. The test images in the experiments were
1506200 pixels wide with 8-bit grey levels. The
computation time per frame of size 1506200 was
0.0071 s (i.e. 142 fps). In the experiments, we also
with that of temporal difference and adaptive back-
ground mixture models proposed by Stauffer and
Grimson.16 The source code of the Gaussian mixture
modelling algorithm for background subtraction is
available through OpenCV.31 In the adaptive back-
ground mixture models, a mixture of three Gaussians
was used, and the learning rate a was given by 0.1 for
all test scenarios. In the proposed method, the short-
term number and the long-term number of image
frames were set with Ns5100 and Nl510 000 for all
test scenarios. In order to show the primary effec-
tiveness of the proposed method, post-processing
operations such as noise removal and connected
component analysis were not applied to the resulting
images.
In order to evaluate the performance of the
proposed method with the currently available meth-
ods that combine short and long frame information,
the dual-background model proposed by Porikli and
Yin28 is used as the benchmark method and the
indoor scenario in Fig. 4 is used for the experiment.
The indoor scenario entails a person walking into the
laboratory with the curtain closed. He then opens the
curtain, faints on the floor, stands up after a while
and finally leaves the room with the curtain still open.
The grey-level images shown in Fig. 4a display the
original video sequence at varying time frames. The
video images were captured at 15 fps.
The results of the proposed method are presented
in Fig. 4b. The detection results show that the
proposed method can well extract the moving person
and is not affected by the changed state of the
curtain. Furthermore, the proposed method also
correctly detected the motionless man after he
The Imaging Science Journal Vol 0
10 W-Y CHIU AND D-M TSAI

4 Experimental results on an indoor laboratory scenario: (a) discrete scene images in a video
sequence taken at 15 fps; (b) detected foreground objects from the proposed method; (c) detec-
tion results from the long-term and short-term combination method;26 (d) detection results
from temporal difference; (e) detection results from the adaptive background mixture model
(the symbol t represents the frame number in the sequence)

fainted. The opened curtain was promptly adapted as
a part of the background. Figure 4c shows the results
from the dual-background model method.28 When
the person fainted at t5550, the method was able to
detect the motionless man. At the same time, it also
detected the opened curtain as a foreground object.
Even after the person left at t51095, the curtain was
still incorrectly detected as a foreground object. The
test images in the experiments were 1506200 pixels
wide with 8-bit grey levels. The computation time per

The Imaging Science Journal Vol 0 IMAG 91 # RPS 2011

 MOVING/MOTIONLESS FOREGROUND OBJECT DETECTION 11

5 Experimental results of an indoor living room with a rotating fan: (a) discrete scene images in
a video sequence taken at 15 fps; (b) detected foreground objects from the proposed method;
(c) detection results from temporal difference; (d) detection results from the adaptive back-
ground mixture model (the symbol t indicates the frame number in the sequence)

frame of our proposed method was 0.0071 s (i.e.
142 fps), and the computation time of the dual-
background model28 was 0.021 s (i.e. 46 fps). The
proposed method is more effective and computation-
ally more efficient.
The indoor scenario in Fig. 4 was also tested with
the temporal difference method and mixture of
Gaussians model. Figure 4d shows the results from
temporal difference. It indicates that temporal
difference cannot detect a stationary foreground
object such as the fainted man. The resulting shape
of the moving object from temporal difference was
not accurate, and only partial edges were detected.
Figure 4e displays the results from the adaptive
background mixture model, in which the walking
person and the opened curtain were both detected as

IMAG 91 # RPS 2011 The Imaging Science Journal Vol 0

12 W-Y CHIU AND D-M TSAI

6 Experimental results on an outdoor parking lot scenario: (a) discrete scene images in a video
sequence taken at 15 fps; (b) detected foreground objects from the proposed method; (c) detec-
tion results from temporal difference; (d) detection results from the adaptive background mix-
ture model (the symbol t indicates the frame number in the sequence)

foreground objects with the learning rate a50.1. If a
larger learning rate is used, the opened curtain
can indeed be updated promptly as a part of the
background. However, the slow-moving person will
not be fully identified as a foreground object. As seen
in the last row of Fig. 4, the proposed method was
able to quickly update the floor area occupied by the
fainted person as a part of the background soon after
he got up from floor. The mixture of Gaussians
model could also detect the walking person when he
got up from the floor, but the floor area he fell onto
still generated severe residuals.
The second indoor example in the experiments
came from a scenario in which a person walks into a
living room equipped with a ceiling fan, and then
leaves the room, while the ceiling fan keeps spinning.

The Imaging Science Journal Vol 0 IMAG 91 # RPS 2011

 MOVING/MOTIONLESS FOREGROUND OBJECT DETECTION 13

7 Background estimation: (a1–a3) real background images for the scenarios in the last row of
Figs. 4a, 5a and 6a, respectively; (b1–b3) estimated background images mst from the proposed
method; (c1–c3) estimated background images from the adaptive background mixture
model

The video sequence was taken at 15 fps. Figure 5a
shows the video sequence at varying time frames.
Figure 5b displays the results from the proposed
method. It shows that the regular rotation of the
ceiling fan was promptly adapted as a part of the
background, and only the walking person was
identified as a foreground object. Figure 5c shows
the results from temporal difference. The detection
result was affected by the rotation of the ceiling fan,
and only a partial shape of the walking person was
extracted. The detection results from the adaptive
background mixture model are demonstrated in the
fourth column Fig. 5d. The mixture model of
adaptive Gaussians was also capable of dealing with
movement of the ceiling fan. However, it was more
sensitive to the shadow of the walking person. The
background model was not promptly updated right
after the person walked out of the room, as seen in
Fig. 5d.
The third outdoor scenario entailed monitoring of
a parking lot, in which a car leaves a parking space in
the parking lot with varying speeds, and then drives
back to the lot and parks in the same space. The
video sequence was also taken at 15 fps. Figure 6a
shows the original video sequence at varying time
frames. There are many trees around the parking lot,
and wind and sunlight produced dynamic changes in
the background. The proposed foreground segmenta-
tion scheme can effectively extract the moving car,
regardless of its speeds, as shown in Fig. 6b. The
detection results from temporal difference are pre-
sented in Fig. 6c. Different driving speeds of the car
caused an incomplete car shape. When the car slowed
down or stopped, the temporal difference method

IMAG 91 # RPS 2011 The Imaging Science Journal Vol 0

14 W-Y CHIU AND D-M TSAI
8 Bar chart of overall performance on the Microsoft benchmark dataset
was unable to detect the car effectively. In the
experiment, the frames from t5569 to t5605 show
that the target car remains still. Thus, the temporal
difference method cannot detect the foreground from
the image sequence. The proposed method can
reliably detect this motionless foreground car.
Figure 6d presents the detection results from the
adaptive background mixture model. It shows that
the mixture of Gaussians model was robust to the
changes in sunlight and waves of leaves. However, the
car shape was not well extracted. The residuals of
the car still presented in the image soon after the car
parked in the lot and became a part of the
background, as seen in Fig. 6d.
In order to further analyse the detection results of
the proposed method and the adaptive background
mixture model, Fig. 7 shows the estimated back-
grounds for the three scenarios. Figure 7a1–a3 are,
respectively, real scene backgrounds of the laboratory
(at image frame t51095 in Fig. 4a), the living room
(at image frame t5363 in Fig. 5a) and the parking
lot (at image frame t51650 in Fig. 6a). Figure 7(b1)–
(b3) are the short-term mean mst given by the
proposed method. They show that the backgrounds
derived from the proposed method are very close to
their real ones. Figure 7(c1)–(c3) present the esti-
mated backgrounds from the mixture of Gaussians
model. The results show that the mixture model can
effectively estimate the scene background of the living
room, as seen in Fig. 7c2. It can well model the non-
stationary blades of the ceiling fan, and adapt the fan
as a part of the background. However, the mixture of
The Imaging Science Journal Vol 0
Gaussians model cannot responsively update the
scene backgrounds of the laboratory and the parking
lot. There were dark residuals presented in the floor
area possessed previously by the fainted person, as
seen in Fig. 7c1. The edges of the white car were not
effectively updated soon after it parked in the lot and
presented some artificial effects, as seen in Fig. 7c3.
In order to verify the effect of parameter K in
equation (5), K value was varied from 1 to 7 and
tested on the Microsoft Wallflower dataset.32 The
dataset contains seven scenarios of image sequences:
Moved objects, Time of day, Light switch, Waving
trees, Camouflage, Bootstrapping and Foreground
aperture. The Wallflower dataset also provides a
hand-segmented ground truth for evaluation. The
accuracy of foreground segmentation results can be
represented by false-positive rate (FP), false-negative
rate (FN) and total error rate (TE). False-positive
refers to the percentage of background pixels that are
misclassified as foreground pixels, and false-negative
indicates the percentage of foreground pixels that are
misclassified as background pixels. The total error
rate is the sum of FP and FN.
In the experiments, we compared the performance
of the proposed method with different control
constants K and two adaptive background models:
single Gaussian11 and mixture of Gaussians.16 In the
background models, a mixture of three Gaussians
was used. We experimented with different settings of
the adjustable parameters of the two compared
algorithms until the results seemed optimal over the
entire datasets. We did not change parameter values
IMAG 91 # RPS 2011
 MOVING/MOTIONLESS FOREGROUND OBJECT DETECTION
between sequences. The hand-segmented ground
truths are also provided by the benchmark for
performance evaluation. Figure 8 summarises the
performance in terms of FP, FN and TE for each
algorithm. When the control constant K is increased
from 1 to 7, the FP is decreased and FN is slightly
increased.
When K is given between 4 and 7, all the resulting
FP values are less than 1% for lower noise in the
background. The TE is stable with K from 2 to 7. In
considering the overall performance of FP and TE,
we chose K55 for robust foreground detection.
Figure 8 also shows that the proposed method can
detect the foreground objects with the least total
errors, compared to the single Gaussian and mixture
of Gaussian methods. In terms of the sensitivity of
the parameter value, the experimental results indicate
that the proposed method has stable and effective
performance in a wide range of K values between 3
and 7.
The proposed foreground segmentation method is
based on the statistical mean and standard deviation
of a series of consecutive image frames. The well-
known single Gaussian updating models also rely on
these two statistics for motion detection. However,
the proposed method updates the background by
precise calculation of the mean and standard devia-
tion and requires no reset operation in a non-stop
monitoring process. Instead, the single Gaussian
updating models estimate these two statistics with a
linear filter and the estimation degrades over time due
to accumulated error.
4 CONCLUSION
In this paper, we have presented a statistic process
control scheme for foreground detection in video
sequences from a static camera. The main advantages
of the proposed method are that it can observe both
long- and short-term backgrounds simultaneously
and the corresponding grey-level mean and variance
can be efficiently and exactly computed without the
introduction of a learning rate for background
estimation. The grey-level mean of long-term frames
represents a static background without moving
objects, and retains all the advantages of background
subtraction. The grey-level mean of short-term
frames represents the most current status of the
dynamic environment, and retains all the advantages
of temporal difference.
IMAG 91 # RPS 2011
15
The variance of image frames in either short- or
long-term processes indicates the degree of scene
changes, thus enabling noise and repetitive move-
ments such as rotating fans and swinging leaves to be
easily removed from the background. Combining the
results of the long- and short-term processes can
detect the motionless objects and eliminate the
influence of non-stationary background objects. The
proposed method is robust in accommodating noise
and grey-level variations of individual pixels. It can
effectively extract the shape of a moving object,
and is very responsive to both gradual and radical
changes in the environment. High process frame rates
can be achieved up to 142 fps for smaller images of
size 1506200 and 15 fps for larger images of size
6406480 on an Intel Core2 2.33 GHz personal
computer. Experimental results have revealed that
the proposed method can be applied to monitoring of
both indoor and outdoor scenarios in which dynamic
environments and foreground objects with various
moving velocities or without motion might occur.
REFERENCES
1 Hu, W., Tan, T., Wang, L. and Maybank, S. A survey
on visual surveillance of object motion and behaviors.
IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev.,
2004, 34, 334–352.
2 Horn, B. K. P. and Schunck, B. G. Determining optical
flow. Artif. Intell., 1981, 17, 185–203.
3 Lucas, B. D. and Kanade, T. An iterative image
registration technique with an application to stereo
vision, Proc. PADRA Imaging Understanding Work-
shop, Washington, DC, USA, April 1981, IEEE, pp.
121–130.
4 Gibson, D. and Spann, M. Robust optical flow
estimation based on a sparse motion trajectory set.
Image Process., 2003, 12, 431–445.
5 Liu, P. R., Meng, M. Q.-H. and Liu, P. X.: Moving
object segmentation and detection for monocular robot
based on active contour model. Electron. Lett., 2005, 41,
1320–1322
6 Kim Y. H. and Kak, A. C. Error analysis of robust
optical flow estimation by least median of squares
methods for the varying illumination model. Patt. Anal.
Mach. Intell., 2006, 28, 1418–1435.
7 Chamorro-Martinez, J. and Fernandez-Valdivia, J. A
new approach to motion pattern recognition and its
application to optical flow estimation. IEEE Trans.
Syst. Man Cybern. Part C: Appl. Rev., 2007, 37, 39–51.
8 Lipton, A. J., Fujiyoshi, H. and Patil, R. S. Moving
target classification and tracking from real-time video,
The Imaging Science Journal Vol 0
16 W-Y CHIU AND D-M TSAI
Proc. 4th IEEE Workshop on Applications of computer
vision: WACV ’98, Princeton, NJ, USA, October 1998,
IEEE, pp. 8–14.
9 Wang, C. M. and Brandstein, M. S. A hybrid real-time
face tracking system, Proc. IEEE Int. Conf. on
Acoustics, speech, signal processing: ICASSP ’98,
Seattle, WA, USA, May 1998, IEEE, pp. 3737–3740.
10 Piccardi, M. Background subtraction techniques: a
review, Proc. IEEE Int. Conf. on Systems, man,
cybernetics, The Hague, The Netherlands, October
2004, IEEE, pp. 3099–3104.
11 Wren, C. R., Azarbayejani, A., Darrell, T. and
Pentland, A. P. Pfinder: real-time tracking of the
human body. IEEE Trans. Patt. Anal. Mach. Intell.,
1997, 19, 780–785.
12 Olson, T. and Brill, F. Moving object detection and
event recognition algorithm for smart camera, Proc.
DARPA Image Understanding Workshop, New
Orleans, LA, USA, May 1997, Morgan Kaufmann,
pp. 159–175.
13 Eveland, C., Konolige, K. and Bolles, R. C.
Background modeling for segmentation of video-rate
stereo sequences, Proc. IEEE Conf. on Computer vision
and pattern recognition: CVPR ’98, Santa Barbara,
CA, USA, June 1998, IEEE Computer Society, pp.
266–
271.
14 Kanade, T., Collins, R., Lipton, A., Burt, P. and
Wixson, L. Advances in cooperative multi-sensor video
surveillance, Proc. DARPA Image Understanding
Workshop, Monterey, , CA, USA, November1998,
IEEE, Vol. 1, pp. 3–24.
15 Cavallaro, A. and Ebrahimi, T. Video object extraction
based on adaptive background and statistical change
detection. Proc. SPIE, 2001, 4310, 465–475.
16 Stauffer, C. and Grimson, W. E. L. Adaptive back-
ground mixture models for real-time tracking, Proc.
IEEE Conf. on Computer vision and pattern recogni-
tion: CVPR ’99, IEEE Computer Society, Vol. 2, pp.
246–252.
17 Stauffer, C. and Grimson, W. E. L. Learning patterns of
activity using real-time tracking. IEEE Trans. Patt.
Anal. Mach. Intell., 2000, 22, 747–757.
18 KaewTrakulPong, P. and Bowden, R. An improved
adaptive background mixture model for real-time
tracking with shadow detection, Proc. 2nd European
Workshop on Advanced video based surveillan: AVBS
2001, Kingston, UK, September 2001, Kingston
University, pp. 149–158.
19 Lee, D.-S. Effective Gaussian mixture learning for video
background subtraction. IEEE Trans. Patt. Anal. Mach.
Intell., 2005, 27, 827–832.
The Imaging Science Journal Vol 0
20 McFarlane, N. and Schofield, C. Segmentation and
tracking of piglets in images. Mach. Vision Appl., 1995,
8, 187–193.
21 Manzanera, A. and Richefeu, J. C. A new motion
detection algorithm based on S–D background estima-
tion. Patt. Recogn. Lett., 2007, 28, 320–328.
22 Elgammal, A., Duraiswami, R., Harwood, D. and
Davis, L. S. Background and foreground modeling
using nonparametric kernel density estimation for visual
surveillance. Proc. IEEE, 2002, 90, 1151–1163.
23 Ianasi, C., Gui, V., Toma, C. I. and Pescaru, D. A fast
algorithm for background tracking in video surveillance
using nonparametric kernel density estimation. Facta
Univ. Ser.: Electron. Energ., 2005, 18, 127–144.
24 Elgammal, A., Duraiswami, R. and Davis, L. Efficient
kernel density estimation using the Fast Gauss
Transform with applications to color modeling and
tracking. IEEE Trans. Patt. Anal. Mach. Intell., 25,
1499–1504.
25 Greengard, L. and Strain, J. The fast Gauss transform.
SIAM J. Sci. Stat. Comput., 1991, 12, 79–94.
26 Huerta, I., Rowe, D., Gonzalez, J. and Villanueva, J. J.
Improving foreground detection for adaptive back-
ground segmentation, Proc. 1st CVC Workshop on
computer vision: progress of research and development:
CVCRD 2006, Bellaterra, Spain, Universitat Autònoma
de Barcelona. October 2006, pp. 1–6.
27 Bayona, A., SanMiguel, J. C. and Martiı́nez, J. M.
Stationary foreground detection using background
subtraction and temporal difference in video surveil-
lance, Proc. 17th IEEE Int. Conf. on Image processing:
ICIP 2010, Hong Kong, China, September 2010, IEEE
Signal Processing Society pp. 4657–4660.
28 Porikli, F. and Yin, Z. Temporally static region
detection in multi-camera systems, Proc. 10th IEEE
Int. Workshop on Performance evaluation of tracking
and surveillance: PETS 2007, Rio de Janeiro, Brazil,
October 2007, IEEE, pp. 79–86.
29 Montgomery, D. C. Introduction to Statistical Quality
Control, 2004 (John Wiley & Sons, New York).
30 Hsieh, K. L., Tong, L. I. and Wang, M. C. The
application of control chart for defects and defect
clustering in IC manufacturing based on fuzzy theory.
Expert Syst. Appl., 2007, 32, 765–776.
31 OpenCV, Intel Open Source Computer Vision
Library. ,http://www.intel.com/technology/computing/
opencv/.
32 Toyama, K., Krumm, J., Brumitt, B. and Meyers, B.
Wallflower: principles and practice of background
maintenance, Proc. 7th IEEE Int. Conf. on Computer
vision: ICCV ’99: Corfu, Greece, September 1999,
IEEE Computer Society, pp. 255–261.
IMAG 91 # RPS 2011