Incorporation of Panoramic View in Fall Detection
Using Omnidirectional Camera
Tên sinh viên (Author)
Nguyễn Minh Quân – CTTT KTYS- K61
Trần Thị Mến – CTTT KTYS – K61
Nguyễn Xuân Bách – CTTT KTYS – K61
Abstract—Falling is one of the major problems that
threaten the health of the elderly and particularly dangerous
for people that live alone. Recently, surveillance systems
using omnidirectional cameras in general and fisheye
cameras in particular have become an attractive choice, as
they provide a wide Field of Vision without the need for
multiple cameras. However, objects captured by fisheye
cameras are highly distorted, therefore computer vision
approaches that are developed for conventional cameras
require modification to work with such systems. The aim of
this work is to incorporate the de-warping of fisheye image
using polar to cartesian transformation to generate a
panoramic view. Objects are detected by background
subtraction method. Depending on the objects lay inside or
outside of a center circle of the omnidirectional frame,
features based on contour and rotated bounding box of the
objects, are extracted correspondingly on original
omnidirectional view or panoramic view. Experiments show
that by incorporating both panoramic and omnidirectional
view, we can achieve significant improvement in fall
detection, particularly in peripheral areas. This result could
be a useful reference for further studies.
Keywords—falling, omnidirectional, panoramic, detection
I. GIỚI THIỆU (HEADING 1)
The world’s population is getting increasingly older.
The proportion of elderly people (aged 60 and over)
increased from 7.15% in 1989 to 8.93% in 2009 [1]. About
8% of all elderly people live alone and 13% elderly
couples live alone . Living alone poses a great danger as
the elderly are easily affected by emergency situations:
natural disasters, health problems such as stroke, heart
attack, and unusual actions in daily life , especially falls,
strokes…[2]. These situations need to be discovered, dealt
with promptly and quickly and accurately notified to
relatives or careers. Therefore, having a video surveillance
system is extremely necessary. Over the past decades,
there has been various researches on computer- vision
based surveillance system. Most of these researches focus
on regular pin-hole cameras. However, while pin-hole
camera has an advantage of simplicity as well as giving an
accurate representation of the human’s visual system,
monitoring a wide area with conventional pinhole cameras
can be troublesome, as it requires a Pan-Tilt-Zoom (PTZ)
or multiple regular cameras. Using multiple camera is not
optimal, as it costs more to maintain and install, as well as
Tên giáo viên hướng dẫn
TS Nguyễn Việt Dũng – ĐTYS
requires more bandwidth and synchronization among
cameras. PTZ cameras on the other hand, while more
flexible than conventional, can still only capture image
from only one direction at any given time, and mechanical
parts also are prone to failure. In order to extend the FOV
of a surveillance system, omnidirectional cameras can also
be used.
Fig. 1. Types of omnidirectional cameras.
Omnidirectional camera, which consists of fisheye-
lens cameras and catadioptric lens cameras (as shown in
Fig. 1), allows a larger field of vision compared to
traditional pin-hole camera by sacrificing resolution in
peripheral areas and image’s structural integrity, which
makes it difficult for conventional action recognition and
object detection algorithms to work on fisheye camera.
Some techniques have been proposed to mitigate the
drawbacks of fisheye cameras. Chiang and Wang [3]
applied HOG+SVM detector and rotate people in fisheye
images so as they appear upright. This practice can be
computationally expensive, as in full-frame fisheye
images, the image must be rotated many times. Other
works modify the features used for conventional camera
to work on omnidirectional camera [ 4], however, this
might get complex and the results fall behind compared to
modern standards. Some papers propose to use
Convolutional Neural Network (CNN) directly on fisheye
dataset [5]. However, the variety of human poses captured
by the fisheye camera can lead to problems, as there are
not enough fisheye databases currently. Other authors use
fisheye calibration to undistort the fisheye image [6].
However, to get a full straight equiangular quadrilateral
shaped image, cutting out a considerable number of pixels
from the edges is required. Meanwhile this method is
applicable in some situations, it is not optimal to do so in
 surveillance applications. Another projection is fisheye
de- warping, in which a fisheye image is unwrapped into
360- degree equirectangular image [7]. This projection is
also flawed, as the nearer the object is to the center of the
fisheye image, the more deformed it becomes when
unwrapped.
In this work, we propose an approach to reduce the
effect of omnidirectional camera distortion by
incorporating de- warped panorama view for a better fall
detection result. Section II describes our method of
incorporating panoramic view in fall detection using
omnidirectional video. Experimental results of the
proposed method are illustrated in Section III. Finally,
conclusions are given in Section IV.
II. METHOD FOR INCORPORATING PANORAMIC
VIEW IN POSE RECOGNITION ON
OMNIDIRECTIONAL VIDEO
A. Fisheye de-warping using polar to cartesian
transformation
An easy visualization of the de-warping techniques is to
imagine the fisheye image as a circle, put a slice in it, and
then uncurl it to normal equirectangular panorama. For this
process, we will need the position of /the center as well as
the radius of the fisheye camera. Then, we find the
corresponding fisheye output point corresponding to an
input cartesian point, with each point in the polar coordinate
represented by four parameters: ( ), and the
corresponding cartesian coordinate pixel position (
),and the height and width of the panorama image is Hp and
Wp, radius of fisheye image as R, center of the fisheye
image is ( )
Then we have:
 the fisheye image.
 objects in the central area of fisheye image. We propose a

 outside the cropped area, we process it in panoramic view.
Then, for each pixel in the equirectangular image, we find
the corresponding pixel in the fisheye image. We then map
each pixel in the omnidirectional image to their
corresponding location in equirectangular image. The
mapping diagram can be seen in Fig.2 below:
Fig.2 Pixel mapping diagram.
B. Background subtraction algorithm
We apply the background subtraction algorithm proposed in
our previous research [8]. First, the first frame is taken as a
comparison background. The background after being
converted to grayscale has a lot of background noise,
therefore we apply the Gaussian Blur function to smooth
and reduce image noise. The background subtraction
algorithm KNN is applied to each frame, people or moving
objects is separated. Then, methods such as shadow removal
and noise elimination are applied to further separate the
main moving object from the background. For the shadow
removal, we use the threshold function – a segmentation
method, in which if pixel value is greater than a threshold
value, it is assigned one value (white), else it is assigned
another value (black). Due to the light change, the
comparison between two frames occurred too quickly,
resulting in noise. To eliminate noise, we use the opening
morphology algorithm.
C. Region detection and choosing processing method.
However, fisheye de-warping algorithm is not without
flaws. As we unwrap the image starting from the center of
the fisheye camera, the closer the object is to the center of
the fisheye image, the more distorted it becomes, as there
are not enough pixels to unwrap to the equirectangular
image. Therefore, this region near the center is not suitable
for processing. To solve this problem, we simply do not
map the distorted areas of the equirectangular image from
However, the panoramic view is now unable to detect
region detection and choosing processing method algorithm.
The algorithm is as follow: If the object is completely
If a part of the object is inside the cropped area, we process
it in omnidirectional view. We can do this simply by
creating a circle with radius r in the omnidirectional view
corresponding to the cropped-out area in the panoramic
view. Then for every frame, we check if the any part of the
object is inside the circle or not. If the object is inside the
circle, process the object in omnidirectional view. Else,
process in equirectangular view. This method is feasible,
due to the fact that the central area of the fisheye image is
much less distorted than the peripheral areas, therefore we
can use this region for processing without having to worry
too much about distortion, as object in this region remain
mostly undistorted.
Our algorithm can be described according to the flow chart
below:
Fig.3 Algorithm flowchart
D. Posture recognition
1. Contour:
Contour is a useful tool used to analyze object shape, detect
and identify objects. Contour is a set of points that make up
a curve around pixels that have the same or near
approximation of a color value.
2. Models of machine learning:
Machine learning is an application of Computer Science and
Artificial Intelligence (AI) that provides systems the ability
to automatically learn and improve from experience without
being explicitly programmed.
2.1 Support vector machine (SVC):
In this algorithm, data points will be expressed in n-
dimensional space, where n is the number of features from
the input data set, the value of each data point will be
expressed as a specific value coordinates according to the
feature axes. After that the Support vector machine
algorithm re-classifies by finding a hyper-plan to separate
data groups, or classes. So SVC can be interpreted as the
best boundary (optimal hyperlane) between classes[11].
2.2 K-Nearest Neighbor (KNN):
The k-Near Neighbor algorithm assumes that similar things
exist close together, in other words, similar things will be
near each other [13]. So the KNN algorithm is similar to
considering the position of the surrounding points group
(neighbors) compared to the classified point to consider
which data class it belong to.
2.3 Naïve Bayes
Naïve Bayes is an algorithm in the Supervised Learning
group. Naïve Bayes classification, a classification problem
based on the probability which applying Bayes theorem
[16].
According to Bayes theorem, we have the formula for
calculating the random probability of event A as follows:
Suppose that event A is divided into n independent parts ,
, …, .
We have:
So:
is proportional[16].
2.4 Decision Tree
Decision Tree is a diagram which has tree shape structure,
includes:
• Nodes: Test the value of a certain attribute
• Edge / Branch: corresponds to the result of a test and
connects to the next node or leaf.
• Leaf nodes: Terminal nodes predict the outcome
(representing class labels or class distributions) [17].
Fig 4. Diagram of Decision Tree
3. Cross validation
Cross validation is the most famous method to help evaluate
machine learning algorithm model. This method helps us
divide the original data into separate training and testing
subsets [20].
We will go further into the K-fold cross validation
method. For k-fold cross validation, the data set will be
divided into k equal parts (k subsets have approximately the
same total data) [21]. Then one subset will be chosen as a
test set (k-1) and other subsets will be combined as a
training set. The selection is repeated k times so that each of
the k subsets is selected as a test set once.

Fig 5. K-fold
4. GridsearchCV
GridsearchCV is the process of making parameter
adjustments to determine the optimal parameter values for a
given model on a data set. This is important because the
performance of the entire model is based on the specified
parameter values.
III. Posture recornization method
As mentioned above, we use four ML models to classify
and identify three postures: walking, sitting, fainting.
 Block diagram:
 Select and extract features:
After applying BS algorithm, six feature is extracted, three
postures are classified based on the following 6 features:
- Area of contour (A1)
- Angle of line (fitting to contour)
- Ratio of width and height ( of rotate
rectangle)
- Area of bounding box (rotate rectangle)
(A2)
- Ratio of area (A1/A2)
- Angle of bounding box
a) b) c)
Fig 6 a) Contour b) Line fitting to contour c) Bounding box
 Model optimization:
- GridSearchCV:
The gridsearchCV2 function is used to pair the values of
each model's input parameters to find out the value of the
model's parameters for the highest accuracy on the data set.
Fig 13. result of optimizing model by gridsearchCV
- Cross validation:
Cross validation is used to check the change of accuracy
over the range of each input parameter. Our aim is to refer
to the best value of each model's input parameters on the
dataset.
gives an accuracy graph as shown:
a) b) c)
Fig 7. result of optimizing model by Cross validation
 K-fold:
K-fold is used to slip dataset into training set and testing set.
III. IMPLEMENTATION AND RESULTS
All step are written mostly using OpenCV Python, using a
computer with core I5- 6300HQ 2.3Ghz processor, and the
video size is 640x480 pixels.
Using the BOMNI [9] database scenario 1. This database
contains two cases that are recorded by 2 full-direction
cameras. A camera mounted on the ceiling, a camera
mounted on the wall.
Scenario #1 when one person performs the following
actions: * walks *, * sits *, * drinks water *, * washes his
hands * opens-closes *, * falls *.
1. Fisheye de-warping and background subtraction
For fisheye de-warping, we are able to achieve an
acceptable frame rate of 10 FPS and good results, as shown
in Fig.5
Fig.8 De-warped fisheye image with distorted region excluded
The optimal radius of the circle corresponding to the
excluded distort region (the black area) is calculated to be
0.145 * R, with R as the radius of the fisheye image, and in
this case, r is approximately 70 pixels.
  Accuracy of Machine learning models:

Table 2. Final results

After applying divide folds based on labels, the accuracy
difference between folds decreases significantly. The
accuracy of models applying on topcut and topano video is
much higher than top video and side video (original video)
and get the value from 97% to 100%.
Although our average result is quite high, there are still
remains some drawbacks as:
- The BS algorithm limitations such as
affection of light (stand near by too bright
places), and the algoritm of shadow
remove hasn’t optimized yet result in in
a) b)
accurate feature data.
Fig.9 Region detection and choosing processing method
- When our object opens or closes door,
a) Object outside r-radius circle, b) Object inside r-radius circle,
bounding box will include both the object
process in panoramic view , process in omnidirectional
view
and the space area when our object
After applying background subtraction methods, our opening and closing the door which lead
detection algorithm is also able to accurately detect the to mistaken by our model between
position of the person and choose the region of process walking and fainted.
automatically. In Fig. 6a , the person is out side of the circle - In the problem of sitting on the chair back
with radius r, therefore is processed in panoramic view. In to the camera, models can just recognize
Fig. 6b , the person is inside the circle with radius r, and our the upper part of the object, the lower part
algoritm automatically process the object in omnidirectional was hidden by the chair so our model
view. can’t identify this case
2. Posture recognition
 Dataset includes: V. CONCLUSION
- Sided view- camera (3549 frames): 2347
We presented a novel approach to top-view posture
frames with 3 labels
recognition for omnidirectional camera using incorporation
- Top view-camera(4058 frames) : 2974 of panoramic and omnidirectional view. Compared to
frames with 3 labels previous researches, our method could reliably deal with the
Topcut: 602frames with 3 labels drawbacks of fisheye distortion when applying conventional
Toppano: 2161 frames with 3 labels detection methods as well as solve the problem of loss of
vital information compared to conventional fisheye
 Model optimization: calibration methods. Using Decision Tree, we can achieve
satisfactory detection results on both panoramic as well as
omnidirectional view. However, our approach is still
sensitive to multiple object detection as well as occlusion or
sudden change in illumination. The authors are currently
working on these problems. The result of this work can be
used for further research on omnidirectional camera, and
with further optimization, it can be reliably used for real-
time omnidirectional surveillance system.

REFERENCES
Table 1. Optimization of input parameters
[1] “Viet Nam prepares to support aging population” ,Vietnam News, 09/2017,
https://vietnamnews.vn/society/health/393500/viet-nam-prepares-to-support-
aging-population.html#o2tLry468d42sApy.97
[2] A.-T. Chiang and Y. Wang. Human detection in fish-eye images
using HOG-based detectors over rotated windows. In ICMEW, pages
1–6, 2014.
[3] Z.Arican and P. Frossard. Omnisift: Scale invariant
featuresinomnidirectionalimages. InIEEEICIP,pp. 3505–3508, 2010.
[4] I. Cinaroglu and Y. Bastanlar. A direct approach
forhumandetectionwithcatadioptricomnidirectional cameras. In 22nd
Signal Processing and Communications Applications Conference
(SIU), 2014.
[5] Y. Dupuis et al. A direct approach for face detection on
omnidirectional images. In IEEE ROSE, pp. 243– 248, 2011
[6] Davide Scaramuzza, Agostino Martinelli, Roland Siegwart, A
Flexible Technique for Accurate Omnidirectional Camera Calibration
and Structure from Motion
[7] Hyungtae Kim, Jaehoon Jung, Joonki Paik, “Fisheye lens camera
based surveilance system for wide field of view monitoring”
[8] Viet Dung Nguyen, Men Tran Thi, Quan Minh Nguyen, Bach
Nguyen Xuan, “EVALUATION OF BACKGROUND
SUBTRACTION METHODS IN OMNIDIRECTIONAL VIDEO”,
KISC, Hanoi University of Science and Technology, Vietnam, 2019
[9] Demiröz B. E., Arı İ., Eroğlu O., Salah A. A., Akarun L., "Feature-
Based Tracking On A Multi-Omnidirectional Camera Dataset",
International Symposium On Communications, Control, And Signal
Processing (ISCCSP12), Rome, Italy, 2012.
[10] Avinash Navlani, “Support Vector Machines with Scikit-learn”, Dec
28th, 2019
[11] Rohith Gandhi, “Support Vector Machine- Introduction to Machine
Learning Algorithms”, Jun 7,2018.
[12] P.S. Bradley and O.L. Mangasarian, “Feature selection via concave
minimization and support vector machines,” in Machine Learning
Proceedings of the Fifteenth International Conference (ICML '98). J.
Shavlik (Ed.), Morgan Kaufmann: San Francisco, California, 1998,
pp. 82-90. ftp://ftp.cs.wisc.edu/math-prog/techreports/98-03.ps.
[13] Wang Ling, Fu Dong-Mei, “Estimation of missing values using a
weighted k-Nearest Neighbors Algorithm”, 4-5 July 2009
[14] Behrooz Kamgar-Parsi and Laveen N Kanal, “An improved branch
and bound algorithm for computing k-nearneighbors”, May 19,2003
[15] Min-Ling Zhang, Jose M.Penam, Victor Robles, “Feature section for
multi-label Naïve Bayes Classification”, June 5, 2019
[16] Yuguang Huang, Lei Li, “Naïve Bayes classification algorithm based
on small sample set”, Sept 2011
[17] Afroz Chakure, “Decision Tree Classification”, Jul 6,2019.
[18] Avinash Navlani, “Decision Tree Classification in Python”,
December 29th, 2018.
[19] Anuja Priyam, Rahul Gupta, Anju Rathee, Saurabh Srivastava,
“Comparative Analysis of Decision Tree Classification Algorithms”,
2003
[20] Raheek Shaikh, “Cross Validation Explained: Evaluating estimator
performance.”, Nov 26, 2018.
[21] Krishni, “K-Fold Cross Validation”, Dec 17,2018.