2022 International Conference on Futuristic Technologies (INCOFT)
Karnataka, India. Nov 25-27, 2022
Human Activity Detection Using Pose Net
Sudharshan Duth P
Department of Computer Science Amrita Vishwa Vidyapeetham
Amrita School of Arts and Sciences
Mysuru, India
p_sudharshanduth@my.amrita.edu
2022 International Conference on Futuristic Technologies (INCOFT) | 978-1-6654-5046-1/22/$31.00 ©2022 IEEE | DOI: 10.1109/INCOFT55651.2022.10094346
Abstract—Human detection and tracking is one of
the important study fields that has attracted a lot of
attention recently.Despite the fact that commercially
available technologies for human identification and
counting are now available, further experimentation is
required in order to address the challenges.
Abnormality detection, commonly called automated
video surveillance, is the way of noticing and describing
human behaviour and interactions in a crowded setting.
People must be discovered and tracked in order to
maintain reliability, welfare, and site management.
Object identification is a critical stage in abnormality
detection and automated video surveillance.
Background subtraction is a technique for detecting
human actions in video segments. The frequent
technique for distinguishing moving objects from
motionless images Background subtraction is the
technique of detaching foreground objects from the
background in a sequence of video frames, as the label
implicit. In this scenario, the primary purpose of
abnormality detection is to recognise and track a
moving object using video recorded images and pose
net.
Keywords—Background subtraction, Pose net, CNN
I. .INTRODUCTION
Computer vision is a multidisciplinary field that focuses
on how computers might be designed to recognise high-
quality digital images or movies. It expects to computerise
things that the visual framework of humans is capable of
performing a designing outlook. Techniques for recording,
processing, analysing, and understanding digital images, also
computer vision problems include extracting high-
geometric data from the real world in order to provide
statistical information like judgments. In this case,
understanding refers to the conversion of visual
representations into world descriptions which can have
interact with other cognitive processes and lead to exact
action. Human activity detection and position estimates have
aroused attention in a variety of applications, including
video-based recognition and human–computer interactions.
However, there is still a lot of study being carried out with
precision and speed. In most cases, activity recognition and
pose estimation are done individually.
Even if the posture is strongly linked with activity
detection, no solution for simultaneously addressing both
difficulties is being researched for the advantage of activity
recognition. One of ML’s primary advantages is its capacity
to execute end-to-end optimization. There are various
advantages to a machine learning problem that can perform
end-to-end optimization.
978-1-6654-5046-1/22/$31.00 ©2022 IEEE
Poojashree B S
Department of Computer Science
Amrita Vishwa Vidyapeetham
Amrita School of Arts and Sciences
Mysuru, India
Poojashreeni111@gmail.com
Convolutional Neural Networks (CNN) have grown in
popularity in recent years as a solution to picture
categorization problems. Finally, the success in identifying
images and description, analysis have begin to use CNNs
for visual categorization more widely. Due to lighting
circumstances, blockage, framework clutter, contortion,
measurements, and inter-group variance, classifying real-life
movies into inconsistent freestyle activities is a difficult
chore.
Human Action Recognition (HAR) is a programme that
analyses human behaviour and assigns a name to each
action. As a result, it has a huge number of applications and
is getting more popular in the world of computer vision.
skeleton, indeep, flaming, specific cloud, voice, acceleration,
radar, and WiFi signals are some of the data procedure that
can be used to depict human actions. Depending on the
application environment, these methods encapsulate several
sources of important but separate data and offer several
benefits. Appropriately, numerous past research has
attempted to investigate individual HAR techniques
employing various modalities. Human action recognition
(HAR) is important for a range of real-world examples.This
can be used to detect potentially dangerous human actions
and assure their safe operation in both visual surveillance
and autonomous navigation systems. It is additionally
helpful for video recovery, human-robot association, and
diversion. HAR motive is to analyse suddenly and recognise
the characteristics of an action in unknown video sequences.
Academics and industry are both interested in HAR because
of the rising demand for automated human activity
interpretation. In real, analysing and understanding Person
behaviour is important for a variety of approach, including
video indexing, biometrics, surveillance, and security.
II. LITERATURE REVIEW
Fatemeh et al. [1] present a graded approach that
includes HOG with background subtraction, as well as the
use of a deep neural network and bone modelling methods.
For feature selection and storing past information, a CNN
and an LSTM recursive network are combined, and
ultimately, a Softmax-KNN classifier is utilized to identify
human actions.
Fernando et al [2] A powerful deep neural networks for
the HAR. The dimensions from multiple body-worn devices
are separated by this network. Three datasets were used:
Opportunity, Pamap2, and an industrial data set. The
architecture was evaluated and found to outperform the
state-of-the-art.
H Mei et al. [3]presents a way of detecting a large figure
of things with an unknown amount that fluctuates over time.
The multiple object tracking approach use a graph structure
to keep track of a number of conjecture regarding the
1
quantity and trajectories of the objects in the video, which
are based on preliminary object identification findings in
each image, which may or may not be right.
Rashmi R.koli et al.[4]The study's purpose is to create a
device that recognises gestures and then converts them into
speech. CNN training receives specific emphasis. The
concept entails creating a device that treats input as a gesture
and then outputs identifiable text using in-depth mastering
standards.
M.Leo et al. [5]The recognition procedure is divided into
two-step processes: The human body position is first
determined picture by picture, and then the detected
postures' temporal sequences are statistically modelled.
Human body postures are calculated using binary shapes
associated with people by selecting horizontal and vertical
histograms as features and feeding them into an
unsupervised clustering technique.
Shi et al. [6] proposed a strategy for spectrogram-based
human action grouping utilising profound learning methods,
including a profound convolutional generative ill-disposed
network for growing and enhancing the preparation set and
an exchange learned profound convolutional network for
highlight abstraction and characterization, which depends on
a DCNN preprepared by a huge scope RGB picture
informational index, that is, ImageNet.
Noor et al.[7] offer a new method for recognising and
summarising numerous human actions in surveillance films.
By separating each person's sequence from the photo, the
suggested method suggests a new representation of the data.
Then, using 3D convolutional neural networks, each
sequence is analyzed to detect and recognize the relevant
behaviours (3DCNNs). An action-based video summary
involves recording each person's actions at various points
throughout the film.
Tingtian Li et al. [8]For complicated events, the authors
of paper present a novel human activity recognition
technique based on group skeletons. Our approach combines
Convolutional networks with multi-scale spatial-temporal
graphs to extract skeleton information from a huge number
of people (MS-G3Ds). In addition to the usual key point
coordinates, the author also gave key point speed values for
the equation.
Amel Ben Mahjoub et al. [9]The paper describes a
method for recognising human actions. The STIP is used by
the authors to discover significant changes in the image. The
author then uses the histogram of Oriented Gradient and
histogram of Optical Flow descriptors to extract the
appearance and motion properties of these interesting
locations. Finally, to apply a label to each video sequence,
the author uses a Bag Of Words (BOW) to match the STIP
descriptor's Support Vector Machine (SVM).
V.Parameswari et al. [10]As input, it uses footage of
COVID-19 patients and searches the hold on photographs
for a match. This method is based on the extraction of Gabor
options using the Gabor filter. To extract features from the
input pictures, the Gabor filter is utilised, and then a
personal sample generation formula is employed.
Kavya et al.[11] Active research subjects include
ubiquitous and mobile computing, surveillance-based
security, context-aware computing, and ambient assisted
living. The purpose of activity recognition is to recognise
2
one or more agents' actions and goals from a set of
observations. The methodology utilised in this article was
based on FSM. This technique is less sophisticated since it
recognises complex processes using a succession of simple
push, pop, enqueue, and dequeue actions. When tested on a
synthetic dataset, the suggested strategy outperforms the
competition, with an accuracy of over 80.9 percent. For the
prescribed actions, the system provides correct and accurate
results. The purpose of activity recognition is to recognise
one or more agents' actions and goals from a set of
observations. The methodology utilised in this article was
based on FSM. This strategy is less sophisticated because To
recognise sophisticated operations, you'll need to do a
succession of simple push, pop, enqueue, and dequeue
actions. When tested on a synthetic dataset, the suggested
strategy outperforms For the prescribed actions, the system
provides correct and accurate results.
Ashokan et al. [12]Due to its numerous applications in
healthcare, security systems, and other fields, human activity
detection has become one of the most popular explored
disciplines in recent decades. Customers can use ATMs to
access financial services quickly and conveniently. It is
critical to create a security system that identifies any unusual
activity in order to ensure the safety of customers. It will
immediately notify the police of any unusual events,
allowing the police to quickly apprehend the culprit. The
work focuses on comparing the performances of a number of
classifiers, including random forest, SVM, and KNN, on the
same dataset. Three distinct classifiers are used to classify
the data.On activity detection, all three approaches produce
satisfactory results. However, based on experimental
findings, it can be concluded that the random forest classifier
outperforms SVM and KNN classifiers in detecting
anomalous events in ATMs. Random forest is quick and
reliable, making it ideal for real-time applications. It's also
good for working with massive datasets.
Amrutha et al. [13]Intelligent video surveillance and
shopping behaviour analysis are only two examples of real-
world applications for human behaviour recognition. The
training phase is 76 percent accurate for the first 10 epochs.
The model's accuracy can be increased by increasing the
number of iterations. For testing purposes, frames from
videos are collected in a single folder. Using our trained
model, the algorithm determines whether the frames are
suspicious (cell phone use on campus, fighting, or fainting)
or regular (walking, running). A notice with the expected
class will be sent to the appropriate authorities in the event
of suspicious behaviour. The accuracy was found to be
87.15 percent.
Chitturi et al.[14] In the smart environment, human
activity discovery, or HAD, is a hotly debated topic. This
paper is about using an unsupervised approach to recognise
human behaviour. For the same, we provided three options.
We either use the LRS method or extract segments as a first
step. substrings with respect to initial step are then grouped
either directly or via sub sequences retrieved from them.
Using phrase frequency, the clusters are then identified. The
Dunns Index and the confusion matrix were used to assess
the project. The proposed systems performed better than the
existing systems, according to the findings. In addition, we
used WEKA 3.6 to compare clustering techniques and
discovered that hierarchical clustering outperforms other
clustering algorithms on the provided data files. compared to
alternative clustering techniques.
Vinayakumar.R et al.[15] The usefulness of the IRNN
and various RNN variations for ID is investigated in this
work. The detection rates achieved by IRNN methods with
respect to KDDCup-99 intrusion datafiles are very similar to
those achieved by other RNN variations. With numerous
tests with IRNN and RNN variant designs, the logic behind
the network topology and its parameters has been thoroughly
studied.
Soman.K.P et al.[16]When compared to typical machine
learning classifiers, experiments using members of RNN
modal produced a lower FP rate. RNN designs are popular
because they can retain detalies for long-period dependency
during time delay and adapt it with subsequent detalies in
sequence connections. The usefulness of RNN designs is
also demonstrated in UNSW-NB15 data files.
Jie Yin et al. [17]To eliminate the actions with a very
high likelihood of becoming normal, our method first uses a
one-class SVM which has been justified on regularly found
usual activities. To lower the wrong positive rate in an
unsupervised way, From a generic normal model, we next
create models of anomalous behaviour using kernel
nonlinear regression. We demonstrate that our method offers
a favourable trade-off between the rate of abnormality
recognition and the incorrect alarm rate and enables the
automatic derivation of abnormal activity models without
the need to exact tag the rare aberrant training data. Using
actual data gathered from a sensor network set up in a
practical environment, we show the efficacy of our
methodology.
Gurkirt Singh et al.[18] The method is easily expanded
to simultaneously detect and classify without requiring
classification scores at the video level, which creates the
possibility for classifying, detecting, and predicting online
activity. The competition organisers offered precomputed
features, which we utilised. Implementation of linear SVM
and random forest was done using SciKit-learn.
Natarajan Kumaran et al.[19]An optical flow covariance
matrix model is presented in this paper. A vital component
of video streams is optical flow, which is regular in the
temporal domain. When there are several people present,
anomalous activity can be found using the logistic regression
method. Finally, benchmark datasets like UMN, UCSD, and
BEHAVE can be used to predict the behaviours of human
crowds. The collected experimental findings demonstrate
that the suggested strategy may accurately identify
anomalous events derived from the desolate setting of
surveillance videos.
Baris Erol et al.[20] In this research, we suggested a
multilinear subspace approach for human behaviour
detection based on the radar data cube (RDC).
Using RDC provides a powerful method for combining
motion data from different domains to capture
crosscorrelations and interdependency. A single
representation that takes use of the interdependence between
the joint-variable domains for fast-time, slow-time, and
Doppler frequency is a key benefit of the suggested subspace
method.
Singh, T et all.[21]Vision-based Due of its many
applications in fields including security and surveillance,
3
human computer interactions, patient monitoring systems,
and robotics, human activity detection is quickly becoming a
popular topic of research. These datasets were split into two
categories by us. First, there are two-dimensional (2D-RGB)
datasets, and then there are three-dimensional (3D-RGB)
datasets. The most accurate algorithms for these datasets
according to state-of-the-art technology are also offered. We
quickly go over both the benefits and drawbacks of using 2D
and 3D datasets.
Furthermore, the document is organised as follows: The
Literature survey is found in the second section. The
methods part is the third component. In the fourth section,
the proposed system is discussed. The fifth portion contains
the results and discussion, while the last section contains the
study's conclusion.
III. METHODOLOGY
A. CNN
Human Pose Estimation is the art of extricating the
body's skeletal central issues and joint areas relating to the
human body parts. It makes use of the enormous number of
central problems and joints to connect the human body's
two-layered structure. In this project, we used the OpenPose
system to measure posture from an information picture. The
image is submitted to the CNN Yield organisation in
OpenPose to extract the highlights from the input. The
element map is then processed through multiple CNN layers
to yield (PAF) Part Affinity Fields and Confidence Maps. To
capture human attitude in the image, the partial affinity
fields and confidence map established above go through a
bipartite diagram matching calculation.
B. PoseNet
It contains two-layered vectors that encode the body
part's positions and directions in a picture. It encodes your
information as a two-fold connection between body parts.
L = (L1, L2, L3 … . Lc)
Lc ‫א‬R^w*h*2
c ‫{ א‬1 … C},
It is a two-layered portrayal of the conviction that a
specific piece of the body can be put on a particular pixel.
P = (P1, P2, P3 … . Pj)
Pj ‫א‬R^w*h
j ‫{ א‬1 … J},
where J is the absolute number of body parts, R is the
genuine number, and P is the arrangement of certainty maps.
The quantity of keypoints recognized through OpenPose is
subject to the dataset having been prepared.
The UCF101 dataset is used in this study, which contains
18 body core issues: R Ankle, R Knee, R Wrist, L Wrist, R
Shoulder, L Shoulder, L_Ankle, L Ear, R Ear, R Elbow, L
Elbow, L Knee, L Eye, R Eye, R Hip, L Hip, Nose
1) Algorithm
The algorithm workflow is as follows:
1. Get Joints with OpenPose.
2. Track each person, using the Euclidean distance
between the joints of two skeletons.
3. extract features of the body, and normalized joint
positions.

A video file or a video stream, is fed into the system.

Then, from each frame, the OpenPose algorithm [12] is used
to detect the human skeleton (joint locations). The skeleton
data from the first N frames is then aggregated using a
sliding window of size N. This skeletal data is preprocessed
and used for feature extraction, with the final recognition
result being put into a classifier.
A feature of a person or any object can be considered a
distinctive attribute that defines the object or person by
itself. In simpler words, it is something that is unique to that Fig. 2. UCF101 data set used for training
object. Once the preprocessing step is done, the joints or key
points are good to use for further processing. Some of the
features which are computed from the input are: direct
concatenation of joints of all the N frames, average height of D. Video Pre-processing
the skeletons, the next position; all the joints' position, length The input video is given to the model. The model will
of limbs, and joint angles computed from the joint positions. extract the frames from the video. The frames is then
IV. PROPOSED SYSTEM preprocessed to remove the noise in the image. Then the
image is processed with CNN and PosNet models to predict
In the proposed system, we have taken video footage the activity of the user.
from a camera for monitoring human activities and sending
messages to the corresponding user when any Anamoly E. Performance Measure
activity occurs. In terms of training and testing, the classifier's
A. System Architecture performance must be evaluated. Three performance
parameters are utilised to measure the performance:
Video capture, video preprocessing, feature extraction, accuracy, precision, and recall. The receiver operating
classification, and prediction are all phases of the characteristics curve is used to assess the performance of the
architecture. trained classifier. The confusion matrix’s number of true FP
and FN is used to calculate the parameters.
୘୔ା୘୒
x  —”ƒ › ൌ
୘୔ା୘୒ା୊୔ା୊୒
୘୔
x ”‡ ‹•‹‘ ൌ
୘୔ା୊୔
୘୔
x ‡ ƒŽŽ ൌ
୘୔ା୊୒

Fig. 1. System Architecture

Precision is defined as the ratio of TP to the total number
of true and FP. Recall is determined by comparing the
B. Video capture number of true positives to the total number of true positives
and false negatives.
The first stage in a video surveillance system is to set up
a camera and monitor the footage. Various types of V. RESULTS AND DISCUSSION
recordings are captured from a variety of cameras across the
The aim of the project is detect the Human activity
surveillance area. The movies must be converted to frames
using input video footage. If any abnormal activity is
because our technology processes data in frames.
detected, it will send the notification to the userInitially,
C. Dataset Description we will upload the video. This was accomplished by using
The UCF101 dataset is a well-known dataset that has video to extract features from the frames. After frames are
been utilised for training. UCF101 is a data set of realistic extracted by using poseNet, human activity is detected.
action videos containing 101 action categories acquired from
The figure 3 and 4 shows the human activity. In Fig 3,
YouTube for action recognition. This data set complements
human activity is detected to be walking. There is an
the UCF50 data collection, which includes 50 different
anamoly detected. Fig 4 shows that human activity detected
activity categories. UCF101 is the most diverse data set to
is fighting. Here anamoly is detected and it sends the
date, with substantial differences in camera movements,
notification to the user .
object look and location, object scale, viewpoint, cluttered
background, illumination settings, and other characteristics. In the below table 1, human activity detection will yield
It contains 13320 images encompassing 101 action a result by considering the contour and edges of the image.
categories. Because most action detection data sets are
illogical and produced by actors, UCF101 intends to support
further action detection research by learning and exploring
new realistic action categories.

4
 [2] Fernando Moya Rueda, Gernot A. Fink, “Convolutional Neural
Networks for Human Activity Recognition Using Body-Worn
Sensors”, 2018
[3] Mei Han; A. Sethi, “A detection-based multiple object tracking
method”, International Conference on Image Processing, 2004
[4] Rashmi R. Koli, Tanveer I. Bagban,”Human Action Recognition
Using Deep Neural Networks”, Fourth World Conference on Smart
Trends in Systems, Security and Sustainability, 2020
[5] M. Leo, T. D'Orazio, I. Gnoni, “Complex human activity recognition
for monitoring wide outdoor environments”, 7th International
Conference on Pattern Recognition, 2004
[6] Xiaoran Shi; Yaxin Li; Feng Zhou,“Human Activity Recognition
Based on Deep Learning Method”, International Conference on
Radar (RADAR), 2018
[7] Noor Almaadeed, Omar Elharrouss, Somaya Al-Maadeed, “A Novel
Approach for Robust Multi Human Action Recognition and
Summarization based on 3D Convolutional Neural Networks”, 2021
[8] Tingtian Li, Zixun Sun, Xiao Chen, “Group-Skeleton-Based Human
Action Recognition in Complex Events”, 2020
[9] Amel Ben Mahjoub, Mohamed Atri,“Human action recognition using
RGB data”, 11th International Design & Test Symposium, 2016
[10] V. Parameswari S. Pushpalatha V. Parameswari S. Pushpalatha V.
Parameswari, S. Pushpalatha, “Human Activity Recognition using
SVM and Deep Learning”, European Journal of Molecular & Clinical
Medicine, Volume 7, Issue 4, 2020
[11] Kavya , J., & Geetha , M. (2016, September). An FSM based
Fig.3 Human activity detected Fig.4 Human activity detected methodology for interleaved and concurrent activity recognition. In
is Walking is fighting 2016 International Conference on Advances in Computing,
Communications and Informatics (ICACCI) (pp. 994-999). IEEE
Classification report: [12] Ashokan, V., & Murthy, O. R. (2017, July). Comparative evaluation
of classifiers for abnormal event detection in ATMs. In 2017
TABLE I. CLASSIFACATION REPORT International Conference on Intelligent Computing, Instrumentation
and Control Technologies (ICICICT) (pp. 1330-1333). IEEE
precision recall f1-score support [13] Amrutha, C. V., Jyotsna, C., & Amudha, J. (2020, March). Deep
learning approach for suspicious activity detectio n from surveillance
1 0.98 0.98 0.98 50 video. In 2020 2nd International Conference on Innovative
0 0.98 0.98 0.98 50 Mechanisms for Industry Applications (ICIMIA) (pp. 335-339).
IEEE.
accuracy 0.98 100 [14] Chitturi, B., Thomas, J., & Indulekha, T. S. (2015, December). New
approaches for discovering unsupervised human activities by mining
Macro avg 0.98 0.98 0.98 100 sensor data. In 2015 International Conference on Computing and
Network Communications (CoCoNet) (pp. 118-123). IEEE
Weighted avg 0.98 0.98 0.98 100
[15] Vinayakumar, R., Soman, K. P., Poornachandran, P. (2019). A
comparative analysis of deep learning approaches for network
VI. .CONCLUSION intrusion detection systems (N-IDSs): deep learning for N-IDSs.
International Journal of Digital Crime and Forensics (IJDCF), 11(3),
The focus of this article is to discuss the most recent 65-89
work in this field of study. It initially discusses the objective [16] Vinayakumar, R., Soman, K. P., & Poornachandran, P. (2017).
of human pose estimation and then presents the human pose Evaluation of recurrent neural network and its variants for intrusion
detection system (IDS). International Journal of Information System
estimation approaches and key point detection methods for Modeling and Design (IJISMD), 8(3), 43-63
pose representation. It also discusses some common datasets [17] Yin, J., Yang, Q., & Pan, J. J. (2008). Sensor-based abnormal human-
and different types of classification methods that provide an activity detection. IEEE transactions on knowledge and data
effective survey study to design and develop the automated engineering, 20(8), 1082-1090.
[18] Singh, G., & Cuzzolin, F. (2016). Untrimmed video classification for
human recognition system. In numerous computer vision activity detection: submission to activitynet challenge. arXiv preprint
applications, the necessity to interpret human actions has arXiv:1607.01979.
become unavoidable. On the other hand, the major [19] Kumaran, N., & Reddy, U. S. (2021). Classification of human
application fields anamoly recognition becoming a activity detection based on an intelligent regression model in video
prominent research work because of developed techniques to sequences. IET Image Processing, 15(1), 65-76.
[20] Erol, B., & Amin, M. G. (2019). Radar data cube processing for
perform anamoly detection in front of system without the human activity recognition using multisubspace learning. IEEE
help of trainer and become self-learner. We conclude that Transactions on Aerospace and Electronic Systems, 55(6), 3617-
our study helps to design and develop the automated human 3628.
recognition system for anomaly recognition systems with [21] Singh, T., & Vishwakarma, D. K. (2019). Human activity recognition
different poses irrespective of many challenges. in video benchmarks: A survey. Advances in Signal Processing and
Communication, 247-259.
REFERENCES
[1] Fatemeh Serpush, Mahdi Rezaei, “Complex Human Action
Recognition Using a Hierarchical Feature Reduction and Deep
Learning-Based Method”, SN Computer Science, 2021