Human Anomalous Activity recognition in Healthcare

TY B.Tech. Mini Project Report

SUBMITTED BY

Mansi Jaju [T204003]

Akanksha Ghatole [T204033]
Chaitra Hiremath [T204059]

GUIDED BY

Dr. Shital Kumar Jain

SCHOOL OF COMPUTER ENGINEERING AND TECHNOLOGY,

MIT ACADEMY OF ENGINEERING, ALANDI (D), PUNE-412105

MAHARASHTRA (INDIA)

MAY, 2021
Human Anomalous Activity recognition in Healthcare

TY B.Tech. Mini Project Report

submitted in partial fulfilment of the

requirements for the award of the degree

of

Bachelor of Engineering

in

Computer Engineering & Technology

BY
Akanksha Ghatole ,Chaitra Hiremath, Mansi Jaju

SCHOOL OF COMPUTER ENGINEERING AND TECHNOLOGY

MIT ACADEMY OF ENGINEERING, ALANDI(D), PUNE-412105

MAHARASHTRA (INDIA)

MAY, 2021
 CERTIFICATE

It is hereby certified that the work which is being presented in the TY B.Tech. Mini Project
Report entitled “Human Anomalous Activity recognition in Healthcare”, in partial fulfillment
of the requirements for the award of the Bachelor of Technology in Computer Engineering &
Technology and submitted to the School of Computer Engineering & Technology,
Alandi(D), Pune, Affiliated to Savitribai Phule Pune University (SPPU), Pune is an authentic
record of work carried out during an Academic Year 2020-2021, under the supervision of Name
of Advisor(s), School of Computer Engineering & Technology.

Mansi Jaju PRN No. 0120180026 Exam Seat No. T204003

Akanksha Ghatole PRN No. 0120180196 Exam Seat No. T204033
Chaitra Hiremath PRN No. 0120180401 Exam Seat No. T204059

Date:

Signature of Project Advisor Signature of Dean

Project Adviser Dean
School of Computer Engineering &Technology, School of Computer Engineering &Technology,
MIT Academy of Engineering, Alandi(D), Pune MIT Academy of Engineering, Alandi(D), Pune

(STAMP/SEAL)

Signature of Internal examiner/s Signature of External examiner/s

Name……………………………… Name………………………………

Affiliation………………………… Affiliation…………………………
 ACKNOWLEDGEMENT

We would like to thank Dr. Shitalkumar Jain Sir for all this effort and guidance to make this
project successful and help us to improve it further. With all the inputs given during guidance we
were able to make this a better project.

We want to express our gratitude towards our respected project advisor/guide Dr. ShitalKumar
Jain for his constant encouragement and valuable guidance during the completion of this project
work. We also want to express our gratitude towards respected School Dean Prof. Ranjana Badre
for her continuous encouragement.

We would be failing in our duty if we do not thank all the other staff and faculty members for
their experienced advice and evergreen co-operation

1. Akanksha Ghatole sign

2. Chaitra Hiremath sign
3. Mansi Jaju Sign

i
 ABSTRACT

Healthcare domain is one of the major sectors of research for vision-based and machine
intelligence. Human action recognition has been improving the service of healthcare. The
objective here is to understand the behavior of patients and sick health activities of patients
using human action recognition. The purpose here is to understand major approaches for Human
activity recognition in the Healthcare domain of sensor-based and video/image-based
recognition. Here we propose a sensorless activity recognition using the video-based
implementation using a deep learning approach for anomalous activity recognition of patients or
elderly people using vision-based implementation. A Convolutional Neural Network model
followed by the Long Short Term Memory Model (CNN-LSTM) will be used for video
classification. Our proposed model will be efficient in recognition given 4 activities, a single
person in a frame at a time.

ii
 LIST OF FIGURES

Fig. No. Fig. Name Page No.

Fig 1 System Architecture for HAAR activity classification 6

Fig 2.1 Example of frames extracted for falling down activity 8

Fig 2.2 Example of frames extracted for vomiting activity 8

Fig 2.3 Example of frames extracted for blowing nose activity 8

Fig 2..4 Example of frames extracted for chest pain activity 8

iii
 LIST OF TABLES

Table. No. Table. Name Page No.

Table 1 Literature survey of research papers 2

Table 2 Comparison of previous w.r.t proposed technique 4

Table 3 Dataset description 7

Table 4 Validation Accuracy 10

iv
 CONTENTS

Acknowledgements i

Abstract ii

List of Figures iii

List of Tables iv

1. Introduction 1

1.1 Motivation for the project 1

1.2 Problem Statement 1

1.3 Objectives and Scope 1

2. Literature Survey 2

3. System Design 6

4. Implementation and Results 7

4.1 Dataset and Pre-processing 7

4.2 Algorithm and flowcharts 9

4.3 Results 10

5. Conclusion and Future scope 11

References 13
 1. INTRODUCTION

1.1 Motivations

Over the decade Human Activity Recognition (HAR) has been a major research domain in the
field of Machine learning and OpenCV. Medical image diagnosis has its major success with the
influence of modern machine learning techniques. We found there is a need for vision-based
monitoring of elderly people and patients. Such an approach would make it easy and the cost of
implementation could be reduced which gives a fair chance of implementing it even at home.

It is found in recently proposed papers that the SVM approach was used to recognize patients'
activity. Also, it is observed that sensor-based monitoring is used to detect the abnormal
behaviour using RNN LSTM , which increases the cost of monitoring a patient and elderly
person, which can be only feasible in hospital areas and some approaches were both vision and
sensor-based . Some implementations had only the vision (camera) based systems for pose-
based human recognition systems and HAR where CNN - LSTM was used to detect normal
activities of humans.

In Human Anomalous Activity Recognition (HAAR), we propose a vision-based (camera-based)

implementation for recognizing the 4 activities that affect health. Implementation can be divided
into 3 broad levels; the first step includes video segmentation where frames are segmented out
from each video. After that important features from the segmented images are extracted. Spatial
features are extracted. After that temporal features are extracted. At the end, the Classification
algorithm is applied to extracted features from segmented video to classify the activity of a
patient or elderly person.

1.2 Problem Statement

Develop a vision based Human Anomalous Activity recognition system for patients in healthcare
domain.

1.3 Objectives and Scope

The objectives covered in this project are –

1. To detect a single human body in the frame.

2. To recognize abnormal activities like sneezing, chest pain, falling down, vomit.
3. To obtain high accuracy for predicted activity

Page | 1
 2. LITERATURE SURVEY

Table 1: Literature survey of research papers

Sr. Paper Name Publication Author Findings

no
1 Activity Publication: Ellsevier B.V 1. Raw dataset is segmented into slices
Recognition The 14th by using a sliding window approach.
and Abnormal International 2. RNNs are trained to recognize daily
Behavior Conference on activities and encode daily-life
Detection with Mobile behavior routines.
Recurrent Systems and 3. The trained model is used to detect
Neural Pervasive anomalies deviating from the normal
Networks. Computing(Mo daily-life sequences.
biSPC 2017) 4. A popular dataset collected by Van
Year: 2017 Kasteren is used from 3
households .The data captures daily-
life activities such as sleeping,
cooking, leaving home, etc. using
sensors placed at the homes in less
than a month

2 An intelligent Publication: Zaineb 1. A context - specific description

knowledge Springer, Liouane, language has been designed that
system for Year: 19th Tayeb includes the daily activities and the
designing, April 2020 Lemlouma,· P unusual activities
modelling and hilippe 2. The language is named Elderly people
recognizing the Roose,Frédéri by room activities description
behaviors of c language
elderly people Weis,Hassani 3. A hybrid temporal hierarchical hidden
in smart space Messaoud. markov model has been proposed for
all types of users to recognize the
activities of the elderly and detect
abnormal activities.
4. The dataset used is collected from e-
health monitoring open data project
integrating the CASAS Aruba and
Tulum datasets.

Page | 2
3 NTU RGB+D: Publication: Amir 1. NTU RGB+D dataset is used which
A Large-Scale CVPR paper Shahroudy, have many action classes.
Dataset for 3D provided by Jun Liu, Tian- 2. They proposed the new recurrent
Human computer Tsong Ng, neural network.
Activity vision Gang Wang 3. They used RNN + LSTM
Analysis foundation methodology. Also they proposed
Year: 2016 part-aware LSTM which gave a good
accuracy upto 70%.
4. Advantages are like 3 cameras are
assigned for better view, Many actions
class, Age group from 10-35, and in
case of more than one body in the
scene, it picks one with highest
amount of 3D body motion.

4 A CNN-LSTM IEEE Ronald 1. Classified 3 (walking, sitting,

Approach to Year : 2020 Mutegeki, standing )activities on iSPL dataset
Human Dong Seog with 99.06 % accuracy
Activity Han 2. classified (laying , standing , sitting ,
Recognition downstairs walking, upstairs walking ,
walking ) activities on UCI HAR
dataset with 92.13 %
3. Challenges faced interclass similarity
4. Time series classification , CNN
followed by LSTM is used

5 Advanced Publication: Lisa Schrader, 1. More than 20 activities classified

Sensing and Springer et al. which are divided in different phases
Human Year: 2020 2. Based on their activities they were
Activity classified as healthy and patients
Recognition in 3. Methodology used Random Forest
Early algorithm classification
Intervention 4. Recognize the data based on BPMS
and sensor and video
Rehabilitation 5. Classification was based on daily
of Elderly activities elder people or patients were
People classified
6. Basic movement like in bed posture ,
getting up from bed and paitent
movement activities were future
research consideration
7. Challenge – interclass similarity ,
interclass variability and NULL class
problem

Page | 3
Various Methodology identified in literature survey

Table 2. Comparison of previous w.r.t proposed technique

Sr Methodology Activities Advantages Disadvantages

no used for Recognized
action
recognition

1 Random forest 20 Activities  Recognize the Activities had an

were data based on intra class similarity,
classified BPMS sensor interclass variability
and video. and NULL class
 Easy monitoring problem
using sensor

2 CNN- RNN Walking, jumping,  Recognize the Sufficient input data,

For pose running data based on large no. of subject
estimation video. performing activities
were some
constraints.
interclass similarity

3 CNN-LSTM On iSPL dataset:  IMU sensor data Efforts in collecting

For HAR walking, sitting, is used to the censored data.
standing. improve
predictive
UCI HAR dataset: accuracy,
laying, standing, reduces
sitting, downstairs complexity of
walking, upstairs model
walking, walking. eliminating the
need of
advanced
feature
engineering.
 Time-
dependent,
Time series
classification
(TCS)

Page | 4
4 KDA and LDA Six activities were  Highly similar This method would
techniques captured from the activities are be a failure if there
used to left side and right- differentiated by is a progressive
differentiate side view namely LDA and KDA decline in the health
between forward fall, vomit, techniques. of the elderly.
activities. chest pain, headache,  This technique
HMM is used faint, backward fall. improves the
for HAR. accuracy of
recognition.

5 Proposed A dataset is created  Each activity in The method will not

methodology- which has four the dataset be able to recognize
different activities includes videos multiple people at a
CNN-LSTM namely falling from three time.
For human down, vomiting, views that are
anomalous blowing nose, chest from left, right,
activity pain. and front.
recognition  Sensor less data
is collected.
 Camera based
implementation.
 The project is
performed on
video activities.

Page | 5
3. SYSTEM DESIGN

Fig 1 depicts the architecture of our proposed technique to solve the problem of HAAR
classification. Considering a vision-based implementation for recognition in testing phase
we get a video through camera and segment those videos. On the basis of the frames obtained
we predict the classification of activity. Based on the classified activity, we will come to
know health issues faced by the person such as vomiting, falling down, chest pain, blowing
nose. The motive behind this implementation is to get a sensor less approach as compared to
previous methodology used as mentioned in Table2.

Page | 6
4. IMPLEMENTATION AND RESULTS

4.1 Dataset and its Pre-processing

A camera-based dataset is created by us which includes four different activities namely
falling down, vomiting, blowing nose, chest pain. The dataset consists videos of 15
people with three different views that is front view, left side view and right-side view.
Each activity consists of 45 videos which sums up to 180 videos. These videos are
splitted into train and test. Frames are extracted accordingly as test frames and train
frames. The frames are saved in a gap of 15. Total number of frames extracted are 2902

Table 3. Dataset description

NUMBER OF NUMBER OF
ACTIVITIES
VIDEOS FRAMES
Falling 45 702
Vomiting 45 573
Blowing nose 45 732
Chest pain 45 895
TOTAL 180 2902

Page | 7
Fig 2.1 : Example of frames extracted for falling down activity

Fig 2.2 : Example of frames extracted for vomiting activity

Fig 2.3 : Example of frames extracted for blowing nose activity

Fig 2.4: Example of frames extracted for chest pain activity

Page | 8
 4.2 Algorithm

We have used Deep Learning Algorithm for designing recognition based system for Human
Anomalous Activity Recognition. Deep learning approach used here is combined Convolutional
Neural Network – Long Short Term Memory (CNN-LSTM) model . Initially we build the model
with CNN structure using convolutional , dense, maxpooling , and flatten layer. Then we have
passed the feature collected from the CNN layer to LSTM layers.

CNN is used for the modelling of image data (frames) and with help of layers used in CNN
model we obtained the insights on the frames extracted. LSTM is used to the check the temporal
change in frames. Fig 3 represents the model we used for HAAR activity classification.

Fig3 Algorithm

Page | 9
 4.3 Result

In process of building CNN Model we build various model with different CNN layer and made if
observation as listed in Table 4.

Table 4 Validation Accuracy

Sr. No. Model Validation Accuracy

1. CNN 57.69%
2. CNN with vggnet 46.30%
3. CNN-LSTM(still working)

Page | 10
 5. CONCLUSION & FUTURE SCOPE

To recognize the normal human activity various approaches have been proposed (HAR). Normal
and abnormal activities are also been recognized with the high accuracy.

We have proposed a method with deep learning technique (CNN-LSTM) to recognize the health
related activities of patients, so the monitoring of patients or elder person becomes easy and
reduces the human resource required for monitoring. Our model would be able to recognize the
four activities falling, chest pain, vomiting, blowing nose considering the 3 side views of a
human body

Vision based deep learning technique can be used for recognizing wide variety of activities the
understand behavior of the patients and elderly person.

Various activity depicting the nature of ill person or abnormal activities that are observed in
patients can be tracked using this approach.

This vision based technique of monitoring can be implemented in any environment without
sensors.

Page | 11
REFERENCES

[1] S. N. Boualia and N. Essoukri Ben Amara, "Pose-based Human Activity Recognition: a
review," 2019 15th International Wireless Communications & Mobile Computing Conference
(IWCMC), Tangier, Morocco, 2019, pp. 1468-1475.

[2] R. Mutegeki and D. S. Han, "A CNN-LSTM Approach to Human Activity Recognition,"
2020 International Conference on Artificial Intelligence in Information and Communication
(ICAIIC), Fukuoka, Japan, 2020, pp. 362-366.

[3] Schrader, L., Vargas Toro, A., Konietzny, S. et al. Advanced Sensing and Human Activity
Recognition in Early Intervention and Rehabilitation of Elderly People. Population Ageing 13,
139–165 (2020).

[4] Gul, Malik A.; Yousaf, Muhammad H.; Nawaz, Shah; Ur Rehman, Zaka; Kim, HyungWon.
2020. "Patient Monitoring by Abnormal Human Activity Recognition Based on CNN
Architecture" Electronics 9, no. 12: 1993.

[5] Arifoglu, Damla, and Abdelhamid Bouchachia. 2017. "Activity Recognition And Abnormal
Behaviour Detection With Recurrent Neural Networks". Procedia Computer Science 110: 86-93.

[6] Z. A. Khan and W. Sohn, "Abnormal human activity recognition system based on R-
transform and kernel discriminant technique for elderly home care," in IEEE Transactions on
Consumer Electronics, vol. 57, no. 4, pp. 1843-1850, November 2011.

[7] Liouane, Zaineb, Tayeb Lemlouma, Philippe Roose, Frédéric Weis, and Hassani Messaoud.
2020. "An Intelligent Knowledge System For Designing, Modeling, And Recognizing The
Behavior Of Elderly People In Smart Space". Journal Of Ambient Intelligence And Humanized
Computing 11 (12): 6059-6075.

[8] Buzzelli, Marco; Albé, Alessio; Ciocca, Gianluigi. 2020. "A Vision-Based System for
Monitoring Elderly People at Home" Appl. Sci. 10, no. 1: 374.

[9] Park, S.U., J.H. Park, M.A. Al-masni, M.A. Al-antari, Md.Z. Uddin, and T.-S. Kim. 2016. "A
Depth Camera-Based Human Activity Recognition Via Deep Learning Recurrent Neural
Network For Health And Social Care Services". Procedia Computer Science 100: 78-84.

[10] B. Jansen, F. Temmermans and R. Deklerck, "3D human pose recognition for home
monitoring of elderly," 2007 29th Annual International Conference of the IEEE Engineering in
Medicine and Biology Society, Lyon, France, 2007, pp. 4049-4051.

Page | 12
[11] R. V. Jagtap and M. A. Potey, "Human activity monitoring using incremental SVM in
health care domain," 2016 International Conference on Computing, Analytics and Security
Trends (CAST), Pune, 2016, pp. 222-226.

[12] R. Bhardwaj, S. Kumar and S. C. Gupta, "Human activity recognition in real world," 2017
2nd International Conference on Telecommunication and Networks (TEL-NET), Noida, 2017,
pp. 1-6.

[13] A. Shahroudy, J. Liu, T. Ng and G. Wang, "NTU RGB+D: A Large Scale Dataset for 3D
Human Activity Analysis," 2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), Las Vegas, NV, USA, 2016, pp. 1010-1019.

[14] S. Ji, W. Xu, M. Yang and K. Yu, "3D Convolutional Neural Networks for Human Action
Recognition," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no.
1, pp. 221-231, Jan. 2013.

[15] Ming-Chih Chen, Yang-Ming Liu, "An Indoor Video Surveillance System with Intelligent
Fall Detection Capability", Mathematical Problems in Engineering, vol. 2013, Article ID
839124, 8 pages, 2013.

[16] J. Yamato, J. Ohya and K. Ishii, "Recognizing human action in time-sequential images
using hidden Markov model," Proceedings 1992 IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, Champaign, IL, USA, 1992, pp. 379-385.

[17] Jian Sun, Yongling Fu, Shengguang Li, Jie He, Cheng Xu, Lin Tan, "Sequential Human
Activity Recognition Based on Deep Convolutional Network and Extreme Learning Machine
Using Wearable Sensors", Journal of Sensors, vol. 2018, Article ID 8580959, 10 pages, 2018.

[18] K. Ashwini, R. Amutha and S. Aswin raj, "Skeletal Data based Activity Recognition
System," 2020 International Conference on Communication and Signal Processing (ICCSP),
Chennai, India, 2020, pp. 444-447.

[19] A. Kovashka and K. Grauman. Learning a hierarchy of discriminative space-time

neighborhood features for human action recognition. In IEEE CVPR 2010.

[20] D. O. Esan, P. A. Owolawi and C. Tu, "Detection of Anomalous Behavioural Patterns In

University Environment Using CNN-LSTM," 2020 IEEE 23rd International Conference on
Information Fusion (FUSION), Rustenburg, South Africa, 2020, pp. 1-8.

[21] T. Gatt, D. Seychell and A. Dingli, "Detecting human abnormal behaviour through a video
generated model," 2019 11th International Symposium on Image and Signal Processing and
Analysis (ISPA), Dubrovnik, Croatia, 2019, pp. 264-270.

Page | 13
[22] Wang, Lei & Xu, Yangyang & Cheng, Jun & Xia, Haiying & Yin, Jianqin & Wu, Jiaji.
(2018). Human Action Recognition by Learning Spatio-Temporal Features With Deep Neural
Networks. IEEE Access. PP. 1-1. 10.1109/ACCESS.2018.2817253.

[23] Rodríguez-Moreno, Itsaso & Martinez-Otzeta, Jose Maria & Sierra, Basilio & Rodriguez
Rodriguez, Igor & Jauregi Iztueta, Ekaitz. (2019). Video Activity Recognition: State-of-the-Art.
Sensors. 19. 3160. 10.3390/s19143160.

[24] I. Jargalsaikhan, S. Little, R. Trichet and N. E. O'Connor, "Action recognition in video

using a spatial-temporal graph-based feature representation," 2015 12th IEEE International
Conference on Advanced Video and Signal Based Surveillance (AVSS), 2015, pp. 1-6, doi:
10.1109/AVSS.2015.7301760.

[25] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action

recognition in videos,” in Advances in Neural Information Processing Systems (NIPS), 2014,
pp. 568–576.

Page | 14