Human Pose Estimation Using Convolutional Neural Networks

Anubhav Singh1, Shruti Agarwal2, Preeti Nagrath3, Anmol Saxena4, Narina Thakur5
1,3,4,5
Computer Science Department, Bharati Vidyapeeth’s College of Engineering, New Delhi, India
2
Electronics and Communication Department, Bharati Vidyapeeth’s College of Engineering, New Delhi, India
1
anubhav.singh2359@gmail.com, 2agarwalshruti2801@gmail.com
3
preeti.nagrath@bharatividyapeeth.edu, 4anmsaxena123@gmail.com, 5narinat@gmail.com

Abstract: Human pose estimation has always been a
challenging problem that holds great attention, it has the
widespread and extensive variety of uses from the
classification of images to activity acknowledgment, main
challenge is the detection and localization of the key points in
the variation of several body poses. To resolve this issue,
substantial research work have been done in this area. This
paper discusses the issues in human pose estimation and
gives the overview of considerable research work in pose
estimation, including deep learning approach and customary
image-based techniques. After analyzing several results and
detecting the restrictions, the author has reconstructed a
simple model using convolutional neural network that
estimates the poses and demonstrates the potential of CNN's.
The author concludes with a few promising bearings and
directions that have to be explored for future research.
I. INTRODUCTION
Human pose estimation holds extraordinary potential from
single, 2D pictures to aid an extensive variety of uses from the
classification of images and recordings, activity
acknowledgment, active investigation, and grabbed great
attention in computer vision and human PC interaction.
However, human posture estimation has always been a
challenging problem that acquires great attention. It involves
huge difficulty for the identification and localization of key
points of the body that mainly includes various joints and body
movement forecast and also shares difficulties in detection, for
example in clustering, lighting, perspective, and scale, are the
significant troubles interesting to human postures[1].
The detection of body key points has been a great problem due
to little joints, impediments, and the need to catch content.
Hence convolutional neural networks have a remarkable
approach to image classification and object identification
issues. Fundamentally they are similar to conventional neural
systems that are comprised of neurons with predefined weights
and predispositions. However, neural systems do not scale well
to larger pictures. The authors rapidly create countlessly and
wind up overfitting on the preparation set as every neuron in a
layer is completely associated with every one of the neurons in
the past layer. CNN took advantage of the fact that comprises
mainly of pictures, so they constrain the engineering in a more
sensible manner which incomprehensibly lessens the number
of parameters. CNN is engaging for human posture estimation
as firstly there's no compelling reason to design feature
representations and identifiers for parts because the model and
its attributes are been demonstrated from the data itself.
Secondly, the model that has been learned is comprehensive,
where the last joint prediction is based on a complex non-
linear change of the entire image, as restricted to local
identifiers whose thinking is a solitary part and can just model
a little subset of between activities and between body parts.
There are some difficulties in the prediction of human pose
coordinates: the foreshortening of appendages, impediment of
appendages, turn and introduction of the figure, and cover of
different subjects. For example, there are various cumbersome
poses to annotate: rotation, foreshortening, occlusion, and
various figures this fluctuation in the info frame proposes that
the entire thinking given by CNN might be a great system. In
this venture, researchers investigate diverse CNN models for
demonstrating human posture estimation and activity
classification[2].
Pose estimation has ample uses and has numerous utilizations,
response to the body to augmented reality to situation related
support, liveliness, wellness uses, and that's just the beginning.
The authors in this paper believe the availability of this model
motivates more designers and producers to try and apply
present recognition to their very own unique projects. While
many substitute posture location frameworks have been
publicly released, all require specific equipment or potentially
cameras, and also a lot of framework setup. this human pose
estimation can be utilized to gauge either a single pose or
multiple poses, which means there is a form of the algorithm
that can recognize just a single individual in a picture and one
form that can identify numerous people in a picture.
Therefore for the advancement and solution for this
challenging problem MPII Human Pose dataset has been used
for analyzing human postures. It involves around 25k images
of 40k people having various illustrated body joints .Every
picture was collected from YouTube videos and has former
and un-illustrated frames involved 3D torso and head
movements.
Therefore the proposed benchmark “MPII Human Posture”
fundamentally progresses in terms of appearance variability
and complexity, and incorporates in excess of 40,000 pictures
of individuals or full-body human posture estimation.

978-1-5386-9346-9/19/$31.00 ©2019 IEEE

While the methodology depends on convolutional systems
(convnets), testing dataset, and furthermore an investigation of
what is required to make convnets work in estimation of
human pose. The model inputs a color image having size w*h
and outputs 2D positions of key points for each person present
in the picture itself. The model has layers used for creation of
input image has two branch multistage CNN where first branch
forecast about the confidence maps of various body parts
localization like elbow,knee etc. and second branch tells about
2D vector fields(L) and tells about the degree of correlation
between parts and therefore produce the 2D key points for all
members present in the picture.The Multi-Person
Dataset(MPII) is learned specialized model and has 15 outputs
points.
Specifically, presenting a two-arrange sifting method whereby
the reaction maps of convnet part identifiers are denoised by a
additional procedure educated by the division pecking order
.The proposed system is tried in the comparing parcels of the
professionally presented benchmarks (i.e. in pictures that have
not been utilized for preparing), and the outcomes are
contrasted with the first system, in this manner approving the
assumption that profundity data helps to assess the human
body's joints. The exploratory approval demonstrates that it is
conceivable to exploit the halfway relative-depth
estimation[3]. The recognition of human poses have
tremendous applications in various fields and do wonders in
several areas, by recognizing the human activities through
poses the world would be more safer, healthier and prove as a
boon to industries in robotics by making them learn and
familiar with human poses, in health and wellness by making
any smart gadget into a scanners, and also recognizing the
medical emergencies, helpful in recognizing pedestrian in
autonomous vehicles and preventing several serious and death
prone accidents, helpful in security purposes while tracking
and identifying human poses, gestures while recognizing
doubtful activity and demanding immediate actions.
II. PROBLEM STATEMENT
The problem consists of human pose estimation. For
estimation of human postures, the network takes a raw image
as input and a vector of coordinates of the body key points as
outputs. The purpose is to identify x-y pixel coordinates for 15
body joints. By training the regression CNN that compensate
and decrease the loss.
III. THEORETICAL FRAMEWORK
Part Affinity Fields(PAFs)[4] is a non-parametric portrayal
approach that is used to learn about various body parts of
individuals in the images distinguishing the 2D postures and
more effectively incorporating numerous individuals in an
image. This global context allows a greedy bottom-up
methodology keeping up a high accuracy independent of
multiple individuals in an image. This engineering is intended
947
to learn part locations and their relationship through two parts
of the same consecutive forecast process.
This PAF approach present us a productive strategy for multi-
individual posture estimation having a high correct nesses in a
few open benchmarks.
Convolutional Neural Networks (CNN) methodology is used
for the estimation of 2D human postures from a single picture.
Recently, for the assessment of human postures, many
techniques and procedures have been created that utilize
postures from physiologically propelled graphical models.
Arrangement of image patches takes as input in CNN and
determine the correct location of the joints in the picture
utilizing CNN approach, achieving both joint recognizable
proof which decides whether an image comprises freeze body
joint and joint constraint finds the right region of joint in the
image plot. Finally, these joint recognition/limitation results
are combined for surveying the posture.
Dual-Source Deep Convolutional Neural Network(DS-CNN)
[5] integrates both local(body) part appearance and holistic
view of every local part to get a better human pose estimation.
Basically, in this paper the author is taking set of image
patches as the input and by using holistic view learns the
appearances of each local part. By DS-CNN they are achieving
both joint detection and joint localization and then estimates
the human pose by combining the both.
The researcher [6] uses Convolution Neural Network for
human posture estimation where primary responsibility is a
CNN fell structure especially planned for adjusting part
associations and spatial contexts. The initial segment performs
part identification heatmaps and second part performs
regression on these heatmaps managing the framework where
to focus in the image and enough encodes part requirements
and context. Likewise, author shows that the planned course is
sufficiently adaptable to expeditiously allow the blend of
various CNN structures for both acknowledgmentand relapse.
The authors [7,8] in their paper proposed named Location
based or relapse based[9,10,11] verbalized human posture
estimation. The Location constructed techniques are depending
with respect to intense CNN-based part identifiers which are
then solidified using a graphical model. Relapse-based
strategies endeavor to take in a mapping from the picture and
CNN features to part areas.
The authors in their paper uses CNN [12] which is trained for
working on scene marking, by characterizing a multi-
classification characterization for every pixel. Rather, they
describe location over sliding windows in the image. Since,
they allow their revelation over each window to contain
various body parts, where every identification task is basically
a paired characterization job in a window. Network
optimization is been done with L2 loss without seeing the
effect on outliers on the training process which would be
affected by outliers.
The author has used a regression network with ConvNets that
attains robustness against these outliers by reducing Tukey’s
bi-weight function an m-estimator robust to outliers. The
author illustrates quicker combination with improved
speculation of the robust loss function for the estimating the
posesfrom images of faces.
A significant Convolution neural system is utilized in
heterogeneous performing various tasks learning structure for
human posture estimation from single images. This framework
takes in a posture joint regression and a sliding-window body-
part indicator in a deep organize design demonstrating that
including the body-part location regularizes the system
producing better solutions. The researcher reports a
competitive and state-of-arts results in manydatasets.[13]
The authors in their paper, utilized a heterogeneous multi-task
regression task method [14] and the arrangement assignment
that urged to have a similar sparsity design. They discovered
that joint-preparing patterns to locate the most valuable
element in the contribution for the two assignments to have a
similar component layer, which brings about learning shared
portrayal that is useful for both thetasks.
The author also illustrated that deep neural system[15] forecast
and identifies the location of the various body parts like head
and hands just by knowing the close by neighbors with learned
lodge features by taking a pose-sensitive embedding with non-
linear NCA (neighborhood components analysis) regression.
Instead, likewise, from regression network systems the work
can directly yield the joint locations from the accessory
undertakings learning shared"posehighlights".
A multi-stage framework with the deep convolutional
network[16] is worked in this paper for foreseeing facial point
areas. They use an arrangement of neural frameworks that
emphasis on various areas of the input picture.
The researchers estimates different CNN structures[17] with
regard to hand shape, joint visibility and articulation
distributions which include isolated 3D hand pose estimation
targeting low mean errors, 3D volumetric representations
outperform 2D CNN capturing the spatial structure of the
depth data.
The authors describes Human Mesh Recovery(HMR)[18]
which is an end to end framework for rebuilding a full 3D
mesh of human body form an RGB image. This paper
computes a much richer depiction that is parameterized by
shape and 3D joint angles running in a real-time with person in
bounding box.
948
Correspondingly, prepared well trained convolutional systems
for human posture estimation. As an alternative of growing the
quantity of stages for enhancement, authors investigate how to
enhance the execution of a monocular regression system
showing add-on task [19].
IV. FORMULATED WORK
The proposed work objective is posed estimation. For the
former task, the authors aim to identify x-y pixel coordinates
for 14 body joints depicted in the Figure 1. For the latter, they
aim to label the images based on the activity category and to
estimate the poses using Convolutional Neural Network
(CNN). This paper proposes the regression approach as a
solution to human posture estimation problem that can be
designed as a hereditary convolutional neural network. The
CNN outputs the pixel coordinate of each body key joints by
taking as input a full raw image of (96*96pixels).
Since 1980s this field of research has been considered
important by computer science communities because of it
having a strong point in giving modified support to other
applications. It has range of implementations from medicine,
human-computer interaction to sociology.
V. RESEARCH METHODOLOGY
In pose estimation, the authors identify the position and
orientation of an object mean and detect keypoints locations
that describe the object. This paper will discuss the human
pose estimation where major parts/joints are detected and
localized in the body analyzing the state of the art in
articulated human pose estimation using a new large-scale
benchmark dataset.
The dataset includes 40,000 images collected through an
recognized catalogue of human activities. The collected images
cover 410 human activities in total. Pictures are analyzed on
the basis of annotations including body pose, body part
occlusion, torso, and head viewpoint, and personal activity The
architecture takes an input of w*h size and producing an
output which includes of 2D locations of anatomically key
points for every person in the image.
The network predicts a set of locations and a set of confidence
maps S which has body part locations and degree association
between parts in the form of 2D vector fields. Where there are
J confidence maps in set S and C vector fields in L.
The procedure involves feeding of images into feedforward
network which predicting a set of locations with a set 2D
vector fields having degree association between parts. Where S
consists of J confidence maps and set L has C vector fields.
Beige color represents the confidence map and lowers branch
represented as blue affinity fields.
MPII dataset outputs 15 points which are :

Fig. 1. MPII Dataset Images[20]

Figure 1 represent various human activities includes in the dataset

TABLE 1: Various joints and their keys

Joint Head Nec Rt. Rt. Rt. Lt. Lt. Lt. Rt. Rt. Rt. Lt. Lt. Lt. Chest Backgroun
k Shoul elbo wrist Shou Elbo Wris Hip Knee Ankl Hip Knee Ank d
der w lder w t e le

Key 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Link to download dataset is in [20].

The model inputs a color image of size w × h outputs the 2D locations of keypoints for every person in the picture. The
recognition procedure is in three stages:

Fig. 2. Architecture of the two-branch multistage CNN. After every stage the
predictions of previous branches and image feature are concatenated .[21]

949
In Stage 0 feature maps for the input image is created using the
first 10 layers of the VGGNet as shown in Figure 2.
The starting 10 layers of the VGGNet would create a feature
map. Then stage 1 consists of dual branches, the first branch
predicts a set of 2D confidence maps(S) of the location of body
parts. Whereas, confidence maps of affinities encode the
degree of association between parts. Then stage 2 both the
maps i.e., greedy inference parse confidence and affinity maps
to produce the 2D key points for all people in the image where
each stage where every subsequent stage concatenation of
predictions from both branches in the previous stage and the
original image is performed.
VI. PROPOSED WORK
Backpropagation is being used optimizing weights w, then
performing mini-batch gradient descent on a batch of 128.
Figure 3 shows the regression architecture for the keypoint
location estimation where the final layer outputs a 2D
dimensional vector representing the x and y coordinate of all
14 points.
Fig. 3. Regression CNN architecture for keypoint location
estimation.
950
as the aim is to identify x-y pixel coordinates for 14 body
joints depicted in the figures above. For the latter, aim is to
label the images based on the activity category and to estimate
the poses using Convolutional Neural Network
(CNN).Formulating the human posture and estimating the
problem as a regression problem that can be designed as a
hereditary convolutional neural network. The CNN outputs the
pixel coordinate of each body key joints by taking as input a
full raw image of (96*96 pixels).
Where number of training examples are N, D is the train-set,k
is the number of body key points, and wi is the weights learned
for the ith body keypoint.
VII. RESULT AND ANALYSIS
Figure 4 represents some results containing viewpoint and
appearance artifacts from model described in Figure 2.
Whereas Figure 5 shows the body points that are been
predicted.
Fig. 4. Results containing viewpoint and appearance variation,
occlusion, crowding, contact, and other common imaging
artifacts.
Table 2 below shows the earlier state of the art significantly their ease and instinctive nature, and their lessened number of
exceeding previous state results on the MPII multi-person a parameter when diverged from the totally related model.
benchmark. It has some qualitative results.
CNN has the advantage of being a model that accepts the
TABLE 2: (a): Evaluation result on the testing subset. whole picture as an info motion for each body points rather
(b): Evaluation result on the whole testing subset. than nearby locators where they are obliged to a single part.
The subset of 288 images
The authors demonstrated that human posture estimation can
Method Hea Sho Elb Wri Hip Kn Ankl s/im be thrown a regression issue and demonstrated with a non-
e e age exclusive CNN. Their use of CNN to the issue of human
posture estimation accomplishes aggressive outcomes on a
Deepcut[ 73.4 71.8 57.9 39. 56. 44. 32.0 579 testing scholarly dataset with a basic model. They guess that
22] 9 7 0 95 they would have the capacity to accomplish shockingly better
outcomes with more processor power and space (the depth of
Iqbal et 70.0 65.2 56.4 46. 52. 47. 44.5 10
al.[23] 1 7 9 their relapse CNN was limited by the RAM of the GPU it was
prepared on.
Deeper 87.9 84.0 71.9 63. 68. 63. 58.1 230
Cut[24] 9 8 8 To diminish the hole among preparing and approval execution
additionally calibrate the hyperparameters of their
demonstrate. Interesting points incorporate altering the base
Full Testing Set learning rate and learning rate approach, trying distinctive
kinds of momentum updates, and tuning the regularization
Method Hea Sho Elb Wri Hip Kne Ank s/im
quality. Besides, display gatherings can be utilized to
age increment performance. They might likewise want to try
different things with a blend of joint estimation and action
Deepcut 73.4 72.5 60.2 51.0 57. 52.0 45.4 48.5 characterization undertakings to check whether knowing the
[24] 2 areas of joints in a picture enhances the air conditioner activity
arrangement execution of a CNN. They would first run the info
Iqbal et 70.0 53.9 44.5 35.0 42. 36.7 31.1 10 picture through the joint estimation relapse model to get the
al.[23] 2 human posture data and utilize this as an optional contribution
(notwithstanding the first info picture) into the characterization
demonstrate. This extra data may enable their model to decide
the movement is performed in the picture. Nonetheless, it is
conceivable that 2D pixel coordinates won't give adequately
valuable data regarding the genuine 3D posture estimation.

REFERENCES
[1] Ντίνου, Ι. (2018). Human pose estimation using convolutional
neural networks (Doctoral dissertation).
[2] Cao, Z., Simon, T., Wei, S. E., & Sheikh, Y. (2016). Real-time
multi-person 2d pose estimation using part affinity fields. arXiv
preprint arXiv:1611.08050.
[3] Jain, A., Tompson, J., Andriluka, M., Taylor, G. W., &Bregler,
C. (2013). Learning human pose estimation features with
convolutional networks. arXiv preprint arXiv:1312.7302.
[4] Cao, Z., Simon, T., Wei, S. E., & Sheikh, Y. (2016). Real-time
multi-person 2d pose estimation using part affinity fields. arXiv
preprint arXiv:1611.08050.
[5] Fan, X., Zheng, K., Lin, Y., & Wang, S. (2015). Combining
local appearance and holistic view: Dual-source deep neural
networks for human pose estimation. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition
Fig. 5. 14 body points that aim to identify (pp. 1347-1355).
[6] Bulat, A., &Tzimiropoulos, G. (2016, October). Human pose
VIII. CONCLUSION AND FUTURE WORK estimation via convolutional part heatmap regression. In
European Conference on Computer Vision (pp. 717-732).
If the authors consider PC vision assignments convolutional Springer, Cham.
neural networks system are most demanding design due to

951
[7] Chen, X., Yuille, A.L.: Articulated pose estimation by a
graphical model with image dependent pairwise relations. In:
NIPS. (2014)
[8] Tompson, J.J., Jain, A., LeCun, Y., Bregler, C.: Joint training of
a convolutional network and a graphical model for human pose
estimation. In: NIPS. (2014)
[9] Toshev, A., Szegedy, C.: Deeppose: Human pose estimation via
deep neural networks. In: CVPR. (2014)
[10] Pfister, T., Simonyan, K., Charles, J., & Zisserman, A. (2014,
November). Deep convolutional neural networks for efficient
pose estimation in gesture videos. In Asian Conference on
Computer Vision (pp. 538-552). Springer, Cham.
[11] Pfister, T., Charles, J., Zisserman, A.: Flowing convnets for
human pose estimation in videos. In: ICCV. (2015)
[12] Belagiannis, V., Rupprecht, C., Carneiro, G., &Navab, N.
(2015). Robust optimization for deep regression. In Proceedings
of the IEEE International Conference on Computer Vision (pp.
2830-2838).
[13] Li, S., Liu, Z. Q., & Chan, A. B. (2014). Heterogeneous multi-
task learning for human pose estimation with the deep
convolutional neural network. In Proceedings of the IEEE
conference on computer vision and pattern recognition
workshops (pp. 482-489).
[14] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning
hierarchical features for scene labeling.IEEE TPAMI, 2013
[15] G. W. Taylor, R. Fergus, G. Williams, I. Spiro, and C. Bregler.
Pose-sensitive embedding by nonlinear nca regression. In NIPS,
pages 2280–2288, 2010
952
[16] Y. Sun, X. Wang, and X. Tang. Deep convolutional network
cascade for facial point detection. In CVPR, 2013
[17] Yuan, S., Garcia-Hernando, G., Stenger, B., Moon, G., Yong
Chang, J., Mu Lee, K., ... & Yuan, J. (2018). Depth-based 3d
hand pose estimation: From current achievements to future
goals. In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR).
[18] Kanazawa, A., Black, M. J., Jacobs, D. W., & Malik, J. (2018,
June). End-to-end recovery of human shape and pose. In The
IEEE Conference on Computer Vision and Pattern Recognition
(CVPR).
[19] A. Toshev and C. Szegedy. Deeppose: Human pose
estimation via deep neural networks. In CVPR, 2014.
[20] http://human-pose.mpi-inf.mpg.de/
[21] Cao, Z., Simon, T., Wei, S. E., & Sheikh, Y. (2016).
Realtime multi-person 2d pose estimation using part affinity
fields. arXiv preprint arXiv:1611.08050.
[22] L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M.
Andriluka, P. Gehler, and B. Schiele. Deepcut: Joint subset
partition and labeling for multi person pose estimation. In
CVPR, 2016.
[23] U. Iqbal and J. Gall. Multi-person pose estimation with local
joint-to-person associations. In ECCV Workshops, Crowd
Understanding, 2016.
[24] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, and
B. Schiele. Deepercut: A deeper, stronger, and faster
multiperson pose estimation model. In ECCV, 2016.