Foot Recognition Using Deep Learning
for Knee Rehabilitation
Abstract—The foot recognition can be applied in many medical
fields such as the gait pattern analysis and the knee exercises of
patients in rehabilitation. Generally, the foot recognition system with
a camera is aimed to captures an image of the patients in a controlled
room and background to recognize the foot in the limited views.
However, this system seems to be not convenience for monitoring
the knee exercises at home. This research proposes a method for the
foot recognition using a deep learning method and compares with
traditional classification method. According to the results, deep learn-
ing method provides better accuracy and complexity to recognize the
foot images from online databases than the traditional classification
method.
Keywords—Foot Recognition, Deep Learning, Knee Rehabilita-
tion, Convolutional Neural Network
I. I NTRODUCTION
I N general, a knee physical rehabilitation [1] aims to in-
crease the strength, mobility and fitness of the knee for a
patient. This exercise is observed by a physical therapist (PT)
to provide a feedback to the patient such as the limited range
of motion of knee (angle). Normally, the PT uses a goniometer
to measure angle of knee joint during knee rehabilitation as
shown in below figure. Nonetheless, it may be not suitable for
home rehabilitation or many patients in the hospital.
Normally, there are many examples of knee rehabilitation by
using body weigh, machines and free weights [2] . Considering
the main postures of these, the foot of the patient is the main
part of the body for moving the direction under the instructions
of PT while the knee joint of the patient is fixed for protection
painful on its . It seems to be measurable the angle of knee
by tracking the foot movement during knee rehabilitation.
Using a marker less system (MLS) [3] for tracking a
foot with camera is able to monitor a patient during do a
rehabilitation program. In addition, it is low cost and easy
to setup. However, there are problems such as an occlusion, a
specific room and background and limited the view of captures
to recognize the foot images. Some patients additionally prefer
to wear shoes/socks than to be a barefoot to do a rehabilitation
program so it is hard to recognize the foot location in the
image.
In machine learning for computer vision application, it
becomes useful for to identify objects in raw images, search
items and translation information. The common model of
machine learning is also supervised learning. To build this
system, it consists of a proper data set of images that are
labelled with its class. For conventional machine learning
techniques, theirs system required a feature generation module
to convert the low level information such as the pixel values
to suitable feature vectors such as the local binary patterns
(LBPs) [4] and the histogram of oriented gradients (HOG)
[5]-[6] . These features are able to represent its class by
using a classifier which is trained from the labelled data set.
Nonetheless, if the data set are variety in same class, it seems
to be that theirs feature vectors tends to be not satisfactory to
fit with the classifier.
Deep learning [7] is a new area of machine learning
application. It can additionally applied to the low level and
high level information in the large data set. The deep learning
architecture is normally similar to artificial neural networks
but its hidden layers and number of nodes are complex. As
a result, it can identify an object in high accuracy such as
handwritten digits, multiple objects and variations of images.
The purpose of this study is to develop the foot recognition
using a deep learning method for knee rehabilitation, and it
is evaluated the deep learning method with the traditional
methods by using an accuracy and complexity.
II. M ETHODS
This paper proposed the foot recognition using Convolu-
tional Neural Network (CNNs) and compared with traditional
cassification methods.
A. Traditional classification methods for foot recognition
As can be seen from Figure 1, there are four main steps of
foot recognition.
1) Preprocessing Images Data: The raw images are nor-
mally labelled, cropped and converted into the suitable color
space. These data set consist of two types of data such as
positive and negative images.
2) Feature Generation: It is mathematical calculation to
produce a feature vector from a raw image. The feature vector
seems to be represented the information of its image. In this
research considers LBP and HOG features.
• LBP is a basic features by using the LBP mask to
calculate the gray images as demonstrated in Figure 2.
This mask is generally divided the images into cells
determined by a radius of its mask. For example, if the
radius is one, the neighbors pixel around the cells is 8
points as showed in Figure 3. The final of LBP feature
are normalized by using histogram of its LBP values and
reduced the dimension by the rotation invariant uniform
LBP [8].
• HOG is a feature that uses the histograms of oriented gra-
dients of a gray image. Generally, its process initializes
a parameters such as the sizes of cell, blocks and bins.
The image is calculated the gradients of both horizontal
axis, vertical axis and a magnitude an angle of gradients.
This edge histogram is created as gradient vote depending
on bins parameter with normalized magnitude of gradient
vote. Finally, the 2-dimensional features are changed into
 single vector of features to represent the information of
the image as illustrated in Figure 4.
3) Classifier: The third step is a classifier. In general, the
feature vectors of image are trained by the labelled theirs
class. The classifier model will learn from both negative and
positive images. In this research, SVM and kNN are used in
the experiment.
• Support Vector Machine (SVM) is a popular classifier
which based on a separation planes of the class by super-
vised learning [5]. The plane is normally divided a group
of items in different class. To identify the suitable plane,
it requires the maximization of the distances between
nearest item point of class and the decision plane. Its
distance is also called as margin. As a result, the selection
of plane with high margin will provide a robustness
classifier.
• k Nearest Neighbor (kNN) is a simple classifier [9].
It is also a lazy algorithm because it does not need
training phase. For kNN, all feature data is calculated a
distance such as Euclidean distance between the unknown
points and every data points. The result of classification is
depending on the class of the nearest k-distances with the
highest frequency of corresponding class to the nearest k-
distances.
4) Recognition Results: The last step aims to evaluate the
performance of foot recognition from the test images without
relating in training images. In this paper, it focuses on two
measurements: sensitivity and specificity.
B. Convolutional Neural Networks (CNNs) methods for foot
recognition
Among the deep learning, the CNNs has continuously been
the effective method in image recognition, object tracking, face
analysis. In general, the CNNs consists of varying component
elements of three main layers as shown in Figure 5.
• Convolutional layers are the main core of filters or
kernels to calculate the features such as lines, edges and
corners of an image. Normally, these filters are the mask
matrix of numbers and moving over the input image to
calculate the specific features. The convolution operation
of filters are also dot product and summation between the
filters and input image. The output of these operation are
usually passed through an activation function depending
on its purpose such as the Rectified Linear Unit (ReLU)
activation function for non-linear input.
• Pooling layers are generally reduced the dimensional
layers. These layers also represent the local and local
or global pooling layers. The pooling operation may be
maximum, average and summation of each of a cluster
of data at the prior layer.
• Fully connected layers is the final layers which connect
every neuron in previous layer to every neuron on the next
layer. The fully connected layer normally uses a softmax
activation function in the output layer to classify the input
image into several classes based on the training image.
fig:ExamFoot
Fig. 1. Diagram of foot recognition
III. E XPERIMENTAL SETUP
A. Dataset
In this study, the data consists of two classes (N=8000) as
demonstrated in Figure 6 and Figure 7: foot images (N=4000)
and non-foot images (N=4000). In case of foot images, they
provide the foot images in different views, varieties of bright-
ness, bare foot and wearing shoe/sock. In term of non-foot
images, they are randomized cropping from the living room
images. Both two classes of image are from online dataset
such as Pascal VOC2012 [10] and image-net.org [11]. In case
of foot recognition with the traditional methods, it requires
for feature generation (LBPs and HOG) of images before
processing in training step to train and test the classifier
model. Considering the training step, the deep learning use
training data (100%): training set (80%) and test set (20%) to
adjust the deep learning model. The validation data is about
20% of training data to report the accuracy. The traditional
methods are using 5-fold validation with training set (80%)
and validation data (20%) to report the average of accuracy
for all 5-fold validation.
B. Configuration
In classifier model step, the deep learning uses Tensorflow
14.0 module in Python 3.6 with simple model running on
400 iterations as configuration in Figure 8. In case of the
two traditional methods for foot recognition, the former is k-
nearest neighbors (kNN) with k = 3, 5 and 7 by calculated
their Euclidean distance. The latter is support vector machine
(SVM) [12] with the linear SVM. Both classifiers process by
scikit-learn 0.19.1 module in the Python 3.6.
C. Evaluation
In this paper, it focuses on three measurements to evaluate
the foot recognition methods: sensitivity, specificity and com-
plexity. The sensitivity is able to assess the accuracy of foot
recognition as positive images. In case of the specificity, it
can evaluate the accuracy of non-foot recognition as negative
images. For complexity evaluation, the number of feature, the
number of connection node and Big-O-notation of classifier
are compared of each methods.
IV. R ESULTS
n this section, there are two main indicator of evaluation for
recognition: performance and complexity of each methods.
Fig. 2. Diagram of LBP calculation

Fig. 3. Example of the LBP pixels calculation along a clockwise

Fig. 8. Configuration on CNNs

A. Performance of Recognition
Fig. 4. Diagram of HOG calculation
The Table I shows the performance for foot recognition by
using LBP, HOG with SVM or kNN classifier and CNNs
methods. First of all, it is clear that HOG+kNN and CNNs
provide the higher sensitivity with over 0.96 than LBP+kNN.
As can be seen from the Table II, it is the performance
for non-foot recognition. Overall, both the kNN and the SVM
achieve high specificity over 0.92 with LBP features while
the HOG feature performs the low specificity. Moreover, the
CNNs is the lowest specificity in this experiment.
Fig. 5. Diagram of CNNs for foot recognition

B. Complexity
The Table III illustrates the number of each features where
Nf is the number of features and Nc is number of convolutions
in the neural network. Beginning with the Nf , the HOG
provides the maximum number of features (Nf = 1764) with
a default parameter [13]. Meanwhile, the LBP with r = 1 [8]
shows that it has a lowest number of features (Nf = 10) in
these study. Turning to the CNNS, it can be seen that the Nc
is about 88 million nodes.
The Table IV shows that the Big-O-notation complexity of
each classifier where Ns is the number of samples for training,
Fig. 6. Example of foot images Nf is the number of features, k is the number of majority
votes of its nearest neighbor and Nc is number of convolutions
in the neural network. To begin with the SVM, it seems to
be the lowest complexity of classifier as O(Nf ) in among
these experiment. By contrast, in case of CNNs, it provides
the highest complexity of classifier depending on the number
of Nc as shown in Table III.

V. D ISCUSSION AND C ONCLUSION

In case of sensitivity, the CNNs and HOG+kNN gain high
recognition for foot images. Meanwhile, the specificity of them
shows is lower than LBP+SVM. It seems to be that these
CNNs model in the experiment may not suitable to detect the
Fig. 7. Example of non-foot images background image with variety of shaped, color, and texture
of image.
 TABLE I
S ENSITIVITY OF FOOT RECOGNITION R EFERENCES
[1] Anwer S, Alghadir A. Effect of isometric quadriceps exercise on muscle
Sensitivity strength, pain, and function in patients with knee osteoarthritis: a
Method kNN randomized controlled study. J Phys Ther Sci. 2014;26(5):745-8.
SVM
k=3 k=5 k=7 [2] Vincent KR, Vincent HK. Resistance exercise for knee osteoarthritis.
LBP PM R. 2012;4(5 Suppl):S45-52.
r=1 0.74 ± 0.02 0.89 ± 0.01 0.88 ± 0.01 0.88 ± 0.01 [3] Ceseracciu E, Sawacha Z, Cobelli C. Comparison of markerless and
r=2 0.83 ± 0.03 0.90 ± 0.01 0.89 ± 0.02 0.89 ± 0.02 marker-based motion capture technologies through simultaneous data
r=3 0.82 ± 0.02 0.89 ± 0.01 0.88 ± 0.01 0.87 ± 0.01 collection during gait: proof of concept. PLoS One. 2014;9(3):e87640.
r=4 0.81 ± 0.02 0.88 ± 0.02 0.86 ± 0.02 0.86 ± 0.02 Published 2014 Mar 4. doi:10.1371/journal.pone.0087640
r=5 0.86 ± 0.01 0.88 ± 0.02 0.87 ± 0.01 0.87 ± 0.01 [4] J. Kwak, S. Xu and B. Wood, ”Efficient data mining for local binary
HOG 0.80 ± 0.12 0.96 ± 0.01 0.97 ± 0.01 0.97 ± 0.01 pattern in texture image analysis”, Expert Systems with Applications,
CNNs 0.98 ± 0.01 vol. 42, no. 9, pp. 4529-4539, 2015.
[5] Ma, Y., Chen, X., Chen, G., Pedestrian detection and tracking using
TABLE II HOG and oriented-LBP features, Network and Parallel Computing,
S PECIFICITY OF FOOT RECOGNITION 2011, pp. 176-184.
[6] L. Hou, W. Wan, K. Han, R. Muhammad and M. Yang, ”Human
Specificity detection and tracking over camera networks: A review,” 2016 Interna-
Method kNN tional Conference on Audio, Language and Image Processing (ICALIP),
SVM Shanghai, 2016, pp. 574-580. doi: 10.1109/ICALIP.2016.7846643
k=3 k=5 k=7
[7] LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature,
LBP
521(7553), 436-444. https://doi.org/10.1038/nature14539
r=1 0.96 ± 0.01 0.92 ± 0.01 0.94 ± 0.01 0.95 ± 0.01
[8] T. Ojala, M. Pietikainen and T. Maenpaa, “Multiresolution gray-scale
r=2 0.93 ± 0.01 0.93 ± 0.01 0.94 ± 0.01 0.94 ± 0.01
and rotation invariant texture classification with local binary patterns”,
r=3 0.92 ± 0.01 0.92 ± 0.01 0.94 ± 0.01 0.95 ± 0.01 IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.
r=4 0.92 ± 0.01 0.92 ± 0.01 0.94 ± 0.01 0.95 ± 0.01 24, no. 7, pp. 971-987, 2002.
r=5 0.92 ± 0.02 0.93 ± 0.01 0.95 ± 0.01 0.95 ± 0.01 [9] L. Hu, M. Huang, S. Ke and C. Tsai, “The distance function effect on k-
HOG 0.59 ± 0.16 0.42 ± 0.02 0.41 ± 0.02 0.41 ± 0.02 nearest neighbor classification for medical datasets”, SpringerPlus, vol.
CNNs 0.03 ± 0.01 5, no. 1, 2016.
[10] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn and A.
TABLE III Zisserman, “The PASCAL Visual Object Classes Challenge 2012
N UMBER OF FEATURES AND NUMBER OF CONVOLUTIONS IN THE NEURAL (VOC2012) Results”, Host.robots.ox.ac.uk, 2012. [Online]. Available:
NETWORK http://host.robots.ox.ac.uk/pascal/VOC/voc2012/. [Accessed: 10- OCT-
2018].
[11] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z.
Feature Nf Nc Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg and L. Fei-Fei,
HOG 1,764 - “ImageNet Large Scale Visual Recognition Challenge”, International
LBP Journal of Computer Vision, vol. 115, no. 3, pp. 211-252, 2015.
r=1 10 - [12] Li, D., et al., Integrating a statistical background-foreground extraction
r=2 18 - algorithm and SVM classifier for pedestrian detection and tracking,
r=3 26 - Integrated Computer-Aided Engineering, 2013, 20, (3), pp. 201-216.
r=4 34 - [13] N. Dalal and B. Triggs, “Histograms of oriented gradients for human
r=5 42 - recognition,” Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1-8,
Jun. 2005.
CNNs - 88,473,600

TABLE IV
C OMPLEXITY OF SVM, K NN AND CNN S C LASSIFIER

Classifier Complexity
SVM O(Nf )
kNN O(Ns ∗ Nf + Ns ∗ k)
CNNs O(Nc )

For the complexity, the CNNs is the highest complexity of

classifiers depending the number of convolutions in this test.
In addition the kNN is the high complexity of classifiers due
to consuming time to calculate distance of all training dataset.

VI. F UTURE WORK

Studying the foot recognition in lateral view relates the knee
rehabilitation program, and a modification of deep learning
model is able to gain high accuracy while it has lower
complexity.

ACKNOWLEDGMENT
This project is supported by the British Council Newton
Fund: Researcher Links Travel Grant 2015/2016.