Professional Documents
Culture Documents
Textbook Intelligent Computing Systems Carlos Brito Loeza Ebook All Chapter PDF
Textbook Intelligent Computing Systems Carlos Brito Loeza Ebook All Chapter PDF
Textbook Intelligent Computing Systems Carlos Brito Loeza Ebook All Chapter PDF
Brito-Loeza
Visit to download the full and correct content document:
https://textbookfull.com/product/intelligent-computing-systems-carlos-brito-loeza/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...
https://textbookfull.com/product/advanced-intelligent-systems-
for-sustainable-development-ai2sd-2018-volume-5-advanced-
intelligent-systems-for-computing-sciences-mostafa-ezziyyani/
https://textbookfull.com/product/electronic-systems-and-
intelligent-computing-proceedings-of-esic-2020-pradeep-kumar-
mallick/
https://textbookfull.com/product/soft-computing-theories-and-
applications-proceedings-of-socta-2019-advances-in-intelligent-
systems-and-computing-millie-pant-editor/
https://textbookfull.com/product/intelligent-computing-
optimization-pandian-vasant/
Recent Advances of Hybrid Intelligent Systems Based on
Soft Computing 1st Edition Patricia Melin
https://textbookfull.com/product/recent-advances-of-hybrid-
intelligent-systems-based-on-soft-computing-1st-edition-patricia-
melin/
https://textbookfull.com/product/smart-and-sustainable-
intelligent-systems-sustainable-computing-and-optimization-1st-
edition-namita-gupta-editor/
https://textbookfull.com/product/intelligent-decision-support-
systems-for-sustainable-computing-paradigms-and-applications-1st-
edition-arun-kumar-sangaiah/
https://textbookfull.com/product/law-as-passion-systems-theory-
and-constitutional-theory-in-peripheral-modernity-miguel-
nogueira-de-brito/
Carlos Brito-Loeza
Arturo Espinosa-Romero (Eds.)
Intelligent Computing
Systems
Second International Symposium, ISICS 2018
Merida, Mexico, March 21–23, 2018
Proceedings
123
Communications
in Computer and Information Science 820
Commenced Publication in 2007
Founding and Former Series Editors:
Alfredo Cuzzocrea, Xiaoyong Du, Orhun Kara, Ting Liu, Dominik Ślęzak,
and Xiaokang Yang
Editorial Board
Simone Diniz Junqueira Barbosa
Pontifical Catholic University of Rio de Janeiro (PUC-Rio),
Rio de Janeiro, Brazil
Phoebe Chen
La Trobe University, Melbourne, Australia
Joaquim Filipe
Polytechnic Institute of Setúbal, Setúbal, Portugal
Igor Kotenko
St. Petersburg Institute for Informatics and Automation of the Russian
Academy of Sciences, St. Petersburg, Russia
Krishna M. Sivalingam
Indian Institute of Technology Madras, Chennai, India
Takashi Washio
Osaka University, Osaka, Japan
Junsong Yuan
Nanyang Technological University, Singapore, Singapore
Lizhu Zhou
Tsinghua University, Beijing, China
More information about this series at http://www.springer.com/series/7899
Carlos Brito-Loeza Arturo Espinosa-Romero (Eds.)
•
Intelligent Computing
Systems
Second International Symposium, ISICS 2018
Merida, Mexico, March 21–23, 2018
Proceedings
123
Editors
Carlos Brito-Loeza Arturo Espinosa-Romero
Autonomous University of Yucatán Autonomous University of Yucatán
Merida Merida
Mexico Mexico
Recent technological advances such as the increase in computing power and wider and
faster communication networks have made computing systems ubiquitous. Intelligent
systems are computing systems capable of simulating the intelligent behavior unique to
living beings, and researchers in this field take advantage of this technological growth
to propose new algorithms and applications.
Intelligent systems are important, since the application of this field of study takes on
a main role in many disciplines, where the development of automatic systems capable
of making decisions on complex environments or assisting humans in tasks where the
unique human perceptual capabilities are difficult to replace.
In this volume you will find many novel examples of research in intelligent systems:
algorithms capable of learning from examples to recognize and classify objects in
images, novel sensors and the computational methods needed to process and interpret
their readings, neural network-based methods to predict time series and control and
manage energy generation systems, variational image processing methods systems to
segment and denoise images, as well as the processing of multimodal signals to use in
the context of human–robot interaction.
This book contains the written contributions of the Second International Symposium
on Intelligent Computing Systems (ISICS) that was held in Merida (Mexico), during
March 21–23, 2018. To further increase the body of knowledge in this specific area of
computer science was the aim of the ISICS 2018, by providing a forum in which to
exchange ideas and discuss state-of-the-art results. ISICS 2018 was committed to the
promotion, preservation, and collaboration of research and practice, focusing on the
fields of artificial intelligence, computer vision, and image processing.
We received 28 submissions from seven countries around the world. Each sub-
mission was evaluated by at least three members of the Program Committee and
external reviewers. Based on these reviews, 12 papers were selected for long oral
presentation. In addition to the contributed papers, four keynote speaker presentations
were included in the conference program.
We want to thank the authors for their contributions, the scientific Program Com-
mittee members for their reviews, and especially our invited speakers, Prof. Andrew
Barto (University of Massachusetts, USA), Prof. Leo Joskowicz (Hebrew University of
Jerusalem, Israel), Prof. Reinhard Klette (Auckland University of Technology, New
Zealand), Prof. Jesús Savage Carmona, (Bio-Robotics Laboratory, Mexico), and
Prof. Sajjad Mohsin (University of Islamabad, Pakistan). We are very grateful to the
Universidad Nacional Autónoma de México (UNAM), the Centro de Investigaciones
Matemáticas (CIMAT), and the Universidad Autónoma de Yucatán (UADY) for their
support in the organization of the ISICS 2018.
Program Committee
Bassam Ali Universidad Autónoma de Yucatán, Mexico
Noor Badshah University of Engineering and Technology, Pakistan
Carlos Brito-Loeza Universidad Autónoma de Yucatán, Mexico
Morgado Dias Universidade da Madeira, Portugal
Arturo Espinosa-Romero Universidad Autónoma de Yucatán, Mexico
Jorge Gomez-Montalvo Universidad Autónoma de Yucatán, Mexico
Benjamín Gutierrez-Becker Technische Universität München, Germany
Jean-Bernard Hayet Centro de Investigación en Matemáticas A. C., Mexico
Francisco Javier Hernández Centro de Investigación en Matemáticas A. C., Mexico
López
Nidiyare Hevia-Montiel IIMAS-UNAM, Mexico
Angel Kuri-Morales Instituto Tecnológico Autónomo de México, Mexico
Ricardo Legarda-Saenz Universidad Autónoma de Yucatán, Mexico
Stacey Levine Duquesne University, USA
Eduardo Lleida Universidad de Zaragoza, Spain
Elena Loli Piccolomini University of Bologna, Italy
José Luis López Martínez Universidad Autónoma de Yucatán, Mexico
Salvador Mandujano Google Inc., USA
Anabel Martin-Gonzalez Universidad Autónoma de Yucatán, Mexico
Rolando Medellín University of Dundee, UK
Raúl Monroy Borja ITESM-CEM, Mexico
Francisco Moo-Mena Universidad Autónoma de Yucatán, Mexico
Nicolás Navarro-Guerrero Aarhus University, Denmark
Luis A. Pineda Cortés IIMAS-UNAM, Mexico
Lavdie Rada Bahçesehir University, Turkey
Alonso Centro de Investigación en Matemáticas A. C., Mexico
Ramirez-Manzanares
Eduardo Universidad Autónoma Metropolitana, Mexico
Rodriguez-Martinez
Israel Sánchez-Domínguez IIMAS-UNAM, Mexico
Asad Safi COMSATS Institute of Information Technology,
Pakistan
Sai Deep Tetali Google Inc., USA
Victor Uc-Cetina Universidad Autónoma de Yucatán, Mexico
VIII Organization
Organizing Committee
Anabel Martin-Gonzalez Universidad Autónoma de Yucatán, Mexico
Víctor Uc-Cetina Universidad Autónoma de Yucatán, Mexico
Carlos Brito-Loeza Universidad Autónoma de Yucatán, Mexico
Bassam Ali Universidad Autónoma de Yucatán, Mexico
Arturo Espinosa-Romero Universidad Autónoma de Yucatán, Mexico
Ricardo Legarda-Saenz Universidad Autónoma de Yucatán, Mexico
Nidiyare Hevia-Montiel Instituto de Investigaciones en Matemáticas Aplicadas
y en Sistemas - UNAM, Mexico
Israel Sánchez-Domínguez Instituto de Investigaciones en Matemáticas Aplicadas
y en Sistemas - UNAM, Mexico
Eric Molino-Minero-Re Instituto de Investigaciones en Matemáticas Aplicadas
y en Sistemas - UNAM, Mexico
Francisco J. Hernández Centro de Investigación en Matemáticas A. C., Mexico
López
Contents
1 Introduction
One relevant issue in computer science is to try to interpret an image for a
specific purpose. This research focuses on recognizing objects in an image with
different occlusion percentages. Right now, computers have the computational
power to achieve this objective with an acceptable running time.
Singh [1] defines video tracking as the process to detect one moving object
throughout a video sequence, and a fundamental task of this tracking is to rec-
ognize the object in every frame. An object of interest can be: an animal, a
machine, a person, etc. A system to detect the existence of objects in a frame
was developed. The dataset consisted of video sequences of people, and this gave
different challenges related to shape transformation (because of object move-
ments) and occlusion (when the object is partially occluded by other objects).
Figure 1 shows an example of occlusion.
The human brain has a neocortex. This element is in charge of visual pattern
recognition and many other cognitive processes. Hierarchical temporal memory
(HTM) is a technology developed to try to mimic the neocortex. It is very similar
to artificial neural network (ANN), but with some architectural and conceptual
c Springer International Publishing AG, part of Springer Nature 2018
C. Brito-Loeza and A. Espinosa-Romero (Eds.): ISICS 2018, CCIS 820, pp. 1–14, 2018.
https://doi.org/10.1007/978-3-319-76261-6_1
2 F. Fallas-Moya and F. Torres-Rojas
differences. For example, the cells1 are connected to different cells all over the
region (see Fig. 2). These connections or synapses can change during the training
period. Even more, they can change during classification. It depends on the
input data. The HTM architecture have cells that are arranged into regions, and
regions that can form a complex hierarchy.
This HTM technology was developed by Hawkins [2] and it is closely related
to the biological structure of the neocortex that has six layers of regions. This
explains why HTM uses the concepts of regions, cells, and hierarchies. The
regions can be constructed using different numbers of cells. An important aspect
is that the connections inside regions can change over time.
The idea behind hierarchies is to learn complex patterns. This is similar to the
idea behind convolutional neural networks, as explained by Fan et al. [3]. There,
every layer tries to detect gradient features to have an accurate classification
process. In the case of HTM, every defined level2 will be learning different degrees
of an image, to get to the final layer. The idea is to learn complex patterns.
See Fig. 3.
As well as in ANN, HTM network has a specific region for the input data. The
outcome of this first layer is passed to the following region to be trained, and
the process continues for every region until the last one.
HTM uses the concept of sparse distributed representations (SDR). It is a
technique to represent different patterns, where a small number of cells are picked
to be active.
1
In the HTM terminology, cells are synonyms of neurons.
2
The terms level and region will be used interchangeably.
Object Recognition Using Hierarchical Temporal Memory 3
Fig. 2. An HTM region with columns of cells, the cells are connected to different cells.
Also, the concept of sparse distributed representations (SDR), the patterns are just a
few and distributed cells along the region.
HTM can receive an enormous input. However, every layer needs just a few
cells to represent it. Higher layers apply the same concept until the final layer is
reached. This process is not uniform or deterministic. This explains why different
cells can be picked. In Fig. 2 shows that it is not mandatory for these cells to be
close to each other. To summarize, only a few cells are need and these cells can
be distributed along the region.
As Schlag [21] explains, the way HTM learns patterns is in a statistical
manner, very similar to Bayesian Networks; however, they differ in the hierarchy
construction and the usage of time. First, it converts raw data into proper HTM
input, using some decoders (libraries developed for HTM). Second, from the data
it looks for activations which occur together (spatial patterns). It then searches
sequences of these patterns over time (temporal patterns). So, there is not a back
propagation step, it does some kind of clustering classification. These learned
patterns are used to perform inference on new inputs. In HTM the concept of
time is important, the order in which the patterns enter the architecture impacts
the connection construction in the network.
Just like a biological neuron, an HTM cell has dendrites, proximal and a distal
dendrites. The dendrites contain many synapses in order to receive signals. The
proximal dendrites receive a feed-forward input from regions in a lower layer. The
distal receive their input from neighboring cells which belong to the same HTM
region. Every cell is made up of a number of synapses, which can be potential
or permanent. During the training process these synapses can change due to the
classification result. Every cell can be in a different state: predictive, active and
fully active. These states will defined the synapses. This cell flexibility to easily
change its connections will help to make a fast training step. According to Schlag
[21] (2016), “This increase and decrease in permanence is an important aspect
of the Hebbian3 learning ability of HTMs”.
3
Hebbian Theory defined by Gerstner [22].
4 F. Fallas-Moya and F. Torres-Rojas
Fig. 3. An HTM hierarchy: in this example, the hierarchy has four levels of regions.
To have a full HTM implementation one can use the online prediction
framework (OPF). It provides all components needed to create an architecture.
A short description of these components follows.
4
“The accuracy is the proportion of true results (both true positives and true
negatives) among the total number of cases examined” [5].
Object Recognition Using Hierarchical Temporal Memory 5
Fig. 5. A point detected by the SIFT algorithm and the descriptor generated around
the point. (a) A grid generated around the interesting point. Its orientation is according
to the gradient direction. (b) Every cell of the grid is represented by an 8 bin histogram.
(c) Get all histograms from the grid. (d) All histograms are concatenated to have a
vector. Image from [7].
Fig. 7. Dense SIFT: there are 12 static points and they have a predetermined radius.
Object Recognition Using Hierarchical Temporal Memory 7
3 Experiments
Design of Experiments (DoE) was used as a statistical model with the R [17]
language. It tries to answer if predefined factors have an influence over a study
response variable, and, if there is a significant influence, the important issue
is to quantify that influence [18]. The ANalysis Of VAriance (ANOVA) is an
instrument of DoE. It measures the variance over the factor values against the
variance of other predefined factors. OSR is the response variable and it has four
factors. Table 1 shows the factors: machine learning technique, training-set size,
the percentage of occlusion and the complexity of the scenario. On the one hand,
a simple scenario is composed of white background and some objects. On the
other hand, a complex scenario has more objects, light contrast, and different
backgrounds.
As shown in Table 1, there are 4 techniques, 3 training-sets, 4 occlusion per-
centages, and 2 scenarios, for a total of 96 combinations. 4 replicas were picked,
for a number of 384 runs (4 × 3 × 4 × 2 = 96 × 4 = 384). It is important to
5
Numenta is the enterprise behind of the HTM technology.
8 F. Fallas-Moya and F. Torres-Rojas
highlight that every run is a sequence of frames. For instance, the training phase
has 38 + 176 + 474 = 688 images for training. With 2 scenarios, that is 1376
images. Also there are 4 replicas, for a total of 5504 images to train our algo-
rithms. Regarding to the testing phase, there are 96 runs of 360 frames, for a
total of 34560. Since there are 4 replicas, the total number of classification tasks
is 138240.
Selected factors
Techniques Training sets Occlusion(%) Scenarios
Levels HTM (Hard) 38 75 Simple
HTM (Soft) 176 50 Complex
ANN 474 25
SVM 0
1. There are 3 video classes: two different persons moving around, one class per
person. And the background with no motion, the final class. These persons
were moving with different percentages of occlusion, this percentage does not
change throughout the video. See Table 1 for the occlusion percentages.
2. One replica has 4 occlusion percentages, 2 different scenarios (completely dif-
ferent videos), and 3 different classes (8 videos per class). That is, 24 different
videos. Since there are 4 replicas, there were 96 videos.
3. Every video has 36 s. Every second has 15 frames, that is, a total number of
540 frames per video. In Table 1 the number of frames taken for training the
algorithms can be different (38, 176 and 474).
DoE needs to accomplish two assumptions. First, the residuals are normally dis-
tributed. Second, they are independent with a constant variance, as explained by
Anderson and Whitcomb [19]. Fortunately, these assumptions were accomplished
and no data transformation was needed. Table 2 shows the output of an ANOVA
(Acronym explanation: Df: Degrees of Freedom, Sum Sq: Sum of Squares, Mean
Sq: Mean of Squares, F val: F values, Pr(>F): Probability values).
Box plots [20] were used to analyzed some aspects. On the one hand, Fig. 8 shows
that ANN has the lowest quartile6 and the mean with the lowest value, which
means that ANN fails more than the others.
On the other hand, the HTM Hard algorithm has its 50% of data enclosed in
the thinnest box, also the upper and lower quartile boundaries have the smallest
range. The mean is near to 0.5 and it has a lot of outliers, this situation is normal
because the upper and lower quartiles form a small range. The lower quartile
shows that this algorithm had fewer failures than the others. This algorithm had
a stable performance.
6
Quartile: they are the values that divide a list of numbers into quarters.
Object Recognition Using Hierarchical Temporal Memory 11
In Fig. 9, there is the box plot of the factor Occlusion against OSR. It can be
seen that all algorithms fail if the occlusion value is incremented, on 75% the
lower boundary of the box is almost 0.0. This is a expected behavior, because
the more the object is occluded, the more difficult to hit.
It can be appreciated that the mean value for 25% and 50% have almost
the same value, which indicates it is easier for an algorithm to hit. This can be
explained because people have more characteristics from the hips towards the
head. Also, there is another interesting aspect: the values for 25%, 50%, and
75%, have their upper box boundary close to 0.5 of accuracy, but, the box of 0%
has it close to 0.6, which indicates that with 0% the algorithms tend to fail less.
However, its mean has the lower value. This situation tells that under a specific
condition, it produces more fails with 0%. An explanation is that with 0% the
objects (people) had more freedom to move, and did not have to be behind of
an object to catch the occlusion.
The final aspect to analyze is one of the most interesting ones. The interaction
between two factors: technique and occlusion. Whitcomb and Anderson give the
definition of interaction [19]: “Interactions happen when the effect of one of the
factors depends on another factor”. Figure 10 shows many relevant situations.
The first one is that all algorithms tend to decrease. It is obvious because as the
occlusion raises its value, it is more difficult for all algorithms to hit.
Another aspect to note is that HTM Soft and ANN do not have an acceptable
performance, unlike SVM and HTM Hard. It was mentioned that with 0% of
occlusion, there is a huge challenge: the targets had more freedom to move all
over the scene. It is considered that ANN and HTM Soft do not have sensibility
12 F. Fallas-Moya and F. Torres-Rojas
under this condition (target movements). Hence, the good performance of SVM
and HTM Hard shows that these algorithms are better implementations for real
world systems.
Fig. 10. Interaction of two significant elements (factors): occlusion and technique.
The third aspect analyzed is that SVM has a remarkable behavior on 0%.
A stable performance on 25% and 50%, it is similar to HTM Hard. However,
it fails on the 75% value. This situation indicates that it could not classify well
when the object was almost totally occluded.
Lastly, there is the HTM Hard implementation. It has an excellent perfor-
mance on 0%, a good sensibility to target movements. Also, good results on 25%
and 50%. As seen in Fig. 10, HTM Hard had the best performance on the value
of 75% of occlusion. As a result, this is the best algorithm for the occlusion
challenge.
5 Conclusions
A modified version of an OPF algorithm was used (HTM Hard). Based on the
statistical results, one conclusion is that this algorithm is better to recognize
objects than the others. But, it was experienced an unexpected situation: it
had the largest running time. It lasts 60 times more comparing with the other
implementations. It is not feasible to implement with streaming data.
Another aspect is that SVM had a good performance. During tests, it was
the fastest implementation. However, it had a poor performance on 75%. Also,
there are the ANN and the HTM Soft performance, which had the worst
results. As a result, their implementation is not recommended with the proposed
architectures.
All algorithms used SIFT to classify. This technique provides enough informa-
tion to feed a machine learning algorithm. For classifying purposes, Dense SIFT
Object Recognition Using Hierarchical Temporal Memory 13
was used to have a fixed amount of data per frame. Only 12 SIFT points were
used. However, with a limited amount of data the results were acceptable and
satisfactory.
Finally, it was discovered that the implementations of HTM Hard and SVM
are more sensitive to object movements. They have an acceptable performance
with a small quantity of data. HTM Hard is the recommended version for this
kind of problem because in real world systems the objects are moving all over
the scene.
6 Future Work
The next step is to test different combinations of algorithms. For instance, recur-
rent neural networks (RNN), convolutional neural networks (CNN), bayesian
networks, etc. It would be interesting to use CNN, due to its similarity to HTM
(many layers, feature patterns reduction, etc.).
Another interesting aspect is to test different HTM combinations. For exam-
ple, to implement the usage of a temporal pooler with a classifier (in this research
kNN was implemented) such as SVM, ANN, RNN, CNN, etc. Furthermore, high-
quality videos can be used and get more information.
Also, to use more points (Dense SIFT values). It can be treated as an hyper-
parameter and adjust it to get better results.
Finally, another feature extraction technique can be used as well as different
preprocessing techniques.
Acknowledgements. We would like to thank our colleagues from the Happy Few
Research Group for their support during the development of this research. In addition,
we thank the PARMA Group for its guidance and support publishing this research.
References
1. Jalal, A.S., Singh, V.: Visual object tracking: state of art. Int. J. Comput. Inform.
3, 227–247 (2011). Ljubljana, Slovenia
2. Hawkins, J., Blakeslee, S.: On Intelligence. St. Martin’s Griffin, USA (2005)
3. Fan, J., Xu, W., Gong, Y.: Convolutional neural networks: human traffic. IEEE
Trans. About Neural Netw., 1610–1623 (2010). https://doi.org/10.1109/TNN.
2010.2066286
4. Hawkins, J., Ahmad, S., Dubinsky, D.: Cortical Learning Algorithms and HTM.
Numenta Inc., California (2011)
5. Metz, C.E.: Principles of ROC Analysis. University of Chicago and the Franklin
McLean Memorial Research Institute, Chicago, USA, pp. 283–298 (1978). 0001-
2998/78/0804-0003S02.00/0
6. Lowe, D.G.: Object recognition based on local scale invariant features. In: Com-
puter Vision: International Conference, pp. 1150–1157. IEEE, Canada (1999).
https://doi.org/10.1109/ICCV.1999.790410
7. Solem, J.E.: Python: Basics on Computer Vision. O’Reilly Medias, Sebastopol
(2012)
14 F. Fallas-Moya and F. Torres-Rojas
8. Hassner, T., Mayzels, V., Zelnik-Manor, L.: About SIFT and its scale. In: IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), Rhode Island,
USA (2012)
9. Han, B., Li, D., Ji, J.: DSIFT Algorithm for People Detection. Stanford University,
California (2011)
10. Costa, A.: A MNIST Classifier Using OPF (2016). http://github.com/allanino/
nupic-classifier-mnist
11. Altman, N.S.: Introduction to kernel and nearest-neighbor nonparametric regres-
sion. Am. Stat. 46, 175–185 (1992)
12. Schaul, T., Bayer, J., Wierstra, D., Sun, Y., Felder, M., Sehnke, F., Rückstieß, T.,
Schmidhuber, J.: The PyBrain library. J. Mach. Learn. Res. 11, 743–746 (2011)
13. Bishop, C.M.: Pattern Recognition Using Neural Networks. Oxford University
Press, Oxford (1995)
14. Blum, A.: Neural Networks (C++). Wiley, New York (1992)
15. Numenta: Nupic: managing vision tasks (2017). http://github.com/numenta
16. Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines (2017).
http://www.csie.ntu.edu.tw/cjlin/libsvm/
17. Ihaka, R.: History of R: Past and Future. The University of Auckland, Auckland
(1998)
18. Ferré, J., Rius, X.: Introduction to the Statistical Design of Experiments. Univer-
sitat Rovira i Virgili, Tarragona (2001)
19. Anderson, M., Whitcomb, P.: Design of Experiments: A Simplified Approach.
Taylor and Francis Group, Boca Raton (2007)
20. Massart, D.L., Smeyers-Verbeke, J., Capron, X., Schlesier, K.: Means of Box Plots:
Visual Presentation of Data. Vrije Universiteit Brussel, Brussel, Belgium (2005)
21. Schlag, I.: On Hierarchical Temporal Memory (2016). http://ischlag.github.io/
2016/04/25/on-hierarchical-temporal-memory
22. Gerstner, W.: Hebbian learning and plasticity. From Neuron to Cognition via Com-
putational Neuroscience, Chap. 9. MIT Press, Cambridge (2011)
Tree-Structured CNN for the Classification
of Surgical Instruments
1 Introduction
The use of convolutional neural networks (CNN) [1] for the classification of objects in
different types of applications has been made possible by the different architectures that
can be proposed for each network, the use of certain types of layers, and to the
dimensions defined for each of their filters, factors that have allowed CNNs with
different depths to acquire the ability to extract and learn specific characteristics of
different training categories, as shown in [1].
The implementation of this type of networks has allowed to develop numerous
applications of recognition that cover diverse subjects like the detection of faces [2, 3],
the detection and classification of positions of the hands [4], text classification in
written documents [5], the classification of traffic signals [6], and even applications for
speech recognition [7]. Also, they have been extended to video object recognition [8, 9]
and to handwriting recognition [10].
However, all of them present a particular characteristic and it is the classification of
categories with numerous differences between them, where diverse characteristics like the
background, the color, or certain proportions of the object allow each category to be
classified by a single convolutional neural network with error rates lower than 5%, as
© Springer International Publishing AG, part of Springer Nature 2018
C. Brito-Loeza and A. Espinosa-Romero (Eds.): ISICS 2018, CCIS 820, pp. 15–30, 2018.
https://doi.org/10.1007/978-3-319-76261-6_2
16 P. C. Useche Murillo et al.
shown in [6], while the problem that is sought to solve in this article is focused on the
classification of surgical instruments through CNNs, where only small details of each tool
are the ones that allow to differentiate one from the others, forcing the network to acquire
the ability to detect small changes in the instruments in order to classify them properly.
A novel method of classification by means of tree-structured convolutional neural
networks is presented in this work, focused on the classification of 10 surgical
instruments, whose main characteristics for the classification of each category cannot
be extracted with a single CNN architecture, as it is demonstrated during the devel-
opment of the application, but with different networks focused on the main charac-
teristics of each category.
The following article is divided into four main sections, where the first one focuses
on the “Classification Tree” within which the classification method with tree-structured
CNN is explained, and the problem to be faced is described. In the next section the tests
made to obtain the final classification structure are presented. In the third section the
behavior of the different trained networks is analyzed. Finally, in the fourth section, the
conclusions reached are presented.
2 Classification Tree
2.1 Database
For the development of the surgical instrument classification algorithm, 10 tools were
selected, which are presented in Fig. 1, and a tolerance of ± 15 degrees of inclination
with respect to the central vertical axis of each tool was set for the taking of the training
and test databases, in order to give a degree of tolerance to the classification algorithm
that allows it to recognize the instruments even if there is a slight inclination of the
element.
With this tolerance limit, that eliminates the need for the network to learn the same
characteristic of each instrument under different rotation situations, allowing to focus the
classification on the shape of the element and not on its inclination, 300 training color
images and 50 test images with size of 128 128 pixels were taken for each instrument,
different backgrounds were used for the capture of the database, whose shades vary
between blue and gray, as shown in Fig. 1, and the lighting conditions were varied to
capture the changes of tonality that appear on the material of each instrument.
The initial classification of the tree consists of a main category called “Scissors”
which contains all the scissors-type surgical instruments, and a secondary category
called “Others” containing more varied surgical instruments, from cutting elements
such as “Scalpel Handle 4” to apprehension tools such as “Splinter Forceps”. The
categories “Flat” and “Thin” are composed of tools with mainly flat bodies, in the case
of “Flat”, or thin instruments, in the case of “Thin”, and the category “Sticks” contains
tools with shapes similar to rods. On the other hand, the category “Open” contains
those scissors whose eye rings are separated from each other, while the “Closed”
category contains the scissors that have the eye rings together.
An example of the two categories set for the first branch of the tree is shown in
Fig. 4, where Fig. 4a contains the instruments of the “Others” category and Fig. 4b, the
instruments of the “Scissors “category. When comparing both categories it is possible
to appreciate that the great difference between the two lies in the existence or not of eye
rings, a factor that becomes a characteristic of the category “Scissors” that cannot be
found in the “Others” category. On the other hand, within each category are great
similarities, such as a thin body for the category “Others”, and the presence of eye rings
for scissors, with small differences in the points that characterize each instrument.
a. b.
Fig. 4. Categories of the first branch, (a) Others and (b) Scissors.
The tree-structured object classification works as follows: the image of the surgical
instrument to be classified is taken as an input and, by means of a CNN, the instrument
is placed in one of the two categories of the first branch of the tree, either “Others” or
“Scissors”. Subsequently, the image enters another CNN that classifies it into one of the
next two or three categories in which the tree is divided, changing the image of branch
to a final classification, where it is named after any of the surgical instrument cate-
gories. Figure 5 shows two examples of successful path classification for the case of
Tree 5.
Tree-Structured CNN for the Classification of Surgical Instruments 19
a. b.
Fig. 5. Operation of the classification tree for (a) “Dissecting Needle Curved” and (b) “Operating
Scissors Sharp-Blunt”.
All that night, Rosabelle could hardly get any sleep for thinking of
that strange fragment of conversation between aunt and nephew
which she had overheard in the afternoon. It was with great difficulty
she kept herself from telling her lover, but she wanted to meditate
well over the matter before confiding it to anyone.
She felt that if anybody ought to know it was her uncle; in fact, was
it not almost her duty to tell him? On the other hand, she had a
considerable affection for her aunt, and shrank from getting her into
trouble. The relations between the two had been for years very
close, and Mrs. Morrice had always shown her great kindness. Since
the introduction of Archie Brookes there had been a certain
diminution of affection on her aunt’s part, the new-comer had
considerably ousted her.
But Rosabelle was a very fair-minded girl, and she did not resent
that. There was no blood-tie between her and Mrs. Morrice. The
husband and wife got on very comfortably together, but it was easy
to see it was a very placid union, that their marriage had not been
prompted by any great depth of feeling on either side, and there
were no children to draw them closer together.
It was only natural, therefore, that she should welcome this young
man so closely related to her, the son of a, probably, deeply loved
sister. On him she could expend that wealth of maternal feeling
which, so far, had never been called into existence, but which
resides in the heart of every good, womanly woman. Small wonder
that Rosabelle, to a considerable extent, should have receded into
the background. Had she been in her aunt’s position, the same thing
would most probably have occurred.
She had not told it to Dick, she shrank from telling it to her uncle;
for the present she was disposed to keep it to herself. Under
ordinary circumstances it would have seemed to her a tragedy of the
first importance, that this good-looking young nephew was preying
upon his aunt’s weakness or fondness for him, to such an extent that
she had declared herself to be half ruined. But the greater tragedy of
her lover lying under a horrible stigma absorbed all minor ones; she
saw them, as it were, only in perspective.
The two things could not be in any way related, she felt pretty
sure. And yet, as she lay in the darkness, pondering and pondering,
suddenly there flashed across her the thought, coming almost with
the force of an inspiration, that the detective ought to be told. He had
especially impressed upon her at the beginning of their business
connection, that she was to report to him any uncommon happening
in the Morrice household, irrespective of whether or not it seemed to
her of importance. What was troubling her now was certainly not a
common or trivial thing.
To think was to act. If the knowledge were of benefit to him, he
would use it as he thought fit—and after all, the greatest concern of
her life at the moment was the restoration of her lover to his former
honourable place in the regard of those who knew the real reason of
his exile from her uncle’s house. And, if the knowledge was useless
to him, she was quite sure of the man; he would never divulge it
unless she gave him permission.
She was round at Lane’s office early the next morning. Mrs.
Morrice had not appeared at breakfast, but Rosabelle noticed at
dinner the night before, and afterwards when they were together in
the drawing-room, that her manner had appeared anxious and
preoccupied.
It could not be said of Lane at any time that he was a man whom
you could read like an open book, but she was sure the information
made a great impression on him. As was his custom after an
important communication had been made to him, he sat silent for
some little time.
“And you have said nothing about this to your uncle, or Mr.
Croxton?” he asked at length. “I am so far the only person to whom
you have revealed it?”
“Because I thought you ought to know,” answered the girl frankly.
“My uncle ought to be told, I feel that, but I shrink from telling him; it
might create an irreparable breach between them, and I should be
very grieved to be the cause of it. I think, or rather I am sure, that my
aunt has not the same affection for me that she had before the
arrival of Archie, but that is only natural, and not a thing to be
resented. She has always shown me unvarying kindness, and made
my life in Deanery Street very happy. And you know, Mr. Lane, it is
not every woman who would have done that in the circumstances.
For my dear uncle has been always very demonstrative in his love
for me, and it might have aroused the jealousy of a great many
wives.”
What a sweet-natured, tolerant-minded girl she was, her listener
thought. Then he said decisively: “Certainly Mr. Morrice ought to
know. You would object to my telling him, I suppose?”
Yes, Rosabelle shrank from that. “It would come to the same thing,
would it not? He would want to know where you got your information
from, and you would have to tell him. I might as well do it myself.
Besides, I expect he would be very angry with me for having told you
at all. He is a very proud man in certain things.”
Yes, there was a good deal of shrewdness in that remark. He
might be able to get through it without bringing her in, for he was a
man of infinite resources, but although by no means scrupulous
when driven to use subterfuge, he did not employ tortuous methods
if it was possible to avoid them.
“Tell me, Miss Sheldon, what do you know of your aunt’s affairs?
Has she money of her own?”
“I should say very little. I have more than once heard her jokingly
allude to her ‘paltry income.’ But I know my uncle makes her a very
handsome allowance, although I don’t know the precise amount. And
he is always making her presents of valuable and expensive
jewellery.”
It was evident, by his serious look, that he was thinking very
deeply. “That allowance, of course, he makes her for her own
personal needs, and to maintain her proper position as the wife of a
wealthy man. If he knew that she was diverting any, or a
considerable portion, of this money to supplying this young man’s
extravagant needs, you are of opinion he would be greatly
incensed.”
“I am sure of it. He is peculiar in many ways, he abhors strongly
anything in the nature of deceit. If she came to him openly and said
she was going to give Archie money, he might remonstrate with her,
actually forbid her, or take the view that it was her own and she could
do what she wished with it. But he would never forgive her doing it
clandestinely, I mean in large sums. He would think it a betrayal of
the trust he had reposed in her.”
Lane’s brain was still working on the problem presented to him.
Morrice, according to Rosabelle, made his wife a handsome
allowance. That might be taken for granted. He had a wide
reputation for generosity, and for pride’s sake he would be especially
lavish to his wife. But what is a very ample allowance for a woman
does not go far when constant drains are made upon it by a young
man who lived in the style that Simmons had described when
speaking of Archie Brookes.
“Have you noticed any diminution in Mrs. Morrice’s expenditure on
herself since the arrival of this nephew on the scene, Miss Sheldon?”
Rosabelle gave her evidence very reluctantly, but it was right Lane
should know the whole circumstances. From the very beginning, her
aunt had appeared to curtail her personal expenditure. For the last
twelve months, her economy in her own direction had been much
more marked. It pointed to the fact that Archie had been draining her
considerably.
Lane thought more than considerably. That poignant exclamation
that she had been half ruined suggested a good deal to him.
“I am going to ask you a rather peculiar question, Miss Sheldon.
Are Mr. and Mrs. Morrice what might be described as a very devoted
couple? You know what I mean, are they wrapt up in each other as
some people are at their time of life when they have married solely
for love?”
It was a peculiar question certainly; to Rosabelle it seemed rather
an irrelevant one. But she was sure the detective never asked
irrelevant questions. He had some good reason for putting this one,
without doubt, and she would give him a perfectly candid answer.
“Why, no, it certainly would not be accurate to describe them as
that. I am certain they have a great respect for each other, and a
very quiet and placid affection. He is the soul of generosity and
courtesy to her; she respects his wishes in everything. You see, he
was devotedly in love with Mr. Croxton’s mother; he kept unmarried
for years for her sake. A man cannot love twice like that, can he, Mr.
Lane?” concluded Rosabelle artlessly.
The detective smiled kindly at the romantic girl. No doubt she was
contrasting the placid affection between the Morrices, and her own
ardent love for young Croxton and his for her. And no doubt she was
sure, like all fervent souls, that when the years had silvered her hair
and stolen the roses from her cheeks, love would burn as brightly as
in the hey-day of their glorious youth.
“I am not a great expert in the tender passion, Miss Sheldon, but I
am quite prepared to believe real love comes but once in a lifetime to
either man or woman. Well, now, I am much obliged to you for telling
me what you have done, and I am glad you told me. For the present
we will keep it to ourselves. But I think you had better face the fact
that, sooner or later, Mr. Morrice will have to be told by one of us.”
When the girl had left, Lane indulged in a long fit of meditation.
Yes, Morrice had better know this at once. He could probably invent
more than one plan by which Rosabelle could be kept out of it, even
if he approached him directly. But Lane had gauged the financier
sufficiently to know that in some respects he was a very peculiar
man. He might resent the detective’s interference in what he
considered a purely private matter, and order him out of the house.
He would adopt a method which he had used more than once
before when he did not wish to appear personally. He went to a small
typewriter which he only used on special occasions; his usual one
had a personality of its own which might be easily identified, for
certain typewriting is sometimes as distinctive as handwriting.
He indited a brief epistle and addressed it to “Rupert Morrice,
Esquire,” taking care to mark it “Private.” He would take it down to
the City and post it there, thus avoiding the tell-tale West End
postmark.
It was an anonymous letter, signed by “A Well-wisher.” “If that
doesn’t stir him to some sort of action, we must think of something
else,” so ran the reflections of this astute man. “It may precipitate an
explosion, and amongst other things reveal to him that Mr. Archie
Brookes is no more his wife’s nephew than I am.”
He walked away from the pillar-box in the City well pleased with
himself. It could not be said that he felt any compunction with regard
to Mrs. Morrice née Miss Lettice Larchester. She had, no doubt,
married the man for his money, and was treating him very badly. But
even if he had, his hand would not have been stayed in
consequence. His first duty was to his clients.
CHAPTER XVI
AT SCOTLAND YARD
D ISMISSING from his mind for the moment the incident which
Rosabelle Sheldon had made a special visit to communicate to
him—the anonymous letter would put in train the machinery for
elucidating the real facts of that—there were two pressing problems
that Lane was anxious to solve without any undue delay. The one
was the actual position of Sir George Clayton-Brookes. Was he a
comparatively poor man, as his paying-in book went to prove; or a
rich one, as his lavish expenditure in certain directions tended to
show? The second problem was the real identity of the young man
calling himself his nephew, and also passing as the nephew of Mrs.
Morrice by the marriage of her sister to the brother of the mysterious
baronet.
The latter of the two puzzles was in the capable hands of Sellars.
Much would depend upon the result of that interview with the friend
of Mrs. Morrice’s youth, Alma Buckley. And the result depended
upon the woman herself. First of all, had she any knowledge of
Lettice Larchester after they had parted company at the little village
of Brinkstone sufficiently intimate to include the details of her life
between that date and her marriage to the wealthy financier? If she
knew them, was she too staunch a friend to the companion of her
youth to satisfy the curiosity of a stranger, or could she be tempted to
open her mouth by a bribe of sufficient magnitude. If she were a
venal person she would, no doubt, require a considerable sum for
any information she gave.
No large sums could be extracted from the meagre resources of
Richard Croxton. Anxious as he appeared to clear himself, he could
not be expected to reduce himself to penury for an investigation
which might not lead to any clear evidence of his innocence. Even if
Alma Buckley knew the real identity of Archie Brookes and sold the
knowledge for an agreed sum of money, the fact of proving him an
impostor would not necessarily acquit Croxton of the suspicions
resting upon him.
In that case the only person to whom application could be made
would be Morrice himself. And that would entail immediate avowal of
what Sellars had found out; and it might be that such an immediate
avowal might be a little too precipitate for Lane’s plans. Anyway, in
that direction he could do nothing till he knew the result of his
lieutenant’s negotiations with this middle-aged actress.
The further investigations into the case of Sir George he was for
the present keeping in his own hands. Later on he must tackle that of
Archie Brookes, not as regards his antecedents, but his expenditure
and the source of his income. Popular rumour credited Sir George
with the financial support of his alleged nephew. Well, a certain light
upon that portion of the problem had been thrown by Rosabelle’s
statement of the conversation in her aunt’s boudoir which she had
overheard.
It was evident, even from the little she had gathered, that money
was the topic of that conversation; equally evident that Mrs. Morrice
had contributed large sums to the young man’s support. But however
generous her allowance, his supposed aunt could not alone maintain
the burden of young Brookes’s lavish expenditure as detailed by
Simmons, who had the information from a reliable source. He must
have other resources, and the nature and extent of those resources
must be discovered.
Lane felt he would like to discuss this matter in strict confidence
with somebody as clever as himself. Sellars was very intelligent in
his own way, had a wonderful nose for investigation when he was
once put on the right track, when, to use a hunting metaphor, he had
picked up the scent. But he lacked experience and he was not very
imaginative—he had little faculty of anticipating facts, in contrast to
Lane, who had moments of inspiration which guided him instinctively
in a puzzling labyrinth.
Casting about in his mind for a helpful confidant, he thought at
once of his old friend MacKenzie, who now occupied a prominent
position at Scotland Yard. They had joined the Force together as
young men, had risen together, step by step, till separation came
when Lane decided to set up for himself. Of the two, Lane was
slightly the better man, owing to the particular streak of imagination
—that frequency of inspiration to which allusion has been made. But
MacKenzie was only slightly inferior; very sound, very painstaking,
very logical.
There was perfect confidence between the two men. If MacKenzie
wanted counsel or sometimes assistance from Lane, he applied to
him without hesitation; and his friend as frequently availed himself of
the shrewd Scotchman’s powers of analysis and deduction. There
was nothing the two men enjoyed more than a long yarn over their
experiences, to the accompaniment of a good cigar and a stimulating
dose of sound whisky.
“Ah, glad to see you, my boy, it’s a little time since we met,” was
MacKenzie’s greeting to his old friend and comrade, uttered in his
rather broad Scotch, which need not be reproduced here. “We are
rather quiet at the moment, nothing very exciting, just a few simple
little things. I hope you have got something really worth taking
trouble about.”
“I’ve got in hand one of the most remarkable cases I think I’ve ever
had in my life,” was Lane’s reply, and he straightway plunged into a
full recital of the Morrice mystery and the salient facts connected with
it.
His friend listened with the deepest attention, and when it was
finished the two men engaged in a long and animated discussion,
exhausting the arguments for and against the various hypotheses
that were thrown out first by the one and then by the other.
“Well, now, about this Clayton-Brookes,” said MacKenzie
presently. “I think I can give you a little assistance. We’ve had him
under observation since a little after you left us, and that’s a few
years ago now.”
“Ah!” Lane drew a deep breath. He was glad that he had paid this
visit to his old friend and wished that he had come sooner.
The Scotchman waved his big hand round his comfortable, roomy
apartment. “I wouldn’t care to say it outside these four walls, and not
to more than a few inside them, because we’ve nothing but very
substantial conjecture, and up to the present we’ve not been able to
lay a finger on him, he’s so devilish clever. But there’s no doubt he’s
a ‘wrong ’un.’”
“Do you mean actually a crook?” queried Lane.
MacKenzie nodded his massive head. “Anyway, the friend of
crooks; he’s been observed in some very queer company quite
outside of his own proper beat, which we know is the West End and
the fashionable clubs and a few smart and semi-smart houses. We
know birds of a feather flock together, and men are known by the
company they keep.”
So Sir George Clayton-Brookes, the elegant man-about-town, led
a double life then—associating at one end of the scale with the
fashionable and semi-fashionable denizens of the west, at the other
with certain flashy members of the underworld.
MacKenzie proceeded to relate that their attention had been first
attracted to him by a series of burglaries committed at certain
country houses and hotels, from the owners of and visitors at which
valuable jewellery and articles of plate had been stolen. At three of
the houses in question he had actually been a guest at the time of
the robberies, and with regard to the others, he had been a visitor a
little time previously. The theory was that he took advantage of his
opportunities to spy out the position of the land, to furnish the actual
thieves with plans of the interior of the different mansions at which
he had stayed, and give details of the jewellery belonging to the
various guests. It was curious, to say the least, that robberies should
occur, as it seemed, automatically either during his actual visits or
very shortly after them.
Further evidence was afforded by the fact that he had frequently
been observed in the company of certain high-class crooks who
engineered and financed various criminal schemes, the practical
working of which was left to subordinates.
Lane could not say he was surprised overmuch, he had long ago
come to the conclusion that there was something very mysterious
about this supposed man of wealth and substance, who could
purchase a thousand pound car one day, and be scared out of his
wits on another lest a cheque for a paltry thirty pounds should be
dishonoured.
“But as I say,” concluded MacKenzie, “he’s as artful as a monkey,
and we can’t get evidence enough to connect him with any one of
the actual thefts. But there is the coincidence I have mentioned, and
that’s evidence for us, although it wouldn’t do for a judge and jury.”
“And what about the young man—his supposed nephew?” asked
Lane.
“Oh, we’ve had him under observation as well, and, of course, he
must be mixed up with Sir George in some way or another, but we
don’t think in these particular things. They see each other pretty
nearly every day, but they appear to lead different lives. Young
Brookes doesn’t go very much into the same sort of society; he
doesn’t stay at country houses, seems on a bit lower plane than the
baronet. But I’ve no doubt they run some little show of their own
together.”
“Do you know anything about his antecedents before he came on
the scene as Sir George’s nephew?”
“Yes, we know a bit. He was a clerk in a City warehouse before Sir
George took him on. He was living then with a woman who had
apparently brought him up from a child.”
“Do you know the name of the woman?” Lane felt he was on the
track of something, but he was more than startled at the answer. “But
of course, you would learn that.”
“Yes, she is a music-hall artist of a third or fourth-rate type, but
pretty well known in the profession. She is called Alma Buckley.”
CHAPTER XVII
LANE VISITS RICHARD