Computerized Medical Imaging and Graphics 95 (2022) 102022

Contents lists available at ScienceDirect

Computerized Medical Imaging and Graphics

journal homepage: www.elsevier.com/locate/compmedimag

A relation-based framework for effective teeth recognition on dental

periapical X-rays
Kailai Zhang a, *, Hu Chen b, Peijun Lyu b, Ji Wu a
a
Department of Electronic Engineering, Tsinghua University, Beijing, China
b
Center of Digital Dentistry, Peking University School and Hospital of Stomatology & National Engineering Laboratory for Digital and Material Technology of
Stomatology, Beijing, China

A R T I C L E I N F O A B S T R A C T

Keywords: Dental periapical X-rays are used as a popular tool by dentists for diagnosis. To provide dentists with diagnostic
Teeth recognition support, in this paper, we achieve automated teeth recognition of dental periapical X-rays by using deep learning
Convolutional neural network techniques, including teeth location and classification. Convolutional neural network(CNN) is a popular method
Label reconstruction
and has made large improvements in medical image applications. However, in our specific task, the performance
Proposal correlation module
of CNN is limited by lack of data and too many teeth positions in X-rays. Addressing this problem, we consider to
utilize the prior dental knowledge, and therefore we propose a relation-based framework to handle the teeth
location and classification task. According to the relation in teeth labels, we apply a special label reconstruction
technique to decompose the teeth classification task, and use a multi-task CNN to classify the teeth positions.
Meanwhile, for teeth location task, we design a proposal correlation module to use the information in teeth
positions, and insert it into the multi-task CNN. A teeth sequence refinement module is used for the post pro­
cessing. Our experiment results show that our relation-based framework achieves high teeth classification and
location performance, which is a big improvement compared to the direct use of famous detection structures.
With reliable teeth information, our method can provide automated diagnostic support for the dentists.

1. Introduction
Oral diseases have been classified as the third largest disease after
cancer and cerebrovascular diseases by the World Health Organization
(WHO), which have a strong impact on life quality (Sanders et al., 2010).
For oral disease diagnosis, dentists first need to recognize the teeth in
dental X-rays, including finding their location and classify the teeth
positions, which means that the teeth recognition is a key step in the
whole process. Meanwhile, unlike the face and fingerprint features
which are difficult to obtain when one died from disaster or traffic ac­
cident, the teeth can remain unbroken because they are the hardest part
of humans, so that the court doctors can use teeth for person recognition
(Tohnak et al., 2007; Nomir and Abdel-Mottaleb, 2005; Zhou and
Abdel-Mottaleb, 2005) and postmortem identification (Daniels and
Troy, 1984; Nomir and Abdel-Mottaleb, 2007; Corrêa et al., 2020).
Therefore, the teeth detection and classification is very important for
both dentists and court doctors.
For these reasons, in this paper, we aim to achieve automated teeth
recognition in dental periapical X-rays. Dental periapical X-rays are used
as a popular tool for dentists, which can provide details for specific teeth
areas, so that this type of radiograph is more representative to show
disease conditions. Unlike the dental bitewing X-rays which depict all
the teeth in a single X-rays, dental periapical X-rays only contains
several teeth, which is more difficult to recognize especially for inex­
perienced dentists, because there is less global information to recognize
the teeth positions. Some examples are shown in Fig. 1. For court doc­
tors, the similar situation also happens when recognizing the teeth from
the dead person, because the teeth may fall from the body, and it is very
hard to get the bitewing image without one’s cooperation. For example,
the jaws of a victim can be easily disintegrated in a flight accident. In
these cases, automated teeth recognition system in dental periapical X-
rays is meaningful to provide reliable assistance for medical experts.
However, automated teeth recognition in dental periapical X-rays
also has some challenges. On the one hand, compared to magnetic
resonance(MR) images and images generated by computed tomography
(CT), the X-rays are less clear, so that they are more difficult for
recognition. On the other hand, an adult has 32 teeth in total, which
means that there are 32 teeth positions for recognition. Compared to the

* Correspondence to: Faculty of Electronic Engineering, Tsinghua University, Beijing, China.

E-mail address: zhangkl17@mails.tsinghua.edu.cn (K. Zhang).

https://doi.org/10.1016/j.compmedimag.2021.102022
Received 27 May 2021; Received in revised form 5 November 2021; Accepted 12 November 2021
Available online 2 December 2021
0895-6111/© 2021 Elsevier Ltd. All rights reserved.
K. Zhang et al.
Fig. 1. Some examples of dental periapical X-rays. Some teeth positions are difficult to recognize for inexperienced dentists. For instance, the teeth in (a) are easily to
mistakenly recognized as tooth 7 and tooth 8 while in fact they are tooth 6 and tooth 7.
large number of teeth positions, the number of labeled dental X-rays is
usually deficient due to the difficulty of data acquisition and labeling,
and it is related to patients’ privacy. This is also a common case in
medical image applications. The large number of teeth positions under
lack of data make it difficult to achieve good performance.
There are some previous attempts related to automated teeth
recognition of dental X-rays. Some methods are based on traditional
feature extraction methods such as contour detection method (Arbelaez
et al., 2011), level set method (Li et al., 2006) and graph-based method
(Carreira and Sminchisescu, 2011; Boykov, 2001; Felzenszwalb and
Huttenlocher, 2004). Kumar and Rajiv (2016) calculated the similarity
of different images which include teeth for person identification. Lin
et al. (2010) used the region and contour information as the feature
vector, then applied the support vector machine for teeth classification
in dental bitewing X-rays. Mahoor and Abdel-Mottaleb (2005) used
Fourier descriptors of teeth contours as features, and applied the
Bayesian method to design a teeth numbering system in dental bitewing
X-rays. Said et al. (2006) applied a mathematical morphology method
2
Computerized Medical Imaging and Graphics 95 (2022) 102022
for teeth segmentation. Shah et al. (2006) used active contour method to
extract teeth contour for teeth recognition. Yuniarti (2012) used a bi­
nary support vector machine method to classify the extracted tooth into
molar or premolar for both bitewing radiographs and panoramic ra­
diographs. Aeini and Mahmoudi (2010) considered both the location
and size of the mesiodistal neck for each tooth position and applied a
linear model for teeth classification. Tangel et al. (2013) analyzed each
tooth based on multiple criteria such as area/perimeter ratio and
width/height ratio for teeth classification on periapical X-rays. Rad et al.
(2013) used clustering method and texture statistics techniques for teeth
segmentation. These methods use hand-designed features which re­
quires less labeling efforts, but the traditional feature extractors can be
easily influenced by the variance in X-rays.
In last few years, deep learning methods have developed rapidly. The
convolutional neural networks(CNNs) is a popular method in deep
learning methods. The CNN methods such as VGG16 (Simonyan and
Zisserman, 2014), Resnet (He et al., 2016) and Densenet (Gao et al.,
2017), have achieved great success in natural image applications. They
K. Zhang et al.
Fig. 2. Our proposed relation-based framework, including the multi-task CNN and a teeth sequence refinement module.
also have made big improvements in medical image applications such as
cancer cell detection (Zhang et al., 2017), lung disease classification (Li
et al., 2014) and brain segmentation (Dey and Hong, 2018), which are
more robust than traditional feature extractors. Therefore we choose
CNN as the basic model in our task. There are also some previous works
using CNN-based methods for teeth recognition. Zhang et al. (2018)
used a special label tree for teeth detection in dental periapical X-rays,
then numbered the teeth according to some rules. Chen et al. (2019)
designed a CNN for teeth classification, and used some teeth templates
for correction. Mahdi et al. (2020) and Motoki et al. (2020) used the
faster R-CNN (Ren et al., 2015) for teeth detection, and then designed
some rules according to prior knowledge for the refinement of teeth
sequence. Tuzoff et al. (2019) first detected teeth in dental panoramic
radiographs, and then used VGG16 (Simonyan and Zisserman, 2014) for
teeth classification. Lin et al. (2014) segmented the tooth area based on
local singularity analysis. Rana et al. (2017) used CNN to detect the
gingivitis in color images with teeth location, and Jae-Hong et al. (2018)
used CNN to directly judge whether the teeth are healthy or not without
specific teeth positions. These methods achieve good performance in
their tasks. However, most of the methods are not designed for dental
periapical X-rays, and they do not consider the prior dental knowledge.
Different from natural image applications, there is much prior
knowledge in medical image applications, which is very important for
improving the performance. In some cases, the performance improve­
ment by using prior knowledge can be even larger than using different
CNN models (Liao et al., 2018; Nosrati and Hamarneh, 2016). As
mentioned above, we use CNN-based methods in our task. Under the
situation that the data is deficient compared to the large number of teeth
positions, we utilize the prior dental knowledge to prevent overfitting of
CNN and get better performance. In dental X-rays, the prior knowledge
is that the 32 teeth positions are dependent. This relationship reflects in
two aspects. First, each tooth label has multiple information. Second,
each tooth position is bounded by its neighbors. We design our
relation-based framework according to these prior knowledge. Based on
the multiple information in teeth labels, we propose a special label
reconstruction technique to decompose the whole teeth classification
task to three subtasks, and design a multi-task CNN accordingly, which
can be seen as a divide-and-conquer method. Considering the informa­
tion between teeth positions, we also design a proposal correlation
module and insert it into multi-task CNN to get better performance.
Finally some rules generated by knowledge are also used for the
refinement of teeth sequence output by our multi-task CNN. The
experiment results show that our relation-based framework achieves big
improvement of teeth recognition performance compared to the direct
use of famous detection structures.
3
Computerized Medical Imaging and Graphics 95 (2022) 102022
The main contributions of this paper are:(1)We design a relation-
based framework for teeth recognition in dental periapical X-rays,
which improves the performance a lot compared to the direct use of
famous detection structures. (2)We propose a special label reconstruc­
tion technique to decompose the teeth classification task according to
the multiple information in teeth labels. (3)We design a proposal cor­
relation module, which can utilize the information between teeth posi­
tions to get better performance.
The remainder of this paper is organized as follows: Section 2 will
introduce our proposed relation-based framework with different key
components. Section 3 will provide the details of our designed compo­
nents, including the label reconstruction technique with multiple
branches design, the proposal correlation module and the teeth
sequence refinement module for post processing. The experiment results
and relevant discussion are shown in Section 4. Section 5 is the final
conclusion and Section 6 is the acknowledgements.
2. Materials and methods
2.1. Dataset description
The dataset of dental periapical X-rays is provided by Peking Uni­
versity School and Hospital of Stomatology, which has 1250 images in
total, including 4336 teeth in 32 teeth positions. Each X-ray has a size of
approximately (300to500) × (300to400) pixels, and it has less than 5
teeth in most cases. All the images are collected anonymously. 32 teeth
positions are labeled with bounding boxes by professional dentists, using
the Federation Dentaire Internationale(FDI) teeth numbering system
(ISO-3950). Therefore the labeling results are thought as the gold
standard in our experiments. In our experiments, 10-folds validation is
used for performance evaluation, which means that in each round, 1125
images are used for network training and 125 images are used for test.
The average performance is considered to ensure that the results are
robust.
2.2. System framework
In this section, we propose our system framework. As mentioned
above, we design the relation-based framework to achieve automated
teeth recognition, which is shown in Fig. 2. Our system is a multi-mask
CNN with the post processing module, which receives a dental periapical
X-ray as input. It outputs all the teeth bounding boxes with classification
results. In our system framework, the basic structure is a object detection
framework. Faster R-CNN (Ren et al., 2015) is a famous object detection
method, which achieve great success on many image recognition tasks,
K. Zhang et al.
Fig. 3. Our multiple branches according to label reconstruction technique. Compared to traditional detection branch and classification branch with many classes (top
figure), we design three classification branches which has less classes (bottom figure). The FL means the full connection layer and the number means the dimension of
output for each branch.
so that we use faster R-CNN as our basic structure. Faster R-CNN in­
cludes a convolution backbone, a region proposal network(RPN) and a
ROIPooling layer. The convolutional backbone usually includes several
convolutional layers, pooling layers and batch normalization layers, and
it can output the processed feature map. After the convolutional back­
bone, the region proposal network can generate proposal candidates
from the feature map. The ROIPooling layer is used to extract these
proposal candidates to fixed-length vector, and finally these proposal
candidates are sent to a classification branch and a detection branch to
get the object classification and detection results.
Based on faster R-CNN, we apply Resnet (He et al., 2016) as the
convolutional backbone. Considering the multiple information in teeth
labels, we propose a special label reconstruction technique, and
accordingly design a multiple branches structure to output the results,
which can be regarded as a multi-task CNN. Based on the relationship
between teeth, we also design a proposal correlation module, and insert
it into the multi-task CNN. A teeth sequence refinement module is used
for post processing after the multi-task CNN. More details are introduced
in the following sections.
2.3. Label reconstruction
As mentioned above, compared to the number of training data, the
number of teeth positions is too large to train network directly. How­
ever, according to the prior knowledge of teeth, the 32 teeth positions
have relationship and they can be manually divided into four parts: the
right upper teeth(RUT), the left upper teeth(LUT), the right down teeth
(RDT), and the left down teeth(LDT). Each part has 8 teeth positions,
named T1 to T8 from the middle to each side of jaw. We order the teeth
sequence in front view so that all the teeth positions can be labeled as
below:
RUT8, RUT7, ..., RUT1, LUT1, ..., LUT7, LUT8
RDT8, RDT7, ..., RDT1, LDT1, ..., LDT7, LDT8
According to the division of labels, there are three different marks: R
or L, U or D, and 1–8. Our label reconstruction technique is to combine
some of these marks to get three kinds of new labels: the teeth(T), the
upper teeth(UT) or the down teeth(DT), teeth 1(T1) to teeth 8(T8). The
mark R and T can be obtained by the output teeth sequence so that we do
4
Computerized Medical Imaging and Graphics 95 (2022) 102022
not consider it on this stage. Accordingly, instead of predicting the
classes in 32 teeth positions directly for proposal candidates, we design
three classification branches and use three kinds of new labels as
groundtruths, which is shown in Fig. 3. This decomposition can be seen
as a divide-and-conquer method, and our framework is a multi-task
CNN. Each branch includes a full connection layer, which represents a
subtask. The first branch is used to determine whether a proposal
candidate is a tooth. The second branch is used to judge whether a
proposal candidate is an upper tooth or a down tooth. The third branch
is used to classify the specific teeth positions from T1 to T8. Therefore all
the three branches have much less classes than the direct prediction of
32 teeth positions. The fourth branch is the detection branch which is
the same as that in faster R-CNN.
On the training stage, for our multi-task CNN, the total loss function
can be defined as below:
losstotal = lossb1 + I[c = 1](λb2 lossb2 + λb3 lossb3 + +λb4 lossb4 ) (1)
where lossb1, lossb2 and lossb3 represent three loss functions of classifi­
cation branches, and the cross entropy loss is used for all of them. lossb4
is the box regression loss. We use the smooth L1 loss (Ren et al., 2015)
for it, which is the same as that in faster R-CNN. λb2, λb3 and λb4 represent
the weight parameters. I is the indicator function which value is 0 or 1. It
means that we only calculate the latter three loss items if the ground­
truth label of a proposal candidate is tooth(c=1), otherwise these loss
items are ignored. The back propagation algorithm is used to update the
network parameters. On the test stage, we only consider the bounding
box that is judged as a tooth in the first branch, and we get other marks
and bounding box prediction from other three branches. These infor­
mation are combined and sent to the teeth sequence refinement module
for posting processing.
2.4. Proposal correlation module
As mentioned before, in our multi-task CNN, after we get the final
feature map F from the convolutional backbone, the RPN with ROI­
Pooling layer can extract a fixed number N of proposal candidates to
feature vectors f nt (n = 1, 2, ..N). For dental X-rays, the tooth has corre­
lation to each other. Each tooth is connected with its neighbor teeth. In
other words, once we find a tooth in the input X-ray, there always exists
K. Zhang et al.
Fig. 4. Our proposal correlation module and its correlation units. The FL represents the full connection layer.
another tooth on a side of it. This information is hardly seen in natural
image applications, and it can be used as patterns to find the neighbor
teeth when there is little global information in dental periapical X-rays.
Therefore, we aim to model this correlation. We design a proposal cor­
relation module accordingly and insert it into our multi-task CNN, which
is shown in Fig. 4(a).
Since we can not know the label of proposal candidates in advance
especially on the test stage, we consider to utilize the information be­
tween each pair of proposal candidates. In our proposal correlation
module, all the feature vectors f nt (n = 1, 2, ..N) are first sent into a full
connection layer to generate the output feature vector f na (n = 1, 2, ..N)
for each proposal candidate. Then these feature vectors are sent to M
correlation units simultaneously. Each correlation unit is aimed to learn
a correlation pattern f pn n correlation unit l(l = 1, 2, . . , M) can be described as:
c (p = 1, 2, .., M) by using all of f a (n = 1, 2, ..N),
which can represent a kind of position relationship. We choose the
output pattern that has the maximum response for all the f na from cor­
relation units, so that the final output of our proposal correlation module
can be described as:
f no = f na + max f ln
c (n = 1, 2, ..N) (2)
l∈[1,M]
5
Computerized Medical Imaging and Graphics 95 (2022) 102022
For our correlation unit design, we consider to extract the information
between proposal candidates, which has a similar form to the self-
attention mechanism (Vaswani et al., 2017). A query matrix Ql, a key
matrix Kl and a value matrix Vl which have trainable parameters are set
for each correlation unit l. Besides the information between proposal
candidates, the environment should also have effects on each proposal
candidates. Therefore, we use a global pooling operation for feature map
F to get the global feature vector gf, and combine it with the feature
vectors of proposal candidates:
f ns = f na + gf (n = 1, 2, ..N) (3)
After we get f ns (n = 1, 2, ..N), for proposal candidate n, the output of
∑
f ln wlmn ⋅(Vl ⋅f ms ) (n = 1, 2, ..N) (4)
c =
m∈[1,N]
where wlmn is the attention weight and its summation is 1 over index m,
which means that the influence of other proposal candidates is weighted
by the attention weights. The wlmn should be related to the both the
feature and position of proposal candidates, so that we consider the
K. Zhang et al. Computerized Medical Imaging and Graphics 95 (2022) 102022

Fig. 5. A visualized example of our system procedure.

information of both feature vectors and relative positions of proposals to image are all the upper teeth or the down teeth, and this prior knowl­
generate the attention weights. For the interaction of feature vector f ms edge can be used for output refinement. Therefore, for the teeth output
and f ns , we use by the multi-task CNN, we define a judgment function to evaluate the
results as below:
wlmn
f = DP(Kl ⋅f ms , Ql ⋅f ns ) (5)
∑
Tu Td
∑
judgement = Tu + Cui − (Td + Cdi ) (9)
to get the scalar weight wlmn
f where DP means the dot product. For the i=1 i=1
relevant position pm and pn of proposal candidate m and n predicted by
RPN, described as [xm, ym, hm, wm] and [xn, yn, hn, wn]. Their relative where Tu and Td are the number of upper teeth and down teeth judged by
position is more important than the absolute position, so that we the second branch in multi-task CNN. Cui and Cdi are the relevant con­
combine their positions to a relative position vector fidence coefficient. If judgment is positive, it means that most of the teeth
in a X-ray are the upper teeth, and we judge all the teeth to upper teeth,
pmn
|xm − xn | |ym − yn | wm hm
(6) otherwise they are judged to down teeth. In most cases, if the number of
r = [ , , , ].
wm hm wn hn right teeth is larger than the number of wrong teeth, all the teeth will be
judged to the right teeth for mark U or D.
For better representation, the embedding method in Vaswani et al.
Since the teeth sequence has an order and the adjacent teeth should
(2017) and Hu et al. (2018) can transform the position vector pmn r to a have the adjacent numbers, we check the teeth positions of output teeth
higher dimension vector pmn for the following-up processing. Then we
h sequence from T1 to T8, and give teeth the mark R or L. According to the
use a trainable vector P to transform pmn
h to a scalar weight, which can be number of teeth in an output teeth sequence, we use all the right teeth
calculated as sequence templates and compare them with the teeth sequence pre­
wlmn = Pl ⋅pmn (7) dicted by multi-task CNN. The difference between a right template and
the output teeth sequence is defined as
p h

After we get the feature information wlmn

f and position information wlmn
p , ∑
N

we can calculate the attention weight wlmn by using a softmax-like difference = wi ⋅I[Ci ∕
= Ti ] (10)
i=1
function, which is defined as

wlmn lmn where the Ci and Ti are the label of tooth candidate output by multi-task
h ⋅exp(ws )
wlmn = (8) CNN and the right tooth position in template respectively. I is the
∑
N
wlin lin
h ⋅exp(ws ) indicate function. If the condition is true, then the value of I is 1,
otherwise it is 0. wi is the weight of Ti. The larger weight means that the
i=1

relevant tooth position in template is more credible. After comparison,

After we get all the attention weights, we can get f ln
c from each corre­
the output teeth sequence is judged to the template that has the mini­
lation unit l and generate the output f no for the proposal correlation
mum difference. It is worth mentioning that there are some special cases
module. These feature vectors are sent to the multiple branches for final
for consideration. If the X-ray only contains one tooth, the teeth
recognition. The trainable parameters are randomly initialized in cor­
sequence refinement module will not work, because a single tooth
relation units so that they are different from each other. During the
cannot be corrected without other teeth. If the X-ray contains several
network training stage, the parameters in our proposal correlation
teeth, but a tooth is lost in the middle of teeth sequence. Then the
module can be directly updated through the back propagation
bounding boxes output by multi-task CNN is not adjacent in the relevant
algorithm.
location. We add a special placeholder, marked as X, which represents
the missing tooth in the teeth sequence. When we use the teeth template
to calculate the difference according to formula 10, this placeholder is
2.5. Teeth sequence refinement module
always regarded as a right one. We use an example to describe the whole
process, if all the wi is set to 1 and there are five teeth detected in an
After we get the teeth information from multiple branches, we can
input X-ray, including a missing tooth. If the ordered teeth sequence is
combine these information to get the teeth sequence, arranged by the x-
judged to T3, T2, T1, X, T1 by multi-task CNN, it is easy to know that the
coordinate of bounding box center. We design a teeth sequence refine­
matching template is T3, T2, T1, T1, T2 by calculation of difference, and a
ment module for post processing. In periapical X-rays, the teeth in an

6
K. Zhang et al.
Fig. 6. Some visualized results of our system output.
wrong tooth position T1 is corrected to T2. According to the order of
teeth, the sequence can be further marked as RT3, RT2, RT1, X, LT2, so
that the R or L information is inserted. With U or D judged by formula 9,
we can get the precise label from 32 teeth positions for each tooth, and
finish the whole teeth recognition task.
3. Experiments and discussion
3.1. Parameters setting and system output
For experiment evaluation, the experiment environment is running
on ubuntu 14.04 with gpu 1080Ti, and the software used in the exper­
iments is python with mixnet. For network training, the stochastic
gradient descent(SGD) optimizer with momentum is used. The initial
learning rate and momentum are set to 0.001 and 0.9 respectively. The
batch size is set to 32 and the number of training epoches is set to 200.
The λb1, λb2 and λb3 in formula 1 are simply set to 1. For our proposal
correlation module, the M and N in formula 2 are set to 16 and 1000
respectively. For our teeth sequence refinement module, the wi in for­
mula 10 is set to the percentage of number of each tooth position in
training data, the larger wi means that the number of relevant tooth
7
Computerized Medical Imaging and Graphics 95 (2022) 102022
position is larger and this tooth position is more credible in templates.
We also use the pre-train model in ImageNet for our multi-task CNN to
get a good parameters initialization.
An example of our system procedure is shown in Fig. 5. The infor­
mation of tooth position is first obtained by each branch. We can
combine these information to get the teeth sequence, and send it to the
teeth sequence refinement module to get the final output results. Our
system can not only provide dentists with comprehensive and reliable
diagnostic support, but also provide visualized intermediate results for
analysis in case that there are wrong final results.
3.2. Evaluation and comparisons
We do a lot of experiments for our proposed framework. We first
show some visualized results output by our framework, which is shown
in Fig. 6. We can find that our system can get right results for different
teeth positions. For quantitative results, the widely used metrics are the
precision, recall and F-score, which are defined as below:
TP
precision = (11)
TP + FP
K. Zhang et al. Computerized Medical Imaging and Graphics 95 (2022) 102022

Table 1 Table 3
The performance of intermediate nodes in our relation-based framework. The comparisons of different methods. The TSRM means the teeth sequence
Branch 1 Branch 2 Branch 3 Teeth sequence refinement
refinement module.
module Precision Recall F-score
Precision 0.968 0.956 0.790 0.951 Fast R-CNN 0.629 0.595 0.612
Recall 0.971 0.958 0.798 0.955 Faster R-CNN 0.672 0.595 0.631
F-score 0.969 0.957 0.794 0.953 R-FCN 0.718 0.487 0.580
Fast R-CNN + TSRM 0.861 0.762 0.808
Faster R-CNN + TSRM 0.890 0.785 0.834
R-FCN + TSRM 0.913 0.609 0.731
Table 2 Zhang et al. (2018) 0.893 0.897 0.895
The comparison for our proposal correlation module and its variants. GF means Chen et al. (2019) 0.904 0.909 0.906
the global feature generated by global pooling operation and it is listed sepa­ Our relation-based framework 0.951 0.955 0.953
rately. PC is our proposed proposal correlation module. The PCta and PCtm are the
proposal correlation module with average response and t correlation units,
proposal correlation module with maximum response and t correlation units. than the number of right teeth, all the teeth in the sequence may be
judged to wrong results. However, this case seldom happens in our ex­
Precision Recall F-score
periments, because each classification branch has a good performance,
Without PC 0.911 0.914 0.912
so that the probability of the case that more than one tooth are predicted
PC1 0.932 0.935 0.933
0.940 0.942 0.941
wrongly in a X-ray is very low. In fact, we only find that the X-ray which
PC16
a
just includes two teeth can lead to this case. If one of the two teeth is
PC16 0.946 0.949 0.947
m
predicted wrongly, then the other teeth may be judged to a wrong one.
PC16
a +GF
0.943 0.944 0.943
This case does not make large influence to our final results.
PC4m +GF 0.939 0.942 0.940
We also evaluate the performance of our proposal correlation mod­
PC8m +GF 0.946 0.950 0.948
ule. The proposal correlation module in multi-task CNN utilizes the in­
PC16 0.951 0.955 0.953
m +GF formation between proposal candidates to get better teeth classification
and location performance. We remove this module to compare the
performance. We also evaluate the performance of some variations, such
TP
recall = (12) as using average response, changing the number of correlation units and
TP + FN
removing the global feature. The experiment results are shown in
2⋅precision⋅recall Table 2.
F-score = (13) From the results, by inserting the proposal correlation module, the
precision + recall
teeth recognition performance is much better than that without proposal
where the TP,TN,FP,FN denote the number of true positive, true nega­ correlation module, because we utilize the location information between
tive, false positive, false negative objects respectively. We use the teeth positions. Each proposal candidate can influence its neighbors. By
intersection over union(IOU) between bounding boxes to count the true ablation study, when the number of correlation units M increases, we
positive predictions. The IOU of two boxes X and Y is defined as can get further performance improvement. We believe this is because
each correlation unit can output a pattern that represent a kind of
area(X ∩ Y)
IOU = (14) relationship between teeth positions. By comparison, the maximum
area(X ∪ Y) response from correlation units is better than the average response. By
the combination of proposal feature and global feature, we can get better
The box predictions output by our relation-based framework are
results, which means that the global information is helpful for local
compared to the groundtruth. If the IOU of a box prediction and the
proposal candidates. Each proposal candidate can interact with the
relevant groundtruth is larger than a threshold, then it is judged as a
global feature vector to get more global information.
right result. We set the IOU threshold to 0.5 in our experiments. We first
For further evaluation, we compare our relation-based framework
evaluate the performance of our multiple branches and teeth sequence
with some famous detection baselines, such as Fast R-CNN (Girshick,
refinement module. Based on the teeth classification and location re­
2015), Faster R-CNN (Ren et al., 2015) and R-FCN (Dai et al., 2016).
sults, we can calculate the precision, recall, and F-score of three
These structures achieve very good performance in many natural image
branches and the teeth sequence refinement module. In Table 1, we test
applications, so that we use these structures to detect 32 teeth positions
the performance of these intermediate nodes in our relation-based
as the direct detection method. For each network structure, since the
framework.
teeth sequence refinement module can check the output teeth sequence
According to the results, if we only judge whether a proposal
and correct a large number of wrong classification results, we also insert
candidate is a tooth or background(branch 1), or whether it is a upper
this module to these baselines for comparison. Meanwhile, some pre­
tooth or a down tooth(branch 2), the training data is enough to get high
vious attempts for teeth location and numbering in dental periapical
precision and recall because there are only two classes for these two
X-rays such as Zhang et al. (2018) and Chen et al. (2019) are also
branches. For the third branch, the precision and recall decrease because
compared with our relation-based framework. All of these methods are
the number of classes increases sharply to 8(T1 to T8), which is large
evaluated on our dataset for comparison. All the results are shown in
compared to the number of training data, and this leads to performance
Table 3.
drop. It also indicates that the direct classification on 32 teeth positions
According to the results, our relation-based framework achieves
can not achieve good performance due to insufficient network training,
higher teeth classification and location performance than these previous
which will be shown in the following experiments, and it is necessary to
works. Compared to the famous detection structures, although these
decompose the teeth classification task. Finally, the teeth sequence
structures and our relation-based framework use the same data for
refinement module corrects a large number of wrong classification re­
network training, our proposed method is much better than these direct
sults according to formula 9 and formula 10, and the final performance
methods. The improvement mainly benefits from our multiple branches
increases to a high level again. In fact, in most cases, if the number of
design. Even with the correction of teeth sequence refinement module,
right teeth is larger than the number of wrong teeth, the teeth sequence
these direct methods can not achieve the performance of our divide-and-
refinement module can correct these wrong classification results and
conquer method. This is because these famous detection structures can
improve the performance a lot. If the number of wrong teeth is no less

8
K. Zhang et al. Computerized Medical Imaging and Graphics 95 (2022) 102022

Table 4 Acknowledgements
The performance of different teeth positions.
Precision Recall F-score This work is sponsored by the National Natural Science Foundation
of China (Grant No. 61571266), Beijing Municipal Natural Science
T1 0.921 0.916 0.918
T2 0.950 0.981 0.965 Foundation (No. L192026) and Tsinghua-Foshan Innovation Special
T3 0.978 0.923 0.950 Fund (TFISF) (No. 2020THFS0111).
T4 0.935 0.969 0.952
T5 0.948 0.971 0.959 References
T6 0.924 0.931 0.927
T7 0.963 0.928 0.945
Aeini, F., Mahmoudi, F., 2010. Classification and numbering of posterior teeth in
T8 0.973 0.948 0.960
bitewing dental images. In: 2010 3rd International Conference on Advanced
Computer Theory and Engineering (ICACTE), vol. 6, pages V6–66. IEEE.
Arbelaez, P., Maire, M., Fowlkes, C., Malik, J., 2011. Contour detection and hierarchical
not extract the relationship of teeth and other information efficiently, image segmentation. Pattern Anal. Mach. Intell. IEEE Trans. 33 (5), 898–916.
under lack of training data and large number of teeth positions, and this Boykov, Y., 2001. Interactive graph cuts for optimal boundary and region segmentation
of objects in n-d images. In: Proceedings of the IEEE Conference on Computer Vision.
leads to insufficient training of network. By using the label reconstruc­ Carreira, J., Sminchisescu, C., 2011. Constrained parametric min-cuts for automatic
tion technique, we insert the prior knowledge to our framework. We object segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 34 (7), 3241–3248.
decompose the teeth classification task manually and insert the proposal Chen, Hu, Zhang, Kailai, Lyu, Peijun, Li, Hong, Zhang, Ludan, Wu, Ji, Lee, Chin-Hui,
2019. A deep learning approach to automatic teeth detection and numbering based
correlation module, and our relation-based framework can extract the on object detection in dental periapical films. Sci. Rep. 9 (1), 3840.
tooth position information by multiple branches. Each branch tries to Corrêa, HSD, Brescia, G., Cortellini, V., Verzeletti, A., 2020. Human identification
classify a part of the whole tooth label, which has less classes under lack through dna analysis of restored postmortem teeth. Forensic Sci. Int. Genet. 47,
102302.
of data and is easier to train. Meanwhile, more prior knowledge can be
Dai, J., Li, Y., He, K., Sun, J., 2016. R-fcn: object detection via region-based fully
easily inserted to the teeth sequence refinement module, which also convolutional networks. In: Advances in Neural Information Processing Systems,
helps the system to improve teeth recognition performance a lot. 379–387.
Therefore, our method achieves very good results for teeth recognition. Daniels, Troy, E., 1984. Human mucosal langerhans cells: postmortem identification of
regional variations in oral mucosa. J. Investig. Dermatol. 82 (1), 21–24.
In dental practice, different teeth positions from T1 to T8 represent Dey, Raunak, Hong, Yi, 2018. Compnet: Complementary segmentation network for brain
different types of teeth, including a wisdom, two molars, two premolars, mri extraction. International Conference on Medical Image Computing and
a canine, and two incisors. Their structures are different and should be Computer-Assisted Intervention. Springer, pp. 628–636.
Felzenszwalb, P.F., Huttenlocher, D.P., 2004. Efficient graph-based image segmentation.
treated accordingly, so that we also evaluate the performance of teeth Int. J. Comput. Vis. 59 (2), 167–181.
recognition in different teeth positions for our proposed method. The Gao, H., Zhuang, L., Van Der Maaten, L., Weinberger, K.Q., 2017. Densely connected
results are shown in Table 4. From the results, our relation-based convolutional networks. In: IEEE Conference on Computer Vision and Pattern
Recognition.
framework achieves consistent performance on different teeth posi­ Girshick, R., 2015. Fast r-cnn. In: Proceedings of the IEEE International Conference on
tions. It means that our relation-based framework can indeed provide Computer Vision, 1440–1448.
efficient and reliable teeth recognition results for dentists, which is very He, K., Zhang, X., Ren, S., Jian, S., 2016. Deep residual learning for image recognition.
In: IEEE Conference on Computer Vision and Pattern Recognition.
helpful for diagnosis support. Hu, H., Gu, J., Zhang, Z., Dai, J., Wei, Y., 2018. Relation networks for object detection.
In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Jae-Hong, Lee, Do-hyung, Kim, Seong-Nyum, Jeong, Seong-Ho, Choi, 2018. Diagnosis
4. Conclusion
and prediction of periodontally compromised teeth using a deep learning-based
convolutional neural network algorithm. J. Periodontal Implant Sci. 48 (2),
In this paper, we propose a novel relation-based framework for teeth 114–123.
recognition of dental periapical X-rays, including teeth location and Kumar, Rajiv, 2016. Teeth recognition for person identification. In: International
Conference on Computation System and Information Technology for Sustainable
classification. The lack of data is the common case in medical field. To Solutions, 13–16.
handle with this problem efficiently, we first analysis the prior knowl­ Li, S., Fevens, T., KrzyAk, A., Song, L., 2006. An automatic variational level set
edge and design the whole framework. We apply the label reconstruc­ segmentation framework for computer aided dental x-rays analysis in clinical
environments. Comput. Med. Imaging Graph. 30 (2), 65–74.
tion technique to decompose teeth classification task and design a multi- Li, Q., Cai, W., Wang, X., Zhou, Y., Feng, D.D., Chen, M., 2014. Medical image
task CNN, while we also design a proposal correlation module by classification with convolutional neural network. In: International Conference on
considering the location information in teeth positions, and the teeth Control Automation Robotics and Vision.
Liao, Haofu, Tang, Yucheng, Funka-Lea, Gareth, Luo, Jiebo, Kevin Zhou, Shaohua, 2018.
sequence refinement module is used for post processing. Our relation- More knowledge is better: cross-modality volume completion and 3d. 2d
based framework has much better teeth recognition performance than segmentation for intracardiac echocardiography contouring. International
previous works and famous direct detection methods, which can provide Conference on Medical Image Computing and Computer-Assisted Intervention.
Springer, pp. 535–543.
reliable and comprehensive diagnostic support for dentists. In the future
Lin, P.L., Huang, P.Y., Huang, P.W., Hsu, H.C., Chen, C.C., 2014. Teeth segmentation of
work, we will continue to improve the teeth recognition performance dental periapical radiographs based on local singularity analysis. Comput. Methods
and expand our method to similar tasks. Prog. Biomed. 113 (2), 433–445.
Lin, PL, Lai, YH, Huang, PW, 2010. An effective classification and numbering system for
dental bitewing radiographs using teeth region and contour information. Pattern
CRediT authorship contribution statement Recognit. 43 (4), 1380–1392.
Mahdi, F.P., Motoki, K., Kobashi, S., 2020. Optimization technique combined with deep
learning method for teeth recognition in dental panoramic radiographs. Sci. Rep. 10
Kailai Zhang: Conceptualization, Methodology, Software, Formal (1).
analysis, Investigation, Writing, Visualization. Hu Chen: Conceptuali­ Mahoor, Mohammad H., Abdel-Mottaleb, Mohamed, 2005. Classification and numbering
zation, Validation, Investigation, Data curation. Peijun Lyu: Supervi­ of teeth in dental bitewing images. Pattern Recognit. 38 (4), 577–586.
Motoki, K., Mahdi, F.P., Yagi, N., Nii, M., Kobashi, S., 2020. Automatic teeth recognition
sion, Resources, Project administration. Ji Wu: Supervision, Resources, method from dental panoramic images using faster r-cnn and prior knowledge
Project administration. model. In: 2020 Joint 11th International Conference on Soft Computing and
Intelligent Systems and 21st International Symposium on Advanced Intelligent
Systems (SCIS-ISIS).
Nomir, O., Abdel-Mottaleb, M., 2005. A system for human identification from x-ray
Declaration of Competing Interest dental radiographs. Pattern Recognit. 38 (8), 1295–1305.
Nomir, O., Abdel-Mottaleb, M., 2007. Human identification from dental x-ray images
The authors declare that they have no known competing financial based on the shape and appearance of the teeth. IEEE Trans. Inf. Forensics Secur. 2
(2), 188–197.
interests or personal relationships that could have appeared to influence
the work reported in this paper.

9
K. Zhang et al.
Nosrati, M.S., Hamarneh, G., 2016. Incorporating prior knowledge in medical image
segmentation: a survey. arXiv:1607.01092.
Rad, A.E., Shafry, M., Rahim, M., Norouzi, A., 2013. Digital dental x-ray image
segmentation and feature extraction. Telkomnika Indonesian. J. Electr. Eng. 11 (6),
3109–3114.
Rana, Aman, Yauney, Gregory, Wong, Lawrence C., Gupta, Otkrist, Muftu, Ali,
Shah, Pratik, 2017. Automated segmentation of gingival diseases from oral images.
2017 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT). IEEE,
pp. 144–147.
Ren, S., He, K., Girshick, R., Sun, J., 2015. Faster r-cnn: towards real-time object
detection with region proposal networks. In: Advances in Neural Information
Processing Systems, 91–99.
Said, E.H., Nassar, Dem, Fahmy, G., Ammar, H.H., 2006. Teeth segmentation in digitized
dental x-ray films using mathematical morphology. IEEE Trans. Inf. Forensics Secur.
1 (2), 178–189.
Sanders, Anne E., Slade, Gary D., Lim, Sungwoo, Reisine, Susan T., 2010. Impact of oral
disease on quality of life in the us and australian populations. Community Dent. Oral
Epidemiol. 37 (2), 171–181.
Shah, S., Abaza, A., Ross, A., Ammar, H., 2006. Automatic tooth segmentation using
active contour without edges. In Biometric Consortium Conference.
Simonyan, K., Zisserman, A., 2014. Very deep convolutional networks for large-scale
image recognition. arXiv:1409.1556.
10
Computerized Medical Imaging and Graphics 95 (2022) 102022
Tangel, M.L., Fatichah, C., Fei, Y., Janet, P.B., Hirota, K., 2013. Dental classification for
periapical radiograph based on multiple fuzzy attribute. In: 2013 IFSA World
Congress NAFIPS Annual Meeting.
Tohnak, S., Mehnert, Ajh, Mahoney, M., Crozier, S., 2007. Synthesizing dental
radiographs for human identification. J. Dent. Res. 86 (11), 1057–1062.
Tuzoff, D.V., Tuzova, L.N., Bornstein, M.M., Krasnov, A.S., Kharchenko, M.A.,
Nikolenko, S.I., Sveshnikov, M.M., Bednenko, G.B., 2019. Tooth detection and
numbering in panoramic radiographs using convolutional neural networks.
Dentomaxillofac. Radiol.
Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion,
Gomez, AidanN, Kaiser, Lukasz, Polosukhin, Illia, 2017. Attention is all you need.
arXiv.
Yuniarti, A., 2012. Classification and numbering of dental radiographs for an automated
human identification system. Telkomnika 10 (1).
Zhang, Kailai, Wu, Ji, Chen, Hu, Lyu, Peijun, 2018. An effective teeth recognition method
using label tree with cascade network structure. Comput. Med. Imaging Graph. 68,
61–70.
Zhang, J., Hu, H., Chen, S., Huang, Y., Qiu, G., 2017. Cancer cells detection in phase-
contrast microscopy images based on faster r-cnn. In: International Symposium on
Computational Intelligence and Design.
Zhou, J., Abdel-Mottaleb, M., 2005. A content-based system for human identification
based on bitewing dental x-ray images. Pattern Recognit. 38 (11), 2132–2142.