Received August 25, 2019, accepted October 13, 2019, date of publication October 17, 2019, date of current

version October 30, 2019.

Digital Object Identifier 10.1109/ACCESS.2019.2948081

Question Answering Systems With Deep

Learning-Based Symbolic Processing
HIROSHI HONDA AND MASAFUMI HAGIWARA, (Senior Member, IEEE)
Faculty of Science and Technology, Keio University, Yokohama 2238522, Japan
Corresponding author: Hiroshi Honda (honda@keio.jp)
This work was supported in part by the Keio University Kenkyu-no-Susume Scholarship, and in part by the KLL Ph.D. Program Research
Grant.

ABSTRACT The authors propose methods to learn symbolic processing with deep learning and to build
question answering systems by means of learned models. Symbolic processing, performed by the Prolog
processing systems which execute unification, resolution, and list operations, is learned by a combination
of deep learning models, Neural Machine Translation (NMT) and Word2Vec training. To our knowledge,
the implementation of a Prolog-like processing system using deep learning is a new experiment that has
not been conducted in the past. The results of their experiments revealed that the proposed methods are
superior to the conventional methods because symbolic processing (1) has rich representations, (2) can
interpret inputs even if they include unknown symbols, and (3) can be learned with a small amount of
training data. In particular (2), handling of unknown data, which is a major task in artificial intelligence
research, is solved using Word2Vec. Furthermore, question answering systems can be built from knowledge
bases written in Prolog with learned symbolic processing, which, with conventional methods, is extremely
difficult to accomplish. Their proposed systems can not only answer questions through powerful inferences
by utilizing facts that harbor unknown data but also have the potential to build knowledge bases from a
large amount of data, including unknown data, on the Web. The proposed systems are a completely new
trial, there is no state-of-the-art methods in the sense of ‘‘newest’’. Therefore, to evaluate their efficiency,
they are compared with the most traditional and robust system i.e., the Prolog system. This is new research
that encompasses the subjects of conventional artificial intelligence and neural network, and their systems
have higher potential to build applications such as FAQ chatbots, decision support systems and energy-
efficient estimation using a large amount of information on the Web. Mining hidden information through
these applications will provide great value.

INDEX TERMS Deep learning, knowledge base, prolog, question answering system, neural machine
translation, symbolic processing, Word2Vec.

I. INTRODUCTION volumes of data as found on the Web, became difficult with

In artificial intelligence research, a wide range of stud-
ies on inferences using symbolic processing exist. From
1970s to 1980s, studies on expert systems [1] that simu-
lated expert decision making, were actively undertaken. Early
expert systems needed to strictly build knowledge bases by
hand. Eventually, studies were conducted to generate rules
where the knowledge base was rendered incomplete [2], [3]
and to obtain inferences based on hypotheses when there
was a shortage of data [4]–[6], which, consequently, helped
to improve these expert systems. However, handling large
The associate editor coordinating the review of this manuscript and
approving it for publication was Victor Hugo Albuquerque
these technologies, and hence, they were only used in limited
fields.
In the 1990s, connectionists studied symbolic processing
by using multilayered neural networks [7], [8]. However,
owing to limitations of the hardware at that time and learning
ability of the multilayered neural networks, the connectionists
were limited to propose methods but were unable to build
practical systems.
With the emergence of deep learning technique emerged
at the start of 2010s, the learning ability of neural networks
drastically improved. In particular, graph structures could be
learned by deep learning [9], [10], and symbolic processing
. could be reexamined by neural networks [11]–[13]. However,

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/
152368 VOLUME 7, 2019
H. Honda, M. Hagiwara: Question Answering Systems With Deep Learning-Based Symbolic Processing
in order to build knowledge bases from large volumes of
data existing on the Web, symbolic processing will need rich
representations to cater to various formats, high robustness to
handle errors or unknown data, and learning capabilities from
small data. Thus, in the subsequent paragraphs, we propose
methods with the following features which have not been
covered by the conventional methods.
1) There is no restriction on the number of terms included
in an atomic formula, regardless of the network config-
uration.
2) There is no restriction on the number of atomic for-
mulas included in a formula, regardless of the network
configuration.
3) There is no need to provide meta-rules to the network.
4) List structures can be used in atomic formulas.
5) Interpretation of inputs is possible even if they include
unknown atoms.
6) Models can be trained with small training data.
The fourth feature regarding handling of unknown data can
be attributed to the implementation of Word2Vec, which is
an important achievement in the recent years in the field of
neural network research. This method represents concepts
as vectors and thereby facilitates the estimation of similar-
ities between concepts. Therefore, similarities between the
known and unknown data can be used to resolve handling of
unknown data.
Furthermore, we show an application by embedding
learned models into question answering systems. Most of the
conventional question answering systems [14], [15] are not
entirely capable of inferring from a large amount of infor-
mation on the Web containing unknown data. The proposed
systems are designed to answer questions through powerful
inferences based on first-order predicate logic by utilizing
facts containing unknown data. By applying the proposed
systems, it will be possible to build applications such as
high-performance frequently asked questions (FAQ) chat-
bots [16], [17], decision support systems (DSS) [18], [19]
energy-efficient estimation in sensor networks [20]–[22]
using information on the Web. If these applications are real-
ized, users might be able to explore novel concepts or mine
hidden information.
Since connectionism in the 1990s, there have been very
few studies in which the research areas of conventional
artificial intelligence and neural network have intersected.
This is a new research area spanning conventional artificial
intelligence and deep learning. Furthermore, this research
aims to accomplish a Prolog-like processing system using
deep learning, and to our knowledge, this is a novel
application.
In this paper, we begin by reviewing related research work
in section II. In section III, we define and describe sym-
bolic processing which is the learning target. In section IV,
we propose learning methods for symbolic processing
and building methods of question answering systems.
Section V reports the experimental results of the proposed
methods.
VOLUME 7, 2019
II. RELATED WORK
A. SYMBOLIC PROCESSING WITH NEURAL NETWORKS
Before the emergence of deep learning, many studies have
attempted to train neural networks on symbolic processing
and use it for inference [23]. Additionally, studies have pre-
viously been conducted to learn propositional logic [24]–[26]
and first-order predicate logic [27]–[29] as well as to perform
unification [30], [31], similar to the present study. However,
these studies were limited to method proposal since they
could not be implemented.
With the emergence of deep learning, studies to learn sym-
bolic processing with graph networks [11], [12], [32]–[34]
and feedforward networks [13] have been performed and
documented. In the case of symbolic learning with graph
networks, it was necessary to provide preliminarily forms of
atomic formulas or formulas to networks and to presume that
forms of formulas were included in the data in advance. In the
case of symbolic learning with feedforward networks, forms
of atomic formulas and formulas depended on the network
configuration and it was necessary to provide meta-rules to
the network in advance. Our proposed methods levy no such
restrictions on the number of terms included in an atomic
formula or the number of atomic formulas included in a
formula, besides there is no need to give forms of the atomic
formulas to networks in advance.
B. QUESTION ANSWERING SYSTEMS
After the emergence of deep learning, studies about question
answering systems with deep learning [15], [35], [36] have
been conducted, and the performance of these systems have
improved.
These systems search for answer candidates from facts,
select an answer from the existing candidates, and respond.
Unlike the proposed system, these systems do not infer, but
merely answer the facts.
Additionally, studies have also been carried out to embed
questions in vector representations and infer answers by deep
learning [37]–[39]. However, even if unknown symbols have
been included in questions, to our knowledge, no studies have
previously embedded unknown symbols in internal repre-
sentations and inferred answers as proposed by the methods
described in this paper.
III. SYMBOLIC PROCESSING
Here, Prolog-like system is used for symbolic processing.
When a Prolog [40] processing system receives a question,
it refers to facts and rules stored in a knowledge base and
infers an answer. A question consists of one or more goals.
A Prolog processing system infers goals by backward rea-
soning. The following is a brief description of the operations
that a Prolog processing system performs to infer answers.
A. UNIFICATION
Unification is a process of determining whether two given
terms are identical. In Prolog programs, it is possible to
152369
 H. Honda, M. Hagiwara: Question Answering Systems With Deep Learning-Based Symbolic Processing
FIGURE 1. An example of unification.
FIGURE 2. An example of SLD resolution.
question whether two terms are identical by connecting the
two terms with the operator ‘‘=’’. If a term contains variables,
the variables are substituted for values which another term
indicates.
Fig. 1 shows an example of unification. If ‘‘male(tom)’’
and the right side of operator ‘‘=’’ are the same, a Pro-
log processing system answers ‘‘true.’’ If they are different,
it answers ‘‘false.’’ In the case of ‘‘male(X)’’ including a
variable, the variable is substituted and answers ‘‘X = tom.’’
B. RESOLUTION
Resolution is the process of resolving a new goal from a given
goal and rules. In Prolog, resolution called Selective Linear
Definite (SLD) limited to Horn clauses is performed. Horn
clause is a clause consisting of one or less positive literals.
Fig. 2 shows an example of SLD resolution. The rule
‘‘father(X, Y):- male(X), parent(X, Y).’’ is applied to the
goal ‘‘father(bob, tom).’’ and the new goal ‘‘male(bob), parent
(bob, tom).’’ is resolved.
C. BACKTRACK
A Prolog processing system takes facts and rules from a
knowledge base and performs unification and resolution.
If unification or resolution fails, then the system retraces steps
to the previous successful operation; subsequently, it identi-
fies new facts and rules and repeats unification and resolution.
This process executed by a Prolog processing system is called
backtrack.
Fig. 3 shows an example of backtrack; ‘‘male(X), parent(X,
tom).’’ is resolved from the rule ‘‘father(X, Y):- male(X),
parent(X, Y).’’ and the goal ‘‘father(X, tom).’’ Subsequently,
the goal ‘‘parent(X, tom)’’ matches the fact ‘‘parent(mary,
tom).’’ and it becomes ‘‘X = mary.’’ However, it fails because
there is no fact unification with the goal ‘‘male(mary).’’ hence
152370
FIGURE 3. An example of backtrack.
FIGURE 4. An example of membership relation.
backtracking is performed. When the facts ‘‘parent(bob,
tom).’’ is matched to goal, it becomes ‘‘X = bob.’’ because
the fact ‘‘male(bob).’’ exists, it is successful.
D. LIST OPERATIONS
Prolog can handle lists in data structures. Prolog programs
can perform operations such as adding new objects to a list
and joining two lists.
Fig. 4 shows an example of membership relation which
is one of the list operations. Membership relation is a pro-
cess of checking whether a specific object exists in the list.
The list ‘‘[colorado, nebraska, kansas]’’contains ‘‘kansas’’.
Thus, ‘‘member(kansas, [colorado, nebraska, kansas]).’’ is
equal to ‘‘true.’’ On the other hand, the list ‘‘[colorado,
nebraska, kansas]’’ does not contain ‘‘georgia’’. Thus, ‘‘mem-
ber(georgia, [colorado, nebraska, kansas]).’’ is equal to
‘‘false.’’
IV. PROPOSED DEEP LEARNING-BASED SYMBOLIC
PROCESSING AND QUESTION ANSWERING SYSTEMS
In this section, we describe the methods of learning symbolic
processing with deep learning. Subsequently, we describe
symbolic processing methods to build question answer sys-
tems from knowledge bases.
A. DEEP LEARNING-BASED SYMBOLIC PROCESSING
We propose methods to learn matching, resolution and
membership relations described in section II by com-
bining Neural Machine Translation (NMT) [41]–[43] and
VOLUME 7, 2019
H. Honda, M. Hagiwara: Question Answering Systems With Deep Learning-Based Symbolic Processing
FIGURE 5. Unification model.
Word2Vec [44]–[46]. NMT is a model that can learn to
convert one sequence into another. Unlike graph networks
[11], [12] and feedforward networks [13], NMT has no
restriction in the number of terms included neither in an
atomic formula nor in the number of atomic formulas
included in a formula. It is also not imperative to provide
to the network with meta-rules in advance. Furthermore,
Word2Vec is used to replace unknown atoms with semanti-
cally similar atoms.
1) LEARNING UNIFICATION
Fig. 5 shows a unification model used in the proposed sys-
tems. The input to this model is questions to determine pos-
sibility of performing the unification process. The model is
trained so that it outputs ‘‘true.’’ if it matches and ‘‘false.’’ if
it does not match. Although questions can include variables
unlike that in Prolog processing systems, these variables are
not substituted and only whether they are identical is deter-
mined. For example, ‘‘male(X) = male(tom).’’ should output
‘‘true.’’ instead of ‘‘X = tom.’’ After checking whether terms
are identical, substitution is performed outside the model.
When a query in the form of a word string is input to
the unification model, the embedding layer for the input
converts the string into a combined vector, a 300-dimensional
word embedding, of Word2Vec and Gray code [47]. Common
nouns of atoms such as ‘‘male’’ are Word2Vec word embed-
dings, however logical symbols such as ‘‘(‘‘, ’’)’’, ‘‘,’’, ‘‘.’’,
and proper nouns of atoms such as ‘‘bob’’ are Gray code
word embeddings. Gray code has a characteristic that the
Hamming distance between adjacent codes is 1. Although
the input could include words not contained in the training
data, output can be obtained, because Word2Vec is used for
embedding atoms.
Subsequently, the combined vector of Word2Vec and Gray
code embeddings is passed to NMT. The output from NMT
consists of the one-hot encoded word inserted in the embed-
ding layer, and also the unification result of the input word
string.
2) LEARNING RESOLUTION
Fig. 6 shows resolution model used in the proposed systems.
When the head of a rule is input to this model, the model
is trained so that it outputs the body of the rule. For exam-
VOLUME 7, 2019
FIGURE 6. Resolution model.
FIGURE 7. Example of increasing training data.
ple, in the rule ‘‘father(bob, mary):- male(bob), spouse(bob,
mary).’’, the input to the proposed model is ‘‘father(bob,
mary).’’ and the output is ‘‘male(bob), spouse(bob, mary).’’
When a word string forming a part of the head is input to
the resolution model, the embedding layer for input, converts
the string into a combined vector of Word2Vec and Gray
code word embeddings, as per conventions defined in the
unification learning processes. Subsequently, the combined
vector of Word2Vec and Gray code is passed to NMT. The
output from NMT consists of the one-hot encoded word
inserted in the embedding layer, and also the resolution result
of the word string forming the body.
When training resolution with NMT using the existing
knowledge base, the volume of training data may not be
sufficient in some cases. In such a case, a method to augment
the volume of proper noun data is applied. For example,
imaginary proper nouns such as ‘‘bob-1’’ and ‘‘bob-2’’ are
generated from ‘‘bob’’, and training data is increased as
shown in Fig. 7.
3) LEARNING MEMBERSHIP RELATION
Fig. 8 shows the membership relation model. The input to
this model is a membership related question such as ‘‘mem-
ber(kansas, [colorado, nebraska, kansas] ).’’ The model is
trained so that it outputs ‘‘true.’’ if the object exists in the list
and ‘‘false.’’ if it does not exist. Besides, questions to these
models do not have variables.
When a query in the form of a word string is input to
the membership relation model, the input embedding layer,
152371
 H. Honda, M. Hagiwara: Question Answering Systems With Deep Learning-Based Symbolic Processing

FIGURE 8. Membership relation model.

converts the string into a combined vector of Word2Vec and

Gray code word embeddings, as per conventions defined in FIGURE 9. Question answering system.
the unification learning processes. Subsequently, the com-
bined vector of Word2Vec and Gray code is passed to NMT.
The output from NMT consists of the one-hot encoded
word inserted in the embedding layer, and also the word
string, result of the object existence operation.
If the volume of training data is not sufficient, the volume
of training data is augmented in the same way as performed
for the resolution learning.

FIGURE 10. User interface.

B. QUESTION ANSWERING SYSTEMS WITH DEEP
LEARNING-BASED SYMBOLIC PROCESSING
We propose a method to build question answering sys-
tems using the model of symbolic processing detailed in
section IV-A. Users can construct questions-and-answers
with the question answering system by inputting texts. This
system supports yes/no-questions [48], what-questions [49],
and why-questions [50].
Fig. 9 shows the proposed question answering system.
When the user interface of the system receives questions
of texts inputted from users, it passes them to the Text-to-
Prolog model. Text-to-Prolog converts user questions from
text to Prolog and routes them to the inference engine. The
inference engine infers the answer to the question described
in Prolog using the facts stored in the Prolog knowledge base
and returns the result to Prolog-to-Text model. Prolog-to-Text
converts the system response described in Prolog into text
and routes it to the user interface. The user interface displays
this response to the user. Details of each component of the
question answering system are described below.
1) USER INTERFACE
Fig. 10 shows the user interface of the question answering
system. In this example, when a user inputs a question ‘‘Is
Bob a male?’’ in natural language, the question answering
system returns ‘‘Yes.’’ as the response in natural language.
After a series of questions-and-answers is completed, a user
can enter the next question.
Furthermore, if either the Text-to-Prolog or inference
engine provides erroneous grammar as input to Prolog sys-
tem, and the system cannot answer, then the user interface
handles the error gracefully and outputs a message, ‘‘I cannot
answer.’’
FIGURE 11. Some examples of translation from texts to Prolog.
2) TEXT-TO-PROLOG
Text-to-Prolog converts texts of user questions into Prolog.
Some examples of conversion by question patterns are shown
in Fig. 11. Yes/no-questions such as ‘‘Is Bob Tom’s father?’’
are converted to Prolog questions that do not use variables
such as ‘‘father( bob, tom ).’’ What-questions such as ‘‘Who
is Tom’s father?’’ are converted into ‘‘father( X, tom ).’’ by
replacing a question target with a variable. Why-questions
such as ‘‘Why is Bob Tom’s father?’’ are converted to Pro-
log questions with ‘‘trace’’ at the beginning such as ‘‘trace,
father( bob, tom ).’’ ‘‘trace’’ means a built-in predicate that
performs all tracing of goals.
Conversion from text to Prolog is learned by NMT.
By using deep learning, various user input questions
can be converted to their corresponding Prolog questions.
Fig. 12 shows the Text-to-Prolog model. The model is trained
so that when questions using regular text are input to this
model, it outputs the equivalent Prolog questions.
When the text string of word is input to Text-to-Prolog
model, the embedding layer for input, converts the string

152372 VOLUME 7, 2019

H. Honda, M. Hagiwara: Question Answering Systems With Deep Learning-Based Symbolic Processing
FIGURE 12. Text-to-Prolog model.
FIGURE 13. Inference algorithm.
into a vector of word embedding by using one-hot encoding
method. Subsequently, this vector is passed to NMT. The out-
put from NMT consists of the one-hot encoded word inserted
in the embedding layer for output, and also the resultant word
string of the Prolog program.
3) INFERENCE ENGINE
The inference engine performs symbolic processing using
the model learned in section IV-A. Fig. 13 shows the algo-
rithm by which the inference engine generates responses from
questions.
VOLUME 7, 2019
FIGURE 14. An example of translation from Prolog to texts.
In this algorithm, a goal list received from Text-to-Prolog is
unified, resolved, and membership relations checked by uni-
fication model, resolution model, and membership relation
model respectively, to obtain a new goal list. Even if unknown
atoms are included in the goal list received from Text-to-
Prolog, they are converted into a new goal list consisting
only of known atoms from unification model and resolution
model. After that, backtracking as performed in Prolog pro-
cess systems is performed on the new goal list. If ‘‘trace’’
is included in the goal list, abovementioned new goal list is
returned as response to the Why-questions.
4) PROLOG-TO-TEXT
Prolog-to-Text converts system response Prolog to regular
text. An example of conversion by question patterns is shown
in Fig. 14. Conversion from Prolog to text is done with rules.
C. NEURAL MACHINE TRANSLATION (NMT)
NMTs are models used in machine translation, which help to
facilitate conversion of one sequence into another. We apply
the widely used Seq2Seq with Attention [42] and Trans-
former [43] methods to the NMT used in our unification,
resolution, membership relation and Prolog-to-Text models.
Seq2Seq with Attention and Transformer are typical NMTs.
Unlike graph networks and feedforward networks, when
NMT is used, there is no restriction on the number of terms
included in an atomic formula and the number of atomic for-
mulas included in a formula, and meta rules are unnecessary.
This is because NMT can handle sequential data.
1) SEQ2SEQ
Seq2Seq with Attention [42] consists of three blocks viz., the
Encoder, Decoder and Attention. When the Encoder receives
an input sequence, it returns a compressed vector. The Atten-
tion block, based on the context of the output sequence, cal-
culates the degree of attention to be given to each word in the
input sequence. The compressed vector is then weighted by
attention and added. When the Decoder receives the resulting
vector output from the Encoder and Attention, it generates an
output sequence.
The Encoder and Decoder are composed of Long Short-
Term Memory (LSTM) [51], which is a type of Recurrent
Neural Network (RNN) [52]. LSTM can handle sequential
data with long-term dependencies that cannot be learned
152373
 H. Honda, M. Hagiwara: Question Answering Systems With Deep Learning-Based Symbolic Processing

with conventional RNNs. The Encoder applies Bi-LSTM

that uses future as well as past information. Bi-LSTM has
128-dimensional output and three layers. The Decoder
applies Stateless LSTM that does not take over short-term
memory. Stateless LSTM has 128-dimensional output and
uses Maxout [53] as the activation function.
The dropout rate is 0.1, the batch size is 128 and 20 epochs
are trained. For optimization, we use the Adam optimizer [54]
with α = 0.001, β1 = 0.9, β2 = 0.999, eps = 1e-08.

2) TRANSFORMER
Transformer [43] consists of two blocks, an Encoder Stack FIGURE 15. Seq2Seq with attention.
and a Decoder Stack. The Encoder Stack receives an input
sequence and the Decoder Stack returns the output sequence.
The Encoder and Decoder Stacks each have 6 Encoders and
6 Decoders.
The Encoder consists of a Feedforward and a Self-
Attention, 512-dimensional output and six layers. The
Decoder consists of a Feedforward, an Encoder-Decoder
Attention and a Self-Attention, 512-dimensional output and
six layers.
The Self-Attention is used to relate different positions
of a single sequence by computing a representation of the
sequence. The Encoder-Decoder Attention helps the Decoder
to focus on the appropriate parts of the input sequence.
The Feedforward uses Leaky ReLU [55] for the activation
function.
The dropout rate is 0.1, the batch size is 48 and 100 epochs
are trained. For optimization, we use Adam optimizer [53]
with α = 5e-5, β1 = 0.9, β2 = 0.98, eps = 1e-9.

D. WORD2VEC
Word2Vec is a method for obtaining the vector representa-
tion of each word from a large amount of text data using
a neural network. We apply pre-trained Word2Vec called
GoogleNews-vectors-negative300 [40]–[42] to the unifica-
tion, resolution and membership relation models.
GoogleNews-vectors-negative300 is trained on a dataset
of approximately 100 billion words. The model has
300-dimensional vectors for 3 million words and phrases.
FIGURE 16. Transformer.

V. EVALUATION EXPERIMENTS
Using knowledge bases described in Prolog, we trained mod-
els, built question answering systems, and evaluated their
performance. Specifically, we conducted experiments using
two kinds of knowledge bases with graph structures.

A. EXPERIMENTS USING KINSOURCES FIGURE 17. An example of kinsources knowledge base.

Kinsources [56] is a collection of data representing blood

relationships, and this knowledge is presented graphically.
We used the data set to build the knowledge base for our
experiment. Fig. 17 shows an example of the knowledge
base. The knowledge base described in Prolog consists
of 5,887 atoms and 10 kinds of predicates, excluding list
structures.
Based on this knowledge base, training data are gener-
ated. Because the number of training data of Kinsources
is sufficiently large, we do not augment the data. The
training data of Text-to-Prolog includes Yes/no-questions,
What-questions which have interrogative pronouns for sub-
jects, What-questions which have interrogative pronouns for

152374 VOLUME 7, 2019

H. Honda, M. Hagiwara: Question Answering Systems With Deep Learning-Based Symbolic Processing
TABLE 1. Results of models by kinsources.
TABLE 2. Result of unification model with unknown data.
objects, and Why-questions, and each of which prepares two
patterns of wording.
The learning results of these models are shown in Table 1.
The test data in Table 1 does not contain the case with
unknown data. A correct answer rate is calculated as follows.
rate = num of exact matches / num of Test data
We can compute each correct answer rate when Seq2Seq and
Transformer are applied to NMT. Furthermore, we conducted
similar experiments with the conventional Prolog system to
derive a correct answer rate, which was then compared to the
figures generated by the proposed systems.
Any unknown data provided as test data input to the uni-
fication models were replaced as shown in Table 1 with
‘‘male’’ as ‘‘man’’ and ‘‘female’’ as ‘‘woman’’. The results
of unification using unknown data are shown in Table 2.
Any unknown data provided as test data input to the resolu-
tion model were replaced as shown in Table 1 with ‘‘father’’
as ‘‘dad’’ and ‘‘mother’’ as ‘‘mom’’. The results of resolution
using unknown data are aggregated in Table 3.
The question answering system was built using each model
as shown in Table 1. Since Seq2Seq has better performance
than Transformer, it was applied to NMT of the question
answering system. In the experiments, questions were gen-
erated for each combination of question and response types
and provided to the question answering system. There are
two kinds of response types, Positive and Negative. Posi-
tive is the case where responses such as ‘‘Yes.’’ for Yes/no-
questions, ‘‘Bob.’’ for What-questions or Why-questions are
obtained. Negative is the case where responses such as ‘‘No’’
for Yes/no-questions, and no valid response in case What-
questions or Why-questions are obtained. Questions included
unknown data, as shown in Tables 2 and 3, and they were
VOLUME 7, 2019
TABLE 3. Results of resolution model with unknown data.
TABLE 4. Results of question answering system.
(1)
FIGURE 18. An example of geoquery knowledge base.
randomly generated with the wording of two patterns for
each question type. This was done with an understanding
that if questions are randomly generated without dividing the
question types, then Negative questions will largely occupy
the dataset.
Then, we decided to carry out the experiment separately
for Positive and Negative questions. Table 4 shows the correct
answer rate when questions are input to the question answer-
ing system.
B. EXPERIMENTS USING GEOQUERY
Geoquery [57] is a database that represents the United States
geography using graph structures. We have used the data set
to build the knowledge base for our experiment. Fig. 18 shows
an example of the knowledge base. The knowledge base,
written in Prolog consists of 574 atoms and 8 kinds of pred-
icates and includes list structures in their bodies. Further-
more, as shown in rules having their predicates of heads as
‘‘located’’ (Fig. 18), the representation of the knowledge base
also contains disjunctions.
152375
 H. Honda, M. Hagiwara: Question Answering Systems With Deep Learning-Based Symbolic Processing
TABLE 5. Results of models by geoquery.
Based on this knowledge base, training data of models
were generated. Due to the small volume of training data
available for the resolution model and membership relation
model, the data were augmented. In the training data of the
resolution model, lists were replaced with the variable ‘‘L’’.
This was the way for making learning easier, wherein lists
could be uniquely determined from logical symbols appear-
ing in rules.
The training data of Text-to-Prolog included Yes/no-
questions, and they were generated with the wording of two
patterns.
The learning results of these models are shown in Table 5.
The test data in Table 5 does not contain the case with
unknown data. We calculated each correct answer rate when
Seq2Seq and Transformer were applied to NMT. Further-
more, we conducted similar experiments with conventional
Prolog system to derive a correct answer rate, which was then
compared to the figures generated by the proposed systems.
Any unknown data provided as test data input to the
resolution model were replaced as shown in Table 5 with
‘‘adjacent’’ as ‘‘adjoining’’ and ‘‘located’’ as ‘‘situated’’. The
results of resolution using unknown data are shown in Table 4.
The question answering system was developed using each
model as illustrated in Table 7. Since Seq2Seq has better
performance than Transformer, it was applied to NMT of
the question answering system. In the experiments, ques-
tions were generated for each combination of question and
response types and provided to the question answering sys-
tem. Question type is one kind of Yes/no-questions. There are
two kinds of response types, Positive and Negative. Positive is
the case where responses such as ‘‘Yes.’’ for Yes/no-questions
are obtained. Negative is the case where responses such as
‘‘No’’ for Yes/no-questions are obtained. Questions included
unknown data, as shown in Table 6, and they were randomly
generated with the wording of two patterns for each question
type. Table 7 shows the correct answer rate when questions
are input to the question answering system.
C. DISCUSSION
Experimental results on the learning of the symbolic process-
ing revealed that the performance was better when Seq2Seq
152376
TABLE 6. Results of resolution model with unknown data.
TABLE 7. Results of question answering system.
was applied than when Transformer was applied to NMT.
With Seq2Seq the correct answer rate of the unification model
was 1.00 and the correct answer rate of the resolution model
was 0.823 or more. We did not restrict primitive logic and
logical formulas, nor did we specify meta-rules in advance for
any of the models and still realized a high correct answer rate.
In the membership relation model with Seq2Seq, the correct
answer rate was 0.917, and data structures including lists
were handled. Therefore, the proposed methods have rich
representations for handling various data structures.
With Seq2Seq the correct answer rate of the unification
model for unknown data was 0.839 or more and the correct
answer rate of the resolution model was 0.727 or more. As for
the membership relation model, because elements of lists
were only proper nouns, we were unable to measure the
correct answer rate of unknown data in this experiment. How-
ever, because the membership relation model employs the
same learning model as those for unification and resolution,
we can infer that the results will apply to unknown data as
well. Thus, by using Word2Vec, this proposed method pro-
vides high robustness for handling unknown data in inputs.
Data that is originally available for learning of the resolu-
tion model in the Geoquery dataset is 496, and data that
can be used for learning the membership relation model is
1,256. These are very small volumes of data. However, with
Seq2Seq the correct answer rates are 0.823 and 0.913, respec-
tively, and the proposed method shows that the resolution
model can learn even from small-scale dataset.
In addition, comparing the proposed systems with the
conventional Prolog system, it is evident that the former
can handle unknown data. Since the conventional Prolog
system has manually built-in rules, when using known data,
the rate of unification, resolution, and membership relation
was 1.00. However, when using unknown data, the rate was
0.00. Although the proposed systems do not have a rate that
is less than 1.0 for known data, not only can it achieve high
performance, but it can also handle unknown data.
In the experimental results of the question answering
systems, the minimum value of the correct answer rate is
VOLUME 7, 2019
H. Honda, M. Hagiwara: Question Answering Systems With Deep Learning-Based Symbolic Processing
0.614 and the maximum value is 1.00. Although it varies
depending on the question types and response types, practical
question answering systems could likely be built based on the
existing knowledge bases. We do not learn the case where res-
olution is impossible in the resolution model incorporated in
the system this time. If it becomes possible to judge whether
or not resolution is impossible in the resolution model, there
is a possibility that the correct answer rate when inputting
‘‘Negative of What-questions’’ which is the lowest this time
is improved.
VI. CONCLUSION AND FUTURE WORK
In this paper, we have proposed the methods to learn symbolic
processing using deep learning and the methods to build ques-
tion answering systems using the trained model. Experimen-
tal results on the training of the symbolic processing models
show that the proposed methods have rich representations and
high robustness and that these models can learn even from
small-scale data. Especially by using Word2Vec, the ability to
handle unknown data will be a great contribution to artificial
intelligence research. Moreover, experimental results of the
question answering systems suggested that practical question
answering systems could be built from the knowledge bases
written in Prolog. Building such systems will be extremely
difficult using a conventional connectionism-based method.
This study pertains a new research domain that spans
the areas of conventional artificial intelligence and neural
networks. We conducted experiments using simple datasets.
However, through the experimental results, we recommend
its potential applications in areas such as FAQ chatbots, DSS
and energy-efficient estimation in sensor networks. If these
applications can respond to information not only by facts but
also by inferring from a large amount of information on the
Web, the proposed systems might be able to contribute great
value to society.
Future work includes symbolic processing to analyze
large-scale data present on the Web and inductive inference
with deep learning-based symbolic processing.
REFERENCES
[1] R. K. Lindsay, B. G. Buchanan, E. A. Feigenbaum, and J. Lederberg,
Applications of Artificial Intelligence for Organic Chemistry: The Dendral
Project. New York, NY, USA: McGraw-Hill, 1980.
[2] J. R. Quinlan, ‘‘Induction of decision trees,’’ Mach. Learn., vol. 1, no. 1,
pp. 81–106, 1986.
[3] S. Muggleton, ‘‘Inductive logic programming,’’ New Gener. Comput.,
vol. 8, no. 4, pp. 295–318, Feb. 1991. [Online]. Available: http://www.
doc.ic.ac.uk/~shm/Papers/ilp.pdf. doi: 10.1007/BF03037089.
[4] G. Brewka, Nonmonotonic Reasoning: Logical Foundations of Common-
sense. Cambridge, U.K.: Cambridge Univ. Press, 1991.
[5] J. Doyle, ‘‘The ins and outs of reason maintenance,’’ in Proc. 8th Int. Joint
Conf. Artif. Intell. (IJCAI), Los Altos, CA, USA, 1983, pp. 349–351.
[6] A. C. Kakas, R. A. Kowalski, and F. Toni, ‘‘Abductive logic programming,’’
J. Log. Comput., vol. 2, no. 6, pp. 719–770, Dec. 1993. doi: 10.1093/log-
com/2.6.719.
[7] G. E. Hinton, ‘‘Preface to the special issue on connectionist symbol pro-
cessing,’’ Artif. Intell., vol. 46, nos. 1–2, pp. 1–4, Nov. 1990.
[8] D. S. Touretzky, ‘‘BoltzCONS: Dynamic symbol structures in a connec-
tionist network,’’ Artif. Intell., vol. 46, nos. 1–2, pp. 5–46, Nov. 1990.
VOLUME 7, 2019
[9] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg, I. Titov, and
M. Welling, ‘‘Modeling relational data with graph convolutional net-
works,’’ in Proc. Eur. Semantic Web Conf. (ESWC), 2018, pp. 593–607.
[10] T. Trouillon, J. Welbl, S. Riedel, E. Gaussier, and G. Bouchard, ‘‘Complex
embeddings for simple link prediction,’’ in Proc. 33nd Int. Conf. Mach.
Learn. (ICML), New York, NY, USA, 2016, 2016 pp. 2071–2080.
[11] T. Rocktaschel and S. Riedel, ‘‘End-to-end differentiable proving,’’ in
Proc. Annu. Conf. Neural Inf. Process. Syst., 2017, pp. 3788–3800.
[12] P. Minervini, M. Bosnjak, T. Rocktäschel, and S. Riedel, ‘‘Towards neural
theorem proving at scale,’’ 2018, arXiv:1807.08204. [Online]. Available:
https://arxiv.org/abs/1807.08204
[13] H. Dong, J. Mao, T. Lin, C. Wang, L. Li, and D. Zhou, ‘‘Neural logic
machines,’’ in Proc. Int. Conf. Learn. Represent., New Orleans, LA, USA,
2019, pp. 1–22.
[14] S. Tellex, B. Katz, J. Lin, G. Marton, and A. Fernandes, ‘‘Quantitative
evaluation of passage retrieval algorithms for question answering,’’ in Proc.
26th Annu. Int. ACM SIGIR Conf. Res. Develop. Inf. Retr., Toronto, ON,
Canada, Aug. 2003, pp. 41–47.
[15] R. Sequiera, G. Baruah, Z. Tu, S. Mohammed, J. Rao, H. Zhang, and J. Lin,
‘‘Exploring the effectiveness of convolutional neural networks for answer
selection in end-to-end question answering,’’ 2017, arXiv:1707.07804.
[Online]. Available: https://arxiv.org/abs/1707.07804
[16] N. T. Thomas, ‘‘An e-business chatbot using AIML and LSA,’’ in Proc. Int.
Conf. Adv. Comput., Commun. Inform. (ICACCI), Jaipur, India, Sep. 2016,
pp. 2740–2742.
[17] L. Cui, F. Wei, S. Huang, C. Tan, C. Duan, and M. Zhou, ‘‘SuperA-
gent: A customer service chatbot for e-commerce websites,’’ in Proc.
55th Annu. Meeting Assoc. Comput. Linguistics-Syst. Demonstrations,
Jul. 2017, pp. 97–102.
[18] H. Bhargava and D. Power, ‘‘Decision support systems and Web technolo-
gies: A status report,’’ in Proc. Amer. Conf. Inf. Syst., Boston, MA, USA,
Dec. 2001, p. 46.
[19] M. S. Kohn, J. Sun, S. Knoop, A. Shabo, B. Carmeli, D. Sow,
T. Syed-Mahmood, and W. Rapp, ‘‘IBM’s health analytics and clini-
cal decision support.,’’ Yearbook Med. Inf., vol. 9, no. 1, pp154-162,
Aug. 2014.
[20] A. Sodhro, Y. Li, and M. Shah, ‘‘Energy-efficient adaptive transmission
power control for wireless body area networks,’’ IET Commun., vol. 10,
no. 1, pp. 81–90, Jan. 2016.
[21] A. Sodhro, S. Pirbhulal, M. Lodro, and M. Shah, ‘‘Energy-efficiency in
wireless body sensor networks,’’ in Networks of the Future Architectures,
Technologies, and Implementations. Boca Raton, FL, USA: CRC Press,
2017, p. 492.
[22] A. Sodhro, A. Sangaiah, G. Sodhro, A. Sekhari, Y. Ouzrout, and
S. Pirbhulal, ‘‘Energy-efficiency of tools and applications on Internet,’’
in Computational Intelligence for Multimedia Big Data on the Cloud
with Engineering Applications (Intelligent Data-Centric Systems: Sensor
Collected Intelligence). Amsterdam, The Netherlands: Elsevier, 2018.
[23] A. S. d’Avila Garcez, K. Broda, and D. M. Gabbay, Neural-Symbolic
Learning Systems: Foundations and Applications. London, U.K.:
Springer-Verlag, 2002.
[24] J. W. Shavlik and G. G. Towell, ‘‘An approach to combining explanation-
based and neural learning algorithms,’’ Connection Sci., vol. 1, no. 3,
pp. 231–253, 1989.
[25] G. G. Towell and J. W. Shavlik, ‘‘Knowledge-based artificial neural net-
works,’’ Artif. Intell., vol. 70, nos. 1–2, pp. 119–165, Oct. 1994. doi:
10.1016/0004-3702(94)90105-8.
[26] A. S. A. Garcez and G. Zaverucha, ‘‘The connectionist inductive learning
and logic programming system,’’ Appl. Intell., vol. 11, no. 1, pp. 59–77,
Jul. 1999. doi: 10.1023/A:1008328630915.
[27] L. Shastri, ‘‘Neurally motivated constraints on the working memory capac-
ity of a production system for parallel processing: Implications of a
connectionist model based on temporal synchrony,’’ in Proc. 14th Annu.
Conf. Cognit. Sci. Soc. Bloomington, IN, USA: Psychology Press, vol. 14,
Jul./Aug. 1992, p. 159.
[28] L. Ding, ‘‘Neural prolog-the concepts, construction and mechanism,’’ in
Proc. IEEE Int. Conf. Syst., Man Cybern., Intell. Syst. 21st Century, vol. 4,
Oct. 1995, pp. 3603–3608.
[29] M. V. M. Franca, G. Zaverucha, and A. S. d’Avila Garcez, ‘‘Fast relational
learning using bottom clause propositionalization with artificial neural
networks,’’ Mach. Learn., vol. 94, no. 1, pp. 81–104, Jan. 2014. doi:
10.1007/s10994-013-5392-1.
152377
 H. Honda, M. Hagiwara: Question Answering Systems With Deep Learning-Based Symbolic Processing
[30] E. Komendantskaya, ‘‘Unification neural networks: Unification by error-
correction learning,’’ Log. J. IGPL, vol. 19, no. 6, pp. 821–847, Dec. 2011.
doi: 10.1093/jigpal/jzq012.
[31] S. Holldobler, ‘‘A structured connectionist unification algorithm,’’ in Proc.
8th Nat. Conf. Artif. Intell., Boston, MA, USA, vol. 2, 1990, pp. 587–593.
[32] G. Sourek, V. Aschenbrenner, F. Zelezny, and O. Kuzelka, ‘‘Lifted rela-
tional neural networks,’’ in Proc. Int. Conf. Cogn. Comput., Integrating
Neural Symbolic Approaches, Montreal, QC, Canada, 2015.
[33] W. W. Cohen, ‘‘Tensorlog: A differentiable deductive database,’’ 2016,
arXiv:1605.06523. [Online]. Available: https://arxiv.org/abs/1605.06523
[34] L. Serafini and A. S. d’Avila Garcez, ‘‘Logic tensor networks: Deep
learning and logical reasoning from data and knowledge,’’ in Proc. 11th
Int. Workshop Neural-Symbolic Learn. Reasoning (NeSy), New York, NY,
USA, 2016, pp. 1–12.
[35] T. Lai, T. Bui, S. Li, and N. Lipka, ‘‘A simple end-to-end question answer-
ing model for product information,’’ in Proc. 1st Workshop Econ. Natural
Lang. Process., Jul. 2018, pp. 38–43.
[36] Y. Tay, L. A. Tuan, and S. C. Hui, ‘‘Hyperbolic representation learning for
fast and efficient neural question answering,’’ in Proc. 11th ACM Int. Conf.
Web Search Data Mining, Los Angeles, CA, USA, Feb. 2018, pp. 583–591.
[37] B. Peng, Z. Lu, H. Li, and K. Wong, ‘‘Towards neural network-based
reasoning,’’ 2015, arXiv:1508.05508. [Online]. Available: https://arxiv.
org/abs/1508.05508
[38] D. Weissenborn, ‘‘Separating answers from queries for neural read-
ing comprehension,’’ 2016, arXiv:1607.03316. [Online]. Available:
https://arxiv.org/abs/1607.03316
[39] Y. Shen, P. Huang, J. Gao, and W. Chen, ‘‘Reasonet: Learning to stop
reading in machine comprehension,’’ in Proc. 23rd ACM SIGKDD Int.
Conf. Knowl. Discovery Data Mining, Barcelona, Spain, Aug. 2017,
pp. 1047–1055.
[40] I. Bratko, Prolog Programming for Artificial Intelligence. 2nd ed. Reading,
MA, USA: Addison-Wesley, 1990, p. 597.
[41] I. Sutskever, O. Vinyals, and Q. V. Le, ‘‘Sequence to sequence learning
with neural networks,’’ in Proc. NIPS, Montreal, QC, Canada, 2014,
pp. 3104–3112.
[42] D. Bahdanau, K. Cho, and Y. Bengio, ‘‘Neural machine translation by
jointly learning to align and translate,’’ in Proc. ICLR, San Diego, CA,
USA, 2015, pp. 1–15.
[43] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez,
L. Kaiser, and I. Polosukhi, ‘‘Attention Is All You Need,’’ in Proc.
31st Conf. Neural Inf. Process. Syst., Long Beach, CA, USA, 2017,
pp. 5998–6008.
[44] T. Mikolov, K. Chen, G. Corrado, and J. Dean, ‘‘Efficient estimation of
word representations in vector space,’’ in Proc. ICLR, Scottsdale, AZ,
USA, 2013, pp. 1–12.
[45] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, ‘‘Distributed
representations of words and phrases and their compositionality,’’ in Proc.
NIPS, Lake Tahoe, NV, USA, 2013, pp. 3111–3119.
[46] T. Mikolov, W. Yih, and G. Zweig, ‘‘Linguistic regularities in continuous
space word representations,’’ in Proc. NAACL HLT, Atlanta, GA, USA,
2013, pp. 746–751.
[47] F. Gray, ‘‘Pulse code communication,’’ U.S. Patent 2 632 058 A,
Mar. 17, 1953.
[48] H. Kanayama, Y. Miyao, and J. Prager, ‘‘Answering Yes/no questions via
question inversion,’’ in Proc. 24th Int. Conf. Comput. Linguistics, Mumbai,
India, Dec. 201, pp. 1377–1392.
[49] D. Ravichandran and E. Hovy, ‘‘Learning surface text patterns for a
question answering system,’’ in Proc. 40th Annu. Meeting Assoc. Comput.
Linguistics, Jul. 2002, pp. 41–47.
[50] R. Higashinaka and H. Isozaki, ‘‘Corpus-based question answering for
why-questions,’’ in Proc. IJCNLP, Hyderabad, India, 2008, pp. 418–425.
152378
[51] S. Hochreiter and J. Schmidhuber, ‘‘Long short-term memory,’’ Neural
Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
[52] J. L. Elman, ‘‘Finding structure in time,’’ Cognit. Sci., vol. 14, no. 2,
pp. 179–211, Mar. 1990.
[53] I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio,
‘‘Maxout networks,’’ in Proc. 30th Int. Conf. Mach. Learn., Atlanta, GA,
USA, 2013, pp. 1–9.
[54] D. Kingma and J. Ba, ‘‘Adam: A method for stochastic optimiza-
tion,’’ 2014, arXiv:1412.6980. [Online]. Available: https://arxiv.org/
abs/1412.6980
[55] A. Maas, A. Hannu, and A. Ng, ‘‘Rectifier nonlinearities improve neural
network acoustic models,’’ in Proc. 30th Int. Conf. Mach. Learn., Atlanta,
GA, USA, 2013, p. 3.
[56] Kinsources: A Collaborative Web Platform for Kinship Data Sharing.
Accessed: May 19, 2018. [Online]. Available: https://www.kinsources.net/
[57] L. R. Tang and R. J. Mooney, ‘‘Automated construction of database inter-
faces: Integrating statistical and relational learning for semantic parsing,’’
in Proc. SIGDAT Conf. Empirical Methods Natural Lang. Process. Very
Large Corpora (EMNLP/VLC), Hong Kong, Oct. 2000, pp. 133–141.
HIROSHI HONDA received the B.E. and M.E.
degrees in administration engineering from Keio
University, Yokohama, Japan, in 2003 and 2005,
respectively, where he is currently pursuing
the Ph.D. degree in information and computer
science.
From 2005 to 2014, he was a Software Engi-
neer with Mitsubishi Electric Corporation. From
2014 to 2016, he was a Software Engineer with
Fuji Xerox Company Ltd. Since 2017, he has been
a Researcher with Honda R&D Company Ltd. His research interests include
symbolic processing using deep learning and dialogue systems.
MASAFUMI HAGIWARA (M’89–SM’04)
received the B.E., M.E., and Ph.D. degrees
in electrical engineering from Keio University,
Yokohama, Japan, in 1982, 1984, and 1987,
respectively. Since 1987, he has been with Keio
University, where he is currently a Professor. From
1991 to 1993, he was a Visiting Scholar with
Stanford University. His research interests include
neural networks, fuzzy systems, and affective
engineering. He is a member of IEICE, IPSJ, JSAI,
SOFT, IEE of Japan, and JNNS. He received from the IEEE Consumer
Electronics Society Chester Sall Award, in 1990, the Author Award from
Japan Society of Fuzzy Theory and Systems, in 1996, the Technical Award
and Paper Awards from Japan Society of Kansei Engineering, in 2003, 2004,
and 2014, respectively, and the Best Research Award from Japanese Neural
Network Society, in 2013. He was the President of the Japan Society for
Fuzzy Theory and Intelligent Informatics (SOFT).
VOLUME 7, 2019