Semantic Recommendation by a Modular Recurrent
Neural Network
Joern David
Institute of Computer Science / I1
Technical University Munich
D-85748 Garching, Germany
email: david@in.tum.de
Keywords:
Recurrent neural networks, text mining, information
retrieval, sequence prediction, dynamic systems, semantic
recommendation.
Abstract— The central problem that is addressed by this
paper is to learn and exploit the intrinsic associations be-
tween knowledge repositories such as document trees, which
semantically belong to one knowledge domain, but are not
explicitly linked together. This class of information retrieval (IR)
and inference problems is characterized by a lack of explicit
associations between the independent knowledge repositories. To
comprehensively understand the knowledge in a domain it is
important to associate knowledge fragments, which have been
created independently.
Thus we focus on the hidden semantic relationships between cor-
related nodes that occur in the involved repositories. We conduct
a combined approach of machine learning, formal grammars
and data mining methods to provide semantic recommendation
functionality upon existing graph-based knowledge bases. The
recommendation is a prediction of a sequence of nodes based
on the recent user behaviour on the knowledge graph. Domain
experts are observed while they navigate through these graphs
and therefore implicitly associate nodes, which have a high
cohesion (since they altogether describe the steps of a certain
business process, for example). These associations together with
some automatically discovered association rules are learned by a
Modular Recurrent Neural Network (MRNN) in a supervised train
process. This recurrent network model represents a dynamic
system that has a novel modular composition and adapts at
runtime to the structure of the variably long train patterns.
In turn new knowledge in form of associations between the
knowledge bases can be inferred from the internal network state,
such that a domain beginner can be recommended proactively to a
sequence of relevant nodes – due to his automatically recognized
navigation history.
I. I NTRODUCTION
Research in artificial intelligence always dealt with sym-
bolic knowledge representations on the one side and with
subsymbolic or connectionistic representations on the other
side. Symbolic or rule-based techniques like expert systems
benefit from their strict systematics and high interpretability.
Symbols are arbitrary, but can be composed to meaningful
symbolic structures like formal grammars according to the
Chomsky-hierarchy or rules in an expert system.
The connectionistic paradigm has the powerful concept of
artificial neural networks at its disposal, which are universally
applicable to every problem that can meaningfully be repre-
sented by a set of vectors {~v1 , . . . , ~vn }, ~vi ∈ Rd .
We believe that a more powerful form of artificial intel-
ligence can be achieved by combining the symbolic with
the connectionistic paradigm. Thus we want to construct
connectionistic models (e.g. neural networks) that behave
as symbol processors and which autonomously develop an
understanding of their environment. This implies that symbols
have to be anchored in the connectionistic representation: A
symbol is nothing without its creator.
On this basis an intelligent hybrid system of both paradigms
can evolve. This article describes such an approach on the
concrete domain of graph-based knowledge structures such as
document trees, that are implicitly correlated to each other.
The behaviour of experts is observed in order to obtain
association rules on the knowledge bases, that are used for
supervised training of the recurrent neural network.
A main benefit of this connectionistic machine learning unit
is its inherent generalization capability. Also not explicitly
learned but meaningful association rules can be inferred from
the internal representation space of the novel recurrent neural
network MRNN. Thus new knowledge in form of associations
between the knowledge bases can be derived from the internal
network state. This knowledge should be provided to the
system actor proactively as a recommendation of a sequence
of target nodes in the respective knowledge base.
II. R ELATED WORK
Since our semantic recommendation approach is composed
by concepts of dynamic systems, formal grammars and text
mining, a brief excerpt concerning relevant research findings
in these areas is outlined.
A. Context prediction
[MRF04] states: “The future states q̄t+1 , q̄t+2 , . . . q̄t+m
for m discrete time steps are predicted recursively by some
arbitrarily complex process q̄t+i = p hqt , q̄t+1 , . . . q̄t+i−1 i;”
This is exactly what we propose in this article, while the
“complex process” is realized by a modular recursive neu-
ral network, that computes the future state q̄t+i based on
all previous states qt−k , . . . , qt , q̄t+1 , . . . q̄t+i−1 . According
to [MRF04], this prediction process enables proactivity in
general. In a proactive system, the system output at time t
can also depend on the predicted future states q̄t+1 , . . . q̄t+i−1 .
This implies that the system states qi are representing a context
and the complex process p is an abstract state machine. “Thus
interpreting the context changes as a state trajectory allows to
forecast the future development of the trajectory, and therefore
to predict the anticipated context.”
This interpretation exactly describes the goal of our MRNN
prediction model with its internal state layer, that represents
the mentioned state trajectory.
B. Formal grammars incorporated by neural networks
Formal languages that are generated by formal grammars
can be learned by artificial neural networks. Recurrent neural
networks (RNNs) have already been applied to internalize the
explicit rules of formal grammars, such that they were capable
to classify or predict words ω ∈ L of the generated language L
(cp. [Smi03]). These words are sequences of terminal symbols
of the underlying alphabet Σ. So different types of RNNs
like Elman Simple Recurrent Networks (SRN, e.g. [JMS]) or
Jordan Nets were utilized to classify the words in terms of
the membership ω ∈ L or ω ∈ / L (the word problem) [Cal03].
By contrast, the prediction task is to continue the symbol
sequence ω̌ = a1 . . . ak , ai ∈ Σ for a given partial word
ω̌, ω = ω̌ ω̂, such that the predicted sequence of subsequent
symbols ω̂ = ak+1 . . . an is conform with the rules of the
grammar.
Many languages that are processed by neural networks orig-
inate from the regular grammars (Chomsky type 3), that can
be represented by a Finite State Machine (FSM).
C. Information Retrieval
[MS05] proposes a Neural Network Information Retrieval
(IR) system, that consists of three layers namely query,
keyword and document layer. This system should enable
information retrieval from text documents in Slovak language.
Here the specialty is the use of neural networks for the
“transition of the information” between each of these layers.
The goal is to associate actor queries with a set of keywords,
which is itself associated with the actual document set, while
the associations between the layers are established by two
neural networks. Like proposed in the paper at hand, [MS05]
Fig. 1. Three layer information retrieval concept enabled by a
common feed-forward neural network architecture.
also wants to exploit the memorization and generalization
capabilities of artificial neural networks in order to associate
significant terms (keywords) with user queries on the one hand
side and with their occurrences in documents on the other hand
side. However the focus of this IR system is not on machine
learning
III. M ODELING NAVIGATION SEQUENCES BY A FORMAL
GRAMMAR
In order to associate sequences of nodes in knowledge bases
we want to model navigation operations on these nodes by a
formal grammar.
Def. Components of a formal grammar
• Σ is the alphabet of terminal symbols (lowercase letters).
• V is the set of variables or non-terminal symbols, V ∩Σ =
∅ (capital letters).
• P is the set of production rules, which is a finite subset:
P ⊂ (V ∪ Σ)+ × (V ∪ Σ)∗ .
∗
is the Cleene star, that stands for an arbitrary number
0..n of single symbol occurrences (+ stands for 1..n).
Now a formal grammar is a 4-tuple G = (V, Σ, P, S), where
S ∈ V is the start symbol.
It is proven that neural networks with one hidden layer and
non-linear activation function (e.g. logistic function, tanh) are
at least as powerful and expressive than the general Turing
machine.1
So context-sensitive rules like αBγ → αβγ, B ∈ V, α, β, γ ∈
(V ∪Σ)∗ can be recognized by neural networks with the appro-
priate learning algorithm theoretically. The expressiveness of
context-sensitive grammars is very high, thus it is in practice
very difficult to conduct context sensitive language learning.
Nevertheless also very complex grammars are expressible by
RNNs. For our purpose it is sufficient to train a recurrent
neural network in a way, that it behaves like a Finite State
Machines (FSM).
A FSM can only generate or respectively accept the regu-
lar languages, according to the Chomsky-hierarchy [Sch01].
These are generated by the grammars of type three in this
hierarchy, whose regular production rules – which are exactly
equivalent to the regular expressions – are the least expressive
ones, but sufficient to describe all relevant actor traces in the
recommendation system.
But also with the restrictions of regular grammars we can
leverage the context of an actor trace in form of the previously
visited nodes ab1 b2 . . . bj c, which altogether represent the
navigation-history for the generic knowledge tree depicted
in figure 2. In that case, the target sequence is represented
by the symbol sequence d1 . . . dk ef g. As a consequence,
the path through the knowledge tree generates a word ω,
that is composed of a sequence of terminal symbols and
ω ∈ L(G) ∈ L3 is derived from the following regular grammar
1 The power of neural networks could even exceed the power of Turing
machines, due to the massive parallelism of information processing (MPP)
by the interconnected neurons.

Fig. 2. Sequence of navigation operations represented by a word
of a formal language L(G) – according to the regular grammar G
given above. The nodes of the knowledge tree represent the terminal
symbols (lower case letters) of the underlying alphabet Σ associated
with L. A single actor trace is depicted as a sequences of arrows.
G:
G = {S → aB1 D1 → d1 D2
B1 → b1 B2 Dk → dk E
.. ..
. .
B j → bj C E → eF
C → cD1 F → fG
G → g}
Therefore L(G) as a regular language is an instance of the
class of type 3 languages L3 . The capital letters are variables,
that are not part of the actual knowledge tree.
By contrast to the common Hidden Markov Model (e.g.
[Rab89]), we want to take the history into account and thus
it is not expedient to build the prediction only on the current
state.
This is illustrated in the knowledge tree, which is shown
in figure 2, where the previously visited nodes resulting in
the navigation-history a . . . c are deciding for the further
proceeding of the actor in the knowledge base. The
recommendation system must finally support this kind of
context recognition, because an actor should be guided to
the subsequent nodes ef g, if his recently observed behaviour
matches to an already learned rule or can be inferred from
it. In terms of the given regular grammar the meaning is,
that the processed terminal symbol c only matches to the rule
C → cD1 , when it occurs subsequent to the history ab1 . . . bj
– which can be considered as the search or navigation context.
In this section we have shown that the task of incorporating
arbitrary navigation sequences, which are implicitly generated
by domain experts, can be considered as learning of an
unknown regular grammar. This means in turn that all learned
rules can be extracted afterwards by methods similar to those
of association rule mining. Like this, a general understanding
of the linked knowledge bases can be derived in form of
symbolic association rules.
IV. S OLUTION APPROACH
A. The Modular Recurrent Neural Network (MRNN)
The main task of the proposed neural network model is
to recognize context-dependent navigation operations on the
artifact tree and automatically predict the most appropriate
target sequence of nodes, that is relevant to the current topic
or task.
1) Theoretical foundation of the network model: Figure 3
illustrates the topology and the information flow of the pro-
posed recurrent neural network. ~yt+1 , . . . , ~yt+m represent the
associated sequence of target nodes, which are predicted by the
neural net to be associated strongest with the original sequence
of nodes in the artifact tree. This means that the prediction
is computed on the basis of the recent navigation-history
~xt−k , . . . , ~xt according to the observed actor behaviour.
An important characteristics of the recurrent design is the
inherent temporal memory of the neural network. So the
internal state layer ~st−k , . . . , ~st can store a temporal sequence
of actor operations in the training phase. Afterwards in the
operational application of the network, it can fully recover a
similar sequence of navigation operations in form of another
actor trace and then is able to give a prediction of the most
likely associated node-sequence in the target tree.
Thus the proposed MRNN can learn the temporal structure of a
set of symbol sequences, what is achieved by the temporally
unfolded network topology. The temporal dynamics of the
history and target sequences are explicitly modeled in the
hidden state layer. The neuron formations of that layer serve
as hidden and as context units, because ~st−1 provides the
context for the recursive computation of the hidden state ~st .
Thus these neurons are called hidden and context records
and one record per time step corresponds to one node of a
knowledge repository such as the artifact or target tree.
Fig. 3. Schematic topology of the proposed modular recurrent
network called MRNN. Internal state transition ~st → ~st+1 , A, B
and C are weight matrices. ~
xt is the external input symbol at time t,
~
yt+1 is the correspondingly predicted output symbol. The depicted
block arrow direction stands for a forward information passage.
B. Dynamical modular network structure
In order to process arbitrarily long node sequences
~xt−k , . . . , ~xt , ~yt+1 , . . . , ~yt+m with a neural network, we intro-
duce a novel modular configuration of an recurrent network,
that adopts to the number of history and sequence nodes at
runtime.
This dynamical modular structure is founded on the following
recurrent propagation model (forward information flow).
Prior definitions
k d k k d k
• A is a R , B is a R and C a R matrix.
• d is the dimension of the feature space.
• h = dim(~ si ), ∀i is an arbitrary number of hidden neurons
per state module or in other words the dimensionality of
the state layer. h is independent from d.
~st = f (B~st−1 + A~xt ) (1)
~yt = f (C~st−1 ) (2)
As introduced above, the ~si stand for the internal state at
each discrete time step. This state layer is the backbone for
learning from the navigation-history and for predicting the
recommended target nodes. The crucial recurrent equation 1
combines an external input ~xt and the previous state ~st−1
to the subsequent state ~st , which indirectly depends on all
foregoing external inputs ~xt−k , . . . , ~xt−1 and internal states
~st−k , . . . , ~st−1 . f : R → [cl , ch ] is a function with the
following properties.
1) f is monotonic increasing.
2) limx→−∞ f (x) = cl , limx→∞ f (x) = ch , ch > cl .
Often f the is chosen as the sigmoidal function
1
f (x) = 1+exp(−x) . a numerical document representation.
The specialty of the suggested model is its modular com-
position like depicted in figure 3. Thus the MRNN has to be
trained with a modified backpropagation through time (BPTT)
algorithm, while the training process itself is not subject of
this article (but will be treated in detail in succeeding papers).
However there is a crucial feature of the modular network
structure:
Independent of the length k of the history sequence or the
length m of the target sequence, the weight matrices A, B
and C are adjusted in the train process, because they are
reused in every time step t. This implies that there are exactly
three matrix instances for the training of all variably long
association rules. During one training epoch2 the network
adapts at runtime to the individual structure of the respective
node sequence with its history and target part – fed into the
neural network after preprocessing and transformation to a
train pattern.
1) Prediction of target sequences: The hidden temporal
representation by the MRNN’s state layer enables theoretically
infinite long prediction sequences, since the MRNN possesses
an continuous internal state layer [GSW05]. Thereby the
current state (~st )t∈N is repeatedly propagated through the
2A single cycle through all train patterns.
matrix B, even if no external inputs ~xt are available anymore.
The prediction-horizon, that means the number of reliably
predictable target nodes m, can normally be recognized by
means of the output behaviour ~yt+m , which attains to a steady
state for m > mmax and m → ∞:
Theorem:
~yt −→ ~y
ψ (3)
m>mmax ,m→∞
A = const. ⇒ ~st −→ ~s
ψ (4)
m>mmax ,m→∞
Since for the internal weights holds B[i][j] < 1, f oralli, j –
due to the chosen sigmoidal activation function with ch = 1
– the mapping B(. . . (B(B~st )) . . .) is a so-called contraction,
which converges to a fixpoint ψ for m → ∞.
This result was also observed in practice during intensive
work with the functional prototype implementation (Java)
of the proposed semantic recommendation system (coming
up in the following paper with working title: “Semantic
Recommendation for the Software Engineering Process”).
V. A PPLICATION TO A TEXTUAL KNOWLEDGE BASE
As a concrete application of the recommendation function-
ality a knowledge base like a document tree should be con-
sidered. Therefore each document node has to be transformed
into a meaningful and machine-processable representation.
A. Node representation model
For each knowledge node a feature vector ~vi ∈ V ⊆ Rd
is to be computed in a way that it represents its content in a
numerically processable form. In case of textual content we
apply well-known text mining methodologies for computing
Stemming..
We give a procedure by means of the term frequency -
inverse document frequency, that computes a characteristical
vector for each document in the knowledge base (KB) by
rating the relevance of all existing keywords.
Def. Term and document ratios
|docs(ti )|
df (ti ) = |KB| (5)
P n(ti ,doc)
tf (ti , doc) = (6)
t∈doc n(t,doc)
P n(ti ,doc) |KB|
tf idf (ti , doc) = · |docs(ti )| (7)
t∈doc n(t,doc)
n(t, doc) counts the number of terms t in a single document
doc.
The document frequency df (ti ) counts the number of docu-
ments that contain a term ti related to the number n of existing
documents in a certain repository KB= {doc1 , . . . , docn } .
tf (ti , doc) gives the relative term frequency of a term ti within
a single document doc. An exemplary ranking table is given
in table I. The idea of tf idf is to assign a higher relevance
to terms that occur often within a single document. At the
same time terms are rated higher, that appear more seldom
in the entire repository KB. The notion behind the tf idf is,
that globally frequent terms are not valuable for differentiating
 term tf idf (doc)
t14 0.41
t35 0.37
t27 0.26
. .
. .
. .
TABLE I
R ANKING TABLE WITH term frequency–inverse document frequency FOR
EACH TERM ti IN A DOCUMENT doc.
between the documents doc1 , . . . , docn , because they are
comprised by the most of the documents. On the other side
locally frequent terms characterize a single document very
well.3
Based on the introduced term and document frequency
measures, a fixed number d of most significant terms has to
be selected. Thus a global term significancy measure required
is, that rates terms over all documents in KB.
The d selected terms serve now for computing the feature
vector, that describes a document by the globally most signif-
icant terms and computes their local value in the respective
document doc by tf idf .
 
tf idf (trank 1 , doc)
 tf idf (trank 2 , doc) 
∀doc ∈ KB : ~vi =  (8)
 
.. 
 . 
tf idf (trank d , doc)
This corresponds to a modified bag of words approach
with d terms, where the terms are locally rated via tf idf
and w.r.t. to a single document. Not until this global
feature vector model the semantics of each document are
approachable by the MRNN. Here global means that the
structure (trank 1 , . . . , trank d ) and therefore the keyterms
describing the content of each document are equal, only the
component-wise tf idf ()-rating distinguishs the document
instances.
In order to ensure a sufficient performance of the neural
network processing, the complexity of the document represen-
tation should be constraint via d to a reasonable magnitude.
Furthermore dimension reduction techniques like Principle
Component Analysis (PCA) can be leveraged to optimize
the document representation regarding its semantics and the
efficiency of neural processing.
The suggested method for constructing a meaningful doc-
ument vector model will be utilized in the generic 2-layered
representation of knowledge nodes, that is presented in the
following section.
B. 2-layer representation of knowledge tree nodes
Knowing that the knowledge tree nodes can be represented
in a vector space V ⊆ Rd , the following 2-layered mapping
3 tf idf plus document length normalization
will serve for the identification and addressing of the nodes,
that participate in an association between artifact and target
tree. The 2-layered mapping is illustrated by figure 4. The
dichotomy of the symbolic and latent semantic representation
enables a fault-tolerant and fuzzy linking of both knowledge
bases via the recurrent neural network. The latter operates
only on the node representation layer and is decoupled from
the unique identifier scheme of the symbolic layer. Therefore
the neural network is exclusively bound to the vector space
layer on the side of the artifact repository as well as on the
side of the target repository.
This second layer ensures that similar or neighbored instances
of the knowledge tree nodes such as text documents are also
neighbored regarding their content representation in the vector
space Rd .
C. Pretraining of the recommendation system
In order to provide a basic recommendation functionality
on a certain knowledge domain – assumed to be represented
by a graph or tree structure consisting of text documents
– the recommendation system automatically discover a set
of fundamental similarity rules. These rules are association
rules of history

and target length 1 and are determined by
a simple n2 similarity search over all document nodes in
KB={doc1 , . . . , docn }. As similarity measure the standard
Euclidean norm on Rd or the so-called cosine distance can
be used.
cos-distance
D E
~a, ~b
d(~a, ~b) = (9)
k~ak ~b
h·, ·i is the standard scalar product.
Document index pairs (i, j) with cond : d(doci , docj ) ≤
t, t ∈ R are considered as semantically similar and lead to
binary association rules
doci ⇒ docj .
The set of document pairs that meet the condition cond are
transformed into train patterns and are pretrained by the
MRNN.
VI. C ONCLUSION
We have introduced a generic semantic recommendation
functionality based on a novel recurrent neural network –
the Modular Recurrent Neural Network MRNN. The MRNN
enables learning of variably long associations rules on graph-
based knowledge bases.
ACKNOWLEDGMENT
The authors would like to thank...
 Fig. 4. Illustration of the 2-layered mapping concept for knowledge node representation and association.

R EFERENCES
[Cal03] Robert Callan. Neuronale Netze im Klartext. In Pearson Studium,
ISBN 3-8273-7071-X, 2003.
[GSW05] Faustino J. Gomez, J”urgen Schmidhuber, and Daan Wierstra.
Modeling Systems with Internal State using Evolino. In Proc.
of the 2005 conference on genetic and evolutionary computation
(GECCO), Washington, D. C., ACM Press, New York, NY, USA,
2005, pages 1795–1802, 2005.
[JMS] Fergal W. Jones, IPL McLaren, and Rainer Spiegel. The
Prediction-Irrelevance Problem in Grammar Learning. In Uni-
versity of Cambridge, Department of Experimental Psychology,
Downing Site, Cambridge, CB2 3EB, UK.
[Kra91] Klaus Peter Kratzer. Neuronale Netze. In Carl Hanser Verlag
M”unchen Wien, 1991.
[MRF04] Rene Mayrhofer, Harald Radi, and Alois Ferscha. Recognizing and
Predicting Context by Learning from User Behavior. In Institut fr
Pervasive Computing, Johannes Kepler Universitt Linz, 2004.
[MS05] Igor MOKRIŁ and Lenka SKOVAJSOV. Neural Network Model
of System for Information Retrieval from Text Documents in Slovak
Language. In Acta Electrotechnica et Informatica No. 3, Vol. 5,
2005.
[Rab89] Lawrence R. Rabiner. A Tutorial on Hidden Markov Models and
Selected Applications in Speech Recognition. 1989.
[Sch01] Uwe Sch”oning. Theoretische Informatik - kurzgefasst. In SPEK-
TRUM, akademischer Verlag, 2001.
[SLD96] I. Syu, S.D. Lang, and N. Deo. A neural network model for
information retrieval using latent semantic indexing. In ICNN
96, The 1996 IEEE International Conference on Neural Networks,
pages 1318–23 vol.2, 1996.
[Smi03] Andrew Smith. Grammar Inference Using Recurrent Neural
Networks. In Department of Computer Science, University of
California, San Diego, La Jolla, CA 92037, 2003.