This article has been accepted for publication in a future issue of this journal, but has not been
fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
Autoencoder for Semisupervised Multiple Emotion
Detection of Conversation Transcripts
Duc-Anh Phan, Yuji Matsumoto, Hiroyuki Shindo
Abstract—Textual emotion detection is a challenge in com-
putational linguistics and affective computing as it involves the
discovery of all associated emotions expressed in a given piece of
text. It becomes even more difficult when applied to conversation
transcripts, as there arises a need to model the spoken utterances
between speakers while keeping in mind the context of the
entire conversation. In this paper, we propose a semisupervised
multi-label method of predicting emotions from conversation
transcripts. The corpus contains conversational quotes extracted
from movies. A small number of them are annotated, whereas
the rest are used for unsupervised training. The word2vec word-
embedding method has been used to build an emotion lexicon
from the corpus and then embed the utterances into vector
representations. A deep-learning auto-encoder is then used to
discover the underlying structure of the unsupervised data. We
fine-tune the learned model based on labeled training data and
measure its performance on a test set. The experiment result
suggests that the method is effective and is only slightly less
effective than human annotators.
Index Terms—Emotion recognition, semisupervised learning,
multilabel, word2vec, autoencoder.
I. I NTRODUCTION
E MOTION detection is an upcoming field of research
closely related to sentiment analysis. Extending beyond
sentiment analysis in terms of complexity, emotion detection
primarily aims to recognize feelings concealed within a media
source. Most of the research works until now have focused on
brain signals, video of facial expressions, audio recording, and
so on, along with multiclass classification of emotions. Little
research has been done regarding the detection of multiple
emotions simultaneously in textual data.
Nowadays, along with the popularity of social networks,
the Internet also contains an enormous amount of unlabeled
data, most of which is textual. By mining and applying
semisupervised emotion detection techniques to such data, we
open ourselves to a wide range of useful applications, such
as measuring citizen happiness, improving customer service,
social mental health care, and early screening of possible
suicides or crimes. While many such researches have been
undertaken for news headlines [1], tweets [2], and paragraphs
[3], the most useful type of textual data, conversational text
such as chat logs and replies on social media, is often
overlooked. Text of this type is of great use to applications
that use emotion detection. It gives us information about the
initial emotional state, how it changes during the conversation,
The authors are with the Graduate School of Information and Science,
Nara Institute of Science and Technology, 8916-5, Takayama, Ikoma, Nara,
630-0192, Japan
Email: phan.duc anh.oq3,matsu,shindo -at- is.naist.jp
1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1
what causes the said changes, what the final state is, and what
kind of actions result from the said final state. A survey on
”Trend analysis in social networking using opinion mining”
[4] predicted the need for emotion detection in streaming data
and live chat.
Conversational text is also more of a challenge than other
types of text. For short text samples, such as news headlines
[1] or tweets [5], the expression of emotion generally depends
on the words being used. Meanwhile, for longer text, the
grammatical structure and syntactic variables such as nega-
tions, embedded sentences, and type of sentence (question,
exclamation, command, or statement) play a part in expressing
emotions [6]. Identifying emotions in conversational text is
also very different from paragraphs because there is often
more than just one party in a conversation. Each party takes
turns, which are called utterances, to express different ideas
and emotions, thus making an impact on the other party’s
emotions. Therefore, to detect emotions in conversational text,
one needs to monitor not only the current utterance but also
the previous utterances, as well as the context of the entire
conversation [7].
In this study, we follow Plutchik’s work [8] and make
assumptions about the complicated nature of human emotions.
Emotions have connections, some are similar to each other but
have different intensities while some are opposite to each other
and some occur together at the same time, resonate, and create
other emotional states: dyads (figure 1).
Robert Plutchik proposed a hybrid model of both basic
emotions and dimensional theory [9]. He also put forward four
axes of bipolar emotions basic emotions: joy-sadness, fear-
anger, trust-disgust, and surprise-anticipation with different
levels of intensity (subfigure 1a). These primary emotions
may blend to form the full spectrum of human emotional
experience. The new complex emotions formed by a mix of
these primary emotions called dyads (subfigure 1b). Plutchik’s
theory explains the connection between emotions, he proposes
that some emotions are similar but have different intensities,
while some emotions do not occur at the same time since they
are on opposite sides of the axis. He also says that complex
emotions can be viewed as combinations of the relevant pri-
mary ones. This idea enables us to approach emotion detection
in a more comprehensive manner. [7]
In this paper, we propose a five-step method for detection
of emotions in a conversation:
1) We follow in [10]’s footsteps by building an emotion
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
(a) Plutchik’s wheel of emotions has four major axes (dimensions)
of basic emotions. Along each axis, there are opposite pairs of basic
emotions and different levels of intensity. The outer circle indicates
milder emotions while the inner circle indicates stronger ones. The basic
emotions from the two axes may blend in together and form more
complex emotions called dyads. For example: the combination of disgust
and anger is contempt and the combination of disgust and sadness is
remorse.
(b) Dyads: It is the combination of two emotions. Conflicting (opposite)
basic emotions cannot combine, whereas the other emotions can blend
in and form emotions that are more complex. Primary dyads are
combinations from neighboring axes, as shown in the above figure 1a,
whereas secondary and tertiary dyads are combinations of emotions that
are distanced by one or two axes, respectively.
Fig. 1. Plutchik’s basic emotions and dyads (combination of two basic
emotions) - image taken from http://twinklet8.blogspot.jp
lexicon using word2vec [11] on IMDb quotes dataset 1
2) Transforming raw input data into feature vectors using
the lexicon.
3) Using autoencoder to learn the underlying structure of
unsupervised data.
4) Retraining deep networks initialized from the parameters
of unsupervised models using labeled training data.
5) Verifying the performance of the system on annotated
1 ftp://ftp.fu-berlin.de/pub/misc/movies/database/
1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
2
test data and analyzing the evaluation result.
The approach described in this paper exploits Plutchik’s
theory in a comprehensive manner, which covers the full
spectrum of human emotions, to work on a challenging multi-
label conversation corpus. Our work is different from the
previous researches in four main ways:
• We have integrated Plutchik’s idea of basic emotion dyads
and intensity in our model, which provides scalability to
address emotions at a fine-grained level in the future.
• We use word2vec to automatically produce an emotion
lexicon, which is essential for automatic feature extrac-
tion of the raw input data.
• Our method is semisupervised: An autoencoder is used
on a large unsupervised dataset and its parameters are
used to retrain a deep network of annotated data. This
allows us to take advantage of both the scarce-labeled
data and the vast reserve of unlabeled data available on
the Internet.
• The output of our system is multi-label, which means that
it is capable of addressing multiple emotions simultane-
ously.
The remainder of this paper is organized into six sections.
Section II summarizes related work on emotion detection.
Section III discusses the nature of our dataset and explains
the annotating scheme. Section IV proposes the visualization
of our lexicon and discusses our approach, which includes
the steps mentioned in the preceding paragraphs. Section V
evaluates the proposed method in general and section VI
presents the conclusion and discusses future work.
II. R ELATED W ORKS
In the previous studies carried out on emotion analysis,
Ekman’s model of six basic emotions (anger, disgust, fear,
happiness, sadness, and surprise) [12] was often employed.
However, this model was developed from the observation of
human facial expressions. Therefore, it becomes irrelevant
when applied to text classification where there are opposing
emotions that cannot occur together. For example, a person
will feel either happy or sad about an event, yet he may feel
both happy and surprised at the same time. Ekman’s notion
also does not address other complicated emotions such as
annoyance, pride, or dominance.
Newer works rely on dimensional models using the valence-
arousal (VA) space [13], [14] where valance indicates the
positive-negative level of the emotion and arousal indicates the
intensity of the emotion. Both works focus on the projection of
emotional words into the VA space that can be used for lexicon
augmentation. [15] proposed building affective resources in
VA dimensions. However, VA space suggests a common and
interconnected neurophysiological system that is responsible
for all affective states [16], which is in contrast to the theory of
basic emotion [8], [12] where different emotions are assumed
to arise from separate neural systems. Mostly, the VA space is
often used to test the stimuli of emotional words or facial
expressions, as it appears to be too noisy for the emotion
classification task.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
One of the most obvious clues to identify emotions in
conversation is the choice of words as suggested in [17].
Therefore, [18], [19] proposed a database of more than English
lemmas giving information about the affective meanings of
words. These works inspired the use of emotional lexicon
in emotion analysis, including the NRC emotion lexicon [2]
and the Wordnet-affect [20]. However, their lexicon size is
small, therefore lacks coverage, and only works on short text
domains such as tweets, news headlines, and so on where the
recognition of emotion heavily depends on words. They relied
on a set of specific seed emotional words and then expanded
the lexicon by finding all its synonyms and antonyms. The
lexicons are often annotated in a multiclass manner, which
means that one lexical item can only be associated with a
specific emotion. However, in reality, one word might express
different emotions in different situations. Their methods not
only oversimplify the complex nature of emotions but also
remove the collocation connections between the lexical items.
As a result, the approaches of previous lexicons show inflexi-
bility when applied to other domains, such as emotion analysis
in customer reviews, restaurants, personal blogs, because they
do not take into account the different meanings of words (word
senses) in different contexts. [21]. In this paper, despite the
fact that we build our lexicon from the IMDb movie quotes
domain, the lexicon is automatically extracted from the corpus
through word embedding and can be reproduced on any other
domains without much effort.
Most of the previous work concentrated on narrowing down
the complexity of the problem by focusing on only a small
set of emotions that barely involved three or four emotional
states [22], [23]. While such approaches may have succeeded
in particular problems and domains, they lacked the capacity
to predict all genres of emotions in different kinds of text.
Another work by [24] performed a multiclass classification
of dialogue data sourced from Twitter in Japanese. The re-
searchers automatically labeled the obtained dialogues using
emotional expression clues, which is similar to our work of
using an emotional word lexicon. The work assumed that one
tweet might portray only one emotion. Although, this may
be the case in short exchanges of tweets, it is not applicable
to real-life conversations where the number of characters
per utterance is not limited, and the context information
complicates the process of emotion analysis.
Existing multi-label classification methods can be classi-
fied into either the problem transformation category or the
algorithm adaptation category. The former group of methods
transforms the multi-label tasks into multiple single-label
classifications. They use the multi-class approach of one-vs-
rest but ignore the correlations among labels similar to the
binary relevance method [25]. The less common approach
in this group is the Label Powerset (LP) method, which
produces new labels from every possible combination of the
original labels. This approach considers label correlations but
also produces a very large number of subsets of labels with
very few examples of each. Random k-Labelsets (RAkEL)
[26] follow LP; however, they train the LP classifiers with
only k random labelsets from all possible subsets. RAkEL
1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
3
takes advantage of the label correlations and at the same time
manages to avoid the problem of a large number of subsets
that traditional LP methods face.
Algorithm adaptation methods include the multi-label ex-
tension of decision trees, support vector machines, neural
networks, and so on. [27]. Meka’s 2 implementation of deep
back-propagation neural network (DBPNN) [28] is also ca-
pable of handling multi-label problems. The method focuses
on using multiple layers of restricted Boltzmann machines
(RBM) to create an autoencoder for pre-training. After that, the
whole network is fine-tuned using back-propagation of error
derivatives. In our study, we implement a similar adaptation of
a multi-label neural network using a stack of fully connected
layers and also incorporate the correlations between labels by
directly modifying the loss function.
Using Plutchik’s basic emotions, [29] proposed a simple
bag-of-words approach and fine-tuned RAkEL for multi-label
classification of movie reviews. We delve further and work
on the conversation data, where the exchange between the
characters and the context of the entire dialogue is of great
importance. The closest example of our work is [3] on para-
graphs and documents, which tried to improve the sentence-
level prediction of some special emotions which, owing to
data sparseness and inherent multi-label classification, were
very difficult to predict. These researchers incorporated label
dependency between labels and context dependency into the
graph model to achieve the goal. However, their work is for
paragraphs in Chinese. In our case, we take advantage of an
autoencoder to capture the abstract representation of context
information.
III. C ORPUS
The IMDb quotes corpus is newly published and gets
updated frequently. It includes approximately 2,107,863 ut-
terances (turns in conversation) from 117,425 movies. There
are several reasons for us to choose the IMDb corpus. First,
movies have more input for the annotators with video and
audio other than just text, which helps them to provide a
more accurate annotation. Second, movies tend to evoke many
emotions among the viewers [30]. Third, there are various
genres in movies such as romance, sci-fi, documentary, and so
on. These genres might be equivalent to real-life domains, for
example: normal life conversations and scientific exchanges
and debates. Lastly, unlike other types of art, such as literature
or theatre plays, moviemakers often try to give their dialogues
as much real-life feelings as possible [31]. In the work of
[32] and [33], it had been concluded that spoken language
in movies resembles spontaneous spoken language, which
supports our choice of using a movie corpus to imitate real-life
conversations.
To produce labeled data, we sample quotes from a few
chosen famous movies. We then try to match the quotes with
subtitle files to get the time of the corresponding scenes from
the movies. Annotators are given a description of the basic
emotions and dyads and then asked to watch the scenes with
2 http://meka.sourceforge.net/\#about
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING 4

using the values in this work. In future work, we hope to be

able to predict the intensity of the emotion and conduct more
thorough annotating sessions.
Emotion class Average Intensity
Anger 0.495 ± 0.200
Fear 0.426 ± 0.181
Disgust 0.411 ± 0.175
Trust 0.498 ± 0.167
Joy 0.564 ± 0.194
Sadness 0.477 ± 0.202
Surprise 0.484 ± 0.190
Anticipation 0.433 ± 0.152
TABLE I
AVERAGE INTENSITY VALUES PER EMOTION AND STANDARD
DERIVATIONS . T HE DERIVATIONS ARE TOO HIGH , INDICATING THAT THE
VARIED VIEWS OF THE ANNOTATOR ON THE INTENSITY OF EMOTIONS .

We define the gold-standard data by applying the majority

(a) UI of the annotating website. Users can choose the appropriate emotions
by adjusting the confidence bars or by typing the emotions or dyads into
the text box. The dyads are then decomposed automatically into primary
emotions and the bars are readjusted.
(b) Examples of annotated transcripts from movie: Brave Heart (1995) - In
the last example, opposing emotions of trust and disgust are both annotated.
Fig. 2. In the annotating scheme of the testing data, each utterance is
annotated with basic emotions or dyads. On the website, the confidence levels
are adjusted using the bar scale in 2a and translated into intensity values in2b
subtitles (figure 2), following the annotation scheme shown
below:
• One utterance may hold zero, one, or more emotions at
the same time. The list of emotions to assign includes
Plutchik’s basic emotions and dyads. The system will
treat the dyads as a combination of basic emotions. In
case an utterance holds no emotion, it should be annotated
with ”None.” The intensity of emotions is also considered
in the labeling phrase, as in subfigure 2a.
• The annotators need to assign the entire utterance, which
may have two or more sentences, with a set of all
emotions expressed inside it. There may be cases where
opposing emotions appear simultaneously in the same
utterance, for example the pair trust-disgust in the last
example of the subfigure 2b.
We understand that the annotated emotions might be linked
to different entities (for example, in the last example of
subfigure 2b, trust is linked to the princess while disgust is
to the king). However, in the scope of this paper, we would
like to cover only the emotions that can be felt by a speaker.
Hence, the annotation scheme does not require the annotators
to point out the target entities of the emotions.
We have several intensity values for each annotation; how-
ever, the standard derivations are very high, as shown in Table
I. While annotators agree on the existence of some emotions,
their views on the intensity vary. Thus, we decided to avoid
rule: any emotion annotated by at least three annotators is
considered a valid label for the utterance. If an utterance has
no valid label, it is considered to be objective and have no
emotion. There might be complex cases where a single sample
has been annotated with different emotions of high-confidence
values by different annotators. We acknowledge that this is
the current blind spot of our scheme, as these cases might
get labeled as objective. We report on the statistics of the
data in Table II. In the 1,000 utterances of gold-standard data,
896 of them are agreed by at least three annotators and have
emotion labels. The remaining 104 utterances that have no
label according to the gold standard are defined above and
are considered to be objective. All these 1,000 utterances are
used in the testing phase.
Following are some statistics of the corpus: a total of
2,107,863 utterances comprising of over 26 million words.
We separated the training data of 10,000 utterances, which
was annotated by only one annotator, from the testing data of
1,000 utterances (gold-standard), which was annotated by five
annotators. The rest of nearly two million utterances are to be
used as unsupervised data.
One of the most common Inter-annotator agreement
measurements is the Kappa statistics [34]. However, these are
not applicable to multi-label datasets because their way of
computing causes a hypothetical probability-of-chance agree-
ment Pe to be greater than 1, since there are cases where two
or more labels are annotated to a given instance. Therefore,
we consider the gold-standard data to be the ground-truth data
and measure the average accuracy of each of the emotions
and the F1 score of the annotators in Table III. A survey by
[35] suggests that low agreement scores are often observed in
multi-label annotating tasks.
From Table III we can see that there is an unbalanced
No. Annotators No. utterances
Agreed by 2 annotators 996
Agreed by 3 annotators (gold-standard) 896
Agreed by 4 annotators 443
Agreed by 5 annotators 253
Total No. utterances 1,000
TABLE II
N UMBER OF UTTERANCES RECEIVING AGREEMENTS BY ANNOTATORS IN
THE TESTING DATA .

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
Emotion class Frequency Accuracy
Anger 0.214
Fear 0.154
Disgust 0.127
Trust 0.279
Joy 0.183
Sadness 0.232
Surprise 0.141
Anticipation 0.064
Average accuracy (by class)
Average accuracy (by annotator)
Average F1 (by annotator)
Total No. utterances
Emotions per utterance
TABLE III
AGREEMENT SCORE AMONG ANNOTATORS WITH GOLD - STANDARD DATA
AS THE GROUND - TRUTH . W E COMPARE THE ANNOTATION OF EACH
ANNOTATOR TO THE GOLD - STANDARD DATA AND THEN INFER THE
ACCURACY AND THE F1 SCORE
distribution of emotions in our gold-standard data. While
emotions such as anger, trust, and sadness are annotated
very frequently, the annotators rarely label anticipation. There
might be two reasons for this, the first being the subtlety of the
emotion itself and the second being the choice of the movie
genre used in our data. However, both seem to be interesting
points to investigate on our future research. On an average,
our data has 1.41 emotions per utterance.
IV. P ROPOSED METHOD
A. Emotional words to vectors
Using a lexicon is proven to provide significant improve-
ments in identifying the emotion conveyed by a word [2].
Therefore, in our case, we build a new lexicon, in which each
lexical item displays not only its association with Plutchik’s
basic emotions but also the strength of the association. We
combine the word2vec features and calculated emotion fea-
tures to form a hybrid vector representation of a lexicon item
as follows:
1) word2vec features: Using word2vec, we generate the
embedding of all words available in the corpus. With the
embedding, the cosine similarity between each word and the
primary emotion words is calculated in the form of an equation
1. In our work, the embedding of a word is a 96-dimensional
vector.
vec(word) · vec(ei )
sim(word, ei ) =
kvec(word)k2 kvec(ei )k2
2) Emotion features: By contrast, we define the primary
emotions and dyads proposed in Plutchik’s theories as the
emotional vectors of our lexicon and give them initial values.
Different levels of intensity of emotional words are also
considered. Each lexical item in the lexicon has a vector of
values on every axis of the basic emotions: joy-sadness, fear-
anger, trust-disgust, and surprise-anticipation. We manually
assign the primary emotions a value vector of 1, 0, or -1, and
others with 1.5, 0.5, or -0.5, -1.5, depending on the intensity
of the emotion according to Plutchik’s theory. For example,
”joy” came from the axis of joy-sadness. Thus, its vector is
[1,0,0,0], while the vector for ”sadness” is [-1,0,0,0]. The word
”ecstasy” is of higher intensity than ”joy”; hence, its vector
1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Transactions on Affective Computing
5
0.72
is [1.5,0,0,0]. It is to be noted that the minus sign indicates
0.673 only that the emotion is on the other side of the axis and does
0.624 not suggest a negative emotion in any case. Figure 3 explains
0.65
0.606
the intensity and polarity of basic emotions according to the
0.584 theory.
0.575
0.491
0.615
0.43
0.626
1,000
1.41
Fig. 3. Intensity and polarity of basic emotions.
For the dyads and other words, we calculate the similarity
between these words and all primary emotions ei . We assume
that the higher the similarity, the closer the emotional state
of the words to the primary emotions. The emotional vector
of one word is the averaged result of all primary emotion
vectors multiplied by the similarity weights, as in an equation
2. Since there are four axes of basic emotions in Plutchik’s
theory, the result of this step is a four-dimensional vector of
emotion features. Pn
sim(word, ei ) × vec(ei )
vec(word) = 1 (2)
n
The final embedding is the concatenation result of the
word2vec-generated vectors and the newly calculated emo-
tional vectors. In this research, our embedding is of 100-
dimensional vectors, 96 of which are generated with word2vec.
The other four are generated using the preceding steps.
B. Visualization of the lexicon
Our lexicon consists of 181,276 lexical words, which is
much larger than most previous lexicons proposed by other
researchers. The NRC Emotion Lexicon [2] and Wordnet-
Affect [20] contain 25,000 and 2,876 synsets, respectively.
Figure 4a is a visualization of the top 5,000 popular lexical
items and some of the basic emotions. The projection is
done directly on 100-dimensional vectors using the matplotlib
(1) library function 3 and t-SNE algorithm from the scikit-learn
packet 4 for dimension reduction. Despite the fact that the
visualization is done by reducing the number of dimensions
of each lexical item to only two dimensions, some interesting
results can be observed in figure 4.
From the figure, we observe can see that except for the pair
of fear and anger, other opposite basic emotions are located
quite far from each other (subfigure 4a), which is the desirable
outcome of the lexicon. Interestingly, in a small cluster of
subfigures 4b, we observe three basic emotions: joy, fear, and
anger. The surrounding lexical items, while appearing to be
random at first, somehow seem more relevant later. Words
3 https://matplotlib.org
4 http://scikit-learn.org/stable/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING 6

(a) Embedding of the top and most frequent 5000 lexical items (small dots) and basic emotions, which are both annotated in
red with arrow signs. We notice some clusters are overlapped by basic emotions.

(b) Items with the most similarity to basic emotions: joy, fear, and anger (c) Items that are most similar to basic emotion trust , such as nods,
. Dyads or intensified/lesser emotions are boxed and annotated in blue. appreciate, together, and agreed.
Rage is the intensification of anger and guilt is a combination of joy and
fear
Fig. 4. Visualization of the lexicon in two dimensions using matplotlib and scikit-learn’s t-SNE. The opposite emotions are often far from each other, while
lexical items with similar meanings are close, as in 4a. However, because we reduced the number of embedding dimensions from the original 100 to only 2,
some clusters are mixed together, as in 4b

such as pain, rage, evil, and curse are close to anger; pride,
happiness, and beauty are close to joy. The dyad guilt, which
according to Plutchik’s theory is a combination of joy and
fear (subfigure 1b) is also present in this small cluster. In the
cluster of trust (subfigure 4c), we see lexical items that suggest
agreement, such as nods, agreed, and appreciate. The results
show that our lexicon learns the association between lexical
items and the basic emotions and dyads.
for each utterance in a conversation, we also have to vectorize
the previous utterance and the entire conversation to capture
the contextual information (Figure 5). As a result, the vector
representation of an utterance is now a 300-dimensional vector
concatenated product of the utterance itself and the above-
mentioned contextual information. This representation is then
fed to the input layer of the neural network in the following
sections:

C. Text to vector D. Autoencoder semisupervised learning

We consider a bag-of-lexical-items approach to transform Similar to [36], the goal of our autoencoder is to understand
the raw input text into vector form. Therefore, for a piece the representation of the input data. We hope that in the process
of text, its representation is the sum vector of all lexical of encoding and reconstructing the input data, the underlying
items inside. As our goal is to predict the emotional label structure is revealed and the model, after retraining on labeled

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
Fig. 5. Vector representation of an utterance is a concatenation of the
word2vec and emotion features of the current utterance, the previous utterance,
and all other utterances in the conversation, as three 100-dimensional vectors.
data, provides better results than when it depended solely
the labeled data. From our perspective, the process of the
autoencoder learning the representation of our data is similar
to the language learning process of an infant in its early years5
6
. Both the processes include the repetition of decomposing
input into smaller components (morphemes in infant’s lan-
guage learning and features vector in an autoencoder) and re-
composing these components to produce meaningful output
(babbling sounds in the infant’s case and the reconstruction of
input in the autoencoder’s case).
In our opinion, an autoencoder is much simpler because it
is limited in the way it can interact with the ”world”. The
”world” to an autoencoder is just the information we feed to
it while an infant can interact freely and deliberately with the
real world to acquire more information and come up with its
own abstract representation of the information it receives. The
features in an autoencoder are just compressed information
while the infant can understand the meaning of each input
and output and transform them into concepts and ideas.
Figure 6 displays the components of our network. During
the unsupervised training phase, we use the encoder and
decoder network and during the supervised training phase,
we use the encoder and the classifier. In our implementation,
Tensorflow [37] was used for its GPU-computing power. The
entire system uses a sigmoid function and gradient descent
optimizer with a learning rate of 0.001. To avoid overfitting,
we use a dropout method with a keep rate of 0.75 at all hidden
layers.
1) Unsupervised training: The encoder is a straight feed-
forward neural network with a 300-dimensional input layer
and two 100-dimensional fully connected hidden layers 7 .
7
Fig. 6. Structure of the autoencoder deep network: 1. Shared encoder layers
(bottom) 2. Unsupervised part (top-left): A straight feed-forward decoder
neural network with mirror settings from the encoder. 3. Supervised part (top-
right): With eight-dimensional output and threshold layers corresponding to
eight basic emotions ei .
epochs at 200.
2) Supervised retraining: After learning the representation
of the input, the model is trained further with 10,000 labeled
utterances. We use the bias and weights of the encoder to
initialize the network and an output and threshold layer are
added to monitor the multi-label prediction (Figure 6. The
output of our system is a set of predicted labels ei for eight
basic emotions. The threshold layer is a simple set of all ti
for each ei . Let oi be an output node( of the output layer. We
1, if oi ≥ ti
have the following equation: ei =
0, otherwise
The thresholds are initialized randomly and then updated
after each epoch; in a manner similar to how we updated the
biases and weights. Only labels with output values greater than
the corresponding threshold are considered valid. Initially, we
determined a fixed threshold for all the emotions but soon
realized that a flexible set of thresholds will be more effective
and reasonable.
As Y is the true labelset and Y 0 is the set of labels predicted
by our model, we define the cross-entropy loss function as
follows:
CrossEntropy = −[Y ln Y 0 + (1 − Y ) ln(1 − Y 0 )] (3)
Logically, the decoder is the mirror image of the encoder l  2 
with the same settings but in the reverse order. Unlike the
X wi
Loss = CrossEntropy + λ (4)
autoencoder in the image-classifying task, we do not add j
2
noise to the network. The least-squares error loss function for
reconstructing the input is as follows: L = (X − X 0 )2 , where The global cost function is regularized by twelve regulariza-
X is the input vector and X 0 is the reconstructed vector. For tions using the weights wj of all layers and the value of lambda
the autoencoder to learn the underlying structure completely, λ is fixed to be 0.01 in this study. As we use only a small part
we set the minibatch size at 128 and the number of training of the corpus as training data, all the twelve regularizations
help us in avoiding overfitting problems.
5 https://www.linguisticsociety.org/resource/faq-how-do-we-learn-language
6 http://www.ling.upenn.edu/courses/Fall 2011/ling001/acquisition.html
7 We are aware of LSTM seq2seq architectures that can handle dialogues
V. E XPERIMENT
directly instead of our method of appending the three vectors. However, LSTM
is a more complex method and has higher computational costs. Our trial run of
A. Experiment setting
LSTM on 100,000 utterances of unsupervised training data took much longer As mentioned earlier, we used a set of 1,000 annotated
than our current method with no real improvement. While taking our current
limits of computing power into consideration, we decided to keep LSTM for utterances for testing our method. For the gold standard of
future research. the test data, we applied the majority rule to the annotations,

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
which states that if a basic emotion is labeled by at least three
among five annotators, we accept it as a true label for the
utterance.
Evaluation metrics In our study, two common evaluation
metrics, which have been popularly used in multi-label clas-
sification problems, [3], [38] are employed to measure the
performance of our system. Let Yi be a set of true labels for
a given instance i , and Yi0 be the set of labels predicted by a
system. Let N be the total number of instances, then:
1) Hamming score, or accuracy in multi-label classifica-
tion, gives the degree of similarity between the ground-
truth set of labels and the predicted set of labels.
N ods, self-learn uses supervised data to produce a model. This
1 X |Yi ∩ Yi0 |
Hammingscore = (5)
N i |Yi ∪ Yi0 |
2) F1-measure: It is the harmonic mean of precision and
recall. In our study, we have given equal importance to
precision and recall.
2 ∗ precision ∗ recall
F1 = (6)
precision + recall
Precision: It is the fraction of correctly predicted labels
among all the predicted labels in the set.
N
1 X |Yi ∩ Yi0 |
P recision = (7)
N i |Yi0 |
Recall: It is the fraction of correctly predicted labels among
all the true labels in the set.
N
1 X |Yi ∩ Yi0 |
Recall = (8)
N i |Yi |
B. Experimental results
To evaluate our system, a comparison must be made against
other systems. We replicated the works of others and applied
them to our new corpus. [29] is a similar work that also used
Plutchik’s theory of basic emotions and worked on multi-
label data without considering the intensity of the emotion
labels. This study achieved an F1 score of 45.6 in its own
dataset of 629 sentences of user-generated movie reviews.
Both the studies used Meka’s RAkEL method and bag-of-
words approach as their work for the first baseline. However,
it would be unfair to apply [29]’s system to our corpus
and make comparisons since it is fine-tuned specifically for
their corpus. Their study neither considers the emotions of
each sentence nor the contextual information. Therefore, the
second system is Meka’s DBPNN, which is said to have a
better performance than RAkEL [39]. We consider RAkEL
and DBPNN to be state-of-the-art systems for multi-label
classification of emotions in text.
To our knowledge, the most important baseline is human an-
notation. To obtain this baseline, we calculated the evaluation
metrics based on the average agreement score between each
annotator and the gold standard, as mentioned in section III.
We elaborated the performance of our system using different
settings: autoencoder semisupervised learning, unsupervised
self-learning, and supervised learning using labeled data only.
Figure 7 compares the performance of our system to the
baselines.
1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
8
vs. Bag-of-words approaches: Our system, under different
settings, performed remarkably better than the simple ap-
proaches using Meka’s DBPNN and RAkEL. The supervised
method used the same dataset as the other two methods and
none of them took advantage of the unsupervised data. Yet, the
supervised method remarkably outperformed the two methods
by 14 and 19 in hamming and the F1 score, respectively. We
believe that the context features and our emotion lexicon were
a deciding factor.
Our system: autoencoder vs. self-learn We can clearly see
that the the autoencoder has a better performance as compared
to the self-learn method. While both are semisupervised meth-
model tries to classify the unsupervised data and assimilates
the obtained result to retrain. Naturally, when the size of the
unsupervised data becomes larger than the supervised one, the
model starts dealing with more and more unseen examples and
the performance drops. Figure 8 confirms this explanation.
Our system: semisupervised vs. supervised The perfor-
mance of the autoencoder is much better than that of the
supervised method. Our system first learns and tries to imitate
the enormous number of unsupervised examples. During the
retraining process, it figures out the connections between the
concepts that it has imitated and the true results. Therefore,
it has a better understanding of the data and makes better
predictions than when using only the supervised examples.
vs. human annotator: This is the most important baseline,
which explains how well our system performs in comparison
with a human. Please note that this evaluation of the human
annotators is the average agreement between each annotator
and the gold-standard data (decided by the majority rule as
discussed in the earlier section). Our system’s performance
is slightly worse than that of the human annotator in both
Hamming score by 4 and the F1 measure by 6. However,
we should also take into consideration that the input for
human annotators are movies with full video, sound signals,
and transcript texts, while the input for our system is only
the transcripts. We acknowledge that the different inputs of
annotators and our system may affect the results. However,
as the purpose of this research is to make a system that can
identify emotions in text messages, we did not incorporate the
features from other modalities.
In short, our system with autoencoder semisupervised learn-
ing performs much better than the existing methods not only
because of the emotion lexicon that we built or its way of
extracting contextual information but also because of its ability
to use largely available unsupervised data. When relying only
on textual data, our system performs slightly worse than
human annotators, for whom full movie clips serve as the
input.
C. Publication of the data and possibility of replicating the
work in other domains
The annotated data and the emotion lexicon will soon be
published in the author’s GitHub repository.
The movie conversation domain is very close to real-life
settings. Therefore, we believe that the model and lexicon
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
Fig. 7. Evaluation of the system: 1) Human annotators, 2) our system using autoencoder, 3) our system using self-learning, 4) our supervised system,
5)RAkEL, and 6)DBPNN.
Fig. 8. Performance comparison between autoencoder, self-learn, and super-
vised settings of the system: The autoencoder achieves the best performance
along with a steady growth in the number of unsupervised data, while
self-learn performs worse and worse. Supervised method does not use any
unsupervised data; thus, it is not affected.
would perform reasonably well when applied to other domains.
Our work can be replicated easily on other domains without
a lot of effort using little annotated data and the automatic
emotion lexicon extraction process.
VI. C ONCLUSION & FUTURE WORK
In this paper, we proposed a method of detecting and
classifying multiple emotions from the IMDb movie quotes
corpus. The corpus is a set of conversation transcripts anno-
tated with multi-label emotions following Plutchik’s notion of
basic emotions and dyads. Our method involves building an
emotion lexicon using the word2vec word-embedding tech-
nique, extracting a vectorized representation of the input, and
classifying the emotions in a semi-supervised manner with
the help of an autoencoder that exploits both the unlabeled
and labeled data. The experiments show that the ability of our
1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
9
method to detect emotion is comparable to that of a human
annotator despite the fact that it received only text as input.
By building our system in a movie conversation domain, we
attempted to produce a close imitation of real conversations.
We hope that our system will perform well in real-life settings,
such as SMS texting and messenger applications, and will play
a supporting role in identifying emotions in speech processing.
We are planning to apply our work to other domains and expect
a similar level of performance.
In the future, it shall be very interesting to incorporate all
features from modalities such as text, speech, and video signals
into our current system, and once again evaluate how well
it performs against a human. Another useful and important
aspect for future research is that we link the emotions towards
entities in the conversation. From our perspectives, such fea-
ture might play an important part in future emotion analysis
systems as it can indicate on how the machine should react
and respond to human emotional remarks.
ACKNOWLEDGMENT
This research was supported by JST CREST Grant Num-
ber JPMJCR1513, Japan. We are grateful to our colleagues
from the Computational Linguistics Lab, NAIST, Japan, who
provided insights and expertise that greatly assisted in the re-
search. We also appreciate all the annotators who participated
in this research for their valuable efforts and patience. We
would like to thank Editage 8 for English language editing
R EFERENCES
[1] C. Strapparava and R. Mihalcea, “Semeval-2007 task 14: Affective
text,” in Proceedings of the 4th International Workshop on Semantic
Evaluations. Association for Computational Linguistics, 2007, pp. 70–
74.
8 www.editage.jp
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
[2] S. Mohammad, “Portable features for classifying emotional text,” in
Proceedings of the 2012 Conference of the North American Chapter
of the Association for Computational Linguistics: Human Language
Technologies. Association for Computational Linguistics, 2012, pp.
587–591.
[3] S. Li, L. Huang, R. Wang, and G. Zhou, “Sentence-level emotion
classification with label and context dependence,” in Proceedings of the
53rd Annual Meeting of the Association for Computational Linguistics
and the 7th International Joint Conference on Natural Language
Processing (Volume 1: Long Papers). Beijing, China: Association
for Computational Linguistics, July 2015, pp. 1045–1053. [Online].
Available: http://www.aclweb.org/anthology/P15-1101
[4] S. Dave and H. Diwanji, “Trend analysis in social networking using
opinion mining a survey,” 2015.
[5] J. Bollen, H. Mao, and A. Pepe, “Modeling public mood and emotion:
Twitter sentiment and socio-economic phenomena.” 2011.
[6] G. Collier, Emotional expression. Psychology Press, 2014.
[7] D.-A. Phan, H. Shindo, and Y. Matsumoto, “Multiple emotions detection
in conversation transcripts,” PACLIC 30, p. 85, 2016.
[8] R. Plutchik, “A general psychoevolutionary theory of emotion,” Theories
of emotion, vol. 1, pp. 3–31, 1980.
[9] ——, “The nature of emotions human emotions have deep evolutionary
roots, a fact that may explain their complexity and provide tools for
clinical practice,” American scientist, vol. 89, no. 4, pp. 344–350, 2001.
[10] M. Li, Q. Lu, Y. Long, and L. Gui, “Inferring affective meanings of
words from word embedding,” IEEE Transactions on Affective Comput-
ing, vol. 8, no. 4, pp. 443–456, 2017.
[11] R. Řehůřek and P. Sojka, “Software Framework for Topic Modelling
with Large Corpora,” in Proceedings of the LREC 2010 Workshop on
New Challenges for NLP Frameworks. Valletta, Malta: ELRA, May
2010, pp. 45–50, http://is.muni.cz/publication/884893/en.
[12] P. Ekman, W. V. Friesen, M. O’Sullivan, A. Chan, I. Diacoyanni-
Tarlatzis, K. Heider, R. Krause, W. A. LeCompte, T. Pitcairn, P. E. Ricci-
Bitti et al., “Universals and cultural differences in the judgments of facial
expressions of emotion.” Journal of personality and social psychology,
vol. 53, no. 4, p. 712, 1987.
[13] R. A. Calvo and S. Mac Kim, “Emotions in text: dimensional and
categorical models,” Computational Intelligence, vol. 29, no. 3, pp. 527–
543, 2013.
[14] L.-C. Yu, J. Wang, K. R. Lai, and X.-j. Zhang, “Predicting valence-
arousal ratings of words using a weighted graph method,” in Proceedings
of the 53rd Annual Meeting of the Association for Computational
Linguistics and the 7th International Joint Conference on Natural
Language Processing (Volume 2: Short Papers), vol. 2, 2015, pp. 788–
793.
[15] L.-C. Yu, L.-H. Lee, S. Hao, J. Wang, Y. He, J. Hu, K. R. Lai,
and X. Zhang, “Building chinese affective resources in valence-arousal
dimensions,” in Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics:
Human Language Technologies, 2016, pp. 540–545.
[16] J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model
of affect: An integrative approach to affective neuroscience, cognitive
development, and psychopathology,” Development and psychopathology,
vol. 17, no. 03, pp. 715–734, 2005.
[17] S. M. Mohammad and P. D. Turney, “Emotions evoked by common
words and phrases: Using mechanical turk to create an emotion lexicon,”
in Proceedings of the NAACL HLT 2010 workshop on computational
approaches to analysis and generation of emotion in text. Association
for Computational Linguistics, 2010, pp. 26–34.
[18] A. B. Warriner, V. Kuperman, and M. Brysbaert, “Norms of valence,
arousal, and dominance for 13,915 english lemmas,” Behavior research
methods, vol. 45, no. 4, pp. 1191–1207, 2013.
[19] M. M. Bradley and P. J. Lang, “Affective norms for english words
(anew): Instruction manual and affective ratings,” Citeseer, Tech. Rep.
[20] C. Strapparava, A. Valitutti et al., “Wordnet affect: an affective extension
of wordnet.” Citeseer, 2004.
[21] B. Heredia, T. M. Khoshgoftaar, J. Prusa, and M. Crawford, “Cross-
domain sentiment analysis: an empirical investigation,” in Information
Reuse and Integration (IRI), 2016 IEEE 17th International Conference
on. IEEE, 2016, pp. 160–165.
[22] S. K. D’Mello, S. D. Craig, J. Sullins, and A. C. Graesser, “Predict-
ing affective states expressed through an emote-aloud procedure from
autotutor’s mixed-initiative dialogue,” International Journal of Artificial
Intelligence in Education, vol. 16, no. 1, pp. 3–28, 2006.
1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
10
[23] C. Yang, K. H.-Y. Lin, and H.-H. Chen, “Emotion classification using
web blog corpora,” in Web Intelligence, IEEE/WIC/ACM International
Conference on. IEEE, 2007, pp. 275–278.
[24] T. Hasegawa, N. Kaji, N. Yoshinaga, and M. Toyoda, “Predicting and
eliciting addressee’s emotion in online dialogue.” in ACL (1), 2013, pp.
964–972.
[25] K. Brinker, J. Fürnkranz, and E. Hüllermeier, “A unified model for
multilabel classification and ranking,” in Proceedings of the 2006
conference on ECAI 2006: 17th European Conference on Artificial
Intelligence August 29–September 1, 2006, Riva del Garda, Italy. IOS
Press, 2006, pp. 489–493.
[26] G. Tsoumakas and I. Vlahavas, “Random k-labelsets: An ensemble
method for multilabel classification,” Machine learning: ECML 2007,
pp. 406–417, 2007.
[27] G. Tsoumakas and I. Katakis, “Multi-label classification: An overview,”
International Journal of Data Warehousing and Mining, vol. 3, no. 3,
2006.
[28] G. Hinton and R. Salakhutdinov, “Reducing the dimensionality of data
with neural networks,” Science, vol. 313, no. 5786, pp. 504–507, 2006.
[29] L. Buitinck, J. Van Amerongen, E. Tan, and M. de Rijke, “Multi-emotion
detection in user-generated reviews,” in Advances in Information Re-
trieval. Springer, 2015, pp. 43–48.
[30] J. T. Hancock, K. Gee, K. Ciaccio, and J. M.-H. Lin, “I’m sad you’re
sad: emotional contagion in cmc,” in Proceedings of the 2008 ACM
conference on Computer supported cooperative work. ACM, 2008, pp.
295–298.
[31] S. I. Rauma, “Cinematic dialogue, literary dialogue, and the art of
adaptation: dialogue metamorphosis in the film adaptation of the green
mile,” 2004.
[32] P. Forchini, “Movie language revisited.”
[33] I. V. Serban, R. Lowe, L. Charlin, and J. Pineau, “A survey of
available corpora for building data-driven dialogue systems.” CoRR,
vol. abs/1512.05742, 2015. [Online]. Available: http://dblp.uni-trier.de/
db/journals/corr/corr1512.html#SerbanLCP15
[34] J. Cohen, “Kappa: Coefficient of concordance,” Educ. Psych. Measure-
ment, vol. 20, p. 37, 1960.
[35] R. Artstein and M. Poesio, “Inter-coder agreement for computational
linguistics,” Computational Linguistics, vol. 34, no. 4, pp. 555–596,
2008.
[36] R. Socher, J. Pennington, E. H. Huang, A. Y. Ng, and C. D. Manning,
“Semi-supervised recursive autoencoders for predicting sentiment dis-
tributions,” in Proceedings of the conference on empirical methods in
natural language processing. Association for Computational Linguis-
tics, 2011, pp. 151–161.
[37] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S.
Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow,
A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser,
M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray,
C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar,
P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals,
P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng,
“TensorFlow: Large-scale machine learning on heterogeneous systems,”
2015, software available from tensorflow.org. [Online]. Available:
http://tensorflow.org/
[38] S. Godbole and S. Sarawagi, “Discriminative methods for multi-labeled
classification,” in Advances in Knowledge Discovery and Data Mining.
Springer, 2004, pp. 22–30.
[39] P. Fernandez-Gonzalez, C. Bielza, and P. Larranaga, “Multidimensional
classifiers for neuroanatomical data,” in ICML Workshop on Statistics,
Machine Learning and Neuroscience (Stamlins 2015), 2015.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2885304, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING 11

Phan Duc Anh received his B.E. degree from Hanoi

University of Science and Technology (HUST), Viet-
nam, in 2011, and his M.Sc. degree from the Uni-
versity of Poitiers, France, in 2013. He is currently
pursuing the Ph.D. degree at Nara Institute of Sci-
ence and Technology (NAIST), Graduate School of
Information Science. His research interests include
natural language processing and machine learning,
in particular, emotion analysis and opinion mining.

Yuji Matsumoto is currently a Professor of In-

formation Science at the Nara Institute of Science
and Technology. He received his M.Sc. and Ph.D.
degrees in information science from Kyoto Univer-
sity in 1979 and 1989, respectively. He joined the
Machine Inference Section of the Electrotechnical
Laboratory in 1979. He then served as an academic
visitor at the Imperial College of Science and Tech-
nology, a deputy chief of the First Laboratory at
ICOT, and an associate professor at Kyoto Univer-
sity. His main research interests are natural language
understanding and machine learning.

Hiroyuki Shindo received his B.E. and M.E. de-

grees from Waseda University, Japan, in 2007 and
2009, respectively, and his Ph.D. degree in engineer-
ing from Nara Institute of Science and Technology
(NAIST), Ikoma, Japan, in 2013. He is currently
working as an assistant professor at NAIST in the
Graduate School of Information Science. From 2009
to 2014, he was a researcher at NTT Communication
Science Laboratories. His research interests include
machine learning and computational linguistics.

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.