2019 IEEE International Conference on Fuzzy Systems

FUZZ-IEEE 2019
June 23-26, 2019
General Chairs The 2019 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2019), the world-leading
Tim Havens, USA event focusing on the theory and application of fuzzy logic, will be held in New Orleans, Louisiana,
Jim Keller, USA USA. Nicknamed the “Big Easy,” New Orleans is known for its round-the-clock nightlife, vibrant
live-music scene, and spicy Cajun cuisine. It is located on the Mississippi River, near the Gulf of
Program Chairs
Mexico, and is popular tourist destination for all ages.
Alina Zare, USA
Derek Anderson, USA FUZZ-IEEE 2019 will be hosted at the JW Marriott, a premier conference venue nestled in
the heart of the world-famous French Quarter. You will be steps away from some of New Orleans’s
Special Sessions Chairs most iconic nightlife and restaurants. Take a walk outside and visit Jackson Square, shop at the
Christian Wagner, UK lively French Market, or dance your way through Bourbon Street.
Thomas Runkler, Germany FUZZ-IEEE 2019 will represent a unique meeting point for scientists and engineers, from
Tutorials Chairs academia and industry, to interact and discuss the latest enhancements and innovations in the
Qiang Shen, UK field. The topics of the conference will cover all aspects of theory and applications of fuzzy logic
Jesus Chamorro, Spain and its hybridisations with other artificial and computational intelligence methods. In particular,
Keynotes Chair
FUZZ-IEEE 2019 topics include, but are not limited to:
Robert John, UK
Posters Chair Mathematical and theoretical foundations of fuzzy sets, fuzzy measures, and fuzzy integrals
Marek Reformat, Canada Fuzzy control, robotics, sensors, fuzzy hardware and architectures
Finance Chair Fuzzy data analysis, fuzzy clustering, classification and pattern recognition
Mihail Popescu, USA Type-2 fuzzy sets, computing with words, and granular computing
Conflict-of-Interest Chairs Fuzzy systems with big data and cloud computing, fuzzy analytics, and visualization
Sansanee Fuzzy systems design and optimization
Auephanwiriyakul, Thailand Fuzzy decision analysis, multi-criteria decision making and decision support
CT Lin, Australia Fuzzy logical and its applications to industrial engineering
Competitions Chairs Fuzzy modeling, identification, and fault detection
Christophe Marsala, France Fuzzy information processing, information extraction, and fusion
Mika Sato-Ilic, Japan Fuzzy web engineering, information retrieval, text mining, and social network analysis
Panel Sessions Chair Fuzzy image, speech and signal processing, vision, and multimedia data analysis
Humberto Bustince, Spain Fuzzy databases and informational retrieval
Publications Chairs Rough sets, imprecise probabilities, possibilities approaches
Anna Wilbik, Netherlands Industrial, financial, and medical applications
Tim Wilkin, Australia Fuzzy logic application in civil engineering and GIS
Registrations Chair Fuzzy sets and soft computing in social sciences
Marie-Jeanne Lesot, France Linguistic summarization, natural language processing
Local Arrangement Chairs Computational intelligence in security
Fred Petry, USA Hardware and software for fuzzy systems and logic
Paul Elmore, USA Fuzzy Markup Language and standard technologies for fuzzy systems
Publicity Chair Adaptive, hierarchical, and hybrid neuro- and evolutionary-fuzzy systems
Daniel Sanchez, Spain
Web Chair The conference will include regular oral and poster presentations, an elevator pitch competition,
Tony Pinar, USA
tutorials, panels, special sessions, and keynote presentations. Full details of the submission
process for papers, tutorials, and panels will be made available on the conference website:
http://www.fuzzieee.org

Important dates
Deadline for special session, tutorial, competition, and panel session proposals: October 8, 2018
Z•IE Notification of acceptance for tutorials, special sessions, and panels: November 2, 2018
Z Deadline for full paper submission: January 11, 2019
E

ANS
LE
Notification of paper acceptance: March 4, 2019
FU

E
OR

Deadline for camera-ready paper submission: April 1, 2019

NEW

Deadline for early registration: April 5, 2019

Conference: June 23-26, 2019

019 http://www.fuzzieee.org
2

Digital Object Identifier 10.1109/MCI.2018.2840740

 Volume 13 Number 3 ❏ August 2018
www.ieee-cis.org

Features
12 
Identifying DNA Methylation Modules Associated
with a Cancer by Probabilistic Evolutionary Learning
b y Je-Keun Rhee, Soo-Jin Kim, and Byoung-Tak Zhang
20 
Augmentation of Physician Assessments with Multi-Omics
Enhances Predictability of Drug Response: A Case Study
of Major Depressive Disorder
by Arjun Athreya, Ravishankar Iyer, Drew Neavin, Liewei Wang,

Richard Weinshilboum, Rima Kaddurah-Daouk, John Rush,
Mark Frye, and William Bobo

Columns
32 Research Frontier
 Optimal Weighted Extreme Learning Machine for Imbalanced Learning
with Differential Evolution
by JongHyok Ri, Liang Liu,Yong Liu, Huifeng Wu,Wenliang Huang, and Hun Kim
Learning Without External Reward
by Haibo He and Xiangnan Zhong
on the cover
©istockphoto.com/kirstypargeter
55 Review Article
 Recent Trends in Deep Learning Based Natural Language Processing
 by Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria

Departments
2 Editor’s Remarks 10 Guest Editorial
3 President’s Message Computational Intelligence
by Nikhil R. Pal Techniques in Bioinformatics
and Bioengineering
5 Conference Reports  by Richard Allmendinger,
Conference Report on IEEE Daniel Ashlock, and
Computational Intelligence Sansanee Auephanwiriyakul
Society ExCom Workshop 2018
 by Jie Lu 76 Conference Calendar
by Bernadette Bouchon-Meunier
8 Publication Spotlight
 by Haibo He, Jon Garibaldi,
Kay Chen Tan, Julian Togelius,
Yaochu Jin, and Yew Soon Ong
Promoting Sustainable Forestry

SFI-01681

IEEE Computational Intelligence Magazine (ISSN 1556-603X) is published quarterly by The Institute of Electrical and Electronics Engineers, Inc. Headquarters: 3 Park Avenue, 17th
Floor, New York, NY 10016-5997, U.S.A. +1 212 419 7900. Responsibility for the contents rests upon the authors and not upon the IEEE, the Society, or its members. The magazine is a
membership benefit of the IEEE Computational Intelligence Society, and subscriptions are included in Society fee. Replacement copies for members are available for US$20 (one copy only).
Nonmembers can purchase individual copies for US$201.00. Nonmember subscription prices are available on request. Copyright and Reprint Permissions: Abstracting is permitted with
credit to the source. Libraries are permitted to photocopy beyond the limits of the U.S. Copyright law for private use of patrons: 1) those post-1977 articles that carry a code at the bottom of
the first page, provided the per-copy fee is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01970, U.S.A.; and 2) pre-1978 articles without fee. For other
copying, reprint, or republication permission, write to: Copyrights and Permissions Department, IEEE Service Center, 445 Hoes Lane, Piscataway NJ 08854 U.S.A. Copyright © 2018 by The
Institute of Electrical and Electronics Engineers, Inc. All rights reserved. Periodicals p­ ostage paid at New York, NY and at additional mailing offices. Postmaster: Send address changes to
IEEE Computational Intelligence Maga­zine, IEEE, 445 Hoes Lane, Piscataway, NJ 08854-1331 U.S.A. Printed in U.S.A. Canadian GST #125634188.

Digital Object Identifier 10.1109/MCI.2017.2770275 August 2018 | IEEE Computational intelligence magazine 1
CIM Editorial Board Hisao Ishibuchi Editor’s
Editor’s
Editor-in-Chief
Hisao Ishibuchi
Southern University of Science Remarks
Southern University of Science and Technology and Technology, CHINA
Department of Computer Science and Engineering Osaka Prefecture University, JAPAN
Shenzhen, Guangdong, CHINA
Osaka Prefecture University
Department of Computer Science
Sakai, Osaka 599-8531, JAPAN
(Email) hisaoi@cs.osakafu-u.ac.jp
depart title
Founding Editor-in-Chief
Gary G. Yen, Oklahoma State University, USA
One Year in China
Past Editor-in-Chief
Kay Chen Tan, City University of Hong Kong,
HONG KONG

Editors-At-Large
Piero P. Bonissone, Piero P. Bonissone Analytics
LLC, USA
David B. Fogel, Natural Selection, Inc., USA
Vincenzo Piuri, University of Milan, ITALY
Marios M. Polycarpou, University of Cyprus,
CYPRUS
Jacek M. Zurada, University of Louisville, USA
Associate Editors
José M. Alonso, University of Santiago de Compos-
tela, SPAIN
Erik Cambria, NTU, SINGAPORE
Raymond Chiong, The University of Newcastle,
AUSTRALIA
Yun Fu, Northeastern University, USA
Robert Golan, DBmind Technologies Inc., USA
Roderich Gross, The University of Sheffield, UK
Amir Hussain, University of Stirling, UK
John McCall, Robert Gordon University, UK
Zhen Ni, South Dakota State University, USA
Yusuke Nojima, OPU, JAPAN
Nelishia Pillay, University of Pretoria, SOUTH
AFRICA
Rong Qu, University of Nottingham, UK
Dipti Srinivasan, NUS, SINGAPORE
Ricardo H. C. Takahashi, UFMG, BRAZIL
Kyriakos G.Vamvoudakis, Virginia Tech, USA
Nishchal K.Verma, IIT Kanpur, INDIA
Dongrui Wu, DataNova, USA
Bing Xue, Victoria University of Wellington, NEW
ZEALAND
Dongbin Zhao, Chinese Academy of Sciences,
CHINA
IEEE Periodicals/
Magazines Department
Editorial/Production Associate, Heather Hilton
Senior Art Director, Janet Dudar
Associate Art Director, Gail A. Schnitzer
Production Coordinator, Theresa L. Smith
Director, Business Development—
Media & Advertising, Mark David
Advertising Production Manager,
Felicia Spagnoli
Production Director, Peter M. Tuohy
Editorial Services Director, Kevin Lisankie
Staff Director, Publishing Operations,
Dawn M. Melley
S
ince April 2017, I have been working in China. I live in an
apartment on the university campus. When I moved to
China last year, I knew only two Chinese words, “ni hao
(hello)” and “xie xie (thank you)”. Surprisingly I have yet to learn
any other Chinese words. I rely on my secretary’s help which I
am very thankful for. My biggest achievement in China has been
the success in my weight loss. I have lost more than 10 kg! I eat
three meals a day at the university’s canteen. It opens seven days a week but only
for five hours a day: two hours for breakfast, one and a half hours each for lunch
and dinner. Due to this, my daily schedule is clock work around the canteen’s
opening hours. Also the large and hilly campus lets me exercise regularly for my
weight control too.
Recently I visited Xiamen for a conference. Three undergraduate students came to
see me at the high-speed train station with a large name board (see the photo below).
I enjoyed dinner and a one-hour metro ride to the conference hotel. Thank you to
the student who brought back
the name board to the univer-
sity. On the last day in Xiamen,
they took me to the train sta-
tion and gave me some local
cakes and nuts. One of the stu-
dents spoke excellent Japanese.
Her Japanese was much better
than my English. When I re-
ceived a phone call from her With three first-year undergraduate students in Xiamen
before meeting them, I thought
that the call came from Japan. She was in Japan for one year when she was a high-
school student. I was so impressed by her language learning ability while comparing it
to my progress with Chinese after one year. Also the other student showed me her
amazing photos of her hometown about 600 km north of Xi’an. She told me that she
often enjoyed camel rides when she was a small girl. Next year, we will host the IEEE
SSCI 2019 conference in Xiamen. I am looking forward to seeing all of you and those
students again.
The feature topic of the current CIM issue is “Bioinformatics and Bioengineer-
ing.” This has been an interesting and challenging application field for CI techniques. I
hope you will enjoy all of the articles in this issue.

IEEE prohibits discrimination, harassment, and bullying.

For more information, visit http://www.ieee.org/web/
abou-tus/whatis/policies/p9-26.html.

Digital Object Identifier 10.1109/MCI.2017.2770276 Digital Object Identifier 10.1109/MCI.2018.2840642

Date of publication: 18 July 2018

2 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

 President’s Nikhil R. Pal CIS Society Officers
Message Indian Statistical President – Nikhil R. Pal,
Indian Statistical Institute, INDIA
Institute, INDIA Past President – Pablo A. Estévez,
University of Chile, CHILE
Vice President – Conferences – Bernadette
Bouchon-Meunier, University Pierre et Marie
Curie, FrANCE
Vice President – Education – Simon M. Lucas,
Queen Mary University of London, UK
Vice President – Finances – Enrique H. Ruspini,
Sharing Some Experience USA
Vice President – Members Activities –
Pau-Choo (Julia) Chung, National Cheng
Kung University,TAIWAN
Vice President – Publications – Jim Keller,
University of Missouri, USA

W
Vice President – Technical Activities –
e held our society’s first Executive Committee (ExCom) Hussein Abbass, University of New South Wales,
AUSTrALIA
meeting of this year on April 8, 2018, in Sydney, Austra-
lia. Although I had attended several ExCom meetings as Publication Editors
the Vice-President for Publications and as the President-Elect, this IEEE Transactions on Neural Networks
was my first meeting as the President and hence was of a special and Learning Systems
Haibo He, University of rhode Island, USA
importance to me. We had a long but very successful meeting. I
IEEE Transactions on Fuzzy Systems
take this opportunity to sincerely thank all members of the Jon Garibaldi, University of Nottingham, UK
ExCom for their very active participation and providing useful suggestions, which IEEE Transactions on Evolutionary Computation
will certainly help to take the society ahead. Kay Chen Tan, City University of Hong Kong,
In our society we make use of every opportunity to strengthen our interaction HONG KONG
IEEE Transactions on Games
with people and it is not restricted to the members of the Computational Intelli-
Julian Togelius, New York University, USA
gence Society (CIS). Keeping this in mind, we carefully choose the locations for our IEEE Transactions on Cognitive and
ExCom meetings and usually organize one (and sometimes even two) one-day Developmental Systems
workshop on computational intelligence and its applications in the region where Yaochu Jin, University of Surrey, UK
the meeting is held. To the extent I know, this initiative was started in 2011 and so IEEE Transactions on Emerging Topics in
Computational Intelligence
far it has been found to be very successful in achieving its intended goals of allow-
Yew Soon Ong, Nanyang Technological University,
ing us to reach out to students, researchers, and practitioners. Generally we find a SINGAPOrE
“champion” from a local university/institute or from the local CIS chapter, if any, to
organize the event. Every ExCom member, who attends the ExCom meeting in Administrative Committee
person, serves as a speaker to the workshop. In addition, to strengthen our bonding Term ending in 2018:
with the local researchers as well as to get a better idea of some of the research Piero P. Bonissone, Piero P. Bonissone Analytics
activities that the local researchers are engaged in, often we invite a few speakers LLC, USA
Carlos A. Coello Coello, CINVESTAV-IPN,
from the neighboring institutes/universities. In the past years we organized very
MEXICO
successful events in different countries including Brazil, Cyprus, Peru, Spain, Ecua- Barbara Hammer, Bielefeld University, GErMANY
dor, Chile, and Argentina. This year, after the ExCom meeting, we have organized Alice E. Smith, Auburn University, USA
two such workshops, one in Sydney (on April 9) and the other in Canberra (on Jacek M. Zurada, University of Louisville, USA
April 10). In Sydney, the workshop was organized by the University of Technology Term ending in 2019:
Sydney (UTS) under the leadership of Prof. Jie Lu. It was a very well-attended Cesare Alippi, Politecnico di Milano, ITALY
event—more than 100 researchers/students participated in this workshop. The sec- James C. Bezdek, USA
ond workshop was organized by the University of New South Wales (UNSW) Gary B. Fogel, Natural Selection, Inc., USA
Hisao Ishibuchi, Southern University of Science
Canberra and Prof. Hussein Abbass took the lead in organizing this event. This was and Technology, CHINA, and Osaka Prefecture
also a well-attended one with more than 80 participants. For this event, in addition University, JAPAN
to the ExCom members, Prof. Jason Scholz, Adjunct Professor, University of New Kay Chen Tan, City University of Hong Kong,
South Wales in Canberra and the Chief Scientist and Engineer, Defence Coopera- HONG KONG
tive Research Centre (DCRC) was invited to give a talk. He spoke on “Trusted Term ending in 2020:
Autonomous Systems Defence CRC”. These workshops covered a wide spectrum Janusz Kacprzyk, Polish Academy of Sciences,
of related topics of contemporary interests including deep learning, recognition POLAND
Sanaz Mostaghim, Otto von Guericke University of
technology, machine learning, big data challenges in astrophysics, fuzzy information
Magdeburg, GErMANY
processing, and human-machine systems. In both workshops the participants were Christian Wagner, University of Nottingham, UK
very enthusiastic and interactive. An important aspect of the Canberra program was Ronald R.Yager, Iona College, USA
that some young high school students were invited to attend the talk by Prof. Gary G.Yen, Oklahoma State University, USA

Digital Object Identifier 10.1109/MCI.2018.2840643

Date of publication: 18 July 2018 Digital Object Identifier 10.1109/MCI.2018.2840198

AUGUST 2018 | IEEE CompUTATIonAl InTEllIGEnCE mAGAzInE 3

 Bernadette Bouchon-Meunier. Her
talk was appropriately adapted to suit
the high school students. After the talk,
the students were given an opportuni-
ty to closely interact with Prof. Berna-
dette Bouchon-Meunier and Prof. Julia
Chung (both are CIS ExCom mem-
bers). This unique initiative was taken
to motivate young minds to get inter-
ested in engineering, particularly in
computational intelligence. Prof. Abbass
deserves a special thank for this inno-
Prof. Bernadette Bouchon-Meunier giving her talk at the UTS, Sydney workshop.
vative idea. Such an interaction with
high school students can have a long
lasting effect. In future, we plan to take
up more such initiatives to make high
school students interested in engineer-
ing and technology.
To conclude, I would like to remind
us that our favorite SSCI 2018 (IEEE
Symposium Series on Computational
Intelligence, 2018) will be held in Bengal-
uru, India during November 18–21, 2018.
I look forward to seeing you all there.
Prof. Pablo Estevez giving his talk at the UNSW, Canberra workshop.

IMAGE LICENSED BY INGRAM PUBLISHING

IEEE connects you to
a universe of information!
As the world’s largest professional
association dedicated to advancing
technological innovation and excellence
for the benefit of humanity, the IEEE
and its Members inspire a global
community through its highly cited
publications, conferences, technology
standards, and professional
and educational activities.

Visit www.ieee.org.

Publications / IEEE Xplore ® / Standards / Membership / Conferences / Education

4 IEEE CompUTATIonAl InTEllIGEnCE mAGAzInE | AUGUST 2018

 Jie Lu
Faculty of Engineering and Information technology,
University of Technology Sydney, AUSTrAlIA
Conference Report on IEEE Computational Intelligence
Society ExCom Workshop 2018
o n Monday 9th April 2018, the
IEEE Computational Intelli-
gence Society (CIS) ExCom
Workshop was hosted by the Faculty of
Engineering and Information Technol-
ogy (FEIT) at University of Technology
Sydney (UTS), chaired by Associate
Dean Research Excellence, Distin-
guished Professor Jie Lu.
Over 120 academic staff, industry
partners and PhD Students from UTS,
University of NSW, University of Syd-
ney and University of Wollongong,
along with IEEE CIS members in the
NSW Chapter attended the workshop.
The UTS IEEE Student Board were
also involved in the facilitation of the
event and on-hand on the day to answer
any questions about IEEE.
In her opening speech, Dist. Prof. Jie
Lu gave an introduction to the IEEE
CIS and its members, mentioning the
vast reach of IEEE and the IEEE CIS
across the world, its international stand-
ing as a leading society for technical
professionals and as a world-leading
publisher in the CI field.
Following this, the six IEEE CIS
ExCom members and world renowned
scholars from institutions in France, Tai-
wan, India, Chile, America and Australia
presented the latest theoretical and
application developments in CI.
During these talks, the speakers cov-
ered many interesting areas and applica-
tions of CI from neural networks to
classification, from fuzzy systems to
Digital Object Identifier 10.1109/MCI.2018.2840644
Date of publication: 18 July 2018
Prof. Nikhil R. Pal presenting to staff, students, and industry at University Technology Sydney.
autonomous behaviour, and from astron-
omy to bush-fires.
The talks gave the audience an
opportunity to hear insights from aca-
demics at the top of their fields, includ-
ing significant contributions to new
knowledge and applications with real-
world impact.
The audience were inspired, captivat-
ed and delighted to hear from renowned
professors on such pertinent topics,
which encouraged a lot of interest-
ing questions.
The program ran as follows:
Prof. Nikhil R. Pal, Electronics and
Communication Sciences Unit, Indian
Statistical Institute & current Presi-
dent of IEEE CIS spoke on “The
open world classification problem:
making neural networks say don’t
know.”
Prof . Ber nadette Bouchon-
Meunier, Emeritus Director of Re-
search, National Centre for Scientific
AUGUST 2018 | IEEE CompUTATIonAl InTEllIGEnCE mAGAzInE
Conference
Reports
Research, Sorbonne University, spoke
on “Fuzzy approaches of informa-
tion quality.”
Prof. Pablo A. Estévez, Electrical
Engineering, Universidad de Chile &
co-founder of the Millennium Institute
of Astrophysics, spoke on “Big data
challenges and deep learning appli-
cations to astronomy.”
Dist. Prof. James M Keller, Elec-
trical Engineering and Computer Sci-
ence, R.L Tatum Professor, University of
Missouri, spoke on “Recognition techn-
ology: Lofti’s look to the future from
the late 1990s.”
Dist. Prof. Pau-Choo (Julia)
Chung, Director General of the
Department of Information and Tech-
nology Education, Ministry of Educa-
tion,Taiwan spoke on “An overview to
deep learning development.”
Prof. Hussein Abbass, School of
Engineering and Information Technolo-
gy, UNSW, Canberra spoke on “Char-
acterising human and autonomous
5
systems behaviour in deceptive and
noisy environments.”
The workshop was very successful
and offered excellent opportunities for
IEEE CIS to promote its research devel-
opments. It provided a high-level inter-
national forum for scientists, engineers,
students and educators to learn state-of-
the-art research and applications in
computational intelligence.

************************

More questions for the CI experts.

While the IEEE CIS ExCom mem- From left to right: Dist. Prof. CT Lin, Dist. Prof. Jie Lu (UTS), Prof. Nikhil R. Pal (President), Prof.
bers were visiting for the workshop, Pablo Estevez, Dist. Prof. James Keller, Dist. Prof. Pau-Choo (Julia) Chung, Prof. Bernadette
UTS also took the opportunity to ask Bouchon-Meunier, Jo-Ellen Snyder, guest (IEEE Computational Intelligence Society).

them a few questions about computa-

tional intelligence, and captured it on
film. The resulting answers were very
insightful. Transcripts of some of these
answers are included below.
Q: What is the difference between
Artificial Intelligence (AI) and Com-
putational Intelligence (CI)?
Answer from Prof. Nikhil R. Pal
I think we need to look at the histo-
ry, AI formally started in late 1950’s and
at that time the emphasis was on sym-
bolic information processing, logical
reasoning—you know, the success sto-
ries of expert systems and other things.
At that time they (AI) did not give
much emphasis on learning or reason-
ing based on data. Research in compu-
tational neural networks (NN) started
earlier—the first computational model
of neuron was proposed in the 1940’s.
At the beginning AI people never
considered NN a part of AI but with
time, when CI and in particular NN,
which is one of the major pillars of CI,
started giving really superb results,
everybody [now] considers NN, espe-
cially deep NN [deep learning] as a part
of AI.
Q: What have been some of the
most successful applications of AI?
Answer from Dist. Prof Pau-
Choo (Julia) Chung
Currently [CI is] used in image
understanding, large image understand-
ing, autonomous systems and smart-
environment understanding [and many
other applications].
In this kind of environment it’s hard
for humans to know what kind of rule
you can embed, so you can use the
algorithm very precisely to do that. In
that case our society [has built] up this
technique, to let the algorithm learn the
situation, learn the environment and
learn from the data so that it will
become smarter and out-perform the
traditional approaches.
Q: What are the current challenges
in AI and robotics?
Answer from Prof. Bernadette
Bouchon-Meunier

Members of the IEEE Computational Intelligence Society with UTS hosts, IEEE Student Board volunteers, staff, and students.

6 IEEE CompUTATIonAl InTEllIGEnCE mAGAzInE | AUGUST 2018

 I think it is the human component
of robots which is not so obvious, for
instance: to understand the feelings, to
understand the subtlety of natural lan-
guage (which is not obvious), robots
are able to perform [normal tasks] very
efficiently. But as far as we go to human-
like behaviour, it is more and more dif-
ficult and this is where fuzzy logic is
very useful in managing a kind of
subjective, informal way of interacting
with people.
Q: What are the new research
directions and applications in CI?
Answer from Prof. Pablo A.
Estévez
There are several areas, in my case I
work with NN, so the idea is to classify
patterns but the traditional way to do
this is with labels. It is like teaching a kid
Are You Moving?
Don’t miss an issue of this magazine—
update your contact information now!
Update your information by:
E-MAIL: address-change@ieee.org
PHONE: +1 800 678 4333 in the United States
or +1 732 981 0060 outside
the United States
If you require additional assistance
regarding your IEEE mailings,
visit the IEEE Support Center
at supportcenter.ieee.org.
IEEE publication labels are printed six to eight weeks
in advance of the shipment date, so please allow sufficient
time for your publications to arrive at your new address.
to read, we do this by telling the kid
how to pronounce the words, and then
after repeating many times the kid
learns. This is the same way for ma-
chines. So we are talking about ma -
chine learning.
You can teach a machine to do
something. But the problem is when
you need labels, this is not useful when
you have say, 100 million of data, there
is no way humans will label that. So
one important area is to do the same
without labelling. To be able to recog-
nize the environment and the patterns
without humans telling the machine
what is there.
Answer from Dist. Prof. James
Keller
The population of the world is get-
ting older so more and more resources
are being pushed into that arena, so
© ISTOCKPHOTO.COM/BRIANAJACKSON
AUGUST 2018 | IEEE CompUTATIonAl InTEllIGEnCE mAGAzInE
that’s where I think a lot of activity is
going. And then of course you have all
the issues of cyber security and physical
security—e.g. landmines and parameters
detection. But all the cyber-attacks going
on, there is going to need to be an enor-
mous amount of work done to try to keep
our [personal] data secure.
Answer from Prof. Hussein
Abbass
The future is with connected data,
but what we need to start thinking
about is [how] to connect the smartness
that exists all over the place, and almost
independent of each other. How we can
actually leverage what we call distribut-
ed intelligence. Once we are able to
connect this intelligence we will quite
quickly see lots of advantages in society.
7
 Publication
Spotlight
CIS Publication Spotlight
IEEE Transactions on Neural
Networks and Learning Systems
Robust C-Loss Kernel Classifiers, by G.
Xu, B. Hu, and J. C. Principe, IEEE
Transactions on Neural Networks and
Learning Systems, Vol. 29, No. 3,
March 2018, pp. 510–522.
Digital Object Identifier: 10.1109/
TNNLS.2016.2637351
“The correntropy-induced loss (C-loss)
function has the nice property of being
robust to outliers. In this paper, we study
the C-loss kernel classifier with the Tik-
honov regularization term, which is used
to avoid overfitting. After using the half-
quadratic optimization algorithm, which
converges much faster than the gradient
optimization algorithm, we find out that
the resulting C-loss kernel classifier is
equivalent to an iterative weighted least
square support vector machine (LS-
SVM). This relationship helps explain the
robustness of iterative weighted LS-SVM
from the correntropy and density estima-
tion perspectives. On the large-scale data
sets which have low-rank Gram matri-
ces, we suggest to use incomplete Cho-
lesky decomposition to speed up the
training process. Moreover, we use the
representer theorem to improve the sparse-
ness of the resulting C-loss kernel classifier.
Experimental results confirm that our
methods are more robust to outliers than
the existing common classifiers.”
Digital Object Identifier 10.1109/MCI.2018.2840645
Date of publication: 18 July 2018
8 IEEE ComputatIonal IntEllIgEnCE magazInE | august 2018
Multicolumn RBF Network, by A. O.
Hoori and Y. Motai, IEEE Transac-
tions on Neural Networks and Learning
Systems,Vol. 29, No. 4, April 2018, pp.
766–778.
Digital Object Identifier: 10.1109/
TNNLS.2017.2650865
“This paper proposes the multicol-
umn RBF network (MCRN) as a
method to improve the accuracy and
speed of a traditional radial basis func-
tion network (RBFN). The RBFN, as a
fully connected artificial neural network
(ANN), suffers from costly kernel inner-
product calculations due to the use of
many instances as the centers of hidden
units. This issue is not critical for small
datasets, as adding more hidden units
will not burden the computation time.
However, for larger datasets, the RBFN
requires many hidden units with several
kernel computations to generalize the
problem. The MCRN mechanism is
constructed based on dividing a dataset
into smaller subsets using the k-d tree
algorithm. N resultant subsets are con-
sidered as separate training datasets to
train N individual RBFNs. Those small
RBFNs are stacked in parallel and
bulged into the MCRN structure dur-
ing testing. The MCRN is considered as
a well-developed and easy-to-use paral-
lel structure, because each individual
ANN has been trained on its own sub-
sets and is completely separate from the
other ANNs. This parallelized structure
reduces the testing time compared with
that of a single but larger RBFN, which
cannot be easily parallelized due to its
fully connected structure. Small infor-
Haibo He, Jon Garibaldi, Kay Chen Tan,
Julian Togelius, Yaochu Jin, and
Yew Soon Ong
mative subsets provide the MCRN with
a regional experience to specify the
problem instead of generalizing it. The
MCRN has been tested on many bench-
mark datasets and has shown better
accuracy and great improvements in
training and testing times compared
with a single RBFN. The MCRN also
shows good results compared with those
of some machine learning techniques,
such as the support vector machine and
k-nearest neighbors.”
IEEE Transactions
on Fuzzy Systems
Analysis and Design of Functionally
Weighted Single-Input-Rule-Modules
Connected Fuzzy Inference Systems, by
C. Li, J. Gao, J. Yi, and G. Zhang,
IEEE Transactions on Fuzzy Systems,
Vol. 26, No 1, February 2018, pp.
56–71.
Digital Object Identifier: 10.1109/
TFUZZ.2016.2637369
“Single-input-rule-modules (SIRMs)
can efficiently solve the fuzzy rule explo-
sion phenomenon, which usually occurs
in the multivariable modeling and/or
control applications. However, the per-
formance of SIRM connected fuzzy
inference systems (SIRM-FIS) is limited
due to its simple input-output mapping.
In this paper, to further enhance the per-
formance of SIRM-FIS, a functionally
weighted SIRM-FIS (FWSIRM-FIS),
which adopts multi-variable functional
weights to measure the important degrees
of the SIRMs, is presented.Then, in order
to show the fundamental differences of
the methods, various properties of a
range of SIRM-FIS models are explored.
These properties demonstrate that the
proposed FWSIRM-FIS has more gen-
eral and complex input-output mapping
than the existing SIRMs methods. Fur-
thermore, based on the least-squares
method, a novel data-driven optimiza-
tion method is presented for parameter
learning, which can also be used to
optimize the parameters of the other
models. Due to the properties of the
least-squares method, the proposed
parameter learning algorithm can over-
come the drawbacks of gradient-based
parameter learning methods and obtain
both smallest training errors and small-
est parameters. Finally, examples and
detailed comparisons are given. Simula-
tion results show that FWSIRM-FIS
can obtain better performance than the
other methods, and, compared with some
well-known methods, can achieve similar
or better performance with fewer param-
eters and faster training speed.”
On Distributed Fuzzy Decision Trees for
Big Data, by A. Segatori, F. Marcello-
ni, and W. Pedrycz, IEEE Transactions
on Fuzzy Systems,Vol. 26, No. 1, Feb-
ruary 2018, pp. 174–192.
Digital Object Identifier: 10.1109/
TFUZZ.2016.2646746
“Fuzzy decision trees (FDTs) have
shown to be an effective solution in the
framework of fuzzy classification. The
approaches proposed so far to FDT
learning, however, have generally neglect-
ed time and space requirements. In this
paper, the authors propose a distributed
FDT learning scheme shaped according
to the MapReduce programming model
for generating both binary and multiway
FDTs from big data. The scheme relies
on a novel distributed fuzzy discretizer
that generates a strong fuzzy partition
for each continuous attribute based on
fuzzy information entropy. The fuzzy
partitions are, therefore, used as an input
to the FDT learning algorithm, which
employs fuzzy information gain for
selecting the attributes at the decision
nodes. The authors have implemented
the FDT learning scheme on the
Apache Spark framework. They have
used ten real-world publicly available
big datasets for evaluating the behavior
of the scheme along three dimensions:
1) performance in terms of classification
accuracy, model complexity, and execu-
tion time; 2) scalability varying the
number of computing units; and 3) abil-
ity to efficiently accommodate an
increasing dataset size. They have dem-
onstrated that the proposed scheme
turns out to be suitable for managing
big datasets even with a modest com-
modity hardware support. Finally, they
have used the distributed decision tree
learning algorithm implemented in the
MLLib library and the Chi-FRBCS-
BigData algorithm, a MapReduce dis-
tributed fuzzy rule-based classification
system, for comparative analysis.”
IEEE Transactions
on Evolutionary Computation
A Review on Bilevel Optimization: From
Classical to Evolutionary Approaches and
Applications, by A. Sinha, P. Malo, K.
Deb, IEEE Transactions on Evolution-
ary Computation,Vol. 22, No. 2, April
2018, pp. 276–295.
Digital Object Identifier: 10.1109/
TEVC.2017.2712906
“Bilevel optimization is defined as a
mathematical program, where an opti-
mization problem contains another
optimization problem as a constraint.
These problems have received signifi-
cant attention from the mathematical
programming community. Only limited
work exists on bilevel problems using
evolutionary computation techniques;
however, recently there has been an
increasing interest due to the prolifera-
tion of practical applications and the
potential of evolutionary algorithms in
tackling these problems. This paper pro-
vides a comprehensive review on bilevel
optimization from the basic principles
to solution strategies; both classical and
evolutionary. A number of potential
application problems are also discussed.
To offer the readers insights on the
prominent developments in the field of
bilevel optimization, the authors have
august 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE
performed an automated text-analysis
of an extended list of papers published
on bilevel optimization to date. This
paper should motivate evolutionary
computation researchers to pay more
attention to this practical yet challeng-
ing area.”
IEEE Transactions on Games
Informed Hybrid Game Tree Search for
General Video Game Playing, by T.
Joppen, M. U. Moneke, N. Schröder,
C. Wirth, and J. Fürnkranz, IEEE
Transactions on Games, Vol. 10, No. 1,
March 2018, pp. 78–90.
Digital Object Identifier: 10.1109/
TCIAIG.2017.2722235
“In this paper, we introduce a univer-
sal game playing agent that is able to suc-
cessfully play a wide variety of video
games. It combines the strengths of
Monte Carlo tree search with conven-
tional heuristic search into a single hybrid
search agent, which is able to select the
appropriate strategy based on its obser-
vations about the game dynamics. In
particular, the agent learns a knowledge
base which provides the agent with
information such as an approximate
transition function, the type of agents
and objects that participate in the game
and the possible effects of interacting
with them, heuristics for focusing and
pruning the search, and more. This hy-
brid strategy proved to be successful in
the 2015 General Video Game Compe-
tition, in which our agent emerged as
the clear winner.”
IEEE Transactions on Cognitive
and Developmental Systems
Bootstrapping Q-Learning for Robotics
From Neuro-Evolution Results, by M.
Zimmer and S. Doncieux, IEEE
Transactions on Cognitive and Develop-
mental Systems, Vol. 10, No. 1, March
2018, pp. 102–119.
Digital Object Identifier: 10.1109/
TCDS.2016.2628817
(continued on page 11)
9
 Guest
Editorial
Computational Intelligence Techniques
in Bioinformatics and Bioengineering
t he field around Bioinformatics
and Bioengineering is rich and
includes important and diverse
problems, such as protein structure pre-
diction, systems and synthetic biology,
feature discovery and induction, health-
care informatics, biomarker discovery,
and development of personalized medi-
cine and treatment, amongst others.
Many of the above problems can be
framed as optimization, modeling and/
or learning problems that are too diffi-
cult to tackle via classical techniques.
Consequently, a consensus is emerging
that current state-of-the-art approaches,
such as sampling-based schemes in the
Rosetta suite for macromolecular mod-
eling, or classical mathematical pro-
gramming methods, have reached a
saturation point and are not very effec-
tive for vast multi-modal landscapes
encountered in this domain.
Over the past decade, it has become
clear that increases in computational
power alone will not be sufficient to
combat this problem, and that there is a
need for the development of specialized
search and learning procedures that
exploit problem-specific features and are
capable of reusing information gathered
during the problem solving procedure.
The field around optimization and
learning via computational intelligence
offers a repertoire of candidate tech-
niques for global optimization and
learning, as well as a rich body of theo-
retical and empirical work relating to
Digital Object Identifier 10.1109/MCI.2018.2840658
Date of publication: 18 July 2018
10 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
Many Bioinformatics and Bioengineering problems
can be framed as optimization, modeling and/or
learning problems that are too difficult to tackle
via classical techniques.
their tuning and performance in differ-
ent problem domains. Large parts of this
expertise are yet to make their debut in
the domain of bioinformatics and bio-
engineering, as the knowledge exchange
between the two fields has been limited.
It is only very recently that this boundary
has started to break down, and promising
preliminary applications have underlined
the potential of this research direction.
Given this, a special issue in a popular
journal, such as IEEE Computational
Intelligence Magazine, is particularly time-
ly and will help further draw attention to
this emerging research area.
The computational intelligence com-
munity in bioinformatics and bioengi-
neering is fragmented and large. The aim
of the special issue is to capture some of
the ongoing interdisciplinary research
that draws upon joint expertise in the
domains of optimization and learning via
computational intelligence techniques
and bioinformatics and bioengineering.
After a rigorous review process, two papers
were selected for publication in the spe-
cial issue.
The first paper, “Identifying DNA
methylation modules associated with a cancer
by probabilistic evolutionary learning” by Je-
Keun Rhee, Soo-Jin Kim, and Byoung-
Tak Zhang, aims at improving our
Richard Allmendinger
University of Manchester, UK
Daniel Ashlock
University of Guelph, Canada
Sansanee Auephanwiriyakul
Chiang Mai University, Thailand
understanding on the effects of DNA
methylation on complex diseases. In
particular, the paper focuses on identify-
ing multiple interactions of many DNA
methylation sites in the context of can-
cer. The authors demonstrate that com-
putational intelligence (more precisely,
an estimation of distribution algorithm-
based evolutionary algorithm) can be
used to identify high-order interactions
of DNA methylated sites that are poten-
tially relevant to a disease. The meth-
odology has been validated successfully
on array- and sequencing-based high-
throughput DNA methylation profil-
ing datasets.
The second paper, “Augmentation of
physician assessments with multi-omics
enhances predictability of drug response: A
case study of major depressive disorder” by
Arjun Athreya, Ravishankar Iyer, Drew
Neavin, Liewei Wang, Richard Weinshil-
boum, Rima Kaddurah-Daouk, John
Rush, Mark Frye, William Bobo, pro-
poses a learning-augmented clinical
assessment workflow to sequentially
augment a physician’s assessment of
patients symptoms and their socio-
demographic measures with heteroge-
neous biological measures to accurately
predict treatment outcomes using
machine learning and computational
intelligence. Using real data from a clini-
cal trial as a case study, the paper dem-
onstrates that the proposed approach can
yield significant improvements in the
prediction accuracy for antidepressant
treatment outcomes in patients with major
depressive disorder, compared to using
only a physician’s assessment as the pre-
dictor. In other words, the paper argues
that a properly tuned prediction model
can be used to assess the therapeutic
efficacy for a new patient prior to treat-
ment. Ultimately, the approach proposed
may find applications beyond psychia-
try, for example, for predicting treatment
Publication Spotlight (continued from page 9)
“Reinforcement learning (RL) prob-
lems are hard to solve in a robotics con-
text as classical algorithms rely on
discrete representations of actions and
states, but in robotics both are continu-
ous. A discrete set of actions and states
can be defined, but it requires an exper-
tise that may not be available, in particu-
lar in open environments. It is proposed
to define a process to make a robot
build its own representation for an RL
algorithm. The principle is to first use a
direct policy search in the sensori-motor
space, i.e., with no predefined discrete
sets of states nor actions, and then extract
from the corresponding learning traces
discrete actions and identify the relevant
dimensions of the state to estimate the
value function. Once this is done, the
robot can apply RL: 1) to be more
robust to new domains and, if required
and 2) to learn faster than a direct policy
search. This approach allows to take the
best of both worlds: first learning in a
continuous space to avoid the need of a
specific representation, but at a price of a
long learning process and a poor gener-
alization, and then learning with an
adapted representation to be faster and
more robust.”
There is a need for the development of specialized
search and learning procedures that exploit problem-
specific features and are capable of reusing information
gathered during the search.
outcomes for other medical conditions,
such as migraine headaches or rheuma-
toid arthritis.
We would like to use this opportuni-
ty to thank all the authors for submit-
ting their high quality papers to the
special issue, and all the reviewers for
IEEE Transactions on Emerging
Topics in Computational
Intelligence
Insights on Transfer Optimization:
Because Experience is the Best Teacher,
by A. Gupta, Y. S. Ong, and L. Feng,
IEEE Transactions on Emerging Topics in
Computational Intelligence, Vol. 2, No.
1, February 2018, pp. 51–64.
Digital Object Identifier: 10.1109/
TETCI.2017.2769104
“Traditional optimization solvers
tend to start the search from scratch by
assuming zero prior knowledge about
the task at hand. Generally speaking, the
capabilities of solvers do not automati-
cally grow with experience. In contrast,
however, humans routinely make use of
a pool of knowledge drawn from past
experiences whenever faced with a new
task. This is often an effective approach
in practice as real-world problems sel-
dom exist in isolation. Similarly, practi-
cally useful artificial systems are expected
to face a large number of problems in
their lifetime, many of which will either
be repetitive or share domain-specific
similarities. This view naturally moti-
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE
their invaluable contribution in assessing
the submissions. Our final thanks go to
the Editor-in-Chief of IEEE CIM, Hisao
Ishibuchi, for the opportunity to publish
the special issue and his support and
advice throughout the process.
vates advanced optimizers that mimic
human cognitive capabilities; leveraging
on what has been seen before to accel-
erate the search toward optimal solu-
tions of never before seen tasks. With
this in mind, this paper sheds light on
recent research advances in the field of
global black-box optimization that
champion the theme of automatic knowl-
edge transfer across problems. We intro-
duce a general formalization of transfer
optimization, based on which the con-
ceptual realizations of the paradigm are
classified into three distinct categories,
namely sequential transfer, multitasking,
and multiform optimization. In addition,
we carry out a survey of different meth-
odological perspectives spanning Bayes-
ian optimization and nature-inspired
computational intelligence procedures
for efficient encoding and transfer of
knowledge building blocks. Finally, real-
world applications of the techniques are
identified, demonstrating the future
impact of optimization engines that
evolve as better problem-solvers over
time by learning from the past and from
one another.”
11
 Identifying DNA Methylation
Modules Associated with
a Cancer by Probabilistic
Evolutionary Learning
Je-Keun Rhee
Cancer Research Institute, College of Medicine,
Catholic University of Korea, Seoul, KoRea
Soo-Jin Kim
Research Institute of agriculture and Life Sciences,
College of agriculture and Life Sciences,
Seoul National University, Seoul, KoRea
Byoung-Tak Zhang
School of Computer Science & engineering,
Seoul National University, Seoul, KoRea
12 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
©istockphoto.com/ciphotos
Abstract—DNA methylation leads to inhibition of downstream
gene expression. Recently, considerable studies have been made
to determine the effects of DNA methylation on complex dis-
ease. However, further studies are necessary to find the multiple
interactions of many DNA methylation sites and their associa-
tion with cancer. Here, to assess DNA methylation modules
potentially relevant to disease, we use an Estimation of Distri-
bution Algorithm (EDA) to identify high-order interaction of
DNA methylated sites (or modules) that are potentially relevant
to disease. The method builds a probabilistic dependency
model to produce a solution that is a set of discriminative
methylation sites. The algorithm is applied to array- and
sequencing-based high-throughput DNA methylation profiling
datasets. The experimental results show that it is able to identify
DNA methylation modules for cancer.
1556-603x/18©2018IEEE
g
I. Introduction
enomic studies mainly aim to find genetic markers
that are associated with a phenotype. Based on DNA
sequences, researchers have searched for causal effects
on biological processes including gene regulatory
mechanisms and diseases. Although several risk factors have
been identified by the association studies, the genetic variants do
not fully explain the abnormal regulation because the biological
regulatory mechanism can be affected by many other factors, as
well as DNA sequence modification [1]–[4].
Epigenomics refers to the study of regulation of various
genomic functions that are controlled by another partially sta-
ble modification, but not DNA sequence variants [5]. Among
these, DNA methylation, which typically occurs at CpG dinu-
cleotides catalyzed by DNA methyltransferase, is a crucial epi-
genetic regulatory mechanism in cellular processes. DNA
methylation of CpG sites mostly causes silencing of the down-
stream gene. The enrichment of the differentially methylated
DNA fractions can contribute to specific abnormalities, includ-
ing complex diseases [6]–[8]. In particular, with the advent of
array and next generation sequencing (NGS) technology, many
researchers have carried out genome-wide DNA methylation
profiling studies [9]–[11], and the genome-wide studies have
reported that many genomic regions are differentially methyl-
ated in normal and abnormal cells [12]–[14].
However, a complex disease is caused by a combination of
dysegulatory effects of multiple genes [15]–[17]. That is, errors
of biological processes are not caused by the alteration of an
individual methylation level. Recently, Easwaran et al. hypothe-
sized that DNA hypermethylation modules preferentially target
important developmental regulators in embryonic stem cells
[18]. They found a set of genes whose DNA methylation con-
tributed to the stem-like state of cancer. Horvath et al. studied
aging effects of DNA methylation and identified co-methylat-
ed modules related to aging in the human brain and blood
tissue [19]. Zhang and Huang investigated the DNA co-meth-
ylation patterns frequently observed in cancer [20].
Here, we identify combinatorial modules of DNA methyla-
tion sites associated with human diseases using an evolutionary
learning approach (Figure 1). Evolutionary algorithms can approx-
imate solutions well for a variety of problems [21]–[25]. They
generate a new population through iterative updates and
selection using a guided search process in a feature space. We
utilized an Estimation of Distribution Algorithm (EDA)-based
learning approach to identify combinations of cancer-relat-
ed DNA methylation sites. In the EDA, the population is
evolved according to the probabilistic distribution in selected
individuals without conventional genetic operators such as cross-
over and mutation. As a result, the EDA can provide answers
in combinatorial optimization problems [26]–[29]. The
EDA-based methods have been previously applied in several
biological studies, and it has offered promising results for
Digital Object Identifier 10.1109/MCI.2018.2840659
Date of publication: 18 July 2018
complex problems where other methods failed to find a
good solution [30]–[32].
We investigated DNA methylation modules relevant to can-
cer, using the DNA methylation profiling datasets produced by
array- and sequencing-based approaches. The experimental
results showed that our method could identify DNA methyla-
tion modules related to cancer.
II. Methods
a. evolutionary Learning Procedure to Identify a Set
of DNa Methylation Sites associated with a Disease
EDAs evolve a population to find the optimal solution probabi-
listically. The initial population is constructed by randomly
selecting individuals. The individuals represent higher order
interactions of the methylated sites. The population size m is
decided empirically and the initial weight w j of the individ-
ual j (0 1 j 1 m) is randomly assigned with a small value
(- 1 1 w j 1 1).
In the evolutionary process, each individual is evaluated for
how discriminative the interaction is for the datasets. Better
individuals are then selected and a dependency tree is built by
fitting to the selected individuals. New individuals of the next
generation are generated using the probability distribution
within the tree structure, and replace the previous individuals.
The overall procedure is as follows:
Step 1) Set g ! 0
Step 2) Initialize population X (g) by random generation
Step 3) Evaluate individuals in X (g)
Step 4) Select a set of individuals by tournament selection from
X (g)
Step 5) Construct a dependency tree G (g) by measuring Kull-
back-Leibler divergence between variables
Step 6) Learn parameters using a probability distribution of the
set of selected at step 4
Step 7) Generate new individuals by sampling with joint distri-
bution from the G (g), and create a new population
X (g + 1)
Step 8) Set g ! g + 1
Step 9) If the termination criterion is not met, go to Step 3
Further details for steps 3 and 5 are explained in follow-
ing sections.
B. Learning Dependency Tree
The dependency tree is built from the selected individuals by
searching conditional dependencies between random variables.
The model is then optimized by a series of incremental updates
[33], [34], as follows:
Suppose that X is a population and X = {X 1, X 2, ..., X n}
represents a vector of variables with n features, i.e., DNA
methylation sites. The probability distribution is denoted by a
joint probability P (X 1, X 2, ..., X n) as to:
Corresponding Author: Byoung-Tak Zhang (Email: btzhang@bi.snu.ac.kr)
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 13
 Generation Sampling

Population New Population

X1 X2 X3 X5 X6
X1 X2 X3 X4 X5

X1 X7 X8 X2 X4 X8

X2 X3 X6 X7 X3 X4 X7 X9

X1 X4 X7 X8 X9 X1 X5 X6 X8 X10
Dependency
X3 X5 X9 X10 Graphs
X4 X6 X7 X9

Selection Evaluation

X1 X2 X3 X4 X5

X1 X2 X3 X5 X6
X2 X4 X8
X2 X4 X8
X2 X3 X6 X7 X3 X4 X7 X9
Training
Datasets
X1 X7 X8
X1 X5 X6 X8 X10

X4 X6 X7 X9 X4 X6 X7 X9

FIgurE 1 Schematic overview for probabilistic evolutionary learning to identify DNA methylation modules.

P (X) = P (X 1, X 2, ..., X n) I (X i; X pa (i)) =

= P (X 1|X 2, ..., X n) P (X 2|X 3, ..., X n) ....P (X n - 1|X n) P (X n).
(1) / / P (X i = x, X pa(i) = y) log PP(X(Xi =i =x)xP, X(Xpapa(i)(i=) =y)y) . (4)
x y

However, it is hard to measure all the joint probabilities

exactly when n, the number of variables, is large. Thus it is The complete graph G searches the maximum spanning tree,
necessary to approximate the probability distribution. In this and then the best dependency tree is constructed.
study, we used a dependency tree, and the distribution is For parameter learning, the most likely values are calculated
approximated as follows: from the frequencies in the selected individuals. That is, the
model parameters are represented as marginal probabilities in a
P (X 1, X 2, ..., X n) = P (X r ) % P (X i|X pa (i)), (2) root node and conditional probabilities in the other nodes. The
i!r
marginal probabilities in the root nodes and the conditional
where X 1, X 2, ..., X n are random variables, r is an index of a probabilities in the child nodes are calculated by Eqs. (5) and
root node, and pa(i) denotes the index of the parent node of (6), respectively, as follows:
X i. The tree structure is built by searching based on Kullback-
c (X r = x)
Leibler divergence between two random variables. The depen- P ( X r = x) = , (5)
N
dency graph is constructed optimally in a direction to
c (X i = x, X pa (i) = y)
maximize total mutual information as follows: P (X i = x|X pa (i) = y) = , (6)
c (X pa (i) = y)

argmax r, pa % I (X i; X pa (i)), (3) where c is the count of a variable X with a specific value and
i!r N is the total number of individuals.

14 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

C. Fitness evaluation in a Population
The fitness function represents how informative the chromo-
some is to classify the samples. That is, the fitness for an individ-
ual is evaluated by measuring the classification accuracy for
interaction of the features. To determine and update the fitness
for each individual, we introduce a gradient descendant rule for
training data D as follows:
w i = w i + h (t j - f ( D j)) v ji, (7)
where w i is the weight value for i-th feature and t j is the target
class in the j-th training instance D j . h is the learning rate and
v ji is the value of the i-th attribute in the j-th instance. f (D j) is
the predicted output value of the j-th training instance by our
model and determined as follows:
n
The DNA methylation levels of the two datasets were rep-
resented as beta-values, which were bounded between 0
(unmethylated) and 1 (totally methlyated).
III. Results
a. DNa Methylation Module associated with Breast Cancer
This analysis was carried out based on DNA methylation pro-
filing datasets that experimentally measured the methylation
statuses using DNA Methylation BeadChip [35]. We extracted
data for DNA methylation profiles on chromosome 17 from
breast cancer and normal samples. Then, the data used at our
experiment consist of total 99 samples with 82 cancer and 17
normal samples with 1,587 features. Figure 2 shows the learn-
ing curves in the evolutionary process. The fitness value was
improved when the number of generations increased. We intro-
duced a term, in the fitness function, for the number of the

f ( D j) = *
1, if / w i $ v ji 2 0, (8)
methylation sites to find an individual with a shorter length;
i=0
- 1, otherwise.

1.0
The difference between the predictions and the target values
specified in the training sequence is used to represent the error 0.9
of the current weight vector. The target function is optimized
0.8
to minimize the classification error. The weight values are eval-
uated against a sequence of training samples and are updated to 0.7
Fitness

improve the classification accuracy. The weight update processes

are repeated until they converge after a number of epochs. 0.6
Using the learning scheme, we identify the most informa-
0.5
tive individuals for classification, where the absolute values of Maximum
their weights are large. In addition, it is better to find the DNA 0.4 Mean
methylation module, whose number of features is small. Finally, Minimum
the fitness function for the k-th individual X k, Fitness (X k ) is 0.3
defined as follows: 0 10 20 30 40 50 60
Generation
(a)
Fitness (X k ) = Acc (X k ) - Order (X k ), (9)
1,000
Maximum
where Acc (X k ) is the classification accuracy for training datasets
Mean
and Order (X k ) denotes the number of methylation sites which Minimum
800
are selected in the individual X k.

D. Dataset 600
Order

The high-throughput DNA methylation profiles of large

genomic regions can be produced by both array and NGS 400
technologies. We applied our approach to these two types of
datasets. The array data were generated by the Illumina Infini- 200
um 27 k Human DNA methylation BeadChip, for surveying
genome-wide DNA methylation profiles in breast cancer and
0
normal samples [35]. We downloaded the dataset from Gene
Expression Omnibus accession number GSE32393, and re- 0 10 20 30 40 50 60
Generation
moved the samples with missing values. Sequence-based datasets
(b)
were produced by MethylCap-seq in matched normal and
colorectal cancer samples and collected at GSE39068 [36]. Nor-
FIgurE 2 Learning curve using breast cancer datasets. The x-axis is
malization and preprocessing were carried out using the ap- the number of generations and the y-axis shows (a) fitness values and
proaches detailed by Simmer et al. [36]. (b) the number of methylation sites.

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 15

therefore, the order decreased with the learning process (Fig- In addition, it has been reported that hypermethylation of
ure 2(b)). After convergence, six sites were selected for the dis- ALOX12 is associated with cancer [39]–[42]. Indeed, the
crimination. These six sites were related to genes, KIAA1267, ALOX12 gene is closely related to apoptosis, and alterations in its
CD79B, ALOX12, TMEM98, KRT19 and FOXJ1 (Table 1). expression caused by DNA methylation can cause a malfunction
ALOX12 has a role in the growth of breast cancer and its in cell death [43]–[45]. Therefore, it is reasonable to hypothesize
inhibition may be a strategy for inhibiting tumor growth [37]. that a change of methylation in the gene is linked to cancer,
The gene can be used as a serum marker for breast cancer [38]. including breast tumors. KRT19 is a well-known marker for
breast cancer [46], [47], and the KRT19 promoter can be aber-
TAbLE 1 Finally Selected Methylation Sites. rantly methylated in cancer cell lines [48]. The CD79B gene has
also been shown to be related to breast cancer in several studies
ID PoSITIoN gENE CgI LoCATIoN
[49], [50]. FOXJ1, a member of the forkhead box (FOX) family,
cg02301815 41605268 KiAA1267 41605074-41605445
may function as a tumor suppressor gene in breast cancer [51].
cg07973967 59363339 cD79B 25467633-25468370 FOXJ1 is hypermethylated and silenced in breast cancer cell lines
cg08946332 6840612 ALoX12 6839463-6841283 [52]. TMEM98 is a transmembrane protein. Recently, Grimm et
cg11833861 28279748 TMeM98 28278827-28279833 al. investigated transmembrane proteins specific for cancer cells,
cg16585619 36938776 KRT19 No cgi* and showed that the transmembrane proteins can be targets for
cg24164563 71647990 FoXJ1 71647419-71649480
antibodies and may form biomarkers for tumor diagnosis, prog-
*This site is not located within a cgi.
nosis, and treatment [53]. The function of KIAA1267 is unclear
yet, but this gene encodes KAT8 regulatory NSL complex sub-
unit 1, and KAT8 regulates p53, a tumor suppressor gene [54],
TAbLE 2 Classification Performance by Splitting Training [55]. Our results suggest that KIAA1267 also can have a role in
and Test Data.
breast cancer.
ALgorIThM ACCurACy SENSITIvITy SPECIFICITy To verify that our method produced good classification
LogiSTic RegReSSioN 0.947 0.919 0.768 performance generally, we calculated the classification perfor-
SVM 0.908 0.975 0.476 mance by randomly separating the original dataset into train-
DeciSioN TRee 0.928 0.894 0.768 ing and test datasets. Table 2 shows the average accuracy,
NAiVe BAyeS 0.935 0.928 0.772 sensitivity and specificity for 20 times repetition of random

TAbLE 3 Classification Performance Using the Selected Sites and Randomly Selected Sites.

ALgorIThM FEATurE* ACCurACy SENSITIvITy SPECIFICITy

LogiSTic RegReSSioN SeLecTeD 0.939 0.987 0.762
f=5 0.834 0.968 0.191
f=6 0.839 0.967 0.224
f = 10 0.855 0.949 0.405
f = 20 0.893 0.950 0.621
SVM SeLecTeD 0.929 0.941 0.857
f=5 0.829 0.999 0.008
f=6 0.830 0.998 0.018
f = 10 0.833 0.995 0.054
f = 20 0.867 0.986 0.304
DeciSioN TRee SeLecTeD 0.939 0.952 0.867
f=5 0.822 0.936 0.269
f=6 0.822 0.93 0.302
f = 10 0.826 0.908 0.431
f = 20 0.849 0.910 0.555
NAiVe BAyeS SeLecTeD 0.919 0.951 0.765
f=5 0.774 0.817 0.568
f=6 0.769 0.802 0.613
f = 10 0.795 0.804 0.753
f = 20 0.837 0.843 0.810
*At the column Feature, f is the number of randomly selected sites, and selected means the selected sites by our method.

16 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

 TAbLE 4 Classification Performance Using Only the Selected
1.0 Sites in Colorectal Cancer.

ALgorIThM ACCurACy SENSITIvITy SPECIFICITy

0.9
LogiSTic 0.900 0.920 0.880
RegReSSioN
0.8
SVM 0.940 0.960 0.920
Fitness

DeciSioN TRee 0.640 0.680 0.600

0.7
NAiVe BAyeS 0.900 0.920 0.880

0.6

0.5 Maximum TAbLE 5 Enriched Geneset in Colorectal Cancer Data.

Mean
gENE SET p-vALuE FDr q-vALuE
Minimum
0.4 NoN-SMALL ceLL LuNg cANceR 2.61e-05 4.25e-03
0 20 40 60 80 100 120 gLioMA 4.56e-05 4.25e-03
Generation
NeuRoTRophiN SigNALiNg 3.25e-04 1.85e-02
pAThwAy
FIgurE 3 Learning curve using colorectal cancer datasets. The x-axis pAThwAyS iN cANceR 3.99e-04 1.85e-02
is the number of generations and the y-axis is fitness values.
wNT SigNALiNg pAThwAy 5.52e-04 2.05e-02
ALDoSTeRoNe-ReguLATeD 9.09e-04 2.22e-02
SoDiuM ReABSoRpTioN
splitting, measured by conventional classification algorithms.
eNDocyToSiS 9.62e-04 2.22e-02
Our algorithm showed good classification results even at the
independent test set. For further verification, we randomly VASopReSSiN-ReguLATeD wATeR 9.97e-04 2.22e-02
ReABSoRpTioN
extracted the methylation sites with 100 times repetition, then
cheMoKiNe SigNALiNg pAThwAy 1.07e-03 2.22e-02
measured the classification performance in each dataset by
10-fold cross-validation. Table 3 shows that our method pro- FocAL ADheSioN 1.26e-03 2.34e-02

duced better results than the others, regardless of the number eNDoMeTRiAL cANceR 1.39e-03 2.35e-02
of the randomly selected sites. In particular, it was noted that BASAL ceLL cARciNoMA 1.55e-03 2.41e-02
the specificity using the selected sites by our method was coLoRecTAL cANceR 1.97e-03 2.73e-02
much better than the others, even though the original data pANcReATic cANceR 2.50e-03 2.73e-02
was highly imbalanced.
MeLANoMA 2.57e-03 2.73e-02
chRoNic MyeLoiD LeuKeMiA 2.72e-03 2.73e-02
B. Modules associated with Colorectal Cancer
using High-Throughput Sequencing Data cyToKiNe-cyToKiNe RecepToR 2.82e-03 2.73e-02
iNTeRAcTioN
Recently, high-throughput sequencing technologies have been
MApK SigNALiNg pAThwAy 2.82e-03 2.73e-02
used to determine DNA methylation profiles. We applied our
method to the sequencing-based methylation profile datasets phoSphATiDyLiNoSiToL 2.94e-03 2.73e-02
SigNALiNg SySTeM
produced by Simmer et al. [36].
VegF SigNALiNg pAThwAy 2.94e-03 2.73e-02
The experiments were carried out using 25 cancer and 25
normal samples with 10,393 genomic regions on chromosome Fc epSiLoN Ri SigNALiNg 3.17e-03 2.81e-02
pAThwAy
17. Figure 3 depicts the improvement of the fitness in iterative
SMALL ceLL LuNg cANceR 3.58e-03 2.98e-02
learning procedures using these datasets, and finally 348 regions
were selected to discriminate colorectal cancer and normal sam- eRBB SigNALiNg pAThwAy 3.83e-03 2.98e-02

ples after convergence. Table 4 shows the average classification
performance by 10-fold cross-validation using the selected sites.
We annotated the 348 selected regions using GPAT [56] and
investigated which genes were located close to the selected
regions. We determined which genes were enriched within the
KEGG pathway using the genes whose transcription start sites
are located within 5,000 bp from the selected genomic regions
[57], [58]. Table 5 summarizes the significantly enriched path-
ways with low p-values and shows that most of these are closely
associated with cancer-related networks. Note that the enriched
signaling pathways were related to colorectal cancer. In colorectal
ApopToSiS 3.92e-03 2.98e-02
pRoSTATe cANceR 4.01e-03 2.98e-02
cancer, the roles of the wnt signaling pathway and MAPK signal-
ing pathway have been studied intensively [59]–[62]. Genetic
mutations affecting the pathway components and the alteration
of their expression can enhance tumorigenicity in cancer cells. In
addition, the neurotrophin signaling pathway could be related to
growth of colorectal cancer cells [63] and the chemokine signal-
ing pathway suppresses colorectal cancer metastasis [64], [65].

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 17

The phosphatidylinositol signaling pathway plays an important
role in the growth, survival and metabolism of cancer cells, and
targeting this pathway has the potential to lead to treatments for
colorectal cancer [66], [67].VEGF and ErbB may be valid thera-
peutic targets for patients with colorectal cancer [68]–[71].
IV. Discussion and Conclusion
DNA methylation may be associated significantly with com-
plex diseases and many genomic regions are differentially
methylated in various cancers, comparing to normal samples.
In this study, we presented a method to identify combinatorial
effects of DNA methylation at multiple sites. From a systematic
perspective, the relationship between DNA methylation
regions and a specific disease is learned by the presented proba-
bilistic evolutionary learning method. The fitness value of a
DNA methylation module measures the level of its responses
to the cancer. In a computational view, our method can solve a
large number of feature problems by identifying modules with
both compactness and high coverage of cancer-related genes.
Applying our method to breast cancer and colorectal cancer
data produced by high-throughput technologies, we detected
cancer-related modules that were confirmed by the literature
and functional enrichment analysis. Interestingly, we observed
that the selected regions were located around genes that are
significantly enriched in cancer-related gene set categories,
which provided evidence that the identified modules in our
study are biologically meaningful.
Moreover, from the result for the array-based dataset, we could
obtain a good accuracy with a very small number of random fea-
tures. However, the specificity was very low in the experiments
with random features. The result suggested that our method could
generate well-balanced classification performance even with a
highly imbalanced dataset, although conventional classifiers would
not work well with imbalanced circumstances. Also in the second
experiment using the NGS-based dataset with large number of
features and small sample size, our method could find the informa-
tive DNA methylation sites with good classification performances,
even though the decision tree, necessary to be discretized in each
value, showed relatively lower results.
Studies on DNA methylation could reveal the process of
tumorigenesis as well as identify biomarkers. Our approach,
which identifies multiple DNA methylation sites that might be
epigenetically regulated, could provide a useful strategy to detect
the epigenetic association related to cancer. By applying our
method to array- and NGS-based data, we showed that it is
applicable to a variety of data types and various disease contexts.
Moreover, recent studies suggest a complex relationship between
genetic variation and DNA methylation. Systems genetics and
epigenetics approaches are required to examine these relation-
ships. Although our framework is based on DNA methylation
profile datasets, it could be used to identify the combinatorial
association of various factors, including gene expression levels,
microRNAs, copy number variations, genetic variations, and
environmental factors. The integration of a variety of data would
provide the basis for new hypotheses and experimental
18 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
approaches in the model of a complex disease. Moreover, the sys-
tematic identification of causal factors and modules would pro-
vide insights into mechanisms underlying complex diseases and
help to develop efficient therapies or effective drugs.
In summary, we presented a method for searching the higher-
order interaction of DNA methylation sites using a probabilistic
evolutionary learning method. We also examined the potential
for the combined effects of various sites on the genome. The
results suggested that the alteration of DNA methylations at
multiple sites affects cancer. Similar to genome-wide association
studies, our approach provided an opportunity to capture the
complex and multifactorial relationships among DNA methyla-
tion sites and to find new factors for future study. Therefore, our
approach would facilitate a comprehensive analysis of genome-
wide DNA methylation datasets and help the interpretation for
the effects of DNA methylation on multiple sites.
Acknowledgment
This work was supported by the National Research Foundation of
Korea (NRF) grant funded by the Ministry of Science and ICT,
Republic of Korea (grant no. NRF-2015R1C1A1A01053824,
NRF-2018R1C1B6005304, NRF-2016R1D1A1B03935676,
and NRF-2018R1D1A1B07050393).
References
[1] P. A. Jones and S. B. Baylin, “The epigenomics of cancer,” Cell, vol. 128, no. 4,
pp. 683–962, 2007.
[2] B. Sadikovic, K. Al-Romaih, J. Squire, and M. Zielenska, “Cause and consequences
of genetic and epigenetic alterations in human cancer,” Current Genomics, vol. 9, no. 6,
pp. 394–408, 2008.
[3] A. E. Handel, G. C. Ebers, and S. V. Ramagopalan, “Epigenetics: Molecular mecha-
nisms and implications for disease,” Trends Mol. Med., vol. 16, no. 1, pp. 7–16, 2010.
[4] J. Sandoval and M. Esteller, “Cancer epigenomics: Beyond genomics,” Current Opin-
ion Genetics Develop., vol. 22, no. 1, pp. 50–55, 2012.
[5] L. Bonetta, “Epigenomics: Detailed analysis,” Nature, vol. 454, pp. 795–798, 2008.
[6] K. Robertson, “DNA methylation and human disease,” Nature Rev. Genetics, vol. 6,
pp. 597–610, 2005.
[7] A. Portela and M. Esteller, “Epigenetic modifications and human disease,” Nature
Biotechnol., vol. 28, no. 10, pp. 1057–1068, 2010.
[8] P. Jones, “Functions of DNA methylation: Islands, start sites, gene bodies and be-
yond,” Nature Rev. Genetics, vol. 13, no. 7, pp. 484–492, 2012.
[9] P. Laird, “Principles and challenges of genomewide DNA methylation analysis,” Na-
ture Rev. Genetics, vol. 11, no. 3, pp. 191–203, 2010.
[10] V. K. Hill, C. Ricketts, I. Bieche, S. Vacher, D. Gentle, C. Lewis, E. R. Maher,
and F. Latif, “Genome-wide DNA methylation profiling of CpG islands in breast cancer
identifies novel genes associated with tumorigenicity,” Cancer Res., vol. 71, no. 8, pp.
2988–2999, 2011.
[11] J.-K. Rhee, K. Kim, H. Chae, J. Evans, P. Yan, B.-T. Zhang, J. Gray, P. Spellman,
T. H.-M. Huang, K. P. Nephew, and S. Kim, “Integrated analysis of genome-wide DNA
methylation and gene expression profiles in molecular subtypes of breast cancer,” Nucl.
Acids Res., vol. 41, no. 18, pp. 8464–8474, 2013.
[12] H. Cheung, T. Lee, A. Davis, D. Taft, O. Rennert, and W. Chan, “Genome-
wide DNA methylation profiling reveals novel epigenetically regulated genes and non-
coding RNAs in human testicular cancer,” Br. J. Cancer, vol. 102, no. 2, pp. 419–427,
2010.
[13] G. Toperoff, D. Aran, J. D. Kark, M. Rosenberg, T. Dubnikov, B. Nissan, J. Wain-
stein, Y. Friedlander, E. Levy-Lahad, B. Glaser, and A. Hellman, “Genome-wide survey
reveals predisposing diabetes type 2-related DNA methylation variations in human pe-
ripheral blood,” Hum. Mol. Genetics, vol. 21, no. 2, pp. 371–383, 2012.
[14] B. A. Walker, C. P. Wardell, L. Chiecchio, E. M. Smith, K. D. Boyd, A. Neri, F. E.
Davies, F. M. Ross, and G. J. Morgan, “Aberrant global methylation patterns affect the
molecular pathogenesis and prognosis of multiple myeloma,” Blood, vol. 117, no. 2, pp.
553–562, 2011.
[15] J. N. Hirschhorn and M. J. Daly, “Genome-wide association studies for common
diseases and complex traits,” Nature Rev. Genetics, vol. 6, no. 2, pp. 95–108, 2005.
[16] A. Janssens and C. van Duijn, “Genome-based prediction of common diseases: Ad-
vances and prospects,” Hum. Mol. Genetics, vol. 17, pp. R166–R173, 2008.
[17] A. Kiezun, K. Garimella, R. Do, N. O. Stitziel, B. M. Neale, P. J. McLaren, N.
Gupta, P. Sklar, P. F. Sullivan, J. L. Moran, C. M. Hultman, P. Lichtenstein, P. Mag-
nusson, T. Lehner, Y. Y. Shugart, A. L. Price, P. I. de Bakker, S. M. Purcell, and S. R.
Sunyaev, “Exome sequencing and the genetic basis of complex traits,” Nature Genetics,
vol. 44, no. 6, pp. 623–630, 2012.
[18] H. Easwaran, S. Johnstone, L. Van Neste, J. Ohm, T. Mosbruger, Q. Wang, M.
Aryee, P. Joyce, N. Ahuja, D. Weisenberger, E. Collisson, J. Zhu, S. Yegnasubramanian,
W. Matsui, and S. Baylin, “A DNA hypermethylation module for the stem/progenitor
cell signature of cancer,” Genome Res., vol. 22, pp. 837–849, 2012.
[19] S. Horvath, Y. Zhang, P. Langfelder, R. Kahn, M. Boks, K. van Eijk, L. van den
Berg, and R. Ophoff, “Aging effects on DNA methylation modules in human brain and
blood tissue,” Genome Biol., vol. 13, no. 10, p. R97, 2012.
[20] J. Zhang and K. Huang, “Pan-cancer analysis of frequent DNA co-methylation pat-
terns reveals consistent epigenetic landscape changes in multiple cancers,” BMC Genomics,
vol. 18, no. 1, p. 1045, 2017.
[21] M. Kumar, M. Husian, N. Upreti, and D. Gupta, “Genetic algorithm: Review
and application,” Int. J. Inf. Technol. Knowl. Manage., vol. 2, no. 2, pp. 451–454, 2010.
[22] K. Deb and R. Datta, “A fast and accurate solution of constrained optimization
problems using a hybrid bi-objective and penalty function approach,” in Proc. IEEE Con-
gr. Evolutionary Computation, 2010, pp. 1–8.
[23] J.-G. Joung, S.-J. Kim, S.-Y. Shin, and B.-T. Zhang, “A probabilistic coevolution-
ary biclustering algorithm for discovering coherent patterns in gene expression dataset,”
BMC Bioinform., vol. 13, no. Suppl 17, p. S12, 2012.
[24] R. Wang, R. C. Purshouse, and P. J. Fleming, “On finding well-spread pareto
optimal solutions by preference-inspired co-evolutionary algorithm,” in Proc. 15th Annu.
Conf. Genetic and Evolutionary Computation Conf., New York, NY, 2013, pp. 695–702.
[25] S.-J. Kim, J.-W. Ha, and B.-T. Zhang, “Bayesian evolutionary hypergraph learning
for predicting cancer clinical outcomes,” J. Biomed. Inform., vol. 49, pp. 101–111, 2014.
[26] T. Chen, P. Lehre, K. Tang, and X. Yao, “When is an estimation of distribution
algorithm better than an evolutionary algorithm?” in Proc. IEEE Congr. Evolutionary Com-
putation, 2009, pp. 1470–1477.
[27] A. Zhou, Q. Zhang, and Y. Jin, “Approximating the set of pareto-optimal solutions
in both the decision and objective spaces by an estimation of distribution algorithm,”
IEEE Trans. Evol. Comput., vol. 13, no. 5, pp. 1167–1189, 2009.
[28] V. Shim, K. Tan, J. Chia, and A. Al Mamun, “Multi-objective optimization with
estimation of distribution algorithm in a noisy environment,” Evol. Comput., vol. 21, no.
1, pp. 149–177, 2013.
[29] J. Ceberio, E. Irurozqui, A. Mendiburu, and J. Lozano, “A distance-based ranking
model estimation of distribution algorithm for the f lowshop scheduling problem,” IEEE
Trans. Evol. Comput., vol. 18, no. 2, pp. 286–300, 2014.
[30] S. Pal, S. Bandyopadhyay, and S. Ray, “Evolutionary computation in bioinfor-
matics: A review,” IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 36, no. 5, pp.
601–615, 2006.
[31] R. Santana, A. Mendiburu, N. Zaitlen, E. Eskin, and J. Lozano, “Multi-marker
tagging single nucleotide polymorphism selection using estimation of distribution algo-
rithms,” Artif. Intell. Med., vol. 50, no. 3, pp. 193–201, 2010.
[32] K. Shelke, S. Jayaraman, S. Ghosh, and J. Valadi, “Hybrid feature selection and
peptide binding affinity prediction using an EDA based algorithm,” in Proc. IEEE Congr.
Evolutionary Computation, 2013, pp. 2384–2389.
[33] M. Pelikan, “Implementation of the dependency-tree estimation of distribution al-
gorithm in C++,” 2006.
[34] M. Pelikan, S. Tsutsui, and R. Kalapala, “Dependency trees, permutations, and qua-
dratic assignment problem,” in Proc. 9th Annu.Conf. Genetic and Evolutionary Computation,
New York, NY, 2007, pp. 629–629.
[35] J. Zhuang, A. Jones, S.-H. Lee, E. Ng, H. Fiegl, M. Zikan, D. Cibula, A. Sar-
gent, H. B. Salvesen, I. J. Jacobs, H. C. Kitchener, A. E. Teschendorff, and M. Wid-
schwendter, “The dynamics and prognostic potential of DNA methylation changes at
stem cell gene loci in women’s cancer,” PLoS Genetics, vol. 8, no. 2, p. e1002517, 2012.
[36] F. Simmer, A. Brinkman, Y. Assenov, F. Matarese, A. Kaan, L. Sabatino, A. Vil-
lanueva, D. Huertas, M. Esteller, T. Lengauer, C. Bock, V. Colantuoni, L. Altucci, and
H. Stunnenberg, “Comparative genome-wide DNA methylation analysis of colorectal
tumor and matched normal tissues,” Epigenetics, vol. 7, no. 12, pp. 1355–1367, 2012.
[37] A. kumar Singh, R. Singh, F. Naz, S. S. Chauhan, A. Dinda, A. A. Shukla, K. Gill,
V. Kapoor, and S. Dey, “Structure based design and synthesis of peptide inhibitor of hu-
man lox-12: In vitro and in vivo analysis of a novel therapeutic agent for breast cancer,”
PLoS One, vol. 7, no. 2, p. e32521, 2012.
[38] A. Singh, S. Kant, R. Parshad, N. Banerjee, and S. Dey, “Evaluation of human
LOX-12 as a serum marker for breast cancer,” Biochem. Biophys. Res. Commun., vol. 414,
no. 2, pp. 304–308, 2011.
[39] A. C. Tan, A. Jimeno, S. H. Lin, J. Wheelhouse, F. Chan, A. Solomon, N. Rajesh-
kumar, B. Rubio-Viqueira, and M. Hidalgo, “Characterizing DNA methylation patterns
in pancreatic cancer genome,” Mol. Oncol., vol. 3, no. 5, pp. 425–438, 2009.
[40] S. Alvarez, J. Suela, A. Valencia, A. Fernández, M. Wunderlich, X. Agirre, F.
Prósper, J. I. Martín-Subero, A. Maiques, F. Acquadro, S. Rodriguez Perales, M. J. Ca-
lasanz, J. Roman-Gómez, R. Siebert, J. C. Mulloy, J. Cervera, M. A. Sanz, M. Esteller,
and J. C. Cigudosa, “DNA methylation profiles and their relationship with cytogenetic
status in adult acute myeloid leukemia,” PLoS One, vol. 5, no. 8, p. e12197, 2010.
[41] O. Ammerpohl, J. Pratschke, C. Schafmayer, A. Haake, W. Faber, O. von Kam-
pen, M. Brosch, B. Sipos, W. von Schönfels, K. Balschun, C. Röcken, A. Arlt, B.
Schniewind, J. Grauholm, H. Kalthoff, P. Neuhaus, F. Stickel, S. Schreiber, T. Becker,
R. Siebert, and J. Hampe, “Distinct DNA methylation patterns in cirrhotic liver and
hepatocellular carcinoma,” Int. J. Cancer, vol. 130, no. 6, pp. 1319–1328, 2012.
[42] R. S. Ohgami, L. Ma, L. Ren, O. K. Weinberg, M. Seetharam, J. R. Gotlib, and
D. A. Arber, “DNA methylation analysis of ALOX12 and GSTM1 in acute myeloid leu-
kaemia identifies prognostically significant groups,” Br. J. Haematol., vol. 159, no. 2,
pp. 182–190, 2012.
[43] X.-Z. Ding, C. A. Kuszynski, T. H. El-Metwally, and T. E. Adrian, “Lipoxygenase inhibi-
tion induced apoptosis, morphological changes, and carbonic anhydrase expression in human
pancreatic cancer cells,” Biochem. Biophys. Res. Commun., vol. 266, no. 2, pp. 392–399, 1999.
[44] G. P. Pidgeon, M. Kandouz, A. Meram, and K. V. Honn, “Mechanisms controlling
cell cycle arrest and induction of apoptosis after 12-lipoxygenase inhibition in prostate
cancer cells,” Cancer Res., vol. 62, no. 9, pp. 2721–2727, 2002.
[45] G. P. Pidgeon, K. Tang, Y. L. Cai, E. Piasentin, and K. V. Honn, “Overexpression
of platelet-type 12-lipoxygenase promotes tumor cell survival by enhancing a v b 3 and
av b5 integrin expression,” Cancer Res., vol. 63, no. 14, pp. 4258–4267, 2003.
[46] A. Ring, I. E. Smith, and M. Dowsett, “Circulating tumour cells in breast cancer,”
Lancet Oncol., vol. 5, no. 2, pp. 79–88, 2004.
[47] M. Lacroix, “Significance, detection and markers of disseminated breast cancer
cells,” Endocrine-Relat. Cancer, vol. 13, no. 4, pp. 1033–1067, 2006.
[48] M. Morris, D. Gentle, M. Abdulrahman, N. Clarke, M. Brown, T. Kishida, M.
Yao, B. Teh, F. Latif, and E. R. Maher, “Functional epigenomics approach to identify
methylated candidate tumour suppressor genes in renal cell carcinoma,” Br. J. Cancer, vol.
98, no. 2, pp. 496–501, 2008.
[49] R. Ellsworth, C. Heckman, J. Seebach, L. Field, B. Love, J. Hooke, and C. Shriver,
“Identification of a gene expression breast cancer nodal metastasis profile,” J. Clin. Oncol.,
vol. 26, no. 15 Suppl, p. 1022, 2008.
[50] A. Prat, J. S. Parker, O. Karginova, C. Fan, C. Livasy, J. I. Herschkowitz, X. He,
and C. M. Perou, “Phenotypic and molecular characterization of the claudin-low intrin-
sic subtype of breast cancer,” Breast Cancer Res., vol. 12, no. 5, p. R68, 2010.
[51] B. C. Jackson, C. Carpenter, D. W. Nebert, and V. Vasiliou, “Update of human and
mouse forkhead box (FOX) gene families,” Hum. Genomics, vol. 4, pp. 345–352, 2010.
[52] B. Demircan, L. M. Dyer, M. Gerace, E. K. Lobenhofer, K. D. Robertson, and K.
D. Brown, “Comparative epigenomics of human and mouse mammary tumors,” Genes
Chromosomes Cancer, vol. 48, no. 1, pp. 83–97, 2009.
[53] D. Grimm, J. Bauer, J. Pietsch, M. Infanger, J. Eucker, C. Eilles, and J. Schoen-
berger, “Diagnostic and therapeutic use of membrane proteins in cancer cells,” Current
Med. Chem., vol. 18, no. 2, pp. 176–190, 2011.
[54] X. Li, L. Wu, C. A. S. Corsa, S. Kunkel, and Y. Dou, “Two mammalian MOF
complexes regulate transcription activation by distinct mechanisms,” Mol. Cell, vol. 36,
no. 2, pp. 290–301, 2009.
[55] S. Zhang, X. Liu, Y. Zhang, Y. Cheng, and Y. Li, “RNAi screening identifies
KAT8 as a key molecule important for cancer cell survival,” Int. J. Clin. Exp. Pathol., vol.
6, no. 5, pp. 870–877, 2013.
[56] A. Krebs, M. Frontini, and L. Tora, “GPAT: Retrieval of genomic annotation from
large genomic position datasets,” BMC Bioinform., vol. 9, no. 1, p. 533, 2008.
[57] A. Subramanian, P. Tamayo, V. K. Mootha, S. Mukherjee, B. L. Ebert, M. A. Gil-
lette, A. Paulovich, S. L. Pomeroy, T. R. Golub, E. S. Lander, and J. P. Mesirov, “Gene
set enrichment analysis: A knowledge-based approach for interpreting genome-wide ex-
pression profiles,” Proc. Natl. Acad. Sci. USA, vol. 102, no. 43, pp. 15 545–15 550, 2005.
[58] A. Liberzon, A. Subramanian, R. Pinchback, H. Thorvaldsdóttir, P. Tamayo, and
J. P. Mesirov, “Molecular signatures database (MSigDB) 3.0,” Bioinformatics, vol. 27, no.
12, pp. 1739–1740, 2011.
[59] E. Å. Jansson, A. Are, G. Greicius, I.-C. Kuo, D. Kelly, V. Arulampalam, and S.
Pettersson, “The WNT/b-catenin signaling pathway targets PPARY activity in colon
cancer cells,” Proc. Natl. Acad. Sci. USA, vol. 102, no. 5, pp. 1460–1465, 2005.
[60] S. Segditsas and I. Tomlinson, “Colorectal cancer and genetic alterations in the
WNT pathway,” Oncogene, vol. 25, no. 57, pp. 7531–7537, 2006.
[61] J. Y. Fang and B. C. Richardson, “The MAPK signalling pathways and colorectal
cancer,” Lancet Oncol., vol. 6, no. 5, pp. 322–327, 2005.
[62] M. L. Slattery, A. Lundgreen, and R. K. Wolff, “Map kinase genes and colon and
rectal cancer,” Carcinogenesis, vol. 33, no. 12, pp. 2398–2408, 2012.
[63] H. Akil, A. Perraud, C. Mélin, M.-O. Jauberteau, and M. Mathonnet, “Fine-tun-
ing roles of endogenous brain-derived neurotrophic factor, TrkB and sortilin in colorectal
cancer cell survival,” PLoS One, vol. 6, no. 9, p. e25097, 2011.
[64] T. Kitamura, T. Fujishita, P. Loetscher, L. Revesz, H. Hashida, S. Kizaka-Kondoh, M.
Aoki, and M. M. Taketo, “Inactivation of chemokine (C-C motif) receptor 1 (CCR1) sup-
presses colon cancer liver metastasis by blocking accumulation of immature myeloid cells in a
mouse model,” Proc. Natl. Acad. Sci. USA, vol. 107, no. 29, pp. 13 063–13 068, 2010.
[65] H. J. Chen, R. Edwards, S. Tucci, P. Bu, J. Milsom, S. Lee, W. Edelmann, Z. H.
Gümüs, X. Shen, and S. Lipkin, “Chemokine 25-induced signaling suppresses colon
cancer invasion and metastasis,” J. Clin. Invest., vol. 122, no. 9, pp. 3184–3196, 2012.
[66] D. W. Parsons, T.-L. Wang, Y. Samuels, A. Bardelli, J. M. Cummins, L. DeLong,
N. Silliman, J. Ptak, S. Szabo, J. K. Willson, S. Markowitz, K. W. Kinzler, B. Vogel-
stein, C. Lengauer, and V. E. Velculescu, “Colorectal cancer: Mutations in a signalling
pathway,” Nature, vol. 436, no. 7052, pp. 792–792, 2005.
[67] T. Yuan and L. Cantley, “PI3K pathway alterations in cancer: Variations on a
theme,” Oncogene, vol. 27, no. 41, pp. 5497–5510, 2008.
[68] L. M. Ellis and D. J. Hicklin, “VEGF-targeted therapy: Mechanisms of anti-tumour
activity,” Nature Rev. Cancer, vol. 8, no. 8, pp. 579–591, 2008.
[69] T. Winder and H.-J. Lenz, “Vascular endothelial growth factor and epidermal
growth factor signaling pathways as therapeutic targets for colorectal cancer,” Gastroenter-
ology, vol. 138, no. 6, pp. 2163–2176, 2010.
[70] R. Roskoski, Jr., “The ERBB/HER receptor protein-tyrosine kinases and cancer,”
Biochem. Biophys. Res. Commun., vol. 319, no. 1, pp. 1–11, 2004.
[71] J. Spano, R. Fagard, J.-C. Soria, O. Rixe, D. Khayat, and G. Milano, “Epidermal
growth factor receptor signaling in colorectal cancer: Preclinical data and therapeutic
perspectives,” Ann. Oncol., vol. 16, no. 2, pp. 189–194, 2005.
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 19
 Image lIcensed by Ingram PublIshIng
Augmentation of Physician
Assessments with Multi-Omics
Enhances Predictability
of Drug Response: A Case Study
of Major Depressive Disorder
Arjun Athreya and Ravishankar Iyer
Department of Electrical and Computer Engineering, Univ. of Illinois at Urbana-Champaign, IL, USA

Drew Neavin, Liewei Wang and Richard Weinshilboum

Department of Molecular Pharmacology and Experimental Therapeutics, Mayo Clinic, MN, USA

Rima Kaddurah-Daouk and John Rush

Department of Psychiatry and Behavioral Sciences, Duke University, NC, USA

Mark Frye
Department of Psychiatry and Psychology, Mayo Clinic, MN, USA

William Bobo
Department of Psychiatry and Psychology, Mayo Clinic, FL, USA

Digital Object Identifier 10.1109/MCI.2018.2840660

Date of publication: 18 July 2018 Corresponding Author: Ravishankar Iyer (rkiyer@illinois.edu)

20 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018 1556-603x/18©2018IEEE

Abstract—This work proposes a “learning-augmented clinical are treated according to subject-reported assessments of symp-
assessment” workflow to sequentially augment physician assess- tom severity.
ments of patients’ symptoms and their socio-demographic

measures with heterogeneous biological measures to accurately
predict treatment outcomes using machine learning. Across
many psychiatric illnesses, ranging from major depressive disor-
der to schizophrenia, symptom severity assessments are subjective
and do not include biological measures, making predictability in
eventual treatment outcomes a challenge. Using data from the
Mayo Clinic PGRN-AMPS SSRI trial as a case study, this
work demonstrates a significant improvement in the prediction
accuracy for antidepressant treatment outcomes in patients
with major depressive disorder from 35% to 80% individualized
by patient, compared to using only a physician’s assessment as
the predictors. This improvement is achieved through an itera-
tive overlay of biological measures, starting with metabolites
(blood measures modulated by drug action) associated with
symptom severity, and then adding in genes associated with
metabolomic concentrations. Hence, therapeutic efficacy for a
new patient can be assessed prior to treatment, using predic-
tion models that take as inputs selected biological measures
and physicians’ assessments of depression severity. Of broader
significance extending beyond psychiatry, the approach pre-
sented in this work can potentially be applied to predicting
treatment outcomes for other medical conditions, such as
migraine headaches or rheumatoid arthritis, for which patients
I
I. Introduction
n diseases that are characterized by the complex pheno-
types (traits) listed in Table 1, such as psychiatric disor-
ders, inflammatory diseases, and migraines, therapeutic/
treatment decisions are primarily based on the subject-
reported/physician-rated severity of symptoms (which are an
example of complex phenotypes/traits) in conjunction with
standard social/demographic factors. The ability of these mea-
sures to predict therapeutic success is slightly better than
chance [1], [2], and is largely limited by the lack of biological
measures that reflect the underlying molecular mechanisms of
therapeutic agents (e.g., drugs) and could therefore potentially
serve as stronger predictors of therapeutic outcomes. The key
contribution of this work in addressing that limitation is a
“learning-augmented clinical assessment” workflow to sequential-
ly augment physicians’ assessments of subject-specific ratings
of symptoms with heterogeneous biological measures (such as
metabolomics and genomics) to significantly enhance the pre-
dictability of drug treatment outcomes as shown in Fig. 1. As
a case study, the workflow proposed in this work demon-
strates improved predictability in antidepressant treatment
outcomes of patients with major depressive disorder (MDD)
by using biological measures (metabolomics and genomics)

TAblE 1 Efforts in integrating multiple measures to predict clinical outcomes in diseases characterized by complex phenotypes.

DiSEASE PRiOR wORk PREDiCTOR vARiAblES COMMEnT

CliniCAl
MEASuRES (nOn- nOn-DRug DRug-bASED
biOMARkERS) biOMARkERS biOMARkERS
MDD ChekrouD et Yes no no abIlItY to establIsh Cross-trIal
al. [1], InIesta preDICtIon of ClInICal outCoMes
et al. [2] usIng onlY ClInICal Measures
thIs work Yes targeteD targeteD augMentIng exIstIng ClInICal Mea-
Metabolo- MetaboloMICs sures wIth funCtIonallY valIDateD
MICs anD anD genoMICs bIoMarkers assoCIateD wIth DIsease
genoMICs pathophYsIologY or Drug MeCha-
nIsMs IMproves preDICtabIlItY In
treatMent outCoMes
reDlICh Yes MagnetIC no establIshIng the abIlItY of IMagIng
et al. [3] resonanCe Data to preDICt ClInICal outCoMes
IMagIng
(MrI)
sChIzophrenIa koutsoulerIs Yes no no abIlItY to establIsh Cross-trIal
et al. [4] preDICtIon of ClInICal outCoMes
usIng onlY ClInICal Measures
bIpolar DIsor- tIghe Yes funCtIonal no establIshIng the abIlItY of IMagIng
Der et al. [5] MrI Data anD transCrIptoMICs Data to
preDICt ClInICal outCoMes
rheuMatoID wIjbranDts Yes no targeteD abIlItY to preDICt treatMent outCoMes
arthrItIs et al. [6] transCrIp- usIng ClInICal Measures anD
toMICs transCrIptoMe varIatIons assoCIateD
wIth outCoMes, but bIoMarkers neeD
replICatIon anD funCtIonal valIDatIon

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 21

 Trial
Patients Treated with Antidepressants
Baseline 4 Weeks 8 Weeks
Antidepressant Treatment Outcome Prediction
e no
G
ics
m
lo
abo
et
M
Baseline
FiguRE 1 the proposed analyses to establish predictability of clinical outcomes at eight weeks.
derived from peripheral blood to augment the severity mea-
sures and other sociodemographic factors currently used in
clinical practice as predictor variables. This improvement in
predictive accuracy of treatment outcomes motivates the need
for developing antidepressant-specific prediction models, so
that choice of antidepressant can be based on highest likeli-
hood of remission of depressive symptoms. Choosing antide-
pressants that maximize therapeutic success marks a major
shift from the current “try and wait” approach, which often
requires multiple trials of antidepressants before patients
achieve remission of their depressive symptoms.
To demonstrate the improved predictability in treatment
outcomes, the workflow was developed using data from the
22 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
Patient Stratification
Increasing Symptom Severity
Baseline 4 Weeks 8 Weeks
–log10 (P )
ics
m
Chromosome
?
?
4 Weeks 8 Weeks
clinical trial of the Mayo Clinic Pharmacogenomics Research
Network Antidepressant Medical Pharmacogenomic Study
(Mayo PGRN-AMPS) [7], which is the largest single-center
selective serotonin reuptake inhibitor (SSRI) trial that has been
conducted in the United States. 603 patients completed the
trial. They were administered citalopram/escitalopram (com-
monly prescribed SSRIs) for eight weeks, and psychiatric
assessments of depression severity at baseline (pre-treatment),
four weeks, and eight weeks were conducted by a clinician
using the Quick Inventory of Depressive Symptomatology
(QIDS-C). In this trial, biological measures for 290 of the 603
patients included genome-wide association study (GWAS) gen-
otype data that, after imputation, included approximately 7 million
single-nucleotide polymorphisms (SNPs) ^G h, and plasma
TAblE 2 Data (D = [S: C: B]).
metabolomic concentrations (B in Table 2) for 31 metabolites
^ M h from patients at three time-points of the trial (at baseline, total patIents: 603.

four weeks, and eight weeks). Through augmentation of those
biological measures with psychiatric assessments and sociode-
mographic factors as predictor variables, the prediction accura-
cy of antidepressant treatment outcomes in MDD patients
improved from 35% to 80% relative to the use of clinical mea-
sures alone as the predictor variables.
The formalism for integrating multiple biological measures
in this case study is as follows and is illustrated in Fig. 2. Just as
tumor subtypes serve as a foundation for integrating biological
measures in oncology, our formalism first established patient
subtypes/stratification C by using mixture-model-based unsu-
Men: total: 222. wIth oMICs: 99.
woMen: total: 381. wIth oMICs: 191.
SOCiAl AnD DEMOgRAPhiC DATA (S) COllECTED Only AT
bASElinE:
age (In Years)
boDY Mass InDex (bMI In kg/m2)
DepressIon In {parents, sIblIngs, ChIlDren}
bIpolar DIsorDer In {parents, sIblIngs, ChIlDren}
alCohol abuse bY {parents, sIblIngs, ChIlDren}
Drug abuse bY {parents, sIblIngs, ChIlDren}

pervised learning techniques. In the first layer of overlaying of seasonal pattern In sYMptoM oCCurrenCe
the biological measures, a set of metabolites m ! M were hIstorY of psYChotherapY
identified based on significant associations of their concentra- DEPRESSivE SEvERiTy ASSESSMEnT (C ):
tions with symptom severity in previously inferred patient ClInICIan-rateD QuICk InventorY of DepressIve sYMptoMatologY
stratification. In the second layer of the overlay of biological (QIDs-C) QuestIonnaIre (16 QuestIons)
measures, in what is referred to as a metabolomics-informed-genom-
QIDs-C total sCore
ics approach, we used GWAS to identify SNPs g ! G that are
biOlOgiCAl DATA (B):
associated with concentrations of metabolites comprising m.
Through iterative overlaying of biological measures starting M: 31 MetabolItes froM the hplC lCeCa platforM
with metabolites (blood measures reflecting drug action) asso- G : 7 MIllIon sIngle nuCleotIDe polYMorphIsM genotYpes
ciated with depressive severity, and then adding in the genes
associated with metabolomic concentrations, the biological
measures became more closely associated with the molecular
mechanisms of antidepressant response. Finally, out of the more Formulation of Multi-Omic Integration
than 7 million possible predictor variables, the proposed
approach identified about 65 predictor variables that comprised
(1) SNPs ( g) identified by the GWAS based on metabolomic
concentrations, (2) metabolites (m) whose concentrations are
significantly associated with depression severity in patient clus- Patient Stratification
ters, and (3) clinical measures (as shown in Table 2). Thus we = arg max (x, µk, σk2)
k (x ) where k (x ) =
made the size of the predictor data computationally tractable to k ∈[1:K ]

predict clinical outcomes yu by using supervised learning meth-

ods F (m, g, S, C, y), where y is the treatment outcome labels
of the training data. Metabolomic Association with Symptom
The overarching significance of this work is suggested by Severity in Patient Clusters
the success of analogous “precision medicine” approaches in | p.value [m α x ( )] ≤ 0.05/| |}
m = {m ∈
breast cancer therapeutics. Today, treatment strategies for each
breast cancer patient are tailored to the tumor’s specific molec-
ular characteristics. That successful approach is facilitated by the
close association of the phenotype (which is the molecular SNP Associations with
Metabolomic Concentrations
characteristics of the tumor, such as whether it is estrogen-
receptor-positive (ER-positive), human-epidermal-growth-fac- g = {g ∈ | p.value [GWAS(m)] ≤ 1E – 06}
tor-receptor-2-amplified (HER2 amplified), and/or triple-negative)
with a set of biomarkers, such as hormone receptors (e.g., ER),
genes, and their SNPs, which, when taken together, can be
Predicting Antidepressant Treatment Outcomes
prognostic of treatment outcomes [8]. However, in the study of
treatment outcomes in patients with MDD (as for other diseas- ~ S: Social and Demographics Data
y= (m, g, x, S, C, y) C: Clinical Data
es with complex phenotypes), some interesting key observa-
y : Training Label
tions can be made. First, GWAS have often failed to associate
SNPs with complex and non-binary phenotypes defined as, for FiguRE 2 the proposed approach to integrating multiple omics
example, “Did patients achieve a 50% reduction in baseline (metabolomics and genomics) measures.

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 23

patient-reported/clinician-recorded symptoms?” As a conse-
quence, it is acknowledged that methods of integrating widely
heterogeneous biological measures without a priori biological
knowledge become computationally intractable as the number
of study variables increases to the order of millions [9], [10].
Second, the predictability of antidepressant treatment out-
comes when clinical measures alone are used is at best slightly
better than chance [1], [2], [11]–[15]. Third, antidepressant
medications such as selective serotonin reuptake inhibitors
(SSRIs) are the standard of care for drug therapy in adults
with MDD, but less than half of patients have favorable out-
comes from this treatment [11]. In light of these observations,
if learning techniques could more accurately predict treatment
outcomes in patients with MDD by integrating a few biologi-
cal measures prognostic of antidepressant treatment’s success
with routine clinical measures, the impact would be far-reach-
ing, because MDD affects over 350 million patients worldwide
and is expected to be the leading cause of disabilities globally
by 2030 [16]–[18].
The approach proposed in this work will also have biological
significance and clinical utility through further methodological
innovations. For example, clusters inferred in this work can
serve as the basis for using probabilistic graphical models to
study the longitudinal behavior of depressive symptoms during
antidepressant treatment; we have previously referred to such
behavior as symptom dynamics [19]. From a clinical perspective,
this method can be applied to other medical conditions as
shown in Table 1, such as rheumatoid arthritis and migraine
headaches, for which patients are subtyped by the degree of
swelling of their joints (which does not directly reflect a specific
mechanistic biomarker), and by the pain ratings reported by
patients using a scale similar to the QIDS-C scale that is used in
this work to rate depressive symptoms [20]–[22]. Furthermore,
the approach proposed in this work could also be used to aug-
ment biological measures with clinical measures in other psy-
chiatric conditions, such as bipolar disorder and schizophrenia.
II. Related Work
Multi-omic integration has been broadly pursued in the con-
text of precision medicine for which patients are subtyped by
the underlying molecular characteristics of their disease.
A. Subtyping in Precision Medicine
Complexity in diseases is due to the manifestation of factors
from both within and outside the genome. Alterations in many
different molecular pathways may lead to similar disease pheno-
types [23]. Therefore, where possible, disease subtypes can be
defined by identifying different biological mechanisms that result
in the same phenotypes. Such subtypes may present unique
predispositions to benefit from different therapeutic options
[24]. Hence, for the success of precision medicine initiatives for
any disease, it is necessary to map therapeutic options to subtypes
(phenotypes) by using molecular mechanisms of the therapeutic
agent [24]. Until two decades ago, all breast cancers were treated
in the same way, and the survival rate was much lower than it is
24 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
today. However, with growing access to omics data from trials
across the globe, it is now possible for breast oncologists to iden-
tify different kinds of tumor based on molecular characteristics,
e.g., estrogen-receptor-positive (ER-positive), human-epidermal-
growth-factor-receptor-2-amplified (HER2 amplified), or tri-
ple-negative. Today, therapeutic options for breast cancer are
tailored to those observed molecular characteristics of the
tumor. As a result, the survival rate among patients with breast
cancer has increased significantly [8]. Similar approaches have
been taken for other aspects of oncology and also for autism, and
are currently being explored for neurodegenerative diseases such
as Alzheimer’s disease [25], [26].
B. Multi-omics Integration
Integration of multi-modal biological data (multi-omics data)
has been proposed in the context of breast cancer, diabetes, glio-
blastoma, and other diseases [9], [27]–[31]. For example, imaging
data have been combined with gene expression and patient
demographics data to better predict the prevalence of cancer
[31]. Such work has looked at identifying unique biological sig-
natures or biomarkers from each of the types of static data, and
then building a function that linearly or nonlinearly combines
the data into a rule-based decision system [9]. Previous efforts
have either used a sequential approach to integrate omics mea-
sures (assuming causal relationships) [32], [33] or used simul-
taneous approaches when studying gene-gene interactions [34].
Another class of methods for integrating data have used net-
work-based approaches that use correlation analyses as their
foundation. Finally, based on knowledge of prior distributions in
omics measures, Bayesian and knowledge-boosting approaches
in both network and network-free approaches have been useful
for incrementally growing the interactions between multi-omics
measures [35]. For all these aforementioned approaches to be
insightful and capable of handling the high-dimensional nature
of multi-omics measures, a necessary condition is the availabil-
ity of a priori biological knowledge of disease/patient sub-
types or the phenotype [9], [10]. In the absence of this a priori
biological knowledge, handling large volumes of data with mil-
lions of features (e.g., 7 million SNPs across the whole genome
in each patient) is computationally intractable [9], [10].
In addressing the limitations of the aforementioned work,
this work makes significant contributions towards the integra-
tion of multi-omics measures for diseases with complex phe-
notypes, using MDD as a case study.
1) Although this work has not subtyped MDD patients by
their molecular characteristics, replication of patient strat-
ification for patients across independent trials with statis-
tically identical symptom severity profiles is an important
step forward in the field of psychiatry. The ability to repli-
cate stratification is important given the widely acknowl-
edged heterogeneity in depressive symptom profiles and
drug response phenotypes [16]–[18].
2) We assert that the stratification established in this work
serves to overlay multiple biological measures that could
potentially improve predictability of treatment outcomes.
 3) We demonstrate that this approach to integrating multi-
omics measures provides mechanistic insights into drug
response that can be experimentally established in clini-
cal laboratories.
III. Data
The Mayo PGRN-AMPS trial (NCT 00613470) was
designed to assess the clinical outcomes of adults (aged 18–84
years) with non-psychotic MDD after four and eight weeks of
open-label treatment with citalopram or escitalopram and to
examine the metabolomic and genomic factors associated
with those outcomes [7]. Subjects were recruited from prima-
ry and specialty care settings in and near Rochester, MN from
March 2005 to May 2013. All psychiatric diagnoses were con-
firmed at the screening visit using modules of the Structured
Clinical Interview for DSM-IV (SCID) administered by
trained clinical research staff. The data D = [S :C : B] analyzed
in this work comprise social and demographic variables (S ),
clinical measures (C ), and biological measures (B ) and are list-
ed in Table 2. The social and demographic data (S ) were
assessed only at baseline. The treatment outcomes were estab-
lished using the 16-item, clinician-rated version of the Quick
Inventory of Depressive Symptomatology (QIDS-C) at base-
line, four weeks, and eight weeks; the results comprise the
clinical data C, which include the responses to the 16 QIDS-
C questions and the total QIDS-C score of the symptom
severity [36]. Biological measures for 290 of the 603 patients
in this trial included GWAS genotype data that, after imputa-
tion, included approximately 7 million SNPs ^G h, and plasma
metabolomic concentrations (B in Table 2) for 31 metabolites
^ M h taken from patients at three time-points of the trial (at
baseline, four weeks, and eight weeks). Samples were assayed
on a high-performance liquid chromatography (HPLC) elec-
trochemical coulometric array (LCECA) platform to obtain
standardized measures of the concentrations of metabolites.
Clinical Definitions
Response is defined as a 50% reduction in baseline symptoms as
measured at four weeks or eight weeks. If the total QIDS-C
score measured at eight weeks is less than or equal to 5, then
the patient is said to have achieved remission.
IV. Patient Stratification and Omics Association
A. Patient Stratification
Precision medicine in practice optimizes therapeutic options
based on patient “subtypes” based on specific biological/clinical
characteristics. Currently, there are no established mechanisms
by which patients with MDD are subtyped/stratified at baseline
(prior to treatment) or during the treatment, although at the
end-point of the trial, patients are triaged into remitters,
responders without remission, and non-responders. In the
absence of predefined stratification of patients, as previously
demonstrated in detail in our earlier work [19], we use unsuper-
vised learning to algorithmically infer patient clusters. Based on
existing knowledge of gender differences in response to antide-
pressants [37], the patients were stratified separately by gender.
observation
The p-value from the Shapiro-Wilk test of the total QIDS-C
score from all three time-points of the trial, and in both men
and women, was less than the significance level (a = 0.05).
This meant that the symptom severity scores were not normal-
ly distributed, as we rejected the null hypothesis of the Shapiro-
Wilk test (i.e., that the data are normally distributed).
approach
The fact that levels of symptom severity are not normally dis-
tributed meant that the k-means clustering algorithm would
not be suitable as a clustering algorithm here. Without a loss in
generality, under the assumption that the data (x; the total
QIDS-C score, which is the sum of a group of individual
depression items in the QIDS-C scale) were distributed as a
mixture of Gaussians (modeled using a Gaussian mixture model
(GMM)), we developed the patient stratification workflow
(Algorithm 1). Starting with an assumption that the data have at
least two components in the GMM, we used the expectation
maximization (EM) algorithm to estimate the sufficient statistics
parameters of the Gaussian components (mean n and variance
2
v ) of the GMM, as shown in Fig. 3(a). 10,000 samples were
randomly drawn from the inferred distributions (generateSam-
ples). Next, the Kolmogorov-Smirnov test (ks.test) was used to
test whether the distribution of the generated data was statisti-
cally similar to that of the original data. If the p-value (p) was
less than the significance level (a = 0.05), then we rejected the
null hypothesis that the two distributions were not similar. If
that happened, the number of components was increased by
one, and tested for similarity in the two distributions. Once we
obtained the minimum number of components K in the
GMM for which the generated and input data’s distributions
were similar, K clusters C = {C k; 6k ! 1: K } ordered by the
Algorithm 1 Patient Stratification.
input: x ! total QIDs-C scores
1: k ! 2
2: C ! Q
3: a ! 0.05
4: p ! 0
5: while p # a do
6: { n, v 2} ! EM (x, k)
7: x l ! generatesamples ^ n, v 2h
8: p ! ks.test (x, xl )
9: if p 2 significancelevel then
10: C ! gmmCluster ^ n, v 2h
11: end if
12: k ! k + 1
13: end while
Output: C
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 25
 0.12
0.09
Probability Density
0.06
0.03
0.00
10 15 20
Baseline Total QIDS−C Score
(a) Estimating Distributions
FiguRE 3 estimating parameters of gaussian mixture model for identifying clusters in the data. (a) the inference of mixtures comprising the
distribution of symptom severity scores. (b) Distribution of symptom severity within the clusters inferred using the sufficient statistics of
components inferred in (a).
increasing mean ( n k) of the components were the outputs of
the workflow [38]. Patients were assigned to the component C
that maximized the likelihood L (x) given the component’s suf-
ficient statistics (gmmCluster), as illustrated in Fig. 3(b) and
described as
C = argmax L k (x) where L k (x) = N ^x, n k, v 2k h .
k ! [1: K ]
Results
At each time-point t ! {b (baseline), f (four weeks), e (eight
weeks)}, we found three clusters of men and women by using
the proposed stratification process. Clusters at the baseline are
C b = " C 1b , C 2b , C 3b ,, at four weeks are C f = " C 1f , C 2f , C 3f ,, and
at eight weeks are C e = " C 1e , C 2e , C 3e , .
The clinical value of the clustering behavior is that C 1e in
both men and women captures all patients who achieved
remission at the end of eight weeks. Furthermore, the C 2e in
both men and women comprised patients who demonstrated
response but did not achieve remission. Finally, patients in C 3f
(both men and women) did not exhibit response or achieve
remission. The same workflow demonstrated identical patient
stratification in the Sequenced Treatment Alternatives to
Relieve Depression (STAR*D) trial [39], i.e., the Kol-
mogorov-Smirnov test for the symptom severity scores
between clusters of similar average symptom severity had a
p-value 2 0.8 [19]. From the analytics perspective, the signifi-
cance of the replication of patient stratification in two independent
clinical trials is that the clustering behavior followed the existing defi-
nitions of clinical outcomes in psychiatry.
B. Omics Associations
Baseline concentrations of key metabolites m = " m ! M p.value
[m a x (C )] # 0.05/ M ,, such as serotonin, kynurenine, tryp-
tophan, tyrosine, and paraxanthine, were significantly correlated
26 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
0.2
Probability Density
0.1
0.0
25 10 15 20 25
Baseline Total QIDS−C Score
(b) Clusters from Inferred GMM
with depression severity in all three clusters at eight weeks.
These correlations were biologically significant because
they have been associated with MDD treatment and
response, as these metabolites are related to the mono-
amine neurotransmitter pathways [40], [41]. Furthermore,
SNPs ^ g = " g ! G p.value [GWAS (m)] # 1E - 06 ,h in the
(1) TSPAN5 (rs10516436), AHR (rs17137566), ERICH3
(rs696692), and DEFB1 (rs5743467, rs2741130, rs2702877)
genes have been associated through use of GWAS with con-
centrations of kynurenine and serotonin [42], [43]. The associa-
tions of these biological measures laid the foundation for
assessing whether they could improve the predictability of clin-
ical outcomes when combined with traditional clinical, social,
and demographic variables.
V. Using Baseline Data to Predict Clinical Outcomes
A recent publication proposed a prediction model that uses
elastic-net regularization for feature selection and a gradient
boosting machine (GBM) for classification, but only using
baseline social, demographic, and clinical data 6S :C@ from the
STAR*D trial [1]. While their prediction accuracies were bet-
ter than chance, the authors acknowledged the limitations of
their work, which suggests that it might be worthwhile to
study whether the addition of baseline biological measures
together with the social, demographic, and clinical data would
increase the predictability of the clinical outcomes. With access
to metabolomics and genomics data in a smaller cohort of the
Mayo PGRN-AMPS trial, we set out to answer the following
questions (illustrated in Fig. 4):
1) Would augmenting social, demographic, and clinical
data 6S :C@ with metabolomics data improve the predic-
tion accuracies of treatment outcomes over using only
social, demographic, and clinical data 6S :C@ as predic-
tor variables?
 2) Would augmenting social, demographic, and clinical data
[ S: C ] with metabolomics and genomics data improve the
prediction accuracies of treatment outcomes over using
only social, demographic, and clinical data 6S :C@ as pre-
dictor variables?
3) If the predictions improved as a result of augmenting
existing clinical measures with biological measures, how
many of the top predictors were biological measures?
Feature Selection and Choice of Classifiers
Three classes of classifiers are used in this work, including ker-
nel, linear, and ensemble methods. For predicting outcomes
using baseline clinical, social, demographic, and metabolomic
data, we used support vector machines with linear kernels
(SVMLinear) and support vector machines that use radial-basis
kernels (SVM-RBF) as kernel methods [44]; a generalized lin-
ear model (GLM) as a linear method [45]; and GBMs as an
ensemble method [46]. As the creators of those methods have
indicated, each of those broader types has its own merits, math-
ematical nuances, and complexities, and all of them have been
used in other classification applications, such as in Kaggle [47].
To use all of the omics and clinical, social, and demographics
data to predict outcomes, we used nonparametric classifiers
such as SVM-RBF and random forests, as they are better suited
to handling correlated features [48], and have been used in pre-
dicting treatment outcomes in other psychiatric diseases such
as schizophrenia.
In addition to elastic-net regularization, recursive feature
elimination (i.e., a wrapper method) was also used for the
GLM and GBM classifiers; that made it possible to estimate the
model performance not only by optimizing the parameters of
the model, but also by searching for the right set of predictor
variables. Based on our datasets, the prediction performance did
not significantly vary with or without the use of any of the fea-
ture selection methods; the prediction accuracy remained
within 4%. This observation could also be in part due to a rea-
sonably small size of predictor variables.
To minimize the effects of overfit and information leak,
nested cross-validation (nested-CV) with 5 repeats was used to
train the classifiers. In each repeat, data were randomized, and
the nested-CV comprised an outer loop and an inner loop. The
outer loop had a fivefold cross-validation to split the data into
training data (80% of the data) and testing data (the remaining
20%). The inner loop used the training data to train the classifi-
er by using a tenfold cross-validation, and the trained classifier
was tested on the testing data. To minimize the effects of class
imbalance (i.e., unequal numbers of responders (60%) and non-
responders (40%)) in the training data, we used the synthetic
minority over-sampling (SMOTE) algorithm [49], which sim-
ulated patient profiles of the under-sampled class and up-sam-
pled the under-sampled class to ensure that the two classes had
equal sizes. Prediction performance was reported using several
metrics (AUC, sensitivity, and specificity), and the statistical sig-
nificance of the classifier’s accuracy was established using the

Clinical Assessments
Depression Severity,
Social and Demographic Data,
Patient History

Metebolomics Genomics

Predictive Models Predictive Models Predictive Models

Accuracy Improvement? Accuracy Improvement? Accuracy

FiguRE 4 the proposed analyses to establish improved predictability in antidepressant treatment outcomes by augmenting the clinicians’
assessments with biological measures.

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 27

null information rate (NIR, which is the prevalence of the class
with the largest samples) that served as a proxy for chance.
Training with and Without Biological Measures
In order to quantitatively assess the benefit of adding biologi-
cal measures to predict outcomes yu , we trained classifiers
F (m, g, S, C, y) using (1) baseline clinical data that included
only social and demographic data, X = [S :C]; (2) all base-
line data (including metabolomics and genomics data),
X = [S :C : B], where B = [m, g]; and (3) training labels y for
treatment outcomes. Metabolites (m) whose baseline concentra-
tions were correlated with the symptom severity at eight weeks,
and SNPs ( g) associated with their concentrations, were then
normalized along with clinical data in order to train the chosen
supervised learning methods. It is important to note that several
other researchers have proposed the combination of other
modalities of biological data [28]–[30], but it remains to be
explored whether combination becomes less effective when
patient-reported data are used, since there is considerable hetero-
geneity in subject-reported measures. Therefore, to the best of
our knowledge, this is the first time that quantified biological
measures comprising metabolomics and genomics measures have
been integrated for analyses with the clinical measures of psychi-
atric assessments that comprise demographic data and patient-
provided responses to symptom questionnaires (such as
QIDS-C). For all the classifiers, we compared the AUC, in addi-
tion to the generalized prediction accuracies, to see whether the
same model’s predictive ability improved with the addition of
metabolomics data. Further, if the predictability improved, we
extracted the top five predictors of the model that provided the
best balance of accuracy and AUC to see whether the top pre-
dictors were dominated by the metabolomics.
A. Results
As shown in Table 3, for both men and women and for both
the outcomes response and remission, there was a 30% improve-
ment in the overall accuracy and corresponding AUC. The
highlighted columns in Table 3 indicate the best-performing
models with the metabolomics data included; four out of the
top five predictors are metabolites, indicating that their addition
to the prediction model likely explains the increase in the pre-
dictability of the outcomes. As shown in Table 4, there was a
further improvement of at least 5% in the AUC and corre-
sponding accuracy when genomics data were integrated with
the metabolomics, clinical, social, and demographic data. We
have two observations about the inclusion of biological mea-
sures in all these predictions. First, the top predictors of out-
comes when biological measures were used were different in
men and women, likely pointing to different biological mecha-
nisms determining how men and women respond to the same
antidepressant. Second, except for the variable seasonal pattern
and the involvement item in the QIDS-C scale, no other clini-
cal/demographic measures were predictive of outcomes. Finally,
it is biologically significant that many of the top predictor
metabolites identified in this work are known to be correlated
28 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
with mood in the behavioral sciences, which has additional
promising implications, as discussed next.
VI. Broader Significance
The approach presented in this work provides opportunities for
further methodological extensions and innovations that will
allow for longitudinal analyses of therapeutic agents (such as
the analysis done for antidepressants in this work). Our work
also has broader clinical significance based on knowledge gained
through the analyses conducted in this work.
A. Methodological Extensions
Patient stratification across the time-points of a clinical trial
could serve as the intelligence needed to build a probabilistic
graph in which stratification could be used as the nodes of the
graph, so that one can study the longitudinal effects of thera-
peutic agents during treatment. By using the proportions of
patients who traverse between clusters of consecutive time-
points (transition probabilities), we can use optimizations such as
the Viterbi algorithm or forward algorithm to establish the
“most likely” paths that patients will traverse during the treat-
ment. Our previous work has demonstrated this capability by
using factor graphs to formalize the depressive symptom severi-
ty data during the antidepressant treatment’s timeline, and then
using the forward algorithm to establish the “most likely” paths
(referred to as symptom dynamics) that patients will traverse based
on changes in their depression severity during treatment [19].
B. In Psychiatry
Patient stratification conforming to known psychiatric definitions
of outcomes, and replicated in both the Mayo PGRN-AMPS
and STAR*D trials, provided the first show of confidence in our
ability to model symptom responses to citalopram/escitalopram
(antidepressant) treatment in depressed patients. Our previous
work demonstrated its biological significance through its
improvement of predictability by integrating metabolomics data
with clinical measures, because metabolites, such as serotonin and
kynurenine, were among the top predictors [19]. That develop-
ment was important because for decades, the treatment of MDD
has focused on biogenic amine neurotransmitter pathways, i.e.,
the synthesis and metabolism of catecholamines (such as norepi-
nephrine) and indoleamines (such as serotonin) [40], [41]. Fur-
thermore, the existing body of knowledge fits well with the
findings of our study; note that the metabolites listed in Table 3
include serotonin (5HT) itself as well as two metabolites from the
competing tryptophan metabolism pathway (KYN and 3OHKYN)
and the major catecholamine metabolite (MHPG), which are
known to play a role in behavior.
The addition of genomics data with metabolomics and
clinical measures as predictor variables has further improved the
predictability of antidepressant outcomes, as shown in Table 4.
This result raises the question of whether the genes associated
with serotonin and kynurenine are also extending their effects
in other metabolites used in this study through other mecha-
nisms. The improvement in predictability should motivate
 TAblE 3 Clinical outcome prediction performance with metabolomics data in the Mayo Clinic PGRN-AMPS trial. Expansions of the
abbreviations of the top predictors are as follows. ATOCO is (+)-alpha-tocopherol; URIC is uric acid; QIDSC-1 is sleep-onset
insomnia [36]; KYN is kynurenine; 3OHKY is 3-hydroxykynurenine; AMTRP is alpha-methyltryptophan; I3PA is indole-3-propionic
acid; GTOCO3 is (+)-gamma-tocopherol (redox state #3); 5HT is serotonin; MHPG is methoxy-hydroxyphenyl glycol; MET is
methionine; QIDS-13 is involvement [36]; HGA is homogentisic acid; PARAXAN is 1,7-dimethylxanthine.

MEn

RESPOnSE

DATA CliniCAl DATA Only CliniCAl AnD METAbOlOMiCS DATA TOP PREDiCTORS
MoDel svM-rbf svM-lInear glM gbM svM-rbf svM-lInear glM gbM atoCo
aCCuraCY 28.2 32 52 40 48 48 64 48 urIC
sensItIvItY 0 16.67 16.67 33.33 33.33 33.33 50 33.33 QIDs-1
speCIfICItY 53.5 46.15 84.62 46.15 61.54 61.54 61.54 61.54 kYn
auC 0.64 0.60 0.63 0.54 0.53 0.53 0.68 0.5 3ohkY
REMiSSiOn

DATA CliniCAl DATA Only CliniCAl AnD METAbOlOMiCS DATA TOP PREDiCTORS
MoDel svM-rbf svM-lInear glM gbM svM-rbf svM-lInear glM gbM aMtrp
aCCuraCY 28 44 44 48 64 68 64 45.65 I3pa
sensItIvItY 38.46 38 53.85 46.15 76.52 76 76.92 65.22 Drug Dosage
speCIfICItY 16.67 50 33.33 50 50 50 50 26.09 gtoCo3
auC 0.8 0.6 0.67 0.6 0.76 0.78 0.62 0.6 5ht

wOMEn

RESPOnSE

DATA CliniCAl DATA Only CliniCAl AnD METAbOlOMiCS DATA TOP PREDiCTORS
MoDel svM-rbf svM-lInear glM gbM svM-rbf svM-lInear glM gbM seasonal pattern
aCCuraCY 52.08 52.08 54.17 50 41.3 72.33 64.58 41.67 5ht
sensItIvItY 18.18 18.18 27.27 18.18 34.78 18.18 36.36 0 Mhpg
speCIfICItY 80.72 80.76 76.92 76.9 47.83 92.83 88.46 76 Met
auC 0.60 0.59 0.63 0.63 0.69 0.74 0.68 0.51 QIDs-13
REMiSSiOn

DATA CliniCAl DATA Only CliniCAl AnD METAbOlOMiCS DATA TOP PREDiCTORS
MoDel svM-rbf svM-lInear glM gbM svM-rbf svM-lInear glM gbM 5ht
aCCuraCY 34.78 50 45.65 36.96 41.3 54.33 52.17 45.65 hga
sensItIvItY 26.09 65.22 56.52 47.83 34.78 56.52 76.92 65.22 3ohkY
speCIfICItY 43.48 34.78 34.78 26.09 47.83 52.17 50 26.09 seasonal pattern
auC 0.64 0.52 0.58 0.58 0.56 0.53 0.53 0.53 paraxan

researchers and clinicians to collect more biological measures
for psychiatric diseases other than major depressive disorder
(e.g., bipolar disorder, schizophrenia, and various dementias)
that would not only help subtype or stratify patients by their
symptom severity profiles, but also combine biological charac-
teristics that would enable treatment strategies closer to the
kinds used in breast cancer therapeutics.
C. In Pharmacogenomics Research
Pharmacogenomics research focuses on understanding the
interplay between drug effects and functions of the genome. In
this context, we reflect on the improvements in breast cancer
therapeutics wherein treatment selection is based on molecular
characteristics of the tumor. In diseases such as major depressive
disorder, for which such biologically based subtyping is not
possible, the approach described in this work for stratifying
patients using symptomatic characteristics is of immense value
to associated pharmacogenomics research. In particular, trials
could be designed in which multi-omics (metabolomics,
genomics, etc.) and other biological measures (neuroimaging,
electrophysiology, etc.) could be collected that help to establish
biological associations with patients stratified using the pro-
posed approach. Then, longitudinal effects of the drug on these
biomarkers can be used to study why patients either respond
well to the intervention or do not. Furthermore, as already
demonstrated in this work, associations of biological markers
with inferred patient stratification can provide improved pre-
dictability for treatment outcomes.

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 29

D. In Other Clinical Applications accurately predict clinical outcomes for new patients prior to
From a biological mechanistic perspective, on one end of the initiating treatment. Use of such a small set of measures (as op-
clinical spectrum there are diseases such as breast cancer for posed to millions of variables, as with genome-wide genotype
which treatment is based on tumor subtypes. At the other end data) makes inference of novel biology, or accurate prediction
of the spectrum are neuropsychiatric diseases such as MDD for of clinical outcomes in diseases with complex phenotypes,
which subtyping of individual patients is possible based on computationally tractable. Of further significance is the ability
their reported symptom severity, as demonstrated in this work. to replicate patient stratification in independent clinical trial
Between these two extremes are migraine headaches and in- datasets while also having clinical validity in the context of
flammatory diseases, such as rheumatoid arthritis (RA), in which studying antidepressant response in patients with major de-
patients are subtyped by the degree of swelling of their joints pressive disorder. The stratification that serves as the basis
(which does not directly reflect a specific mechanistic biomark- for sequential overlaying of biological measures has not only im-
er) and by pain ratings reported by the patients using validated proved the predictability of antidepressant outcomes, but also
scales that are similar to the QIDS-C scale used to rate depres- shown that top predictors have mechanistic associations with an-
sive symptoms in this work [20]–[22]. Therefore, there is suffi- tidepressant response. These findings together motivate the
cient heterogeneity in RA symptoms to make treatment response use of this approach for other common diseases, such as rheu-
phenotypes so complex that the methodological innovation matoid arthritis or migraine headaches, for which a similar
presented in this work could be used to overlay biological mea- complexity in phenotype is seen, and will also motivate re-
sures that provide a significant mechanistic perspective. This searchers and clinicians to collect additional biological measures
approach could then be tested for the prediction of outcomes in for other psychiatric diseases for which the methods proposed
response to the drug therapy for migraine headaches or RA, as in this work could identify novel mechanisms of therapeutic effi-
additional examples of a possible broader application of the cacy. Furthermore, the workflow could be further enhanced by
approach described here. considering other omics data such as transcriptomics, and/or
proteomics, in addition to profiling of the microbiome. While
VII. Conclusions, Limitations, and Future Work the work has shown promise in our ability to combine multi-
Use of data-driven approaches such as the one proposed in this ple biological measures, our future work will focus on estab-
work provides a way to identify and combine a small set of tar- lishing replicability of the findings in external datasets of
geted biological measures to augment physician assessments to MDD patients treated with citalopram/escitalopram and other
antidepressants. Such replication in
findings would represent a strong
TAblE 4 Clinical outcome prediction performance when clinical measures were
foundation for investigating biological
combined with metabolomics and genomics data from the Mayo Clinic PGRN-AMPS factors associated with pathophysiolo-
trial. Expansion of abbreviations of the top predictors are as follows. 5HT is serotonin; gy of this disease with heterogeneous
MHPG is methoxy-hydroxyphenyl glycol; MET is methionine; QIDSC-13 is involvement
[36]; HGA is homogentisic acid; 3OHKY is 3-hydroxykynurenine; PARAXAN is
disease states, and additional mecha-
1,7-dimethylxanthine; ATOCO is (+)-alpha-tocopherol; URIC is uric acid; 4HPLA is nisms of drug action.
4-hydroxyphenyllactic acid; 4HBAC is 4-hydroxybenzoic acid; XAN is xanthine; 4HPAC is
4-hydroxyphenylacetic acid; CYS is cysteine; I3PA is indole-3-propionic acid; DEFB1_1 is
VIII. Acknowledgments
DEFB1 SNP rs5743467.
This material is based upon work par-
gEnDER MEn wOMEn tially supported by a Mayo Clinic and
OuTCOME RESPOnSE Illinois Alliance Fellowship for Tech-
MODEl SvM-RbF RAnDOM FOREST SvM-RbF RAnDOM FOREST nology-Based Healthcare Research; a
aCCuraCY (%) 76 80 75 71
CompGen Fellowship; an IBM Fac-
ulty Award; the National Science
sensItIvItY 0.66 0.83 0.5 0.6
Foundation (NSF) under grant CNS
speCIfICItY 0.84 0.77 0.9 0.88
13-37732; the National Institutes of
auC 0.82 0.88 0.88 0.73 Health (NIH) under grants U19
top preDICtors urIC, atoCo, 3ohkY, I3pa, Defb1_1 Mhpg, Met, 5ht, 4hpaC, xan GM61388, U54 GM114838, RO1
gEnDER MEn wOMEn GM28157, R24 GM078233, T32
OuTCOME REMiSSiOn GM072474 and RC2 GM092729;
MODEl SvM-RbF RAnDOM FOREST SvM-RbF RAnDOM FOREST and The Mayo Clinic Center for
aCCuraCY (%) 68.75 80 86.7 73.7
Individualized Medicine. Any opin-
ions, findings, and conclusions or rec-
sensItIvItY 0.63 0.9 0.73 0.78
ommendations expressed in this
speCIfICItY 0.74 0.7 0.8 0.68
material are those of the author(s) and
auC 0.74 0.87 0.9 0.88 do not necessarily reflect the views of
top preDICtors 4hpla, I3pa, 3ohkY, urIC, QIDsC-13 paraxan, 5ht, 4hbaC, CYs, 3ohkY the NSF and NIH. We thank IBM,

30 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

Intel, and Xilinx for hardware donations, which were used to
develop and test the methods. We thank Jenny Applequist for her
help in preparing the manuscript. Finally, we thank the MDD
patients who participated in the PGRN-AMPS and STAR*D
SSRI study and the psychiatrists who cared for them.
References
[1] A. M. Chekroud, R. J. Zotti, Z. Shehzad, R. Gueorguieva, M. K. Johnson, M. H.
Trivedi, T. D. Cannon, J. H. Krystal, and P. R. Corlett, “Cross-trial prediction of treat-
ment outcome in depression: A machine learning approach,” Lancet Psychiatry, vol. 3, no. 3,
pp. 243–250, Apr. 2016.
[2] R. Iniesta, K. Malki, W. Maier, M. Rietschel, O. Mors, J. Hauser, N. Henigsberg,
M. Z. Dernovsek, D. Souery, D. Stahl, et al., “Combining clinical variables to optimize
prediction of antidepressant treatment outcomes,” J. Psychiatric Res., vol. 78, pp. 94–102,
2016.
[3] R. Redlich, N. Opel, D. Grotegerd, K. Dohm, D. Zaremba, C. Burger, S. Munker,
L. Muhlmann, P. Wahl, W. Heindel, et al., “Prediction of individual response to electro-
convulsive therapy via machine learning on structural magnetic resonance imaging data,”
JAMA Psychiatry, vol. 73, no. 6, pp. 557–564, July 2016.
[4] N. Koutsouleris, R. S. Kahn, A. M. Chekroud, S. Leucht, P. Falkai, T. Wobrock, E.
M. Derks, W. W. Fleischhacker, and A. Hasan, “Multisite prediction of 4-week and 52-
week treatment outcomes in patients with first-episode psychosis: A machine learning
approach,” Lancet Psychiatry, vol. 3, no. 10, pp. 935–946, Oct. 2016.
[5] S. K. Tighe, P. B. Mahon, and J. B. Potash, “Predictors of lithium response in bipolar
disorder,” Therapeutic Advances Chronic Disease, vol. 2, no. 3, pp. 209–226, May 2011.
[6] C. Wijbrandts and P. Tak, “Prediction of response to targeted treatment in rheumatoid
arthritis,” Mayo Clin. Proc., vol. 92, no. 7, pp. 1129–1143, July 2017.
[7] D. A. Mrazek, J. M. Biernacka, D. J. O’kane, J. L. Black, J. M. Cunningham, M. S.
Drews, K. A. Snyder, S. R. Stevens, A. J. Rush, and R. M. Weinshilboum, “CYP2C19
variation and citalopram response,” Pharmacogenetics Genomics, vol. 21, no. 1, p. 1, Jan.
2011.
[8] A. A. Onitilo, J. M. Engel, R. T. Greenlee, and B. N. Mukesh, “Breast cancer subtypes
based on ER/PR and Her2 expression: Comparison of clinicopathologic features and
survival,” Clin. Med. Res., vol. 7, no. 1–2, pp. 4–13, June 2009.
[9] M. D. Ritchie, E. R. Holzinger, R. Li, S. A. Pendergrass, and D. Kim, “Methods of
integrating data to uncover genotype-phenotype interactions,” Nature Rev. Genetics, vol.
16, no. 2, pp. 85–97, Feb. 2015.
[10] M. Bersanelli, E. Mosca, D. Remondini, E. Giampieri, C. Sala, G. Castellani, and L.
Milanesi, “Methods for the integration of multi-omics data: Mathematical aspects,” BMC
Bioinformatics, vol. 17, no. Suppl 2, p. 15, Dec. 2016.
[11] M. H. Trivedi, M. Fava, S. R. Wisniewski, M. E. Thase, F. Quitkin, D. Warden,
L. Ritz, A. A. Nierenberg, B. D. Lebowitz, M. M. Biggs, et al., “Medication augmenta-
tion after the failure of SSRIs for depression,” New Eng. J. Med., vol. 354, no. 12, pp.
1243–1252, Mar. 2006.
[12] F. A. Jain, A. M. Hunter, J. O. Brooks, and A. F. Leuchter, “Predictive socioeconomic
and clinical profiles of antidepressant response and remission,” Depression Anxiety, vol. 30,
no. 7, pp. 624–630, July 2013.
[13] R. Hirschfeld, J. M. Russell, P. L. Delgado, J. Fawcett, R. A. Friedman, W. M.
Harrison, L. M. Koran, I. W. Miller, M. E. Thase, R. H. Howland, et al., “Predictors of
response to acute treatment of chronic and double depression with sertraline or imipra-
mine,” J. Clin. Psychiatry, vol. 59, pp. 669–675, July 1998.
[14] R. M. Bagby, A. G. Ryder, and C. Cristi, “Psychosocial and clinical predictors of
response to pharmacotherapy for depression,” J. Psychiatry Neurosci., vol. 27, no. 4, pp.
250, July 2002.
[15] A. C. Altamura, C. Montresor, D. Salvadori, and E. Mundo, “Does comorbid sub-
threshold anxiety affect clinical presentation and treatment response in depression? A
preliminary 12-month naturalistic study,” Int. J. Neuropsychopharmacol., vol. 7, no. 4, pp.
481–487, Dec. 2004.
[16] N. Olchanski, M. M. Myers, M. Halseth, P. L. Cyr, L. Bockstedt, T. F. Goss, and R.
H. Howland, “The economic burden of treatment-resistant depression,” Clin. Therapeu-
tics, vol. 35, no. 4, pp. 512–522, Apr. 2013.
[17] K. Martinowich, D. Jimenez, C. Zarate, and H. Manji, “Rapid antidepressant effects:
Moving right along,” Mol. Psychiatry, vol. 18, no. 8, pp. 856–863, Aug. 2013.
[18] R. C. Kessler, H. S. Akiskal, M. Ames, H. Birnbaum, P. Greenberg, R. M. A.
Hirschfeld, R. Jin, K. R. Merikangas, G. E. Simon, and P. S. Wang, “Prevalence and ef-
fects of mood disorders on work performance in a nationally representative sample of US
workers,” Amer. J. Psychiatry, vol. 163, no. 9, pp. 1561–1568, Sept. 2006.
[19] A. P. Athreya, S. S. Banerjee, D. Neavin, R. Kaddurah-Daouk, A. J. Rush, M. A.
Frye, L. Wang, R. M. Weinshilboum, W. V. Bobo, and R. K. Iyer, “Data-driven longitu-
dinal modeling and prediction of symptom dynamics in major depressive disorder: Inte-
grating factor graphs and learning methods,” in Proc. IEEE Conf. Computational Intelligence
Bioinformatics and Computational Biology, Aug. 2017, pp. 1–9.
[20] W. Bardwell, P. Nicassio, M. Weisman, R. Gevirtz, and D. Bazzo, “Rheumatoid
arthritis severity scale: A brief, physician-completed scale not confounded by patient
self-report of psychological functioning,” Rheumatology, vol. 41, no. 1, pp. 38–45, Jan.
2002.
[21] W. Kwong and D. Pathak, “Validation of the eleven-point pain scale in the mea-
surement of migraine headache pain,” Cephalalgia, vol. 27, no. 4, pp. 336–342, Apr.
2007.
[22] M. Kosinski, M. Bayliss, J. Bjorner, J. Ware, W. Garber, A. Batenhorst, R. Cady, C.
Dahlöf, A. Dowson, and S. Tepper, “A six-item short-form survey for measuring headache
impact: The hit-6,” Quality Life Res., vol. 12, no. 8, pp. 963–974, Dec. 2003.
[23] D. S. Charney and H. K. Manji, “Life stress, genes, and depression: Multiple pathways
lead to increased risk and new opportunities for intervention,” Sci. STKE, vol. 2004, no.
225, p. re5, Mar. 2004.
[24] R. Weinshilboum and L. Wang, “Pharmacogenomics: Bench to bedside,” Nature Rev.
Drug Discovery, vol. 3, no. 9, pp. 739–748, Sept. 2004.
[25] D. H. Geschwind and M. W. State, “Gene hunting in autism spectrum disorder: On
the path to precision medicine,” Lancet Neurol., vol. 14, no. 11, pp. 1109–1120, Nov. 2015.
[26] R. Higdon, R. K. Earl, L. Stanberry, C. M. Hudac, E. Montague, E. Stewart, I.
Janko, J. Choiniere, W. Broomall, N. Kolker, et al., “The promise of multi-omics and
clinical data integration to identify and target personalized healthcare approaches in au-
tism spectrum disorders,” Omics: J. Integrative Biol., vol. 19, no. 4, pp. 197–208, Apr. 2015.
[27] W. Zhang, F. Li, and L. Nie, “Integrating multiple omics analysis for microbial biolo-
gy: Application and methodologies,” Microbiology, vol. 156, no. 2, pp. 287–301, Feb. 2010.
[28] D. Lahat, T. Adali, and C. Jutten, “Multimodal data fusion: An overview of methods,
challenges, and prospects,” Proc. IEEE, vol. 103, no. 9, pp. 1449–1477, Sept. 2015.
[29] B. Ray, M. Henaff, S. Ma, E. Efstathiadis, E. R. Peskin, M. Picone, T. Poli, C. F.
Aliferis, and A. Statnikov, “Information content and analysis methods for multi-modal
high-throughput biomedical data,” Sci. Rep., vol. 4, p. 4411, Mar. 2014.
[30] P.-Y. Wu, C.-W. Cheng, C. D. Kaddi, J. Venugopalan, R. Hoffman, and M. D.
Wang, “–Omic and electronic health record big data analytics for precision medicine,”
IEEE Trans. Biomed. Eng., vol. 64, no. 2, pp. 263–273, Feb. 2017.
[31] J. Kong, L. A. Cooper, F. Wang, D. A. Gutman, J. Gao, C. Chisolm, A. Sharma, T.
Pan, E. G. Van Meir, T. M. Kurc, et al., “Integrative, multimodal analysis of glioblas-
toma using TCGA molecular data, pathology images, and clinical outcomes,” IEEE Trans.
Biomed. Eng., vol. 58, no. 12, pp. 3469–3474, Dec. 2011.
[32] B. P. Coe, E. A. Vucic, R. Chari, W. L. Lam, and W. W. Lockwood, “An integrative
multi-dimensional genetic and epigenetic strategy to identify aberrant genes and pathways
in cancer,” BMC Syst. Biol., vol. 4, no. 1, p. 67, Dec. 2010.
[33] M. R. Aure, I. Steinfeld, L. O. Baumbusch, K. Liestøl, D. Lipson, S. Nyberg, B.
Naume, K. K. Sahlberg, V. N. Kristensen, A.-L. Børresen-Dale, et al., “Identifying in-
trans process associated genes in breast cancer by integrated analysis of copy number and
expression data,” PLoS One, vol. 8, no. 1, p. e53014, Jan. 2013.
[34] B. Wang, A. M. Mezlini, F. Demir, M. Fiume, Z. Tu, M. Brudno, B. Haibe-Kains,
and A. Goldenberg, “Similarity network fusion for aggregating data types on a genomic
scale,” Nature Methods, vol. 11, no. 3, pp. 333–337, Mar. 2014.
[35] D. Kim, J.-G. Joung, K.-A. Sohn, H. Shin, Y. R. Park, M. D. Ritchie, and J. H.
Kim, “Knowledge boosting: A graph-based integration approach with multi-omics data
and genomic knowledge for cancer clinical outcome prediction,” J. Amer. Med. Informatics
Assoc., vol. 22, no. 1, pp. 109–120, July 2014.
[36] A. J. Rush, M. H. Trivedi, H. M. Ibrahim, T. J. Carmody, B. Arnow, D. N. Klein, J.
C. Markowitz, P. T. Ninan, S. Kornstein, R. Manber, et al., “The 16-item quick inven-
tory of depressive symptomatology (QIDS), clinician rating (QIDS-C), and self-report
(QIDS-SR): A psychometric evaluation in patients with chronic major depression,” Biol.
Psychiatry, vol. 54, no. 5, pp. 573–583, Sept. 2003.
[37] M. Piccinelli and G. Wilkinson, “Gender differences in depression critical review,”
Br. J. Psychiatry, vol. 177, no. 6, pp. 486–492, Dec. 2000.
[38] J. D. Banfield and A. E. Raftery, “Model-based Gaussian and non-Gaussian cluster-
ing,” Biometrics, pp. 803–821, Sept. 1993.
[39] M. H. Trivedi, A. J. Rush, S. R. Wisniewski, A. A. Nierenberg, D. Warden, L. Ritz,
G. Norquist, R. H. Howland, B. Lebowitz, P. J. McGrath, et al., “Evaluation of outcomes
with citalopram for depression using measurement-based care in STAR*D: Implications
for clinical practice,” Amer. J. Psychiatry, vol. 163, no. 1, pp. 28–40, Jan. 2006.
[40] J. J. Schildkraut, “Neuropsychopharmacology and the affective disorders,” New Eng.
J. Med., vol. 281, no. 6, pp. 302–308, Aug. 1969.
[41] J. Axelrod and R. Weinshilboum, “Catecholamines,” New Eng. J. Med., vol. 287, no.
5, pp. 237–242, Aug. 1972.
[42] M. Gupta, D. Neavin, D. Liu, J. Biernacka, D. Hall-Flavin, W. V. Bobo, M. A. Frye,
M. Skime, G. D. Jenkins, A. Batzler, et al., “TSPAN5, ERICH3 and selective serotonin
reuptake inhibitors in major depressive disorder: Pharmacometabolomics-informed phar-
macogenomics,” Mol. Psychiatry, Dec. 2016.
[43] D. Liu, B. Ray, D. R. Neavin, J. Zhang, A. P. Athreya, J. M. Biernacka, W. V. Bobo,
D. K. Hall-Flavin, M. K. Skime, H. Zhu, et al., “Beta-defensin 1, aryl hydrocarbon
receptor and plasma kynurenine in major depressive disorder: Metabolomics-informed
genomics,” Translational Psychiatry, vol. 8, no. 1, p. 10, Jan. 2018.
[44] B. Scholkopf and A. J. Smola, Learning with Kernels: Support Vector Machines, Regular-
ization, Optimization, and Beyond. Cambridge, MA: MIT Press, 2001.
[45] J. Friedman, T. Hastie, and R. Tibshirani, “Regularization paths for generalized
linear models via coordinate descent,” J. Stat. Softw., vol. 33, no. 1, p. 1, 2010.
[46] J. H. Friedman, “Stochastic gradient boosting,” Computational Statist. Data Anal., vol.
38, no. 4, pp. 367–378, Feb. 2002.
[47] J. Friedman, T. Hastie, and R. Tibshirani, The Elements of Statistical Learning. Springer
Series in Statistics, vol. 2. Berlin: Springer, 2009.
[48] Y. Tang, Y.-Q. Zhang, N. V. Chawla, and S. Krasser, “SVMs modeling for highly
imbalanced classification,” IEEE Trans. Syst., Man, Cybern., Part B, vol. 39, no. 1, pp.
281–288, Feb. 2009.
[49] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthet-
ic minority over-sampling technique,” J. Artif. Intell. Res., vol. 16, pp. 321–357, June 2002.
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 31
 Research
Society
Society
Frontier
Briefs
Optimal Weighted Extreme Learning Machine
for Imbalanced Learning with Differential Evolution
Abstract
I n this paper, we present a formal
model for the optimal weighted
extreme learning machine (ELM)
on imbalanced learning. Our model
regards the optimal weighted ELM as an
optimization problem to find the best
weight matrix. We propose an approxi-
mate search algorithm, named weighted
ELM with differential evolution (DE),
that is a competitive stochastic search
technique, to solve the optimization
problem of the proposed formal
imbalanced lear ning model.
We perfor m experiments on
standard imbalanced classifica-
tion datasets which consist of 39
binary datasets and 3 multiclass
datasets. The results show a
significant performance improve-
ment over standard ELM with
an average Gmean improvement
of 10.15% on binary datasets
and 1.48% on multiclass datasets,
which are also better than other
state-of-the-art methods. We
also demonstrate that our pro-
posed algorithm can achieve
Digital Object Identifier 10.1109/MCI.2018.2840707
Date of publication: 18 July 2018
32 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
JongHyok Ri
Institute of Cyber-Systems and Control, Zhejiang University, Zhejiang, China
Institute of Information Technology, Kim Il Song University, Pyongyang, DPR of Korea
Liang Liu
Institute of Cyber-Systems and Control, Zhejiang University, Zhejiang, China
Yong Liu
State Key Laboratory of Industrial Control Technology and Institute
of Cyber-Systems and Control, Zhejiang University, Zhejiang, China
Huifeng Wu
College of Computer Science and Technology, Zhejiang Dianzi University,
Zhejiang, China
Wenliang Huang
China Unicom Ltd. Beijing, China
Hun Kim
Department of Computer Science, Kim Il Song University, DPR of Korea
high accuracy in representation learn- neuron parameters are randomly assigned
ing by performing experiments on
MNIST, CIFAR-10, and YouTube-8M,
with feature representation from convo-
lutional neural networks.
which may be independent of training
data and the output weights can be analyt-
ically decided by the Moore-Penrose gen-
eralized inverse [5], [6]. Thus, it provides
simpler and faster implementation than
I. Introduction other machine learning techniques.
Extreme learning machine (ELM) [1]–[4], In recent years, the imbalanced learn-
is an effective and efficient machine ing problem [7]–[10] has drawn a signif-
learning technique that has attracted icant amount of interest from academia,
attention in various fields. The essential industry, and government funding agen-
advantage of ELM is that the hidden cies as data continues to accumulate. As
most of the standard learning
algorithms assume the distri-
butions among all classes are
equal, the equal misclassification
costs are acceptable on algo-
rithms learning balanced datas-
ets. However, when dealing with
a complex imbalanced dataset,
standard algorithms [2]–[4] will
fail to represent the distribu-
tion characteristics of the data
©istockphoto.com/valdum
and thus will lead to unfavor-
able accuracies across the data
classes [11]. Unfortunately, clas-
sic ELM does not solve the
problem of imbalanced data
Corresponding Author: Yong Liu (Email: yongliu@iipc.zju.edu.cn).
1556-603x/18©2018IEEE
distribution. When using classic ELM for
imbalanced data, the majority class tends
to push the separating boundary toward
the minority side to gain better classifica-
tion results for itself.Therefore data in the
minority class will be easily misclassified.
The most straightforward solution for
imbalanced data learning is to assign the
misclassification cost inversely to the class
distribution, which may be simply calcu-
lated as the number of samples in each
class. Thus Zong et al. [7] proposed
weighted ELM to overcome the disad-
vantages of the original ELM for an
imbalanced data problem. Their solution
is based on the work regarding weighted
regularized ELM presented by Toh [9]
and Deng et al. [10], and the key essence
of weighted ELM in Zong et al. [7] is to
assign an extra weight to each sample to
strengthen the impact of the minority
class while weakening the relative impact
of the majority class. Experimental results
in their work showed superior perfor-
mance of weighted ELM compared with
original ELM on various imbalanced
datasets. However, the two weighting
schemes used in their approach can only
obtain empirical sub-optima, and global
optima cannot be guaranteed. Following
work [8] introduced the boosting meth-
od to obtain better weighting schemes;
however, how to set the optimal weight-
ing scheme remains an open problem.
In this paper, we present a formal
model for optimal weighted ELM
applied to imbalanced learning. Our
model assumes that the optimal weight-
ed ELM can be presented as an optimi-
zation problem to search the optimal
weight matrix for the weighted ELM.
We also present an approximate solution
for the optimal weight matrix searching
problem based on differential evolution
(DE) [12], which is a competitive sto-
chastic search technique that performs
well in various standard test functions
and real-world optimization problems.
We also evaluated the effectiveness of
our learning machine algorithm by per-
forming experiments on various stan-
dard classification datasets, which consist
of 39 binary datasets and 3 multiclass
datasets: the MNIST [13], CIFAR-10
[14], and YouTube-8M [15] datasets.
The contributions of our approach
are as follows,
❏ We present a formal mathematical
model to obtain the optimal weighting
scheme. We introduce DE to calculate
the approximate optimal solution.
❏ Our approach can achieve significant
improvement in classification per-
for mance compared with other
state-of-the-art methods on various
imbalanced datasets.
❏ Our approach can narrow the search
range greatly compared with other
state-of-the-art methods, which indi-
cates our approach may be more effi-
cient for the practical imbalanced
learning problem.
The remaining sections are organized
as follows. Sections II and III introduce
the theoretical background of ELM and
related ELM methods on imbalanced
learning. Section IV presents the pro-
posed method. Section V reports the
experimental results and performance
analysis. Finally, the conclusions are sum-
marized in Section VI.
II. Theoretical Background
This section introduces a brief theoreti-
cal background of ELM.
ELM [1], [2] was originally proposed
for single-hidden layer feedforward
neural networks (SLFNs) and then ex-
tended to ‘generalized’ SLFNs where
the hidden layer does not require tun-
ing [3].
The main feature of ELM is the ran-
dom generation of hidden nodes which
may be independent of the training data
and the analytical calculation of output
weights by the Moore-Penrose general-
ized inverse [3]. The hidden layer
output (with l nodes) can be presented
by a row vector h (x) = [h 1 (x), ..., h l (x)]
where x is the input sample. Given n
training samples (x i, t i), the mathemati-
cal model of the SLFNs is
Hb = T. (1)
where H is the hidden layer output
matrix, b is the output weight and T is
the target vector.
The least squares solution with mini-
mal norm is analytically determined
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE
using the Moore-Penrose “generalized”
inverse [3], [16] as follows,
n 1 l : b = H @ T = H T ` I + HH T j T
-1
C
l 1 n : b = H @ T = ` I + H T H j H T T.
-1
C
(2)
ELM can also be explained from the
optimization view. ELM tries to mini-
2
mize both Hb - T 2 and b . There-
fore, a solution for formula (1) can be
obtained [2] from
n
Minimize : L ELM = 1 b + C 1 / p i 2
2
2 2 i=1
Subject to : h (x i) b = t Ti - p Ti , i = 1, ..., n.
(3)
where p i = [p i, 1, ..., p i, m] is the training
error vector of the m output nodes cor-
responding to training sample x i . C is
the trade-off regularization parameter
between the minimization of training
errors and the maximization of the mar-
ginal distance. Based on the Karush-
Kuhn-Tucker (KKT) theorem [16], we
can solve the optimization problem of
formula (3) and obtain the same solution
as formula (2).
Given a new sample x, the output
function of ELM is obtained as follows.
f ( x) =
h (x) H T ` I + HH T j T, when n 1 l
-1
* C
h (x) ` I + H T H j H T T, when n $ l.
-1
C
(4)
Here, f (x) = 6 f1 (x), ..., fm (x)@ is the out-
put function vector. Users may determine
the predication label of x as follows.
label (x) = arg max fi (x), i ! [1, ..., m].
i
(5)
Recently, further focus has been
placed on ELM algorithms and applica-
tions [17]–[19], and some advanced algo-
rithms [20]–[23] have been proposed to
improve performance. In [20], a Bayes-
ian-based ELM (BELM), which incor-
porates the advantages of both ELM and
Bayesian models, is proposed. It can
build the corresponding confidence
33

Gmean is a conventional evaluation metric in the case
of imbalanced learning; it is the geometric mean of the
recall values of all m classes.
interval without using additional meth-
ods such as bootstrap, and requires low
computational cost.
Bai et al. [21] proposed a sparse ELM
(S-ELM) by replacing the equality con-
straints in traditional ELM model with
inequality constraints, which can reduce
the storage space and testing time. Bai
et al. [22] also proposed a S-ELM for
regression analysis that can reduce the
storage space and testing time. In addi-
tion, they have developed an efficient
training algorithm based on iterative
computation, which scales quadratically
with respect to the number of training
samples. In [23], a random projection-
based ELM (RP-ELM), which is esti-
mated on the analysis of the random
projection feature mapping schema in
the ELM, is proposed. RP-ELM can sig-
nificantly reduce the number of neurons
in the hidden layer without affecting the
accuracy of the generalization perfor-
mance. As a result, the final learning
machine will benefit from a consider-
able simplification in the feature-map-
ping stage.
III. Related ELM Methods
on Imbalanced Learning
A. Weighted ELM [7]
The main goal of the ELM classifier is
to find a boundary to separate data from
two or multiple parts with maximal
margin distance between any two parts.
This separating boundary is supposed to
be pushed toward the side of the minor-
ity class for imbalanced data so that
the minority classes are easy misclassi-
fied. To resolve this issue, weighted
ELM (WELM) [7] has been recent-
ly proposed.
WELM improves the classification
performance for data with imbalanced
class distribution while maintaining the
advantages of the original ELM stated
above. Specifically, each training sample
is assigned with an extra weight. Mathe-
34 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
matically, an n # n diagonal matrix W
associated with every training sample x i
is defined. Usually, if x i comes from a
minority class, the associated weight w ii
is relatively larger than samples from a
majority class. Therefore, the impact of
the minority class is strengthened while
the relative impact of the majority class
is weakened. Considering the diagonal
weight matrix W, the optimization for-
mula of ELM can be revised [7] as
Minimize :
n
L WELM = 1 b + C 1 / (w ii
2 2
pi
2 2 i=1
Subject to :
h (x i) b = t Ti - p Ti , i = 1, ..., n.
According to the KKT theorem [16],
the solution to formula (6) is
b = H @T =
H T ` I + WHH T j WT, when n 1 l
-1
* I C perienced WELM.
` C + H T WH j H T WT, when n $ l.
-1
(7)
They also proposed two empirical weight-
ing schemes [7] as follows,
W 1: w ii = 1
#t i
Z 0.618
]] if #t i 2 AVG (#t i)
#t i
W 2: w ii = [ 1
] #t if #t i # AVG (#t i) .
i
\
(8)
where #t i is the number of samples
belonging to class t i, and AVG (#t i) re-
presents the average number of samples
for all classes.
Although WELM proposes two
weighting schemes in formula (8) for
imbalanced data, these matrices are not
optimal, which can be proven by the
following toy experiment. In the toy
experiment, we first randomly generate
an imbalanced dataset containing two
categories with a category ratio of 10:1,
and we also apply the standard ELM,
WELM with W 1, WELM with W 2 and
two other WELMs with randomly
selected weights. The original dataset and
the classification results are shown in
Fig. 1, where the ground truth labels of
every data instance are denoted by dif-
ferent symbols and the classification
results are denoted by different colors.
The results in Fig. 1 show that although
WELM with W 1 and W 2 policies can
achieve better results than the ELM
method in this dataset, their perfor-
mances are worse than the two WELMs
with randomly selected weights, which
indicates the policies with W 1 and W 2
in WELM are not optimal. In this paper,
these two WELM classifiers are referred
to as WELM-W1 and WELM-W2.
WELM [7] can improve the perfor-
)
mance of ELM greatly for imbalanced
data. The classification error of the class
(6) with fewer elements was reduced by set-
ting an unequal cost distribution for
each class with the weight matrix. How-
ever, the approach relied on empirical
weighting schemes, which were designed
according to the element number of
each class. Thus, it can also be called ex-
B. Boosting Weighted ELM [8]
To overcome this shortcoming of ex-
per ienced WELM, boosting-based
WELM (BWELM) [8] was proposed.
BWELM tries to determine the opti-
mal weight matrix using an AdaBoost
algorithm [24].
Inspired by the distribution weights
updating mechanism of AdaBoost, they
embedded WELM seamlessly into a
modified AdaBoost framework. Intui-
tively, the distribution weights in Ada-
Boost, which reflect the importance of
training samples, are input into WELM
as training sample weights. Further-
more, such training sample weights are
dynamically updated during iterations
of AdaBoost.
Considering the characteristics of
imbalanced learning, they modify the
original AdaBoost.M1 [24] in two
aspects. First, the initial distribution
weights are set to be asymmetric to make
AdaBoost converge at a faster speed.
 1.0 1.0 1.0
0.8 0.8 0.8
0.6 0.6 0.6
0.4 0.4 0.4
0.2 0.2 0.2
0.0 0.0 0.0
–0.2 –0.2 –0.2
–0.6 –0.6 –0.6
–0.4 –0.4 –0.4
–0.8 –0.8 –0.8
–1.0 –1.0 –1.0
–1.0 –0.5 0.0 0.5 1.0 –1.0 –0.5 0.0 0.5 1.0 –1.0 –0.5 0.0 0.5 1.0
(a) Original Data (b) ELM (Gmean: 95.3463) (c) WELM-W1 (Gmean: 98.5347)

1.0 1.0 1.0

0.8 0.8 0.8
0.6 0.6 0.6
0.4 0.4 0.4
0.2 0.2 0.2
0.0 0.0 0.0
–0.2 –0.2 –0.2
–0.6 –0.6 –0.6
–0.4 –0.4 –0.4
–0.8 –0.8 –0.8
–1.0 –1.0 –1.0
–1.0 –0.5 0.0 0.5 1.0 –1.0 –0.5 0.0 0.5 1.0 –1.0 –0.5 0.0 0.5 1.0
(d) WELM-W2 (Gmean: 98.9949) (e) Random Selected (f) Random Selected
Weight (1) (Gmean: 99.6357) Weight (2) (Gmean: 99.7269)

Figure 1 Classification results produced by ELM, WELM-W1, WELM-W2, and WELM with random selected weights on a randomly-generated
dataset with an imbalanced ratio of 1:10. Blue circles and red circles represent the correctly classified instances and incorrectly classified instanc-
es in the majority class, respectively. Red crosses and blue crosses represent the correctly classified instances and incorrectly classified instances
in the minority class, respectively.

This results in a boosting classifier with
a smaller number of WELM classifiers
and can save much computational time.
Second, the distribution weights are
updated separately for different classes
to avoid destroying the distribution
weights’ asymmetry.
In this paper, we also address the
problem of obtaining the best weighted
matrix for the ELM-based imbalanced
learning problem, and we propose a DE-
based WELM (DE-WELM) for imbal-
anced dataset learning.
DE was first presented by R. Storn
and Price [12], and it has been recog-
nized as a powerful method for solving
optimization problems. It resembles the
structure of evolutionary algorithms, but
differs from their traditional versions in
the generation of new candidate solu-
tions and the use of a greedy selection
scheme. As shown in previous research
[25], DE is far more efficient and robust
(with respect to reproducing the results
in several runs) compared to other evo-
lutionary computation algorithms such
as particle swarm optimization (PSO)
[26] and evolutionary programming (EP)
[27]. In addition, it has few parameters to
set, and the same settings can be adapted
to many different problems. Thus, this
method has been actively used in various
fields [28]–[30]. Some works [31]–[33]
have focused on updating ELM using
DE. However, they only use DE to search
the hidden neuron parameters instead of
searching the weighted matrix for imbal-
anced learning.
As parameters such as NP, F, and CR
in the DE algorithm are critical for its
perfor mance, there are also many
improved DE algorithms [34]–[40],
and [41] which can adaptively control
those parameters.
IV. Our Approach
Although the two weighting schemes in
formula (8) can obtain superior results
compared with the original ELM, they
are only empirical schemes and cannot
guarantee the optima, which was proven
in the toy experiment. In this section,
we present the formal mathematical
model to obtain the optimal weighting
scheme, and we also introduce the DE
method [12] to calculate the approxi-
mate optimal weight matrix used in
WELM, as the calculation for the formal
model is infeasible.
A. Mathematical Model for
Optimal Weighted Scheme
The problem of calculating the optimal
weight matrix in WELM can be formal-
ly defined as follows, Given training data
X = {(x i, t i), t i ! {1, 2, ..., m}, i = 1, 2, ...,
n}, and there are m classes in X, the

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 35


We compare the performance of our proposed
method with four state-of-the-art methods, i.e., the
Softmax classifier [43], the classic ELM classifier [1],
the experienced WELM classifier [7], and the
BWELM classifier [8].
optimal weight matrix can be obtained
by optimizing the following formula,
W) =
arg min (1 - Gmean (Hb (X,W),T ))
W tion size (NP), scaling factor (F ), and
Subject to:
b (X, W) =
H T ` I + WHH T j WT, when n 1 l
-1
* I C
` C + H T WH j H T WT, when n $ l.
-1
(9)
where H is the hidden layer output
matrix, T is the target vector, and l is
the number of hidden layer nodes in
WELM. As b is the output weight
computed in WELM using W as the
weight matrix, this formula can be con-
sidered as the function of W and X.
Gmean is a conventional evaluation
metric in the case of imbalanced learn-
ing; it is the geometric mean of the
recall values of all m classes and is
defined as follows,
1 which is used to avoid overfitting.
pjG
Gmean (Y,T ) = = %
m
qj m
(10)
j=1
where Y is the class prediction vector,
q j is the number of elements belonging
in class j correctly classified among Y,
and p j is the number of elements
belonging in class j among T. Although
we may use the Levenberg Marquardt
(LM) algorithm [42] to search for the
optimal solution to this problem, the
results are quite sensitive to the initial
values, so we introduce the DE algo-
rithm to calculate the approximate opti-
mal solution.
B. DE-based Approximate
Weight Calculation Algorithm for
Weighted ELM
DE is a method that optimizes a prob-
lem by iteratively trying to improve a
36 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
candidate solution with regard to a
given measure of quality; it has been
successfully applied in many domains
[28]–[30]. There are three major control
factors in the DE algorithm, i.e., popula-
crossover rate (CR). There are also three
operations involved, i.e., mutation oper-
ation, crossover operation, and selection
operation. We present our approximate
weight calculation algorithm based on
the three factors and three operations
in DE.
There are two stages in our algorithm:
the initial stage and the update stage.
1) Initial Stage
The initial stage generates an initial
population that contains NP candidates
in the weight matrix W, and searches for
the best weight from the current candi-
date set.
First, the values in training set X are
normalized to [-1,1], and we also select a
subset X vd 1 X as the validation set,
The initial population contains NP
candidate weight matrixes, NP – 2 can-
didates are generated randomly in [0,1],
and the remaining two candidates are set
as W 1 and W 2 , respectively, which are
presented in WELM [7]. Thus the initial
population W R consists of NP n-dimen-
sional vectors as follows,
W R = {W i, R W i, R = (w 1i, R, w i2, R, ..., w ni, R),
i = 1, 2, ..., NP - 2} , {W 1, W 2}.
(11)
Note that the weight matrix in WELM
is a diagonal matrix, so all elements are zero
except for the diagonal elements. There-
fore we will express a diagonal matrix
using a vector.
Second, we transfor m W i, R (i =
1, 2, ..., NP ) into a weight matr ix
diag (W i, R) 1 and compute the er ror
E Wi,R (i = 1, ..., NP) of the WELM clas-
sifier on validation set X vd, and the
candidate optimization weight vector
which produces the lowest error is
defined as W best, R .
E W i, R =
1 - Gmean (Hb (X vd, diag (W i, R)),T )
W best, R =
arg min E Wi,R (i = 1, 2, ..., NP ). (12)
W i, R
2) update Stage
The update stage will generate a new
candidate population based on the
results of the previous iteration and per-
forms the operations of mutation, cross-
over, and selection.
The input of the update stage is a
candidate population of the previous
stage and the candidate optimal weight
vector from the previous stage, and the
output of the update stage is a candidate
population of the next stage and a can-
didate optimal weight vector for the
next stage.
First, we transform candidate popula-
tion W R to mutant population VR with
the mutation operation. Each mutant
vector of VR is calculated according to
the following equation.
VR = {Vi, R Vi, R = W i, R + F (W best, R
- W i, R) + F (W ri1, R - W ri2, R),
i = 1, 2, ..., NP}. (13)
The indices r i1 and r i2 are mutually
exclusive integers randomly generated
within the range [1, NP], which are also
different from the index i. Indices are
randomly generated once for each mutant
vector. The scaling factor F is a positive
control parameter for scaling the differ-
ences among vectors.
After the mutation operation, we
apply the crossover operation to each
pair of target vector W i, R and its corre-
sponding mutant vector Vi, R to gener-
ate a trial vector U i, R = (u 1i, R, u 2i, R, ..., u ni, R)
as follows,
1
diag(.) is the function to map input vectors to a diago-
nal matrix as diagonal elements and all elements are
zero except for the diagonal elements.
u ij, R =
v ij, R if (rand j [0, 1) # CR) or ( j = j rand)
) vector will remain in the population for
w ij, R otherwise
( j = 1, 2, ..., n, i = 1, 2, ..., NP).
(14)
Here if the values of the newly gen-
erated trial vectors exceed the correspond-
ing upper and lower bounds, they will
be re-normalized into the range [0, 1].
CR is defined by users as a constant
within the range [0, 1], which controls
the fraction of parameter values copied
from the mutant vector. j rand is a ran-
domly chosen integer in the range [1, n].
This operation copies the jth parameter
of the mutant vector Vi, R to the corre-
sponding element in trial vector U i, R if
(rand j [0, 1) # CR) or ( j = j rand). Other-
wise, the parameter is copied from the
corresponding target vector W i, R . The
condition j = j rand is introduced to
ensure that the trial vector U i, R will be
different from its corresponding target
vector W i, R by at least one parameter.
We then use diag (U i, R) (i = 1, 2, ...,
NP ) as a weight matrix to compute the
error of the WELM classifier on valida-
tion set X vd . The element which has the
lowest error is defined to U best.R. We can
calculate the error of U i, R as follows.
E Ui,R = 1 - Gmean (Hb (X vd, diag (U i, R)), T )
(i = 1, ..., NP) . (15)
the target vector and enter the next can-
didate population. Otherwise, the target
the next generation.
The update stage will be executed
iteratively several times, and we obtain
the approximate optimal weight matrix
diag (W best,Rmax) once the algorithm com-
pletes, where R max is the number of it-
erations in the DE algorithm.
In short, our method obtains the ini-
tial population and the candidate weight
vectors in the initial stage and calculates
an approximate optimized matrix by
updating the population using the DE
algorithm in the update stage. The pseu-
do-code of our proposed algorithm is
given in Algorithm 1.
V. Performance Evaluation
In this section, we conduct comparison
experiments to evaluate the classification
capability of our DE-WELM on imbal-
anced learning problems and the analy-
ses on the results are also presented. We
compare the performance of our pro-
posed method with four state-of-the-art
methods, i.e., the Softmax classifier [43],
the classic ELM classifier [1], the experi-
enced WELM classifier [7], and the
BWELM classifier [8].
We select a subset of the KEEL
(Knowledge Extraction based on Evolu-
tionary Learning) imbalanced dataset
repository2 to evaluate the methods. We
also use the datasets of MNIST [13]
(handwritten digit image dataset), the
CIFAR-10 [14] (tiny image dataset), and
the YouTube-8M database in our experi-
ments. The results are averaged over ten
runs. Brief descriptions of the datasets are
provided in the following subsections.
In our experiments, we also intro-
duce the imbalanced ratio (IR) [7] to
quantitatively measure the imbalanced
degree of a dataset.
# (+ 1 )
Binary : IR =
# ( - 1)
min (# (t i))
Multiclass : IR = , i = 1, ..., m.
max (# (t i))
(17)
Above, # (+ 1) is the number of samples
in a minority class, # (- 1) is the num-
ber of samples in the majority class, and
# (t i) is the number of samples in class t i
for a multiclass dataset.
The attributes of all the datasets are
normalized to [–1,1].
In ELM theory, there are many
choices for feature mapping. In our
experiments, we use the sigmoid addi-
tive based feature mapping function
[4], which is a popular choice for
researchers. There are two parameters
2
http://sci2s.ugr.es/keel/study.php?cod=24

Then the next candidates popula-
tion W R + 1 = {W i, R + 1, i = 1, 2, ..., NP}
and its best candidate can be decid-
ed according to the following select
operation.
W i, R + 1 = '
U i, R if E Ui,R # E Wi,R
W i, R otherwise
W best, R if E Wbest,R 1 E Ubest,R
W best, R + 1 = '
U best, R otherwise.
(16)
We compare the E Ui,R of each trial
vector U i, R with the E Wi,R of its corre-
sponding target vector in the current
population. If the trial vector has less or
equal error value to the corresponding
target vector, the trial vector will replace
Algorithm 1 De-WeLM.
input: Training set X = {(x i, t i), t i ! {1, 2, ..., m}, i = 1, 2, ..., n}, R max .
Output: Approximate optimal weight matrix W *.
1: All data values x i of training set X are normalized in [–1,1], and then select valida-
tion set X vd. Set R = 0.
2: Generate initial population W R = {W i, R, i = 1, 2, ..., NP} by formula (11).
3: Compute the error E W i,R (i = 1, ..., NP) of WELM classifier with weight matrix
diag (W i, R) on validation set X vd and the candidate optimal weight vector W best,R by
formula (12).
4: while R 1 R max do
5: Compute the trial vectors U i, R (i = 1, ..., NP) from W R by formulas (13) and (14).
6: Compute the error E U i,R(i = 1, ..., NP) of WELM classifier with weight matrix
diag (U i, R) on validation set X vd by formula (15).
7: Obtain next stage population W R + 1 and a candidate optimal weight vector of the
next stage W best,R + 1 by formula (16).
8: R = R + 1.
9: end while
10: W * = diag (W best,Rmax).

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 37

 proach. The number of iterations R max
TAbLe 1 Specification of binary classification problems.
is set to 10. The validation set is random-
DATASeTS # ATTri # TrAiN # TeST IR ly assigned as a subset with half the ele-
YEAST05679VS4 8 422 106 0.1047 ments of the training set.
YEAST1458VS7 8 554 139 0.0453 In the experiments involving ELM,
YEAST1289VS7 8 385 97 0.0327 experienced WELM [7], and BWELM,
the grid search ranges for C and l are
YEAST1VS7 7 367 92 0.0700
conducted over {2 -18, 2 -16, ..., 2 48, 2 50}
ECOLI1 7 268 68 0.2947
and {10, 20, ..., 990, 1000}, respectively.
ECOLI2 7 268 68 0.1806 The search range of m for the Soft-
ECOLI3 7 268 68 0.1167 max classifier is set as {- 8, - 7, - 6,
ECOLI4 7 268 68 0.0635 ..., 1, 2, 3}.
YEAST2VS4 8 411 103 0.1078
To compare our algorithm with
other computational methods, we also
YEAST2VS8 8 385 97 0.0434
introduce the approximate weight search
GLASS0123VS456 9 171 43 0.3053
method based on PSO [44], [45]. Parti-
GLASS016VS2 9 153 39 0.0929 cle swarm optimization is one of the
GLASS016VS5 9 147 37 0.0500 most popular nature-inspired meta-
PIMA 8 614 154 0.5350 heuristic optimization algorithms, devel-
YEAST1 8 1187 297 0.4064
oped by Kennedy and Eberhart in 1995
[44], [45]. Since its development, many
YEAST3 8 1187 297 0.1230
variants have also been developed for
YEAST4 8 1187 297 0.0349
solving practical issues related to optimi-
YEAST5 8 1187 297 0.0304 zation [46]–[48]. In our paper, we use
SHUTTLEC0VSC4 9 1463 366 0.0726 the PSO proposed in [26].
SHUTTLEC2VSC4 9 103 26 0.0404 In the following experiments involv-
SEGMENT0 19 1846 462 0.1661
ing the PSO method, two acceleration
parameters c 1 and c 2 are set to 2, the
WISCONSIN 9 546 137 0.5380
inertia factor w is set to [0.4, 0.9], and
HABERMAN 3 244 62 0.3556 the maximal iteration is set to 50. The
IRIS0 4 120 30 0.5000 size of the initial population (swarm)
VOWEL0 13 790 198 0.1002 that is the initial set of candidate weights
NEW-THYROID1 5 172 43 0.1944 is set to 50, and W 1, W 2 , which are pre-
sented in [7] are included in the initial
NEW-THYROID2 5 172 43 0.1944
population (swarm) and the remaining
PAGE-BLOCKS1 10 377 95 0.0620
elements are randomly generated
GLASS0 9 173 43 0.4786 between [0,1]. The grid search ranges
GLASS1 9 171 43 0.5405 for C and l are also conducted over
GLASS2 9 171 43 0.0823 {2 -18, 2 -16, ..., 2 48, 2 50} and {10, 20, ...,
GLASS4 9 171 43 0.0621
990, 1000}, respectively. We call this
algorithm “PSO-WELM” in the follow-
GLASS5 9 171 43 0.0427
ing sections.
GLASS6 9 171 43 0.1554

VEHICLE0 18 676 170 0.3075 A. Dataset Specification

VEHICLE1 18 676 170 0.3439

VEHICLE2 18 676 170 0.3466 1) Standard Classification Datasets

VEHICLE3 18
YEAST6 8 1187
to tune for ELM, i.e., the trade-off
constant C and the number of hidden
nodes l. In our exper iments, these
two parameters for DE-WELM are
set as follows,
676 170 0.3330
Similar to the data specifications men-
tioned in the experienced WELM [7]
297 0.0243
and BWELM [8], we have selected 39
binary datasets and 3 multiclass datasets
C ! {2 -6, 2 -4, 2 -2, 2 0, 2 2, 2 4, 2 6} from the KEEL dataset repository
l ! {10, 110, 210, ..., 810, 910} . (18) which have different degrees of imbal-
ance. Details of the datasets used in
We set the control parameters to our experiments are shown in Tables 1
NP = 50, F = 1, CR = 0.8 in our ap - and 2, where the number of attributes

38 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

(#ATTRI), the number of classes
(#CLASS), the number of training
samples (#TRAIN), the number of test
samples (#TEST), and IR are listed.
The IR of the binary datasets selected
from the KEEL dataset varies from
0.0243 to 0.5405, and the IR of the
multiclass datasets varies from 0.0061
to 0.5882.
ten digits that is commonly used for images and 10,000 test images with each
training various image processing sys- image labeled by an integer, and 7,000
tems. The database is also widely used images per class. Each example is a 28 ×
for training and testing in the field of 28 single channel grayscale image. Exam-
machine learning [13], [49]–[52]. The ple images of the MNIST dataset are
MNIST dataset contains 60,000 training shown in Fig. 2(a).
TAbLe 2 Specification of multiclass classification problems.

DATASeTS # ATTri # CLASS # TrAiN # TeST IR

2) mnISt Dataset [13]
HAYES-ROTH 4 3 105 27 0.5882
The MNIST dataset (Mixed National
Institute of Standards and Technology NEW-THYROID 5 3 172 43 0.2066

database) is a large database of handwrit- PAGE-BLOCKS 10 5 438 110 0.0061

Airplane
Automobile
Bird

Cat
Deer
Dog
Frog
Horse
Ship
Truck
(a) MNIST (b) CIFAR-10

Arts and Entertainment 2,884,721

Games 1,202,429
Autos and Vehicles 1,149,528
Sports 1,099,091
Food and Drink 580,627
Computers and Electronics 489,413
Hobbies and Leisure 435,991
Pets and Animals 394,785
Science 388,718
Business and Industrial 370,558
Beauty and Fitness 327,335
Vertical

Internet and Telecom 252,385

Shopping 226,161
Home and Garden 160,147
Travel 152,482
People and Society 106,686
News 71,306
Reference 70,971
Jobs and Education 62,252
Law and Government 50,448
Books and Literature 50,448
Health 43,941
Real Estate 15,945
Finance 13,756
104 105 106 107
Number of Videos
(c) YouTube-8M

Figure 2 Example digital images of MNIST database and CIFAR-10 database, and number histograms of top training videos in YouTube-8M.

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 39

 TAbLe 3 Experimental results of binary problem.

DATASeTS GMEAN (SigMOiD MODe)

SOFTMAX eLM WeLM-W1 WeLM-W2 bWeLM PSO-WeLM De-WeLM

(%) (%) (%) (%) (%) (%) (%)

YEAST05679VS4 62.58 84.13 86.02 80.02 85.56 85.69 86.53

YEAST1458VS7 45.12 61.07 67.10 64.26 68.09 68.55 72.18

YEAST1289VS7 46.52 59.23 75.83 73.92 74.21 78.48 90.16

YEAST1VS7 49.72 74.70 76.65 81.36 86.23 88.24 85.45

ECOLI1 86.40 87.77 90.69 90.26 93.02 90.80 95.13

ECOLI2 91.82 89.44 90.19 88.64 90.80 94.50 95.04

ECOLI3 73.70 77.38 90.17 90.00 93.21 92.84 93.61

ECOLI4 85.15 91.96 97.83 95.90 100 100 99.19

YEAST2VS4 85.28 86.25 91.56 90.02 93.78 97.75 98.01

YEAST2VS8 81.63 72.83 75.56 76.01 81.32 80.32 81.65

GLASS0123VS456 97.10 95.34 95.66 95.55 100 100 100

GLASS016VS2 49.28 67.78 83.77 83.06 92.85 94.11 95.26

GLASS016VS5 100 92.41 98.55 98.70 100 100 100

PIMA 74.32 70.10 74.74 71.51 75.79 74.97 78.06

YEAST1 54.19 63.26 72.57 70.32 73.36 73.37 77.06

YEAST3 73.42 80.75 93.25 91.08 92.52 93.92 94.06

YEAST4 40.54 65.52 87.92 73.92 83.51 93.05 93.01

YEAST5 61.02 81.04 95.39 95.15 98.25 98.79 99.04

SHUTTLEC0VSC4 100 100 100 100 100 100 100

SHUTTLEC2VSC4 100 93.54 100 100 100 100 100

SEGMENT0 100 99.24 99.75 99.70 99.78 100 100

WISCONSIN 95.58 97.85 97.62 96.95 97.34 98.25 98.48

HABERMAN 23.57 49.16 65.11 59.26 73.74 77.24 78.95

IRIS0 100 100 100 100 100 100 100

VOWEL0 85.58 100 100 100 100 100 100

NEW-THYROID1 98.64 98.24 99.44 99.72 100 100 100

NEW-THYROID2 100 95.55 99.72 100 100 100 100

PAGE-BLOCKS1 97.78 99.09 99.43 99.42 100 100 100

GLASS0 74.65 78.51 80.29 81.35 86.66 88.98 87.18

GLASS1 38.29 78.79 78.96 80.87 79.54 84.04 86.09

GLASS2 57.01 90.61 91.33 88.34 90.15 93.07 94.08

GLASS4 69.79 85.72 91.34 91.46 96.45 96.74 97.33

GLASS5 97.47 90.81 95.99 96.60 100 100 100

GLASS6 89.44 94.96 95.72 95.90 100 100 100

VEHICLE0 95.97 98.51 99.12 99.09 96.71 100 99.62

VEHICLE1 69.54 84.14 85.21 82.75 75.98 87.23 87.47

VEHICLE2 95.99 98.43 99.12 98.78 97.31 100 100

VEHICLE3 60.92 78.15 85.13 84.34 85.37 86.52 87.37

YEAST6 70.59 70.77 87.77 88.29 90.68 89.67 90.02

AVERAGE 76.37 84.18 89.60 88.52 91.07 92.49 93.33

40 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

3) CIFaR-10 Dataset [14]
The CIFAR-10 dataset is a labeled sub-
set of the 80 Million Tiny Images data-
set3, containing 60,000 color images (32
× 32) in 10 classes, with 6,000 images
per class. They are split into 50,000
training images and 10,000 test images.
Example images of the CIFAR-10 data-
base are shown in Fig. 2(b).
4) Youtube Dataset
The YouTube-8M dataset [15] contains
approximately 8 million YouTube vid-
eos. Each video is annotated with one
or multiple tags for a total of 4,716 class-
es. All 4,716 classes can be grouped into
24 top-level vertical categories. In Fig. 2(c),
we show histograms with the number of
training videos in each top-level vertical
(or top-level) category.
B. Experimental Results on
Standard Classification Datasets
The experimental results of seven meth-
ods (Softmax, ELM, WELM-W1,
WELM-W2, BWELM, PSO-WELM,
and DE-WELM) under 39 binary im-
balanced datasets and 3 multiclass imbal-
anced datasets are given in Tables 3 and 4,
respectively. The results in both tables
show that our DE-WELM can perform
much better than all the other state-of-
the-art methods, and the evaluation
metric Gmean can improve by 30%
compared to the classic ELM classifier.
The average performance metrics also
indicate significant improvement using
our approach.
1) Comparison with
Softmax Classifier
As can be seen in Tables 3 and 4, our
DE-based algorithm can achieve satis-
factory performance for all datasets
compared with the Softmax classifier.
Specifically, on the datasets YEAST-
1289VS7, HABERMAN, YEAST4,
GLASS1, and GLASS016VS2, DE-WELM
can achieve an improvement of more
than 40% on Gmean. There are six datas-
ets where both classifiers achieve perfect
100% Gmean values.The Softmax classifi-
er can perform much better than other
3
http://groups.csail.mit.edu/vision/TinyImages/
algorithms except for our DE-WELM
on the datasets of ECOLI2 and GLASS-
0123VS456, which also indicates that
WELM-W1, WELM-W2, and BWELM
are not optimal.
2) Comparison with
Experienced Weighted Elm
Based on the results in Table 3, our DE-
WELM can perform much better in all
datasets compared with experienced
WELM, especially on the datasets
YEAST1289VS7, YEAST4, and
GLASS016VS2, achieving an improve-
ment of more than 10% on Gmean.
Regarding the balanced datasets such as
GLASS1, WISCONSIN, PIMA, etc., the
experimental results of DE-WELM are
also better than experienced WELM,
which shows that our method is applica-
ble to not only imbalanced datasets, but
also to balanced datasets. It also indicates
that the two experienced weighting
schemes employed in [7] are not optimal.
3) Comparison with
Boosting Weighted Elm
As shown in Table 3, DE-WELM can
achieve superior performance on 39
datasets (92.85%) versus BWELM
except for ECOLI4, YEAST6, and
YEAST1VS7. There are 11 datasets
(26.19%) where both DE-WELM and
BWELM achieve perfect 100% Gmean
values, which indicates that optimal
weights are reached for those datasets.
We further focus on the three datasets
where DE-WELM performs worse than
BWELM. The Gmean values of our
approach are all slightly lower (less than
1%) than BWELM. The Gmean values
of our approach on other datasets are
TAbLe 4 Experimental results of multiclass problems.
DATASeTS GMEAN (SigMOiD MODe)
WeLM-
SOFTMAX eLM W1
(%) (%) (%)
HAYES-ROTH 42.51 67.94 69.65
NEW-THYROID 80.69 93.11 92.83
PAGE-BLOCKS 88.46 94.41 89.71
AVERAGE 70.55 85.15 84.06
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE
better than the values of BWELM,
where the Gmean values in some datas-
ets, such as VEHICLE1, YEAST4, and
YEAST1289VS7, are 10% higher than
the values of BWELM. The overall com-
parable results indicate that DE-WELM
potentially represents a superior solution
as an optimal WELM classifier com-
pared with BWELM. This conclusion is
further supported by the results for aver-
age Gmean values.
4) multiclass Imbalanced learning
The experimental results on multiclass
data are shown in Table 4, the results
also show that DE-WELM can achieve
much better performance compared
with other methods except on NEW-
THYROID, where the Gmean value of
our method is 0.04% lower than the
value of BWELM. The results in Table 4
indicate that our DE-WELM potential-
ly represents the best solution as an opti-
mal WELM classifier in multiclass
imbalanced datasets.
5) analysis on the Search Ranges
In ELM related methods, the best
parameters for the trade-off constant C
and the number of hidden nodes l are
normally obtained by a grid search pol-
icy, which pre-sets search ranges for
these two parameters and exhaustively
searches all possible combinations of
these two parameters. Obviously, the
computation efficiencies are negatively
related to the search ranges, while the
probabilities to obtain the optimal
solution are positively related to the
search ranges.
In the beginning of our experiment
section, we mentioned that the settings
WeLM- PSO- De-
W2 bWeLM WeLM WeLM
(%) (%) (%) (%)
63.52 71.65 71.20 72.20
92.58 93.15 93.51 93.11
94.33 93.52 92.85 94.60
83.47 86.10 85.85 86.63
41
of C and l in our DE-WELM are based Fig. 3. As shown in Fig. 3, a minimum Gmean values. Comparing with PSO-
ove r {2 -6, 2 -4, 2 -2, 2 0, 2 2, 2 4, 2 6} a n d of 9 iterations are required for the train- WELM, there are six datasets (14.29%)
{10, 110, 210, ..., 810, 910}, respectively, ing process to converge in WISCON- (YEAST1458VS7, YEAST1289VS7,
and the settings of C and l in other SIN, 10 in PIMA, 6 in GLASS5, 7 in ECOLI1, PIMA, YEAST1, GLASS1)
methods are {2 -18, 2 -16, ..., 2 48, 2 50} and SEGMENT0, 10 in HABERMAN, and where DE-WELM achieves an improve-
{10, 20, ..., 990, 1000}, respectively. The 10 in GLASS0. ment of over 2% on Gmean, 11 datasets
experimental results in Tables 3 and 4 The results in Fig. 3 indicate that (26.19%) (YEAST05679VS4, ECOLI2,
indicate that DE-WELM uses a smaller although the two sets of parameters are ECOLI3, YEAST2VS8, GLASS016VS2,
search range compared to other state-of- quite different, DE-WELM will quickly HABERMAN, and so on) where DE-
the-art methods, and can achieve much converge to the optimum within 10 iter- WELM can achieve an improvement
better performance compared to other ations on all the test datasets, which indi- greater than 0.5% on Gmean, and six data-
state-of-the-art methods. Thus, our DE- cates DE-WELM can easily converge sets (14.29%) where DE-WELM achieves
WELM represents a more efficient opti- toward the optimal solution. Therefore, an improvement of more than 0.14% on
mal WELM classifier. we set the number of iterations to Gmean. DE-WELM achieves an improve-
R max = 10 in the previous experiments. ment above 0.84% regarding average
6) analysis of the Convergence Gmean for binary datasets, and an
on Iterations 7) Comparison with improvement greater than 0.78% on aver-
In this section, we will further evaluate other Computational age Gmean for multiclass datasets com-
the convergence on iteration of our method Based WElm pared with PSO-WELM.
DE-WELM. We carry out two experi- As shown in Tables 3 and 4, our DE-
ments on six datasets, i.e., WISCONSIN, WELM can achieve superior perfor- 8) temporal Complexity analysis
HABERMAN, GLASS5, SEGMENT0, mance to PSO-WELM on 36 datasets Regarding the temporal complexity,
GLASS0, and PIMA. We use two sets of (85.71%) except for ECOLI4, YEAST4, assuming that the temporal computation
C and l parameters for the two experi- GLASS0, VEHICLE0, YEAST1VS7, cost of a single WELM on the given
ments and plot the training Gmean val- and NEW-THYROID. Among all datas- parameters of C, l is denoted as t WELM ,
ues after every iteration of the DE ets, there are 13 datasets (30.95%) where then the temporal complexities of
algorithm. The results are shown in both methods achieve perfect 100% WELM based methods can be expressed

Wisconsin (IR = 0.5380) PIMA (IR = 0.5350) Glass 5 (IR = 0.0427)

1.1 1.1 1.1
1.0 1.0 1.0
Training Gmean

Training Gmean

Training Gmean

0.9 0.9 0.9

0.8 0.8 0.8
0.7 0.7 0.7
0.6 0.6 0.6
L : 310, C : 24 L : 410, C : 2 L : 110, C : 2
0.5 L : 610, C : 2–4 0.5 L : 810, C : 2–2 0.5 L : 210, C : 2–4
0.4 0.4 0.4
0 5 10 15 20 0 5 10 15 20 0 5 10 15 20
Iteration Steps Iteration Steps Iteration Steps
(a) (b) (c)

Segment 0 (IR = 0.1661) Haberman (IR = 0.3556) Segment 0 (IR = 0.1661)

1.1 1.1 1.1
1.0 1.0 1.0
Training Gmean

Training Gmean

Training Gmean

0.9 0.9 0.9

0.8 0.8 0.8
0.7 0.7 0.7
0.6 0.6 0.6
L : 110, C : 22 L : 310, C : 22 L : 110, C : 2–2
0.5 L : 710, C : 2–6 0.5 L : 510, C : 2–4 0.5 L : 710, C : 2–6
0.4 0.4 0.4
0 5 10 15 20 0 5 10 15 20 0 5 10 15 20
Iteration Steps Iteration Steps Iteration Steps
(d) (e) (f)

Figure 3 Gmean values after every iteration of our DE-based method on six standard classification datasets.

42 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

as the search numbers multiplied by the
basic temporal computation cost of a
single ELM as follows,
T = t WELM #
(Psize # I max # C number # L number) (19)
where Psize is the size of the candidate
population, I max is the number of itera-
tions, C number is the number of possible
trade-off constants C, and L number is the
number of possible hidden nodes l.
Considering the above equation and
our experiment settings, the temporal
complexity of our algorithm is t WELM
# 50 # 10 # 7 # 10 = t WELM # 35, 000
and the temporal complexity of PSO-
WELM is t WELM # 50 # 100 # 35 # 100
= t WELM # 17, 500, 000. Consider ing
that the initial candidate weight of
BWELM is one and requires 10 itera-
tions, the temporal complexity of
BWELM is t WELM # 1 # 10 # 35 # 100
= t WELM # 35, 000. That is to say, our
DE-WELM is 500 times faster than
PSO-WELM and is equal to BWELM
in terms of temporal complexity.
As mentioned for BWELM in [8]
and considering the very fast learning
speed of ELM, such costs for DE-WELM
are quite acceptable.
ECOLI 1 (IR = 0.2947)
1.00
0.95
Gmean
0.90
0.85
1 2 3 4 5 6 7
Experiment Number
(a)
Figure 4 Experimental results of the convergence on initialization of weight on two datasets (ECOLI1, GLASS2).
9) analysis of the Convergence
on Different Initializations
Both our DE-WELM and PSO-WELM
employ candidate weight populations
that are randomly generated at the
beginning of the algorithm. In the case
of BWELM, the initial weights are
determined directly by the data. In this
section, we will further evaluate the con-
vergence on different initial candidate
weight populations. We set the same ini-
tial population, which is generated ran-
domly, for the former two methods and
perform comparative experiments on
two datasets (ECOLI1, GLASS2). The
experiments were carried out ten times
with different initializations. At the same
time, corresponding experiments on
BWELM were also performed. The
results can be seen in Fig. 4.
In Fig. 4, the horizontal axis indicates
the sequence number of the different ini-
tial candidate weight populations and the
vertical axis denotes the resulting Gmean
values obtained in each experiment.
Among 10 experiments for ECOLI1,
DE-WELM achieved better classification
performance than PSO-WELM in 9
experiments (90%), and in 7 experiments
(70%) compared to BWLEM. Among
the 10 experiments for GLASS2, DE-
1.00
0.95
Gmean
0.90
0.85
8 9 10 1 2
DE-WELM PSO-WELM
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE
WELM achieved better classification
performance than PSO-WELM in 8 ex-
periments (80%), and in 9 experiments
(90%) compared to BWLEM.
C. Experimental Results
on MNIST Dataset
We introduce the LeNet-5 model to
evaluate the performance of DE-WELM
versus three state-of-the-art methods
(classic ELM classifier, experienced
WELM classifier, and BWELM) on the
MNIST dataset.The LeNet-5 model [13]
is a classic convolutional neural network
(CNN) architecture proposed by LeCun
et al. in 1998, which was applied in hand-
written digit character recognition.
We perform experiments with the
LeNet-5 model regarding the first few
layers of the network as feature extrac-
tors to evaluate DE-WELM.
We first arbitrar ily select three
datasets that do not overlap and bal-
ance among the 70,000 MNIST datas-
ets (60,000 training sets, 10,000 test
sets). These three datasets are referred
to as RES_DATA (representation
training set), C_TRAIN (classifier
training set), and C_TEST (classifier
test set). The details of the selected
datasets are provideds in Table 5,
Glass 2 (IR = 0.0823)
3 4 5 6 7 8 9 10
Experiment Number
(b)
BWELM
43
 nected layer on top for classification. We
We perform experiments with the LeNet-5 model use the simplest implementation to train
regarding the first few layers of the network as feature a CifarNet model as a baseline model
without any preprocessing such as image
extractors to evaluate DE-WELM. translations or transformations.
We perform experiments on Cifar-
where the number of attributes (#ATTRI), The experimental results of six Net regarding the first few layers of the
the number of classes (#CLASS), the num- methods (LeNet-5, ELM, WELM-W1, network as feature extractors to evaluate
ber of selected data (#SAMPLE) and, IR WELM-W2, BWELM, and DE-WELM) our classifier algorithm.
are listed. under selected datasets are given in We first arbitrarily select three datasets
Then we train the LeNet-5 model Table 7. The parameters used in the that do not overlap and balance (among
on Res_Data by a popular deep learning experiment were the same as those used 60,000 images from the CIFAR-10 data-
framework known as Caffe [53]. The for the standard classification problem set: 50,000 for the training set, 10,000 for
parameters we used for training the datasets. Table 7 indicates that DE- the test set), as RES_DATA (representa-
CNN are the default values in Caffe. WELM achieves better performance tion training set), C_TRAIN (classifier
The LeNet-5 model outputs fea- than the other methods. training set) and C_TEST (classifier test
ture images of original image data set).The details of the selected datasets are
which have 500 dimensions in its first D. Experimental Results on shown in Table 6, where the number of
fully connected layer. We regard the CIFAR-10 Dataset attributes (#ATTRI), the number of
output of the LeNet-5 model’s first We introduce the CifarNet model [14] to classes (#CLASS), the number of selected
fully connected layer on C_TRAIN as evaluate the performance of our DE- data (#SAMPLE), and IR are listed.
the training set to evaluate our classifi- WELM with four state-of-the-art meth- We train CifarNet on RES_DATA
er algorithm, and we take the output ods (ELM, WELM-W1, WELM-W2, using Caffe and its default parameters [53].
of LeNet-5 model’s first fully connect- and BWELM) on CIFAR-10. Proposed We extract and reshape the output of
ed layer on C_TEST as the test set to by Alex Krizhevsky, CifarNet is the state- the “Pool3” layer, which has 1024 dimen-
evaluate our DE-WELM. We also eval- of-the-art method for object classification sions, and use the extracted features of
uate the classification of LeNet-5 using on the CIFAR-10 dataset. It has three the CIFAR-10 dataset to evaluate five
the above two data to compare with convolution layers and three pooling lay- classifiers (ELM, WELM-W1, WELM-
DE-WELM. ers for feature extraction, and a fully con- W2, BWELM, and DE-WELM). We
regard the output of CifarNet model’s
“Pool3” layer on C_TRAIN as the
training set, and the output of CifarNet
TAbLe 5 Selected imbalanced datasets from MNIST.
model’s “Pool3” layer on C_TEST as
DATASeTS # ATTri # CLASS # SAMPLe IR the test set to evaluate DE-WELM. We
RES_DATA 500 10 13470 0.0848 also evaluated classification by the Cifar-
C_TRAIN 500 10 3329 0.0872 Net model using the above two datasets
to compare with our DE-WELM.
C_TEST 500 10 4991 0.0872
The experimental results for six meth-
ods (CifarNet, ELM, WELM-W1,
WELM-W2, BWELM, and DE-WELM)
TAbLe 6 Selected imbalanced datasets from CIFAR-10. under selected imbalanced datasets are
given in Table 7. The parameters in the
DATASeTS # ATTri # CLASS # SAMPLe IR
experiment were the same as previous
RES_DATA 1024 10 12351 0.0839
experiments. As shown in Table 7,
C_TRAIN 1024 10 3121 0.0918 DE-WELM achieves better performance
C_TEST 1024 10 4679 0.0918 than other methods except for CifarNet.

TAbLe 7 Experimental results on MNIST and CIFAR-10.

DATAbASeS gMeAN (SigMOiD MODe)

LENET-5 CIFARNET ELM WELM-W1 WELM-W2 BWELM DE-WELM
(%) (%) (%) (%) (%) (%) (%)
MNIST 96.82 95.57 97.58 97.5 97.48 97.72
CIFAR-10 58.84 37.33 53.94 55.73 55.54 57.25

44 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

 Airplane 0.57 0.05 0.06 0.04 0.06 0.05 0.05 0.05 0.04 0.03 Airplane 0.42 0.01 0.07 0.06 0.06 0.12 0.06 0.11 0.08 0.02
Automobile 0.02 0.65 0.03 0.05 0.05 0.04 0.04 0.05 0.06 0.02 Automobile 0.04 0.37 0.09 0.08 0.08 0.09 0.09 0.06 0.07 0.04
Bird 0.00 0.01 0.67 0.04 0.05 0.03 0.04 0.03 0.05 0.06 Bird 0.01 0.08 0.32 0.03 0.06 0.11 0.09 0.13 0.13 0.04
True Expression

True Expression
Cat 0.04 0.07 0.07 0.48 0.05 0.07 0.06 0.07 0.06 0.03 Cat 0.03 0.07 0.06 0.43 0.06 0.09 0.06 0.08 0.07 0.05
Deer 0.03 0.05 0.05 0.06 0.55 0.06 0.06 0.05 0.07 0.02 Deer 0.05 0.06 0.10 0.09 0.36 0.05 0.09 0.08 0.09 0.02
Dog 0.03 0.06 0.06 0.06 0.08 0.48 0.06 0.05 0.09 0.03 Dog 0.02 0.08 0.08 0.08 0.06 0.42 0.06 0.08 0.07 0.05
Frog 0.02 0.05 0.03 0.05 0.05 0.05 0.62 0.04 0.05 0.03 Frog 0.04 0.08 0.08 0.08 0.08 0.08 0.35 0.07 0.09 0.03
Horse 0.04 0.04 0.06 0.03 0.08 0.06 0.04 0.60 0.04 0.02 Horse 0.04 0.07 0.08 0.08 0.07 0.10 0.10 0.32 0.09 0.07
Ship 0.03 0.03 0.09 0.04 0.04 0.04 0.05 0.06 0.58 0.02 Ship 0.02 0.05 0.05 0.06 0.10 0.05 0.08 0.08 0.45 0.05
Truck 0.05 0.05 0.02 0.00 0.03 0.02 0.05 0.03 0.02 0.72 Truck 0.03 0.05 0.07 0.07 0.07 0.14 0.09 0.10 0.03 0.34

Ai

Au

Bi

Fr

Sh

Tr
Ai

Au

Bi

Fr

Sh

Tr

at

ee

og

or
at

ee

og

or

rp

rd

og

uc
rp

rd

og

uc

to

ip
to

ip

se
se

r
l

an

k
m
an

k
m

ob
ob

e
e

ile
ile

Recognized Expression Recognized Expression

(a) CifarNet (b) ELM

Airplane 0.55 0.03 0.06 0.06 0.06 0.05 0.06 0.06 0.07 0.01 Airplane 0.57 0.04 0.04 0.06 0.04 0.08 0.06 0.01 0.06 0.03
Automobile 0.03 0.60 0.04 0.05 0.04 0.05 0.06 0.05 0.06 0.02 Automobile 0.03 0.60 0.04 0.04 0.05 0.04 0.06 0.05 0.05 0.03
Bird 0.01 0.11 0.61 0.03 0.03 0.07 0.05 0.03 0.02 0.03 Bird 0.00 0.06 0.63 0.03 0.05 0.02 0.05 0.11 0.03 0.01
True Expression

True Expression

Cat 0.03 0.07 0.08 0.45 0.06 0.07 0.06 0.09 0.06 0.03 Cat 0.04 0.06 0.06 0.45 0.07 0.07 0.08 0.06 0.08 0.03
Deer 0.03 0.07 0.04 0.06 0.49 0.08 0.07 0.07 0.06 0.02 Deer 0.02 0.05 0.07 0.08 0.50 0.05 0.07 0.07 0.05 0.03
Dog 0.04 0.07 0.08 0.07 0.06 0.48 0.06 0.05 0.05 0.02 Dog 0.04 0.06 0.06 0.06 0.05 0.50 0.07 0.05 0.05 0.05
Frog 0.02 0.04 0.05 0.06 0.03 0.05 0.65 0.04 0.04 0.03 Frog 0.02 0.04 0.05 0.05 0.03 0.04 0.65 0.04 0.04 0.03
Horse 0.02 0.06 0.12 0.05 0.05 0.05 0.04 0.51 0.07 0.03 Horse 0.03 0.10 0.06 0.05 0.08 0.08 0.04 0.51 0.04 0.01
Ship 0.04 0.05 0.06 0.06 0.06 0.09 0.05 0.06 0.50 0.02 Ship 0.02 0.06 0.06 0.06 0.08 0.05 0.06 0.04 0.52 0.02
Truck 0.05 0.09 0.03 0.07 0.07 0.03 0.03 0.02 0.00 0.60 Truck 0.03 0.03 0.05 0.02 0.00 0.07 0.09 0.02 0.00 0.69
Ai

Au

Bi

Fr

Sh

Tr

Ai

Au

Bi

Fr

Sh

Tr
at

ee

og

or

at

ee

og

or
rp

rd

og

uc

rp

rd

og

uc
to

ip

to

ip
se

se
la

la

r
k
m

k
m
ne

ne
ob

ob
ile

ile

Recognized Expression Recognized Expression

(c) WELM-W1 (d)WELM-W2

Airplane 0.56 0.02 0.03 0.03 0.08 0.08 0.03 0.06 0.06 0.04 Airplane 0.56 0.02 0.06 0.08 0.06 0.06 0.05 0.03 0.05 0.03
Automobile 0.01 0.60 0.05 0.05 0.05 0.06 0.05 0.06 0.05 0.02 Automobile 0.03 0.60 0.04 0.06 0.04 0.04 0.04 0.05 0.07 0.03
Bird 0.03 0.02 0.63 0.06 0.04 0.04 0.08 0.03 0.04 0.01 Bird 0.01 0.02 0.79 0.03 0.02 0.04 0.04 0.03 0.00 0.01
True Expression

True Expression

Cat 0.02 0.06 0.07 0.45 0.08 0.06 0.08 0.06 0.07 0.04 Cat 0.04 0.06 0.06 0.45 0.06 0.07 0.10 0.08 0.06 0.02
Deer 0.04 0.09 0.05 0.03 0.50 0.05 0.05 0.08 0.08 0.04 Deer 0.05 0.06 0.07 0.07 0.50 0.05 0.05 0.07 0.05 0.03
Dog 0.04 0.07 0.05 0.07 0.06 0.50 0.08 0.05 0.06 0.03 Dog 0.03 0.08 0.08 0.05 0.06 0.49 0.06 0.05 0.06 0.03
Frog 0.03 0.05 0.04 0.03 0.05 0.04 0.65 0.03 0.04 0.03 Frog 0.02 0.07 0.03 0.04 0.03 0.05 0.63 0.04 0.04 0.03
Horse 0.01 0.06 0.06 0.10 0.04 0.06 0.05 0.52 0.08 0.02 Horse 0.07 0.07 0.07 0.04 0.07 0.04 0.06 0.51 0.06 0.01
Ship 0.05 0.04 0.05 0.08 0.05 0.07 0.07 0.05 0.50 0.02 Ship 0.03 0.08 0.07 0.05 0.04 0.04 0.05 0.06 0.51 0.05
Truck 0.02 0.09 0.02 0.05 0.02 0.00 0.07 0.02 0.03 0.69 Truck 0.02 0.07 0.10 0.00 0.00 0.03 0.00 0.00 0.02 0.76
Ai

Au ne

Bi ob

Fr

Sh

Tr

Ai

Au ne

Bi ob

Fr

Sh

Tr
at

ee

og

or

at

ee

og

or
rp

rd

rp

rd
og

uc

og

uc
to

ip

to

ip
se

se
la

la
r

r
k

k
m

m
ile

ile

Recognized Expression Recognized Expression

(e) BWELM (f) DE-WELM

Figure 5 Confusion matrices obtained by applying six algorithms (CifarNet, ELM, WELM-W1, WELM-W2, BWELM, DE-WELM) on CIFAR-10 dataset.

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 45


TAbLe 8 Selected imbalanced datasets from YouTube.
DATASeTS # ATTri # CLASS
TRAINDATA 1024 24
TESTDATA 1024 24
TAbLe 9 Experimental results on YouTube dataset.
eXPeriMeNT
iNDeX WeLM-W1 WeLM-W2
GMEAN(%) 47.78 47.40
TOTAL RUNNING 2287.57 2236.45
TIME(S)
The Gmean value of our approach is
1.59% lower than CifarNet with the
original Softmax classifier. We can argue
that the model has been custom-trained
in the network training stage for the
Softmax classifier layer by back-propaga-
tion from the classification layer to the
feature extraction part. Hence the origi-
nal structure of the classifier works bet-
ter for the feature extractor than our
independent classifier. A relatively similar
work to this paper is [54]. The authors
compared a “one-shot” fine-tuning
with Softmax to a multi-stage training
method using support vector machines
(SVMs) and the results indicated that
Softmax slightly outperforms SVMs. On
the other hand, the number of parame-
ters in the classifier is much smaller than
the number in the feature extraction
layers, which leads to a fast recovery for
the Softmax layer during the classifier
training stage. To some degree, the Soft-
max layer recalls the samples in the
network training set. In the case with
an independent classifier (DE-WELM),
the results are close to the original clas-
sifier performance of CifarNet, indicat-
ing the effectiveness and robustness of
our algorithm.
In Fig. 5, we show the confusion
matrices obtained by applying the six
methods (CifarNet, ELM, WELM-W1,
WELM-W2, BWELM, and DE-
WELM). As can be observed, our meth-
od achieves better performance than
CifarNet for four classes (bird, dog, frog,
truck). It is worth considering here that
“bird” and “truck” classes belong to the
minority class. The sample numbers of
46 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
# SAMPLe IR
10534 0.0018
5267 0.0018
bWeLM PSO-WeLM De-WeLM
46.25 47.84 49.51
23301.35 3535679.05 24805.23
“bird” and “truck” in C_TRAIN are
respectively 95 and 58, which are the
two smallest classes among the classes of
C_TRAIN. This shows that our DE-
WELM is more suitable for imbalanced
data classification.
The experimental results on the
selected feature data also show that DE-
WELM achieves better performance
compared with the other four methods.
E. Experimental Results
on YouTube Dataset
In this dataset, visual and audio features
are pre-extracted. Visual features are
obtained using the Google Inception
CNN, which is pre-trained on ImageN-
et [55], and those features are then
reduced by PCA-compression into a
1024-dimensional vector. The audio fea-
tures are extracted from a pre-trained
VGG [56] network. In the official split,
the dataset is divided into three parts:
70% for training, 20% for validation, and
10% for testing. In our experiment, we
randomly selected 15,801 videos, which
have unique top-level vertical categories
with visual features from the official
dataset, to validate our algorithm on
video classification problems. The de-
tails of the selected datasets are shown
in Table 8.
The experimental results of these five
methods, i.e., WELM-W1, WELM-W2,
BWELM, PSO-WELM, and DE-
WELM, under selected imbalanced
datasets are given in Table 9. The param-
eters for the experiment were the same
as the ones in the experiment on stan-
dard classification datasets. As shown in
Table 9, our DE-WELM achieves better
performance than the other methods.
The experimental results indicate
that our DE-WELM model is an effec-
tive and efficient algorithm for solving
the class imbalanced problem in varied
datasets. However, owing to an embed-
ded optimization search procedure in
the algorithm, DE-WELM requires
more training time. Table 9 provides the
total running time of various learning
algorithms on the YouTube datasets.
From Table 9, the execution time of
DE-WELM is several times higher than
WELM-W1 and WELM-W2. As our
proposed DE-WELM employs the vali-
dation set in the training step, our algo-
rithm is a little slower than BWELM.
VI. Conclusions
In this paper, we present the formal
mathematical model to obtain the opti-
mal weight scheme in WELM for
imbalanced learning problems. We also
propose DE-WELM to calculate the
approximate optimal weight matrix of
the model. The effectiveness of DE-
WELM is proven by experiments con-
ducted using 39 binary datasets and 3
multiclass datasets which have different
degrees of imbalance, as well as two
large-scale image datasets from MNIST
and CIFAR-10, and one large-scale
video dataset from YouTube-8M.
Our DE-based approximate weight
calculation algorithm requires only 10
iterations to converge to a solution and
uses a smaller search range than other
state-of-the-art methods for the trade-
off between constant C and the number
of hidden nodes l. As the initial popula-
tion in our approach contains two expe-
rienced weighting schemes, W1 and W2,
our algorithm can preserve the advan-
tages of WELM [7]. We also performed
experiments comparing other computa-
tional methods, and the experimental
results show that DE-WELM can im-
prove overall classification performance
significantly with less temporal com-
plexity. Future work will focus on more
effectively structuring the initial popula-
tion into the DE algorithm, and apply-
ing DE-WELM to datasets with large
variety in class distributions.
Acknowledgments
This work was supported in part by the
National Natural Science Foundation of
China under Grant U1509210 and
U1609210, and the National Key Re-
search and Development Program of
China under Grant 2017YFB1302003.
References
[1] G.-B. Huang, Q.-Y. Zhu, and C. K. Siew, “Extreme
learning machine: Theory and applications,” Neurocom-
puting, vol. 70, no. 1–3, pp. 489–501, Dec. 2006.
[2] G.-B. Huang, H. Zhou, X. Ding, and R. Zhang, “Ex-
treme learning machine for regression and multiclass clas-
sification,” IEEE Trans. Syst., Man, Cybern., Syst. B, vol.
42, no. 2, pp. 513–529, Mar. 2012.
[3] G.-B. Huang and L. Chen, “Convex incremental
extreme learning machine,” Neurocomputing, vol. 70, no.
16–18, pp. 3056–3062, Aug. 2007.
[4] G.-B. Huang, D. Wang, and Y. Lan, “Extreme learn-
ing machines: A survey,” Int. J. Mach. Learn. Cybern., vol.
2, no. 2, pp. 107–122, May 2011.
[5] C. R. Rao and S. K. Mitra, Generalized Inverse of Matri-
ces and its Applications. New York, NY: Wiley, 1971.
[6] D. Serre, Matrices: Theory and Applications. New York,
NY: Springer-Verlag, 2002.
[7] W. Zong, G.-B. Huang, and Y. Chen, “Weight-
ed extreme learning machine for imbalance learn-
ing,” Neurocomputing, vol. 101, no. 3, pp. 229–242,
Aug. 2013.
[8] K. Li, X. Kong, Z. Lu, L. Wenyin, and J. Yin, “Boost-
ing weighted ELM for imbalanced learning,” Neurocom-
puting, vol. 128, no. 5, pp. 15–21, May 2014.
[9] K.-A. Toh, “Deterministic neural classification,” Neu-
ral Comput., vol. 20, no. 6, pp. 1565–1595, Apr. 2008.
[10] W. Deng, Q. Zheng, and L. Chen, “Regularized
extreme learning machine,” in Proc. IEEE Symp. Compu-
tational Intelligence and Data Mining, Nashville, TN, Apr.
2009, pp. 389–395.
[11] H. He and E. A. Garcia, “Learning from imbalanced
data,” IEEE Trans. Knowl. Data Eng., vol. 21, no. 9, pp.
1263–1284, May 2009.
[12] R. Storn and K. Price, “Differential evolution: A
simple and efficient adaptive scheme for global optimi-
zation over continuous spaces,” International Computer
Science Institute, Berkeley, CA, Tech. Rep. TR-95-012,
May 1995.
[13] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner,
“Gradient-based learning applied to document recogni-
tion,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov.
1998.
[14] A. Krizhevsky, “Learning multiple layers of features
from tiny images,” University of Toronto, Toronto, ON,
Tech. Rep., Apr. 2009.
[15] S. Abu-El-Haija, N. Kothari, J. Lee, A. Natsev,
G. Toderici, B. Varadarajan, and S. Vijayanarasimhan,
“Youtube-8m: A large-scale video classification bench-
mark,” CoRR, vol. abs/1609.08675, Nov. 2016.
[16] R. Fletcher, Practical Methods of Optimization, Vol-
ume 2: Constrained Optimization. New York, NY: Wi-
ley, 1981.
[17] E. Cambria, G.-B. Huang, L. L. C. Kasun, H. Zhou,
C. M. Vong, J. Lin, J. Yin, Z. Cai, Q. Liu, K. Li, V. C.
M. Leung, L. Feng, Y.-S. Ong, M.-H. Lim, A. Akusok,
A. Lendasse, F. Corona, R. Nian, Y. Miche, P. Gastaldo,
R. Zunino, S. Decherchi, X. Yang, K. Mao, B.-S. Oh, J.
Jeon, K.-A. Toh, A. B. J. Teoh, J. Kim, H. Yu, Y. Chen,
and J. Liu, “Extreme learning machines [trends & con-
troversies],” IEEE Intell. Syst., vol. 28, no. 6, pp. 30–59,
Nov. 2013.
[18] G.-B. Huang, E. Cambria, K. A. Toh, B. Widrow,
and Z. Xu, “New trends of learning in computational
intelligence [guest editorial],” IEEE Comput. Intell. Mag.,
vol. 10, no. 2, pp. 16–17, May 2015.
[19] I. Chaturvedi, E. Ragusa, P. Gastaldo, R. Zunino,
and E. Cambria. (2017, July). Bayesian network based
extreme learning machine for subjectivity detection, J.
Franklin Inst. [Online]. Available: http://www.sciencedirect
.com/science/article/pii/S0016003217303009
[20] E. Soria-Olivas, J. Gomez-Sanchis, J. D. Martin, J.
Vila-Frances, M. Martinez, J. R. Magdalena, and A. J.
Serrano, “BELM: Bayesian extreme learning machine,”
IEEE Trans. Neural Netw., vol. 22, no. 3, pp. 505–509,
Mar. 2011.
[21] Z. Bai, G.-B. Huang, D. Wang, H. Wang, and M.
B. Westover, “Sparse extreme learning machine for
classification,” IEEE Trans. Cybern., vol. 44, no. 10, pp.
1858–1870, Oct. 2014.
[22] Z. Bai, G.-B. Huang, and D. Wang, “Sparse extreme
learning machine for regression,” in Proc. Extreme Learn-
ing Machine, Hangzhou, China, pp. 471–490, vol. 2, Dec.
2015.
[23] P. Gastaldo, R. Zunino, E. Cambria, and S.
Decherchi, “Combining ELMs with random projec-
tions,” IEEE Intell. Syst., vol. 28, no. 6, pp. 46–48, Nov.
2013.
[24] R. E. Schapire, “The boosting approach to ma-
chine learning: An overview,” in Nonlinear Estimation
and Classification. New York, NY: Springer, 2003, pp.
149–171.
[25] J. Vesterstrom and R. Thomsen, “A comparative
study of differential evolution, particle swarm opti-
mization, and evolutionary algorithms on numerical
benchmark problems,” in Proc. IEEE Congr. Evolution-
ary Computation, Portland, OR, June 2004, vol. 2, pp.
1980–1987.
[26] M. N. Alam, B. Das, and V. Pant, “A comparative
study of metaheuristic optimization approaches for direc-
tional overcurrent relays coordination,” Electr. Power Syst.
Res., vol. 128, pp. 39–52, June 2015.
[27] S. M. Lucas, “Evolving finite state transducers: Some
initial explorations,” in Proc. European Conf. Genetic Pro-
gramming, Essex, Apr. 2003, pp. 130–141.
[28] J. Ilonen, J.-K. Kamarainen, and J. Lampinen, “Dif-
ferential evolution training algorithm for feed-forward
neural networks,” Neural Process. Lett., vol. 17, no. 1, pp.
93–105, Mar. 2003.
[29] R. Storn, “On the usage of differential evolu-
tion for function optimization,” in Proc. North American
Fuzzy Information Processing, Berkeley, CA, June 1996, pp.
519–523.
[30] T. Rogalsky, S. Kocabiyik, and R. W. Derksen,
“Differential evolution in aerodynamic optimization,”
Can. Aeronaut. Space J., vol. 46, no. 4, pp. 183–190, Apr.
2000.
[31] J. Cao, Z. Lin, and G.-B. Huang, “Self-adaptive
evolutionary extreme learning machine,” Neural Process.
Lett., vol. 36, no. 3, pp. 285–305, July 2012.
[32] Q.-Y. Zhu, A. K. Qin, P. N. Suganthan, and G.-
B. Huang, “Evolutionary extreme learning machine,”
Pattern Recog., vol. 38, no. 10, pp. 1759–1763, Oct.
2005.
[33] Y. Qu, C. Shang, W. Wu, and Q. Shen, “Evolution-
ary fuzzy extreme learning machine for mammographic
risk analysis,” Int. J. Fuzzy Syst., vol. 13, no. 4, pp. 282–
291, Dec. 2011.
[34] V. L. Huang, A. K. Qin, and P. N. Suganthan, “Self-
adaptive differential evolution algorithm for constrained
real-parameter optimization,” in Proc. IEEE Congr.
Evolutionary Computation, Vancouver, BC, Canada, July
2006, pp. 17–24.
[35] A. K. Qin, V. L. Huang, and P. N. Suganthan,
“Differential evolution algorithm with strategy ad-
aptation for global numerical optimization,” IEEE
Trans. Evol. Comput., vol. 13, no. 2, pp. 398–417,
May 2009.
[36] Y. Wang, Z. Cai, and Q. Zhang, “Differential evolu-
tion with composite trial vector generation strategies and
control parameters,” IEEE Trans. Evol. Comput., vol. 15,
no. 1, pp. 55–66, May 2011.
[37] R. A. Sarker, S. M. Elsayed, and T. Ray, “Differ-
ential evolution with dynamic parameters selection for
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE
optimization problems,” IEEE Trans. Evol. Comput., vol.
18, no. 5, pp. 689–707, May 2014.
[38] J. Zhang and A. C. Sanderson, “JADE: Adaptive dif-
ferential evolution with optional external archive,” IEEE
Trans. Evol. Comput., vol. 13, no. 5, pp. 945–958, May
2009.
[39] Y. Wang, Z. Cai, S. Member, Q. Zhang, and S.
Member, “Differential evolution with composite trial
vector generation strategies and control parameters,”
IEEE Trans. Evol. Comput., vol. 15, no. 1, pp. 55–66,
Mar. 2011.
[40] Z. Yang, K. Tang, and X. Yao, “Self-adaptive dif-
ferential evolution with neighborhood search,” in Proc.
IEEE Congr. Evolutionary Computation, Hong Kong, Chi-
na, June 2008, pp. 1110–1116.
[41] S. Das, A. Abraham, U. K. Chakraborty, and A.
Konar, “Differential evolution using a neighborhood-
based mutation operator,” IEEE Trans. Evol. Comput., vol.
13, no. 3, pp. 526–553, June 2009.
[42] D. W. Marquardt, “An algorithm for least-squares
estimation of nonlinear parameters,” J. Soc. Ind. Appl.
Math., vol. 11, no. 2, pp. 431–441, Aug. 1963.
[43] C. M. Bishop, Pattern Recognition and Machine Learn-
ing. New York, NY: Springer, 2006.
[44] J. Kennedy and R. Eberhart, “Particle swarm op-
timization,” in Proc. IEEE Int. Conf. Neural Networks,
Perth, Australia, Nov. 1995, vol. 4, pp. 1942–1948.
[45] R. Eberhart and J. Kennedy, “A new optimizer us-
ing particle swarm theory,” in Proc. 6th Int. Symp. Micro
Machine and Human Science, Nagoya, Japan, Oct. 1995, pp.
39–43.
[46] J.-H. Seo, C.-H. Im, C.-G. Heo, J.-K. Kim, H.-
K. Jung, and C.-G. Lee, “Multimodal function op-
timization based on particle swarm optimization,”
IEEE Trans. Magn., vol. 42, no. 4, pp. 1095–1098, Apr.
2006.
[47] Z. H. Zhan, J. Zhang, Y. Li, and Y. H. Shi, “Or-
thogonal learning particle swarm optimization,” IEEE
Trans. Evol. Comput., vol. 15, no. 6, pp. 832–847, Dec.
2011.
[48] C. Li, S. Yang, and T. T. Nguyen, “A self-learning
particle swarm optimizer for global optimization prob-
lems,” IEEE Trans. Syst., Man, Cybern., Syst. B, vol. 42,
no. 3, pp. 627–646, June 2012.
[49] D. Ciresan, U. Meier, and J. Schmidhuber,
“Multi-column deep neural networks for image clas-
sif ication,” in Proc. IEEE Int. Conf. Computer Vision
and Pattern Recognition, Providence, RI, June 2012, pp.
3642–3649.
[50] D. Keysers, T. Deselaers, C. Gollan, and H. Ney,
“Deformation models for image recognition,” IEEE
Trans. Pattern Anal. Mach. Intell., vol. 29, no. 8, pp. 1422–
1435, Nov. 2007.
[51] A. T. Visual, P. Y. Simard, D. Steinkraus, and J.
C. Platt, “Best practices for convolutional neural net-
works,” in Proc. IEEE Int. Conf. Document Analysis
and Recogntion, Edinburgh, Scotland, Aug. 2003, pp.
958–962.
[52] D. C. Ciresan, U. Meier, L. M. Gambardella, and J.
Schmidhuber, “Deep big simple neural nets for handwrit-
ten digit recognition,” Neural Comput., vol. 22, no. 12,
pp. 3207–3220, Dec. 2010.
[53] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long,
R. Girshick, S. Guadarrama, and T. Darrell, “Caffe:
Convolutional architecture for fast feature embedding,”
in Proc. 22nd ACM Int. Conf. Multimedia, Orlando, FL,
Nov. 2014, pp. 675–678.
[54] R. Girshick, “Fast R-CNN,” in Proc. IEEE Int.
Conf. Computer Vision, Santiago, Chile, Dec. 2015, pp.
1440–1448.
[55] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and
F.-F. Li, “Imagenet: A large-scale hierarchical image data-
base,” in Proc. IEEE Int. Conf. Computer Vision and Pattern
Recognition, Miami, FL, Nov. 2009, pp. 248–255.
[56] K. Simonyan and A. Zisserman, “Very deep convo-
lutional networks for large-scale image recognition, in
Proc. IEEE Int. Conf. Learning Representations, CO, June
2015, pp. 1150–1210.
47
 Haibo He
Department of Electrical, Computer and Biomedical
Engineering, University of Rhode Island, Kingston, RI, USA
Xiangnan Zhong
Department of Electrical Engineering,
University of North Texas, Denton, TX, USA
Learning Without External Reward
Abstract
I n the traditional reinforcement
learning paradigm, a reward signal is
applied to define the goal of the
task. Usually, the reward signal is a
“hand-crafted” numerical value or a
pre-defined function: it tells the agent
how good or bad a specific action is.
However, we believe there exist situa-
tions in which the environment cannot
directly provide such a reward signal to
the agent. Therefore, the question is
whether an agent can still learn without
the external reward signal or
not. To this end, this article
develops a self-learning ap -
proach which enables the
agent to adaptively develop an
internal reward signal based on
a given ultimate goal, without
requiring an explicit external
reward signal from the envi-
ronment. In this article, we aim
to convey the self-learning
idea in a broad sense, which
could be used in a wide range
of existing reinforcement
learning and adaptive dynamic
programming algorithms and architec-
tures. We describe the idealized forms of
this method mathematically, and also
demonstrate its effectiveness through a
triple-link inverted pendulum case study.
I. Introduction
When we first think about the nature of
learning, we probably start with the idea
Digital Object Identifier 10.1109/MCI.2018.2840727
Date of publication: 18 July 2018
48 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018
that we learn by interacting with the
environment [1]–[3]. For instance, when
we try to hold a conversation with oth-
ers, we need to decide what to say based
on the people we are talking to as well as
the conversational context. Over the past
several decades, many researchers have
explored computational approaches to
learn from active interactions with the
environment, such as reinforcement
learning (RL) and adaptive dynamic pro-
gramming (ADP). Imagine we hope to
train a monkey to learn the result of
image licensed by ingram publishing
“1+1”. At first, we present two cards with
number “1” on them to the monkey: If it
picks a card with the number “2” on it
from a box, we present a banana as a
reward. In this way, although the monkey
does not know the exact meaning of
math, it knows that a banana can be
given when it picks the appropriate card.
Therefore, the banana reward plays an
Corresponding Author: Haibo He (Email: haibohe@uri.edu)
important role in the learning process. In
general, the key element of RL is defined
by the reward signal, which is given by
the environment [1], [4]–[6]. In order to
achieve goals, the agent chooses a set of
actions that maximize the expected total
rewards it receives over time. Therefore,
RL achieves goals by defining the
interaction between an agent and its
environment in terms of states, actions,
and rewards [1], [7]. Recently, the
development of deep RL [8]–[10] has
attracted increasing attention, especially
for the level of intelligence it
has achieved.
So far, many RL/ADP
designs focus on how to calcu-
late and maximize the cumula-
tive rewards [11]–[15]. Usually,
it is assumed that the agent
knows what the immediate
reward is or how the immedi-
ate reward is computed as a
function of the actions and
states in which they are taken
[16]. There are several ap -
proaches in the literature to
define such a reward signal. For
instance, a typical approach is to use a
binary signal, e.g., using a “0” or “−1” to
represent “success” or “failure” of an
action [17], or a semi-binary reward sig-
nal, e.g., using “0, −0.4, −1” as a more
informative representation [18]. Another
way to define the reward signal is to use
a quadratic function based on the system
states and actions [19]–[22]. This type of
1556-603x/18©2018IEEE
reward signal is more adaptive and very
popular in system stabilization. Recently,
a goal representation heuristic dynamic
programming (GrHDP) method [23],
[24] has been proposed in the literature.
By introducing an internal reward signal,
this method can automatically and adap-
tively provide information to the agent
instead of hand-crafting. Generally, all
these existing approaches require the
environment to define the reward signal
with a priori knowledge or domain
expertise. This can be seen as an external
teacher or trainer who provides the
reward value or function for each action
performed by the agent. However, what
if this kind of teacher is unavailable or
he/she cannot provide such a direct
feedback to the agent for some reasons?
Can the agent learn by itself in this case
under the RL framework?
In this article, by considering the
relationship between the goal and the
system states and actions, we propose a
method that enables the agent to learn
only with the ultimate goal but no
explicit external reward signals. To this
end, the key contribution of this work is
the development of a self-learning
approach based on the ultimate goal,
rather than obtaining the external super-
vision (reward signals) directly from the
environment. We further develop the
computational formulation of this self-
learning idea based on a specific ADP
design, and validate its performance and
effectiveness based on a triple-link invert-
ed pendulum case study. We would like
to note that this article focuses on the
self-learning of an agent’s own goals, in
contrast to observational learning or
apprenticeship learning about other’s
goals such as in inverse RL [25].
II. The Key Idea:
Self-Learning Design
In contrast to the traditional RL/ADP
design, in which a specific reward signal
passes from the environment to the agent
to tell the effects (“good” or “bad”), our
proposed self-learning ADP method
enables the agent to establish the reward
signal itself, which is called the internal
reward signal in this article.The comparison
of the agent-environment interaction in
traditional and self-learning ADP design is
described in Fig. 1. We can observe that
instead of receiving the immediate reward
signal from the environment (Fig. 1(a)), the
agent in the self-learning ADP method,
estimates an internal reward s (t) to help
achieve the goal (Fig. 1(b)). Hence, the
communications between the environ-
ment and the agent at each time step are
only the states and actions, which is funda-
mentally different to the existing RL and
ADP methods.
Note that, in the traditional RL/ADP
design, the use of the reward signal is to
define the agent’s goal of a task. However,
in the self-learning ADP method, the
reward signal is unavailable in the interac-
tion. Since the reward signal reflects the
evaluation of an action’s effect, which is
always paired with the goal, the agent in
this learning process needs to learn what
the reward signal is according to the ulti-
mate goal. The effect of the action is
compared to the goal within a common
reference frame in order to assess the
achievement [26]. The agent then learns
how “good” or “bad” the action is at each
time step by itself via the guidance from
the ultimate goal. After that, based on the
estimated internal reward signal and the
system state, the agent generates the con-
trol action. This is to say, in order to
achieve the ultimate goal, instead of
learning to make the decision directly, the
agent needs first to learn what the best
reward signal is to represent the informa-
tion upon which to base a certain action.
More specifically, the interaction
between the agent and the environment
Traditional RL/ADP
State
x (t ) Agent
Reward
r (t ) Action
a (t )
Environment
(a)
Figure 1 The conceptual diagram of agent-environment interaction: (a) Traditional RL/ADP
design: the external reinforcement signal r (t) passing from the environment to the agent; (b)
Self-learning RL/ADP design: no external reinforcement signal needed during this process, and
the agent estimating an internal reward signal s (t) itself.
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE
happens sequentially, in discrete time
steps. At each time step t, the agent
selects an action a (t) according to the
representation of the environment x (t) .
In consequence, the agent finds itself in a
new state x (t + 1) and then estimates
the corresponding internal reward sig-
nal s (t + 1) at next time step. For exam-
ple, when we train a battery-charged
robot to collect trash, the robot decides
at each time step whether it should
move forward to search more trash or
find its way back to its battery charger.
Its decision is based on its position and
speed. As a consequence of its action and
state, the robot estimates the reward sig-
nal s (t) = f (x (t), a (t)) . Initially, the robot
randomly assigns a value s (0) since no
prior knowledge is available about what
to do. However, after trial-and-error
learning, we want the robot to learn how
to represent an action in a given state.
Let us stipulate that the agent’s goal is
to maximize the reward it estimates in
the long term. If the reward sequence
after time step t is as s (t + 1), s (t + 2),
s (t + 3), f, then the value function can
be described as
V (t) = s (t) + cs (t + 1) + c 2 s (t + 2)
+ c 3 s (t + 3) + g
= s (t) + cV (t + 1) (1)
where 0 1 c 1 1 is the discount factor.
The discount factor determines if an
immediate reward is more valuable than
the rewards received in the far future. If
c = 0, the value function is equal to the
Self-Learning RL/ADP
State Internal
Agent Reward
x (t ) s (t )
Action
a (t )
Environment
(b)
49

In contrast to the traditional RL/ADP design, in which
a specific reward signal passes from the environment
to the agent to tell the effects (“good” or “bad”), our
proposed self-learning ADP method enables the agent
to establish the reward signal itself, which is called the
internal reward signal in this article.
immediate reward and the agent’s goal
becomes to maximize the immediate re-
ward. This may reduce the possibility of
accessing future rewards. When c ap-
proaches 1, future rewards will increase
their influence in selecting actions, there-
fore the agent becomes more farsighted
[1]. We note here that s (t) is the estimated
internal reward signal, rather than the ex-
ternal reward signal as in the classic RL/
ADP literature [1], [12], [13], [18], [27].
In this way, the optimal value func-
tion can be defined as
V ) (t) = max {s ) (t) + cV ) (t + 1)}. (2)
a (t), s (t)
Here s ) (t) is the optimal internal re-
ward signal that the agent learns to rep-
resent how good or bad an action is in a
given state. Our objective is to ensure
this signal can guide the agent to achieve
the goal, i.e.,
s ) (t) = argmax {V ) (t)}.
s (t )
Then, according to Bellman’s opti-
mality principle [28], the control action
a (t) can be given as
a ) (t) = argmax {s ) (t) + cV ) (t + 1)}. (4)
a (t )
Note that since the internal reward
signal s (t) is estimated by the agent, it is
random at first during the training pro-
cess. However, after trial-and-error
learning, the agent can learn how to
represent the internal reward signal
based on the ultimate goal and update it
accordingly. In this way, the internal
reward signal is automatically decided
within the agent in this approach. In
Fig. 2, we use the neural network tech-
nique as an example to illustrate this
idea. In different applications, one can
choose any type of function approxima-
tors for implementation depending on
the scenario and/or design. The internal
reward signal estimation will be achieved
by goal network with inputs x (t), a (t)
and an output s (t). The value function
V (t) will be generated by the critic net-
work to adjust the updating process.
Furthermore, the action network will
(3) provide the control action. Therefore,
we can observe that instead of obtain-
ing an explicit external reward signal
directly from the environment, the agent
in the proposed method self-learns an
internal reward signal to guide its action
based on the ultimate goal. This clearly
distinguishes self-learning ADP from the
existing ADP methods: (i) No external
supervision is needed in this fully autono-
mous learning. The basic principle
becomes finding an internal reward func-
tion which makes the agent achieve the
ultimate goal. (ii) Communication burden
is reduced since the interaction between

V (t )

s (t )
Goal Network Critic Network Action Network

V (t )

x (t )
x (t )
s (t ) V (t ) a (t )
x (t )
a (t )
a (t )
s (t )

a (t ) a (t )

Figure 2 Neural network training process. The goal network takes the system state x (t) and control action a (t) as its input, and outputs an
estimated internal reinforcement signal s (t). The critic network similarly applies the neural network structure, but takes an additional input s (t)
and outputs the value function V (t) . Moreover, the action network uses a similar neural network structure with input x (t) and output a (t) .

50 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

the agent and the environment at each
time step only becomes the states and In order to achieve the ultimate goal, instead of learning
actions. This is important when the com- to make the decision directly, the agent needs first to
munication bandwidth or power resourc-
es are constrained.
learn what the best reward signal is to represent the
information upon which to base a certain action.
III. Learning Architecture
and Implementation
The key of implementing self-learning the goal network, we set the estimated where b c is the learning rate of the crit-
ADP design is to estimate Eqs. (2), (3), internal reward signal s (t) as one of the ic network. Chain backpropagation rule
2E c (t)
and (4). In our current research, we use inputs to the critic network. Therefore, can be applied to further calculate 2~c (t) :
three neural networks to design the goal the inputs of the critic network include
network, action network, and critic network, the state x (t), the action a (t), and the 2E c (t) 2E c (t) 2V (t)
= . (10)
as shown in Fig. 3. Here, we assume that internal reward signal s (t). The critic 2~ c (t) 2V (t) 2~ c (t)
the solution of the task exists and it is network aims to minimize the following
unique. The mathematical analysis of error function over time: C. Action Network
these three neural networks is provided At any time instant, action network pro-
as follows. e c (t) = cV (t) - [V (t - 1) - s (t - 1)] vides the control action a (t) for the sys-
tem based on the system states x (t) .
E c (t) = 1 e 2c (t) . (8)
A. Goal Network 2 Therefore, we define the error function
The goal network estimates the internal as e a (t) = V (t) - U c and E a (t) = 12 e 2a (t).
reward signal s (t) in Eq.(3) based on the Note that, although both V (t) and The weights updating rule can be
ultimate goal U c . Since the internal V (t - 1) depend on critic network’s defined as:
reward signal is a response of the exter- weights ~ c, we do not account for the
~ a (t) - b a c
2E a ( t)
nal environment, we set the inputs of dependence of V (t - 1) on weights ~ c ~ a (t + 1) = m (11)
2~ a (t)
the goal network as the state x (t) and when minimizing the error in Eq. (8)
action a (t). With these information, the [16]. Therefore, the updating scheme of where b a is the learning rate of the
goal network provides an estimation of the critic network can be defined as: action network. With the chain back-
the internal reward signal in order to propagation rule, we have:
optimize the value function V (t). There-
~ c (t) - b c c
2E c ( t)
m
2E a (t) 2E a (t) 2V (t) 2 a (t)
fore, the error can be described as: ~ c (t) = (9) = . (12)
2~ c (t) 2~ a (t) 2V (t) 2 a (t) 2~ a (t)
e g (t) = V (t) - U c
E g (t) = 1 e 2g (t) (5)
2
where U c is the ultimate goal. The value
Environment
of U c is critical in the design and it
could be variant in different applications. x (t + 1)
We apply backpropagation to train the
goal network and update the goal net- x (t ) a (t )
work weights ~ g (t) as following:
s (t )
~ g ( t) - b g c
2E g (t)
m (6)
Action a (t ) Goal
~ g (t + 1) =
2~ g (t) Network Network + +

where b g is a learning rate of the goal x (t ) Internal Reward Signal

network. Followed by the chain back-
Critic V (t ) V (t –1)
propagation rule, we further have γ
Network + –
2E g (t) 2E g (t) 2V (t) 2s (t)
= . (7) + U c(t )
2~ g (t) 2V (t) 2s (t) 2~ g (t)
– Ultimate Goal
Agent

B. Critic Network
The value function V (t) in Eq. (2) is
estimated by the critic network. To
closely connect the critic network with
Figure 3 Implementation architecture of self-learning adaptive dynamic programming
design: three neural networks are established as the action network, the critic network and
the goal network.

auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 51

 The algorithm for the proposed self- IV. Results and Discussions nonlinearities. Therefore, this benchmark
learning ADP approach is given in We consider a triple-link inverted pen- is frequently used to evaluate the perfor-
[Algorithm 1], where step 1 setups the dulum case here. As seen in Fig. 4, this mance of new control strategies. The
variables and ingredients of three neural pendulum includes three poles connect- system model and system parameters are
networks, and then steps 2-26 describe ed by three links. This model is highly identical as those in [18]. In this task, our
the online training process. unstable and exhibits non-negligible goal is to balance the inverted pendu-
lum under the following constraints: (1)
the cart track should be within 1.0 m to
both sides from the center point; and (2)
Algorithm 1 Self-learning adaptive dynamic programming. each link angle should be within the
range of [- 20c, 20c] with respect to the
1: a ! fa (x a, ~ a), control action selection; vertical axis. For these two conditions, if
fa: the action network; either one fails or both fail, we consider
x a: inputs of action network, x a = [x]; that the current controller fails to
~ a: weights in action network; accomplish the task.
a: control action; In our study, the triple-link inverted
s ! fg (x g, ~ g), internal reward signal choosing; pendulum is the environment and the
fg: the goal network; designed controller, which produces the
x g: input of goal network, x g = [x, a]; control action, is the agent. The control
~ g: weights in goal network; unit a (t) (in voltage) is converted into
s: internal reward signal; force by an analog amplifier (with gain
V ! fc (x c, ~ c), value function mapping; K s = 24.7125N/V ) to the DC servo
fc: the critic network; motor. Each link here only rotates in a
x c: input of critic network, x c = [x, a, s]; vertical plane. At each time step t, the
~ c: weights in critic network; controller receives an eight-dimension
V: value function; vector which represents the state, i.e., the
N a, N g, and N c: internal cycles of the action network, goal network, and critic net- position of the cart on the track (x), the
work, respectively; vertical angles of three links ^i 1, i 2, i 3h,
Ta, Tg, and Tc: internal training error thresholds for the action network, goal network, the cart velocity (xo ), the angular velocities
and critic network, respectively; of three links ^io 1, io 2, io 3h . On this basis,
2: for 1 to MaxRun do the controller chooses an action. Note that
3: Initialize ~ a (0), ~ g (0), ~ c (0); the sign and the number of the action
4: x (t) ! System (x (t - 1), a (t - 1)); // execute action and obtain current state denote direction and magnitude of the
5: // online training of goal network force, respectively. Then one time step
6: while ^E g (t) 2 Tg & cyc 1 N gh do later, as a result of the action, the con-
7: ~ g (t) = ~ g (t) + D~ g (t) via (6) and (7); // update the weights recursively troller receives a new state vector and
8: s (t) ! fg (x g (t), ~ g (t)); estimates the internal reward signal
9: e g (t) = V (t) - U c, E g (t) = 1 e 2g (t); according to the current state and the
2
10: end while produced action. Therefore, during this
11: // online training of critic network training process, there is no external
12: while ^E c (t) 2 Tc & cyc 1 N c h do reward signal transmitting from the envi-
13: ~ c (t) = ~ c (t) + D~ c (t) via (9) and (10); ronment to the agent.
14: s (t) ! fg (x g (t), ~ g (t)); V (t) ! fc (x c (t), ~ c (t)); To test the performance of the self-
15: e c (t) = cV (t) - [V (t - 1) - s (t - 1)], E c (t) = 1
2 e 2c (t); learning ADP approach, one hundred
16: end while runs were conducted. We set the initial
17: // online training of action network cart position at the center of the track
18: while ^E a (t) 2 Ta & cyc 1 N ah do and its velocity to be zero. For different
19: ~ a (t) = ~ a (t) + D~ a (t) via (11) and (12); runs, the initial values of three angles
20: a (t) ! fa (x a (t), ~ a (t)); s (t) ! fg (x g (t), ~ g (t)); V (t) ! fc (x c (t), ~ c (t)); and angular velocity of the triple links
21: e a (t) = V (t) - U c, E a (t) = 12 e 2a (t); were chosen randomly within the range
22: end while of [-1c, 1c] and [- 0.5, 0.5] rad /s, re -
23: ~ a (t + 1) = ~ a (t); spectively. The maximum number of tri-
24: ~ g (t + 1) = ~ g (t); als in each run was 3000. A run is
25: ~ c (t + 1) = ~ c (t); // update weights through each trial considered successful if the controller
26: end for can balance the triple-link inverted

52 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

pendulum within the assigned number
of trials. The structures of goal, critic,
and action networks established for this
case refer to Fig. 2 with the number of
neurons in each layer as 9-14-1, (i.e., the
neural network has 9 input nodes, 14
hidden nodes, and 1 output node),
10-16-1, and 8-14-1, respectively. Since
the optimal equilibrium of all states are
near zero, we define the mathematical
representation of the ultimate goal, in
this example, as U c = 0.
Our simulation demonstrated that
92% of the runs resulted in a successful
balance. The average number of trials
to success was 1071.6. Table 1 shows
the comparative results of our method
with respect to those reported in [18]
and [23].
Table 1 shows that the perfor-
mance of the proposed self-learning
ADP method is not as good as the
perfor mance of the existing ADP
methods with pre-defined reward sig-
nal. This is because the agent in our
approach needs to learn what the
reward signal is. However, the key ob-
servation from this research is that, the
learning process of the proposed meth-
od can be accomplished without the
explicit external reward directly from
the environment. Instead, this reward
will be automatically and adaptively
learned and developed by the goal net-
work according to the ultimate goal.
This is the key fundamental contribu-
tion of this article.
To further examine the performance
of the self-learning ADP method, a typ-
ical trajectory for each of the state vari-
ables for the task is shown in Fig. 5,
including (a) the position of the cart on
the track; (b)–(d) the vertical angle of
the 1st, 2nd and 3rd links of the pendu-
lum, respectively; (e) the velocity of the
cart; and (f )–(h) the angular velocity of
the 1st, 2nd and 3rd links of the pendu-
lum, respectively. We observe that the
cart position on the track and all the
joint angles of the links are balanced
within a small range of the balance
point. This indicates that the proposed
method can effectively control the sys-
tem to achieve desired performance
and the controller can estimate the
The learning process of the proposed method can be
accomplished without the explicit external reward
directly from the environment. Instead, this reward will
be automatically and adaptively learned and developed
by the goal network according to the ultimate goal.
effect of an action during the learning an explicit reward signal directly from
process automatically. the external environment, the agent
itself can learn an internal reward signal
V. Summary and Conclusion s (t) by the goal network according to
We have designed a self-learning meth- the ultimate goal U c and guide itself to
od without the explicit external reward accomplish the task. This also means in
signal directly given by the environ- our approach, only two interaction ele-
ment. Comparing with the traditional ments, state x (t) and action a (t), are
RL/ADP methods in the literature, the required at each time step during the
key contribution of our approach is that learning process. From simulations, we
we introduce a new goal network to observe that the success rate of the
automatically and adaptively develop an designed self-learning method is lower
internal reward signal based on the ulti- than that of the traditional ADP meth-
mate goal to facilitate the self-learning ods. This is because the agent in the
process. Therefore, instead of receiving proposed method needs to learn how to
State
Environment Agent
x (t )
θ3
s (t )
θ2
Internal
Reward Signal
θ1
Self-Learning
Force
a (t )
Cart
Figure 4 Triple-link inverted pendulum case and its interaction with the agent.
TAbLe 1 Comparison with existing ADP learning algorithms.
SeLF-LeArNiNg TrADiTiONAL gOAL rePreSeN-
ADP ADP [18] TATiON ADP [23]
SUCCESS RATE 92% 97% 99%
NUMBER OF TRAILS 1071.6 1194 571.4
NEED EXTERNAL REWARDS NO YES YES
auguSt 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE 53
 [8] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J.
Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A.
0.02 0.04 K. Fidjeland, G. Ostrovski, et al. “Human-level control
0.02 through deep reinforcement learning,” Nature, vol. 518,

θ1 (rad)
x (m)

no. 7540, pp. 529–533, Feb. 2015.

0 0
[9] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre,
–0.02 G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V.
–0.02 –0.04 Panneershelvam, M. Lanctot, et al. “Mastering the game
0 2,000 4,000 6,000 0 2,000 4,000 6,000 of go with deep neural networks and tree search,” Nature,
Time Step Time Step vol. 529, no. 7587, pp. 484–489, Jan. 2016.
[10] D. Silver, J. Schrittwieser, K. Simonyan, I. Antono-
(a) (b) glou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A.
0.02 0.01 Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, T. G. G.
van den Driessche, and D. Hassabis, “Mastering the game
θ2 (rad)

θ3 (rad)
of go without human knowledge,” Nature, vol. 550, no.
0 0 7676, pp. 354–359, Oct. 2017.
[11] D. P. Bertsekas, Dynamic Programming and Optimal
Control. Belmont, MA: Athena Scientific, 1995.
–0.02 –0.01 [12] P. J. Werbos, “Intelligence in the brain: A theory of
0 2,000 4,000 6,000 0 2,000 4,000 6,000
how it works and how to build it,” Neural Netw., vol. 22,
Time Step Time Step no. 3, pp. 200–212, Apr. 2009.
(c) (d) [13] P. J. Werbos, “ADP: The key direction for future
research in intelligent control and understanding brain
intelligence,” IEEE Trans. Syst., Man, Cybern. B, Cybern.,
0.2 1
θ1 (rad/s)

vol. 38, no. 4, pp. 898–900, Aug. 2008.

x (m/s)

[14] F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, “Rein-

0 0
forcement learning and feedback control: Using natural deci-
.

–0.2 –1 sion methods to design optimal adaptive controllers,” IEEE

.

Control Syst. Mag., vol. 32, no. 6, pp. 76–105, Dec. 2012.
0 2,000 4,000 6,000 0 2,000 4,000 6,000 [15] W. B. Powell, Approximate Dynamic Programming: Solv-
Time Step Time Step ing the Curses of Dimensionality. New York: Wiley, 2007.
[16] D.V. Prokhorov and D. C.Wunsch,“Adaptive critic designs,”
(e) (f) IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997–1007, Sept. 1997.
1 0.4 [17] F. Liu, J. Sun, J. Si, W. Guo, and S. Mei, “A bound-
edness result for the direct heuristic dynamic program-
θ2 (rad/s)

θ3 (rad/s)

0.2 ming,” Neural Netw., vol. 32, pp. 229–235, Aug. 2012.
0 0 [18] J. Si and Y.-T. Wang, “Online learning control by as-
sociation and reinforcement,” IEEE Trans. Neural Netw.,
–0.2
.

vol. 12, no. 2, pp. 264–276, Mar. 2001.

–1 –0.4
0 2,000 4,000 6,000 0 2,000 4,000 6,000
Time Step Time Step
(g) (h)
Figure 5 Typical trajectory of a successful trial on the triple-link inverted pendulum balancing
task during the learning process: (a) the position of the cart, (b) the vertical angle of the 1st
link joint to the cart, (c) the vertical angle of the 2nd link joint to the 1st link, (d) the vertical
angle of the 3rd link joint to the 2nd link, (e) the cart velocity, (f) the angular velocity of the
1st link joint to the cart, (g) the angular velocity of the 2nd link joint to the 1st link, (h) the
angular velocity of the 3rd link joint to the 2nd link.
represent the reward signals by himself References
[1] A. G. Barto, Reinforcement Learning: An Introduction.
based on the ultimate goal, rather than Cambridge, MA: MIT Press, 1998.
is given explicitly by an exter nal [2] R. A. Brooks, “Intelligence without reason,” in Proc.
teacher in the existing methods. The 12th Int. Joint Conf. Artificial Intelligence, Sydney, NSW,
Australia, Aug. 1991, pp. 569–595.
implications of self-learning with no [3] R. Pfeifer and C. Scheier, Understanding Intelligence.
external supervision available at each Cambridge, MA: MIT Press, 1999.
[4] F.-Y. Wang, H. Zhang, and D. Liu, “Adaptive dynam-
time step during the learning process is ic programming: An introduction,” IEEE Comput. Intell.
significant, resulting in potential far- Mag., vol. 4, no. 2, pp. 39–47, Apr. 2009.
[5] B. Bathellier, S. P. Tee, C. Hrovat, and S. Rumpel, “A mul-
reaching applications across a wide tiplicative reinforcement learning model capturing learning
range of fields. dynamics and interindividual variability in mice,” Proc. Natl.
Acad. Sci. USA, vol. 110, no. 49, pp. 19950–19955, Nov. 2013.
[6] P. W. Glimcher, “Understanding dopamine and re-
Acknowledgment inforcement learning: The dopamine reward prediction
error hypothesis,” Proc. Natl. Acad. Sci. USA, vol. 108, no.
This work was supported in part by the Suppl 3, pp. 15647–15654, Mar. 2011.
National Science Foundation (NSF) [7] D. Zhao, Z. Xia, and Q. Zhang, “Model-free optimal
control based intelligent cruise control with hardware-
under CMMI 1526835 and ECCS in-the-loop demonstration [research frontier],” IEEE
1731672. Comput. Intell. Mag., vol. 12, no. 2, pp. 56–69, Apr. 2017.
[19] A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, “Dis-
crete-time nonlinear HJB solution using approximate dy-
namic programming: Convergence proof,” IEEE Trans. Syst.,
Man, Cybern. B, vol. 38, no. 4, pp. 943–949, June 2008.
[20] F. L. Lewis and K. G. Vamvoudakis, “Reinforce-
ment learning for partially observable dynamic processes:
Adaptive dynamic programming using measured output
data,” IEEE Trans. Syst., Man, Cybern. A, vol. 41, no. 1,
pp. 14–25, Feb. 2011.
[21] D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, “Neu-
ral-network-based optimal control for a class of unknown
discrete-time nonlinear systems using globalized dual heu-
ristic programming,” IEEE Trans. Autom. Sci. Eng. (from
July 2004), vol. 9, no. 3, pp. 628–634, July 2012.
[22] X. Zhong, H. He, H. Zhang, and Z. Wang, “Optimal
control for unknown discrete-time nonlinear Markov
jump systems using adaptive dynamic programming,” IEEE
Trans. Neural Netw. Learn. Syst., vol. 25, no. 12, pp. 2141–
2155, Mar. 2014.
[23] H. He, Z. Ni, and J. Fu, “A three-network architec-
ture for on-line learning and optimization based on adap-
tive dynamic programming,” Neurocomputing, vol. 78, no.
1, pp. 3–13, Feb. 2012.
[24] Z. Ni, H. He, J. Wen, and X. Xu, “Goal represen-
tation heuristic dynamic programming on maze naviga-
tion,” IEEE Trans. Neural Netw. Learn. Syst., vol. 24, pp.
2038–2050, Dec. 2013.
[25] A. Y. Ng and S. J. Russell, “Algorithms for inverse
reinforcement learning,” in Proc. 17th Int. Conf. Machine
Learning, San Francisco, CA, July 2000, pp. 663–670.
[26] M. Rolf and M. Asada, “Where do goals come from?
A generic approach to autonomous goal-system develop-
ment,” arXiv Preprint, arXiv:1410.5557, Oct. 2014.
[27] X. Zhong, H. He, D. Wang, and Z. Ni, “Model-free
adaptive control for unknown nonlinear zero-sum dif-
ferential game,” IEEE Trans. Cybern., vol. 48, no. 5, pp.
1633–1646, May 2018.
[28] R. Bellman, Dynamic Programming. Princeton, NJ:
Princeton Univ. Press, 1957.

54 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

 Tom Young
School of Information and Electronics, Beijing Institute
of Technology, Beijing, China
Devamanyu Hazarika
School of Computing, National University of Singapore,
Singapore
Soujanya Poria
Temasek Laboratories, Nanyang Technological University,
Singapore
Erik Cambria
School of Computer Science and Engineering,
Nanyang Technological University, Singapore
Recent Trends in Deep Learning Based Natural Language Processing
Abstract
D eep learning methods employ
multiple processing layers to learn
hierarchical representations of
data, and have produced state-of-the-art
results in many domains. Recently, a
variety of model designs and methods
have blossomed in the context of natu-
ral language processing (NLP). In this
paper, we review significant deep
learning related models and
methods that have been
employed for numerous
NLP tasks and provide a
walk-through of their
evolution. We also sum-
mar ize, compare and
contrast the various models
and put forward a detailed
understanding of the past, present
and future of deep learning in NLP.
I. Introduction
Natural language processing (NLP) is a
theory-motivated range of computa-
tional techniques for the automatic
analysis and representation of human
language. NLP research has evolved
from the era of punch cards and batch
Digital Object Identifier 10.1109/MCI.2018.2840738
Date of publication: 18 July 2018
1556-603x/18©2018ieee
processing, in which the analysis of a
sentence could take up to 7 minutes, to
the era of Google and the likes of it, in
which millions of webpages can be
Image lIcensed by Ingram PublIshIng
processed in less than a second [1]. NLP
enables computers to perform a wide
range of natural language related tasks at
all levels, ranging from parsing and part-
of-speech (POS) tagging, to machine
translation and dialogue systems.
Deep learning architectures and al-
gorithms have already made impressive
advances in fields such as computer vision
and pattern recognition. Following this
trend, recent NLP research is now in-
Corresponding Author: Erik Cambria (cambria@ntu.edu.sg)
august 2018 | ieee Computational intelligenCe magazine
Review
Article
creasingly focusing on the use of new deep
learning methods (see Fig. 1). For decades,
machine learning approaches targeting
NLP problems have been based on shal-
low models (e.g., SVM and logistic regres-
sion) trained on very high dimensional and
sparse features. In the last few years, neural
networks based on dense vector represen-
tations have been producing superior
results on various NLP tasks.This
trend is sparked by the success
of word embeddings [2], [3]
and deep learning meth-
ods [4]. Deep learning
enables multi-level auto-
matic feature representa-
tion learning. In contrast,
traditional machine learning
based NLP systems liaise heavily
on hand-crafted features. Such hand-
crafted features are time-consuming and
often incomplete.
Collobert et al. [5] demonstrated that
a simple deep learning framework
outperforms most state-of-the-art ap-
proaches in several NLP tasks such as
named-entity recognition (NER), se-
mantic role labeling (SRL), and POS
tagging. Since then, numerous complex
deep learning based algorithms have
been proposed to solve difficult NLP
55
 70 ACL EACL
EMNLP NAACL
60
(%)
50
40
30
2012 2013 2014 2015
Year
Figure 1 Percentage of deep learning papers in ACL, EMNLP, EACL,
NAACL over the last 6 years (long papers).
tasks. We review major deep learning
related models and methods applied to
natural language tasks such as convolu-
tional neural networks (CNNs), recur-
rent neural networks (RNNs), and
recursive neural networks. We also dis-
cuss memory-augmenting strategies,
attention mechanisms and how unsu-
pervised models, reinforcement learning
methods and recently, deep generative
models have been employed for lan-
guage-related tasks.
To the best of our knowledge, this
work is the first of its type to compre-
hensively cover the most popular deep
learning methods in NLP research
today. The work by Goldberg [6] only
presented the basic principles for apply-
ing neural networks to NLP in a tuto-
rial manner. We believe this paper will
give readers a more comprehensive idea
of current practices in this domain. The
structure of the paper is as follows: Sec-
tion II introduces the concept of dis-
tributed representation, the basis of
sophisticated deep learning models;
next, Sections III, IV, and V discuss
popular models such as convolutional,
recurrent, and recursive neural net-
works, as well as their use in various
NLP tasks; following, Section VI lists
recent applications of reinforcement
learning in NLP and new developments
in unsupervised sentence representation
learning; later, Section VII illustrates the
recent trend of coupling deep learning
models with memory modules; finally,
Section VIII summarizes the perfor-
mance of a series of deep learning meth-
56 ieee Computational intelligenCe magazine | august 2018
King
(–) Man
(+) Woman
2016 2017 Queen
Figure 2 Distributional vectors exhibits compositionality.
ods on standard datasets about major
NLP topics.
II. Distributed Representation
Statistical NLP has emerged as the pri-
mary option for modeling complex nat-
ural language tasks. However, in its
beginning, it often used to suffer from
the notor ious curse of dimensionality
while learning joint probability func-
tions of language models. This led to the
motivation of learning distributed repre-
sentations of words existing in low-
dimensional space [7].
A. Word Embeddings
Distributional vectors or word embed-
dings (Fig. 2) essentially follow the dis-
tributional hypothesis, according to
which words with similar meanings tend
to occur in similar context. Thus, these
vectors try to capture the characteristics
of the neighbors of a word. The main
advantage of distributional vectors is that
they capture similarity between words.
Measuring similarity between vectors is
possible, using measures such as cosine
similarity. Word embeddings are often
used as the first data processing layer in a
deep learning model. Typically, word
embeddings are pre-trained by optimiz-
ing an auxiliary objective in a large
unlabeled corpus, such as predicting a
word based on its context [3], [8], where
the learned word vectors can capture
general syntactical and semantic infor-
mation. Thus, these embeddings have
proven to be efficient in capturing con-
text similarity, analogies and due to its
smaller dimensionality, are fast and effi-
cient in computing core NLP tasks.
Over the years, the models that cre-
ate such embeddings have been shallow
neural networks and there has not been
need for deep networks to create good
embeddings. However, deep learning
based NLP models invariably represent
their words, phrases and even sentences
using these embeddings. This is in fact a
major difference between traditional
word count based models and deep
learning based models. Word embed-
dings have been responsible for state-of-
the-art results in a wide range of NLP
tasks [9]–[12]. For example, Glorot et al.
[13] used embeddings along with
stacked denoising autoencoders for
domain adaptation in sentiment classifi-
cation and Hermann and Blunsom [14]
presented combinatory categorial auto-
encoders to learn the compositionality
of sentence. Their wide usage across the
recent literature shows their effectiveness
and importance in any deep learning
model performing a NLP task.
Distributed representations (embed-
dings) are mainly learned through con-
text. During 1990s, several research
developments [15] marked the founda-
tions of research in distributional seman-
tics. A more detailed summary of these
early trends is provided in [16], [17].
Later developments were adaptations of
these early works, which led to creation
of topic models like latent Dirichlet
allocation [18] and language models [7].
These works laid out the foundations of
representation learning.
 In 2003, Bengio et al. [7] proposed a
neural language model which learned dis- i th Output = P (wt = i | Context)
tributed representations for words (Fig. 3).
Authors argued that these word represen- Softmax ..... .....
tations, once compiled into sentence rep- Classification
resentations using joint probability of word
sequences, achieved an exponential num-
Tanh Activation
ber of semantically neighboring sentences.
This, in turn, helped in generalization Concatenation .....

since unseen sentences could now gather

higher confidence if word sequences with
similar words (in respect to nearby word
representation) were already seen.
Collobert and Weston [19] was the
first work to show the utility of pre-
trained word embeddings. The authors
proposed a neural network architecture
that forms the foundation to many cur-
rent approaches. The work also estab-
lishes word embeddings as a useful tool
for NLP tasks. However, the immense
popularization of word embeddings was
arguably due to [3], who proposed the
continuous bag-of-words (CBOW) and
skip-gram models to efficiently con-
struct high-quality distributed vector
representations. Propelling their popu-
larity was the unexpected side effect of
the vectors exhibiting compositionality,
i.e., adding two word vectors results in a
vector that is a semantic composite of
the individual words, e.g., ‘man’ + ‘royal’
= ‘king’. The theoretical justification for
this behavior was recently given by Git-
C (wt – n +1) C (wt – 1)
..... .....
Table Look-Up
Using Matrix C
Word Index wt – n +1 Word Index wt – 1
Figure 3 Representation of the Neural Language Model proposed by Bengio et al. [7]. C(i) is
the ith word embedding.
the CBOW and skip-gram models. respectively. Each word from the vocab-
CBOW computes the conditional proba- ulary is finally represented as two
bility of a target word given the context learned vectors v c and v w, correspond-
words surrounding it across a window of ing to context and target word represen-
size k. On the other hand, the skip-gram tations, respectively. Thus, kth word in
model does the exact opposite of the the vocabulary will have
CBOW model, by predicting the sur-
v c = W (k,.) and v w = Wl(.,k) . (1)
rounding context words given the central
target word. The context words are Overall, for any word wi with given
assumed to be located symmetrically to context word c as input,
the target words within a distance equal to
the window size in both directions. In P` wi j = yi = V
e ui (2)
c
unsupervised settings, the word embed- / eu i

tens et al. [20], which stated that com-
positionality is seen only when certain
assumptions are held, e.g., the assump-
tion that words need to be uniformly
distributed in the embedding space.
Pennington et al. [21] is another
famous word embedding method which
is essentially a “count-based” model. Here,
the word co-occurrence count matrix is
i= 1
ding dimension is determined by the
T
accuracy of prediction. As the embedding where, u i = v .v c . wi
dimension increases, the accuracy of pre- The parameters i = {Vw, Vc} are
diction also increases until it converges at learned by defining the objective func-
some point, which is considered the optimal tion as the log-likelihood and finding its
embedding dimension as it is the shortest gradient as
without compromising accuracy.
Let us consider a simplified version l (i) = / log ` P ` w jj (3)
of the CBOW model where only one w ! Vocabulary c

preprocessed by normalizing the counts
and log-smoothing them. This matrix is
then factorized to get lower dimensional
representations which is done by minimiz-
ing a “reconstruction loss”.
Below, we provide a brief description
of the word2vec method proposed by
Mikolov et al. [3].
B. Word2vec
Word embeddings were revolutionized
by Mikolov et al. [3], [8] who proposed
word is considered in the context.
This essentially replicates a bigram lan-
guage model.
The CBOW model is a simple fully
connected neural network with one
hidden layer. The input layer, which
takes the one-hot vector of context
word has V neurons while the hidden
layer has N neurons. The output layer is
softmax of all words in the vocabulary.
The layers are connected by weight
matrix W ! R V # N and Wl ! R H #V ,
= Vc ` 1 - P ` w jj .
2l (i)
(4)
2Vw c
In the general CBOW model, all the
one-hot vectors of context words are
taken as input simultaneously, i.e,
h = W T (x 1 + x 2 + g + x c). (5)
One limitation of individual word
embeddings is their inability to represent
phrases, where the combination of two
or more words (e.g., idioms like “hot

august 2018 | ieee Computational intelligenCe magazine 57

potato” or named entities such as “Bos-
ton Globe”) does not represent the
combination of meanings of individual
words. One solution to this problem, as
explored by Mikolov et al. [3], is to
identify such phrases based on word
co-occurrence and train embeddings for
them separately. More recent methods
have explored directly learning n-gram
embeddings from unlabeled data [22].
Another limitation comes from
learning embeddings based only on a
small window of surrounding words,
sometimes words such as good and bad
share almost the same embedding [23],
which is problematic if used in tasks
such as sentiment analysis [24]. At times
these embeddings cluster semantically-
similar words which have opposing sen-
timent polarities. This leads the down-
stream model used for the sentiment
analysis task to be unable to identify this
contrasting polarities leading to poor
performance. Tang et al. [25] addresses
this problem by proposing sentiment
specific word embedding (SSWE).
Authors incorporate the supervised sen-
timent polarity of text in their loss func-
tions while learning the embeddings.
A general caveat for word embeddings
is that they are highly dependent on the
applications in which it is used. Labutov
and Lipson [26] proposed task specific
embeddings which retrain the word
embeddings to align them in the current
task space. This is very important as train-
ing embeddings from scratch requires large
amount of time and resource. Mikolov et
al. [8] tried to address this issue by propos-
ing negative sampling which is frequency-
based sampling of negative terms while
training the word2vec model.
Traditional word embedding algo-
rithms assign a distinct vector to each
TABLe 1 Frameworks providing embedding tools and methods.
FrAmework LAnguAge
S-SPACE JAvA
SEMANTiCvECToRS JAvA
gENSiM PYTHoN
PYDSM PYTHoN
DiSSECT PYTHoN
fASTTExT PYTHoN
58 ieee Computational intelligenCe magazine | august 2018
word. This makes them unable to
account for polysemy. In a recent work,
Upadhyay et al. [27] provided an inno-
vative way to address this deficit. The
authors leveraged multilingual parallel
data to learn multi-sense word embed-
dings. For example, the English word
bank, when translated to French provides
two different words: banc and banque
representing financial and geographical
meanings, respectively. Such multilingual
distributional information helped them
in accounting for polysemy.
Table 1 provides a directory of existing
frameworks that are frequently used for
creating embeddings which are further
incorporated into deep learning models.
C. Character Embeddings
Word embeddings are able to capture
syntactic and semantic information, yet
for tasks such as POS-tagging and NER,
intra-word morphological and shape
information can also be very useful. Gen-
erally speaking, building natural language
understanding systems at the character
level has attracted certain research atten-
tion [28]–[31]. Better results on morpho-
logically rich languages are reported in
certain NLP tasks. Santos and Guimaraes
[30] applied character-level representa-
tions, along with word embeddings for
NER, achieving state-of-the-art results in
Portuguese and Spanish corpora. Kim et
al. [28] showed positive results on build-
ing a neural language model using only
character embeddings. Ma et al. [32]
exploited several embeddings, including
character trigrams, to incorporate proto-
typical and hierarchical information for
learning pre-trained label embeddings in
the context of NER.
A common phenomenon for lan-
guages with large vocabularies is the
urL
HTTPS://giTHuB.CoM/fozziETHEBEAT/S-SPACE
HTTPS://giTHuB.CoM/SEMANTiCvECToRS/
HTTPS://RADiMREHuREk.CoM/gENSiM/
HTTPS://giTHuB.CoM/JiMMYCALLiN/PYDSM
HTTP://CLiC.CiMEC.uNiTN.iT/CoMPoSES/TooLkiT/
HTTPS://fASTTExT.CC/
unknown word issue or out-of-vocabu-
lar y word (OOV) issue. Character
embeddings naturally deal with it since
each word is considered as no more than
a composition of individual letters. In
languages where text is not composed of
separated words but individual characters
and the semantic meaning of words map
to its compositional characters (such as
Chinese), building systems at the charac-
ter level is a natural choice to avoid word
segmentation [33]. Thus, works employ-
ing deep learning applications on such
languages tend to prefer character
embeddings over word vectors [34]. For
example, Peng et al. [35] proved that rad-
ical-level processing could greatly
improve sentiment classification perfor-
mance. In particular, the authors pro-
posed two types of Chinese radical-based
hierarchical embeddings, which incorpo-
rate not only semantics at radical and
character level, but also sentiment infor-
mation. Bojanowski et al. [36] also tried
to improve the representation of words
by using character-level information in
morphologically-rich languages. They
approached the skip-gram method by
representing words as bag-of-characters
n-grams. Their work thus had the effec-
tiveness of the skip-gram model along
with addressing some persistent issues of
word embeddings. The method was also
fast, which allowed training models on
large corpora quickly. Popularly known
as FastText, such a method stands out
over previous methods in terms of speed,
scalability, and effectiveness.
Apart from character embeddings,
other approaches have been proposed for
OOV handling. Herbelot and Baroni [37]
provided OOV handling on-the-fly by
initializing the unknown words as the
sum of the context words and refining
these words with a high learning rate.
However, their approach is yet to be
tested on typical NLP tasks. Pinter et al.
[38] provided an interesting approach of
training a character-based model to recre-
ate pre-trained embeddings. This allowed
them to learn a compositional mapping
from character to word embedding, thus
tackling the OOV problem.
Despite the ever growing popularity
of distributional vectors, recent discussions
on their relevance in the long run have
cropped up. For example, Lucy and
Gauthier [39] has recently tried to evalu-
ate how well the word vectors capture
the necessary facets of conceptual mean-
ing. The authors have discovered severe
limitations in perceptual understanding
of the concepts behind the words, which
cannot be inferred from distributional
semantics alone. A possible direction for
mitigating these deficiencies will be
grounded learning, which has been gain-
ing popularity in this research domain.
III. Convolutional Neural Networks
Following the popularization of word
embeddings and its ability to represent
words in a distributed space, the need
arose for an effective feature function
that extracts higher-level features from
constituting words or n-grams. These
abstract features would then be used for
numerous NLP tasks such as sentiment
analysis, summarization, machine trans-
lation, and question answering (QA).
CNNs turned out to be the natural
choice given their effectiveness in com-
puter vision tasks [40]–[42].
The use of CNNs for sentence model-
ing traces back to Collobert and Weston
[19].This work used multi-task learning to
output multiple predictions for NLP tasks
such as POS tags, chunks, named-entity
tags, semantic roles, semantically-similar
words and a language model. A look-up
table was used to transform each word
into a vector of user-defined dimensions.
Thus, an input sequence {s 1, s 2, ..., s n} of
n words was transformed into a series of
vectors {w s , w s 2, ..., w s n} by applying the
1
sentence to create an informative latent
semantic representation of the sentence
for downstream tasks. This application
was pioneered by Collobert et al. [5],
Kalchbrenner et al. [43], Kim [44],
which led to a huge proliferation of
CNN-based networks in the succeeding
literature. Below, we describe the work-
ing of a simple CNN-based sentence
modeling network:
A. Basic CNN
1) sentence modeling
For each sentence, let w i ! R d repre-
sent the word embedding for the i th
word in the sentence, where d is the
dimension of the word embedding.
Given that a sentence has n words, the
sentence can now be represented as an
embedding matrix W ! R n # d . Fig. 5
depicts such a sentence as an input to
the CNN framework.
Softmax Classification
Fully Connected Layer
Max-Pool
over Time
Convolution
Layer
Let w i:i + j refer to the concatenation
of vectors w i, w i + 1, ..., w j . Convolution
is performed on this input embedding
layer. It involves a filter k ! R hd which is
applied to a window of h words to pro-
duce a new feature. For example, a fea-
ture c i is generated using the window of
words w i:i + h - 1 by
c i = f (w i:i + h - 1 .k T + b) (6)
where b ! R is the bias term and f is a
non-linear activation function, for exam-
ple the hyperbolic tangent. The filter k is
applied to all possible windows using the
same weights to create the feature map.
c = [c 1, c 2, ..., c n - h + 1] . (7)
In a CNN, a number of convolu-
tional filters, also called kernels (typically
hundreds), of different widths slide over
the entire word embedding matrix.
Each kernel extracts a specific pattern of
n-gram. A convolution layer is usually

look-up table to each of its words (Fig. 4).

This can be thought of as a primitive Lookup Table
word embedding method whose weights
were learned in the training of the net- Feature k
work. In [5], Collobert extended his
work to propose a general CNN-based Feature 1
framework to solve a plethora of NLP
tasks. Both these works triggered a huge
popularization of CNNs amongst NLP
Input
researchers. Given that CNNs had
Sentence
already shown their mettle for computer
vision tasks, it was easier for people to wo w1 wN –1
believe in their performance.
CNNs have the ability to extract Figure 4 CNN framework used to perform word-wise class prediction proposed by Collobert
salient n-gram features from the input and Weston [19].

august 2018 | ieee Computational intelligenCe magazine 59


Textual Processing
I
Love
This
Movie
Very
Much
Input Embedding Convolution with
Sequence of Multiple Filter
Sentence Widths and Multiple
Feature Maps
Figure 5 CNN modeling on text.
followed by a max-pooling strategy,
ct = max {c}, which subsamples the
input typically by applying a max opera-
tion on each filter. This strategy has two
primary reasons.
Firstly, max pooling provides a fixed-
length output which is generally required
for classification. Thus, regardless the size
of the filters, max pooling always maps the
input to a fixed dimension of outputs.
Secondly, it reduces the output’s dimen-
sionality while keeping the most salient
n-gram features across the whole sen-
tence. This is done in a translation invari-
ant manner where each filter is now able
to extract a particular feature (e.g., nega-
tions) from anywhere in the sentence and
add it to the final sentence representation.
The word embeddings can be initial-
ized randomly or pre-trained on a large
unlabeled corpora (as in Section II). The
latter option is sometimes found benefi-
cial to performance, especially when the
amount of labeled data is limited [44].
This combination of convolution layer
followed by max pooling is often stacked
to create deep CNN networks. These
sequential convolutions help in improved
mining of the sentence to grasp a truly
abstract representation comprising rich
semantic information. The kernels through
deeper convolutions cover a larger part of
the sentence until finally covering it fully
and creating a global summarization of
the sentence features.
2) Window approach
The above-mentioned architecture allows
for modeling of complete sentences into
sentence representations. However, many
60 ieee Computational intelligenCe magazine | august 2018
Softmax
Max-Pooling Dense Layer
Over Time
NLP tasks, such as NER, POS tagging,
and SRL, require word-based predictions.
To adapt CNNs for such tasks, a window
approach is used, which assumes that the
tag of a word primarily depends on its
neighboring words. For each word, thus, a
fixed-size window surrounding itself is
assumed and the sub-sentence ranging
within the window is considered. A
standalone CNN is applied to this sub-
sentence as explained earlier and predic-
tions are attributed to the word in the
center of the window. Following this
approach, Poria et al. [45] employed a
multi-level deep CNN to tag each word
in a sentence as a possible aspect or non-
aspect. Coupled with a set of linguistic
patterns, their ensemble classifier managed
to perform well in aspect detection.
The ultimate goal of word-level clas-
sification is generally to assign a sequence
of labels to the entire sentence. In such
cases, structured prediction techniques
such as conditional random field (CRF)
are sometimes employed to better cap-
ture dependencies between adjacent class
labels and finally generate cohesive label
sequence giving maximum score to the
whole sentence [46].
To get a larger contextual range, the
classic window approach is often coupled
with a time-delay neural network
(TDNN) [47]. Here, convolutions are
performed across all windows throughout
the sequence. These convolutions are
generally constrained by defining a kernel
having a certain width. Thus, while the
classic window approach only considers
the words in the window around the
word to be labeled, TDNN considers all
windows of words in the sentence at the
same time. At times, TDNN layers are
also stacked like CNN architectures to
extract local features in lower layers and
global features in higher layers [5].
B. Applications
In this section, we present some of the
crucial works that employed CNNs on
NLP tasks to set state-of-the-art bench-
marks in their respective times.
Kim [44] explored using the above
architecture for a variety of sentence
classification tasks, including sentiment,
subjectivity and question type classifica-
tion, showing competitive results. This
work was quickly adapted by researchers
given its simple yet effective network.
After training for a specific task, the ran-
domly initialized convolutional kernels
became specific n-gram feature detec-
tors that were useful for that target task.
This simple network, however, had
many shortcomings with the CNN’s
inability to model long distance depen-
dencies standing as the main issue.
This issue was partly handled by Kal-
chbrenner et al. [43], who published a
prominent paper where they proposed a
dynamic convolutional neural network
(DCNN) for semantic modeling of sen-
tences. They proposed dynamic k -max
pooling strategy which, given a sequence
p selects the k most active features. The
selection preserved the order of the fea-
tures but was insensitive to their specific
positions (Fig. 6). Built on the concept of
TDNN, they added this dynamic k-max
pooling strategy to create a sentence
model. This combination allowed filters
with small width to span across a long
range within the input sentence, thus
accumulating crucial information across
the sentence. In the induced subgraph
(Fig. 6), higher order features had highly
variable ranges that could be either short
and focused or global and long as the
input sentence. They applied their model
on multiple tasks, including sentiment
prediction and question type classifica-
tion, achieving significant results. Overall,
this work commented on the range of
individual kernels while trying to model
contextual semantics and proposed a way
to extend their reach.
 Tasks involving sentiment analysis also
require effective extraction of aspects
along with their sentiment polarities [48].
Ruder et al. [49] applied a CNN where
in the input they concatenated an aspect
vector with the word embeddings to get
competitive results. CNN modeling
approach varies amongst different length
of texts. Such differences were seen in
many works like Johnson and Zhang
[22], where performance on longer text
worked well as opposed to shorter texts.
Wang et al. [50] proposed the usage of
CNN for modeling representations of
short texts, which suffer from the lack of
available context and, thus, require extra
efforts to create meaningful representa-
tions. The authors proposed semantic
clustering which introduced multi-scale
semantic units to be used as external
knowledge for the short texts. CNN was
used to combine these units and form
the overall representation. In fact, this
requirement of high context information
can be thought of as a caveat for CNN-
based models. NLP tasks involving
microtexts using CNN-based methods
often require the need of additional
information and external knowledge to
perform as per expectations. This fact was
also observed in [51], where authors per-
formed sarcasm detection in Twitter texts
using a CNN network. Auxiliary support,
in the form of pre-trained networks
trained on emotion, sentiment and per-
sonality datasets was used to achieve
state-of-the-art performance.
CNNs have also been extensively
used in other tasks. For example, Denil
et al. [52] applied DCNN to map mean-
ings of words that constitute a sentence
to that of documents for summarization.
The DCNN learned convolution filters
at both the sentence and document
level, hierarchically learning to capture
and compose low-level lexical features
into high-level semantic concepts. The
focal point of this work was the intro-
duction of a novel visualization tech-
nique of the learned representations,
which provided insights not only in the
learning process but also for automatic
summarization of texts.
CNN models are also suitable for cer-
tain NLP tasks that require semantic
x1 x2 ........ xn
Figure 6 Representation of a DCNN subgraph. With dynamic pooling, a filter with small width
at the higher layers can relate phrases far apart in the input sentence. DCNN was proposed by
kalchbrenner et al. [43].
matching [53]. A similar model to the
above CNN architecture (Fig. 5) was
explored in [54] for information retrieval.
The CNN was used for projecting que-
ries and documents to a fixed-dimension
semantic space, where cosine similarity
between the query and documents was
used for ranking documents regarding a
specific query. The model attempted to
extract rich contextual structures in a
query or a document by considering a
temporal context window in a word
sequence. This captured the contextual
features at the word n-gram level. The
salient word n-grams is then discovered
by the convolution and max-pooling lay-
ers which are then aggregated to form
the overall sentence vector.
In the domain of QA, Yih et al.
[55] proposed to measure the semantic
similarity between a question and en-
tries in a knowledge base (KB) to
determine what supporting fact in the
KB to look for when answering a
question. To create semantic repre-
sentations, a CNN similar to the one
in Fig. 5 was used. Unlike the classifica-
tion setting, the supervision signal came
from positive or negative text pairs (e.g.,
query-document), instead of class
labels. Subsequently, Dong et al. [56]
introduced a multi-column CNN
(MCCNN) to analyze and under-
stand questions from multiple aspects
and create their representations.
MCCNN used multiple column net-
works to extract information from as-
pects comprising answer types and
august 2018 | ieee Computational intelligenCe magazine
x1 x2 ........ xn
context from the input questions. By
representing entities and relations in the
KB with low-dimensional vectors, they
used question-answer pairs to train the
CNN model so as to rank candidate
answers. Severyn and Moschitti [57]
also used CNN network to model opti-
mal representations of question and
answer sentences. They proposed
additional features in the embeddings in
the form of relational information giv-
en by matching words between the
question and answer pair. These param-
eters were tuned by the network. This
simple network was able to pro-
duce comparable results to state-of-
the-art methods.
CNNs are wired in a way to capture
the most important information in a sen-
tence. Traditional max-pooling strategies
perform this in a translation invariant
form. However, this often misses valuable
information present in multiple facts
within the sentence. To overcome this
loss of information for multiple-event
modeling, Chen et al. [58] proposed a
modified pooling strategy: dynamic
multi-pooling CNN (DMCNN). This
strategy used a novel dynamic multi-
pooling layer that, as the name suggests,
incorporates event triggers and argu-
ments to reserve more crucial informa-
tion from the pooling layer.
CNNs inherently provide certain
required features like local connectivity,
weight sharing, and pooling. This puts
forward some degree of invariance
which is highly desired in many tasks.
61
Speech recognition also requires such
invariance and, thus, Abdel-Hamid et al.
[59] used a hybrid CNN-HMM model
which provided invariance to frequency
shifts along the frequency axis. This vari-
ability is often found in speech signals
due to speaker differences. They also
perfor med limited weight shar ing
which led to a smaller number of pool-
ing parameters, resulting in lower com-
putational complexity. Palaz et al. [60]
performed extensive analysis of CNN-
based speech recognition systems when
given raw speech as input. They showed
the ability of CNNs to directly model
the relationship between raw input and
phones, creating a robust automatic
speech recognition system.
Tasks like machine translation re -
quire perseverance of sequential infor-
mation and long-ter m dependency.
Thus, structurally they are not well suit-
ed for CNN networks, which lack these
features. Nevertheless, Tu et al. [61] ad-
dressed this task by considering both the
semantic similarity of the translation
pair and their respective contexts. Al-
though this method did not address the
sequence perseverance problem, it al-
lowed them to get competitive results
amongst other benchmarks.
Overall, CNNs are extremely effec-
tive in mining semantic clues in contex-
tual windows. However, they are very
data heavy models. They include a large
number of trainable parameters which
require huge training data. This poses a
problem when scarcity of data arises.
Another persistent issue with CNNs is
their inability to model long-distance
contextual information and preserving
sequential order in their representations
[43], [61]. Although CNNs prove an
effective way to capture n-gram features,
which is approximately sufficient in cer-
tain sentence classification tasks, their
sensitivity to word order is restricted
locally and long-term dependencies are
typically ignored.
IV. Recurrent Neural Networks
RNNs [62] use the idea of processing
sequential information. The term “recur-
rent” applies as they perform the same
computation over each token of the
62 ieee Computational intelligenCe magazine | august 2018
sequence and each step is dependent
on the previous computations and results.
Generally, a fixed-size vector is pro-
duced to represent a sequence by feeding
tokens one by one to a recurrent unit. In
a way, RNNs have “memory” over pre-
vious computations and use this infor-
mation in current processing.This template
is naturally suited for many NLP tasks
such as language modeling [2], [63],
[64], machine translation [65]–[67], speech
recognition [68]–[71], image caption-
ing [72]. This made RNNs increas-
ingly popular for NLP applications in
recent years.
A. Need for Recurrent Networks
In this section, we analyze the funda-
mental properties that favored the popu-
larization of RNNs in a multitude of
NLP tasks. Given that an RNN per-
forms sequential processing by modeling
units in sequence, it has the ability to
capture the inherent sequential nature
present in language, where units are
characters, words or even sentences.
Words in a language develop their
semantical meaning based on the previ-
ous words in the sentence. A simple
example stating this would be the differ-
ence in meaning between “dog” and
“hot dog”. RNNs are tailor-made for
modeling such context dependencies in
language and similar sequence modeling
tasks, which turned to be a strong moti-
vation for researchers to use RNNs over
CNNs in these areas.
Another factor aiding RNN’s suit-
ability for sequence modeling tasks lies
in its ability to model variable length of
text, including very long sentences,
paragraphs and even documents [73].
Unlike CNNs, RNNs have flexible
computational steps that provide better
modeling capability and create the pos-
sibility to capture unbounded context.
This ability became one of the selling
points of major works using RNNs [74].
Many NLP tasks require semantic
modeling over the whole sentence. This
involves creating a gist of the sentence
in a fixed dimensional hyperspace.
RNN’s ability to summarize sentences
led to their increased usage for tasks like
machine translation [75] where the
whole sentence is summarized to a fixed
vector and then mapped back to the
variable-length target sequence.
RNN also provides the network sup-
port to perform time distributed joint
processing. Most of the sequence label-
ing tasks like POS tagging [31] come
under this domain. More specific use
cases include applications such as multi-
label text categorization [76], multi-
modal sentiment analysis [77]–[79], and
subjectivity detection [80].
The above points enlist some of the
focal reasons that motivated researchers
to opt for RNNs. However, it would be
gravely wrong to make conclusions on
the superiority of RNNs over other
deep networks. Recently, several works
provided contrasting evidence on the
superiority of CNNs over RNNs. Even
in RNN-suited tasks like language
modeling, CNNs achieved competitive
performance over RNNs [81]. Both
CNNs and RNNs have different objec-
tives when modeling a sentence. While
RNNs try to create a composition of an
arbitrarily long sentence along with
unbounded context, CNNs try to
extract the most important n-grams.
Yin et al. [82] provided interesting
insights on the comparative performance
between RNNs and CNNs. After testing
on multiple NLP tasks that included sen-
timent classification, QA, and POS tag-
ging, they concluded that there is no
clear winner: the performance of each
network depends on the global seman-
tics required by the task itself.
Below, we discuss some of the RNN
models extensively used in the literature.
B. RNN Models
1) simple Rnn
In the context of NLP, RNNs are pri-
marily based on Elman network [62]
and they are originally three-layer net-
works. Fig. 7 illustrates a more general
RNN which is unfolded across time to
accommodate a whole sequence. In the
figure, x t is taken as the input to the
network at time step t and s t repre-
sents the hidden state at the same time
step. Calculation of s t is based as per
the equation:
 s t = f (Ux t + Ws t - 1) . (8)
o ot −1
Thus, s t is calculated based on the
current input and the previous time step’s W W
hidden state. The function f is taken to V h
be a non-linear transformation such as Unfold
U
tanh, ReLU and U, V, W account for
x
weights that are shared across time. In the
context of NLP, x t typically comprises of
Figure 7 Simple RNN network.
one-hot encodings or embeddings. At
times, they can also be abstract represen-
tations of textual content. q t illustrates nation of these three gates as per the
the output of the network which is also equations below:
often subjected to non-linearity, espe-
x=; E
cially when the network contains further ht - 1
layers downstream. xt
The hidden state of the RNN is typ- ft = v (W f . x + b f)
ically considered to be its most crucial
it = v (W i . x + b i)
element. As stated before, it can be con-
sidered as the network’s memory ele- ot = v (W o . x + b o)
ment that accumulates information from c t = ft 9 c t - 1 + i t 9 tanh (W c . x + b c)
other time steps. In practice, however, (13)
these simple RNN networks suffer from
h t = o t 9 tanh (c t) .
the infamous vanishing gradient problem,
which makes it really hard to learn and
tune the parameters of the earlier layers 3) gated Recurrent units
in the network. Another gated RNN variant called GRU
This limitation was overcome by [75] (Fig. 8) of lesser complexity was
various networks such as long short- invented with empirically similar perfor-
term memory (LSTM), gated recur- mances to LSTM in most tasks. GRU
rent units (GRUs), and residual net- comprises of two gates, reset gate and
works (ResNets), where the first two update gate, and handles the flow of infor-
are the most used RNN variants in mation like an LSTM without a memory
NLP applications. unit. Thus, it exposes the whole hidden
content without any control. GRU can be
2) long short-term memory a more efficient RNN than LSTM. The
LSTM [83], [84] (Fig. 8) has additional working of GRU is as follows:
“forget” gates over the simple RNN,
which allows the error to back-propa- z= v (U z . x t + W z . h t - 1)
gate through an unlimited number of r= v (U r . x t + W r . h t - 1)
time steps. Consisting of three gates:
input, forget and output gates, it calcu- s t = tanh (U z . x t + W s . (h t - 1 9 r)) (17)
lates the hidden state by taking a combi- h t = (1 - z) 9 s t + z 9 h t - 1.
f
i o
xt
ht –1 C̃ C tanh ht
(1) Long Short-Term Memory
Figure 8 illustration of an LSTM and gRu gate.
august 2018 | ieee Computational intelligenCe magazine
ot ot +1
W W
ht −1 ht ht +1
V V V
U U U
xt −1 xt xt +1
Researchers often face the dilemma
of choosing the appropriate RNN. This
also extends to developers working in
NLP. Throughout the history, most of
(9)
the choices over the RNN variant
(10) tended to be heuristic. Chung et al. [74]
did a critical comparative evaluation of
(11)
the three RNN variants mentioned
(12) above, although not on NLP tasks. They
evaluated their work on tasks relating to
polyphonic music modeling and speech
signal modeling. Their evaluation clearly
(14)
demonstrated the superiority of the
gated units (LSTM and GRU) over the
traditional simple RNN (in their case,
using tanh activation). However, they
could not make any concrete conclusion
about which of the two gating units was
better. This fact has been noted in other
works too and, thus, people often lever-
age on other factors like computing
power while choosing between the two.
C. Applications
1) Rnn for Word-level Classification
RNNs have had a huge presence in the
(15) field of word-level classification. Many
(16) of their applications stand as state of the
art in their respective tasks. Lample et al.
[85] proposed to use bidirectional
(18) LSTM for NER. The network captured
r
xt s z ht
(2) Gated Recurrent Unit
63
arbitrarily long context information
around the target word (curbing the
limitation of a fixed window size) result-
ing in two fixed-size vector, on top of
which another fully-connected layer was
built. They used a CRF layer at last for
the final entity tagging.
RNNs have also shown considerable
improvement in language modeling over
traditional methods based on count statis-
tics. Pioneering work in this field was
done by Graves [86], who introduced the
effectiveness of RNNs in modeling com-
plex sequences with long range context
structures. He also proposed deep RNNs
where multiple layers of hidden states
were used to enhance the modeling. This
work established the usage of RNNs on
tasks beyond the context of NLP. Later,
Sundermeyer et al. [87] compared the gain
obtained by replacing a feed-forward neu-
ral network with an RNN when condi-
tioning the prediction of a word on the
words ahead. In their work, they proposed
a typical hierarchy in neural network
architectures where feed-forward neural
networks gave considerable improvement
over traditional count-based language
models, which in turn were superseded by
RNNs and later by LSTMs. An important
point that they mentioned was the appli-
cability of their conclusions to a variety of
other tasks such as statistical machine
translation [88].
2) Rnn for sentence-level
Classification
Wang et al. [24] proposed encoding entire
tweets with LSTM, whose hidden state is
p1
Image
used for predicting sentiment polarity.
This simple strategy proved competitive
to the more complex DCNN structure
by Kalchbrenner et al. [43] designed to
endow CNN models with ability to cap-
ture long-term dependencies. In a special
case studying negation phrase, the authors
also showed that the dynamics of LSTM
gates can capture the reversal effect of the
word “not”.
Similar to CNN, the hidden state of
an RNN can also be used for semantic
matching between texts. In dialogue sys-
tems, Lowe et al. [89] proposed to match
a message with candidate responses with
Dual-LSTM, which encodes both the
message and response as fixed-size vectors
and then measure their inner product as
the basis to rank candidate responses.
3) Rnn for generating language
A challenging task in NLP is generating
natural language, which is another natural
application of RNNs. Conditioned on
textual or visual data, deep LSTMs have
been shown to generate reasonable task-
specific text in tasks such as machine
translation, image captioning, etc.
In [67], the authors proposed a gen-
eral deep LSTM encoder-decoder
framework that maps a sequence to
another sequence. One LSTM is used to
encode the “source” sequence as a fixed-
size vector, which can be text in the
original language (machine translation),
the question to be answered (QA) or the
message to be replied to (dialogue sys-
tems). The vector is used as the initial
state of another LSTM, named the
Output
p2 pN – 1
decoder. The decoder generates tokens
one by one, while updating its hidden
state with the last generated token.
Sutskever et al. [67] experimented
with 4-layer LSTM on a machine trans-
lation task in an end-to-end fashion,
showing competitive results. In [91], the
same encoder-decoder framework is
employed to model human conversations.
When trained on more than 100 million
message-response pairs, the LSTM de-
coder is able to generate very interesting
responses in the open domain. It is also
common to condition the LSTM de-
coder on additional signal to achieve cer-
tain effects. In [92], the authors proposed
to condition the decoder on a constant
persona vector that captures the personal
information of an individual speaker. In
the above cases, language is generated
based mainly on the semantic vector rep-
resenting textual input. Similar frame-
works have also been successfully used in
image-based language generation, where
visual features are used to condition the
LSTM decoder (Fig. 9).
Visual QA is another task that
requires language generation based on
both textual and visual clues. Malinowski
et al. [93] were the first to provide an
end-to-end deep learning solution
where they predicted the answer as a
sequence of words conditioned on the
input image modeled by a CNN and
text modeled by an LSTM.
D. Attention Mechanism
One potential problem that the tradi-
tional encoder-decoder framework faces
is that the encoder at times is forced to
encode information which might not be
fully relevant to the task at hand. The
problem arises also if the input is long or
very information-rich and selective
encoding is not possible.
For example, the task of text summa-

CNN
rization can be cast as a sequence-to-
LSTM LSTM LSTM
sequence learning problem, where the
input is the original text and the output
is the condensed version. Intuitively, it is
w1 w2 wN – 1 unrealistic to expect a fixed-size vector to
True Image Description encode all information in a piece of text
whose length can potentially be very
Figure 9 image captioning using CNN image embedder followed by LSTM decoder. This long. Similar problems have also been
architecture was proposed by vinyals et al. [90]. reported in machine translation [94].

64 ieee Computational intelligenCe magazine | august 2018

 In tasks such as text summarization
and machine translation, certain align-
ment exists between the input text and
the output text, which means that each
token generation step is highly related to
a certain part of the input text.This intu-
ition inspires the attention mechanism.
This mechanism attempts to ease the
above problems by allowing the decoder
to refer back to the input sequence. Spe-
cifically during decoding, in addition to
the last hidden state and generated token,
the decoder is also conditioned on a
“context” vector calculated based on the
input hidden state sequence.
Bahdanau et al. [94] first applied the
attention mechanism to machine trans-
lation, which improved the perfor-
mance especially for long sequences. In
their work, the attention signal over the
input hidden state sequence is deter-
mined with a multi-layer perceptron by
the last hidden state of the decoder. By
visualizing the attention signal over the
input sequence during each decod-
ing step, a clear alignment between
the source and target language can
be demonstrated.
A similar approach was applied to the
task of text summarization by Rush et al.
[95] where each output word in the
summary was conditioned on the input
sentence through an attention mecha-
nism. The authors performed abstractive
summarization which is not very con-
ventional as opposed to extractive sum-
marization, but can be scaled up to large
data with minimal linguistic input.
In image captioning, Xu et al. [96]
conditioned the LSTM decoder on dif-
ferent parts of the input image during
each decoding step. Attention signal was
determined by the previous hidden state
and CNN features. In [97], the authors
cast the syntactical parsing problem as a
sequence-to-sequence learning task by
linearizing the parsing tree. The atten-
tion mechanism proved to be more
data-efficient in this work. A further step
in referring to the input sequence was
to directly copy words or sub-sequences
of the input onto the output sequence
under a certain condition [98], which
was useful in tasks such as dialogue gen-
eration and text summarization. Copy-
Weighted Combination
α
Attention
Aspect Embedding
Hidden h1
Representation
LSTM
Input Sentence
w1
Figure 10 Aspect classification using attention. The original attention-based model in this
application was proposed by Wang et al. [100].
ing or generation was chosen at each
time step during decoding [99].
In aspect-based sentiment analysis,
Wang et al. [100] proposed an attention-
based solution where they used aspect
embeddings to provide additional sup-
port during classification (Fig. 10). The
attention module focused on selective
regions of the sentence which affected
the aspect to be classified. Recently, Ma
et al. [101] augmented LSTM with a
hierarchical attention mechanism con-
sisting of a target-level attention and a
sentence-level attention to exploit com-
monsense knowledge for targeted
aspect-based sentiment analysis.
Given the intuitive applicability of
attention modules, they are still being
actively investigated by NLP researchers
and adopted for an increasing number
of applications.
V. Recursive Neural Networks
RNNs represent a natural way to model
sequences. Arguably, however, language
exhibits a natural recursive structure,
where words and sub-phrases combine
into phrases in a hierarchical manner.
Such structure can be represented by a
constituency parsing tree. Thus, tree-
structured models have been used to
better make use of such syntactic inter-
august 2018 | ieee Computational intelligenCe magazine
hn
LSTM LSTM
w2 wn
pretations of sentence structure [4]. Spe-
cifically, in a recursive neural network,
the representation of each non-terminal
node in a parsing tree is determined by
the representations of all its children.
A. Basic Model
In this section, we describe the basic
structure of recursive neural networks.
As shown in Fig. 11, the network g
defines a compositional function on the
representations of phrases or words (b, c
or a, p 1) to compute the representation
of a higher-level phrase ( p 1 or p 2). The
representations of all nodes take the
same form.
p2 = g (a, p1)
(on the mat)
p1 = g (b, c)
(the mat)
a (on) b (the) c (mat)
Figure 11 Recursive neural networks
iteratively form high-level representation
from lower-level representations.
65

Recurrent Neural Networks use the idea of processing
sequential information. The term “recurrent” applies as
they perform the same computation over each token
of the sequence and each step is dependent on the
previous computations and results.
In [4], the authors described multiple model is trained with the max-margin
variations of this model. In its simplest objective [103].
form, g is defined as: Based on recursive neural networks
and the parsing tree, Socher et al. [4]
p 1 = tanh c W ; Em, p 2 = tanh c W ; Em
b a
proposed a phrase-level sentiment an-
c p1
alysis framework, where each node in
(19)
the parsing tree can be assigned a sen-
in which the representation for each timent label.
node is a d-dimensional vector and Socher et al. [102] classified semantic
W ! R D # 2D. relationships such as cause-effect or
Another variation is the MV-RNN topic-message between nominals in a
[102]. The idea is to represent every sentence by building a single composi-
word and phrase as both a matrix and a tional semantics for the minimal constit-
vector. When two constituents are com- uent including both terms. Bowman et
bined, the matrix of one is multiplied al. [104] proposed to classify the logical
with the vector of the other: relationship between sentences with
recursive neural networks. The represen-
p 1 = tanh c W ; Em, P1 = tanh c W M ; Em
Cb B tations for both sentences are fed to
Bc C another neural network for relationship
(20) classification. They show that both
vanilla and tensor versions of the recur-
in which b, c, p 1 ! R D, B, C, P1 ! R D # D, sive unit performed competitively in a
and W M ! R D # 2D. Compared to the textual entailment dataset.
vanilla form, MV-RNN parameterizes To avoid the gradient vanishing
the compositional function with matri- problem, LSTM units have also been
ces corresponding to the constituents. applied to tree structures in [105]. The
The recursive neural tensor network authors showed improved sentence rep-
(RNTN) is proposed to introduce more resentation over linear LSTM models,
interaction between the input vectors as clear improvement in sentiment
without making the number of parame- analysis and sentence relatedness tests
ters exceptionally large like MV-RNN. was observed.
RNTN is defined by:
VI. Deep Reinforced Models and
p 1 = tanh c; E V [1: D] ; E + W ; Em
b T b b Deep Unsupervised Learning
c c c ternal environment (the words and the
(21)
A. Reinforcement Learning for
where V ! R 2D # 2D # D is a tensor that Sequence Generation
defines multiple bilinear forms. Reinforcement learning is a method of
training an agent to perform discrete
B. Applications actions before obtaining a reward. In
One natural application of recursive NLP, tasks concerning language genera-
neural networks is parsing [10]. A scor- tion can sometimes be cast as reinforce-
ing function is defined on the phrase ment learning problems.
representation to calculate the plausibili- In its original formulation, RNN
ty of that phrase. Beam search is usually language generators are typically trained
applied for searching the best tree. The by maximizing the likelihood of each
token in the ground-truth sequence
given the current hidden state and the
previous tokens. Termed “teacher forc-
ing”, this training scheme provides the
real sequence prefix to the generator
during each generation (loss evaluation)
step. At test time, however, ground-truth
tokens are then replaced by a token gen-
erated by the model itself. This discrep-
ancy between training and inference,
termed “exposure bias” [106], [107], can
yield errors that can accumulate quickly
along the generated sequence.
Another problem with the word-
level maximum likelihood strategy,
when training auto-regressive language
generation models, is that the training
objective is different from the test met-
ric. It is unclear how the n-gram overlap
based metrics (BLEU, ROUGE) used to
evaluate these tasks (machine translation,
dialogue systems, etc.) can be optimized
with the word-level training strategy.
Empirically, dialogue systems trained
with word-level maximum likelihood
also tend to produce dull and short-
sighted responses [108], while text sum-
marization tends to produce incoherent
or repetitive summaries [99].
Reinforcement learning offers a pro-
spective to solve the above problems to a
certain extent. In order to optimize the
non-differentiable evaluation metrics di-
rectly, Ranzato et al. [107] applied the
REINFORCE algorithm [109] to train
RNN-based models for several se-
quence generation tasks (e.g., text sum-
marization, machine translation and
image captioning), leading to improve-
ments compared to previous supervised
learning methods. In such a framework,
the generative model (RNN) is viewed
as an agent, which interacts with the ex-
context vector it sees as input at every
time step). The parameters of this agent
defines a policy, whose execution results
in the agent picking an action, which
refers to predicting the next word in the
sequence at each time step. After taking
an action the agent updates its internal
state (the hidden units of RNN). Once
the agent has reached the end of a se-
quence, it observes a reward. This reward
can be any developer-defined metric

66 ieee Computational intelligenCe magazine | august 2018

tailored to a specific task. For example,
Li et al. [108] defined 3 rewards for a
generated sentence based on ease of an-
swering, information flow, and seman-
tic coherence.
There are two well-known shortcom-
ings of reinforcement learning. To make
reinforcement learning tractable, it is
desired to carefully handle the state and
action space [110], [111], which in the
end may restrict expressive power and
learning capacity of the model. Secondly,
the need for training the reward func-
tions makes such models hard to design
and measure at run time [112], [113].
Another approach for sequence-level
supervision is to use the adversarial
training technique [114], where the
training objective for the language gen-
erator is to fool another discrimina-
tor trained to distinguish generated
sequences from real sequences. The gen-
erator G and the discriminator D are
trained jointly in a min-max game
which ideally leads to G, generating
sequences indistinguishable from real
ones. This approach can be seen as a
var iation of generative adversar ial
networks in [114], where G and D are
conditioned on certain stimuli (for ex-
ample, the source image in the task of
image captioning). In practice, the above
scheme can be realized under the rein-
forcement learning paradigm with poli-
cy gradient. For dialogue systems, the
discriminator is analogous to a human
Turing tester, who discriminates be-
tween human and machine-produced
dialogues [115].
B. Unsupervised Sentence
Representation Learning
Similar to word embeddings, distributed
representation for sentences can also be
learned in an unsupervised fashion. The
result of such unsupervised learning are
“sentence encoders”, which map arbi-
trary sentences to fixed-size vectors that
can capture their semantic and syntactic
properties. Usually an auxiliary task has
to be defined for the learning process.
Similar to the skip-gram model [8]
for learning word embeddings, the skip-
thought model [116] was proposed for
learning sentence representation, where
Recent success in generating realistic images has
driven a series of efforts on applying deep generative
models to text data. The promise of such research is
to discover rich structure in natural language while
generating realistic sentences from a latent code space.
the auxiliary task was to predict two
adjacent sentences (before and after)
based on the given sentence. The
seq2seq model was employed for this
learning task. One LSTM encoded the
sentence to a vector (distributed repre-
sentation). Two other LSTMs decoded
such representation to generate the tar-
get sequences. The standard seq2seq
training process was used. After training,
the encoder could be seen as a generic
feature extractor (word embeddings
were also learned in the same time).
Kiros et al. [116] verified the quality
of the learned sentence encoder on a
range of sentence classification tasks,
showing competitive results with a sim-
ple linear model based on the static fea-
ture vectors. However, the sentence
encoder can also be fine-tuned in the
supervised learning task as part of the
classifier. Dai and Le [117] investigated
the use of the decoder to reconstruct the
encoded sentence itself, which resembled
an autoencoder [118].
Language modeling could also be
used as an auxiliary task when training
LSTM encoders, where the supervision
signal came from the prediction of the
next token. Dai and Le [117] conducted
experiments on initializing LSTM mod-
els with learned parameters on a variety
of tasks. They showed that pre-training
the sentence encoder on a large unsu-
pervised corpus yielded better accuracy
than only pre-training word embeddings.
Also, predicting the next token turned
out to be a worse auxiliary objective
than reconstructing the sentence itself, as
the LSTM hidden state was only respon-
sible for a rather short-term objective.
C. Deep Generative Models
Recent success in generating realistic
images has driven a series of efforts on
applying deep generative models to text
data. The promise of such research is to
discover rich structure in natural lan-
guage while generating realistic sentences
from a latent code space. In this section,
we review recent research on achieving
this goal with variational autoencoders
(VAEs) [119] and generative adversarial
networks (GANs) [114].
Standard sentence autoencoders, as in
the last section, do not impose any con-
straint on the latent space, as a result,
they fail when generating realistic sen-
tences from arbitrary latent representa-
tions [120]. The representations of these
sentences may often occupy a small
region in the hidden space and most of
regions in the hidden space do not nec-
essarily map to a realistic sentence [121].
They cannot be used to assign probabili-
ties to sentences or to sample novel sen-
tences [120].
The VAE imposes a prior distribu-
tion on the hidden code space which
makes it possible to draw proper samples
from the model. It modifies the autoen-
coder architecture by replacing the
deterministic encoder function with a
learned posterior recognition model.
The model consists of encoder and gen-
erator networks which encode data
examples to latent representation and
generate samples from the latent space,
respectively. It is trained by maximizing
a variational lower bound on the log-
likelihood of observed data under the
generative model.
Bowman et al. [120] proposed an
RNN-based variational autoencoder
generative model that incorporated dis-
tributed latent representations of entire
sentences (Fig. 12). Unlike vanilla RNN
language models, this model worked
from an explicit global sentence repre-
sentation. Samples from the prior over
these sentence representations produced
diverse and well-formed sentences.

august 2018 | ieee Computational intelligenCe magazine 67


A generative neural network decodes latent representation
to a data instance, while the discriminative network is
simultaneously taught to discriminate between instances
from the true data distribution and synthesized instances
produced by the generator.
Hu et al. [122] proposed generating criminate between instances from the
sentences whose attributes are con- true data distribution and synthesized
trolled by learning disentangled latent instances produced by the generator.
representations with designated seman- GAN does not explicitly represent the
tics. The authors augmented the latent true data distribution p (x).
code in the VAE with a set of structured Zhang et al. [121] proposed a frame-
variables, each targeting a salient and work for employing LSTM and CNN
independent semantic feature of sen- for adversarial training to generate realis-
tences. The model incorporated VAE tic text. The latent code z was fed to the
and attribute discriminators, in which LSTM generator at every time step.
the VAE component trained the genera- CNN acted as a binary sentence classifier
tor to reconstruct real sentences for gen- which discriminated between real data
erating plausible text, while the discrim- and generated samples. One problem
inators forced the generator to produce with applying GAN to text is that the
attributes coherent with the structured gradients from the discriminator cannot
code. When trained on a large number properly back-propagate through discrete
of unsupervised sentences and a small variables. In [121], this problem was
number of labeled sentences, Hu et al. solved by making the word prediction at
[122] showed that the model was able to every time “soft” at the word embedding
generate plausible sentences conditioned space. Yu et al. [123] proposed to bypass
on two major attributes of English: tense this problem by modeling the generator
and sentiment. as a stochastic policy. The reward signal
GAN is another class of generative came from the GAN discriminator
model composed of two competing judged on a complete sequence, and was
networks. A generative neural network passed back to the intermediate state-
decodes latent representation to a data action steps using Monte Carlo search.
instance, while the discriminative net- The evaluation of deep generative
work is simultaneously taught to dis- models has been challenging. For text, it
Decoder
y1 y2 <EOS>
z LSTM LSTM LSTM
µ σ
<EOS> y1 y2
Linear Linear
LSTM LSTM
LSTM
x1 x2 x3
Encoder
Figure 12 RNN-based vAE network for sentence generation proposed by Bowman et al. [120].
68 ieee Computational intelligenCe magazine | august 2018
is possible to create oracle training data
from a fixed set of grammars and then
evaluate generative models based on
whether (or how well) the generated
samples agree with the predefined
grammar [124]. Another strategy is to
evaluate BLEU scores of samples on a
large amount of unseen test data. The
ability to generate similar sentences to
unseen real data is considered a mea-
surement of quality [123].
VII. Memory-Augmented Networks
The attention mechanism stores a series
of hidden vectors of the encoder, which
the decoder is allowed to access during
the generation of each token. Here, the
hidden vectors of the encoder can be
seen as entries of the model’s “internal
memory”. Recently, there has been a
surge of interest in coupling neural net-
works with a form of memory, which
the model can interact with.
In [135], the authors proposed mem-
ory networks for QA tasks. In synthetic
QA, a series of statements (memory
entries) were provided to the model as
potential supporting facts to the ques-
tion. The model learned to retrieve one
entry at a time from memory based on
the question and previously retrieved
memory. In large-scale realistic QA, a
large set of commonsense knowledge in
the form of (subject, relation, object) tri-
ples were used as memory.
Sukhbaatar et al. [136] extended this
work and proposed end-to-end mem-
ory networks, where memory entries
were retrieved in a “soft” manner with
attention mechanism, thus enabling end-
to-end training. Multiple rounds (hops)
of information retrieval from memory
were shown to be essential to good per-
formance and the model was able to
retrieve and reason about several sup-
porting facts to answer a specific ques-
tion. They also showed a special use of
the model for language modeling,
where each word in the sentence was
seen as a memory entry. With multiple
hops, the model yielded results compa-
rable to deep LSTM models.
Furthermore, dynamic memory net-
works (DMN) [128] improved upon
previous memory-based models by
 TABLe 2 POS tagging.
PAPer modeL wSJ-PTB (Per-Token AccurAcy %)

giMéNEz AND MARquEz [125] SvM WiTH MANuAL fEATuRE PATTERN 97.16

CoLLoBERT ET AL. [5] MLP WiTH WoRD EMBEDDiNgS + CRf 97.29

SANToS AND zADRozNY [31] MLP WiTH CHARACTER + WoRD EMBEDDiNgS 97.32

HuANg ET AL. [126] LSTM 97.29

HuANg ET AL. [126] BiDiRECTioNAL LSTM 97.40

HuANg ET AL. [126] LSTM-CRf 97.54

HuANg ET AL. [126] BiDiRECTioNAL LSTM-CRf 97.55

ANDoR ET AL. [127] TRANSiTioN-BASED NEuRAL NETWoRk 97.45

kuMAR ET AL. [128] DMN 97.56

TABLe 3 Parsing (UAS/LAS = Unlabeled/labeled Attachment Score; WSJ = The Wall Street Journal Section of Penn Treebank).
PArSing TyPe PAPer modeL wSJ

DEPENDENCY PARSiNg CHEN AND MANNiNg [129] fuLLY-CoNNECTED NN WiTH fEATuRES iNCLuDiNg PoS 91.8/89.6 (uAS/LAS)

WEiSS ET AL. [130] DEEP fuLLY-CoNNECTED NN WiTH fEATuRES iNCLuDiNg PoS 94.3/92.4 (uAS/LAS)

DYER ET AL. [131] STACk-LSTM 93.1/90.9 (uAS/LAS)

zHou ET AL. [132] BEAM CoNTRASTivE MoDEL 93.31/92.37 (uAS/LAS)

CoNSTiTuENCY PARSiNg PETRov ET AL. [133] PRoBABiLiSTiC CoNTExT-fREE gRAMMARS (PCfg) 91.8 (f1 SCoRE)

SoCHER ET AL. [10] RECuRSivE NEuRAL NETWoRkS 90.29 (f1 SCoRE)

zHu ET AL. [134] fEATuRE-BASED TRANSiTioN PARSiNg 91.3 (f1 SCoRE)

viNYALS ET AL. [97] SEq2SEq LEARNiNg WiTH LSTM+ATTENTioN 93.5 (f1 SCoRE)

employing neural network models for in- TABLe 4 Named-Entity Recognition.

put representation, attention, and answer
PAPer modeL conLL 2003 (F1%)
mechanisms. The resulting model was ap-
plicable to a wide range of NLP tasks CoLLoBERT ET AL. [5] MLP WiTH WoRD 89.59
EMBEDDiNgS+gAzETTEER
(QA, POS tagging, and sentiment analysis),
as every task could be cast to the <memo- PASSoS ET AL. [138] LExiCoN iNfuSED PHRASE EMBEDDiNgS 90.90

ry, question, answer> triple format. Xiong CHiu AND NiCHoLS [139] Bi-LSTM WiTH WoRD+CHAR+LExiCoN 90.77
et al. [137] applied the same model to vi- EMBEDDiNgS

sual QA and proved that the memory
module was applicable to visual signals.
VIII. Performance of Different
Models on Different NLP Tasks
We summarize the performance of a
series of deep learning methods on stan-
dard datasets developed in recent years
on 7 major NLP topics in Tables 2–7.
Our goal is to show the readers com-
mon datasets used in the community
and state-of-the-art results along with
different models.
A. POS Tagging
The WSJ-PTB (the Wall Street Journal
part of the Penn Treebank Dataset) cor-
pus contains 1.17 million tokens and has
Luo ET AL. [140] SEMi-CRf JoiNTLY TRAiNED WiTH LiNkiNg 91.20
LAMPLE ET AL. [85] Bi-LSTM-CRf WiTH WoRD+CHAR 90.94
EMBEDDiNgS
LAMPLE ET AL. [85] Bi-LSTM WiTH WoRD+CHAR EMBEDDiNgS 89.15
STRuBELL ET AL. [141] DiLATED CNN WiTH CRf 90.54
been widely used for developing and cy between adjacent tags. With a simple
evaluating POS tagg ing systems. left-to-right tagging scheme, this meth-
Giménez and Marquez [125] employed od modeled dependencies between
one-against-all SVM based on manual- adjacent tags only by feature engineer-
ly-defined features within a seven- ing. In an effort to reduce feature engi-
word window, in which some basic neering, Collobert et al. [5] relied on
n-gram patterns were evaluated to form only word embeddings within the word
binary features such as: “previous word is window with a multi-layer perceptron.
the”, “two preceding tags are DT NN”, Incorporating CRF was proven useful
etc. One characteristic of the POS tag- in [5]. Santos and Zadrozny [31] concat-
ging problem was the strong dependen- enated word embeddings with character

august 2018 | ieee Computational intelligenCe magazine 69

 TABLe 5 Semantic Role Labeling.
PAPer modeL conLL2005 (F1%) conLL2012 (F1%)

CoLLoBERT ET AL. [5] CNN WiTH PARSiNg fEATuRES 76.06

TäCkSTRöM ET AL. [142] MANuAL fEATuRES WiTH DP foR iNfERENCE 78.6 79.4

zHou AND xu [143] BiDiRECTioNAL LSTM 81.07 81.27

HE ET AL. [144] BiDiRECTioNAL LSTM WiTH HigHWAY CoNNECTioNS 83.2 83.4

TABLe 6 Sentiment Classification (SST-1 = Stanford Sentiment Treebank,

made based on a stack containing avail-
fine-grained 5 classes Socher et al. [4]; SST-2: the binary version of SST-1; able tree nodes, a buffer containing
Numbers are accuracies (%)). unread words and the obtained set of
PAPer modeL SST-1 SST-2 dependency arcs. Chen and Manning
SoCHER ET AL. [4] RECuRSivE NEuRAL TENSoR NETWoRk 45.7 85.4 [129] modeled the decision making at
each time step with a neural network
kiM [44] MuLTiCHANNEL CNN 47.4 88.1
with one hidden layer. The input layer
kALCHBRENNER ET AL. [43] DCNN WiTH k-MAx PooLiNg 48.5 86.8
contained embeddings of certain words,
TAi ET AL. [105] BiDiRECTioNAL LSTM 48.5 87.2 POS tags and arc labels, which came
LE AND MikoLov [145] PARAgRAPH vECToR 48.7 87.8 from the stack, the buffer and the set of
TAi ET AL. [105] CoNSTiTuENCY TREE-LSTM 51.0 88.0
arc labels.
Tu et al. [61] extended the work of
Yu ET AL. [146] TREE-LSTM WiTH REfiNED WoRD 54.0 90.3
EMBEDDiNgS
Chen and Manning [129] by employing
a deeper model with 2 hidden layers.
kuMAR ET AL. [128] DMN 52.1 88.6
However, both Tu et al. [61] and Chen
and Manning [129] relied on manual
embeddings to better exploit morpho- context multiple times by treating each feature selecting from the parser state,
logical clues. In [31], the authors did not RNN hidden state as a memory entry, and they only took into account a lim-
consider CRF, but since word-level each time focusing on different parts of ited number of latest tokens. Dyer et al.
decision was made on a context win- the context. [131] proposed stack-LSTMs to model
dow, dependencies between adjacent arbitrarily long history. The end pointer
tags were modeled implicitly. Huang B. Parsing of the stack changed position as the
et al. [126] concatenated word embed- There are two types of parsing: depen- stack of tree nodes could be pushed and
dings and manually-designed word-level dency parsing, which connects individu- popped. Zhou et al. [132] integrated
features and employed bidirectional al words with their relations, and beam search and contrastive learning for
LSTM to model arbitrarily long context. constituency parsing, which iteratively better optimization.
A series of ablative analysis suggested that breaks text into sub-phrases. Transition- Transition-based models were applied
bi-directionality and CRF both boosted based methods are a popular choice to constituency parsing as well. Zhu et al.
performance. Andor et al. [127] showed a since they are linear in the length of the [134] based each transition action on
transition-based approach that produces sentence. The parser makes a series of features such as the POS tags and con-
competitive result with a simple feed- decisions that read words sequentially stituent labels of the top few words of
forward neural network. When applied from a buffer and combine them incre- the stack and the buffer. By uniquely
to sequence tagging tasks, DMNs [128] mentally into the syntactic structure representing the parsing tree with a lin-
essentially allowed for attending over the [129]. At each time step, the decision is ear sequence of labels, Vinyals et al. [97]

TABLe 7 Machine translation (Numbers are BLEU scores).

wmT2014 wmT2014
PAPer modeL engLiSh2germAn engLiSh2French

CHo ET AL. [75] PHRASE TABLE WiTH NEuRAL fEATuRES 34.50

SuTSkEvER ET AL. [67] RERANkiNg PHRASE-BASED SMT BEST LiST WiTH LSTM SEq2SEq 36.5

Wu ET AL. [147] RESiDuAL LSTM SEq2SEq + REiNfoRCEMENT LEARNiNg REfiNiNg 26.30 41.16

gEHRiNg ET AL. [148] SEq2SEq WiTH CNN 26.36 41.29

vASWANi ET AL. [149] ATTENTioN MECHANiSM 28.4 41.0

70 ieee Computational intelligenCe magazine | august 2018

applied the seq2seq learning method to
this problem.
C. Named-Entity Recognition
CoNLL 2003 has been a standard Eng-
lish dataset for NER, which concentrates
on four types of named entities: people,
locations, organizations and miscellaneous
entities. NER is one of the NLP prob-
lems where lexicons can be very useful.
Collobert et al. [5] first achieved compet-
itive results with neural structures aug-
mented by gazetteer features. Chiu and
Nichols [139] concatenated lexicon fea-
tures, character embeddings and word
embeddings and fed them as input to a
bidirectional LSTM. On the other hand,
Lample et al. [85] only relied on charac-
ter and word embeddings, with pre-train-
ing embeddings on large unsupervised
corpora, they achieved competitive results
without using any lexicon. Similar to
POS tagging, CRF also boosted the per-
formance of NER, as demonstrated by
the comparison in [85]. Overall, we see
that bidirectional LSTM with CRF acts
as a strong model for NLP problems
related to structured prediction.
Passos et al. [138] proposed to modify
skip-gram models to better learn entity-
type related word embeddings that can
leverage information from relevant lexi-
cons. Luo et al. [140] jointly optimized
the entities and the linking of entities to
a KB. Strubell et al. [141] proposed to use
dilated convolutions, defined over a
wider effective input width by skipping
over certain inputs at a time, for better
parallelization and context modeling.
The model showed significant speedup
while retaining accuracy.
D. Semantic Role Labeling
Semantic role labeling (SRL) aims to
discover the predicate-argument struc-
ture of each predicate in a sentence. For
each target verb (predicate), all constit-
uents in the sentence which take a
semantic role of the verb are recognized.
Typical semantic arguments include
Agent, Patient, Instrument, etc., and also
adjuncts such as Locative, Temporal,
Manner, Cause, etc. [143]. Table 5 shows
the performance of different models on
the CoNLL 2005&2012 datasets.
Recently, there has been a surge of interest in coupling
neural networks with a form of memory, which the model
can interact with.
Traditional SRL systems consist of
several stages: producing a parse tree,
identifying which parse tree nodes rep-
resent the arguments of a given verb,
and finally classifying these nodes to
determine the corresponding SRL tags.
Each classification process usually entails
extracting numerous features and feed-
ing them into statistical models [5].
Given a predicate, Täckström et al.
[142] scored a constituent span and its
possible role to that predicate with a
series of features based on the parse tree.
They proposed a dynamic programming
algorithm for efficient inference. Collob-
ert et al. [5] achieved comparable results
with a convolution neural networks aug-
mented by parsing information provided
in the form of additional look-up tables.
Zhou and Xu [143] proposed to use
bidirectional LSTM to model arbitrarily
long context, which proved to be suc-
cessful without any parsing tree informa-
tion. He et al. [144] further extended this
work by introducing highway connec-
tions [150], more advanced regulariza-
tion and ensemble of multiple experts.
E. Sentiment Classification
The Stanford Sentiment Treebank (SST)
dataset contains sentences taken from
the movie review website Rotten Toma-
toes. It was proposed by Pang and Lee
[151] and subsequently extended by
Socher et al. [4]. The annotation scheme
has inspired a new dataset for sentiment
analysis, called CMU-MOSI, where
sentiment is studied in a multimodal
setup [152].
Socher et al. [4] and Tai et al. [105]
were both recursive networks that relied
on constituency parsing trees. Their dif-
ference shows the effectiveness of
LSTM over vanilla RNN in modeling
sentences. On the other hand, tree-
LSTM performed better than linear
bidirectional LSTM, implying that tree
structures can potentially better capture
the syntactical property of natural sen-
august 2018 | ieee Computational intelligenCe magazine
tences. Yu et al. [146] proposed to refine
pre-trained word embeddings with a
sentiment lexicon, observing improved
results based on [105].
Kim [44] and Kalchbrenner et al.
[43] both used convolutional layers. The
model [44] was similar to the one in
Fig. 5, while Kalchbrenner et al. [43]
constructed the model in a hierarchical
manner by interweaving k-max pooling
layers with convolutional layers.
F. Machine Translation
The phrase-based SMT framework
[160] factorized the translation model
into the translation probabilities of
matching phrases in the source and tar-
get sentences. Cho et al. [75] proposed
to learn the translation probability of a
source phrase to a corresponding target
phrase with an RNN encoder-decoder.
Such a scheme of scoring phrase pairs
improved translation performance.
Sutskever et al. [67], on the other hand,
re-scored the top 1000 best candidate
translations produced by an SMT system
with a 4-layer LSTM seq2seq model.
Dispensing the traditional SMT system
entirely, Wu et al. [147] trained a deep
LSTM network with 8 encoder and
8 decoder layers with residual connec-
tions as well as attention connections.
Wu et al. [147] then refined the model
by using reinforcement learning to
directly optimize BLEU scores, but they
found that the improvement in BLEU
scores by this method did not reflect in
human evaluation of translation quality.
Recently, Gehring et al. [148] pro-
posed a CNN-based seq2seq learning
model for machine translation. The rep-
resentation for each word in the input is
computed by CNN in a parallelized
style for the attention mechanism. The
decoder state is also determined by
CNN with words that are already pro-
duced. Vaswani et al. [149] proposed a
self-attention-based model and dispensed
convolutions and recurrences entirely.
71

We expect to see more deep learning models whose
internal memory (bottom-up knowledge learned from
the data) is enriched with an external memory
(top-down knowledge inherited from a knowledge base).
G. Question Answering
QA problems take many forms. Some
rely on large KBs to answer open-
domain questions, while others answer a
question based on a few sentences or a
paragraph (reading comprehension). For
the former, we list (see Table 8) several
experiments conducted on a large-scale
QA dataset introduced by [153], where
14 M commonsense knowledge triples
are considered as the KB. Each question
can be answered with a single-relation
query. For the latter, we consider (see
Table 8) the synthetic dataset of bAbI,
which requires the model to reason over
multiple related facts to produce the right
answer. It contains 20 synthetic tasks that
test a model’s ability to retrieve relevant
facts and reason over them. Each task
focuses on a different skill such as basic
coreference and size reasoning.
The central problem of learning to
answer single-relation queries is to find
the single supporting fact in the data-
base. Fader et al. [153] proposed to
tackle this problem by learning a lexicon
that maps natural language patterns to
database concepts (entities, relations and
question patterns) based on a question
paraphrasing dataset. Bordes et al. [154]
embedded both questions and KB tri-
ples as dense vectors and scored them
with inner product.
Weston et al. [135] took a similar
approach by treating the KB as long-
term memory, while casting the prob-
lem in the framework of a memory net-
work. On the bAbI dataset, Sukhbaatar
et al. [136] improved upon the original
memory networks model [135] by mak-
ing the training procedure agnostic of
the actual supporting fact, while Kumar
et al. [128] used neural sequence mod-
els (GRU) instead of neural bag-of-
words models as in [136] and [135] to
embed memories.
H. Dialogue Systems
Two types of dialogue systems have
been developed: generation-based mod-
els and retrieval-based models.
In Table 9, the Twitter Conversation
Triple Dataset is typically used for eval-
uating generation-based dialogue
systems, containing 3-turn Twitter con-
versation instances. One commonly
used evaluation metric is BLEU [161],
although it is commonly acknowledged
that most automatic evaluation metrics
are not completely reliable for dialogue
evaluation and additional human evalu-
ation is often necessary. Ritter et al.
[155] employed the phrase-based statis-
tical machine translation (SMT) frame-
work to “translate” the message to its
appropriate response. Sordoni et al.
[156] reranked the 1000 best responses
produced by SMT with a context-sen-
sitive RNN encoder-decoder frame-
work, observing substantial gains. Li
et al. [157] reported results on replacing
the traditional maximum log likelihood
training objective with the maximum
mutual information training objective,
in an effort to produce interesting and
diverse responses, both of which are
tested on a 4-layer LSTM encoder-
decoder framework.

TABLe 8 Question answering.

PAPer modeL bAbi (meAn AccurAcy %) FArBeS (AccurAcy %)

fADER ET AL. [153] PARAPHRASE-DRivEN LExiCoN LEARNiNg 0.54

BoRDES ET AL. [154] WEEkLY SuPERviSED EMBEDDiNg 0.73

WESToN ET AL. [135] MEMoRY NETWoRkS 93.3 0.83

SukHBAATAR ET AL. [136] END-To-END MEMoRY NETWoRkS 88.4

kuMAR ET AL. [128] DMN 93.6

TABLe 9 Dialogue systems.

TwiTTer converSATion uBunTu diALogue dATASeT
PAPer modeL TriPLe dATASeT (BLeu) (recALL 1@10%)

RiTTER ET AL. [155] SMT 3.60

SoRDoNi ET AL. [156] SMT+NEuRAL RERANkiNg 4.44

Li ET AL. [157] LSTM SEq2SEq 4.51

Li ET AL. [157] LSTM SEq2SEq WiTH MMi oBJECTivE 5.22

LoWE ET AL. [89] DuAL LSTM ENCoDERS foR SEMANTiC MATCHiNg 55.22

DoDgE ET AL. [158] MEMoRY NETWoRkS 63.72

zHou ET AL. [159] SENTENCE-LEvEL CNN-LSTM ENCoDER 66.15

72 ieee Computational intelligenCe magazine | august 2018

 The response retrieval task is defined
as selecting the best response from a re-
pository of candidate responses. Such a
model can be evaluated by the recall1@k
metric, where the ground-truth re-
sponse is mixed with k - 1 random re-
sponses. The Ubuntu dialogue dataset
was constructed by scraping multi-turn
Ubuntu trouble-shooting dialogues
from an online chatroom [89]. Lowe et
al. [89] used LSTMs to encode the mes-
sage and response, and then inner prod-
uct of the two sentence embeddings is
used to rank candidates.
Zhou et al. [159] proposed to better
exploit the multi-turn nature of human
conversation by employing the LSTM
encoder on top of sentence-level CNN
embeddings, similar to [162]. Dodge et
al. [158] cast the problem in the frame-
work of a memory network, where the
past conversation was treated as memory
and the latest utterance was considered
as a “question” to be responded to. The
authors showed that using simple neural
bag-of-word embedding for sentences
can yield competitive results.
IX. Conclusion
Deep learning offers a way to harness
large amount of computation and data
with little engineering by hand [163].
With distributed representation, various
deep models have become the new
state-of-the-art methods for NLP prob-
lems. Supervised learning is the most
popular practice in recent deep learning
research for NLP. In many real-world
scenarios, however, we have unlabeled
data which require advanced unsuper-
vised or semi-supervised approaches. In
cases where there is lack of labeled data
for some particular classes or the appear-
ance of a new class while testing the
model, strategies like zero-shot learning
should be employed. These learning
schemes are still in their developing
phase but we expect deep learning
based NLP research to be driven in the
direction of making better use of unla-
beled data. We expect such trend to
continue with more and better model
designs. We expect to see more NLP
applications that employ reinforcement
learning methods, e.g., dialogue systems.
We also expect to see more research on
multimodal learning [164] as, in the real
world, language is often grounded on
(or correlated with) other signals.
Finally, we expect to see more deep
learning models whose internal mem-
ory (bottom-up knowledge learned
from the data) is enr iched with an
external memory (top-down knowledge
inherited from a KB). Coupling sym-
bolic and sub-symbolic AI will be key
for stepping forward in the path from
NLP to natural language understanding.
Relying on machine learning, in fact, is
good to make a ‘good guess’ based on
past experience, because sub-symbolic
methods encode correlation and their
decision-making process is probabilistic.
Natural language understanding, how-
ever, requires much more than that. To
use Noam Chomsky’s words, “you do
not get discoveries in the sciences by
taking huge amounts of data, throwing
them into a computer and doing statisti-
cal analysis of them: that’s not the way
you understand things, you have to have
theoretical insights”.
References
[1] E. Cambria and B. White, “Jumping NLP curves: A
review of natural language processing research,” IEEE
Comput. Intell. Mag., vol. 9, no. 2, pp. 48–57, May 2014.
[2] T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and
S. Khudanpur, “Recurrent neural network based lan-
guage model,” in Proc. Interspeech, vol. 2, p. 3, 2010.
[3] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado,
and J. Dean, “Distributed representations of words and
phrases and their compositionality,” in Proc. Advances
Neural Information Processing Systems, 2013, pp. 3111–3119.
[4] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D.
Manning, A. Ng, and C. Potts, “Recursive deep mod-
els for semantic compositionality over a sentiment tree-
bank,” in Proc. Conf. Empirical Methods Natural Language
Processing, 2013, pp. 1631–1642.
[5] R. Collobert, J. Weston, L. Bottou, M. Karlen, K.
Kavukcuoglu, and P. Kuksa, “Natural language process-
ing (almost) from scratch,” J. Mach. Learn. Res., vol. 12,
pp. 2493–2537, Aug. 2011.
[6] Y. Goldberg, “A primer on neural network models for
natural language processing,” J. Artif. Intell. Res., vol. 57,
pp. 345–420, Nov. 2016.
[7] Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin,
“A neural probabilistic language model,” J. Mach. Learn.
Res., vol. 3, pp. 1137–1155, Feb. 2003.
[8] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Effi-
cient estimation of word representations in vector space,”
arXiv Preprint, arXiv:1301.3781, 2013.
[9] J. Weston, S. Bengio, and N. Usunier, “Wsabie: Scal-
ing up to large vocabulary image annotation,” Proc. Int.
Joint Conf. Artificial Intelligence., 2011, vol. 11, pp. 2764–
2770.
[10] R. Socher, C. C. Lin, C. Manning, and A. Y. Ng,
“Parsing natural scenes and natural language with recur-
sive neural networks,” in Proc. 28th Int. Joint Conf. Machine
Learning, 2011, pp. 129–136.
[11] P. D. Turney and P. Pantel, “From frequency to
meaning: Vector space models of semantics,” J. Artif. In-
tell. Res., vol. 37, pp. 141–188, Nov. 2010.
august 2018 | ieee Computational intelligenCe magazine
[12] E. Cambria, S. Poria, A. Gelbukh, and M. Thelwall,
“Sentiment analysis is a big suitcase,” IEEE Intell. Syst.,
vol. 32, no. 6, pp. 74–80, Nov. 2017.
[13] X. Glorot, A. Bordes, and Y. Bengio, “Domain ad-
aptation for large-scale sentiment classification: A deep
learning approach,” in Proc. 28th Int. Conf. Machine Learn-
ing, 2011, pp. 513–520.
[14] K. M. Hermann and P. Blunsom, “The role of syntax
in vector space models of compositional semantics,” in
Proc. 51st Annu. Meeting Association Computational Linguis-
tics, 2013, vol. 1, pp. 894–904.
[15] J. L. Elman, “Distributed representations, simple
recurrent networks, and grammatical structure,” Mach.
Learn., vol. 7, no. 2–3, pp. 195–225, 1991.
[16] A. M. Glenberg and D. A. Robertson, “Symbol
grounding and meaning: A comparison of high-dimen-
sional and embodied theories of meaning,” J. Memory
Lang., vol. 43, no. 3, pp. 379–401, Oct. 2000.
[17] S. T. Dumais, “Latent semantic analysis,” Annu. Rev.
Inf. Sci. Tech., vol. 38, no. 1, pp. 188–230, Nov. 2004.
[18] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent
dirichlet allocation,” J. Mach. Learn. Res., vol. 3, pp.
993–1022, 2003.
[19] R. Collobert and J. Weston, “A unified architecture
for natural language processing: Deep neural networks
with multitask learning,” in Proc. 25th Int. Conf. Machine
Learning, 2008, pp. 160–167.
[20] A. Gittens, D. Achlioptas, and M. W. Mahoney,
“Skip-gram-zipf + uniform = vector additivity,” in Proc.
55th Annu. Meeting Association Computational Linguistics,
2017, vol. 1, pp. 69–76.
[21] J. Pennington, R. Socher, and C. D. Manning,
“Glove: Global vectors for word representation,” in Proc.
Conf. Empirical Methods Natural Language Processing, 2014,
vol. 14, pp. 1532–1543.
[22] R. Johnson and T. Zhang, “Semi-supervised con-
volutional neural networks for text categorization via
region embedding,” in Proc. Advances Neural Information
Processing Systems, 2015, pp. 919–927.
[23] R. Socher, J. Pennington, E. H. Huang, A. Y. Ng,
and C. D. Manning, “Semi-supervised recursive auto-
encoders for predicting sentiment distributions,” in Proc.
Conf. Empirical Methods Natural Language Processing, 2011,
pp. 151–161.
[24] X. Wang, Y. Liu, C. Sun, B. Wang, and X. Wang,
“Predicting polarities of tweets by composing word em-
beddings with long short-term memory,” in Proc. Annu.
Meeting Association Computational Linguistics, 2015, pp.
1343–1353.
[25] D. Tang, F. Wei, N. Yang, M. Zhou, T. Liu, and B.
Qin, “Learning sentiment-specific word embedding for
twitter sentiment classification,” in Proc. Annu. Meeting
Association Computational Linguistics, 2014, pp. 1555–1565.
[26] I. Labutov and H. Lipson, “Re-embedding words,”
in Proc. Annu. Meeting Association Computational Linguistics,
2013, pp. 489–493.
[27] S. Upadhyay, K. Chang, M. Taddy, A. Kalai, and
J. Zou, “Beyond bilingual: Multi-sense word embed-
dings using multilingual context,” arXiv Preprint, arX-
iv:1706.08160, 2017.
[28] Y. Kim, Y. Jernite, D. Sontag, and A. M. Rush,
“Character-aware neural language models,” in Proc. As-
sociation Advancement Artificial Intelligence Conf., 2016, pp.
2741–2749.
[29] C. N. Dos Santos and M. Gatti, “Deep convolutional
neural networks for sentiment analysis of short texts,” in
Proc. Int. Conf. Computational Linguistics, 2014, pp. 69–78.
[30] C. N d Santos and V. Guimaraes, “Boosting named
entity recognition with neural character embeddings,”
arXiv Preprint, arXiv:1505.05008, 2015.
[31] C. D. Santos and B. Zadrozny, “Learning character-
level representations for part-of-speech tagging,” in Proc.
31st Int. Conf. Machine Learning, 2014, pp. 1818–1826.
[32] Y. Ma, E. Cambria, and S. Gao, “Label embedding
for zero-shot fine-grained named entity typing,” in Proc.
Int. Conf. Computational Linguistics, Osaka, 2016, pp.
171–180.
[33] X. Chen, L. Xu, Z. Liu, M. Sun, and H. Luan, “Joint
learning of character and word embeddings,” in Proc. Int.
Joint Conf. Artificial Intelligence, 2015, pp. 1236–1242.
[34] X. Zheng, H. Chen, and T. Xu, “Deep learning for
chinese word segmentation and pos tagging,” in Proc.
73
Conf. Empirical Methods Natural Language Processing, 2013,
pp. 647–657.
[35] H. Peng, E. Cambria, and X. Zou, “Radical-based
hierarchical embeddings for chinese sentiment analysis
at sentence level,” in Proc. Int. Florida Artificial Intelligence
Research Society Conf., 2017, pp. 347–352.
[36] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov,
“Enriching word vectors with subword information,”
arXiv Preprint, arXiv:1607.04606, 2016.
[37] A. Herbelot and M. Baroni, “High-risk learning:
Acquiring new word vectors from tiny data,” arXiv Pre-
print, arXiv:1707.06556, 2017.
[38] Y. Pinter, R. Guthrie, and J. Eisenstein, “Mimicking
word embeddings using subword rnns,” arXiv Preprint,
arXiv:1707.06961, 2017.
[39] L. Lucy and J. Gauthier, “Are distributional repre-
sentations ready for the real world? Evaluating word vec-
tors for grounded perceptual meaning,” arXiv Preprint,
arXiv:1705.11168, 2017.
[40] A. Krizhevsky, I. Sutskever, and G. E. Hinton,
“Imagenet classification with deep convolutional neural
networks,” in Proc. Advances Neural Information Processing
Systems, 2012, pp. 1097–1105.
[41] A. Sharif Razavian, H. Azizpour, J. Sullivan, and
S. Carlsson, “CNN features off-the-shelf: An astound-
ing baseline for recognition,” in Proc. IEEE Conf. Com-
puter Vision and Pattern Recognition Workshops, 2014, pp.
806–813.
[42] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J.
Long, R. Girshick, S. Guadarrama, and T. Darrell,
“Caffe: Convolutional architecture for fast feature em-
bedding,” in Proc. 22nd ACM Int. Conf. Multimedia, 2014,
pp. 675–678.
[43] N. Kalchbrenner, E. Grefenstette, and P. Blunsom,
“A convolutional neural network for modelling sentenc-
es,” in Proc. 52nd Annu. Meeting Association Computational
Linguistics, 2014, vol. 1, pp. 655–665.
[44] Y. Kim, “Convolutional neural networks for sen-
tence classification,” arXiv Preprint, arXiv:1408.5882,
2014.
[45] S. Poria, E. Cambria, and A. Gelbukh, “Aspect ex-
traction for opinion mining with a deep convolutional
neural network,” Knowl.-Based Syst., vol. 108, pp. 42–49,
June 2016.
[46] A. Kirillov, D. Schlesinger, W. Forkel, A. Zelenin,
S. Zheng, P. Torr, and C. Rother, “Efficient likelihood
learning of a generic CNN-CRF model for semantic
segmentation,” arXiv Preprint, arXiv:1511.05067, 2015.
[47] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and
K. J. Lang, “Phoneme recognition using time-delay neu-
ral networks,” IEEE Trans. Acoust., Speech, Signal Process.,
vol. 37, no. 3, pp. 328–339, Mar. 1989.
[48] A. Mukherjee and B. Liu, “Aspect extraction through
semi-supervised modeling,” in Proc. 50th Annu. Meeting
Association Computational Linguistics, 2012, pp. 339–348.
[49] S. Ruder, P. Ghaffari, and J. G. Breslin, “Insight-1
at semeval-2016 task 5: Deep learning for multilingual
aspect-based sentiment analysis,” arXiv Preprint, arX-
iv:1609.02748, 2016.
[50] P. Wang, J. Xu, B. Xu, C. Liu, H. Zhang, F. Wang,
and H. Hao, “Semantic clustering and convolutional
neural network for short text categorization,” in Proc.
Annu. Meeting Association Computational Linguistics, 2015,
pp. 352–357.
[51] S. Poria, E. Cambria, D. Hazarika, and P. Vij, “A
deeper look into sarcastic tweets using deep convolu-
tional neural networks,” in Proc. Int. Conf. Computational
Linguistics, 2016, pp. 1601–1612.
[52] M. Denil, A. Demiraj, N. Kalchbrenner, P. Blunsom,
and N. de Freitas, “Modelling, visualising and summaris-
ing documents with a single convolutional neural net-
work,” 26th Int. Conf. Computational Linguistics, 2014,
pp. 1601–1612.
[53] B. Hu, Z. Lu, H. Li, and Q. Chen, “Convolutional
neural network architectures for matching natural lan-
guage sentences,” in Proc. Advances Neural Information Pro-
cessing Systems, 2014, pp. 2042–2050.
[54] Y. Shen, X. He, J. Gao, L. Deng, and G. Mesnil,
“A latent semantic model with convolutional-pooling
structure for information retrieval,” in Proc. 23rd ACM
Int. Conf. Information and Knowledge Management, 2014,
pp. 101–110.
74 ieee Computational intelligenCe magazine | august 2018
[55] W. Yih, X. He, and C. Meek, “Semantic parsing
for single-relation question answering,” in Proc. Annu.
Meeting Association Computational Linguistics, 2014, pp.
643–648.
[56] L. Dong, F. Wei, M. Zhou, and K. Xu, “Question
answering over freebase with multi-column convolu-
tional neural networks,” in Proc. Annu. Meeting Association
Computational Linguistics, 2015, pp. 260–269.
[57] A. Severyn and A. Moschitti, “Modeling relational
information in question-answer pairs with convolutional
neural networks,” arXiv Preprint, arXiv:1604.01178,
2016.
[58] Y. Chen, L. Xu, K. Liu, D. Zeng, and J. Zhao, “Event
extraction via dynamic multi-pooling convolutional
neural networks,” in Proc. Annu. Meeting Association Com-
putational Linguistics, 2015, pp. 167–176.
[59] O. Abdel-Hamid, A.-R. Mohamed, H. Jiang, L.
Deng, G. Penn, and D. Yu, “Convolutional neural
networks for speech recognition,” IEEE Trans. Acoust.,
Speech, Signal Process., vol. 22, no. 10, pp. 1533–1545,
Oct. 2014.
[60] D. Palaz, M. Magimai.-Doss, and R. Collobert,
“Analysis of CNN-based speech recognition system us-
ing raw speech as input,” Idiap, Tech. Rep, 2015.
[61] Z. Tu, B. Hu, Z. Lu, and H. Li, “Context-dependent
translation selection using convolutional neural net-
work,” arXiv Preprint, arXiv:1503.02357, 2015.
[62] J. L. Elman, “Finding structure in time,” Cogn. Sci.,
vol. 14, no. 2, pp. 179–211, 1990.
[63] T. Mikolov, S. Kombrink, L. Burget, J. Černocký,
and S. Khudanpur, “Extensions of recurrent neural net-
work language model,” in Proc. Int. Conf. Acoustics, Speech
and Signal Processing, 2011, pp. 5528–5531.
[64] I. Sutskever, J. Martens, and G. E. Hinton, “Gener-
ating text with recurrent neural networks,” in Proc. 28th
Int. Conf. Machine Learning, 2011, pp. 1017–1024.
[65] S. Liu, N. Yang, M. Li, and M. Zhou, “A recursive
recurrent neural network for statistical machine transla-
tion,” in Proc. 52nd Annu. Meeting Association Computa-
tional Linguistics, 2014, pp. 1491–1500.
[66] M. Auli, M. Galley, C. Quirk, and G. Zweig, “Joint
language and translation modeling with recurrent neural
networks,” in Proc. Conf. Empirical Methods Natural Lan-
guage Processing, 2013, pp. 1044–1054.
[67] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence
to sequence learning with neural networks,” in Proc.
Advances Neural Information Processing Systems, 2014, pp.
3104–3112.
[68] T. Robinson, M. Hochberg, and S. Renals, “The use
of recurrent neural networks in continuous speech recog-
nition,” in Proc. Automatic Speech and Speaker Recognition,
1996, pp. 233–258.
[69] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech
recognition with deep recurrent neural networks,” in
Proc. Int. Conf. Acoustics, Speech and Signal Processing, 2013,
pp. 6645–6649.
[70] A. Graves and N. Jaitly, “Towards end-to-end speech
recognition with recurrent neural networks,” in Proc. 31st
Int. Conf. Machine Learning, 2014, pp. 1764–1772.
[71] H. Sak, A. Senior, and F. Beaufays, “Long short-term
memory based recurrent neural network architectures for
large vocabulary speech recognition,” arXiv Preprint,
arXiv:1402.1128, 2014.
[72] A. Karpathy and L. Fei-Fei, “Deep visual-semantic
alignments for generating image descriptions,” in Proc.
IEEE Conf. Computer Vision and Pattern Recognition, 2015,
pp. 3128–3137.
[73] D. Tang, B. Qin, and T. Liu, “Document modeling
with gated recurrent neural network for sentiment classi-
fication,” in Proc. Conf. Empirical Methods Natural Language
Processing, 2015, pp. 1422–1432.
[74] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Em-
pirical evaluation of gated recurrent neural networks on
sequence modeling,” arXiv Preprint, arXiv:1412.3555,
2014.
[75] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bah-
danau, F. Bougares, H. Schwenk, and Y. Bengio, “Learn-
ing phrase representations using RNN encoder-decoder
for statistical machine translation,” arXiv Preprint, arX-
iv:1406.1078, 2014.
[76] G. Chen, D. Ye, E. Cambria, J. Chen, and Z. Xing,
“Ensemble application of convolutional and recurrent
neural networks for multi-label text categorization,” in
Proc. Int. Joint Conf. Neural Networks, 2017, pp. 2377–2383.
[77] S. Poria, E. Cambria, D. Hazarika, N. Mazumder, A.
Zadeh, and L. Morency, “Context-dependent sentiment
analysis in user-generated videos,” in Proc. Annu. Meeting
Association Computational Linguistics, 2017, pp. 873–883.
[78] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.
Morency, “Tensor fusion network for multimodal sen-
timent analysis,” in Proc. Conf. Empirical Methods Natural
Language Processing, 2017, pp. 1114–1125.
[79] E. Tong, A. Zadeh, C. Jones, and L.-P. Morency,
“Combating human trafficking with deep multimodal
models,” arXiv Preprint, arXiv:1705.02735, 2017.
[80] I. Chaturvedi, E. Ragusa, P. Gastaldo, R. Zunino,
and E. Cambria, “Bayesian network based extreme learn-
ing machine for subjectivity detection,” J. Franklin Inst.,
vol. 355, no. 4, pp. 1780–1797, July 2018.
[81] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier,
“Language modeling with gated convolutional net-
works,” arXiv Preprint, arXiv:1612.08083, 2016.
[82] W. Yin, K. Kann, M. Yu, and H. Schütze, “Com-
parative study of CNN and RNN for natural language
processing,” arXiv Preprint, arXiv:1702.01923, 2017.
[83] S. Hochreiter and J. Schmidhuber, “Long short-term
memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780,
1997.
[84] F. A. Gers, J. Schmidhuber, and F. Cummins, “Learn-
ing to forget: Continual prediction with LSTM,” in Proc.
9th Int. Conf. Artificial Neural Networks, pp. 850–855, 1999.
[85] G. Lample, M. Ballesteros, S. Subramanian, K.
Kawakami, and C. Dyer, “Neural architectures for named
entity recognition,” arXiv Preprint, arXiv:1603.01360,
2016.
[86] A. Graves, “Generating sequences with recurrent
neural networks,” arXiv Preprint, arXiv:1308.0850,
2013.
[87] M. Sundermeyer, H. Ney, and R. Schlüter, “From
feedforward to recurrent LSTM neural networks for lan-
guage modeling,” IEEE Trans. Audio, Speech, Language
Process., vol. 23, no. 3, pp. 517–529, Mar. 2015.
[88] M. Sundermeyer, T. Alkhouli, J. Wuebker, and H.
Ney, “Translation modeling with bidirectional recurrent
neural networks,” in Proc. Conf. Empirical Methods Natural
Language Processing, 2014, pp. 14–25.
[89] R. Lowe, N. Pow, I. Serban, and J. Pineau, “The
ubuntu dialogue corpus: A large dataset for research in
unstructured multi-turn dialogue systems,” arXiv Pre-
print, arXiv:1506.08909, 2015.
[90] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan,
“Show and tell: A neural image caption generator,” in
Proc. IEEE Conf. Computer Vision and Pattern Recognition,
2015, pp. 3156–3164.
[91] O. Vinyals and Q. Le, “A neural conversational mod-
el,” arXiv Preprint arXiv:1506.05869, 2015.
[92] J. Li, M. Galley, C. Brockett, G. P. Spithourakis, J.
Gao, and B. Dolan, “A persona-based neural conversation
model,” arXiv Preprint, arXiv:1603.06155, 2016.
[93] M. Malinowski, M. Rohrbach, and M. Fritz, “Ask
your neurons: A neural-based approach to answering
questions about images,” in Proc. IEEE Int. Conf. Com-
puter Vision, 2015, pp. 1–9.
[94] D. Bahdanau, K. Cho, and Y. Bengio, “Neural ma-
chine translation by jointly learning to align and trans-
late,” arXiv Preprint, arXiv:1409.0473, 2014.
[95] A. M. Rush, S. Chopra, and J. Weston, “A neural
attention model for abstractive sentence summarization,”
arXiv Preprint, arXiv:1509.00685, 2015.
[96] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R.
Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend
and tell: Neural image caption generation with visual at-
tention,” in Proc. Int. Conf. Machine Learning, 2015, pp.
2048–2057.
[97] O. Vinyals, Ł. Kaiser, T. Koo, S. Petrov, I. Sutskever,
and G. Hinton, “Grammar as a foreign language,” in
Proc. Advances Neural Information Processing Systems, 2015,
pp. 2773–2781.
[98] O. Vinyals, M. Fortunato, and N. Jaitly, “Pointer
networks,” in Proc. Advances Neural Information Processing
Systems, 2015, pp. 2692–2700.
[99] R. Paulus, C. Xiong, and R. Socher, “A deep rein-
forced model for abstractive summarization,” arXiv Pre-
print, arXiv:1705.04304, 2017.
[100] Y. Wang, M. Huang, X. Zhu, and L. Zhao, “At-
tention-based LSTM for aspect-level sentiment classifica-
tion,” in Proc. Conf. Empirical Methods Natural Language
Processing, 2016, pp. 606–615.
[101] Y. Ma, H. Peng, and E. Cambria, “Targeted aspect-
based sentiment analysis via embedding commonsense
knowledge into an attentive LSTM,” in Proc. Association
Advancement Artificial Intelligence Conf., 2018, pp. 5876–
5883.
[102] R. Socher, B. Huval, C. D. Manning, and A. Y. Ng,
“Semantic compositionality through recursive matrix-
vector spaces,” in Proc. Joint Conf. Empirical Methods Natu-
ral Language Processing and Computational Natural Language
Learning, 2012, pp. 1201–1211.
[103] B. Taskar, C. Guestrin, and D. Koller, “Max-mar-
gin Markov networks,” in Proc. Advances Neural Informa-
tion Processing Systems, 2004, pp. 25–32.
[104] S. R. Bowman, C. Potts, and C. D. Manning,
“Recursive neural networks can learn logical semantics,”
arXiv Preprint, arXiv:1406.1827, 2014.
[105] K. S. Tai, R. Socher, and C. D. Manning, “Im-
proved semantic representations from tree-structured
long short-term memory networks,” arXiv Preprint,
arXiv:1503.00075, 2015.
[106] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer,
“Scheduled sampling for sequence prediction with recur-
rent neural networks,” in Proc. Advances Neural Information
Processing Systems, 2015, pp. 1171–1179.
[107] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba,
“Sequence level training with recurrent neural net-
works,” arXiv Preprint, arXiv:1511.06732, 2015.
[108] J. Li, W. Monroe, A. Ritter, M. Galley, J. Gao, and
D. Jurafsky, “Deep reinforcement learning for dialogue
generation,” arXiv Preprint, arXiv:1606.01541, 2016.
[109] R. J. Williams, “Simple statistical gradient-follow-
ing algorithms for connectionist reinforcement learning,”
Mach. Learn., vol. 8, no. 3–4, pp. 229–256, 1992.
[110] S. Young, M. Gašič, S. Keizer, F. Mairesse, J.
Schatzmann, B. Thomson, and K. Yu, “The hidden in-
formation state model: A practical framework for POM-
DP-based spoken dialogue management,” Comput. Speech
Lang., vol. 24, no. 2, pp. 150–174, June 2010.
[111] S. Young, M. Gašič, B. Thomson, and J. D. Wil-
liams, “POMDP-based statistical spoken dialog systems:
A review,” Proc. IEEE, vol. 101, no. 5, pp. 1160–1179,
2013.
[112] P.-h. Su, V. David, D. Kim, T.-h. Wen, and S.
Young, “Learning from real users: Rating dialogue suc-
cess with neural networks for reinforcement learning in
spoken dialogue systems,” in Proc. Interspeech Conf., 2015,
pp. 2007–2011.
[113] P.-H. Su, M. Gasic, N. Mrksic, L. Rojas-Barahona,
S. Ultes, D. Vandyke, T. Wen, and S. Young, “On-line
active reward learning for policy optimisation in spoken
dialogue systems,” arXiv Preprint, arXiv:1605.07669,
2016.
[114] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio,
“Generative adversarial nets,” in Proc. Advances Neural In-
formation Processing Systems, 2014, pp. 2672–2680.
[115] J. Li, W. Monroe, T. Shi, A. Ritter, and D. Jurafsky,
“Adversarial learning for neural dialogue generation,”
arXiv Preprint, arXiv:1701.06547, 2017.
[116] R. Kiros, Y. Zhu, R. R. Salakhutdinov, R. Zemel,
R. Urtasun, A. Torralba, and S. Fidler, “Skip-thought
vectors,” in Proc. Advances Neural Information Processing
Systems, 2015, pp. 3294–3302.
[117] A. M. Dai and Q. V. Le, “Semi-supervised sequence
learning,” in Proc. Advances Neural Information Processing
Systems, 2015, pp. 3079–3087.
[118] D. E. Rumelhart, G. E. Hinton, and R. J. Williams,
“Learning internal representations by error propagation,”
DTIC Document, Tech. Rep, 1985.
[119] D. P. Kingma and M. Welling, “Auto-encoding
variational bayes,” arXiv Preprint, arXiv:1312.6114,
2013.
[120] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R.
Jozefowicz, and S. Bengio, “Generating sentences from a
continuous space,” arXiv Preprint, arXiv:1511.06349, 2015.
[121] Y. Zhang, Z. Gan, and L. Carin, “Generating text
via adversarial training,” in Proc. Neural Information Pro-
cessing Systems Workshop Adversarial Training, 2016.
[122] Z. Hu, Z. Yang, X. Liang, R. Salakhutdinov, and E.
P. Xing, “Controllable text generation,” arXiv Preprint,
arXiv:1703.00955, 2017.
[123] L. Yu, W. Zhang, J. Wang, and Y. Yu, “Seqgan: se-
quence generative adversarial nets with policy gradient,”
in Proc. Association Advancement Artificial Intelligence Conf.,
2017, pp. 2852–2858.
[124] S. Rajeswar, S. Subramanian, F. Dutil, C. Pal, and
A. Courville, “Adversarial generation of natural lan-
guage,” arXiv Preprint, arXiv:1705.10929, 2017.
[125] J. Giménez and L. Marquez, “Fast and accurate
part-of-speech tagging: The SVM approach revisited,”
Recent Adv. Natural Lang. Process., pp. 153–162, 2004.
[126] Z. Huang, W. Xu, and K. Yu, “Bidirectional
LSTM-CRF models for sequence tagging,” arXiv Pre-
print, arXiv:1508.01991, 2015.
[127] D. Andor, C. Alberti, D. Weiss, A. Severyn, A.
Presta, K. Ganchev, S. Petrov, and M. Collins, “Glob-
ally normalized transition-based neural networks,” arXiv
Preprint, arXiv:1603.06042, 2016.
[128] A. Kumar, O. Irsoy, P. Ondruska, M. Iyyer, J. Brad-
bury, I. Gulrajani, V. Zhong, R. Paulus, and R. Socher,
“Ask me anything: Dynamic memory networks for natu-
ral language processing,” in Proc. Int. Conf. Machine Learn-
ing, 2016, pp. 1378–1387.
[129] D. Chen and C. D. Manning, “A fast and accurate
dependency parser using neural networks,” in Proc. Conf.
Empirical Methods Natural Language Processing, 2014, pp.
740–750.
[130] D. Weiss, C. Alberti, M. Collins, and S. Petrov,
“Structured training for neural network transition-based
parsing,” arXiv Preprint, arXiv:1506.06158, 2015.
[131] C. Dyer, M. Ballesteros, W. Ling, A. Matthews,
and N. A. Smith, “Transition-based dependency pars-
ing with stack long short-term memory,” arXiv Preprint,
arXiv:1505.08075, 2015.
[132] H. Zhou, Y. Zhang, C. Cheng, S. Huang, X. Dai,
and J. Chen, “A neural probabilistic structured-predic-
tion method for transition-based natural language pro-
cessing,” J. Artif. Intell. Res., vol. 58, pp. 703–729, Mar.
2017.
[133] S. Petrov, L. Barrett, R. Thibaux, and D. Klein,
“Learning accurate, compact, and interpretable tree an-
notation,” in Proc. 21st Int. Conf. Computational Linguistics,
2006, pp. 433–440.
[134] M. Zhu, Y. Zhang, W. Chen, M. Zhang, and J.
Zhu, “Fast and accurate shift-reduce constituent pars-
ing,” in Proc. Annu. Meeting Association Computational Lin-
guistics, 2013, pp. 434–443.
[135] J. Weston, S. Chopra, and A. Bordes, “Memory net-
works,” arXiv Preprint, arXiv:1410.3916, 2014.
[136] S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus,
“End-to-end memory networks,” in Proc. Advances Neural
Information Processing Systems, 2015, pp. 2440–2448.
[137] C. Xiong, S. Merity, and R. Socher, “Dynamic
memory networks for visual and textual question an-
swering,” in Proc. Int. Conf. Machine Learning, 2016, pp.
2397–2406.
[138] A. Passos, V. Kumar, and A. McCallum, “Lexicon
infused phrase embeddings for named entity resolution,”
arXiv Preprint, arXiv:1404.5367, 2014.
[139] J. P. Chiu and E. Nichols, “Named entity recogni-
tion with bidirectional LSTM-CNNs,” arXiv Preprint,
arXiv:1511.08308, 2015.
[140] G. Luo, X. Huang, C. Lin, and Z. Nie, “Joint
named entity recognition and disambiguation,” in Proc.
Conf. Empirical Methods Natural Language Processing, 2015,
pp. 879–880.
[141] E. Strubell, P. Verga, D. Belanger, and A. McCal-
lum, “Fast and accurate sequence labeling with iterated
dilated convolutions,” arXiv Preprint, arXiv:1702.02098,
2017.
[142] O. Täckström, K. Ganchev, and D. Das, “Efficient
inference and structured learning for semantic role label-
ing,” Trans. Assoc. Comput. Linguistics, vol. 3, pp. 29–41,
Jan. 2015.
[143] J. Zhou and W. Xu, “End-to-end learning of se-
mantic role labeling using recurrent neural networks,” in
Proc. Annu. Meeting Association Computational Linguistics,
2015, pp. 1127–1137.
[144] L. He, K. Lee, M. Lewis, and L. Zettlemoyer,
“Deep semantic role labeling: What works and what’s
august 2018 | ieee Computational intelligenCe magazine
next,” in Proc. Annu. Meeting Association for Computational
Linguistics, 2017, pp. 473–483.
[145] Q. Le and T. Mikolov, “Distributed representations
of sentences and documents,” in Proc. 31st Int. Conf. Ma-
chine Learning, 2014, pp. 1188–1196.
[146] L. Yu, J. Wang, K. R. Lai, and X. Zhang, “Re-
fining word embeddings for sentiment analysis,” in Proc.
Conf. Empirical Methods Natural Language Processing, 2017,
pp. 545–550.
[147] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. No-
rouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K.
Macherey, et al., “Google’s neural machine translation
system: Bridging the gap between human and machine
translation,” arXiv Preprint, arXiv:1609.08144, 2016.
[148] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y.
N. Dauphin, “Convolutional sequence to sequence learn-
ing,” arXiv Preprint, arXiv:1705.03122, 2017.
[149] A. Vaswani, N. Shazeer, N. Parmar, and J. Uszko-
reit, “Attention is all you need,” arXiv Preprint, arX-
iv:1706.03762, 2017.
[150] R. K. Srivastava, K. Greff, and J. Schmidhuber,
“Training very deep networks,”in Proc. Advances Neural
Information Processing Systems, 2015, pp. 2377–2385.
[151] B. Pang and L. Lee, “Seeing stars: Exploiting class
relationships for sentiment categorization with respect to
rating scales,” in Proc. 43rd Annu. Meeting Association Com-
putational Linguistics, 2005, pp. 115–124.
[152] A. Zadeh, R. Zellers, E. Pincus, and L. Morency,
“Multimodal sentiment intensity analysis in videos: Fa-
cial gestures and verbal messages,” IEEE Intell. Syst., vol.
31, no. 6, pp. 82–88, Nov. 2016.
[153] A. Fader, L. S. Zettlemoyer, and O. Etzioni, “Para-
phrase-driven learning for open question answering,” in
Proc. Annu. Meeting Association Computational Linguistics,
2013, pp. 1608–1618.
[154] A. Bordes, J. Weston, and N. Usunier, “Open
question answering with weakly supervised embedding
models,” in Proc. Joint European Conf. Machine Learning and
Knowledge Discovery Databases, 2014, pp. 165–180.
[155] A. Ritter, C. Cherry, and W. B. Dolan, “Data-driv-
en response generation in social media,” in Proc. Conf.
Empirical Methods Natural Language Processing, 2011, pp.
583–593.
[156] A. Sordoni, M. Galley, M. Auli, C. Brockett,
Y. Ji, M. Mitchell, J.-Y. Nie, J. Gao, and B. Dolan, “A
neural network approach to context-sensitive genera-
tion of conversational responses,” arXiv Preprint, arX-
iv:1506.06714, 2015.
[157] J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan,
“A diversity-promoting objective function for neural
conversation models,” arXiv Preprint, arXiv:1510.03055,
2015.
[158] J. Dodge, A. Gane, X. Zhang, A. Bordes, S. Cho-
pra, A. Miller, A. Szlam, and J. Weston, “Evaluating
prerequisite qualities for learning end-to-end dialog sys-
tems,” arXiv Preprint, arXiv:1511.06931, 2015.
[159] X. Zhou, D. Dong, H. Wu, S. Zhao, D. Yu, H.
Tian, X. Liu, and R. Yan, “Multi-view response selec-
tion for human-computer conversation,” in Proc. Conf.
Empirical Methods Natural Language Processing, 2016, pp.
372–381.
[160] P. Koehn, F. J. Och, and D. Marcu, “Statistical
phrase-based translation,” in Proc. Conf. North American
Chapter Association Computational Linguistics, 2003, pp.
48–54.
[161] K. Papineni, S. Roukos, T. Ward, and W. Zhu,
“Bleu: A method for automatic evaluation of machine
translation,” in Proc. 40th Annu. Meeting Association Com-
putational Linguistics, 2002, pp. 311–318.
[162] I. V. Serban, A. Sordoni, Y. Bengio, A. C. Cour-
ville, and J. Pineau, “Building end-to-end dialogue sys-
tems using generative hierarchical neural network mod-
els,” in Proc. Association Advancement Artificial Intelligence
Conf., 2016, pp. 3776–3784.
[163] Y. LeCun, Y. Bengio, and G. Hinton, “Deep
learning,” Nature, vol. 521, no. 7553, pp. 436–444, May
2015.
[164] T. Baltrušaitis, C. Ahuja, and L. Morency, “Multi-
modal machine learning: A survey and taxonomy,” arXiv
Preprint, arXiv:1705.09406, 2017.
75
Conference Bernadette Bouchon-Meunier
Calendar University Pierre et Marie Curie,
FRANCE

* Denotes a CIS-Sponsored Conference

∆ Denotes a CIS Technical
Co-Sponsored Conference

∆ 13th International Workshop * 2018 IEEE Symposium Series
on Semantic and Social Media on Computational Intelligence
Adaptation and Personalization (IEEE SSCI 2018)
(SMAP 2018) November 18-21, 2018
September 6-7, 2018 Place: Bangalore, India
Place: Zaragoza, Spain General Co-Chairs: Sundaram Suresh
General Chairs: Sergio Ilarri, and Koshy George
Fernando Bobillo, Raquel Trillo-Lado, Website: http://ieee-ssci2018.org
and Martín López-Nores
Website: http://smap2018.unizar.es * 2019 IEEE Conference on
Computational Intelligence
* 2018 IEEE International for Financial Engineering and
Conference on Data Science Economics (IEEE CIFEr 2019)
and Advanced Analytics May 4-5, 2019
(IEEE DSAA 2018) Place: Shenzhen, China
October 1-4, 2018 General Chairs: Hisao Ishibuchi
Place: Turin, Italy and Dongbin Zhao
General Chairs: Francesco Bonchi Website: TBA
and Foster Provost
Website: https://dsaa2018.isi.it * 2019 IEEE Congress on Evolutionary
Computation (IEEE CEC 2019)
* 2018 IEEE Smart World Congress June 10-13, 2019
(IEEE SWC 2018) Place: Wellington, New Zealand
October 8-12, 2018 General Co-Chairs: Mengjie Zhang
Place: Guangzhou, China and Kay Chen Tan
General Chairs: Guojun Wang Website: http://www.cec2019.org
and Yew Soon Ong
Website: http://www.smart-world.org/ * 2019 IEEE International
2018/ Conference on Fuzzy Systems
(FUZZ-IEEE 2019)
* 2018 IEEE Latin American June 23-26, 2019
Conference on Computational Place: New Orleans, USA
Intelligence (2018 IEEE LA-CCI) General Chairs: Timothy C. Havens
November 7-9, 2018 and James M. Keller
Place: Guadalajara, Mexico Website: http://www.fuzzieee.org
General Chairs: Alma Y. Alanis
and Marco A. Perez-Cisneros * 2019 IEEE Conference on
Website: http://la-cci.org Computational Intelligence in
Bioinformatics and Computational
Biology (IEEE CIBCB 2019)
July 24-27, 2019
Place: Siena, Italy
Digital Object Identifier 10.1109/MCI.2018.2840739 General Chair: Giuseppe Nicosia
Date of publication: 18 July 2018 Website: TBA
* The 9th Joint IEEE International
Conference of Developmental
Learning and Epigenetic Robotics
(IEEE ICDL-EpiRob 2019)
August 19-22, 2019
Place: Oslo, Norway
General Chairs: Jim Torresen
and Kerstin Dautenhahn
Website: TBA
* 2019 IEEE International
Conference on Games
(IEEE CoG 2019)
August 20-23, 2019
Place: London, United Kingdom
General Chairs: Diego Perez Liebana
and Sanaz Mostaghim
Website: TBA
* 2019 IEEE Latin American
Conference on Computational
Intelligence (2019 IEEE LA-CCI)
November 11-15, 2019
Place: Guayaquil, Ecuador
General Chair: Otilia Alejandro
Website: http://la-cci.org
* 2019 IEEE Symposium
Series on Computational
Intelligence (IEEE SSCI 2019)
December 6-9, 2019
Place: Xiamen, China
General Chair: Tingwen Huang
Website: http://ssci2019.org
* 2020 IEEE World Congress
on Computational Intelligence
(IEEE WCCI 2020)
July 19-24, 2020
Place: Glasgow, UK
General Chairs: Amir Hussain,
Marios M. Polycarpou, and Xin Yao
Website: TBA

76 IEEE ComputatIonal IntEllIgEnCE magazInE | auguSt 2018

 New Journal Title

al
on
ati
ut a
p c ts l,
om o
 e
C t r efl nica
s on ed nge tech
n m a
c tio rena le ch s of
a
r a ns was he tit ll formes.
E E T mes es. T ass a gam
a
8 , IE in G Gam omp h on
2 01 AI on enc earc
a ry and ns e to res
a nu nce ctio cop ring
J a
In ellige ans the s inee
r
Int E T g of eng m es on
IEE enin c an
d e lco pers r
l w f pa e fo cial
wid entifi a
ur ons o genc artifi
n
c i j o
s e si lli r
Th mis l inte es fo an–
s s ub cia am um on, nd
a me a rtifi es, g ce, h racti nal a e
n G lity a m gen inte atio war
g t
n s o h-qu ,
a
n t elli uter educ , sof es,
i
c tio l hig tific o mp ics, mes gam in
n sa ina cien ring c ph ga in
r a s t i ng
a i g s g iou rin g u
E Tr s or ing inee e r e m p nd
a me
e r g s gin e o a g s.
IEE blish cove d en s.
 i u s e c al ty,
u i i c
ge
l en ectiv virt real r top
pu icles al, an ame o
r t i c f g a n T aff mes, ted othe
a hn o uli ga gmen and
tec ects ief
: J
p h au sign,
as C

 r - in- de
ito
Ed

Digital Object Identifier 10.1109/MCI.2018.2840758

 2019 IEEE Congress on Evolutionary Computation
10–13 June 2019, Wellington, New Zealand

IEEE CEC 2019

Wellington New Zealand

Advisory Board
Hussein Abbass, Australia
Kalyanmoy Deb, USA
David Fogel, USA
Kwong Tak Wu Sam, Hong Kong SAR
Simon Lucas, UK
Zbigniew Michalewicz, Australia
Xin Yao, UK
The IEEE Congress on Evolutionary Computation (IEEE CEC) is a world-class event in the field of Evolutionary Gary Yen, USA
Computation. It provides a forum to bring together researchers and practitioners from all over the world to General Co-Chairs
present and discuss their research findings on Evolutionary Computation. Mengjie Zhang, New Zealand
Kay Chen Tan, Hong Kong SAR
Program Chair
IEEE CEC 2019 will be held in Wellington, New Zealand. Wellington is known as the ‘Coolest Little Capital’. It is
Carlos A. Coello Coello, Mexico
famous for a vibrant creative culture fuelled by events and great food. Wellington offers a wide range of Technical Co-Chairs
cosmopolitan amenities in downtown that is safe, clean, and pedestrian friendly. Jürgen Branke, UK
Oscar Cordón, Spain
Call for Papers Hisao Ishibuchi, Japan
Jing Liu, China
Papers for IEEE CEC 2019 should be submitted electronically through the Congress website at Gabriela Ochoa, UK
www.cec2019.org, and will be refereed by experts in the fields and ranked based on the criteria of Dipti Srinivasan, Singapore
originality, significance, quality and clarity. Plenary Talk Chair
Yaochu Jin, UK
Call for Special Sessions Special Session Chair
Chuan-Kang Ting, Taiwan
Special session proposals are invited to IEEE CEC 2019. Special session proposals should include the title, aim Tutorial Chair
and scope (including a list of main topics), and the names of the organizers of the special session, together Xiaodong Li, Australia
with a short biography of all organizers. A list of potential contributors will be very helpful. All special Competition Chair
sessions proposals should be submitted to the Special Session Chair: Prof Chuan-Kang Ting Jialin Liu, UK
(ckting@pme.nthu.edu.tw). Workshop Chair
Handing Wang, UK
Submission Chair
Call for Tutorials Huanhuan Chen, China
IEEE CEC 2019 solicits proposal for tutorials covering specific topics in Evolutionary Computation. If you are Sponsorship Chair
interested in proposing a tutorial, would like to recommend someone who might be interested, or have Andy Song, Australia
Poster Chair
questions about tutorials, please contact the Tutorial Chair: Prof Xiaodong Li (xiaodong.li@rmit.edu.au).
Kai Qin, Australia
Publicity Co-Chair
Call for Competitions Stefano Cagnoni, Italy
Competitions will be held as part of the Congress. Prospective competition organizers are invited to submit Anna I Esparcia-Alcázar, Spain
their proposals to the Competition Chair: Dr Jialin Liu ( jialin.liu@qmul.ac.uk). Emma Hart, UK
Bin Hu, Austria
Sanza Mostaghim, Germany
Call for Workshops
Yew Soon Ong, Singapore
Workshops will be held to provide participants with the opportunity to present and discuss novel research Jun Zhang, China
ideas on active and emerging topics in Evolutionary Computation. Prospective workshop organizers are Finance Chair
invited to submit their proposals to the Workshop Chair: Dr Handing Wang (handing.wang@surrey.ac.uk). Bing Xue, New Zealand
Local Organising Co-Chairs
Will Browne, New Zealand
Important Dates Hui Ma, New Zealand
Registration Chair
26 26 7 7 31 Aaron Chen, New Zealand
October November January March March Proceedings Chair
2018 2018 2019 2019 2019 Yi Mei, New Zealand
Web Masters
Special Session Proposal Competition Proposal Paper Submission & Notification Deadline Camera-ready & Harith Al-Sahaf
Deadline Deadline Workshop Proposal & Early Registration Deadline
Tutorial Proposal Deadline Yiming Peng
Qi Chen
Sponsored by Find Us At
www.cec2019.org
TE WHARE WANANGA O TE UPOKO O TE IKA A MAUI

VICTORIA
UNIVERSITY OF WELLINGTON
admin@cec2019.org
Digital Object Identifier 10.1109/MCI.2018.2840759