Organising Committee

General Chairs
IEEE Symposium Series on Computational Intelligence
Keeley Crockett, UK
Sanaz Mostaghim, Germany IEEE SSCI 2021
Program Chairs
Alice Smith, USA
December 4th - 7th 2021, Orlando, Florida, USA
Carlos A. Coello Coello, Mexico
IEEE SSCI is an established flagship annual international series of symposia on
Finance Chair
computational intelligence sponsored by the IEEE Computational Intelligence Society. IEEE
Piero Bonissone, USA
SSCI 2021 promotes and stimulates discussion on the latest theory, algorithms,
Publications Chairs
applications and emerging topics on computational intelligence across several unique
Dipti Srinivasan, Singapore
Anna Wilbik, Netherlands symposia, providing an opportunity for cross-fertilization of research and future
collaborations. For more information visit http://attend.ieee.org/ssci-2021
Conflict of Interest Chair
Marley Vellasco, Brazil
Registration Chairs
IEEE Computational Intelligence Symposia on
Julia Cheung, Taiwan Adaptive Dynamic Programming and Explainable Data Analytics in Computational
Steven Corns, USA
Reinforcement Learning (IEEE ADPRL) Intelligence (IEEE EDACI)
Keynote Chairs CI in Agriculture (IEEE CIAg) Evolutionary Learning (IEEE EL)
Bernadette Bouchon-Meunier,
CI for Brain Computer Interfaces (IEEE Evolutionary Neural Architecture Search and
France
Gary Yen, USA CIBCI) Applications (IEEE ENASA)
Manuel Roveri, Italy Computational Intelligence in Big Data (IEEE Evolutionary Scheduling and Combinatorial
Special Session Chair CIBD) Optimisation (IEEE ESCO)
Marde Helbig, Australia CI in Biometrics and Identity Management Ethical, Social and Legal Implications of
Tutorial Chairs (IEEE CIBIM) Artificial Intelligence (IEEE ETHAI)
Sansanee Auephanwiriyakul, CI in Control and Automation (IEEE CICA) CI in Feature Analysis, Selection and
Thailand CI in Healthcare and E-health (IEEE Learning in Image and Pattern Recognition
Daniel Ashlock, Canada CICARE) (IEEE FASLIP)
Manuel Roveri, Italy CI in Cyber Security (IEEE CICS) Foundations of CI (IEEE FOCI)
Submission Chair CI in Data Mining (IEEE CIDM) Evolvable Systems (IEEE ICES)
Nicolò Navarin, Italy CI in Dynamic and Uncertain Environments Immune Computation (IEEE IComputation)
Local Organizing Chair (IEEE CIDUE) Intelligent Agents (IEEE IA)
Zhen Ni, USA CI and Ensemble Learning (IEEE CIEL) Multi-agent System Coordination and
Travel Grant Chair CI for Engineering Solutions (IEEE CIES) Optimization (IEEE MASCO)
Pauline Haddow, Norway CI for Human-like Intelligence (IEEE CIHLI) Model-Based Evolutionary Algorithms (IEEE
Webmaster Chair CI in IoT and Smart Cities (IEEE CIIoT) MBEA)
Jen-Wei Huang CI for Multimedia Signal and Vision Multicriteria Decision-Making (IEEE MCDM)
Processing (IEEE CIMSIVP) Nature-Inspired Computation in Engineering
Whova Chair
Albert Lam, Hong Kong CI in Remote Sensing (IEEE CIRS) (IEEE NICE)
CI for Security and Defence Applications Robotic Intelligence in Informationally
Conference Activities Chair
Bing Xue, New Zealand (IEEE CISDA) Structured Space (IEEE RiiSS)
Deep Learning (IEEE DL) Cooperative Metaheuristics (IEEE SCM)
Social Event Chair
Jo-Ellen Synder, USA Evolving and Autonomous Learning Systems Differential Evolution (IEEE SDE)
(IEEE EALS) Swarm Intelligence Symposium (IEEE SIS)
Publicity Chairs
Jialin Liu, China
Joao Carvalho, Portugal
Matt Garrat, Australia
Important Dates Call for Papers
Special Session Proposals: May 28th 2021 Papers for IEEE SSCI 2021 should be submitted
Tutorial Proposals: May 28th 2021 electronically using the conference website: http://
Paper Submissions: August 6th 2021 attend.ieee.org/ssci-2021 and will be reviewed by
experts in the fields and ranked based on the crite-
Paper Acceptance: September 17th 2021
ria of originality, significance, quality and clarity.
Camera ready Paper: October 15th 2021

Call for Tutorials Call for Special Sessions

IEEE SSCI 2021 will feature tutorials covering fun-
damental and advanced topics in computational in-
telligence. A tutorial proposal should include title,
short introduction to the topic, an outline of the tu-
torial, length of the tutorial, level and names, affilia-
Special session proposals should include title, a de-
scription of the scope, a list of topics covered, a list
of potential contributors, and names, affiliations,
websites, and bios of special session organizers.

Digital Object Identifier 10.1109/MCI.2021.3084497

 Volume 16 Number 3 ❏ August 2021
www.ieee-cis.org

Features
10 Evolutionary Multi-Objective Model
Compression for Deep Neural Networks
by Zhehui Wang, Tao Luo, Miqing Li, Joey Tianyi Zhou,
Rick Siow Mong Goh, and Liangli Zhen
22 
Fast and Unsupervised Neural Architecture
Evolution for Visual Representation Learning
by Song Xue, Hanlin Chen, Chunyu Xie, Baochang Zhang,
Xuan Gong, and David Doermann
33 
Self-Supervised Representation Learning
for Evolutionary Neural Architecture Search
by Chen Wei,Yiping Tang, Chuang Niu, Haihong Hu,Yue Wang, and Jimin Liang
50 
Forecasting Wind Speed Time Series
Via Dendritic Neural Regression
on the cover
©SHUTTERSTOCK.COM/ANDREY SUSLOV
by Junkai Ji, Minhui Dong, Qiuzhen Lin, and Kay Chen Tan
67 
A Self-Adaptive Mutation Neural Architecture
Search Algorithm Based on Blocks
by Yu Xue,Yankang Wang, Jiayu Liang, and Adam Slowik

Promoting Sustainable Forestry

SFI-01681

IEEE Computational Intelligence Magazine (ISSN

Columns
1556-603X) is published quarterly by The Institute of
Electrical and Electronics Engineers, Inc. Headquarters:
79 Application Notes
3 Park Avenue, 17th Floor, New York, NY 10016-5997,  Multi-View Feature Construction Using Genetic Programming
U.S.A. +1 212 419 7900. Responsibility for the contents for Rolling Bearing Fault Diagnosis
rests upon the authors and not upon the IEEE, the Soci- 
by Bo Peng,Ying Bi, Bing Xue, Mengjie Zhang, and Shuting Wan
ety, or its members. The magazine is a membership benefit
of the IEEE Computational Intelligence Society, and sub-
scriptions are included in Society fee. Replacement copies
for members are available for US$20 (one copy only).
Nonmembers can purchase individual copies for
US$220.00. Nonmember subscription prices are available
on request. Copyright and Reprint Permissions:
Abstracting is permitted with credit to the source. Librar-
ies are permitted to photocopy beyond the limits of the
U.S. Copyright law for private use of patrons: 1) those
post-1977 articles that carry a code at the bottom of the Departments
first page, provided the per-copy fee is paid through the 2 Editor’s Remarks 8  Guest Editorial
Copyright Clearance Center, 222 Rosewood Drive, Dan-
vers, MA 01970, U.S.A.; and 2) pre-1978 articles without 3 President’s Message by Yanan Sun, Mengjie Zhang,
fee. For other copying, reprint, or republication permis-
 by Bernadette and Gary G.Yen
sion, write to: Copyrights and Permissions Department,
IEEE Service Center, 445 Hoes Lane, Piscataway NJ Bouchon-Meunier 96 Conference Calendar
08854 U.S.A. Copyright © 2021 by The Institute of Elec-
5 Publication Spotlight by Marley Vellasco
trical and Electronics Engineers, Inc. All rights reserved.
Periodicals ­postage paid at New York, NY and at addi-  by Haibo He, Jon Garibaldi,
tional mailing offices. Postmaster: Send address changes to Carlos A. Coello Coello,
IEEE Computational Intelligence Maga­zine, IEEE, 445
Hoes Lane, Piscataway, NJ 08854-1331 U.S.A. PRINT- Julian Togelius,Yaochu Jin,
ED IN U.S.A. Canadian GST #125634188. Yew Soon Ong, and Hussein Abbass

Digital Object Identifier 10.1109/MCI.2021.3056259 AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 1
CIM Editorial Board Chuan-Kang Ting Editor’s
Editor’s
Editor-in-Chief
Chuan-Kang Ting
National Tsing Hua University, TAIWAN Remarks
National Tsing Hua University
Department of Power Mechanical Engineering
No. 101, Section 2, Kuang-Fu Road
Hsinchu 30013, TAIWAN
(Phone) +886-3-5742611
(Email) ckting@pme.nthu.edu.tw

Founding Editor-in-Chief
Gary G.Yen, Oklahoma State University, USA
The Evolution of Neural Networks
Past Editors-in-Chief
Kay Chen Tan, Hong Kong Polytechnic
University, HONG KONG
Hisao Ishibuchi, Southern University of Science and
Technology, CHINA

Editors-At-Large
Piero P. Bonissone, Piero P Bonissone Analytics
LLC, USA
David B. Fogel, Natural Selection, Inc., USA
Vincenzo Piuri, University of Milan, ITALY
Marios M. Polycarpou, University of Cyprus,
The focus on evolutionary neural architecture search in this
issue has driven me to ponder the evolution of biological
neural networks, or rather, the human brain. Researchers in
neuroanatomy found that the cognitive and mental development
of humans is attributed to the increase of our brain size through-
CYPRUS
Jacek M. Zurada, University of Louisville, USA out evolution. Yet bigger is not always better. It was also discov-
ered that, once the brain reaches a certain size, further growth
Associate Editors
José M. Alonso, University of Santiago de will only render the brain less efficient. In addition, the brain is limited and affect-
Compostela, SPAIN ed by its inherent architecture and signal processing time. These discoveries in neu-
Erik Cambria, Nanyang Technological University,
SINGAPORE roscience are interestingly analogous to the advances in neural networks and
Liang Feng, Chongqing University, CHINA evolutionary computation. Stacking of perceptrons empowers artificial neural net-
Barbara Hammer, Bielefeld University, GERMANY
Eyke Hüllermeier, University of Munich, GERMANY works to solve complex problems but detracts from their efficiency. Indeed, the
Sheng Li, University of Georgia, USA human brain evolution and the artificial neural network evolution both face the
Hsuan-Tien Lin, National Taiwan University,
TAIWAN trade-off between capacity and efficiency. The workings of nature truly give us a
Hongfu Liu, Brandeis University, USA lot to mull over.
Zhen Ni, Florida Atlantic University, USA
Yusuke Nojima, Osaka Prefecture University, JAPAN This issue includes five Features articles. The first article proposes an evolutionary
Nelishia Pillay, University of Pretoria, SOUTH AFRICA multi-objective model compression approach to simultaneously optimize the model
Danil Prokhorov, Toyota R&D, USA
Kai Qin, Swinburne University of Technology, size and accuracy of a deep neural network. The second and third articles adopt the
AUSTRALIA notion of self-supervised learning in the neural architecture evolution, resulting in
Rong Qu, University of Nottingham, UK
Ming Shao, University of Massachusetts state-of-the-art performance. To improve the accuracy of wind speed forecasting, the
Dartmouth, USA fourth article uses an evolutionary algorithm to optimize the architecture of dendritic
Vincent S. Tseng, National Chiao Tung University,
TAIWAN neural regression. The fifth article presents a self-adaptive mutation strategy for block-
Kyriakos G.Vamvoudakis, Georgia Tech, USA based evolutionary neural architecture search on convolutional neural networks.
Nishchal K.Verma, Indian Institute of Technology
Kanpur, INDIA In the Columns, the article proposes a multi-view feature construction method
Handing Wang, Xidian University, CHINA based on genetic programming and ensemble techniques. The approach can automati-
Dongrui Wu, Huazhong University of Science and
Technology, CHINA cally generate high-level features from multiple views; moreover, a new fitness func-
Bing Xue, Victoria University of Wellington, tion is developed to improve the discriminability of these features. Experimental
NEW ZEALAND
results show the effectiveness of the proposed approach in rolling bearing fault diag-
IEEE Periodicals/ nosis with a limited amount of data.
Magazines Department
Managing Editor, Mark Gallaher
We hope you will enjoy the articles in this issue. If you have any suggestions or
Senior Managing Editor, Geri Krolin-Taylor feedback for this magazine, please do not hesitate to contact me at ckting@pme.nthu
Senior Art Director, Janet Dudar .edu.tw.
Associate Art Director, Gail A. Schnitzer
Production Coordinator, Theresa L. Smith
Director, Business Development—
Media & Advertising, Mark David
Advertising Production Manager,
Felicia Spagnoli
Production Director, Peter M. Tuohy
Editorial Services Director, Kevin Lisankie
Senior Director, Publishing Operations,
Dawn Melley

IEEE prohibits discrimination, harassment, and bullying.

For more information, visit http://www.ieee.org/web/
abou-tus/whatis/policies/p9-26.html.
Digital Object Identifier 10.1109/MCI.2021.3084387
Digital Object Identifier 10.1109/MCI.2021.3056260 Date of current version: 15 July 2021

2 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

 President’s CIS Society Officers
Message
Bernadette Bouchon-Meunier
CNRS—Sorbonne Université, President – Bernadette Bouchon-Meunier,
FRANCE Sorbonne Université, FRANCE
President Elect – Jim Keller, University of
Missouri, USA
Vice President – Conferences – Marley M. B. R.
Vellasco, Pontifical Catholic University of
Rio de Janeiro, BRAZIL
Vice President – Education – Pau-Choo (Julia)
Chung, National Cheng Kung University,TAIWAN
Ethics, Diversity and Consciousness in AI Vice President – Finances – Pablo A. Estévez,
University of Chile, CHILE
Vice President – Members Activities – Sanaz
Mostaghim, Otto von Guericke University of
Magdeburg, GERMANY

E
Vice President – Publications – Kay Chen Tan,
thics in Artificial Intelligence (AI) is discussed everywhere, Hong Kong Polytechnic University, HONG KONG
in governmental circles as well as scientific forums. It is very Vice President – Technical Activities –
often associated with the two widely used concepts of Luis Magdalena, Universidad Politecnica
de Madrid, SPAIN
diversity and eXplainable Artificial Intelligence (XAI). The latter
was promoted by DARPA1, and it is expected to propose meth- Publication Editors
ods preserving the rights of users to understand how the AI sys- IEEE Transactions on Neural Networks
tems work and why decisions are made. Computational and Learning Systems
Haibo He, University of Rhode Island, USA
Intelligence methods have much to contribute to this effort to ensure that AI sys- IEEE Transactions on Fuzzy Systems
tems are ethical. Jon Garibaldi, University of Nottingham, UK
In July 2020, the European Union has published an Assessment List for Trustwor- IEEE Transactions on Evolutionary Computation
thy Artificial Intelligence (ALTAI)2 covering a wide range of ethical issues. Among the Carlos A. Coello Coello, CINVESTAV-IPN,
MEXICO
requirements listed, we find transparency, that encompasses the traceability of the data IEEE Transactions on Games
and model used to make a decision, the explainability of decisions and the communi- Julian Togelius, New York University, USA
cation with the user. Transparency mainly corresponds to the general approach of XAI IEEE Transactions on Cognitive and
Developmental Systems
and Computational Intelligence has much to do to participate in the effort of trans-
Yaochu Jin, University of Surrey, UK
parency of XAI, by means of the various methods of automatic generation of explana- IEEE Transactions on Emerging Topics in
tions, the use of counterfactuals, the help of visualization or the construction of fuzzy Computational Intelligence
systems providing interpretability and expressiveness, for example. Yew Soon Ong, Nanyang Technological University,
SINGAPORE
Another ALTAI requirement sounds familiar to us, dealing with diversity, non-dis-
IEEE Transactions on Artificial Intelligence
crimination and avoidance of unfair bias. Such concerns are at the heart of the IEEE Hussein Abbass, University of New South Wales,
Computational Intelligence’s actions, which is very committed to their respect in its AUSTRALIA
scientific and associative life. It is therefore natural to also consider these properties as
Administrative Committee
essential for all algorithms and products that are parts of AI systems, not only in the
collection of data and choice of attributes leading to a decision, but also in the ability Term ending in 2021:
David Fogel, Natural Selection, Inc., USA
of all categories of users to interact with the AI system regardless of their characteris- Barbara Hammer, Bielefeld University,
tics. Privacy and data protection is another aspect of the respect of human users. GERMANY
Other requirements focus more on the cognitive components of the construction Yonghong (Catherine) Huang,
of AI systems, the effect they can have on human users, the possible governance of McAfee LLC, USA
Xin Yao, Southern University of Science
human agents in AI systems and the integration of AI systems in the human society. and Technology, CHINA
The technical robustness of the AI system, its reliability, its security and its capacity to Jacek M. Zurada, University of Louisville, USA
identify and mitigate risks in a transparent way are not less important. Term ending in 2022:
Machine ethics and the notions of responsibility of AI systems or artificial moral Cesare Alippi, Politecnico di Milano, ITALY
agents3 are also topics of research that deserve a lot of attention. Consciousness in James C. Bezdek, USA
Gary Fogel, Natural Selection, Inc., USA
AI has been debated for a long time4 and it can take a form different from human Yaochu Jin, University of Surrey, UK
consciousness5. Multidisciplinarity, and in particular the cooperation with cognitive Alice E. Smith, Auburn University, USA
Term ending in 2023:
Piero P. Bonissone, Piero P Bonissone Analytics
1
https://www.cc.gatech.edu/~alanwags/DLAI2016/(Gunning)%20IJCAI-16%20DLAI%20WS.pdf LLC, USA
2
https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai Oscar Cordón, University of Granada, SPAIN
3
https://plato.stanford.edu/entries/ethics-ai/#MachEthi
4 Pauline Haddow, Norwegian University of Science
McCarthy, John (1996). Making robots conscious of their mental states. In S. Muggleton (ed.), _Machine Intelligence 15_.
Oxford University Press. and Technology, NORWAY
5
Pitrat, Jacques (2009). Artificial Beings : The Conscience of a Conscious Machine. Wiley, 2009 Haibo He, University of Rhode Island, USA
Hisao Ishibuchi, Southern University of Science
and Technology, CHINA
Digital Object Identifier 10.1109/MCI.2021.3084388
Date of current version: 15 July 2021 Digital Object Identifier 10.1109/MCI.2021.3056261

 AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 3

 IEEE Consortium On The Landscape
A special issue of the IEEE Computational Intelligence of AI Safety (CLAIS) and, on a more
Magazine dedicated to Explainable and Trustworthy specific level, it participates in the IEEE
Brain Community.
Artificial Intelligence will be published in 2022. I am convinced that the IEEE CIS
needs to pay even more attention to
these crucial issues. At a time when the
sciences and computational neurosci- Technical Committee task forces and IEEE CIS is strengthening its ties to
ence, is important in this regard. special issues of CIS publications are industry by creating a Committee for
The Computational Intelligence also focused on XAI. In particular, a Industrial and Governmental Activities
society (CIS) is at the heart of all these special issue of the IEEE Computation- starting in January 2022, the trust-
aspects of ethics in AI. They represent al Intelligence Magazine dedicated to worthiness of Artificial Intelligence
fascinating challenges for researchers Explainable and Trustworthy Artificial appears even more as a key factor in
and they can highlight the power of all Intelligence will be published in 2022. the realization and success of industrial
Computational Intelligence methods: The IEEE CIS Cognitive and Develop- products based on Computational
neural networks, learning methods, mental Systems Technical Committee Intelligence methods.
fuzzy systems and evolutionary compu- and the IEEE Transactions on Cogni- If you are interested in these topics
tation. Efforts are already done through tive and Developmental Systems also or any other activity of the IEEE CIS,
the IEEE CIS Task Force on Ethical address some of these concerns on a you can always contact me at b.bouchon
and Social Implications of Computa- cognitive basis. Moreover, the IEEE -meunier@ieee.org.
tional Intelligence. Several other CIS CIS is the oversight committee for the

TAP.
CONNECT.
NETWORK.
SHARE.
Connect to IEEE–no matter where you are–with the IEEE App.
Stay up-to-date Schedule, manage, or Get geo and interest-based
with the latest news join meetups virtually recommendations

Read and download Create a personalized Locate IEEE members by location,

your IEEE magazines experience interests, and affiliations

4 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

 Haibo He, University of Rhode Island, USA
Jon Garibaldi, University of Nottingham, UK
Carlos A. Coello Coello, CINVESTAV-IPN, MEXICO
Julian Togelius, New York University, USA
Yaochu Jin, University of Surrey, UK
Yew Soon Ong, Nanyang Technological University, SINGAPORE
Hussein Abbass, University of New South Wales, AUSTRALIA
CIS Publication Spotlight
IEEE Transactions on Neural
Networks and Learning Systems
A Comprehensive Survey on Graph
Neural Networks, by Z. Wu, S. Pan, F.
Chen, G. Long, C. Zhang, and P. S.
Yu, IEEE Transactions on Neural Net-
works and Learning Systems, Vol. 32,
No. 1, January 2021, pp. 4–24.
Digital Object Identifier: 10.1109/
TNNLS.2020.2978386
“Deep learning has revolutionized
many machine learning tasks in recent
years, ranging from image classification
and video processing to speech recogni-
tion and natural language understanding.
The data in these tasks are typically repre-
sented in the Euclidean space. However,
there is an increasing number of applica-
tions, where data are generated from non-
Euclidean domains and are represented as
graphs with complex relationships and
interdependency between objects. The
complexity of graph data has imposed sig-
nificant challenges on the existing
machine learning algorithms. Recently,
many studies on extending deep learning
approaches for graph data have emerged.
In this article, we provide a comprehen-
sive overview of graph neural networks
(GNNs) in data mining and machine
learning fields.We propose a new taxono-
my to divide the state-of-the-art GNNs
into four categories, namely, recurrent
GNNs, convolutional GNNs, graph auto-
encoders, and spatial-temporal GNNs. We
further discuss the applications of GNNs
Digital Object Identifier 10.1109/MCI.2021.3084390
Date of current version: 15 July 2021
©SHUTTERSTOCK.COM/BEERENTOCHTER
across various domains and summarize
the open-source codes, benchmark data
sets, and model evaluation of GNNs.
Finally, we propose potential research
directions in this rapidly growing field.”
A Survey of the Usages of Deep Learn-
ing for Natural Language Processing, by
D. W. Otter, J. R. Medina, and J. K.
Kalita, IEEE Transactions on Neural
Networks and Learning Systems,
Vol. 32, No. 2, February 2021, pp.
604–624.
Digital Object Identifier: 10.1109/
TNNLS.2020.2979670
“Over the last several years, the field
of natural language processing has been
propelled forward by an explosion in the
use of deep learning models. This article
provides a brief introduction to the field
and a quick overview of deep learning
architectures and methods. It then sifts
through the plethora of recent studies
and summarizes a large assortment of
relevant contributions. Analyzed research
areas include several core linguistic pro-
cessing issues in addition to many appli-
cations of computational linguistics. A
discussion of the current state of the art
is then provided along with recommen-
dations for future research in the field.”
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE
Publication
Spotlight
IEEE Transactions on
Fuzzy Systems
An Effective Multiresolution Hierarchical
Granular Representation Based Classifier
Using General Fuzzy Min-Max Neural
Network, by T. T. Khuat, F. Chen,
and B. Gabrys, IEEE Transactions on
Fuzzy Systems, Vol. 29, No. 2, Feb-
ruary 2021, pp. 427–441.
Digital Object Identifier: 10.1109/
TFUZZ.2019.2956917
“Motivated by the practical demands
for simplification of data toward being
consistent with human thinking and
problem-solving, as well as tolerance of
uncertainty, information granules are
becoming important entities in data pro-
cessing at different levels of data abstrac-
tion. This article proposes a method to
construct classifiers from multiresolution
hierarchical granular representations using
hyperbox fuzzy sets. The proposed
approach forms a series of granular infer-
ences hierarchically through many levels
of abstraction. An attractive characteristic
of our classifier is that it can maintain a
high accuracy in comparison to other
fuzzy min-max models at a low degree of
granularity based on reusing the knowl-
edge learned from lower levels of abstrac-
tion. In addition, our approach can
reduce the data size significantly as well as
handle the uncertainty and incomplete-
ness associated with data in real-world
applications. The construction process of
the classifier consists of two phases. The
first phase is to formulate the model at the
greatest level of granularity, while the later
stage aims to reduce the complexity of
5
the constructed model and deduce it from
data at higher abstraction levels. Experi-
mental analyses conducted comprehen-
sively on both synthetic and real datasets
indicated the efficiency of our method in
terms of training time and predictive per-
formance in comparison to other types of
fuzzy min-max neural networks and
common machine learning algorithms.”
Discrete and Smoothed Resampling
Methods for Interval-Valued Fuzzy
Numbers, by M. Romaniuk and O.
Hryniewicz, IEEE Transactions on
Fuzzy Systems, Vol. 29, No. 3,
March 2021, pp. 599–611.
Digital Object Identifier: 10.1109/
TFUZZ.2019.2957253
“In this article, we propose two new
resampling algorithms for the simulation
of bootstrap-like samples of interval-val-
ued fuzzy numbers (IVFNs). These
methods (namely, the d-method and the
s-method) reuse a primary sample (an
initial set) of IVFNs to generate a sec-
ondary sample, which also consists of
this type of fuzzy numbers, and simulta-
neously utilize existing dependencies in
pairs of some characteristic points of
IVFNs. During a corresponding resam-
pling step, a non-parametric approach is
used. Additionally, we apply a widely
used assumption about the Gaussian ker-
nel densities. The proposed methods in
some way resemble Efron’s bootstrap,
but, contrary to this classical approach,
they generate “not exactly the same as
previous” IVFNs, so it leads to a greater
diversity of the obtained secondary sam-
ple. We also numerically check the qual-
ity of the introduced methods using a
few more statistically oriented approach-
es together with four similarity measures
and three types of IVFNs.”
IEEE Transactions on Evolutionary
Computation
Realistic Constrained Multiobjective
Optimization Benchmark Problems
From Design, by C. Picard and
J. Schiffmann, IEEE Transactions on
Evolutionary Computation, Vol. 25, No. 2,
April 2021, pp. 234–246.
6 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
Digital Object Identifier: 10.1109/
TEVC.2020.3020046
“Multiobjective optimization is
increasingly used in engineering to design
new systems and to identify design trad-
eoffs. Yet, design problems often have
objective functions and constraints that are
expensive and highly nonlinear. Combina-
tions of these features lead to poor conver-
gence and diversity loss with common
algorithms that have not been specifically
designed for constrained optimization.
Constrained benchmark problems exist,
but they do not necessarily represent the
challenges of engineering problems. In this
article, a framework to design electro-
mechanical actuators, called multiobjective
design of actuators (MODAct), is present-
ed and 20 constrained multiobjective opti-
mization test problems are derived from
the framework with a specific focus on
constraints. The full source code is made
available to ease its use. The effects of the
constraints are analyzed through their
impact on the Pareto front as well as on
the convergence performance. A con-
straint landscape analysis approach is fol-
lowed and extended with three new
metrics to characterize the search and
objective spaces. The features of MODAct
are compared to existing test suites to
highlight the differences. In addition, a
convergence analysis using NSGA-II,
NSGA-III, and C-TAEA on MODAct
and existing test suites suggests that the
design problems are indeed difficult due to
the constraints. In particular, the number
of simultaneously violated constraints in
newly generated solutions seems key in
understanding the convergence challenges.
Thus, MODAct offers an efficient frame-
work to analyze and handle constraints in
future optimization algorithm design.”
IEEE Transactions on Games
A Multifaceted Surrogate Model for
Search-Based Procedural Content Gener-
ation, by D. Karavolos, A. Liapis, and
G. Yannakakis, IEEE Transactions on
Games, Vol. 13, No. 1, March 2021,
pp. 11–22.
Digital Object Identifier: 10.1109/
TG.2019.2931044
“This paper proposes a framework
for the procedural generation of level and
ruleset components of games via a surro-
gate model that assesses their quality and
complementarity. The surrogate model
combines level and ruleset elements as
input and gameplay outcomes as output,
thus constructing a mapping between
three different facets of games. Using this
model as a surrogate for expensive game-
play simulations, a search-based generator
can adapt content toward a target game-
play outcome. Using a shooter game as
the target domain, this paper explores
how parameters of the players’ character
classes can be mapped to both the level’s
representation and the gameplay out-
comes of balance and match duration.
The surrogate model is built on a deep
learning architecture, trained on a large
corpus of randomly generated sets of lev-
els, classes, and simulations from game
playing agents. Results show that a
search-based generative approach can
adapt character classes, levels, or both
toward designer-specified targets. The
model can thus act as a design assistant or
be integrated in a mixed-initiative tool.
Most importantly, the combination of
three game facets into the model allows it
to identify the synergies between levels,
rules, and gameplay and orchestrate the
generation of the former two toward
desired outcomes.”
IEEE Transactions on Cognitive
and Developmental Systems
Exoskeleton Online Learning and
Estimation of Human Walking
Intention Based on Dynamical
Movement Primitives, by S. Qiu, W.
Guo, D. Caldwell, and F. Chen, IEEE
Transactions on Cognitive and Develop-
mental Systems, Vol. 13, No. 1, March
2021, pp. 67–79.
Digital Object Identifier: 10.1109/
TCDS.2020.2968845
“Human walking intention estimation
is a critical step for the active assistance
control of lower limb exoskeleton, be­­
cause the purpose of active assistance
control is human motion assistance rather
than human motion tracking. Complying
with human walking intention is the
basic requirement of human walking
assistance. Hence, the human walking
intention must be estimated first to
ensure the exoskeleton will not impede
human motion. Actually, estimating
human walking intention is to estimate
human joint torque during walking. In
order to estimate a smooth personalized
human joint torque profile, an online
learning and prediction algorithm of
human joint trajectory and joint torque
is proposed in this article. The algorithm
is based on the dynamical movement
primitives model which is used for
online learning and predicting human
joint trajectory which is substituted into
the human dynamics model to estimate
the human joint torque. The results of
human walking experiments demon-
strate that the proposed algorithm can
not only predict a smooth human joint
trajectory and joint torque profile in real
time but also compensate the phase
delay caused by sensor signal filtering.
Hence, the proposed algorithm is suit-
able for the active walking assistance
control of exoskeleton.”
IEEE Transactions on Emerging
Topics in Computational Intelligence
Evaluating Evolving Structure in Stream-
ing Data With Modified Dunn’s Indices,
(continued on page 94)

Call for Papers for Journal Special Issues

Special Issue on “Causal Discovery and Causality-Inspired Machine Learning”
Journal: IEEE Transactions on Neural Networks and Learning Systems
Guest Editors: Kun Zhang, Ilya Shpitser, Sara Magliacane, Davide Bacciu, Fei Wu, Changshui Zhang, and
Peter Spirtes
Submission Deadline: October 22, 2021
https://cis.ieee.org/images/files/Publications/TNNLS/special-issues/CFP_TNNLS_SI_Causal_Discovery_and_Caus
ality-Inspired_Machine_Learning.pdf

Special Issue on “Cyborg Intelligence: Human Enhancement with Fuzzy Sets”

Journal: IEEE Transactions on Fuzzy Systems
Guest Editors: Zhijun Li, Jian Huang, Hang Su, and Zhaojie Ju
Submission Deadline: September 30, 2021
https://cis.ieee.org/images/files/Publications/TFS/special-issues/Special_Issue_on_Cyborg_Intelligence.pdf

Special Issue on “Recent Advances in Fuzzy-based Intelligent IoT and Cyber-physical

Systems”
Journal: IEEE Transactions on Fuzzy Systems
Guest Editors: Mainak Adhikari, Varun G Menon, Jessie Park, and Danda Rawat
Submission Deadline: October 31, 2021
https://cis.ieee.org/images/files/Publications/TFS/special-issues/TFS_SI_-_Recent_Advances_in_Fuzzy-based_Intel
ligent_IoT_and_Cyber-physical_Systems.pdf

Special Issue on “Benchmarking Sampling-Based Optimization Heuristics: Methodology

and Software”
Journal: IEEE Transactions on Evolutionary Computation
Guest Editors: Thomas Bäck, Carola Doerr, Bernhard Sendhoff, and Thomas Stützle
Submission Deadline: August 31, 2021
https://cis.ieee.org/images/files/Documents/call-for-papers/tevc/cfp-bench-updated-may2021.pdf

Special Issue on “Computational Intelligence to Edge AI for Ubiquitous IoT Systems”

Journal: IEEE Transactions on Emerging Topics in Computational Intelligence
Guest Editors: Honghao Gao, Biao Luo, Ramón J. Durán Barroso, and Walayat Hussain
Submission Deadline: November 20, 2021
https://cis.ieee.org/images/files/Publications/TETCI/SI23_CFP_EAI-IoT.pdf

Special Issue on “Security and Privacy of Machine Learning”

Journal: IEEE Transactions on Artificial Intelligence
Guest Editors: Wenjian Luo, Yaochu Jin, and Catherine Huang
Submission Deadline: August 15, 2021
https://cis.ieee.org/publications/ieee-transactions-on-artificial-intelligence/special-issues

Digital Object Identifier 10.1109/MCI.2021.3084499

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 7

 Guest Yanan Sun
Editorial Sichuan University, CHINA
Mengjie Zhang
Victoria University of Wellington,
NEW ZEALAND
Gary G. Yen
Oklahoma State University, USA

Evolutionary Neural Architecture Search and Applications

D eep neural networks (DNNs) have
shown significantly promising per-
formance in addressing real-world
problems, such as image recognition, nat-
ural language processing and self-driving.
The achievements of DNNs owe largely
to their deep architectures. However,
designing an optimal deep architecture for
a particular problem requires rich domain
knowledge on both the investigated data
and the neural network domains, which is
not necessarily held by the end-users.
Neural architecture search (NAS), as an
emerging technique to automatically
design the optimal deep architectures
without requiring such expertise, is draw-
ing increasing attention from industry and
academia. However, NAS is theoretically a
non-convex and non-differentiable opti-
mization problem, and existing methods
are incapable of well addressing it. Evolu-
tionary computation approaches, particu-
larly genetic algorithms, particle swarm
optimization and genetic programming,
have shown superiority in addressing real-
world problems due largely to their pow-
erful abilities in searching for global
optima, dealing with non-convex/non-
differentiable problems, and requiring no
rich domain knowledge. In this regard,
deep neural architecture designed by evo-
lutionary computation approaches, so
called evolutionary neural architecture
search (ENAS), have attracted the interest
of many researchers.
Digital Object Identifier 10.1109/MCI.2021.3084391
Date of current version: 15 July 2021
This special issue has brought together
researchers to report state-of-the-art con-
tributions on the latest research and devel-
opment, up-to-date issues, challenges, and
applications in the field of ENAS. Follow-
ing a rigorous peer review process, five
papers have been accepted for publication
in this special issue.
The first paper included in the special
issue is entitled “Evolutionary Multi-
Objective Model Compression for Deep
Neural Networks” authored by Z.Wang et
al., which aims at accelerating the inference
speed of DNNs by optimizing the model
size and accuracy simultaneously with evo-
lutionary algorithms.The architecture pop-
ulation evolution was employed to explore
and exploit the network space of pruning
and quantization. In addition, a two-stage
co-optimizing strategy of pruning and
quantization was proposed to significantly
reduce time cost during the architecture
search process. To further lower energy
consumption and reduce the model size,
various dataflow designs and parameter
coding schemes were also considered in the
optimization process. Unlike most related
research solely focusing on reducing the
model size while maintaining the model
accuracy, the work in this paper can achieve
a trade-off between different model sizes
and model accuracies, which meets the
requirements for most edge devices in real-
world scenarios. The experimental results
demonstrated that the proposed algorithm
can obtain a broad range of compact
DNNs for diverse memory usage and
energy consumption requirements.
The second paper, titled “Fast and
Unsupervised Neural Architecture Evolu-
tion for Visual Representation Learning”
by S. Xue et al., proposes the FaUNAE
algorithm to improve the unsupervised
visual representation learning ability
against the supervised peer competitors.
Specifically, FaUNAE employed the evo-
lutionary algorithm to search for promis-
ing neural architectures from an existing
architecture designed by experts or exist-
ing NAS algorithms that focus on the
transferability from a small dataset to a
larger dataset. To reduce the search cost
and enhance the search efficiency, the
prior knowledge and the inferior, as well
as least promising operations, were used
during the evolutionary process. In addi-
tion, the contrast-loss function was uti-
lized as the evaluation metric in a
student-teacher framework to achieve the
self-supervised evolution. FaUNAE was
evaluated on four widely used large-scale
benchmark datasets. The results demon-
strated the effectiveness of FaUNAE
upon various downstream applications,
including object recognition, object
detection, and instance segmentation.
The third paper entitled “Self-Super-
vised Representation Learning for Evolu-
tionary Neural Architecture Search” by
C. Wei et al. proposes a novel performance
predictor, aiming to enhance the efficien-
cy of ENAS algorithms. Specifically, the
ENAS algorithms are often computation-
ally expensive in practice because hun-
dreds of DNNs needed to be trained
during the search process. Performance

8 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

preceptors are a kind of regression model
that can directly predict the performance of
a neural network without any training and
is a hot research topic among the commu-
nity. However, the training of performance
predictors requires a large number of well-
trained neural networks, which is often
scarce in practice. To address this issue, this
paper first developed a new encoding strat-
egy of architectures for calculating the
graph edit distances among different archi-
tectures. Also, two self-supervised learning
methods were designed to improve the pre-
diction performance by learning the mean-
ingful representations of neural architectures.
This can help to enhance the quality of the
training data fed to the performance pre-
dictors. The experiments demonstrated
promising performance of the proposed
performance predictor against the peer
competitors. In addition, the proposed per-
formance predictor is integrated into an
ENAS algorithm for validation, and the
results also showed its superiority of search-
ing for promising neural architectures.
In the fourth paper, “Forecasting Wind
Speed Time Series Via Dendritic Neural
Regression” by J. Ji et al., proposes a
regressive version of the dendritic neuron
model (DNM), i.e., dendritic neural
regression (DNR), to forecast wind
power. Particularly, wind energy is one of
the fastest-growing green energy resourc-
es. Precise forecasting wind power is cru-
cial in planning the power system and
operating the wind farm. However, since
the wind speed time series is with chaotic
properties and high volatility, traditional
methods are incapable of producing satis-
factory forecasts. DNM is a plausible bio-
logical neural model and has the potential
to forecast wind power well. However,
DNM is originally designed for classifica-
tion problems.To this end, DNR is devel-
oped based on DNM to forecast the
wind power that is a regression task. Like
other neural network-based models, the
performance of DNR is also voluntary to
its architecture design. To address this
problem, the states of matter search (SMS)
algorithm is used to search for the prom-
ising architecture of DNR without much
manual effort. The experiments were
conducted on two benchmark datasets
with two different time intervals. The
results revealed that DNR can provide population degradation and slow conver-
superior performance compared to its gence. In addition, motivated by the
competitors, and DNR-SMS was an effi- recent advances of DNN models, the
cient tool for wind speed prediction. building blocks of DenseNet and ResNet
The fifth paper, “A Self-Adaptive were also designed as the search units of
Mutation Neural Architecture Search SaMuNet. The performance of SaMuNet
Algorithm Based on Blocks” by Y. Xue et was compared with 17 peer competitors
al., proposes the algorithm named SaMu- on CIFAR10 and CIFAR100 bench-
Net to tackle the problem of “loss of mark datasets. The results demonstrated
experience” caused by ENAS algorithms that SaMuNet can outperform most of
in their early search stage. Specifically, them in terms of both the classification
most of the existing ENAS algorithms accuracy and the consumed computa-
mainly focused on investigating the search tional resource.
space or evaluation strategy to enhance The guest editors of this special issue
their performance. However, the search would like to thank Prof. Chuan-Kang
strategy also plays an important role dur- Ting, the Editor-in-Chief of IEEE Com-
ing the whole process. In addition, the putational Intelligence Magazine, for his
information of individuals in different great support in initiating and developing
generations are also crucial to the perfor- this special issue together. Many thanks to
mance of the corresponding evolutionary all members of the editorial team for their
algorithm, which is ignored by most kind support during the editing process of
existing ENAS algorithms. To address this special issue. Last but not least, we
both issues, SaMuNet incorporated a self- would also like to thank the authors for
adaptive mutation component into the submitting their valuable research out-
framework of the evolutionary algorithm comes as well as the reviewers who have
to effectively search for the neural archi- critically evaluated the papers. We sincere-
tectures. Furthermore, a semi-complete ly hope and expect that readers will find
binary competition selection strategy was this special issue useful.
also designed into SaMuNet to prevent
Introducing IEEE Collabratec™
The premier networking and collaboration site
for technology professionals around the world.
IEEE Collabratec is a new, integrated online community
where IEEE members, researchers, authors, and
technology professionals with similar fields of interest
can network and collaborate, as well as create and
manage content all in one place!
Learn about IEEE Collabratec at
ieee-collabratec.ieee.org

Evolutionary Multi-Objective
Model Compression
for Deep Neural Networks
Zhehui Wang and Tao Luo
Agency for Science, Technology and Research (A*STAR),
SINGAPORE
Miqing Li,
University of Birmingham, UK
Joey Tianyi Zhou, Rick Siow Mong Goh,
and Liangli Zhen
Agency for Science, Technology and Research (A*STAR),
SINGAPORE
Abstract—While deep neural networks (DNNs) deliver state-of-
the-art accuracy on various applications from face recognition to
language translation, it comes at the cost of high computational
and space complexity, hindering their deployment on edge
devices. To enable efficient processing of DNNs in inference, a
novel approach, called Evolutionary Multi-Objective Model
Digital Object Identifier 10.1109/MCI.2021.3084393
Date of current version: 15 July 2021
10 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
©SHUTTERSTOCK.COM/DMITRIY PRAYZEL
Compression (EMOMC), is proposed to optimize energy effi-
ciency (or model size) and accuracy simultaneously. Specifically,
the network pruning and quantization space are explored and
exploited by using architecture population evolution. Further-
more, by taking advantage of the orthogonality between pruning
and quantization, a two-stage pruning and quantization co-opti-
mization strategy is developed, which considerably reduces time
cost of the architecture search. Lastly, different dataflow designs
and parameter coding schemes are considered in the optimiza-
tion process since they have a significant impact on energy con-
sumption and the model size. Owing to the cooperation of the
evolution between different architectures in the population, a set
of compact DNNs that offer trade-offs on different objectives
(e.g., accuracy, energy efficiency and model size) can be obtained
in a single run. Unlike most existing approaches designed to
reduce the size of weight parameters with no significant loss of
accuracy, the proposed method aims to achieve a trade-off
between desirable objectives, for meeting different requirements
of various edge devices. Experimental results demonstrate that the
proposed approach can obtain a diverse population of compact
Corresponding author: Liangli Zhen (e-mail: zhenll@ihpc.a-star.edu.sg).
1556-603X/21©2021IEEE
DNNs that are suitable for a broad range of different memory
usage and energy consumption requirements. Under negligible
accuracy loss, EMOMC improves the energy efficiency and
model compression rate of VGG-16 on CIFAR-10 by a factor of
more than 8.9 # and 2.4 #, respectively.
D
I. Introduction
eep neural networks (DNNs) are artificial neural net-
works with more than three layers (i.e., more than one
hidden layer), which progressively extract higher-level
features from the raw input in the learning process.
They have delivered the state-of-the-art accuracy on various
real-world problems, such as image classification, face recogni-
tion, and language translation [1]. The superior accuracy of
DNNs, however, comes at the cost of high computational and
space complexity. For example, the VGG-16 model [2] has about
138 million parameters, which requires over 500 MB memory
for storage and 15.5G multiply-and-accumulates (MACs) to
process an input image with 224 × 224 pixels. In myriad applica-
tion scenarios, it is desirable to make the inference on edge
devices rather than on cloud, for reducing the latency and
dependency on connectivity and improving privacy and security.
Many of the edge devices that draw the DNNs inference have
stringent limitations on energy consumption, memory capacity,
etc. The large-scale DNNs [3], [4] are usually difficult to be
deployed on edge devices, thus hindering their wide application.
Efficient processing of DNNs for inference has become increas-
ingly important for the deployment on edge devices. For generat-
ing efficient DNNs, many neural architecture search (NAS)
approaches have been developed in recent years [5]–[7]. One way
of carrying out NAS is to search from scratch [8], [9]. In contrast,
model compression1 [10] searches for the optimal networks starting
from a well-trained network. For instance, to reduce the storage
requirement of DNNs, Han et al. proposed a three-stage pipeline
(i.e., pruning, trained quantization, and Huffman coding) to com-
press redundant weights [10]. Wang et al. suggested removing
redundant convolution filters to reduce the model size [11]. Rather
than reducing the model size, a few attempts [12], [13] are conduct-
ed to compress DNNs directly by taking the energy consumption
as the feedback signals. They have achieved promising results in
reducing the size of weight parameters (or energy consumption).
However, these approaches require the model to achieve approxi-
mately no loss of accuracy, rendering the solution less flexible.
In practice, different users often have distinct preferences on
desirable objectives, e.g., accuracy, model size, energy efficiency, and
latency, when they select the optimal DNN model for their appli-
cations. In this paper, a novel approach, called Evolutionary Multi-
Objective Model Compression (EMOMC), is proposed to
optimize energy efficiency/model size and accuracy simultaneous-
ly. By considering network pruning and quantization, the model
compression is formulated as a multi-objective problem under dif-
ferent dataflow designs and parameter coding schemes. Each can-
didate architecture can be regarded as an individual in the
1
The technique aims to shrink the size of the neural network model without a signifi-
cant drop of accuracy.
evolutionary population. Owing to the cooperation and interplay
of the evolution between different architectures in the population,
a set of compact DNNs that offer trade-offs on different objectives
(e.g., accuracy, energy efficiency, and model size) can be obtained in
a single run. Unlike most existing approaches which aim to reduce
the size of weight parameters or the energy consumption with no
significant loss of accuracy, the proposed approach attempts to
achieve a good balance between desired objectives, for meeting the
requirements of different edge devices. Experimental results dem-
onstrate that the proposed approach can obtain a diverse popula-
tion of compact DNNs for customized requirements of accuracy,
memory capacity, and energy consumption.
The novelty and main contributions of this work can be
summarized as follows:
❏❏ The model compression problem is formulated as a multi-
objective problem. The optimal solutions are searched in the
network pruning and quantization space using a popula-
tion-based algorithm.
❏❏ To speed up the population evolution, a two-stage pruning/
quantization co-optimization strategy is developed based on
the orthogonality between pruning and quantization.
❏❏ The trade-offs between accuracy, energy efficiency, and
model size in model compression are explored by consider-
ing different dataflow designs and parameter coding schemes.
The experimental results demonstrate that the proposed
method can obtain a set of diverse Pareto optimal solutions in
a single run. Also, it achieves a considerably higher energy
efficiency than current state-of-the-art methods.
II. Preliminaries
Network pruning and quantization are two commonly used
model compression techniques to improve the energy efficiency
in model inference and/or to shrink the size of the model.
Moreover, the dataflow design employed by edge devices and the
coding scheme applied to store the weight matrix both have a
significant impact on the performance of model compression.
A. Network Pruning and Quantization
For making the training easy, the networks are usually over-
parameterized [14]. Pruning is a widely-used model compression
technique that can effectively reduce the energy consumption of
edge devices and shrink the model size [10]. Network pruning
removes some of the redundant parameters in the network by
setting their values as zeros. A well-trained neural network usually
contains a large number of weights whose values are relatively
small compared to other parameters. In most cases, these parame-
ters are not particularly important when performing model infer-
ence. Hence, one can sort all the parameters in the model and
replace those parameters with the least absolute values by zeros,
while the accuracy of the model can still be maintained. For
instance, the pruning amount to be 33%, then one-third of the
parameters in the model will be replaced by zeros. In the infer-
ence process, if the processing elements (PEs)2 whose input
2
The PE is a basic unit to conduct computation in processors.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 11

Dataflow design is one of the most important features
in hardware accelerators, which allows the system to
reuse the data among different processing elements.
weight parameters are zeros, the computation process can be
skipped in those PEs, thus reducing energy consumption.
Quantization is another critical model compression tech-
nique that is used to accelerate DNNs and reduce model size
[10]. It involves mapping data to a small set of quantization lev-
els and aims at minimizing the error between the reconstructed
data from the quantization levels and the original data. The
quantization level reflects the precision and ultimately the
number of bits representing a parameter. After the quantization,
the low precision parameters may still store enough informa-
tion for model inference, and the accuracy of the model can be
maintained. In practical implementations, if the weights are
quantized, one can use multipliers with simpler structures, thus
reducing energy consumption. For instance, a high precision
parameter with 32-bit float point (32FP) data type requires
23 bit × 23 bit multipliers. Such type of multiplier contains
506 adders in total. If quantizing the activation from 32FP to
16-bit float point (16FP) and quantizing the weights from
32FP to 8-bit integer (8INT), only 10 bit × 8 bit multipliers
are required, each of which contains only 72 adders in total.
The fewer adders in multipliers, the lower energy consumption
for computation.
Pruning and quantization can reduce not only the energy
consumption on the computation process but also the energy
consumption on the data movement, which is roughly propor-
tional to the total amount of data transmitted from the memo-
ry module in terms of bits [15]. For instance, if pruning 80% of
the parameters in the model and quantizing all the parameters
from 16 bits to 8 bits, then about 90% of the energy consump-
tion on data movement can be reduced.
B. Dataflow Design in Hardware Accelerators
The dataflow design decides how data is reused among differ-
ent PEs. Since a large portion of the energy consumption of
hardware accelerators is on the data movement, the dataflow
design needs to be considered when optimizing the energy
efficiency. Algorithm 1 shows the computation procedure of a
Algorithm 1 Computation of a typical convolutional layer.
  for co in range(CO) do
  for ci in range(CI) do
   for x in range(X ) do
    for y in range(Y ) do
     for fx from -(FX -1)/2 to (FX -1)/2 do
      for fy from -(FY -1)/2 to (FY -1)/2 do
       O[co][x][y]
       +  = I[ci][x + fx][y + fy] # W [co][ci][fx][fy]
12 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
typical convolutional layer. It contains six
loops, each of which corresponds to one
dimension either in the weight filter or in the
feature map. More specifically, C O and C I are
the numbers of output and input channels, X
and Y are the width and height of the feature
map, and FX and FY are the width and height of the weight
filter. In each iteration of the innermost loop, a basic arithmetic
operation called multiply-accumulate (MAC) is performed. In
one convolutional layer, there are C O ·C I ·X·Y·FX ·FY MAC
operations in total.
In typical hardware accelerators,3 there are a set of process-
ing elements. Each PE can execute one MAC operation inde-
pendently. How to map the MAC operation into each PE and
how the data flow among those PEs become key consider-
ations in the design of hardware accelerators. Theoretically,
there are many mapping methods, resulting in different data-
flow designs. For example, suppose that the device has an array
of PEs, one can unroll any one of the loops in Algorithm 1
and map each iteration in the unrolled loop into each PE of
the array. Similarly, if the device has a matrix of PEs, one can
unroll any two loops in Algorithm 1 and map the MAC oper-
ations into each PE in the matrix. Thus, with six loops as
shown in Algorithm 1, there are C 26 = 15 possible dataflow
designs in total. To simplify the problem, in this work, only four
popular dataflow designs are evaluated, as shown in Table I.
These dataflow designs are named as A : B, where A and B
stand for the names of the two unrolled loops.
Figure 1 shows the schematic diagram of those four popu-
lar dataflow designs, where only four PEs are involved in each
example. Each PE contains one multiplier and one adder,
which can execute one MAC operation each time. The PE also
contains register files, which can temporarily store input or
output data. In X:Y, the MAC operation results are stored in
registers at output ports of PEs. At each iteration, the last MAC
operation result is read from registers. In C I :C O, at each itera-
tion, the input feature map is reused C O times, and C I MAC
operation results are summed up. In FX :FY , FX ·FY weights are
stored in registers at input ports of PEs. At each iteration,
FX ·FY MAC operation results are summed up. In X :FX , FX
weights are stored in registers at input ports of PEs. At each
iteration, the weights are reused X times, and FX MAC opera-
tion results are summed up.
TABLE I Popular dataflow designs
that are applied in literature.
DATAFLOW APPLIED BY DATAFLOW APPLIED BY
X :Y [16] [17] CI : CO [18] [19]
FX : FY [20] X : FX [21] [22]
3
The devices that are specialized to execute a certain task, such as the graphics pro-
cessing unit (GPU), field-programmable gate array(FPGA), and application-specific
integrated circuit (ASIC).
C. Coding Scheme for the
Parameters I0 I1 I0
After pruning, the filter in the model
becomes a sparse matrix, which means
W0 O0 W1 O1 W0 W10
that it contains plenty of zero elements.
To store the sparse matrix into the
I2 I3
memory, many coding schemes are
developed, and the choice of the cod-
ing scheme mainly depends on the W2 O2 W3 O3 W1 W11
O0 O1
characteristics of the matrix. In this
work, three coding schemes are consid- Dataflow - X :Y Dataflow - C l :C o
ered. The first one is the normal coding
scheme, where it stores those zero ele- I0 I1 I0 I1
ments in the same way as those non-
zero elements. In other words, it keeps W0 W1 W0
the space for all the zero elements in
the matrix. Therefore, the storage size of I2 I3 I1 I2
the normal coding scheme is N $ q i,
where N is the total number of weight W2 W3 W1
elements in the matrix and q i is the O0 O1
quantization depth of the weight. O
Dataflow - Fx:FY Dataflow - X:F x
In some cases, it is a waste of the
memory capacity to store all these zero Processing Element Register File Data Direction
weights. For saving the memory space,
various new coding schemes are pro-
FIGURE 1 Examples of the four popular dataflow designs [15]. For simplicity, only four PEs are
posed to shrink the size of the sparse shown in each example. W k /W kk, Ik, and Ok are the elements in the weight, the input feature
matrix. One common coding scheme is map, and the output feature map, respectively.
called the Coordinate (COO) coding.
In the COO coding scheme, the non-zero elements are stored, model size, number of floating-point operations per second
along with the row index and the column index, and zero ele- (FLOPs), latency, and energy efficiency. The initial intention of
ments are ignored. Another popular coding scheme is called the model compression is to alleviate the on-chip storage limit for
compressed sparse row (CSR) coding. In the CSR coding complicated CNN models [10]. Since then, many approaches
scheme, only the values of the non-zero elements are stored have been proposed to shrink the model size of CNNs [23],
along with the column index and the row offset. To further [24]. There are two major branches in this area. The first branch
save the memory space, one version of the CSR coding focuses on the computation cost, and they target the number of
scheme only stores the relative distance between two non-zero FLOPs [25]. For example, Li et al. [26] proposed to prune
elements, which is shown in Figure 2. For this example, the whole filters from CNNs, avoiding sparse connectivity patterns
weight matrix has eight elements, and three of them are non- and reducing the computational cost significantly. Lemaire et al.
zero elements. Only the values of these three elements and proposed a budgeted regularized pruning framework for deep
their relative positions in the array are stored. The second non- CNNs [25], which makes the compressed model less computa-
zero element is three slots away from the first non-zero ele- tion-intensive. The second branch targets the inference speed
ment, and the integer 2 is recorded as a relative row index for [27], [28]. For instance, He et al. leveraged reinforcement
the second element. It is assumed that each non-zero element
requires three bits to store the relative row index. If one non-
zero element is far away from the previous element, zero-pad- Distance = 2 Distance = 3
ding elements are inserted into the array to avoid overflow.
Therefore, the storage size of the CSR coding scheme with NA 2 3
relative positions is n· (p i + 3), where n is the number of non-
zero elements and q i is the quantization depth of the weight.
XXX 0 0 XXX 0 0 0 XXX
III. Related Work
W0 W1 W2 W3 W4 W5 W6 W7

A. Model Compression
FIGURE 2 The CSR coding scheme with relative positions, which
Model compression aims to compress and accelerate DNN stores the values of non-zero elements and their relative positions in
models. Different approaches target different objectives, such as the array.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 13


Evolutionary multi-objective optimization has been
widely used to search for the optimal solutions,
in the presence of trade-offs between multiple
conflicting objectives.
learning to provide the model compression policy, which can
accelerate the inference on mobiles considerably.
Recently, edge devices have become increasingly popular
for AI applications. However, considering the large amount of
energy consumed for the model inference, the deployment of
CNN on edge devices becomes challenging. To solve this
problem, some scholars proposed model compression
approaches to reduce the energy consumption directly, using
quantization [13] and/or pruning [12] techniques. In [12], an
energy-aware network pruning approach is proposed to reduce
the overall energy across all layers by 3.7 # for AlexNet [29]
and 1.6 # for GoogLeNet [30].
From the above, it can be seen that model compression is
essentially a multi-objective optimization problem, with several
objectives to be considered, including accuracy, energy con-
sumption, model size, etc. Previous studies rarely deal with
multiple objectives at the same time. A common way adopted
in literature is to optimize only one of the objectives while set-
ting the remaining ones to be hard constraints. In this work, the
evolutionary multi-objective optimization technique is applied
to tackle these objectives simultaneously.
B. Evolutionary Multi-Objective Optimization
In the real-world systems, there exist plenty of problems having
two or more (often conflicting) objectives which one needs to
consider simultaneously. Such problems are called the multi-
objective optimization problems (MOPs). Without loss of gen-
erality, a multi-objective optimization problem (MOP) can be
formulated as the following minimization problem:
min F (x) = (f1 (x), f2 (x), f, fM (x)) T
x MOEAs to search for efficient neural architectures. For
s.t. g j (x) # 0, j ! {1, 2, f, J},

h k (x) = 0, k ! {1, 2, f, K},
x ! X, (1)
where J denotes the number of inequality constraints, K is the
number of equality constraints, X 3 R n is the decision space,
x = (x 1, x 2, f, x n) T is a candidate solution, and F : X " R M
consists of M (conflicting) objective functions.
Let a and b be two feasible solutions for an MOP defined in
Equ at ion (1), one can say that a dominates b if
6u fu (a) # fu (b) and 7v fv (a) 1 fv (b), where u, v ! {1, 2, f, M}.
A solution is Pareto optimal if it is not dominated by any other
solutions. Due to the conflict of the objectives in MOPs, there
are a set of Pareto optimal solutions, which represent the best
possible trade-offs among different objectives. The optimal solu-
tion set in the decision space is called the Pareto set (PS), and its
mapping in the objective space is called the Pareto front (PF).
14 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
In the literature, many approaches have
been developed to solve MOPs since the
1950s [31]. Among them, evolutionary algo-
rithms (EAs) stand out thanks to the nature of
population-based search that aims to approxi-
mate the whole Pareto front in a single execu-
tion. Also, EAs are typically exempt from the
characteristics of the PF than conventional math-
ematical programming techniques [31]. They can handle the
MOPs with discontinuous and non-convex PFs well.
Since the seminal work, called Vector Evaluated Genetic
Algorithm (VEGA) [32], was proposed by Schaffer in 1985, a
large number of multi-objective evolutionary algorithms
(MOEAs) have been developed and adopted in various appli-
cations. In MOEAs, the selection strategy of individuals in the
population plays a key role in the evolutionary process. Since
the optimal solutions are those non-dominated to each other
in the whole search space, Pareto dominance naturally becomes
a viable criterion for selecting promising solutions during the
evolutionary process. The Pareto dominance criterion, however,
may fail to provide sufficient selection pressure, making the
algorithm hard to converge. This situation can be usually
encountered when the objective space is enormous, e.g., in
many-objective optimization problems [33]–[35]. To push the
population towards the PF, Goldberg proposed a mechanism
called Pareto ranking [36] for the selection in MOEAs. A niche
method is then used in the Nondominated Sorting Genetic
Algorithm (NSGA) [37] to maintain stable sub-populations.
Later on, in its new version Nondominated Sorting Genetic
Algorithm-II (NSGA-II) [38], a crowding degree comparison
operator is adopted to make the ranking scheme more effective
and efficient. NSGA-II is widely used to solve MOPs, despite
its limitations in handling the MOPs with more than three
objectives [39]. Recently, many MOEAs tend to consider other
selection strategies since they may converge fast towards the PF,
such as indicator-based MOEAs, decomposition-based
MOEAs, and bi-goal criterion MOEAs [33].
Recently, there have been a few attempts to exploit
instance, Lu et al. proposed a method, called NSGA-Net [40],
which formulates the neural architecture search as a multi-
objective problem and uses the NSGA-II algorithm to solve it.
NSGA-Net considers two objectives: the classification error
and the computation cost (measured by the number of MACs).
It has achieved promising results compared with other neural
architecture search methods, e.g., DARTS [5] and ENAS [41],
on the CIFAR-10 dataset [42].
This work studies how the evolutionary multi-objective
(EMO) method can be used in model compression, given its
multi-objective nature.
IV. Our Proposed Method
In real-world applications, users usually have different preferenc-
es on the prediction model’s objectives, including accuracy,
energy efficiency, model size, etc. In this section, the evolutionary
multi-objective model compression method is presented. The
model compression problem is formulated as a multi-objective
problem (MOP), which has several objective functions and con-
straints [43]. Then, an evolutionary algorithm is adopted to solve
the MOP. The goal of the optimization is to find a set of Pareto
optimal solutions that represents various trade-offs on the
desired objectives, thus enabling the deployment of the AI mod-
els on edge devices with different resource constraints.
A. Problem Formulation
This work aims to compress a well-trained model to achieve
high accuracy, low energy consumption, and low model size.
By providing different pruning amount, p, and quantization
depth, q, the compressed model should result in different accu-
racy, energy consumption, and model size. The goal of the opti-
mization is to reduce the energy consumption or model size
while at the same time making the accuracy of the model as
high as possible. The relationship between the accuracy, the
pr uning amount, p, and the quantization depth
q = [q 1, q 2, f, q L] is denoted as
Accuracy = f1 (p, q), (2)
where f1 (·) represents the accuracy score of the model
obtained by pruning p of the weight parameters in each layer
of the original model, then quantizing the parameters in the
i-th layer with the depth of q i bits, and L denotes the number
of layers in the original model.
The energy consumption of the inference is constrained by
the battery’s capacitance of edge devices. Exceeding the ener-
gy budget of the edge device will greatly limit the implemen-
tation of AI applications. From the perspective of users, it is
usually acceptable to trade a bit of loss of accuracy for a large
amount of reduction on energy consumption, especially for
edge devices. For a trained model, the energy consumption in
inference is also related to the exact dataflow design d applied
on the edge devices. The relationship among the pruning
amount p, the quantization depth q, and the dataflow design d
is denoted as follows:
Energy = f2 (p, q, d ) .(3)
The model size is constrained by the capacities of on-chip
memory modules in edge devices. If the model size exceeds
the limitation, the model inference procedure requires to load
and save weights/features maps through the off-chip memory.
Given the fact that off-chip memory access consumes much
larger energy consumption than the on-chip memory access
[10], the energy consumption of the inference process increas-
es tremendously. Furthermore, the app stores are sensitive to
the size of the binary files, e.g., App Store has the restriction
“apps above 100 MB will not download until you connect to
Wi-Fi” [10]. Hence, it is important to shrink the size of the
model and to make sure that the entire model can be fit into
the memory constraint of the edge devices. For a given model,
the model size highly depends on the coding scheme c applied
to store the weights. The relationship between the model size,
the pruning amount p, the quantization depth q and the coding
scheme c is defined as
Model Size = f3 (p, q, c) .(4)
There are L + 3 variables, and L denotes the number of lay-
ers in the original model. The value of the variable p is a real
number that indicates the pruning amount in all the layers of
the model. The value of the variable q i is an integer that
reflects the quantization depth in the i-th layer of the model.
The constraints on these variables are as follows:
p l # p # p u,
q l # q i # q u,

d ! {d 1, d 2, d 3, d 4},
c ! {c 1, c 2, c 3}, (5)
where p l and p u are the upper and lower bounds of the prun-
ing amount, q l and q u are the upper and lower bounds of the
quantization depth, d 1, d 2, d 3 , and d 4 correspond to the four
dataflow designs of X: Y, C I : C O, FX : FY and X: FX , and c 1, c 2
and c 3 indicate three parameter coding schemes of the normal
coding, COO and CSR, respectively. In this work, the pruning
amount is assumed to be from 0% to 100%, and the quantiza-
tion depth of each layer ranges from 1 bit to 23 bits.
Two bi-objective optimization problems are studied. In the
first problem, it explores possible combinations of pruning
amount and quantization depth, and aims to maximize the
model accuracy f1 and minimize the energy consumption f2,
assuming the dataflow design to be d. Mathematically, the
bi-objective problem can be formulated as following:
p l # p # p u,
' s.t. * q l # q i # q u,
max f1 (p, q),
(6)
min f2 (p, q,d ),
d ! {d 1, d 2, d 3, d 4} .
The second bi-objective problem considers to maximize
the accuracy f1 and minimize the model size f3 simultaneous-
ly, assuming the coding scheme to be c, namely, the following
problem:
p l # p # p u,
s.t. * q l # q i # q u, (7)
'
max f1 (p, q),
min f3 (p, q, c),
c ! {c 1, c 2, c 3} .
Note that this work formulates two bi-objective optimiza-
tion problems rather than a three-objective optimization prob-
lem. There are two reasons. Firstly, if one optimizes the energy
consumption and the model size simultaneously (i.e., different
dataflow designs and different coding schemes will be consid-
ered at the same time), the decision space will be increased
considerably, making the optimization much harder and con-
suming more computation resource. Secondly, as the evalua-
tion of each individual has a high computational cost, the
population size cannot be a large number. Typically, the pop-
ulation size is set to be smaller than 100. A three-objective
space will lead to the solution set to be much more sparse
than a bi-objective space.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 15
B. Multi-Objective Optimization and Speedup
Instead of pruning the model directly in one step, a more effec-
tive approach employed is to prune the model in multiple steps.
If pruning the model in one step, the accuracy will decrease
apparently, and it will be too difficult to restore the model [44].
Figure 3 demonstrates the comparison between the multi-step
pruning method and the single-step pruning method. The
model compression of a well-trained VGG-16 model is tested
on the CIFAR-10 dataset [15], [42]. For the multi-step prun-
ing, it gradually increases the pruning amount from 0 to 95%
in 32 steps. In each step, the model is pruned partially and re-
trained by one epoch. In the single-step pruning, the model is
pruned by 95% immediately, and then re-trained by 32 epochs.
As shown in Figure 3, it can be seen that the multi-step prun-
ing method outperforms the single-step pruning method in
terms of accuracy with a large margin.
1 1
Non-Zero Weights (%)
A challenge in the multi-step pruning process is that it usu-
ally has high computational complexity. Specifically, each step
requires fine-tuning the model by one or several epochs. If one
attempts to find the optimal pruning amount and quantization
depth for a model, the multi-step pruning process will consid-
erably delay the optimization progress. To obtain the accuracy
of the compressed model at a given pruning amount and quan-
tization depth, the model needs to be compressed first, which
usually includes many training epochs. Due to the large search
space, it is almost impossible to pre-store all the compressed
models under any combinations of pruning amount and quan-
tization depth. For example, the parameters in each layer of the
model can be quantized from 23 bits to 1 bit. The pruning
amount in each layer can range from 0 to 100%. In general, an
L-layer model can have 100 # 23 L possible combinations of
pruning amount and quantization depth, assuming 1% pruning
amount granularity.
The EMO technique is adopted to solve this problem.
However, since an evolutionary algorithm is essentially a sto-

chastic search, it may need thousands of trials (candidate solu-

0.75 0.75
tions) to find a high-quality solution. Once a new solution
Accuracy

(architecture) is produced, it takes a substantial amount of time

0.5 0.5
to perform the training for the evaluation. Consequently, it
0.25 0.25
may make the EMO-based search impossible.
To address this issue, by taking advantage of the orthogonal-
0 0
ity between pruning and quantization [45], a two-stage prun-
0 4 8 12 16 20 24 28 32 ing and quantization co-optimization method is proposed,
Time (Epoch) which can effectively reduce the computational cost. Specifical-
Multi-Step Pruning ly, the optimization process is divided into two stages. In the
Single-Step Pruning first stage, it prunes the model by multiple independent loops.
In each loop, it starts from a well-trained model, prunes the
FIGURE 3 The comparison between multi-step pruning and single-step model with a different pruning amount, fine-tunes the model,
pruning, tested on CIFAR-10 using VGG-16 (figure adopted from [15]). and saves the pruned model into a library. The set of pruning
amounts cover all the possible pruning
amounts which can be referenced by
the multi-objective solver. This is to
Pruning Amount p Pruning Amount p Quantizing Depth q guarantee that no pruning process is
required in the second stage. In the sec-
ond stage, the multi-objective solver
Prune Model Pre-Pruned Models Load Quantize Model starts to explore the design space and
Based on p Library Based on q tries to find the optimal combinations
Save of pruning amount and quantization
depth. During this process, the solver
Pre-Pruned Models
Energy Estimator Model Inference needs to know the accuracy, energy
Library
consumption, and model size under a
given combination of pruning amount
Energy and quantization depth. At this step, one
p=p+ρ Accuracy
Consumption just needs to load the corresponding
pruned model from the library and
quantize it.
(a) (b) Figure 4 shows an overview of the
proposed approach. Instead of pruning
FIGURE 4 The process of the proposed two-stage pruning and quantization co-optimization and quantizing the models at the same
method. (a) Stage-I: prune the model and save the pre-pruned models to the library; (b) Stage-
II: load the pre-pruned model from the library, quantize the parameter, and calculate the accuracy time, these two actions are taken into
as well as the energy consumption/the model size. two different stages. In the first stage, it

16 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

only prunes the model. Specifically, assuming t
as granularity, it prunes the model by 100/t Pruning and quantization are two popular techniques
times. At the i-th time step, it starts from the in DNN model compression, which show substantial
well-trained model and prunes the model
gradually using the multi-step pruning method,
improvement in energy efficiency.
until the target pruning amount reaches
i $ t. After that, it saves the compressed model into a pre-pruned first place on the image localization and the second place on
models library. In the second stage, it loads one of the pre- the image classification task in the ImageNet Large-Scale Visual
pruned models from the library based on the required pruning Recognition Challenge (ILSVRC) in 2014. LeNet-5 is a sim-
amount p, and then quantizes the parameters on the pre-pruned ple network for handwritten and machine-printed character
model based on the required quantization depth q. Since prun- recognition. It consists of only two sets of convolutional and
ing and quantization are two orthogonal operations, the final average pooling layers, followed by a flattening convolutional
compressed model will be equivalent to the compressed model layer, two fully-connected layers, and a Softmax classifier.
that is pruned and quantized at the same time. Lastly, it obtains MobileNet and VGG-16 are tested for color image classifica-
the accuracy by performing the model inference and read the tion on the CIFAR-10 dataset [42], and LeNet-5 is applied to
energy consumption from an energy estimator. recognize handwriting digits in the MNIST dataset [47].
The proposed approach can efficiently speed up the optimiza-
tion process.To obtain the accuracy and energy consumption under A. Experimental Setting
a given pruning amount and quantization depth, it does not need The NSGA-II algorithm in the python-based tool Pymoo [43]
to fine-tune the model anymore. Before the optimization process, it is used to solve the formulated multi-objective problem. The
completes the procedures in stage-I and saves only 100 compressed neural network is implemented in PyTorch4. During the net-
models into the library, assuming 1% granularity. The number of work training, the initial learning rate is set to be 0.01, and it
saved models is much less than 100 # 23 L , i.e., the number of pos- decays by half every 30 epochs. The batch size is set to be 256.
sible compressed models in the whole exploration space. For each During the multi-objective optimization process, the popula-
combination of pruning amount and quantization depth, the time tion size is set to be 40, and it runs 250 generations in each
cost of evaluating the individual is roughly equal to the inference execution. The multi-objective optimization and network
time cost of the model. training are performed on an NVIDIA Titan Xp graphics
processing unit (GPU) card. Four dataflow designs are consid-
V. Experimental Results and Analysis ered as they are the most commonly used dataflow designs
The proposed method is evaluated on three baseline CNN X : Y, C I : C O, FX : FY , and X : FX . The resource requirement is
models: MobileNet [46],VGG-16 [2] and LeNet-5 [47], which calculated based on the Xilinx Virtex UltraScale FPGA and the
have different characteristics. MobileNet is a neural network energy consumption from the Xilinx XPE toolkit [48]. In the
specially designed for mobile and embedded vision applica-
tions. VGG is a typical deep neural network, which was in the 4
PyTorch Open Source Toolkit at https://github.com/pytorch/pytorch.

X :Y (3.4) CI :CO (3.2) X :Y (23.1) CI :CO (21.2) X :Y (31.2) CI :CO (33.1)

FX :FY (4.6) X:FX (4.7) FX :FY (24) X:FX (22.8) FX :FY (31.5) X:FX (28.1)

1 2.5 6
Energy Consumption (mJ)

Energy Consumption (mJ)

Energy Consumption (mJ)

0.8 2 5

0.6 1.5 4

0.4 1 3

0.2 0.5 2

0 0 1
0.7 0.75 0.8 0.85 0.9 0.75 0.8 0.85 0.9 0.95 0.8 0.85 0.9 0.95 1
Accuracy Accuracy Accuracy
MobileNet VGG-16 LeNet-5

FIGURE 5 The solution sets obtained from the bi-objective optimization of accuracy and energy consumption on CIFAR-10 (MobileNet and VGG-
16) and MNIST (LeNet-5). The four different dataflow designs are marked with different colors. In the legends, the quoted number after the data-
flow design indicates its energy consumption (mJ) on the original model before the model compression.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 17

implementation, the multipliers and adders are implemented
on LUTs (lookup tables). An M # N multiplier requires
M/2 # (N + 1) LUTs [49]. To save the memory space, there is
no need to keep the feature map in local memory after the
computation of each layer. Hence, the size of the local memory
modules must support the weights in all layers and the tempo-
rary feature maps. The pruning and quantization approaches are
described in [10]. For pruning, the , 1 -norm based unstructured
pruning method is adopted and a mask is added to filter out
the pruned weight. For quantization, the linear (uniform)
quantization method is adopted and a scaling factor is used to
lower the precision of the weights.
B. Bi-Objective Optimization of
Accuracy and Energy Consumption
Due to the proposed two-stage pruning and quantization co-
optimization method, one can complete the model compres-
sion and obtain the solution set efficiently. The entire
optimization process includes two stages. The first stage is for
the pre-processing, which takes around 24 hours. The second
stage is for the multi-objective optimization. The solver can
generate optimal solutions within one hour by using a single
NVIDIA Titan Xp graphics processing unit (GPU) card.
Figure 5 shows the solution sets obtained from the bi-objec-
tive optimization of accuracy and energy consumption,
under the four different dataflow designs. Each point in the
figure corresponds to one compressed model in the solution
set obtained by the bi-objective optimization. From the
results, one can see that:
❏❏ The points marked in different colors cover a large range of
accuracy scores and energy consumption, which means that
EMOMC obtains a solution set with a high diversity for
the model compression of the three baseline CNN models,
COO CSR-Relative
Normal (12.5)
7.5 25
Model Size (MBytes)
Model Size (MBytes)
6 20
4.5 15
3 10
1.5 5 25
0 0
0.7 0.75 0.8 0.85 0.9 0.75
Accuracy
MobileNet
FIGURE 6 The solution sets obtained from the bi-objective optimization of accuracy and model size on CIFAR-10 (MobileNet and VGG-16) and
MNIST (LeNet-5). The three different coding schemes are marked with different colors. In the legends, the quoted number after normal coding
scheme indicates the size of the original model before the model compression.
18 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
under the four dataflow designs. For example, under the
dataflow design of X : Y, the accuracy scores of MobileNet
range from around 75% to 90%, and the energy consump-
tion from around 0.2 mJ to 0.58 mJ. It offers the right
trade-offs between the two objectives for meeting the con-
straints of various edge devices.
❏❏ From the perspective of energy consumption, if searching
solutions from the one with the highest energy consumption
to the one with the lowest energy consumption, the loss on
accuracy is negligible at the first few points. For instance,
under the dataflow design of X : Y, the energy consumption
of VGG-16 decreases from around 2.3 mJ to 0.5 mJ with an
accuracy drop less than 2%. However, after a certain thresh-
old, the accuracy loss becomes extremely large. By consider-
ing the model’s accuracy, if searching for solutions from the
one with the highest accuracy to the one with the lowest
accuracy, the reduction of energy consumption is remarkable
at the first few points. However, after a certain threshold, the
energy consumption becomes relatively stable.
❏❏ Different models prefer different dataflow designs. Specifi-
cally, C I : C O achieves the highest energy efficiency among
the four dataflow designs for MobileNet. However, it is
inferior to other dataflow designs for VGG-16. The reason is
that the convolution layers of different models have different
shapes. In addition to energy consumption, the latency and
cost of edge devices also depend on dataflow designs. The
selection of dataflow designs involves many factors, which
makes it very difficult in practice. This work explores the
optimization results on the four popular dataflow designs.
C. Bi-Objective Optimization of Accuracy and Model Size
Figure 6 demonstrates the solution sets obtained from the bi-
objective optimization of accuracy and model size, under three
COO CSR-Relative COO CSR-Relative
Normal (59.5) Normal (240)
125
Model Size (MBytes)
100
75
50
0
0.8 0.85 0.9 0.95 0.8 0.85 0.9 0.95 1
Accuracy Accuracy
VGG-16 LeNet-5
 10 125 10

8 100 8
Normalized Score

Normalized Score

Normalized Score
6 75 6

4 50 r /τ = 5 4 r /τ = 5
r/τ = 5
r /τ = 1 r /τ = 1
2 r/τ = 1 25 2
r /τ = 0.2 r /τ = 0.2
r/τ = 0.2
0 0 0
0.7 0.75 0.8 0.85 0.9 0.75 0.8 0.85 0.9 0.95 0.8 0.85 0.9 0.95 1
Accuracy Accuracy Accuracy
MobileNet VGG-16 LeNet-5

FIGURE 7 The aggregation scores on CIFAR-10 (MobileNet and VGG-16) and MNIST (LeNet-5) under three different reward over penalty ratios.
The scores are individually normalized by the aggregation score obtained by the uncompressed models.

different parameter coding schemes. Each point stands for one
compressed model in the solution set obtained by EMOMC.
The results demonstrate that in terms of diversity the solution
sets show similar patterns with the bi-objective optimization of
accuracy and energy consumption.
Furthermore, it can be observed that although COO and
CSR are developed to store a sparse matrix, sometimes they
do not save the memory space for the compressed models,
compared with the normal coding scheme. For example, if
pursuing high model accuracy, the normal coding scheme is
the best one among the three coding schemes for
MoblieNet. The reason is that although the COO and CSR
coding schemes only store non-zero elements, they still need
where f1 is the accuracy of the model, and f2 is the corre-
sponding energy consumption. When classifying an image, if
the result is correct, a reward r can be obtained; otherwise, a pen-
alty x is performed. By giving a fixed amount of energy budget,
the number of images that can be classified is inversely propor-
tional to the energy consumed per image f2 . From Equation (8),
it can be seen that one of the key parameters in this aggregation
score system is the ratio between the reward and the penalty r/x ,
which indicates the significance of accuracy. The selection of the
optimal solution highly depends on the ratio r/x .
Ratio of Energy Cons.

several extra bits to record the position of each non-zero ele- 22

ment. If attempting to keep the model accuracy at a high 21 X :Y CI :CO
level, the compression rate cannot be high, making the 1 FX :FY X:FX
memory space saved from the sparsity of the filter less than
2–1
the overhead of those extra bits. In this case, the normal cod-
2–2
ing scheme is a better choice. However, if allowing a certain 0.70 0.75 0.80 0.85 0.90 0.95
level of accuracy loss, then CSR is the best among the three Accuracy
coding schemes.
FIGURE 8 The energy consumption of VGG-16 over the energy con-
sumption of MobileNet under different accuracy scores on CIFAR-10.
D. Aggregation of Accuracy and Energy Efficiency
Theoretically, higher accuracy comes with higher energy con-
sumption. Most previous model compression approaches only
allow a negligible loss of accuracy. For applications on edge
COO CSR-Relative
devices, it will be acceptable to sacrifice a little bit of accuracy
Normal
to achieve substantial improvement in energy efficiency. For
Ratio of Model Size

23
VGG-16, as shown in Figure 5, if 2% of accuracy loss is 22
acceptable, the energy consumption can be reduced by around 21
1
80%. In the solution sets displayed in Figure 5, there are some
2–1
knee points if considering the balance of both the model accu- 2–2
racy and the energy consumption. To help users select the 2–3
0.70 0.75 0.80 0.85 0.90 0.95
model for deployment on edge devices, a new metric called
Accuracy
aggregation score is defined as:
FIGURE 9 The model size of VGG-16 over the model size of MobileNet
AScore = (f1 $ r + (1 - f1) $ x) /f2, (8) under different accuracy scores on CIFAR-10.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 19

 TABLE II Energy consumption comparison of the compressed models obtained by EMOMC
and the peer methods for VGG-16 on CIFAR-10.

ACCURACY ENERGY CONSUMPTION (MJ) EFFICIENCY IMPROVEMENT (×) AGGREGATION SCORE

METHOD LOSS X :Y CI : CO FX : FY X : FX X :Y CI : CO FX : FY X : FX r/x = 5.0 r/x = 1.0 r/ x = 0.2

EMOMC 0.3% 1.7 1.7 2.3 2.6 14.0 12.2 10.4 8.9 125.6 97.0 30.0
(OURS)
PRUNING 0.2% 18.5 19.6 18.5 19.3 1.2 1.1 1.3 1.2 1.2 1.2 1.4
FILTERS [26]
PLAY AND 0.1% 9.5 12.6 9.5 11.2 2.4 1.7 2.5 2.0 2.2 2.2 2.5
PRUNE [50]

occupies around 50% less memory

TABLE III Model size comparison of the compressed models obtained by MOMC
and the peer methods for VGG-16 on CIFAR-10.
ACCURACY MODEL SIZE (MB)
METHOD LOSS NORMAL CSR COO
EMOMC (OURS) 0.3% 9.8 8.0 24.5
PRUNING 0.2% 34.7 37.9 57.3
FILTERS [26]
PLAY AND 0.1% 15.1 16.5 24.6
PRUNE [50]
Figure 7 displays the aggregation scores of different solu-
tions under three different values of r/x . Each curve is
plotted based on the results from one execution of the
multi-objective optimization. As the multi-objective solver
generates discrete points (i.e., solutions), they are plotted in a
line using a smooth function called cspline, which connects
consecutive points by natural cubic splines after rendering
the data monotonic. The scores are individually normalized
by the aggregation score obtained by original uncompressed
models. From the results, it can be observed that most accu-
racy-energy curves have one peak point. For the complex
neural networks such as VGG-16, the highest scores will be
125 # higher than the uncompressed model due to its high
compression rate. For the simpler networks such as
MobileNet and LeNet, the highest aggregation score is
around 9 # more than the original model.
E. On Selection of Neural Networks
On the same dataset, the selection of an optimal neural net-
work depends on how one compresses the model. For instance,
MobileNet is specially designed for computation efficiency;
although its accuracy is slightly lower than VGG-16, it uses
much fewer hardware resources than VGG-16, in terms of
energy efficiency and model size. However, this statement is
true only for the original uncompressed MobileNet and VGG-
16. After model compression, VGG-16 may be more efficient
than MobileNet. Figures 8 and 9 show the ratios of energy
consumption and model size between VGG-16 and MobileNet,
in four dataflow designs and three coding schemes. The results
show that apart from dataflow design C I : C O and the normal
coding scheme,VGG-16 consumes around 50% less energy and
space than the MobileNet when the
accuracy is below 88%. This observa-
COMPRESSION RATE (×)
tion shows that although MobileNet is
NORMAL CSR COO designed for computation efficiency,
6.1 7.4 2.4 one should select a compressed model
1.7 1.6 1.0 from a more complex neural network
such as VGG-16. It is more efficient
3.9 3.6 2.4 than the compressed model from simpler
neural networks, in terms of energy effi-
ciency and model size. The reason is that
the number of the parameters or the
precision of the parameters in VGG-16 can be lower than those
of MobileNet after the mode compression.
F. Comparison to the State-of-the-Art
Tables II and III report the results of different model com-
pression methods, in terms of the energy consumption and
the aggregation scores, and model size, respectively. Table II
shows that under negligible accuracy loss (typically, less than
0.5% accuracy loss), EMOMC improves the energy efficiency
and model compression rate by a factor of 11.4 # and 5.3 #,
on average. There are two reasons for such improvements.
Firstly, the evolutionary multi-objective solver optimizes the
problem generation by generation. By allowing a certain
range of accuracy loss, it can generate many intermediate
results, and these results contribute to the improvements in
energy efficiency or compression rate. Compared with previ-
ous methods which take accuracy loss as a hard constraint,
EMOMC is more likely to find better results. Secondly, the
exploration space of the model compression process is signifi-
cantly reduced by adopting both pruning and quantization
techniques. Without the proposed two-stage pruning and
quantization co-optimization strategy, previous approaches
suffer from too high computation cost to explore and exploit
such a huge search space. In addition to energy efficiency and
compression rate, the proposed method also shows an average
84.2 # improvement on aggregation scores.
In practice, one needs to select an optimal solution
(from the solution set obtained by an EMO algorithm) for
the machine learning task on a specific device. For instance,
after solving the bi-objective optimization problem of accura-
cy and energy efficiency, a set of solutions can be obtained

20 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

which trade-off the two objectives. The constraint on the
energy can be calculated based on the energy capacity and
battery life. Then, the solution that achieves the highest accu-
racy will be selected as the optimal solution for the task.
Alternatively, one can select the knee point from the solution
set as the preferred solution.
VI. Conclusion
In this paper, an evolutionary multi-objective model compres-
sion approach is proposed to accelerate and compress DNNs by
optimizing multiple objectives (e.g., accuracy, energy efficiency,
and model size) simultaneously. As the evaluation of each archi-
tecture is extremely time-consuming during the evolution, a
two-stage pruning and optimization co-optimization strategy is
developed to speed up the architecture searching process.
Extensive experimental results demonstrate that the proposed
method can obtain a set of diverse networks in a single execu-
tion. Furthermore, the proposed method outperforms the peer
methods in terms of energy efficiency and model size for model
compression of three popular DNNs.
Acknowledgement
This work is partly supported by the Agency for Science, Tech-
nology and Research (A*STAR) under its AME Programmatic
Funding Scheme (No. A18A1b0045 and No. A1687b0033).
References
[1] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep Learning, vol. 1. Cam-
bridge MA: MIT Press, 2016, no. 2.
[2] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale
image recognition,” 2014. [Online]. Available: https://arxiv.org/abs/1804.09081
[3] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “XLNet:
Generalized autoregressive pretraining for language understanding,” in Adv. Neural Inf.
Process. Syst., 2019, pp. 5753–5763.
[4] L. Zhen, P. Hu, X. Peng, R. S. M. Goh, and J. T. Zhou, “Deep multimodal transfer
learning for cross-modal retrieval,” IEEE Trans. Neural Netw. Learn. Syst., to be published.
doi: 10.1109/TNNLS.2020.3029181.
[5] H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiable architecture search,” in
Proc. Int. Conf. Learn. Representations, 2018.
[6] Y. Sun, H. Wang, B. Xue, Y. Jin, G. G. Yen, and M. Zhang, “Surrogate-assisted evolution-
ary deep learning using an end-to-end random forest-based performance predictor,” IEEE
Trans. Evol. Comput., vol. 24, no. 2, pp. 350–364, 2020. doi: 10.1109/TEVC.2019.2924461.
[7] Y. Sun, G. G. Yen, and Z. Yi, “Evolving unsupervized deep neural networks for learn-
ing meaningful representations,” IEEE Trans. Evol. Comput., vol. 23, no. 1, pp. 89–103,
2018. doi: 10.1109/TEVC.2018.2808689.
[8] Y. Sun, B. Xue, M. Zhang, and G. G. Yen, “Completely automated CNN architec-
ture design based on blocks,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 4, pp.
1242–1254, 2020. doi: 10.1109/TNNLS.2019.2919608.
[9] Y. Sun, B. Xue, M. Zhang, and G. G. Yen, “Evolving deep convolutional neural networks
for image classification,” IEEE Trans. Evol. Comput., vol. 24, no. 2, pp. 394–407, 2020.
[10] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural net-
works with pruning, trained quantization and Huffman coding,” 2015. [Online]. Avail-
able: https://arxiv.org/abs/1510.00149
[11] Y. Wang, C. Xu, J. Qiu, C. Xu, and D. Tao, “Towards evolutionary compression,”
in Proc. ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2018, pp. 2476–2485.
[12] T.-J. Yang, Y.-H. Chen, and V. Sze, “Designing energy-efficient convolutional neu-
ral networks using energy-aware pruning,” in Proc. IEEE Conf. Comput. Vision Pattern
Recognit., 2017, pp. 5687–5695.
[13] K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, “HAQ: Hardware-aware automated quan-
tization with mixed precision,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2019.
[14] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient processing of deep neural
networks: A tutorial and survey,” Proc. IEEE, vol. 105, no. 12, pp. 2295–2329, 2017. doi:
10.1109/JPROC.2017.2761740.
[15] Z. Wang, T. Luo, J. T. Zhou, and R. S. M. Goh, “EDCompress: Energy-aware model
compression with dataf low,” 2020. [Online]. Available: https://arxiv.org/abs/2006.04588
[16] Z. Du et al., “ShiDianNao: Shifting vision processing closer to the sensor,” in Proc.
Annu. Int. Symp. Computer Architecture, 2015, pp. 92–104.
[17] M. Song et al., “Towards efficient microarchitectural design for accelerating unsu-
pervized GAN-based deep learning,” in Proc. IEEE Int. Symp. High Performance Comput.
Architecture, 2018, pp. 66–77.
[18] N. P. Jouppi et al., “In-datacenter performance analysis of a tensor processing unit,”
in Proc. Annu. Int. Symp. Comput. Architecture, 2017, pp. 1–12.
[19] M. Alwani, H. Chen, M. Ferdman, and P. Milder, “Fused-layer CNN accelerators,”
in Proc. Annu. IEEE/ACM Int. Symp. Microarchitecture, 2016, pp. 1–12.
[20] J. Qiu et al., “Going deeper with embedded FPGA platform for convolutional neu-
ral network,” in Proc. ACM/SIGDA Int. Symp. Field-Programmable Gate Arrays, 2016,
pp. 26–35.
[21] Y.-H. Chen et al., “Eyeriss: A spatial architecture for energy-efficient dataf low for
convolutional neural networks,” ACM SIGARCH Comput. Architecture News, vol. 44, no.
3, pp. 367–379, 2016. doi: 10.1145/3007787.3001177.
[22] H. Li, X. Fan, L. Jiao, W. Cao, X. Zhou, and L. Wang, “A high performance FPGA-
based accelerator for large-scale convolutional neural networks,” in Proc. Int. Conf. Field
Programmable Logic Appl., 2016, pp. 1–9.
[23] Y. Guo, A. Yao, and Y. Chen, “Dynamic network surgery for efficient DNNs,” in
Proc. Adv. Neural Inf. Process. Syst., 2016, pp. 1379–1387.
[24] F. Manessi, A. Rozza, S. Bianco, P. Napoletano, and R. Schettini, “Automated
pruning for deep neural network compression,” in Proc. Int. Conf. Pattern Recognit., 2018,
pp. 657–664.
[25] C. Lemaire, A. Achkar, and P.-M. Jodoin, “Structured pruning of neural networks
with budget-aware regularization,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit.,
2019, pp. 9108–9116.
[26] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for ef-
ficient convnets,” 2016. [Online]. Available: https://arxiv.org/abs/1608.08710
[27] Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, “AMC: AutoML for model
compression and acceleration on mobile devices,” in Proc. Eur. Conf. Comput. Vision, 2018,
pp. 784–800.
[28] Z. Liu, J. Xu, X. Peng, and R. Xiong, “Frequency-domain dynamic pruning for con-
volutional neural networks,” in Proc. Adv. Neural Inf. Process. Syst., 2018, pp. 1043–1053.
[29] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep
convolutional neural networks,” in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1097–1105.
[30] C. Szegedy et al., “Going deeper with convolutions,” in Proc. IEEE Conf. Comput.
Vision Pattern Recognit., 2015, pp. 1–9.
[31] K. Deb, “Multi-objective optimization,” in Search Methodologies. Springer-Verlag,
2014, pp. 403–449.
[32] J. D. Schaffer, “Multiple objective optimization with vector evaluated genetic algo-
rithms,” in Proc. Int. Conf. Genetic Algorithms, 1985, pp. 93–100.
[33] B. Li, J. Li, K. Tang, and X. Yao, “Many-objective evolutionary algorithms: A sur-
vey,” ACM Comput. Surv., vol. 48, no. 1, 2015. doi: 10.1145/2792984.
[34] L. Zhen, M. Li, R. Cheng, D. Peng, and X. Yao, “Adjusting parallel coordinates for
investigating multi-objective search,” in Proc. Int. Conf. Simulated Evol. Learn., Shenzhen,
China, 2017, pp. 224–235.
[35] M. Li, L. Zhen, and X. Yao, “How to read many-objective solution sets in parallel
coordinates,” IEEE Comput. Intell. Mag., vol. 12, no. 4, pp. 88–100, 2017. doi: 10.1109/
MCI.2017.2742869.
[36] D. E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning. Addi-
son-Wesley Longman Publishing, 1989.
[37] N. Srinivas and K. Deb, “Muiltiobjective optimization using nondominated sorting
in genetic algorithms,” Evol. Comput., vol. 2, no. 3, pp. 221–248, 1994. doi: 10.1162/
evco.1994.2.3.221.
[38] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective
genetic algorithm: NSGA-II,” IEEE Trans. Evol. Comput., vol. 6, no. 2, pp. 182–197,
2002. doi: 10.1109/4235.996017.
[39] C. A. C. Coello, S. G. Brambila, J. F. Gamboa, M. G. C. Tapia, and R. H. Gómez,
“Evolutionary multiobjective optimization: Open research areas and some challenges ly-
ing ahead,” Complex Intell. Syst., vol. 6, no. 2, pp. 221–236, 2020.
[40] Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf,
“NSGA-Net: Neural architecture search using multi-objective genetic algorithm,” in
Proc. Genetic Evol. Comput. Conf., 2019, pp. 419–427.
[41] H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture
search via parameters sharing,” in Proc. Int. Conf. Mach. Learn., 2018, pp. 4095–4104.
[42] A. Krizhevsky, V. Nair, and G. Hinton, “CIFAR-10 (Canadian Institute for Advanced
Research),” Tech. Rep., 2010. [Online]. Available: http://www.cs.toronto.edu/kriz/
cifar.html
[43] J. Blank and K. Deb, “Pymoo: Multi-objective optimization in python,” IEEE Access,
vol. 8, pp. 89,497–89,509, 2020.
[44] M. Zhu and S. Gupta, “To prune, or not to prune: Exploring the efficacy of prun-
ing for model compression,” 2017. [Online]. Available: https://arxiv.org/abs/1710.01878
[45] Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “A survey of model compression and
acceleration for deep neural networks,” 2017. [Online]. Available: https://arxiv.org/
abs/1710.09282
[46] A. G. Howard et al., “Mobilenets: Efficient convolutional neural networks for mobile
vision applications,” 2017. [Online]. Available: https://arxiv.org/abs/1704.04861
[47] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner et al., “Gradient-based learning applied
to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. doi: 10.1109/
5.726791.
[48] Xilinx, “Vivado design suite user guide,” Technical Publication, 2018.
[49] E. G. Walters, “Array multipliers for high throughput in Xilinx FPGAs with 6-input
LUTs,” Computers, vol. 5, no. 4, p. 20, 2016. doi: 10.3390/computers5040020.
[50] P. Singh, V. K. Verma, P. Rai, and V. P. Namboodiri, “Play and Prune: Adaptive
filter pruning for deep model compression,” 2019. [Online]. Available: https://arxiv.org/
abs/1905.04446

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 21
 ©SHUTTERSTOCK.COM/OMELCHENKO
Fast and Unsupervised
Neural Architecture Evolution
for Visual Representation
Learning

Abstract—Unsupervised visual representation learning is one

Song Xue, and Hanlin Chen of the hottest topics in computer vision, yet performance still
Beihang University, CHINA lags behind compared with the best supervised learning meth-
Chunyu Xie ods. At the same time, neural architecture search (NAS) has pro-
Qihoo 360 AI Research, CHINA; Beihang University, CHINA duced state-of-the-art results on various visual tasks. It is a
natural idea to explore NAS as a way to improve unsupervised
Baochang Zhang representation learning, yet it remains largely unexplored. In this
Beihang University, CHINA paper, we propose a Fast and Unsupervised Neural Architecture
Xuan Gong, and David Doermann Evolution (FaUNAE) method to evolve an existing architecture,
University at Buffalo, USA manually constructed or the result of NAS on a small dataset, to
a new architecture that can operate on a larger dataset. This par-
tial optimization can utilize prior knowledge to reduce search
cost and improve search efficiency. The evolution is self-super-
vised where the contrast loss is used as the evaluation metric in

Digital Object Identifier 10.1109/MCI.2021.3084394 Corresponding author: Baochang Zhang (e-mail: bczhang@buaa.edu.cn).
Date of current version: 15 July 2021 Co-first authors: Song Xue, and Hanlin Chen

22 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021 1556-603X/21©2021IEEE

a student-teacher framework. By eliminating the inferior or
least promising operations, the evolutionary process is greatly
accelerated. Experimental results show that we achieve state-of-
the-art performance for downstream applications, such as object
recognition, object detection, and instance segmentation.
L
I. Introduction
earning high-level representations from labeled data and
deep learning models in an end-to-end manner is one of
the biggest successes in computer vision in recent history.
These techniques make manually specified features largely
redundant and have greatly improved the state-of-the-art for many
real-world applications, such as medical imaging, security, and
autonomous driving. However, there are still many challenges. For
example, a learned representation from supervised learning for
image classification may lack information such as texture, which
matters little for classification but can be more relevant for later
tasks. Yet adding it makes the representation less general, and it
might be irrelevant for tasks such as image captioning. Thus,
improving representation learning requires features to be focused
on solving a specific task. Unsupervised learning is an important
stepping stone towards robust and generic representation learning
[1]. The main challenge is the significant performance gap com-
pared to supervised learning. Neural architecture search (NAS) has
shown extraordinary potential in deep learning due to the cus-
tomization of the network for target data and tasks. Intuitively,
using the target data and searching the tasks directly without a
proxy gap will result in the least domain bias. For performance rea-
sons however, early NAS [2] methods only searched small datasets.
Later, transfer methods were utilized to search for an optimal
architecture on one dataset, then adapt the architecture to work on
larger datasets [3]. ProxylessNAS [4] was proposed to search larger
datasets after directly learning the architecture for the target task
and hardware, instead of using a proxy. Recent advances in NAS
show a surge of interest in neural architecture evolution (NAE)
[3], [5]. NAE first searches for an appropriate architecture on a
small dataset and then transfers the architecture to larger dataset, by
simply adjusting weights. One key to the success of NAS or NAE
is the use of large-scale data, suggesting that more data leads to bet-
ter performance.The problem, however, is that the cost of labeling
these larger datasets may become prohibitive as their size grows.
In scenarios where we can not obtain sufficient annotations,
self-supervised learning is a popular approach to leverage the
mutual information of unlabeled training data. However, the per-
formance of unsupervised methods is still unsatisfactory compared
with the supervised methods. One obstacle is that only the param-
eters are learned using conventional self-supervised methods. To
address the performance gap, a natural idea is to explore the use of
NAE to optimize the architecture along with parameter training.
Tackling these bottlenecks will allow us to design and implement
efficient deep learning systems that will help to address a variety of
practical applications. Specifically, we can initialize with an archi-
tecture found using NAS on a small supervised dataset and then
evolve the architecture on a larger dataset using unsupervised
learning. Currently, existing architecture evolution methods [3],
[5] are inefficient and can not deal effectively with unsupervised
representation learning. Our approach is extremely efficient with
a complexity of O (n 2) where n is the size of the operation space.
Here we propose our Fast and Unsupervised Neural Architecture
Evolution (FaUNAE) method to search architectures for repre-
sentation learning. Although UnNAS [6] discusses the value of a
label and discovers that labels are not necessary for NAS, it cannot
solve the aforementioned problems because it is computationally
expensive and is trained using supervised learning for real applica-
tions. FaUNAE is introduced to evolve an architecture from an
existing architecture manually designed or searched from one
small-scale dataset on another large-scale dataset. This partial opti-
mization can utilize the existing models to reduce the search cost
and improve search efficiency. The strategy is more practical for
real applications, as it can efficiently adapt to the new scenarios’
minimal data labeling requirements.
First, we adopt a trial-and-test method to evolve the initial
architecture, which is more efficient than the traditional evolution
methods, which are computationally expensive and require large
amounts of labeled data. Second, we note that the quality of the
architecture is hard to estimate due to the absence of labeled data.
To address this, we explore contrastive loss [1] as the evaluation
metric for the operation evaluation. Although our method is built
based on contrastive loss [1], we model our method on the teach-
er-student framework to mimic supervised learning and then
estimate the operation performance even without annotations.
Then the architecture can be evolved based on the estimated per-
formance. Third, we address the fact that one bottleneck in NAS
is its explosive search space of up to 148. The search space issue is
even more challenging for unsupervised NAS built on an ambig-
uous performance estimation that further deteriorates the training
process. To address this issue, we build our search algorithm based
on the principles of survival of the fittest and elimination of the
inferior. This significantly improves search efficiency. Our frame-
work is shown in Fig. 1. Our contributions are as follows:
❏❏ We propose unsupervised neural architecture evolution,
which utilizes prior knowledge to search for an architecture
using large-scale unlabeled data.
❏❏ We design a new search block and limit the model size
according to the initial architecture during evolution to deal
with the huge search space of ResNet. The search space is
further reduced through a contrastive loss in a teacher-stu-
dent framework by abandoning operations with less poten-
tial to significantly improve the search efficiency.
❏❏ Extensive experiments demonstrate that the proposed
methods achieve better performance than prior art on Ima-
geNet, PASCAL VOC, and COCO.
II. Related Work
A. Unsupervised Learning
Recent progress on unsupervised/self-supervised1 learning
began with artificially designed pretext tasks, such as patch
1
Self-supervised learning is a form of unsupervised learning.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 23
relative location prediction [7] and rotation prediction [8]. Large
scale experiments on these pretext tasks [9] reveal that neither is
the ranking of architectures consistent across different methods
nor is the ranking of methods consistent across architectures. Thus,
pretext tasks for self-supervised learning should be considered in
conjunction with their underlying architectures.
Self-supervised representation learning can also be formu-
lated as learning an embedding where the semantically similar
images are very close, and the semantically different images are
far away. One method is to contrast positive pairs with negative
pairs. Contrastive learning [10] learns mappings that are invari-
ant to certain transformations of the inputs. He, et al. [1] build a
dynamic dictionary with a queue and a moving-averaged
encoder using a contrastive loss. He, et al. [1] update the dic-
tionary in the form of a queue, decoupling the size of the dic-
tionary and the batch size, so that the size of the dictionary
does not need to be constrained by batch size. In addition, the
Momentum-based Moving Average keeps the encoder updated
all the time. Subsequently, Chen, et al. [11] combine the meth-
ods of [1], [12], and use an MLP projection head and more
data Augmentation to further improve performance.
B. Neural Architecture Search
With the rapid development of computer vision, various net-
work structures have achieved significant performance improve-
ments, but most of them are manually designed. Recently, a
method called NAS has attracted more and more attention,
which can automate neural net architecture design [2], [13].
NAS performs full-fledged training and validation over thou-
sands of architectures from scratch. Traversing all candidate
architectures in a very large search space requires tremendous
computation and memory resources. A cell-based architecture
[3] is widely used to reduce the size of search space by stacking
cells to make NAS more efficient. However, the search space is
still noncontinuous and high-dimensional. A one-shot architec-
ture search method [14], [15] is proposed for learning efficiency,
making it possible to identify an optimal architecture within a
few GPU days. Pham, et al. [15], share parameters among child
models and trains a controller to search the network architec-
ture. Liu, et al. [14] introduce a differentiable framework that
enables NAS to use gradient descent for search. Due to the
considerable memory requirement of [14], [16] proposes a par-
tial channel connection method, which reduces memory con-
sumption by sampling a small part of the network and improves
search efficiency. Chen, et al. [17] successfully apply NAS to
model defense, sampling the network through the combination
of valid accuracy and bandit, and achieve good results in adver-
sarial training. The use of NAS for unsupervised learning is a
new research hotspot, which needs further exploration.
C. Object Detection and Segmentation
Object detection [18], [19] is one of the basic tasks of comput-
er vision. Object detection algorithms typically include one-
stage methods and two-stage methods. The two-stage methods
use a sliding window detector. First, the algorithm generates a
24 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
series of anchors as samples and then classifies the samples
through the convolutional neural network. Common algo-
rithms include R-CNN [20], Fast R-CNN [21], and Faster
R-CNN [22]. The one-stage methods use a single shot detector,
and the methods are divided into anchor-based method and
anchor-free methods. The anchor-free method does not generate
anchors and directly transforms the problem of anchor positioning
into regression. Common algorithms are YOLO [23], SSD [24]
and so on. Compared with object detection, instance segmentation
requires more stringent requirements. It not only requires seg-
menting the object’s mask and identifying its category but also
needs to distinguish different instances in the same category. The
main algorithms include Mask R-CNN [25], SOLO [26], and
SOLOv2 [27]. For object detection and segmentation, we mainly
use Faster R-CNN and Mask R-CNN. Faster R-CNN proposes
the Region Proposal Network on the basis of Fast R-CNN and
integrates feature extraction, region proposal, bounding box
regression (rect refine), and classification into one network, thereby
effectively improving detection accuracy and detection efficiency.
Mask R-CNN is a framework for instance segmentation, which
adds a branch to Faster R-CNN for segmentation. In addition,
Mask R-CNN introduces a region of interest (ROI) alignment to
replace ROI pooling in Faster R-CNN, which improves the
accuracy of the mask.
D. Neural Architecture Evolution
Before the advent of the term “deep learning,” neuroevolution
[28], [29] used genetic algorithms to change the topology along
with hyperparameters and connection weights of artificial neu-
ral networks (ANNs) by allowing high performing structures to
reproduce and mutate. Recent works focus on evolving deep
neural architecture on a larger scale. Real, et al. [13] start small
and mutates architectures to navigate large search spaces.
Encouraged by the idea of stacking the same modules in many
successful architectures such as ResNet and DenseNet, a variant
of this method [30] shows better results. It repeatedly evolves
small neural network modules by using a larger hand-coded
blueprint. Similarly, CoDeepNEAT [31] evolves the topology,
the components, and hyperparameters simultaneously for
cheaper computation. Cai, et al. [5] use reinforcement learning
as a meta-controller and Bi-LSTM as an encoder for function-
preserving transformations to make the network wider or
deeper. Zoph, et al. [3] use a recurrent neural network as a con-
troller to learn an architectural building block (cell) on a small
dataset and then transfer the block to a larger dataset. Hierarchi-
cal genetic representation is proposed to mutate from a low
level to a high level for complex topologies [32].
Although a lot of advances have been made in NAE,
few works have investigated unsupervised learning, partially
because of the inefficiency in the search process. We design
a new search block and limit the model size according to
the initial architecture during evolution. The search effi-
ciency of our method can also be significantly improved
by abandoning operations with less potential in terms of
contrastive loss.
III. FaUNAE
In this section, we elaborate on how our network efficiently

FIGURE 1 The main framework of the proposed Teacher-Student search strategy. Left: Search space of FaUNAE. Middle: Evolution process of FaUNAE. Right: FaUNAE reduces the search space by
SAConv
5×5
adapts to new datasets with unsupervised learning. We first
introduce an evolution strategy to effectively evolve the

SAConv
architecture from an existing architecture to a new one. We

3×3

Search Space
then train our network by exploiting the contrastive learn-

OUTPUT
INPUT
ing to discriminate between positive samples and negative
samples. Our framework is shown in Fig. 1.

Conv
5×5
A. Search Space
For different datasets, our architecture space is different. For

Conv
3×3
CIFAR10 [33], we search for two kinds of computation

Ω ← Ω – {arg max w(ok,n)}

cells, normal cells and reduction cells, to build the final

i
2logN
architecture. The reduction cells are located at 1/3 and 2/3

saki
Search Space
of the total depth of the network, and the rest are normal

w(ok,n) = l(ok,n) –
Reduce

k
cells. Following [14], the set of operations include skip
connection (identity), 3 # 3 convolutions, 5 # 5 convolu-

i
tions, 3 # 3 depth-wise separable convolutions, 5 # 5

i
depth-wise separable convolutions,3 # 3 max pooling,

i
Contrastive

i
3 # 3 average pooling, and zero. The search space of a cell

Loss
consists of operations on all edges.
On ImageNet, we have experimentally determined that

s0
s1
s2
t
for unsupervised learning, ResNet [34] is better than cell-
based methods for building an architecture space. We denote

Teacher

Student
SAConv

SAConv
this space as {X i}, where i represents a given block. Rather 5×5

5×5
than repeating the bottleneck (building block in ResNet)
with various operations, however, we allow a set of search
SAConv

SAConv
3×3

3×3
blocks shown in Fig.2 (a) with various operations, including
OUTPUT

OUTPUT
traditional convolution with kernel size {3, 5}, split-atten-
INPUT

INPUT
tion convolution (SAConv) [35] with kernel sizes {3, 5} and
radixes {2, 4}. SAConv is a Split-Attention block [35] which
Conv

Conv
5×5

5×5
incorporates feature map split attention within the individu-
al network blocks. Each block divides the feature map into
Conv

Conv

several groups and fine-grained subgroups or splits, where

3×3

3×3

the feature representation of each group is determined by

the weighted combination of its split representations. This
reduces the model size by sharing the 1 # 1 convolution to
Mutation and Train

improve efficiency. To enable a direct trade-off between

K Times

depth and block size (indicated by the parameters of the

selected operations), we initiate a deeper over-parameterized
network and allow a block to be skipped by adding the
eliminating the operations with the least potential.

identity operation to the candidate set of its mixed opera-

tion. So the set of operations X i in the ith block consists of
M = 7 operations. With a limited model size, the network
SAConv
5×5

can either choose to be shallower by skipping more blocks

and using larger blocks or choose to be deeper by keeping
SAConv

more blocks with smaller size.

3×3

Search Space

The initial structure a 0 is first manually designed (e.g.,

OUTPUT
INPUT

ResNet-50, without weight parameters) or searched for

by another NAS (e.g., ProxylessNAS [4]) on different
Conv
5×5

datasets in a supervised manner2, then remapped to the

search space to accelerate the evolution process and make
use of prior knowledge.
Conv
3×3

2
The evolution is based on unsupervised learning.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 25

B. Evolution Z 1 - , oi = oi
]] e k, n k, n - 1
The evolutionary strategy is summarized in Alg. 1. Unlike Amoe- p mt ^o k, nh = [ sa ik
f / sa ikl pe, otherwise
i 1 1- (1)
baNet [36], which evaluates the performance of the sub-networks ]K-1
sampled from the search space in a population, our method evolves \ kl

the operation in each block in a trial-and-test manner. We first where sa k represents the sampling times of the operation o ik .
i

mutate the operation based on its mutation probability, followed
by an evaluation step to make sure the mutation is ultimately used.
Mutation: An initial structure a 0 is manually designed (e.g.,
ResNet-50)3 or searched by another NAS (e.g., ProxylessNAS
[4]) on a different dataset using supervised learning. The initial
subnetwork fis, which is generated by searching over-parameter-
ized network based on a 0, is then trained using (6) for k steps to
obtain the evaluation metric l ^o ikh . A new architecture a n (o k, n) is
then constructed from the old architecture a n - 1 (o k, n - 1) by a
transformation or a mutation. The mutation probability p mt is
defined as
3
No weight parameters.
The operation o ik represents the kth operation in the ith block
and n represents current number of evolutions or the index of
epoch. In general, the operation in each block is kept constant
with a probability 1 - e in the beginning. For the mutation,
younger (less sample time) operations are more likely to be
selected, with a probability of e, Intuitively, keeping the opera-
tion constant can be thought of as providing exploitation,
while mutations provide exploration [36]. We use two main
mutations that we call the depth mutation and the op muta-
tion, as in AmoebaNet [36], to modify the structure generated
by the search space described above.
The operation mutation pays attention to the selection of
operations in each block. Once the operation in a block is

Performance Evaluation
InfoNCE Loss

t = fθt (xt) s0, s1, s2...

Reduce t = fθt (xt) s0, s1, s2...

Teacher Student

Teacher EMA Student

Search Space
FaUNAE TS Contrastive Learning

(a)

Conv Conv
1×1 1×1

Conv Conv Conv SAConv SAConv

3×3 fmap in Memory
3×3 5×5 3×3 5×5
fmap not in Memory

Conv Conv
1×1 1×1

1) Bottleneck 2) Search Block

(b)

FIGURE 2 (a) The main framework of the Teacher-Student model, which focuses on both the unsupervised neural architecture evolution (left)
and contrastive learning (right). (b) Compared with the original bottleneck (1) in ResNet, a new search block is designed for FaUNAE (2).” fmap
in memory” means that we select the operation to participate in the feature map calculation.

26 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

chosen to mutate, the mutation picks one of the other opera-
tions based on (1). The depth mutation can change the depth
of the subnetwork in the over-parameterized network by set-
ting the operation of one block to the ‘Identity’ operation. To
search more efficiently and evaluate operations that are more
reasonable, we limit the model size as a restriction metric. The
structure can then evolve into a subnetwork that has the same
computational burden. The probability of the restriction metric
p irm is defined as
- exp MS (o ik, n)
p rm (o ik, n) =
/ exp MS (o ikl,n) ,(2)
kl
i
where MS (o ) represents the number of parameters of the kth
k, n
operation in the ith block and n represents the index of epoch
or current number of evolutions. The final evolution probabili-
ty p that combines p mt and p rm is defined as
p (o ik, n) = m 1 ) p mt (o ik, n) + (1 - m 1) ) p rm (o ik, n), (3)
where m 1 is a hyperparameter.
Mutation validation: After each evolution, the subnet-
work is trained using (6), and the loss is used as the evaluation
metric. We observe the current validation loss a and accordingly
update the loss l (o ik, n), which historically indicates the valida-
(i, j)
tion loss of all the sampled operations o k as
l (o ik, n) = m 2 ) l (o ik, n - 1) + (1 - m 2) ) a, (4)
where m 2 is a hyperparameter. If the operation which is mutat-
ed performs better (less loss), we apply it as the base of the next
evolution; otherwise, we use the original operation as the base
of the next evolution.
o ik, n = )
o ik, n, l (o ik, n) 1 l (o ik, n - 1)
(5)
o ik, n - 1, else
C. Contrastive Learning
Contrastive learning [37] greatly improves the performance of
unsupervised visual representation learning. The goal is to keep
the negative sample pairs away and the positive samples close in
the latent space. Prior works [1], [38] usually investigate contras-
tive learning by exploring the sample pairs calculated from the
encoder and the momentum encoder [1]. Based on the investi-
gation, we reformulate the unsupervised/self-supervised NAS as
a teacher-student model, as shown in Fig. 2 (a). Following [1],
we build dynamic dictionaries, and the “keys” t in the dictionary
are sampled from data and are represented by the teacher net-
work. In general, the key representation is t = fit (x t), where
fit (.) = f (o ik, n; i t; .) is a teacher network, and x t is a key sample.
Likewise, the “query” x s is represented by s = fis (x s), where
fis = f (o ik, n; i s; .) is a student network. Unsupervised learning
performs dictionary look-up by training a network of students.
The student model and teacher model are subnetworks of NAE
from the over-parameterized network described in 3.1.
We use contrastive loss to match an encoded query s with the
encoded key dictionary to train the visual representation student
model. The value of the contrastive loss is lower when s and t are
from the same (positive) sample and higher when s and t are
from different (negative) samples. The contrastive loss is also
deployed in FaUNAE as a measure to guide the evolution of the
structure to obtain the optimal structure based on the unlabeled
dataset. InfoNCE [39] shown in Fig.2 (a), measures the similarity
using the dot product and is used as our evaluation metric:
exp (s $ t + /x)
L = - log , (6)
N
/ exp (s $ t n /x)
n=0
where x is a temperature hyperparameter per [40] and t + rep-
resents the feature calculated from the same sample with s.
InfoNCE is over one positive and M negative samples. Our
method is general and can be based on other contrastive loss
functions [37], [40]–[42]. Following [1], [43], the teacher
model is updated as an exponential moving average (EMA) of
the student model:
it = m ) i t + (1 - m) ) i s, (7)
where i s and i t are the weight of the student model and the
teacher model, respectively, updated by back-propagation in
contrastive learning, and m ! [0, 1) is a smoothing coefficient
hyperparameter.
Algorithm 1 FaUNAE.
Input: Training data, validation data, and the initial structure a 0
Parameter: Searching hypergraph
Output: Optimal structure a
 1: Let n = 1.
 2: while (K >1) do
 3:   for t = 1, ..., T epoch do
 4:    Evolve architecture a n from the old architecture a n - 1
based on the evolution probability p using (3);
 5:   Construct the Teacher model and Student model with
the same architecture a n, and then train Student mod-
els by gradient descent and update Teacher model by
EMA using (7);
  6:   Get the evaluation loss on the validation data using (6);
Use (4) to calculate the performance and assign it to all
sampled operations;
Update a n using (5);
 7:   end for
 8:   if t == K ) E then
 9:    Update w ^o ki , nh using (9);
10:    Reduce the search space: X i ! X i - " argmax w ^o ki , nh,;
11:    K ! K - 1;
k
12:   end if
13:  end while
14:  return a
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 27
D. Fast Evolution by Eliminating Operations
One of the most challenging aspects of NAS lies in the ineffi-
cient search process, and we address this issue by eliminating
the operations with the least potential. The multi-armed bandit
problem is similar to the NAS problem. The Upper Confi-
dence Bound (UCB) is used for dealing with the exploration-
exploitation dilemma in the multi-armed bandit problem.
2 log N
s = rk + , (8)
nk
where rk represents the average reward obtained by arm k. n k
represents the number of times arm k has been selected in the
total number of trials N.
After | X i | ) E epochs, we remove the operations in each
block based on performances (loss) and the sampling times.
Inspired by the UCB algorithm, we define the combination of
the two as
2 log N
w (o ik, n) = l (o ik, n) - , (9)
sa ik
where N is the total number of evolutions and mutations, and
sa ik is the number of times the kth operation of the ith block
has been selected. The first item l (o ik, n) in (9) is calculated based
on an accumulation of the validation loss, and the second term
is the exploration term. The operation with the maximum w

Airplane

Automobile

Bird

Cat

Deer

Dog

Frog

Horse

Ship

Truck

FIGURE 3 Some pictures and corresponding labels of the CIFAR10 dataset.

28 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

for every block is abandoned. This means that the operations
that are given more opportunities but result in poor perfor-
mance are removed. In order to improve search efficiency, we
eliminate the operation with the maximum w for every block
simultaneously. With this strategy, the search space which has v
blocks is significantly reduced from | X i | v to (| X i | - 1) v,
and the reduced space becomes
X i ! X i - { argmax w (o ik, n)} .(10)
k split into training, validation, and testing data with 24,640
The reduction procedure is carried out repeatedly until the
optimal structure is obtained when only one operation is left in
each block.
IV. Experiments
In this section, we describe experiments on the ImageNet and
CIFAR10 datasets and compare FaUNAE with other NAS
methods and human-designed networks. In addition, we
describe experiments on the PASCAL VOC and COCO data-
sets. The evolved architecture on ImageNet is applied as the
backbone of object detection.
A. Datasets
1) CIFAR-10
CIFAR-10 (Fig. 3) is a dataset containing 60,000 images.
Fifty thousand pictures are used for the training set, and the
remaining ten thousand are used for the test set. Each photo
is a 32 × 32 color photo, each pixel includes three values of
RGB, and the value range is 0-255. All photos belong to 10
different categories.
2) ImageNet
ImageNet is currently the world’s largest image recognition
database. It is an ongoing research effort aimed at providing an
easy-to-access image database for researchers around the world.
At present, there are tens of millions of images in ImageNet,
and more than 20000 categories.
3) PASCAL VOC
VOC2007 contains 9963 annotated pictures and 20 categories
objects marked. There are 11530 images in VOC2012. For the
detection task, the trainval of VOC2012 has 11540 images and
a total of 27450 objects.
4) MS COCO
The coco dataset is widely used in areas such as object detec-
tion and instance segmentation. COCO2017 contains 118,287
training images, 5000 validation images, and 40670 testing,
with more than 80 object categories.
B. Evolution and Training Protocol
The evolution and training protocol used in our experiments
are described in this section. We first set a global average pool-
ing and a 2-layer MLP [12] head (hidden layer 2048-d, with
ReLU), which has a fixed-dimensional output (128-d [40])
after the hypernet and searched network. The temperature x in
6 is set to 0.2 [40], and the smoothing coefficient hyperparam-
eter m in (7) is set as 0.999. We use the same data enhancement
settings as MoCoV2 [11]
Experiments first evolve the initial structure a 0 on an
over-parameterized network, which uses ResNet50 as the
backbone to build the architecture space (details can be found

skip_connect

conv_3×3 sep_conv_3×3
N_{–1} N_4
max_pool_3×3 sep_conv_5×5 N_3

N_1
conv_3×3 skip_connect Output
skip_connect N_2
N_{0}

(a)

conv_3×3
N_{–1} avg_pool_3×3 skip_connect N_4

N_1
conv_3×3
avg_pool_3×3 Output
conv_5×5
N_2 conv_5×5
N_{0} skip_connect
N_3
(b)

FIGURE 4 Detailed structure of the best cells discovered on CIFAR-10. (a) normal cells found on CIFAR10. (b) reduction cells found on CIFAR10.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 29

in 3.1) on ImageNet. We set the initial structure a 0 as a ran- train an epoch, we immediately verify the performance on the
dom structure, ResNet50, and the structure searched by Proe- validation set. In the experiment using ImageNet, we perform
xyless on ImageNet100, respectively, to show the importance unsupervised pretraining and then train a supervised linear clas-
of prior knowledge. During the architecture search, the 128M sifier for 100 epochs. Finally, we report top-1 classification
training samples of ImageNet are divided into two subsets, accuracy on the validation set.
80% for the training set, and the remainder for the validation
set for mutation validation and search space reduction. We set 1) Results on CIFAR10
the channel as half of that of ResNet50 for efficiency and pay On CIFAR10, we use an experimental setting described in
attention to the evolution of the operation rather than the MoCo v1 [1], not the settings used in v2 [11] for ImageNet. In
channel. We set the hyperparameter E = 3, so the number of addition, we use cell-based search space. In the search process,
the epoch of the search is R Mm = 2 k ) E. The hyperparameter m 1 we search for normal cells and reduction cells. The total num-
and m 2 are set to 0.9 and 0.3. After evolution, we train the ber of layers is set to 6, and reduction cells are located at 1/3
searched network on ImageNet in an unsupervised manner and 2/3. We set the intermediate nodes to M = 4, the momen-
for 200 epochs. tum to 0.9, the weight decay to 3 # 10 -4 and the initial learn-
ing rate to 0.025 described in [14]. We use 90% of the training
C. Results for Classification set as the training data in the search phase, and 10% of the
In this section, we perform experiments on CIFAR10 and training set as the validation data. After the search, we train the
ImageNet. In the experiment using CIFAR10, every time we final architecture for 600 epochs on CIFAR10. We set the total
number of cell layers to 10 and the
batch size to 128.
TABLE 1 Comparison under the linear classification protocol on CIFAR10.
We use two NVIDIA Titan V GPUs
SEARCH to search approximately 1.5 hours. We
COST (GPU SEARCH
ARCHITECTURE METHOD ACCURACY (%) PARAMS (M) DAYS) METHOD
set the initial number of channels to 36
and 100 for better performance. It can
AMDIMSMALL AMDIM [44] 89.5 194 — Manual
clearly be seen in Table 1 whether
AMDIMLARGE AMDIM [44] 91.2 626 — Manual
it is FaUNAE (channel  =  36) or
RESNET18 MOCO V1 [1] 84.09 11.1 — Manual FaUNAE (channel  =  100), the model
FAUNAE size of FaUNAE is smaller than that
(CHANNEL = 36) MOCO V1 88.83 9.1 0.125 Evolution
of AMDIM large and AMDIM small.
FAUNAE Note that the results in Table 1 are
(CHANNEL = 100) MOCO V1 92.26 69.8 0.125 Evolution
our best, but they are unstable and
we need additional trials to obtain
TABLE 2 Comparisons under the linear classification protocol on ImageNet. sults. Compared with AMDIM large,
re­­
FaUNAE (channel  =  100) achieves
ACCURACY PARAMS SEARCH COST SEARCH
ARCHITECTURE METHOD (%) (M) (GPU DAYS) METHOD
not only a better performance (91.2
vs. 92.26), but also has fewer parame-
RESNET50 INSTDISC [40] 54.0 24 — Manual
ters (626M vs. 69.8M). When com-
RESNET50 LOCALAGG [45] 58.8 24 — Manual
pared with ResNet18, FaUNAE is
RESNET101 CPC V1 [42] 48.7 28 — Manual slightly better than ResNet in terms
RESNET170WIDER CPC V2 [46] 65.9 303 — Manual of model size (11.1M vs. 9.1M), and
RESNET50L+AB CMC [47] 64.1 47 — Manual FaUNAE improved performance by
AMDIMSMALL AMDIM [44] 63.5 194 — Manual 4.74% (84.09 vs. 88.83).
AMDIMLARGE AMDIM [44] 68.1 626 — Manual
2) Results on ImageNet
RESNET50 MOCO V1 [1] 60.6 24 — Manual
During the search, we set an initial
RESNET50 MOCO V2 [11] 67.5 24 — Manual
learning rate of 0.03, a momentum
RESNET50 SIMCLR [12] 66.6 24 — Manual of 0.9, a weight decay of 0.0001, and
RANDOM MOCO V2 66.2 23 — Random then we use 8 Tesla V100 GPUs to
PROXYLESSNAS MOCO V2 67.8 23 23.1 Gradient-base search approximately 46 hours. In
FAUNAE the process of training a supervised
(RANDOM) MOCO V2 67.4 24 15.3 Evolution linear classifier, we set the initial
FAUNAE learning rate as 30 and the weight
(RESNET50) MOCO V2 67.8 24 15.3 Evolution decay 0 as described in [1]. Table 2
FAUNAE shows that FaUNAE outperforms
(PROXYLESSNAS) MOCO V2 68.3 30 15.3 Evolution
ResNet50, ResNet101, ResNet170,

30 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

 32×112×112
3×224×224

128×28×28

256×14×14
64×56×56

512×7×7
SAConv4 3×3

SAConv2 3×3

SAConv4 5×5

SAConv4 3×3

SAConv2 3×3

SAConv4 3×3

SAConv4 3×3

SAConv4 3×3

SAConv4 3×3

SAConv4 3×3

SAConv4 3×3

SAConv2 3×3

SAConv4 3×3

SAConv2 3×3

SAConv4 3×3
Conv 3×3

Conv 3×3
FIGURE 5 Detailed structures of the best structure discovered on ImageNet. “SAConv2” and “SAConv4” denote split-attention bottleneck convolu-
tion layer with radix of 2 and 4, respectively.

and AMDIM small/large with higher accuracy. Compared with 20]). Next, we perform experiments on the COCO [18] data
AMDIM large, which is by far the best method of manual set and the PASCAL VOC [19] data set, and compared FaUNAE
design that we know of, FaUNAE (ProxylessNAS) achieves with other methods.
not only a better performance (68.1 vs. 68.3), but the
model size was also reduced by about 20 times (626M vs. 1) Results on PASCAL VOC
30M). FaUNAE also performs better than the structure We use Faster R-CNN [22] as the detector and the evalu-
sampled randomly from the search space described in 3.1 ated network obtained on ImageNet as the backbone,
on Top-1 accuracy (68.3 vs. 66.2). When compared with with batch normalization tuned, implemented in [48]. All
other NAS methods which use the same search space, such layers are fine-tuned end-to-end. We fine-tune for 18k
as ProxylessNAS, FaUNAE obtains a better performance iterations (~18 epochs) on PASCAL VOC trainval07+12.
with higher accuracy (67.8 vs. 68.3) and is much faster We evaluate the default VOC metric of AP50 on the VOC
(23.1 vs. 15.3 GPU days). In addition, we calculated the test2007 set.
FLOPs of different models separately. The FLOPs of In Table 3, we can see the results on trainval07+12. FaU-
ResNet50, FaUNAE (ResNet50) and FaUNAE (Proxyless- NAE with ImageNet is better than ResNet50 with the same
NAS) are respectively: 4.11G, 3.87G and 4.08G. When the method (MoCo v2): up to + 0.9 AP50, + 2.3 AP, and + 2.6
FLOPs are approximately the same, FaUNAE (Proxyless- AP75. Also, our FaUNAE is better than the ImageNet super-
NAS) achieves better performance. vised counterparts, which shows the usefulness and effective-
We also set different initial structures a 0 including random ness of unsupervised learning.
structure, ResNet50, and the structure searched by Proxyless-
NAS on ImageNet100. As shown in Table 2, we find that the
better the initial structure, the better the performance, which TABLE 3 Results under object detection on PASCAL VOC
with Faster R-CNN.
shows the importance of the prior knowledge. For the struc-
ture (Fig. 5) obtained by FaUNAE on ImageNet, we find ARCHITECTURE METHOD AP50 AP AP75
that the structure on unsupervised learning prefers a small RESNET50 Super. 81.3 53.5 58.8
kernel size and a split-attention convolution [35], which also RESNET50 MOCO V1 [1] 81.5 55.9 62.6
shows the effectiveness of split-attention convolution and the
RESNET50 MOCO V2 [11] 82.4 57.0 63.6
rationality of FaUNAE. We searched the architecture on Ima-
FAUNAE MOCO V2 83.3 59.3 66.2
geNet for many times, which needs further research in our
future work.

D. Results on Object Detection TABLE 4 Results of object detection and instance segmentation on COCO with
and Segmentation Mask R-CNN. APbb means bounding-box AP and APmk means mask AP.
Learning transferable features is the main ARCHITECTURE METHOD APbb APbb APbb APmk mk
AP50 mk
AP75
50 75
goal of unsupervised learning. When used as
RESNET50 Super. 40.0 59.9 43.1 34.7 56.5 36.9
fine-tuning initialization for object detection
RESNET50 MOCO V1 [1] 40.7 60.5 44.1 35.4 57.3 37.6
and segmentation, ImageNet supervised pre-
training is the most influential (e.g., [21, 22, FAUNAE MOCO V2 43.1 63.0 47.2 37.7 60.2 40.6

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 31

2) Results on COCO
We apply Mask R-CNN [25] with the C4 backbone as the
detector. The rest of the settings are the same as those for the
PASCAL VOC experiment. In Table 4, we can see the results
on the COCO dataset with the C4 backbone. With the 2 #
schedule, FaUNAE is better than its ImageNet supervised
counterpart on all metrics. Due to the absent result of MoCo
v2 [11], we do not compare it with our FaUNAE. We run their
code for this comparison, which is even worse than v1. Also,
FaUNAE is better than ResNet50 trained with unsupervised
MoCo v1 [1].
V. Conclusions
We proposed a fast and unsupervised neural architecture evolu-
tion (FaUNAE) to evolve an existing architecture manually
designed or searched for on one small dataset to a new archi-
tecture on another larger dataset. The evolution is self-super-
vised, where contrast loss is used as the evaluation metric in the
student-teacher framework. The evolution process is signifi-
cantly accelerated by eliminating the inferior or the least prom-
ising operation. Experimental results demonstrate the
effectiveness of our unsupervised NAE for the downstream
applications, such as object recognition, object detection, and
instance segmentation. FaUNAE addresses core technical chal-
lenges in improving the performance of unsupervised deep
learning to make it more scalable and applicable, which will
ultimately impact many meaningful applications.
Acknowledgments
This study was supported by Grant NO.2019JZZY011101
from the Key Research and Development Program of Shan-
dong Province to Dianmin Sun. This work was supported in
part by the National Natural Science Foundation of China
under Grant 62076016 and 61876015.
References
[1] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised
visual representation learning,” in Proc. CVPR, 2020, pp. 9726–9735.
[2] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” in
Proc. ICLR, Toulon, France, Apr. 24–26, 2017.
[3] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures
for scalable image recognition,” in Proc. CVPR, 2018, pp. 8697–8710.
[4] H. Cai, L. Zhu, and S. Han, “ProxylessNAS: Direct neural architecture search on
target task and hardware,” in Proc. ICLR, 2019.
[5] H. Cai, T. Chen, W. Zhang, Y. Yu, and J. Wang, “Efficient architecture search by
network transformation,” in Proc. AAAI, 2018, pp. 2787–2794.
[6] C. Liu, P. Dollár, K. He, R. Girshick, A. Yuille, and S. Xie, “Are labels necessary for
neural architecture search?” 2020, arXiv:2003.12056.
[7] C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning
by context prediction,” in Proc. ICCV, 2015, pp. 1422–1430.
[8] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by
predicting image rotations,” in Proc. ICLR, 2018.
[9] A. Kolesnikov, X. Zhai, and L. Beyer, “Revisiting self-supervised visual representa-
tion learning,” in Proc. CVPR.
[10] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an
invariant mapping,” in Proc. CVPR, 2006.
[11] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum
contrastive learning,” 2020, arXiv:2003.04297.
[12] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for con-
trastive learning of visual representations,” 2020, arXiv:2002.05709.
32 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
[13] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. Le, and A. Kurakin,
“Large-scale evolution of image classifiers,” in Proc. ICML, pp. 2902–2911.
[14] H. Liu, K. Simonyan, and Y. Yang, “Darts: Differentiable architecture search,” in
Proc. ICLR, 2019.
[15] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean, “Efficient neural architecture
search via parameter sharing,” in Proc. ICML, 2018, pp. 4092–4101.
[16] Y. Xu, L. Xie, X. Zhang, X. Chen, G.-J. Qi, Q. Tian, and H. Xiong, “Pc-darts: Par-
tial channel connections for memory-efficient architecture search,” in Proc. ICLR, 2020.
[17] H. Chen, B. Zhang, S. Xue, X. Gong, H. Liu, R. Ji, and D. Doermann, “Anti-bandit
neural architecture search for model defense,” 2020, arXiv:2008.00698.
[18] T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C.
L. Zitnick, “Microsoft COCO: Common objects in context,” in Proc. ECCV, Springer-
Verlag, 2014, vol. 8693, pp. 740–755.
[19] M. Everingham, L. V. Gool, C. K. I. Williams, J. M. Winn, and A. Zisserman, “The
pascal visual object classes (VOC) challenge,” IJCV, vol. 88, no. 2, pp. 303–338, 2010. doi:
10.1007/s11263-009-0275-4.
[20] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for
accurate object detection and semantic segmentation,” in Proc. CVPR, 2014, pp. 580–587.
[21] R. Girshick, “Fast R-CNN,” in Proc. ICCV, 2015, pp. 1440–1448.
[22] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object
detection with region proposal networks,” in Proc. NeurIPS, 2015, pp. 91–99.
[23] J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi, “You only look once:
Unified, real-time object detection,” in Proc. CVPR, Las Vegas, NV, 2016, pp. 779–788.
[24] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed, C. Fu, and A. C. Berg,
“SSD: Single shot multibox detector,” in Proc. Comput. Vision - ECCV 2016 - 14th Eur.
Conf., Amsterdam, The Netherlands, Springer-Verlag, 2016, vol. 9905, pp. 21–37.
[25] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in Proc. ICCV,
2017, pp. 2980–2988.
[26] X. Wang, T. Kong, C. Shen, Y. Jiang, and L. Li, “SOLO: Segmenting objects by
locations,” CoRR, vol. abs/1912.04488, 2019.
[27] X. Wang, R. Zhang, T. Kong, L. Li, and C. Shen, “SOLOv2: Dynamic, faster and
stronger,” 2020, arXiv:2003.10152.
[28] D. Floreano, P. Durr, and C. Mattiussi, “Neuroevolution: from architectures to
learning,” Evol. Intell., vol. 1, pp. 47–62, 2008. doi: 10.1007/s12065-007-0002-4.
[29] K. O. Stanley and R. Miikkulainen, “Evolving neural networks through augmenting
topologies,” MIT Press Journals, 2002.
[30] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image
classifier architecture search,” in Proc. AAAI, 2019, pp. 4780–4789.
[31] R. Miikkulainen et al., “Evolving deep neural networks,” 2017, arXiv:1703.00548.
[32] H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu, “Hierarchical
representations for efficient architecture search,” in Proc. ICLR, 2018.
[33] A. Krizhevsky et al., “Learning multiple layers of features from tiny images,” 2009.
[34] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,”
in Proc. CVPR, 2016, pp. 770–778.
[35] H. Zhang et al., “Resnest: Split-attention networks,” 2020, arXiv:2004.08955.
[36] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image clas-
sifier architecture search,” in Proc. AAAI, 2019, pp. 4780–4789. Doi: 10.1609/aaai.v33
i01.33014780.
[37] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an
invariant mapping,” in Proc. CVPR, 2006, pp. 1735–1742.
[38] Y. Tian, D. Krishnan, and P. Isola, “Contrastive representation distillation,” in Proc.
ICLR, 2020.
[39] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive pre-
dictive coding,” 2018, arXiv:1807.03748.
[40] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-
parametric instance discrimination,” in Proc. CVPR, 2018, pp. 3733–3742.
[41] X. Wang and A. Gupta, “Unsupervised learning of visual representations using vid-
eos,” in Proc. CVPR, 2015, pp. 2794–2802.
[42] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A.
Trischler, and Y. Bengio, “Learning deep representations by mutual information estima-
tion and maximization,” in Proc. ICLR, 2019.
[43] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-aver-
aged consistency targets improve semi-supervised deep learning results,” in Proc. NIPS,
2017.
[44] P. Bachman, R. D. Hjelm, and W. Buchwalter, “Learning representations by maxi-
mizing mutual information across views,” in Proc. NeurIPS, 2019, pp. 15,509–15,519.
[45] C. Zhuang, A. L. Zhai, and D. Yamins, “Local aggregation for unsupervised learning
of visual embeddings,” in Proc. ICCV, 2019, pp. 6001–6011.
[46] O. J. Hénaff, A. Srinivas, J. De Fauw, A. Razavi, C. Doersch, S. Eslami, and A. v. d.
Oord, “Data-efficient image recognition with contrastive predictive coding,” 2019,
arXiv:1905.09272.
[47] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” 2019, arX-
iv:1906.05849.
[48] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2.” 2019.
https://github.com/facebookresearch/detectron2

Self-Supervised Representation
Learning for Evolutionary
Neural Architecture Search
Chen Wei
Xidian University and Xi’an University of
Posts & Telecommunications, CHINA
Yiping Tang
Xidian University, CHINA
Chuang Niu
Rensselaer Polytechnic Institute, USA
Haihong Hu,Yue Wang, and Jimin Liang
Xidian University, CHINA
Digital Object Identifier 10.1109/MCI.2021.3084415
Date of current version: 15 July 2021
1556-603X/21©2021IEEE
©SHUTTERSTOCK.COM/YURCHANKA SIARHEI
Abstract—Recently proposed neural architecture search (NAS)
algorithms adopt neural predictors to accelerate architecture
search. The capability of neural predictors to accurately predict
the performance metrics of the neural architecture is critical to
NAS, but obtaining training datasets for neural predictors is often
time-consuming. How to obtain a neural predictor with high
prediction accuracy using a small amount of training data is a
central problem to neural predictor-based NAS. Here, a new
architecture encoding scheme is first devised to calculate the
graph edit distance of neural architectures, which overcomes the
drawbacks of existing vector-based architecture encod-
ing schemes. To enhance the predictive performance of neural
Corresponding Author: Jimin Liang (email: jimleung@mail.xidian.edu.cn).
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 33
predictors, two self-supervised learning methods are proposed to
pre-train the architecture embedding part of n ­ eural predictors to
generate a meaningful representation of neural architectures.
The first method designs a graph neural network-based model
with two independent branches and utilizes the graph edit dis-
tance of two different neural architectures as a supervision to
force the model to generate meaningful architecture representa-
tions. Inspired by contrastive learning, the second method pres-
ents a new contrastive learning algorithm that utilizes a central
feature vector as a proxy to contrast positive pairs against nega-
tive pairs. Experimental results illustrate that the pre-trained
neural predictors can achieve comparable or superior perfor-
mance compared with their supervised counterparts using only
half of the training samples. The effectiveness of the proposed
methods is further validated by integrating the pre-trained neu-
ral predictors into a neural predictor guided evolutionary neural
architecture search (NPENAS) algorithm, which achieves state-
of-the-art p­ erformance on NASBench-101, NASBench-201,
and DARTS benchmarks.
N
I. Introduction
eural architecture search (NAS) refers to the use of cer-
tain search strategies to find the best performing neural
architecture in a pre-defined search space with minimal
search costs [1]. The search strategies sample potentially
promising neural architectures from the search space and perfor-
mance metrics of the sampled architectures obtained from time-
consuming training and validation procedures, which are used to
optimize the search strategies.To alleviate the time cost of training
and validation procedures, some recently proposed NAS search
strategies employed neural predictors to accelerate the perfor-
mance estimation of the sampled architectures [2]–[6]. The capa-
bility of neural predictors to accurately predict the performance
of the sampled architectures is critical to downstream search strat-
egies [2], [5]–[8]. Because of the significant time cost of obtaining
labeled training samples, acquiring accurate neural predictors
using a fewer number of training samples is one of the key issues
in NAS methods employing neural predictors.
Self-supervised representation learning, a type of unsupervised
representation learning, has been successfully applied in areas such
as image classification [9], [10] and natural language processing
[11]. If a model is pre-trained by self-supervised representation
learning and then fine-tuned by supervised learning using a few
labeled training data, then it is highly likely to outperform its
supervised counterparts [9], [10], [12]. In this paper, self-super-
vised representation learning is investigated and applied to the
NAS domain to enhance the performance of neural predictors
built from graph neural networks [13] and employed in the
downstream evolutionary search strategy.
Effective unsupervised representation learning falls into one of
two categories: generative or discriminative [9]. Existing unsuper-
vised representation learning methods for NAS [8], [14] belong to
the generative category. Their learning objective is to compel the
neural predictor to correctly reconstruct the input neural archi-
tecture, but it has limited relevance to NAS.This may result in the
34 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
trained neural predictor producing a less effective representation
of the input neural architecture. Discriminative unsupervised rep-
resentation learning, also known as self-supervised learning,
requires designing a pretext task [15], [16] from an unlabeled
dataset and using it as the supervision to learn meaningful feature
representation. Inspired by previous findings that “close by” archi-
tectures tend to have similar performance metrics [17], [18], this
paper uses the graph edit distance (GED) as supervision for self-
supervised learning, because GED can reflect the distance of dif-
ferent neural architectures in the search space. Commonly used
GED is computed based on the graph encoding of two different
neural architectures (adjacency matrices and node operations)
[17], but this scheme cannot identify isomorphic graphs. Path-
based encoding [3] is another commonly used neural architecture
encoding scheme, but it cannot recognize the position of each
operation in the neural architecture, e.g., two different operations
in a neural architecture may have the same path-encoding vectors.
To overcome the above drawbacks, this paper proposes a new
neural architecture encoding scheme, a position-aware path-based
encoding, which can recognize the positions of different opera-
tions in the neural architecture and efficiently identify unique
neural architectures.
Since different pretext tasks may lead to different feature rep-
resentations, two self-supervised learning methods are proposed
from two different perspectives to improve the feature representa-
tion of neural architectures, and to investigate the effect of differ-
ent pretext tasks on the predictive performance of neural
predictors. The first method utilizes a handcrafted pretext task,
while the second one learns feature representation by contrasting
positive pairs against negative pairs.
The pretext task of the first self-supervised learning method
is to predict the normalized GED of two different neural archi-
tectures in the search space. A graph neural network-based
model with two independent identical branches is devised and
the concatenation of the output features from both branches is
used to predict the normalized GED. After the self-supervised
pre-training, only one branch of the model is adopted to build
the neural predictor. This method is termed as self-supervised
regression learning.
The second self-supervised learning method is inspired by the
prevalently contrastive learning for image classification [9], [10],
[12], which maximizes the agreement between differently aug-
mented views of the same image via a contrastive loss in latent
space [9]. Since there is no guarantee that a neural architecture
and its transformed form will have the same performance met-
rics, it is not reasonable to directly apply contrastive learning to
NAS. This paper proposes a new contrastive learning algorithm,
termed self-supervised central contrastive learning, that uses the
feature vector of a neural architecture and its nearby neural
architectures’ feature vectors (with small GEDs) to build a cen-
tral feature vector. Then, the contrastive loss is utilized to tightly
aggregate the feature vectors of the architecture and its nearby
architectures onto the central feature vector and push the fea-
ture vectors of other neural architectures away from the central
feature vector.
 After self-supervised pre-training, two neural predictors are
built by connecting a fully connected layer to the architecture
embedding modules of the pre-trained models. Finally, the pre-
trained neural predictors are respectively integrated into a neural
predictor guided evolutionary neural architecture search (NPE-
NAS) algorithm [6] to verify their performance.
Our main contributions can be summarized as follows.
❏❏ A new position-aware path-based neural architecture encod-
ing scheme is devised to overcome the drawbacks of adjacen-
cy matrix encoding and path-based encoding methods. The
experimental results illustrate its superiority in identifying
unique neural architectures.
❏❏ A self-supervised regression learning method is proposed,
which defines a pretext task to predict the normalized GED
of two different neural architectures and design a graph neural
network-based model with two independent identical
branches to learn meaningful representation of neural archi-
tectures. The neural predictor pre-trained by this method
achieves its performance upper bound with a small search
budget and a few training epochs. In the best case, it can
achieve better performance only using half of the search bud-
get compared to its supervised counterparts.
❏❏ A self-supervised central contrastive learning algorithm is pro-
posed, which forces neural architectures with small GED to
lean closer together in the feature space, while neural architec-
tures with large GED are divided further apart.The pre-trained
neural predictor fine-tuned with a quarter of the search budget
can achieve comparable performance to its supervised counter-
parts; with the same search budget, the fine-tuned neural pre-
dictor outperforms its supervised counterparts by about 1.5
times. The proposed central contrastive learning algorithm can
also be extended to the domain of graph unsupervised repre-
sentation learning without any modifications.
❏❏ Incorporating the pre-trained neural predictors, NPENAS
achieves state-of-the-art performance on NASBench-101 [17],
NASBench-201 [19], and DARTS [20] benchmarks. On
NASBench-101 and NASBench-201 search space, the
searched neural architectures even achieve comparable or equal
results to the ORACLE baseline (performance upper bound).
II. Related Work
A. Neural Architecture Search
Due to the huge size of the pre-defined search space, NAS usually
searches for potential superiority neural network architectures by
utilizing a search strategy. Reinforcement learning (RL) [2], [21]–
[23], evolutionary algorithms [6], [24]–[28], gradient-based meth-
ods [20], [29]–[32], and Bayesian optimization (BO) [3], [6], [33]
are the commonly used search strategies. A search strategy adjusts
itself by exploiting the selected neural architectures’ performance
metrics to explore the search space better.
As it is time-consuming to estimate the performance metrics
of a given neural architecture through training and validation pro-
cedures, many performance estimation strategies are proposed to
speed up this task. Commonly used strategies include using a
proxy dataset and proxy architecture, early stopping, inheriting
weights from a trained architecture, and weight sharing [1]. A
neural predictor that is employed to estimate the performance
metrics of the neural network architectures can also be recog-
nized as a kind of performance estimation strategy. Recently,
many search strategies have adopted neural predictors to explore
the search space [2]–[4], [6]–[8], [34], [35].The capability of neural
predictors to accurately predict the performance of neural archi-
tectures is critical to the search strategies using neural predictors.
The neural predictors are trained on a dataset consisting of a
number of neural architectures, along with corresponding perfor-
mance metrics, which are obtained through time-consuming
training and validation procedures. Wen et  al. [7] designed an
architecture encoding method and proposed a neural predictor
composed of Graph Convolutional Networks (GCNs). BRP-
NAS [35] adopts GCNs to construct a latency neural predictor
and employs transfer learning to transfer knowledge from the
trained latency predictor to a binary relation predictor, which is
used to rank neural architectures. GATES [5] views the embed-
ding of neural architectures as the information flow in the archi-
tectures and presents two different encoding processes: operation
on node and operation on edge. Each operation node employs a
soft attention mask to enhance the input features, and the output
of the output node is used as the embedding of the neural archi-
tecture. SemiNAS [34] presents a semi-supervised iterative train-
ing scheme to reduce the number of architecture-accuracy data
pairs required to train a high-performance neural predictor. Semi-
NAS first trains the neural predictor with a small number of
architecture-accuracy data pairs, then utilizes the trained neural
predictor to predict the performance of a large number of archi-
tectures, and finally adds the generated pseudo data pairs to the
original training set to update the neural predictor.
In this paper, self-supervised representation learning is applied
to the NAS domain. Two self-supervised representation learning
methods are proposed to improve the feature representation of
neural predictors, thus enhancing the prediction performance of
neural predictors.
B. Neural Architecture Encoding Scheme
Neural architecture is usually defined as a direct acyclic graph
(DAG).The adjacency matrix of the graph is used to represent the
connections of operations, and the nodes are used to represent the
operations. The commonly used neural architecture encoding
schemes can be categorized into the vector encoding scheme and
graph encoding scheme.
Adjacency matrix encoding [4], [29], [36] and path-based
encoding [3] are two frequently used vector encoding schemes.
The adjacency matrix encoding is the concatenation of the flat-
tened adjacency matrix and the one-hot encoding vector of each
node, but it cannot identify the isomorphic graphs [8]. Path-based
encoding is the encoding of the input-to-output paths of the
neural architecture, but as demonstrated in the Supplementary
Materials1, this scheme cannot recognize the position of
1
The supplementary material is available at https://github.com/auroua/SSNENAS.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 35
operations in the neural architecture.The graph encoding scheme
represents the neural architecture by its adjacency matrix and the
one-hot encoding of each node.
In this paper, a new vector encoding scheme denoted as posi-
tion-aware path-based encoding is proposed. This encoding
scheme can identify different graphs more efficiently and recog-
nize the position of operations in the neural architecture. A graph
neural network is adopted to embed the neural architectures into
feature space. Since the graph encoding scheme cannot identify
isomorphic graphs [8], position-aware path-based encoding is
used to filter out isomorphic graphs. It is also utilized to calculate
the GED of different neural architectures.
C. Unsupervised Representation Learning for NAS
Unsupervised representation learning methods fall into two cate-
gories: generative and discriminative [9]. The learning objective
of existing generative unsupervised learning methods for NAS,
arc2vec [8], and NASGEM [14] is to reconstruct the input neu-
ral architectures using an encoder-decoder network, which has
little relevance to NAS. Moreover, arc2vec [8] adopts the varia-
tional autoencoder [37] to embed the input neural architectures
into a high dimensional continuous feature space, and the feature
space is assumed to follow Gaussian distribution. Since there is
no guarantee that the real underlying distribution of the feature
space is Gaussian, this assumption may harm the representation
of neural architectures. NASGEM [14] adds a similarity loss to
improve the feature representation. However, the similarity loss
only considers the adjacency matrix of the input neural architec-
ture and ignores the node operations, resulting in the failure to
identify isomorphic graphs.
In this paper, two self-supervised learning methods are pro-
posed for NAS. The first one is inspired by unsupervised graph
representation learning. GMNs [38] adopts the graph neural net-
work as building blocks and presents a cross-graph attention-
based mechanism to predict the similarity of the two input
graphs. SimGNN [39] takes two graphs as input, embeds the
graph and each node of the graph into the feature space using a
graph convolutional neural network, and then uses graph feature
similarity and node feature similarity to predict the similarity of
the input graphs. UGRAPHEMB [40] takes two graphs as input,
adopts the graph isomorphism network (GIN) [41] to embed the
input graphs into feature space, and utilizes a multi-scale node
attention mechanism to predict the similarity of the input graphs.
Our work is similar to UGRAPHEMB, but it designs a new
neural network model without using complex multi-scale node
attention and applies unsupervised learning to the field of neural
architecture representation learning.
The second method is inspired by contrastive learning for
image classification. The contrastive learning used in image classi-
fication forces the image and its transformations to be similar in
the feature space [9], [10], [42]. Since there is no guarantee that a
neural architecture and its transformed form will have the same
performance metrics, it is not reasonable to directly apply contras-
tive learning for image classification to the NAS domain. This
paper proposes a new contrastive learning algorithm,
36 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
self-supervised central contrastive learning, to learn meaningful
representation of neural architectures. To the best of our knowl-
edge, this is the first study applying contrastive learning to the
NAS domain.
III. Methodology
To enhance the prediction performance of neural predictors, two
self-supervised representation learning methods are proposed to
improve the feature representation ability of the neural predictors.
A new neural architecture encoding scheme is designed to calcu-
late the GED of graphs in Section III-B. The  self-supervised
regression learning that utilizes a carefully designed model with
two independent identical graph neural network branches to pre-
dict the GED of neural architectures is discussed in Section III-C.
The self-supervised central contrastive learning is introduced in
Section III-D. The utilization of the pre-trained neural predictors
for the downstream search strategies is elaborated in Section III-E.
A. Problem Formulation
In a pre-defined search space S, a neural architecture s can be rep-
resented as a DAG
s = ^V, E h, s ! S,(1)
where V = " v i ,i = 1: H is the set of nodes representing operations
in s, E = " v i, v j ,i, j = 1: H is the set of edges describing the connec-
tion of operations, and H is the number of nodes.
The prediction process of neural predictor can be formu-
lated as
yt = f ^ s h,(2)
where f is the neural predictor, and it takes a neural architecture s
as input and predicts the performance metric yt of s.
B. Position-Aware Path-Based Encoding
Since the proposed self-supervised learning methods utilize GED
to measure the similarity of different neural architectures, it is crit-
ical to calculate GED effectively. This paper presents a new vector
encoding scheme, position-aware path-based encoding, which
improves path-based encoding [3] by recording the position of
each operation in the path.The scheme consists of two steps: gen-
erating the position-aware path-based encoding vectors for the
input-to-output paths of the neural architecture, and concatenat-
ing the vectors of all of the paths.
As shown in Eq. 1, a neural architecture can be defined by
DAG with its nodes representing the operations in the neural
architecture. DAG consists of an input node, some operation
nodes, and an output node connected in sequence.The adjacency
matrix of DAG is used to represent the connections of the differ-
ent nodes. Since each node in DAG has a fixed position, each
node is assigned with a unique index, which implies that each
operation associated with the node has a unique index.
NASBench-101 [17] is a widely used NAS search space. It
contains three different operations: convolution 3 # 3,
convolution 1 # 1, and max-pooling 3 # 3. Figure 1a illustrates a
neural architecture in NASBench-101 search space and uses
green and red lines to indicate two different input-to-output
paths. The operations and their corresponding indices of the
neural architecture are shown in Figure 1b. Figure 1c demon-
strates two input-to-output paths of the neural architecture in
Figure 1a. Unlike path-based encoding [3], when all the input-
to-output paths are extracted from the neural architecture, the
index of operations in the input-to-output paths is also record-
ed.The position-aware path-based encoding of the two different
paths in Figure 1c is indicated in Figure 1d.The vector length of
each input-to-output path is fixed, which equals the multiplica-
tion of the number of operation nodes and the number of
operation types. All the operation nodes are traversed in the
neural architecture. If an operation node appears in the input-
to-output path, then the operation is represented by its one-hot
operation type vector; otherwise, it is represented by a zero vec-
tor. Since there are three different operations in NAS-
Bench-101, the length of the one-hot operation vector and the
zero vector is three.
The final encoding vector is the concatenation of the posi-
tion-aware path-based encoding vectors for all the input-to-out-
put paths in the neural architecture. In order to maintain the
consistency of the connection, the following operations are per-
formed sequentially:
1) Firstly, sort all the input-to-output paths in ascending order
by path length.
2) Secondly, sort all the input-to-output paths of the same path
length in ascending order by the operation index.
3) Finally, concatenate the sorted input-to-output paths’ posi-
tion-aware path-based encoding vectors.
The vector length of path-based encoding [3] increases expo-
nentially with the number of operation nodes, whereas the vector
length of the position-aware path-based encoding increases lin-
early with the number of input-to-output paths. Therefore, the
position-aware path-based encoding is a more efficient vector
encoding scheme than path-based encoding. As the number of
input-to-output paths may be different in different neural archi-
tectures, the short vectors will be padded with zeros to keep all
vectors the same length.
Since only the structural information of neural architectures is
considered, the position-aware path-based encoding scheme does
not cover other important properties of neural architectures such
as input resolution and kernel numbers. However, the position-
aware path-based encoding scheme is flexible and scalable. With
some slight modifications, the current encoding scheme can
accommodate other neural architecture settings. A modified posi-
tion-aware path-based encoding scheme, including the neural
architecture’s input resolution and kernel numbers, is presented in
the Supplementary Materials.
C. Self-Supervised Regression Learning
The pretext task of the proposed self-supervised regression learn-
ing is to predict the normalized GED of the two input neural
architectures. GED is defined as
K
GED ^s i, s j h = / p ki - p kj , s i, s j ! S,(3)
k=i

[Input, Conv 3 × 3, Conv 3 × 3,Conv 1 × 1, Max-Pool, Output] Input-to-Output Path 1 Encoding

Output 3×3
Node [1,0,0] [0,0,0] [0,1,0] [0,0,0]
Input-to-Output Path 2 Encoding
[Input, 1, 2, 3, 4, Output] [1,0,0] [1,0,0] [0,1,0] [0,0,1]
Max-Pool
Conv 3×3 (b) (d)
1×1

Input Conv Conv Output

Conv Node 3×3 1×1 Node
3×3
Input-to-Output
Path 1 1 3
Conv
3×3
Input Conv Conv Conv Max-Pool Output
Node 3×3 3×3 1×1 3×3 Node

Input Input-to-Output
Node Path 2 1 2 3 4

(a) (c)

FIGURE 1 Overview of the position-aware path-based encoding. (a) A neural architecture in NASBench-101 search space. The green and red
lines indicate two input-to-output paths. (b) Operations and their corresponding unique indices. (c) Two different input-to-output paths and
their operation indices. (d) Position-aware path-based encoding of the two input-to-output paths in (c).

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 37

where p i and p j are the position-aware path-based encoding
vectors of architecture s i and s j, p ki and p kj are the kth elements
of the position-aware path-based encoding vector p i and p j, and
K is the vector length.
Following [39], the normalized GED is defined as
GED ^s i, s j h
nGED ^s i, s j h = exp -dist, where dist = ,(4)
V
where V is the number of nodes in the neural architectures
and exp represents the exponential function with base e. The
performance of normalized GED and non-normalized GED
are compared on NASBench-201 and the result shows that
the normalized GED performs slightly better than non-nor-
malized GED.
As the architecture in search space is represented as a DAG, it
is straightforward to adopt graph neural networks to aggregate
features for each node and generate the graph embedding by
averaging the nodes’ features. In this paper, both the self-super-
vised models and neural predictors utilize the spatial-based graph
neural network GIN layers.
Since the pretext task is to predict the normalized GED of
two different neural architectures, a regression model frl consist-
ing of two independent identical graph neural network branches
is designed, as shown in Figure 2. Each branch is composed of
three sequentially connected GIN layers and a global mean pool-
ing (GMP) layer. The GMP layer outputs the mean of the node
features of the last GIN layer.The outputs of the two branches are
concatenated, and then sent to two sequentially connected fully
connected layers to predict the two input architectures’ normal-
ized GED. The regression loss function to optimize the parame-
ters w rl of frl is formulated as
w )rl = argmin / ^ frl ^s i, s j h - nGED ^s i, s j hh2.(5)
w rl (s i , s j ) ! S
After the self-supervised pre-training, any branch of frl can
be selected to embed the neural architectures into the feature
space. A neural predictor is constructed by connecting a fully
connected layer to the architecture embedding module (as
illustrated in the red rectangle of Figure 2) of the pre-trained
models. Regression loss is employed to fine-tune the neural
predictor. The parameters of the neural predictor, denoted as w,
are optimized as
w ) = argmin
w
/ ^ f^s ih - y ih2,(6)
si ! S
where y i is the performance metric of s i.
The structure of self-supervised regression learning method is
similar to the binary relation predictor proposed in BRP-NAS
[35]. However, the binary relation predictor requires the perfor-
mance metrics of neural architectures to learn how to rank
architectures, and the performance metrics are very time-con-
suming to obtain.The self-supervised regression learning method
uses GED as supervision to learn a meaningful representation of
the neural architectures, and the time-cost of computing GED
can be neglected.
D. Self-Supervised Central Contrastive Learning
This paper proposes a central contrastive learning algorithm to
force neural architectures with a small GED to lean closer togeth-
er in the feature space, while neural architectures with a large
GED are divided further apart.
As illustrated in Figure 3, a graph neural network model fccl is
developed to embed the neural architecture into feature space.
Following SimCLR [9], fccl consists of a neural architecture
embedding module and two fully connected layers. For a fair
comparison, the architecture embedding module is identical to
that of frl .
Given a batch of neural architectures S b = " s k ,Nk = 1 and a neu-

ral architecture s i ! S b, the minimum GED of s i from all other

architectures is denoted as g min . The neural architectures whose
Architecture GED from si is equal to g min, together with s i, constitute the
Embedding
positive sample set S pos of s i . The set of neural architectures in
Performance
GMP
GIN

GIN

GIN

FC

Input Predictor this batch but not in S pos is denoted as S neg . The model fccl is
used to embed all of the neural architectures, and the feature vec-
pred
FC
FC

tor sets of S pos and S neg are denoted as E pos and E neg, respective-
ly. A central vector e c is calculated as the average of all of the
GMP
GIN

GIN

GIN

feature vectors in E pos . The contrastive loss is used to aggregate all

FC

Input

Architecture of the feature vectors in E pos to the central vector e c and push
Embedding the feature vectors in E neg far away from e c . The detailed proce-
dure of central contrastive learning is summarized in Algorithm 1.
FIGURE 2 Structure of the regression model frl. An example of the central contrastive learning is illustrated in the
Supplementary Materials.
To reduce the interaction between the central vectors, a cen-
Architecture Embedding tral vector regularization term is added to the loss function that
forces each pair of the central vectors to be orthogonal. The cen-
Embedding
GMP
GIN

GIN

GIN

tral vector regularization term is defined as

FC

FC

Input Feature
M M
L reg = 1 / / 1 [ j ! i] e i< e j,(7)
FIGURE 3 Structure of the feature embedding model fccl. 2 i=0 j=0

38 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

where M is the number of central vectors, and e i and e j are the paper, the pre-trained neural predictors are integrated respectively
central vectors. with NPENAS to illustrate the performance gains that result
After the self-supervised central contrastive pre-training, a from applying self-supervised learning to NAS.
fully connected layer is appended to the architecture embed- Since our experiments demonstrate that the neural predictor
ding module to predict the input neural architecture’s perfor- built from a self-supervised pre-trained model can significantly
mance. The regression loss in Eq. 6 is adopted to fine-tune the outperform its supervised counterpart and achieve comparable
neural predictor. performance with smaller training datasets, the NPENAS method
is modified to utilize only a fixed search budget to carry out the
E. Fixed Budget NPENAS neural architecture search. The fixed budget NPENAS is summa-
NPENAS [6] combines the evolutionary search strategy with a rized in Algorithm 2, which is modified from NPENAS and only
neural predictor and utilizes the neural predictor to guide the the differences are presented.
evolutionary search strategy to explore the search space. In this
IV. Experiments and Analysis
In this section, experiments are conducted to illustrate that the
performance of the proposed neural predictors can be significant-
Algorithm 1 Central Contrastive Learning. ly improved by self-supervised learning. The benefit of self-super-
 1: Input:  batch size N, number of training architectures M and vised learning for NAS is also demonstrated by integrating the
M # N, temperature x, regularization weight m, model fccl . pre-trained neural predictors with NPENAS.
 2: for sampled minibatch S b = " s k ,Nk = 1 do All experiments are implemented in Pytorch [43]. GIN is
 3:   E c = Q implemented using the publicly available graph neural network
 4:   for all t ! " 1, f, M , do library pytorch_geometric [44].The code of this paper is provided
  5:    Randomly draw one neural architecture s i ! S b . in reference [45].
 6:    g min = min GED (s i, s j) where j = " 1, f, i -1, i +1, f, N ,
   # GED:  Eq. 3 A. Benchmark Datasets
 7:    E pos = Q and E neg = Q All experiments are performed on NASBench-101 [17], NAS-
 8:    for all j ! " 1, f, i - 1, i + 1, N , do Bench-201 [19], and DARTS [32] search spaces.
 9:     e j = fccl (s j)
10:    if GED (s i, s j) == g min then a) NASBench-101
11:      E pos ! E pos , e j NASBench-101 [17] contains 423k neural architectures, and each
12:    else architecture is trained three times on the CIFAR-10 [46] training
13:      E neg ! E neg , e j datasets independently.The structure of the neural architectures, as
14:    end well as their validation accuracies, and test accuracies correspond-
15:    end for ing to the three independent trainings on CIFAR-10 are report-
16:    E pos ! E pos , e i ed. The architecture in this search space is defined by DAG, utilizing
nodes to represent the operations of the neural architecture and
17:    e c = 1 ) / e # vector average
E pos e ! Epos
18:    E c ! E c , e c
19:    for all idx, e p ! E pos do # idx is the index of e p in Epos
Algorithm 2 Fixed Budget NPENAS.
20:     E pair = Q
21:     sim p, c = e <p e c / ^x e p e c h Input: initial population size n 0 , initial population D = " (s i, y i),
1: 
22:     E pair ! E pair , sim p, c i = 1, 2, f, n 0 ,, neural predictor f, number of the total
23:    for all e n ! E neg do searched architectures total_num, number of the evaluated
24:      sim n, c = e <n e c / ^x e n ec h architectures (budget) to fine-tune neural predictor ft_num.
25:      E pair ! E pair , sim n,c for n from n 0 to total_num do
2: 
26:    end for 3:  if n # ft_num then
exp (sim p, c)
27:     l t, idx = - log 4:    Initialize the weights of neural predictor f with the
/ exp (sim vec)
weights from the pre-trained model.
sim vec ! E pair

28:    end for 5:    Fine-tune the neural predictor f with dataset D = " (s i, y i),
29:    l t = / l t, idx i = 1, 2, f, n , .
idx
6:  end
30:   end for
M 7:  Utilize the neural predictor f to guide the evolutionary neu-
31:   L = 1 / l t + m L reg (E c) # Lreg:  Eq. 7
M t=1 ral architecture search. # Detailed code can be found in
32:   Update model fccl to minimize L. Algorithm 2 of NPENAS [6].
33:  end for end for
8: 
34:  return model fccl Output: s ) = argmin ( y i), (s i, y i) ! D
9: 

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 39

using the adjacency matrix to represent the connection of differ-
ent operations. Only convolution 1 # 1, convolution 3 # 3, and
max-pooling 3 # 3 are allowed to be used to build the neural
architectures. The best architecture achieves a mean test error of
5.68%, and the mean test error of the architecture with the best
validation error is 5.77%.
b) NASBench-201
NASBench-201 [19] is a recently proposed NAS benchmark, and
it contains 15.6k trained architectures for image c­lassification.
Each architecture is trained once on the CIFAR-10, CIFAR-100
[46], and ImageNet-16-120, and ImageNet-16-120 is a down-
sampled variant of ImageNet [47].The structure of each architec-
ture and its evaluation details such as training error, validation
error, and test error of each architecture are reported. Each archi-
tecture is defined by a DAG, utilizing nodes to represent the fea-
ture maps and using the edges to represent the operation. The
convolution 1 # 1, convolution 3 # 3, average pooling 3 # 3,
skip connection, and zeroize operation are allowed to be used to
construct the neural architectures. On CIFAR-10, CIFAR-100,
and ImageNet-16-120, the best test percentage errors are 8.48%,
26.49%, and 52.69%, respectively. On CIFAR-10, CIFAR-100,
and ImageNet-16-120, architectures with the best validation
error achieve the test percentage errors of 8.91%, 26.49%, and
53.8%, respectively.
c) DARTS
Unlike the two search spaces above, DARTS [32] is a more realis-
tic-sized search space that contains 10 18 neural architectures. The
architectures in DARTS are not evaluated by the training and val-
idation procedures. The optimal normal and reduction cells need
to be found in the search space, and the final neural network is
composed of sequentially connected normal and reduction cells.
Each cell has two inputs and four nodes, each with two opera-
tions. Each node can use the following operations: 3 # 3 and
5 # 5 separable convolutions, 3 # 3 and 5 # 5 dilated separable
convolutions, 3 # 3 max pooling, 3 # 3 average pooling, identity,
and zero.
B. Position-Aware Path-Based Encoding Analysis
In this section, the improvement and limitation of the posi-
tion-aware path-based encoding scheme are analyzed. Experi-
ments on NASBench-101 and NASBench-201 search spaces
are conducted to confirm the proposed encoding scheme’s
ability to uniquely identify neural architectures. NAS-
Bench-101 and NASBench-201 contain 423k and 6466
TABLE I Unique encoding vectors of different encoding
methods.
PATH- POSITION-AWARE UNIQUE
SEARCH SPACE BASED PATH-BASED ARCHS*
NASBENCH-101 170K 423K 423K
NASBENCH-201 8046 9741 6466
*ARCHITECTURES is shortened as ARCHS.
40 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
unique topology structures [17], [19], respectively. All neural
architectures are encoded separately using the path-based
encoding and position-aware path-based encoding schemes,
and then the numbers of unique neural architecture encodings
are counted respectively.
As shown in Table I, the position-aware path-based encoding
can identify all the unique neural architectures in the NAS-
Bench-101 search space, while path-based encoding maps many
unique neural architectures to the same encodings. In the NAS-
Bench-201 search space, both encoding schemes fail to identify
the 6466 unique neural architectures. The failure of the position-
aware path-based encoding scheme is attributed to the presence
of skip connection operations and zeroize operations in the
NASBench-201 search space. This is a limitation of the proposed
encoding scheme.
Since the vector length of position-aware path-based encod-
ing only increases linearly with the number of input-to-output
paths, it is a more efficient encoding method than path-based
encoding. For the NASBench-101, NASBench-201, and
DARTS search spaces, the encoding vector lengths of the posi-
tion-aware path-based encoding scheme are 120, 96 and 1224,
respectively, while those of the path-based encoding are 364, 5461
and 18724, respectively.
C. Prediction Analysis
1) Model Details
This paper first utilizes self-supervised regression learning to train
the model frl in Figure 2 and self-supervised central contrastive
learning to train the model fccl in Figure 3. The architecture
embedding module consists of three sequentially connected
GIN layers. The hidden layer size of the GIN layer is 32, and
each GIN layer is followed with a batch normalization and a
ReLU layer. The hidden dimension size of the fully connected
layer of the model frl and fccl is 16 and 8, respectively. After self-
supervised pre-training, the neural predictors are constructed by
connecting the pre-trained architecture embedding modules
with a single fully connected layer with the hidden dimension
size of 8. The neural predictors constructed by the architecture
embedding modules of frl and fccl are denoted as SS-RL and
SS-CCL, respectively.
The same neural architecture encoding method as NPE-
NAS is employed [6]. The architecture in NASBench-101 is
represented by a 7 # 7 upper triangle adjacency matrix and a
collection of 6-dimensional one-hot encoded node features,
and that in the NASBench-201 is represented by an 8 # 8
upper triangle matrix and several 8-dimensional one-hot
encoded node features.
2) Training Details
Self-supervised regression learning requires the construction of
paired training samples from the training data, and the number of
paired training samples increases by the square of the number of
training data. When the number of neural architectures in the
search space is large, it is impractical to train the self-supervised
regression learning model using all paired training samples within
a reasonable time cost. For example, on NASBench-101, it costs
about 1.7 minutes to train the self-supervised regression learning
model if 380k (the number equals to 90% of the total number of
architectures) of the paired training samples are used. If the model
is trained with all the paired samples ^^1 + 423kh # 423k # 0.5h
for 300 epochs, it will take about 252 years. Therefore, in our
experiments, a fixed number of paired training samples are ran-
domly selected to pre-train the model frl . The number is set to
90% of the total number of the neural architectures in the search
space, i.e., 380k for NASBench-101 and 13.7k for NAS-
Bench-201. The training epoch is 300 and the batch size is 64.
The Adam optimizer [48] is employed to optimize the parameters
of the model frl, the initial learning rate is 5e - 4, and the
weight decay is 1e - 4. A cosine learning rate schedule [49]
without restart is adopted to anneal down the learning rate to
zero. On NASBench-201, the initial learning rate is 1e - 4 and it
is trained for 1000 epochs. Other training details are the same as
NASBench-101.
The self-supervised central contrastive learning utilizes all of
the architectures in NASBench-101 to pre-train the model fccl .
The training epoch is 300, the regularization weight m is 0.5, and
the temperature x is 0.07. The batch size is 140k, and the train-
ing architectures are 140k. When pre-training on NAS-
Bench-201, the batch size is 10k, and the training architectures
are 1k. For both search spaces, the initial learning rate is 5e - 3.
Other training details like the optimizer, weight decay, and the
learning rate schedule are identical with those for self-supervised
regression learning.
The initial learning rates of the supervised neural predictor
on NASBench-101 and NASBench-201 are 5e - 3 and
1e - 3, respectively. The other training details of the supervised
neural predictor are the same as self-supervised central contras-
tive learning.
After pre-training, the neural architectures and their corre-
sponding validation accuracies are used to fine-tune the neural
predictors. SS-RL and SS-CCL are fine-tuned with an initial
learning rate of 5e - 5 and 5e - 3, respectively. The weight
decay is 1e - 4. The optimizer and the learning schedule are the
same as the self-supervised pre-training.
3) Setup
The search budget and training epochs of neural predictors
directly affect the time cost of NAS. The supervised neural
predictor, SS-RL and SS-CCL are compared under the search
budgets of 20, 50, 100, 150, and 200. According to a search
budget, neural architectures are randomly selected from the
search space as the training dataset, and the remaining neural
architectures constitute the test dataset. To illustrate the effect
of the training epochs, the neural predictors with different
search budgets are trained under 50, 100, 150, 200, 250, and
300 training epochs. After fine-tuning, the correlation between
the validation accuracy of the neural architectures and their
performance predicted by the neural predictors is evaluated
using the Kendall tau rank correlation. All the experimental
results are averaged over 40 independent runnings using differ-
ent random seeds.
4) Results
The predictive performance measurements of the neural predic-
tors on NASBench-101 and NASBench-201 are shown in Fig-
ure 4 and Figure 5, respectively.
On the NASBench-101 search space, SS-RL achieves its best
performance with fewer training epochs and gradually decreases
with more training epochs, and finally drops to a range of 0.2 to
0.3. The supervised neural predictor performs significantly better
than SS-RL when the training epoch is above 150 and the
search budget is more than 100. The predictive performance of
SS-CCL is significantly better than that of SS-RL and the super-
vised neural predictor. It increases with the number of training
epochs, and approaches saturation when the search budget
exceeds 150. In addition, SS-CCL achieves better performance
than the supervised neural predictor while using half of the
number of training neural architectures. In extreme cases,
SS-CCL achieves comparable performance to the supervised
neural predictor using only a quarter of the training neural archi-
tectures (Figure 4e and Figure 4f).
On the NASBench-201 search space, SS-RL and SS-CCL
have comparable performance and both consistently outper-
form the supervised neural predictor. SS-CCL slightly out-
performs SS-RL when trained for more than 150 epochs. As
shown in Figure 5b and Figure 5f, even with fewer training
epochs, SS-RL and SS-CCL can approach their optimal per-
for mance. When trained over 200 epochs, SS-RL and
SS-CCL can outperform the supervised neural predictor using
only a quarter of the training samples (Figure 5e, Figure 5f).
Nvidia TITAN V GPUs are used to train the two self-super-
vised representation learning methods and report the time-cost in
Table II. When trained on the NASBench-101 search space with
the same epochs, the time cost of self-supervised central contras-
tive learning is about six times higher than that of self-supervised
regression learning. On the small search space of NASBench-201,
their training time costs are comparable and both methods are
quite efficient in training.
The above results show that the performance of SS-RL on
the small search space of NASBench-201 is comparable to that of
TABLE II Time cost of self-supervised training.
SEARCH TRAINING BATCH GPU
METHODS SPACE EPOCHS SIZE DAYS
Self-Supervised
Regression 101* 300 — 0.92
Self-Supervised Cen-
tral-Contras* 101* 300 14K 5.97
Self-Supervised
Regression 201* 1000 — 0.05
Self-Supervised Cen-
tral-Contras* 201* 300 10K 0.02
*NASBench-101, NASBench-201 and Central Contrastive are shortened as 101, 201
and Central-Contras, respectively.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 41
 Training Epoch: 50 Training Epoch: 100
0.6 0.6

0.5 0.5
Kendall Tau Correlation

Kendall Tau Correlation

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0.0 0.0
20 50 100 150 200 20 50 100 150 200
Search Budget Search Budget
(a) (b)
Training Epoch: 150 Training Epoch: 200
0.6 0.6

0.5 0.5
Kendall Tau Correlation

Kendall Tau Correlation

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0.0 0.0
20 50 100 150 200 20 50 100 150 200
Search Budget Search Budget
(c) (d)
Training Epoch: 250 Training Epoch: 300
0.6 0.6

0.5 0.5
Kendall Tau Correlation

Kendall Tau Correlation

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0.0 0.0
20 50 100 150 200 20 50 100 150 200
Search Budget Search Budget
(e) (f)
SUPERVISED SS-RL SS-CCL

FIGURE 4 Predictive performance of neural predictors on NASBench-101 with different training epochs.

42 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

 Training Epoch: 50 Training Epoch: 100
0.6 0.6

0.5 0.5
Kendall Tau Correlation

Kendall Tau Correlation

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0.0 0.0
20 50 100 150 200 20 50 100 150 200
Search Budget Search Budget
(a) (b)
Training Epoch: 150 Training Epoch: 200
0.6 0.6

0.5 0.5
Kendall Tau Correlation

Kendall Tau Correlation

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0.0 0.0
20 50 100 150 200 20 50 100 150 200
Search Budget Search Budget
(c) (d)
Training Epoch: 250 Training Epoch: 300
0.6 0.6

0.5 0.5
Kendall Tau Correlation

Kendall Tau Correlation

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0.0 0.0
20 50 100 150 200 20 50 100 150 200
Search Budget Search Budget
(e) (f)
SUPERVISED SS-RL SS-CCL

FIGURE 5 Predictive performance of neural predictors on NASBench-201 with different training epochs.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 43

SS-CCL, but on the large search space of NASBench-101, its
performance drops significantly. We conjecture that this is because
SS-RL does not get enough paired training samples and training
time, so it is difficult to learn to generate meaningful feature rep-
resentations. This suggests that self-supervised regression learn-
ing may not be suitable for large search space. Unlike
self-supervised regression learning, self-supervised central con-
trastive learning does not need paired training samples. The
optimization purpose of central contrastive learning is more
intuitive, which pulls similar neural architectures to gather close-
ly in the feature space and push dissimilar neural architectures
far away. When the search space is large, we advocate the adop-
tion of the self-supervised central contrastive learning method,
and self-supervised regression learning can be considered when
the search space is small.
D. Effect of Batch Size
Since the number of negative pairs of central contrastive learning
is determined by the batch size, in this section, experiments are
conducted to investigate the effect of batch size on the perfor-
mance of neural predictors. The batch size N is set to 10k, 40k,
70k, and 100k, and the corresponding neural predictors are
denoted as SS-CCL-10k, SS-CCL-40k, SS-CCL-70k, and SS-
CCL-100k, respectively. The number of training ­architectures M
is set to half of the batch size. To compare the performance of
neural predictors with larger M, SS-CCL pre-trained with
N = 140k and M = 140k is also included, and it is denoted as
SS-CCL-140k. The initial learning rate of the above neural pre-
dictors is set to 5e - 3, and all the experiments in this section are
carried out on NASBench-101. Other experimental settings are
the same as in Section IV-C. All results are averaged over 40 inde-
pendent runnings using different seeds.
As shown in Figure 6, SS-CCL outperforms the supervised
neural predictor despite the batch size used. When the search
budget is greater than 100 and the training epoch is greater
than 100, the performance of SS-CCL trained with different
batch sizes tends to be the same (Figure 6c-6f). Unlike the
findings in the study of contrast learning-based image classifica-
tion [9], i.e., where model performance increases consistently
with batch size, using larger batch sizes does not improve the
performance of SS-CCL. The results demonstrate that SS-CCL
is not sensitive to batch size when the search budget is large.
Therefore, when using the self-supervised central contrastive
learning method to pre-train the neural predictor’s embedding
part, a relatively small batch size can be selected to obtain better
memory efficiency.
E. Predictive Performance Comparison
The neural predictors proposed in this paper are compared with
SemiNAS [34], Multilayer Perceptron (MLP) [4], and that pro-
posed in Wen et al. [7] (denoted as NP-NAS). The comparison is
performed on NASBench-101, and all the predictors are trained
for 200 epochs under the search budgets of 20, 50, 100, and 200.
The experimental results are averaged over 40 independent run-
nings using different random seeds.
44 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
As illustrated in Figure 7, the neural predictor SS-CCL
achieves the best performance for all of the search budgets.
The supervised neural predictor outperforms SemiNAS and
MLP when the search budget is less than 100, but it is sur-
passed by MLP and NP-NAS when the search budget is
greater than 100. Since self-supervised regression learning is
not efficient for learning meaningful representations when the
search space has numerous neural architectures, the perfor-
mance of SS-RL is only slightly better than that of SemiNAS,
which has the worst performance.
F. Fixed Budget NPENAS
1) Setup
The pre-trained neural predictors are integrated with NPENAS
[6] and the integration of SS-RL and SS-CCL with NPENAS
are denoted as NPENAS-SSRL and NPENAS-SSCCL, respec-
tively. The fixed budget version of NPENAS-SSRL and NPE-
NAS-SSCCL are denoted as NPENAS-SSRL-FIXED and
NPENAS-SSCCL-FIXED, respectively. The experimental set-
tings are the same as NPENAS [6]. The algorithms of random
search (RS) [50], regularized evolutionary (REA) [25], BANAN-
AS [3] with path-based encoding (BANANAS-PE), BANANAS
with position-aware path-based encoding (BANANAS-PAPE),
NPENAS-NP [6], and NPENAS-NP with a fixed search budget
(NPENAS-NP-FIXED) are compared to illustrate the benefit of
the two self-supervised learning methods for NAS. Each algo-
rithm has a search budget of 150 and 100 on the NASBench-101
and NASBench-201 search space, respectively. The fixed search
budgets for NASBench-101 and NASBench-201 are set to 90
and 50, respectively. All the experimental results are averaged over
600 independent trails, every update of the population, each algo-
rithm returns the architecture with the lowest validation error so
far and reports its test error, so there are 15 or 10 best architec-
tures in total. The methods proposed in this paper are also com-
pared with arc2vec [8] that is a recently proposed unsupervised
representation learning for NAS. As the search strategies employ
the neural architectures’ validation error to explore the search
space, a reasonable best performance of NAS is the test error of
the neural architecture that has the best validation error in the
search space, which is called the ORACLE baseline [7]. The
ORACLE baseline is used as the upper bound of performance.
Since self-supervised central contrastive learning is more suitable
for a large search space, the experiments on the DARTS search
space are conducted only using the self-supervised central contras-
tive learning method. As the DARTS search space contains 1018
neural architectures, it is not possible to train the self-supervised
central contrastive representation learning model using all of the
neural architectures.The model is trained with 50k randomly sam-
pled neural architectures.The training details on the DARTS search
space are the same as those on NASBench-101. After self-super-
vised pre-training, NPENAS-SSCCL-FIXED is adopted to search
architectures in the DARTS search space. The NPENAS-SSCCL-
FIXED has a search budget of 100 but can only evaluate 70
searched architectures. When the number of searched architectures
 Training Epoch: 50 Training Epoch: 100
0.6 0.6

0.5 0.5
Kendall Tau Correlation

Kendall Tau Correlation

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0.0 0.0
20 50 100 150 200 20 50 100 150 200
Search Budget Search Budget
(a) (b)
Training Epoch: 150 Training Epoch: 200
0.6 0.6

0.5 0.5
Kendall Tau Correlation

Kendall Tau Correlation

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0.0 0.0
20 50 100 150 200 20 50 100 150 200
Search Budget Search Budget
(c) (d)
Training Epoch: 250 Training Epoch: 300
0.6 0.6

0.5 0.5
Kendall Tau Correlation

Kendall Tau Correlation

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0.0 0.0
20 50 100 150 200 20 50 100 150 200
Search Budget Search Budget
(e) (f)

SUPERVISED SS-CCL-10K SS-CCL-40K SS-CCL-70K SS-CCL-100K SS-CCL-140K

FIGURE 6 Predictive performance of neural predictors pre-trained by different batch size N on NASBench-101.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 45

 Training Epoch: 200
0.6 7.4
RS

Testing Error of Best Neural Net

7.2 REA
0.5 BANANAS-PE
7 BANANAS-PAPE
Kendall Tau Correlation

0.4 NPENAS-NP
6.8 NPENAS-SSRL
NPENAS-SSCCL
0.3 6.6

6.4
0.2 MLP
NP_NAS 6.2
SS-CCL
0.1 SS-RL 6
SUPERVISED
SemiNAS 5.8
0.0
10 30 50 70 90 110 130 150
20 50 100 200 Number of Samples
Search Budget

FIGURE 7 Performance comparison of different neural predictors. FIGURE 8 Performance comparison of NAS algorithms on NASBench-101.

TABLE III Performance comparison of NAS algorithms on NASBench-101.

METHODS SEARCH BUDGET TEST ERR (%) AVG ARCHITECTURE EMBEDDING SEARCH METHOD
RS [50] 150 6.42 ± 0.20 – RANDOM SEARCH
REA [25] 150 6.32 ± 0.23 DISCRETE EVOLUTION
BANANAS-PE [3] 150 5.90 ± 0.15 SUPERVISED BAYESIAN OPTIMIZATION
BANANAS-AE [3] 150 5.85 ± 0.14 SUPERVISED BAYESIAN OPTIMIZATION
BANANAS-PAPE [3] 150 5.86 ± 0.14 SUPERVISED BAYESIAN OPTIMIZATION
NPENAS-NP [6] 150 5.83 ± 0.11 SUPERVISED EVOLUTION
†
NPENAS-NP-FIXED [6] 90 5.90 ± 0.16 SUPERVISED EVOLUTION
ARCH2VEC-RL [8] 400 5.90 UNSUPERVISED REINFORCE
ARCH2VEC-BO [8] 400 5.95 UNSUPERVISED BAYESIAN OPTIMIZATION
NPENAS-SSRL 150 5.86 ± 0.14 SELF-­SUPERVISED EVOLUTION
NPENAS-SSRL-FIXED 90† 5.88 ± 0.15 SELF-­SUPERVISED EVOLUTION
NPENAS-SSCCL 150 5.83 ± 0.11 SELF-­SUPERVISED EVOLUTION
NPENAS-SSCCL-FIXED 90† 5.85 ± 0.13 SELF-­SUPERVISED EVOLUTION
†
The neural predictor is trained with 90 evaluated neural architectures, while other algorithms use 150 neural architectures for evaluation.

TABLE IV Impact of fixed search budget on NASBench-101. is greater than 70, the performance of the extra sampled neural
architectures is predicted by the neural predictor. Upon comple-
METHODS SEARCH BUDGET† TEST ERR (%) AVG
tion of the search, the best performing neural architecture is
NPENAS-SSRL 20 6.04 ± 0.25 selected for evaluation on the CIFAR-10 dataset, and the archi-
NPENAS-SSRL 50 5.94 ± 0.18 tecture is evaluated five times with different seeds. All the other
NPENAS-SSRL 80 5.87 ± 0.14 evaluation settings are the same as DARTS [32]. These experi-
NPENAS-SSRL 110 5.86 ± 0.11 ments are executed on two Nvidia RTX 2080Ti GPUs.
NPENAS-SSRL 150 5.86 ± 0.14
2) NAS Results on NASBench-101
NPENAS-SSCCL 20 5.99 ± 0.21
The performance of different NAS algorithms on the NAS-
NPENAS-SSCCL 50 5.87 ± 0.15
Bench-101 benchmark is illustrated in Figure 8, and the quantita-
NPENAS-SSCCL 80 5.83 ± 0.12
tive comparison is also provided in Table III. Except for RS and
NPENAS-SSCCL 110 5.83 ± 0.12 REA, the other algorithms achieve comparable p­ erformance
NPENAS-SSCCL 150 5.83 ± 0.12 on NASBench-101 (Figure 8), while NPENAS-SSCCL and
†
The neural predictor is trained with the given number of search budgets. NPENAS-NP have the best performance overall, as shown in

46 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

 Table III. The performance of BANANAS using the position-
10.4 aware path-based encoding exceeds that of using path-based
RS encoding, proving that the proposed position-aware path-based
Testing Error of Best Neural Net

10.2 REA
BANANAS-PE
encoding is an efficient and effective encoding scheme. In addi-
10 BANANAS-PAPE tion, the performance of NPENAS-NP improves from 5.86% [6]
NPENAS-NP to 5.83% after filtering out isomorphic graphs using the proposed
9.8 NPENAS-SSRL
NPENAS-SSCCL position-aware path-based encoding. Table III also shows that
9.6 NPENAS methods using the proposed self-supervised pre-
9.4 trained neural predictors (NPENAS-SSRL and NPENAS-SSC-
CL) have only a slight performance drop when the search budget
9.2 is reduced from 150 to 90, but still outperform the unsupervised
9 arch2vec that uses a large search budget of 400.
The impact of the search budget on NPENAS-SSRL and
8.8
10 20 30 40 50 60 70 80 90 100
NPENAS-SSCCL methods are shown in Table IV. The perfor-
Number of Samples mance of NPENAS-SSRL continues to improve as the search
budget increases, while NPENAS-SSCCL achieves its best per-
FIGURE 9 Performance comparison of NAS algorithms on NASBench-201 formance using only 80 evaluated neural architectures. These
for CIFAR10 classification. results are consistent with those in Section IV-C and again show

31.5 56.9
RS RS
31
Testing Error of Best Neural Net
Testing Error of Best Neural Net

REA REA
56.4
30.5 BANANAS-PE BANANAS-PE
BANANAS-PAPE BANANAS-PAPE
30 NPENAS-NP 55.9 NPENAS-NP
29.5 NPENAS-SSRL NPENAS-SSRL
NPENAS-SSCCL 55.4 NPENAS-SSCCL
29
28.5 54.9
28
54.4
27.5
27 53.9
26.5 53.4
10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100
Number of Samples Number of Samples

FIGURE 10 Performance comparison of NAS algorithms on NASBench-201 FIGURE 11 Performance comparison of NAS algorithms on NASBench-201
for CIFAR-100 classification. for ImageNet-16-120 classification.

TABLE V Performance comparison of NAS algorithms on NASBench-201.

METHODS SEARCH BUDGET TEST ERR (%) AVG CIFAR-10 TEST ERR (%) AVG CIFAR-100 TEST ERR (%) AVG IMAGENET-16-120
RS [50] 100 9.27 ± 0.32 28.55 ± 0.86 54.65 ± 0.80
REA [25] 100 8.98 ± 0.24 27.20 ± 0.89 53.89 ± 0.71
BANANAS-PE [3] 100 9.06 ± 0.31 27.41 ± 1.03 53.85 ± 0.64
BANANAS-AE [3] 100 8.95 ± 0.14 26.77 ± 0.67 53.67 ± 0.34
BANANAS-PAPE [3] 100 8.93 ± 0.16 26.86 ± 0.67 53.71 ± 0.46
NPENAS-NP [6] 100 8.95 ± 0.13 26.70 ± 0.64 53.90 ± 0.59
NPENAS-NP-FIXED [6] 50† 8.95 ± 0.15 26.72 ± 0.57 53.84 ± 0.57
NPENAS-SSRL 100 8.94 ± 0.11 26.55 ± 0.34 53.75 ± 0.51
NPENAS-SSRL-FIXED 50† 8.92 ± 0.11 26.57 ± 0.30 53.63 ± 0.38
NPENAS-SSCCL 100 8.94 ± 0.09 26.50 ± 0.12 53.80 ± 0.36
NPENAS-SSCCL-FIXED 50† 8.93 ± 0.10 26.58 ± 0.33 53.66 ± 0.36
†
The neural predictor is trained with 50 evaluated neural architectures, while other algorithms use 100 neural architectures for evaluation.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 47

 TABLE VI Performance comparison of NAS algorithms on DART for CIFAR-10 Classification.

MODEL PARAMS (M) ERR(%) AVG ERR(%) BEST NO. OF SAMPLES EVALUATED GPU DAYS
NASNET-A [23] 3.3 – 2.65 20000 1800
NAONET [51] 10.6 – 3.18 1000 200
ASHA [50] 2.2 – 2.85 700 9
DARTS [20] 3.3 – 3.00 ± 0.14 – 1.5
BANANAS [3] 3.6 2.64 ± 0.05 2.57 100 11.8
GATES [5] 4.1 – 2.58 800 –
ARCH2VEC-RL [8] 3.3 2.65 ± 0.05 2.60 100 9.5
ARCH2VEC-BO [8] 3.6 2.56 ± 0.05 2.48 100 10.5
NPENAS-NP [6] 3.5 2.54 ± 0.10 2.44 100 1.8
NPENAS-SSCCL-FIXED 3.9 2.49 ± 0.06 2.41 70 1.6

that the neural predictor SS-CCL outperforms SS-RL on the NASBench-101, the performance of BANANAS using the posi-
large search space of NASBench-101. tion-aware path-based encoding exceeds that of using path-based
encoding. Furthermore, it can be seen from Figures 9-11 that the
3) NAS Results on NASBench-201 performance of our methods improves faster as the search budget
The above algorithms are compared on CIFAR-10, CIFAR-100, increases. Due to space limitation, more comparisons are attached
and Image­Net-16-120 on NASBench-201, and the results are in the Supplementary Materials.
shown in Figure 9, Figure  10 and Figure 11, respectively. The
quantitative comparison is presented in Table V. As arc2vec [8] does 4) NAS Results on DARTS
not report queries on this benchmark, it is not compared here. As shown in Table VI, NPENAS-SSCCL-FIXED achieves the
Table V shows that the methods proposed in this paper obtain best performance compared with the recently proposed NAS
the best performance on all three datasets. Specifically, NPENAS- algorithms, and its search speed is nearly the same as the gradient-
SSCCL achieves the best performance on both CIFAR-100 and based method DARTS [20].The searched normal cell and reduc-
ImageNet-16-120, almost reaching the ORACLE baseline on tion cell are illustrated in Figure 12.
CIFAR-100 (26.5% vs. 26.49%). In particular, the performance
of NPENAS-SSCCL is the same as the ORACLE baseline on V. Conclusion
ImageNet-16-120. On CIFAR-10, NPENAS-SSRL-FIXED This paper presents a new neural architecture encoding scheme,
achieves the best performance comparable to the ORACLE position-aware path-based encoding, to calculate the GED of
baseline (8.92% vs 8.91%). In addition, like the results on neural architectures. To enhance the performance of neural
predictors, two self-supervised learn-
ing methods are proposed to pre-train
the neural predictors’ architecture
sep_conv_3×3 embedding modules to generate
skip_connect meaningful representation of neural
c_{k-2} sep_conv_3×3
2 architectures. Extensive experiments
0
sep_conv_5×5 demonstrate the superiority of the
c_{k}
sep_conv_3×3 self-supervised pre-training. The
1 dil_conv_5×5
3 results advocate the adoption of the
sep_conv_3×3
c_{k-1} sep_conv_3×3 self-supervised central contrastive rep-
(a) resentation learning method, while
self-supervised regression learning
none can be considered when the search
0
c_{k-1} avg_pool_3×3 sep_conv_5×5 space is small. When integrating the
sep_conv_5×5
max_pool_3×3
3 pre-trained neural predictors with
c_{k}
­NPENAS, it achieves state-of-the-art
c_{k-2} 1 max_pool_3×3 performance on the NASBench-101,
none 2
max_pool_3×3 NASBench-201 and DARTS search
space. Since neural predictors can be
(b) combined with different search strate-
gies, the proposed self-supervised
FIGURE 12 (a) The normal cell and (b) reduction cell searched by NPENAS-SSCCL-FIXED on DARTS. representation learning methods are

48 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

applicable to any search strategy that employs graph neural
networks as neural predictors.
For future works, combining the pre-trained neural predictors
to other neural predictor-based NAS to verify their generalization
ability is worth further investigation. Extending the integration of
pre-trained neural predictors with NPENAS to other tasks, such
as image segmentation, object detection and natural language
processing, is also promising research.
VI. Acknowledgment
This work was supported in part by the National Natural Science
Foundation of China under Grants Nos 61976167, U19B2030
and 11727813, the Key Research and Development Program in
the Shaanxi Province of China under Grant 2021GY-082, and
the Xi’an Science and Technology Program under Grant
201809170CX11JC12. The authors would like to thank Dr.
Karen von Deneen for her professional language editing.
References
[1] T. Elsken, J. H. Metzen, and F. Hutter,“Neural architecture search: A survey,” J. Mach. Learning Res.,
vol. 20, pp. 55:1–55:21, 2018.
[2] C. Liu et al., “Progressive neural architecture search,” in Proc. Computer Vision—ECCV 2018—
15th European Conf., Munich, Part I, Sept. 8–14, 2018, vol. 11205, pp. 19–35.
[3] C. White, W. Neiswanger, and Y. Savani, “Bananas: Bayesian optimization with neu-
ral architectures for neural architecture search,” 2019. [Online]. Available: https://arxiv
.org/abs/1910.11858
[4] L.Wang,Y. Zhao,Y. Jinnai,Y.Tian, and R. Fonseca, “Neural architecture search using deep neural
networks and Monte Carlo tree search,” in Proc. 34th AAAI Conf. Artif. Intell. (AAAI 2020) 32nd In-
novative Appl.Artif. Intell. Conf. (IAAI 2020) 10th AAAI Symp. Educational Adv.Artif. Intell., EAAI 2020,
New York, Feb. 7–12, 2020, pp. 9983–9991.
[5] X. Ning, Y. Zheng, T. Zhao, Y. Wang, and H. Yang, “A generic graph-based neural architecture
encoding scheme for predictor-based NAS,” in Proc. Comput.Vision—ECCV 2020—16th European
Conf., Glasgow, U.K., Aug. 23–28, 2020, pp. 189–204.
[6] C. Wei, C. Niu,Y. Tang,Y. Wang, H. Hu, and J. Min Liang, “NPENAS: Neural predictor guided
evolution for neural architecture search,” 2020. [Online].Available: https://arxiv.org/abs/2003.12857
[7] W. Wen, H. Liu,Y. Chen, H. H. Li, G. Bender, and P. Kindermans, “Neural predictor for neural
architecture search,” in Proc. Comput. Vision—ECCV 2020—16th European Conf., Glasgow, U.K.,
Aug. 23–28, 2020, pp. 660–676.
[8] S.Yan,Y. Zheng,W. Ao, X. Zeng, and M. Zhang, “Does unsupervised architecture representation
learning help neural architecture search?” in Proc. Adv. Neural Inf. Process. Syst. 33: Annu. Conf. Neural
Inf. Process. Syst. 2020, NeurIPS 2020, Dec. 6–12, 2020.
[9] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, “A simple framework for con-
trastive learning of visual representations,” 2020. [Online]. Available: https://arxiv
.org/abs/2002.05709
[10] K. He, H. Fan,Y. Wu, S. Xie, and R. B. Girshick, “Momentum contrast for unsupervised visual
representation learning,” in Proc. IEEE/CVF Conf. Comput.Vision and Pattern Recogn., CVPR 2020,
Seattle,WA, June 13–19, 2020, pp. 9726–9735.
[11] J. Devlin, M. Chang, K. Lee, and K.Toutanova,“BERT: pre-training of deep bidirectional trans-
formers for language understanding,” in Proc. Conf. North American Chapter Assoc. Comput. Linguistics:
Human Language Technol. (NAACL-HLT 2019), Minneapolis, MN, June 2–7, 2019, pp. 4171–4186.
[12] Y. M. Asano, C. Rupprecht, and A.Vedaldi, “Self-labelling via simultaneous clustering and repre-
sentation learning,” in Proc. 8th Int. Conf. Learning Representations, ICLR 2020, Addis Ababa, Ethiopia,
Apr. 26–30, 2020.
[13] Z.Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S.Yu,“A comprehensive survey on graph neural
networks,” IEEE Trans. Neural Netw. Learning Syst., 2020.
[14] H. Cheng et al., “NASGEM: Neural architecture search via graph embedding method,” 2020.
[Online]. Available: https://arxiv.org/abs/2007.04452
[15] M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw
puzzles,” in Proc. Comput.Vision—ECCV 2016—14th European Conf., Amsterdam,The Netherlands,
Oct. 11–14, 2016, pp. 69–84.
[16] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting
image rotations,” in Proc. 6th Int. Conf. Learning Representations, ICLR 2018,Vancouver, BC, Canada,
April 30–May 3, 2018.
[17] C.Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter,“NAS-bench-101:Towards
reproducible neural architecture search,” in Proc. 36th Int. Conf. Machine Learning, ICML 2019, June
9–15, 2019, vol. 97, pp. 7105–7114.
[18] J.You, J. Leskovec, K. He, and S. Xie,“Graph structure of neural networks,” in Proc. 37th Int. Conf.
Machine Learning, ICML 2020, July 12–18, 2020.
[19] X. Dong and Y.Yang, “NAS-Bench-201: Extending the scope of reproducible neural architec-
ture search,” in Proc. 8th Int. Conf. Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, Apr.
26–30, 2020.
[20] H. Liu, K. Simonyan, and Y.Yang, “DARTS: Differentiable architecture search,” in Proc. 7th Int.
Conf. Learning Representations, ICLR 2019, New Orleans, LA, May 6–9, 2019.
[21] B. Zoph and Q.V. Le, “Neural architecture search with reinforcement learning,” in Proc. 5th Int.
Conf. Learning Representations, ICLR 2017,Toulon, France, Apr. 24–26, 2017.
[22] H. Pham, M.Y. Guan, B. Zoph, Q.V. Le, and J. Dean, “Efficient neural architecture search via
parameter sharing,” in Proc. 35th Int. Conf. Machine Learning, ICML 2018, Stockholmsmässan, Stock-
holm, Sweden, July 10–15, 2018, vol. 80, pp. 4092–4101.
[23] B. Zoph,V.Vasudevan, J. Shlens, and Q.V. Le,“Learning transferable architectures for scalable im-
age recognition,” in Proc. IEEE Conf. Comput.Vision and Pattern Recogn., CVPR 2018, Salt Lake City,
UT, June 18–22, 2018, pp. 8697–8710.
[24] E. Real et al.,“Large-scale evolution of image classifiers,” in Proc. 34th Int. Conf. Machine Learning,
ICML 2017, Sydney, NSW, Australia, Aug. 6–11, 2017, vol. 70, pp. 2902–2911.
[25] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for im-
age classifier architecture search,” in Proc. 33rd AAAI Conf. Artif. Intell., AAAI 2019, 31st
Innovative Appl.Artif. Intell. Conf., IAAI 2019, 9th AAAI Symp. Educational Adv.Artif. Intell., EAAI 2019,
Honolulu, HI, Jan. 27–Feb. 1, 2019, pp. 4780–4789.
[26]Y. Sun, B. Xue, M. Zhang, G. G.Yen, and J. Lv,“Automatically designing CNN architectures using
the genetic algorithm for image classification,” IEEE Trans. Cybernetics, 2020, pp. 1–15. doi: 10.1109/
TCYB.2020.2983860.
[27] Y. Sun, B. Xue, M. Zhang, and G. G. Yen, “Completely automated CNN archi-
tecture design based on blocks,” IEEE Trans. Neural Netw. Learning Syst., vol. 31, no. 4,
pp. 1242–1254, 2020. doi: 10.1109/TNNLS.2019.2919608.
[28] Y. Sun, B. Xue, M. Zhang, and G. G.Yen,“Evolving deep convolutional neural networks for im-
age classification,” IEEE Trans. Evolutionary Comput., vol. 24, no. 2, pp. 394–407, 2020.
[29] H. Zhou, M.Yang, J. Wang, and W. Pan, “Bayesnas: A Bayesian approach for neural architecture
search,” in Proc. 36th Int. Conf. Machine Learning, ICML 2019, June 9–15, 2019, Long Beach, CA,
2019, vol. 97, pp. 7603–7613.
[30] S. Xie, H. Zheng, C. Liu, and L. Lin,“SNAS: Stochastic neural architecture search,” in Proc. 7th Int.
Conf. Learning Representations, ICLR 2019, New Orleans, LA, May 6–9, 2019.
[31] X. Chen, L. Xie, J.Wu, and Q.Tian, “Progressive differentiable architecture search: Bridging the
depth gap between search and evaluation,” pp. 1294–1303, 2019.
[32]Y. Xu et al.,“PC-DARTS: Partial channel connections for memory-efficient architecture search,”
in 8th Int. Conf. Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, Apr. 26–30, 2020.
[33] K. Kandasamy,W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing,“Neural architecture search
with Bayesian optimisation and optimal transport,” in Proc. Adv. Neural Information Process. Syst. 31:
Annu. Conf. Neural Information Process. Syst. 2018, NeurIPS 2018, Montréal, Canada, Dec. 3–8, 2018,
pp. 2016–2025.
[34] R. Luo, X. Tan, R. Wang, T. Qin, E. Chen, and T. Liu, “Semi-supervised neural architecture
search,” in Proc. Adv. Neural Information Processing Syst. 33: Annu. Conf. Neural Information Process. Syst.
2020, NeurIPS 2020, Dec. 6–12, 2020.
[35] L. Dudziak, T. C. P. Chau, M. S. Abdelfattah, R. Lee, H. Kim, and N. D. Lane, “BRP-NAS:
Prediction-based NAS using GCNS,” in Proc. Adv. Neural Inf. Process. Syst. 33: Annu. Conf. Neural Inf.
Process. Syst. 2020, NeurIPS 2020, Dec. 6–12, 2020.
[36] B. Baker, O. Gupta, R. Raskar, and N. Naik, “Accelerating neural architecture search using
performance prediction,” in Proc. 6th Int. Conf. Learning Representations, ICLR 2018,Vancouver, BC,
Canada, Apr. 30–May 3, 2018.
[37] D. P. Kingma and M.Welling, “Auto-encoding variational bayes,” in Proc. 2nd Int. Conf. Learning
Representations, ICLR 2014, Banff, AB, Canada, Apr. 14–16, 2014.
[38]Y. Li, C. Gu,T. Dullien, O.Vinyals, and P. Kohli,“Graph matching networks for learning the simi-
larity of graph structured objects,” in Proc. 36th Int. Conf. Machine Learning, ICML 2019, Long Beach,
CA, June 9–15, 2019, pp. 3835–3845.
[39] Y. Bai, H. Ding, S. Bian,T. Chen,Y. Sun, and W.Wang,“SIMGNN:A neural network approach to
fast graph similarity computation,” in Proc. 12th ACM Int. Conf.Web Search and Data Mining,WSDM
2019, Melbourne,VIC, Australia, Feb. 11–15, 2019, pp. 384–392.
[40] Y. Bai et al.,“Unsupervised inductive graph-level representation learning via graph-graph prox-
imity,” in Proc. 28th Int. Joint Conf. Artif. Intell., IJCAI 2019, Macao, China, Aug. 10–16, 2019, pp.
1988–1994.
[41] K. Xu,W. Hu, J. Leskovec, and S. Jegelka,“How powerful are graph neural networks?” in Proc. 7th
Int. Conf. Learning Representations, ICLR 2019, New Orleans, LA, May 6–9, 2019.
[42] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning
of visual features by contrasting cluster assignments,” 2020. [Online]. Available: https://arxiv.org/
abs/2006.09882
[43] A. Paszke et al., “Pytorch: An imperative style, high-performance deep learning library,” in Proc.
Adv. Neural Information Processing Syst. 32: Annu. Conf. Neural Information Process. Syst. 2019, NeurIPS
2019, Dec. 8–14, 2019,Vancouver, BC, Canada, 2019, pp. 8024–8035.
[44] M. Fey and J. E. Lenssen,“Fast graph representation learning with PyTorch Geometric,” in Proc.
Int. Conf. Learning Representations Workshop Representation Learning Graphs and Manifolds, 2019.
[45] C. Wei, “Code for self-supervised representation learning for evolutionary neural architecture
search,” 2020. [Online]. Available: https://github.com/auroua/SSNENAS
[46] A. Krizhevsky and G. E. Hinton, “Learning multiple layers of features from tiny images,” Tech.
Rep., 2009.
[47] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional
neural networks,” in Proc. Adv. Neural Information Process. Syst. 25: 26th Annu. Conf. Neural Information
Process. Syst. 2012. Proc. Meeting, Lake Tahoe, NV, Dec. 3–6, 2012, pp. 1106–1114.
[48] D. P. Kingma and J. Ba,“Adam:A method for stochastic optimization,” CoRR, vol. abs/1412.6980,
2014.
[49] I. Loshchilov and F. Hutter,“SGDR: Stochastic gradient descent with warm restarts,” in Proc. 5th
Int. Conf. Learning Representations, ICLR 2017,Toulon, France, Apr. 24–26, 2017, 2017.
[50] L. Li and A. Talwalkar, “Random search and reproducibility for neural architecture search,” in
Proc. 35th Conf. Uncertainty Artif. Intell., UAI 2019,Tel Aviv, Israel, July 22–25, 2019, p. 129.
[51] R. Luo, F.Tian,T. Qin, and T.-Y. Liu, “Neural architecture optimization,” in Proc. Adv. Neural Inf.
Process. Syst. 31: Annu. Conf. Neural Inf. Process. Syst. 2018, NeurIPS 2018, Dec. 3–8, 2018, Montréal,
Canada, 2018, pp. 7827–7838.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 49

Forecasting Wind Speed
Time Series Via Dendritic
Neural Regression
Junkai Ji, Minhui Dong, and Qiuzhen Lin
Shenzhen University, CHINA
Kay Chen Tan
The Hong Kong Polytechnic University, HONG KONG SAR.
Abstract—Wind energy is considered one of the fastest
growing renewable (‘green’) energy resources. Precise wind
power forecasting is imperative to ensure reliable power sys-
tem planning and wind farm operation. However, traditional
methods cannot yield satisfactory forecasts because of the
chaotic properties and high volatility of wind speed time
Digital Object Identifier 10.1109/MCI.2021.3084416
Date of current version: 15 July 2021
50 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
©SHUTTERSTOCK.COM/JIMMOYHT
series. To address this issue, the use of artificial neural net-
works has attracted increasing attention owing to their sig-
nificantly enhanced prediction accuracy. Based on these
considerations, a novel neural model with a dynamic den-
drite structure, known as the dendritic neuron model
(DNM), can be adopted for wind speed time series predic-
tion. The DNM is a plausible biological neural model that
was originally designed for classification problems; accord-
ingly, this study proposes the use of a regressive version of
the DNM, named dendritic neural regression (DNR), in
which the dendrite strength of each branch is considered. To
enhance the prediction performance, the recently proposed
states of matter search (SMS) optimization algorithm is used
to optimize the neural architecture for DNR. By virtue of
the powerful search ability of the SMS algorithm, DNR can
Corresponding Author: Qiuzhen Lin (e-mail: qiuzhlin@szu.edu.cn).
1556-603X/21©2021IEEE
efficiently capture the nonlinear correlations among distinct
features and dendritic branches. Extensive experimental
results and statistical tests demonstrate that compared with
other state-of-the-art prediction techniques, DNR can
achieve highly competitive results in wind speed forecasting.
W
I. Introduction
ith the rapid development of society, it has
become challenging for the remaining fossil
fuel supply to meet societal requirements world-
wide; thus, the demand for sustainable renewable
energy is growing. In recent years, wind energy, as a sustain-
able renewable energy source, has attracted particular atten-
tion. Wind is an abundant, pollution-free energy source that
is widely distributed, has few geographical restrictions, and
can be easily used anywhere. Building wind farms is one of
the main challenges in the use of wind energy. Nevertheless,
the cost of building wind farms is low, no additional energy
source is required, and such farms do not impact the sur-
rounding environment.
Since the energy supplied by wind farms mainly depends
on the wind power, precise wind power forecasting is imper-
ative to enable reliable power system planning and wind
farm operation [1, 2]. Wind power is closely related to wind
speed, and these two parameters share several characteristics,
such as randomness, uncontrollability and intermittency [3].
Because of these characteristics, wind farm management is
extremely challenging. Calculating the energy production
of a wind farm is essential for assessing the economic feasi-
bility of such a project prior to construction planning [4].
Therefore, accurate wind speed prediction is being strongly
prioritized in this context [5]. More accurate forecasting
capabilities correspond to larger reductions in the construc-
tion costs of wind farms [6].
To date, various models have been used for wind speed
prediction. These methods can be classified into three cate-
gories: physical models, statistical models, and artificial
intelligence models [7]. Examples of physical models
include the Mesoscale Model Version 5 [8] and the Weather
Research and Forecasting Model [9]. These numerical pre-
diction models can achieve satisfactory performance in
long-term wind speed prediction; however, such models
require complex atmospheric information pertaining to
pressure, temperature and other environmental factors and
exhibit high computational complexity [10], [11].
Compared to physical models, statistical models are more
widely used to forecast wind speed. Examples of such models
include direct random time series models [12], autoregressive
models [13] and autoregressive integrated moving average
(ARIMA) models [14]. ARIMA models are regarded as a
typical class of statistical models, and their prediction perfor-
mance in short-term wind speed forecasting has been veri-
fied [15]. However, in general, the prediction performance of
statistical models is f lawed [16] because most statistical mod-
els are based on the assumption that the wind speed series
follows a normal distribution, although this assumption is not
valid in all cases. Moreover, statistical models have linear
correlation structures and always yield large errors when
applied to intermittent and stochastic wind speed series [17].
The third category pertains to artificial intelligence mod-
els, which can effectively overcome the shortcomings of the
abovementioned methods. Considering the nonlinear nature
of wind speed series, artificial intelligence algorithms that are
designed for effectively solving nonlinear problems, such as
artificial neural networks (ANNs) [16] and support vector
machines (SVMs) [18], are suitable for wind speed forecast-
ing. A previous comparison of prediction performance has
demonstrated that artificial intelligence algorithms are faster
and more accurate than statistical models [19]. SVMs are
commonly used in prediction frameworks, and they can out-
perform ANNs in certain cases. However, the performance
of an SVM is limited by its penalty settings and kernel
parameters; consequently, algorithms for tuning these hyper-
parameters are necessary [20]. For example, a genetic algo-
rithm was employed to enhance the prediction results of an
SVM in [21]; a reduced SVM with feature selection, trained
using the particle swarm optimization algorithm, was used to
optimize the parameters of an SVM in [22]; and the perfor-
mance of an SVM was enhanced using the cuckoo search
algorithm in [23]. In addition to SVMs, an increasing num-
ber of ANNs and their variants have been proposed for wind
speed prediction. For instance, a backpropagation (BP) neu-
ral network was employed to forecast a wind speed series in
[24], a combination of an ANN and Markov chains was pro-
posed for forecasting in [25], and a functional network was
utilized for multistep wind speed prediction in [26]. Further-
more, a fine-tuned long short-term memory (LSTM) neural
network hybridized with the crow search algorithm, the
wavelet transform and feature selection was applied for short-
term wind speed forecasting in [27]. In [28], a hybrid model
involving a causal convolutional network and a gated recur-
rent unit architecture was used in wind speed prediction.
In general, physical models and traditional statistical mod-
els both have several limitations pertaining to the precision
and robustness of wind speed time series prediction, whereas
artificial intelligence models can effectively overcome these
problems to offer more powerful prediction performance.
Based on these considerations, the dendritic neuron model
(DNM), which was recently developed based on inspiration
from biological neurons in vivo [29], is adopted in this study
for the prediction of wind speed time series. In the DNM,
synaptic nonlinearity is implemented in a dendritic structure
to effectively solve linearly inseparable problems, and this
model has been applied to a variety of complex continuous
functions [30]–[32]. The original DNM was specifically
designed for classification problems. By discarding the unnec-
essary synapses and dendritic branches in the DNM, diverse
dendritic structures can be produced to pursue an extremely
high classification speed for each task. Notably, however, the
structure of the original DNM is extremely simple, and it
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 51

The demand for sustainable renewable energy is
growing. Wind energy, as a sustainable renewable
energy source, has attracted particular attention.
ignores the mechanism by which the signal transformation
strengths of the dendritic branches vary with their thickness
[33]. Therefore, this study proposes the use of a variant of the
DNM called dendritic neural regression (DNR), in which the
intensity of each dendritic signal is considered to enhance the
regression ability.
The neural architecture considerably inf luences the model
performance of ANNs [34]. Accordingly, evolutionary algo-
rithms have been widely used to realize neural architecture
search tasks for performance optimization. For instance, Sun
et al. proposed an automatic method of designing convolu-
tional neural network (CNN) architectures for solving
image classification problems based on genetic algorithms
[35]. Lu et al. used a multiobjective evolutionary algorithm
for CNN architecture design [36]. In contrast to that of
conventional ANNs, the neural architecture of a DNR
model is determined by the values of the weights and
thresholds rather than by hyperparameters. Subsequently, an
inherent pruning mechanism can be implemented to simpli-
fy the DNR model to produce unique neural architectures
for particular real-world tasks, as described in our previous
research [29], [37]. To further enhance the performance of
wind speed forecasting, the recently proposed states of mat-
ter search (SMS) algorithm [38] can be used to optimize the
neural architecture of the DNR model. The SMS algorithm
is a global search algorithm that can escape from local mini-
ma more effectively than gradient-based optimization algo-
rithms. In fact, many researchers have attempted to employ
evolutionary algorithms to enhance the performance of
ANNs for chaotic time series prediction. For example, par-
ticle swarm optimization has been introduced into echo
state networks as a pretraining tool to optimize the
untrained weights to address time series forecasting prob-
lems [39]. Furthermore, a genetic algorithm has been intro-
duced into the Elman neural network to optimize the
connection weights and thresholds to prevent the optimiza-
tion from becoming trapped in local minima and enhance
the training speed and success rate [40]. Moreover, a modi-
fied cuckoo search algorithm has been used to optimize
wavelet neural networks to achieve higher generalization
capabilities in chaotic time series forecasting [41]. In addi-
tion, an evolving fuzzy neural network predictor has been
proposed to effectively capture the dynamic properties of
multidimensional datasets and accurately track the system
characteristics [42], while in [43], a novel strategy was pro-
posed for evolving the structure of deep recurrent neural
networks by means of ant colony optimization.
Compared with other ANNs, a DNR model can help
enhance the performance of wind speed prediction because
52 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
its plastic neural architecture can efficiently
capture the nonlinear correlations among
distinct features and different dendritic
branches. In addition, a one-dimensional
wind speed time series typically appears to
exhibit intermittent and random features
since it contains complex information projected from a high-
er-dimensional space. Thus, the phase space should be recon-
structed based on Takens’ theorem [44]. The time delays and
numbers of embedding dimensions for a wind speed time
series can be calculated using a mutual information (MI)
approach and the false nearest neighbors (FNN) algorithm,
respectively. To evaluate the forecasting results of the DNR-
SMS method, a comprehensive experiment has been con-
ducted in which the performance of the DNR approach has
been compared with that of other commonly used time series
prediction algorithms.
The main contributions of this study are as follows:
First, a novel DNR model that considers the dendrite
strength of each individual branch is proposed. This
approach can significantly enhance the regression perfor-
mance for wind speed forecasting. Second, due to its pow-
erful search ability, the SMS algorithm is used to optimize
the neural architecture of the DNR model. The SMS algo-
rithm can escape from local minima more effectively than
gradient-based optimization algorithms can. Third, the
results of extensive experiments demonstrate that a DNR
model trained using the SMS algorithm can achieve highly
competitive performance in wind speed forecasting com-
pared with other state-of-the-art prediction techniques.
The remainder of the paper is organized as follows: Section
II describes the DNR approach. Section III introduces the
process of wind speed prediction. Section IV presents and
discusses the experimental results. Finally, concluding
remarks are presented in Section V.
II. Dendritic Neural Regression
In this study, we demonstrate the utilization of DNR for
wind speed prediction and evaluate its performance at vari-
ous time scales. As illustrated on the left side of Fig. 1, the
neural architecture for DNR includes four layers. The first
layer is the synaptic layer, which represents the specific tissue
that receives electrical or chemical signals from other neu-
rons. The second layer is the dendrite layer, which consists of
many branches that integrate the output signals from the
synaptic layer. The third layer is the membrane layer, which
sums the outputs of the dendritic layer and transfers the result
to the next layer. The final layer is the cell body (soma),
which compares the signal against a given threshold. If the
signal is larger than the threshold, the soma fires; otherwise,
no action is performed.
A. Synaptic Layer
Synapses are a kind of neural tissue that conveys information
between dendrites and axons or among dendrites of different
 Condition Cases
Branch-m
Constant 1 Condition
Synapse Synapse Synapse Dendrite
Learning Algorithm Constant 0 Condition

Direct Condition

Inverse Condition
Branch-3

Synapse Synapse Synapse Dendrite

Direct Connection Inverse Connection
1 1
0.8 0.8
0.6 0.6

y
y

Branch-2 0.4 0.4

0.2 0.2
0 0
Synapse Synapse Synapse Dendrite –2 –1.5 –1 –0.5 0 0.5 1 1.5 2 –2 –1.5 –1 –0.5 0 0.5 1 1.5 2
x x
(a) (b)
Constant 1 Connection
1 1
Branch-1 0.8 0.8
0.6 0.6

y
y

0.4 0.4
Synapse Synapse Synapse Dendrite 0.2 0.2
0 0
–2 –1.5 –1 –0.5 0 0.5 1 1.5 2 –2 –1.5 –1 –0.5 0 0.5 1 1.5 2
(c)
Constant 0 Connection
1 1
0.8 0.8
X1 0.6 0.6
y

X2 Xn
y

0.4 0.4
Soma 0.2 0.2
0 0
–2 –1.5 –1 –0.5 0 0.5 1 1.5 2 –2 –1.5 –1 –0.5 0 0.5 1 1.5 2
Neuron Architecture x x
(d)

Condition Functions

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

53
FIGURE 1 Illustration of the DNR architecture.

The neural architecture of DNR consists of four layers,
namely the synaptic layer, dendritic layer, membrane
layer, and soma layer.
neurons. These elements are distributed throughout the den-
dritic tree and possess various receptors for specific ions.
Depending on the potential of the ions entering a receptor,
the synapse changes its connection state and enters either an
excitatory or inhibitory state [45]. The process of signal
transmission can be described using the following equation:
1 1 + e -k (V - qs)
y im = , (1)
1 + e -k (w im xi - qim)
where x i represents the i-th input feature, whose range is
[0,1], with i ! 61, 2, f, I @; y im is the output of the i-th syn-
apse on the m-th dendritic branch, with m ! 61, 2, f, M @; k
is a hyperparameter that is a positive constant; and w im and
q im are connection parameters that represent a weight and a
threshold, respectively. To obtain the appropriate values for
each problem, these connection parameters in a DNR model
can be trained using a learning algorithm.
B. Dendrite Layer
Each branch in the dendrite layer receives the output signals
from all synapses on that branch. The nonlinear relationship
among these signals plays a key role in neural information
processing for several sensory systems in biological networks,
such as the visual and auditory systems [46], [47]; in DNR,
this relationship can be expressed in terms of multiplication
operations. Let Z m represent the output of the m-th dendritic
branch. The equation for a dendritic branch can be expressed
as follows:
I drite layer, the parameter space of the model appears to be
Z m = % y im .(2)
i=1
C. Membrane Layer
The membrane layer combines all outputs from the dendrite
layer through a summation operation. Let V represent the out-
put of the membrane layer. The corresponding equation can
be expressed as follows:
M
V= / ^u m ) Z mh,(3)
m=1
where u m represents the strength of the m-th dendritic
branch. This value is constant and is always set to 1 for each
branch in the original DNM to simplify the neural architec-
ture to accelerate the computation process [29]. However, in
reality, the thicknesses and signal transformation strengths of
the dendritic branches vary; thus, using a uniform u m value
for all branches may degrade the regression ability of the
54 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
DNM. Therefore, in DNR, each u m is
regarded as a parameter that needs to be
optimized via a learning algorithm; accord-
ingly, these values are specified differently
for different problems.
D. Cell Body (Soma)
The soma fires depending on whether the membrane poten-
tial exceeds a given threshold. This process can be mathemat-
ically described as a sigmoid operation on the product terms,
as follows:
O= 1 , (4)
where V represents the output of the membrane layer and k
and qs are positive constant hyperparameters.
E. Connection Cases
On the right side of Fig. 1, the six functions of the synaptic
layer are illustrated for various combinations of w im and q im.
According to these six different functions, the connection
states of the synaptic layer can be divided into four main cate-
gories, defined as follows: constant 1 connections, whose
parameters satisfy w im 1 0 1 q im or 0 1 w im 1 q im , implying
that regardless of the input, the output is always excitatory;
constant 0 connections, whose parameters satisfy q im 1 w im 1 0
or q im 1 0 1 w im , implying that regardless of the input, the
output remains inhibitory; excitatory connections, whose
parameters satisfy 0 1 q im 1 w im , implying that the input and
output are directly correlated; and inhibitory connections,
whose parameters s­atisfy w im 1 q im 1 0, implying that the
input and output are inversely correlated.
F. Learning Algorithm
Because of the multiplication operations applied in the den-
extremely large and complicated. Additionally, weights are
added to the output of each dendritic branch in the DNR
model, leading to an increase in the dimensionality of the
parameter space, which further increases the difficulty of
optimizing the parameters. Consequently, it is difficult to
perfectly train the parameters when using the traditional BP
algorithm. Therefore, in this study, the SMS algorithm is
adopted as a more suitable global optimization algorithm to
optimize the DNR model. The SMS algorithm is an evolu-
tionary algorithm that mimics the variation in the states of
matter. Compared with the traditional BP algorithm, the
SMS algorithm exhibits a higher search ability, is less likely to
fall into local optima, and seldom results in overfitting. In this
subsection, this algorithm is described in detail.
The process of searching for the best solution in the SMS
algorithm can be expressed as a series of physical motions
among molecules, which mimic the state transformations of
matter [38]. Specifically, the SMS algorithm can be divided
into three phases: the gas phase, the liquid
phase, and the solid phase. The gas phase The process of searching for the best solution in the
corresponds to the previous solution set. In SMS algorithm can be expressed as a series of physical
the gas state, the molecules are far from one
another and widely distributed, and intense
motions among molecules, which mimic the state
motions and collisions occur among them. transformations of matter.
The liquid phase represents the intermediate
solution set. In the liquid state, the distances between mole- denote the corresponding distance threshold, which is calcu-
cules are considerably smaller than those in the gas state, the lated as follows:
distribution is relatively concentrated, and the movements are
/ ^b mhigh - b mlowh
n
relatively limited. Finally, the solid phase represents the later
solution set. In the solid state, the distances between mole- r = m=1 ) b, (8)
n
cules are minimal, the distribution is highly concentrated,
and only a small amount of motion occurs. where b is a constant positive value with a range of [0,1].
In each phase, three operations are executed between the If the distance between two individuals is less than r, they
molecules, defined as follows: will undergo collision, and this phenomenon can be
(1) Direction Vector Operator: The purpose of the direc- expressed as follows:
tion vector operator is to push other individuals to move
toward the best individual in the population. Let P i repre- d m = d i AND d i = d m .(9)
sent a member of the population, and let p best represent the
current best individual. P best attracts other members of the (3) Random Behavior: Following the rules of the SMS
population, causing them to move toward P best. Let d i be the algorithm, during the iterative process, each individual will
direction vector of the i-th member. The corresponding likely exhibit random behavior, i.e., the elements of the indi-
equation is as follows: viduals will randomly change. The function for this random
behavior can be expressed as follows:
d ti + 1 = d ti ) c 1 - t m ) 0.5 + P best - Pi , (5)
]] b m + Rand 2 ) ^b m - b m h,
Z low high low
epoch P best - Pi
t+1
P i, m = [ with probability H, (10)
\ P i, m , with probability ^1 - H h,
where t denotes the current iteration number and epoch ] t+1
denotes the total number of iterations. The obtained value d i
is used to calculate the velocity vector of the i-th member of where H represents the occurrence probability of the random
the population, as follows: behavior.
The entire iterative process in the SMS algorithm is
/ ^b high
m - bm h
n
low divided into three phases by establishing different c, a, b
and H values. In accordance with [38], we adopt the parame-
vi = di ) m = 1 ) c, (6)
n ters presented in Table II for the three phases. The complete
SMS process is described in Algorithm 1.
where c is a constant positive value with a range of [0,1] and
n represents the dimensionality of each member, namely, the III. Wind Speed Forecasting Framework
number of parameters in the model. The upper and lower The process of applying DNR for wind speed prediction is
bounds on the m-th parameter are b high m and b low
m , respectively. illustrated in Fig. 2. First, the time delay and embedding
Next, we can employ v i to update the position p i, where the dimensionality of the series are calculated to reconstruct the
update function is defined as follows: phase space. Next, the maximum Lyapunov exponent is cal-
culated from the time delay and embedding dimensionality.
P it,+m1 = p ti, m + v m ) Rand 1 ) ^b mhigh - b mlowh ) a, (7) When this exponent exceeds zero, the wind speed series
data are regarded as a chaotic time series. Subsequently, the
where Rand 1 is a random number between 0 and 1 and a is a
constant positive value with a range of [0,1].
(2) Collision Operator: When two molecules approach
TABLE I Parameter settings of the SMS algorithm
each other, they may collide. The collision operator emu- in different states.
lates this process; if two individuals have collided, their
STATE DURATION c a b PROBABILITY H
direction vectors are exchanged, which ensures the diversi-
ty of the population and prevents premature convergence in GAS 50% 0.8 0.8 [0.8,1] 0.9

the evolutionary process. Collision behavior occurs when LIQUID 40% 0.4 0.2 [0.0,0.6] 0.2
two individuals are sufficiently close to each other. Let r SOLID 10% 0.1 0 [0.0,0.1] 0.0

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 55

wind speed time series data are processed through a normal-
ization operation. Finally, the DNR model can be employed
for wind speed prediction.
A. Phase Space Reconstruction
It is difficult to forecast the future trend of a chaotic time
series such as a wind speed series because of its irregulari-
ty. A chaotic time series can be considered to represent a
type of random motion in a definite dynamical system.
Thus, we need to restore the original dynamical system
of the chaotic time series. The most common method
to accomplish this is the delayed coordinate approach
proposed by Takens [44]. For a chaotic time ser ies
" x (i ) ; i = 1, 2, f, N ,, under the assumption that the time
delay x and vector embedding dimensionality m have
Algorithm 1 Pseudocode for the SMS algorithm.
Input: P  opulation size N, Number of dimensions n, Maximum
number of iterations epoch.
Result: Best solution P best .
begin
 Initialize the population X = " P1, f, PN ,, the direction vector set
 D 0 = " d 01, f, d 0N ,, the maximum number of iterations epoch =
1000, the state count phase = 1, and the current iteration
number t = 1;
 repeat
  if phase == 1 then
    c = 0.8, a = 0.8, b = 0.9, H = 0.9, Dend = epoch ) 0.5;
  if phase == 2 then
    c = 0.4, a = 0.2, b = 0.5, H = 0.2, Dend = epoch ) 0.9;
  if phase == 3 then
    c = 0.1, a = 0.0, b = 0.0, H = 0.0, Dend = epoch;
  for t, t 1 = Dend, t = t + 1 do
    Evaluate the fitness of the population
    F (X ) = " f (P1), f, f (PN) ,;
    Set the individual with the best fitness as P best;
    /* Perform direction vector operations */
    Calculate the new direction vector set D t with Eq. (5);
   Use D t to calculate the velocity vector set V = " v 1, f, v N ,
  with Eq. (6);
   Update each individual P n with Eq. (7);
    /* Perform collision operations */
   Calculate the threshold r using Eq. (8);
   Calculate the distance between each pair of points; if the
 distance is less than r, exchange the direction vectors of
the points, as indicated in Eq. (9);
    /* Perform random behavior */
   Draw a random number RA in the range of [0,1]; if RA is
 less than H, execute random behavior with Eq. (10);
   phase = phase + 1;
 until phase 2 3;
return the best solution P best .
56 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
been calculated using a certain approach, a group of new
vectors can be constructed as follows:
J x1 x2 f x N - 1 - x (m - 1)N
K O
x x f x N - 1 - x (m - 2)O
X =K
x+1 x+2
,(11)
K h h h h O
K O
Lx x (m - 1) + 1 x x (m - 1) + 2 f xN - 1 P
T = ^x x (m - 1) + 2, x x (m - 1) + 3, f, x N h, (12)
where X and T are the input signals of the neural model and
the target outputs, respectively. The matrix X can efficiently
describe the original dynamical system if the appropriate val-
ues of the time delay x and embedding dimensionality m are
selected. The operation of constructing X from a time series
" x (i ) ; i = 1, 2, ..., N , is termed phase space reconstruction.
The core problem in performing this operation is to obtain
the appropriate values for x and m. Takens only proved the exis-
tence of the time delay and embedding dimensionality in theory
and did not provide a method for determining their values. In
general, a time series always contains finite sequences with
noise. There is no single method that can accurately obtain the
time delay and embedding dimensionality for every time series;
instead, the appropriate algorithm must be chosen depending
on the actual situation. In this study, the FNN algorithm is
utilized to calculate the embedding dimensionality [48], and
the MI algorithm is employed to obtain the time delay [49].
The details of these two methods are presented in the two
subsequent subsections.
B. Mutual Information Algorithm
The MI algorithm is one of the most widely used methods for cal-
culating the time delay of a time series and is based on the theory
of mutual information. The information entropy, H(X ), which is
used to represent the degree of uncertainty of X, can be expressed
as follows:
n
H (X ) = - / P (x i) log P (x i), (13)
i=1
where X denotes discrete random variables, P (x i) is the prob-
ability of the occurrence of event x, and n is the total number
of states x. H (X ; Y ) represents the conditional information
entropy and can be expressed as
n m
H (X ; Y ) = - / / P (y j ) P (x i ; y j ) log P (x i ; y j ),(14)
i=1 j=1
where m represents the total number of states y, P (y j ) is the
probability of event y occurring alone, and P (x i ; y j) denotes
the conditional probability of event x occurring under the
condition of the occurrence of event y. The MI entropy,
I(X, Y ), can be calculated as
I (X, Y ) = H (X ) + H (Y ) - H (X, Y ), (15)
where H(X, Y ) represents the joint information of X and Y
and can be calculated using the following expression:
n m
H (X, Y ) = - / / P (x i, y j) log P (x i, y j).(16)
i=1 j=1
For the time series " x (i ) ; i = 1, 2, f, N ,, P (x l) and
P (x l + x) are the probabilities that x l and x l + x, respectively,
appear in " x (i ) ; i = 1, 2, f, N , . Based on these definitions,
the MI entropy I (x l, x l + x) for a time delay x can be specified
as follows:
I (x) = I (x l, x l + x) = H (x l) + H (x l + x) - H (x l, x l + x).(17)
According to the MI algorithm, the value of x when
I (x) reaches a local minimum for the first time is taken as
the final solution.
C. False Nearest Neighbors Algorithm
The FNN algorithm, which was proposed by Kennel in
1992 [48], is used to calculate the embedding dimensionali-
ty for a chaotic time series. This algorithm is based on the
premise that a chaotic time series, such as a series of wind
speed data, can be regarded as a set of continuously varying
particles in a high-dimensional space mapped to a one-
dimensional space. If the number of embedding dimensions
is too small, the particles will be compressed and folded
onto one another due to the insufficient extent of their spa-
tial orbits, meaning that two adjacent points in the one-
dimensional space may cor respond to t wo particles
separated by a large distance in the high-dimensional space.
Two such adjacent particles are def ined as false nearest
neighbor points. In this case, the embedding dimensionali-
ty should be gradually increased to fully expand the spatial
orbits. With an increase in the number of embedding
dimensions, the particles are expected to gradually separate,
and the number of false nearest neighbor points is expected
to gradually decrease. Once the embedding dimensionality
is set to a sufficient value such that all false nearest neighbor
points are eliminated, the corresponding solution is consid-
ered to be the optimal solution.
Suppose that z i (m) = (y (i ), y (x + i ), f, y ((m - 1) x + i ))
is a vector in an m-dimensional phase space and that
z j(m) is the nearest neighbor point of z i; the distance
between z i(m) and z j (m) can be calculated using the fol-
lowing equation:
R i (m) = z i (m) - z j (m) .(18)
This distance changes as the number of dimensions
increases. Thus, the updated distance between z i (m + 1) and
z j (m + 1) can be expressed as follows:
R i2 (m + 1) = R i2 (m) + z i (i + mx) - z j ( j + mx ) .(19)
If R i (m + 1) is considerably larger than R i (m), the two
points are considered to be false nearest neighbor points.
By modifying this equation, a more convenient function
for judging false nearest neighbor points can be obtained
as follows:
z i ( i + m x ) - z j ( j + mx )
Rx = .(20)
R i ( m)
R x is compared against a given threshold i, whose range is
set to [10, 50]. If R x is larger than this threshold, the points are
determined to be false nearest neighbor points. For a real-
world chaotic time series, such as a wind speed series, the ini-
tial number of embedding dimensions is usually set to 2. Once
the proportion of false nearest neighbor points is less than 5%,
the corresponding number of embedding dimensions is select-
ed as the final solution. In certain extreme cases, the propor-
tion of false nearest neighbor points cannot decrease to 5%; in
such a case, the critical dimensionality at which the number of
false nearest neighbor points stops decreasing is considered the
final solution.
D. Maximum Lyapunov Exponent
Before a machine learning technique is employed to forecast
the future wind speed, the chaotic characteristics of the histor-
ical wind speed series must be confirmed. Lyapunov expo-
nents are commonly used for this purpose. In 1983, it was
proven that if at least one of the Lyapunov exponents in a
dynamical system is positive, the corresponding time series can
be considered chaotic [50]. Therefore, the maximum Lyapu-
nov exponent of the time series needs to be calculated. If this
exponent exceeds zero, the time series is considered to have
chaotic characteristics. Among the various approaches for cal-
culating Lyapunov exponents, Wolf ’s method [51], which is
based on the phase space reconstruction theory of Takens, is
considered to be the most effective.
For a series X (t ) = (x (t ), x (x + t ), f, x ((m - 1)x + t ) re­­
constructed using Takens’ theory, as described above, the
distance L i between X (t i) and the closest point X (t n) can be
expressed as follows:
L i = X (t n) - X (t i) .(21)
According to Wolf ’s method, the maximum Lyapunov
exponent can be calculated as
M
m max = 1 / ln L i , (22)
t M - t 0 i = 0 L li
where t 0 and t M represent the initial time and the f inal
time, respectively, and M = N - (m - 1) x. In addition,
the interval for time series prediction can be calculated
as follows [52], [53]:
Tt = 1 .(23)
m max
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 57

It is difficult to forecast the future trend of a chaotic
time series such as a wind speed series because of
its irregularity. But it can be considered to represent a
type of random motion in a definite dynamical system.
This equation indicates that larger and smaller m max values
correspond to closer and more distant predictions, respectively.
E. Normalization
For a raw time series, the first task is normalization. Normalization
is a necessary and useful preprocessing step in the field of ANNs. By
normalizing the data, all features can be mapped to the same scale,
thereby reducing the computational cost and increasing the conver-
gence speed of optimization algorithms. In this study, the one-
dimensional wind speed time series " x (i ) ; i = 1, 2, f, N , is
normalized using the min-max method, as follows:
x t - MIN (x t)
Yt = , (24)
MAX (x t) - MIN (x t)
where MAX (X t) and MIN (X t) represent the maximum and
minimum values of the input vector, respectively. Using Eq.
(24), both the training and test samples are normalized to the
range [0,1]. In addition to min-max normalization, other
methods, such as mean and variance normalization and simple
normalization, are also commonly used [54], [55].
Start
Wind Speed Time Series
Divide Into the Training and
Testing Datasets
Calculate Time Delay and
Embedding Dimension
Conduct Phase Space
Reconstruction
Calculate Maximum
Lyapunov Exponent
FIGURE 2 Flowchart of DNR-based wind speed forecasting.
58 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
IV. Experiments and Discussion
A. Experimental Setup
The wind speed data used for experimental
analysis in this study pertain to a wind farm
in Galicia and are available for free at http://
www.evwind.es. To comprehensively evaluate
the prediction performance of DNR, two sets of
wind speed data with different time intervals, specifically, ten
minutes and one day, were collected; these datasets are illus-
trated in Figs. 3(a) and (b), respectively. The two datasets were
used to verify long-term and short-term predictions. Each
dataset contained 1,150 data points. The collected data were
divided into two subsets: training data and test data. The
training data were used to train the DNR parameters, and the
test data were utilized to evaluate the performance of the
trained model in terms of several indicators. The use of differ-
ent ratios of training data to test data can help determine the
dependence of an algorithm on the number of training data;
thus, three ratios were considered in our experiments: 1:1, 4:1
and 9:1. All experiments were conducted on a personal PC
with a 2.90 GHz Intel(R) Core i7 CPU and 16 GB of memory
using MATLAB R2018b.
B. Evaluation Indicators
To fairly evaluate the performance of each algorithm, several
evaluation methods were adopted in this study. The details of
these methods are given as follows.
No Chaotic
No
Estimate Whether
It Is Chaotic
Yes
Implement the
Normalization Operator
Train DNR on the Training
Dataset
Evaluate DNR on the Test
Dataset
End
 (1) Evaluation Indicators: To compare the prediction Before employing the DNR model to forecast the wind
performance of the algorithms, four widely used indica- speed, three hyperparameters, M, pop, and epoch, which could
tors were evaluated in our experiments: the mean-squared inf luence the performance of the proposed algorithm, were
error (MSE), the mean absolute error (MAE), the root- established. M denotes the number of dendritic branches, and
mean-squared error (RMSE) and the correlation coeffi- Pop and epoch represent the population size and maximum iter-
cient of the predictions (R). These indicators are defined ation time, respectively, for the SMS algorithm. It should be
as follows: noted that determining the best hyperparameter combination
is a difficult and time-consuming task. Although several
n
approaches have been proposed for the automatic tuning of
MSE = 1 / (O - T )2, (25)
n i=1 hyperparameters, the results are comparable to those of the tri-
n al-and-error approach. Based on empirical evidence, k, qs, M,
MAE = 1 / ;T - O ; , (26) pop, and epoch were set to 6, 0.8, 9, 100 and 1000, respectively,
n i=1
in this experiment.
n
1 / (O - T )2 , (27) Moreover, several commonly used prediction techniques
RMSE =
n i=1 were compared with the proposed model, including the
n
/ (T - Tr ) (O - Or )
i =1
R= n n , (28) Wind Speed Time Series Curve (One Day)
25
/ (T - Tr )2 / (O - Or )2 20
i =1 i =1 15
M/S
10
5
where O is the prediction output and T is the actual 0
0 200 400 600 800 1,000 1,200
result. MSE, MAE, and RMSE represent different kinds
of errors between the outputs and targets. Smaller indi- Wind Speed Time Series Curve (Ten Minutes)
16
cator values correspond to higher prediction perfor- 12
mance. R represents the degree of fit between the output
M/S

8
curve of a prediction algorithm and the original curve. 4
0
If R is equal to 1, the predicted curve fits the original 0 200 400 600 800 1,000 1,200
curve perfectly.
(2) Nonparametric Statistical Test: This test detects wheth- FIGURE 3 Wind speed curves for long-term and short-term predictions.
er a significant difference exists between the proposed meth-
od and another algorithm. In this study, the Wilcoxon rank
sum test [56], [57] was conducted using the KEEL software TABLE II Results for the time delays, embedding
[58]. The significance level was set to 5%. If the p-value of the dimensionalities and maximum Lyapunov exponents of
wind speed series at different time scales.
Wilcoxon rank sum test was less than 5%, a significant differ-
ence in performance was considered to exist between the two MAXIMUM
TIME EMBEDDING LYAPUNOV
algorithms being compared. TIME DELAY ­DIMENSIONALITY EXPONENT
(3) Relative Graphs: Graphs related to an experiment INTERVAL (x) (m) (mmax) CHAOTIC
enable more intuitive observation of the experimental ONE DAY 2 4 0.1181 YES
results. The following graphs were generated in this study. TEN MIN 5 7 0.0330 YES
First, fit graphs and correlation coefficient graphs were plot-
ted to visualize how well the predicted results matched the
target values. Second, a convergence graph was produced to TABLE III Parameter settings of algorithms for predicting
illustrate the speed and stability of the proposed model dur- wind speed time series.
ing the convergence process. ALGORITHM PARAMETERS
MLP HIDDENLAYER = 10, LEARNINGRATE = 0.01,
C. Performance Comparison and Analysis EPOCH = 1000
Following the approaches described above, the time delays, ENN LEARNINGRATE = 0.01, EPOCH = 1000
embedding dimensionalities and maximum Lyapunov SVR LINEAR, POLYNOMIAL, RBF AND SIGMOID KERNELS
­exponents of the wind speed series at different time scales were
LSTM HIDDENUNITS = 200, EPOCH = 1000
calculated, and the results are presented in Table II. The maxi-
DNM-BP LEARNINGRATE = 0.01, EPOCH = 1000
mum Lyapunov exponents presented in Table II imply that the
two wind speed time series are chaotic. Next, phase space EDNM POPSIZE = 100, EPOCH = 1000

reconstruction was conducted based on the time delay and DNR-BP LEARNINGRATE = 0.01, EPOCH = 1000
embedding dimensionality. DNR-SMS POPSIZE = 100, EPOCH = 1000

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 59

 TABLE IV Long-term prediction performance of all models for the experimental wind speed time series.

LONG-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 1:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))

ENN 1.19E-01±3.77E-01 9.13E-07 2.28E-01±2.59E-01 9.13E-07 1.99E+00±3.50E-01 9.13E-07 4.01E-01±1.26E-01

MLP 2.86E-02±5.42E-03 9.13E-07 1.68E-01±1.54E-02 9.13E-07 2.24E+00±2.18E-01 9.12E-07 2.94E-01±1.18E-01

DT 2.33E-02±0.00E+00 9.13E-07 1.53E-01±0.00E+00 9.12E-07 2.04E+00±0.00E+00 9.12E-07 4.45E-01±0.00E+00

SVR-L 1.66E-02±0.00E+00 1.00E+00 1.29E-01±0.00E+00 1.00E+00 1.68E+00±0.00E+00 1.00E+00 5.59E-01±0.00E+00

SVR-P 2.07E-02±0.00E+00 9.13E-07 1.44E-01±0.00E+00 9.12E-07 1.98E+00±0.00E+00 9.12E-07 4.56E-01±0.00E+00

SVR-R 1.65E-02±0.00E+00 1.00E+00 1.28E-01±0.00E+00 1.00E+00 1.70E+00±0.00E+00 1.00E+00 5.68E-01±0.00E+00

SVR-S 1.68E-02±0.00E+00 9.94E-01 1.30E-01±0.00E+00 9.93E-01 1.69E+00±0.00E+00 1.00E+00 5.51E-01±0.00E+00

LSTM 1.99E-02±2.10E-04 9.13E-07 1.41E-01±7.48E-04 9.11E-07 1.95E+00±1.27E-02 9.12E-07 4.90E-01±1.25E-02

DNM-BP 1.70E-02±3.63E-04 5.69E-01 1.30E-01±1.39E-03 5.69E-01 1.70E+00±4.40E-02 9.41E-01 5.50E-01±1.97E-02

EDNM 1.72E-02±4.20E-04 2.10E-03 1.31E-01±1.60E-03 2.10E-03 1.72E+00±3.48E-02 2.11E-01 5.50E-01±1.35E-02

DNR-BP 1.75E-02±1.86E-03 3.04E-01 1.32E-01±6.59E-03 3.04E-01 1.73E+00±9.29E-02 5.73E-01 5.41E-01±7.08E-02

DNR-SMS 1.69E-02±2.04E-04 - 1.30E-01±7.85E-04 - 1.72E+00±1.07E-02 - 5.53E-01±5.73E-03

LONG-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 4:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))

ENN 3.15E-02±4.48E-02 9.13E-07 1.62E-01±7.20E-02 9.13E-07 1.83E+00±1.21E-01 9.13E-07 3.92E-01±7.47E-02

MLP 2.72E-02±4.10E-03 9.13E-07 1.64E-01±1.24E-02 9.13E-07 2.12E+00±1.63E-01 9.13E-07 2.83E-01±1.21E-01

DT 2.57E-02±0.00E+00 9.11E-07 1.60E-01±0.00E+00 9.12E-07 2.13E+00±0.00E+00 9.12E-07 3.54E-01±0.00E+00

SVR-L 1.73E-02±0.00E+00 3.32E-06 1.32E-01±0.00E+00 3.32E-06 1.69E+00±0.00E+00 1.37E-06 5.01E-01±0.00E+00

SVR-P 2.01E-02±0.00E+00 9.13E-07 1.42E-01±0.00E+00 9.12E-07 1.88E+00±0.00E+00 9.12E-07 4.03E-01±0.00E+00

SVR-R 1.71E-02±0.00E+00 3.75E-01 1.31E-01±0.00E+00 3.83E-01 1.68E+00±0.00E+00 8.35E-05 5.08E-01±0.00E+00

SVR-S 1.75E-02±0.00E+00 9.11E-07 1.32E-01±0.00E+00 9.12E-07 1.70E+00±0.00E+00 9.12E-07 4.94E-01±0.00E+00

LSTM 1.86E-02±1.96E-04 9.12E-07 1.36E-01±7.19E-04 9.12E-07 1.81E+00±1.42E-02 9.13E-07 4.55E-01±7.19E-03

DNM-BP 1.76E-02±1.38E-03 1.30E-03 1.32E-01±4.80E-03 1.40E-03 1.71E+00±8.51E-02 2.70E-03 4.97E-01±8.60E-02

EDNM 1.76E-02±6.87E-04 6.91E-04 1.33E-01±2.55E-03 7.15E-04 1.70E+00±2.67E-02 7.70E-05 5.03E-01±2.40E-02

DNR-BP 1.74E-02±3.52E-04 2.00E-03 1.32E-01±1.32E-03 2.10E-03 1.68E+00±4.09E-02 1.38E-01 5.09E-01±5.46E-03

DNR-SMS 1.71E-02±1.84E-04 - 1.31E-01±7.05E-04 - 1.67E+00±9.46E-03 - 5.14E-01±6.22E-03

LONG-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 9:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))

ENN 2.70E-02±3.88E-03 1.01E-06 1.64E-01±1.10E-02 1.01E-06 2.01E+00±7.23E-02 9.13E-07 4.05E-01±5.49E-02

MLP 3.94E-02±6.43E-03 9.13E-07 1.98E-01±1.60E-02 9.13E-07 2.56E+00±2.31E-01 9.13E-07 2.05E-01±1.03E-01

DT 3.36E-02±0.00E+00 9.12E-07 1.83E-01±0.00E+00 9.12E-07 2.36E+00±0.00E+00 9.12E-07 3.31E-01±0.00E+00

SVR-L 2.25E-02±0.00E+00 4.88E-06 1.50E-01±0.00E+00 4.88E-06 1.89E+00±0.00E+00 9.12E-07 4.76E-01±0.00E+00

SVR-P 2.58E-02±0.00E+00 9.12E-07 1.61E-01±0.00E+00 9.12E-07 2.08E+00±0.00E+00 9.12E-07 3.70E-01±0.00E+00

SVR-R 2.22E-02±0.00E+00 1.38E-01 1.49E-01±0.00E+00 1.33E-01 1.88E+00±0.00E+00 4.88E-06 4.85E-01±0.00E+00

SVR-S 2.28E-02±0.00E+00 9.12E-07 1.51E-01±0.00E+00 9.12E-07 1.91E+00±0.00E+00 9.12E-07 4.68E-01±0.00E+00

LSTM 2.41E-02±9.15E-04 9.13E-07 1.55E-01±2.93E-03 9.13E-07 2.08E+00±1.05E-01 9.13E-07 4.90E-01±5.96E-03

DNM-BP 2.24E-02±1.81E-03 4.75E-01 1.50E-01±5.53E-03 4.75E-01 1.89E+00±1.02E-01 1.16E-01 4.75E-01±1.05E-01

EDNM 2.25E-02±1.17E-03 8.10E-02 1.50E-01±3.80E-03 8.10E-02 1.88E+00±5.46E-02 9.23E-02 4.88E-01±3.28E-02

DNR-BP 2.24E-02±4.79E-04 1.18E-02 1.50E-01±1.58E-03 1.12E-02 1.87E+00±3.47E-02 5.66E-02 4.90E-01±1.22E-02

DNR-SMS 2.21E-02±3.55E-04 - 1.49E-01±1.20E-03 - 1.86E+00±1.35E-02 - 4.94E-01±9.41E-03

60 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

TABLE V Short-term prediction performance of all models for the experimental wind speed time series.

SHORT-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 1:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))

ENN 4.35E-02±6.37E-02 9.13E-07 1.80E-01±1.06E-01 9.13E-07 9.84E-01±2.42E-01 9.13E-07 6.03E-01±2.53E-01

MLP 2.87E-02±1.08E-02 9.13E-07 1.66E-01±3.15E-02 9.13E-07 1.21E+00±2.24E-01 9.13E-07 5.75E-01±2.01E-01

DT 1.21E-02±0.00E+00 9.13E-07 1.10E-01±0.00E+00 9.13E-07 7.94E-01±0.00E+00 9.13E-07 8.09E-01±0.00E+00

SVR-L 4.69E-03±0.00E+00 1.00E+00 6.85E-02±0.00E+00 1.00E+00 4.89E-01±0.00E+00 1.00E+00 9.26E-01±0.00E+00

SVR-P 2.45E-02±0.00E+00 9.13E-07 1.56E-01±0.00E+00 9.13E-07 1.18E+00±0.00E+00 9.13E-07 8.13E-01±0.00E+00

SVR-R 5.85E-03±0.00E+00 8.59E-06 7.65E-02±0.00E+00 8.59E-06 5.50E-01±0.00E+00 1.37E-06 9.25E-01±0.00E+00

SVR-S 5.22E-03±0.00E+00 9.97E-01 7.22E-02±0.00E+00 9.97E-01 5.18E-01±0.00E+00 9.81E-01 9.15E-01±0.00E+00

LSTM 1.47E-02±1.99E-03 9.13E-07 1.21E-01±8.00E-03 9.13E-07 9.29E-01±6.56E-02 9.13E-07 8.63E-01±5.28E-02

DNM-BP 5.96E-03±3.04E-03 4.19E-01 7.59E-02±1.44E-02 4.67E-01 5.41E-01±1.08E-01 5.08E-01 9.15E-01±5.45E-02

EDNM 2.26E-02±2.96E-02 1.24E-06 1.30E-01±7.56E-02 1.24E-06 7.24E-01±2.95E-01 4.44E-06 7.25E-01±2.65E-01

DNR-BP 9.19E-03±1.98E-02 4.51E-01 8.29E-02±4.82E-02 4.75E-01 6.03E-01±3.84E-01 3.11E-01 8.99E-01±1.38E-01

DNR-SMS 5.43E-03±3.54E-04 - 7.36E-02±2.41E-03 - 5.24E-01±1.58E-02 - 9.21E-01±5.28E-03

SHORT-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 4:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))

ENN 1.03E-02±3.13E-02 9.13E-07 7.87E-02±6.42E-02 9.13E-07 5.12E-01±6.91E-02 9.13E-07 8.86E-01±1.24E-01

MLP 1.80E-02±7.20E-03 9.13E-07 1.32E-01±2.66E-02 9.13E-07 1.01E+00±2.14E-01 9.13E-07 6.03E-01±1.78E-01

DT 5.89E-03±0.00E+00 9.13E-07 7.68E-02±0.00E+00 9.13E-07 5.90E-01±0.00E+00 9.13E-07 8.82E-01±0.00E+00

SVR-L 3.42E-03±0.00E+00 3.02E-06 5.85E-02±0.00E+00 2.48E-06 4.48E-01±0.00E+00 2.48E-06 9.35E-01±0.00E+00

SVR-P 1.49E-02±0.00E+00 9.13E-07 1.22E-01±0.00E+00 9.13E-07 9.48E-01±0.00E+00 9.13E-07 7.73E-01±0.00E+00

SVR-R 3.70E-03±0.00E+00 9.13E-07 6.09E-02±0.00E+00 9.13E-07 4.73E-01±0.00E+00 9.13E-07 9.31E-01±0.00E+00

SVR-S 3.63E-03±0.00E+00 1.01E-06 6.02E-02±0.00E+00 1.01E-06 4.66E-01±0.00E+00 9.13E-07 9.30E-01±0.00E+00

LSTM 7.46E-03±3.75E-04 9.13E-07 8.63E-02±2.17E-03 9.13E-07 6.96E-01±1.85E-02 9.13E-07 8.57E-01±8.12E-03

DNM-BP 7.54E-03±2.37E-02 6.67E-01 6.64E-02±5.60E-02 6.75E-01 5.12E-01±4.89E-01 8.67E-01 8.87E-01±2.75E-01

EDNM 3.43E-03±8.12E-04 5.90E-02 5.82E-02±6.05E-03 6.40E-02 4.29E-01±3.11E-02 7.71E-01 9.31E-01±1.69E-02

DNR-BP 9.29E-03±3.26E-02 7.79E-02 6.92E-02±6.70E-02 8.41E-02 5.39E-01±6.00E-01 4.35E-01 9.12E-01±1.43E-01

DNR-SMS 3.15E-03±1.50E-04 - 5.61E-02±1.32E-03 - 4.27E-01±1.14E-02 - 9.36E-01±3.12E-03

SHORT-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 9:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))

ENN 4.47E-03±4.25E-04 9.13E-07 6.68E-02±3.12E-03 9.13E-07 4.98E-01±2.39E-02 9.13E-07 8.77E-01±1.21E-02

MLP 1.92E-02±4.41E-03 9.13E-07 1.37E-01±1.70E-02 9.13E-07 1.08E+00±1.46E-01 9.13E-07 4.58E-01±1.36E-01

DT 6.73E-03±0.00E+00 9.13E-07 8.20E-02±0.00E+00 9.13E-07 6.23E-01±0.00E+00 9.13E-07 8.14E-01±0.00E+00

SVR-L 3.81E-03±0.00E+00 9.13E-07 6.18E-02±0.00E+00 9.13E-07 4.64E-01±0.00E+00 9.13E-07 8.93E-01±0.00E+00

SVR-P 9.06E-03±0.00E+00 9.13E-07 9.52E-02±0.00E+00 9.13E-07 7.75E-01±0.00E+00 9.13E-07 8.24E-01±0.00E+00

SVR-R 3.85E-03±0.00E+00 9.13E-07 6.21E-02±0.00E+00 9.13E-07 4.69E-01±0.00E+00 9.13E-07 8.93E-01±0.00E+00

SVR-S 3.88E-03±0.00E+00 9.13E-07 6.23E-02±0.00E+00 9.13E-07 4.65E-01±0.00E+00 9.13E-07 8.93E-01±0.00E+00

LSTM 5.56E-03±2.16E-04 9.12E-07 7.46E-02±1.45E-03 9.13E-07 5.81E-01±1.42E-02 9.13E-07 8.51E-01±6.16E-03

DNM-BP 8.09E-03±2.36E-02 3.02E-06 7.10E-02±5.51E-02 3.02E-06 5.39E-01±4.84E-01 1.16E-01 8.90E-01±5.53E-02

EDNM 3.58E-03±2.74E-04 1.33E-01 5.98E-02±2.15E-03 1.29E-01 4.41E-01±1.30E-02 9.98E-01 9.02E-01±2.43E-03

DNR-BP 9.46E-03±3.09E-02 1.59E-04 7.29E-02±6.44E-02 1.59E-04 5.66E-01±5.90E-01 2.19E-02 8.83E-01±1.14E-01

DNR-SMS 3.50E-03±7.30E-05 - 5.91E-02±6.17E-04 - 4.46E-01±7.34E-03 - 9.03E-01±2.07E-03

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 61

Elman neural network (ENN) [59]; a multilayer perceptron
(MLP); a decision tree (DT) model; support vector regres-
sion with a linear kernel (SVR-l), a polynomial kernel
(SVR-p), a radial basis function (RBF) kernel (SVR-r), and
a sigmoid kernel (SVR-s) [60]; a long short-term memory
(LSTM) network; and several variants of the DNM,
0.4
0.3
MSE
specifically, the original DNM, the DNM trained with the
L-SHADE algorithm (EDNM), and a DNR model trained
with the BP algorithm. The parameter settings for all algo-
rithms are listed in Table III. To ensure fair comparisons,
the experiments with all algorithms were conducted 30
times for each time series.
Since the size of the training dataset can impact the fore-
casting accuracy, the prediction results of DNR-SMS were
evaluated on wind speed data with different partition ratios
between the training and test datasets, specifically, 1:1, 4:1
One-Day 1:1 and 9:1. The average and standard deviation for each
One-Day 4:1
One-Day 9:1 approach are summarized in Tables IV and V. Note that

0.2 Ten-Min 1:1 MSE and RMSE were calculated using the normalized data,
Ten-Min 4:1 whereas MAE and R were calculated using the raw data.
0.1 Ten-Min 9:1
Tables IV and V show that for most algorithms, the predic-
0 tion results for both the long-term and short-term wind
0 100 200 300 400 500
Learning Epoch
speed time series with a partition ratio of 4:1 were superior to
those for a ratio of 1:1; moreover, the prediction results for a
FIGURE 4 Convergence curves of the DNR model for the experimental partition ratio of 9:1 were inferior to those for 4:1. These
wind speed time series. f indings suggest that merely increasing the number of

Output ~= 0.34 ∗ Target + 4.6

Training and Prediction (One-Day 1:1) 2 4 6 8 10 12 14 16
The Values of InFlow (M/S)

20
Data 16
Actual Value Training Fit
15 Prediction 14
Predictive Value Y=T 12
10 10
8
5 6
4
0 2
0 200 400 600 800 1,000 1,200 Target
(a)
Output ~= 0.29 ∗ Target + 4.8

Training and Prediction (One-Day 4:1) 2 4 6 8 10 12 14 16

The Values of InFlow (M/S)

20 16
Actual Value Training Prediction Data 14
15 Predictive Value Fit
Y=T 12
10 10
8
5 6
4
0 2
0 200 400 600 800 1,000 1,200 Target
(b)

Training and Prediction (One-Day 9:1) 2 4 6 8 10 12 14 16

Output ~= 0.27 ∗ Target + 5
The Values of InFlow (M/S)

20 16
Actual Value Data
Training Prediction Fit 14
15 Predictive Value
Y=T 12
10 10
8
5 6
4
0 2
0 200 400 600 800 1,000 1,200 Target
(c)

FIGURE 5 Training and prediction results and predicted correlation coefficients of DNR-SMS for the wind speed time series with an interval of
one day and partition ratios of (a) 1:1, (b) 4:1, and (c) 9:1.

62 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

training samples does not always enhance
the prediction performance and that a par- To fairly evaluate the performance of each algorithm,
tition ratio of 4:1 may be a suitable choice. several evaluation methods were adopted in this
In addition, for a partition ratio of 1:1,
SVR-r and SVR-l yielded superior results
study, such as the evaluation indicators, nonparametric
on both the long-term and short-term statistical test, and relative graphs.
wind speed data compared to those
obtained using the other algorithms. Howev-
er, when the partition ratio was 4:1 or 9:1, DNR-SMS SVR-r and SVR-l outperform DNR-SMS for the ratio of
achieved the highest prediction performance among the test- 1:1, the corresponding p-values are larger than 5%, indicat-
ed algorithms. These findings imply that the training pro- ing that the performance is not significantly higher than
cesses for SVR-r and SVR-l require less data than those for that of DNR-SMS. However, for ratios of 4:1 and 9:1,
other algorithms. Thus, when the length of the wind speed DNR-SMS can produce significantly superior results com-
time series is insufficient, SVR-r and SVR-l can provide pared to the other algorithms, such as the ENN, the MLP,
more satisfactory results. However, if an adequate amount of the various SVR methods and the LSTM network, based
wind speed time series data is available, DNR can demon- on the corresponding p-values. Moreover, compared with
strate highly competitive performance. the prediction performance of the original DNM-BP algo-
To identify the signif icant differences between the rithm and EDNM, that of the proposed DNR model is
compared algorithms, the p-values of the Wilcoxon rank significantly higher, which implies that considering the
sum test are also provided in Tables IV and V. Although strength of each dendritic branch individually can enhance

Training and Prediction (Ten-Min 1:1) 1 2 3 4 5 6 7

Output ~= 0.74 ∗ Target + 1

The Values of InFlow (M/S)

10 7
Data
Actual Value Training Prediction Fit 6
8 Predictive Value Y=T 5
6
4
4 3
2 2
1
0
0 200 400 600 800 1,000 1,200 Target
(a)
Output ~= 0.85 ∗ Target + 0.58

Training and Prediction (Ten-Min 4:1) 1 2 3 4 5 6

The Values of InFlow (M/S)

10
Actual Value Training Data
Data 6
8 Prediction Fit
Fit
Predictive Value 5
Y == TT
Y
6 4
4 3
2 2

0 1
0 200 400 600 800 1,000 1,200 Target
(b)
Output ~= 0.84 ∗ Target + 0.66

Training and Prediction (Ten-Min 9:1) 2 3 4 5 6

The Values of InFlow (M/S)

10
Actual Value Data
Data
8 Training Prediction Fit
Fit 6
Predictive Value
Y
Y == TT
6 5

4 4

2 3

0 2
0 200 400 600 800 1,000 1,200 Target
(c)

FIGURE 6 Training and prediction results and predicted correlation coefficients of DNR-SMS for the wind speed time series with an interval of
ten minutes and partition ratios of (a) 1:1, (b) 4:1, and (c) 9:1.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 63


Experimental results demonstrate that the DNR
can provide satisfactory and stable forecasting
performances for wind speed time series.
the regression ability. Thus, DNR is more suitable for
solving prediction problems than the DNM. In addition, a
comparison of DNR-SMS and DNR-BP is presented in
Tables  IV and V. DNR-SMS is significantly superior to
DNR-BP in most cases, suggesting that the SMS algo-
rithm, as a global optimization approach that does not eas-
ily become trapped in local minima, can offer enhanced
optimization performance compared to that achieved using
the BP algorithm.
Furthermore, the convergence curves of DNR-SMS for
the wind speed time ser ies are provided in Fig. 4.
Although the maximum number of iterations of the SMS
algorithm was set to 1000, convergence was attained in less
than 200 iterations. Thus, the DNR model trained using
the SMS algorithm could achieve prompt convergence,
indicating that the SMS algorithm is an efficient optimiza-
tion tool for real-world prediction problems. In addition,
Figures 5 and 6 show the fit graphs and corresponding
correlation coefficient graphs of DNR-SMS for long-term
and short-term wind speed prediction,
respectively. Figures 5 and 6 indicate that
long-term wind speed time series prediction
is more complex than short-term prediction.
Although the forecast curves of DNR-SMS
closely fit the actual curves for both long-
term and short-term wind speed prediction with different
partition ratios, the long-term prediction performance of
DNR-SMS is higher than the short-term prediction perfor-
mance, as indicated by the correlation coefficient graphs. In
addition, for both long-term and short-term wind speed pre-
diction, the performance of the DNR model trained using
the SMS algorithm was compared with that of DNR models
trained using other state-of-the-art evolutionary algorithms,
namely, a genetic algorithm (GA), particle swarm optimiza-
tion (PSO), adaptive differential evolution with an optional
external archive (JADE) [61] and differential evolution with
success-history-based parameter adaptation and linear popula-
tion size reduction (L-SHADE) [62]. The experimental
results are presented in Tables VI and VII. The SMS algo-
rithm notably outperforms the other algorithms. Conse-
quently, the SMS algorithm is recommended as the main
optimization approach for DNR. These experimental results
demonstrate that the proposed approach can enable satisfacto-
ry and stable forecasting for wind speed time series.

TABLE VI Long-term prediction performance of DNR models trained using different evolution algorithms
for the experimental wind speed time series.

LONG-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 1:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))
DNR-GA 1.91E-02±1.55E-03 1.01E-06 1.38E-01±5.50E-03 1.01E-06 1.84E+00±1.16E-01 4.89E-06 5.02E-01±5.98E-02
DNR-PSO 1.73E-02±6.64E-04 1.01E-02 1.32E-01±2.50E-03 1.06E-02 1.72E+00±1.73E-02 3.48E-01 5.46E-01±1.68E-02
DNR-JADE 1.74E-02±6.89E-04 9.13E-07 1.32E-01±2.59E-03 9.12E-07 1.73E+00±4.62E-02 9.12E-07 5.46E-01±2.10E-02
DNR-L-SHADE 1.81E-02±2.15E-03 1.46E-02 1.34E-01±7.48E-03 1.54E-02 1.76E+00±1.08E-01 8.41E-02 5.27E-01±6.00E-02
DNR-SMS 1.69E-02±2.04E-04 - 1.30E-01±7.85E-04 - 1.72E+00±1.07E-02 - 5.53E-01±5.73E-03

LONG-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 4:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))
DNR-GA 1.91E-02±1.43E-03 9.13E-07 1.38E-01±5.03E-03 9.13E-07 1.79E+00±9.18E-02 1.01E-06 4.66E-01±5.43E-02
DNR-PSO 1.74E-02±4.84E-04 1.50E-02 1.32E-01±1.82E-03 1.54E-02 1.69E+00±1.97E-02 6.67E-04 5.06E-01±1.40E-02
DNR-JADE 1.78E-02±1.01E-03 2.65E-05 1.34E-01±3.66E-03 2.54E-05 1.72E+00±3.98E-02 7.13E-06 5.00E-01±3.17E-02
DNR-L-SHADE 1.85E-02±1.66E-03 1.03E-05 1.36E-01±5.93E-03 1.13E-05 1.73E+00±6.58E-02 7.13E-06 4.81E-01±4.08E-02
DNR-SMS 1.71E-02±1.84E-04 - 1.31E-01±7.05E-04 - 1.67E+00±9.46E-03 - 5.14E-01±6.22E-03

LONG-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 9:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))
DNR-GA 2.46E-02±2.20E-03 2.25E-06 1.57E-01±6.81E-03 2.25E-06 1.97E+00±1.05E-01 3.66E-06 4.34E-01±6.40E-02
DNR-PSO 2.35E-02±2.80E-03 2.27E-03 1.53E-01±8.46E-03 2.27E-03 1.91E+00±6.54E-02 5.51E-05 4.66E-01±5.01E-02
DNR-JADE 2.30E-02±1.19E-03 8.25E-04 1.52E-01±3.91E-03 9.49E-04 1.90E+00±4.89E-02 1.86E-03 4.82E-01±2.57E-02
DNR-L-SHADE 2.35E-02±1.49E-03 2.13E-05 1.53E-01±4.80E-03 2.13E-05 1.90E+00±4.06E-02 6.52E-05 4.67E-01±3.16E-02
DNR-SMS 2.21E-02±3.55E-04 - 1.49E-01±1.20E-03 - 1.86E+00±1.35E-02 - 4.94E-01±9.41E-03

64 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

 TABLE VII Short-term prediction performance of DNR models trained using different evolution algorithms
for the experimental wind speed time series.

SHORT-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 1:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))

DNR-GA 1.14E-02±4.36E-03 1.12E-06 1.05E-01±1.94E-02 1.12E-06 7.56E-01±1.47E-01 1.24E-06 8.46E-01±7.47E-02

DNR-PSO 1.09E-02±1.51E-02 1.37E-06 9.63E-02±4.08E-02 1.37E-06 6.11E-01±1.49E-01 3.02E-06 8.46E-01±1.52E-01

DNR-JADE 8.55E-03±5.98E-03 2.54E-04 8.91E-02±2.47E-02 2.75E-04 5.99E-01±1.13E-01 2.12E-03 8.80E-01±9.16E-02

DNR-L-SHADE 1.27E-02±1.50E-02 9.13E-07 1.05E-01±4.06E-02 9.13E-07 6.41E-01±1.44E-01 1.12E-06 8.11E-01±1.45E-01

DNR-SMS 5.43E-03±3.54E-04 - 7.36E-02±2.41E-03 - 5.24E-01±1.58E-02 - 9.21E-01±5.28E-03

SHORT-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 4:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))

DNR-GA 7.04E-03±3.07E-03 1.85E-06 8.20E-02±1.79E-02 1.85E-06 6.31E-01±1.44E-01 2.25E-06 8.80E-01±6.31E-02

DNR-PSO 3.35E-03±5.19E-04 6.15E-02 5.78E-02±4.26E-03 6.40E-02 4.28E-01±2.23E-02 5.00E-01 9.32E-01±9.73E-03

DNR-JADE 3.92E-03±1.38E-03 8.25E-04 6.19E-02±9.57E-03 8.85E-04 4.61E-01±7.11E-02 2.92E-02 9.29E-01±2.09E-02

DNR-L-SHADE 3.66E-03±1.53E-03 3.40E-01 5.96E-02±1.05E-02 3.40E-01 4.21E-01±2.54E-02 9.19E-01 9.27E-01±2.95E-02

DNR-SMS 3.15E-03±1.50E-04 - 5.61E-02±1.32E-03 - 4.27E-01±1.14E-02 - 9.36E-01±3.12E-03

SHORT-TERM WIND SPEED TIME SERIES WITH A PARTITION RATIO OF 9:1

MSE (MEAN ± STD RMSE (MEAN ± STD MAE (MEAN ± STD R (MEAN ± STD
ALGORITHM (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD)) P-VALUE (OR SD))

DNR-GA 7.52E-03±2.95E-03 1.24E-06 8.50E-02±1.72E-02 1.24E-06 6.47E-01±1.37E-01 9.13E-07 8.26E-01±8.36E-02

DNR-PSO 3.98E-03±2.90E-03 9.79E-01 6.14E-02±1.46E-02 9.79E-01 4.65E-01±1.19E-01 7.95E-01 8.86E-01±1.06E-01

DNR-JADE 4.17E-03±5.79E-04 3.02E-06 6.44E-02±4.41E-03 3.02E-06 4.80E-01±3.67E-02 2.02E-04 8.95E-01±1.43E-02

DNR-L-SHADE 3.55E-03±1.95E-04 5.25E-01 5.95E-02±1.59E-03 5.08E-01 4.46E-01±1.34E-02 7.64E-01 9.02E-01±6.46E-03

DNR-SMS 3.50E-03±7.30E-05 - 5.91E-02±6.17E-04 - 4.46E-01±7.34E-03 - 9.03E-01±2.07E-03

V. Conclusion model. Second, DNR-SMS can be applied to solve other pre-

A novel DNR method inspired by biological neurons is
employed to predict wind speed time series. To improve the
prediction results, the neural architecture for DNR is trained
using the SMS algorithm, which is a global optimization algo-
rithm with powerful search capabilities. Before the prediction
process, the time delay and the number of embedding dimen-
sions are calculated using the MI algorithm and the FNN
algorithm, respectively. Based on the results, phase space
reconstruction of the wind speed data is performed, and the
maximum Lyapunov exponent is calculated to detect the cha-
otic nature of the wind speed series. Comprehensive experi-
ments conducted to evaluate the effectiveness of the proposed
DNR-SMS approach are reported, in which several common-
ly used prediction approaches are considered for comparison
and four diverse evaluation indicators are employed. The
experimental results and statistical tests demonstrate that DNR
can outperform the other compared approaches. The proposed
DNR-SMS model is thus an efficient tool for wind speed
­prediction due to its high prediction accuracy and robust sta-
bility. Further research can be conducted in the following
directions. First, other optimization algorithms can be evaluat-
ed to enhance the forecasting performance of the DNR
diction problems, such as container throughput forecasting and
power load prediction.
Acknowledgments
This work was supported by a Project of the Guangdong Basic and
Applied Basic Research Fund (No. 2019A1515111139), by the
National Natural Science Foundation of China under Grants
61876110 and 61876162, by a Key Program of the Joint Funds
of the National Natural Science Foundation of China under
Grant U1713212, and in part by the Research Grants Council
of the Hong Kong Special Administrative Region, China,
under Grant PolyU11202418 and Grant PolyU11209219.
References
[1] W. Yang, J. Wang, H. Lu, T. Niu, and P. Du, “Hybrid wind energy forecasting and
analysis system based on divide and conquer scheme: A case study in China,” J. Cleaner
Production, vol. 222, pp. 942–959, 2019. doi: 10.1016/j.jclepro.2019.03.036.
[2] C. Tian, Y. Hao, and J. Hu, “A novel wind speed forecasting system based on hybrid
data preprocessing and multi-objective optimization,” Appl. Energy, vol. 231, pp. 301–319,
2018. doi: 10.1016/j.apenergy.2018.09.012.
[3] H. Li, J. Wang, H. Lu, and Z. Guo, “Research and application of a combined model
based on variable weight for short term wind speed forecasting,” Renewable Energy, vol.
116, pp. 669–684, 2018. doi: 10.1016/j.renene.2017.09.089.
[4] M. Imal, M. Sekkeli, C. Yildiz, and F. Kececioglu, “Wind energy potential estimation
and evaluation of electricity generation in Kahramanmaras, Turkey,” Energy Educ. Sci.
Technol. A, Energy Sci. Res., vol. 30, no. 1, pp. 661–672, 2012.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 65

[5] J. Zhao, Z.-H. Guo, Z.-Y. Su, Z.-Y. Zhao, X. Xiao, and F. Liu, “An improved multi-
step forecasting model based on WRF ensembles and creative fuzzy systems for wind
speed,” Appl. Energy, vol. 162, pp. 808–826, 2016. doi: 10.1016/j.apenergy.2015.10.145.
[6] C. Y ild iz, M. Tekin, A. Gani, Ö. F. Keçeciog˘lu, H. Aç ikgöz, and M. S¸ekkeli, “A
day-ahead wind power scenario generation, reduction, and quality test tool,” Sustainabil-
ity, vol. 9, no. 5, p. 864, 2017. doi: 10.3390/su9050864.
[7] D. Wang, H. Luo, O. Grunder, and Y. Lin, “Multi-step ahead wind speed forecasting using an
improved wavelet neural network combining variational mode decomposition and phase space re-
construction,” Renewable Energy, vol. 113, pp. 1345–1358, 2017. doi: 10.1016/j.renene.2017.06.095.
[8] J. Wang, P. Du, T. Niu, and W. Yang, “A novel hybrid system based on a new proposed
algorithm–multi-objective whale optimization algorithm for wind speed forecasting,”
Appl. Energy, vol. 208, pp. 344–360, 2017. doi: 10.1016/j.apenergy.2017.10.031.
[9] J. Song, J. Wang, and H. Lu, “A novel combined model based on advanced opti-
mization algorithm for short-term wind speed forecasting,” Appl. Energy, vol. 215, pp.
643–658, 2018. doi: 10.1016/j.apenergy.2018.02.070.
[10] T. Howard and P. Clark, “Correction and downscaling of NWP wind speed fore-
casts,” Meteorol. Appl., J. Forecasting, Pract. Appl., Train. Techn. Model., vol. 14, no. 2, pp.
105–116, 2007. doi: 10.1002/met.12.
[11] P. Du, J. Wang, W. Yang, and T. Niu, “A novel hybrid model for short-term wind power
forecasting,” Appl. Soft Comput., vol. 80, pp. 93–106, 2019. doi: 10.1016/j.asoc.2019.03.035.
[12] Y. Wang, J. Wang, G. Zhao, and Y. Dong, “Application of residual modification ap-
proach in seasonal ARIMA for electricity demand forecasting: A case study of China,”
Energy Pol., vol. 48, pp. 284–294, 2012. doi: 10.1016/j.enpol.2012.05.026.
[13] J. Wang, J. Heng, L. Xiao, and C. Wang, “Research and application of a combined
model based on multi-objective optimization for multi-step ahead wind speed forecast-
ing,” Energy, vol. 125, pp. 591–613, 2017. doi: 10.1016/j.energy.2017.02.150.
[14] R. G. Kavasseri and K. Seetharaman, “Day-ahead wind speed forecasting using f-ARIMA
models,” Renewable Energy, vol. 34, no. 5, pp. 1388–1393, 2009. doi: 10.1016/j.renene.2008.09.006.
[15] R. Li and Y. Jin, “A wind speed interval prediction system based on multi-objective
optimization for machine learning method,” Appl. Energy, vol. 228, pp. 2207–2220, 2018.
doi: 10.1016/j.apenergy.2018.07.032.
[16] H. Liu, Z. Duan, Y. Li, and H. Lu, “A novel ensemble model of different mother
wavelets for wind speed multi-step forecasting,” Appl. Energy, vol. 228, pp. 1783–1800,
2018. doi: 10.1016/j.apenergy.2018.07.050.
[17] Q. Zhou, C. Wang, and G. Zhang, “Hybrid forecasting system based on an optimal
model selection strategy for different wind speed forecasting problems,” Appl. Energy, vol.
250, pp. 1559–1580, 2019. doi: 10.1016/j.apenergy.2019.05.016.
[18] Y. Hao and C. Tian, “A novel two-stage forecasting model based on error factor and
ensemble method for multi-step wind power forecasting,” Appl. Energy, vol. 238, pp.
368–383, 2019. doi: 10.1016/j.apenergy.2019.01.063.
[19] A. Sfetsos, “A comparison of various forecasting techniques applied to mean hourly
wind speed time series,” Renewable Energy, vol. 21, no. 1, pp. 23–35, 2000. doi: 10.1016/
S0960-1481(99)00125-1.
[20] Y. Wang, Y. Yu, S. Cao, X. Zhang, and S. Gao, “A review of applications of artificial
intelligent algorithms in wind farms,” Artif. Intell. Rev., pp. 1–54, 2019. doi: 10.1007/
s10462-019-09768-7.
[21] A. Liu, Y. Xue, J. HU, and L. LIU, “Ultra-short-term wind power forecasting based
on SVM optimized by GA,” Power Syst. Protection Control, vol. 43, no. 2, pp. 90–95, 2015.
[22] X. Kong, X. Liu, R. Shi, and K. Y. Lee, “Wind speed prediction using reduced sup-
port vector machines with feature selection,” Neurocomputing, vol. 169, pp. 449–456, 2015.
doi: 10.1016/j.neucom.2014.09.090.
[23] P. Jiang, Y. Wang, and J. Wang, “Short-term wind speed forecasting using a hybrid
model,” Energy, vol. 119, pp. 561–577, 2017. doi: 10.1016/j.energy.2016.10.040.
[24] Z.-h. Guo, J. Wu, H.-y. Lu, and J.-z. Wang, “A case study on a hybrid wind speed
forecasting method using bp neural network,” Knowl.-Based Syst., vol. 24, no. 7, pp.
1048–1056, 2011.
[25] A. Ahmed and M. Khalid, “An intelligent framework for short-term multi-step wind
speed forecasting based on functional networks,” Appl. Energy, vol. 225, pp. 902–911,
2018. doi: 10.1016/j.apenergy.2018.04.101.
[26] S. P. Kani and M. Ardehali, “Very short-term wind speed prediction: A new artifi-
cial neural network–Markov chain model,” Energy Convers. Manage., vol. 52, no. 1, pp.
738–745, 2011. doi: 10.1016/j.enconman.2010.07.053.
[27] G. Memarzadeh and F. Keynia, “A new short-term wind speed forecasting method
based on fine-tuned LSTM neural network and optimal input sets,” Energy Convers. Man-
age., vol. 213, p. 112,824, 2020.
[28] G. Zhang and D. Liu, “Causal convolutional gated recurrent unit network with
multiple decomposition methods for short-term wind speed forecasting,” Energy Convers.
Manage., vol. 226, p. 113,500, 2020.
[29] J. Ji, S. Gao, J. Cheng, Z. Tang, and Y. Todo, “An approximate logic neuron model
with a dendritic structure,” Neurocomputing, vol. 173, pp. 1775–1783, 2016. doi: 10.1016/j.
neucom.2015.09.052.
[30] T. Zhou, S. Gao, J. Wang, C. Chu, Y. Todo, and Z. Tang, “Financial time series
prediction using a dendritic neuron model,” Knowl.-Based Syst., vol. 105, pp. 214–224,
2016. doi: 10.1016/j.knosys.2016.05.031.
[31] W. Chen, J. Sun, S. Gao, J.-J. Cheng, J. Wang, and Y. Todo, “Using a single dendritic
neuron to forecast tourist arrivals to Japan,” IEICE Trans. Inf. Syst., vol. 100, no. 1, pp.
190–202, 2017. doi: 10.1587/transinf.2016EDP7152.
[32] J. Ji, S. Song, Y. Tang, S. Gao, Z. Tang, and Y. Todo, “Approximate logic neu-
ron model trained by states of matter search algorithm,” Knowl.-Based Syst., vol. 163, pp.
120–130, 2019. doi: 10.1016/j.knosys.2018.08.020.
[33] J. K. Makara, A. Losonczy, Q. Wen, and J. C. Magee, “Experience-dependent com-
partmentalized dendritic plasticity in rat hippocampal ca1 pyramidal neurons,” Nature
Neurosci., vol. 12, no. 12, p. 1485, 2009. doi: 10.1038/nn.2428.
66 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
[34] T. Elsken, J. H. Metzen, F. Hutter et al., “Neural architecture search: A survey.” J.
Mach. Learn. Res., vol. 20, no. 55, pp. 1–21, 2019.
[35] Y. Sun, B. Xue, M. Zhang, G. G. Yen, and J. Lv, “Automatically designing CNN
architectures using the genetic algorithm for image classification,” IEEE Trans. Cybern.,
vol. 50, no. 9, pp. 3840–3854, 2020. doi: 10.1109/TCYB.2020.2983860.
[36] Z. Lu, I. Whalen, Y. Dhebar, K. Deb, E. Goodman, W. Banzhaf, and V. N. Boddeti,
“Multi-objective evolutionary design of deep convolutional neural networks for image
classification,” IEEE Trans. Evol. Comput., 2020. doi: 10.1109/TEVC.2020.3024708.
[37] J. Ji, Y. Tang, L. Ma, J. Li, Q. Lin, Z. Tang, and Y. Todo, “Accuracy versus simplifica-
tion in an approximate logic neural model,” IEEE Trans. Neural Netw. Learn. Syst., 2020.
[38] E. Cuevas, A. Echavarría, and M. A. Ramírez-Ortegón, “An optimization algorithm
inspired by the states of matter that improves the balance between exploration and exploi-
tation,” Appl. Intell., vol. 40, no. 2, pp. 256–272, 2014. doi: 10.1007/s10489-013-0458-0.
[39] N. Chouikhi, B. Ammar, N. Rokbani, and A. M. Alimi, “PSO-based analysis of
echo state network parameters for time series forecasting,” Appl. Soft Comput., vol. 55, pp.
211–225, 2017. doi: 10.1016/j.asoc.2017.01.049.
[40] W. Jia, D. Zhao, Y. Zheng, and S. Hou, “A novel optimized GA–Elman neural net-
work algorithm,” Neural Comput. Appl., vol. 31, no. 2, pp. 449–459, 2019. doi: 10.1007/
s00521-017-3076-7.
[41] P. Ong and Z. Zainuddin, “Optimizing wavelet neural networks using modified
cuckoo search for multi-step ahead chaotic time series prediction,” Appl. Soft Comput.,
vol. 80, pp. 374–386, 2019. doi: 10.1016/j.asoc.2019.04.016.
[42] D. Z. Li, W. Wang, and F. Ismail, “An evolving fuzzy neural predictor for multi-
dimensional system state forecasting,” Neurocomputing, vol. 145, pp. 381–391, 2014. doi:
10.1016/j.neucom.2014.05.014.
[43] T. Desell, S. Clachar, J. Higgins, and B. Wild, “Evolving deep recurrent neural
networks using ant colony optimization,” in Proc. Eur. Conf. Evol. Comput. Combinatorial
Optimization. Springer-Verlag, 2015, pp. 86–98.
[44] F. Takens, “Detecting strange attractors in turbulence,” in Dynamical Systems and
Turbulence, Warwick 1980. SpringerVerlag, 1981, pp. 366–381.
[45] C. Koch, Biophysics of Computation: Information Processing in Single Neurons. London:
Oxford Univ. Press, 1998.
[46] E. Salinas and L. Abbott, “A model of multiplicative neural responses in parietal
cortex,” Proc. Nat. Acad. Sci., vol. 93, no. 21, pp. 11,956–11,961, 1996. doi: 10.1073/
pnas.93.21.11956.
[47] F. Gabbiani, H. G. Krapp, C. Koch, and G. Laurent, “Multiplicative computation
in a visual neuron sensitive to looming,” Nature, vol. 420, no. 6913, pp. 320–324, 2002.
[48] M. B. Kennel, R. Brown, and H. D. Abarbanel, “Determining embedding dimension
for phase-space reconstruction using a geometrical construction,” Phys. Rev. A, vol. 45,
no. 6, p. 3403, 1992. doi: 10.1103/PhysRevA.45.3403.
[49] A. M. Fraser and H. L. Swinney, “Independent coordinates for strange attractors
from mutual information,” Phys. Rev. A, vol. 33, no. 2, p. 1134, 1986. doi: 10.1103/
PhysRevA.33.1134.
[50] C. Grebogi, E. Ott, and J. A. Yorke, “Crises, sudden changes in chaotic attractors,
and transient chaos,” Phys. D, Nonlinear Phenomena, vol. 7, nos. 1–3, pp. 181–200, 1983.
doi: 10.1016/0167-2789(83)90126-4.
[51] A. Wolf, J. B. Swift, H. L. Swinney, and J. A. Vastano, “Determining Lyapunov
exponents from a time series,” Phys. D, Nonlinear Phenomena, vol. 16, no. 3, pp. 285–317,
1985. doi: 10.1016/0167-2789(85)90011-9.
[52] P. Shang, X. Na, and S. Kamae, “Chaotic analysis of time series in the sediment
transport phenomenon,” Chaos, Solitons Fractals, vol. 41, no. 1, pp. 368–379, 2009. doi:
10.1016/j.chaos.2008.01.014.
[53] H. Abarbanel, Analysis of Observed Chaotic Data. Springer Science & Business Media,
2012.
[54] J. Sola and J. Sevilla, “Importance of input data normalization for the application of
neural networks to complex industrial problems,” IEEE Trans. Nuclear Sci., vol. 44, no. 3,
pp. 1464–1468, 1997. doi: 10.1109/23.589532.
[55] G. Zhang, B. E. Patuwo, and M. Y. Hu, “Forecasting with artificial neural networks:
The state of the art,” Int. J. Forecasting, vol. 14, no. 1, pp. 35–62, 1998. doi: 10.1016/
S0169-2070(97)00044-7.
[56] S. García, D. Molina, M. Lozano, and F. Herrera, “A study on the use of non-
parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the
CEC–2005 special session on real parameter optimization,” J. Heurist., vol. 15, no. 6, p.
617, 2009. doi: 10.1007/s10732-008-9080-4.
[57] J. Derrac, S. García, D. Molina, and F. Herrera, “A practical tutorial on the use
of nonparametric statistical tests as a methodology for comparing evolutionary and
swarm intelligence algorithms,” Swarm Evol. Comput., vol. 1, no. 1, pp. 3–18, 2011. doi:
10.1016/j.swevo.2011.02.002.
[58] J. Alcalá-Fdez et al., “Keel: a software tool to assess evolutionary algorithms for
data mining problems,” Soft Comput., vol. 13, no. 3, pp. 307–318, 2009. doi: 10.1007/
s00500-008-0323-y.
[59] J. L. Elman, “Finding structure in time,” Cognit. Sci., vol. 14, no. 2, pp. 179–211,
1990. doi: 10.1207/s15516709cog1402_1.
[60] C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vector machines,” ACM
Trans. Intell. Syst. Technol., vol. 2, pp. 27:1–27:27, 2011. [Online]. Available: http://www
.csie.ntu.edu.tw/~cjlin/libsvm
[61] J. Zhang and A. C. Sanderson, “Jade: adaptive differential evolution with optional
external archive,” IEEE Trans. Evol. Comput., vol. 13, no. 5, pp. 945–958, 2009. doi:
10.1109/TEVC.2009.2014613.
[62] R. Tanabe and A. S. Fukunaga, “Improving the search performance of shade using
linear population size reduction,” in Proc. 2014 IEEE Congr. Evol. Comput. (CEC), pp.
1658–1665. doi: 10.1109/CEC.2014.6900380.

 Michael Margaliot
Tel Aviv University, Israel
A Self-Adaptive Mutation
Neural Architecture Search
Algorithm Based on Blocks
Yu Xue and Yankang Wang
Nanjing University of Information Science and Technology, CHINA
Jiayu Liang
Tiangong University, CHINA
Adam Slowik
Koszalin University of Technology, POLAND
Abstract—Recently, convolutional neural networks (CNNs)
have achieved great success in the field of artificial intelligence,
including speech recognition, image recognition, and natural
language processing. CNN architecture plays a key role in
CNNs’ performance. Most previous CNN architectures are
hand-crafted, which requires designers to have rich expert
domain knowledge. The trial-and-error process consumes a lot
of time and computing resources. To solve this problem,
Digital Object Identifier 10.1109/MCI.2021.3084435
Date of current version: 15 July 2021
1556-603X/21©2021IEEE
©SHUTTERSTOCK.COM/BETTERVECTOR
researchers proposed the neural architecture search, which
searches CNN architecture automatically, to satisfy dif-
ferent requirements. However, the blindness of the search
strategy causes a ‘loss of experience’ in the early stage of
the search process, and ultimately affects the results of the
later stage. In this paper, we propose a self-adaptive
mutation neural architecture search algorithm based on
ResNet blocks and DenseNet blocks. The self-adaptive
mutation strategy makes the algorithm adaptively adjust
the mutation strategies during the evolution process to
achieve better exploration. In addition, the whole search pro-
cess is fully automatic, and users do not need expert knowl-
edge about CNNs architecture design. In this paper, the
proposed algorithm is compared with 17 state-of-the-art
algorithms, including manually designed CNN and automatic
search algorithms on CIFAR10 and CIFAR100. The results
indicate that the proposed algorithm outperforms the com-
petitors in terms of classification performance and consumes
fewer computing resources.
Corresponding Author: Yu Xue (e-mail: xueyu@nuist.edu.cn).
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 67
D
I. Introduction
eep learning has achieved great success in many
fields [1], such as speech recognition [2], semantic
segmentation [3], [4], image recognition [5], [6], and
natural language processing [7]. With the excellent
performance in these fields, convolutional neural networks
(CNNs) have become one of the most widely used models in
deep learning [8]. In general, conventional CNNs consist of
convolutional layers, pooling layers, and fully-connected layers.
Typical CNNs include AlexNet [9], VGG [10], GoogleNet
[11], ResNet [5], and DenseNet [12]. Although these networks
have achieved huge accuracy improvements in vision tasks, the
design of the CNN architectures is still a difficult task because
of the large number of parameters [13]. These models are all
hand-crafted and they cannot learn the architecture by them-
selves, thus, designers need to have much expert knowledge in
CNN architecture design [14]. Moreover, the design of CNN
architecture is guided by problems; that is, the architectures of
CNNs are decided by different data distributions, and a manu-
ally designed architecture lacks flexibility. The two main fac-
tors that affect the performance of CNNs are architectures
and weights. For weight optimization, the gradient descent
algorithm has been proved to offer a significant advantage [15].
Unfortunately, for the optimization of CNNs’ architectures,
there are no explicit functions to directly calculate the optimal
architecture [16]. To solve the problems above, Google
researchers Zoph et al. [17] proposed the concept of the neural
architecture search (NAS) for deep neural networks. Following
this concept, a large number of researchers have paid attention
to this field, and it has become one of the most popular
research topics in the automatic machine learning community.
The purpose of NAS is to automatically search for parameters
such as the optimal architectures of CNNs so that CNNs can
outperform those that are hand-crafted. Moreover, the NAS
can reduce the high cost of trial-and-error design.
The existing NAS algorithms can be categorized into three
types: 1) reinforcement learning based methods [18]; 2) gradient
based methods [19]; 3) evolutionary computation based [20]
(ENAS) methods. Among these methods, the NAS method
based on reinforcement learning requires a large number of
computing resources, i.e., thousands of graphics processing units
(GPUs) run on a medium-sized dataset such as CIFAR10 for
more than 10 days. Zoph et al. [17] proposed a NAS method
based on reinforcement learning. They used 800 GPUs and
spent 28 days completing the entire search process, which is not
practical for most interested users. The gradient-based method is
more efficient than the reinforcement learning method, and one
of the most famous works is proposed by Liu et al. [19]. Because
of the lack of theoretical support, the architectures searched by
them are inadequate and unstable, and the process of construct-
ing a cell consumes significant computing resources and requires
much prior knowledge. The ENAS algorithms use the evolu-
tionary computation (EC) technique to design the CNN archi-
tecture. In particular, the EC is a population-based technique to
obtain the global optimum. Many EC techniques have been
68 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
proposed, such as genetic algorithm (GA) [21], particle swarm
optimization (PSO) [22], and artificial ant colony algorithm
[23]. Due to the gradient-free characteristic, the EC technique
has been widely used in optimization problems [24] and even in
black box problems that do not have mathematical equations
explicitly expressed. Therefore, EC techniques have been used
to solve NAS problems.
In fact, two decades ago, the EC technique was already used to
optimize neural networks and it was called neuroevolution [25].
The goal is to use the EC technique to optimize the architecture
and weights of neural networks. Neuroevolution is proposed only
for small or medium scale neural networks [16]. With the devel-
opment of neural networks, the CNNs based on deep neural net-
works (DNNs) have been proposed and widely used in computer
vision tasks. However, the CNNs have a large number of parame-
ters.Therefore, to solve the problem of designing CNNs’ architec-
tures, the concept of ENAS was proposed in the evolutionary
computation community. According to references, the first ENAS
work was completed by Google’s Real et al. [26] and they pro-
posed a LargeEvo algorithm. The LargeEvo algorithm uses the
GA to design CNNs’ architectures. However, they only use muta-
tion operators. Experimental results have proven the effectiveness
of LargeEvo on CIFAR10 and CIFAR100 [27]. LargeEvo direct-
ly evolves basic units in open space, then samples and evaluates
each candidate solution, which consumes a lot of computational
resources. Since Real et al. [26] used layers as the search space, this
space contains a relatively simple architecture, LargeEvo initializes
the model with the basic layers, and each individual contains only
a fully connected layer. Xie et al. [28] searched through the same
space, and their results indicated that this trivial search space is
capable of evolving a competitive architecture.Yet, the large search
space brings high computational cost. To overcome this phenom-
enon, some methods have been proposed to constrain the search
space, such as block-based design methods. These methods com-
bine the state-of-the-art architecture in the initial state, so it can
greatly reduce the search space and maintain the performance.
Many block-based methods have been proposed, such as ResNet
blocks [5], DenseNet blocks [12], Inception blocks [11]. Block-
based methods are quite promising because of the fewer parame-
ters which greatly speed up the search process.
Some researchers directly used aforementioned blocks to
design CNN architectures [29], [30], whereas others have pro-
posed different kinds of blocks. For example, Chen et al. [31]
proposed eight kinds of blocks, including ResNet block and
Inception block, and encoded them into 3-bit strings. After that,
they used the Hamming distance to identify blocks that have
similar performance. Song et al. [32] proposed three types of
residual dense blocks, which greatly constrain the search space
and reduce computational consumption. After determining the
search space, different evolution operators were employed to
generate new architectures. Sun et al. [14] used polynomial
mutation on the encoded information which is represented by
real numbers. To make mutations less random, Lorenzo et al.
[33] proposed a new Gaussian mutation guided by Gaussian
regression. Gaussian regression can predict the architectures
which have good performance, and can also search and sample
in a search space where the fitness value may be better.
The studies above do not effectively record the evolution infor-
mation, which leads to their inability to guide the entire search pro-
cess based on past experience. Most of them tend to focus on the
improvement of the search space and evaluation methods, but the
search strategy also affects an algorithm when finding a competitive
architecture. At the same time, the general search strategy loses the
‘experience’ in the entire search process, resulting in the algorithm
being unable to obtain more favorable architectures. This paper
focuses on finding a suitable way to guide the whole search process.
For this purpose, this paper proposes a self-adaptive mutation based
on blocks to design CNN architectures. The self-adaptive mecha-
nism has been widely used in EC, but no one has applied it to the
NAS. In addition, the traditional mutation strategy does not focus
on retaining the evolution information. To choose an appropriate
mutation strategy, the self-adaptive mechanism is added into the
evolutionary process. To prevent the phenomenon of population
degradation and slow convergence, a semi-complete binary com-
petition selection strategy has also been designed and it can suffi-
ciently prevent the promising individuals in the population from
being easily abandoned. Moreover, the ENAS algorithm proposed
in this paper is completely automatic, which means that it does not
require the user to have rich expert knowledge about CNN archi-
tecture design.
The rest of the paper is organized as follows: Section II intro-
duces related work. Section III describes the proposed algorithm in
details. Section IV introduces the designs of the experiment and
Section V analyzes the experimental results. Finally, Section VI pro-
vides our conclusions and directions for future work.
II. Related Work
A. Genetic Algorithm
The GA is an evolutionary algorithm inspired by the natural
selection. Because it has gradient-free characteristics, it can be
used to optimize nonconvex and nondifferentiable problems
[21], [34]. In addition, the GA has the characteristic of being
insensitive to local optima, which enables it to find global opti-
ma. The GA usually uses a series of biologically inspired opera-
tors to solve optimization problems, such as crossover and
mutation. Generally speaking, the main steps of GA are as fol-
lows: 1) initialize a population; 2) evaluate fitness; 3) execute
crossover and mutation operators; 4) evaluate individuals; 5)
environmental selection. Among them, steps 3) - 5) are con-
trolled by a loop, and the stop condition is whether the maxi-
mum generation is reached.
In this paper, each individual represents the architecture of
one CNN, and the fitness is the classification accuracy of each
CNN on the validation dataset. The higher the classification
accuracy, the higher the fitness.
B. Block-Based Design Method
Because a constraint represents human intervention in the
search space, the suitable constraints on the coding space are
very important. A method with a large number of constraints
can easily obtain a good architecture, but it also will be unable
to find a novel architecture. In addition, the size of the search
space greatly affects the efficiency of search techniques. With
the development of the NAS, more researchers use block-based
techniques to define the search space [35], [36], which com-
bines various types of layers as the basic unit. These block-based
techniques have good performance and fewer parameters are
required to build the architecture.
ResNet and DenseNet are two kinds of DNNs with bet-
ter performance in hand-designed CNNs. Their success
depends on the construction of ResNet blocks and
DenseNet blocks, and the block-based method has been
proven to have better effectiveness than the design method
which is composed of layers [32], [37], [38]. This design
method can solve the problem of gradient disappearance
[39]. Therefore, the block-based search space can achieve
promising prospects. In this paper, ResNet blocks and
DenseNet blocks are used as the basic units to determine the
search space. For the convenience of representation, ResNet
blocks and DenseNet blocks are respectively called RBs and
DBs. The units composed of RBs or DBs are called RUs or
DUs, respectively. Figure 1 shows the structure of an RU.
Figure 2 shows the internal architecture of an RB. In this
RB, there are three convolutional layers, which are called
conv1, conv2, and conv3. In conv1, 1 × 1 convolution ker-
nels are used to decrease the number of channels of input
features and the computing complexity. In conv2, the 3 × 3
filters are used to learn the characteristics of the input. In
conv3, the filters with the size of 1 × 1 are used to increase
the number of output channels of the previous convolu-
tional layer, so that the number of input and output chan-
nels is consistent throughout the entire RB. In addition, a
RU
RB RB RB
FIGURE 1 An example of the ResNet Unit.
RB
Conv2
Conv1
Conv3
Output
Input
FIGURE 2 An example of the ResNet block with three convolutional
layers and one skip connection.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 69

The existing NAS algorithms can be categorized
into three types: 1) reinforcement learning based
methods; 2) gradient based methods; 3) evolutionary
computation based (ENAS) methods.
skip-connection is used to connect the input layer and the
output layer, and when the sizes of input and output chan-
nels are different, some 1 × 1 filters are added to adjust the
size of channels so that the size of the input channels is the
same as the size of output channels. Figure 3 shows an example
of a DB. As shown in the figure, three convolutional layers
are used to form a DB (the number of convolutional layers
in each DB can be set by the users). In this DB, the input of
each convolutional layer contains the output of each previ-
ous layer. In addition, to better control the input and output
size of each convolutional layer, a parameter k is used. For
example, if the input size of the first layer is a, then the out-
put size of the first layer is a+k, the input size of the second
layer is a+k, and the output size of the second layer is a+2k.
Through similar operations, some DBs can be constructed.
The two hierarchical representation block-based methods
above can not only reduce the search space, but also enable
deeper convolutional layers to learn features in the shallow
layers. This hierarchical representation of input can also
improve the classification accuracy and make the search pro-
cess more flexible [37].
DB
Conv1
C. Self-Adaptive Mechanism
In recent years, self-adaptive mechanisms have
been increasingly used in the evolutionary
computation community. Moreover, many
EC algorithms based on self-adaptive mecha-
nisms have been proposed [34]. In the early
years of such research, Qin et al. [40] intro-
duced the self-adaptive mechanism into the
differential evolution (DE) algorithm, combining different
types of DE, and proposed a new algorithm called self-adap-
tive DE. The experimental results showed that self-adaptive
DE can obtain higher quality solutions. In recent years, Xue
et al. [24] applied a self-adaptive PSO to solve large-scale fea-
ture selection problems, and experimental results showed its
effectiveness in reducing redundant features. In the field of
neural networks, adaptive mechanisms, such as adaptive gra-
dient algorithm [41], adadelta [42], and adam [43] have been
applied for selecting the learning rate. These algorithms
can adjust the learning rate to meet the needs of different
learning rates in different training stages. To the best of our
knowledge, there is no work that introduces the adaptive
mechanism into the CNN architecture design. Therefore, this
paper proposes a self-adaptive mutation method to search the
CNNs’ architectures so that the algorithm can adaptively
choose appropriate mutation operations, such as adding,
removing, and modifying units.
III. Details of SaMuNet
In this section, the framework of the algorithm is described,
and its main components are discussed in details. A self-adaptive
mutation network (SaMuNet) is proposed in this paper. The
proposed NAS method is mainly used in image classification
tasks. For clarity, the frequently used notations and their
descriptions are shown in Table I.

Conv2

Conv3

Output
Input

A. Framework of SaMuNet
The framework of SaMuNet is shown in Algorithm 1, first, a
population with size N is initialized. Then, each individual in the
initialized population is evaluated. After that, the architecture of
FIGURE 3 An example of the DenseNet block with three convolutional
the CNN is evolved by the crossover and self-adaptive mutation
layers. operators. A new population will be generated by the environ-
ment selection operator. This
process continues until the
TABLE I Notations and Corresponding Descriptions. maximum generation number
is reached. Finally, the best
NOTATIONS DESCRIPTIONS NOTATIONS DESCRIPTIONS
generated individual is decod-
N Population size P Current population
ed into a CNN with the best
nlter Maximal iteration Q Offspring population architecture. In the following
m Crossover probability nsflag Success matrix in a generation parts, encoding strategy, popu-
n Mutation probability nfflag Fail matrix in a generation lation initialization, individual
Dtrain Train datasets S Success matrix in defined generation evaluation, crossover opera-
tors, self-adaptive mutation
Dvalid Validation datasets F Fail matrix in defined generation
mechanism, and the proposed
p1, p2 Two parent individuals COGS Candidate offspring generate strategy
environment selection opera-
q1, q2 Two offspring individuals NS Number of strategies
tor are introduced.

70 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

B. Encoding Strategy
Encoding is very important when solving In this paper, each individual represents the architecture
practical problems, and different problems of one CNN, and the fitness is the classification
require different coding strategies. For the
EC technique, encoding strategies are gener-
accuracy of each CNN on the validation dataset. The
ally divided into two types: direct encoding higher the classification accuracy, the higher the fitness.
and indirect encoding. Direct encoding
directly encodes the precise information into
each individual. Indirect encoding is different from directly use layers as basic units. Inspired by the success of ResNet [5]
representing the weight or connection of the network, and it and DenseNet [12], in this paper, RUs, DUs, and PUs are used as
usually represents each individual through a new generation the basic units. RUs are composed of RBs, DUs are composed
rule [16]. This encoding method is widely used in neuroevolu- of DBs, and PUs are composed of pooling layers. The proposed
tion [44]. Because of the increasing number of network layers algorithm does not optimize the fully-connected layer.The main
in CNNs, the number of parameters that need to be opti- reason is that the neurons will cause overfitting due to the char-
mized has also increased exponentially. The backpropagation acteristics of the fully-connected layers [45]. To alleviate this
algorithm [15] has been proven to be superior in optimizing phenomenon, operations such as dropout [46] need to be used,
weights and it is directly applied to the weight optimization of but this will increase the number of parameters that need to be
the CNN. Thus, it is not necessary to consider the optimiza- optimized and increase the number of computing resources.
tion of the weights of CNN, and only some hyperparameters
need to be optimized.
To make operators such as crossover and mutation easier to Algorithm 1 Framework of SaMuNet.
implement and maintain the flexibility of CNNs’ architectures,
this paper uses a direct variable-length encoding strategy to Input: The population size (N), the number of maximal iteration
represent CNN architecture. This encoding strategy uses basic (nIter), the crossover probability (m), the mutation probability
units RU, DU and PU. For the RU, a random number is used (μ), the current iteration (t).
to represent the amount of RBs, and each RB contains at least O utput: The best individual with the architecture of
one 3 × 3 convolution kernel. Each RB also includes a skip the CNN.
connection operation.   1:  Randomly initialize population P0;
For the DU, a random number is used to represent the number   2:  Evaluate each individual;
of DBs, and each DB also contains at least one 3 × 3 convolu-  3:  while t < nIter do
tion kernel. The difference between DU and RU is that the skip  4:   Qt ! {};
connections in the DB are densely connected (i.e., the input of  5:   while Q t < N do
each layer contains all the previous layer outputs). Meanwhile,   6:    Select two parents p1, p2 from Pt by binary tourna-
each PU includes only one pooling layer of either type: average ment selection;
pooling layer and maximum pooling layer. In summary, RUs,   7:    Generate two offspring q1, q2 from parents p1, p2 by
DUs and PUs are composed of RBs, DBs and pooling layers, crossover and mutation;
respectively. For different RBs, the unknown parameters are the   8:    Qt ! Qt ∪ q1 ∪ q2;
number of input and output channels. For different DBs, in addi-  9:   end while
tion to the number of input and output channels, the unknown 10:   Evaluate the individuals in Qt and update the probabili-
parameters include a growth rate k (to calculate the number of ties of strategies in mutation strategy pool (Adding,
input channels for each layer). To express individual information Removing, Replacing) by using self-adaptive mechanism;
and make the genetic operator more flexible, this article uses the 11:   Utilize the proposed environment selection strategy to
variable-length encoding strategy [15]. As shown in Figure 4, an select N individuals from Pt ∪ Qt and generate the new
individual contains RUs, DUs, and PUs. In the later stage of the population Pt + 1;
evolution process, the block-based method in the figure is used 12:   t = t + 1;
as the basic unit. 13:  end while
14: Output the best individual with the best architecture of
C. Initialization of the Population the CNN;
Population initialization is the first stage in the evolution process,
and it involves a random initialization method that obeys a uni-
form distribution for each individual. As described in Part A,
each individual represents a CNN architecture. To find the best Individual:
individual, GA with self-adaptive mutation is used in the evolu- RU DU PU RU DU PU PU RU DU PU
tion process. The proposed algorithm uses a block-based design
method, which is different from traditional NAS methods that FIGURE 4 The architecture of an individual.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 71

 The method of population initialization is presented in E. Crossover Operator
Algorithm 2. For each individual, we need to randomly gen- The offspring inherit the genes from the parents and are
erate a positive integer k to determine the length of the indi- expected to obtain a higher fitness than the parents. Therefore,
vidual (i.e., the number of units that need to be initialized), individuals with higher fitness values should be selected as par-
and then randomly generate a vector with length k. The vec- ents. In this paper, the binary competition strategy is used to
tor stores each unit that needs to be initialized. Then, the sec- select parents [47]. This strategy first randomly selects two
ond random number is used to determine whether to individuals from the population, and then it chooses the fitter
generate RUs, DUs, or PUs. Note that, in line 7, the number of the two as a parent. The second parent is selected in the
of PUs is limited. This is because the function of the pooling same way.
layer is to halve the size of the input, and this operation is After selecting the parents, the crossover operator which has
widely used in many state-of-the-art CNNs [5], [10]–[12]. local ability is performed. To make each individual have better
However, if an individual starts with the pooling layer, a lot of flexibility, the fixed-length coding strategy is replaced by the
information will be lost. In addition, if a 64 × 64 input image variable-length coding strategy. Because of the different lengths
passes through seven pooling layers continuously, the output of individuals, the traditional crossover operator is no longer
size will be 1 × 1; this operation will lead to logic errors and is directly applicable.Yet, the crossover operator plays a key role in
thus disallowed. the search process; a modified single point crossover operator is
used based on variable-length individuals. The specific opera-
D. Evaluation of Individuals tion is shown in Figure 5.
The fitness value of each individual indicates the individual’s It should be noted that the number of input channels at the
adaptivity to the environment. In the proposed algorithm, the splicing place after the crossover needs to be adjusted, and the
fitness value is the classification accuracy on the image datasets.
In EC, individuals with higher fitness values have a higher
probability of generating offspring with better performance
than themselves. In this paper, to obtain the fitness value, the Algorithm 3 Evaluation of the Individuals.
proposed algorithm decodes each individual into a CNN I nput: The population Pt for fitness evaluation, training dataset
architecture and trains it on the training set, then tests the Dtrain, validation dataset Dvalid, i = 0.
trained architecture on the validation set. In Algorithm 3, each Output: The fitness of the population.
individual is decoded into a CNN architecture and its weights  1:  for each individual in Pt do
are initialized, then the CNN will be trained until the iteration  2:   Decode the encoded information to the architecture of
reaches its maximum value. Finally, the trained CNN model CNN;
searched by the proposed algorithm is put on the validation   3:   Initialize the weights of CNN;
dataset to obtain the fitness value.  4:   while i < epoch do
  5:   Train the CNN;
  6:    i + + ;
Algorithm 2 Initialize Population.  7:   end while
  8:   Evaluate the accuracy for the CNN on the Dvalid;
Input: The population size (N), the maximal PUs poolmax,  9:  end for
j = 0, i = 0.
10:  Output the fitness values of the individuals in Pt;
Output: The initialized population P0.
 1:  P0 ! {};
 2:  while i < N do
  3:   Randomly generate a positive number k to decide the
length of individual;
  4:   Initialize an array with the size of 1# k; Parent 1: RU DU PU RU DU PU PU DU PU
 5:   for j < k do
  6:     U ! Randomly choose one unit from {PU, RU, DU};
  7:     if The last unit of U is not PU and the number of PU
Parent 2: RU DU DU PU RU DU PU RU
is less than poolmax then
  8:     U ! randomly choose one unit from {RU, DU};
  9:     end if
10:     Put the information of U into the jth element of a; Offspring 1: RU DU PU RU RU DU PU RU
11:  end for
12:  P0 ! P0 ∪ a; Offspring 2: RU DU DU RU DU PU PU DU PU
13:   i = i + 1;
14:  end while FIGURE 5 The crossover operator based on variable-length encoding
15:  Output the initialized population P0; strategy.

72 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

unit number also needs to be modified. The pseudo-codes of
the crossover operator are described in Algorithm 4. A number
is randomly generated to determine whether to perform the
crossover operator or not. If it is determined to perform the
crossover operator, the binary competition strategy is used to
select parents firstly, then two crossover points are randomly
generated for the parents. Finally, the offspring are generated by
the crossover operator.
F. Self-Adaptive Mechanism Strategy Based on Blocks
The mutation operator usually explores the search space to
obtain individuals with promising performance. The pro-
posed algorithm uses three types of mutation methods, i.e.,
adding, removing, and replacing. Adding refers to adding a
unit at the selected location of the individual, removing
refers to removing the current unit at the designated location
of the individual, and replacing refers to randomly replacing
a new type of unit with a new one at the selected location of
the individual.
The self-adaptive mechanism has been widely used in the
evolutionary computation community [24], [34]. In this
paper, a self-adaptive mutation strategy pool with three candi-
date offspring generation strategies (COGSs) is proposed. This
mechanism enables the algorithm to adaptively find mutation
operators that are more suitable for the current task. Thus it
can determine a more suitable generation strategy to match
the evolution process at different stages.
The self-adaptive mutation operator mechanism is described
as follows: First, each COGS has an initial probability 1/N s,
where Ns represents the number of COGS in the strategy
pool. Let Pq represent the probability of the qth strategy being
selected (q = 1, 2,…, Ns). Next, the roulette strategy is used to
select a COGS. Through this strategy, a new individual is gen-
erated, and it is compared with its parent. The result is record-
ed in the initial zero matrix nsflag i, s and nfflag i, s (i= 1, 2,…, N,
s = 1, 2,…, Ns) where N represents the number of individuals
in the population. After a generation, the sums of each col-
umn of nsflag i, s and nfflag i, s are calculated, and they are
recorded in two new matrices S k, q and Fk, q (k= 1,…, N g,
q = 1,…, Ns, where Ng represents the updated period). S
records the number of successes of the qth strategy in the kth
generation, and F records the number of failures of qth strate-
gy in the kth generation. After a generation, both nsflag N, Ns
and nsflag N, Ns are reset.
After Ng generations, the total number of successes and fail-
ures of all COGSs is calculated, and the probability of each
COGS is updated as follows:
Ng
S 1q = / S k,q (1)
k=1
S 2q = )
m, if S 1q = 0
1 (2)
S , otherwise
q
where q represents the strategy in the qth strategy pool, and m is
set to 0.0001 to prevent the probability of some strategies from
becoming 0. Next, the generation probability of a new COGS
is calculated using the following formulas:
Ng
S 3q = S 2q e S 2q + / Fk,q o (3)
k=1
Nq
Pq = S 3q / S 3q (4)
q=1
where S 3q represents the new probability of the qth COGS
after the update. In addition, this probability needs to be nor-
malized by Equation (4). The above equations are used to
update the probability of each COGS according to the perfor-
mance of the offspring. Obviously, the better the performance
of a COGS, the higher the probability that it will be selected
during the entire evolutionary process.
In Algorithm 5, the implementation of the self-adaptive
mutation strategy is shown in details. First, three opera-
tions are added to the strategy pool (line 1). Then, a ran-
dom number is generated to determine whether to perform
mutation operations. If a mutation occurs, a block opera-
tion is selected to generate new offspring. The information
of generating offspring is used to update the probability
of the COGS. Finally, the mutated offspring population
is output.
G. Environmental Selection Based on Semi-Complete
Binary Competition
In the environmental selection stage, suitable individuals need to
be selected from the population to serve as the next generation
population. The simplest way to accomplish this is to select the
individual with the highest fitness value from the population
each time, but this will cause the population to lose diversity
and fall into a local optimum [48], [49].
For NAS applications, researchers have proposed many
selection strategies. Real et al. [50] used aging evolution to
Algorithm 4 Crossover operator of SaMuNet.
Input: Two parents p1, p2, the crossover probability m.
Output: Two offspring q1, q2.
  1:  Randomly generate a number v in range of [0,1];
 2:  if v < m then
 3:    Select two parents p1, p2 from Pt by binary tourna-
ment selection;
  4:   Randomly generate two integer numbers to separate p1,
p2 into two parts, respectively;
  5:   Combine the first part of p1 and the second part of p2;
  6:   Combine the first part of p2 and the second part of p1;
 7:  else
 8:   q1 ! p1;
 9:   q2 ! p2;
10:  end if
11:  Output two offspring q1, q2;
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 73
kill the oldest individual from the population, thus avoiding
the premature convergence problem. Zhu et al. [51] com-
bined two methods while removing the oldest and worst
individuals. In addition, Xie et al. [28] used the roulette strat-
egy to assign probabilities to each individual. Individuals with
higher fitness values have a higher probability of survival. Sun
et al. [14], [29] used a binary competition strategy for envi-
ronmental selection.
In the traditional binary competition, two individuals are
randomly selected; the individual with lower fitness value is
eliminated. This means that if two good individuals are
selected, the relatively bad one will be killed. This process
can have a negative effect on the population. Therefore, this
paper proposes an environmental selection operator based on
a semi-complete binary competition. The implementation of
the method is shown as follows: First, a number k is
Algorithm 5 Self-adaptive mutation operator of SaMuNet.
Input: The maximal generation N, number of strategies N s = 3 ,
the mutation probability μ, the number of offspring q, i = 0, cur-
rent iteration (cIter), the number of iterations (Iter), flagIter = 0,
Ng = 5.
Output: The population of mutated offspring.
 1: Put three operation {Adding, Removing, Replacing} into
strategy pool;
 2:  while i < Iter do
 3:   for j < q do
 4:     Randomly generate a number r in range from [0,1];
 5:     if r < μ then
 6:      Choose one strategy with roulette from strategy
pool to generate q new
j ;
 7:     if the new offspring q new j is better than qj then
 8:       nsflag i, s = 1;
 9:     else
10:       nfflag i, s = 1;
11:     end if
12:      j = j + 1;
13:     end if
14:  end for
15:   S N g # N s = sum of each column in nsflag i, s;
16:   FN g # N s = sum of each column in nfflag i, s;
17:  Reset matrices nsflag i, s and nfflag i, s;
18:  if cIter – flagIter = = Ng then
19:    tempS N g # N s = sum of each column in S N g # N s;
20:    tempFN g # N s = sum of each column in FN g # N s;
21:    U p d a t e t h e p r o b a b i l i t y : P1 # N s = (tempS N g # N s) /
(temp S N g # N s + FN g # N sh;
22:   Reset matrices S N g # Nq and FN g # Nq;
23:   flagIter = cIter;
24:  end if
25:   i = i + 1;
26:  end while
27:  Output the population of mutated offspring;
74 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
randomly generated, then k individuals with the highest fit-
ness values are selected from the parent population and the
offspring population. We retain the k individuals and use
the binary competition strategy to select the needed indi-
viduals. In this way, it is ensured that the excellent indi-
viduals in the population will not be eliminated. This
method, which incorporates the ‘elite’ [52] mechanism in
GAs, can effectively prevent the population from degener-
ating during evolution.
Algorithm 6 shows the details of the proposed environ-
mental selection based on the semi-complete binary competi-
tion. First, randomly generate a number k from (0, N/2], select
k best individuals from the parent population and the offspring
population, and then use the binary competition strategy to
select N – k from the remaining individuals. Finally, the new
population is obtained.
IV. Experiment Design
This section verifies the effectiveness of the CNN architecture
searched by our proposed algorithm on image classification
tasks. We first introduce the benchmark datasets (in Part A),
then introduce the comparison algorithms (in Part B), and
finally describe the parameter setting (in Part C).
A. Benchmark Datasets
CIFAR10 and CIFAR100 are color image datasets that are
close to universal objects. For state-of-the-art CNNs,
CIFAR10 and CIFAR100 are the most widely used bench-
mark datasets [27]. Therefore, this paper uses CIFAR10
and CIFAR100 to verify the effectiveness of our pro-
posed algorithm.
CIFAR10 contains a total of 10 classes of RGB color imag-
es: airplane, automobile, bird, cat, deer, dog, frog, horse, ship and
truck. The size of each image is 32 × 32. Each class has 6,000
images. CIFAR10 has a total of 50,000 training images and
Algorithm 6 Environmental selection based on semi-complete
binary competition.
Input: The parent population Pt, the offspring population Qt, the
population size N.
Output: The next population Pt+1.
 1:  Pt + 1 ! { }, U t ! { }, q ! { };
  2:  Randomly generate a number k in range of (0,N/2];
 3:  while j < k do
  4:   p ! Select the k individuals with highest values from
Pt , U t;
 5:   Pt + 1 ! Pt + 1 , P;
 6:  end while
 7:  for j = k + 1 to N do
 8:   P1, P2 ! Randomly selected two individuals from Ut;
 9:   q ! Select the one with higher fitness from Ut;
10:   Pt + 1 ! Pt + 1 , q;
11:  end for
12:  Output the next population Pt+1;
10,000 test images. We show the examples of CIFAR10 in
Figure 6. In contrast, CIFAR100 has 100 classes, each with
500 training images and 100 test images. In the experiment, in
order to match the conventions of state-of-the-art CNNs, each
side of each image is filled with four zeros, and then the image
is randomly cropped to its original size. After that, each side is
horizontally flipped randomly, and the images are input finally
in the proposed algorithm.
B. Comparison Algorithms
To verify the effectiveness of the proposed algorithm, many
state-of-the-art algorithms are used for comparison. Ac­­
cording to the literature [29], the existing state-of-the-art
CNNs are divided into the following three types: The first
type is purely hand-designed CNNs, which require a lot of
exper t knowledge, such as VGG [10], ResNet [5],
DenseNet [12], Network in Network [53], Highway Net-
work [54], Maxout [55], and ALL-CNN [56]. The purpose
of choosing these hand-designed networks is to prove the
superiority of the automatically searched network archi-
tecture. The second type of network is designed with a
semi-automatic architecture, such as Genetic CNN [28],
Hierarchical Evolution [57], and EAS [58]. The third is a
fully-automatically designed network architecture, such as
Large-scale Evolution [26], NAS [17], MetaQNN [59],
AE-CNN [29], Firefly-CNN [60], EPSO-CNN [61], and
NSGANet [62] which is based on a multi-objective NAS
algorithm [63].
C. Parameter Setting
We compare the existing state-of-the-art algorithms by
consulting the results presented in other papers. The reason
for this is that the algorithms in other papers generally use
the best accuracy, which saves a lot of time. In this paper,
because the proposed algorithm is based on GA architec-
ture search [29], the population size is set to 20, the cross-
over probability is set to 0.9, and the three mutation
operation probabilities (adding, removing, and replacing)
are set to 0.4, 0.3, and 0.3, respectively. Following the rec-
ommendation of the machine learning community, the
datasets are randomly divided into one-fifth as validation
datasets. Finally, all classification error rates are obtained on
the same validation datasets.
In this paper, when training the obtained neural network,
Adam is used as the optimizer to learn the weights of the neu-
ral network [43]. The number of epochs is set to 250, the initial
learning rate is set to 0.1, and the batch size is set to 80, the
learning rate is adjusted according to the epoch.
In addition, this paper uses a block-based design method.
Each block and its parameters in the block are set as follows:
The maximum number of RUs, DUs, and PUs is set to three,
and the maximum number of RBs in each RU is set to four.
The selectable parameters in the DB are set to 12, 20, and 40.
The maximum number of convolutional layers is set to five in
the DB. Note that, because the experimental device used in
this paper is one GeForce GTX 2080Ti, and the video mem-
ory is 11 GB, some of the parameters can be adjusted by the
user. Because of the device’s video memory limitation, out of
memory error may occur if the parameter setting is too large.
V. Experimental Results and Analysis
This paper evaluates the performance of the algorithms using
not only the classification accuracy, but also the number of
parameters (i.e., model size) and GPU/Days. According to the
concept of GPU/Days proposed by some previous researchers,
the time consumption in the final search process of the algo-
rithm can be calculated as follows: If n GTX 1080Ti cards are
used to run on the CIFAR10 dataset for m days, the computa-
tional resource consumed by the algorithm on CIFAR10 is
m ) n GPU/Days. In addition, the proposed algorithm uses the
standard in [29] in which Sun et al. divided the current state-
of-the-art CNNs into three categories: hand-crafted, semi-
automatic, and completely automatic.
A. Classification Performance of the
Algorithms on CIFAR10
Table II shows the performance comparison of the proposed
algorithm SaMuNet and the comparison algorithms on
CIFAR10 and CIFAR100. The second and third columns rep-
resent the validation classification error rates of different CNNs
on CIFAR10 and CIFAR100, respectively. The fourth column
represents the sizes of the final models. The fifth column repre-
sents the sizes of GPU/Days consumed by each algorithm, and
the sixth column is the constructive method of each model.
The ‘-’ in the table indicates that this indicator has not been
reported in the relevant reference.
As shown in Table II, compared with the purely hand-craft-
ed model, which includes ResNet and DenseNet, the classifi-
cation accuracy of SaMuNet on CIFAR10 has achieved the
second lowest error rate, which is a little higher than Firefly-
CNN. However, the number of parameters from the final
model which is found by our algorithm is much lower than
Firefly-CNN. The proposed algorithm has an error rate 1.6%
Airplane
Automobile
Horse
FIGURE 6 Three examples from CIFAR10.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 75
lower than the best model (DenseNet, which is hand-crafted
on CIFAR10). Since the hand-crafted model does not have
the process of searching the model, its GPU/Days is represent-
ed by ‘-’. Compared with the semi-automatic architecture
design methods, our proposed algorithm has the lowest classi-
fication error rate on CIFAR10, and consumes less GPU/
Days than Hierarchical Evolution. Compared with the com-
pletely automatic search algorithm, the proposed algorithm
has a classification 1.8% lower than Large-scale Evolution,
2.4% lower than NAS, 3.2% lower than MetaQNN, and 0.7%
lower than AE-CNN. Since our proposed algorithm and AE-
CNN are both block-based constructive methods and both
involve GAs, this also proves the effectiveness of our proposed
adaptive mutation strategy.
It can be seen that our algorithm is less time-consum-
ing than most algorithms, except for EPSOCNN. Because
EPSOCNN uses a multi-server distributed computing meth-
od, it is faster than our algorithm which did not use a weight
sharing strategy.
B. The Classification Performance of the
Algorithms on CIFAR100
As one of the most widely used datasets for machine learning
algorithms, the classification task on CIFAR100 is more diffi-
cult than that on CIFAR10. It can be seen from Table II that
all models have higher classification errors on CIFAR100. The
classification error rate of our proposed algorithm on
CIFAR100 is 20.2%.
Compared with the hand-crafted networks, the error rate of
our proposed algorithm is 4.2% lower than DenseNet which is
the best of the hand-crafted models. Compared to other hand-
crafted models, SaMuNet has remarkable advantages; in partic-
ular, when compared to Maxout, the error rate of SaMuNet is
18.6% lower. In addition, the table shows that the automatically
searched architectures generally have a lower classification error
rate than the manually designed networks. In addition, expert
knowledge is not required when designing architectures
for SaMuNet.
Compared with the semi-automatic and completely auto-
matic architecture design methods, SaMuNet also achieved the
lowest error rate. Among them, the classification error rate of
our proposed algorithm is 0.6% lower than that of AE-CNN.
This also proves the effectiveness of the adaptive mutation strat-
egy proposed in this paper. Compared with the Large-Scale
Evolution and NAS algorithms, SaMuNet is dozens of times
faster. Because NAS is based on reinforcement learning, it usu-
ally consumes more computing resources. In general, SaMuNet
has achieved very promising performance in classification accu-
racy, and the consumption of computing resources is relatively
lower than its comparison algorithms.

TABLE II THE PERFORMANCE OF ALGORITHMS ON CIFAR10 AND CIFAR100.

DATASETS CIFAR10(%) CIFAR100(%) PARAMETERS GPU/DAYS NETWORK CONSTRUCTIONS

VGG [10] 6.66 28.05 20.04M – Hand-crafted
ResNet [5] (depth=1,202) 7.93 27.82 1.7M – Hand-crafted
DenseNet (K=12) [12] 5.24 24.42 1.0M – Hand-crafted
Maxout [55] 9.3 38.6 – – Hand-crafted
Network in Network [53] 8.81 35.68 – – Hand-crafted
Highway Network [54] 7.72 32.39 – – Hand-crafted
ALL-CNN [56] 7.25 33.71 – – Hand-crafted
Genetic CNN [28] 7.1 29.05 – 17 Semi-automatic search
Hierarchical [57] Evolution 3.63 – – 300 Semi-automatic search
EAS [58] 4.23 – 23.4M 10 Semi-automatic search
Large-Scale Evolution [26] 5.4 – 5.4M 2,750 Automatic search
Large-Scale Evolution – 23 40.4M 2,750 Automatic search
NAS [17] 6.01 – 2.5M 22,400 Automatic search
MetaQNN [59] 6.92 27.14 – 100 Automatic search
AE-CNN [29] 4.3 – 2.0M 27 Automatic search
AE-CNN – 20.85 5.4M 36 Automatic search
NSGANet [62] 4.67 – 0.2M 27 Automatic search
NSGANet – 25.17 0.2M 27 Automatic search
Firefly-CNN [60] 3.3 22.3 3.21M – Automatic search
EPSO-CNN [61] 3.69 – 6.77M 4– Automatic search
SaMuNet (ours) 3.6 – 1.5M 23 Automatic search
SaMuNet (ours) – 20.2 4.6M 25 Automatic search

76 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

C. Evolution Process on CIFAR10
To prove the effectiveness of the self-adaptive The simplest way to choose parents is to select the
mutation mechanism and semi-complete bina- individual with the highest fitness value from the
ry competition strategy proposed in this paper,
the classification accuracy of the final searched
population each time, but this will cause the population
individuals is collected, and plot the statistical to lose diversity and fall into a local optimum.
results. In this paper, the population size is 20,
and the total number of generations is 20.
Figure 7 compares the proposed algorithm and GA. To CNN architecture design process, and uses an adaptive muta-
intuitively show how the self-adaptive mutation mechanism tion strategy to make the algorithm adaptively select mutation
guides the evolution process, a tendency line is drawn by con- strategies in different evolution stages, so that the algorithm can
necting the averages over ten points before and after the cur- guide the search process effectively. In addition, a semi-com-
rent point. It is noted that the accuracy of some individuals is plete binary competition strategy is also designed for environ-
0% in Figure 7. Because the variable-length encoding strategy mental selection. This strategy can better retain the elites. It can
is used, the length that the offspring individuals generated by be seen from the experimental results that our proposed algo-
the crossover individuals is too long. Due to the memory limi- rithm achieves better results in comparison with different
tation of our experimental equipment, an out of memory hand-crafted, semi-automatic, and completely automatic struc-
occurs. As shown in Figure 7, for SaMuNet, from the 1st gen- tural design methods in terms of accuracy and consumption of
eration to the 7th generation, the average classification accuracy computing resources. SaMuNet still consumes a lot of comput-
rises faster. From the 12th generation, the average classification ing resources, thus, in the future, our team will be committed
accuracy is stable at about 90%, and the gap between individu- to accelerating the process of evaluating individuals, and reduc-
als narrows, which shows that the algorithm is gradually con- ing consumption of resources.
verging. SaMuNet can achieve about 96% accuracy in the last
generation. For the comparison method GA, due to the lack of
guidance from the self-adaptive mechanism, the individual
1.0
accuracy rises slowly. After the 12th generation, the traditional
GA gradually converges too, but the final convergence accura- 0.8
Fitness (Accuracy)

cy is only about 94%. The analysis proves that the self-adaptive

mechanism has better performance in the later stage, and it can 0.6
also guide the later evolution. Tendency (SaMuNet)
In addition, to prove the effectiveness of the semi-complete 0.4 Tendency (GA)
binary competition strategy, this paper makes a comparison Individual (SaMuNet)
between SaMuNet with semi-complete binary competition 0.2
Individual (GA)
and SaMuNet with the binary competition algorithm. The
0.0
conditions remain unchanged from the previous experiment. It
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
can be seen from Figure 8 that, compared with the binary
competition, the semi-complete binary competition strategy Generations
appears to be more stable throughout the entire evolution pro-
FIGURE 7 Scatter diagram of SaMuNet and GA.
cess. The competition method tends to retain bad individuals.
Especially after the 15th generation, there are some poor indi-
viduals with an accuracy of 80%.
The self-adaptive mutation mechanism proposed in this 1.0
paper can effectively improve the search efficiency of the
algorithm, and guide the later evolution through the experi- 0.8
Fitness (Accuracy)

ence accumulated in the early stage. In addition, the pro-

0.6
posed semi-complete binary competition strategy can
effectively maintain population stability and prevent popula-
0.4 SaMuNet With Semi-Complete
tion degradation.
Binary Competition
0.2 SaMuNet With Binary Competition
VI. Conclusions and Future Work
The goal of this paper is to design a CNN architecture search 0.0
algorithm by using SaMuNet with an adaptive mutation strate-
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

gy and a semi-complete binary competition strategy. This goal Generations

is successfully achieved on CIFAR10 and CIFAR100. This
paper uses a block-based design method to accelerate the entire FIGURE 8 Scatter diagram of semi-completely binary competition.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 77

VII. Acknowledgments
This work was partially supported by the National Natural Sci-
ence Foundation of China (61876089, 61876185, 61902281), the
Engineering Research Center of Digital Forensics, Ministry of
Education, the Priority Academic Program Development of Jiang-
su Higher Education Institutions, and the Postgraduate Research &
Practice Innovation Program of Jiangsu Province (KYCX21_1019).
References
[1] I. Al Ridhawi, S. Otoum, M. Aloqaily, and A. Boukerche, “Generalizing AI: Chal-
lenges and opportunities for plug and play AI solutions,” IEEE Netw., 2020. doi: 10.1109/
MNET.011.2000371.
[2] G. Hinton et al., “Deep neural networks for acoustic modeling in speech recognition:
The shared views of four research groups,” IEEE Signal Process. Mag., vol. 29, no. 6, pp.
82–97, 2012.
[3] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic
segmentation,” in Proc. IEEE Conf. Comput. Vision Pattern Recogn., 2015, pp. 3431–3440.
[4] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab:
Semantic image segmentation with deep convolutional nets, atrous convolution, and fully
connected CRFs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834–848,
2017. doi: 10.1109/TPAMI.2017.2699184.
[5] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,”
in Proc. IEEE Conf. Comput. Vision Pattern Recogn., 2016, pp. 770–778.
[6] X. Yao and Md M. Islam, “Evolving artificial neural network ensembles,” IEEE Com-
put. Intelli. Mag., vol. 3, no. 1, pp. 31–42, 2008. doi: 10.1109/MCI.2007.913386.
[7] R. Collobert and J. Weston, “A unified architecture for natural language processing:
Deep neural networks with multitask learning. in Proc. 25th Int. Conf. Machine Learning,
2008, pp. 160–167.
[8] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image cap-
tion generator,” in Proc. IEEE Conf. Comput. Vision Pattern Recogn., 2015, pp. 3156–3164.
[9] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep
convolutional neural networks,” in Proc. Adv. Neural Information Process. Syst., 2012, pp.
1097–1105.
[10] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale
image recognition,” 2014, arXiv:1409.1556.
[11] C. Szegedy et al., “Going deeper with convolutions,” in Proc. IEEE Conf. Comput.
Vision and Pattern Recogn., 2015, pp. 1–9.
[12] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected
convolutional networks,” in Proc. IEEE Conf. Comput. Vision and Pattern Recogn., 2017,
pp. 4700–4708.
[13] M. Suganuma, S. Shirakawa, and T. Nagao, “A genetic programming approach to de-
signing convolutional neural network architectures,” in Proc. Genetic Evolutionary Comput.
Conf., 2017, pp. 497–504. doi: 10.1145/3071178.3071229.
[14] Y. Sun, B. Xue, M. Zhang, and G. G Yen, “Evolving deep convolutional neural
networks for image classification,” IEEE Trans. Evolutionary Comput., vol. 24, no. 2, pp.
394–407, 2019. doi: 10.1109/TEVC.2019.2916183.
[15] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representa-
tions by back-propagating errors,” Nature, vol. 323, no. 6088, pp. 533–536, 1986. doi:
10.1038/323533a0.
[16] Y. Liu, Y. Sun, B. Xue, M. Zhang, and G. Yen, “A survey on evolutionary neural
architecture search,” 2020, arXiv:2008.10937.
[17] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,”
2016, arXiv:1611.01578.
[18] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A sur-
vey,” J. Artif. Intell. Res., vol. 4, pp. 237–285, 1996.
[19] H. Liu, K. Simonyan, and Y. Yang, “Darts: Differentiable architecture search,” 2018,
arXiv:1806.09055
[20] T. Bäck, D. B. Fogel, and Z. Michalewicz, “Handbook of evolutionary computa-
tion,” Release, vol. 97, no. 1, p. B1, 1997.
[21] K. Deb, A. Pratap, S. Agarwal, and T. A. M. T. Meyarivan, “A fast and elitist multi-
objective genetic algorithm: NSGA-II,” IEEE Trans. Evolutionary Comput., vol. 6, no. 2,
pp. 182–197, 2002.
[22] J. Kennedy and R. Eberhart, “Particle swarm optimization,” in Proc. ICNN’95—Int.
Conf. Neural Netw., vol. 4, pp. 1942–1948, 1995.
[23] M. Dorigo, M. Birattari, and T. Stutzle, “Ant colony optimization,” IEEE Comput.
Intell. Mag., vol. 1, no. 4, pp. 28–39, 2006. doi: 10.1109/MCI.2006.329691.
[24] Y. Xue, B. Xue, and M. Zhang, “Self-adaptive particle swarm optimization for large-
scale feature selection in classification,” ACM Trans. Knowl. Discovery Data (TKDD), vol.
13, no. 5, pp. 1–27, 2019. doi: 10.1145/3340848.
[25] D. Floreano, P. Dürr, and C. Mattiussi, “Neuroevolution: From architectures to
learning,” Evolution. Intell., vol. 1, no. 1, pp. 47–62, 2008.
[26] E. Real et al., “Large-scale evolution of image classifiers,” 2017, arXiv:1703.01041.
[27] A. Krizhevsky et al., “Learning multiple layers of features from tiny images,” 2009.
[28] L. Xie and A. Yuille, “Genetic CNN,” in Proc. IEEE Int. Conf. Comput. Vision, 2017,
pp. 1379–1388.
[29] Y. Sun, B. Xue, M. Zhang, and G. G Yen, “Completely automated CNN architec-
ture design based on blocks,” IEEE Trans. Neural Netw. Learning Syst., vol. 31, no. 4, pp.
1242–1254, 2019. doi: 10.1109/TNNLS.2019.2919608.
78 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
[30] L. Frachon, W. Pang, and G. M Coghill, “Immunecs: Neural committee search by an
artificial immune system,” 2019, arXiv:1911.07729.
[31] Z. Chen, Y. Zhou, and Z. Huang, “Auto-creation of effective neural network archi-
tecture by evolutionary algorithm and ResNet for image classification,” in Proc. IEEE Int.
Conf. Systems, Man Cybernetics (SMC), 2019, pp. 3895–3900. doi: 10.1109/SMC.2019.8914267.
[32] D. Song, C. Xu, X. Jia, Y. Chen, C. Xu, and Y. Wang, “Efficient residual dense
block search for image super-resolution,” in Assoc. the Adv. Artif. Intell., 2020, pp. 12,007–
12,014. doi: 10.1609/aaai.v34i07.6877.
[33] P. R. Lorenzo and J. Nalepa, “Memetic evolution of deep neural networks,” in Proc.
Genetic Evolutionary Comput. Conf., 2018, pp. 505–512.
[34] Y. Xue, J. Jiang, B. Zhao, and T. Ma, “A self-adaptive artificial bee colony algorithm
based on global best for global optimization,” Soft Comput., vol. 22, no. 9, pp. 2935–2952,
2018. doi: 10.1007/s00500-017-2547-1.
[35] M. Suganuma, M. Kobayashi, S. Shirakawa, and T. Nagao, “Evolution of deep con-
volutional neural networks using Cartesian genetic programming,” Evolut. Comput., vol.
28, no. 1, pp. 141–163, 2020. doi: 10.1162/evco_a_00253.
[36] K. Chen and W. Pang, “Immunetnas: An immune-network approach for searching
convolutional neural network architectures,” 2020, arXiv:2002.12704.
[37] A. Emin Orhan and X. Pitkow, “Skip connections eliminate singularities,” 2017,
arXiv:1701.09175.
[38] T. Elsken, J.-H. Metzen, and F. Hutter, “Simple and efficient architecture search for
convolutional neural networks,” 2017, arXiv:1711.04528.
[39] Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradi-
ent descent is difficult,” IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157–166, 1994. doi:
10.1109/72.279181.
[40] A. Kai Qin, V. L. Huang, and P. N. Suganthan, “Differential evolution algorithm
with strategy adaptation for global numerical optimization,” IEEE Trans. Evolut. Comput.,
vol. 13, no. 2, pp. 398–417, 2008. doi: 10.1109/TEVC.2008.927706.
[41] M. D. Zeiler, “ADADELTA: an adaptive learning rate method,” 2012, arXiv:1212.5701.
[42] J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning
and stochastic optimization,” J. Machine Learning Res., vol. 12, no. 7, pp. 2121–2159, 2011.
[43] D. P. Kingma and J. Ba, “ADAM: A method for stochastic optimization,” 2014,
arXiv:1412.6980.
[44] K. O. Stanley, J. Clune, J. Lehman, and R. Miikkulainen, “Designing neural net-
works through neuroevolution,” Nat. Mach. Intell., vol. 1, no. 1, pp. 24–35, 2019. doi:
10.1038/s42256-018-0006-z.
[45] G. C. Cawley and N. L. C. Talbot, “On over-fitting in model selection and sub-
sequent selection bias in performance evaluation,” J. Machine Learning Res., vol. 11, pp.
2079–2107, 2010.
[46] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Drop-
out: A simple way to prevent neural networks from overfitting,” J. Machine Learning Res.,
vol. 15, no. 1, pp. 1929–1958, 2014.
[47] B. L. Miller et al., “Genetic algorithms, tournament selection, and the effects of
noise,” Complex Syst., vol. 9, no. 3, pp. 193–212, 1995.
[48] Y. Leung, Y. Gao, and Z.-B. Xu, “Degree of population diversity—A perspective on
premature convergence in genetic algorithms and its Markov chain analysis,” IEEE Trans.
Neural Netw., vol. 8, no. 5, pp. 1165–1176, 1997. doi: 10.1109/72.623217.
[49] L. Davis, “Handbook of genetic algorithms,” 1991.
[50] E. Real, A. Aggarwal, Y. Huang, and Q. V Le, “Regularized evolution for image
classifier architecture search,” in Proc. AAAI Conf. Artif. Intell., vol. 33, pp. 4780–4789,
2019. doi: 10.1609/aaai.v33i01.33014780.
[51] H. Zhu, Z. An, C. Yang, K. Xu, E. Zhao, and Y. Xu, “EENA: Efficient evolution of
neural architecture,” in Proc. IEEE Int. Conf. Comput. Vision Workshops, 2019, pp. 1–13.
[52] J. A. Vasconcelos, J. A. Ramirez, R. H. C. Takahashi, and R. R. Saldanha, “Im-
provements in genetic algorithms,” IEEE Trans. Magn., vol. 37, no. 5, pp. 3414–3417,
2001. doi: 10.1109/20.952626.
[53] M. Lin, Q. Chen, and S. Yan, “Network in network,”, 2013, arXiv:1312.4400.
[54] R. K. Srivastava, K. Greff, and J. Schmidhuber, “Highway networks,” 2015, arX-
iv:1505.00387.
[55] I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, “Maxout
networks,” in Proc. Int. Conf. Machine Learning, 2013, pp. 1319–1327.
[56] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for sim-
plicity: The all convolutional net,” 2014, arXiv:1412.6806.
[57] H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu, “Hierarchical
representations for efficient architecture search,” 2017. arXiv:1711.00436.
[58] H. Cai, T. Chen, W. Zhang, Y. Yu, and J. Wang, “Efficient architecture search by
network transformation,” 2017, arXiv:1707.04873.
[59] B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing neural network architec-
tures using reinforcement learning,” 2016, arXiv:1611.02167.
[60] A. I. Sharaf and E.-S. F. Radwan, “An automated approach for developing a convo-
lutional neural network using a modified firef ly algorithm for image classification,” in
Applications of Firefly Algorithm and Its Variants. Springer-Verlag, 2020, pp. 99–118.
[61] B. Wang, B. Xue, and M. Zhang, “Particle swarm optimisation for evolving deep neural
networks for image classification by evolving and stacking transferable blocks,” in Proc. IEEE
Congr. Evolut. Comput. (CEC), 2020, pp. 1–8. doi: 10.1109/CEC48606.2020.9185541.
[62] Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf,
“NSGA-NET: Neural architecture search using multi-objective genetic algorithm,” in
Proc. Genetic Evolut. Comput. Conf., 2019, pp. 419–427.
[63] Y. Xue, Y. Tang, X. Xu, J. Liang, and F. Neri, “Multi-objective feature selection with
missing data in classification,” IEEE Trans. Emerg. Topics Comput. Intell. doi: 10.1109/
TETCI.2021.3074147.

 Bo Peng
North China Electric Power University, CHINA
Ying Bi, Bing Xue, Mengjie Zhang
Victoria University of Wellington, NEW ZEALAND
Shuting Wan
North China Electric Power University, CHINA
Multi-View Feature Construction Using Genetic Programming
for Rolling Bearing Fault Diagnosis
Abstract
R olling bearing fault diagno-
sis is an important task in
mechanical engineering.
Existing methods have several limi-
tations, such as requiring domain
knowledge and a large number of
training samples. To address these
limitations, this paper proposes a
new diagnosis approach, i.e., multi-
view feature construction based on
genetic programming with the idea
of ensemble learning (MFCGPE),
to automatically construct high-level fea-
tures from multiple views and build an
effective ensemble for identifying different
fault types using a small number of train-
ing samples.The MFCGPE approach uses
a new program structure to automatically
construct a flexible number of features
from every single view. A new fitness
function based on accuracy and distance is
developed in MFCGPE to improve the
discriminability of the constructed fea-
tures. To further improve the generaliza-
tion performance, an ensemble of classifiers
based on k-nearest neighbor is created by
using the constructed features from every
single view. Three bearing datasets and 19
competitive methods are used to validate
the effectiveness of the new approach.The
Digital Object Identifier 10.1109/MCI.2021.3084495
Date of current version: 15 July 2021
1556-603X/21©2021IEEE AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE
©SHUTTERSTOCK.COM/SERGEY TARASOV
results show that MFCGPE achieves
higher diagnostic accuracy than all the
compared methods on the three datasets
with a small number of training samples.
I. Introduction
Rolling bearings are important support-
ing equipment and have been used in all
kinds of rotating machinery, such as elec-
tric motors, turbine generators, and high-
speed trains [1]. The performance of
rolling bearings can be affected by the
high temperature, high pressure, and alter-
nating load during the equipment opera-
tion [2]. The rolling bearings are often
inevitably damaged as the running time
increases [2]. The damage of rolling bear-
ings will lead to mechanical equipment
Corresponding Author: Ying Bi (e-mail: ying.bi@ecs
.vuw.ac.nz).
Application
Notes
failures and the percentage of failures
caused by rolling bearing damage
reaches nearly 30% [3]. Therefore, it
is important to identify the faults of
rolling bearing to monitor bearing
status, ensure machinery safety, reduce
economic losses, and avoid casualties.
In recent years, many methods
have been developed for rolling
bearing fault diagnosis. Because the
vibration signals are easy to collect
and often contain information of
the running states, the studies on
using vibration signals for fault diagnosis
have gained much attention. The col-
lected vibration signals often have back-
ground noise or information loss. A
variety of signal processing methods, such
as wavelet transform [4], empirical mode
decomposition [5], fast spectral kurtosis
[6], maximum correlated kurtosis decon-
volution [7], and variational mode decom-
position [8], have been used to suppress
the interference of noise and harmonics
and strengthen signal characteristics.
Experts have conducted the spectrum
analysis on the processed signals and rec-
ognized the fault characteristic frequency
for fault diagnosis [9]. However, these
methods do not provide satisfactory results
and require extensive domain expertise,
which is time-consuming and expensive.
To automatically identify faults,
methods based on machine learning
79
have been developed for rolling bearing
feature analysis and fault diagnosis. These
methods extract various types of features
from the vibration signals and use tradi-
tional classification algorithms to per-
form fault diagnosis [10]. The commonly
used features include simple statistical
features of time-domain and frequency-
domain, and the non-linear evaluation
indicators, such as fractal dimension [11],
Lyapunov exponent [12], and entropy-
based features [13]–[15]. Typically, a large
number of features are extracted to
describe the vibration signal and these
features may contain redundant or irrele-
vant features, which may reduce the fault
diagnosis accuracy. Therefore, feature
selection methods, such as max-relevance
and min-redundancy [16], ReliefF [17],
Laplacian score [18], principal compo-
nent analysis [19], local discriminant
analysis [20], and margin fisher analysis
[21], have been employed to select a subset
of important features for effective fault
diagnosis. To further improve the fault
diagnosis performance, different classifi-
cation methods have been explored for
different tasks, including k-nearest neigh-
bor (KNN) [22], artificial neural net-
work (ANN) [23], extreme learning
machine (ELM) [24], and support vector
machine (SVM) [25]. However, these
methods typically contain several steps of
signal processing, feature design, feature
selection, and classifier learning. These
steps need to be appropriately connected
to achieve accurate fault diagnosis. In
addition, most of these steps require
domain expertise, which causes the diag-
nosis method to be effective only on a
specific fault diagnosis task and unable to
be generalized to other even similar tasks.
Deep learning is an advanced machine
learning approach that has been applied
to fault diagnosis [26], [27]. Most deep
learning methods are based on neural net-
works (NNs). Different types of neural
networks, such as deep belief networks,
sparse autoencoders, and convolutional
neural networks, have also been devel-
oped for effective fault diagnosis [28]–
[30]. These methods can automatically
learn features from the vibration signals
and train classifiers for effective fault diag-
nosis. However, the NN-based methods
80 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
often use a large number of training sam-
ples to build models/classifiers for fault
diagnosis. In real-world scenarios, the
training samples are often difficult to
obtain and require extensive manual effort
to label. Furthermore, designing an effec-
tive architecture for the NN model typi-
cally requires rich expertise in the
problem and NN domains. Therefore, it is
necessary to develop a new intelligent
fault diagnosis method that can achieve
good performance using a small number
of training samples.
Typically, the feature quality is impor-
tant for effective rolling bearing fault
diagnosis. Traditional methods use many
ways, such as signal processing, feature
selection, and feature learning, to improve
the quality of the extracted features [15],
[21], [25]. Feature construction is an
effective way to generate new informa-
tive and high-level features from the
original low-level features [31]. As shown
in existing work [11]–[15], the features
of vibration signals can be extracted from
multiple views and each view represents
different characteristics. However, feature
construction, particularly constructing
features from multiple views, which can
create informative features to improve
the fault diagnosis, has not been exten-
sively explored in this field.
Evolutionary computation (EC) meth-
ods ordinarily do not use extensive
domain knowledge to find solutions and
have been successfully applied to many
difficult problems [32]–[34]. Genetic pro-
gramming (GP) is an EC technique that
has been widely used as one of the most
popular feature construction methods [31],
[35]. Unlike other EC methods, which use
a fixed-length representation, GP has a
variable-length tree-based representation,
enabling it to automatically construct
high-level features from low-level features
in a more flexible way [35]. For feature
construction, GP typically evolves models
that consist of a set of operators (such as +,
− and ×) and the original features. The
original features are operated by these
operators and new features are then gener-
ated. The tree-based solutions of GP
potentially have high interpretability to
provide insights into what features are
important for construction and why the
constructed features are effective. Owing
to these advantages, many GP methods
have been developed to feature construc-
tion in various problems and achieved
promising results [36]–[38].
However, existing GP methods need
several improvements to construct effec-
tive features for fault diagnosis using a
small number of training samples. First,
the features extracted from the vibration
signals can be multiple views, such as
time-domain view and the frequency-
domain view [39]. Each view represents
different characteristics of the data.
Existing methods often simply concate-
nate all the features from different views,
which may not be effective. GP has sel-
dom been developed for constructing
high-level features from different views
(i.e., multi-view feature construction).
Second, existing fitness measures of GP
for feature construction often use classi-
fication accuracy, which may cause the
overfitting issue, particularly when the
number of training samples is small.
Third, the generalization performance
of the features constructed by GP can
be further improved by constructing
ensembles of classifiers for classification.
However, this has seldom been explored.
Therefore, this paper develops a new GP
approach to address these limitations.
The goal of this paper is to propose a
new intelligent approach, i.e., multi-
view feature construction based on GP
with the idea of ensemble learning
(hereafter called MFCGPE), to rolling
bearing fault diagnosis using a small
number of training samples. The pro-
posed approach is able to construct a
flexible number of high-level features
from every single view and build an
effective ensemble using the constructed
multi-view features to achieve high gen-
eralization performance when the num-
ber of training samples is small. The
performance of MFCGPE will be tested
on three datasets of varying difficulty
and compared with 19 competitive
methods. Further analysis will be con-
ducted to show the effectiveness of the
constructed features and the ensemble.
The main contr ibutions of this
paper are summarized in the follow-
ing four aspects.
 1) A new program structure is devel-
oped in MFCGPE to allow it to
construct high-level features from
multiple views. More importantly,
the number of constructed features
can be adaptively determined with-
out being pre-defined.
2) A new fitness function based on
accuracy and distance is proposed in
MFCGPE to enable the constructed
features to be accurate and discrimi-
native when the number of training
samples is small. The new fitness
function is able to maximize the
classification performance and the
inter-class distances of training sam-
ples, and minimize the intra-class
distances of training samples.
3) An ensemble is created using the fea-
tures constructed by MFCGPE from
multiple views and using KNN to
make predictions for unseen test sam-
ples. Using ensemble for classification
can further improve the generalization
performance, particularly when the
number of training samples is small.
4) A new intelligent diagnosis approach,
namely MFCGPE, is developed to
achieve effective fault diagnosis of roll-
ing bearings with the use of a small
number of training samples. MFCGPE
can achieve better results than 19 com-
petitive methods on three datasets. Spe-
cifically, MFCGPE can achieve a
maximal and average accuracy of 100%
and above 99% on three fault datasets
with only five training samples per class.
The rest of this paper is organized as
follows: Section II briefly introduces the
GP algorithm and reviews its applications
on feature construction and fault diagno-
sis. Section III describes the MFCGPE
approach in detail. Section IV designs the
experiments. The experimental results are
analyzed and compared in Section V. Sec-
tion VI further analyses the proposed
approach. Section VII presents conclu-
sions and future work.
II. Background and Related Work
A. Genetic Programming (GP)
Unlike other EC techniques such as
genetic algorithms (GAs) that have a
fixed-length representation, GP has a
variable-length representation [40]. An
individual of GP is typically represented
using a tree-based structure, as shown in
Figure 1. This example tree/program
consists of internal nodes (the functions
or operators selected from the function
set) and leaf nodes (the arguments or
constants selected from the terminal set).
This example tree can be mathematical-
ly expressed as 5 ' (y + 8) + (2 # x),
where +, ×, and ' (protected division,
return 0 if the divisor is 0) are the inter-
nal nodes, and x, y, 2, 5, and 8 are the
leaf nodes. This equation can also be
treated as a newly constructed feature,
where x and y are two original features.
GP can automatically select functions
from the function set and terminals from
the terminal set to evolve the best trees
for solving a task via an evolutionary pro-
cess. Figure 2 shows the overall evolu-
tionary process of GP. First, a population
of computer programs are randomly ini-
tialized in the search space. Then, each
individual (program) in the population is
evaluated using a fitness function and
assigned a fitness value. At each genera-
tion, a new population of individuals are
Start
Initial Population
Fitness Function
No
Terminate
Yes
Best Individual
End
FIGURE 2 Evolutionary process of GP.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE
generated using genetic operators, i.e.,
Elitism, Crossover, and Mutation operators.
The Elitism operation is to copy the best
individuals from the current generation
to the next generation. A number of
individuals are selected based on their
fitness values via Tournament selection to
be used as parents to generate new
individuals using Crossover and Muta-
tion operators. The Crossover operation
is to exchange the subtrees of two
parents to generate new offspring.
The Mutation operation is to random-
ly delete a subtree of the parent and
grow a new subtree from that node. The
new population of individuals are evalu-
ated and evolved generation by genera-
tion. When a termination criterion is
+
÷ ×
5 + 2 x
y 8
FIGURE 1 An example program of GP.
Generate New Population
Crossover
Mutation Elitism
81
satisfied, the evolutionary process stops,
and the best individual is obtained.
B. GP for Feature Construction
In recent years, many GP based feature
construction methods have been pro-
posed, in which the arithmetic and logi-
cal operators are usually used as the
function set, and the low-level features
are usually used as the terminal set.
Otero et al. [41] used information gain
ratio as the fitness function of GP to
construct features. Muharram et al. [42]
comprehensively compared four differ-
ent fitness functions based on informa-
tion gain in GP for feature construction.
Guo et al. [43], [44] used the Fisher cri-
teria and its improved version as the fit-
ness function of GP. Neshatian et al.
[45] proposed a multiple feature con-
struction method for symbolic learning
classifiers, where the constructed fea-
tures are evaluated by a fitness function
that maximizes the purity of the class
interval. The above methods belong to
the filter-based feature construction
methods. Guo et al. [36] proposed a
method based on GP and KNN to clas-
sify EEG signals, which achieved a clas-
sification accuracy of 99% on one
dataset. Bi et al. [37] proposed a GP
method combined with image-related
operators and SVM for image classifica-
tion, which obtained better accuracy
than the deep learning methods on
some datasets. Aslam et al. [38] com-
bined GP and KNN for automatic
modulation classification, and the meth-
od achieved better classification perfor-
mance. The above methods belong to
the wrapper-based feature construction
methods. Tran et al. [46] developed a fit-
ness function using the classification
accuracy and a distance measure to
improve the performance of the con-
structed features on high-dimensional
data classification tasks and discussed the
impact of the number of constructed
features on classification accuracy. Ma et
al. [47] designed a hybrid fitness func-
tion that combined information gain
ratio and the error rate of a classification
algorithm, and proposed a feature con-
struction strategy that obtained multiple
high-level features using a single GP.
82 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
C. GP for Fault Diagnosis
To the best of our knowledge, GP has
rarely been applied to fault diagnosis of
mechanical equipment. In [48], GP was
used as a binary classifier for fault diag-
nosis of rolling bearing. The classifica-
tion performance of GP with the use of
statistical features, spectral features, and
the combination of statistical and spec-
tral features are compared. The results
showed that the combined features
achieved better fault classification accu-
racy. Guo et al. [49] proposed a GP-
based rolling bearing fault diagnosis
method, in which a fitness function
based on Fisher criteria was developed
to enable GP to construct the high
order of moments of vibration signals as
informative features. This method used
two classification algorithms for multi-
class fault classification. Xuan et al. [50]
proposed a gear fault diagnosis method
that combines GP and SVM, where a
distance measure based fitness function
was developed and the power spectral
features of vibration signals were used to
construct high-level features. These two
methods only construct a single feature
for fault diagnosis, which may not be
effective when the machine working
conditions become more complicated.
In summary, although these methods
successfully show that GP offers possi-
bilities for dealing with fault diagnosis,
there are still some problems that need
to be addressed. The GP based feature
construction methods in [45]–[47] have
discussed the effect of the number of
constructed features on classification
performance. However, these methods
set the number of constructed features
according to prior knowledge and mul-
tiple trials, therefore the adaptability of
them is poor. The GP based fault diag-
nosis method in [48]–[50] only used the
features of a single view for feature con-
struction. However, since the features of
different views have both internal rela-
tions and interval differences, using only
single view features may ignore the
characteristics of the samples. Moreover,
these GP based fault diagnosis methods
use sufficient training data to construct
features and perform classification and
do not consider the scenario of a small
number of training samples. In general, a
small number of training samples can
not well represent the class distribution
information comprehensively and often
leads to poor generalization perfor-
mance. To address the above issues, this
paper proposes a new GP based fault
diagnosis approach (i.e., MFCGPE) to
adaptively construct a flexible number
of informative features from multiple
views for representing sample compre-
hensively, and create an ensemble using
these constructed features for effective
fault diagnosis with the use of a small
number of training samples.
III. The Proposed Approach
In this section, the details of the MFC-
GPE fault diagnosis approach will be
introduced, including the algorithm
overview, the program structure, the
function set, the terminal set, the fitness
function, and ensemble construction for
fault diagnosis.
A. Overview of MFCGPE
Figure 3 shows the overall structure of
MFCGPE to rolling bearing fault diag-
nosis with a small number of training
samples. First, the collected vibration
signals of rolling bearings are trans-
formed into three different-view fea-
tures by calculating the statistical values
of time waveform and frequency spec-
trum. The 16 time-domain features
(TDF), the 13 frequency-domain fea-
tures (FDF), and the combination of
TDF and FDF (named TFDF features)
are represented by View1, View2, and
View3, respectively. Second, the trans-
formed data set is divided into the train-
ing set and the test set, and three
independent GPs are utilized to con-
struct high-level features of each single-
view feature set, respectively. The
program structure and the function set
used for feature construction from dif-
ferent views are the same, but the termi-
nal set is different. Only the training set
is used for the evolutionary process of
GP to construct high-level features that
are expected to have a small intra-class
distance and a large inter-class distance
for effective fault diagnosis. Third, the
constructed features are provided as the
inputs of classification algorithms for
fault diagnosis. We use the features con-
structed from each view to feed three
classifiers/KNNs individually, which are
combined via majority voting to form
an ensemble to make a good prediction
for the unseen test set.
B. Program Structure
To enable MFCGPE to construct a flex-
ible number of high-level features, a
new program structure with the input,
feature construction, feature combina-
tion, and output layers are developed
based on strongly-typed GP (STGP)
[51]. These layers are connected in a
bottom-up manner. Each layer has a
outputs/features can be further used for
feature construction. To achieve feature
combination, two new operators (FC 2
and FC m) are designed to combine
multiple features into a vector, where
the inputs of FC 2 are two features, and
the inputs of FC m can be one feature,
one vector, or two vectors. The use of
these feature combination operators
enables MFCGPE to adaptively con-
struct multiple high-level features
Vibration Signal
S1
16 View1 Features
S2
without presetting the number of con-
structed features.
D. Terminal Set
The terminal set of MFCGPE is com-
posed of the low-level features extract-
ed from the vibration signals. The
terminal sets for feature construction
from different views are different. For
View1 (only consider the time-domain
characteristics of rolling bearing signals),
Multi-View Feature Set
T1×16 F1×13 TF1×29

...

...

...
specific function and type constraint. ...
13 View2 Features Tp×16 Fp×13 TFp×29
The input layer takes the original fea- Sp

...

...

...
tures as inputs, which are the terminals
...

Sq 29 View3 Features Tq ×16 Fq ×13 TFq ×29

of GP. The feature construction layer
transforms the original features into
Step 1: Original Feature Extraction
high-level features. The feature combi-
nation layer combines multiple con-
structed features into a vector to Train Set Test Set
comprehensively describe the sample.
S1 ... ... ... S1 ... ... ...
The output layer returns the construct-
ed feature vector as outputs for fault
...

...

...

...

...

...

...

...
diagnosis. The tree depths of the input Sm ... ... ... Sn ... ... ...
and output layers are one, and those of
the feature construction and feature
Evolutionary Learning
combination layers are automatically
adjusted according to the given task. GP1 GP2 GP3
Figure 4 shows the program structure
and an example program of MFCGPE,
respectively. This example program rep- Best Model
resents a feature vector containing three
constructed features. The detailed
description of the operators and the ter-
minals will be described in the follow-
Step 2: Multi-View Feature Construction
ing subsections.

C. Function Set Training Set and Test Set With the Constructed Features
The function set of MFCGPE is com-
posed of two types of operators, i.e., the
Classifier 1 Classifier 2 Classifier 3
feature construction operators and the
feature combination operators. Table I
lists the detailed information of the Majority Voting
function set. Four arithmetic operators
including +, −, ×, and ' are for feature
construction, where ' is protected by Final Diagnosis
returning 0 if the divisor is 0. The inputs Step 3: Ensemble Diagnosis
of +, −, ×, and ' operators are two fea-
tures, and the output of them is a new
feature. It should be noted that their FIGURE 3 Overall structure of the MFCGPE fault diagnosis approach.

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 83

the terminal set contains 16 statistical where s(i) represents the time signal, variance, maximum value, minimum
features T1 + T16, which are extracted and N represents the number of the sig- value, peak-to-peak value, waveform
through performing the simple mathe- nal data points. T1 + T16 represent the index, peak index, pulse index, margin
matical calculation on the amplitude of vibration signal’s mean value, standard index, skewness index and kurtosis
the raw vibration signals [25]. Table II deviation, square root amplitude, abso- index, respectively.
lists the terminal set used under View1, lute mean value, skewness, kurtosis, For View2 (only consider the fre-
quency-domain characteristics of rolling
bearing signals), the terminal set con-
tains 13 statistical features F1 + F13,
Output
which are extracted through performing
the mathematical calculation on the
Output FCm
amplitude of frequency spectrum that
FC2 – was obtained through Fast Fourier
Feature Combination Transform for the vibration signals [25].
+ – ÷ ÷
Table III lists the terminal set used
Feature Construction under View2, where f ( j ) represents the
× T5 + T4 T14 × T9 T11
frequency spectrum of time signal s(i),
T1 T8 × T5 T2 T7 M represents the number of spectrum
Input
lines, and A j represents the frequency
T14 T3 value of the jth spectrum line. F1 repre-
sents the energy of the vibration signal
FIGURE 4 Program structure of MFCGPE (left) and an example program (right) that can be in frequency-domain, F2 + F5 represent
evolved by MFCPGE. the position variation of the main fre-
quency band, and F6 + F13 represent the
TABLE I Function set of GP.
concentration and dispersion of the fre-
quency spectrum.
SYMBOL INPUT OUTPUT DESCRIPTION For View3 (considering both the time-
+ 2 Features 1 Feature Addition domain and frequency-domain characteris-
− 2 Features 1 Feature Subtraction tics of rolling bearing signals), the terminal
× 2 Features 1 Feature Multiplication set contains 29 features (TF1 + TF29 ) com-
' 2 Features 1 Feature Protected division
posed by the features of View1 and View2,
where the features TF1 + TF16 are equal to
FC 2 2 Features 1 Vector Concatenation
the features T1 + T16 in View1, and the fea-
FC m 2 Vectors or 1 Vector/Feature 1 Vector Concatenation
tures TF17 + TF29 are equal to the fea-
tures F1 + F13 in View2. There are two
reasons for using TDF features and FDF
TABLE II Terminal set corresponding to View1.
features to form the multi-view features.
SYMBOL FORMULA SYMBOL FORMULA One is that the extraction of TDF and
T1 N
1 / s (i ) T9 min s (i ) FDF features is simple statistical calcula-
N i=1 tions, while the extraction of the time-fre-
T2 N
1 / [s (i ) - T ] 2 T10 T8 - T9 quency domain features typically needs
N - 1 i=1 1
complicated signal processing opera-
T3 N 2 T11 T2
tions and manual participation. The
eN s (i ) o
1 / T4 other is that [48] shows that using the
i=1
combination of the TDF and FDF fea-
T4 N
1 / s (i ) T12 T8
N i=1 F2 tures for feature construction is effective
for constructing new features. Different
T5 N
1 / (s (i ))3 T13 T8 from most of the existing works, the
N i=1 F4
proposed method constructs features
T6 N
1 / (s (i ))4 T14 T8 from these three views, individually,
N i=1 T3
which aims to generate a group of
T7 N
1 / (s (i ))2 T15 T5 effective and diverse features to com-
N i=1 ( T7 ) 3 prehensively describe a sample. The
T8 max s (i ) T16 T6 effectiveness and diversity of the con-
( T7 ) 2 structed features can also improve the

84 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

high generalization performance of the
ensemble, which will be described in
the later subsection.
E. Fitness Function
scheme. That is, the new generated fea- ence value of the inter-class and intra-
tures and labels of the training set are class distances into the range of [0, 1].
divided into 5 folds, evenly. Each time, n
one fold is used as the sub-test set, and d (X k, X l) = / (x ko - x lo) 2 (2)
the other two folds are used as the sub- o=1

An effective fitness function is crucial
for guiding GP to construct high-level
features for fault diagnosis. When the
number of training samples is small, the
features constructed by GP can easily
achieve good classification performance
on the training set, but poor generaliza-
tion performance on the test set. To
address this issue, a new fitness function
based on the classification accuracy and
the distance measure of the training
samples is proposed to guide the search
of MFCGPE. To calculate the diagnosis
accuracy, KNN is used to perform clas-
sification based on the constructed fea-
tures. The reason for using KNN is that
it is a simple classification algorithm,
easy to implement, and treats each
feature equally without any feature
weighting or selection [22], [52]. With
the use of KNN, MFCGPE can auto-
matically construct discriminative fea-
tures and avoid redundant or irrelevant
features. The distance measure is to
minimize the intra-class distance and
maximize the inter-class distance of the
training samples based on the con-
training set. The average value of the Ni Ni
D intra (X i) = 1 / / d (X k , X l )
(i) (i)
three sub-test sets is set as the diagno- Ni Ni k = 1 l = 1
sis accuracy. (3)
The Dist is calculated according to Ni Nj
D inter (X i, X j) = 1 / / d (X k , X l )
(i) (j)
Equations (2)–(5), which evaluate the Ni N j k = 1 l = 1
distance of the training samples with the  (4)
constructed features. The calculation of Dist = 1
Dist is based on the Euclidean distance. 1 + e - (min (Dinter) - max (Dintra))
(5)
For the given samples X k = {x k1, x k2,
f, x kn} and X l = {x l1, x l2, f, x l n}, the The proposed fitness function opti-
Euclidean distance between them can mizes the classification accuracy and the
be calculated as Equation (2), where x ko distances of the training samples, simul-
and x lo represent the features of samples taneously. When the fitness value
X k and X l, respectively, and n is the approaches one, it indicates that the fea-
number of features. The calculation of tures constructed by GP have the best
the intra-class and inter-class distances is classification performance and the train-
based on Equation (3) and Equation ing samples have a small intra-class dis-
(4), where X represents the samples in tance and a large inter-class distance.
one single class, and N is the number of
samples in the X class. Equation (5) is F. Ensemble for Fault Diagnosis
the calculation of Dist, where the sigmoid The MFCGPE system is able to find
function is used to transform the differ- the three best GP trees/programs that
TABLE III Terminal set corresponding to View2.
SYMBOL FORMULA SYMBOL FORMULA
F1 M F8 M

structed features. Using such a distance / f( j) / [f ( j ) - F1]4

j=1 j=1

measure will group the samples in the M M (F6) 2

same class and enlarge the differences of F2 M F9 M
/ (A j f ( j )) / [(A j - F5) 2 f ( j )]
the samples in different classes, which j=1 j=1
M
potentially improves the discriminabili- M
/ f( j)
j=1
ty of the constructed features. This also
helps to improve the effectiveness of F3 M F10 F9
/ [A 2j f ( j )] F2
KNN as it is based on distance. Based j=1
M
on the above analysis, the new fitness / f( j)
j=1
function to be maximised is formulated
F4 M F11 M
as follows. / [A 4j f ( j )] / [(A j - F2) 3 f ( j )]
j=1 j=1

(Acc + Dist) M M (F9) 3

Fit = (1) / [A 2j f ( j )]
2 j=1

F5 M F12 M
where Acc represents the diagnosis accu- / [A 2j f ( j )] / [(A j - F2) 4 f ( j )]
j=1 j=1
racy of KNN using the constructed fea- M M M (F9) 4
= / (A j f ( j ))G= / f ( j )G
4
tures and Dist represents the distance j=1 j=1

measure. Only the training set is used in F6 M F13 M

the fitness evaluation step, and the con- / [f ( j ) - F1] 2 /[ A j - F2 f ( j )]

j=1 j=1
structed features are transformed into M-1 M F9
the range of [0, 1] through the min-max F7 M — —
normalization method. The Acc is calcu- / [f ( j ) - F1] 3

j=1

lated using the 5-fold cross-validation M ( F6 )3

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 85


View1 GP
Features Construct Features
Raw View2 GP
Signal Features Construct Features
View3 GP
Features Construct Features
FIGURE 5 Architecture of ensemble fault diagnosis system.
FIGURE 6 NCEPU test rig.
construct features from different views.
Since the high-level features are con-
structed from different views, they are
diverse and effective. To effectively use
the constructed features, an ensemble is
created for fault diagnosis. Consistent
with the fitness evaluation process, the
ensemble uses three KNNs as the base
classifiers, and each KNN uses the fea-
tures constructed from a single view.
The overall fault diagnosis system using
ensemble is illustrated in Figure 5.
The training set (including the trans-
formed feature sets from three different
views and the corresponding class labels)
are fed into three KNNs, and these
three KNNs are used as base classifiers
of an ensemble. This ensemble will have
high generalization performance since
the inputs are the features constructed
TABLE IV Description of the NCEPU dataset.
CLASS
LABEL RUNNING CONDITION
1 Normal
2 Inner ring fault
3 Outer ring fault
4 Rolling element fault
5 Inner and outer ring compound fault
6 Rolling element and outer ring
compound fault
86 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
KNN
Majority Final
KNN Voting Diagnosis
KNN
from different views. This constructed
ensemble is applied to predict the class
labels for the unseen samples in the test
set. In the process, the majority voting
algorithm is employed because it is a
simple and common method to com-
prehensively consider the results
obtained by the multiple base classifiers
[53]. For a test sample, among the class
labels/fault types obtained by three
KNNs, the fault type with the largest
number of votes is chosen as the final
fault type.
IV. Experimental Design
A. Dataset Description
The mechanical equipment typically
operates normally, and the failure data
that can be collected is limited. Collect-
ing a large number of labeled samples
requires high cost and a long period of
time. Therefore, the goal of this paper is
to deal with fault diagnosis using the
limited number of training samples. In
the experiments, three datasets with a
small number of training samples are
employed to evaluate the effectiveness
of MFCGPE. In each dataset, only five
samples of each running condition are
TRAINING TEST
SAMPLES SAMPLES
5 45
5 45
5 45
5 45
5 45
5 45
randomly selected as the training set,
and the remaining samples are used
as the test set. The three datasets are
described as follows.
1) NCEPU [25]: It is a bearing fault
dataset collected by North China
Electric Power University (NCEPU)
and contains vibration signals of six
fault types. Figure 6 shows the used
test rig of NCEPU. The six rolling
bearing running conditions are the
normal state (NOR), inner ring fault
(IRF), outer ring fault (ORF), rolling
element fault (REF), inner & outer
ring compound fault (IOCF), and
rolling element & outer ring com-
pound fault (ROCF), which are sim-
ulated by manufacturing defects on
normal bearings using electrical dis-
charge machining (EDM) technolo-
gy. All vibration signals are collected
with a sampling frequency of 12,000
Hz. The first 102,400 data points of
vibration signals under each running
condition are divided into 50 sam-
ples on average and there is no over-
lap between each sample. Table IV
lists the detailed information of the
NCEPU dataset. Figure 7 shows the
time domain waveform of vibration
signals under six running conditions
in NCEPU.
2) CWRU [54]: It is a bearing fault
dataset collected by Case Western
Reserve University (CWRU) and
contains vibration signals of ten
fault types and fault degrees. Fig-
ure 8 shows the used test rig of CWRU.
All faults are generated by manufac-
turing defects on the inner ring,
outer ring, and rolling element sur-
faces using EDM, and the same fault
category with three defect diame-
ters (0.007, 0.014, and 0.021 inches,
respectively). The ten different roll-
ing bearing running conditions are
the normal state (NOR), three kinds
of inner ring fault (IRF07, IRF14,
and IRF21), three kinds of outer
ring fault (ORF07, ORF14, and
ORF21), and three kinds of rolling
element fault (REF07, REF14, and
REF21), respectively. IRF, ORF, and
REF indicate inner ring fault, outer
ring fault, and rolling element fault,
 respectively. 07, 14, and 21 indicate
that the defect diameter is 0.007, NOR IRF
50 300
0.014, and 0.021 inches, respectively.
0 0
The vibration signals of driver-end –50 –300
bearing under 1,797 r/min are col- 0 0.08 0.16 0 0.08 0.16
lected with a sampling frequency of

Amplitude (m/s2)
ORF REF
12,000 Hz. Similar to the NCEPU 400 500
dataset, the first 102,400 data points 0 0
of vibration signals under each run- –400 –500
ning condition are divided into 50 0 0.08 0.16 0 0.08 0.16
samples on average. Five samples of IOCF ROCF
400 200
each running condition are ran-
0 0
domly selected to form the training
–400 –200
set, and the remaining samples are 0 0.08 0.16 0 0.08 0.16
used as the test set. Table V lists the Time (s) Time (s)
detailed information of the CWRU
dataset. Figure 9 shows the time FIGURE 7 Time domain waveform of vibration signals under six running conditions in NCEPU.
domain waveform of vibration sig-
nals under ten running conditions domain waveform of vibration sig- sification. The TDF and FDF features
in CWRU. nals under four running conditions have been described in Section III-D.
3) X JTU [55]: It is a run-to-failure in XJTU. The numbers of features in MDF,
bearing fault dataset collected by MMSDE, and IMDE are 37, 20, and 20,
Xi’an Jiaotong University (XJTU), B. Comparison Methods respectively. The comparisons aim to
in which the reason for fault occur- To verify the effectiveness of MFCGPE, investigate whether the features con-
rence is the natural damage as the four categories of competitive methods, structed by MFCGPE are more effective
running time increases. Figure 10 i.e., 19 methods, are employed for com- for fault diagnosis than these manually
shows the used test rig of XJTU. In parisons. The first category includes five crafted features.
the experiments, the vibration sig- different classification algorithms, i.e., The third category contains six GP
nals of the test bearings are collect- KNN, SVM, Naive Bayes (NB), logistic based methods using TDF, FDF, and
ed with a sampling frequency of regression (LR), and multilayer percep- TFDF (described in Section III-D) as
25,600 Hz until the bearing fails, tron (MLP), which use raw signals
and the time interval for each sam- amplitude to train the classifiers for fault
pling is one minute. The vibration classification. The second category are
signals of the last two recorded data KNN using five manually crafted fea-
file of the Bearing 2_1, Bearing tures, i.e., TDF, FDF, multi-domain fea-
2_2, and Bearing 2_3 datasets are tures (MDF) [25], modified multi-scale
used as the analyzed data. The fault symbolic dynamic entropy (MMSDE)
types are inner ring fault (IRF), [14], and improved multi-scale disper-
outer ring fault (ORF), and cage sion entropy (IMDE) [15], for fault clas- FIGURE 8 CWRU test rig.
fault (CF). The signals of the first
two recorded data file of the Bear-
TABLE V Description of the CWRU dataset.
ing 2_1 dataset is used as the nor-
mal (NOR) vibration signals for CLASS DEFECT TRAINING TEST
LABEL RUNNING CONDITION SIZE (IN) SAMPLES SAMPLES
analysis. Every vibration signal
contains 32,768 data points and is 1 Normal 0 5 45
divided into 32 samples on average 2 Inner ring fault 0.007 5 45
for conducting the experiments. 3 Inner ring fault 0.014 5 45
Each sample contains 2,048 data 4 Inner ring fault 0.021 5 45
points. Five samples under each 5 Outer ring fault 0.007 5 45
running condition are randomly
6 Outer ring fault 0.014 5 45
selected to form the training set,
7 Outer ring fault 0.021 5 45
and the remaining samples are used
as the test set.Table VI lists the 8 Rolling element fault 0.007 5 45

detailed information of the XJTU 9 Rolling element fault 0.014 5 45

dataset. Figure 11 shows the time 10 Rolling element fault 0.021 5 45

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 87

 MFCGPE can outperform other GP
NOR IRF07 based feature construction methods
0.5 2
that construct features.
0 0
The fourth category of the compari-
–0.5 –2
0 0.08 0.16 0 0.08 0.16 son methods are three ensemble diagno-
sis methods (i.e., OFE, GPSE, and
IRF14 IRF21
5 5 GPME). These methods use three
0 0 KNNs as base classifiers and the majori-
–5 –5 ty voting algorithm to build an ensem-
0 0.08 0.16 0 0.08 0.16 ble. In OFE (i.e., ensemble using the
Amplitude (m/s2)

ORF07 ORF14 original features), the TDF, FDF, and

1 2
TFDF features are fed into three KNNs,
0 0
respectively. In GPSE (i.e., ensemble
–1 –2
0 0.08 0.16 0 0.08 0.16 using the features constructed by GPS),
the features constructed by GPS-TDF,
ORF21 REF07 GPS-FDF, and GPS-TFDF are fed into
1 5
0 0 three KNNs to build an ensemble. In
–1 –5 GPME (i.e., ensemble using the features
0 0.08 0.16 0 0.08 0.16 constructed by GPM), the features con-
REF14 REF21
structed by GPM-TDF, GPM-FDF, and
0.5 10 GPM-TFDF are fed into three KNNs,
0 0 respectively. The comparisons aim to
–0.5 –10 investigate whether MFCGPE can beat
0 0.08 0.16 0 0.08 0.16
Time (s) Time (s) the ensemble diagnosis methods that use
the original features and the features
FIGURE 9 Time domain waveform of vibration signals under ten running conditions in CWRU.
constructed by GP.

GPS methods (using GP to construct a C. Parameter Settings

FIGURE 10 XJTU test rig.
the input features to construct high-
level features. The constructed features
are used as the inputs of the KNN clas-
sifier for fault classification. Two types
of GP based methods are used, i.e., the
TABLE VI Description of the XJTU dataset.
CLASS LABEL RUNNING CONDITION
1 Normal
2 Inner ring fault
3 Outer ring fault
4 Cage fault
single feature) and the GPM methods
(using GP to construct multiple fea-
tures). The fitness function of these
methods is the same as MFCGPE. The
comparisons can investigate the effect
of the number of constructed features
on diagnosis accuracy. The GPS/GPM-
TDF, GPS/GPM-FDF, and GPS/
GPM-TFDF methods use the TDF,
FDF, and TFDF features for feature
construction, respectively. According to
[46], the number of features construct-
ed by the GPM based methods is the
same as the number of the classes in
the dataset. For example, GPM-TDF,
GPM-FDF, and GPM-TFDF construct
10 new features for classifying the
CWRU dataset with 10 classes. The
comparisons aim to investigate whether
TRAINING SAMPLES TEST SAMPLES
5 27
5 27
5 27
5 27
In MFCGPE, the GP related parameters
are common settings and refer to those
in [56]. The population size and the
maximal number of generations are set
as 100 and 50, respectively. The rates of
crossover, mutation, and elitism are set as
0.8, 0.19 and 0.01, respectively. The tree
depth is between 2-6. The tree genera-
tion method is ramped half-and-half.
The tournament selection with size 5 is
used to select parents for crossover and
mutation operations. The parameters of
the GP based comparison methods are
the same as MFCGPE.
The GP based methods are imple-
mented using DEAP [57] and the clas-
sification algorithms are implemented
using scikit-learn [58]. The number of
the neighbors in KNN is set as 3
according to [22]. The parameters in
the classification algorithms are the
default values of the scikit-learn pack-
age. The parameter values of the meth-
ods using manually extracted features
are set according to [14], [15], [25].
Each method has been executed 30
independent runs on each dataset with

88 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

different random seeds. The evolution-
ary process of GP and the classifier
training step only use the training set.
The fault diagnosis results of the test
sets are reported.
V. Results and Discussions
This section discusses and analyses the
fault diagnosis results obtained by MFC-
GPE and the 19 comparison methods
on the three different datasets with a small
number of training samples. Table VII
lists the classification accuracy of the 20
methods, including the maximum
(Max) value, average (Avg) value and
standard deviation (Std) of the accuracy
of the 30 runs, where the best results of
each dataset are highlighted in bold.
the significance test results on each data- maximal and average accuracy of
set is listed in the last row of Table VII. 52.78%, which is better than other clas-
Rows 1-5 of Table VII list the classi- sifiers. Compared with these five meth-
fication results of the five traditional ods, MFCGPE achieves much higher
classifiers using raw signals amplitude accuracy, i.e., over 99%, on the three
(RSA). It can be seen that the diagnosis datasets. The results show that construct-
accuracy of these methods is very low. ing high-level features is very important
On the NCEPU and CWRU datasets, for fault diagnosis of rolling bearings.
MLP achieves better results than KNN, Rows 6 to 10 of Table VII list the
LR, SVM, and NB. Specifically, MLP classification results of the KNN classifi-
achieves an average accuracy of 22.01% er using five different types of manually
on NCEPU and of 21.92% on CWRU. crafted features (i.e., TDF, FDF, MDF,
On the XJTU dataset, NB achieves a MMSDE, and IMDE). It can be seen
NOR IRF
5 50
0 0
Amplitude (g)

–5 –50
Wilcoxon rank-sum test with a 5% sig- 0 0.04 0.08 0 0.04 0.08
nificance level is employed to evaluate ORF CF
30 50
the significant difference in performance
0 0
improvement of MFCGPE compared to
–30 –50
a method. In Table VII, the “+” symbol 0 0.04 0.08 0 0.04 0.08
indicates that the performance of MFC- Time (s) Time (s)
GPE is significantly better than the
comparison method. The summary of FIGURE 11 Time domain waveform of vibration signals under four running conditions in XJTU.

TABLE VII Diagnosis accuracy (%) of MFCGPE and the comparison methods on the NCEPU, CWRU, and XJTU datasets.

ROW METHOD NCEPU CWRU XJTU

MAX AVG±STD MAX AVG±STD MAX AVG±STD

1 RSA+KNN 16.67 16.67±0.00+ 10.00 10.00±0.00+ 25.00 25.00±0.00+
2 RSA+LR 17.04 17.04±0.00+ 21.33 21.33±0.00+ 28.70 28.70±0.00+
3 RSA+SVM 16.67 16.67±0.00+ 16.44 16.44±0.00+ 28.70 28.70±0.00+
4 RSA+NB 17.41 17.41±0.00+ 23.33 23.33±0.00+ 52.78 52.78±0.00+
5 RSA+MLP 31.85 22.01±4.24+ 32.22 21.92±1.10+ 37.96 31.45±3.26+
6 TDF 79.26 79.26±0.00+ 76.67 76.67±0.00+ 87.96 87.96±0.00+
7 FDF 82.96 82.96±0.00+ 83.56 83.56±0.00+ 97.22 97.22±0.00+
8 MDF 88.15 88.15±0.00+ 90.22 90.22±0.00+ 75.93 75.93±0.00+
9 MMSDE 87.78 87.78±0.00+ 92.67 92.67±0.00+ 83.33 83.33±0.00+
10 IMDE 91.11 91.11±0.00+ 85.78 85.78±0.00+ 82.41 82.41±0.00+
11 GPS-TDF 90.74 87.83±2.76+ 88.89 84.62±2.56+ 91.67 88.22±3.71+
12 GPS-FDF 92.59 89.42±3.78+ 91.78 87.52±3.08+ 99.07 96.45±2.36+
13 GPS-TFDF 94.81 90.73±3.53+ 93.78 90.17±2.91+ 100.0 96.67±3.37+
14 GPM-TDF 94.44 92.37±1.94+ 96.22 93.37±2.42+ 95.37 93.09±2.98+
15 GPM-FDF 95.92 93.30±2.17+ 97.11 95.34±1.58+ 100.0 97.87±2.36+
16 GPM-TFDF 97.03 94.56±2.35+ 97.56 96.16±1.76+ 100.0 98.21±1.97+
17 OFE 89.25 89.25±0.00+ 90.44 90.44±0.00+ 95.37 95.37±0.00+
18 GPSE 95.55 93.84±1.56+ 94.46 92.81±1.45+ 100.0 98.37±1.81+
19 GPME 98.51 97.47±1.12+ 98.45 97.65±0.85+ 100.0 99.02±1.09+
20 MFCGPE 100.0 99.56±0.30 100.0 99.23±0.67 100.0 99.61±0.26
21 OVERALL 19+ 19+ 19+

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 89

that the accuracy of these five methods
is higher than the KNN, LR, SVM, NB,
and MLP methods using raw signals
amplitude. Among these five methods
using the manually crafted features, the
IMDE method achieves the best results
(the maximal and average diagnosis
accuracy are 91.11%) on NCEPU, the
MMSDE method achieves the best
results (the maximal and average diag-
nosis accuracy are 92.67%) on CWRU,
and the FDF method achieves the best
results (the maximal and average diag-
nosis accuracy are 97.22%) on XJTU.
The results show that it is necessary to
extract a set of effective features accord-
ing to the datasets to perform fault
diagnosis because the performance of
the manually extracted features varies
with datasets. Compared with these
methods, MFCGPE achieves much bet-
ter performance by automatically con-
structed multi-view high-level features
and building an ensemble for fault diag-
nosis. The adaptability of MFCGPE is
much higher than these five methods as
it achieves the best results on all the
three datasets.
Rows 11 to 13 of Table VII are the
classification results of the GP based
comparison methods for constructing a
single feature, i.e., GPS-TDF, GPS-FDF,
and GPS-TFDF. Among these methods,
GPS-TFDF achieves the best results on
all three datasets by obtaining an aver-
age accuracy of 90.73% on NCEPU,
90.17% on CWRU and 96.67% on
XJTU. Compared with these three
methods, MFCGPE achieves better and
stable classification performance. The
results show that only constructing one
high-level feature may not be effective
for multi-class fault diagnosis. Com-
pared with these methods, MFCGPE is
able to construct a flexible number of
effective features from every single view
for fault diagnosis.
Rows 14 to 16 of Table VII are the
classification results of the GP based
comparison methods that construct
multiple features for fault diagnosis, i.e.,
GPM-TDF, GPM-FDF, and GPM-
TFDF. Among these three methods,
GPM-TFDF achieves better results on
all three datasets. Specifically, GPM-
90 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
TFDF achieves an average diagnosis
accuracy of 94.56%, 96.16%, and
98.21% on NCEPU, CWRU and
XJTU, respectively. Compared with the
GPS-based methods, the GPM-based
methods achieve higher average accura-
cy and smaller standard deviation values.
These results indicate that using multi-
ple constructed high-level features is
more effective than using a single con-
structed high-level feature to improve
diagnosis performance. Compared with
these three methods, MFCGPE achieves
better performance on the three datasets.
MFCGPE builds an effective ensemble
using the features constructed from
multiple views, which allows it to obtain
higher generalization performance than
these methods using a single classifier.
Rows 17 to 19 of Table VII are the
classification results of the ensemble
diagnosis methods, i.e., OFE, GSE, and
GPME. It can be found that on the
NCEPU, CWRU, and XJTU datasets,
the diagnostic performance of OFE,
GPSE, and GPME is ranked the third,
the second, and the first, respectively.
Compared with these three methods,
MFCGPE achieves higher average accu-
racy and smaller standard deviation values.
The reason is that MFCGPE has a new
program structure, function set, terminal
set, and fitness function, which allow it to
construct more discriminative features
and build a more effective ensemble to
obtain better diagnosis accuracy.
Row 20 and Row 21 of Table VII
lists the classification results and the sig-
nificance test results of the MFCGPE
approach, respectively. On the NCEPU,
CWRU, and XJTU dataset, MFCGPE
achieves the maximal accuracy of 100%
and the average accuracy of above 99%.
The results of the significance tests
show that the diagnosis performance of
MFCGPE is significantly better than
the 19 comparison methods on the
three datasets.
In summary, the results show that
MFCGPE is able to achieve excellent
fault diagnosis performance on three
datasets with a small number of training
samples. MFCGPE can adaptively con-
struct a flexible number of features for
fault diagnosis, which can not only
achieve a comprehensive description of
the raw signal but also avoid redundancy
and information interference. The fitness
function based on accuracy and distance
enables MFCGPE to construct new
effective features that achieve a small
intra-class distance and a large inter-class
distance of the training samples, which
can improve the generalization perfor-
mance on the unseen test set. MFCGPE
constructs multiple high-level features
from different views, individually, and
constructs an effective ensemble based
on these constructed features through
the majority voting, which further
improves the effectiveness of the diag-
nostic approach and avoids the overfit-
ting issue caused by using a small
number of training samples. Thus, MFC-
GPE is a practical and promising
approach to engineering applications.
VI. Further Analysis
This section further analyses the evolved
GP tree/models, the features construct-
ed by GP from different views, and the
constructed ensemble to provide insights
into why the new approach is effective.
A. Example Trees/Models
Figure 12 shows the best trees/models
evolved by MFCGPE from the three
different views on the NCEPU dataset,
where the white oval nodes are the fea-
ture combination operators, the white
circle nodes are the feature construction
operators, and the orange, blue, and pink
rectangle nodes are the features of View1,
View2, and View3, respectively.
As it can be seen from Figure 12, for
View1, six distinguished features among
the input 16 features are selected to
construct four high-level features. For
View2, six distinguished features among
the 13 input features are selected to
construct four high-level features. For
View3, five distinguished features among
the 29 input features are selected to
construct three high-level features. This
indicates that MFCGPE can automati-
cally select the useful features and deter-
mine the number of constructed
features, which avoid information inter-
ference and feature redundancy. The
depths of the trees evolved from View1,
 FCm

FCm ×

FC2 F11 F5
FCm
× – ×
FC2 FC2
F5 – F5 F5 – –
÷ ÷ ÷ ÷ FCm
F8 F1 F7 – F12 F5
T2 – T10 T5 ÷ T10 T2 T5 – FC2
F8 –

T10 ÷ T7 × + – TF7 TF28 + ÷

T7 T13 T5 T9 F1 F12 F8 F1 TF3 TF17 TF2 TF28

(a) (b) (c)

FIGURE 12 Example trees evolved by MFCGPE on the NCEPU dataset. (a) Best tree obtained under View1 (b) Best tree obtained under View2 (c)
Best tree obtained under View3.

View2, and View3 are 5, 7, and 3, respec-

tively, which indicates that MFCGPE View1 View2 View3
can adaptively evolve the optimal vari-
able-length models with the use of the
time-domain, frequency-domain,
time&frequency-domain characteristics
of vibration signals. The above analysis
shows that MFCGPE has the ability to
learn effective and diverse features from
multi-view input features to capture dif- (a)
ferent patterns of the vibration signal,
which is beneficial in describing data View1 View2 View3
characteristics comprehensively.

B. Visualization of the GP-Constructed

Features
To better demonstrate that the evolved
models/trees shown in Figure 12 are
effective for fault diagnosis, the original
features and the newly constructed fea- (b)
tures are visualized for comparisons. The
features of the samples in the test set are NOR IRF ORF REF IOCF ROCF
visualized by using t-distributed stochas-
tic neighbor embedding (t-SNE) [59] FIGURE 13 Feature visualization on the NCEPU dataset using t-SNE. (a) Original features
method. Figure 13 shows the visualiza- (b) GP constructed features.

tion results, where six colored points

represent six running conditions of the
rolling bearing, i.e., normal (NOR),
inner ring fault (IRF), outer ring fault
(ORF), rolling element fault (REF),
inner & outer ring compound fault
(IOCF), and rolling element & outer
ring compound fault (ROCF ).
As shown in Figure 13(a), using the
original features of View1 for visualiza-
tion, the points in different classes are all
overlapping; when using the original
features of View2 and View3 for visual-
ization, the points in the NOR class are
clustered separately, but the points in
other classes are also overlapping. These
visualization results indicate that the
original features of vibration signals in
different running conditions cannot
effectively separate the fault types into
different classes to achieve a high diag-
nostic accuracy. As shown in Figure 13(b),

AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 91

using the features constructed from
View1 for visualization, few points in the
NOR, IRF, and REF classes are overlap-
ping, and the other colored points are
gathered together and do not overlap.
Using the features constructed from
View2 for visualization, only a few points
in the IOCF and ROCF classes are over-
lapping. Using the features constructed
from View3 for visualization, only a few
points in the ORF and ROCF classes are
overlapping. Compared with the original
features, these newly constructed features
have a good similarity among the same
class and have a big difference between
different classes. These visualization
results indicate that the constructed
features can effectively represent the
vibration signals in different running
conditions, which makes fault diagnosis
simpler and more accurate.
View1
6 6
5 5
Class Label
C. Diagnosis Results Using the
GP-Constructed Features and the
Constructed Ensemble
To analyze the effectiveness of the con-
structed features of Figure 13(b) and
the built ensemble, Figure 14 shows
their diagnosis results and the results of
the built ensemble, where the blue box
represents the true label of the sample,
the red asterisks represent the predicted
label of the sample, and the class labels 1,
2, 3, 4, 5, and 6 represent the NOR, IRF,
ORF, REF, IOCF, and ROCF running
conditions, respectively. The overlap of
boxes and asterisks indicates that the
diagnosis is correct. No-overlapping of
boxes and asterisks indicates that the
diagnosis is wrong.
It can be seen from Figure 14(a) that
using the features constructed from
View1 for fault diagnosis will cause seven
View2
samples to be misclassified, where two
samples in the IRF class are identified as
the NOR class and five samples in the
REF class are identified as the IRF class.
Figure 14(b) shows the diagnosis results
of using the features constructed from
View2. It shows that two types of com-
pound faults are misclassified, i.e., two
samples in the IOCF class are classified
into the ROCF class and three samples
in the ROCF class are identified as the
IOCF class. Figure 14(c) shows the diag-
nosis results of using the features con-
structed from View3, where three samples
in the ORF class are identified as the
ROCF class. The diagnosis accuracy of
using the TDF, FDF, and TFDF based
constructed features are 97.40%, 98.14%
and 98.88%, respectively. The ensemble
diagnosis accuracy is obtained by inte-
grating the three diagnosis results via the
majority voting method and shown in
Figure 14(d), in which no samples are
misclassified. Obviously, the ensemble
strategy improves the diagnosis accuracy.
By analyzing the best models evolved
by MFCGPE, the feature construction

Class Label

4 4 process is clearly displayed. In addition,

3 3 the visualization results and the diagnosis
results of the newly constructed features
2 2 are demonstrated for interpretability.
1 1
Owing to the program structure and the
0 45 90 135 180 225 270 0 45 90 135 180 225 270 new fitness function, MFCGPE con-
Sample Number Sample Number structs a flexible number of diverse and
(a) (b) effective high-level features from differ-
ent views. The constructed features can
View3 Ensemble
6 6 make the classification of different fault
types easier and more accurate. Owing
5 5 to the ensemble diagnosis strategy, MFC-
Class Label

Class Label

4 4
GPE constructs multi-view features and
gains a higher diagnosis accuracy by
3 3 using an ensemble built from the features
constructed from different views.
2 2

1 1 VII. Conclusions
0 45 90 135 180 225 270 45 90 135 180 225 270 The goal of this paper was to develop a
Sample Number Sample Number new GP-based approach to achieving
(c) (d)
effective fault types diagnosis of rolling
Ture bearings using a small number of train-
Predict ing samples. This goal has been suc-
cessfully achieved by developing the
FIGURE 14 Diagnosis results on the NCEPU dataset using the constructed features of differ- MFCGPE approach. A new GP pro-
ent views and the constructed ensemble. (a) Diagnosis results of using the features construct- gram structure, a function set, and a
ed on View1 (b) Diagnosis results of using the features constructed on View2 (c) Diagnosis
results of using the features constructed on View3 (d) Diagnosis results of using the construct- terminal set were developed to enable
ed ensemble. MFECPE to construct a flexible ­number

92 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021

of high-level features from three different
views of features. A new fitness function
based on the measures of accuracy and
distance was developed to enable these
newly constructed features to be accu-
rate and discriminative. To improve the
diagnosis performance, an ensemble clas-
sifier was created by using constructed
features from multiple views and using
KNN. With these designs, MFCGPE can
not only automatically select and con-
struct informative and discriminative fea-
tures from different views but also build
an effective ensemble to achieve a high
generalization performance.
The effectiveness of MFCGPE was
evaluated on three rolling bearing fault
datasets and compared with the 19 com-
petitive methods. The results showed that
MFCGPE achieved the best diagnosis
accuracy on the three datasets among all
the methods. The highlight of MFCGPE
was that multiple discriminative features
were automatically constructed using
MFCGPE, and the ensemble diagnosis
can address the issue of poor generaliza-
tion of the diagnosis model caused by
using a small number of training samples.
This paper shows that the proposed
MFCGPE approach is effective for fault
diagnosis of rolling bearings. In addition
to fault diagnosis, remaining life predic-
tion can also help to analyze rolling
bearing degradation. In the future, we
will investigate how GP is used to con-
struct the health index to predict the
remaining life of the rolling bearing.
Acknowledgments
This work was supported in part by the
National Natural Science Foundation of
China under grant 51777075, the Natu-
ral Science Foundation of Hebei Prov-
ince under grant E2019502064, the
Marsden Fund of New Zealand Govern-
ment under Contracts VUW1509 and
VUW1615, the Science for Technologi-
cal Innovation Challenge (SfTI) fund
under grant E3603/2903, the University
Research Fund at Victoria University of
Wellington grant number 223805/3986,
MBIE Data Science SSIF Fund under
the contract RTVU1914, and National
Natural Science Foundation of China
(NSFC) under Grant 61876169.
References
[1] R. Liu, B. Yang, E. Zio, and X. Chen, “Artificial
intelligence for fault diagnosis of rotating machinery: A
review,” Mech. Syst. Signal Process., vol. 108, pp. 33–47,
2018. doi: 10.1016/j.ymssp.2018.02.016.
[2] H. Li, T. Liu, X. Wu, and Q. Chen, “Enhanced fre-
quency band entropy method for fault feature extrac-
tion of rolling element bearings,” IEEE Trans. Ind. In-
form., vol. 16, no. 9, pp. 5780–5791, 2020. doi: 10.1109/
TII.2019.2957936.
[3] Y. Zhang and R. Randall, “Rolling element bear-
ing fault diagnosis based on the combination of genetic
algorithms and fast kurtogram,” Mech. Syst. Signal Pro-
cess., vol. 23, no. 5, pp. 1509–1517, 2009. doi: 10.1016/j.
ymssp.2009.02.003.
[4] R. Yan, R. X. Gao, and X. Chen, “Wavelets for
fault diagnosis of rotary machines: A review with ap-
plications,” Signal Process., vol. 96, pp. 1–15, 2014. doi:
10.1016/j.sigpro.2013.04.015.
[5] Y. Lei, J. Lin, Z. He, and M. Zuo, “A review on em-
pirical mode decomposition in fault diagnosis of rotating
machinery,” Mech. Syst. Signal Process., vol. 35, nos. 1–2,
pp. 108–126, 2013. doi: 10.1016/j.ymssp.2012.09.015.
[6] J. Antoni, “The spectral kurtosis: a useful tool for
characterising non-stationary signals,” Mech. Syst. Signal
Process., vol. 20, no. 2, pp. 282–307, 2006. doi: 10.1016/j.
ymssp.2004.09.001.
[7] G. L. McDonald, Q. Zhao, and M. J. Zuo, “Maxi-
mum correlated kurtosis deconvolution and application
on gear tooth chip fault detection,” Mech. Syst. Signal
Process., vol. 33, pp. 237–255, 2012. doi: 10.1016/j.yms-
sp.2012.06.010.
[8] K. Dragomiretskiy and D. Zosso, “Variational mode
decomposition,” IEEE Trans. Signal Process., vol. 62, no.
3, pp. 531–544, 2013. doi: 10.1109/TSP.2013.2288675.
[9] R. B. Randall, J. Antoni, and S. Chobsaard, “The rela-
tionship between spectral correlation and envelope analysis
in the diagnostics of bearing faults and other cyclostationary
machine signals,” Mech. Syst. Signal Process., vol. 15, no. 5,
pp. 945–962, 2001. doi: 10.1006/mssp.2001.1415.
[10] S. Maurya, V. Singh, N. K. Verma, and C. K.
Mechefske, “Condition-based monitoring in variable
machine running conditions using low-level knowledge
transfer with DNN,” IEEE Trans. Autom. Sci. Eng., 2020.
[11] J. Yang, Y. Zhang, and Y. Zhu, “Intelligent fault
diagnosis of rolling element bearing based on SVMS
and fractal dimension,” Mech. Syst. Signal Process., vol.
21, no. 5, pp. 2012–2024, 2007. doi: 10.1016/j.yms
sp.2006.10.005.
[12] W. Caesarendra, B. Kosasih, A. K. Tieu, and C. A.
Moodie, “Application of the largest Lyapunov exponent
algorithm for feature extraction in low speed slew bearing
condition monitoring,” Mech. Syst. Signal Process., vol. 50,
pp. 116–138, 2015. doi: 10.1016/j.ymssp.2014.05.021.
[13] J. Zheng, H. Pan, and J. Cheng, “Rolling bearing
fault detection and diagnosis based on composite mul-
tiscale fuzzy entropy and ensemble support vector ma-
chines,” Mech. Syst. Signal Process., vol. 85, pp. 746–759,
2017. doi: 10.1016/j.ymssp.2016.09.010.
[14] Y. Li, Y. Yang, G. Li, M. Xu, and W. Huang, “A fault
diagnosis scheme for planetary gearboxes using modi-
fied multi-scale symbolic dynamic entropy and MRMR
feature selection,” Mech. Syst. Signal Process., vol. 91, pp.
295–312, 2017. doi: 10.1016/j.ymssp.2016.12.040.
[15] X. Yan and M. Jia, “Intelligent fault diagnosis of ro-
tating machinery using improved multiscale dispersion
entropy and MRMR feature selection,” Knowl. Based
Syst., vol. 163, pp. 450–471, 2019. doi: 10.1016/j.kno
sys.2018.09.004.
[16] V. Singh and N. K. Verma, “Intelligent condition-
based monitoring techniques for bearing fault diagnosis,”
IEEE Sens. J., 2020.
[17] X. Zhang, Q. Zhang, M. Chen, Y. Sun, X. Qin, and
H. Li, “A two-stage feature selection and intelligent fault
diagnosis method for rotating machinery using hybrid
filter and wrapper method,” Neurocomputing, vol. 275, pp.
2426–2439, 2018. doi: 10.1016/j.neucom.2017.11.016.
[18] Z. Huo, Y. Zhang, L. Shu, and M. Gallimore, “A
new bearing fault diagnosis method based on fine-to-
coarse multiscale permutation entropy, Laplacian score
and SVM,” IEEE Access, vol. 7, pp. 17,050–17,066, 2019.
doi: 10.1109/ACCESS.2019.2893497.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE
[19] Y. Gu, X. Zhou, D. Yu, and Y. Shen, “Fault diagno-
sis method of rolling bearing using principal component
analysis and support vector machine,” J. Mech. Sci. Tech-
nol., vol. 32, no. 11, pp. 5079–5088, 2018. doi: 10.1007/
s12206-018-1004-0.
[20] J. Yu, “Machinery fault diagnosis using joint global
and local/nonlocal discriminant analysis with selective
ensemble learning,” J. Sound Vib., vol. 382, pp. 340–356,
2016. doi: 10.1016/j.jsv.2016.06.046.
[21] X. Zhao and M. Jia, “Fault diagnosis of rolling bear-
ing based on feature reduction with global-local margin
fisher analysis,” Neurocomputing, vol. 315, pp. 447–464,
2018. doi: 10.1016/j.neucom.2018.07.038.
[22] J. Tian, C. Morillo, M. H. Azarian, and M. Pecht,
“Motor bearing fault detection using spectral kurto-
sis-based feature extraction coupled with k-nearest
neighbor distance analysis,” IEEE Trans. Ind. Electron., vol.
63, no. 3, pp. 1793–1803, 2016. doi: 10.1109/TIE.2015.
2509913.
[23] M. Unal, M. Onat, M. Demetgul, and H. Kucuk,
“Fault diagnosis of rolling bearings using a genetic al-
gorithm optimized neural network,” Measurement,
vol. 58, pp. 187–196, 2014. doi: 10.1016/j.measure
ment.2014.08.041.
[24] M. Luo, C. Li, X. Zhang, R. Li, and X. An, “Com-
pound feature selection and parameter optimization of
ELM for fault diagnosis of rolling element bearings,”
ISA Trans., vol. 65, pp. 556–566, 2016. doi: 10.1016/j.
isatra.2016.08.022.
[25] X. Yan and M. Jia, “A novel optimized svm clas-
sification algorithm with multi-domain feature and its
application to fault diagnosis of rolling bearing,” Neu-
rocomputing, vol. 313, pp. 47–64, 2018. doi: 10.1016/j.
neucom.2018.05.002.
[26] M. He and D. He, “Deep learning-based approach
for bearing fault diagnosis,” IEEE Trans. Ind. Appl., vol.
53, no. 3, pp. 3057–3065, 2017. doi: 10.1109/TIA.2017.
2661250.
[27] S. Zhang, S. Zhang, B. Wang, and T. G. Habetler,
“Deep learning algorithms for bearing fault diagnos-
tics—A comprehensive review,” IEEE Access, vol. 8,
pp. 29,857–29,881, 2020. doi: 10.1109/ACCESS.2020.
2972859.
[28] W. Deng, H. Liu, J. Xu, H. Zhao, and Y. Song, “An
improved quantum-inspired differential evolution al-
gorithm for deep belief network,” IEEE Trans. Instrum.
Meas., vol. 69, no. 10, pp. 7319–7327, 2020.
[29] S. Maurya, V. Singh, and N. K. Verma, “Condi-
tion monitoring of machines using fused features from
EMD-based local energy with DNN,” IEEE Sens. J., vol.
20, no. 15, pp. 8316–8327, 2019. doi: 10.1109/JSEN.
2019.2927754.
[30] C. Cheng, B. Zhou, G. Ma, D. Wu, and Y. Yuan,
“Wasserstein distance based deep adversarial transfer
learning for intelligent fault diagnosis with unlabeled or
insufficient labeled data,” Neurocomputing, vol. 409, pp.
35–45, 2020. doi: 10.1016/j.neucom.2020.05.040.
[31] H. Al-Sahaf et al., “A survey on evolutionary ma-
chine learning,” J. Roy. Soc. N. Z., vol. 49, no. 2, pp.
205–228, 2019. doi: 10.1080/03036758.2019.1609052.
[32] W. Deng, J. Xu, X.-Z. Gao, and H. Zhao, “An en-
hanced MSIQDE algorithm with novel multiple strate-
gies for global optimization problems,” IEEE Trans. Syst.
Man. C., 2020.
[33] W. Deng, J. Xu, Y. Song, and H. Zhao, “Differen-
tial evolution algorithm with wavelet basis function
and optimal mutation strategy for complex optimization
problem,” Appl. Soft Comput., 2020.
[34] Y. Song et al., “MPPCEDE: Multi-population paral-
lel co-evolutionary differential evolution for parameter
optimization,” Energ. Convers. Manage., vol. 228, 2021.
[35] K. Krawiec, “Genetic programming-based construc-
tion of features for machine learning and knowledge dis-
covery tasks,” GPEM, vol. 3, no. 4, pp. 329–343, 2002.
[36] L. Guo, D. Rivero, J. Dorado, C. R. Munteanu, and
A. Pazos, “Automatic feature extraction using genetic
programming: An application to epileptic EEG clas-
sification,” Expert Syst. Appl., vol. 38, no. 8, pp. 10,425–
10,436, 2011.
[37] Y. Bi, B. Xue, and M. Zhang, “Genetic program-
ming with image-related operators and a flexible program
structure for feature learning in image classification,”
93
IEEE Trans. Evol. Comput., vol. 25, no. 1, pp. 87–101,
2021. doi: 10.1109/TEVC.2020.3002229.
[38] M. W. Aslam, Z. Zhu, and A. K. Nandi, “Automatic
modulation classification using combination of genetic
programming and KNN,” IEEE Trans. Wirel. Com-
mun., vol. 11, no. 8, pp. 2742–2750, 2012. doi: 10.1109/
TWC.2012.060412.110460.
[39] Z. Gao, C. Cecati, and S. X. Ding, “A survey of fault
diagnosis and fault-tolerant techniques—Part I: Fault di-
agnosis with model-based and signal-based approaches,”
IEEE Trans. Ind. Electron., vol. 62, no. 6, pp. 3757–3767,
2015. doi: 10.1109/TIE.2015.2417501.
[40] J. R. Koza, Genetic Programming: On the Programming
of Computers by Means of Natural Selection. Cambridge,
MA: MIT Press, 1992.
[41] F. E. Otero, M. M. Silva, A. A. Freitas, and J. C.
Nievola, “Genetic programming for attribute construc-
tion in data mining,” in Proc. EuroGP. Berlin: Springer-
Verlag, 2003, pp. 384–393.
[42] M. Muharram and G. D. Smith, “Evolutionary
constructive induction,” IEEE Trans. Knowl. Data Eng.,
vol. 17, no. 11, pp. 1518–1528, 2005. doi: 10.1109/
TKDE.2005.182.
[43] H. Guo and A. K. Nandi, “Breast cancer diagnosis
using genetic programming generated feature,” Pattern
Recogn., vol. 39, no. 5, pp. 980–987, 2006. doi: 10.1016/j.
patcog.2005.10.001.
[44] H. Guo, Q. Zhang, and A. K. Nandi, “Feature ex-
traction and dimensionality reduction by genetic pro-
Publication Spotlight (continued from page 7)
by O. A. Ibrahim, J. M. Keller, and J. C.
Bezdek, IEEE Transactions on Emerging
Topics in Computational Intelligence, Vol.
5, No. 2, April 2021, pp. 262–273.
Digital Object Identifier: 10.1109/
TETCI.2019.2909521
“Dunn’s internal cluster validity
index is used to assess partition quality
and identify a “best” crisp c-partition of
n objects built from static data sets. This
index is quite sensitive to inliers and
outliers in the input data, so a subse-
quent study developed a family of 17
generalized Dunn’s indices that extend
and improve the original measure in
various ways. This paper presents online
versions of two modified generalized
Dunn’s indices that can be used for the
dynamic evaluation of an evolving (clus-
ter) structure in streaming data. We
argue that this method is a good way to
monitor the ongoing performance of
streaming clustering algorithms, and we
illustrate several types of inferences that
can be drawn from such indices. Stream-
ing clustering algorithms are incremen-
tal, process incoming data points only
once and then discard them, adapt as the
94 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
gramming based on the fisher criterion,” Expert Syst.,
vol. 25, no. 5, pp. 444–459, 2008. doi: 10.1111/j.1468
-0394.2008.00451.x.
[45] K. Neshatian, M. Zhang, and P. Andreae, “A filter
approach to multiple feature construction for symbolic
learning classifiers using genetic programming,” IEEE
Trans. Evol. Comput., vol. 16, no. 5, pp. 645–661, 2012.
doi: 10.1109/TEVC.2011.2166158.
[46] B. Tran, B. Xue, and M. Zhang, “Genetic pro-
gramming for multiple-feature construction on high-
dimensional classification,” Pattern Recogn., vol. 93, pp.
404–417, 2019. doi: 10.1016/j.patcog.2019.05.006.
[47] J. Ma and G. Teng, “A hybrid multiple feature con-
struction approach for classification using genetic pro-
gramming,” Appl. Soft Comput., vol. 80, pp. 687–699,
2019. doi: 10.1016/j.asoc.2019.04.039.
[48] L. Zhang, L. B. Jack, and A. K. Nandi, “Fault de-
tection using genetic programming,” Mech. Syst. Signal
Process., vol. 19, no. 2, pp. 271–289, 2005. doi: 10.1016/j.
ymssp.2004.03.002.
[49] H. Guo, L. B. Jack, and A. K. Nandi, “Feature gen-
eration using genetic programming with application to
fault classification,” IEEE Trans. Syst., Man, Cybern. C,
Cybern., vol. 35, no. 1, pp. 89–99, 2005. doi: 10.1109/
TSMCB.2004.841426.
[50] J. Xuan, H. Jiang, T. Shi, and G. Liao, “Gear fault
classification using genetic programming and support vec-
tor machines,” Int. J. Inf. Technol., vol. 11, no. 9, pp. 18–27,
2005.
data stream evolves, flag outliers, and
most importantly, spawn new emerging
structures. We compare the two new
indices to the incremental Xie-Beni and
Davies-Boudin indices, which to our
knowledge offer the only comparable
approach, with numerical examples on a
variety of synthetic and real datasets.”
IEEE Transactions on Artificial
Intelligence
Procedural Memory Augmented Deep
Reinforcement Learning, by Y. Ma, J.
Brooks, H. Li, and J. C. Principe, IEEE
Transactions on Artificial Intelligence, Vol.
1, No. 2, Oct 2020, pp. 105–120.
Digital Object Identifier: 10.1109/
TAI.2021.3054722
“Inspired by the human brain, we pro-
pose an external memory-augmented
decision-making architecture for video
processing. A self-organizing object detec-
tor is employed as a frontend to decon-
struct the environment. This is done by
extracting events from the flow of time
and detecting objects within the frames.
By employing an extra working memory
[51] D. J. Montana, “Strongly typed genetic program-
ming,” Evol. Comput., vol. 3, no. 2, pp. 199–230, 1995.
doi: 10.1162/evco.1995.3.2.199.
[52] Y. Lei, B. Yang, X. Jiang, F. Jia, N. Li, and A. K.
Nandi, “Applications of machine learning to machine
fault diagnosis: A review and roadmap,” Mech. Syst. Signal
Process., vol. 138, 2020.
[53] X. Dong, Z. Yu, W. Cao, Y. Shi, and Q. Ma, “A survey
on ensemble learning,” Front. Comput. Sci., pp. 1–18, 2020.
[54] “Case western reserve university bearing data cen-
ter.” Accessed: Apr. 2018. [Online]. http://csegroups
.case.edu/bearingdatacenter/home/
[55] B. Wang, Y. Lei, N. Li, and N. Li, “A hybrid prog-
nostics approach for estimating remaining useful life of
rolling element bearings,” IEEE Trans. Reliab., vol. 69, no.
1, pp. 401–412, 2020. doi: 10.1109/TR.2018.2882682.
[56] Y. Bi, B. Xue, and M. Zhang, “An effective feature
learning approach using genetic programming with im-
age descriptors for image classification [research fron-
tier],” IEEE Comput. Intell. Mag., vol. 15, no. 2, pp. 65–
77, 2020. doi: 10.1109/MCI.2020.2976186.
[57] F.-A. Fortin et al., “DEAP: Evolutionary algorithms
made easy,” J. Mach. Learn. Res., vol. 13, pp. 2171–2175, 2012.
[58] F. Pedregosa et al., “Scikit-learn: Machine learning in
Python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.
[59] L. v. d. Maaten and G. Hinton, “Visualizing data using T-
SNE,” J. Mach. Learn. Res., vol. 9, pp. 2579–2605, Nov. 2008.

where objects are temporarily stored, the
system can extract properties of the stored
objects related to the task. We propose a
deep reinforcement learning (RL) neural
network to learn affordances, i.e., a
sequence of actions to manipulate these
objects. The RL network and object
detector are trained alternatively. After
both the network and detector are
trained, the objects and their affordances
are transferred to an external memory.
They are then utilized when the same
objects are detected in input frames. Here,
we use a combination of a dictionary and
a linked list for the external memory that
can be accessed by either content or tem-
poral order. This dual access is motivated
by the temporal property of human pro-
cedural memory. The proposed memory-
augmented RL framework br ings
advantages of transferability, explainability
and computational efficiency with respect
to conventional deep learning architec-
tures. We validate the framework on the
video game Super Mario Brothers to
show superiority to some classical deep
RL architectures and exemplify these
three advantages.”

 We have 30
million reasons to
be proud.
Thanks to our donors, supporters and volunteers who answered the call of the
Realize the Full Potential of IEEE Campaign,
helping impact lives around the world through the power of
technology and education.

Illuminate Educate Engage Energize

Realize Your Impact

Learn how: ieeefoundation.org/campaign
Conference
Calendar
T 6th South-East Europe Design
* Denotes a CIS-Sponsored Conference
Automation, Computer Engineering,
T Denotes a CIS Technical
Computer Networks and Social Media
Co-Sponsored Conference
Conference (SEEDA CECNSM 2021)
September 24–26, 2021
* 2021 IEEE Conference on Games Place: Preveza, Greece
(IEEE CoG 2021) General Co-Chairs: Markos G.Tsipouras,
August 17–20, 2021 Alexandros T.Tzallas, Michael F. Dossis
Place: Virtual (Copenhagen, Denmark) Website: https://seeda2021.uowm.gr/
General Co-Chairs: Miguel Sicart &
Paolo Burelli * 2021 IEEE International
Website: https://ieee-cog.org/2021/ Conference on Data Science
index.html and Advanced Analytics
(IEEE DSAA 2021)
T 2nd International Conference October 6–9, 2021
on Industrial Artificial Intelligence Place: Porto, Portugal
(IAI 2021) General Co-Chairs: João Gama &
August 20–22, 2021 Francisco Herrera
Place: Shenyang, China Website: https://dsaa2021.dcc.fc.up.pt/
General Chair: Yaochu Jin
Website: http://conf.kzgc.com.cn/ * 2021 IEEE Conference on
iai2021 Computational Intelligence in
Bioinformatics and Computational
* 2021 IEEE International Biology (IEEE CIBCB 2021)
Conference on Development October 13–15, 2021
and Learning (IEEE ICDL 2021) Place: Melbourne, Australia
August 23–26, 2021 General Chairs: Madhu Chetty
Place: Beijing, China Website: http://federation.edu.au/cibcb2021
General Co-Chairs: Dingsheng Luo and
Angelo Cangelosi * 2021 IEEE Smart World
Website: https://icdl-2021.org Conference (IEEE SWC 2021)
October 18–21, 2021
T 2021 International Conference Place: Atlanta, USA
on Emerging Techniques in General Co-Chairs:Yi Pan, Rajshekhar
Computational Intelligence Sunderraman,Yanqing Zhang
(ICETCI 2021) Website: https://grid.cs.gsu.edu/
August 25–27, 2021 ~smartworld2021/index.php
Place: Hyderabad, India
General Chair: Atul Negi, Naresh T 5th International Conference
Mallenahalli, Akira Hirose on Intelligent Computing in
Website: http://ietcint.com Data Sciences (ICDS 2021)
October 20–22, 2021
Place: Fez, Morocco
General Chairs: Cesare Alippi, Ahmed
Azough,Youness Oubenaalla
Digital Object Identifier 10.1109/MCI.2021.3084496 Website: http://www.researchnetwork
Date of current version: 15 July 2021 .ma/icds2021/
96 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
Marley Vellasco
Pontifícia Universidade
Católica do Rio de Janeiro,
BRAZIL
T 5th Asian Conference on
Artificial Intelligence (ACAIT 2021)
October 29–31, 2021
Place: Haikou, China
General Chairs: Qionghai Dai, Cesare
Alippi, Jong-Hwan Kim
Website: http://www.acait.cn/
T 2021 International Conference
on Process Mining (ICPM 2021)
October 31–November 4, 2021
Place: Eindhoven, The Netherlands
General Co-Chairs: Boudewijn
van Dongen
Website: https://icpmconference.org/
2021/
* 2021 IEEE Latin American
Conference on Computational
Intelligence (IEEE LA-CCI 2021)
November 2–4, 2021
Place: Temuco, Chile
General Co-Chairs: Millaray Curilem,
Doris Saez and Nelson Aros
Website: http://la-cci.org/
T International Workshop on
Semantic and Social Media
Adaptation and Personalization
(SMAP 2021)
November 4–5, 2021
Place: Virtual
General Chair: Phivos Mylonas
Website: https://hilab.di.ionio.gr/
smap2021/
* 2021 IEEE Symposium
Series on Computational
Intelligence (IEEE SSCI 2021)
December 5–8, 2021
Place: Orlando, FL, USA
General Co-Chairs: Sanaz Mostaghim
and Keeley Crockett
Website: https://attend.ieee.org/
ssci-2021/

Share Your
Preprint Research
with the World!

TechRxiv is a free preprint server for unpublished

research in electrical engineering, computer
science, and related technology. Powered by
IEEE, TechRxiv provides researchers across a
broad range of fields the opportunity to share
early results of their work ahead of formal
peer review and publication.

BENEFITS:
• Rapidly disseminate your research findings

• Gather feedback from fellow researchers

• Find potential collaborators in the

scientific community

• Establish the precedence of a discovery

• Document research results in advance

of publication

Upload your unpublished research today!

Follow @TechRxiv_org
Learn more techrxiv.org Powered by IEEE
 IEEE Computational Intelligence Society Publications
IEEE
Transactions
on
 IEEE
Transactions
on
Evolutionary
Neural
Networks
and
Learning
Systems
Editor-in-Chief:

Haibo
He
Impact Factor: 8.793
Eigenfactor: 0.04821
Article Influence Score: 2.584
Ranking (JCR 2019):
#3 Computer Science—Theory & Methods
#3 Computer Science—Hardware & Architecture
IEEE
Computational
Intelligence
Magazine
Editor-in-Chief:

Chuan-Kang
Ting
Impact Factor: 9.083
Eigenfactor: 0.00277
Article Influence Score: 2.468
Ranking (JCR 2019):
#9 Computer Science—Artificial Intelligence
IEEE
Transactions
on
 IEEE
Press
Books
Cognitive
and
Developmental
Systems
Editor-in-Chief:

Yaochu
Jin
Impact Factor: 2.667
Eigenfactor: 0.00123
Article Influence Score: 0.694
Focuses on advances in the study of
development and cognition in natural
(humans, animals) and artificial (robots,
agents) systems.
Additionally,
IEEE
CIS
technically
co-sponsors
the
following
other
titles;
The
IEEE
Transactions
on
Smart
Grid,
the
IEEE
Transactions
on
Big
Data,
the
IEEE
Transactions
on
Nanobioscience,
the
IEEE
Transactions
on
Information
Forensics
and
Security,
and
the
IEEE
Transactions
on
Affective
Computing.
Digital Object Identifier 10.1109/MCI.2021.3084500


IEEE
Transactions
on
Fuzzy
Systems
Computation
Editor-in-Chief:

Jonathan
Garibaldi Editor-in-Chief:

Carlos
A.
Coello
Coello
Impact Factor: 9.518 Impact Factor: 11.169
Eigenfactor: 0.02514 Eigenfactor: 0.0117
Article Influence Score: 2.203 Article Influence Score: 3.059
Ranking (JCR 2019): Ranking (JCR 2019):
#7 Computer Science—Artificial Intelligence #2 Computer Science—Theory & Methods
#3 Computer Science—Artificial Intelligence
#10 Computer Science—Electrical and Electronic Engineering
IEEE
Transactions
on
Emerging
Topics
in

IEEE
Transactions
on
Artificial
Intelligence
Computational
Intelligence
Editor-in-Chief:

Yew
Soon
Ong Editor-in-Chief:

Hussein
Abbass
Indexing: Scopus Sole transactions for all AI topics.
Uniquely communicate AI benefits for every
Publishes original articles on emerging
paper using published open-access impact
aspects of computational intelligence,
statements.
including theory, applications, and surveys.
DBLB indexed.
IEEE
Transactions
on

Games
Computational
Intelligence
Series
Editor-in-Chief:

Julian
Togelius Editor-in-Chief:

David
Fogel
Impact Factor: 1.886
CIS
Liaison:

Alice
Smith
Eigenfactor: 0.00019
Article Influence Score: 0.449 The IEEE Press Series on Computational
Publishes original high-quality articles Intelligence includes books on neural, fuzzy,
covering scientific, technical, and and evolutionary computation, and related
engineering aspects of games. technologies, of interest to the engineering
and scientific communities.