Evaluating EEG_EMG Fusion for Wearable Upper-Limb Rehabilitation Robots

Western University
Scholarship@Western
Electronic Thesis and Dissertation Repository
8-17-2023 10:30 AM
Evaluating EEG–EMG Fusion-Based Classification as a Method for

Improving Control of Wearable Robotic Devices for Upper-Limb
Rehabilitation
Jacob G. Tryon, Western University
Supervisor: Trejos, Ana Luisa, The University of Western Ontario

A thesis submitted in partial fulfillment of the requirements for the Doctor of Philosophy degree
in Biomedical Engineering
© Jacob G. Tryon 2023
Follow this and additional works at: https://ir.lib.uwo.ca/etd
Part of the Biomedical Commons, Biomedical Devices and Instrumentation Commons, Controls and
Control Theory Commons, Electro-Mechanical Systems Commons, Robotics Commons, Signal
Processing Commons, and the Systems Engineering Commons
Recommended Citation
Tryon, Jacob G., "Evaluating EEG–EMG Fusion-Based Classification as a Method for Improving Control of
Wearable Robotic Devices for Upper-Limb Rehabilitation" (2023). Electronic Thesis and Dissertation
Repository. 9591.
https://ir.lib.uwo.ca/etd/9591
This Dissertation/Thesis is brought to you for free and open access by Scholarship@Western. It has been accepted
for inclusion in Electronic Thesis and Dissertation Repository by an authorized administrator of
Scholarship@Western. For more information, please contact wlswadmin@uwo.ca.
Abstract
Musculoskeletal disorders are the biggest cause of disability worldwide, and wearable mecha-
tronic rehabilitation devices have been proposed for treatment. However, before widespread adop-
tion, improvements in user control and system adaptability are required. User intention should be
detected intuitively, and user-induced changes in system dynamics should be unobtrusively identi-
fied and corrected. Developments often focus on model-dependent nonlinear control theory, which
is challenging to implement for wearable devices.
One alternative is to incorporate bioelectrical signal-based machine learning into the system,
allowing for simpler controller designs to be augmented by supplemental brain (electroencephalog-
raphy/EEG) and muscle (electromyography/EMG) information. To extract user intention better,
sensor fusion techniques have been proposed to combine EEG and EMG; however, further devel-
opment is required to enhance the capabilities of EEG–EMG fusion beyond basic motion classifi-
cation. To this end, the goals of this thesis were to investigate expanded methods of EEG–EMG
fusion and to develop a novel control system based on the incorporation of EEG–EMG fusion
classifiers.
A dataset of EEG and EMG signals were collected during dynamic elbow flexion–extension
motions and used to develop EEG–EMG fusion models to classify task weight, as well as motion
intention. A variety of fusion methods were investigated, such as a Weighted Average decision-
level fusion (83.01 ± 6.04% accuracy) and Convolutional Neural Network-based input-level fusion
(81.57 ± 7.11% accuracy), demonstrating that EEG–EMG fusion can classify more indirect tasks.
A novel control system, referred to as a Task Weight Selective Controller (TWSC), was imple-
mented using a Gain Scheduling-based approach, dictated by external load estimations from an
EEG–EMG fusion classifier. To improve system stability, classifier prediction debouncing was also
proposed to reduce misclassifications through filtering. Performance of the TWSC was evaluated
using a developed upper-limb brace simulator. Due to simulator limitations, no significant differ-
ence in error was observed between the TWSC and PID control. However, results did demonstrate
i
ABSTRACT ii
the feasibility of prediction debouncing, showing it provided smoother device motion. Continued
development of the TWSC, and EEG–EMG fusion techniques will ultimately result in wearable
devices that are able to adapt to changing loads more effectively, serving to improve the user
experience during operation.
Index terms— Adaptive control, convolutional neural networks, electroencephalography,
EEG, electromyography, EMG, gain scheduling, human–machine interfaces, machine learning, sen-
sor fusion, upper-limb rehabilitation, wearable robotics.
Lay Summary
Worldwide, many people suffer from medical conditions or injuries that limit their ability to
move. This lowers their quality of life, as activities of daily living become a challenge. To help
patients regain their ability to move, researchers have developed wearable robotic braces that use
built-in motors to help someone move when their muscles are too weak. While early results are
promising, these devices require more improvements in the way they are controlled before they
can be widely utilized. Operating the wearable robot should feel like natural movement and the
device should work reliably, regardless of the task being performed.
To meet these objectives, researchers have developed systems that can utilize brain or muscle
activity to know when/how a person wants to move. Using sensors placed on the skin, electri-
cal signals generated by the brain (electroencephalography/EEG) and occurring inside muscles
(electromyography/EMG) can be measured to determine that a person is thinking about moving
(EEG) and is trying to move (EMG). Typically, wearable robotic devices only use one signal type;
however, recent work has shown that combing EEG and EMG (EEG–EMG fusion) can improve
accuracy for simple tasks. Further research is required to determine new techniques of integrating
EEG–EMG fusion into device control.
Therefore, the goals of this thesis were to investigate methods of using EEG–EMG fusion
to determine when/how a person is trying to move and to develop techniques to utilize this
information when controlling a wearable robotic brace. To accomplish this, EEG and EMG signals
were recorded during elbow motions, and machine learning was used to train/evaluate various
EEG–EMG fusion models to detect the weight held during movement. A control system was
developed that can modify its calibration settings based on the output of these models, providing
the ability to intelligently adapt to weight variations. This work demonstrated that EEG–EMG
fusion can successfully detect movement information and it developed a method to utilize this
information for adaptable device control. These results begin to address the limitations preventing
widespread use of wearable robotic devices, moving towards a future where they routinely help
people improve their quality of life.
iii
Statement of Co-Authorship
The work presented herein has been written by Jacob Tryon under the supervision of Dr. Ana
Luisa Trejos, who has secured the funding and provided guidance for the development of these
projects. Four publications were chosen to form a portion of the main body of the thesis—two
articles published in peer reviewed journals, one article published in a conference proceeding, and
one book chapter published in a peer-reviewed collection. The extent of the collaboration of the
co-authors is listed below.
Chapter 2. Bioelectrical Signal Processing, Classification, and Fusion
Book Chapter: Interpreting Bioelectrical Signals for Control of Wearable Mechatronic De-
vices
Current Status: Published as Chapter 5 of the book Human–Robot Interaction: Control,
Analysis, and Design. Cambridge Scholars Publishing, 2020.
Tyler Desplenter: First Author, developed the chapter outline, wrote sections related to
EMG processing and modelling, as well as introductory and concluding discussion, edited and
corrected the manuscript.
Jacob Tryon: Co-author, wrote sections related to EEG processing and EEG–EMG fusion
(only these sections are used in this work), and edited and corrected the manuscript.
Emma Farago: Co-author, wrote section discussing to muscle health modelling using EMG.
Taylor Stanbury: Co-author, wrote section discussing to EMG classification.
Dr. Ana Luisa Trejos: Corresponding author, supervised the development of the chapter,
and edited and corrected the manuscript.
Chapter 3. Comparing EEG–EMG Fusion Methods for Motion Classification
Paper: Performance Evaluation of EEG–EMG Fusion Methods for Motion Classification

Current Status: Published in the Proceedings of the IEEE International Conference on
Rehabilitation Robotics (ICORR), Toronto, Canada, June 24–28, 2019
iv
Statement of Co-Authorship v
Jacob Tryon: First Author, developed the bioelectrical signal measurement protocol, collected
experimental data, implemented EEG–EMG fusion classifier models, analyzed the results, and
wrote the manuscript.
Dr. Evan Friedman: Co-author, provided insight on implementation of the bioelectrical
signal measurement protocol, supplied all measurement equipment.
Dr. Ana Luisa Trejos: Corresponding author, supervised development/implementation of
the bioelectrical signal measurement protocol, supervised model development and data analysis,
and edited and corrected the manuscript.
Chapter 4. Task Weight Classification Using EEG–EMG Fusion
Paper: Classification of Task Weight During Dynamic Motion Using EEG–EMG Fusion
Current Status: Published in IEEE Sensors Journal, 2021
Jacob Tryon: First Author, contributed to conception and design of the study, performed
data collection, model development, results analysis, and wrote the manuscript.
Dr. Ana Luisa Trejos: Corresponding author, contributed to conception and design of the
study, supervised project execution, and edited and corrected the manuscript.
Chapter 5. Evaluating Convolutional Neural Networks for Input-Level Fusion
Paper: Evaluating Convolutional Neural Networks as a Method of EEG–EMG Fusion

Current Status: Published in Frontiers in Neurorobotics, 2021
Jacob Tryon: First Author, contributed to conception and design of the study, performed
data collection, model development, results analysis, and wrote the manuscript.
Dr. Ana Luisa Trejos: Corresponding author, contributed to conception and design of the
study, supervised project execution, and edited and corrected the manuscript.
Dedicated to:
Alexandra, for your endless love and support.
vi
Acknowledgements
There are many people I need to thank who have helped me throughout my academic career. First
and foremost, I would like to thank my supervisor, Dr. Ana Luisa Trejos, for the immeasurable
support she has given me during my time at the Wearable Biomechatronics Laboratory. I never
thought that I would end up pursuing a PhD, and I do not think I would have even tried with
any other supervisor. I will always be grateful for the incredible opportunity she gave me, and
for the constant support and encouragement she provided that made the completion of this thesis
possible.
I would also like to acknowledge other mentors who have guided me over the years. Thank
you to my advisory committee, Dr. Ilia Polushin and Dr. Andrew Pruszynski, for their valuable
insights and critique of my work. As well, I would like to give a very big thank you to Eugen
Porter for teaching me so much about robots and electronics. I can confidentially say that I have
learned more about engineering from him than any class I have ever taken.
Of course, my work would not have been possible without support from our sources of funding.
This research was funded by the Natural Sciences and Engineering Research Council (NSERC)
of Canada under grant RGPIN-2020-05648, by the Canadian Foundation for Innovation (CFI),
by the Ontario Research Fund (ORF), by the Ontario Ministry of Economic Development, Trade
and Employment, and by the Ontario Ministry of Research and Innovation through the Early
Researcher Award. I would also like to thank our industry collaborator, Dr. Evan Friedman from
Intronix Technologies, for supplying all of the measurement hardware and supplies used for data
collection. His generosity made much of this research possible in the first place.
I have been lucky enough to have many incredible colleagues at the lab who have supported me
vii
ACKNOWLEDGEMENTS viii
in many ways. Thank you to José Guillermo Colli Alfaro and Dr. Anas Ibrahim for teaching me
so much about machine learning over the years. Their willingness to always volunteer their time
and knowledge gave my work a huge head-start, and I know I would have wasted a lot of time if I
tried to figure everything out on my own. I would also like to thank Tyler Desplenter for teaching
me many things about the research process and for all the advice he gave me when I was a new
member of the lab. As well, thank you to Alex Ouellet for all the work he did manually analyzing
the video recordings from my data collection trials to record timestamps. I know this was a very
tedious undertaking, but it was an important resource that was crucial for the development of the
wearable robotic brace simulator. Thank you to all the members of the WearME Lab, past and
present. I feel very fortunate to have been able to work with so many wonderful people during my
time there.
I must also thank all my family and friends who have been a great source of encouragement
as I worked towards the completion of this thesis. Thank you to my parents, Robert and Nancey
Tryon, as well as my parents-in-law, Dennis and Janice Mayberry, for all of their support.
Most importantly, I want to give the biggest thanks possible to my incredible wife Alexandra,
our son, Maximillian, our cats Triscuit and Gandalf, and our dog Timber. Without their love
and support I know I never would have made it to the end of this long journey and I would be
completely lost without them. I will be eternally grateful to Alexandra for the unbelievable amount
of patience she shows me on a daily basis and for always pushing me to believe in myself. Without
her I never would have attempted a PhD and her willingness to support us through my schooling
is the only reason it was even possible. Everything great in my life I owe to her and words cannot
properly express how much I love and appreciate her. Thank you Ali for everything you do.
Contents
Abstract i
Lay Summary iii
Statement of Co-Authorship iv
Acknowledgements vii
Table of Contents ix
List of Figures xvi
List of Tables xxi
Nomenclature and Acronyms xxiii
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Wearable Robotic Devices for Rehabilitation . . . . . . . . . . . . . . . . . 2
1.1.2 Wearable Device Control and Adaptability . . . . . . . . . . . . . . . . . . 4
1.2 General Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Overview of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Bioelectrical Signal Processing, Classification, and Fusion 10
ix
CONTENTS x
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 Signal Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.1 Wet and Dry Electrodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.2 Signal Transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.3 Electrode Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.4 Electrode Placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.5 Electrode Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3 Signal Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.1 EMG Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.2 EEG Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.4 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.4.1 Segmentation and Windowing . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.4.2 EMG Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.4.3 EEG Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.5 Machine-Learning Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.5.1 Support Vector Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.5.2 Convolutional Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.6 EEG–EMG Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.6.1 Classifier-based Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.6.1.1 Input-Level Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.6.1.2 Decision-Level Fusion . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.6.2 Fusing EEG and EMG with Other Biosignals . . . . . . . . . . . . . . . . . 59
3 Comparing EEG–EMG Fusion Methods for Motion Classification 61

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.2.1 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.2.1.1 Speed/Load Varying Motions . . . . . . . . . . . . . . . . . . . . . 63
3.2.1.2 Muscle Fatiguing Motions . . . . . . . . . . . . . . . . . . . . . . . 63
CONTENTS xi
3.2.2 Signal Acquisition and Processing . . . . . . . . . . . . . . . . . . . . . . . 64

3.2.3 Feature Extraction and Classification . . . . . . . . . . . . . . . . . . . . . 65
3.2.4 Fusion Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.2.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.3.1 Overall Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.3.2 Speed/Load Varying Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.3.3 Muscle Fatiguing Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4 Task Weight Classification Using EEG–EMG Fusion 76

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.2.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.2.2 Signal Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.3 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.2.4 Model Training and Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.2.5 Model Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4.3.2 Speed–Weight Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.3.3 Classifier Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.3.4 Feature Selection Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5 Evaluating Convolutional Neural Networks for Input-Level Fusion 100

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
CONTENTS xii
5.2.1 Data Collection and Signal Processing . . . . . . . . . . . . . . . . . . . . . 104

5.2.2 Image Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.2.2.1 Spectrogram Images . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.2.2.2 Signal Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.2.2.3 Qualitative Image Response . . . . . . . . . . . . . . . . . . . . . 111
5.2.3 CNN Model Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.2.3.1 Spectrogram CNN Models . . . . . . . . . . . . . . . . . . . . . . 115
5.2.3.2 Signal CNN Models . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.2.4 Model Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.3.1 Model Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.3.2 Speed-Specific Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.3.3 Classifier Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
6 Investigating the Effect of Image Normalization on CNN-Based Fusion 134

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
6.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.2.1 Signal Processing and Model Training . . . . . . . . . . . . . . . . . . . . . 139
6.2.2 Normalization Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.2.2.1 Subject-wise Normalization . . . . . . . . . . . . . . . . . . . . . . 143
6.2.2.2 Speed-wise Normalization . . . . . . . . . . . . . . . . . . . . . . . 144
6.2.2.3 Image-wise Normalization . . . . . . . . . . . . . . . . . . . . . . . 145
6.2.2.4 Channel-wise Normalization . . . . . . . . . . . . . . . . . . . . . 146
6.2.2.5 No Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.2.3 Qualitative Effect of Normalization . . . . . . . . . . . . . . . . . . . . . . . 148
6.2.4 Evaluation of Normalization Methods . . . . . . . . . . . . . . . . . . . . . 151
6.2.4.1 Overall Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
CONTENTS xiii
6.2.4.2 Speed-Specific Accuracy . . . . . . . . . . . . . . . . . . . . . . . . 153

6.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
6.3.1.1 Grouped Spectrogram Images . . . . . . . . . . . . . . . . . . . . 156
6.3.1.2 Stacked Spectrogram Images . . . . . . . . . . . . . . . . . . . . . 156
6.3.1.3 Signal 1D Convolution Images . . . . . . . . . . . . . . . . . . . . 157
6.3.2 Speed-Specific Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
6.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.4.1 Comparing Overall Accuracy of Normalization Methods . . . . . . . . . . . 162
6.4.2 Performance of Normalization Methods While Varying Motion Speed . . . . 165
6.4.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
7 Development of a Task Weight Selective Control System 174

7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
7.2 Project Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7.3 Upper-Limb Brace Simulator Development . . . . . . . . . . . . . . . . . . . . . . . 182
7.3.1 General System Torques Model . . . . . . . . . . . . . . . . . . . . . . . . . 183
7.3.2 Brace Actuator Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
7.3.2.1 Modelling the DC Motor and Gearhead . . . . . . . . . . . . . . . 190
7.3.2.2 Modelling Actuator Non-linearities . . . . . . . . . . . . . . . . . . 193
7.3.3 Actuator Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
7.3.4 Modelling Load Inertias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
7.3.4.1 Inertia of the Subject’s Arm . . . . . . . . . . . . . . . . . . . . . 210
7.3.4.2 Inertia of the Dumbbells . . . . . . . . . . . . . . . . . . . . . . . 211
7.3.5 Generating Input Motion Trajectories . . . . . . . . . . . . . . . . . . . . . 214
7.3.6 Modelling Dynamic Weight Changes During Motion . . . . . . . . . . . . . 223
7.3.6.1 Variable Inertia Transfer Function Subsystem . . . . . . . . . . . . 225
7.3.6.2 Generating Synthetic Input Motion Data . . . . . . . . . . . . . . 228
CONTENTS xiv
7.3.7 Upper-Limb Brace Performance Metrics . . . . . . . . . . . . . . . . . . . . 233

7.3.7.1 Root Mean Square Error . . . . . . . . . . . . . . . . . . . . . . . 234
7.3.7.2 Normalized Root Mean Square Error . . . . . . . . . . . . . . . . 236
7.3.7.3 Log Dimensionless Jerk . . . . . . . . . . . . . . . . . . . . . . . . 237
7.4 Implementing the Task Weight Selective Controller . . . . . . . . . . . . . . . . . . 242
7.4.1 Gain Scheduling PID Controllers . . . . . . . . . . . . . . . . . . . . . . . . 245
7.4.2 Simulink Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
7.4.3 Generating the Classifier Prediction Control Signal . . . . . . . . . . . . . . 252
7.4.3.1 Calculating Timestamps . . . . . . . . . . . . . . . . . . . . . . . 254
7.4.3.2 Initializing the Prediction Signal . . . . . . . . . . . . . . . . . . . 258
7.4.4 Debouncing the Prediction Signal . . . . . . . . . . . . . . . . . . . . . . . . 261
7.4.4.1 Hold-Delay Debouncing Method . . . . . . . . . . . . . . . . . . . 264
7.4.4.2 Majority-Vote Debouncing Method . . . . . . . . . . . . . . . . . 266
7.4.5 Tuning Task Weight Specific PID Gains . . . . . . . . . . . . . . . . . . . . 270
7.4.5.1 Optimizing Subject-Specific Proportional Gain . . . . . . . . . . . 274
7.4.5.2 Reduced Subject Pool . . . . . . . . . . . . . . . . . . . . . . . . . 281
7.5 Control System Evaluation Procedure . . . . . . . . . . . . . . . . . . . . . . . . . 284
7.5.1 Experiment 1: Task Weight Variation Between Motions . . . . . . . . . . . 285
7.5.2 Experiment 2: Task Weight Variation During Motion . . . . . . . . . . . . 288
7.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
7.6.1 Experiment 1: Task Weight Variation Between Motions . . . . . . . . . . . 291
7.6.1.1 Debounced Classifier Switching Results . . . . . . . . . . . . . . . 291
7.6.1.2 No Switching Results . . . . . . . . . . . . . . . . . . . . . . . . . 297
7.6.2 Experiment 2: Task Weight Variation During Motion . . . . . . . . . . . . 300
7.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
7.7.1 Simulated TWSC Performance . . . . . . . . . . . . . . . . . . . . . . . . . 305
7.7.2 Classifier Prediction Debouncing Effectiveness . . . . . . . . . . . . . . . . . 311
7.7.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
7.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
CONTENTS xv
8 Conclusions and Future Work 320

8.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
8.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
References 331
Appendices 363
A Additional Details for the Upper-Limb Brace Simulator Model 363

A.1 DC Motor and Gearhead Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
A.1.1 Modelling the DC Motor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
A.1.2 Modelling the Gearhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
A.2 Dumbbell Inertia Formula Derivation . . . . . . . . . . . . . . . . . . . . . . . . . . 373
B Permissions and Approvals 381
Curriculum Vitae 387

List of Figures
1.1 Photographs of an example wearable upper-limb mechatronic brace. . . . . . . . . 3
2.1 Example of a recorded EMG Signal. . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.2 Example of a recorded EEG signal. . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Example wet and dry EMG electrodes. . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4 Example wet and dry EEG electrodes. . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5 EMG electrodes placed according to the SENIAM Project guidelines. . . . . . . . . 22
2.6 EEG electrode placed according to the 10–20 International System. . . . . . . . . . 23
2.7 DC Offset removal applied to an EMG signal. . . . . . . . . . . . . . . . . . . . . . 28
2.8 Effect of bandpass filtering on an EMG signal. . . . . . . . . . . . . . . . . . . . . 29
2.9 Example EEG signal corrupted by EMG and ECG. . . . . . . . . . . . . . . . . . . 32
2.10 Effect of bandpass filtering on an EEG signal. . . . . . . . . . . . . . . . . . . . . . 33
2.11 Demonstration of the sliding window technique. . . . . . . . . . . . . . . . . . . . . 37
2.12 Comparing EEG and EMG signal response to motion. . . . . . . . . . . . . . . . . 43
2.13 Visualisation of the Support Vector Machine training procedure. . . . . . . . . . . 48
2.14 Visualisation of the Convolutional Neural Network model structure. . . . . . . . . 53
2.15 Comparing input-level and decision-Level fusion. . . . . . . . . . . . . . . . . . . . 57
3.1 EEG and EMG electrode placement during the motion classification experiment. . 65
3.2 Protocol for implementing the EEG–EMG fusion-based motion classifiers. . . . . . 66
3.3 Motion classification accuracy results for each fusion method across all motion types. 69
xvi
LIST OF FIGURES xvii
3.4 Motion classification accuracy at three weight levels during slow speed movement
for each fusion method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.5 Motion classification accuracy at three weight levels during fast speed movement
for each fusion method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.6 Motion classification accuracy of each fusion method calculated across all trials of
muscle fatiguing movement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.1 Concept for EEG–EMG fusion-based task weight classification. . . . . . . . . . . . 79

4.2 EEG and EMG electrode placement during the task weight classification experiment. 81
4.3 Protocol used to process the bioelectrical signals and implement the EEG–EMG
fusion methods used to classify task weight. . . . . . . . . . . . . . . . . . . . . . . 82
4.4 Mean task weight classification accuracy of each fusion method calculated across all
weights and motion speeds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.5 Mean task weight classification accuracy of each fusion method calculated for each
combination of speed and weight. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.6 Confusion matrices for the task weight classification results of each fusion method. 91
4.7 EEG feature selection results using the ReliefF algorithm . . . . . . . . . . . . . . 92
4.8 EMG feature selection results using the ReliefF algorithm . . . . . . . . . . . . . . 93
5.1 Data processing protocol used to implement the EEG–EMG fusion-based CNN models.107
5.2 Sample normalized spectrogram image to demonstrate the three EEG–EMG fusion
methods used. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.3 Graphical representation of a sample normalized signal image. . . . . . . . . . . . . 112
5.4 Example normalized spectrogram images and graphical representations of sample
normalized signal images for each of the three weight levels. . . . . . . . . . . . . . 113
5.5 Base model configuration used for all three spectrogram CNN model types. . . . . 116
5.6 Base model configurations used for the split convolution and 1D convolution models. 117
5.7 Mean accuracy of all spectrogram and signal based CNN models, calculated across
both speeds and all task weights. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.8 Mean accuracy for all CNN models, separated by the two speed levels. . . . . . . . 122
LIST OF FIGURES xviii
5.9 Confusion matrices for the single-channel spectrogram-based CNN models. . . . . . 123
5.10 Confusion matrices for the multi-channel spectrogram-based CNN models. . . . . . 124
5.11 Confusion matrices for the split convolution signal-image-based CNN models. . . . 125
5.12 Confusion matrices for the 1D convolution signal-image-based CNN models. . . . . 126
6.1 Data processing protocol for developing EEG–EMG CNN models using separate
image-sets and multiple normalization methods. . . . . . . . . . . . . . . . . . . . . 142
6.2 Visual representation of image-wise and channel-wise normalization . . . . . . . . . 147
6.3 Example spectrograms normalized using the various methods. . . . . . . . . . . . . 150
6.4 Mean overall accuracy of each normalization method when applied to grouped spec-
trogram images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
6.5 Mean overall accuracy of each normalization method when applied to stacked spec-
trogram images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
6.6 Mean overall accuracy of each normalization method when applied to signal 1D
convolution images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
6.7 Speed-specific accuracy for the best performing normalization methods when used
alongside EEG–EMG fusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
7.1 Previously developed wearable mechatronic upper-limb brace prototype. . . . . . . 183

7.2 Free-body diagram of the upper-limb brace during use. . . . . . . . . . . . . . . . . 184
7.3 Hand/forearm measurements and COM locations. . . . . . . . . . . . . . . . . . . 188
7.4 Equivalent electrical and mechanical system diagrams modelling a combined DC
motor/gearhead actuator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
7.5 System block diagram of a DC motor combined with a gearhead. . . . . . . . . . . 193
7.6 System block diagram of a combined DC motor/gearhead including actuator non-
linearities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
7.7 Simulink implementation of the actuator plant model. . . . . . . . . . . . . . . . . 209
7.8 Photograph of the dumbbells used during data collection. . . . . . . . . . . . . . . 212
7.9 Photograph of a participant holding the dumbbell vertically. . . . . . . . . . . . . . 213
7.10 Relating the rotation and inertia of the dumbbell centre of mass back to the brace. 213
LIST OF FIGURES xix
7.11 Motion paths with overlaid video and trigger markers. . . . . . . . . . . . . . . . . 217
7.12 Motion trajectories obtained using the quintic polynomial interpolation method. . 224
7.13 Simulink implementation of a variable first-order transfer function subsystem. . . . 227
7.14 Simulink implementation of the modified actuator model to allow for dynamic task
weight variations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
7.15 Motion path from the synthetic dataset with weight change conditions labelled. . . 230
7.16 Motion path with time labels to describe weight change condition timestamp calcu-
lation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
7.17 Weight change signals used to control the Multiport Switch output supplied to
the variable transfer function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
7.18 Block diagram of a generic closed-loop feedback control system. . . . . . . . . . . . 242
7.19 Simulink implementation of the TWSC . . . . . . . . . . . . . . . . . . . . . . . . . 249
7.20 Process for generating classifier prediction control signals. . . . . . . . . . . . . . . 253
7.21 Assumptions made when initializing prediction control signals. . . . . . . . . . . . 260
7.22 Utilizing debouncing to correct a brief misclassification in a prediction control signal.262
7.23 A visual representation of the Hold-Delay debouncing method applied to a predic-
tion control signal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
7.24 Effect of changing delay value in the Hold-Delay debouncing method. . . . . . . . . 267
7.25 A visual representation of the Majority-Vote debouncing method being applied to
a prediction control signal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
7.26 Demonstration of the vote window dynamically resizing during startup in the Majority-
Vote debouncing method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
7.27 Plots showing the relationship between RMSE/LDLJ and the value of Kp . . . . . . 273
7.28 Process diagram for the subject-specific proportional gain optimization procedure. 275
7.29 Plot showing the derivative of RMSE with respect to proportional gain and the
threshold cut-off used during optimization. . . . . . . . . . . . . . . . . . . . . . . 277
7.30 Plot showing how threshold value affects the selection of proportional gain. . . . . 280
7.31 Box plots showing various debouncing methods performance metric results for static
weight trials. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
LIST OF FIGURES xx
7.32 Example measurements of brace output position obtained when using each debounc-
ing method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
7.33 Effect of prediction debouncing on output velocity. . . . . . . . . . . . . . . . . . . 296
7.34 Box plots showing no-switching performance metric results for static weight trials. 298
7.35 Example measurements of brace output position obtained when using no switching
for a static weight trial. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
7.36 Box plots showing performance metric results for 0 lbs to 5 lbs dynamic weight
change trials. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
7.37 Box plots showing performance metric results for 5 lbs to 0 lbs dynamic weight
change trials. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
7.38 Example measurements of brace output position obtained when using manual and
no switching for a dynamic weight change trial. . . . . . . . . . . . . . . . . . . . . 304
7.39 Simulink scope showing the effects of a strong and weak actuator. . . . . . . . . . . 307
7.40 Robotic manipulator configuration diagrams conceptually demonstrating device–
human interaction effects in the simulator. . . . . . . . . . . . . . . . . . . . . . . . 310
7.41 Block diagram showing concept for a new system plant model in an updated simulator.311
A.1 Equivalent electrical and mechanical system diagrams modelling a DC motor. . . . 364
A.2 System block diagram of a DC motor. . . . . . . . . . . . . . . . . . . . . . . . . . 368
A.3 Equivalent mechanical system diagram modelling a DC motor combined with a
gearhead. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370
A.4 System block diagram of a DC motor combined with a gearhead. . . . . . . . . . . 373
A.5 Dumbbell dimension labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
A.6 Cylinder approximation used for the hexagonal dumbbell heads. . . . . . . . . . . 374
A.7 Photograph of a participant holding the dumbbell vertically. . . . . . . . . . . . . . 375
A.8 Relating the rotation and inertia of the dumbbell centre of mass back to the brace. 376
A.9 Dumbbell partitions used during inertia calculations. . . . . . . . . . . . . . . . . . 377
A.10 Axes and dimensions of a cylinder. . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
List of Tables
2.1 EEG frequency bands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.2 EEG–EMG fusion methods used for decision-level fusion. . . . . . . . . . . . . . . 58
3.1 p values obtained from the one-way ANOVA comparing overall motion classification
accuracy of fusion methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2 p values obtained from the one-way ANOVA comparing fusion method motion clas-
sification accuracy during muscle fatiguing movement. . . . . . . . . . . . . . . . . 72
4.1 List of the EEG and EMG features calculated for task weight classification. . . . . 84
5.1 Literature summary of CNNs that use EEG or EMG. . . . . . . . . . . . . . . . . 103

5.2 CNN hyperparameters tuned during optimization, with the range of possible values
used by the Random Search algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.3 p values obtained from pairwise accuracy comparisons of the different CNN based
EEG–EMG fusion methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.1 Mean overall accuracy scores for each normalization method with all image types. 155
6.2 p values obtained from the comparison of normalization methods applied to the
grouped spectrogram images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
6.3 p values obtained from the comparison of normalization methods applied to stacked
spectrogram images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
6.4 p values obtained from the comparison of normalization methods applied to signal
1D convolution images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
xxi
LIST OF TABLES xxii
7.1 Anthropometric data percentages used to approximate body segment mass and
COM distance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
7.2 Chosen BLDC Motor specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . 199
7.3 Chosen Gearhead specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
7.4 Actuator parameter values used in the simulator model. . . . . . . . . . . . . . . . 208
7.5 Step sizes and range of the proportional gain values used during optimization. . . . 276
7.6 Classifier accuracy scores for the reduced subject pool. . . . . . . . . . . . . . . . . 283
7.7 Selected proportional gain values obtained from the subject-specific optimization
procedure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
7.8 Summary of the TWSC simulations run for Experiment 1. . . . . . . . . . . . . . . 287
7.9 Summary of the TWSC simulations run for Experiment 2. . . . . . . . . . . . . . . 290
A.1 Dumbbell dimension measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . 374

Nomenclature and Acronyms
Acronyms
1D One Dimensional
2D Two Dimensional
ADAM Adaptive Moment Estimation
ADC Analog-to-Digital Converter
AC Alternating Current
AND Logical AND Operation
ANOVA Analysis of Variance
ADRC Active Disturbance Rejection Control
AR1 Autoregressive Coefficient One
AR2 Autoregressive Coefficient Two
AR3 Autoregressive Coefficient Three
AR4 Autoregressive Coefficient Four
BBP Beta Band Power
BCI Brain Computer Interface
BLDC Brushless Direct Current
CAD Computer-Aided Design
CNN Convolutional Neural Network
xxiii
NOMENCLATURE AND ACRONYMS xxiv
COM Centre of Mass
COR Centre of Rotation
CSP Common Spatial Pattern
DASDV Difference Absolute Standard Deviation Value
DC Direct Current
DOF Degree Of Freedom
ECG Electrocardiography
ECoG Electrocorticography
EEG Electroencephalography
EMG Electromyography
EOG Electrooculography
ERD Event-Related Desynchronization
ERP Event-Related Potential
ERS Event-Related Synchronization
FC Fully Connected
FD Fractal Dimension
FFT Fast Fourier Transform
FMG Force Myography
fNIRS Functional Near-Infrared Spectroscopy
FPS Frames Per Second
HA Hjorth Parameter: Activity
HC Hjorth Parameter: Complexity
HD Hold-Delay
HM Hjorth Parameter: Mobility
HMI Human–Machine Interface

NOMENCLATURE AND ACRONYMS xxv
ICA Independent Component Analysis
IMU Inertial Measurement Unit
LBBP Lower Beta Band Power
LSTM Long Short-Term Memory
MAE Mean Absolute Error
MAMP Mean Amplitude
MAV Mean Absolute Value
MAVS Mean Absolute Value Slope
MBP Mu Band Power
MDF Median Frequency
MFL Maximum Fractal Length
MI Motor Imagery
MNF Mean Frequency
MR Motion Region
MRCP Movement-related cortical potential
mRMR Minimum Redundancy Maximum Relevance
MSD Musculoskeletal Disorder
MV Majority-Vote
OR Logical OR Operation
PID Proportional Integral Derivative
PMN Point of Maximum Negativity
PSD Power Spectral Density
ReLu Rectified Linear Unit
RGB Red, Green, and Blue
RMS Root Mean Square

NOMENCLATURE AND ACRONYMS xxvi
RMSE Root Mean Square Error
ROG Radius of Gyration
SC Single Classifier
SG Static Gains
SGD Stochastic Gradient Decent
STFT Short-Time Fourier Transform
SVM Support Vector Machine
TWSC Task Weight Selective Controller
UBBP Upper Beta Band Power
WA Weighted Average
WL Window Length
ZC Zero Crossings
Variables
a0 to a5 Quintic polynomial coefficients
ap Auto-regressive coefficients
A Auto-regressive model order
bG Gearhead coefficient of viscous friction
bM Motor coefficient of viscous friction
bMG Combined actuator coefficient of viscous friction
B Total number of frequency bins
COMA Anthropometric percentage to approximate arm (hand+forearm) centre of mass
COMh Anthropometric percentage to approximate hand centre of mass
d Distance from body segment centre of mass to centre of rotation
dA Distance from elbow joint to arm centre of mass

NOMENCLATURE AND ACRONYMS xxvii
dDb Distance from elbow joint to dumbbell centre of mass
d1 Distance from head segment centre of mass to dumbbell centre of mass
e Back electromotive force voltage
err System error
f Bin frequency
FA Arm (hand + forearm) weight force
FDb Dumbbell weight force
g Acceleration due to gravity
h Cylinder height
h Cylinder height
h1 Dumbbell head segment height
h3 Dumbbell handlebar segment height
i Armature current
imax Current saturation limit
iNL No load current
inom Nominal current (max. continuous)
istall Stall current
JA Arm inertia
JDb Dumbbell inertia
JG Gearhead inertia
JL Load inertia
JM Rotor inertia
JRL Load inertia applied to input rotor shaft
Jrot Moment of inertia about the body segment centre of rotation
Jtotal Effective inertia acting on DC motor

NOMENCLATURE AND ACRONYMS xxviii
Jx , Jy Moment of inertia of a cylinder about its x /y axes
J0 Moment of inertia about the centre of mass
J1 , J2 Moment of inertia of dumbbell head segments about their centre of mass
J1 ′ , J2 ′ Moment of inertia of head segments about the dumbbell centre of mass
J3 Moment of inertia of dumbbell handlebar segment
k Number of nearest neighbours evaluated in ReliefF algorithm
Kd Derivative gain
Ke Back emf constant
Ki Integral gain
Kp Proportional gain
Kp,T hresh Proportional gain value at RMSE decrease rate threshold
Kt Torque constant
lf Forearm length
lh Hand length
L Armature inductance
Lphase Terminal inductance phase to phase
m Mass
mA Arm (hand + forearm) mass
mb Self-reported bodyweight
mDb Dumbbell mass
m1 Dumbbell head segment mass
m3 Dumbbell handlebar segment mass
n Gear ratio
N Filter coefficient
NPredict Number of predictions in motion region

NOMENCLATURE AND ACRONYMS xxix
N1 Number of teeth on input gear
N2 Number of teeth on output gear
p Current image pixel value
pmax Maximum image pixel value
pmin Minimum image pixel value
pnorm Normalized image pixel value
P Power spectral density
r Cylinder radius
r1 Dumbbell head segment radius
r3 Dumbbell handlebar segment radius
R Armature resistance
Rphase Terminal resistance phase to phase
s Laplace variable
SMA Anthropometric percentage to approximate arm (hand+forearm) segment mass
t Time
tf Time duration of via pair
ti Via point time (i is index denoting via number)
tMRi,start Motion region start time (i is index denoting region number)
tWi Weight change time (i is index denoting condition number)
T hresh RMSE decrease rate threshold
u Plant transfer function input
v Input armature voltage
vmax Voltage saturation limit
vnom Nominal voltage
V Volume of a cylinder
NOMENCLATURE AND ACRONYMS xxx
V1 , V1 Dumbbell head segment volume
V3 Dumbbell handlebar segment volume
w Current window number
W Total number of window segments in a signal
WL Window length
x Time domain signal
x̄ Mean of time-domain signal x
y Plant transfer function output
∆tMRi,end Final prediction timestamp (i is index denoting motion region number)
∆tMRi,start Initial prediction timestamp (i is index denoting motion region number)
∆tWi Duration to midpoint for weight change (i is index denoting condition number)
ηG Max. gearhead efficiency
θ Angular position
θBkl Backlash
θf End via point angular position
θG Gearhead angular position
θin Brace input angular position
θM Rotor angular position
θmax Angular position saturation safety limits
θout Brace output angular position
θvia Set of motion trajectory via points
θvia,synth Set of synthetic motion trajectory via points
θ0 Starting via point angular position
θ̇, ω Angular velocity
θ̇f End via point angular velocity

NOMENCLATURE AND ACRONYMS xxxi
θ̇G Gearhead angular velocity
θ̇G,nom Max. continuous gearhead input speed (nominal)
θ̇M Rotor angular velocity
θ̇NL No load speed
θ̇0 Starting via point angular velocity
θ̈, α Angular acceleration
θ̈f End via point angular acceleration
θ̈0 Starting via point angular acceleration

...
θ,ζ Angular jerk
ρ Radius of gyration
ρp Proximal measurement of elbow radius of gyration
ρ0 Radius of gyration with respect to centre of mass
τ Torque
τA Torque due to weight of subject’s arm (hand + forearm)
τactuator Nominal output toque of chosen actuator
τDb Torque due to dumbbell weight
τG Output gearhead torque
τG,max Max. intermittent gearhead torque
τG,nom Max. continuous gearhead torque (nominal)
τM Output motor torque
τmax Torque saturation limit
τM,nom Nominal motor torque (max. continuous)
τRL External rotor load torque
τstall Stall torque
ψ White noise
NOMENCLATURE AND ACRONYMS xxxii
ωpeak Peak angular velocity
Units
A Amperes
deg Degrees
◦ Degrees
◦/s Degrees per second
H Henry
Hz Hertz
lbs Pounds
mm Millimetres
rad Radians
rad/s Radians per second
rad/s2 Radians per second squared
s Seconds
V Volts
Chapter 1
Introduction
The musculoskeletal system is a complex structure of muscles, bones, ligaments, tendons, and other
soft tissues that act as a framework to support the body’s weight and to enable movement through
the application of controlled forces supplied by muscle contractions [1]. A healthy and functioning
musculoskeletal system is crucial to maintaining a person’s quality of life, as the inability to
properly move will severely limit one’s independence [1–3]. A Musculoskeletal Disorder (MSD)
is a classification used to describe any medical condition that impairs these parts of the body,
which can originate from many different causes such as falls, workplace/sports injury, arthritis,
lacerations, and surgery [1]. One particularly prevalent source of motor impairments related to
MSD is stroke [2, 3], which can result in long-term disability for up to 50% of survivors [2]. Overall,
MSDs are the largest contributor to disability worldwide [1, 4], and impact the lives of many people
daily. Upper-limb MSDs can be especially limiting, since the ability to properly move your arms
and hands is essential for completing many activities of daily living independently.
1.1 Motivation
The primary treatment method for the symptoms brought on by MSD is rehabilitation through
physiotherapy [3, 5]. A patient will undergo various exercises and stretches while guided by a
physiotherapist to regain lost range of motion and strengthen weakened muscles in the hopes of
restoring their lost motor function. Research has suggested that regular access to a physiotherapist
1
1.1 Motivation 2
can result in better health outcomes for patients [5]; however, this is not always possible due
to various barriers that may limit a patient’s ability to seek treatment [1]. Physiotherapy is
often expensive and time consuming, which can make it an infeasible solution for certain groups
depending on their socio-economic status. Likewise, people living in rural or underdeveloped
communities may not have access to a nearby physiotherapy clinic and may be required to travel
large distances to receive treatment. The nature of many physiotherapy procedures make them
a poor candidate for telehealth, as the therapist often requires direct contact with the patient to
manipulate their body during various stretches, exercises, and assessments. All of these factors
have led to a need for new treatment solutions for patients suffering from MSDs that allow them
to undergo self-led at-home therapy in a safe and clinically effective manner.
1.1.1 Wearable Robotic Devices for Rehabilitation
One emerging technology that has been proposed as new tool to facilitate at-home physiotherapy
is wearable robotic devices for rehabilitation [6–13]. These novel devices are mechatronic systems
constructed in the form factor of an orthotic brace and are worn over a patient’s affected limb.
Often referred to as robotic exoskeletons, the system is comprised of various actuators used to apply
force to the patient’s limb to generate movement, sensors used to detect the wearer’s intention
(allowing them to operate the device), and an embedded system (often microcontroller-based)
to process input/output data and allow for intelligent control of the brace [6, 12, 14]. Wearable
mechatronic braces are typically separated into two categories based on where they are worn on
the body: upper-limb devices or lower-limb devices. Upper-limb systems are used to treat MSD
affecting the arms/hands, and seek to aid with rehabilitation of these body parts and provide
assistance during various activities of living [6, 11–13]. Conversely, lower-limb systems target the
rehabilitation of the legs, and focus on providing assistance to restore a patient’s ability to walk
[7–10]. For an example of a wearable upper-limb mechatronic brace, see Figure 1.1.
Through the utilization of robotic exoskeletons, MSD patients will be able to perform their
physiotherapy exercises in a home setting without requiring frequent trips to a clinic. A wearable
mechatronic brace can be programmed to move their limbs and apply forces the same way that
a physiotherapist does during their treatment; performing the same exercises, but in a self-led
1.1 Motivation 3
(a) Front-side view (b) Back view
Figure 1.1: Photographs showing a front–side view (a) and back view (b) of an example wearable
upper-limb mechatronic brace. This device specifically targets the elbow, and provides
supportive torque to assist with elbow flexion–extension movement. The brace is
secured to the user using cuffs with straps fixed to the wrist and upper arm. Actuation
force is supplied by two motors that can be seen in (b) and is transmitted to the elbow
joint using the pulley system seen in (a). Electromyography electrodes can be seen
attached to the biceps in (a), indicating the brace is controlled using muscle activity
measurements.
manner. There is a growing body of research that has shown that using wearable robotic devices
for rehabilitation has led to favourable treatment outcomes and improved patient recovery [15–21],
demonstrating the feasibility of these new tools. If programmed for more general use, these devices
can also operate for assistance-based tasks; providing extra support torque to patients with partial
limb function to overcome their muscle weakness and allow them to move freely during various
activities of daily living [11, 12, 22]. Finally, these systems also have the potential to not only
perform rehabilitation/assistance functions, but to also conduct patient monitoring. Using the
sensors built into the brace, data about the patient’s recovery can be collected and utilized by a
clinician to tailor their care to them specifically [23]. Overall, wearable robotic devices have shown
great potential as a new healthcare solution that can provide many benefits to MSD patients.
1.1 Motivation 4
1.1.2 Wearable Device Control and Adaptability
Despite the promise that wearable mechatronic devices display, they have yet to be widely adopted
as a treatment practice [6, 11, 12]. This is due to several challenges still facing the design of these
systems due to limitations that arise from the wearable nature of the robotic exoskeletons: namely
that the brace should be fully portable, light, and comfortable to wear so the patient can easily
integrate it into their daily life. The result of these constraints is that available battery power,
computational resources, and device size are severely limited [6, 11, 12, 24, 25], making it hard to
implement conventional mechatronic technologies typically used when designing robotic systems.
Another substantial challenge faced by wearable robotic devices that has prevented their
widespread adoption is the method of user control [6, 11, 13, 26–28]. The user needs to be
able to operate the brace in a way that feels natural and intuitive, otherwise they may become
frustrated and abandon the device [11, 15, 26, 29]. When surveyed, patients who tested wearable
robotic devices consistently highlighted comfort [7, 29] and ease of use [7, 26] as critical aspects
of the system to improve before widespread adoption is possible. Given the intended use of a
wearable mechatronic brace, conventional command input methods typically utilized in robotic
systems, such as buttons or joysticks, are not suitable [27]. For example, if an upper-limb robotic
exoskeleton restores the use of a patient’s injured arm, but at the cost of the ability to use their
healthy arm (which is now occupied manipulating a set of buttons and joysticks to operate the
system), then the device has not really improved their quality of life in a meaningful way. Wearable
mechatronic systems need a method of detecting the user’s intention in a way that feels seamless
and unobtrusive during operation.
This problem extends beyond just input commands and also occurs during task execution as
well. To be effective for daily use, a wearable robotic device needs to be able to operate safely
during a variety of activities [11–13, 22, 30, 31]; however, it is a well known problem that the con-
trol systems of these devices struggle to adapt to changing conditions and external disturbances
[6, 11–13, 22, 24, 26, 30–46]. A system that requires the user to manually indicate every time
a task changes would be cumbersome to use, and this applies to a robotic exoskeleton used for
rehabilitation applications (such as changing the exercise being performed, or modifying the inten-
1.2 General Problem Statement 5
sity level), as well as assistive applications (such as varying the external load by lifting an object,
or changing between different activities of daily living). The device should instead automatically
detect these changes and have the control system adapt accordingly. This challenge becomes even
more complicated when focusing on upper-limb systems, since the loading conditions and range of
expected tasks is much more varied and unpredictable than those for a lower-limb system (which
typically only has to focus on locomotion) [11]. Attempts to develop adaptable control systems for
wearable robotic devices have largely focused on complex nonlinear control theory techniques that
are heavily dependent on accurate modelling [13, 26], which is difficult to obtain for a human-driven
wearable device [12, 26], and require more computational resources to properly implement than
what may be available in an embedded system [6, 41, 47–49]. Other approaches focus on control
system methods that utilize expensive and cumbersome force sensors that may be challenging to
implement into a wearable device and could impact user comfort [14, 45, 50]. There is a need to
develop adaptable control systems for robotic exoskeletons that can be implemented easily to meet
the hardware constraints of a wearable device and that can decode task intention from the user in
a way that is unobtrusive.
1.2 General Problem Statement
One approach that has been proposed to address the method of control for wearable robotic de-
vices is the use of bioelectrical signals to determine intention from the user. These signals are
electrical currents generated by the nervous system during control of various bodily functions,
and they can be measured using electrode-based sensors on the surface of the skin to infer which
biological processes are presently active (and how this relates to the action currently being per-
formed by the user). Two bioelectrical signal types commonly used for wearable device control
[6, 11, 14, 23, 27, 51–54] are electromyography and electroencephalography. Electromyography
(EMG) measures muscle activity and can detect that a person is trying to move [53, 55–57],
whereas electroencephalography (EEG) measures brain activity and can detect that a person is
thinking about/executing motion [58–61]. This decoded information can then be used by the con-
trol system of a robotic exoskeleton to properly follow operator commands. Traditionally, wearable
1.3 Research Objectives 6
robotic devices have been designed to only utilize one bioelectrical signal type at time, choosing
to use either EEG or EMG for control [6, 11, 28, 62, 63].
While this approach has been demonstrated successfully, both EEG and EMG are plagued by
their own issues that can affect reliability and act as a barrier to widespread adoption of these
bioelectrical signals for control purposes. It can be difficult to decode EEG signals, as information
from different active regions of the brain can overlap and the EEG signals do not provide a di-
rect measure of muscle activity [62, 64–67]. Conversely, EMG signals do measure muscle activity
directly, but can be affected by other factors, such as muscle fatigue, which unpredictably alter
the EMG signal [11, 54, 64, 66, 68, 69]. To address these limitations, the use of sensor fusion
techniques has been proposed to combine the information from both EEG and EMG simultane-
ously, leveraging the benefits of both bioelectrical signals [54, 62, 63, 69–74]. EEG–EMG fusion
has been shown to improve intention detection over using EEG or EMG alone [63, 68, 69, 75–82];
however, development of these techniques has been preliminary [63, 69, 70, 72, 74] and have been
demonstrated primarily for basic motion classification tasks often performed under constrained
conditions that do not always mimic those observed during the operation of a wearable robotic
device [75, 79–89] (for example, simple button presses instead of full dynamic movements).
This work proposes an investigation into the use of EEG–EMG fusion techniques to detect
more complex task information from the bioelectrical signals, such that it can be utilized by the
control system of a wearable upper-limb robotic device to improve adaptability. Various EEG–
EMG fusion methods were developed and evaluated to classify relevant task information, such
as the weight held by a user during motion, and a novel adaptive control system based on the
incorporation of these EEG–EMG fusion classification models into the control law was proposed
and investigated.
1.3 Research Objectives
There were two main goals outlined for this thesis. The first goal was to expand machine-learning-
based EEG–EMG fusion methods to allow for the detection of more relevant task information,
and to evaluate design elements of the various fusion techniques to optimize performance. The
1.4 Overview of the Thesis 7
second goal was to develop a novel approach of integrating these EEG–EMG fusion models into
the control system of a wearable upper-limb mechatronic brace to improve device adaptability. To
achieve these goals, work has focused on the following objectives:
To evaluate previously demonstrated binary classification EEG–EMG fusion methods to

provide a performance comparison when used for motion classification during dynamic elbow
flexion–extension movements.
To expand EEG–EMG fusion techniques for use with more complex multi-output classifiers,
and to evaluate their performance when used for task weight classification during dynamic
elbow flexion–extension motion. At the same time, also evaluating various EEG and EMG
features for their potential to correlate with changes in task weight and investigating the
effect that including speed information as an input feature has on EEG–EMG fusion classifier
accuracy.
To investigate the use of Convolutional Neural Networks (CNNs) as a new method of input-
level EEG–EMG fusion, focusing on an evaluation of how different methods of representing
the EEG–EMG inputs for fusion (time-domain vs. time–frequency domain images), different
normalization techniques used during pre-processing, and different methods of fusing the
bioelectrical signals inside the CNN based on model configuration affect EEG–EMG fusion
performance.
To propose and implement a new adaptive control system method for wearable robotic
devices based on the integration of EEG–EMG fusion task weight classifiers into a Gain
Scheduling-based approach, and to investigate the performance of this controller through
the development of a wearable upper-limb mechatronic brace simulator used to evaluate the
system.
1.4 Overview of the Thesis
The structure of the remainder of this thesis is summarized in the outline below:
Chapter 2 Bioelectrical Signal Processing, Classification, and Fusion: Provides relevant

background information regarding EEG and EMG signals, explaining the full
bioelectrical signal model development pipeline including signal acquisition,
signal processing, feature extraction and classification. The concept of EEG–
EMG fusion is introduced, with a review of current fusion methods presented.
Chapter 3 Comparing EEG–EMG Fusion Methods for Motion Classification: Presents
the procedure and results for a comparison study focused on evaluating mul-
tiple commonly used EEG–EMG fusion methods for binary motion classi-
fication; demonstrating how each technique is affected by changing motion
conditions during dynamic elbow flexion–extension motion.
Chapter 4 Task Weight Classification Using EEG–EMG Fusion: Demonstrates the suc-
cessful implementation of EEG–EMG fusion methods used for multi-output
classification with a focus on classifying a novel task: the weight held during
elbow flexion–extension motion performed at varying speeds (task weight).
Provides an initial investigation into potential EEG and EMG features that
correlate well with task weight changes and evaluates the effect that including
speed information into the classifiers has on EEG–EMG fusion performance.
Chapter 5 Evaluating Convolutional Neural Networks for Input-Level Fusion: Proposes
the use of CNNs as a potential new method of input-level EEG–EMG fusion.
Evaluates task weight classification accuracy to demonstrate the feasibility
of this fusion technique and assess various model design parameters, such as
EEG/EMG image representation and CNN model configuration, to determine
how modifying the approach of fusion within the CNN affects classifier per-
formance.
Chapter 6 Investigating the Effect of Image Normalization on CNN-Based Fusion: Out-
lines a follow-up study performed to iterate on the model design of the CNN-
based input-level EEG–EMG fusion classifiers developed in Chapter 5 to fur-
ther improve performance; evaluating how different methods of EEG/EMG
image normalization used as a pre-processing step before CNN training affect
task weight classification accuracy.
Chapter 7 Development of a Task Weight Selective Control System: Proposes and imple-
ments a novel adaptive control system, referred to as Task Weight Selective
Controller (TWSC), based on the incorporation of EEG–EMG fusion clas-
sifiers into a Gain Scheduling-based control approach. Outlines the devel-
opment of an upper-limb mechatronic brace simulator used to evaluate the
performance of the TWSC.
Chapter 8 Conclusions and Future Work: Highlights the contributions of this thesis and
provides recommendations for future work.
Appendix A Additional Details for the Upper-Limb Brace Simulator Model: Provides sup-
plementary explanations and formula derivations for specific aspects of the
upper-limb brace simulator model, namely regarding the modelling process
used for the DC motor and gearhead, as well as the full derivation of the
dumbbell inertia formula.
Appendix B Permissions and Approvals: Presents approval letters for copyrighted material,
as well as proof of ethics approval for the EEG/EMG signal collection trials
that involved human subjects.
Chapter 2
Bioelectrical Signal Processing,

Classification, and Fusion
Portions of this chapter are adapted from select sections of “Interpreting Bioelectrical Signals for
Control of Wearable Mechatronic Devices”, published as Chapter 5 of the book Human–Robot
Interaction: Control, Analysis, and Design. Cambridge Scholars Publishing, 2020 [90].
2.1 Introduction
The human body is an extremely complex system with many different biological processes running
continuously. Through various sensing technologies, researchers have sought methods of measuring
and observing these events, and one such approach towards this is the use of bioelectrical signals.
Bioelectrical signals are a broad classification applied to any electrical signal measured from the
human body (through various means) that provides information on the biological processes being
observed. These signals are the result of various bioelectric phenomena occurring inside the nervous
system, generated by the brain to control different bodily functions. To subdivide this class of
signals, each major bioelectric phenomena of interest has been given a specific name to specify
what type of biological process it affects. In the context of wearable robotic systems there are two
bioelectrical signals that are especially relevant due to their frequent utilization for device control
[6, 11, 14, 23, 27, 51–54]: electromyography and electroencephalography.
10
2.1 Introduction 11
The first bioelectrical signal of interest, electromyography (EMG), measures the electrical ac-
tivity present in the motor neurons used by the body to contract muscle fibres and enable motion
of the limbs. When a person wants to move, their brain will send an electrical signal through the
nervous system to the limb in question. This electrical impulse will cause the appropriate muscles
fibres to contract, which applies a force onto the limb through the musculoskeletal system and
causes it to move (in the case of a static muscle contraction task with no limb movement, such
as pushing an object, the force is transferred through the musculoskeletal system to the object)
[53, 56, 57, 73, 91, 92]. By measuring the EMG signal, information about the muscle contraction
(and hence limb motion) can be inferred based on its observed behaviour [53, 55–57]. A typical
EMG signal is characterized as a series of high-frequency oscillations occurring around 0 V [53, 93],
with the usual voltage range of the signal being around 0–10 mV peak-to-peak [55, 72, 92, 94–97].
An example EMG signal can be seen in Figure 2.1. Since an EMG signal measured from the surface
of the skin is a combination of many individual motor neuron action potentials recorded simul-
taneously [55, 56, 91], the amplitude of the oscillations will partially correlate to the strength of
the muscle contraction (i.e., large amplitudes are observed when the muscle contracts, whereas the
signal will be close to 0 V when the muscle is at rest) [55, 55, 56, 72, 98]. However, this behaviour
is not a set rule by any means and the appearance/behaviour of EMG can vary greatly both be-
tween people and even within the same person [53, 55–57]. This fact makes EMG a challenging
signal to use when controlling wearable robotic devices since its behaviour can be very erratic,
unpredictable, and non-repeating. It is difficult to know how to correctly interpret the EMG re-
sponse to accurately determine the desired movement it is supposed to be generating. Much of
the research regarding EMG has been devoted to developing different methods of modelling the
EMG–musculoskeletal system relationship [53, 72, 99–101] and investigating different methods of
extracting relevant patterns from the signal [55–57, 72, 98], so it can be used to determine user
intention and drive mechatronic devices.
The second bioelectrical signal of interest, electroencephalography (EEG), measures the elec-
trical activity present in the brain [60, 71, 73, 102–104]. As the brain executes conscious and
unconscious functions of the human body, various neurons in relevant regions will generate elec-
tricity to process information and send commands. By measuring this activity, inferences can be
2.1 Introduction 12
Amplitude [mV] 0.5
-0.5
0 2 4 6 8 10 12 14 16 18 20
Time [s]
Figure 2.1: An example of an EMG signal recorded during elbow flexion–extension motion, demon-
strating its high-frequency oscillatory characteristic behaviour. Amplitude of the os-
cillations correlates roughly to muscle activity, with regions where muscle contractions
have occurred measuring higher peak voltage values and regions where the muscle was
at rest demonstrating almost no electrical activity (the small amount of activity re-
maining is likely caused by measurement noise).
made about a person’s motion intention as the brain sends commands to the relevant motor neu-
rons in the muscles being contracted [58, 60, 61]. The appearance of EEG is difficult to characterize
definitively, due to the highly variable and unpredictable nature of the signal (unlike EMG, which
typically follows a more visually recognizable pattern of amplitude varying oscillations around 0
V). Often, EEG will appear as a time-varying signal with constantly changing amplitudes around
0.5–100 µV [60, 102, 103], with the signal variation occurring between positive/negative voltage
values and containing a combination of oscillation frequencies. An example EEG signal can be
seen in Figure 2.2. With EEG, the distinction is made that it specifically refers to a non-invasive
measure of brain activity; therefore, it is actually measuring the propagation, through the skull, of
the electromagnetic field generated by the combined bioelectric phenomena occurring within each
individual brain cell over the region of interest [60, 104, 105]. This means that an EEG signal
may contain a information from summation of many different continuously active brain functions
during recording, making it even more challenging to decode than EMG [62, 64–67]. Like EMG,
EEG is also highly variable both between subjects and within the same person [59–61, 103], making
2.2 Signal Acquisition 13
its behaviour even more unpredictable. Many researchers have focused on developing methods of
extracting user intention from this complex signal [58, 59, 61, 102, 104, 106–108], to move towards
the goal of “seamless” device control where a user need only think about moving their affected
limb to actuate the system (as if moving naturally).
0.1
Amplitude [mV]
0.05
-0.05
-0.1
0 2 4 6 8 10 12 14 16 18
Time [s]
Figure 2.2: An example of an EEG signal recorded during elbow flexion–extension motion. The
signal displays unpredictable and non-repeating behaviour, with the amplitude varying
erratically throughout the recording and no discernable pattern visible in the response.
The purpose of this chapter is to provide a comprehensive overview regarding the use of EEG
and EMG, as it relates to wearable device control, providing the background information necessary
to understand the work described in this thesis. Methods of acquiring, processing, decoding, and
fusing these bioelectric signals are described in the following sections. Other literature review
specifically relevant to individual projects in this thesis are provided in their respective chapters,
to organize information better.
2.2 Signal Acquisition
To be able to utilize EEG or EMG for control of a wearable mechatronic device, a method of
measuring and recording these bioelectrical signals first needs to be determined. This process is
referred to as signal acquisition, where sensors read the electrical impulses generated by the body
and convert them to digital measurement values suitable for computation and analysis. In general,
signal acquisition requires the use of two main components: electrodes to capture the electrical
activity occurring in the body, as well as a combined amplifier and Analog-to-Digital Converter
(ADC) circuit that takes the bioelectrical signals from the electrodes and processes them into a
digital file format [60, 92, 96, 97, 109, 110]. Of course, this breakdown is a large over-simplification
of the complexities involved in bioelectrical signal acquisition, overlooking in particular the many
components and design choices required to develop a measurement circuit [92, 95, 97, 110, 111].
However, as analog circuit design is not the focus of this thesis, it is considered out of the scope of
the review presented here. Therefore, the main goal of this section is to discuss signal acquisition
specifically from the perspective of the someone collecting EEG/EMG signals from a human study
participant (providing the most relevant background information for this task), as opposed to
developing bioelectrical signal measurement equipment.
For both EEG and EMG signals, the principle by which the electrodes function is the same:
a conductive material (traditionally a conductive metal such as gold [109, 110, 112, 113] or silver
silver-chloride [56, 60, 72, 96, 97, 109, 110, 113, 114], depending on the specific electrode type)
makes contact with the subject’s body and provides a pathway for the bioelectricity to follow that
feeds into the amplifier and ADC measurement circuitry [95, 97, 109, 110, 113]. The main differ-
ence between EEG and EMG electrodes mainly comes down to their form factor and application
methods, tailored to the different locations on the body where the bioelectric signals are recorded.
As a point of clarification, the review provided in this chapter focuses specifically on non-invasive
(also called surface) electrodes for both EEG and EMG. Since the application of interest for these
bioelectrical signals is wearable robotic exoskeleton control for activities of daily living, the use
of any invasive methods of signal acquisition are obviously out of the question. For both EMG
and EEG there exist analogues that utilize invasive approaches: intramuscular EMG using needle
electrodes [73, 97, 115] for muscle activity, and Electrocorticography (known as ECoG, where elec-
trodes are placed directly onto the surface of the brain) [73, 116] or implantable microelectrode
arrays [116] for brain activity. Invasive methods do have the potential to record higher quality
bioelectrical signals [56, 57, 73, 96, 116], due to the fact that the electrodes are closer to the
neurons generating the electricity, as opposed to measuring it through the skin, and can more
accurately target specific neurons. However, this comes at the steep cost of pain/discomfort for
the patient [55–57] and potentially the need for risky surgical procedures particularly in the case
of implantable brain electrodes [73, 116]. Given this fact, only surface EEG and EMG electrodes
were used in the projects outlined in this thesis; hence, the discussion here focuses exclusively on
those.
2.2.1 Wet and Dry Electrodes
One classification used to differentiate electrode types for both EEG and EMG signal acquisition is
the distinction between wet and dry electrodes. Often, an electrode will be coated in an electricity
conductive gel/paste to ensure a good connection with the subject’s skin. Electrodes that utilize
this approach are referred to as wet electrodes [53, 56, 60, 61, 61, 72, 96, 102, 109, 113], and in
many cases are considered the gold standard for bioelectrical signal acquisition (with this applying
to both EMG [53, 56, 72, 96, 112] and EEG [61, 61, 102, 109, 112], historically). The quality of the
electrical connection made at the interface between the subject’s skin and the electrode material
can have a huge impact on the system’s ability to accurately record the signal, and can be affected
by many factors such as electrode movement/air gaps, sweat/dirt on the skin, or hair caught
between the electrode (to name a few examples) [53, 55, 56, 60, 72, 97, 109, 110, 112]. Using
conductive gel can alleviate some of these factors by ensuring a proper electrical connection is
maintained even when these obstacles are present. While useful in maintaining a good connection,
there are drawbacks that arise from wet electrodes, namely reduced user comfort and quality
of life [53, 56, 61, 72, 96, 102, 109, 112]. The ultimate goal of wearable robotic devices is for
patients to use them for extended periods of time as they complete their activities of daily living,
and many users may not wish to wear electrodes with conductive gel for that long (and/or may
find the need to constantly reapply the gel as it wears away throughout the day cumbersome).
To try and address this limitation, more recent developments have focused on the design of dry
electrodes: named so because they do not require the use of any conductive fluid for the electrical
connection [53, 56, 61, 72, 96, 102, 109, 112]. These electrodes instead utilize only the conductive
material of the electrode pressed against the user’s skin to establish a path for current to flow, and
because of this the quality of bioelectrical signals they can obtain is often lower than wet electrodes
[53, 56, 61, 72, 96, 102, 109, 112, 113, 117]. Some equipment manufacturers will compensate for
this fact using more advanced electronics in the measurement system, or more advanced filtering
techniques, to process the bioelectrical signals during/after data collection and remove unwanted
noise [112, 113, 117]; however, doing so may increase the cost and complexity of the device.
Therefore, a trade-off between cost, signal quality, and ease-of-use/subject comfort needs to be
evaluated when considering whether to utilize wet or dry electrodes during signal acquisition. In
the context of wearable rehabilitation robots, dry electrodes would be the ideal choice to reduce
the impact that wearing the device has on user comfort/quality of life, but only if the measured
bioelectrical signals are of sufficient quality to allow for proper device control. In a laboratory
setting however, comfort may not be a higher priority than signal quality for short data collection
sessions, thus wet electrodes may be a suitable option despite the inconvenience.
2.2.2 Signal Transmission
Along with the choice between wet and dry electrodes, there is another category used to classify
electrode (and measurement system) types: the signal transmission method. This refers to how the
bioelectrical signal information is transmitted to the acquisition circuit and is typically grouped
as either wired or wireless systems. Wired systems use cables connected to electrode to carry
the measured bioelectric signal back to the ADC circuit, whereas in a wireless system no cables
are needed and the signal is instead transmitted using a wireless communication protocol (like
Bluetooth or Wi-Fi) [61, 102, 104, 109, 110, 114, 118]. The decision between a wired or wireless
system presents another trade-off: wired systems can provide higher quality signal recording (which
is why they have traditionally been considered the gold-standard for EMG [119, 120] and EEG
[102, 109] measurement, particularly in clinical settings) at the cost of decreased comfort, since
the subject’s movement may be limited by the cables physically connecting them to the ADC
circuit [102, 104, 114]. Conversely, wireless systems allow for freer movement; however, they
can result in lower quality signals (due to data transfer rate of wireless communications limiting
the number/resolution of channels) [104, 109, 114, 118] and may cost more due to the increased
electronics required to facilitate the transfer [102, 109]. These factors need to be evaluated when
selecting a bioelectrical signal acquisition system to use during data collection, to balance system
cost and signal quality with subject comfort and the level of freedom required during movement.
For operation of a wearable robotic exoskeleton during daily life, wireless electrodes are basically a
necessity, since the user needs to be able to walk around and move freely to complete their desired
tasks and cannot be tethered to a computer/bench-top equipment. If wired electrodes are used,
the furthest they could travel, realistically, is to an amplifier/ADC circuit built-in to the embedded
system of the wearable mechatronic device (contained somewhere on the user’s body), and even
this may still impact user comfort as the cables could restrict movement or become tangled if
proper precautions are not taken. Regarding data collection for research purposes however, wired
electrodes are a more viable option (depending on the level of freedom required to perform the
motions under investigation) and the potential for higher quality signal measurement may be a
more desirable trait.
2.2.3 Electrode Examples
The electrodes traditionally used for EMG signal acquisition typically resemble the example seen
in Figure 2.3(a). The electrode itself is a conductive piece of silver silver-chloride metal (the most
commonly utilized conductor [56, 72, 96, 97, 110]) surrounded by an adhesive glue used to adhere
the electrode onto the person’s skin. A small amount of conductive gel surrounds the electrode
to ensure a good connection (making this a wet electrode) and the snaps on the bottom of the
electrode are used to attach cables for a wired EMG signal acquisition system. An example of a
dry EMG electrode, belonging to the wireless Delsys Trigno system (Delsys Incorporated, Natick,
United States of America) can be seen in Figure 2.3(b). In this system, the electrodes are the
conductive metal bars [91], which press against the subject’s skin directly. As with the example
wet electrode discussed above, the dry electrode here uses a strip of adhesive glue to temporarily
attach the electrode to the subject’s skin.
While both EMG electrode examples shown here utilize metal as the conductive interface of
the electrode (as is traditionally done in EMG measurement systems), it should be noted that
this is not the only conductive material that can be used. Though yet to be widely adopted,
research into the use of soft textile-based EMG electrodes has become popular in recent years,
where conductive thread is woven into fabrics (or the fabric is coated with conductive chemicals)
(b) Dry EMG electrodes

(a) Wet EMG electrodes
Figure 2.3: Photographs showing examples of wet (a) and dry (b) EMG electrodes. Looking at
the wet electrode, electrically conducive gel can be seen surrounding the silver silver-
chloride conductive metal. The snap connector allows cables to be attached to the wet
electrodes to use them with a wired amplifier/ADC signal acquisition device. For the
dry electrodes, the four conductive metal bars act as the actual electrode, which are
connected directly to the electronics circuit contained within the plastic housing; used
to measure the signals and transmit recordings wirelessly to the main bench-top unit
of the signal acquisition device.
to create pliable electrodes [112, 121]. The appeal of textile-based EMG electrodes lies in their
potential for increased comfort and unobtrusive incorporation into daily life. The nature of this
technique means that electrodes can conform better to the shape of the subject’s body (compared
to rigid metal-based electrodes), and since they are made of thread they have the potential to be
integrated into normal clothing directly, allowing for easier setup and providing a more discrete
profile for the wearer (which is an attractive benefit for wearable mechatronic devices, as they are
intended for regular use) [112, 121]. As with other electrode design choices, textile-based solutions
introduce yet another trade-off. Similar to the comparison between wet and dry electrodes, the
signal quality of textile electrodes can be poor, even more so than metal-based dry electrodes
[112, 121]. As this approach is still very much in the research stage, many other challenges, such
as integration into embedded electronic circuits and signal processing improvements, still need to
be resolved before textile-based EMG electrodes can see widespread adoption [112, 121]. Due to
their exploratory nature, textile electrodes were not used for the work presented in this thesis;
however, they have been included in the information discussed here for completeness’ sake.
For EEG signal acquisition, the use of gold-cup electrodes (along with ear-clip reference elec-
trodes), like the ones shown in Figure 2.4(a), have been a popular approach historically [122]. The
electrode is cup shaped to allow it to be filled with electrically conductive paste, which serves two
purposes: ensuing a good electrical connection between the electrode and the subject’s scalp and
helping to adhere the electrode [60, 102]. Even more so than EMG, obtaining a good connection
with EEG is particularly challenging due to the hair on the subject’s head. EMG electrodes can
typically be placed directly on the skin with little blocking it; however, for most people (depending
on their hair style) this is not easily doable for the skin on their head. Therefore, the purpose of
the conductive paste is to work around hair and down to the scalp so an electrical connection can
be established. Since the use of adhesive glue strips is not likely feasible on a hair-covered head,
the sticky conductive paste also helps to hold the electrode in place (at the cost of a potential
increase in subject discomfort).
Given the practical issues that arise from requiring paste to be applied to a person’s hair when
performing EEG signal acquisition (especially if this process is required daily for the operator
of a wearable mechatronic device), dry electrode solutions have also been developed for EEG to
overcome this inconvenience [102, 109, 113]. The challenge of establishing an electrical connection
through hair is still present for dry EEG electrodes, thus the approach taken to address this in
many examples is the use of spike shaped electrodes that can work their way through the hair
down to the scalp (similar to how the teeth of a comb pass through the hair) [109, 113]. Some
systems will also utilize springs on the electrodes to apply a gentle force downwards, keeping the
electrodes pressed through the hair [109]. An example of a dry EEG electrode, belonging to the
DSI-7 wireless headset system (Wearable Sensing, San Diego, United States of America), can be
seen in Figure 2.4(b). It should be noted that while this spike-based dry electrode form factor
does reduce the issue of hair blocking the electrode, it is far from a perfect solution. Depending on
how much downward force against the electrode is required to established a good connection, the
user may find it uncomfortable (especially if wearing the headset long periods of time) [109]. Even
if using spike-based dry electrodes, EEG signal acquisition can still potentially fail depending on
hair type and style (there is a limit to how much hair the electrode spikes can reasonably work
through to make contact with skin) [102, 109]. Given these drawbacks, researchers are continuing
to investigate other dry EEG electrode solutions and acquisition methods that may be able to
circumvent the need measure from the scalp directly. One example of this is Ear-EEG, where
electrodes are built around an ear piece (similar in appearance to earbud-style headphones) and
inserted into the ear [112, 123]. However, projects such as this are ongoing and are not always
commercially available.
(a) Wet EEG electrodes (b) Dry EEG electrodes
Figure 2.4: Photographs showing examples of wet (a) and dry (b) EEG electrodes. In (a) the
gold-cup EEG electrode can be seen on the right, showing the concave shape that gets
filled with electrically conductive paste when attaching the electrode to the subject’s
scalp. The ear-clip reference electrode can be seen on the left, which is clipped onto the
subject’s ear lobe. Both of these have cables pre-attached to connect the wet electrodes
to a wired amplifier/ADC signal acquisition device. In (b) example dry electrodes,
incorporated into a wireless EEG headset, can be observed. The grey spikes serve as
the actual conductive electrodes, which use springs to apply gentle force downward to
push them through the subject’s hair and onto their scalp.
2.2.4 Electrode Placement
When setting up for EMG signal measurement, it is common practice to clean the skin around
the area of interest with an alcohol swab before adhering an electrode [53, 72, 97, 110]. This is
done to remove any dirt/oil on the subject’s skin that may interfere with the electrical connection.
Some procedures may even direct the experimenter to shave any body hair present at the electrode
site as well (to ensure the electrode can make full contact with the skin and not be obstructed by
hair) [53, 72, 96]. However, this is a somewhat extreme step that many subjects may not wish to
participate in (especially when considering the burden this would place on a robotic exoskeleton
user who wears their device daily), hence it may be omitted from some data collection protocols.
Depending on the strength of the EMG electrode adhesive, medical tape may also be placed over
the electrodes as well, to further secure them on the subject’s skin and ensure they do not fall off
during data collection (see Figure 2.5 for an example).
Electrode location is an important aspect in EMG signal acquisition, as where the electrodes are
placed on the body will change which muscles contribute to the bioelectric signal being measured
[57, 96]. This spatial information is important, because it provides key insight into what motions
are being performed when observing that the EMG channel corresponding to certain muscles
is activating [57]. For example, if EMG activity is observed from an electrode placed over the
biceps, but not one placed over the triceps, then it is reasonable to conclude that the subject
is performing an elbow flexion movement. Ideally, a recorded EMG signal would only contain
bioelectric measurements from one muscle (so that each channel can be mapped to a muscle
directly); however, this is not always possible. Due to the way electricity conducts through the skin,
EMG signals from other nearby muscles can also be picked up by an electrode in a phenomenon
known as crosstalk [55, 91, 92, 124]. To help reduce this effect, proper electrode placement is
important to try and ensure adequate spacing such that the electrode is only covering the muscle
of interest (as much as is feasibly possible) [55, 91, 124]. Another option to reduce crosstalk is
to use a larger number of EMG channels on all surrounding muscles, so the undesired muscle
activations are known and can be removed from the EMG signal of interest using spatial filters
[124]; however, this obviously increases the complexity of the measurement setup. In an attempt
to ensure proper and consistent electrode placement, a set of EMG setup standards known as
the Surface EMG for a Non-Invasive Assessment of Muscles (SENIAM) Project guidelines have
been developed [119, 120]. This procedure uses measurements taken from anatomy-based reference
points on the subject’s body to determine where to place an EMG electrode for a specific muscle.
For an example of EMG electrodes placed over the biceps and triceps using the SENIAM Project
guidelines, see Figure 2.5.
EEG electrode setup follows a similar approach to EMG, where the subject’s scalp is typically
cleaned before electrode placement. In the case of EEG, a mildly abrasive gel is often used to
remove both dirt/grease and dead skin from the scalp at the desired electrode location, as these
can interfere with the electrical connection being established [60, 113]. The ear lobe where the
reference ear-clip electrode is being placed should also be cleaned using an alcohol swab before
(a) Biceps electrodes (b) Triceps electrodes
Figure 2.5: Photographs showing bipolar EMG electrodes attached to the biceps (a) and triceps
(b), placed according to the SENIAM Project guidelines. Medical tape is applied over
the electrodes to help secure them in place.
it is attached to the subject, for similar reasons. As discussed above for wet gold-cup electrodes,
the cup is then slightly overfilled with electrically conductive paste and pressed onto the scalp to
adhere the electrode to the subject’s skin directly (being careful to avoid pressing the electrode
into the hair instead and not making proper contact, parting the hair as necessary). Medical tape
or gauze can be used to further adhere the EEG electrode to the subject’s scalp; however, caution
must be taken when employing this solution since incorrect placement of the tape/gauze in the
subject’s hair can potentially cause the electrode to lift away from the skin if the tape/gauze pulls
it upwards. An example of an attached EEG electrode can be seen in Figure 2.6(b). Dry electrodes
are applied in a simpler manner, and merely require that the spikes of the electrode are worked
down through the hair, such that they are contacting the skin directly.
During EEG signal acquisition, as with EMG, ensuring the correctness of electrode locations
plays a vital role in the success of a data collection protocol. EEG spatial information (where
the electrode is located on the scalp and what region of the brain this lies over) can provide
significant insight into the nature of the observed signal. It is well known that different regions of
the brain are used to control specific biological functions; therefore, this knowledge can be used
when interpreting signal behaviour [60, 61]. For example, high electrical activity being observed
in the electrodes located over the motor cortex could indicate that the user is engaged in (or
thinking about) a motor action such as moving their arm. For this reason, electrode placement
is critical in EEG applications, and selecting the correct channels to record for a desired action
can greatly improve the quality of an EEG dataset [125, 126]. To ensure proper and consistent
electrode placement, a series of instructions and standards referred to as the 10–20 International
System have been developed [60, 102]. This protocol uses anatomy-based reference points on the
subject’s skull, along with measurements of head size, to place electrodes in a grid-like pattern
across the scalp (with each electrode location being determined proportionally, using percentages
of the person’s total head size). A diagram showing the various electrode locations included in the
10–20 International System can be seen in Figure 2.6(a). Since electrode locations are determined
using a subject’s specific measurements, these specified electrode locations should roughly correlate
to the same regions of the brain for each unique person (understanding that minor setup variations
are to be expected due to human error).
(a) 10-20 International System (b) EEG electrode setup
Figure 2.6: A diagram explaining the naming convention and electrode locations used in the 10–
20 International System (a), as well as a photograph showing an example wet EEG
electrode placed accordingly (b). The electrode shown is placed in the C3 location and
the reference ear-clip electrode is placed in the A1 location to complete the referential
montage.
Ideally, an EEG measurement protocol would be designed in a manner such that the electrodes
are always placed perfectly in the most biologically relevant locations and only record bioelectrical
information from that specific brain region (for example: electrodes would be placed directly over
the motor cortex when trying record motor activity); however, this is not always obtainable. EEG
signal analysis is an ongoing area of research, and it is not always possible to know with certainty
which regions of the brain are most relevant for certain applications. Also, due to variations in the
human body, there can be different optimal electrode locations from person to person. Utilization
of the 10–20 International System seeks to mitigate this problem, but it is only an estimate of
the relationship between head size and brain regions and is by no means a perfect physiological
mapping (given its non-invasive nature). Unlike with EMG, where it is much easier to ensure
relatively accurate placement due to the ability to physically see which muscle (and where on the
body) the electrode is being attached, it can be hard to confirm the source of an EEG signal
being recorded since the brain is contained within the skull. Likewise, the issue of crosstalk is
even more prevalent for EEG, since the bioelectric signal is not able to be measured until after
it has propagated through the skull (far away from its point of origin, the specific neurons in the
brain tissue that generated the signal) [60, 104, 105]. For this reason, it is common in many EEG
applications to perform signal acquisition with a large number of electrodes, then use a spatial
filter to select the most relevant channels for each person (this process is discussed further in
Section 2.3.2). It is important to consider however, that this approach is unlikely to be feasible in
a wearable mechatronic system, since the number of possible EEG electrodes will be limited due
to the hardware restrictions that come as consequence of the device being portable/wearable.
2.2.5 Electrode Configurations
The final component of bioelectrical signal acquisition that needs to be considered is which elec-
trode configuration (sometimes referred to as a montage [60, 91, 92, 109, 110]) to use. An electrode
configuration dictates how the reference point of each electrode channel is defined, which has im-
plications for how the bioelectrical voltage is measured by the ADC circuitry [56, 60, 91, 92, 97,
109, 110, 127]. In many cases, the type of configuration used will be dictated by the measure-
ment system chosen for data collection, as often times the montage is setup at the hardware level.
Due to the small voltage range of both EEG (0.5–100 µV [60, 102, 103]) and EMG (0–10 mV
[55, 72, 92, 96, 97]), signal amplification is a necessary step before the ADC process can take place
[56, 60, 91, 92, 97, 110, 127]. Depending on how the operational amplifier circuit in the mea-
surement device has been designed, certain electrode configurations may be mandatory to ensure
proper operation. When selecting hardware to use for signal acquisition, the desired electrode
configuration should be considered to ensure the system will meet desired specifications of the
data collection protocol.
A common electrode configuration used for EMG signal acquisition is bipolar electrodes. In
this montage, a differential amplifier circuit is used alongside two electrodes (one for the positive
terminal and one for the negative terminal of the operational amplifier) placed next to each other on
the same muscle [53, 56, 91, 92, 96, 97, 110, 127]. This means that for each EMG signal measured
(i.e., each channel), two EMG electrodes will need to be placed on the subject (one acting as
the measurement point and another acting as the reference). The use of bipolar electrodes is
recommended in the SENIAM Project guidelines, and it specifies that the electrodes should be
spaced approximately 20 mm apart [119, 120]. An example of a bioplar EMG electrode setup
can be seen in Figure 2.5. The benefit of a bipolar electrode configuration is that it provides a
property known as common-mode rejection during signal recording [56, 91, 97, 110, 127]. Since
the two EMG electrodes are near each other they will likely be affected by the same sources of
noise (due to factors such as electromagnetic interference), and because the amplification is based
on the difference between the two electrodes, the common noise elements in the EMG signals will
be effectively cancelled out during amplification (providing a basic signal filtering effect during
signal acquisition). Other examples of EMG electrode configurations include monopolar (using
one electrode per EMG signal with a shared system-wide reference point) [53, 91, 92, 127] and
high-density EMG (using a grid of small EMG electrodes for each muscle, similar in many ways to
EEG) [91, 92]. Depending on the measurement device, it may also be necessary to define a system
ground point (shared for all channels) to act as the reference ground for the differential amplifier
circuit. In this case, another electrode should be placed on the subject over a bony area away
from muscle activity (to avoid fluctuations from other EMG signals) [91, 95, 97, 110]. A common
location for this is to place an electrode over the subject’s elbow bone.
2.3 Signal Processing 26
For EEG signal acquisition, a commonly used electrode configuration is a setup referred to as
a referential montage. In this configuration, two electrodes are used (like with the bipolar EMG
configuration); however, instead of utilizing two EEG gold-cup electrodes per channel, only one
electrode is placed on the head to measure EEG while an ear-clip electrode placed on the ear lobe
(the A1 and A2 locations in the 10–20 International System, shown in Figure 2.6(a)) to act as
the reference electrode instead [58, 60, 109, 113]. An example of EEG electrodes setup with a
referential montage can be seen in Figure 2.6(b). Utilizing this approach greatly simplifies the
EEG signal acquisition setup procedure, as it reduces the number of electrodes that need to be
attached to the subject’s scalp. Since the ear-clip electrode used for the amplification reference
is still making contact with the person’s body, some of the common-mode rejection property is
maintained [60, 109]. It is possible, and often employed, to instead setup EEG electrode using a
bipolar montage (two electrodes on the scalp per EEG signal channel) [61, 109]; however, doing
so increases the number of electrodes and setup time required (aspects which are not ideal for a
wearable device).
2.3 Signal Processing
Once the desired EEG and EMG signals have been recorded, the next step is to filter and process
the signals to prepare them for use. As with all sensors, there is a possibility that measured sig-
nals can become corrupted by unwanted noise, and this is especially true for bioelectrical signals
like EEG and EMG given their relatively small amplitudes. There are many potential sources of
noise found in EEG and EMG signals, with some common examples being powerline interference
[55, 60, 72, 95, 102, 110], electromagnetic interference from nearby electronics [55, 57, 60, 95, 102,
111], motion artifacts caused by electrodes moving [55, 60, 71, 72, 95, 102, 110, 111], and other
bioelectric signals in the body dominating the recording [55, 60, 71, 102, 103, 110]. Signal filtering
refers to the act of removing (or at the very least reducing) noise from the EEG/EMG recordings
to improve their quality and allow for better extraction of relevant information. Depending on
the measurement device used, this step may even begin during signal acquisition as many manu-
facturers will build hardware filters directly into their amplifier and ADC circuitry. Regardless, it
is still advisable, even in cases where hardware filtering is available, to also implement a separate
filtering step in software after the signals have been acquired.
2.3.1 EMG Filtering
Though EMG processing can vary depending on the application, the typical procedure involves
Direct Current (DC) Offset removal followed by filtering [53, 80, 93, 94]. Pure EMG presents as
an Alternating Current (AC) signal, meaning it should appear visually as a series of oscillations
centred around 0 V. However, bias in the measurement equipment may introduce a constant DC
component into the signal, causing the AC EMG signal to appear shifted with the oscillations
centering around a non-zero voltage value. See Figure 2.7(a) for an example an EMG signal with
a DC offset present. This shift is referred to a DC Offset, and it must be removed if wishing to
maintain the native properties of EMG within the recorded signal. DC Offset removal is a straight-
forward procedure, which involves calculating the value of the offset then simply subtracting it
from the EMG signal (shifting it back so it once again centres around 0 V) [53, 80, 93, 94]. See
Figure 2.7(b) for an example of an EMG signal with the DC Offset removed. Finding the DC
Offset value can be done by measuring the EMG amplitude voltage when the subject is at rest
and no muscle activations are being performed. In theory, if no motor neurons are being activated
to cause a muscle to contract then the EMG signal should not measure any electrical activity.
This can be observed in Figures 2.1 and 2.7, where the EMG signal remains flat until movement
begins. Therefore, any voltage value measured during rest should, in theory, be due only to the
DC Offset and can be subtracted from the EMG signal. Now, in reality this ideal scenario never
actually occurs because measurement noise in the EMG signal will result in small oscillations to
always be present in the signal (even when the EMG signal is “off”) [55, 72, 95, 110]. Thus, to
compensate for this noise the baseline EMG recording should be averaged to calculate the mean
voltage measured during rest, and this estimation of the DC Offset value can then be subtracted
from the EMG signal [53, 93]. There are multiple approaches, depending on the application, to
obtain the rest-state EMG signal with which to calculate the DC Offset value. One could perform
a calibration recording where the subject sits at rest for a specified period of time (for example,
one minute), or a specified number of samples could be taken from the beginning of every EMG
recording (before motion begins) to use for the average.

0.6 0.6
0.4 0.4
Amplitude [mV]
Amplitude [mV]
0.2 0.2
0 0
-0.2 -0.2
-0.4 -0.4
-0.6 -0.6
0 1 2 3 4 5 6 0 1 2 3 4 5 6
Time [s] Time [s]
(a) Before DC Offset removal (b) After DC Offset removal
Figure 2.7: Two plots showcasing the difference in EMG signal response before (a) and after (b)
DC Offset removal. After the offset value (shown as the red line on both plots) is
subtracted from the EMG signal, the entire signal shifts upwards to centre around 0
V.
Once the DC Offset has been removed from an EMG signal, the next processing step is to
apply a bandpass filter to remove unwanted frequencies. A bandpass filter works by only allowing
a specific range of frequencies through (specified by the designer) and frequency components outside
of this threshold are attenuated to remove them from the signal [55–57, 72, 95, 96, 110, 111, 127].
It can be thought of as a combination of a highpass filter and a lowpass filter, allowing only the
specified frequency band to remain. The use of bandpass filtering as a preprocessing step with
EMG signals is extremely common, and one would be hard pressed to find an EMG project that did
not make use of them at some point in their procedure. By choosing the filter cut-off frequencies
such that the pass band matches the expected frequency range of EMG, typically considered to
be 0–500 Hz [55, 57, 72, 94–96, 111, 128], applying a bandpass filter will remove noise from the
signal that contains frequencies outside of this range. Unfortunately, there is no consensus on
what exactly the EMG frequency cut-offs should be in the literature [55, 56, 96, 110], likely due to
bioelectrical signals highly variable nature making it a challenge to definitively define the expected
frequency range of EMG for a people (leading to different groups using a range of values). Many
researchers agree that a pass band of 20–500 Hz is suitable for EMG and recommend this be used
during filtering [55, 56, 95, 96, 111]. This will keep the filtered signals within the expected 0–500
Hz range of EMG while at the same time rejecting motion artefacts, which typically lay in the
0–20 Hz frequency range [55, 72, 72, 95, 111]. However, despite this, there exist examples of a
variety of pass bands being employed such as 5–500 Hz [55], 10–500 Hz [57, 127], 20–450 Hz [96],
and 10–400 Hz [72]. See Figure 2.8 for an example of the effect that bandpass filtering has on an
EMG signal.
1
Amplitude [mV]
0.5
-0.5
-1
0 0.5 1 1.5 2 2.5 3
Time [s]
(a) Before bandpass filtering
0.5
Amplitude [mV]
-0.5
0 0.5 1 1.5 2 2.5 3

Time [s]
(b) After bandpass filtering
Figure 2.8: Two plots showcasing the difference in EMG signal response before (a) and after (b)
applying a 20–500 Hz bandpass filter. The high frequency noise components in (a),
recognized as the jittery behaviour of the signal (which is particularly noticeable during
rest where the signal should be at 0 V), is significantly reduced in the EMG signal
shown in (b) after bandpass filtering.
Following bandpass filtering, another filter type is often applied when processing EMG signals:
the notch filter. Unlike a bandpass filter, which only lets a certain range of frequencies through
and removes the rest, a notch filter is designed to remove one specific frequency value and let the
rest remain. Regarding EMG processing, notch filters are specifically utilized to remove electrical
powerline interference [56, 72, 95, 96, 110, 111], since this occurs at a single frequency. Based on the
country that EMG signal acquisition occurred in, the notch filter will be set to either 50 Hz or 60 Hz
(North America uses 60 Hz power, for reference) [56, 72, 95]. It should be noted that the correctness
of utilizing a notch filter with EMG signals is something of a debate in the literature. Since there
is no such thing as an ideal filter (that perfectly removes all noise at the specified frequencies and
allows all other frequencies to pass through with no alterations), if a notch filter is used it will also
attenuate some of the frequencies on either side of the 60 Hz (or 50 Hz) powerline interference.
Knowing that the expected frequency range of EMG is 0–500 Hz [55, 57, 72, 95, 96, 111], this
means that the notch filter will inevitably also remove some potentially relevant information from
the signal. Now, some groups will decide that this trade-off is worthwhile and will use a notch
filter anyway [56, 72, 96]; however, there is research to suggest that the information removed by
the notch filter is an important component of the EMG signal response and recommends against
using a notch filter [95, 110, 111]. Thus, the specific requirements of the desired application should
be considered when designing EMG filters to determine if notch filtering is a suitable procedure
or not.
After DC Offset removal and filtering, the EMG signal should be processed enough that it
can be useable (barring any instances of extreme signal corruption, which may require the use of
specialized filters on a case-by-case basis). Many groups will stop the signal processing procedure
here and will use EMG in its filtered form for feature extraction and model training; however,
other researchers may decide to process the signal further to transform the EMG signal to a more
suitable form, depending on its intended use. One example of this is rectification, where the
absolute value of EMG is taken to remove all negative amplitude portions of the signal and flip
them to be positive [53, 55, 56, 72, 95, 96]. Doing so allows for easier calculation of things like the
linear envelope (a function that essentially traces the peaks of the EMG signal to create a line that
represents EMG amplitude), which can be used to easily represent signal strength in relation to
time [53, 56, 96]. Another example of post-filtering processing is normalization, where the EMG
signal values are rescaled to be between a set lower and upper bound value (such as 0 and 1), to
map the varied EMG measurements to a consistent scale and allow for easier comparison between
different signals [55–57, 96, 127]. Note that normalization, which is also utilized occasionally for
EEG, is discussed in much greater detail in Chapter 6. While these techniques do have their
benefits, they also come at the cost of modifying the observed response of the EMG signal, which
may cause other potentially relevant information inferred from the behaviour of EMG to be lost.
For example, if EMG is rectified all information regarding the negative voltage portions of the
signal is lost, and certain metrics, such as zero crossings, can no longer be calculated. Likewise, if
the signal is normalized its specific voltage measurement values can no longer be examined as the
scale has been modified to a new range. It is up to the discretion of the engineer processing an
EMG signal if further transformations such as these are suitable for their intended application.
2.3.2 EEG Filtering
As with EMG, it is challenging to use EEG in its raw form and processing is required to refine the
signal to a useable state. In general, EEG signals are processed similarly to EMG: filtering noise
from unwanted frequencies. The main difference for EEG comes in the complexity of the signal and
its susceptibility to noise. The amplitude of EEG is several orders of magnitude smaller than EMG,
meaning that even more so other signals can easily contaminate it, ruining the signal-to-noise ratio
[57, 72, 73, 96]. This corruption is not limited to external sources, and even other bioelectrical
signals within the subject’s own body, for example electrocardiography signals generated by the
heart (known as ECG) or EMG signals generated by the neck/jaw muscles, can easily dominate
an EEG signal [59, 60, 102, 103, 109, 129] (as shown in Figure 2.9). For this reason, filtering
is a crucial step of EEG processing, and many different approaches are used, depending on the
application.
Just as it was with EMG, the first step of EEG processing is to remove unwanted noise using
a bandpass filter. The frequency range used for this technique is highly variable and can change
depending on the application. It is understood that the frequency range for EEG falls below
100 Hz [103, 109]; however, as the frequency of EEG components increases, the amplitude of
Figure 2.9: An example EEG signal showing contamination from EMG signals generated in the
neck muscles (top) and ECG signals (bottom). EMG contamination is recognized by
the presence of higher frequency, higher amplitude components. ECG contamination
is recognized by larger amplitude periodic spikes present in the EEG signal. In both
cases, an example of the contamination is outlined in red.
these components decreases. This makes the higher frequencies hard to measure, since it becomes
difficult to distinguish these components from noise in the signal. For this reason, a smaller range
of frequencies is often used when filtering that will exclude the higher frequency components in
the EEG signal. For example, using a high frequency cut-off for the bandpass filter that is below
the expected upper limit of the EEG frequency range, such as 40 Hz [60, 61, 108] or 50 Hz [109].
See Figure 2.10 for an example of the effect that bandpass filtering has on an EEG signal.
Beyond being used for noise removal, bandpass filtering is also used to separate the EEG signal
into different frequency components. It is common in many EEG applications to decompose the
signal further into different frequency ranges called EEG bands, or sometimes called rhythms,
with each band labelled using a letter of the Greek alphabet [61, 102–104, 130]. The frequency
ranges for these bands are defined based on observations made that link signal activity in certain
frequencies to different biological actions. It can be useful in some applications to isolate certain
frequency bands depending on what action the EEG signal is trying to detect (using their specified
frequency range as the limits of a bandpass filter). It should be noted that the cut-offs for each
0.1
Amplitude [mV]
0.05
-0.05
-0.1
0 2 4 6 8 10 12 14 16 18 20
Time [s]
(a) Before bandpass filtering
0.1
Amplitude [mV]
0.05
-0.05
-0.1
0 2 4 6 8 10 12 14 16 18 20
Time [s]
(b) After bandpass filtering
Figure 2.10: Two plots showcasing the difference in EEG signal response before (a) and after (b)
applying a 0.5–40 Hz bandpass filter. The EEG signal in (a) is significantly corrupted
by high-frequency noise, which is reduced significantly after bandpass filtering.
frequency band are not an exact number and different researchers may vary the numbers slightly
[61, 102–104, 130]. However, the general range of each band is typically agreed upon. Table 2.1
lists the various EEG bands with commonly used frequency cut-off ranges, as well as corresponding
actions often associated with each band [130].
Table 2.1: Frequency ranges and physiological action assigned to each EEG band [130].
Name Frequency Range [Hz] Physiological Significance

Delta (δ) <4 Deep sleep
Theta (θ) 4–7 Drowsiness, meditation
Alpha (α) 8–15 Relaxation, closed eyes
Beta (β) 16–31 Active concentration
Gamma (γ) >30 Sensory processing, short-term memory
Mu (µ) 8–12 Like α, but recorded over motor cortex, motion rest state
After bandpass filtering, spatial filtering can also be applied in an attempt to improve the per-
formance of EEG-driven devices. Spatial filtering is employed to reduce the number of EEG signal
channels by combining nearby channels, or selecting optimal channels, using various methods. The
reason for combining channels is that information generated in the brain does not perfectly corre-
late to one specific channel location, due to crosstalk between electrodes. Neighbouring electrodes
can pick up portions of the EEG signal generated over an area because of how the electrical signal
propagates through the skull [59, 105]. Some spatial filters attempt to reconstruct the proper EEG
signal, by removing other EEG information picked up by nearby electrodes [59, 118]. There are
multiple spatial filtering algorithms used to achieve this, with a popular method being the Lapla-
cian filter [59, 105, 108]. Another use of spatial filtering is to perform channel selection, choosing
an optimal subset of electrodes for each subject to improve performance [125]. A commonly used
method to do this is the Common Spatial Pattern (CSP) algorithm, which weights the channels
to maximize variance between different mental tasks being detected [125, 126, 130].
The main drawback of spatial filtering is that it requires a larger number of EEG electrodes
to be effective. For each electrode location you wish to measure you need enough neighbouring
electrodes to properly apply the spatial filter of choice. For wearable devices this can be a challenge,
as allowable hardware size and processing power is limited, and each measurement channel added
increases the overall complexity of the system. In some cases where not enough electrodes are
available, research as shown it is actually better to forgo spatial filtering and use all channels
instead. A previous study found that when only using 3 EEG channels, classification accuracy
dropped when spatial filtering was applied, compared to the accuracy obtained when using all
of the original channels [131]. The requirements and limits of the wearable mechatronic system
should be considered when deciding if spatial filtering is appropriate for a certain device.
Finally, for some EEG applications, further noise removal is necessary to be able to use the
signal properly. Noise caused by other bioelectrical signals or external sources can greatly affect the
EEG signal, and sometimes bandpass filtering is not sufficient to completely remove all interference.
In this case, other more advanced filtering methods can be used to try and remove unwanted
contributions to the EEG signal [129]. One method used for this is called Independent Component
Analysis (ICA). ICA breaks the EEG signal into multiple base components and attempts to remove
2.4 Feature Extraction 35
the unwanted components contained within the signal before reconstructing it. A common example
of this technique is to use ICA to remove ECG signal contamination from the recorded EEG
signal [132]. Since ECG is a very recognizable waveform, it is possible to identify its components
within the EEG signal and remove them using ICA. Note that the use of ICA (and other similar
techniques [124]) is not necessarily exclusive to EEG, and the technique can also be applied to
EMG if the situation demands it [133]. Unfortunately, more advanced filtering techniques, like
ICA, are often very computationally expensive and are usually performed during offline analysis
where computation power/time is not as limited. In a wearable application, it may not be feasible
to make use of these methods, in which case the system will need to be robust enough to use EEG
signals containing noise from other sources.
2.4 Feature Extraction
Once the EEG and EMG signals are processed, they can then be employed to decode user intention
for controlling a wearable device. Due to the stochastic nature of both bioelectrical signals, they
are not suitable for use as direct control system inputs: their time-domain behaviour is highly
variable and erratic (as can be see when observing both signal types visually) and it is hard to
extract any meaningful information by just observing the raw signals. For this reason, researchers
have developed metrics, called features, that can be calculated to quantify different aspects of
EEG/EMG signal behaviour [53, 55, 56, 59, 59, 61, 72, 96, 98, 102, 103, 128, 134–136]. Each feature
takes a segment of the signal and provides a single numerical value to summarize the behaviour of
that signal over the specified segment. The actual real-world meaning of that “score,” and what
movement/task it relates to, will depend on what feature is being calculated and what type of
information from the signal it focuses on. Different features have been designed to target specific
signal behaviours, based on what components of the signal researchers believe are most biologically
relevant. There are many possible features that could be calculated when seeking to extract
meaningful information from EEG/EMG, and the decision of which to employ can have a large
impact on the success of intention decoding procedures [53, 55, 56, 59, 59, 61, 72, 96, 98, 102, 103,
134–136]. Feature sensitivity can be very task specific and the determination of an optimal feature-
set for EEG/EMG signals is very much an ongoing area of research [55, 61, 98, 102, 103, 134–136].
2.4.1 Segmentation and Windowing
Before feature extraction can occur, the bioelectrical signals need to be segmented to ensure that
only the areas of interest are included in the calculations. When recording EEG and EMG there
are often unnecessary portions of the signal, such as the small segment of data collected during
the time between when the recording was started and the subject starts moving, or similarly the
section of the signal measured after the subject finished the motion trial, but before the recording
was stopped. These should not be included in the feature extraction procedure since they are
not relevant to the task being studied and may skew the results. Segmentation may also be
used to separate different regions of interest to sort the data based on the different events/tasks
that occurred. It may be beneficial to the analysis, for example, to have the EEG/EMG signals
separated into two groups based on regions where motion occurred and regions where the subject
was at rest. Segmentation can be performed using many techniques depending on the information
available from the data collection protocol. For example, markers placed in the data using an
external system or analysis of video recording during EEG/EMG signal acquisition can be used
to determine the timestamps for segmentation.
After segmentation, it is common to divide the regions of interest further using a process
referred as windowing. After calculating a feature, a single value is obtained for the entire length
of the signal used during the calculation, and while it is possible to calculate a feature over the
entire recorded EEG/EMG signal it would result in a large loss of information. Not only does
this result in less data points to use during model development (something which is always a
challenge when utilizing data-driven machine-learning techniques), it also removes any ability to
analyse signal changes over time, as the temporal resolution of the feature measurement is all but
destroyed. Instead, it is much better to calculate the same feature many times over the length
of the acquired signal’s region of interest, to provide more data points and to observe how the
feature value varies with time [53, 55–57, 59, 61, 72, 96, 108, 128, 135]. Calculating features in this
manner is referred to as windowing (also referred to as a sliding window). In windowing, a short
duration of the signal (referred to as a window) is used to extract the desired features, then the
window is slid forward to the next segment of EEG/EMG and the features are calculated again
for the new duration of the signal [55, 56, 96, 98, 108, 134, 135]. This process continues until the
window has moved across the entire length of the signal recording. When implementing feature
windows there are two parameters to consider. The first parameter is the duration of signal that
fits inside the window, referred to as the window length, and it is usually expressed as a unit of
time (for example, a window length of 250 ms) [55, 56, 61, 96, 108, 128]. The second parameter is
the amount of window overlap, expressed either as a percentage of the window length of as a unit
of time directly (for example, a window overlap of 50% or an overlap of 125 ms for a 250 ms long
window) [55, 56, 96, 108, 128]. The overlap, while specifying how much (if any) of the previous
window is included in the current window, also controls the rate at which the window advances
along the signal (for example, given a window length of 200 ms and an overlap of 75%, the window
would step forward in 50 ms increments). For a visual representation of how windows are applied
to a signal, see Figure 2.11.
0.1
Window Length
0.05
Amplitude [mV]
-0.05
Window Overlap
-0.1
0 50 100 150 200 250 300 350 400 450 500
Time [ms]
Figure 2.11: A demonstration of how a sliding window would be applied for an example EEG
signal. The windows are shown on the plot using red lines, with this example utilizing
a window length of 200 ms with a 50% (or 100 ms) window overlap. Both parameters
are labelled on the plot using the black dimension lines. The first window spans from
0 ms to 200 ms, as dictated by the window length. Then it slides forward 100 ms
such that second window spans between 100 ms and 300 ms, with the amount that
the window advances being dictated by the window overlap. This pattern continues
for the third window, and so on, until the full length of the EEG signal was been
windowed.
Both window parameter values need to be chosen in a manner that allows for proper feature
extraction, ensuring that enough of the signal is included to adequately capture the desired effect
while still providing acceptable temporal resolution and efficient computation. In the context of
a real-time device, these window parameters would affect the amount of time required in between
each update of the system. If a 250 ms window length is used (with no overlap), this means that
the system must wait until at least 250 ms worth of EEG/EMG has been recorded before feature
values can be recalculated and decisions can be made using the new information (not including
any delays incurred due to processing time). If bigger window lengths are implemented, it may
result in a much longer delay in system response (which may or may not be acceptable depending
on the application). For wearable mechatronic devices, these longer window lengths can present a
significant challenge. For real-time applications, research has shown that a delay greater than 300
ms can have a major impact on a subject’s ability to control a device [55, 137]; therefore, these
systems are often limited in the window sizes they can utilize.
For EMG feature extraction, a variety of different window lengths and overlaps have been
demonstrated in the literature, and there is no universally agreed upon choice [55, 56, 96]. Re-
searchers have utilized wide variety of window lengths, with durations of 100 ms [96], 125 ms
[55, 56], 200 ms [55, 57, 96], 250 ms [57, 96], 256 ms [55, 56], 300 ms [56, 96], 400 ms [56, 96], and
500 [55, 56] being demonstrated by a many different groups. Determining optimal EMG window
length and overlap is still an ongoing research topic [55, 56, 96, 138] and it is likely that these
parameters are highly task dependent and should be tuned for each specific system.
For EEG feature extraction, since the response of the EEG signal is slower (compared to EMG),
longer window lengths are sometimes used. However, like EMG, there is no general consensus on
optimal EEG window length/overlap and they can vary greatly for different applications. Some
researchers may use shorter window lengths, such as 250 ms [107], while others may use window
lengths of 1 s [106, 108], 2 seconds [61] or longer, ranging from 10 s to 120 s [136]. As discussed
above, for wearable mechatronic devices these longer window lengths can present a significant
challenge. This means that EEG windows should be kept below 300 ms if the data can influ-
ence real-time performance of the system. EEG models need to be optimized to provide suitable
intention detection accuracy, even with window sizes smaller than what is often used in EEG
applications.
2.4.2 EMG Features
When looking to decode intention from an EMG signal, there are many different features that
have been proposed by various research groups. Each feature will extract different information
from the EMG response, and the decision of which features to include in an analysis is hardly
straight forward. Given the variable nature of EMG, it can be hard to predict which aspects
of the signal will best correlate to the underlying biological process one is attempting to detect,
especially for a novel movement task that has yet to be studied thoroughly. For this reason,
the EMG features used across the literature will vary greatly between studies, and it is best to
try and calculate many features at once to determine which are best for the current application.
Despite this, there are some trends that exist which demonstrate the popularity of certain EMG
features, with some appearing to be utilized more often in movement intention detection studies
[53, 55, 56, 72, 96, 98, 134]. To provide some examples of EMG features, a small sampling of
commonly utilized features will be discussed here; however, it is important to keep in mind that this
list is hardly exhaustive and there are countless other EMG features that could also be calculated
to decode motion intention.
One common trend observed in EMG feature extraction is the use of time-domain features
[53, 55, 56, 72, 96, 98, 134]. This refers to features that can be extracted from the filtered EMG
signal directly and do not require that the signal be converted to a frequency domain representation
using a technique like the Fourier Transform [72]. When observing EMG behaviour visually this
approach begins to make sense. The appearance of EMG changes very distinctly during movement
in a manner that can be easily recognized (the amplitude of the oscillating signal begins to increase
dramatically, as seen in Figure 2.12). Thus, it is a logical approach to employ features that attempt
to quantify this particular behaviour. Also, time-domain features have the benefit of requiring less
computation than features taken from the frequency domain, since no additional step of applying
the Fourier transformation needs to be performed on the measured signals before feature extraction
can occur [53, 55, 72, 96, 98, 134]. For wearable robotic systems this is an inherent benefit, since
minimizing computational complexity is always a target objective (both to keep control delays
small, and to account for the fact that hardware capabilities are limited in wearable devices).
Examples of a few commonly utilized time-domain EMG features are described below.
Mean Absolute Value (MAV) calculates the mean amplitude for a signal segment of
window length WL after applying the absolute value, as follows [96, 98, 134]:
WL
1 X
M AV = |xi | , (2.1)
WL
i=1
where xi is the value of sample i from the time domain signal x. This feature provides
information regarding EMG strength by quantifying amplitude height.
Mean Absolute Value Slope (MAVS) calculates the difference in MAV results between
the current window segment w and the adjacent window, for the total number of window
segments W , as follows [96, 134]:
M AV Sw = M AV w+1 − M AV w w = 1, . . . , W − 1. (2.2)
where M AV Sw is the MAVS value for the current window, M AV w is the MAV result for
the current window, and M AV w+1 is the MAV result for the next window. MAVS describes
how the EMG amplitude changes over time.
Waveform Length (WL) calculates the cumulative length of the signal over the duration
of the current window segment, as follows [96, 98, 134]:
L −1
WX
WL = |xi+1 − xi | , (2.3)
i=1
where WL is the length of window, and xi is the value of sample i from the time domain
signal x.
Zero Crossings (ZC) calculate the number of times the signal amplitude crosses the zero
value (0 mV) [96, 98, 134]. This provides frequency information about the signal using a
time-domain measurement (more zero crossings means the signal is oscillating faster). To
prevent low voltage measurement noise from affecting the calculation, a crossing is only
counted when the signal exceeds a specified threshold th. For the work in this thesis, a
threshold value of 0.02, was used. The ZC value is calculated as follows [98, 134]:
L −1
WX
ZC = fZC (xi , xi+1 ) (2.4)
i=1
where the function fZC (xi , xi+1 ) is defined as:


1
 if xi · xi+1 < 0 and |xi − xi+1 | ≥ th
fZC (xi , xi+1 ) = (2.5)


0 otherwise
Auto-regressive Coefficients (AR) are a calculated through the use of an auto-regressive

prediction model that describes frequency characteristics of the EMG signal. An auto-
regressive model represents each sample xi of the signal as the linear combination of the
previous samples xi−p with white noise added (ψ). The AR model is defined as follows
[96, 98, 134]:
A
X
xi = ap xi−p + ψi , (2.6)
p=1
where A is the AR order and the coefficients ap are used as the EMG features. When AR
features were calculated for the work in this thesis, a 4th order AR model was used (as
recommend in the literature [134]).
While the use of time-domain features with EMG is more commonly demonstrated, that does
not mean that frequency-domain EMG features are without merit and never see any utilization.
Some research has suggested that muscle fatigue information is encoded in the frequency behaviour
of the EMG signal [55, 96, 134, 139, 140], thus frequency-domain feature extraction may be worth-
while if this property is a target of investigation. Two features in particular, mean frequency
and median frequency, have shown to be the most popular features used to detect muscle fatigue
[134, 139, 140], and they are are described below to provide an example of frequency-domain EMG
features.
Mean Frequency (MNF) calculates the average frequency of the signal, computed using
its Power Spectral Density (PSD) obtained by applying the Fourier Transform. The mean
frequency is calculated as follows [134, 139, 140]:
B
X B
X
fj Pj Pj , (2.7)
j=1 j=1
where Pj is the power spectrum at the current frequency bin j, fj is the frequency of the
current bin, and B is the total number of frequency bins (Fourier Transform length/resolu-
tion).
Median Frequency (MDF) calculates the frequency at which the power spectrum can be
divided into two sections of equal amplitude, as follows [134, 139, 140]:
B
1X
Pj , (2.8)
2
j=1
The time and frequency domain features discussed here are only a small sample of the large
potential features that could be used with EMG, included due to their popularity. Many more
features exist, and researchers are constantly developing new metrics to calculate from EMG
signals to decode useful information. It is also important to note that while the features above are
discussed in the context of EMG, nothing precludes their use in EEG feature extraction as well.
The two bioelectrical signal types demonstrate separate sets of commonly implemented features
in the literature due to their unique behaviour (since certain features target different patterns
more effectively). However, it is important to realize that there is no strict rule against applying
these calculations to either signal type and one may come across examples in the literature where
commonly used EMG features are also being calculated on EEG signals (and vice versa). One
example of this is auto-regressive coefficients, which have also been used extensively with EEG as
well [135].
2.4.3 EEG Features
Understanding the fundamental meaning behind EEG signals has been a large barrier to its use as a
control signal for wearable devices. While EMG analysis has traditionally focused on time-domain
information, much of the key information encoded in EEG has been studied in the frequency-
domain [59, 61, 141]. This can make it harder to intuitively understand what is happening in the
EEG signal visually by direct observation, without significant signal transformation. The function
of the brain itself also contributes greatly to this complexity. It is constantly active, processing
different information and controlling various bodily functions. Therefore, it is more difficult for
EEG than it is for EMG to correlate EEG signals directly to motion. As shown in Figure 2.12, the
times at which the person is either in motion or at rest is clearer in the EMG signal than it is in
the EEG signal. In general, the EEG signal shows no distinct visual pattern to help differentiate
between periods of motion and rest. This makes the selection and calculation of features an
important aspect of successfully utilizing EEG within a mechatronic device, to ensure the proper
information is extracted from the signal.
Figure 2.12: A sample EEG (top) and biceps EMG (bottom) signal recorded during three repeti-
tions of elbow flexion–extension motion. The queues given to start and stop motion
are indicated by the green and red lines, respectively. By observing the EMG signal,
a distinction between motion and rest can be observed by looking at the change in
overall muscle activity. However, it cannot be as clearly observed in the EEG signal,
as it does not present a distinct visual change between motion and rest.
Many different types of brain activity can be detected using EEG signals, such as Steady
State Visually Evoked Potentials, which are an oscillatory response to a flashing visual stimulus,
or the P300 Event-Related Potential (ERP), which presents as a positive voltage peak in EEG
amplitude that occurs roughly 300 ms after an unexpected stimulus [59, 71]. Another ERP that can
particularly relevant for wearable robotics is the Movement-Related Cortical Potential (MRCP),
where EEG amplitude undergoes a negative voltage shift corresponding to the planning and onset of
voluntary movement [142, 143]. MRCPs can be detected from EGG using time-domain amplitude
measurement features, such as Mean Amplitude (MAMP) [144].
One of the most commonly used types of brain activity sensing for control of wearable mecha-
tronic devices is Motor Imagery (MI). MI activity is detected in the motor cortex during the
presence of actual or imagined motion, and can be used by a mechatronic device to know when the
person wants to actuate the system [58, 59, 61, 71, 108]. By using the appropriate EEG features,
MI brain activity can be measured and provided as an input signal to the system.
Within the EEG signal, MI activity is recognized as a change in the signal power of certain
EEG frequency bands, namely the µ and β bands [59, 61, 108, 145]. During a real or imagined
motor action the power in these bands will decrease, which is referred to as an Event-Related
Desynchronization (ERD). Likewise, the power will increase in the absence of an MI action, known
as Event-Related Synchronization (ERS) [59, 61, 108, 145]. Detecting ERD or ERS is the primary
focus of MI-based systems and is done commonly using band power as the main feature [59, 146].
Band power calculates average signal power over a specific frequency range, which in this case
would be the ranges for the µ and β bands (see Table 2.1), to capture any increase or decrease
over time. Band power can be calculated using various methods [141], a basic approach being to
take the average PSD value (i.e., area under the PSD curve) over the desired frequency range [59].
Another common set of features used for MI detection are the Hjorth parameters [135, 136, 146].
Hjorth parameters are time-domain features that provide frequency information from EEG. This
means that they can be calculated directly from the time-domain EEG signal without requiring
any transformation to the frequency domain. This is a significant benefit, as it allows the Hjorth
parameters to provide frequency information in a less computationally expensive way, which can
be of great use for systems with limited computational resources, such as those used in wearable
mechatronic devices. There are three Hjorth parameters that provide different information on the
EEG signal. They are expressed, for the EEG signal x(t), as follows [135, 147]:
WL
X (x(i) − x̄)2
Activity = variance(x(t)) = (2.9)
WL
i=1
s
Activity( dx(t)
dt )
M obility = (2.10)
Activity(x(t))
s
M obility( dx(t)
dt )
Complexity = (2.11)
M obility(x(t))
where x̄ is the mean of x(t) and WL is the total number of samples of x(t) in the current window.
Activity provides a measurement of the average power of the EEG signal. A high value for
Activity indicates that a large number of high frequency components are present and vice versa for
low activity [135]. Mobility provides an estimate of the mean frequency of the EEG signal [147].
Complexity indicates an estimate of bandwidth [147], which provides information on the change
in frequency by the EEG signal. For scale, the lowest complexity score possible is 1, which would
indicate that the EEG signal is a pure sine wave [135]. These three parameters, while simple to
calculate, have shown promise as efficient EEG features that can be used to extract MI information
from the time-domain signal.
While these features are commonly used in EEG control tasks based on MI changes, they are
by no means the only features available. Other features, such as Wavelet Coefficients [145, 148],
have shown potential for EEG applications. There are countless features used with EEG [136],
and indeed one could likely write a chapter solely on EEG features given how many are present
in the literature. The complex nature of EEG means that no universally optimal features exist,
and it can be very application and subject specific. Care should be taken when using EEG to
develop a feature set that is optimal for the specific scenario in question, to ensure that the correct
information is being captured from the signal.
2.5 Machine-Learning Classification 46
2.5 Machine-Learning Classification
Once feature extraction is complete, a dataset of metrics that quantify various aspects of the
EEG/EMG signal response have been collected; however, in many cases, these feature values are
not suitable to act as inputs for control of a wearable robotic exoskeleton. Given the complex
underlying biology of EEG/EMG, and the highly variable behaviour displayed by the stochastic
bioelectrical signals, it can be hard to interpret the values of the extracted features directly.
Any patterns that exist in the data to relate the feature results to a person’s movement are
often concealed by this unpredictability and can be challenging to recognize without the use of
statistical analysis tools. This is where the use of machine-learning techniques can be beneficial:
to determine how trends in the EEG/EMG features relate to motion and using these trends to
make control-relevant decisions [59, 61, 102, 108, 149, 150]. Machine learning (a subset of the larger
computer-science field known as artificial intelligence [151]) is a set of statistics-based optimization
methods designed to excel at establishing a relationship between input data and its corresponding
output value (for example EEG/EMG features and the motion being performed, respectively).
When new unknown input data is provided to the model it can predict what that output value
should be based on the patterns previously observed [149–153]. This is why the output of a
machine learning model is often referred to as a prediction [149–151, 153]. Even if the model is not
necessarily trying to forecast any kind future event (as in how the term predict is commonly used),
it does not know what the correct answer should be for these unknown data, so it is “guessing”
(i.e., predicting) the output based on its knowledge of past data. In machine learning terminology,
a new point of input data is often referred to as an observation [150, 152], which comprises all
features calculated for that time point. It is important to keep in mind that a machine-learning
model has no intuitive/underlying knowledge regarding the significance of these data, or their
real-world meaning: it simply accepts a dataset of numbers, optimizes a model based on their
statistical relationships, and outputs a series of numbers as predictions. It is up to the engineer
developing the model to apply their domain knowledge to interpret the patterns and predictions
provided via machine learning techniques to determine their real-world meaning.
One method of grouping different machine-learning approaches is by the type of information
they output. Classification models, or classifiers, output predictions that are broken up into a
discrete number of classes (for example, an output being either A, B, or C) [56, 96, 128, 149–153]
and can be described by their number of output classes (a binary classifier has two outputs, a
three-class model has three outputs, etc.). Conversely, regression models are machine learning
methods that output continuous values as predictions (for example, something like temperature or
cost that can be a range of potential values) [53, 56, 72, 96, 128, 149–153]. Of the two, classification
has been the more popular choice for both EEG [59, 102] and EMG [53, 56, 72] model development.
The development of a machine learning model is referred to as training. In this procedure, a set
of training data, where the correct output predictions for the respective inputs are known, is fed
into the model and an optimization process is undergone to develop mathematical equations that
relate the input data to the output values. The model’s performance is then tested by giving it new,
unknown data that was not used for training (but where the correct outputs are still known) to see
how accurate the model’s predictions are. This type of training is referred to as Supervised Learning
[149–151], because the correct outputs of the training/testing data are known. There is also a type
of training referred to as Unsupervised Learning, where only input data is available and the correct
outputs are unknown, requiring the model also to learn how the input data should be grouped
during training [149–151]. Supervised learning is the primary method used for classifier/regressor
development [149–151] given its foundations as an input/output pattern recognition approach. Use
of Supervised Learning is typically advised if the collected dataset allows for it, which is likely why
this is the dominant approach used for EEG/EMG machine learning [61, 72].
There are many different machine-learning techniques used to develop models, so much so that
covering them all here would be far out of the scope of this Chapter. Two classifier types are
introduced in the following sections, chosen for their relevance to the work in this thesis and for
their popular use in EEG/EMG classification tasks. While these two techniques are commonly
implemented with EEG and EMG it is important to keep in mind that they are not used exclusively,
and other machine learning approaches have been demonstrated with EEG/EMG as well.
2.5.1 Support Vector Machines
The first type of machine learning used in this thesis is referred to as Support Vector Machines
(SVMs) [152, 153]. SVM classifiers work by determining a boundary line (also called a hyperplane)
that separates observations from different classes. SVMs make use of observations located on/be-
yond the borders of each class group (also called the margins) to calculate the optimal placement
of the border line to maximize its distance to both margins [152, 153]. These selected data points
are referred to as Support Vectors (and it is from them that the SVM method derives its name).
A visual representation of the SVM training process can be seen in Figure 2.13. Note that this
example shown is very basic, with easily distinguishable groups, only two features per observation,
and a linear boundary line; done so to demonstrate the concept of SVM training in a way that is
easier to understand.
7
Class A
Class B
6 Support Vector
Boundary Line
Upper Margin
5 Lower Margin
Feature 2
1
4 5 6 7 8
Feature 1
Figure 2.13: A visual representation of the process used to train an SVM classifier. Each obser-
vation, consisting of two features, is plotted on the feature-space with their colour
and shape representing their respective classes. The margins of both groups can be
seen plotted as dashed lines, and the observations that fall on/beyond them, chosen
as Support Vectors, are circled in black. The boundary line (shown as a solid purple
line) is placed in such a way that it maximizes its distance from both margins, sepa-
rating the two classes.
Real-world data is rarely so organized and easily separable, especially as the number of features
increases (it is easy to imagine how the dimensionality of a feature-set could quickly grow to a point
where it is impossible to comprehend visually). To accommodate this, SVMs make use of more ad-
vanced mathematical techniques to project the feature-set into higher dimensional representations
through the use of kernel functions, and will employ non-linear hyperplanes when separating class
groups [152, 153]. By design, SVMs are only able to perform binary classification (determine a
separation for two output classes); however, their use has been extended into multi-class problems
by simply training multiple SVMs for each comparison required until all pairs have been evaluated.
Training a separate SVM for every pairwise comparison is referred to as a One-Vs-One approach;
whereas, if looking to reduce the number of classifiers to train, a One-Vs-All approach (combining
all observations of the other classes into one, so that each desired class only needs one SVM trained
to evaluate it from all others) can be applied as instead [152, 153].
SVMs are an approach that could be considered to be part of the group of techniques referred
to as traditional machine learning [57]. These model types (comprising of the methods that
historically made up the majority of machine-learning work, named thusly) are differentiated by
their reliance on the manual selection of input features. Like many machine learning classifiers,
SVMs use input features to train the model and output predictions, and it is up to the engineer
developing the classifier to decide which features specifically to extract from the EEG/EMG signals
to accomplish the desired task. Traditional machine learning methods, like SVMs, make no attempt
to calculate or optimize the features being used, this all must be done ahead of time as a pre-
training step.
For this reason, the decision of which features to use for EEG and EMG classifiers can have
a huge impact on model performance. As discussed in Section 2.4, there are countless potential
EEG/EMG features that could be implemented, which makes feature selection a complicated
task. While domain knowledge of the underlying signal proprieties and literature recommendations
should always be the primary approaches to choosing features, there does exist a series of methods
developed to aid in this process, referred to as feature selection techniques [57]. Feature selection
involves the utilization of a pre-training optimization on the collected feature set, with the goal
of determining which features demonstrate the best likelihood of capturing relevant information
for the classifier (and thus should be used for training). There are many approaches to this, such
as the ReliefF algorithm, which ranks features based on their similarity/dissimilarity to nearest-
neighbours [154], or Minimum Redundancy Maximum Relevance (mRMR), which uses a statistical
measurement called Mutual Information to rank feature relevance [155].
In a similar vein to feature selection (where the goal is to reduce the size of the feature-set
through the selection of optimal features) is a separate set of techniques referred to as feature
reduction [57]. Feature reduction differs from feature selection in that it does not remove any
features, but instead combines them into a smaller number of component features (containing the
same information, but projected into a more compact representation) to use as the new reduced
feature set. A popular example of this is a statistics-based approach known as Principal Component
Analysis (PCA) [57, 156]. Both feature selection and feature reduction are commonly utilized
as part of the SVM model development pipeline, to begin the optimization process even before
classifier training has actually begun. The success of these techniques is very application dependent,
and will vary on a case-by-case basis. Usually some experimentation is required to determine which
feature selection/reduction methods to use during model development, if any at all (in some cases
they can result in worse performance and it is better to use the full feature-set instead for training).
The use of SVMs have been demonstrated extensively in both EEG [61, 65, 102, 108, 152, 157,
158] and EMG [56, 57, 152, 158] applications, with comparison studies regularly reporting that
using SVMs with EEG/EMG outperformed other classifier types. SVMs offer many benefits that
make them suitable for use with EEG and EMG driven wearable robots, such as good generalization
(avoiding overfitting and providing a degree of insensitivity to the effects of a large feature-set) and
fast prediction computation time [102, 152]. Based on recommendations from the literature and
their well demonstrated track record with EEG/EMG, SVMs were chosen as the machine learning
method to use when developing many of the classifiers described in later Chapters.
2.5.2 Convolutional Neural Networks
The other classifier type relevant to this work is referred to as a Convolutional Neural Network
(CNN). This classifier is part of a new group of machine learning techniques (often referred to as
deep learning) that differ from traditional machine learning in that they do not require the input
of manually calculated features. Deep learning techniques are designed to operate directly on raw
data inputs and as part of the training process they also seek to learn which features to extract to
provide the most relevant information [61, 102, 159]. This provides the benefit of automating more
of the model-development pipeline, as feature calculation and selection is now performed within
the classifier itself.
CNNs were originally developed for the image-processing domain [61, 159, 160]; however, their
use has expanded well beyond this area [57, 61, 70, 102, 149]. Since a digital image is nothing more
than an array of numbers representing pixel colour/intensity values, this means that CNNs can be
applied to a variety of different datasets in the same manner as other traditional machine learning
techniques. Despite their more generalized use in modern times, much of the image-processing
terminology has remained in CNN literature, leading CNN inputs to commonly be referred to as
images even if the input data is not a picture in the traditional sense [159].
CNNs operate by sliding a small array of numbers, called a kernel, around an image while
performing the dot product of the kernel values and the pixel values in the portion of the image
covered by the kernel. The resulting output is a new image that has been modified by the kernel,
increasing or diminishing certain pixels in the image based on the structure of the kernel [159, 160].
Essentially, the kernel acts as a filter, changing what information is highlighted in the image. In
fact, it is for this reason that some people use the term filter and kernel interchangeably (not to be
confused, however, with the signal processing filters discussed in Section 2.3). Depending on the
model design, multiple kernels can be applied at once during one convolution step, leading to an
output image with multiple channels that match the number of filters specified (an image channel
here referring to the number of pixel values assigned to each pixel location, like how an RBG
picture will have a separate value for each colour) [159, 160]. The output array obtained through
convolution can be thought of as the features that have been extracted from the input image, with
the kernel controlling how information is interpreted. The values in the kernel array are modified
during training to optimize the discrimination of relevant features within the image, and it is by
this process that CNNs are able to automatically “learn” which features to use [159, 160]. It very
is common practice to follow a convolution layer with something known as a max pooling layer.
Max pooling slides another window across the kernel-filtered image, this time taking the maximum
pixel value in that section of image to use in the new output while discarding the others [159, 160].
It should be noted that this action is performed on each image channel individually, meaning
that max pooling does not modify the number of channels in any way (an image with 8 channels
will still have a depth of 8 after max pooling). The purpose of max pooling is to downsample
the image to allow for more efficient computation while retaining the essential information in the
image. In some ways, max pooling can be thought of as being akin to feature selection/reduction
in traditional machine learning; reducing the dimensionality of the feature space while utilizing
the most relevant feature information [159].
This process of feature extraction using convolution and max pooling layer pairs can happen
multiple times sequentially within the same classifier, depending on how many convolution steps
are included in the model design. Filtered images outputted by a previous convolution step will
be passed along to the next convolution layer as new input images, where a different kernel will
search for new features (further refining the feature extraction process) and the output will again
be downsampled with max pooling [159, 160]. Once every convolution step has been completed,
the actual decision making in the CNN is performed using a classical neural network, referred to
as a Fully Connected layer (these can also be stacked in multiple layers, if desired) [159, 160]. A
visual representation of the CNN classification process can be seen in Figure 2.14.
The trade-off that comes with utilizing a CNN over traditional machine learning is that while
manual feature calculation is no longer required it is instead replaced by the cost of increased
model complexity [61, 70, 161]. With a CNN there are many decisions that need to be made
regarding the model configuration: number of layers, kernel size/stride, and number of filters
being just a few examples. This can result in a very involved model design process that requires
a lot of trial-and-error testing and hyperparameter optimization to properly implement the CNN.
Another drawback of CNNs is that they typically require larger amounts of data to facilitate
proper training [57, 63, 102, 149, 159, 160]. This can be a challenge when attempting to utilize
them for EEG/EMG-driven wearable devices, as biological-based data collection that relies on
human volunteers can be harder to obtain [161].
Despite these drawbacks, CNNs have shown extensive use in recent years with both EEG
[102, 161–173] and EMG [161, 174–183] classification tasks. In fact, of the deep learning techniques,
CNNs are the most commonly utilized approach for both EEG [184] and EMG [185]. This shows
2.6 EEG–EMG Fusion 53
Figure 2.14: A visual representation of the model structure of an example CNN classifier. The
input image is 128 × 128 pixels, with only one channel (i.e., a greyscale image). The
first convolution layer uses 8 filters, with a 5 × 5 kernel size, moving with a stride (i.e.,
step size) of 1, which results in an output image of size 124 × 124 with 8 channels.
This is followed by a max pooling layer with a size of 2 × 2 and a stride of 2, resulting
in an output image of size 62 × 62 with 8 channels. The second convolution layer takes
this as an input and uses the same kernel size/stride as before, but with 16 filters this
time, resulting in an output image of size 58 × 58 with 16 channels. After another
max pooling layer (size 2 × 2, stride of 2) an image of size 58 × 58 with 16 channels is
obtained. These extracted features are then supplied to two Fully Connected neural
network layers, the first using 256 units (i.e., number of neurons) and the second using
128 units, which use the information extracted by the convolution steps to make a
classification prediction.
that this modern machine learning approach does have potential as a tool for bioelectrical signal-
based classifier development, which prompted its use for some of the work presented in this thesis.
2.6 EEG–EMG Fusion
Both EEG and EMG have shown potential to be used for control of wearable devices; however,
both are hindered by unique challenges. It can be difficult to decode EEG signals, as they do not
provide a direct measurement of muscle activity and can require complex computation to be useable
[62, 64–67]. On the other hand, EMG does offer a more direct measurement of muscle activity, but
it is affected by changes in the body, such as muscle fatigue, which can drastically alter the expected
behaviour of the signal[11, 54, 64, 66, 68, 69]. One method that has been proposed to address
these challenges is the simultaneous use of both EEG and EMG to control a wearable device,
referred to as EEG–EMG fusion [54, 62, 63, 69–72, 74]. The concept of sensor fusion, combining
information from multiple sources to make a decision, has shown a lot of success in many different
domains and is become a popular technique to employ [25, 54, 70, 73, 74, 128, 161, 186, 187].
In EEG–EMG fusion, information is taken from both bioelectrical signals and combined using
various fusion algorithms into a single control signal that can be used by the mechatronic system.
Obtaining motion intention information from multiple bioelectrical sources can allow the system to
leverage the strengths of each signal to try and address their drawbacks, providing a more robust
control scheme to the user. Like all sensor fusion, this approach has also gained more popularity
in recent years [54, 62], and research into EEG–EMG fusion has become an active area of interest
[54, 62, 63, 69–72].
An important distinction to make for the discussion here is to define what exactly is consid-
ered EEG–EMG fusion for the purposes of this work. The main interest of this thesis was the
simultaneous use of EEG and EMG, where both bioelectrical signals are contributing information
towards the same decision. Therefore, the term EEG–EMG fusion here will apply only to models
where EEG and EMG are being used together at the same time and working towards the same
task [73, 186]. Some examples in the literature will report the use of EEG–EMG fusion; however,
it will be implemented by having EMG control some processes of a system while EEG controls
others, with both signals acting separately on their respective tasks [73, 188–192]. To provide
an example, EEG and EMG were used to control a robotic upper-limb prosthetic, where EMG
classification was used to position the arm and EEG classification was used to select the hand
gesture [188, 190]. An argument could be made (rightfully so) that the combined use of EEG and
EMG together in the same device still counts as EEG–EMG fusion, but for the purposes of the
work here such examples are excluded. One of the objectives of this work was to see if using the
signals together for the same task can improve performance by allowing each signal’s strengths
to compensate for their weaknesses, and implementations where EEG and EMG control different
aspects of a device would not demonstrate this trait. Future work could consider investigating
this type of EEG–EMG fusion, but it is considered outside the scope of the work outlined in this
thesis.
2.6.1 Classifier-based Fusion
The most common method of preforming EEG–EMG fusion is through the use of machine learning
classifiers, with this being the overwhelmingly dominant approach demonstrated in the literature
[64, 66, 75, 78–89, 158, 193–202]. Typically, the goal of EEG–EMG fusion is to classify some
aspect of user intention with the idea that this could act as a command signal for a wearable
mechatronic device. A popularly demonstrated application of EEG–EMG fusion is binary motion
classification, where the model outputs one of two possible decisions indicating a desired movement
action by the user [67, 68, 76, 77, 79, 81, 83, 88, 193, 193, 198, 200–209]. For example, classifying
between motion/rest [67, 77, 81, 83, 193, 203–206], knee joint flexion/extension [79], movement of
the left/right limb [68, 76, 88], hand open/closed [201, 202], or intentional/unintentional movement
(e.g., fall detection for lower-limb applications, perception assist for upper-limb applications) [198,
207–209]. Though less common, some groups have also extended EEG–EMG motion to three-class
motion intention problems, combining two motion classes (e.g., flexion/extension, left/right limb,
open/closed hand) and a third rest class indicating no movement [64, 75, 80, 88, 89, 193, 210].
Beyond that, there are also several examples EEG–EMG fusion being applied to higher-class
problems [64–66, 78, 85, 86, 158, 193–197, 201, 211]. For upper-limb applications, 4/5 class hand
gesture recognition [66, 78, 85, 86, 201, 211] appears to be the most commonly example of this,
while for lower-limb applications larger multi-output classification problems focusing on gait phase
recognition/movement type identification have been demonstrated [64, 158, 194–196]. In general,
the field of EEG–EMG fusion is still young, relatively speaking, and the fusion approaches employed
are typically based on simple combination techniques [54, 63, 70, 72, 74] and developed primarily
for basic motion classification tasks. Further work is needed to develop and validate more advanced
EEG–EMG fusion methods and use-cases, to move it beyond an experimental laboratory technique
evaluated through offline classification to full integration into a mechatronic system and use in
actual wearable robotic device control.
2.6.1.1 Input-Level Fusion
There are many different approaches to classifier-based EEG–EMG fusion, each of which can be
sorted into two general categories based on where in the model pipeline EEG and EMG are being
combined. The first group is referred to here as input-level fusion, where EEG and EMG signals
are processed separately until feature extraction is performed. At this point the EEG and EMG
features are combined into one feature set, which is then used to train a single classifier that
outputs a decision based on information from both signals (see Figure 2.15 for a summary of this
process). In this approach, the fusion of the EEG and EMG is done inside the classifier, which
uses the information from both feature sets to train the model and make predictions [25, 62, 63,
69, 128, 161, 186].
Many sensor fusion papers will further sub-divide input-level fusion, making a distinction be-
tween data-level fusion (the raw signals are combined in some manner before feature extraction)
and feature-level fusion (described above) [25, 70, 161, 161, 186, 194, 195]. For the purposes of
the work here, this differentiation is not applied and instead the terminology input-level, or Single
Classifier (SC), will be used to describe both of these types of fusion. For those specifically inter-
ested in these sub-groups of input-level fusion, the work in Chapters 3 and 4 present examples of
feature-level fusion, while the work in Chapters 5 and 6 present examples of data-level fusion.
The use of input-level EEG–EMG fusion has been demonstrated many times by various different
research groups in the literature [64, 66, 78–88, 158, 193–195, 200–202], providing strong evidence
supporting its feasibility. Historically, input-level fusion has been performed using traditional
machine learning techniques, such as SVMs [64, 80, 82–84, 158, 200, 202], Linear Discriminant
Analysis [78, 81, 82, 88, 158, 193, 194, 201], and Neural Networks [79, 82], with this comprising the
majority of examples of SC fusion in the literature. Only very recently has work begun to occur
where deep learning techniques are utilized for EEG–EMG fusion [66, 85–87, 195].
2.6.1.2 Decision-Level Fusion
The other group of EEG–EMG fusion methods is referred to here as decision-level fusion. In
decision-level fusion, both signals are processed separately and are used to train their own classifiers
(a) Input-level fusion
(b) Decision-level fusion
Figure 2.15: The processes used for both fusion types. Input-level fusion is where the information
from both bioelectrical signals is combined before classifier training and only one
model is trained using features from both EEG and EMG. Decision-level fusion is
where information from each bioelectrical signal is used to train their own classifier
and the output of these classifiers is combined into one decision, using various methods
of fusion.
(one for EEG and one for EMG). Then, each classifier will output a prediction of motion intention
and this output decision is what is fused (using various combination algorithms) to come to a
singular output prediction that considers both EEG and EMG [25, 63, 69, 128, 161, 186]. See
Figure 2.15(b) for a summary of this process (and to see how it differs from input-level fusion).
Many different techniques for fusing EEG and EMG classifier decisions have been demonstrated
by a number of research groups. A summary of the various decision-level fusion methods can be
seen in Table 2.2.
In general, decision-level fusion is slightly more common than input-level fusion for combining
EEG and EMG, although both approaches still see frequent use. When compared, two studies
Table 2.2: EEG–EMG fusion methods used for decision-level fusion.
Name Method of Fusion Reference

Weighted Average Outputs are combined using a weighted average [65, 68, 76, 82, 210–212]
of class probabilities
Belief Theory Outputs are combined using Bayesian or [68, 75, 76, 89]
Dempster-Shafer probability statistics
AND Activates if both EEG and EMG detect inten- [77, 88, 203–206]
tion
OR Activates if either EEG or EEG detect intention [77, 88, 198, 206]
EEG Gating Checks EEG first for intention. If it detects in- [64, 67, 197, 199, 209]
tention, then checks EMG
EMG Gating Checks EMG first for intention. If it detects [196, 208]
intention, then checks EEG
Majority Vote Multiple classifiers are trained for EEG and [82, 207]
EMG. Result is the most common output
found that decision-level fusion-based approaches outperformed input-level fusion for lower-limb
classification tasks [64, 88]; however, a different study found the opposite result for their lower-
limb EEG–EMG fusion models [82]. While only three comparison studies can hardly allow for a
definitive conclusion, these results do suggest that the success of each individual EEG–EMG fusion
method is likely very application/task dependent and that experimentation may be required to
determine the best technique to utilize for each situation [71]. There is an overall trend showing a
lack of comparison between fusion methods, both between input and decision-level fusion, as well
as between the various decision-level fusion methods presented in Table 2.2. Studies will typically
only focus on applying one fusion method at a time [65, 66, 70, 75, 78, 78–80, 80, 81, 81, 83,
84, 86, 89, 158, 193, 194, 196–203, 205, 208, 209, 211, 212], will use small sample sizes [62, 66–
68, 70, 78, 80, 88, 158, 195, 196, 198–201, 205, 206, 209–211], or will only compare a few fusion
methods under basic motion conditions that do not replicate real-world usage of a wearable upper-
limb robotic device [77, 82, 206]. For this reason, no one fusion method can be said to confidentially
outperform the others, and more work is needed to evaluate and compare each approach.
2.6.2 Fusing EEG and EMG with Other Biosignals
As previously mentioned, the focus of this thesis was specifically on EEG–EMG fusion and how this
could be applied to wearable robotic exoskeleton control systems. However, it is also important
to recognize that sensor fusion has been demonstrated for EEG and EMG with many other signal
types as well [23, 25, 54, 57, 70–73, 82, 161, 186]. Bioelectrical signals are not the only physiology-
based measurements that can be sensed from a person to decode motion intention, and many other
sensor types have been developed to measure other non-electric biosignals from the human body.
In the same vein that EEG–EMG fusion is hypothesized to improve performance by leveraging
the benefits of both bioelectrical signals, other work has sought to utilize this same approach by
fusing EEG or EMG with other complementary biosignals.
The first, and perhaps most obvious, example of this is fusion with motion data collected using
Inertial Measurement Units (IMUs). IMUs are sensors that contain multi-axis accelerometers and
gyroscopes to measure angular acceleration and position, respectively [57, 72, 161, 187]. Using
these, information regarding a person’s movement can be measured and quantified. When looking
to detect user intention during movement tasks IMUs present a very clear benefit, as measuring the
user’s motion directly will provide significant insight into the task the user is attempting to perform.
To validate this theory, EEG–IMU [54, 128, 186] and EMG–IMU fusion [57, 72, 128, 161, 187] have
both been demonstrated successfully in previous studies.
For detecting muscle activity, another biosignal that is often used Force Myography (FMG).
FMG is a relatively new sensor type, compared to EMG, that works by placing a band of force
sensors around a subject’s arm. When the person moves, the muscles under their skin will contract
and the force sensors in the band will detect this movement [73, 213]. FMG provides a way to
detect activity in muscles without relying on bioelectrical signal acquisition like EMG. The use of
force sensors in FMG can result in more stable signal behaviour during measurement while not
being affected as severely by skin-sweat interference as EMG [214, 215]; however, this comes at
the cost of increased hysteresis [214], which may affect detection accuracy. Given that both signal
types relate to muscle usage, it is a logical choice to utilize both EMG and FMG simultaneously
using sensor fusion techniques. EMG–FMG fusion has been demonstrated recently in the literature
[214, 215], providing another example of a different biosignal that can be fused with EMG.
Regarding brain activity detection, there is also an alternative biosignal that provides this
measurement as well. Functional Near-Infrared Spectroscopy (fNIRS) is a technique that detects
brain activity through the use of optical imaging [216, 217], as opposed to bioelectrical signal
measurement like EEG. In this method, infrared light is shone through the scalp/skull into the
brain and will reflect back to photo-detector sensors placed on against the scalp. Depending on
the blood oxygen levels present in certain regions of the in brain, the infrared light will reflect
back differently and this can be measured/quantified to infer brain activity information. Basing
the measurement technology on light instead of electricity allows fNIRS to be less susceptible
electromagnetic interference; however, the response time of fNIRS is very slow compared to EEG
(using window lengths on the order of 2–7 s [218]), which often present a challenge when trying to
implement fNIRS in real-world control settings. This makes fNIRS a perfect candidate for fusion
with EEG (to use each signal type to compensate for the other’s drawbacks) proven by the fact
that EEG–fNIRS fusion has been demonstrated previously [216, 218, 219].
These examples highlight that sensor fusion for wearable devices is a broad topic that spans
many different biosensing modalities. The scope of this thesis is specifically on EEG–EMG fusion,
but this is only one potential solution to detect user intention and many other fusion opportunities
exist. While these are not covered in this work specifically, fusion with other biosignals is something
that should be recognized here to provide proper context for this thesis, and is something that
should be considered for investigation in future work.
Chapter 3
Comparing EEG–EMG Fusion

Methods for Motion Classification
This chapter is adapted from “Performance Evaluation of EEG–EMG Fusion Methods for Motion
Classification,” published in the Proceedings of the IEEE International Conference on Rehabilita-
tion Robotics (ICORR), Toronto, Canada, June 24–28, 2019 [220].
3.1 Introduction
Wearable robotic systems have shown exciting potential for improving the lives of musculoskeletal
disorder (MSD) patients [221]. A smart brace can help restore lost function to affected limbs and
can aid in rehabilitation. An area of ongoing development related to these devices is the method
of user control. The user needs to be able to indicate to the device that a new position is required
in a way that feels comfortable and intuitive. Being able to detect when the user is trying to
move is critical for the control of assistive robotics, as it is needed as an input to the device control
system. One method of accomplishing this is through the use of a Human-Machine Interface (HMI)
designed to react to biosignals generated during motion. These HMIs use a classifier trained to
recognize patterns indicating that the user intended to move. One common example of this is
the use electromyography (EMG) as a control signal. When the user tries to move their body,
sensors detect the electrical signals generated by their muscles and the device uses this to know
61
3.1 Introduction 62
when to move. Another signal commonly used is electroencephalography (EEG). EEG signals are
the electrical activity generated by the brain, measured from the scalp, and can be used to detect
motion intention (among other things).
Previous work has shown that EMG [222] and EEG [108] signals can be used to detect motion
intention, however both methods have disadvantages. EEG signals can be difficult to decode,
as they do not necessarily provide a direct measure of muscle activity [62, 64–67]. EMG signals
do provide a direct measurement of the muscle activity, but are affected by other factors, such
as muscle fatigue altering the EMG signal [11, 54, 64, 66, 68, 69]. A possible solution to these
limitations is to combine EEG and EMG signals to allow each to compensate for the weaknesses of
the other. Using two classifiers, one for each signal type, and combining their outputs may provide
better accuracy when determining motion intention than using just EEG or EMG alone.
Other research groups have already proposed several algorithms for fusing the output of EEG–
EMG classifiers [67, 68, 77, 78, 206, 211] and while initial development has been promising, more
testing needs to be performed on these algorithms to evaluate their effectiveness. Testing often
was completed using small sample sizes [67, 68, 78, 206, 211] and has mainly focused on evaluating
a single fusion method at a time. Little work has been performed to investigate how the fusion
methods compare to each other. Two comparison evaluations have been completed on a small
subset of EEG–EMG fusion methods; however, the motions tested were restricted to button presses,
which is not representative of what is seen while operating a wearable robotic system used for
rehabilitation [77, 206]. As well, the effect of different motion parameters (such as motion speed
or weight being lifted) on the performance of EEG–EMG fusion methods has not been explored.
Before development of an EEG–EMG controlled device can progress, the methods of EEG–EMG
fusion need to be evaluated under more robust test conditions to determine which method is best
suited for control.
This work provides an evaluation of EEG–EMG fusion methods during various elbow flexion–
extension motions. EEG–EMG signals were recorded during elbow motions at different weights,
speeds, and levels of muscle fatigue and an analysis was completed to observe how different EEG–
EMG fusion methods perform as these parameters change. The performance of various EEG–
EMG fusion methods to classify motion intention was compared to using EEG and EMG alone,
3.2 Methods 63
to demonstrate the effect that each fusion method has on classification accuracy and to provide
a more comprehensive evaluation of their performance. These data help contribute to a better
understanding of the performance of EEG–EMG fusion during realistic motions, and will be used
to inform the development of EEG–EMG control methods for wearable robotic systems, improving
the reliability and safety of such devices while being controlled through HMIs.
3.2 Methods
3.2.1 Experimental Design
For this study, 18 healthy subjects (mean age 22.8 ± 3.3 years) were recruited to perform elbow
flexion–extension motions while having their EEG and EMG signals recorded. The experiment
was split into two parts (recorded during the same session) based on the motions being performed,
described in the following subsections.
3.2.1.1 Speed/Load Varying Motions
The subject completed elbow flexion–extension motions with a combination of two speeds (10 °/s,
150 °/s) while holding three different weights (0 lbs, 3 lbs, 5 lbs) for a total of six combinations.
The order of the speed/load combinations was randomized for each subject. For each combination,
the subject performed three repetitions of elbow flexion–extension, with a three second pause in-
between each. A rest period of one minute was given between combinations. The purpose of this
experiment was to examine the effect that changing the speed of motion and the load held can
have on the performance of EEG–EMG fusion methods.
3.2.1.2 Muscle Fatiguing Motions
The subject completed elbow flexion–extension motions using an increased weight and number of
repetitions. The weight was increased to 10 lbs with the number of repetitions increased to ten,
with only a one second pause in-between each. Three muscle fatiguing trials were completed for a
total of 30 repetitions, with a one minute rest given between trials. The purpose of this experiment
was to examine the effect of muscle fatigue on the performance of EEG–EMG fusion methods. The
3.2 Methods 64
intent was to use a higher load/number of repetitions to induce fatigue in the subject’s arm muscles
(without causing pain). This allowed for an evaluation using real fatigue signals, as opposed to
merely simulating fatigue by artificially degrading amplitude of the EMG signals, as other groups
have done [68, 211].
3.2.2 Signal Acquisition and Processing
Data collection was performed using an Intronix 2024F Physiological Amplifier System (Intronix
Technologies, Bolton, Canada) configured to collect EEG–EMG signals. Both signal types were
sampled at a rate of 4000 Hz.
EEG signals were recorded at the C3, C4, and Cz electrode locations specified by the Inter-
national 10–20 System (shown in Figure 3.1). These locations were chosen as they lie over the
motor cortex of the brain, meaning that they should provide relevant brain activity during motion.
Gold-cup electrodes with conductive paste were utilized to ensure a proper electrical connection
through the subject’s hair and the scalp was cleaned prior to electrode placement using an abrasive
skin-prepping gel. A referential montage was used for the bipolar electrode channels, using ear
clip electrodes as reference points (shown in Figure 3.1). After recording, the EEG signals were
filtered using a 0.5–40 Hz bandpass filter (3rd order Butterworth).
EMG signals were recorded over the biceps and triceps, placed according to the SENIAM
Project guidelines (shown in Figure 3.1). These muscles were chosen as they are the main muscles
activated during elbow flexion–extension. Adhesive electrodes were utilized and the skin was
cleaned using an alcohol swab before placement. After recording, the EMG signals had the DC
Offset removed and were filtered using a 20–500 Hz bandpass filter (4th order Butterworth).
Markers were placed in the data to indicate the beginning and end of the motion using the
trigger system included with the Intronix 2024F amplifier. The trigger was operated by the exper-
imenter running the trials, marking the data while the subject was moving. Video footage, synced
to the data, was also collected during the experiment.
3.2 Methods 65
(a) EEG (b) EMG
Figure 3.1: Photographs of the electrode placement of the EEG (a) and EMG (b) electrodes used
during the experiment. In (a) the C3 and Cz locations of the EEG electrodes are
shown and in (b) the biceps EMG electrode location is shown.
3.2.3 Feature Extraction and Classification
The overall method used to process the EEG/EMG signals, classify the data, and fuse the classifier
outputs can be seen in Figure 3.2. Following data collection and processing, both signals were
segmented into regions where motion occurred (the subject was moving their arm) and where
the subject was resting (the subject’s arm was relaxed at their side). This was completed using
the markers generated by the trigger system during data collection. Once data were segmented,
features were calculated from the EEG and EMG signals using the motion and rest segments, to be
used as inputs to train the classifiers. For EEG, the features chosen were Hjorth Parameters (time-
domain) and band power of the µ (8–13 Hz) and β (18–25 Hz) frequency bands (frequency domain).
These features were chosen due to their common use as features for classifying motor actions
using EEG [146, 147]. For EMG, five time domain features were chosen, namely Auto-Regressive
Coefficients, Mean Absolute Value, Mean Absolute Slope, Waveform Length, and Zero Crossings.
These features were chosen based on work performed by other research groups evaluating various
EMG features [134, 223]. For both EEG and EMG, features were calculate using a window size of
250 ms and a 125 ms overlap. Both feature sets were standardized using z-scores prior to classifier
training to provide a consistent scale, regardless of units.
3.2 Methods 66
Figure 3.2: The data analysis protocol used to implement and evaluate each fusion method.
Using the calculated feature sets, three Support Vector Machine (SVM) classifiers [108] were
trained to classify two states: motion and rest. The SVMs were implemented using the Statistics
and Machine Learning Toolbox available in MATLAB. As can be seen in Figure 3.2, one SVM
was trained using only EEG features and one was trained using only EMG features. The third
classifier deviates from Figure 3.2 slightly in that this classifier was trained using both EEG and
EMG features, providing an example of using Single Classifier fusion (which is discussed in more
detail in the following section). All three classifiers were trained using a leave-one-out strategy,
where one subject’s data were removed and the classifier was trained using the remaining 17
subjects. The left out subject was then used to test the classifier performance using new data that
were not included in the training. This process was iterated 18 times (once per subject), for each
of the three SVM models.
3.2.4 Fusion Methods
For this experiment, three fusion methods were implemented to combine the outputs of the EEG
and EMG classifiers, as seen in Figure 3.2, with the fourth using a combined classifier. The fusion
methods used were as follows:
AND: Computes the logical AND combination of the EEG and EMG classifier outputs. It
will output motion only if both EEG and EMG detect motion [77, 206].
OR: Computes the logical OR combination of the EEG and EMG classifier outputs. It will
3.2 Methods 67
output motion if either EEG, or EMG, or both detect motion [77, 206].
Weighted Average (WA): Computes a weighted average of the probabilities (outputted

by the EEG/EMG classifiers) that motion has occurred [68, 211]. Using the Statistics and
Machine Learning Toolbox, the SVM classifiers were configured to output a probability
of motion (as a percentage). If the probability was above 50%, the classifier would output
motion (less than 50% indicates rest). Three sets of weights were evaluated with this method:
50% EEG and 50% EMG (WA-50/50), 75% EEG and 25% EMG (WA-75/25), and 25% EEG
and 75% EMG (WA-25/75).
Single Classifier (SC): Uses both EEG and EMG features to train a single classifier to
predict motion/rest [78]. Unlike what is shown in Figure 3.2, this method is not fusing
the output of two classifiers; instead fusion happens before classification by combing the
calculated EEG and EMG features into one set. Prior to this however, EEG and EMG are
still processed separately, as is shown in Figure 3.2.
3.2.5 Evaluation
After the classifiers were trained, and the outputs were fused, the performance of each fusion
method was evaluated. Using data acquired from the subject that was left out of training, predic-
tions were obtained to classify if they were moving or at rest, based on the output of the different
fusion methods (as well as the output of using just the EEG and EMG classifiers). These outputs
were compared to what the subject was actually doing at that time, using the segmented motion
regions made using the trigger markers. The number of correct responses across all of the left-out
subject’s data was totaled and was used to calculate an accuracy score (in percentage of correct
predictions) for each fusion method. This process was then iterated for each left-out subject and
the accuracies were averaged across all iterations to obtain a performance metric for each fusion
method.
Once accuracy scores were obtained, an analysis was performed to evaluate different aspects
of EEG–EMG fusion performance. First, the accuracies across all motion types were averaged to
obtain an overall accuracy for each fusion method. This was done to allow for a general compari-
3.3 Results 68
son of each fusion method. Following this, accuracy scores were calculated for each speed/weight
combination, to see the effects that changing these parameters have on EEG–EMG fusion perfor-
mance. Finally, an average accuracy was calculated during the muscle fatiguing motions to see the
effect that fatigue has on EEG–EMG fusion performance.
Statistical analysis was performed using IBM SPSS Statistics to determine if the obtained
accuracies showed significant differences between groups. For the average accuracies calculated
overall and during muscle fatigue, a one-way analysis of variance (ANOVA) using Tukey’s test
for post-hoc analysis was utilized to compare means across fusion methods. A repeated measures
ANOVA was performed on the speed/load varying accuracies to determine if speed and weight
showed a statistically significant effect on the accuracy of each fusion method. This analysis was
also performed on levels of muscle fatigue, to determine if the accuracy of each fusion method
changed as fatigue increased.
3.3 Results
3.3.1 Overall Accuracy
The overall accuracy results for each fusion method are summarized in Figure 3.3. EMG alone
showed the highest accuracy, with an average of 86.81 ± 3.98% and EEG alone had the low-
est accuracy, with an average of 70.98 ± 3.03%. Of the fusion methods AND, Single-Classifier,
Weighted Average 50/50, and Weighted Average 25/75 performed comparably to EMG, with no
statistically significant difference found between their accuracies. The OR and Weighted Average
75/25 fusion methods were found to perform at the same level as EEG, showing no statistically
significant difference between means. A significant difference was found between EMG and EEG,
indicating that EMG, along with similarly performing fusion methods, showed better classification
accuracy. The p values obtained from the one-way ANOVA comparing each method can be found
in Table 3.1.
3.3 Results 69
Figure 3.3: The mean accuracy of each fusion method calculated across all motion types. Error
bars represent ± one standard deviation.
Table 3.1: p values obtained from the one-way ANOVA comparing overall accuracy of fusion meth-
ods.
p values for comparisons of overall accuracy

Fusion Methods EEG EMG SC AND OR WA-50/50 WA-75/25 WA-25/75
EEG - <0.001 <0.001 <0.001 0.870 <0.001 0.094 <0.001
EMG <0.001 - 1.000 0.870 <0.001 1.000 <0.001 1.000
SC <0.001 1.000 - 0.919 <0.001 1.000 <0.001 1.000
AND <0.001 0.870 0.919 - <0.001 0.944 <0.001 0.833
OR 0.870 <0.001 <0.001 <0.001 - <0.001 0.821 <0.001
WA-50/50 <0.001 1.000 1.000 0.944 <0.001 - <0.001 1.000
WA-75/25 0.094 <0.001 <0.001 <0.001 0.821 <0.001 - <0.001
WA-25/75 <0.001 1.000 1.000 0.883 <0.001 1.000 <0.001 -
3.3.2 Speed/Load Varying Accuracy
The accuracy results for each fusion method while moving at the slow speed with different weights
can be seen in Figure 3.4. The accuracy results for all weights at the fast speed can be seen in
Figure 3.5. Comparing the performance of each fusion method shows a similar trend to what was
seen with overall accuracy, while moving at the fast speed. While moving at a slow speed, there
3.3 Results 70
was much less variation in performance across methods, especially when the weight was increased
from 0 lbs.
A statistically significant difference in accuracy was found between the two speeds (p<0.001),
which can be seen by the decrease in overall performance when movement speed increases. No
significant difference in performance was found when the weight was modified (p=0.500), how-
ever, the observed power obtained for this was quite low (15.9%). It is likely that weight did
affect performance and that the analysis completed in this experiment merely failed to capture it.
The combined effect of changing weight and speed was found to significantly alter performance
(p<0.001). This can be seen clearly when looking at the combination of slow speed with 0 lbs,
shown in Figure 3.4. This shows increased variation in performance and a different tend in accuracy
across fusion methods, compared to other speed/weight combinations.
Figure 3.4: Mean accuracy of each fusion method at three weight levels while moving at a slow
speed. Error bars represent ± one standard deviation.
3.3.3 Muscle Fatiguing Accuracy
Average accuracy during muscle fatigue (shown in Figure 3.6) continued to follow the same trend
as the overall accuracy when comparing fusion methods. EMG, AND, Single-Classifier, Weighted
3.3 Results 71
Figure 3.5: Mean accuracy of each fusion method at three weight levels while moving at a fast
speed. Error bars represent standard deviation.
Average 50/50, and Weighted Average 25/75 performed significantly better than EEG, OR, and
Weighted Average 75/25. The p values obtained from the one-way ANOVA comparing each
method can be found in Table 3.2. The Single-Classifier method had the highest accuracy of 77.48
± 8.73% and EEG had the lowest with 66.43 ± 5.94%; however, neither performed significantly
better or worse than other methods with comparable accuracies. While comparisons of each
method followed a similar trend to overall accuracy, during muscle fatigue, the performance of all
of the methods was lower, demonstrating that fatigue does affect performance (as expected). A
weak statistically significant difference in accuracy was found between trials (p=0.017, observed
power of 71.9%), with pairwise comparisons only showing a small difference between the first and
last trial (p=0.048), indicating that continuing to increase levels of muscle fatigue did not have a
large effect on performance.
3.4 Discussion 72
Figure 3.6: The mean accuracy of each fusion method calculated across all three trials of muscle
fatiguing motions. Error bars represent ± one standard deviation.
Table 3.2: p values obtained from the one-way ANOVA comparing fusion method accuracy during
muscle fatiguing motions.
p values for comparisons of muscle fatigue accuracy

Fusion Methods EEG EMG SC AND OR WA-50/50 WA-75/25 WA-25/75
EEG - 0.002 0.001 0.011 1.000 0.014 0.986 0.005
EMG 0.002 - 1.000 1.000 0.011 1.000 0.046 1.000
SC 0.001 1.000 - 0.966 0.004 0.992 0.019 1.000
AND 0.011 1.000 0.966 - 0.042 1.000 0.139 1.000
OR 1.000 0.011 0.004 0.042 - 0.052 1.000 0.020
WA-50/50 0.014 1.000 0.992 1.000 0.052 - 0.166 1.000
WA-75/25 0.986 0.046 0.019 0.139 1.000 0.166 - 0.075
WA-25/75 0.005 1.000 1.000 1.000 0.020 1.000 0.075 -
3.4 Discussion
The results of this experiment provided a robust evaluation of EEG–EMG fusion methods under
various motion conditions, allowing for a more detailed analysis of their performance. Looking at
overall accuracy, it is interesting to see that no fusion method was able to obtain significantly better
3.4 Discussion 73
performance than what was achieved when only using EMG. This is likely due to the relatively
worse performance shown by EEG overall when compared to EMG. It is likely the lower accuracy
of EEG would pull down the performance of fusion, not allowing it to surpass EMG. Certain
fusion methods (namely OR and WA-75/25) appear to be more sensitive to this, causing their
performance to be closer to EEG alone, while other methods keep the same performance as EMG
alone, while not surpassing it. It should be noted that part of the lower performance of EEG may
be due to the process used to train the classifier. Simple filtering and feature extraction techniques
were used with EEG, which might not perform as well as more complex processes. Future work
will look at improving EEG classification performance and re-evaluating the overall performance
of the fusion methods.
Varying speed and load allowed for an analysis of how these motion parameters affect the
performance of EEG–EMG fusion. Changing speed was found to have a significant effect on the
performance of EEG–EMG fusion. EEG in particular was very sensitive to changes in speed,
performing much better during slow motions. This may be, in part, due to noise caused by motion
artifacts present during movement corrupting the EEG signal. Since subjects are likely to move
more aggressively when moving quickly, it is possible that this induced more motion artifacts in the
EEG signals than when moving at the slow speed. Implementing more complex filtering methods
on the EEG signal may alleviate this.
The combined effect of varying speed and load was also found to significantly affect the perfor-
mance of EEG–EMG fusion. This can be clearly demonstrated during the combination of a slow
speed while holding 0 lbs. For this combination, the accuracy of EMG was lower than EEG and
displayed high variability in performance across subjects. This may be due to the lower overall
amplitude of the EMG signal seen during slower motions with no weight, which would be harder
to distinguish from the base level of noise seen in the EMG recording. This phenomenon provides
one example for the justification of using EEG–EMG fusion over EMG alone, despite its overall
accuracy never surpassing EMG. During slow motion with 0 lbs, the weighted average fusion meth-
ods did not display the same accuracy drop and high variability that EMG did. This means that,
while they may not be more accurate, some EEG–EMG fusion methods can perform more consis-
tently than using EMG alone, and are not necessarily as sensitive as EMG to changes in motion
3.4 Discussion 74
parameters (due to the ability to use EEG to compensate during motions where amplitude of the
EMG signal is lower). This is a meaningful conclusion since it demonstrates that using EEG–EMG
fusion will allow for similar performance, regardless of the combination of speed/weight. This ro-
bust behaviour can be used to simplify the control of a wearable robotic device. Having consistent
performance across different motions means that the device can expect the same motion intention
signal to be delivered, regardless of the action the user is performing (holding something or empty
handed, fast or slow). This will allow for a reduction in the complexity of the device control
system by not requiring it to respond to task specific signals. A simplified control system allows
for implementation using less powerful hardware, reducing the cost and the size of the device.
This is particularly crucial for wearable robotic systems, where the availability of computationally
powerful hardware is limited. It should be noted that a small range of weights and speeds was
evaluated in this experiment and further testing should be performed using more speed/weight
combinations to expand upon the results proposed here.
Finally, muscle fatigue was shown to have a significant effect on EEG–EMG fusion performance.
Introducing fatigue caused an overall drop in performance for all fusion methods, although it did
not change the trend seen in the comparison of methods. Another interesting conclusion from this
experiment was that different fatigue levels tested did not demonstrate a large effect on fusion
performance; after the initial reduction in accuracy, increasing fatigue further resulted in only
minor changes. This also provides useful implications towards the control of an assistive robot. If
fatigue levels only alter performance slightly, less complex methods of compensation can be used,
since the device does not need to adapt to extreme changes. It is important to note that this
experiment did not look into severe levels of fatigue, or maintaining fatigue past initial discomfort.
This was done to ensure the safety of study participants, but it may be possible that increasing
fatigue levels further will eventually result in a larger effect on performance. A more varied analysis
of fatigue should be completed in future work to expand upon the results proposed here.
3.5 Conclusion 75
3.5 Conclusion
This work provided a robust evaluation of EEG–EMG fusion methods when classifying motion
intention during elbow flexion–extension motions. It showed the effect that varying speed and
load can have on classification accuracy, as well as the effect of muscle fatigue on EEG–EMG
fusion methods. The results of this work can provide further justification for the use of EEG–EMG
fusion methods, showing that while they may not improve overall accuracy what they do offer is
more consistent performance that is not as sensitive to changes seen during different combinations
of weight and speed. This work also showed that different levels of muscle fatigue have a small
effect on the performance of EEG–EMG fusion methods, after the initial reduction in accuracy.
These findings will allow for simplification of control strategies used in wearable robotic systems,
reducing cost and complexity of their implementation. Future work will focus on expanding the
initial analysis performed here to evaluate performance at greater levels of speed, weight, and
fatigue, as well as improving performance of EEG classification. Improving EEG–EMG fusion will
ultimately lead to smarter assistive devices that are able to recognize the user’s intention better,
allowing for increased comfort and safety during use.
Chapter 4
Task Weight Classification Using

EEG–EMG Fusion
This chapter is adapted from “Classification of Task Weight During Dynamic Motion Using EEG–
EMG Fusion,” published in IEEE Sensors Journal, 2021 [224].
4.1 Introduction
Musculoskeletal disorders are a variety of conditions that affect a patient’s ability to move their
body. They can range from short-term health issues, like muscle sprains, to life-long conditions,
such as arthritis. Worldwide, musculoskeletal disorders are the biggest contributor to disability
[4]. Recently, advances in technology have allowed for a new method of assistance and therapy to
emerge as a potential way to improve the lives of patients afflicted with musculoskeletal disorders:
wearable mechatronic rehabilitation devices. These systems are worn by the patient to provide
assistance in motion by helping them move their injured limbs, or to simulate therapeutic exercises
normally administered by a clinician. While progress has been made in the development of these
devices [221], further improvements are required before they can be widely adopted by the medical
community. One large barrier still impeding their use is the method of user control [6]. To be
accepted by patients, the device needs to be comfortable to wear and simple to operate; it should
not impede the user in any way and, ideally, feel as natural to control as moving their unaffected
76
4.1 Introduction 77
limb. Currently, an issue preventing this intuitive usability is the robustness of the control systems
used to operate such devices. They are often tuned to move a user’s limb under very specific
conditions (for example empty handed at a controlled speed) and can become unstable in the
presence of unknown external disturbances caused by different tasks, such as the user holding a
weighted object. Many current control system techniques used in wearable mechatronic devices lack
the adaptability needed to operate during changing motion conditions and require re-optimization
for each individual motion task [6, 101]. To properly assist the user with daily life, the mechatronic
system should be able to be operated during many different activities where motion parameters may
change. The user should be able to wear the device continuously as they change activities, without
having to manually update the system between each task. Thus, to move towards control systems
that are capable of dynamically tuning their optimization to respond to external disturbances
caused by changing motion tasks, a method needs to be devised to provide information to the
system about the task being performed. The challenge then is to find a reliable method of sensing
such disturbances. Changes in motion speed can be easily measured using conventional sensing
technology; many wearable mechatronic rehabilitation devices already employ the use of encoders,
or other similar sensors, in their control systems. Changes in task weight (i.e., the weight being
supported by the limb) are not as simple to measure. Traditional force sensors are difficult to
implement in a wearable device in a way that is unobtrusive to the user, and may be limited in the
types of forces they can measure [14, 45, 50]. For example, it is possible with current technology to
embed a force sensor into a glove worn in tandem with an upper-limb mechatronic device, but the
user will likely find this glove cumbersome and may not like having to wear it to perform all tasks
[56]. Also, depending on the sensor configuration, it may only be able to measure forces when
these are applied in a specific direction, such as down into the palm of the glove. To accommodate
the unique constraints required by a wearable device, new sensing methods need to be explored
to address the challenge of detecting changes in task weight when using an upper-limb wearable
mechatronic rehabilitation device.
One potential solution to this problem is the use of bioelectrical signals to classify task weight.
Bioelectrical signals, such as electroencephalography (EEG) generating brain activity, or elec-
tromyography (EMG) generating muscle activity, are produced by the body during planning and
4.1 Introduction 78
execution of motion tasks (among other things). Encoded in these signals (which can be mea-
sured through the use of surface electrodes on the skin) is information on how the body is altering
muscle force [140, 225–227] while regulating movement in the presence of external disturbances,
such as a weight being held. Through the use of signal processing and machine learning to decode
this information, it can be possible to detect changes in task weight without requiring the use of
traditional force sensors embedded in glove-like devices. This will offer an alternative to the user
that will not restrict hand motion during tasks, allowing them to move more comfortably while
measuring only their bioelectrical signals from electrodes placed elsewhere on the body. Both EEG
[144, 228, 229] and EMG [139, 230, 231] have been used before in the classification of muscle force
(often expressed as a percentage of subject’s maximum voluntary contraction); however, these
models are limited in their ability to be used for wearable mechatronic rehabilitation devices.
Previous studies using EEG and EMG focus mainly on isometric muscle contractions recorded
while the subject is not moving [139, 144, 226–228, 230, 231] (for example finger pinch strength
[230]). Other studies aim to classify the type of movement (e.g., power grasp vs. precision grasp)
[229], but not to quantify the actual disturbance that the system would see as the user changes
task weight. While many experiments have managed to obtain high accuracies while classifying
muscle force (for example, 73% using EEG [144] or even 99.8% using EMG [230]), these types of
simple isometric motion tasks do not fully represent the dynamic motions that would be seen by
a mechatronic system during activities of daily living. Dynamic motion introduces variability in
muscle force classification that affects its feasibility as a metric used by the control system while
compensating for changes in weight held by the user. Holding the same weight, but moving at a
different speed, requires a different amount of muscle force. Hence, only providing levels of muscle
force to the control system may not properly convey information about the task weight without
further adjustment/classification to compensate for motion speed. Classifying task weight allows
the system to skip these extra steps by directly informing the control system about changes to the
held weight.
While many of these studies show that EEG and EMG can be used to classify direct muscle
force, this does not demonstrate that the same will be true for task weight. Classifying the weight
that the user is holding during dynamic motion is much less straight-forward than classifying
4.1 Introduction 79
muscle force, as changes to task weight may have a less predictable effect on EEG and EMG
signals. For example, since EMG is a measure of the electric signals used to contract muscle fibres,
at some level, it is directly related to muscle force, whereas task weight is attempting to classify
something not directly related to the subject’s biology (the weight they are holding). As such,
improvements are needed for bioelectrical signal classification methods to expand their capabilities
to task weight classification. One unexplored opportunity to achieve this is through EEG–EMG
fusion (described in in Figure 4.1). The use of sensor fusion has shown to improve machine
learning models [128, 187] and EEG–EMG hybrid models have been shown to be beneficial for
classification tasks [68, 77, 78, 128, 220]. Previous work involving EEG–EMG fusion, however, has
mainly focused on classifying motion using binary models [68, 77, 220] or basic gesture recognition
[78]. Further development is needed to expand the capabilities of EEG–EMG fusion to allow for
multi-class outputs that focus on more challenging tasks. Towards this goal, the specific objective
of this study is to use EEG–EMG fusion to develop models that allow for task weight classification.
Figure 4.1: A visual representation of the concept of EEG–EMG fusion for task weight classifica-
tion. When the user of a wearable mechatronic brace holds an object, their EEG and
EMG signals are recorded via attached electrodes and utilized to determine the weight
of said object. This information is then passed to the brace control system so it can
adjust to the external load accordingly.
4.2 Methods 80
4.2 Methods
4.2.1 Data Collection
To develop the task weight classification models, 32 healthy subjects (mean age 24.9 ± 5.4 years)
were recruited to have their EEG and EMG signals recorded while performing elbow flexion–
extension motions for different combinations of movement speed and weight being held. A portion
of the data obtained from these subjects were used in the preliminary study presented in Chapter 3.
Recruitment of volunteer study participants began following approval from the Human Research
Ethics Board at Western University (Project ID: 112023). Data collection implemented a 2 ×
3 full factorial repeated measures study design for varying weight and speed. Two speed levels
(approximately 10°/s and 150°/s) and three weight levels (0 lbs, 3 lbs, 5 lbs) were combined for
a total of six pairings with each combination being recorded as one trial. Motion speed was self-
regulated by the study participant, who was shown by the experimenter how to count seconds
during each flexion–extension repetition to achieve a 30 second motion duration for the slow speed
(10°/s) and a two second duration for the fast speed (150°/s). This resulted in approximately
the desired speed, assuming a 150° range of motion [232–234]. The order in which the speed–
weight combinations were performed was randomized for each subject. Three repetitions of elbow
flexion–extension motion were performed per speed–weight combination trial, with a three second
pause in-between each repetition. Including the three elbow flexion–extension repetitions, and
the two pauses in-between each repetition, the total duration of a slow speed and fast speed trial
was approximately 96 seconds and 12 seconds, respectively. A one minute rest period was given
between each speed–weight combination trial. For all motions, the subject was instructed to use
their dominant arm (30 right handed, 2 left handed).
Bioelectrical signals were acquired using an Intronix 2024F Physiological Amplifier System
(Intronix Technologies, Bolton, Canada), configured to collect EEG and EMG signals. Both
bioelectrical signals were sampled at 4000 Hz. A ground electrode was placed over the elbow bone
of the subject’s non-dominant arm to act as the system ground for the differential amplifier used
by the Intronix 2024F Physiological Amplifier System.
In order to provide a sufficient electrical connection through the subject’s hair, EEG signals
4.2 Methods 81
were measured using gold-cup electrodes with conductive paste. The electrode site was cleaned
with an abrasive gel prior to electrode placement, in order to remove any oil/dead skin that may
cause poor signal measurement. Three electrodes were placed over the C3, C4, and Cz locations
on the head, following the 10–20 International System (shown in Figure 4.2), chosen because
these locations correlate with the motor cortex of the brain and will provide relevant signals
during movement. Signal recording was done using bipolar channels configured with a referential
montage, using ear clip electrodes connected to the subject’s ear lobe as the reference point. A
small number of EEG channels were used to simulate a signal collection paradigm similar to what
would be possible with a wearable mechatronic device. Wearable devices are heavily limited in
both power and computational resources, making high-density EEG recording often not feasible.
Such EEG measurement systems can also be uncomfortable to wear for extended periods and
require complex setup procedures. This places a large burden on the user, which can negatively
impact adoption of the wearable mechatronic device.
(a) EEG electrodes (b) EMG electrodes
Figure 4.2: The placement of the EEG (a) and EMG (b) electrodes used during the experiment.
(a) EEG electrode placement according to the 10–20 International system. The red
locations indicate the electrodes used. (b) EMG electrode placement on a subject’s
arm following the SENIAM Project guidelines.
EMG signals were measured using bipolar electrodes placed over the biceps and triceps muscles,
as these are major muscles used during elbow flexion–extension motion. Electrodes were placed
4.2 Methods 82
according to the SENIAM Project guidelines (shown in Figure 4.2) and the subject’s skin was
cleaned prior to placement using an alcohol swab.
Each data collection session lasted approximately two hours, including obtaining informed con-
sent, setting up and calibrating the signal acquisition system, and implementing the experimental
protocol. Depending on the participant, setup was typically one hour long and data collection was
approximately 45 minutes in duration.
4.2.2 Signal Processing
Once the EEG and EMG signals were collected, an offline analysis was performed in MATLAB
R2019b to process the signals and train the EEG–EMG fusion models to classify task weight. The
Signal Processing Toolbox, as well as the Statistics and Machine Learning Toolbox, were utilized
along with custom developed code. The process followed to develop the models is summarized in
Figure 4.3.
Figure 4.3: The protocol used to process the bioelectrical signals, train the SVM classifiers, and
implement both EEG–EMG fusion methods used to classify task weight. For Weighted
Average fusion (shown in green), separate classifiers are trained for each signal and
their outputs are combined by averaging class probabilities. For Single Classifier fusion
(shown in purple), features from both signals are combined into one set and used to
train one classifier.
EEG signals were bandpass filtered between 0.5 and 100 Hz at the time of recording using the
built-in hardware filters in the Intronix 2024F. The signals were then filtered further in software
using a 0.5–40 Hz bandpass filter (3rd order Butterworth).
4.2 Methods 83
EMG signals were bandpass filtered between 20 and 500 Hz at the time of recording using the
built in hardware filters in the Intronix 2024F. In software, the DC Offset was removed from the
signals and they were filtered again using another 20–500 Hz bandpass filter (4th order Butter-
worth).
Following signal processing, both EEG and EMG signals were segmented to only include the
sections of each recording in which the subject was moving. This was accomplished using start/stop
markers placed in the data during recording by an external trigger system, operated by the exper-
imenter. Synchronized video footage of the subject was also recorded during the experiment.
4.2.3 Feature Extraction
Once the signals had been filtered and processed, features were calculated from the EEG and
EMG signals to extract relevant information that relates to the weight a subject was holding
during motion. For both EEG and EMG, features were collected from similar studies found in the
literature. While these studies do not perfectly match the task of classifying task weight during
dynamic motion, their feature analysis presents a good starting point for potentially relevant
features. The features calculated for this dataset are summarized in Table 4.1. Every feature was
calculated for all channels of their respective signal types. All features were standardized before
classifier training using z-scores to ensure a consistent scale between different units.
To increase the number of observations from each signal recording, features were calculated
across a 250 ms sliding window with 50% overlap for both EEG and EMG. The features calculated
from every window, across the entire duration of elbow flexion–extension motion, were used as
inputs for the classifiers. It has been shown that real-time control targets need to limit system
delay to be below 300 ms for the device to be useable [137]. Even though this study conducted an
offline analysis, a window size that could accommodate this target was used to simulate conditions
closer to what would be seen if the model was implemented in an actual wearable mechatronic
device.
Along with the EEG/EMG features shown in Table 4.1, a categorical feature was added to
the dataset so that information regarding motion speed would be included in the model. Changes
in motion speed for the same task weight require different levels of muscle force to be applied by
4.2 Methods 84
Table 4.1: List of the EEG and EMG features calculated during this experiment. References and
the abbreviations used in this paper are included.
Feature Signal Abbr. Reference

Hjorth Parameters: Activity EEG HA [229]
Mobility EEG HM [229]
Complexity EEG HC [229]
Fractal Dimension EEG FD [226, 235]
Mean Amplitude EEG MAMP [144]
Point of Maximum Negativity EEG PMN [144]
Band Power: µ (8 – 13 Hz) EEG MBP [228]
β (14 – 30 Hz) EEG BBP [228]
Upper β (21 – 30 Hz) EEG UBBP [228]
Lower β (14 – 20 Hz) EEG LBBP [228]
Mean Absolute Value EMG MAV [134, 230]
Window Length EMG WL [134, 230]
Zero Crossings EMG ZC [134]
Autoregressive Coefficients
EMG AR1 – AR4 [134]
(4th Order)
Mean Frequency EMG MNF [134, 139, 140]
Median Frequency EMG MDF [134, 139, 140]
Maximum Fractal Length EMG MFL [230, 236]
Difference Absolute Standard
EMG DASDV [230]
Deviation Value
Mean Absolute Value Slope EMG MAVS [134]
Root Mean Square EEG, EMG RMS [134, 227, 230]
the subject, which can result in changes in the EEG and EMG signals. To be able to consistently
classify task weight at different speeds, the classifier may require information about motion speed
that cannot be easily obtained from the biosignals. To provide this, a discrete speed label denoting
if the features were calculated from a slow motion trial (0) or a fast motion trial (1) were included
along with the other features. In a wearable mechatronic device, speed information will be easy
to obtain using conventional sensors (for example, encoders used to track actuator motion) so it
4.2 Methods 85
is reasonable to think that this information could be provided to the classifier. To evaluate the
robustness of EEG–EMG classification, a second dataset using the same EEG and EMG features
calculated for the first dataset, but without the speed label included was also used to train a
second set of classifiers, to see how sensitive the model would be if not provided with any speed
information.
4.2.4 Model Training and Fusion
To develop the EEG and EMG classification models, Support Vector Machine (SVM) classifiers
with a Radial Basis Function kernel [157] were trained using the calculated EEG/EMG features.
Models were developed using a subject-specific training method, where each subject had classifiers
trained using only their data. To accomplish this, data from each subject were separated into
training and testing portions. Features from the first two motion repetitions of each speed–weight
combination were used for training, and features from the third repetition were used as test data
to evaluate the model. Each subject had three classifiers trained: one using only EEG features,
one using only EMG features, and one using both EEG and EMG features combined into one
dataset (shown in Figure 4.3). This was repeated twice, using both the dataset with speed labels
and the one without, resulting in six classifiers trained per subject.
Prior to training, feature selection was performed for each subject to find an optimal set of fea-
tures based on their recorded signals. The ReliefF algorithm was chosen as the method of feature
selection. ReliefF works by calculating a weight for each feature, based on its similarity/dissim-
ilarity to nearest-neighbours of the same/different classes, which is used to rank the potential of
each feature as a predictor of the output classes [154]. A selectable parameter of the ReliefF al-
gorithm is the number of nearest-neighbours to evaluate for each selected observation (k ). If k is
too small, it will not provide enough examples to calculate an accurate weight for each feature;
if k is too large, execution time will be inefficient as unnecessary calculations will be performed.
A k value of 250 was used for all subjects. This value was chosen by grouping all training data
together and iteratively running the ReliefF algorithm using k values from 1–500 with a step-size
of 50. The resulting feature weights were plotted, and a k value was chosen by inspection to be
where the feature weights plateaued. Feature selection was done on EEG and EMG separately,
4.2 Methods 86
with the top ranked 20 EEG features and 15 EMG features being used from their totals of 33
and 26, respectively (which is 60% of the original number of features remaining after selection).
Other feature selection (Minimum Redundancy Maximum Relevance [155]) and reduction (Princi-
pal Component Analysis [156]) methods were attempted, but in both cases, they resulted in worse
classifier performance for all models when compared to ReliefF, so they were not included in the
final data analysis.
To combine EEG and EMG, two methods of fusion were used (shown in Figure 4.3). The first
method, mentioned above, was Single Classifier (SC) fusion, where EEG and EMG features are
combined into one dataset and used together to train one classifier. The other type of fusion used
Weighted Average (WA) fusion. In this method, EEG and EMG features are separated and used
to train two signal-specific classifiers (one for EEG, one for EMG). Both classifiers were configured
to output class probabilities when given new test data. These probabilities represent the likelihood
that the new observation belongs to each class and the class with the highest probability is chosen as
the final classification output. In the Weighted Average fusion method, the probabilities obtained
from the EEG and EMG classifier were combined using a weighted average, and the class with the
highest resulting probability was chosen as the output classification for the model (shown in Figure
4.3). Three weightings were evaluated: 50% EEG and 50% EMG (WA-50/50), 25% EEG and 75%
EMG (WA-25/75), and 75% EEG and 25% EMG (WA-75/25). These weights were chosen to
broadly cover three cases of balance between the two signals during fusion: both signals equally
considered (WA-50/50), EMG considered more than EEG (WA-25/75), and EEG considered more
than EMG (WA-75/25).
4.2.5 Model Evaluation
To evaluate the trained models, features calculated from each window of the third motion repetition
of each speed–weight combination (excluded from training) were passed into each model as test
data, to obtain a prediction regarding the weight that the subject was holding at each time point.
The output of both fusion methods, as well the output obtained from using just the EEG and
EMG classifiers, was compared to the weight actually being held by the subject at that time,
to determine if the model output was correct. The number of correct outputs was divided by
4.2 Methods 87
the total number of test inputs to obtain an accuracy score for each method. This accuracy
score was averaged across all subjects and all speed–weight combinations to obtain a metric for
assessing the overall performance of task weight classification using EEG–EMG fusion. This overall
accuracy was calculated for both sets of models (with and without speed labels). To determine if
the differences observed in the overall accuracy of the different fusion methods were statistically
significant, IBM SPSS 25 was used to perform a one-way Within-Subjects Analysis of Variance
(ANOVA). Pairwise comparisons were performed following the ANOVA using the Bonferroni post
hoc test. Furthermore, a two-way Within-Subjects ANOVA was performed between the with-
speed-label and without-speed-label models to determine if the effect of adding speed labels resulted
in statistically significant differences in accuracy. Normality was evaluated on the standardized
residuals of the overall accuracy results using the Shapiro-Wilk test and it was found that only the
EEG and EMG distributions were non-normal. To verify the results found by the Within-Subjects
ANOVA, the Friedman test, along with the Wilcoxon Signed Ranks test (using the Bonferroni
correction) for post-hoc pairwise comparison, were applied. The results of the non-parametric
testing methods matched the results obtained using the Within-Subjects ANOVA, so only the
ANOVA results are reported, for simplicity.
The accuracy of all fusion methods was also separated for each speed–weight combination to
see what effect different levels of speed and weight have on model performance. A three-way
Within-Subjects ANOVA was performed on the separated speed–weight accuracies obtained using
each model, with the Bonferroni post hoc test used for pairwise comparison of different levels of
weight.
To fully assess the classifiers, the class outputs for all subjects were combined and used to plot
a confusion matrix for each model. This provides a more direct method of observing classifier
performance for different classes, and provides further confirmation that the classifier training was
successful. Using the confusion matrices, class-wise precision (how likely is it that the output
class is correct) and recall (how likely is it that all observations of a specific class were correctly
classified) scores were calculated to evaluate the robustness of the models across different task
weights.
Finally, a basic evaluation of the calculated features was done by determining how many times
4.3 Results 88
each feature was selected by the ReliefF algorithm for the 32 subjects. Features that are selected
more frequently than others may imply that they are more generalizable for the larger population.
This does not necessarily imply that those features are objectively better for classification of task
weight, but it does show that they may be more useful when looking to develop subject-independent
models, and will help inform future work focusing on a more detailed feature analysis.
4.3 Results
The overall accuracy results for both of the models with and without speed labels are summarized
in Figure 4.4. For all methods, the models trained with speed labels had a higher accuracy than the
models trained without them. This difference in accuracy was found to be statistically significant
(p < 0.001). Due to the fact that the with-speed-label models performed better in all cases, the
remainder of the results presented will focus only on these models, to simplify reporting.
Comparing the different fusion methods, the WA-50/50 model had the highest accuracy (83.01
± 5.21%). This was an statistically significant improvement over using EEG (51.09 ± 13.48%, p
< 0.001) and EMG (80.52 ± 4.78%, p = 0.021) alone. The WA-25/75 model had the next highest
accuracy of 81.71 ± 4.62% with the increase over EEG (p < 0.001) and EMG (p = 0.004) being
statistically significant; however, it was not found to be statistically significantly different than
the WA-50/50 accuracy (p = 0.103). Since the increase between EMG and WA-25/75 was small,
even though the difference was statistically significant, the result is not a major finding of this
work and will not be a focus of the discussion. The SC model achieved an accuracy of 80.75 ±
6.04%, which was a statistically significant increase over EEG (p < 0.001), but not EMG (p =
1.00). Finally, the WA-75/25 model had improved accuracy over using EEG (p < 0.001), but was
worse than EMG (p = 0.003) and all other fusion methods (p < 0.001 for all comparisons). All
classifiers performed well above the chance level (33.33%) for a three-class model.
4.3 Results 89
100
With Speed Labels
Without Speed Labels
90
Percentage of correct classifications
80
70
60
50
40
30
EE
G
EM
G SC /50 /75 /25
-50 A-
25 -75
WA W WA
Figure 4.4: The mean accuracy of each fusion method calculated across all task weights and motion
speeds. Models developed with speed labels are shown in blue on the left and models
developed without speed labels are shown in orange on the right. Error bars represent
one standard deviation. Note that the y axis begins at 30% (chance level for these
models is 33.33%).
4.3.2 Speed–Weight Accuracy
The results of the speed–weight separated accuracies can be seen in Figure 4.5. The effect of speed
(p < 0.001) and weight (p < 0.001) on accuracy were both found to be statistically significant.
This can also be observed when looking at the change in accuracy seen between both levels of
speed in Figure 4.5. For all methods, increasing speed caused a decrease in average accuracy
across subjects, although the reduction was significantly less for the EEG classifier. It can also be
seen from the different lines on both plots that task weight had a large effect on average accuracy
for all methods except EEG, which remained fairly consistent. Of the three weights, 3 lbs resulted
in the worst performance for all methods (other than EEG), for both speeds. The difference in
accuracy between 3 lbs and the other two weights was statistically significant in both comparisons
(p < 0.001 and p = 0.002 for 0 lbs and 5 lbs, respectively).
4.3 Results 90
100
90
Percentage of correct classifications 80
70
60
50
40
30
20
Slow, 0 lbs Fast, 0 lbs
10 Slow, 3 lbs Fast, 3 lbs
Slow, 5 lbs Fast, 5 lbs
0
EE
G
EM
G SC
50/50 /75 /25
25 75
W A- W A- WA-
Figure 4.5: The mean accuracy of each fusion method calculated for each combination of speed
and weight. Error bars represent ± one standard deviation.
4.3.3 Classifier Performance
To simplify reporting, only the confusion matrices for the best performing fusion Weighted Average
fusion method (WA-50/50) are shown here, along with the EEG, EMG, and SC models. They
present the most relevant results to further analyze classier performance, since EEG, EMG, and
SC are the base SVM classifiers trained using different datasets and WA-50/50 presents the best
performing fusion method based on a combination of outputs from EEG and EMG. The confusion
matrices (showing the precision and recall scores) for EEG, EMG, SC, and WA-50/50, using the
combined classification results for all subjects, can be found in Figure 4.6.
Looking at the precision and recall scores, it can be seen that all models are relatively balanced
between both precision and recall metrics. The class outputs, along with precision and recall, show
that the performance of the EMG, SC, and WA-50/50 methods is affected by the task weight class,
confirming what was seen in the speed–weight accuracy plots. The classifiers had a harder time
distinguishing between the two weights being held and had a much easier time predicting when no
weight was in the subject’s hand (although WA-50/50 did perform better than EMG and SC at
4.3 Results 91
0 lbs 4419 2117 2232 50.4% 49.6% 0 lbs 8222 358 188 93.8% 6.2%
3 lbs 2023 4389 2228 50.8% 49.2% 3 lbs 262 6117 2261 70.8% 29.2%
True Class
True Class
5 lbs 2171 2066 4598 52.0% 48.0% 5 lbs 159 1863 6813 77.1% 22.9%
51.3% 51.2% 50.8% 95.1% 73.4% 73.6%
48.7% 48.8% 49.2% 4.9% 26.6% 26.4%
0 lbs 3 lbs 5 lbs 0 lbs 3 lbs 5 lbs

Predicted Class Predicted Class
(a) EEG (51.09 ± 13.48%) (b) EMG (80.52 ± 4.78%)
0 lbs 8141 421 206 92.8% 7.2% 0 lbs 8238 344 186 94.0% 6.0%
3 lbs 366 6185 2089 71.6% 28.4% 3 lbs 313 6479 1848 75.0% 25.0%
True Class
True Class
5 lbs 238 1679 6918 78.3% 21.7% 5 lbs 208 1521 7106 80.4% 19.6%
93.1% 74.7% 75.1% 94.1% 77.6% 77.7%
6.9% 25.3% 24.9% 5.9% 22.4% 22.3%

(c) SC (80.75 ± 6.04%) (d) WA-50/50 (83.01 ± 5.21%)
Figure 4.6: Confusion matrices, using the combined classification results for all subjects, for the (a)
EEG, (b) EMG, (c) Single Classifier, and (d) Weighted Average 50/50 models. Each
matrix contains a positive/negative precision score summary in the final two rows,
and a positive/negative recall score summary in final two columns. For reference, the
overall accuracy of each classifier is included next to their label.
distinguishing both 3 lbs and 5 lbs). To also confirm what was shown in the speed–weight plots,
EEG was not as affected by task weight and showed consistent performance, regardless of what
class was being output; however, its overall performance was lower than that of other methods.
4.3 Results 92
4.3.4 Feature Selection Results
Finally, plots showing the number of times each feature was selected for EEG and EMG are shown
in Figure 4.7 and Figure 4.8, respectively. For EEG, the FD and RMS features showed common
adoption for most subjects and for most channels. The selection of the remaining features appears
relatively variable across subjects, however HC and PMN showed high rates of selection as well.
MBP in particular was chosen infrequently, being used by few subjects. There appears to be no
general consensus among channels, with different features favouring different channels.
32 C3
C4
28 Cz
24
Number of subjects
20
16
12
0
HA HM HC FD AMP PMN RMS MBP BBP BBP BBP
M U L
Figure 4.7: Number of subjects for which each EEG feature was selected using the ReliefF algo-
rithm. Features from the C3 channel are on the left in red, the C4 channel are in the
middle in orange, and the Cz channel are on the right in yellow.
EMG feature selection showed less variance across subjects than EEG. Many features, namely
MAV, WL, ZC, RMS, MFL, and DASDV from the biceps (and MFL from the triceps) were used by
all subjects. MAVS and the AR features showed very little use, with AR3 and AR4 in particular
never being selected. In general, the feature selection favoured the biceps channel, except in the
case of the frequency features MNF and MDF, where the triceps channel was selected more often.
4.4 Discussion 93
32
Biceps
Triceps
28
24
Number of subjects
20
16
12
0
V WL ZC R1 R2 R3 R4 MS NF DF FL DV VS
MA A A A A R M M M AS A
M
D
Figure 4.8: Number of subjects for which each EMG feature was selected the using the ReliefF
algorithm. Features from the biceps channel are on the left in purple and the triceps
channel are on the right in blue.
4.4 Discussion
The objective of this work was to develop EEG–EMG fusion models capable of predicting the
weight being held by a subject during dynamic elbow flexion–extension motion. This work con-
tributes to the improvement of wearable mechatronic rehabilitation devices and presents an alter-
native method of sensing external disturbances caused by changes in task weight that is not reliant
on traditional force sensing technology. Providing this information to device control systems will
result in smarter wearable rehabilitation systems that are able to account better for changes in
system dynamics caused by these external disturbances, resulting in improved stability and per-
formance. This will make the device easier for the user to control and will serve to improve the
overall safety and reliability of these devices.
This study presents a step forward for the field of bioelectrical signal classification by presenting
a model capable of classifying a new measurement: task weight. Literature focusing on muscle force
classification using EEG and EMG have mainly targeted level of muscle contraction [139, 144, 228,
4.4 Discussion 94
230, 231], or grasp type [229], which may not correlate directly to the weight being held by a subject,
limiting its usefulness in a wearable mechatronic device. Many studies also only demonstrate
classification during isometric muscle contractions when the subject is not moving [139, 144, 226–
228, 230, 231]. While this does simplify the classification problem by removing variability in the
bioelectrical signals, it does not represent the types of signals seen during dynamic motion while
using a wearable mechatronic device. This study demonstrated the feasibility of classifying task
weight (a less direct measurement than muscle contraction) during dynamic movement, information
that is needed to implement adaptable control models in a real system. This work also showed
developments in the literature for EEG–EMG fusion. A large majority of prior EEG–EMG fusion
work focused on binary classification of motion intention [68, 77, 220], or basic gesture recognition
[78]. The model presented here focuses on using EEG–EMG fusion for multi-class classification
of more advanced tasks while evaluating multiple EEG–EMG fusion strategies, demonstrating an
advancement in the usage of EEG–EMG fusion.
Looking at the overall accuracy, it can be seen that every model performed well above the
chance level (33.33%) and the precision/recall scores obtained from the confusion matrices are
relatively balanced between both metrics, demonstrating good performance by the trained models.
This verifies that the methods used for model development were successful in decoding task weight
information from both the EEG and EMG signals. A statistically significant improvement was
found when using the Weighted Averaging fusion method, demonstrating that there is a benefit to
using EEG–EMG fusion over just EEG or EMG alone. As well, the overall accuracy results show
that the inclusion of a speed label has a statistically significant improvement on accuracy. This
is likely because the inclusion of speed information in training helps the classifier recognize when
changes in feature values are only due to variations in speed, and allows it to be more robust when
classifying task weight. Since dynamic motion can result in greater variability in the recorded
biosignals, it is important to provide speed information to the model as a method to combat
this challenge. The speed information provided in this experiment was merely a categorical label;
however, in a real mechatronic device, it is likely that continuous speed measurements will be
possible (though the use of sensors such as encoders). Future work should evaluate the effect of
including continuous speed measurements in the model to see if it improves accuracy further.
4.4 Discussion 95
When looking at the speed–weight separated accuracies, it can easily be seen that both dynamic
motion speed, and the task weight being held, had an effect on classifier performance. Looking
at Figure 4.5 it can be observed that increasing speed decreased performance for all methods.
One potential explanation for this is that when moving at a faster speed, more muscle force is
required, even for lower weights. This may cause the biosignals for lower task weights to more
closely resemble higher task weights at lower speed, causing misclassification. Including speed
labels in the model may improve this slightly (causing the improvement in accuracy seen between
both methods); however, this technique does not fully account for the effects caused by changes
in motion speed. It is possible that providing more detailed speed information to the classifier
(such as continuous measurements, or information about the current direction of motion, flexion
or extension) may improve the robustness of the model further. As well, more features to extract
from the biosignals could be investigated to find ones that are sensitive enough to capture speed
changes encoded within EEG and EMG. In regards to the effect of task weight on accuracy, it can
be seen that the 3 lb weight showed consistently lower accuracy when compared to the other two
weights. This is likely due to the models having a hard time separating the middle weight (3 lbs)
from the low end (0 lbs) and high end (5 lbs) weights. The larger gap between task weights causes
more significant changes in the biosignals, making them easier to classify. This is supported by
looking at the confusion matrices in Figure 4.6; it can be seen that for all models, except EEG,
the 3 lb weight had the lowest performance compared to the other classes and would frequently
get labelled incorrectly as 5 lbs. As well, the 5 lb class was also often misclassified as 3 lbs, for the
same reason. Future work should focus on continuing to refine the models with new features and
training methods to improve the resolution of task weight classification possible. A larger range
of speeds and weights should be examined to expand upon the 3 class models shown here.
An interesting observation made from the speed–weight accuracies, is that the accuracy of
the EEG model was not affected nearly as much as the other methods to changes in speed and
weight. In particular, task weight had very little effect on EEG. This is seen in the confusion
matrix as well; EEG classifies all task weights relatively the same. This is likely due to the nature
of the biosignal itself. EMG is a direct measurement of muscle activity so it stands to reason that
it would be affected more by the changes in muscle force required for different speeds/weights.
4.4 Discussion 96
EEG on the other hand is a less direct measurement, correlating to many different actions of the
body. It is possible that the changes detected in EEG may not relate purely to muscle activation
alone and may instead incorporate other factors, such as mental effort or perception of the weight
being held, that allow the classifier to operate with less sensitivity to changes in motion. This
increased complexity in the signal does come at a cost, as seen in the lower overall accuracy of
EEG compared to EMG. This trade-off is where the benefit of EEG–EMG fusion seems to lie.
Using EEG–EMG fusion allows the model to leverage the strengths of both signals, which resulted
in higher accuracy overall compared to using EEG or EMG alone. The WA-50/50 model was
able to use the more direct measurement of EMG, as well as the in-direct information of EEG, to
outperform both methods. This effect can be observed when looking at the confusion matrices of
each model. The precision/recall scores for 0 lbs is similar between EMG and WA-50/50; however,
the scores for 3 lbs and 5 lbs are improved for WA-50/50 when compared to EMG. The inclusion
of EEG information helped boost performance for task weights that EMG had a harder time
classifying. This demonstrates the benefit of Weighted Average EEG–EMG fusion and provides
further evidence for its use in classifying task weight during dynamic motion.
While this study found that EEG–EMG fusion was beneficial, and showed it is a feasible
method of task weight classification, it should be noted that the gains demonstrated by the fusion
models were small when compared to using EMG alone. This may be due to the overall lower
accuracy obtained when using EEG. The small number of EEG channels, as well as the simple signal
processing techniques used for pre-processing, do represent a limitation to the current EEG model
presented in this work. While its current implementation may not fully justify the addition of EEG
measurement into a wearable device to utilize EEG–EMG fusion, the results of this experiment
show that the addition of EEG, along with EMG signals, does have potential as method for
improving classification. Future work should focus on finding ways to improve the performance of
EEG by implementing more advanced signal processing techniques to remove noise and increase
usability of the EEG signal. It should also focus on evaluating the optimal number of EEG
channels needed to improve classification performance while remaining feasible for a wearable
system. The inclusion of more EEG channels will likely result in improved performance [237], so
future experiments should investigate the effect that increasing the number of EEG channels has
4.4 Discussion 97
on task weight classification models.

One potential improvement for Weighted Average fusion that should be pursued in future work
is the optimization of the weights used for EEG and EMG in the model. This work used simple
weightings to provide an investigation of the efficacy of Weighted Average fusion; however, it is
likely that optimization of the weightings will result in further performance improvements. For
example, subject-specific weightings, or updating the weights based on changes in the task (such as
different weights for different motion speeds) are two potential methods to increase the intelligence
of the model in a way that may result in better performance. Of note is the fact that the Single
Classifier fusion method did not improve performance the same way that Weighted Averaging did;
overall, its accuracy was closer to EMG. This may be due to the classifier having a harder time
fitting the model to a combination of EEG and EMG features (as opposed to training classifiers
on only one biosignal as the Weighted Average fusion method does). Since the biosignals react
differently to changes in task weight, it is likely that the features may have also changed differently
from each other, or at different scales, confusing the classifier. As well, a larger feature set can
sometimes result in overfitting, caused by the model being unable to optimize many parameters.
For future work, the Weighted Averaging fusion method should be used, as it outperforms the
Single Classifier fusion method.
Since this type of classification (classifying task weight during dynamic elbow flexion–extension
motion) has not been performed before, there is no definitive answer available for which features
will perform the best; however, the work outlined here does begin to provide insight into this
question. The results of the feature selection performed for each subject, while not an exhaustive
analysis of the performance of each feature, can be useful in deciding which features may perform
best for the general population. The models trained in this work were subject-specific; however, in
an ideal setting, the types of models used in a wearable mechatronic rehabilitation device would
be general across the population. This simplifies calibration of each system and would not require
the user to have to train their device before use, which can be time consuming, especially on the
limited computer processing hardware available in wearable systems. Future work should focus
on optimizing the classifiers used to allow for general models. One method of doing this is by
choosing proper features that are suitable for all people. Since biosignals can vary so much from
4.5 Conclusion 98
person to person, features used need to be robust enough to adapt to these changes in way that
can be predicted by the classifier. Features that were chosen by the ReliefF algorithm for many
subjects imply a greater ability to generalize towards a larger population. This insight can be
valuable when deciding which features should be the subject of further investigation. For EMG,
several features were used by all subjects namely, MAV, WL, ZC, RMS, MFL, and DASDV from
the biceps and MFL from the triceps. These features should definitely be considered for use in
future work attempting to complete this task. Other features from the triceps, namely MAV,
RMS, ZC, and MNF were chosen for the majority of subjects, meaning that they may present
good options for additional features to use along with the ones selected for all subjects. In general,
features from the biceps channel were chosen more often than the triceps channel, except in the
case of frequency domain features, where the triceps was more dominant. This information can
be useful when deciding the priority of EMG electrode locations. The results of EEG feature
selection were less straight forward than EMG; however, some trends did emerge. Both FD and
RMS, while not chosen for all subjects, were used in a majority of subjects and HC and PMN
follow closely behind as the next highest selected. Of the features used, these four show the greatest
selection across subjects and may present the best from this feature set in terms of generalization.
Unlike with EMG, there does not seem to be a clear favouring in regards to channels, as different
features were selected more often on different channels. This, along with the less definitive features,
demonstrates the increased difficulty seen when generalizing an EEG model to a large population.
It should be noted that this result is not unexpected, as the complexity of information encoded
in EEG, along with its high variability in behaviour between subjects, make it challenging to use
in general models. Future work should investigate more EEG features and processing techniques
when moving towards optimizing EEG–EMG fusion methods for general models.
4.5 Conclusion
This work demonstrated the feasibility of using EEG–EMG fusion to classify task weight during
dynamic elbow flexion–extension motions, an operation that can be used to improve the robustness
of wearable mechatronic rehabilitation devices. It showed that these biosignals can offer an alter-
4.5 Conclusion 99
native method of sensing changes in task weight that is more suitable for wearable devices than
traditional sensors (such as force sensors). The experiment outlined here showed that Weighted
Average EEG–EMG fusion performed better than using EEG or EMG alone, and better than the
Single Classifier fusion method. It also found that including speed information in the model im-
proves classification accuracy. Finally, it also found a subset of EEG and EMG features that show
promise in generalizing to a larger population, which is beneficial for future work that will focus
on developing general EEG–EMG fusion models. Future work should also focus on optimizing
the method of calculating EEG/EMG fusion weights, as well as looking at more combinations of
speed/weight, and incorporating more complex speed information into the models. The utiliza-
tion of task weight classification, using EEG–EMG fusion, will ultimately lead to better wearable
mechatronic rehabilitation devices that are able to more accurately adapt to the task being per-
formed by the user. This will result in increased comfort and usability during operation, which
will ultimately help improve the acceptance of such devices.
Chapter 5
Evaluating Convolutional Neural

Networks for Input-Level Fusion
This chapter is adapted from “Evaluating Convolutional Neural Networks as a Method of EEG–EMG
Fusion”, published in Frontiers in Neurorobotics, 2021 [238].
5.1 Introduction
The field of assistive and rehabilitation robotics is rapidly growing, seeking to leverage modern
technological advancements to help patients suffering from mobility issues to restore their quality
of life. With musculoskeletal disorders being the largest contributor to worldwide disability [4],
there is a large market for such devices to help supplement the treatment provided by traditional
therapy. Wearable upper-limb robotic exoskeletons, in particular, present a promising option for
rehabilitation and assistance, since the patient can use the device during daily life to help assist
with tasks, and they are not constrained to a single location during rehabilitation therapy. These
devices, however, are still limited in their use, and one reason for this is that further development
is required to advance the intelligence of the control methods used in these systems [6]. The
devices should be controlled in such a way that their use feels natural and comfortable for the
user, regardless of the task being performed.
One popularly explored method to achieve this is the use of bioelectrical signals, produced
100
5.1 Introduction 101
by the body during motion, to directly control the wearable robotic exoskeletons by detecting
the user’s motion intention and movement activity based on the information encoded in these
signals. Two popularly used bioelectrical signals are electroencephalography (EEG), recorded
from brain activity, and electromyography (EMG), recorded from muscle activity [104, 239]. These
signals are measured using electrodes on the skin and can be decoded (often through the use of
machine learning techniques) to facilitate the control of wearable robotic systems. Typically,
devices will only make use of one bioelectrical signal type at a time [6]; however, studies have
emerged that have shown that the simultaneous use of EEG and EMG together can improve
system performance [68, 69, 75, 77–79, 81, 88, 89, 128, 220, 224]. It has been shown that EEG–
EMG fusion can improve classification accuracy as well as reliability, by leveraging the benefits of
both signal types simultaneously. An example of this is the use of EEG–EMG fusion as a method
to combat the effect of muscle fatigue on system performance. Studies have shown that EEG–EMG
fusion models can maintain sufficient accuracy even during EMG signal attenuation brought on
by muscle fatigue [68, 75], demonstrating the increased reliability that can be obtained through
the use of multiple signal types simultaneously. Typically, EEG–EMG fusion is used with machine
learning to perform a classification task relevant to the control of a robotic exoskeleton device
(for example, motion intention detection). A commonly used method to incorporate EEG–EMG
fusion into machine-learning-based classification is to perform EEG–EMG fusion at the decision
level, meaning that two classifiers are trained (one for EEG, one for EMG) and their outputs
are combined using various techniques [68, 75, 77, 88, 89, 220, 224]. Use of this method has
been successfully demonstrated for tasks such as motion classification, for example, obtaining an
accuracy of 92.0% while outperforming EEG and EMG only models [68]. Some examples exist of
EEG–EMG fusion happening at the input level, meaning that EEG and EMG features are used
simultaneously to train one classifier [78, 79, 81, 88, 220, 224]. Studies that focus on this technique
have been able to show accuracies similar to decision-level fusion studies, in one example obtaining
an accuracy of 91.7% using a single classifier for gesture recognition [78]; however, when compared
with decision-level fusion in the same study, input-level fusion is often found to yield poorer results
[88, 224].
Despite promising results, further development is needed for EEG–EMG fusion techniques to
improve their viability for use in wearable robotic systems. The vast majority of EEG–EMG fusion
has been done using traditional machine learning methods that rely on manual feature extraction
before training the classifier. Recently, new machine learning methods (often referred to as deep
learning) have emerged that are capable of automatically extracting feature information from
inputs. One of the most notable implementations of deep learning is the Convolutional Neural
Network (CNN). These CNN models, originally developed for the image processing domain, work
by using convolution layers that extract information from around an image before feeding it into
traditional neural network layers (called Fully Connected layers). The model is not only able
to learn patterns from within the data, like traditional machine learning, but also automatically
learn what relevant information to extract from the input (instead of relying on the user to specify
this manually through selection of appropriate features). The success of CNN classifiers have
caused them to move beyond the image processing domain into other areas, with bioelectrical
signal classification being one of them. For both EEG [184] and EMG [185], CNNs are the most
popularly used deep learning technique. Many studies have shown great results when using CNNs
with EEG [162–173] and EMG [174–183] signals and have been able to perform many tasks relevant
to control, such as hand gesture recognition or the implementation of a Brain Computer Interface,
based on Motor Imagery, to send device commands. A brief selection of CNN-based EEG/EMG
literature with control-relevant tasks can be seen in Table 5.1. Despite the popularity of CNN
models in EEG and EMG literature, the area of EEG–EMG fusion has yet to widely adopt the
use of this technique. One study showed that CNNs can be used to fuse EEG and EMG (along
with Electrooculography, known as EOG) for sleep stage classification [87]; however, it remains
to be seen how an EEG–EMG CNN classifier would perform if used during motion tasks that
are relevant for control of assistive and rehabilitation robots. It is possible that the CNN model
may extract information about the relationship between the two signals, recorded while the user is
moving, that is not currently captured using manually selected features that have been combined
for input-level fusion. There is further evidence of CNNs being able to extract information from
both EEG and EMG, since a study was done where transfer learning (initially training a classifier
for one type of data, then using that classifier with a different set of data) was performed between
EEG and EMG datasets with CNNs. The study found that transfer learning was possible between
the two signal types to classify concentration levels (EEG) and hand gestures (EMG) [240]. This
may indicate that there is a relationship between the bioelectrical signals that a CNN can detect;
therefore, more experimentation is needed to further evaluate CNNs as a method of input level
EEG–EMG fusion.
Table 5.1: A summary of select literature examples using CNN models with EEG or EMG signals.
The references are grouped by signal type and application.
Signal Type Application Reference

EEG 2 Class Motor Imagery (e.g., left hand, right hand) [162–166]
4 Class Motor Imagery (e.g., left hand, right hand, [167, 168, 170, 171, 241]
feet, tongue)
6 Class Motor Imagery (i.e., elbow flexion/extension, [169]

forearm supination/pronation, hand open/close)
Person Identification [172]
P300 Classification [173]

EMG Hand Gesture Classification [174, 176–178, 180, 182, 183]
Wrist Movement Classification [179]
Hand Movement/Gesture Classification [176, 181]
Hand Position Estimation [175]
The objective of this work was to evaluate CNNs as a method of EEG–EMG fusion, and to
perform an analysis of the feasibility of this technique when used for a classification task rele-
vant to the control of assistive and rehabilitation robots. Multiple methods of representing and
combing the EEG/EMG signals at the input level were investigated to see which method of EEG–
EMG fusion would provide the best performance within the CNN classifier. This work provides
an example of EEG–EMG fusion happening within the CNN classifier, and highlights the most
promising methods to use for further development. To facilitate this evaluation, it was decided to
train models to classify task weight during dynamic elbow flexion–extension motion. Task weight
5.2 Methods 104
is the weight a user is holding during movement. This is relevant to the control of wearable robotic
exoskeletons during assistance and rehabilitation because the presence of an external weight can
affect the stability of a bioelectrical-signal-based control system [6, 101], as well as the accuracy
of control-relevant classification tasks, such as hand gesture recognition [242]. These control sys-
tems are often tuned for specific movement conditions; hence, being able to detect what the user
is holding, will allow the control system to dynamically adapt to the new disturbance and pro-
vide more robust performance as the user changes tasks during their daily life. Measuring task
weight during dynamic elbow flexion–extension motion provides a more realistic evaluation of the
models (as opposed to isometric muscle contraction), since the end goal of EEG–EMG fusion is
to use it within a wearable robotic exoskeleton during different motions. Dynamic movement, as
well as the more indirect force measurement of task weight, can greatly increase the challenge of
performing classification tasks with EEG and EMG signals; hence, EEG–EMG fusion provides a
good opportunity to investigate potential improvements to address these limitations. The previous
work described in Chapter 4 evaluated EEG–EMG fusion methods for task weight classification,
and obtained accuracies of 83.01% using decision-level fusion and 80.75% using input-level fusion;
however, this was done using fusion methods based on traditional machine learning classifiers with
manual feature extraction [224]. This chapter focuses on evaluating CNN-based EEG–EMG fusion
on the same classification task as a means of comparison.
5.2 Methods
5.2.1 Data Collection and Signal Processing
To develop EEG–EMG-fusion-based CNN models, a dataset of EEG and EMG signals were col-
lected during elbow flexion–extension motion from 32 healthy subjects (mean age 24.9 ± 5.4 years)
who were voluntarily recruited following approval from the Human Research Ethics Board at West-
ern University (Project ID: 112023). The data obtained from these subjects were also used in the
previous studies outlined in the preceding chapters [220, 224]. The subjects were instructed to
perform the motion at two speeds level (approximately 10°/s and 150°/s) and three weight levels
(0 lbs, 3 lbs, 5 lbs), implementing a 2 × 3 full factorial repeated measures study design. This re-
5.2 Methods 105
sulted in six combinations of weight and speed being recorded (each pairing referred to as a trial).
The order in which the trials were performed was randomized for each subject to limit any poten-
tial biasing effects caused by the ordering of the speed/weight pairings. Within each trial, elbow
flexion–extension motion was performed for three repetitions using the subject’s dominant arm
(30 right handed, 2 left handed), with a three second pause in-between repetitions. Between each
trial, subjects were given a one-minute rest period. While performing the elbow flexion–extension
motion, the subject would self-regulate their motion speed to achieve an approximation of the
targeted speed. Subjects were instructed by the experimenter to count seconds while performing
each elbow flexion–extension repetition such that a 30 second motion duration was obtained for the
slow speed (10°/s) repetitions and a two second duration was obtained for the fast speed (150°/s)
repetitions. Assuming a 150° range of motion [232–234], this resulted in approximately the desired
speed for each targeted speed level, while still allowing the subject to move dynamically in an
unrestricted manner.
During data collection, the EEG and EMG signals were recorded using an Intronix 2024F
Physiological Amplifier System (Intronix Technologies, Bolton, Canada). Both EEG and EMG
were sampled at 4000 Hz and a ground electrode was placed over the elbow bone of the subject’s
non-dominant arm to act as the system ground for the differential amplifier. The sampling rate of
the measurement system was fixed for all channels and could not be altered, which is why it was
higher than necessary, particularly for the EEG signals. In an actual wearable robotic device, this
sampling rate would be lower to reduce hardware demands.
To record EEG signals, wired gold-cup electrodes, filled with electrically conductive paste,
were placed on the subject’s scalp above the C3, C4, and Cz locations, as specified by the 10–20
International System. These locations were chosen for this study since they correspond with the
motor cortex of the brain, and should provide relevant signal information during movement. Prior
to placing the electrodes, the subject’s scalp was cleaned at the location of electrode placement with
an abrasive gel to ensure that a proper electrical connection was established. Signals were recorded
using bipolar channels, configured for a referential montage, with the reference point being an ear-
clip electrode attached to the subject’s ear lobe. During recording, the EEG signals were filtered
with a 0.5–100 Hz bandpass filter built into the Intronix 2024F system. After recording, the EEG
5.2 Methods 106
signals were filtered again in software using a 0.5–40 Hz bandpass filter (3rd order Butterworth)
[103].
To record EMG signals, bipolar electrodes were placed over the biceps and triceps of the
subject’s dominant arm, as specified by the SENIAM Project guidelines. These muscles were
chosen for this study since they are two of the main muscles that contribute to elbow flexion–
extension motion. Prior to electrode placement, the subject’s skin at the location of electrode
placement was cleaned using an alcohol swab. During recording, the EMG signals were filtered with
a built-in 20–500 Hz bandpass filter included in the measurement system. Following recording, the
EMG signals had the DC offset removed and were filtered again with another 20–500 Hz bandpass
filter (4th order Butterworth) [95].
After filtering, the signals were segmented to remove the portions of the recording where the
subject was not moving. This was done using markers that were placed at the beginning and end
of the subject’s movement. The markers were placed manually by the experimenter during data
recording using an external trigger system. Synchronized video recordings of the subject moving
were also recorded for verification.
All signal processing and image generation was done offline using MATLAB 2019b with the
Signal Processing Toolbox. An overview of the full data processing pipeline can be seen in Figure
5.1.
5.2.2 Image Generation
Once the EEG and EMG signals were processed and segmented, the next step was to convert
the dataset into images that can act as suitable inputs to a CNN classifier. Since CNNs were
developed initially as a tool for image recognition problems, their architecture relies on images as
inputs; however, since an image is simply an array with a numerical value at each pixel location,
it is possible to represent bioelectrical signals in such a way. In previous works that have used
EEG and EMG signals as inputs to CNN models, there are two commonly used methods for
representing the signals as images: calculating a time–frequency domain representation of the
signal to generate spectrogram images [162–165, 174–177] or organizing the processed signals in
the time domain to create signal images [164, 166–171, 177–183]. Note that the term image here
5.2 Methods 107
Figure 5.1: The protocol followed to process the EEG/EMG signals, generate the spectrogram and
signal images, and train the CNN models using different EEG–EMG fusion methods.
The top path (purple) shows the steps used to develop the CNN models based on
spectrogram image inputs, while the bottom path (green) shows the steps used to
develop the CNN models based on signal image inputs. For all EEG–EMG-fusion-
based CNN model types (represented by the final step of all paths), an EEG and EMG
only version was also trained, to provide a baseline comparison for evaluating EEG–
EMG fusion.
refers merely to a CNN input and does not require the use of an image in the colloquial sense (such
as a picture). For example, signal images are just the time series data reshaped into a proper CNN
input (discussed further in Section 5.2.2.2) and the convolution is actually being done on the time
series signal data directly. Both methods show prominent use with EEG and EMG-based models,
with neither method demonstrating an obvious supremacy when it comes to model performance.
Also, since EEG and EMG have never been used simultaneously as inputs to a CNN model to
classify task weight, it is unclear which image type will allow for the best fusion of EEG and
EMG within the classifier. Since both image types use a different domain representation, there
is a chance they may target different responses of the signal, offering different information to the
CNN classifier. Spectrogram images (time–frequency domain) may trend towards representing the
oscillatory behaviour of the signals, while the signal images (time domain representation) may
trend towards representing time-varying behaviour, such as changes in amplitude. However, this
is not a given, as the CNN model is free to extract information it deems relevant from the inputs,
5.2 Methods 108
and it remains to be seen which input method will provide the best performance when classifying
task weight. For these reasons, both image types (spectrogram images and signal images) will be
evaluated to determine which is the most suitable method to use for EEG–EMG fusion.
To increase the number of images to use for classifier training, during image generation the
signals were windowed using a 250 ms window with 50% overlap. This windowing was used for
both image types, with both a spectrogram and signal image being generated for each window. A
window length of 250 ms was chosen, since studies have shown that 300 ms is the maximum amount
of delay a system can experience before the user becomes unable to control the device [137]. Even
though this study was performed offline, limiting the window length to a time that fits within the
real-time delay target allows for a more realistic evaluation of the EEG–EMG-fusion-based CNN
models as a potential method of control for assistive and rehabilitation robots.
5.2.2.1 Spectrogram Images
To generate the spectrogram images, a Short-Time Fourier Transform (STFT) was calculated for
each window of the EEG and EMG signals, providing a time–frequency domain representation of
the signals. The time and frequency resolution of the STFT was chosen so the resulting images
would be of a suitable size for use as an input to a CNN model: large enough to have an appropriate
time/frequency resolution, but not so large as to require an infeasible amount of memory and
computational power. Using trial and error, a spectrogram image size of 68 × 32, for each signal
channel, was chosen. For the time resolution, the STFT was calculated using a Hann window with
a length of 56 samples and 75% overlap, which resulted in an image width of 68 pixels (for the 4000
Hz sampling rate of the measurement system). The frequency resolution of the STFT was chosen
so that an image height of 32 pixels would be obtained for the frequency range of interest for both
EEG (0.5–40 Hz) and EMG (20–500 Hz). Due to the differences in bandwidth, this meant that
EEG and EMG had different STFT frequency resolutions, but their image height was kept the
same to simplify their combination into a single image during fusion. The STFT was calculated
across the entire frequency range of 0–4000 Hz using an Fast Fourier Transform (FFT) size of
3200 for EEG and 256 for EMG. Then, the images were cropped to only include the portions of
the image within the respective bandwidth of each signal type. This resulted in five spectrogram
5.2 Methods 109
images (3 EEG channels and 2 EMG channels) of size 68 × 32 for each time window.
Following image generation, the pixel values of the spectrogram images were normalized to be
between 0 and 1. Due to the highly variable nature of EEG and EMG signals between subjects,
and the different scale in frequency magnitudes for EEG and EMG obtained from the STFT, the
images were normalized for each subject and each signal type. After all spectrogram images were
calculated for one subject, the max/min frequency magnitude value for EEG and the max/min
frequency magnitude value for EMG were recorded and used to normalize all spectrogram images of
that respective signal type for that subject. This ensured that both EEG and EMG spectrograms
were given equal proportion within the image, regardless of the differences in signal amplitude
present when recording both bioelectrical signals. This also ensured that differences observed in
subject recordings did not cause certain images in the dataset to be improperly scaled based on
an outlier subject.
Once the spectrogram images had been normalized, they were combined to facilitate the fu-
sion of EEG and EMG at the input level. Multiple methods of combining the EEG and EMG
spectrogram images were performed, to investigate which method of fusing the EEG and EMG
spectrogram images would provide the best model performance. In the first method of fusion (re-
ferred to here as the grouped method), the EEG and EMG spectrograms were grouped by signal
type and stacked vertically to create a single 68 × 160 image comprised of the five spectrograms.
The three EEG spectrograms were placed at the top of the image (in the order of C3, C4, Cz from
top to bottom) and the two EMG spectrograms were placed on the bottom of the image (in the
order of biceps, then triceps from top to bottom). This fusion method grouped spectrograms of
the same signal type together within the image, causing the convolution of the image to initially
happen within the same signal type and only fusing the signals initially along the single border
between EEG and EMG. An example of this method can be seen in Figure 5.2A. The second
fusion method (referred to here as the mixed method) stacked images vertically once again, but
this time EEG and EMG spectrograms were alternated to provide a better mix between signal
types. The order from top to bottom went C3, biceps, C4, triceps, Cz. This method of fusion
provides more areas within the image were EEG and EMG will be convolved together during the
initial CNN layer, since there are more borders between the EEG and EMG portions of the im-
5.2 Methods 110
age. An example of this method can be seen in Figure 5.2B. The final fusion method (referred
to here as the stacked method) stacked the images depth-wise to create a multi-channel image,
similar to how a colour picture will have three values per pixel location to represent levels of red,
green, and blue. In this case, every pixel location contained 5 values (one for each EEG and EMG
spectrogram) to result in an image with a shape of 68 × 32 × 5. An example of this method
can be seen in Figure 5.2C. To provide a baseline comparison for evaluating the fusion methods,
spectrograms containing only EEG and only EMG signal information were also generated to see
if fusion can outperform using one signal alone. Two spectrogram types were generated for both
EEG and EMG: vertically stacked spectrograms (68 × 96 for EEG and 68 × 64 for EMG) to
provide single-channel spectrograms to compare to the grouped/mixed methods, and depth-wise
stacked spectrograms (68 × 32 × 3 for EEG and 68 × 32 × 2 for EMG) to provide multi-channel
spectrograms to compare to the stacked method.
Figure 5.2: A sample normalized spectrogram image to demonstrate the three EEG–EMG fusion
methods used, where (A) and (B) show single-channel spectrograms and (C) visualizes
a multi-channel spectrogram. (A) shows the grouped method, where signal channels of
the same type are grouped together within the image. (B) shows the mixed method,
where EEG and EMG channels are alternated to mix signal types. (C) provides a
visualization of the stacked method, where a multi-channel spectrogram is generated
by combining the different EEG/EMG spectrograms in depth-wise manner.
5.2 Methods 111
5.2.2.2 Signal Images
Conversely, generating the signal images only required the time series signals to be organized into
an array to form the image, since the convolution is being performed on the time-series data
directly. After filtering, the five signal channels from each window were stacked vertically to create
an image where the width was the number of time samples in that window, and the height was
the number of signal channels. This resulted in a 1000 × 5 image for each window, in which the
pixels values of the image were the signal amplitude at that time point (in mV). The width of 1000
resulted from the 250 ms window length with the 4000 Hz sample rate used by the measurement
system.
The signal images were normalized using the same method as the spectrogram images, by
subject and by signal type. The max/min amplitude value of EEG and EMG for each subject was
recorded and used to scale all signal values between 0 and 1. To account for magnitude differences
between the two signal types, the EEG portion of the image was scaled using the EEG min/max
and the EMG portion of the image was scaled using the EMG min/max, preventing the larger
EMG values from dominating the image by diminishing the contribution of the smaller magnitude
EEG signals. A graphical representation of the normalized signal image can be seen in Figure 5.3.
Similar to the spectrogram images, signal images comprising of only EEG and only EMG were
also generated to provide a comparison point for evaluating EEG–EMG fusion.
5.2.2.3 Qualitative Image Response
To help illustrate the response of the EEG/EMG signals during task weight changes, an example
normalized spectrogram image along with a plot of the normalized signals for all three weight levels
(0 lbs, 3 lbs, and 5 lbs) can be seen in Figure 5.4. Based on this qualitative assessment of the
signal and spectrogram images, it can be seen that the images show different behaviour in both
the time domain and the time–frequency domain, depending on task weight. The distribution of
frequency magnitudes across time/channels is different in the spectrogram images and the shape
of the time domain signal varies in the signal images. This provides a qualitative demonstration
that there are changing patterns within the images for different task weights, which may be able
5.2 Methods 112
C3
C4
Cz
Biceps
Triceps
0 250
Time [ms]
Figure 5.3: A graphical representation of a sample normalized signal image. The image height
contains 5 rows, one for each signal channel, and the image width is dictated by the
number of samples in each 250 ms window (1000 samples at the 4000 Hz sampling
rate).
to be detected by the CNN models and used to train a classification model.
5.2.3 CNN Model Training
Once the dataset of images was developed, the CNN models based on fused EEG–EMG inputs
were trained to classify task weight. Model training was done using TensorFlow 2.3.0 with Keras
2.4.3 [243] in Python 3.8. The models trained were subject specific, meaning that each subject had
a model trained using only their data. To accomplish this, each subject’s data were split into three
parts: training, validation, and testing. The first two repetitions of each speed–weight combination
were dedicated as training data, while images generated from the third repetition were separated
into two equally sized groups: validation and testing data. To ensure that no bias was induced
by the split, the order of the windows within the third motion repetition was randomized and a
stratified split was used to ensure a 50/50 division, while keeping the number of observations of
each class balanced within the validation and testing set. The validation dataset was used during
model optimization while the testing set was kept separate until the final model evaluation, in
order to reduce potential for model bias and overfitting.
Model training had two stages. First, the base configuration of the model was determined (via
trial and error) to determine design factors such as number of layers, batch size, and optimizer
5.2 Methods 113
0 lbs 3 lbs 5 lbs
Figure 5.4: Example normalized spectrogram images and graphical representations of sample nor-
malized signal images for each of the three weight levels, showing the qualitative
variations in the images as task weight changes. During different task weights, the
distribution of frequency magnitudes across time/channels is different in the spectro-
gram images and the shape of the time domain signal varies in the signal images. The
columns each represent a different task weight level (described by the label above),
with the rows being a matched spectrogram and signal image taken from the same
time window. The spectrograms shown use the grouped fusion method to arrange the
channels. The images shown follow the same labelling convention as the sample images
shown in Figure 5.2 and Figure 5.3, excluded here to avoid clutter.
choice, among others. The base configuration used for each model type was the same for all subjects
and is discussed further in Sections 5.2.3.1 and 5.2.3.2. Once a base configuration for the model had
been determined, the second stage of training was to tune the model further using hyperparameter
optimization. This tuning focused on finding optimal parameter values for the setting of the layers
within the set base model design. The structure of the model (e.g., number of layers, types of
layers used, etc.) was not changed during this optimization, only select hyperparameter values
were updated. Using Keras-Tuner 1.0.1 [244], the values of select hyperparameters were tuned
using the Random Search optimization method to find the set that resulted in the best validation
performance. The search space checked 50 random combinations of hyperparameters, and trained
each combination twice to account for variance in model training. Using the validation dataset, the
5.2 Methods 114
hyperparameters were evaluated and the set that resulted in the lowest validation loss was selected
as the final hyperparameters to use for model training. Bayesian optimization was also tested as
a potential method for hyperparameter tuning, but it was found to result in a slight reduction
in performance compared to the Random Search method, so it was not used during training of
the final models. Early Stopping (using a patience value of five and an epoch limit of 50) was
also implemented into model training, using Keras, to stop classifier training once improvements
were no longer seen in the validation loss of the model. This was done to prevent overfitting
and to speed up training time. All models were optimized and trained using batch size of 32,
which was found using trial and error. Categorical Cross-Entropy was used as the loss function
with Adaptive Moment Estimation (ADAM) being used as the optimizer for all model types. A
Stochastic Gradient Decent (SGD) optimizer was also tested, but it resulted in a reduction in
accuracy and longer training times, so ADAM was chosen instead. The hyperparameters being
tuned, and their range of possible values, were the same for all subjects; however, each subject had
their own hyperparameter optimization performed to adjust the models better to the behaviour
seen in their specific EEG and EMG signals. The hyperparameters that were tuned for each model
type can be seen in Table 5.2 and are discussed further in Sections 5.2.3.1 and 5.2.3.2.
Table 5.2: The hyperparameters tuned during optimization, with the range of possible values used
by the Random Search algorithm. Unless specified (in brackets next to the hyperpa-
rameter name), all hyperparameters and value ranges shown were used for all model
types. Note: two exceptions to this are the kernel size for the stacked models, which
were limited to 3 × 3 to account for the smaller image size, and the split convolution
filter, which did not include the 1024 filter setting to prevent an out of memory error
while training.
Hyperparameter Parameter Values

Kernel Size (Spectrogram) 3×3, 5×5, (third layer only) 7×7
Kernel Width (Signal) 3–55 (step size of 2)
Filters 8, 16, 32, 64, 128, 256, 512, 1024
Dropout % 0.0–0.5 (step size of 0.05)
Units (FC Layers) 20–500 (step size of 20)
ADAM Learning Rate 10−5 –10−2 (logarithmic sampling)
5.2 Methods 115
5.2.3.1 Spectrogram CNN Models
A summary of the base model configuration for the spectrogram models can be seen in Figure
5.5. The base configuration for the spectrogram CNN models consisted of three convolution layers
followed by two Fully Connected (FC) layers, with a third FC layer used to output the class
probabilities. All convolution was done using valid padding, a stride of 1 × 1 and the Rectified
Linear Unit (ReLu) for the activation function. Each convolution layer included three sub-layer
steps: convolution, followed by a max pooling layer (with a size and stride of 2 × 2), and then a
dropout layer to reduce overfitting. Both FC layers contained two sub-layers: the FC step, followed
by a dropout layer. Batch Normalization was tested as an alternative to using dropout for the
convolution layers, but it led to a reduction in accuracy so it was not used. The output FC layer
used a softmax activation function to perform classification. This configuration was used for both
the single-channel and multi-channel models (as well as their EEG and EMG only equivalents); the
only difference between model types being the size of the inputted image. The hyperparameters
chosen for tuning, and the range of values included in the search space, are shown in Table 5.2.
Note that these are the same for both model types except for one deviation: the kernel size. For
the multi-channel models, the kernel size was fixed at 3 × 3. This was to account for the smaller
image size being fed into the model; with certain combinations of larger kernels, the tensor that
was passed between convolution layers could be reduced below the minimum allowable size, causing
an error in model training.
5.2.3.2 Signal CNN Models
For the signal CNN models, two base configurations were tested, shown in Figure 5.6. The first
configuration employed a method commonly used when developing CNN models based on time
domain signal inputs for EEG [167–170], referred to here as split convolution. The name arises from
that fact that it takes the first convolution layer and splits it into two back-to-back convolution
steps. This method sets the kernel size of the first two convolution layers such that convolution is
only happening across one axis of the image at time, with Layer 1 having a kernel size of 1 × kernel
width (to only convolve temporally across the time axis of the image) and Layer 2 having a kernel
5.2 Methods 116
Figure 5.5: The base model configuration used for all three spectrogram CNN model types. All
spectrogram model types used three convolution layers, followed by two FC layers and
an output FC layer to perform the final classification. Each convolution layer had
three sub-layer steps (convolution, max pooling, and dropout) and each FC layer had
two sub-layer steps (the FC step followed by dropout). Note, that repeated layers only
show the sub-layers for the first layer, to reduce redundancy and condense the diagram.
size of image height × 1 (to only convolve spatially across signal channels). The output of the
temporal convolution layer is fed directly into the spatial convolution layer, with both layers using
valid padding, stride of 1 × 1, and ReLu for the activation function. The output of the temporal
convolution layer is fed into a max pooling layer (with a size and stride of 1 × 2), followed by a
dropout layer. This is followed up by two FC layers (both using ReLu as the activation function and
a dropout sub-layer), then a third output FC layer using a softmax activation function to perform
the final classification. A summary of the base model configuration for the split convolution signal
model can be seen in Figure 5.6A.
The second base configuration tested for the signal-image-based CNNs used regular one di-
mensional (1D) convolution layers to train the models. Unlike the split convolution, this layer
type convolves across both the time and signal channel axis simultaneously as it moves across the
time axis of the image (for this reason only a kernel width is specified, since all signal channels
are always included in the convolution). This is a common method of using CNNs for time series
signals, so it is useful to see how it compares to the split convolution method commonly seen in the
EEG literature. This configuration was similar in makeup to the spectrogram base configuration
(except using 1D convolution instead of 2D convolution), comprising of three convolution layers
followed by two FC layers and a third FC layer for classification. All convolution layers used valid
padding, a stride of 1 and ReLu for the activation function. Each convolution layer followed up
5.2 Methods 117
Figure 5.6: The base model configurations used for the (A) split convolution and (B) 1D convolu-
tion models. Visual representations of the differences between both convolution types
are shown in the expanded view below each diagram, detailing the changes in kernel
size used to facilitate both types of convolution. Split convolution used one split con-
volution layer comprised of temporal and spatial convolution sub-layers, followed by
a max pooling and dropout sub-layer. 1D convolution used three convolution layers,
each with three sub-layer steps (convolution, max pooling, and dropout). All signal
model types followed convolution with two FC layers (containing two sub-layer steps:
the FC step followed by dropout) and an output FC layer to perform the final clas-
sification. Note, that repeated layers only show the sub-layers for the first layer, to
reduce redundancy and condense the diagram.
the convolution step with a max pooling layer (with a size and stride of 2) then a dropout layer
to reduce overfitting. Both FC layers used a dropout layer after the FC step. The output FC
layer used a softmax activation function to perform the final classification. A summary of the base
model configuration for the 1D convolution signal model can be seen in Figure 5.6B.
5.2 Methods 118
Both signal image model types used similar hyperparameter tuning settings; however, there
were slight variations between them to account for the differences in the configurations. Due to
an out-of-memory error while training, the split convolution models could not use a filter setting
of 1024 and was limited to 512 as the maximum number of filters for any one convolution layer.
For both model types, the hyperparameters chosen for tuning, and the range of values included in
the search space, are shown in Table 5.2.
5.2.4 Model Evaluation
Once the optimized models for each subject were trained, they were evaluated to assess the perfor-
mance of CNN-based EEG–EMG fusion. To achieve this, the withheld test data for each subject
were inputted to their final models to obtain predictions about what task weight was being held
during each test image. Since three task weights were used during data collection (0 lbs, 3 lbs,
and 5 lbs), each classifier was trained to output a three-class prediction, where each output label
corresponded to one of three task weights. This output was compared with the actual class label
to obtain an accuracy score for each model. This accuracy was then averaged across all subjects
to obtain an overall accuracy score for each fusion method, which was then used to compare per-
formance via statistical analysis (performed using IBM SPSS 27). First, the merits of each fusion
method were evaluated by comparing EEG–EMG fusion to using EEG and EMG alone. The accu-
racy scores for each fusion method were compared to the accuracy scores of the EEG/EMG only
models of the same model type to see if the increase in accuracy obtained via EEG–EMG fusion
was statistically significant. A one-way Within-Subjects Analysis of Variance (ANOVA), followed
by pairwise comparisons with the Bonferroni post hoc test, was performed on the accuracy scores
for the models of each type (four one-way ANOVAs in total). Separate ANOVAs were used for
each model type to account for the different number of models present, depending on the type (the
single-channel spectrogram model type contained 4 models, because of the use of both the grouped
and mixed fusion methods, while the other model types only contained three models each). This
prevents model type from being a factor for a two-way ANOVA, so separate one-way ANOVAs
were used instead. Following this, the methods of EEG–EMG fusion were compared to each other
using a one-way Within-Subjects ANOVA, (using the Bonferroni post hoc test for pairwise compar-
5.3 Results 119
isons) to determine if statistically significant differences exist between the accuracy obtained from
each fusion method. The purpose of this was to see if any particular EEG–EMG fusion method
provided a clear advantage in regard to classification accuracy.
To evaluate the robustness of each model further, the effect of movement speed on accuracy
was also evaluated. The classifier output predictions were separated depending on the speed at
which the movement was being performed, and accuracy was calculated separately for the fast
and slow movement speed groups. Since changes in movement speed during dynamic motion
can greatly affect bioelectrical signals, it is important to know how the CNN EEG–EMG fusion
models will perform in the presence of such variability. To be useful in the control of robotic
devices, the models need to be able to operate adequately during the different speeds required to
perform various rehabilitation and assistance tasks. To see if the effect of speed was statistically
significant, a two-way Within-Subjects ANOVA was performed on the speed-separated accuracies
for each model type. Similar to the model accuracy one-way ANOVA, the two-way ANOVA was
performed between models of the same type, resulting in four two-way ANOVAs in total. Note, for
all statistical tests performed (on both the overall model accuracy and the speed-specific accuracy),
a significance threshold of p < 0.05 was used.
As a final analysis of model performance, the class predictions from every subject were combined
and used to plot a confusion matrix for each CNN model. This was done to observe how the models
performed for each task weight and to further verify that the classifiers were adequately trained. To
evaluate the model fitting of each classifier further, the confusion matrices were used to calculate
the class-wise precision (the likelihood that a class prediction is correct) and recall (the likelihood
that all observations of a specific class are correctly classified) scores, to check the balance between
both metrics.
5.3 Results
5.3.1 Model Accuracy
The accuracy results for the spectrogram-based CNN models are summarized in Figure 5.7A.
For all models, the mean accuracy was above chance level (33.33%). The highest accuracy was
5.3 Results 120
obtained by the grouped fusion method (80.51 ± 8.07%). This was higher than the other single-
channel models, beating the EEG (50.24 ± 17.06%, p < 0.001) and mixed fusion method (79.72
± 8.19%, p = 0.025) models, and trending towards a higher mean accuracy than EMG (78.98 ±
4.66%, p = 1.000), but the difference between these two was not statistically significant. The next
highest performing spectrogram model was the stacked fusion method (80.03 ± 7.02%), which
outperformed the multi-channel EEG model (48.44 ± 15.32%, p < 0.001), and trended towards a
higher accuracy than the multi-channel EMG model (78.09 ± 5.65%, p = 0.382), but again this
increase in accuracy was not statistically significant. The stacked fusion method also showed a
higher mean accuracy than all other single-channel models (except for the grouped fusion method).
When comparing the spectrogram fusion methods to their equivalent EEG/EMG model types, the
increase in accuracy for all fusion models was statistically significant for EEG, but not EMG;
however, a clear trend did emerge, where mean accuracy increased when using EEG–EMG fusion.
The accuracy results for the signal-based CNN models are summarized in Figure 5.7B. Again,
all models showed a mean accuracy higher than chance level. The highest accuracy was observed
from the 1D convolution EEG–EMG fusion model (78.40 ± 8.70%), which showed a statistically
significant increase in accuracy over using EEG alone (41.44 ± 12.25%, p < 0.001), but not EMG
alone (74.73 ± 6.90%, p = 0.054), even though the trend is towards an increase in accuracy.
The split convolution EEG–EMG fusion model (74.28 ± 7.42%), while lower than 1D convolution
fusion, also showed a statistically significant improvement over using only EEG (42.16 ± 13.67%,
p < 0.001), but not EMG (72.70 ± 7.60%, p = 0.401); however, as with 1D convolution, the mean
accuracy tends to increase between the split convolution fusion and EMG only models. When
comparing the signal fusion methods to their equivalent EEG/EMG model types, the increase in
accuracy for both fusion models was statistically significant for EEG, but not EMG; however, once
again a trend did emerge where mean accuracy increased when using EEG–EMG fusion.
For comparing the EEG–EMG fusion methods of all model types together, the results of the
pairwise comparisons can be seen in Table 5.3. The mean accuracy for split convolution was found
to be statistically significantly lower than all other fusion methods, indicating that it is the worst
performing method of fusion. The difference in accuracy between grouped and mixed fusion was
also found to be statistically significant, meaning that grouped fusion performed better than mixed
5.3 Results 121
A - Spectrogram Models B - Signal Models

100 100

Single-Channel Split Conv.
90 Multi-Channel 90 1D Conv.
80 80
70 70
60 60
50 50
40 40
30 30
d d d G G d d
pe xe MG EEG ke E se G
EE
G se G
EE
G
rou Mi E t ac EM E Fu EM Fu EM
G S
Figure 5.7: The mean accuracy of all (A) spectrogram and (B) signal based CNN models, calcu-
lated across both speeds and all task weights. Error bars represent one standard devi-
ation. Note that the y axis begins at 30% (chance level for these models is 33.33%).
within this sample group. Stacked, grouped, and 1D convolution fusion showed no statistical signif-
icance in their accuracy differences, meaning that these methods demonstrate similar performance
within this sample group. In general, there was a trend of spectrogram-based methods having a
higher mean accuracy than signal-based methods (which held true for both EEG–EMG fusion, as
well as EEG and EMG alone).
Table 5.3: The p values obtained from the pairwise comparisons in the one-way ANOVA comparing
the accuracy of the different CNN based EEG–EMG fusion methods. Statistically
significant values (p<0.05) are shown in bold.
Fusion Method Grouped Mixed Stacked Split Conv. 1D Conv.

Grouped - 0.041 1.000 <0.001 0.431
Mixed 0.041 - 1.000 0.003 1.000
Stacked 1.000 1.000 - <0.001 1.000
Split Conv. <0.001 0.003 <0.001 - 0.018
1D Conv. 0.431 1.000 1.000 0.018 -
5.3.2 Speed-Specific Accuracy
The accuracy results, separated into the fast and slow speed groups, can be seen in Figure 5.8. For
all four model types, the effect of speed was statistically significant (p < 0.001 for all). Looking
5.3 Results 122
at the plot, it can be seen that performance was significantly worse during the fast speed for all
models. All models still remained above the chance level during the fast motion speed; however,
EEG accuracy decreased almost to this point (with 1D convolution in particular being essentially
at the chance level). It can also be seen that, even when accounting for speed, the trend of
EEG–EMG fusion outperforming EEG and slightly increasing accuracy over EMG still remained;
however, the increase was much less during fast motion (and in the case of 1D convolution, EMG
alone was slightly higher than fusion during the fast speed).
100
Slow
90 Fast
80
70
60
50
40
30
20
10
0
ed ed G G ke
d G G sed MG EG ed MG EG
oup Mix EM EE ac EM EE Fu E E Fus E E
Gr St n v. v.
Co C on
lit 1D
Sp
Figure 5.8: The mean accuracy for all CNN models, separated by the two speed levels (fast and
slow). Models of the same type are grouped together, with the order of the groups from
left to right as follows: single-channel spectrogram models, multi-channel spectrogram
models, split convolution signal models, and 1D convolution signal models. Error bars
represent ± one standard deviation.
5.3.3 Classifier Performance
The confusion matrices for all four model types can be seen in Figures 5.9 to 5.12, with each figure
corresponding to one type of model. For each model type, a confusion matrix is presented for every
model (fusion, EEG, and EMG), shown as sub-figures. Looking at the class outputs, it can be seen
that all models successfully classified 0 lbs at a much higher rater rate when compared to 3 lbs
and 5 lbs (which were similar to each other in performance). An exception to this trend is the two
5.4 Discussion 123
signal-based EEG models (shown in Figure 5.11B and Figure 5.12B for split and 1D convolution,
respectively), which had generally poor performance for all weight classes. The precision and
recall scores for the spectrogram-based models are relatively similar between the two metrics,
demonstrating that on average the fit of the models was balanced in its performance. The signal-
based models show less balance between the two metrics comparatively, although not to a large
degree.
A - Grouped B - Mixed
0 lbs 4110 187 92 93.6% 6.4% 0 lbs 4091 220 78 93.2% 6.8%
3 lbs 222 3115 986 72.1% 27.9% 3 lbs 198 3206 919 74.2% 25.8%
True Class
True Class
5 lbs 124 907 3388 76.7% 23.3% 5 lbs 115 1085 3219 72.8% 27.2%
92.2% 74.0% 75.9% 92.9% 71.1% 76.4%

7.8% 26.0% 24.1% 7.1% 28.9% 23.6%
C - EEG D - EMG
0 lbs 2227 998 1164 50.7% 49.3% 0 lbs 4152 177 60 94.6% 5.4%
3 lbs 1066 2099 1158 48.6% 51.4% 3 lbs 192 3149 982 72.8% 27.2%
True Class
True Class
5 lbs 1092 1001 2326 52.6% 47.4% 5 lbs 107 1227 3085 69.8% 30.2%
50.8% 51.2% 50.0% 93.3% 69.2% 74.8%

49.2% 48.8% 50.0% 6.7% 30.8% 25.2%
Figure 5.9: Confusion matrices, using the combined classification results for all subjects, for the
single-channel spectrogram-based CNN models. (A) shows the matrix for the grouped
fusion method while (B) shows the matrix for the mixed fusion method. (C) and (D)
show the matrices for the EEG and EMG only models, respectively. Each matrix
contains a positive/negative precision score summary in the final two rows, and a
positive/negative recall score summary in the final two columns.
5.4 Discussion
The goal of this study was to evaluate if CNNs could be used as a new method of input level EEG–
EMG fusion to classify task weight during dynamic elbow flexion–extension motion. The hope was
that the ability of the CNN to automatically learn relevant information from an inputted image
5.4 Discussion 124
A - Stacked B - EEG
0 lbs 4095 213 81 93.3% 6.7% 0 lbs 2160 920 1309 49.2% 50.8%
3 lbs 226 3239 858 74.9% 25.1% 3 lbs 1272 1913 1138 44.3% 55.7%
True Class
True Class
5 lbs 113 1119 3187 72.1% 27.9% 5 lbs 1272 872 2275 51.5% 48.5%
92.4% 70.9% 77.2% 45.9% 51.6% 48.2%

7.6% 29.1% 22.8% 54.1% 48.4% 51.8%
C - EMG
0 lbs 4159 170 60 94.8% 5.2%
3 lbs 227 3083 1013 71.3% 28.7%

True Class
5 lbs 118 1258 3043 68.9% 31.1%
92.3% 68.3% 73.9%

7.7% 31.7% 26.1%
0 lbs 3 lbs 5 lbs
Predicted Class
multi-channel spectrogram-based CNN models. (A) shows the matrix for the stacked
fusion method, while (B) and (C) show the matrices for the EEG and EMG only
models, respectively. Each matrix contains a positive/negative precision score sum-
mary in the final two rows, and a positive/negative recall score summary in the final
two columns.
may capture aspects of the EEG–EMG relationship not yet found when using manual feature
extraction techniques. To this end, this study investigated several methods of representing the
EEG–EMG signals as images (to convert the bioelectrical signals into a form suitable for input
into a CNN), as well as ways to fuse EEG/EMG during convolution while in image form. This was
done to act as a preliminary analysis of these methods, to see which CNN-based EEG–EMG fusion
techniques show the most promise to justify their further development. This will ultimately benefit
the field of rehabilitation and assistive robotics by providing a new method of EEG–EMG fusion
that can be used by the control system of such devices to detect user tasks to adapt accordingly,
resulting in devices that are safer and more comfortable to control.
Looking at the model accuracy for each method, it can be seen that all models performed above
the chance level (33.33%), and that the precision/recall scores were relatively balanced between
the two metrics (albeit less so for the signal-based models than the spectrogram models). This
5.4 Discussion 125
A - Fused B - EEG
0 lbs 4114 193 82 93.7% 6.3% 0 lbs 1809 859 1721 41.2% 58.8%
3 lbs 412 2691 1220 62.2% 37.8% 3 lbs 1336 1258 1729 29.1% 70.9%
True Class
True Class
5 lbs 247 1287 2885 65.3% 34.7% 5 lbs 1211 778 2430 55.0% 45.0%
86.2% 64.5% 68.9% 41.5% 43.5% 41.3%

13.8% 35.5% 31.1% 58.5% 56.5% 58.7%
C - EMG
0 lbs 4143 155 91 94.4% 5.6%
3 lbs 427 2486 1410 57.5% 42.5%

True Class
5 lbs 242 1312 2865 64.8% 35.2%
86.1% 62.9% 65.6%

13.9% 37.1% 34.4%
0 lbs 3 lbs 5 lbs
Predicted Class
split convolution signal-image-based CNN models. (A) shows the matrix for the
EEG–EMG fusion model, while (B) and (C) show the matrices for the EEG and
EMG only models, respectively. Each matrix contains a positive/negative precision
score summary in the final two rows, and a positive/negative recall score summary
in the final two columns.
shows that the CNN classifiers were successfully able to decode task weight information from
the EEG/EMG signals, indicating that this classification method is feasible for this task. When
comparing EEG–EMG fusion to using EEG or EMG alone, a clear trend is seen where using
EEG–EMG fusion improves the performance of the models. For all model types, EEG–EMG
fusion resulted in some level of accuracy improvement, as well generally higher precision and recall
scores (and for the classes where the precision/recall scores were not higher, they were almost the
same). Even though no statistically significant difference was found between EEG–EMG fusion
and using EMG alone, this does not completely invalidate the use of this new method. Despite the
current iteration of these models showing that the improvements gained from using EEG–EMG
fusion compared EMG are small, the fact that improvements are consistently observed when using
fusion demonstrates that the method shows potential as a tool to improve task weight classification
and should be investigated further. By focusing future work on developing improvements to model
5.4 Discussion 126
A - Fused B - EEG
0 lbs 4126 170 93 94.0% 6.0% 0 lbs 1705 1157 1527 38.8% 61.2%
3 lbs 316 2930 1077 67.8% 32.2% 3 lbs 1153 1664 1506 38.5% 61.5%
True Class
True Class
5 lbs 220 988 3211 72.7% 27.3% 5 lbs 1172 1152 2095 47.4% 52.6%
88.5% 71.7% 73.3% 42.3% 41.9% 40.9%

11.5% 28.3% 26.7% 57.7% 58.1% 59.1%
C - EMG
0 lbs 4152 161 76 94.6% 5.4%
3 lbs 355 2764 1204 63.9% 36.1%

True Class
5 lbs 224 1318 2877 65.1% 34.9%
87.8% 65.1% 69.2%

12.2% 34.9% 30.8%
0 lbs 3 lbs 5 lbs
Predicted Class
1D convolution signal-image-based CNN models. (A) shows the matrix for the EEG–
EMG fusion model, while (B) and (C) show the matrices for the EEG and EMG
only models, respectively. Each matrix contains a positive/negative precision score
summary in the final two rows, and a positive/negative recall score summary in the
final two columns.
performance, the accuracy gains of using EEG–EMG fusion may be increased, providing a stronger
justification for its use over EMG alone. Based on the trend, it is highly likely that increasing study
power through the recruitment of more subjects may result in the difference in accuracy becoming
statistically significant. Also, improving the quality of the EEG signals may improve the EEG–
EMG fusion models further. Looking at the EEG models, a clear drop in accuracy and classifier
performance can be seen when compared to EMG and EEG–EMG fusion, which is likely due to
the noisy nature of EEG signals. Due to their significantly smaller signal amplitude, EEG is more
prone to signal contamination from motion artifacts and magnetic interference when compared to
EMG, which can make it harder to use for classification. The use of more advanced noise rejection
techniques and better measurement hardware may improve EEG task weight classification, which
should in turn improve the EEG–EMG fusion models. Increasing the amount of EEG channels
being used may also help improve the EEG models, as well as EEG–EMG fusion, since it will
5.4 Discussion 127
allow the classifier to draw from more sources from different areas in the brain. However, this
trade-off needs to be balanced when using this application for wearable robotics, as these devices
are very limited in the hardware resources available. Even though EEG showed worse performance
compared to EMG, it was still clearly able to be of some benefit to the EEG–EMG fusion models,
since their mean accuracy always tended to be higher than the models based on EMG alone. As a
preliminary analysis of EEG–EMG fusion, this work was able to demonstrate that there is a clear
benefit to using CNN-based EEG–EMG fusion over just using EEG or EMG alone. It showed a
trend of CNN-based EEG–EMG fusion resulting in an increase in mean accuracy, demonstrating
the feasibility of these methods and providing a justification for their continued development.
Future work should focus on improving these models further to increase the improvements that
these techniques provide.
Another objective of this work was to see which methods of combining EEG/EMG would result
in the best performance when using CNN models. Looking at the accuracy results of each fusion
method, it is clear to see that the CNNs models did perform differently depending on the method
used. Of all the fusion methods, split convolution using signal images as inputs performed the
worst (and this difference was found to be statistically significant when compared to all other
model types). Even though other studies have used this method successfully for classification
when only using EEG signals [167–170], it is clear from this work that it is not suitable when used
with EEG/EMG together for task weight classification. For signal-image-based models, using a
traditional 1D convolution to perform CNN-based EEG–EMG fusion results in better performance.
For the spectrogram-image-based models, it was less obvious which fusion type is superior. The
grouped method had the highest mean accuracy, and the increase over the mixed method was
statistically significant, which implies that of the two ways to mix EEG/EMG spectrograms, using
the grouped method is better. Between grouped and stacked methods though, the difference in
accuracy was not statistically significant, so it is less clear which method is best. It should be noted
that the stacked spectrogram method is much more computationally efficient than the grouped
method (CNNs can perform convolution on a smaller image with multiple channels faster than a
larger image with only one channel), which may be a reason to use the stacked method. Since
both methods have similar accuracy, the faster method is more ideal, as the end goal of these
5.4 Discussion 128
models is to be used in real time in wearable robotic exoskeletons. Regardless, both methods
should be developed further in future work to investigate which method is ultimately superior.
Comparing between spectrogram-image-based models and signal-image-based models, it can be
seen that, in general, the mean accuracy of spectrogram models was higher. This is also confirmed
when looking at the confusion matrices, as the precision and recall scores are not as balanced for
the signal models. This even held true for the EEG and EMG only models, in particular EEG,
which showed a significant drop in accuracy (as well as precision and recall) for the signal models.
This makes sense, since it is well known that much of the relevant information related to motor
tasks is encoded in the frequency of the EEG signals [103]. It is likely that the time-domain-
based representation of the signal images was not able to capture this information as well as the
time–frequency-based representation used in the spectrogram images could. This, in turn, would
also affect the EEG–EMG fusion methods, which are drawing information EEG, as well as EMG.
Despite the lower mean accuracy, no statistically significant difference was found between the 1D
convolution, grouped, and stacked methods. This means that even though the trend would make it
seem like the 1D convolution method is worse, it should still be considered for future development.
One potential benefit of the 1D convolution method is that it requires fewer processing steps to
generate the images. Performing a calculation like a STFT can be comparatively time consuming,
and computationally expensive, so the use of signal-image-based models may be justified when
used in a real-time context for a wearable robotic system. The slight decline in model performance
may be outweighed by the efficiency provided by the simpler method; however, further testing and
development is needed to confirm this. Since the purpose of this experiment was to investigate the
initial feasibility of the different CNN-based EEG–EMG fusion methods, an extensive evaluation of
the computational complexity of each algorithm was not performed. The discussion here is based
merely on qualitative observations; however, next steps should focus on additional quantitative
evaluations of model complexity, which will become essential for moving the models towards a
real-time application when integrating them into a wearable device. Ultimately, all three fusion
methods (grouped, stacked, and 1D convolution) should continue to be improved and investigated,
since there was not one method shown to definitely have better performance and all three methods
have clear benefits.
5.4 Discussion 129
The models can be evaluated further by looking at the speed separated results, as well as the
confusion matrices, to examine how robust the classifiers are to changes in task weight and motion
speed. Looking at the confusion matrices in Figures 5.9 to 5.12, it can be seen that task weight
affected classification accuracy. All models were able to recognize the 0 lbs class at a much higher
rate than the 3 lbs and 5 lbs classes. While both of these classes still had relatively good precision
and recall scores, 3 lbs and 5 lbs were often misclassified as each other, but not 0 lbs, which implies
that the models had a harder time distinguishing the smaller difference in weight. This still does
present some level of benefit to a wearable robotic exoskeleton, since even knowing that the user
is holding something or not, could be useful for allowing the control system to adapt; however,
future work should look at improving the model results further to make them more consistent
across different task weights. It is clear from Figure 5.8 that speed also has a great effect on
performance for all models, with the fast speed having a significantly lower accuracy than the slow
speed. The EEG–EMG fusion models were still above chance level when moving at the fast speed,
which means that they are still able to recognize relevant patterns in the EEG/EMG signals,
just not as effectively. It also should be noted that the trend of EEG–EMG fusion having higher
accuracy than using EEG or EMG alone continued, even when separated by speed; however, the
increase was very small during fast speed (and the 1D convolution model was actually slightly
less accurate than EMG during fast motion). There are multiple things that may be causing this
phenomenon. First, faster movements are more likely to cause the EEG and EMG signals to be
corrupted by motion artifacts. The more aggressive movements performed by the subject during
the fast motion speed may be causing more motion artifacts, which in turn makes the signals harder
to use for classification. To alleviate this, more advanced filtering techniques should be used during
signal processing to remove this noise. The second reason why the fast motion may be harder to
classify is due the nature of task weight classification itself. Despite being related to muscle force
(a heavy weight needs more muscle force to move), the task weight itself is not actually a direct
measurement of muscle force. The muscle force required to perform an elbow flexion–extension
repetition will be a combination of the speed at which the subject was moving and the weight they
are holding. It is possible that this is causing smaller weights, moving at a faster speed, to have the
appearance of a larger weights at a slower speed, causing the misclassification. EMG in particular
5.4 Discussion 130
may be prone to showing this pattern, since EMG is a measurement of muscle activation. This
theory is supported by the previous work described in Chapter 4, which classified task weight using
EEG–EMG-fusion based on traditional machine learning techniques that rely on manual feature
extraction. In this study, it was found that all EEG–EMG fusion models showed a statistically
significant improvement in accuracy when adding a feature for speed information (in this case a
categorical label for fast and slow), seeing improvements of 1.2% for the best performing fusion
method [224]. Basic knowledge about the speed of the motion given to the classifier was enough
to help improve accuracy, so it stands to reason this could be possible for the CNN models as well.
Future work should investigate ways to include speed information into the input of the CNN, and
evaluate the effect that this has on classifier performance. Finally, the reduction in accuracy seen
during the fast motion trials could be due to the way the CNN models fit to the data. The nature
of how the EEG/EMG signals were windowed mean that there are more observations of movement
during the slow speed than the fast speed (since for slow motion it took longer to complete an elbow
flexion–extension repetition, and there were the same number of repetitions for both speeds). It is
possible that the models became fitted more heavily towards the slow speed data points, causing
poorer performance for the fast speed. To account for this, future work should look at collecting
more repetitions for the fast motion speed to balance out the classifier training.
Based on the results of this work, CNN-based EEG–EMG fusion has shown to be a feasible
method for classification of task weight, and warrants further development to increase the im-
provements provided by this technique. One potential area for improvement is in the dataset used
to train the models. As previously discussed, increasing the number of subjects may improve
study power and allow for more statistically significant results; however, this can also allow for the
development of generalized models that do not need to be subject specific. Ideally, to allow for
ease of use, a wearable robotic exoskeleton should be able to function for any user with minimal
training/calibration required. With a large enough sample of the population, general classifica-
tion models can be pre-trained so that new users can skip the time consuming step of classifier
training. An improved dataset can also benefit subject specific models by collecting more elbow
flexion–extension repetitions, as well as more combinations of speed and weight. One aspect of
CNN models is that their performance can be reduced for smaller training datasets [245], so col-
5.4 Discussion 131
lecting more data per subject should improve performance. More speed/weight combinations will
help to provide a more in-depth analysis of the robustness of the classifiers, and will improve their
functionality. Since this was the first analysis of CNN-based EEG–EMG fusion, only a small range
of weights (0lbs to 5 lbs) and two speeds (approximately 10°/s and 150°/s) were evaluated. It is
possible that the inclusion of more task weights, and a larger range of allowable dynamic motion
speeds, will affect the classifier performance further, so this effect should be investigated in future
work. The current task weight resolution of the classifiers (three weight levels) may limit their
use for assistance with daily-living tasks, where the user is unpredictability lifting many objects
of varied weights; however, this resolution could still be relevant for more controlled tasks, such
as rehabilitation. During rehabilitation exercises, the movement patterns and weight changes per-
formed by the user will be more predictable than activities of daily living, making the use of these
classifiers more feasible. The models developed for this work could be used to help the control
system of a wearable robotic rehabilitation device automatically adapt changing weights as the
user performs different exercises, and will not require the user/therapist to enter the weight change
manually, via some external input method, which may feel cumbersome for the user (for example
a smartphone app). The ultimate goal, however, is to keep improving the CNN-based EEG–EMG
fusion models to increase their resolution, making them a viable tool for use in many different
applications, such as assistance with daily tasks.
One method that may improve CNN-based EEG–EMG fusion is to increase the complexity of
the models via the inclusion of other deep learning architectures into the model configurations. One
popular example of this is the development of models that combine CNNs with Long Short-Term
Memory (LSTM) classifiers. LSTM models are beneficial for the classification of information that
changes over time, by retaining a memory of inputs [246]. Since the behaviour of EEG and EMG
signals will change depending on what stage of elbow flexion–extension motion is currently being
evaluated (for example the biceps muscle should be more dominant during flexion), LSTMs may
benefit the model by being able to incorporate this information better than using only a CNN.
Other studies have shown that CNNs, combined with LSTMs, can be used for EEG [171–173]
and EMG [175] classification, and LSTMs alone have been used during decision-level EEG–EMG
fusion [75], so there is evidence to suggest that this can be a beneficial technique for improving
5.5 Conclusion 132
EEG/EMG models. Future work should evaluate the use of combined CNN–LSTM models for
input-level EEG–EMG fusion. Another potential way of improving CNN-based EEG–EMG fusion
is to explore other methods of calculating time–frequency signal images. While the STFT is a
popular time–frequency representation method, it is far from the only one. Other studies have
shown that Wavelet-Transform-based images can also work for EEG [162, 241] and EMG [177]
CNN models, so future work should investigate these methods as an alternative to using STFT
spectrograms for CNN-based EEG–EMG fusion. Improving these models will move them closer
to being practically implemented within a wearable robotic exoskeleton, where they can improve
the usability of these devices during rehabilitation and assistive tasks.
5.5 Conclusion
This work demonstrated the feasibility of using CNNs as a method of input level EEG–EMG fusion
for task weight classification during dynamic elbow flexion–extension. It presents a new EEG–EMG
fusion method that can be used to improve the performance of bioelectrical signal controlled robotic
devices for assistance and rehabilitation. During the experiment performed, it was shown that a
trend exists where EEG–EMG fusion resulted in a higher mean accuracy compared to using EEG
and EMG alone. Different methods of representing the EEG/EMG signals for use in the CNNs were
also evaluated, and it was found that time–frequency-image-based models (spectrograms) tended
to outperform time domain (signal) models; however, signal models using 1D convolution may still
have the potential to match spectrogram model performance. Future work should expand upon the
results shown here, and focus on improving performance by increasing model complexity through
the inclusion of other deep learning architectures (such as Long Short-Term Memory networks),
as well as, investigating other time–frequency image representation methods (such as Wavelet
Transforms). It should also focus on improving the training dataset used by collecting EEG/EMG
signals during more speed/weight combinations, collecting more motion repetitions from each
subject, and collecting data from a larger population of subjects, to allow for a more in-depth
analysis of model robustness, as well as better trained models. Using CNNs to facilitate EEG–EMG
fusion presents a new way to utilize these bioelectrical signals for the control of wearable robotic
5.5 Conclusion 133
devices, and implementing EEG–EMG fusion for task weight classification will allow such devices
to adapt to changes in system dynamics so that they can perform assistive and rehabilitation
tasks in a more stable and robust way. This will ultimately improve the user experience, leading
to safer devices that can be more widely adopted as a new treatment and assistance solution for
musculoskeletal disorders.
Chapter 6
Investigating the Effect of Image

Normalization on CNN-Based Fusion
6.1 Introduction
In the study outlined in the previous Chapter, the concept of using CNNs as an EEG–EMG fusion
method for task weight classification was introduced, multiple methods of combining EEG and
EMG signals within the CNN were evaluated, and the feasibility of these new EEG–EMG fusion
techniques was successfully demonstrated. While the evaluation of different CNN-based EEG–
EMG fusion methods performed in Chapter 5 provides a good starting point for the implementation
of these models, there are still many other aspects of the model architecture that could be varied to
further optimize classifier accuracy and robustness. As discussed in Section 5.4, model performance
needs to be improved before CNN-based EEG–EMG fusion models can be integrated into wearable
mechatronic devices, and future work should focus on iterating on model design now that the initial
feasibility has been demonstrated.
One aspect of model implementation that was not investigated in Chapter 5 was the method
used for image normalization used during pre-processing. The concept of normalizing a dataset of
images before using them to train a CNN classifier is a widely adopted practice, and is considered
a standard step in many CNN development pipelines [159, 160, 247]. Normalizing an image-
set before training has many potential benefits for a CNN model and, if looking at the practice
134
more generally, normalization of datasets is also typically beneficial for most machine learning
applications (both traditional and deep learning classifiers). In traditional machine learning, where
various manual features are extracted, normalization is used to scale these features so that their
values are within the same range [248]. Since different features represent different aspects of a
signal, and are calculated using various formulas/methods, the range of the final output values
can vary drastically. Failing to account for this can lead to poor model optimization [248] and
can cause the features with larger value ranges to dominate the model (even if their underlying
information is not actually the most relevant to the classification task).
For CNNs specifically, normalization is also an important pre-processing technique that can
affect classifier training and optimization. Depending on the activation function used after the
convolution layers in the CNN classifier, large input values to the model can cause the output
of these activation functions to grow very large (or even saturate, depending on the function
type), while small inputs result in near-zero output values. This phenomenon is often referred
to as an exploding/vanishing gradient problem [160], and should be avoided as it can negatively
impact model training/optimization. Beyond computational optimization benefits, normalization
is also used with image inputs to highlight different information that may be relevant during
classification. In the image processing and recognition domain, normalization is often used to
regulate image intensity [249–253], by shifting all pixel values in the image-set to be within the
same range, regardless of what background noise/changes may have been present when the image
was captured. Examples of this include removing the effect of ambient lighting conditions from
images [251–253], compensating for changing factors, such as skin tone in medical imaging [252],
or reducing background noise introduced by the sensing method employed for image capture [249].
Normalization can also provide the benefit of increasing contrast between the “bright” (high pixel
value) and “dark” (low pixel value) portions of an image [249], which may expose more clearly
the relationship between these regions and help the CNN extract relevant information in a more
efficient manner [160]. As one may expect, the method used for normalization has consequences in
the way intensity and contrast are changed within an image, which in turn can affect CNN classier
performance. In the field of image-recognition CNNs, studies have been performed previously that
evaluate the impact normalization techniques on performance [247, 249, 252], demonstrating this
fact quantitatively.
Normalization is also a key step in EEG/EMG signal processing, analysis, and classifier de-
velopment; making the method used an important choice when working with these signals. Since
bioelectrical signals are highly variable, with their behaviour changing drastically between par-
ticipants, and even within the same participant over time, normalization can be used to reduce
the effect of these variations [254–257]. Given the importance of this procedure for EEG/EMG
signal analysis, studies have investigated the effect of various EMG signal normalization methods
[255, 257–260], EEG signal normalization methods [261], and EEG feature normalization methods
[256, 262–264] to see which approach proves to be the most effective. Studies have also com-
pared normalized and non-normalized EMG features and found that normalization leads to better
outcomes during model development [265–267].
Knowing that the importance of EEG/EMG normalization has been demonstrated for tradi-
tional machine-learning models (using manual feature extraction), and that image normalization
has been shown to affect intensity/contrast in image-recognition CNN models, it stands to reason
that image normalization will also play a role in the performance of EEG-based and EMG-based
CNN classifiers. Despite numerous examples of CNNs trained using image representations of EEG
or EMG signals, the method employed to normalize EEG/EMG images seems to be inconsistent
between studies. For CNN models using time–frequency image representations (e.g., spectrograms)
as inputs, there are examples in both EEG [163, 164, 241, 268] and EMG [177] studies where the
normalization method used is not mentioned in the procedure, leaving it unclear whether raw
values were used as inputs. When the method was described, a range of techniques to normalize
time–frequency EMG images have been demonstrated, such as rescaling values to be between 0 and
1 [176, 269, 270], subtracting mean pixel values [175], or standardizing pixel values using z -scores
[271]. Another common method for both EEG [162, 165, 272] and EMG [174] time–frequency image
inputs is to represent the resulting spectrogram/scaleogram as a standard RGB colour image. This
transforms the classification problem into a conventional image-recognition task, aligning with the
traditional use of CNNs as an image classification tool. In this case, images are usually normalized
to be between 0–1 by simply diving the pixel values in each colour channel by 255 (this number
arising from the fact that each colour channel is represented per pixel via one byte, giving it a
possible decimal value of 0–255). This simple method for normalizing colour images is commonly
demonstrated in the image-recognition domain [253, 273, 274], which is likely why is was imple-
mented in several EEG/EMG CNN studies. For CNN models that perform convolution directly on
the time domain EEG/EMG signals (these inputs referred to here as signal images), there appears
to be a more concrete trend in the normalization method used. For both EEG [167, 169, 275]
and EMG [178–181, 183], there are multiple examples where no normalization is performed, and
the raw signal was inputted directly into the CNN classifier. Despite the previously mentioned
benefits of normalization for both bioelectrical signals and images, some researchers chose to use
non-normalized signals for multiple reasons. Some feel that transforming how the EEG/EMG
signals are represented (using pre-processing like normalization or time–frequency image represen-
tations) may result in a loss of critical information that could have been extracted by the CNN
during convolution [167, 178]. Other work has sought to specifically evaluate the potential of
CNNs for extracting worthwhile information from raw EEG/EMG signals [169, 180, 275] (poten-
tially in an effort to see if pre-processing steps can be removed), or has focused on comparisons
between the features automatically extracted from raw EMG signals and traditionally used, man-
ually extracted, features [179, 183]. Regardless of this trend, there does exist examples of signal
image-based CNNs using normalization methods as a pre-processing step: one such example being
the standardization of EEG/EMG signals using z-scores [164, 168, 181, 276]. In the context of
both EEG-only and EMG-only CNN classification, the topic of if/how to normalize image repre-
sentations of the bioelectrical signals still requires more investigation before any strong consensus
can be made.
Not only are comparisons between normalization techniques for EEG and EMG images largely
unexplored, there is very little work in the literature currently that evaluates normalization of com-
bined EEG–EMG images. Due to the nature of the bioelectrical signals themselves, their behaviour
is very different, which leads to a large range of possible values when recording the two signal types
during data collection. It would not be appropriate to normalize them together as one image, since
the larger voltage amplitudes of the EMG signals would diminish the contribution of the smaller
amplitude EEG signals. Ideally, to properly fuse the information provided by both signal types,
they should provide an equal contribution within the model. While not focusing on EEG/EMG
specifically, previous work has begun to investigate how normalization should be implemented in
sensor fusion CNN systems working with Inertial Measurement Unit (accelerometers, gyroscopes)
and altimeter (altitude measurement using air pressure) sensor data. This study found that sensor
specific normalization should be used (as opposed to normalizing the different signal types as one
group) [277]. This makes sense intuitively: the normalization process used for different types of
signals should be tailored to its unique response to maximize the amount of information it can
provide. This adds an extra layer of complication when implementing normalization pre-processing
for combined EEG–EMG image representations, since different regions of the image need to be
normalized separately.
One question of particular note regarding normalization used in EEG–EMG fusion models is
how to choose what range of boundary values to use for rescaling the signals. During normalization,
image pixel values are rescaled to be between 0 and 1, where 0 represents the minimum image pixel
value and 1 represents the maximum image pixel value. The selection of the maximum/minimum
limits used for normalization can have a consequences on the final resolution and appearance of
the image, ultimately affecting the performance of the CNN classifier. This has been demonstrated
in both traditional image recognition models [249, 252], as well as EMG-based signal image CNNs
[181]. In the previous study outlined in Chapter 5, the images used to train the subject-specific
models were normalized using the maximum/minimum EEG and EMG values across the partici-
pant’s entire image-set, essentially normalizing all of their data together simultaneously. This was
chosen since it mimics a standard approach in traditional machine learning model development,
where the entire feature-set is standardized before training; however, the potential to normalize
across different groupings of the data exists. For example, during image recognition tasks, pixel
values are often normalized only within a single image (e.g., normalizing each image individually)
[249–252]. This allows the normalization to be centred around the specific variation present in each
image, the effect of which may be diminished if normalizing using dataset wide upper/lower bounds
[252]. Image specific normalization has also been demonstrated for EMG images used with CNN
classifiers [176, 181, 269, 270], indicating this method may be appropriate for EEG–EMG combined
images. Going beyond image processing specifically, the decision of what range of boundary values
to use for rescaling the signals may have relevance based on the physiological behaviour of the
6.2 Methods 139
bioelectrical signals themselves. Multiple studies have investigated which reference point to use as
the min/max boundary when normalizing EMG signals [255, 257, 260], and have found that using
a task-specific reference point may reduce inter-subject variation [255] and improve normalization
for EMG signals collected during high-velocity movements [257]. For EEG signals, it has been
shown that normalizing over a smaller set of data can improve performance when compared to
normalizing across the entire dataset, since important patterns in the EEG response may become
obscured when the signal is rescaled across a larger range of data [261]. For both signal types,
the choice of what portions of data to use during normalization can clearly impact how relevant
information can be extracted for classification; therefore, this aspect needs to be studied further
in the context of EEG–EMG images used for CNN classifiers.
Thus, the purpose of the study presented here is to investigate various methods of normalizing
EEG–EMG combined images used for CNN-based task weight classification and to evaluate which
method provides the best classifier accuracy. The goal of this work is to iterate on the EEG–
EMG fusion-based CNN models introduced in Chapter 5, and to begin the process of refining the
techniques implemented during model development, ultimately leading to improved performance.
The analysis performed here will provide quantitative evidence when choosing which normaliza-
tion method to use for future development of CNN-based EEG–EMG fusion classifiers and will
move these tools closer towards the eventual goal of successful implementation within a wearable
mechatronic device.
6.2 Methods
6.2.1 Signal Processing and Model Training
Since this study serves as a follow-up to the work done in Chapter 5, the same EEG and EMG
signals collected for that study were used to generate the necessary input images required for train-
ing the subject-specific CNN models. Likewise, the same filtering/processing methods described in
Section 5.2.1 were used to prepare the EEG and EMG signals for image generation. The idea be-
hind this experiment was to develop CNN models using the same methods followed in the previous
work, while only changing the method of image normalization applied to the spectrogram/signal
6.2 Methods 140
images. This means that all base model configurations, hyperparameter optimization methods,
and model training settings match what was reported in Section 5.2.3, and all image generation
techniques (excluding normalization) that were discussed in Section 5.2.2 remained the same. The
intention behind fixing the model design between studies was to allow for a proper comparison
between CNN models developed using the previous normalization method and the models devel-
oped using the new normalization techniques described in this chapter. Changing other aspects
of image generation, or model training, could introduce variations in model performance that are
not related to the normalization method used. This would make a comparison of methods chal-
lenging, as it would be difficult to determine which aspects of the varied performance were due
to normalization specifically. Reducing the number of variables that can affect model behaviour
will allow for a more direct comparison of normalization methods and improve the strength of the
evaluation.
To reduce the number of models that need to be trained, this study only focused on three of
the previously reported CNN-based EEG–EMG fusion methods: grouped spectrogram, stacked
spectrogram, and signal 1D convolution. These methods were chosen because the results of the
experiment in Chapter 5 indicated that these fusion techniques had the best performance, and
showed the most promise for future development; therefore, it is the logical choice to use them in
a follow-up analysis. The mixed spectrogram and signal split convolution models were found to
have worse performance when compared to the other three (split convolution had a statistically
significant reduction in accuracy compared to all other methods and the accuracy of the mixed
spectrogram method was statistically significantly lower than the grouped spectrogram method);
therefore, they were excluded from the work presented here. The main reason for this was a
time saving measure; model training and optimization is a computationally expensive and time
consuming process, and the evaluation of normalization methods required repetitious training of
each model type (using differently normalized input images each time). Reducing the number
of model types in the evaluation allowed for a significant drop in the overall computation time.
Future work could potentially investigate the effect of image normalization on the performance of
mixed spectrogram and signal split convolution models; however, previous results imply that the
effort will be better spent on refining the other methods instead.
6.2 Methods 141
Similar to the EEG–EMG fusion studies outlined in previous chapters, an EEG only and EMG
only version of the grouped spectrogram, stacked spectrogram, and signal 1D convolution CNN
models were also trained for each normalization method. This was done to provide a baseline
control group to which compare EEG–EMG fusion, as well as to investigate the individual effect
of image normalization on EEG and EMG alone. It is possible that normalization could affect the
signals differently; therefore, it is worthwhile to include this factor in the analysis.
Repeating the procedure outlined in Chapter 5, all signal processing, image generation, and
image normalization was done offline using MATLAB 2019b with the Signal Processing Toolbox.
Model training was done using TensorFlow 2.3.0 with Keras 2.4.3 [243] in Python 3.8, and hy-
perparameter optimization was implemented using Keras-Tuner 1.0.1 [244]. An overview of the
full data processing pipeline, showing the different normalization methods and image types used
during CNN classifier training can be seen in Figure 6.1.
6.2.2 Normalization Methods
As stated previously, the major change made to the CNN models in this analysis was the method
used when normalizing the EEG/EMG images provided for training the CNN classifiers. In the
previous study, this factor was not varied during model development; all signal/spectrogram images
used the same normalization method. Normalization, however, can have a significant effect on the
quality/resolution of an image; therefore, it is important to use a method that will allow for the
best transfer of relevant information to the classifier during training.
All of the normalization methods evaluated here work on the same principle, rescaling the
image pixel values to be between 0 and 1 based on an upper bound maximum value (rescaled to
be 1) and a lower bound minimum value (rescaled to be 0). This type of normalization is often
referred to colloquially as the min–max method [249–251] and the formula used is as follows:
p − pmin
pnorm = , (6.1)
pmax − pmin
where p represents the current image pixel value being normalized, pmax represents the maximum
pixel value from the image-set used as the upper bound, pmin represents the minimum pixel value
6.2 Methods 142
Figure 6.1: The data processing protocol used to implement EEG–EMG fusion-based CNN models
trained using separate image-sets normalized using the various methods. The top path
shows the steps used to develop the CNN models based on spectrogram image inputs,
while the bottom path shows the steps used to develop the CNN models based on signal
image inputs. Note that the top path includes two spectrogram image types (grouped
and stacked), so it results in twice as many trained CNN models. For all EEG–
EMG-fusion-based CNN model types implemented using each normalization method
(represented by the final step of all paths), an EEG only and EMG only version was also
trained, to provide a baseline comparison for evaluating the effect of the normalization
methods on EEG–EMG fusion performance.
6.2 Methods 143
from the image-set used as the lower bound, and pnorm is the final normalized image pixel value
(scaled to be between 0–1).
The differences between each method evaluated in this study come from how the maximum
and minimum values are chosen within the context of the collected training data. The previous
study used subject-wise normalization (using the maximum/minimum value from all of the sub-
ject’s data); however, this is not the only approach possible when rescaling the images. Selecting
maximum/minimum values across alternate groupings may highlight different EEG/EMG signal
behaviour within the images, as the resolution of the pixel values will change when normalization
is calculated using various upper/lower bound values. The following sections outline the different
normalization methods evaluated in this study. One thing to note is that for all normalization
methods (and image/model types), EEG and EMG were normalized separately. This means that
for EEG–EMG fused images, the maximum/minimum EEG values were used to rescale the EEG
portions of the image to be between 0 and 1, and the maximum/minimum EMG values were
used to rescale the EMG portions of the image to be between 0 and 1. This was done to allow
both signal types to have an equal contribution within the image. Since the typical amplitudes of
the two bioelectrical signals are vastly different (mV in the case of EMG, and µV in the case of
EEG), normalizing both together will cause the smaller scale EEG signal to be diminished within
a time-domain image when rescaled to the larger EMG signal range. Likewise, the time–frequency
behaviour of the signals is very different, so normalizing the spectrogram images together will have
similarly undesirable effects on image resolution. Thus, it is important that both EEG and EMG
are normalized separately.
6.2.2.1 Subject-wise Normalization
Subject-wise normalization was the method used initially during the evaluation of CNN-based
EEG–EMG fusion reported in Chapter 5. For this reason, the same images and models developed
for that study are simply reused here (new models/images were not generated and the performance
results from Chapter 5 are reported again here as an example of subject-wise normalization). For
completeness, the method will be described here again, and the model performance results will be
repeated in this chapter, to make the comparison between normalization methods easier.
6.2 Methods 144
Subject-wise normalization finds the maximum/minimum EEG and EMG pixel values across
the entire image-set for the current subject and rescales all images to be between 0 and 1 based on
those values. In essence, it puts emphasis on scaling images to a specific person and the range of
their recorded EEG and EMG values. Since EEG/EMG signals can very so much from person-to-
person [254–257], this method is a reasonable choice for rescaling images when developing classifiers
using human data. However, this method also has the potential to reduce the impact of low and
mid-range pixel values, since they may be obscured during rescaling if the total range of values
for the entire image-set is large [261]. This may occur if undesired noise is present during signal
recording, resulting in outlier values in the EEG/EMG images being incorrectly selected as the
upper/lower bounds for normalization.
6.2.2.2 Speed-wise Normalization
Speed-wise normalization organizes the training images for the current subject into two groups
(fast and slow) based on the speed at which the subject was completing elbow flexion–extension
motions during the respective trial used for image generation. For both EEG and EMG, two sets of
maximum/minimum image pixel values were chosen from the grouped image-sets: one for the fast
motion speed images and one for the slow motion speed images. These two maximum/minimum
value pairs are then used to rescale the image pixels within their respective speed groups to be
between 0 and 1.
This method puts an emphasis on scaling images towards the task-specific motion speed be-
ing performed by the subject, a further subdivision of the image-set compared to subject-wise
normalization. To be used effectively, wearable mechatronic devices need to be able to operate
during varying motion speed conditions (required for the completion of different activities of daily
living); however, these changing speeds can impact the behaviour of EEG/EMG signals, making it
a challenging task to train a robust classifier. Normalizing images to their respective speed groups
may reduce speed-induced variations within the images, allowing the CNN classifier to focus on
extracting more relevant features during model training. The work done in Chapter 4 showed that
including basic speed information into an SVM classifier as a feature can improve classifier perfor-
mance, and it is possible this approach my also benefit CNN classifiers. Speed-wise normalization
6.2 Methods 145
presents an opportunity to incorporate speed information into the CNNs in an indirect way (by
allowing speed to influence image normalization, which in turn may affect model training). As
further evidence for the feasibility of this method, EMG signal normalization studies have shown
that using task-specific reference points can improve outcomes [255, 257], and it has been observed
that this practice can be beneficial during dynamic high-speed motions in particular [257]. Given
these findings, it is possible these benefits may carry over to EEG–EMG image normalization as
well, so this technique should be investigated as part of this experiment.
6.2.2.3 Image-wise Normalization
Image-wise normalization rescales each image individually, as opposed to within a group as with
subject-wise or speed-wise normalization. For each image in the image-set, the maximum/mini-
mum values for the EEG portion of the image and the maximum/minimum values for the EMG
portion of the image are found and used to rescale their respective image portions to be between
0 and 1. See Figure 6.2 for a visual representation of how normalization is split across the image.
With this method, each individual image is given more weight within the image-set, since its
pixel values will always be scaled between the full range of 0 to 1. Using a normalization method
that scales across a group of images (i.e., subject-wise or speed-wise normalization) can cause the
resolution of some images to be reduced, depending on the maximum/minimum values chosen
for rescaling. If an outlier value in the image-set (caused by an unwanted phenomenon, such
as measurement noise) is used as the maximum/minimum value, it can cause actually relevant
variations within the image to be diminished, if the image resolution is reduced when the new
scale is applied during normalization. Normalizing across each individual image can avoid this
issue; however, it is not without faults. A consequence of this trade-off is that within-image noise
can also be concurrently amplified, as the pixels in each individual image are given more weight.
Image-wise normalization has been used extensively in image recognition problems [249–252], as
well as EMG-based CNN classifiers [181, 269, 270], so it is worthwhile to evaluate its effect on
combined EEG–EMG images to see if the trade-off is beneficial.
6.2 Methods 146
6.2.2.4 Channel-wise Normalization
Channel-wise normalization operates similarly to image-wise normalization (normalizing each im-

age individually), but it divides the normalization further by rescaling each signal channel within
the image separately. This means that each EEG channel (C3, C4, Cz) and EMG channel (biceps,
triceps) portions of the image are normalized using their own maximum/minimum values. See
Figure 6.2 for a visual representation of how the normalization is split across the image, and how
channel-wise normalization differs from image-wise normalization.
This method presents the same potential benefits as image-wise normalization, but further
applies this concept by giving not only each image more weight, but each channel within that
image as well. EEG and EMG are already normalized separately in the previously described
methods, but in each one, the channels within those two groups are still normalized together.
Since each EEG/EMG channel corresponds to a different location on the body, the bioelectrical
signals recorded from the various channels will carry different physiological information during
signal acquisition (e.g., activation of different brain regions for EEG, and different muscle groups
for EMG). The scale of those signals can be different, even within the same bioelectrical sig-
nal type (for example, the triceps often has a smaller amplitude than the biceps during vertical
elbow flexion–extension [278]); however, that does not mean that the information from smaller
signals is less relevant, since it may still be relevant to the desired task. Despite their potential
importance, the effect of the less active channels may be reduced via normalization based on the
maximum/minimum scale chosen from the more active channels. Channel-wise normalization looks
at each channel separately, allowing them to contribute equally within each image; however, this
can lead to the same trade-off present in image-wise normalization, where unwanted noise can be
inadvertently amplified (with the effect being potentially more pronounced as intra-channel noise
may now be amplified even further). Channel-wise normalization has been implemented success-
fully in an EMG-based CNN classification model [176], providing an initial demonstration of the
feasibility of this normalization method. Further investigation into the viability of channel-wise
normalization for combined EEG–EMG images is needed; thus, this technique will be evaluated
in this study as well.
6.2 Methods 147
Figure 6.2: A visual representation of how image-wise and channel-wise normalization is performed
across an image. The left side shows image-wise normalization, where the EEG portion
of the image is rescaled using the maximum/minimum EEG values from that specific
image and the EMG portion of the image is rescaled using the maximum/minimum
EMG values from that specific image. The right side shows channel-wise normaliza-
tion, where the portions of the image that correspond to each channel are normalized
separately, using the maximum/minimum value from their own channel from that spe-
cific image.
6.2.2.5 No Normalization
Finally, as a baseline control group, models were trained using an image-set of non-normalized
images. These images simply use the raw value present after image generation for both image
types (bioelectrical signal amplitude in case of the signal images, and STFT magnitude in the case
of the spectrogram images).
Choosing to forgo normalization may offer a more accurate representation of pixel values within
the images, since they will not be mapped into a new 0–1 scale during normalization. This
procedure could potentially compress smaller values and/or over-emphasize larger ones (along
with changing the relationship between different pixel values). However, this approach could
also cause the variation between images to be higher, since the range of potential values is no
longer bounded to known limits through normalization. This may negatively impact the classifier’s
ability to learn relevant information from the input images (since higher variance may make it
harder to notice relevant trends in the data), and has the potential to negatively impact classifier
training/optimization if large values are inputted to the CNN and cause the internal activation
6.2 Methods 148
functions to saturate [160]. Despite these potential drawbacks, the use of non-normalized images,
specifically for signal images, has been popular for both EEG-based [167, 169, 275] and EMG-based
[178–181, 183] CNN models. In one example, normalized and non-normalized EMG signals used
for CNN classification were compared, and it was found that using no normalization was superior
[181]; hence, this method (or lack thereof) should be included in an evaluation of normalization
methods for combined EEG–EMG images.
6.2.3 Qualitative Effect of Normalization
To further demonstrate the differences between the normalization methods used, sample spec-
trogram images taken from the same subject and time window have been generated using the
aforementioned approaches (shown in Figure 6.3). By taking an example spectrogram and nor-
malizing it using each technique, the effect that various rescaling strategies has on the range of
pixel intensities observed within an image can be visualized. This provides a qualitative demon-
stration of the impact that changing normalization methods can have on an image, highlighting
the differences between them.
To generate these images, the sample spectrogram used in Figure 5.2 was normalized using
subject-wise, speed-wise, image-wise, and channel-wise normalization, then plotted using the same
colour scale (0 corresponding to the darkest colour, 1 corresponding to the lightest colour). Note,
that due to the nature of how a colour scale is plotted when visualizing a spectrogram, an example
showing no normalization could not be generated in a meaningfully representative way. Thus,
it has been excluded from the qualitative assessment here. This is because to properly map a
spectrogram magnitude value to a colour (so it can be visualized in a plot), a range of expected
values must be provided during graphing to define what pixels to assign to the brightest and
darkest colours. In essence, this will cause the resulting image to appear normalized, since the
act of displaying it visually requires the spectrogram to be rescaled to some degree. There is no
way to truly visualize a non-normalized spectrogram, since it is an array comprised of magnitude
values, not pixel intensities; therefore, attempting to plot a non-normalized spectrogram will not
properly convey how the image would appear to the CNN when inputted. For the same reason,
sample signal images were also excluded from the comparison. Since the signal 1D convolution
6.2 Methods 149
model is not actually using an image at all, but is performing convolution directly on the time
domain signal, visualizing it requires the drawing of a graphical representation of the signals (such
as what was done in Figure 5.3). The appearance of the signals within this image will be mainly
dominated by the method used to represent them as a picture, so attempting to comment on the
difference between normalization methods will be unreliable and potentially misleading. Sample
spectrograms generated using the four normalization methods will provide the most objective rep-
resentation of the difference between the images; therefore, this will be the focus of the qualitative
analysis.
Looking at the spectrograms, the effect that normalization has on the behaviour of the image
can be clearly observed. Subject-wise and speed-wise normalization present a similar appearance,
since they both rescale pixel values over a large group of images. However, upon closer observa-
tion, slight differences can be seen between them, indicating how changing the range used during
rescaling can affect image resolution and behaviour. Image-wise normalization shows a signifi-
cantly greater range of pixel intensity than subject-wise and speed-wise normalization, due to the
fact that each individual image is scaled to be between 0 and 1; thereby providing the full range
of intensity in every image, resulting in an image that appears “brighter.” With image-wise nor-
malization; however, it can be seen that much of the high intensity pixels for each signal type are
more concentrated in one channel. Since the different EEG/EMG channels demonstrate varied
signal amplitudes within their respective signal type, image-wise normalization gives preference
to the more dominant channels. Channel-wise normalization, on the other hand, shows the same
increase in intensity range as image-wise normalization, but this behaviour is demonstrated in all
channels (since each channel is normalized to contain the full intensity range). This results in an
image where all channels show high contrasts and increased “brightness,” as opposed to just the
dominant channels.
Using the sample spectrogram images, the qualitative effect of normalization can be shown,
highlighting how the selection of a normalization technique can impact the information provided by
an image. The behaviour observed in the example images aligns with the hypothesized behaviour
of each normalization method discussed in Section 6.2.2, helping to reinforce the concepts discussed
there. While qualitative analyses can be used as a demonstrative tool for explanations of theory,
6.2 Methods 150
(a) Subject-wise (b) Speed-wise
(c) Image-wise (d) Channel-wise
Figure 6.3: An example spectrogram image normalized using (a) subject-wise, (b) speed-wise,
(c) image-wise, and (d) channel-wise normalization. Comparison between the images
demonstrates the qualitative impact that different normalization methods can have on
an image, and how it can affect the range of pixel intensities observed. The spectrogram
depicted uses the grouped fusion method to arrange the channels and follows the same
labelling convention as the sample images shown in Figure 5.2 (the labels were excluded
here to avoid clutter).
6.2 Methods 151
to provide a true comparison of the effect of normalization methods a quantitative analysis is

required.
6.2.4 Evaluation of Normalization Methods
The following subsections discuss the methods used to evaluate the aforementioned normalization
methods based on classifier accuracy. To simplify reporting, it was decided to exclude confusion
matrices from the analysis described in this chapter, since the large number of CNN models
trained for this study would require the inclusion of a substantial number of plots. The ability
of EEG–EMG fusion-based CNN classifiers to properly fit to the training dataset has already
been demonstrated and discussed extensively in Chapter 5, and since the models developed for
this work are based on the same architecture, they exhibit fundamentally the same behaviour
during training. The inclusion of a confusion matrix for every CNN model of all image types,
normalization methods, and fusion methods (EEG–EMG, EEG only, EMG only) does not provide
any new results of significant value and would only serve to confuse the analysis. For these reasons,
the evaluations completed for this study solely focus on classifier accuracy as the metric by which
the normalization methods are quantitatively compared.
6.2.4.1 Overall Accuracy
After image-sets for all normalization methods were generated, and a CNN classifier was trained
using each image-set, performance of the various models was analyzed to investigate how normal-
ization affects the ability of the CNN to classify task weight. To accomplish this, model accuracy
was calculated using the same method described in Section 5.2.4. The withheld test data for each
subject were used to obtain task weight predictions from each model, which, when validated against
the true task weight labels, were used to calculate classifier accuracy by dividing the number of
correct predictions by the total number of predictions in the test dataset. The corresponding clas-
sifier accuracy for each subject’s CNN models were then averaged across all subjects to quantify
an overall accuracy score for each CNN model trained using each normalization method (and using
each image type).
To compare the effectiveness of the normalization methods, this accuracy score was analyzed
6.2 Methods 152
through a statistical analysis performed using IBM SPSS 27. For each image type (grouped
spectrogram, stacked spectrogram, signal 1D convolution), a two-way Within-Subjects Analysis
of Variance (ANOVA) was completed where the two factors assigned were image fusion (three
levels: fused, EMG only, EEG only) and normalization method (five levels: subject, speed, image,
channel, none). A follow-up evaluation was done using the Bonferroni post hoc test to look at
pairwise comparisons of the normalization methods. The final result of this evaluation is three
separate two-way ANOVAs, one for each image type. The use of multiple two-way ANOVAs (as
opposed to a single three-way ANOVA, where image type is included as a factor as well) was chosen
to allow for a more detailed analysis of how normalization methods affect model performance for
each image type. If a three-way ANOVA is used, the pairwise comparisons of each normalization
method will be calculated across all image types, which does not allow for an observation of
how effects induced specifically via normalization techniques will differ for a single image type.
Since the multi/single-channel spectrograms, and signal images, convey EEG/EMG information
differently to the CNN classifier, it is possible that the various normalization methods may affect
these image types in unique ways. Using separate ANOVAs will allow for any differences between
normalization methods within a specific image type to be quantified.
To verify that the two-way Within-Subjects ANOVA is an appropriate method for analysing
the obtained results, normality was evaluated on the standardized residuals of the overall accu-
racies from each image type using the Shapiro-Wilk test. The results of the Shapiro-Wilk test
found that there were some violations in the assumption of normality in all three sets of distribu-
tions. Of the fifteen distributions collected from each image type (as a result of the combination
of three levels of fusion type and five levels of normalization method), the grouped spectrogram
and signal 1D convolution image types contained seven non-normal distributions each, while the
stacked spectrogram image type presented five non-normal distributions. Even though it is under-
stood that ANOVAs are generally robust against violations of normality, follow-up analysis was
performed using non-parametric testing to ensure that the comparison of normalization methods
was executed correctly. To verify the results found by the two-way Within-Subjects ANOVAs,
the Friedman test, along with the Wilcoxon Signed Ranks test (using the Bonferroni correction)
for post hoc pairwise comparisons, were applied. One limitation of these tests (and in fact many
6.2 Methods 153
non-parametric testing methods) is that they do no allow for multi-factor analysis: instead com-
paring all distributions pairwise without taking into account any relationships between them. To
perform an analogous comparison of the normalization methods using the non-parametric tests,
the accuracy distributions first had to be averaged across the three levels of fusion, transforming
the data from a two-way Within-Subjects design into a one-way Within-Subjects design where
the sole factor was normalization method. These new one-way design accuracies could then be
properly evaluated using the non-parametric tests. Following the transformation, the results of the
non-parametric testing methods matched the results obtained using the two-way Within-Subjects
ANOVAs for all three image types (the significances from pairwise comparisons using the Wilcoxon
Signed Ranks test matched the post hoc ANOVA pairwise comparisons exactly). Since there was
no change in the results found between the parametric and non-parametric statistics methods, this
indicates that the ANOVA is still a suitable approach, despite minor violations in the assumption
of normality. For this reason, only the ANOVA results are reported here, for simplicity.
6.2.4.2 Speed-Specific Accuracy
Similar to the evaluation done in Section 5.2.4, speed-specific accuracy was also calculated for the
CNN models trained using the previously discussed normalization methods. This was done by
separating the test images into two groups (slow and fast) based on the motion speed the subject
was performing when that image was captured, then calculating a different accuracy score for each
group. The purpose of this evaluation was to see how the normalization methods are affected by
changing motion factors during EEG/EMG signal collection (namely motion speed in this case).
To be useable in a real wearable mechatronic device, the classifiers need to be robust to changing
operating condition as the user dynamically employs the system for different tasks. Therefore,
evaluating the effect of motion speed on model accuracy can be used to investigate the robustness
of the classifiers and analyse their potential for use in a real-world system. Of particular interest to
this evaluation was how speed-wise normalization would perform during varying speed conditions.
Since this method normalizes the images based on fast/slow speed groups, it indirectly introduces
speed information into the CNN. It is possible that this additional information may have an effect
on the robustness of the CNN model as it encounters varying motion speeds.
6.3 Results 154
One thing to note, is that to simplify the analysis and reporting of the speed-specific accuracy
results, the number of models included in the evaluation was reduced. For reasons that will
be evident from the results shown in Section 6.3.1, image-wise and channel-wise normalization
methods were not included in the speed-specific accuracy analysis (since their overall accuracy was
shown to be consistently worse than the other normalization methods, further analysis was deemed
unnecessary). Also, the EEG/EMG only models were not included in the speed-specific accuracy
analysis either; since Chapter 5 already performed a detailed evaluation of the differences between
EEG–EMG fusion and EEG/EMG only models during varying speeds, it would be redundant to
repeat it here.
To facilitate an investigation of the effect that the normalization methods have on model perfor-
mance when changing speeds, two two-way Within-Subjects ANOVAs were performed (separated
by speed: one for fast, one for slow) where the factors assigned were image type (three levels:
grouped spectrogram, stacked spectrogram, signal 1D convolution) and normalization method
(three levels: subject, speed, none). The reason that two ANOVAs were completed was to allow
the tests to calculate across the fast/slow speed groups separately, this choice made using the same
logic that was applied to the overall accuracy analysis (previously discussed in Section 6.2.4.1).
6.3 Results
The overall accuracy results for all normalization methods when used with every image type can
be seen in Table 6.1. Note that the accuracy scores for the subject-wise normalization methods
are the same results that were shared in Chapter 5, since this was the normalization method used
during that study. The results have been repeated again here to improve readability and to make
it easier to compare subject-wise normalization with the other methods used in this evaluation.
Looking at the overall accuracy scores, some trends can be noted across all image types. All
models achieved an accuracy above chance level (33.33%), indicating that CNN classifier was suc-
cessful. The highest mean accuracy (81.57 ± 7.11%) was obtained using the grouped spectrogram
image type with no normalization. For all models, using no normalization resulted in the highest
6.3 Results 155
Table 6.1: The mean overall accuracy scores, calculated across both speeds and all task weights,
obtained from all models trained using each normalization method with all image types.
The mean accuracy scores for EEG–EMG fusion models, EMG only models, and EEG
only models are shown. Mean accuracy is reported as a percent (%) along with ± one
standard deviation.
Image Normalization Method

Fusion
Type Subject Speed Image Channel None
Fused 80.51 ± 8.07 81.28 ± 7.95 72.86 ± 7.93 71.85 ± 6.45 81.57 ± 7.11
Grouped
EMG 78.98 ± 4.66 79.46 ± 5.06 71.74 ± 5.58 72.78 ± 6.84 79.47 ± 4.96
Spectrogram
EEG 50.24 ± 17.06 51.63 ± 17.17 48.20 ± 12.09 42.88 ± 10.08 52.94 ± 17.20
Fused 80.03 ± 7.02 79.94 ± 6.61 70.34 ± 7.74 66.32 ± 7.72 80.48 ± 7.04
Stacked
EMG 78.09 ± 5.65 78.49 ± 5.83 71.06 ± 6.68 70.35 ± 6.23 79.33 ± 5.08
Spectrogram
EEG 48.44 ± 15.32 47.99 ± 15.16 45.14 ± 13.04 42.38 ± 9.44 49.1 ± 17.10
Fused 78.4 ± 8.70 79.96 ± 8.73 67.76 ± 12.2 63.12 ± 8.96 81.00 ± 7.60
Signal
EMG 74.73 ± 6.90 75.4 ± 7.10 63.97 ± 8.88 65.88 ± 13.51 78.08 ± 6.29
1D Conv.
EEG 41.44 ± 12.25 42.78 ± 15.85 48.1 ± 15.50 44.16 ± 10.50 51.08 ± 19.35
accuracy, outperforming all other normalization methods. For the grouped spectrogram and signal
1D convolution image types, speed-wise normalization had the second highest accuracy, beating the
subject-wise normalization method; however, the opposite was true for the stacked spectrogram
method (where subject-wise normalization demonstrated the second highest accuracy). For all
models, image-wise and channel-wise normalization methods resulted in a relatively large decrease
in accuracy, performing much worse than the other three normalization techniques. Regarding the
difference between EEG–EMG fusion and EEG/EMG only classifiers, for subject-wise, speed-wise,
and no normalization methods the accuracy of the EEG–EMG fusion models was higher than using
EEG/EMG alone for all image types; however, this trend was not always maintained for image-
wise and channel-wise normalization methods (in some cases EMG had a higher accuracy than
EEG–EMG fusion when using these).
The following sections discuss the results for each individual image type separately and look
at the comparisons between each normalization method within that image type in more detail.
6.3 Results 156
6.3.1.1 Grouped Spectrogram Images
The accuracy results for the grouped spectrogram image type are summarized in Figure 6.4 and
the results of the pairwise comparisons can be seen in Table 6.2. The highest accuracy was
obtained using EEG–EMG fusion models trained with no normalization (81.57 ± 7.11%), followed
by the speed-wise normalized models (81.28 ± 7.95%). Both normalization methods resulted in
a higher mean accuracy than the previously utilized subject-wise normalization method (80.51
± 8.07%); however, of the two, only the no-normalization accuracy was a statistically significant
improvement from the subject-wise normalization accuracy (p = 0.009 for no normalization, p =
0.369 for speed-wise normalization). As well, the difference between speed-wise normalization and
no normalization was not statistically significant (p = 1.000). The decrease in accuracy observed
when using the image-wise normalization (72.86 ± 7.93%) and channel-wise normalization (71.85
± 6.45%) methods was statistically significant when compared against the other three methods
(however, they were not statistically different from each other, p = 0.122).
Table 6.2: The p values obtained from the pairwise comparisons using the two-way ANOVA be-
tween the accuracies of different normalization methods when applied to the grouped
spectrogram image type. Statistically significant values (p<0.05) are shown in bold.
Pairwise Comparisons - Grouped Spectrogram

Norm. Method Subject Speed Image Channel None
Subject - 0.369 <0.001 <0.001 0.009
Speed 0.369 - <0.001 <0.001 1.000
Image <0.001 <0.001 - 0.122 <0.001
Channel <0.001 <0.001 0.122 - <0.001
None 0.009 1.000 <0.001 <0.001 -
6.3.1.2 Stacked Spectrogram Images
The accuracy results for the stacked spectrogram image type are summarized in Figure 6.5 and the
results of the pairwise comparisons can be seen in Table 6.3. The highest accuracy was obtained
from the EEG–EMG fusion models using no normalization (80.48 ± 7.04%), with subject-wise
normalization providing the second highest accuracy (80.03 ± 7.02%). Speed-wise normalization
6.3 Results 157
100
Grouped
EMG
90
EEG
80
70
60
50
40
30
ct eed age nel ne
bje Sp Im an No
Su Ch
Figure 6.4: The mean overall accuracy of each normalization method when used with the grouped
spectrogram image type. Models developed using EEG–EMG fusion are shown on the
left in purple, models developed using EMG only are shown in the centre in blue, and
models developed using EEG only are shown on the right in red. Error bars represent
models is 33.33%).
EEG–EMG fusion models had a slightly lower mean accuracy of 79.94 ± 6.61%; however, the
accuracy differences between these three normalization methods was not statistically significant for
any combination of comparisons. Again, image-wise and channel-wise normalization both resulted
in a statistically significant reduction in accuracy when compared to subject-wise/speed-wise/no
normalization; however, for the stacked spectrogram image type, the accuracy difference between
image-wise (70.34 ± 7.74%) and channel-wise (66.32 ± 7.72%) normalization was statistically
significant (p = 0.032).
6.3.1.3 Signal 1D Convolution Images
The accuracy results for the signal 1D convolution image type are summarized in Figure 6.6 and
the results of the pairwise comparisons can be seen in Table 6.4. The results obtained from
the signal 1D convolution image type mirror the pattern observed from the grouped spectrogram
6.3 Results 158
100
Stacked
EMG
90
EEG
80
70
60
50
40
30
ct eed age nel ne
bje Sp Im an No
Su Ch
Figure 6.5: The mean overall accuracy of each normalization method when applied to the stacked
spectrogram image type. Models developed using EEG–EMG fusion are shown on the
left in purple, models developed using EMG only are shown in the centre in blue, and
models developed using EEG only are shown on the right in red. Error bars represent
models is 33.33%).
models: no normalization accuracy is the highest (81.00 ± 7.60%), followed by speed-wise (79.96 ±
8.73%) then subject-wise (78.4 ± 8.70%) normalization. However, unlike what was observed for the
grouped spectrogram models, no normalization resulted in a statistically significant improvement
over both speed-wise and subject-wise normalization (p = 0.001 in both cases), indicating that
it was the best performing method by a noteworthy degree. The difference between speed-wise
and subject-wise normalization, as previously observed for both other image types, was found
to not be statistically significant (p = 1.000). As well, The decrease in accuracy observed when
using image-wise normalization (67.76 ± 12.2%) and channel-wise normalization (63.12 ± 8.96%)
methods was statistically significant when compared against the other three techniques (however,
they were not statistically different from each other, p = 1.000, similar to what was observed for
the grouped spectrogram image type).
6.3 Results 159
tween the accuracies of different normalization methods when applied to the stacked
spectrogram image type. Statistically significant values (p<0.05) are shown in bold.
Pairwise Comparisons - Stacked Spectrogram

Subject - 1.000 <0.001 <0.001 0.346
Speed 1.000 - <0.001 <0.001 0.175
Image <0.001 <0.001 - 0.032 <0.001
Channel <0.001 <0.001 0.032 - <0.001
None 0.346 0.175 <0.001 <0.001 -
100
1D Conv.
EMG
90
EEG
80
70
60
50
40
30
ct eed age nel ne
bje Sp Im an No
Su Ch
Figure 6.6: The mean overall accuracy of each normalization method when applied to the signal
1D convolution image type. Models developed using EEG–EMG fusion are shown on
the left in purple, models developed using EMG only are shown in the centre in blue,
and models developed using EEG only are shown on the right in red. Error bars
represent one standard deviation. Note that the y axis begins at 30% (chance level for
these models is 33.33%).
6.3.2 Speed-Specific Accuracy
The accuracy results showing speed-specific performance of the subject-wise normalization, speed-
wise normalization, and no normalization methods can be seen in Figure 6.7. Looking at the plot, it
6.3 Results 160
tween the accuracies of different normalization methods when applied to the signal 1D
convolution image type. Statistically significant values (p<0.05) are shown in bold.
Pairwise Comparisons - Signal 1D Conv.

Subject - 1.000 0.011 0.001 0.001
Speed 1.000 - 0.003 0.001 0.001
Image 0.011 0.003 - 1.000 <0.001
Channel 0.001 0.001 1.000 - <0.001
None 0.001 0.001 <0.001 <0.001 -
can be seen, as was found in Chapter 5, that model accuracy during slow motions was significantly
higher than during fast motions for all image types and normalization methods. When focusing
on the mean accuracy values for the fast motion speed, a slight trend emerged where speed-wise
normalization resulted in minor improvements, for all image types, when compared to subject-wise
and no normalization. The main effect of normalization for the fast speed analysis was not found to
be statistically significant however, (p = 0.155), so little can be said about the importance of this
trend. For this reason, no post hoc statistics results will be presented for the fast speed analysis,
since no claims can be confidentially stated regarding the pairwise comparisons of normalization
methods without a statistically significant main effect. For the slow speed analysis, the main effect
of normalization was found to be statistically significant (p = 0.004); however, the observed power
for this evaluation was found to be right on the cusp of acceptability (observed power = 85.4%).
Hence, this result could be made stronger through further study to confirm that this significance
still applies for a larger population. From the post hoc analysis of the slow speed normalization
methods, the only pairwise comparison that was found to be statistically significant was between
subject-wise and no normalization (p = 0.014), which aligns with the finding from the overall
accuracy results.
6.4 Discussion 161
100

90
80
70
60
50
40
Slow
Fast
30
t d e t ed ne t d ne
jec ee on jec pe No bj
ec ee
No
ub -S
p N ub S u Sp
-S d- S - d -
-S v.
- . -
ed pe
d pe d- ed ke nv
.
on nv
p ou ou c ke a ck t ac o C Co
ou Gr Gr St
a S t S C 1D
Gr 1D 1D
Figure 6.7: The mean accuracy, separated by the two speed levels (fast and slow), for the EEG–
EMG fused CNN models developed using the three best normalization methods:
subject-wise, speed-wise, and no normalization. Models of the same image type are
grouped together, with the order of the groups from left to right as follows: grouped
spectrogram models, stacked spectrogram models, and 1D convolution signal models.
Error bars represent ± one standard deviation.
6.4 Discussion
The goal of this study was to evaluate the effect of image normalization on CNN-based EEG–
EMG fusion used for task weight classification. Following the work done in Chapter 5, CNN
classifiers were trained using spectrogram/signal images that had been normalized using various
methods, and the accuracy obtained from resultant models was compared to determine the best
normalization technique to use during classifier training. An investigation into the effect that
normalization has on classifier robustness was also performed by analysing speed-specific accuracy,
showing how the model performs during changing motion conditions (simulating the variations
in motion speed required during different activities of daily living). By evaluating normalization
techniques for EEG–EMG fusion CNN classifiers, this work provides the first step in refining the
model development process introduced in the previous study presented in Chapter 5. The use of
new normalization techniques demonstrated an increase in model accuracy when compared to the
previous work and shows how these models can continue to be improved in many ways through
6.4 Discussion 162
rigorous investigation into the various aspects of their design/architecture. This will ultimately
result in more accurate task weight classification models that can be integrated into wearable
robotic systems to improve device usability and safety.
6.4.1 Comparing Overall Accuracy of Normalization Methods
Using the overall accuracy scores, a comparison of each normalization method for the three image
types can be performed. The purpose of this evaluation is to see which normalization method will
result in the highest accuracy, thus presenting the greatest potential as a pre-processing technique
for future model development. All normalization methods resulted in an accuracy over the chance
level (33.33% for a 3 class output) for all image and signal types, implying that every normal-
ization method evaluated in this study presents at least a rudimentary feasibility for use with
CNN-based EEG–EMG fusion. In general, the trend demonstrated in Chapter 5 of EEG–EMG
fusion outperforming EMG only models was also maintained for the majority of normalization
methods investigated (namely subject-wise, speed-wise and no normalization), further indicating
the feasibility of these techniques for use with EEG–EMG fusion, since each normalization ap-
proach still allows for proper extraction and combination of relevant EEG and EMG information
from the rescaled images.
However, some of the normalization methods consistently demonstrated worse performance
compared to others, providing strong evidence to exclude these procedures from future work. The
normalization methods that rescaled pixel values across a single image at a time (i.e., image-
wise and channel-wise normalization) resulted in statistically significant decreases in accuracy
when compared against the other methods, and this held true for all image/signal types. These
techniques also did not always maintain the trend of EEG–EMG fusion outperforming EMG only
models, meaning that using these normalization methods interfered with the ability of the CNN
to properly extract and combine EEG/EMG information. It is likely that normalizing across a
single image is causing noise in the image to be amplified to an unacceptable degree during the
rescaling of pixel values. As was speculated in Section 6.2.2.3 and 6.2.2.4, the benefits of giving
individual images more weight within the dataset were ultimately cancelled out by the associated
amplification of noise within these images. It can be concluded from the overall accuracy results
6.4 Discussion 163
that these methods of normalization are not nearly as effective as the methods that rescale images
across a group, and single-image normalization should not be implemented during any future work
that focuses on the development EEG–EMG fusion-based CNN classifiers. While single image
normalization techniques may be beneficial in some image-recognition applications [249–252], and
certain EMG-only CNN classifiers [181, 269, 270], it is clear that they are not suitable for EEG–
EMG fusion-based CNN classification of task weight.
Focusing on the other three normalization methods, it can be seen that no normalization was
the most effective method to use with CNN-based EEG–EMG fusion models. For all image/signal
types, it resulted in the highest accuracy, demonstrating a consistent trend of improved classifier
performance through its use. This finding does align with the opinion of some researchers, that
the use of EEG/EMG normalization should be avoided since it can obscure relevant information
[167, 178] and its exclusion during pre-processing can result in better performance when compared
to normalized signals [179]. It should be noted however, that while this trend can be very clearly
established, the results of the statistical analyses performed for each image type is not entirely
conclusive and leaves some room to debate the superiority of forgoing normalization during im-
age pre-processing. For the signal 1D convolution image type, this result is clear, as there was
a statistically significant difference between no normalization and both subject-wise and speed-
wise normalization. However, for the grouped spectrogram method, no normalization was only
statistically significantly different from subject-wise normalization (but not speed-wise normaliza-
tion). For the stacked spectrogram image type, none of the three methods showed a statistically
significant difference from each other. Only when using signal 1D convolution image type-based
models can a confident claim be made that no normalization results in the highest classification
accuracy; however, the other two image types do show an obvious trend in this direction and it
is highly likely that increasing study power (by collecting data from more subjects, for example)
would result in the difference becoming statistically significant. While it is temping to say that
no normalization is the best method to use with EEG–EMG fusion-based CNN classifiers, and
to recommend the use of this pre-processing technique universally, there are some caveats to this
claim that need to be addressed, arising from the limitations of this study and how they may affect
model training. It is possible that one reason why not normalizing input images provided the best
6.4 Discussion 164
accuracy in this study is because all models developed are subject specific (meaning that each
person has CNN classifiers that are trained using their data only). Since the models are optimized
to that subject’s bioelectrical signals specifically, the classifier can fit itself around their unique
range of values without needing to account for any drastic changes that would be seen if a second
subject’s signals were then used. This means that the need for standardized inputs is much smaller
than it would be for a general-population model that has to handle EEG/EMG signals from many
different people. Therefore, many of the usual benefits of normalization (e.g., reconciling differ-
ences between varied inputs) may not be as prevalent for the subject-specific models implemented
here, and instead, the improved resolution obtained from the non-normalized images that were
allowed to keep their original appearance (since their pixel values were not altered to fit between 0
and 1 during rescaling) is providing greater returns in model performance. It is important to note
that it is likely that this behaviour will not carry over into multi-person models that are designed
to work for many subjects, and for these types of models, it may be better to use a different
normalization method. Future work should continue to analyse these normalization approaches on
general-population CNN classifier models to see how they perform under such conditions. Along
the same vein, another limitation of this study is that data collection only happened during one
session (meaning that the subjects did not return on a different day to record a second set of
EEG/EMG signals). It has been demonstrated that EEG and EMG signal behaviour can vary
greatly over time, even within the same subject [254–257]. This means that even subject-specific
classifier models will still need to be able to handle changes in input value ranges. Similar to the
discussion of the multi-subject models, it remains to be seen if subject-specific CNN classifiers
using non-normalized images would still perform the best under these conditions; therefore, fur-
ther work should be performed to investigate this question. Despite no normalization appearing
superior in this study, it is important to consider that this result may not hold as the application
of these models moves to more complex problems emulating real-world conditions; hence, other
approaches that use normalized images should still be evaluated in future work.
With this in mind, a comparison of the methods that do normalize image values should be
discussed to see if one of them demonstrates more potential for use in future model development.
Based on the discussions above, this leaves subject-wise and speed-wise normalization as the two
6.4 Discussion 165
potential methods to consider. Unfortunately, it is hard to draw a strong conclusion from the overall
accuracy results regarding which normalization technique is superior. For all three image types, no
statistically significant difference was observed between subject-wise and speed-wise normalization.
Looking at the mean accuracy itself, speed-wise normalization resulted in a higher accuracy for the
grouped spectrogram and signal 1D convolution image types, but, when using stacked spectrogram
images, subject-wise normalization had the higher accuracy between the two (although it should
be noted that the mean accuracy was very close in this case, with only a 0.09% difference). It
is clear that the performance of both methods is somewhat similar, making it challenging to
confidently recommend one; however, looking at the trend that develops in the mean accuracies,
it seems that speed-wise normalization holds the most promise as the method to use during model
development. For two of the three image types (spectrogram grouped and signal 1D convolution),
it resulted in a higher mean accuracy and, for the third image type (stacked spectrogram), the
accuracy difference was very small between the two normalization methods, to the point of being
almost equivalent. Based on this trend, one could expect that using speed-wise normalization over
subject-wise normalization will provide an improvement in model accuracy, or will at the very least
offer functionally equivalent performance. It can be concluded that providing indirect motion speed
information to the CNN classifiers will either improve performance slightly (for some image types),
or will not change performance significantly (and at the same time will not significantly hinder
performance either). This preliminary result shows the feasibility of using speed-wise normalization
when developing task weight classification models using EEG–EMG fusion-based CNNs. It shows
that this normalization method has potential to improve model accuracy and should be investigated
further in future work. Using this technique, the models developed in Chapter 5 were iterated upon,
beginning the process of further refining the EEG–EMG fusion-based CNN classifiers introduced
in that study.
6.4.2 Performance of Normalization Methods While Varying Motion Speed
To further evaluate the effectiveness of each normalization method, speed-specific accuracies were
calculated to see how motion speed affects performance, and to see if changing the approach
used when normalizing input images modifies this behaviour in any way. Ideally, the task weight
6.4 Discussion 166
classification models should be robust enough to remain accurate during varying motion speeds,
since the ultimate end-goal for these classifiers is to be used as part of a wearable mechatronic
device that can perform a range of different rehabilitation and assistance tasks. Looking at the plot
of speed-specific accuracies, a similar trend can be seen as the one observed in Chapter 5, where
the slow-speed motions have a higher average accuracy than the fast-speed motions for all image
types and normalization methods. This is to be expected, since the base CNN model configuration
for each image type was not changed in this study (only the input images were modified using
different normalization methods), so it stands to reason that the behaviour of the new models
would be similar to that of the previous ones. The same factors discussed in Chapter 5 that may
cause the slow speed motions to have higher accuracy will still apply for the models developed
here; however, it is still worthwhile to verify this finding in case normalization methods did have
an effect on this behaviour.
Looking more closely at how motion speed affects the accuracy observed when using each
normalization method, it is unfortunately hard to draw any concrete conclusions from the results
obtained. For the fast speed motions, the main effect of normalization on accuracy was not
statistically significant. For the slow speed motions, the main effect was statistically significant, but
the observed power was lower than desired, indicating the effect may require further investigation
to be able to confidently apply it to a wider population. From this evaluation, no normalization
was found to be significantly higher than subject-wise normalization during slow speed movements,
which demonstrated that, at least for slower speeds, the trend seen in the overall accuracy results
of no normalization performing better was somewhat upheld (although, with the other pairwise
comparisons failing to find significance, it is hard to make this claim definitively). It seems that
this experiment was ultimately unable to fully capture the differences in task weight classifier
performance developed using different normalization methods when operating under changing
motion speeds. While it is possible that the results here are “true” and that motion speed does
not affect the various normalization methods in a different manner, the more likely scenario is, as
with all statistical analyses, that the test just could not capture the small effect present in these
data, so further work should be done to investigate this phenomenon more thoroughly. Increasing
study power by adding more subjects, or having subjects perform more motion repetitions, could
6.4 Discussion 167
help strengthen the evaluation. Future work could also look at evaluating a larger number of
speeds/weight combinations to see if greater changes in motion conditions elicit a larger effect on
model performance.
While no strong claims can be made about the speed-specific accuracy results, a qualitative
analysis of the plot can be useful for looking at possible trends in the data that may highlight areas
to focus on in future investigations. Looking at the plot, one very minor trend that does appear is
the slight increase in mean accuracy when using speed-wise normalization during the fast motion
speed trials. For all three image types, the highest mean accuracy observed during fast motion
was obtained using speed-wise normalized images and, while this increase was not statistically
significant, it does seem to be too consistent to be a mere coincidence. It is possible that indirectly
incorporating speed information into the model via normalization is slightly improving the ability
of the CNN to classify images generated during the more aggressive movements performed at a fast
speed. This hypothesis lines up with the behaviour observed in Chapter 4, where adding a speed
feature into an SVM model improved task weight classification accuracy. The slight increase in fast-
speed accuracy seen when using speed-wise normalization may be why it slightly edged out subject-
wise normalization in the overall accuracy results (and interestingly, this increase in fast-speed
accuracy is lower for the stacked spectrogram image type, which saw subject-wise normalization
have a slightly higher overall accuracy compared to speed-wise normalization). It is important to
note however, that this trend was not observed during the slow speed motions, so either the indirect
speed information imparted onto the CNN model was not relevant during slow motions speeds, or
the benefit was small in comparison to the other factors affecting model performance, offsetting its
contribution. Regardless, without strong quantitative statistical results to support this analysis,
the potential trend discussed here is more speculation than anything, but the presence of this
repeating behaviour does justify further investigation of speed-wise normalization, as it appears
there may be a worthwhile reason to use it for model development, if refined further. While not
conclusive on their own, the speed-specific accuracy results provide a secondary justification for
the decision to recommend speed-wise normalization as the primary method for future evaluation.
This evaluation supports the conclusions drawn from the overall accuracy results, and demonstrates
the presence of interesting behaviour when using speed-wise normalization during varying motion
6.4 Discussion 168
speeds, alluding that there may be untapped potential within this approach that should be analyzed
in-depth in future work.
6.4.3 Future Work
By investigating different normalization methods, this study resulted in EEG–EMG fusion-based

CNN task weight classifiers that obtained higher accuracies than the initial subject-wise normalization-
based models shown in Chapter 5. Future work should continue to investigate new techniques to
use during classifier development/training that will increase performance through the refinement
of the model architecture. Along with the recommendations that have already been discussed in
this section, there are several directions on which to focus to move towards achieving this goal.
One avenue of future work with many options to potentially explore is the investigation of other
methods of image normalization. Since this work was a preliminary analysis of normalization tech-
niques used with EEG–EMG fusion-based CNN classifiers, it only evaluated a small number of
methods, but many other normalization approaches remain untested that may improve the models
further. One aspect that could be evaluated is the use of channel-specific normalization in con-
junction with methods that normalize across a group of images. The channel-wise normalization
method implemented in this study only rescaled each channel within a single image, basically
serving as a variant of image-wise normalization. There is no reason, however, that this approach
(using the max/min values for each channel separately, allowing all channels to contribute equally
within the image) could not be applied to techniques where images are rescaled across a larger
dataset, such as subject-wise and speed-wise normalization. It would only require the maximum
and minimum pixel values for each subject/speed to be found per channel. For the five channel
images used in this study, this would result in five max/min pairs for subject-wise normalization
and ten max/min pairs for speed-wise normalization (due to the two speed levels used during el-
bow flexion–extension motion). The poor performance of channel-wise normalization observed in
this study is likely due to its implementation being tied with image-wise normalization, which also
demonstrated lower accuracy. Regardless of this finding, it does not mean that the approach of
giving all channels equal weight within an image holds no merit entirely. Implementing normaliza-
tion in this way may allow the signals to retain more channel-specific characteristics after rescaling
6.4 Discussion 169
and provide more physiologically relevant information to the CNN. Since each channel corresponds
to different electrodes placed at various body locations, their respective signals may demonstrate
unique behaviour based on what muscle group or brain region is being recorded. As such, there
may be benefits in allowing each bioelectrical signal channel to normalize around its specific char-
acteristics. This approach emulates how feature standardization is performed during traditional
machine learning model development (where each feature, for each channel, is normalized sepa-
rately across the entire feature-set when the matrix columns are standardized using z -scores), so it
stands to reason that this practice may successfully transfer over to image normalization as well.
There is no guarantee that this approach will be superior however, since allowing each channel to
contribute a similar amount within the image may result in a loss of information regarding the
relationship between channels that is demonstrated by proportional variations in their pixel values.
For example, when performing elbow flexion, one would expect a much larger EMG response from
the biceps channel than the triceps channel (since the biceps muscle is primarily responsible for
this motion). However, if a channel-wise normalization method is used, it may cause the smaller
triceps EMG values to appear larger (and thus closer to the already large biceps EMG values),
which may make the specific elbow motion being performed harder to identify by a classification
model. Future work should examine speed-channel-wise and subject-channel-wise normalization
to see if this concept has the potential to improve EEG–EMG fusion-based CNN classifiers further,
and should compare this normalization strategy to the equivalent group-wise normalization meth-
ods already demonstrated here to evaluate which approach provides more relevant information
from the bioelectrical signals.
Another option for future work is to evaluate other mathematical techniques employed to
modify pixel values during image normalization. In this study, the differences between the stated
normalization methods arose from how the upper and lower bounds (i.e., the EEG and EMG
max/min pixel values) were chosen from the image-sets. The actual normalizing operation was
performed using the same method, rescaling the images to translate the lower/upper bound pixel
values into 0 and 1, respectively. While rescaling using the min–max method is a common im-
plementation of image normalization, it is far from the only technique used. Other statistical
methods, such as standardizing the images by calculating z -scores for each pixel [247], or centering
6.4 Discussion 170
the image by subtracting the mean pixel value [247, 252] are also used to prepare images for use
with CNN classifiers (among other methods). It is possible that these more advanced formulas
may result in better model performance (compared to images rescaled using the min–max method)
by providing a better representation of image behaviour after the transformation of pixel values.
In particular, standardization has been demonstrated extensively in EEG/EMG-based applica-
tions, with studies calculating z -scores for manually extracted EEG [262–264] and EMG [187, 279]
features, as well as standardizing EEG/EMG signal images [164, 168, 181, 276] and EMG spec-
trogram images [271] for use with CNN classifiers. The use of centering for spectrogram image
normalization has also been shown previously for an EMG-based CNN model [175]. Given that
these techniques have been implemented successfully in EEG-only and EMG-only CNN classifiers,
they should also be evaluated during future work as a potential pre-processing step for developing
EEG–EMG fusion-based CNN models.
Finally, future work should also explore other methods of refining EEG–EMG fusion-based
CNN model architectures that extends beyond normalization techniques. While this study only
focused on an evaluation of different normalization methods in particular, there are many other as-
pects of the CNN model configuration that can be iterated upon to improve classification accuracy.
The work done in Chapter 5 to develop base model configurations and optimize hyperparameters,
as well as the work done here to evaluate various methods of EEG–EMG image normalization,
present merely a preliminary investigation into the optimal way to utilize CNNs as a method of
EEG–EMG fusion. There are a near limitless number of model parameters and configuration de-
sign decisions that could be made during CNN classifier development; the work outlined here only
begins to scratch the surface of what could be done. Now that the feasibility of using CNNs as
a method of input-level EEG–EMG fusion has been demonstrated, and work has begun to refine
these models via iteration on the normalization method used during image generation, further
work should be completed to continue to explore new ways of increasing model performance. One
CNN model configuration that could be investigated in future work is the use of parallel convolu-
tion layers as part of the classifier architecture. Parallel convolution layers are a CNN technique
that use multiple branching paths of convolution to initially extract different features from an
image/dataset separately, which are later combined in the model pipeline when the parallel layers
6.4 Discussion 171
converge into a single layer. This contrasts the more commonly strategy of sequential convolution
layers (used for the work done here), where there is only one path for the features to travel in the
model pipeline. Parallel convolution can be implemented in a way where each branching path is
provided with the entire dataset. The benefit of this approach is that parallel paths can perform
convolution using different hyperparameters, such as kernel size, to extract different features from
the same image [280]. However, of more interest to the work here is the use of parallel convolution
layers to extract features from different types of data (where each path is provided with unique
inputs). In the context of a combined image input, parallel convolution can be implemented by
separating the initial convolution of the image such that each parallel layer passes their kernel along
only one region of the image (where the separate regions contain unique types of information).
Parallel convolution has been demonstrated successfully as a sensor fusion method, where different
sensing modalities are used as inputs to the parallel paths and are only combined after features
have been extracted through convolution [277, 281, 282]. The benefit of this approach is that
each branch can learn sensor-specific features; since only one type of sensing modality is provided
to each convolution path, the CNN can optimize feature extraction to the specific information
provided by each input type (and model architecture/hyperparameters can be optimized for each
sensor, since the parallel paths can be implemented with different settings). It also allows for the
extraction of higher-level features from the individual sensing modalities, since multiple sequential
convolution layers can be used in each parallel branch before combining them. In the context
of EEG–EMG fusion, the use of parallel convolution presents an obvious opportunity: the paths
could be configured to initially perform convolution on EEG and EMG separately, then fuse them
within the model after extracting higher-level features. While parallel convolution has been used
with EEG [166–168, 171, 272] and EMG [177, 178, 276] before, it has not yet been evaluated for
EEG–EMG fusion, so it should be investigated in future work to see if it can provide improvements
to model performance.
Another method that should be considered for future work is the use of CNNs for decision-level
EEG–EMG fusion. The CNN models developed in this work all operate using input-level fusion,
meaning the EEG and EMG signals are combined before being inputted into a single classifier
(in this case, they are combined together within the image); however, nothing precludes CNNs
6.5 Conclusion 172
from being used for decision-level fusion as well (where two CNNs are trained, one for EEG and
one for EMG, and their output classifications are merged to result in a singular final output). As
a point of clarification, it should be noted that the key distinction of decision-level fusion is the
training of two separate classifiers (one for each signal type), thus parallel convolution would still
be considered input-level fusion, as it ultimately results in a single CNN classifier being trained
using both EEG and EMG image information simultaneously. Work should be done to compare
input-level fusion and decision-level fusion when using CNNs for EEG–EMG fusion. As a starting
point, decision-level fusion could be implemented using the methods investigated in Chapter 4;
however, other methods should be explored as well. A study in the literature has utilized deep
learning-based decision-level EEG–EMG fusion [75]; however, it was not performed using CNNs
specifically (instead using LSTMs) or developed for task weight classification. It remains to be
seen how decision-level EEG–EMG fusion will perform when used with CNN classifiers. Further
investigation of new model architectures for CNN-based EEG–EMG fusion, both mentioned here or
otherwise, will ultimately help improve the performance of these models and move them to a point
where they exhibit the robustness required for integration into real-world wearable mechatronic
systems. The utilization of EEG–EMG fusion-based task weight classifiers will help improve the
control of wearable robotic devices by allowing the user to more easily operate the device, leading
to better comfort/safety and increased adoption of these novel rehabilitation and assistive tools.
6.5 Conclusion
This work provided an evaluation of different image normalization methods used when training
EEG–EMG fusion-based CNN models for task weight classification. Serving as a follow-up to the
work done in Chapter 5, this study used the same model configurations developed previously and
retrained them with multiple EEG–EMG image-sets that were normalized using several different
approaches. This facilitated an evaluation of EEG/EMG image normalization methods to inves-
tigate how they affect classification accuracy and to compare each method in an effort to find the
best technique to use during future classifier development. The results of this analysis found that
non-normalized images provided the best model accuracy for all image types; however, this finding
6.5 Conclusion 173
may be limited to subject-specific and single-session trained CNNs. Speed-wise normalization gen-
erally provided the second best model accuracy, and demonstrated an interesting trend of slightly
increased accuracy during fast motion speed trials (likely due to the indirect incorporation of speed
information into the model when normalizing). Both factors suggest that speed-wise normalization
presents a promising alternative for situations where non-normalized images may not be suitable
(such as development of general population models). Future work should continue to investigate
other methods of normalizing images for CNN model training by both iterating on the methods ex-
plored here (through an evaluation of channel-specific speed-wise normalization) and by analysing
new normalization methods that use different underlying formulas when modify image pixel values
(e.g., standardization using z -scores). Beyond normalization methods, future work should also
explore other strategies of refining CNN-based EEG–EMG fusion models by investigating new
CNN classifier configurations (such as, parallel convolution) and evaluating decision-level fusion
when implemented with CNNs specifically. This work demonstrates the first step towards refining
the previously introduced CNN-based EEG–EMG fusion models for task weight classification by
iterating the image processing methods used during classifier training through the use of new nor-
malization methods. This iteration resulted in an improvement in classification accuracy over the
previous procedure, showing that EEG–EMG fusion-based CNN models can, in fact, be improved
further via continued model development, which encourages their use in future work. Improving
task weight classification will ultimately lead to wearable mechatronic devices that can respond
more intelligently to varying tasks, allowing for increased usability and safety during operation.
Through these improvements, the field of wearable mechatronics will be one step closer to actual-
izing the goal of wide-spread adoption of robotic rehabilitation and assistance devices, providing
such systems the chance to improve the lives of more patients worldwide.
Chapter 7
Development of a Task Weight

Selective Control System
7.1 Introduction
Up until this point, the focus of the work in this thesis has been exclusively on machine learning
problems. The projects outlined in the previous chapters all focus on the design and evaluation
of various EEG–EMG fusion-based classifiers, with a specific emphasis given to task weight (the
load the subject is holding during motion) prediction. However, the actual motivation behind
this thesis, as stated in Chapter 1, was to determine a way to improve the controllability of
wearable upper-limb mechatronic devices used for assistance and rehabilitation: allowing them to
safely operate during situations where an applied external load (i.e., the weight the user is holding)
varies. Thus, the project presented here moved away from machine learning as a focus, and instead
began an investigation into control system methodologies for wearable robotic devices. The final
goal of this project was to develop a method for integrating the previously developed EEG–EMG
fusion classifiers into a wearable device control system, creating a novel method of control that
improves performance.
It is a well known problem that upper-limb wearable mechatronic devices often lack adapt-
ability to changing task conditions and external disturbances [6, 11–13, 22, 24, 26, 30–46]. Many
control systems are tuned specifically for one situation (optimized to only that set of modelled
174
dynamics) and require re-tuning each time a new task needs to be executed to maintain suit-
able output behaviour. Changes in the desired motion trajectory (speed/acceleration), as well as
changes in external load (weight the user is holding) can easily overwhelm traditional control sys-
tem approaches and result in poor/unstable performance [6, 13, 22, 26, 27, 30–33, 37, 39, 42, 101].
This property presents a huge barrier towards to the adoption of these systems in rehabilitation
and assistance scenarios. If the device is uncomfortable or difficult to control, the user will be-
come frustrated and abandon it as an assistance/rehabilitation solution [11, 15, 26, 29]. Surveys
of patients who have tested wearable robotic devices have consistently listed comfort [7, 29] and
ease of use [7, 26] as key areas of improvement to help increase adoption of these novel solutions.
In a similar vein, the wearable device needs to allow for generalized operation of the system across
a diverse range of tasks for it to be beneficial for everyday use [11–13, 22, 30, 31]. The desire
of many in the field of wearable mechatronic rehabilitation/assistance tools is a task-independent
device that the user can wear constantly for help with all of their activities of daily living [11].
To accomplish this objective, the control system used by the device needs to be able to respond
appropriately during a range of different dynamics and motion conditions, which is currently a
challenging endeavour and an active area of research.
There are many examples in the literature of control system approaches that were developed
to allow wearable robotic devices to adapt better to task weight variations. Many of these studies
focus on solutions based on methods of modern nonlinear control theory, to implement adaptive
and robust control systems into the devices. These are advanced control law techniques that
seek to improve system response in the presence of disturbances and modelling uncertainties, with
each employing a different approach to the problem. Adaptive control is a category of techniques
that use various methods to update control system parameters in real time, so that the device
can change its output behaviour as the situation demands it [283]. Robust control, on the other
hand, is a category of techniques where the control law does not change during run-time (unlike
in adaptive control); however, various methods are instead utilized to tune parameters in such a
way that the system will display stable performance across its predetermined range of expected
conditions. This stability will be maintained regardless of the presence of external disturbances or
modelling uncertainty within that range (essentially ensuring that the system is optimized for every
situation it may expect to face) [284]. Note that the use of these approaches is not exclusive, and
the combination of adaptive and robust control law techniques can also be implemented to further
improve performance [285]. Regarding external load compensation in upper-limb wearable robots,
the use of adaptive [12, 13, 23, 24, 26, 27, 39, 44, 286] and robust [12, 13, 23, 24, 41, 42, 46, 50]
methods (or a combination of both [31, 34, 35, 38, 40, 42, 43, 45, 287]) has been demonstrated
frequently, with the use of various implementations of a robust technique called Sliding Mode
Control in particular being a popular choice to improve performance during the presence of external
disturbances [30, 34, 35, 40, 41, 50, 287]. Other control-law methods based on the relation of
position and force feedback during control, called impedance and admittance control, have also
been demonstrated frequently in upper-limb wearable devices [13, 24, 27, 32, 45, 47, 100, 288].
Impedance control works on the principle of controlling output force based on input position
(applying actuator force to correct for a position error), while admittance control does the opposite:
controlling output position based on an input force (moving the robot accordingly to maintain the
desired force levels) [13, 47]. Of course, impedance/admittance control can be combined with
robust/adaptive techniques to develop very advanced control-law approaches to drive upper-limb
wearable robots.
In addition to methods based solely on modern nonlinear control theory [38–42, 46], there is
another approach that has been demonstrated to improve the ability of upper-limb wearable devices
to adapt to disturbances caused by changing external loads: the use of various sensor measurements
in tandem with physiological/biomechanical human models and upper-limb brace system models.
By measuring EMG signals and rigorously modelling as many aspects of the human muscle motor
response as possible (through the use of biomechanics-based kinematic models of human motion
and muscle-force modelling approaches), research groups have showcased devices that are able to
adapt to changes in task weight based on the human model output [11–13, 22–24, 26, 27, 33, 47,
48, 50, 100]. Other groups have utilized force/torque sensors built into the mechatronic upper-limb
brace directly, to measure the human–robot interaction forces and make inferences about external
loading conditions on the device using models of the system so that compensation can be applied
accordingly using various control law approaches [31, 32, 36, 37, 39, 45, 47, 288, 289].
These examples of upper-limb wearable device control, while successful in demonstrating im-
proved external disturbance compensation, share similar drawbacks that make them difficult to
implement within wearable devices specifically. The first drawback is computational complex-
ity. Modern nonlinear control theory typically relies on state-space implementations of system
dynamics models, which often necessitate the use of matrix-math operations to execute, and
can require the calculation of complex control law equations. Likewise, physiological modelling
approaches typically uses a series of detailed equations during device control to calculate hu-
man muscle force and body motion based on the kinematic/physiology models employed. Such
processes may be difficult to realize fully in a wearable device where computational capabilities
are limited, due to the frequent use of microcontroller-based embedded systems in their designs
[6, 41, 47–49]. Regarding the control methods based on built-in force/torque sensors specifically,
beyond the computational limitations, a major challenge with these techniques is the added cost
and hardware complexity required to incorporate these sensors into a wearable robotic device.
Force/torque sensors can be expensive and their size/method of integration into the mechani-
cal system of the brace can impact the user’s comfort and hinder their ability to move freely
[14, 45, 50]. The second drawback of these methods, applying to all techniques discussed thus far,
is a reliance on accurate system models in their control law implementations. Modern nonlinear
control theory (i.e., adaptive/robust control and impedance/admittance control), as well as the
physiological modelling/sensing approach, both require accurate models of the system being con-
trolled to utilize during their calculations. Upper-limb wearable devices, by their very nature as a
human-driven system, can be challenging to model in a way that captures all important behaviour
[11, 12, 23, 26, 27, 31, 32, 34–38, 40, 41, 43, 45, 46, 290]. The extreme complexity and variability
(both between different users and within the same user) of the internal processes/anatomy of the
human body [11, 12, 23, 26, 27, 36, 40, 41, 43, 45, 286, 290], combined with unpredictable/com-
plicated human–robot interaction forces as the user performs a variety of activities of daily living
[11, 12, 31, 32, 34, 35, 37, 290], and the presence of actuator/sensor non-linearities and uncertain
dynamics in the wearable robotic system [12, 34, 35, 37, 40, 41, 45, 286, 290] make developing a
completely accurate model for all use cases highly unlikely. This limits the practicality of these
advanced control system techniques when used for upper-limb wearable devices, which has been a
contributor to the difficulties observed when translating these methods from research settings to
real-world widespread use with patients.

A potential solution that has shown a lot popularity recently is the use of machine-learning
models incorporated into traditional control system designs to improve upper-limb wearable robotic
devices and address the limitations of control-law-only approaches [23, 24, 26, 290]. The idea
behind this method is that the information provided from the classifier can be used to improve
the controller by estimating additional system information or optimizing parameters (removing
some of the model-dependency from the design problem) [291]. A common example of this, that
is relevant to task weight compensation, is using machine learning techniques (typically Neural
Networks) to estimate model uncertainties and unknown disturbances, and then including this
estimate into the system dynamics model as an adjustment term to correct the calculations made
when executing the chosen control law [34, 35, 50, 286, 287, 289, 291, 292]. Another approach is
to utilize supplemental use-case information supplied by the machine learning model to improve
control system adaptability: incorporating the classification of movement state [48] or payload
estimation [32, 288] into the dynamics model directly to provide estimated values for otherwise
hard-to-assume measurements. These techniques however, do not fully address the modelling
difficulties present in wearable systems. Even though the estimation of unknown parameters using
machine learning can correct partially incomplete models, it still necessitates the use of a base
model of some kind when building the control law (so that the classifier has something in which
to compensate). Likewise, the systems that employed machine learning techniques to provide
supplemental information still utilized system models alongside the classifier models to execute
their control law. These approaches did not completely remove the need for a model in the control
system design, which still leaves the device open to the complications discussed above.
One alternate approach to improving the adaptability of wearable devices to changes in task
weight, without being affected as harshly by the drawbacks listed above, is to find a way to
utilize simpler conventional (i.e., model free) control system approaches while, at the same time,
finding alternative methods to boost their performance. Using a control system that can be
tuned experimentally without any prior system information (as opposed to a method that requires
detailed models to calculate optimal settings) is an attractive option for wearable devices, where the
ability to model the system is severely limited. Likewise, focusing on a control law technique that
is easier to implement (requiring less intense calculations during execution) will greatly simplify
its transition from computer simulation-based research to actual use in an embedded system on a
wearable mechatronic device. Knowing that the inclusion of machine learning models has benefited
other control systems previously, it stands to reason that the integration of EEG–EMG fusion-based
task weight classifiers into a simplified control approach may improve the ability of upper-limb
wearable robotic devices to adapt to changing external loads during use (while providing a method
that is easier to implement). The previous projects presented in Chapters 4 to 6 have already
developed multiple classifiers designed to predict task weight using EEG–EMG fusion, hence the
next step of development (presented in this chapter) is to determine a method of integrating
these machine learning models into a wearable device control system. This task is the objective
of the project described here: to develop a method of using EEG–EMG fusion-based task weight
classifiers to improve the adaptability of a wearable upper-limb mechatronic brace control system to
changes in external load. To achieve this goal, a novel control system was conceived named a Task
Weight Selective Controller (TWSC). The high-level concept behind this control system technique
is to utilize multiple controllers based on the same simple controller design, with each optimized
experimentally for one specific task weight (the same number of controllers as potential class
prediction outputs from the EEG–EMG fusion task weight classifier). Based on the current task
weight prediction supplied by the classification model, the control system will select the appropriate
controller, tuned specifically for that weight level, and utilize it to drive the device. If the predicted
task weight changes at any point, the control system will switch between the multiple controllers,
as needed, selecting whichever one is optimized for the current external load. With this approach,
the actual control law implemented can be kept simple, as traditional controller designs (such
as Proportional Integral Derivative control, discussed in more detail in Section 7.4) can be used
on a moment-to-moment basis, simplifying implementation; however, the control system still has
the capability to adapt to task weight changes through the incorporation of the machine-learning
classifier to select the appropriate controller. The use of classification models for task weight
prediction, as well as the implementation of simpler controller techniques, prevent the need for a
complex system model when executing the TWSC, providing an advantage over other techniques
based on adaptive/robust control and physiology-based modelling/sensing. Given the difficulty of
7.2 Project Methodology 180
modelling all aspects of a wearable system, and the limitations imposed by hardware constraints
of these devices, the TWSC presents the possibility of an alternative solution that seeks to balance
control law complexity and adaptable performance. Therefore, the focus of the project described
here was the design, implementation, and evaluation of a TWSC into the control system of a
wearable mechatronic upper-limb brace for assistance/rehabilitation; to both develop/document
the methodology used to create this system and to provide initial testing results to inform future
designs. This work serves to act as a starting-point for the use of a TWSC in wearable robotic
devices, and contributes key insights into its performance/limitations that will be beneficial for
future projects looking to further its development.
7.2 Project Methodology
To facilitate an evaluation of the proposed TWSC, a platform in which to implement this novel
controller first needed to be developed. Doing so provides a method to compare system perfor-
mance when using the TWSC against traditional control methods, through the utilization of both
approaches within the developed platform. It was decided that an evaluation of the TWSC would
first be done in an offline simulator that models the behaviour of an upper-limb mechatronic brace.
This simulation study will provide preliminary performance data showing how a TWSC may im-
pact the accuracy and usability of an upper-limb device, which can be used during future work to
inform the next steps taken during continued development.
As a first step towards the investigation of this new control approach, the use of a simulation-
based experiment makes logical sense over other methods (such as the implementation of a TWSC
into an actual wearable mechatronic device). While data obtained from an experiment performed
using real-world hardware would provide the strongest demonstration of TWSC performance,
such an experiment introduces many barriers to its completion. Development of an upper-limb
mechatronic brace, as well as the implementation of a TWSC within the embedded system of the
device, would be time consuming and costly; undertaking the prototyping of these devices is often
a project unto itself. Pre-existing devices could potentially be retrofitted to allow for this new
control approach to be integrated into their embedded system (skipping prototype development);
7.2 Project Methodology 181
however, the few devices currently available commercially are still rather costly and would also
require development time anyway for hardware modifications, if such a thing is even possible
(depending on the level of configurability provided by the manufacturer). Utilizing a simulator in
the initial stages of the project will allow for the feasibility of the concept to be evaluated before
investing time/resources into a physical prototype. It may also help reduce the lead time of future
prototype development, as various design choices regarding the control system can be investigated
beforehand using the simulator, allowing only the best concepts to make it through to the prototype
phase (reducing the amount of design work needed using real-world hardware). Also, there are
safety related reasons for starting development using a simulation-based platform. The nature of
wearable mechatronic rehabilitation and assistance devices means that their evaluation ultimately
needs to be performed using human participants, which can present many risks (such as injury
or discomfort) when allowing users to interact with new prototypes. Demonstrating feasibility
(and more importantly, safe operation) of novel systems in a simulated environment first will
allow the progression towards testing protocols that include human participants to be performed
with a higher level of confidence. The use of model-based simulations has also been a frequently
demonstrated technique in the literature for developing wearable upper-limb mechatronic systems
[233, 234, 293], showing that the use of this method has previously demonstrated merits and is
accepted by the larger scientific community involved in wearable upper-limb mechatronic system
development. These reasons provide a strong argument for the use of a simulator, which is why
this approach was adopted for the project outlined in this chapter.
Another justification for the use of an offline simulator as a primary evaluation platform are the
challenges that were introduced by the COVID-19 pandemic. As many already know, the pandemic
necessitated the closure of public spaces to prevent the spread of COVID-19, which meant that
access to laboratory equipment was limited (making the construction of new prototypes difficult).
Focusing development efforts on a simulator instead allowed the project to be more flexible to
the changing working conditions brought on by public-health related lockdowns. The COVID-19
pandemic also meant that the collection of new human participant data would not be a possible
during this project. Work had already begun on the development of a new EEG/EMG signal
collection protocol (including the acquisition of new bioelectrical signal measurement hardware
7.3 Upper-Limb Brace Simulator Development 182
and the development of a wireless mechatronic elbow motion tracking device) when the pandemic
began, which had to be cancelled given the situation. Thus, the offline simulation experiment
described here was performed using the same EEG/EMG dataset outlined in previous chapters.
While this dataset was not initially collected with the intention of using it for the development of
an upper-limb mechatronic brace simulator, an effort has been made to use the resources available
to implement the brace model appropriately. Any assumptions made to address gaps in the dataset
have been clearly explained in the sections below and are accompanied with a suitable justification.
The next several sections will explain the methods employed for this project to investigate the
performance of the novel TWSC. The following section outlines the development work done to
create the upper-limb mechatronic brace simulator: explaining the mathematical model used, its
implementation in Simulink, and the metrics used to describe the performance of the upper-limb
brace. The next section discusses the implementation of the TWSC into the simulator model and,
finally, the last methods-focused section will explain the evaluation procedure used during the
experiments performed to quantify TWSC system behaviour.
7.3 Upper-Limb Brace Simulator Development
The first stage of the TWSC project was to develop a simulator that can replicate the behaviour
of an upper-limb mechatronic brace. This was used for an evaluation of TWSC performance
to assess its feasibility as a new control system technique. The basis of the simulator was a
mathematical model that attempted to capture the physics of the mechanical/electrical properties
of the mechatronic brace (namely the actuator, as this has the largest impact on system behaviour),
as well as the properties of the human subject operating the device. To implement the simulator,
MATLAB 2020a, along with Simulink, were used to build the model, automate data processing,
and run the time-based simulation experiments. Note, that some minor calculations were also
performed using Excel, for simplicity.
To begin developing a model of an upper-limb brace, the design of said device first needed to be
decided upon, so that logical assumptions could be made regarding its parameters. Even though
there were no plans to actually build a prototype of the simulated system as part of this project,
having an example device to use as an assumed starting point provided a realistic foundation
for the simulator when building the system model. It was decided that the simulator for this
experiment would base the model on a wearable mechatronic upper-limb prototype previously
developed in our lab [221]. The device in question is an elbow brace meant to support the joint
during rehabilitation. It offers one degree-of-freedom motion, to allow the user to perform elbow
flexion–extension, and is driven using a single DC motor coupled to a gearbox. The brace is secured
to the user by cuffs with straps that wrap around the forearm (at approximately the mid point)
and the upper arm (at approximately the middle of the biceps). See Figure 7.1 for a CAD model
and a picture of the wearable mechatronic brace prototype used as the basis for the hypothetical
device modelled in the simulator.
(a) CAD Model (b) Prototype
Figure 7.1: A CAD model (a) and photograph (b) of the wearable mechatronic upper-limb brace
prototype previously developed in our lab [221]. The design of this device was used as
the basis for the hypothetical upper-limb brace modelled in the simulator.
7.3.1 General System Torques Model
Knowing the general design of the brace to be modelled, the next step was to begin identifying
the key physical phenomenon that needed to be represented mathematically. To achieve this, a
basic free-body diagram of the brace was devised; the purpose of this was to highlight the different
forces applied to the brace during operation to determine what should be included in the model.
The free-body diagram of the upper-limb brace being worn by a human user can be seen in Figure
7.2.
Figure 7.2: A free-body diagram showing the torques acting on the upper-limb brace while being
operated by a human user. The torques τDb and τA are caused by the weight of
the dumbbell being held (FDb ) and the combined weight of the subject’s hand and
forearm (FA ), respectively. The measurements dDb and dA indicate the distances from
the actuator of the brace at which these forces are applied. The torque τM is applied
by the brace to counteract the other torques, holding the brace in a stable position.
As a basic starting point, the diagram was developed showing the static forces present on the
device if it were supporting the user’s arm to hold a specific position without moving. There are
three torques acting on the brace based on these static forces: the torque caused by the weight of
the user’s arm (τA ), the torque caused by the weight of the load (i.e., dumbbell) in the subject’s
hand (τDb ), and the torque caused by the DC motor (τM ), which counteracts the previous two to
hold the user arm in a stable position. Knowing that the sum of static forces must equal 0, the
equation for the sum of these torques can be written as follows:
X
τ = τDb + τA − τM = 0, (7.1)
where anticlockwise is assumed to be the positive direction of rotation. Solving for motor torque,
Equation 7.1 can be rewritten as
τM = τDb + τA , (7.2)
to provide an equation for describing the amount of torque that the DC motor must output to
successfully hold the subject’s arm in a static position.
Note that some assumptions have been applied to this model. First, the weight of the brace
itself was not considered when determining the torques present in the system. Normally, the
torque applied by the DC motor would also need to counteract the weight of the mechanical
components used by the brace (such as the frame or the cuffs); however, since the simulator
modelled a hypothetical device that does not actually exist, there is no logical way to assume what
that weight would be in a non-speculative manner (especially given the fact that the previously
developed brace prototype used as the inspiration for this simulator was not accessible at the time
of the project). It was decided instead to simply remove this variable for the sake of simplicity
(with the understanding that this limitation will result in a slight underestimation of motor torque
from the simulator output, which may need to be accounted for in future work).
The second assumption applied is that the subject is not outputting any muscle force themselves
to hold the static elbow position: all of the counteracting torque is being applied by the motor. This
presents a “worst case” scenario for the brace in regards to a hypothetical assistance task, where
the patient has lost complete use of their arm and all support must come from the DC motor.
A patient who has partial use of their arm may still be able to contribute some counteracting
torque (depending on the nature of their injury), lowering the amount of force required from
the DC motor. However, the brace should be designed to handle the most severe injury cases
where the user is incapable of providing this assistance, which is why this assumption was used
for the simulator. Part of this assumption though is that the patient is still able to generate some
level of EMG signal (allowing for EEG–EMG fusion to still be employed by the control system).
While this sounds paradoxical initially, it is not without some real-world merit. Such a situation
would be an example of an MSD patient with a severely weakened muscle, one that will still try
to contract when an EMG signal is supplied by the nervous system, but does not have enough
strength to actually cause their arm to move. In this example the brace would need to counteract
the full weight of the user’s arm, as no meaningful torque is being provided by their muscle to
provide partial assistance. Regarding the actual implementation of this equation, there was also
no feasible way to realistically model human-generated muscle torques using the dataset available
for simulator development. No muscle force and/or elbow joint torque information was collected
during EEG/EMG signal recording (for example, using a dynamometer on the participant’s elbow
joint while they moved); therefore, attempting to incorporate this into the simulator with the
information available would be challenging and likely inaccurate. Instead, it was decided that the
simulator would focus on the use-case scenario where the patient has lost full use of their arm and
that the brace must apply all support needed.
Now that the main torques were identified, a method for calculating the torque caused by the
weight of the subject’s arm and the torque caused by the held load (where that load was a 3 lbs
or 5 lbs dumbbell) needed to be determined. Knowing that torque is simplify force multiplied by
a distance, Equation 7.2 can be expanded to
τM = FDb dDb + FA dA , (7.3)
where FDb and FA are the forces caused by the weight of the load (dumbbell) and arm (hand
and forearm combined), respectively, and dDb and dA are the distance from the elbow joint to the
point where each respective force is applied (their centre of mass). Note that the elbow joint was
chosen to be the point of reference for the distance measurements used during force calculation,
since that is where the motor torque is applied, and since it is a distinct physiological feature on
the body (making it easy to locate when taking measurements). Also, it was chosen to draw the
free-body diagram (Figure 7.2) specifically with the user’s forearm parallel to the ground because
this is the joint position that will result in the maximum amount of torque on the motor, due to
the moment arm of each force being at furthest possible distance from the elbow. It also simplifies
torque calculations by not requiring the 2D components of the force vectors to be determined as
a function of joint angle, since all force vectors are applied only in the vertical direction.
To complete the expansion of the motor torque equation, the forces were further broken down
using the definition of force as dictated by Newton’s Second Law (force is equal to mass multiplied
by acceleration), providing the following equation:

τM = g mDb dDb + mA dA , (7.4)
where mDb and mA are the mass of the dumbbell and the subject’s arm, respectively, and g is
acceleration due to gravity (approximated as 9.81 m/s2 in this work).
With the mass of the load (i.e., each of the dumbbells) known, the final unknowns that were
left to solve were the mass of the subject’s arm (mA ) and the distances of each applied force (dDb
and dA ). All of these factors are related to the anatomy of the subjects themselves, with each
person having different values depending on their size/body type. To estimate these values, anthro-
pometric data approximations, obtained from biomechanics studies of human body proportions
[294] found in the literature, were employed, along with basic body measurements of the study
participants taken during the data collection sessions. Using the anthropometric data, one can ap-
proximate the Center of Mass (COM) of a body segment as a percentage of body segment length.
Likewise, the mass of a body segment can be expressed as a percentage of total bodyweight. For
calculating torque, the COM can be considered the point at which the force caused by the weight
of that body segment acts, thus the anthropometric approximations can be used to solve for the
rest of the unknowns in the static motor torque equation (Equation 7.4). This approach of using
anthropometric data to estimate human subject measurements has been successfully demonstrated
before in an upper-limb exoskeleton simulator development study [293], hence it was decided that
this technique was suitable for the work performed here. The anthropometric data percentages
used for during calculation are summarized in Table 7.1.
Before having their EEG/EMG signals recorded, study participants had their hand length (lh )
and forearm length (lf ) measured, and were asked to self-report their bodyweight (mb ). Using
these measurements, the force distances (dDb and dA ) can be calculated. See Figure 7.3 for a
visual representation of the lengths of the hand/forearm body segments, and how this relates to
the COM. The distance for arm weight was calculated simply using the anthropometric arm COM
Table 7.1: Anthropometric data used to approximate body segment mass and COM distance,
based on a percentage of total segment mass/length, respectively. The percentage values
listed here are taken from a larger table of Anthropometric data found in [294], with
only the values relevant to the calculations performed for this work being summarized
here.
Body Segment(s) Measurement Symbol Percentage Value

Hand COM COMh 0.506
Forearm and Hand COM COMA 0.682
Forearm and Hand Segment Mass SMA 0.022
approximation and the subject’s arm length, expressed by the following equation:
dA = (lh + lf )COMA . (7.5)
Figure 7.3: A diagram showing the COM locations for the combined hand/forearm (dA ) and the
hand dDB , measured from the subject’s elbow. These are the same distances shown in
Figure 7.2. The lengths lh and lf are the recorded measurements of the subject’s hand
and forearm, respectively. The distance lh COMh indicates the location of the centre
of mass of the hand, measured from the subject’s wrist.
To calculate the load distance, an assumption first needed to be made. It is difficult to know
the exact point at which the subject was holding the dumbbell in their hand; therefore, it was
decided to simplify this calculation in the model. It was assumed that the dumbbell was being
held at the COM of the hand during data collection. While this may not be exactly true in all
cases, it should provide a close enough estimate of where the load force was applied, and it presents
a consistent method for calculating this distance for all subjects (given the data available). The
equation for calculating the distance of the load force is given by
dDb = lh COMh + lf . (7.6)
Substituting Equations 7.5 and 7.6, as well as the calculation of arm segment mass using the
anthropometric approximation percentage multiplied by bodyweight, into Equation 7.4 resulted
in the final equation for required static motor torque:

τM = g mDb (lh COMh + lf ) + (mb SMA )(lh + lf )COMA , (7.7)
where mDb (mass of the 3 lbs/5 lbs dumbbell, or 0 lbs if the subject was empty handed, depending
on the trial), COMA , COMh , and SMA (anthropometric approximation percentages) are known
and lh , lf , and mb are subject body measurements taken during data collection. Substituting the
numerical value of known constants, in this case the anthropometric approximation percentages
obtained from Table 7.1, into Equation 7.7 to improve readability of the formula, resulted in the
following equation:

τM = g mDb (0.506lh + lf ) + 0.022mb · 0.682(lh + lf ) . (7.8)
Now that an equation was determined to calculate the amount of static motor torque required
to hold each subject’s arm at a 90° angle, the body measurement values for each subject, and the
mass of the dumbbells, were used to calculate the required motor torque for all subjects and load
weights. Then the maximum torque value from these results was determined. The purpose of
this was to have a starting point when determining the requirements needed for motor selection.
The simulator, while hypothetical, still needs to model realistic actuator properties, and to do
this, a real-world DC motor was chosen as the actuator used for the simulated upper-limb brace
being modelled. This provided a feasible reference to utilize when determining motor parameters.
While static torque will be much lower than the actually required dynamic torque used to move
the subject’s arm, static torque does provide at least a rough starting estimate of how powerful
the DC motor must be (motors rated below the static torque can be confidentially rejected from
selection, as there will be no way it could produce enough torque to move the subject’s arm).
This required torque was calculated for all subjects to ensure that the simulator developed could
be used to model the data collected from all 32 study participants, as they presented a range of
different body types (which resulted in a range of required torque values). The maximum static
motor torque requirement obtained from the results of all subjects was
τM,max = 16.84 Nm ≈ 17 Nm,
which was used to inform the selection of an actuator in later stages of the simulator development
process.
7.3.2 Brace Actuator Model
Having identified motor torque as the controllable output supplied by the brace to move the
subject’s arm, a mathematical model then needed to be developed of the actuator used to drive
the device. As mentioned previously, a DC motor (with an attached gearhead) was chosen as
the actuator type for the simulated brace, as it matches the design of the previously developed
prototype brace [221] used as the foundation for the simulator. Since a DC motor is inherently
an electromechanical system, it will require the coupling of two mathematical models to fully
explain its behaviour: one to define its electrical characteristics and one to define its physical
(i.e., mechanical) characteristics. The mechanical model also needs to incorporate the effect of the
gearhead on system actuation, and the coupled electrical–mechanical model should also consider
non-linearities present in the motor/gearhead response, since these can have a significant impact on
control performance. The following subsections show the full mathematical model used to simulate
the actuator in the upper-limb brace simulator. For a detailed explanation of each component in the
model, and how their various aspects reflect the physical behaviour of the actuator, see Appendix
A.1.
7.3.2.1 Modelling the DC Motor and Gearhead
To represent the properties of the combined DC motor/gearhead actuator, an equivalent circuit

diagram was used to describe the electrical model (Figure 7.4(a)), and an equivalent mechanical
system diagram was used to describe the physical model (Figure 7.4(b)). The equivalent circuit
models input voltage supplied to the motor (v(t)) and the DC current (i(t)) flowing through the
motor armature (i.e., the internal coils), which generates the magnetic field used to spin the rotor.
The equivalent circuit also includes the effect of back electromotive force (back emf) voltage with
respect to time (e(t)). Back emf is the voltage applied backwards into the circuit by the motor, as
the rotor spinning with the angular velocity θ̇M (t) will act like a generator and induce a voltage
within the armature coils (related proportionally by the back emf constant Ke ). The mechanical
system diagram models the rotational motion of the rotor, considering its inertia (JM ) and internal
friction (bM ) as it spins. This simple method of modelling a DC motor is a commonly used approach
that has been well explored in the literature [293, 295–310].
The mechanical system diagram of the DC motor was extended to include the effect of the
gearhead on the system, modelling the torque reduction of the gears (n) determined by the ratio
of the number of teeth on the input and output gear (N1 and N2 , respectively), as well as the losses
in the system due to the mechanical behaviour of the gearhead: the inertia of the gears (JG ), which
requires torque to move and the losses due to friction (represented with the gearhead coefficient of
viscous friction, bG ) [295, 297, 298, 302, 304]. The model also includes the inertia of an external
load attached to the gearhead rotor (JL ), which represents the object that the actuator is trying
to move (in this case the subject’s arm and the dumbbell) with the output gearhead torque (τG ).
Using Kirchhoff’s Voltage Law on the equivalent circuit (Figure A.1(a)), and Using Newton’s
equations of motion and d’Alembert’s principle on the mechanical model (Figure A.1(b)), two
differential equations were obtained to mathematically describe the behaviour of the actuator. To
combine the electrical and mechanical differential equations, a coupling equation was used that
defines a proportional relationship between armature current and motor torque (τM (t)); quantified
through the use of a motor-specific parameter called the torque constant (Kt ). Note that the
value of the torque constant will equal the back emf constant if a consistent set of units is used
(in this case SI units) [295, 307], reducing the number of parameters required. After applying the
Laplace transform to convert the differential equations from time domain equations (using t) into
Laplace domain equations (using s, the Laplace variable), the actuator system was represented
using a block diagram where electrical and mechanical components of the model are described
(a) Electrical model
(b) Mechanical model
Figure 7.4: An equivalent circuit schematic (a) and equivalent mechanical system diagram (b)
describing the electrical and mechanical properties, respectively, of a DC motor with
an attached gearhead. In the electrical model (a), v(t) is the input armature voltage,
i(t) is the armature current, e(t) is the back electromotive force voltage (calculated
using the motor back emf constant, Ke , and the rotor angular velocity, θ̇M (t)), R is the
armature resistance, and L is the armature inductance. In the part of the mechanical
model (b) representing the DC motor, τM (t) is the output motor torque, JM is the
rotor inertia, and bM is the coefficient of viscous friction of the motor. In the part of
the mechanical model (b) representing the gearhead, N1 and N2 are the number of
teeth on the input and output gear, respectively, JG is inertia of the gears, bG is the
gearhead coefficient of viscous friction, θ̇G is the output velocity of the gearhead, τG is
the output torque of the gearhead, and JL is the inertia of the external load acting on
the gearhead.
using different transfer function blocks. The equivalent block diagram representing the combined
DC motor/gearhead actuator system can be seen in Figure 7.5.
Figure 7.5: A system block diagram outlining a Laplace domain representation of a DC motor
combined with a gearhead. The block diagram provides an equivalent description
of the model shown in Figure 7.4. Note that capital letters were used to represent
all previously discussed time-domain variables, indicating a transformation into the
Laplace domain (as is convention).
7.3.2.2 Modelling Actuator Non-linearities
Now that the DC motor and gearhead were modelled mathematically, the final step in simulating
the actuator response was to incorporate non-linear behaviour into the system. Up until this
point, the model of the actuator represented only the “ideal” system behaviour, where its output
conforms perfectly to the described mechanical/electrical system equations and can provide an
infinite range of torque/speed values for an infinite range of input voltage values. While such ideal
system behaviour is possible in a hypothetical mathematical model, it does not properly showcase
how a chosen actuator will behave in a real-world system. To demonstrate a more realistic actuator
within the simulator, non-linearities were included into the model. Actuator non-linearities are
caused by physical limitations and manufacturing imperfections found in the motor/gearhead,
which cause their output to behave in a non-linear manner outside of what is defined by the model
equations. Non-linear behaviour can cause particular issues for a control system and can have
drastic consequences on the performance of the device being controlled [295, 305, 311, 312]. Thus,
it is important to include these factors into the simulator so it can more closely match how a real
system would behave, allowing for a stronger evaluation of the proposed TWSC.
The first non-linearity that was considered is known as backlash, and is a consequence of
manufacturing tolerances within the gearhead. When gear teeth mesh together, there will be
some amount of space between them and this spacing will cause a small dead zone to occur when
the gearhead changes its direction of rotation [295, 305]. The teeth of the input gear (which are
pressed against the teeth of the output gear on the side that causes its current rotational direction)
will have to travel through the small air gap before making contact with the backwards rotating
edge of the teeth on the output gear. This means that for the brief moment when the teeth
change over, no rotation will occur on the gearbox output shaft, despite rotation occurring on the
DC motor input shaft (all while the controller continues to assume that the system is actually
moving). Backlash can be modelled as deadband on the output angular speed or position value
(θ̇G or θG , respectively). The width of the deadband caused by gear backlash (defined here as θBkl )
is a parameter that is usually determined by the manufacturer and is typically reported on their
supplied datasheet. The specific backslash value used for the simulator will be discussed below in
Section 7.3.3.
The second type on non-linearity included in the simulator was saturation. Saturation rep-
resents the physical limits of a real-world actuator and is represented by a cutting-off of output
values above/below the specified limits [295, 311, 312]. Constraints dictated by the mechanical
and electrical design of the DC motor/gearhead will be such that the actuator can only operate
safely within a specified range of values (and exceeding these limits could damage the actuator
and/or potentially cause it to fail entirely). These limits need to be considered, since the control
system must be designed so that the actuator operates appropriately within a safe range to avoid
any discomfort/injury occurring for the human user of the brace. Multiple saturation limits were
added into the simulated system, based on potential failure points of the actuator. The first sat-
uration implemented was applied to input voltage. Any DC motor will be designed such that its
input voltage can only be within a specific range, otherwise it may overheat and/or damage the
armature coils. As such, a voltage limit, vmax , was placed on the system input voltage v(t) to
ensure that this range was maintained. Conveniently, this limit also played a role in limiting the
effects of integral windup, which is discussed further in Section 7.4. Note, that the defined limit
value (vmax ) is actually describing a range of input voltages from −vmax to +vmax . Due to the
physics by which DC motors operate, negative input voltage will result in the same output speed
as a positive voltage, but with the rotor spinning in the opposite direction. For simplicity, the
saturation value is described using only the absolute value, with the implicit understanding that,
in practice, the actual limit spans the positive/negative range of said value. All actuator-based
saturation limits discussed here follow this convention. In a similar manner to the voltage limit,
there is also a current limit, imax , that must be maintained for similar reasons (damage to the
armature coils can occur). Thus, a current saturation was also included in the model. Finally, it is
important to also consider the mechanical limitations of the motor. Based on the operating princi-
ples of DC motors, there is a maximum amount of torque they can output after which they become
unable to move the rotor (the force generated by the maximum energization of the armature coils
is less than the external load applied to the rotor), referred to as the stall torque. Even before
this limit, many manufacturers will advise a smaller torque limit to ensure motor safety (avoiding
overheating or other mechanical damage) and improve the usable lifetime of the device. For these
reasons, a torque saturation, τmax , was included in the model. All three of the aforementioned
saturation limits are specific to the chosen actuator and are dictated by the manufacturer and
therefore will be discussed further in Section 7.3.3.
Finally, there was one more saturation non-linearity included into the simulator, with this limit
having the distinction of not being related specifically to the actuator parameters, but instead to
the desired operating behaviour of the upper-limb brace itself. A very important factor to consider
when developing mechatronic rehabilitation/assistive devices is safety (as would be expected for
all medical devices). It is crucial that the brace does not injure the patient in any way, and one
key aspect of this for a wearable system is ensuring that its range of motion is limited [12, 14]. If
the device were to extend the user’s elbow joint beyond its comfortable range of motion, it could
lead to unwanted discomfort or, in extreme cases, severe injuries. Typically, such devices would
have mechanical stops in place to physically restrict the actuator from moving beyond those limits;
however, running into the mechanical stops could potentially damage the brace, hence it should
not be used regularly and instead be considered as a last resort. To avoid this, software limits
are usually also implemented into the control system as a first line of defence [12, 14], which will
command the actuator to stop moving before a collision with the mechanical limits occurs. As such,
it is important that these position limits are incorporated in the model as well, since saturation
on the system outputs can significantly affect controller behaviour [295, 311, 312] (making this
behaviour beneficial to include as part of the evaluation). A saturation on angular position, θmax ,
was added to the model to limit the final brace output angle (θout (t)) to be between 0◦ and 150◦,
defined thusly:
θmax = 0◦, 150◦ ⇒ θout (t) ∈ R | 0◦ ≤ θout (t) ≤ 150◦ .

(7.9)
These limits were chosen for θmax since 0◦ to 150◦ was the range of motion assumed for elbow
flexion–extension motion performed by the study participants during data collection [232–234].
In the convention used for the simulator, 0◦ was defined as the elbow being fully extended and
150◦ was defined as the elbow being fully flexed. While range of motion will vary from subject
to subject (due to different biological factors like body type, flexibility, etc.), this approximation
was used to define clear limits for the purpose of the simulator, since actual joint range of motion
measurements for the study participants was not available during development.
After outlining which non-linearities to include in the actuator model, the system block dia-
gram was updated to include the previously discussed deadband/saturation blocks. The updated
equivalent block diagram representing the combined DC motor/gearhead actuator system with
non-linear behaviour modelled can be seen in Figure 7.6.
Figure 7.6: A system block diagram for a combined DC motor/gearhead with non-linear actuator
behaviour included in the model. This system block diagram was updated from Figure
7.5 to include the relevant non-linearities, namely saturation of motor inputs/outputs
(voltage, current, and torque), backlash caused by the gears, and limits on brace
output position applied for safety. Note that capital letters were used to represent
all previously discussed time-domain variables, indicating a transformation into the
Laplace domain.
7.3.3 Actuator Selection
Once the mathematical model of the combined DC motor/gearhead actuator was outlined, the
next step taken to develop the upper-limb brace simulator was to determine a method of defining
actual numerical values to use for the system actuator parameters (which can be seen in Figure
7.6). It is often useful to first derive a model in general terms, since it can highlight the individual
contribution of various aspects of the physical system more clearly, and will allow the simulator to
be generalizable to other mechatronic devices (if desired for future work). However, to be able to
actually evaluate the previously shown equations, and to be able calculate numerical results using
the simulator, the various generalized actuator parameters related to the DC motor/gearbox must
be specified. The values chosen for these parameters will have a large effect on the performance
and behaviour of the simulator (since they determine how the hypothetical actuator will respond to
inputs). Thus, they should be chosen methodically, to ensure that the simulator acts in a realistic
manner.
To make sure this was the case, it was decided to select a real-world combined DC motor/gear-
head actuator that is available for purchase, and use data provided by the manufacturer to assign
the actuator parameter values of the simulator. The idea was to select a DC motor/gearbox that
could hypothetically be used to build the upper-limb mechatronic brace being modelled in the
simulator (assuming it was in fact a real physical device being developed). Even though no pro-
totype was actually being assembled for this project, working off this assumption ensured that
the response of the simulated brace was still within a reasonable range and not too far off from
reality (although it is important to remember that there is no such thing as a perfect model,
due to necessary assumptions made during development, therefore the response, while close, will
never be perfect). A true actuator selection process is a very thorough and lengthy procedure that
considers many factors of the design and optimizes the selection process based on the objectives/-
constrains of the system. For the purposes of the work done here, this would have been excessive
and not worth the time/resources it would have consumed. The main objective of actuator selec-
tion for the simulator was merely to find an DC motor/gearhead that would properly approximate
the behaviour of a real system, while at the same time provide sufficient information from the
manufacturer so that it can actually be modelled. There were no attempts made to claim that
the actuator chosen is the “best” choice for a real-world upper-limb brace prototype, since such
arguments about prototype optimization are outside the scope of this project.
As with many design choices relating the hypothetical brace being modelled, it was decided to
refer back to the prototype previously developed in the lab as a starting point, since the simulator
brace was developed to mimic this system closely. That device utilized a maxon Brushless DC
(BLDC) motor/gearhead combination that was capable of delivering 6 Nm of torque [221]. Based
on the initial torque calculations outlined in Section 7.3.1, the maximum amount of torque needed
to hold the user’s arm in a static position (for the worst-case scenario, using the highest value
obtained across all subjects) was 17 Nm. Based on this requirement, it was obvious that the
previously used actuator was not suitable for the application here, thus an alternative solution
needed to be found. To narrow the scope of the search, and simplify the selection process (as
well as to keep the design of the “new” brace similar to the previous prototype), it was decided
to continue using a maxon actuator like before; however, this time choosing a more powerful
brushless DC motor/gearhead. Using 17 Nm as the base selection criteria, the maxon catalogue
of motors/gearheads was reviewed to find an actuator strong enough to move the required load.
It should be noted that 17 Nm is only the static torque requirement, and that the dynamic-torque
needs of the system will be significantly higher when moving the user’s arm. The use static torque
as the selection criteria was done mainly to provide a rough estimate of the needs of the system,
providing a hard cut-off to reject obviously unsuitable candidates (as was done above with the
old actuator), since it is simple enough to pre-calculate static torque using the basic equations
outlined in Section 7.3.1. Dynamic torque is much harder to pre-calculate by hand and, in fact,
this is essentially why this simulator development was required in the first place: to provide a
platform to feasibly calculate dynamic torque and motion values through the use of Simulink.
Thus static torque needed to be utilized for actuator selection instead, while applying a generous
factor of safety when considering the output torque of various actuators, to make sure one is chosen
that has the potential to meet the higher requirements.
With this consideration in mind, a new maxon BLDC motor and compatible gearhead were
chosen when assigning realistic values to the various actuator parameters outlined in the math-
ematical model presented in Section 7.3.2. The motor selected was the EC 90 flat 90 mm,
brushless, 260 watt (part number: 500266) and the gearhead chosen was the Planetary Gearhead
GP 52 C 52 mm, 4.0–30.0 Nm (part number: 223089). Relevant specifications for the chosen
BLDC motor and gearhead, taken from the manufacturer’s datasheets for both products, can be
found in Table 7.2 and 7.3, respectively.
Using the datasheet specifications, it can be shown that the selected brushless DC motor/gear-
head combination meets the actuator selection criteria to surpass the static torque requirement
Table 7.2: Relevant specifications for the chosen BLDC motor, taken from the manufacturer sup-
plied datasheet [313]. When necessary, parameters were converted from their reported
values to SI units.
maxon EC 90 flat (part number: 500266)

Parameter Symbol Value
Nominal Voltage vnom 30 V
Nominal Torque (max. continuous) τM,nom 988 mNm = 0.988 Nm
Nominal Current (max. continuous) inom 7.06 A
No Load Speed θ̇NL 2080 rpm ≈ 217.817 rad/s
No Load Current iNL 490 mA = 0.490 A
Stall Torque τstall 14600 mNm = 14.6 Nm
Stall Current istall 107 A
Terminal Resistance phase to phase Rphase 0.28 Ω
Terminal Inductance phase to phase Lphase 0.369 mH = 0.369×10−3 H
Torque Constant Kt 136 mNm/A = 0.136 Nm/A
Rotor Inertia JM 5060 gcm2 = 0.506×10−3 kgm2
Radial Play (backlash) - Preloaded (none)
Table 7.3: Relevant specifications for the chosen gearhead, taken from the manufacturer supplied
datasheet [314]. When necessary, parameters were converted from their reported values
to SI units.
maxon Planetary Gearhead GP 52 C (part number: 223089)

Reduction - 43:1
Absolute Reduction n 343/8 = 42.875
Max. Continuous Torque (nominal) τG,nom 30 Nm
Max. Intermittent Torque τG,max 45 Nm
Max. Continuous Input Speed (nominal) θ̇G,nom 6000 rpm ≈ 628.3185 rad/s
Mass Inertia JG 17.3 gcm2 = 1.730×10−6 kgm2
Max. Efficiency ηG 75 %
Average Backlash No Load θBkl ◦
1 ≈ 0.017453 rad
of 17 Nm. Using Equation A.11 and substituting the nominal motor torque value from Table 7.2
and the absolute reduction value from Table 7.3 (a more accurate calculation of the gear reduction
provided by the manufacturer), the nominal output toque of the chosen actuator can be calculated
as
τactuator = (42.875)(0.988 Nm) = 42.3605 Nm,
which is more than double the maximum required static torque of 17 Nm (providing a large range
of excess torque available to address the potentially higher dynamic-torque requirements). The
calculated output torque is also below the maximum intermittent torque (τG,max ) rating of 45
Nm for the chosen gearhead (the absolute maximum peak torque the gearhead can output before
mechanical damage is likely to occur), confirming that the chosen DC motor/gearhead combination
is suitable as each component would be operating within their manufacturer specified limits. Based
on this result, it was concluded that the chosen actuator will provide a suitable reference to use
when defining the DC motor/gearbox parameters in the mathematical model outlined for the
upper-limb brace simulator.
Before determining the actuator model parameters, one important factor about the selected
motor, and how it affects the proposed model, needs to be discussed. This is the fact that the
chosen motor is brushless, which will have consequences on the electrical model of the actuator. In
traditional brushed DC motors, commutation (switching the direction of the magnetic field to facil-
itate a full 360◦ rotation of the rotor) is done through a mechanical connection by the eponymous
brushes [296, 315, 316]; however, in a BLDC motor this action is performed electronically using
a highly controlled sequence of pules applied to a series of separate armature coils placed around
the motor. By turning off and on the coils in a specific pattern, the magnetic poles placed on the
rotor will be pushed into a full 360◦ revolution, spinning the motor at a set speed. Using this
approach has many benefits (for example, removing a potential mechanical failure point caused
by wear in the commutation brushes), but it requires more complex driver circuitry to implement
the digital logic necessary to control the coil powering sequence [296, 315–324].
To fully model all electrical behaviour of a BLDC motor, the equivalent circuit for a DC
motor (i.e., what is shown in Figure A.1(a)) needs to be modified. Instead of representing the
armature coil with an single equivalent resistor (R) and inductor (L) pair, each coil used to
execute the commutation sequence is represented with its own resistor–inductor pair [315–318, 320–
323]. In the case of the motor selected for the simulator, there would be three resistor–inductor
pairs, since the chosen BLDC motor is labelled as three-phase by the manufacturer (which is a
common BLDC motor configuration) [313]. Winding connection methods can vary; however, one
of the most common coil configurations is the “Star” setup (also referred to as a “Wye” or “Y”
connection), where the three resistor–inductor pairs are connected together at one end (forming
a shape similar in appearance to the letter “Y”) [296, 315–318, 320–323, 325]. Comprehensive
BLDC motor models will calculate the current passing through each coil separately, providing a
full implementation of the alternating commutation sequence and modelling the magnetic-field
interactions between the various coils as they are switched on/off, to obtain a more accurate
representation of the torque output of the motor for a given commutation signal [293, 296, 315,
317, 318, 320–324]. While this method will undoubtedly result in more realistic motor behaviour, it
also greatly increases the complexity and computational requirements of the model. On the other
hand, while potentially less comprehensive, there are also BLDC motor models in the literature that
represent the equivalent electrical behaviour of the motor using the traditional simplified (single
armature coil) DC motor model [293, 296, 304, 316, 318, 319, 324], and some work has shown that it
can result in similar torque/speed outputs as the three-phase model [296]. For the purposes of the
upper-limb brace simulator, it was assumed that, when controlling the BLDC motor, commutation
will be handled using a pre-made driver circuit (either purchased from the motor manufacturer
or an alternative supplier). Thus, it was not really necessary to incorporate this aspect into the
simulation model. Likewise, maxon themselves instruct purchasers to not concern themselves with
the internal windings/configuration of their BLDC motors, instead advising them to treat the
motor as a black box and consider the overall input current and output torque/speed behaviour
when selecting which motor to purchase [325]. This may be why their motor datasheet contains
little information on the internal winding setup of the motor, instead providing high-level actuator
performance measures, such as the torque constant. For these reasons, it was decided to continue
using the traditional (single armature coil) DC motor equivalent electrical circuit in the actuator
model for the upper-limb brace simulator.
That being said, effort was made to ensure that the single coil approximation used would still
properly model the chosen BLDC motor when assigning the actuator parameter values in the
simulator. The coil resistance (Rphase ) and inductance (Lphase ) values provided by the manufac-
turer in the datasheet are listed as “terminal...phase to phase” [313], meaning that they are the
measurements for one of the three coils (which are usually assumed to be of equal value [296, 315–
318, 320–324]). For a three-phase BLDC motor, the commutation sequence is implemented in such
a way that two of the three coils are always turned on (i.e., have current flowing through them)
[296, 316, 317, 320, 323, 324]. This means that, if approximating the three Star-connected coils as
a single equivalent armature coil, two times the phase-to-phase resistance and inductance must be
used, according to the following equations [316–318, 320, 323, 324]:
R = 2 · Rphase , (7.10)
L = 2 · Lphase . (7.11)
These new values are referred to as line-to-line (sometime shortened to just line) resistance/induc-
tance, and they measure the effective resistance/inductance acting on the current path through
the motor windings [317, 320, 322, 323]. Substituting the appropriate values from Table 7.2 into
Equations 7.10 and 7.11), the line-to-line resistance and inductance values were evaluated as fol-
lows:
R = 2(0.28 Ω) = 0.56 Ω,
L = 2(0.369 × 10−3 H) = 0.738 × 10−3 H,
which were used in the electrical transfer function block of the DC motor model to approximate
the chosen BLDC motor armature using the equivalent circuit shown in A.1(a).
Regarding the other aspects of the chosen motor/gearhead, when looking at the information
provided by the manufacturer in the datasheets (shown in Tables 7.2 and 7.3), it can be seen
that values for many of the actuator parameters defined in the mathematical model can be pulled
directly from these specifications (with an occasional unit conversion required to ensure all pa-
rameters use SI units). Motor/gearhead inertia (JM and JG , respectively), as well as the torque
constant (Kt ) are all clearly stated in the motor/gearhead datasheets, with their values also re-
stated here in the aforementioned tables. It was decided that the absolute gear reduction value
provided in the gearhead datasheet would be used as the reduction value in the model (n), since
this provides a more accurate measurement of the total effect that the gear ratio has on the system.
The amount of backlash present in the actuator (θBkl ) can also be determined from the gearhead
datasheet directly; since the BLDC motor datasheet explains that the rotor shaft is preloaded
(meaning that it was designed specifically to not have any radial play) it will not contribute any
backlash to the system. Finally, the value used for the saturation on input voltage (vmax ) was also
taken directly from the datasheet. It was decided that the nominal voltage (vnom ) value provided
by the manufacturer would be used for this limit, since it provides a known input for which the
motor can be expected to function properly. As well, all other reported motor parameters were
measured by the manufacturer at this voltage [313], hence ensuring that the motor input stays
within this range will help keep the behaviour of the motor closer to what is described in the
datasheet (make it more predictable in the simulation).
For the torque/current limits of the BLDC motor (τmax and imax , respectively), the decision of
which values to use in the model was less straight forward compared to the previous parameters.
The BLDC motor datasheet lists multiple torque/current specifications that are relevant to the
BLDC motor limits, but each carries with it different consequences for how the motor will behave
under those circumstances. The first set of specifications to consider are the stall torque/current
values (τstall and istall , respectively). Stall torque is the amount of force applied by the motor
when the rotor is stationary, typically because the attached load is too great for the motor to
move. Likewise, stall current simply is the amount of current drawn by the motor to apply this
torque (remembering that DC motor torque and current are proportionally related through the
torque constant, using Equation A.5) [295, 326]. Stall torque represents, essentially, the absolute
maximum amount of torque that the motor can produce and, simultaneously, the maximum load
it can move, as the stationary rotor demonstrates that the strength of the magnetic field generated
by the armature coils is no longer sufficient to generate any rotational movement. Knowing this,
it may seem logical to use the stall torque/current for the saturation values in the model (since
stall represents literally the absolute physical limitations of the BLDC motor); however, this is
likely an unrealistic assumption for how the motor will be used in a real-world system. Running
a motor at stall torque can cause it to overheat (due to the large amount of stall current passing
through the coils), which often damages the motor. For this reason, the manufacturer does not
recommend running the motor under these conditions for long periods of time, to ensure the
life cycle of the device [326]. In the context of a wearable mechatronic device, this behaviour is
even more important to avoid, since an overheated motor could potentially burn the user and
seriously compromise device safety. Thus, stall torque/current was deemed unsuitable for use as
the saturation limits in the model. With this understood, the next specification to consider is
the nominal torque/current (τM,nom and inom , respectively). These values are provided by the
manufacturer to outline the torque/current values in which the temperature of the motor coils will
stay within the allowed range during continuous operation [326]. This essentially means that, at
these values, the motor can run non-stop safely without overheating. Once again, these seem like
potential candidates to use as the saturation limits; however, they present the opposite problem
that the stall values demonstrated: being overly conservative. The manufacturer states that
exceeding the nominal values for brief periods is safe [313, 326], therefore limiting the BLDC motor
to only torques/currents below these values will greatly underutilize its capabilities (resulting in a
suboptimal design solution). For the application modelled by the simulator (wearable rehabilitation
brace), it can be assumed that the need for long periods of continuous operation at high torques
will be low. The intended purpose of device is to operate using short bursts of motion, moving
between set-points and holding stable positions as the system moves the user’s arm to the specified
joint angle required to execute their desired task (as opposed to a task like driving the wheel of a
car, which would require long stretches of continuous operation to keep the wheel spinning). For
these reasons, it was decided that the nominal torque/current were also unsuitable for use in the
model as saturation limits.
Given that both sets of torque/current specifications provided in the datasheet could not be
used directly to model the saturation limits, another method needed to be employed to choose
these values in a reasonable way (finding some middle ground between the two extremes of nom-
inal and stall). Accurately modelling/calculating the amount of safe overloading allowed on the
motor is a challenging and imprecise task because it depends on so many physical/environmental
factors (current input voltage, starting temperature of the motor, ambient temperature, amount
of torque applied, duration of the overloading conditions, time between overloading, etc.) [326].
Without access to the actual device to take performance measurements, any attempt to develop
a worthwhile model of motor temperature would likely be insufficient in capturing the desired
actuator behaviour (and would be out of the scope of this project). After experimenting with
different torque/current limit values, it was decided that, for purposes of the simulator discussed
here, a simple approximation would be used to determine the saturation limits. Since current is
what actually causes the motor to overheat, it was decided to limit current to twice the nominal
value, allowing for peak currents above nominal, but staying well below the stall level. The torque
saturation value was then calculated simply as whatever torque value corresponds to twice nominal
current limit. This process is described using the following equations:
imax = 2 · inom , (7.12)
τmax = Kt · imax . (7.13)
Substituting the appropriate values from Table 7.2 into Equations 7.12 and 7.13), the torque/cur-
rent saturation values were evaluated as
imax = 2(7.06 A) = 14.12 A,
τmax = (0.136 Nm/A)(14.12 A) = 1.9203 Nm/A,
which can be shown to be well below the stall current (107 A) and stall torque (14.6 Nm) values
reported in the datasheet (shown in Table 7.2), while also being slightly higher than the nominal
values, providing a less harshly-conservative estimate of motor performance.
The final set of values that need to be determined for the mathematical model are the co-
efficients of viscous friction for the DC motor (bM ) and gearhead (bG ), so friction losses in the
actuator could be accurately simulated. Unfortunately, nether parameter was provided by the
manufacturer in the datasheets; therefore, a method to estimate both values using the information
available needed to be determined.
To obtain an estimate of the DC motor coefficient of friction, an approach from the literature
based on the no-load motor parameters in Table 7.2 was employed. If the motor is operating under
steady-state conditions (the rotor is spinning at a constant velocity, θ̈M (t) = 0 rad/s), and with
no external load attached (the rotor spins unopposed, τRL = 0 Nm), it can then be extrapolated
that the only resistive torque present in the system is being caused by internal losses in the motor.
By assuming that these losses are all the result of internal friction in the motor, a value for
the coefficient of friction can then be solved by rearranging the differential equation representing
the DC motor (Equation A.4) and substituting in no-load measurement values. Note, a no-load
torque value was not provided by the manufacturer, therefore one needed to be calculated using
the reported no-load current (iNL ) and the torque constant (Kt ). Thankfully, a measurement of
no-load speed (θ̇NL ) was provided in the datasheet, so it could be used directly in the formula.
Rearranging the mechanical model differential equation to solve for the motor coefficient of friction,
and applying no-load conditions, resulted in the following equation [297, 327]:
Kt · iNL
bM = . (7.14)
θ̇NL
Substituting the appropriate values from Table 7.2 into Equation 7.14, the DC motor coefficient
of friction was calculated as follows:
(0.136 Nm/A)(0.490 A)
bM = ≈ 3.0594 × 10−4 Nms/rad.
217.817 rad/s
While this method gives an approximate estimate of the motor coefficient of viscous friction, it is
important to note that this value will not match the real-world actuator perfectly. Internal friction
losses will not scale perfectly with speed and it is likely that there will be other sources of loss in
the motor besides friction. This approach should provide a reasonable estimate of the DC motor
coefficient of friction however, which can be expected to be within the realistic range of values for
the selected actuator, hence it was used in this work here.
The method used to determine the gearhead coefficient of viscous friction was similar to that
of the motor: using internal losses to calculate an estimated value. If the assumption is made
that all losses within the gearhead are due to friction (which will likely not be the case in an
actual gearhead, but is a reasonable simplification for purposes of the work done here), then the
coefficient of friction can be approximated using the reported gearhead efficiency (ηG ) and the
nominal torque/speed (τG,nom and θ̇G,nom , respectively) [297, 327]. The idea of this approach is to
use the manufacturer supplied efficiency score to determine the percentage of losses in the gearhead
(how far the efficiency score is from 100%). Then, following the assumption that this difference is
entirely due to friction, it is applied to the nominal measurements to relate the lost torque to the
friction coefficient. Following this method, the formula for calculating the gearhead coefficient of
friction is given by [297, 327]:
(1 − ηG ) · τG,nom
bG = . (7.15)
θ̇G,nom
Substituting the appropriate values from Table 7.3 into Equation 7.15, the gearhead coefficient of
viscous friction was evaluated thusly:
(1 − 0.75)(30 Nm)
bG = ≈ 0.0119 Nms/rad.
628.3185 rad/s
As with the DC motor, the gearhead coefficient of friction calculated here is only an approximation
of the real-world value, due to the assumptions taken to obtain this estimate. The act of attempting
to actually model all the real-world friction effects inside a gearhead, as well as inside the motor, is
a much more complicated process, which involves developing a system of equations that considers
static and kinetic friction as functions of velocity [328–330]. This approach would greatly increase
the complexity of the actuator model and is outside of the scope of the work done here, therefore
the basic approximation was used instead. Despite the simplicity of the approach employed here to
estimate gearhead coefficient of friction, the resulting value should still be expected to be within a
realistic range for the chosen actuator. Therefore, it fulfils the desired goal of attempting to include
gearhead damping into the system, and will be used with the mathematical model employed by
the simulator.
Once all of the BLDC motor/gearhead parameters were obtained, the mathematical model of
the actuator could finally be completed. To clearly describe and organize the actuator parameters,
the final values for all of the variables defined in Figure 7.6 are summarized in Table 7.4 (with
previously stated values being repeated here, to collect all parameters in one place for the sake of
clarity).
Table 7.4: The final parameter values used to when implementing the actuator model for the upper-
limb brace simulator. All values have been collected here in one place for convenience.
Combined DC Motor/Gearhead Actuator Model Parameters

Armature Resistance R 0.56 Ω
Armature Inductance L 0.738×10−3 H
Torque Constant Kt 0.136 Nm/A
Motor Inertia JM 0.506×10−3 kgm2
Gearhead Inertia JG 1.730×10−6 kgm2
Motor Coefficient of Friction bM 3.0594×10−4 Nms/rad
Gearhead Coefficient of Friction bG 0.0119 Nms/rad
Gear Reduction n 42.875
Input Voltage Limits vmax ±30 V
Motor Current Limits imax ±14.12 A
Motor Torque Limits τmax ±1.9203 Nm/A
Backlash θBkl 0.017453 rad
Output Position Safety Limits θmax 0 rad – 2.61799 rad
Once the mathematical model of the actuator was fully defined, it was implemented in Simulink
to facilitate all of the calculations for the upper-limb brace simulator. Using the values of the ac-
tuator parameters shown in Table 7.4, the final actuator system block diagram (shown in Figure
7.6) was implemented in Simulink to act as the plant being controlled by the TWSC. MATLAB
code was also developed to load all actuator parameter values and automate the running of the
Simulink model for each study participant, greatly simplifying the use of the simulator. A screen-
shot of the actuator plant model (encompassing the BLDC motor and gearhead) implemented in
Simulink can be seen in Figure 7.7.
In addition to the system blocks shown in Figure 7.6, the Simulink model shown in Figure 7.7 also
utilized a From Workspace block to load signals from the MATLAB workspace (in this case, input
voltage), as well as To Workspace blocks to export signal measurements obtained as results from
Figure 7.7: A screenshot showing the actuator plant model implemented in Simulink. Param-
eter values used in the various Simulink blocks were loaded from the MATLAB
workspace during initialization (these values match what is shown in Table 7.4). The
Simulink model also utilized a From Workspace block to load signals from the MAT-
LAB workspace, as well as To Workspace blocks to export signal measurement results
to the MATLAB workspace for further analysis.
running the simulation (in this case, various motor performance metrics, such as torque/current,
as well as the output position of the brace) to the MATLAB workspace for further analysis. These
were used in conjunction with the developed MATLAB code to manage all data, automate the
process of running the simulator for each study participant, and to analyse the results. Using
this approach, the simulation results could be easily recorded in a data array and saved for later
analysis using other MATLAB scripts.
7.3.4 Modelling Load Inertias
As discussed in Section A.1.2, the moment of inertia of both the subject’s arm, as well as the
dumbbell they are holding during movement, needed to be determined to incorporate the effect of
both loads into the mechanical plant model of the actuator. After calculating both inertias sepa-
rately, they can be summed together according to the following equation to act as the equivalent
load inertia (JL ):
JL = JA + JDb , (7.16)
where JA is the moment of inertia of the user’s arm and JDb is the moment of inertia of the
dumbbell. Since both of these values depend on the mass/geometry of the object in question, both
dumbbells (3 lbs and 5 lbs) will have different inertia value and all participants’ arms will also have
unique inertia values (similar to how the required static torque values calculated using Equation
7.8 were specific to each subject). The following sub-sections will discuss the method used to
calculate the moment of inertia for the subject’s arm, as well as both dumbbells, respectively.
7.3.4.1 Inertia of the Subject’s Arm
Similar to how the calculation of torque caused by the weight of subject’s arm was based on
anthropometric COM data, the method used to approximate the inertia of each subject’s arm
also utilized anthropometric data obtained from biomechanics literature. Normally, finding the
moment of inertia of a complex shape requires said object to be broken into smaller sections,
with the shapes of these portions approximated as less-complex geometry so known moment of
inertia formulas can be used. However, when determining the inertia of human body segments,
this process is not required thanks to previous work in human biomechanics literature, which
quantified approximations for a parameter known as the radius of gyration (ROG), represented
with the symbol ρ [293, 294]. The ROG is a parameter that defines the distance between the COM
of a body segment and two equal point masses that split the segment mass in half [294, 331]. The
consequence of this is that the calculated inertia of the two equal point masses will end up being
the same as the moment of inertia for the entire segment at the COM. Essentially, the ROG allows
for a simplified calculation of the moment of inertia of a body segment using the following formula
[293, 294, 331]:
J0 = mρ0 2 , (7.17)
where J0 is the moment of inertia of the body segment about the COM, ρ0 is the ROG with respect
to the COM, and m is the mass of the body segment. It is important to note that Equation 7.17
only provides the inertia about the COM, which may not coincide with the axis of rotation of
the body segment. This is the case for the simulator, as the assumption was that the centre of
rotation (COR) is located at the elbow joint (or put another way, the COR is at the proximal end
of the arm body segment). Normally, to reconcile this difference, the parallel axis theorem would
need to be used to find the moment of inertia about the defined COR, according to the following
equation: [294, 331]:
Jrot = J0 + md2 , (7.18)
where Jrot is the moment of inertia about the COR (at the end of the segment) and d is the
distance from the segment COM to the COR. However, this second step was not necessary when
calculating the moment of inertia of the participants’ arms (forearm and hand combined), since
the biomechanics anthropometric data table provided a version of the ROG already expressed as
a proximal distance with respect to a rotation around the elbow joint, as opposed to being defined
by the arm COM. This proximal version of the ROG, ρp , is calculated using the following equation
[294]:
ρp = 0.827(lh + lf ). (7.19)
The proximal ROG for the arm can then be used with Equation 7.17 to find the moment
of inertia of the subject’s arm rotating about the elbow. In this case, the segment mass, m, in
Equation 7.17 refers to the mass of the subject’s arm, mA , which was determined previously in
Section 7.3.1 using the anthropometric body weight percentage approximation shown in Table 7.1.
The final equation for calculating the subject’s arm inertia was as follows:
2
JA = 0.022mb · 0.827(lh + lf ) . (7.20)
Once the arm inertia was calculated for all participants, it could then be substituted into Equation
7.16 to incorporate its effect on the total load inertia for the system.
7.3.4.2 Inertia of the Dumbbells
Once arm inertia was calculated, the only remaining effect of the load inertias to determine was
that of the dumbbells. As mentioned previously, two dumbbells, with weights of 3 lbs and 5 lbs
(shown in Figure 7.8), were held by the participants during elbow flexion–extension motion to
add more load onto their biceps/triceps muscles. These same dumbbells were chosen to act as the
hypothetical load that the user is trying to move for the brace simulator, and thus their inertial
effect on the motor needed to be considered. Modelling the inertia of the dumbbells was less
straightforward than calculating the subject’s arm inertia, and required several approximations
during formulation. For a full explanation of the methodology used to determine an equation for
dumbbell inertia, see Appendix A.2. A summary of the final result of this process will be presented
here.
Figure 7.8: A photograph of the dumbbells used during data collection. The 5 lb dumbbell is
shown on the left (with black coloured end weights) and the 3 lb dumbbell is shown
on the right (coloured pink).
As well, it was important to take into account how the dumbbell would rotate with respect
to the elbow joint (the reference point used as the COR for all mathematical modelling in the
simulator). A crucial aspect to note is that all study participants were instructed to hold the
dumbbell vertically (as shown in Figure 7.9).
This was done to minimize activation of the wrist and forearm muscles during elbow flexion–
extension so that movement was provided solely from the biceps/triceps, allowing for collection
of more isolated EMG signals (it was observed that when holding the dumbbell horizontally, the
participants had a tendency to curl their wrist inwards when performing elbow flexion). For
the purposes of this simulator, it was assumed that the dumbbells were also being held in this
orientation. Based on this assumption, the rotation of the dumbbell about its own COM and the
COR could then be considered, as outlined in Figure 7.10.
From this diagram, it is clear that the parallel axis theorem [294, 331] must be used to translate
the dumbbell moment of inertial about its own COM (J0 ) to the COR of the brace (JDb ), located
at the elbow joint. Based on the geometry of the dumbbells, an approximation was employed to
Figure 7.9: A photograph of a study participant holding a dumbbell during the data collection
process. It can be seen that the participant is holding the dumbbell in a vertical
position (with the bar perpendicular to their forearm) to reduce the contribution of
wrist muscles during elbow flexion–extension motion.
Figure 7.10: A diagram describing the rotation and inertia of the dumbbell with respect to its own
COM and with respect to the brace. During elbow flexion–extension, the dumbbell
will move along path θ(t) and the components of this rotation occurring around the
dumbbell COM will result in the inertia J0 . This inertia can be related to a rotation
about the COR of the brace using the parallel axis theorem and the distance dDb ,
resulting in the inertia JDb . The dashed lines are used to indicate the axis of rotation
for the brace and the dumbbell, and the dots on both objects are used to indicate
their COM.
simplify the inertia calculations. It was decided to approximate the shape of each of the twin heads
of the dumbbells as a cylinder, as opposed to modelling them as hexagonal prisms (which more
closely matches their shape in reality). The dumbbell was also sectioned into three components
(the two heads on either end of the dumbbell, J1 and J2 , and the handlebar, J2 ), each treated
as a complete cylinder. The purpose of this segmentation was to simplify calculations, as the
moment of inertia for each section could be found using the basic inertia formula for a cylinder
[331], and then these inertias values could simply be added together to find the total inertia of the
dumbbell. Applying these approximations, and utilizing the parallel axis theorem along with the
inertia formula for a cylinder, the final equation for the moment of inertia of the dumbbell, with
respect to the COR of the brace (the elbow joint), was found to be
1 1
m1 3r1 2 + h1 2 + m1 d1 2 + m3 3r3 2 + h3 2 + mDb dDb 2 ,

JDb = 2 (7.21)
12 12
where mDb is the total mass of the dumbbell and dDb is the distance from the COR to the dumbbell
COM (previously solved in Equation 7.6). For each approximated cylinder section m is the mass,
r is the radius, and h is the height; the subscript 1 is used to indicate the head segment of the
dumbbell and the subscript 3 is used to indicate the handlebar segment of the dumbbell; and
d1 is the distance between the head segment COM and the dumbbell COM. To approximate the
mass of each dumbbell section, the total mass of the dumbbell was scaled by the percentage of the
total volume that section occupies (assuming a uniform density of the dumbbell material). A full
explanation of this approach can be found in Appendix A.2.
Once evaluated, the values obtained from Equation 7.21 for the 3 lb and 5 lb dumbbell were
then substituted into Equation 7.16 to obtain the total load inertia values to use with the simulator.
Following this, the next step of the project was to determine a method for generating reference
motion paths to supply as inputs to the brace model for the upper-limb simulator.
7.3.5 Generating Input Motion Trajectories
To be able to properly use the simulator as a control system evaluation tool, goal motion trajec-
tories were required to provide a reference input for the hypothetical brace to follow. An output
motion trajectory demonstrated by the simulated upper-limb brace (calculated using the devel-
oped mathematical model in Simulink, described above) could be compared to the reference input
motion trajectory to provide qualitative measurements of the accuracy of the control system. Put
another way, this assessment will essentially try to answer “how well does the wearable mecha-
tronic upper-limb brace follow the motion path that the user is trying to perform?” However,
before experiments could be run using the simulator, a method for determining these input motion
trajectories needed to be established.
In an actual wearable mechatronic device, the system needs a method of detecting what trajec-
tory the user is trying to follow, so that it can properly actuate the device and move it to the user’s
intended position. There are multiple approaches that have been adopted to determine a user’s
desired motion path in real time. Many studies have attempted to decode the desired trajectory
from EEG [332–335] or EMG signals [101, 336–340], using developed models to interpret a user’s
muscle or brain activity as intended movements to generate a system input. Implementing meth-
ods such as these is a complex area of study in itself, and is outside of the scope of this project.
However, the inclusion of such models into the system should be considered for future work and
will be a necessity when progressing the project towards a real-time control implementation. For
the purposes of an offline experiment, such as the one discussed in this Chapter, is it usually
more efficient to merely use pre-recorded motion trajectory data, recorded from study participants
during data collection (often measured using other sensors such as IMUs [187, 341, 342] or motion
camera technology [342]). Unfortunately, such data were not available during the development of
the wearable upper-limb brace simulator described here, since motion measurements were not in-
cluded as part of the study protocol during the initial data collection period. Plans were in motion
to undergo a second phase of EEG/EMG signal data collection that included motion measurements
(work had begun on a wireless wearable brace that could record elbow joint angle using a rotary
encoder); however, due to the COVID-19 pandemic, these plans could not be actualized because
of quarantine policies implemented by the university. As such, the work done on the simulator had
to make due with the initially collected dataset and determine a method of interpolating input
motion trajectories based on what was available.
While motion measurements were not recorded directly during data collection, there is still some
amount of participant movement information that can be extracted from the dataset obtained.
During the EEG/EMG signal recording sessions, the experimenter manually placed markers in the
data at the beginning and end of every elbow flexion–extension repetition using en external trigger
system. As well, webcam video was recorded of the study participant that was automatically
synchronized with the bioelectrical signal recordings by the measurement system software. Using
these resources, it was possible to extrapolate participant movement data for use when generating
input motion trajectories for the simulator. To accomplish this, the recorded videos were manually
analyzed to add more markers to the data by hand, signalling relevant motion actions by the study
participant. It was decided to mark the videos at every point where the subject’s arm underwent a
change in its motion state, meaning the moment when they stop moving or the moment when they
start moving again. These were chosen as the marker points since they are an easy action for the
person analysing the videos to notice visually; they can compare the image frame-by-frame to find
where the subject’s arm begins/stops moving within the video. Attempting to add more complex
markers during the elbow flexion–extension motion would result in inconsistent measurements, as
it would be challenging for the video analyser to quantitatively gauge relevant aspects of the frame
image (for example, elbow angle) due to various changing factors in how the video was recorded
(e.g, the camera angle used at the time). Using this philosophy for video analysis, markers were
added to the data when the subject began elbow flexion, when elbow flexion ended (at the top
of their range of motion), when elbow extension began, and when elbow extension ended (their
arm is fully extended). These markers were repeated for each elbow flexion-extension repetition
in the trial video recording (three repetitions per trial). A plot showing example sets of elbow
flexion–extension repetitions for both speeds (fast, approximately 150°/s, and slow, approximately
10°/s), with the video markers and trigger markers overlaid, can be seen in Figure 7.11.
The actual implementation of the video markers was done by counting video frames and record-
ing the specific frame number of a relevant motion event. The person analysing the videos manually
scrubbed through each video frame-by-frame and noted the frame number where a marker would
be placed. Knowing that the Frames Per Second (FPS) of the webcam used during data collection
was 25 FPS, the frame number could then be converted to a time measurement (in seconds) using
the following equation:
F rame N umber
V ideo M arker [s] = . (7.22)
25 FPS
To simplify the integration of the video markers with the bioelectrical signal data and the trigger
150
125
Angular Position [deg]
100
75
50
25
0
0 2 4 6 8 10 12 14 16
Time [s]
(a) Fast speed
150
125
100
75
50
25
0
0 10 20 30 40 50 60 70 80 90
Time [s]
(b) Slow speed
Figure 7.11: Example motion paths for a fast (a) and a slow (b) speed set of elbow flexion–extension
repetitions from a selected study participant. The obtained timestamp markers have
been overlaid on top of the path, with video markers appearing as solid green lines
and trigger markers appearing as dashed red lines. Angular position is provided in
units of degrees to allow for more intuitive interpretation of the path.
markers collected using the measurement device, it was useful to also convert the video markers
into units of “samples”, to match them with the 4000 Hz sample rate used by the system. This
conversion was performed using the following equation:
V ideo M arker [samples] = V ideo M arker [s] · 4000 Hz. (7.23)

This method of extrapolating motion information from the previously recorded videos is not a
perfect approach for measuring the movement of the subject. For starers, the time resolution of
the markers is rather low when compared to what can be achieved through other sensing methods.
With a camera FPS of 25, the time resolution (the amount of time between each frame of the video)
is 40 ms, which means that there is a large window of time where the subject’s motion state may
have changed without being captured on video. Also, the qualitative nature of the video analysis
method may also limit the accuracy of the video markers. Having a human manually inspect the
videos was the easiest solution to implement; however, any kind of manual data marking can be
subject to the bias of the investigator (especially in cases where the task is highly repetitive as
their attention may wane). Effort was made to ensure consistency during the video analysis by
attempting to minimize these effects, but they should still be considered when contemplating the
limitations of this approach. Despite these factors, this method of determining motion data is
still a viable way of generating motion trajectories that will be based on the movements actually
performed by the study participants. With the absence of actually measured movement data, using
the video to create timestamps will allow the generated motion trajectories to be close to what was
actually observed, and, through seemingly crude, this method is still better than the alternative
of generating completely arbitrary input trajectories. While not exact, using the video/trigger
marker timestamps will at least keep the input trajectories within a range of similar values to the
true motion path and is likely the best method possible with the data/resources available at the
time of this work.
Once the marker timestamps were calculated for each subject, the next step was to build
an actual motion trajectory to fill the time points in between each marker. To accomplish this
task, it was decided to follow a technique used in robotics trajectory planning theory (which
has also been demonstrated with wearable mechatronic devices as well): the quintic polynomial
method [44, 293, 343–347]. This is a method of generating a point-to-point motion between a
starting point and end point by interpolating the points in between using a fifth order (quintic)
polynomial. To more directly control the shape of the resulting path calculated using a quintic
polynomial, a series of intermediary points, referred to as vias or via points, can be defined between
the start/end point and used in the polynomial interpolation to ensure the final trajectory crosses
these points at the defined times. The quintic polynomial method can be used to develop a motion
trajectory between a sequence of points by building a different trajectory polynomial in between
each via pair in sequence (as if those points were the start/end point of a smaller trajectory) and
stitching together the via-pair polynomials into the final motion trajectory that encapsulates the
entire desired path. It was decided to use the quintic polynomial method to create the input
motion trajectories for the simulator because this approach, by its mathematical definition, results
in continuous position, velocity, and acceleration profiles by allowing the velocity/acceleration
values at the via points to be directly specified [343–345]. Lower order polynomial interpolation
methods (for example a cubic polynomial, which lack the capability to define velocity/acceleration
targets), can sometimes result in discontinuous acceleration trajectories, making the movements
feel “jerky” and aggressive [344–346]. This behaviour is undesirable for a wearable mechatronic
rehabilitation device, as it should feel comfortable for the patient. Likewise, the study participants
were directed to try and move smoothly and to attempt to maintain a constant velocity as much
as possible during the elbow flexion–extension motions. While the smoothness of their motion
was not guaranteed, as their movements were self monitored, it still makes sense to attempt and
emulate this type of behaviour with the input trajectories used for the simulator, as it is more likely
to match what was actually done during data collection. For these reasons, the quintic polynomial
interpolation method is the most logical choice for generating the input motion trajectories.
The formula for the quintic polynomial used to calculate the interpolated trajectory dictating
the user’s elbow angle, θ(t), as well as its derivatives describing angular velocity (θ̇(t)) and angular
acceleration (θ̈(t)) are as follows [44, 343–346]:
θ(t) = a0 + a1 t + a2 t2 + a3 t3 + a4 t4 + a5 t5 , (7.24)
θ̇(t) = a1 + 2a2 t + 3a3 t2 + 4a4 t3 + 5a5 t4 , (7.25)
θ̈(t) = 2a2 + 6a3 t + 12a4 t2 + 20a5 t3 . (7.26)

The polynomial coefficients, labelled a0 to a5 , are calculated for each via pair and are based on
the initial/final conditions at both via points (allowing the position, velocity, and acceleration at
each via point to be specified). The formula for the five coefficients are as follows [44, 343, 344]:
a0 = θ0 , (7.27)
a1 = θ̇0 , (7.28)
θ̈0
a2 = , (7.29)
2
20θf − 20θ0 − (8θ̇f + 12θ̇0 )tf − (3θ̈0 − θ̈f )tf 2

a3 = , (7.30)
2tf 3
30θ0 − 30θf + (14θ̇f + 16θ̇0 )tf + (3θ̈0 − 2θ̈f )tf 2

a4 = , (7.31)
2tf 4
12θf − 12θ0 − (6θ̇f + 6θ̇0 )tf − (θ̈0 − θ̈f )tf 2

a5 = , (7.32)
2tf 5
where θ0 , θ̇0 , and θ̈0 are the initial angular position, velocity, and acceleration, respectively, at
the starting via point; θf , θ̇f , and θ̈f are the final angular position, velocity, and acceleration,
respectively, at the end via point; and tf is the final time value at the end via point (put another
way, it is the time duration of the via pair if considering the time value at the starting via point
to be 0 seconds). Note, that the terms “initial” and “final” used here are done so with respect
to the current via pair. As the calculation moves through the via pairs sequentially, recalculating
the polynomial coefficients each time, the previous “final” via position will then become the new
“initial” via position, with the next unused via point in the sequence becoming the new “final”
position (and so on, with this pattern repeating until the end of the trajectory is reached). The
number of interpolated points included between each via pair in the calculation was determined
such that the data points in the motion trajectory would match the bioelectrical measurement
system sampling rate of 4000 Hz. This was done primarily to simplify the integration of the
calculated motion trajectory data within the recorded EEG/EMG signal dataset, to avoid any
issues potentially caused by mismatched sampling rates.
Using Equations 7.24 to 7.32, and using the obtained marker timestamps as via points, motion
trajectories were calculated for all subjects to approximate their elbow flexion–extension move-
ments during their data recording session. It was decided to construct the via points for the motion
path using the timestamps obtained from the video markers, as well as the first and last trigger
marker from the trial recordings. This approach will encapsulate all three elbow flexion-extension
repetitions in the trajectory, as well as the rest periods in between repetitions. The purpose of
including the first and last trigger markers was to ensure that the length of the generated trajecto-
ries matched the length of the bioelectrical signals in the EEG/EMG dataset, since the start/end
trigger markers were the reference points used to crop trial recordings during the segmentation
phase of the signal processing procedure utilized during development of the EEG–EMG fusion-
based classifiers (as outlined in the preceding chapters). This resulted in a set of via points defined
thusly:
h i
θvia = 0◦ 0◦ 150◦ 150◦ 0◦ 0◦ 150◦ 150◦ 0◦ 0◦ 150◦ 150◦ 0◦ 0◦ ,
where each entry in the array is a position value that aligns with its respective timestamp obtained
from the video/trigger markers. The quintic polynomial method was then applied to interpolate
between these values, ensuring that the final motion trajectory followed the desired path properly.
Note, that the angular position values shown above are in degrees, to improve readability and
allow for a more intuitive interpretation of the values, but in the actual calculation radian units
must be used instead (to ensure SI units are maintained consistently throughout all aspects of the
simulator). As discussed previously in Section 7.3.2.2, the assumed range of motion for all study
participants was chosen to be from 0◦ (elbow fully extended) to 150◦ (elbow fully flexed). Thus,
these were chosen as the specific angular position values used for the via points when defining the
set.
To evaluate Equations 7.28 to 7.32 and obtain polynomial coefficients for each via pair, the
velocity and acceleration values at all via points needed to be defined. This was done by applying
the assumption that the subject was always stationary at the instance of all via points, setting all
velocity/acceleration conditions to zero thusly:
θ̇0 = θ̇f = 0 rad/s, (7.33)
θ̈0 = θ̈f = 0 rad/s2 . (7.34)
The assumption applied above was a consequence of how the video marker timestamps, and there-
fore the via points themselves, were extracted from the study participant recordings. Recall, that
the markers were placed specifically at points in the video where the subject had momentarily
paused their motion (either after completing a movement or at the moment before motions be-
gins), hence for all via points it can be assumed that velocity and acceleration are zero. In the
absence of actual velocity/acceleration measurements, this approach seems like a suitable com-
promise, since the quintic polynomial method should interpolate continuous velocity/acceleration
values in between the via points that match the distance travelled by the subject during the de-
fined amount of time and provide a good approximation of the motion trajectory actually observed
during data collection. Knowing that the velocity/acceleration values at all via points is the same,
and that this value is always 0 (as per Equations 7.33 and 7.34), the polynomial coefficient formulas
(Equations 7.27 to 7.32) can be simplified to the following [293, 345]:
a0 = θ0 , (7.35)
a1 = 0, (7.36)
a2 = 0, (7.37)
20θf − 20θ0
a3 = , (7.38)
2tf 3
30θ0 − 30θf
a4 = , (7.39)
2tf 4
12θf − 12θ0
a5 = . (7.40)
2tf 5
After the simplification of the polynomial coefficient equations, the motion trajectories were
calculated for all participants using the timestamps obtained from their recorded trigger signals
and videos. This motion trajectory (describing both the path travelled by their arm during elbow
flexion–extension, as well as the velocity/acceleration with which it moved) was then used as
an input signal for the simulator to act as the desired response of the device that the control
system should seek to follow. An example of the resultant motion trajectory obtained using the
quintic polynomial interpolation method, for both a representative fast motion speed and slow
motion speed trial, can be seen in Figures 7.12(a) and 7.12(b), respectively. Of particular note,
it can be observed that position, velocity, and acceleration all display continuous behaviour, as
expected, and that the defined values at all via points are met. This demonstrated that the quintic
polynomial method was able to successfully interpolate the trajectory and should provide a suitable
approximation of the motions undertaken by the study participant during data collection.
7.3.6 Modelling Dynamic Weight Changes During Motion
In the previous subsections, the developed actuator model for the upper-limb simulator was out-
lined and shown to be implemented in Simulink. However, this implementation has a key limitation
that needed to be addressed to provide the simulator with the full functionality required to perform
the planned experiments: the ability to dynamically modify the user’s task weight during motion
trials. As a consequence of how it was built in Simulink, the actuator model shown in Figure 7.7
can only operate with an assumed static load while the simulator is running (meaning that the
only behaviour it could simulate was the user holding the same weight while moving, and only
Pos. [rad]
2
0
0 3 6 9 12 15
5
Vel. [rad/s]
-5
0 3 6 9 12 15
Accel. [rad/s2]
20
-20
0 3 6 9 12 15
Time [s]
(a) Fast speed

Pos. [rad]
0
0 10 20 30 40 50 60 70 80 90
0.5
Vel. [rad/s]
-0.5
0 10 20 30 40 50 60 70 80 90
Accel. [rad/s2]
0.1
0
-0.1
0 10 20 30 40 50 60 70 80 90
Time [s]
(b) Slow speed
Figure 7.12: An example motion trajectory from a selected study participant showing angular
position, velocity, and acceleration for a fast (a) and slow (b) speed motion trial,
obtained using the quintic polynomial interpolation method. For both trajectories,
it can be observed that the resulting polynomials are continuous, indicating that the
interpolation was successful.
changing that weight in between elbow flexion–extension repetitions). While this is an important
use case to consider (how well the TWSC performs for a set weight), how the TWSC behaves
during a sudden change in the system dynamics (for example, the user dropping the weight while
moving) is also an interesting question to consider. Given that the idea behind the conception
of the TWSC was to make wearable mechatronic devices more adaptable, it seems pertinent to
include such an evaluation as part of the planned experimentation.

Any change in the weight being held would result in a different value for the dumbbell inertia
(JDb ), as per Equation A.24, which in turn will modify the load inertia (JL ) value (Equation 7.16);
ultimately requiring the total inertia (Jtotal ) value to be updated (based on Equation A.14) in
the mechanical model transfer function block. Such behaviour was not possible however, with the
current implementation of the model in Simulink (shown in Figure 7.7), since the basic transfer
function block operates using static parameter values while the simulator is running. The MAT-
LAB script would load in the correct Jtotal value (depending on the current subject and load they
were holding for that given trial) during model initialization, then Simulink would run the sim-
ulation for the entire current motion trial (three elbow flexion–extension repetitions) using that
value. To allow for a version of the simulator that could vary the value of Jtotal while running
(thus modelling dynamic weight changes), modifications were required to the Simulink implemen-
tation of the actuator system model. Likewise, a new set of input data needed to be generated for
use when running the simulator. Since dynamic weight variations were not a part of the original
data collection protocol (the study participants always performed elbow flexion–extension motions
holding a set weight), no recordings were performed that captured this behaviour. As such, a
method needed to be devised to create synthetic motion data to simulate this type of movement
in a meaningful way. The following subsections will outline the changes made to the simulator to
add the ability to model the system response caused by dynamic weight variations during motion.
7.3.6.1 Variable Inertia Transfer Function Subsystem
The first challenge that needed to be addressed was to modify the actuator model to include
a transfer function block that permits time-varying parameters. Simulink actually includes a
Varying Transfer Function block that allows for such behaviour; however, it unfortunately
cannot be configured to use the form of the first-order transfer function specified by the actuator
mechanical model (the block denominator is set as some value plus s, whereas the equation for
the actuator mechanical model requires that Jtotal be multiplied by s). Thus, a varying first-order
transfer function block needed to be implemented manually using the basic math blocks available
in Simulink and incorporated into the actuator model as a subsystem. This is actually how the
Varying Transfer Function block provided by Simulink works at the base level, so by observing
its formulation it was possible to reverse engineer a method for creating a varying transfer function
block that uses the required form instead.
To match the desired form of the actuator mechanical model, the transfer function needed to
be expressed in the form:
1
y(s) = u(s) · , (7.41)
Jtotal s + bMG
where u(s) is the input to the transfer function and y(s) is the output of the transfer function.
However, observing this form directly, it is hard to determine what basic mathematical operations
need to be completed in the subsystem to end up with a transfer function of this arrangement.
Thus, the above equation was rearranged as follows:
1 1
y(s) = u(s) − y(s)bMG · · . (7.42)
s Jtotal
Note, that this rearrangement was completed in such a way as to specifically ensure that all
Laplace operations were divisions of s (integration) and that no multiplications of s (derivation)
occurred. This is because a true derivative is a non-causal operation (as per the definition of the
derivative, it requires future information) and, as such, it cannot be properly executed in a time-
based simulation (which is why Simulink forbids this operation and has no “s multiplication block”
available to its users). Simulink does have the functionality to approximate a derivative operation
using a numerical derivative method; however, in the Simulink documentation they recommend
avoiding its use and to structure all models in a way that uses only integrators instead. This is
because the numerical derivative can be sensitive to the size of the timesteps used when running
the model, which can result in fluctuating errors in its output. Based on this recommendation,
the use of derivates was avoided by rearranging the transfer function equation to only require
integrators.
Using the rearranged first-order transfer function formula shown in Equation 7.42, a time-
varying transfer function subsystem block was implemented in Simulink. The developed subsystem
can be seen in Figure 7.13.
Figure 7.13: A screenshot showing how the variable first-order transfer function subsystem was
implemented in Simulink to use in the actuator mechanical model portion of the
dynamic task weight simulator. The rearranged transfer function formula shown in
Equation 7.42 was implemented using basic math blocks. Annotations have been
added on each signal line (shown as the smaller blue text) to help explain how the
block diagram relates to the rearranged transfer function equation.
Now that a transfer function subsystem was developed that can vary its parameters while the
simulation is running, the next step was to determine how to automate the control of this variation
using MATLAB/Simulink. It was implemented simply using a Multiport Switch block available
in Simulink. This block allows an inputted control signal to switch the output of the block between
multiple supplied inputs, in this case those inputs are the various values of Jtotal that correspond
to the user holding the different task weights. The output of this block feeds directly into the
varying transfer function subsystem to modify the dynamics while the simulator is running. The
full Simulink implementation of the dynamic task weight simulator with the Multiport Switch
and time-varying transfer function subsystem block can be seen in Figure 7.14.
The control signal for the Multiport Switch block was pre-generated prior to running the
simulation and loaded into Simulink by the MATLAB script during model initialization. The next
subsection will discuss how this control signal, as well as the input motion trajectories for these
experiments, were generated.
Figure 7.14: A screenshot showing the modified actuator model, which includes the variable trans-
fer function subsystem and the Multiport Switch that controls it, to allow for the
simulation of dynamic task weight variations during motion. As with the other actu-
ator parameters, the various total inertia values corresponding to each weight level,
and the control signal used to determine the output of the Multiport Switch block,
are loaded from the MATLAB workspace during initialization.
7.3.6.2 Generating Synthetic Input Motion Data
A large obstacle impeding the dynamic task weight evaluation experiments for the TWSC was the
lack of recorded data to use as inputs to the system. As previously mentioned, new data collection
protocols had to be suspended due to the COVID-19 pandemic, therefore all simulator development
had to be performed using the previously acquired dataset. Since these data were not collected
with the intention of developing a simulator for evaluating a control system during dynamic task
weight variations, the study participants never varied the weight they were holding during the
motion trials. The participant would hold the same dumbbell for the full duration of each motion
trial (three elbow flexion–extension repetitions) and would switch dumbbells in between recordings.
Thus, to be able to perform the desired experiments described here, it was decided to develop a
synthetic dataset of simulator inputs that mimic a variable task weight situation using the data
already available. Even though this is not as rigorous as an evaluation performed using real-world
motion data recorded from human volunteers, the use of a synthetic dataset can still provide some
insights on the behaviour of the TWSC in a dynamic situation and can act as a good starting
point for determining its effectiveness. Future work should, of course, pursue an evaluation using
real data.
The first step towards developing the synthetic simulator inputs for the dynamic task weight
variation experiments was to generate the input motion trajectories to use as the reference signal
that the brace will attempt to follow. Since the only motion information available was the trig-
ger/video markers (as previously discussed in Section 7.3.5), the synthetic motion trajectories had
to be based-upon those recorded movement trials. It was decided to use the first elbow flexion–
extension repetition from both the slow and fast motion trial when the subject was holding 5 lbs.
This way, both an example of a slow and fast motion (with dynamic task weight changes) would be
included in the evaluation. The decision to take this repetition from the 5 lbs trials was somewhat
arbitrary, as any of the motion trials would likely have been sufficient, as there is little variation
between their motion paths. The main reason behind this choice is that the initial conception of
the dynamic weight variation experiment was to simulate the subject dropping a weight during
motion (going from 5 lbs to 0 lbs), hence using the 5 lbs trial as the basis for the motion trajectory
seemed logical. Later on, the experiment was expanded to include another variable task weight
scenario (as discussed below); however, the project continued to utilize the 5 lbs trials during tra-
jectory generation for simplicity. The decision to only use one repetition for each speed was made
purely to simplify implementation. Since the dataset is synthetic anyway, its main purpose is to
demonstrate an example of a dynamic task weight experiment, not to make any strong statistical
claims about the quantitative performance of the TWSC under such conditions. More repetitions
of synthetic data could have been included, but the benefit of running more samples of the same
artificial example remain to be seen. Thus, to reduce simulation time, the motion trajectory was
kept to one elbow flexion–extension repetition per speed. As in Section 7.3.5, the key points of the
trajectory were defined using the video markers, and the trigger markers describing the start/end
point of the motion repetition, as via points during interpolation. An example of the synthetic
input motion path, with the associated trigger/video markers used for the via points, can be seen
in Figure 7.15.
Once the parameters of the synthetic motion trajectory were defined, it was calculated simply
using the same quintic polynomial interpolation method outlined in Section 7.3.5. The only dif-
ference was that the list of via points (shown in Figure 7.15) was shortened to only include the
first elbow flexion–extension repetition, given as follows:
h i
θvia,synth = 0◦ 0◦ 150◦ 150◦ 0◦ 0◦ .
Using these via positions aligned with the associated trigger/video markers, along with Equa-
tions 7.24 to 7.26 and Equations 7.35 to 7.40, an input motion trajectory was calculated for both
the slow 5 lbs and fast 5 lbs trials for each subject, to be used with the simulator for the dynamic
task weight variation experiments.
150
2
125
100
1 3
75
50
25
0
0 0.5 1 1.5 2 2.5 3 3.5
Time [s]
Figure 7.15: An example motion path for a fast speed elbow flexion–extension repetition from a
selected study participant, generated for the synthetic dataset. The obtained times-
tamp markers used as via points have been overlaid on top of the path, with video
markers appearing as solid green lines and trigger markers appearing as dashed red
lines. Angular position is provided in units of degrees to allow for more intuitive
interpretation of the path. Time points in the motion path where the three weight
change conditions will occur are marked with a red “X” and numbered accordingly.
Now that synthetic input motion trajectories were calculated, the next step was to determine
what hypothetical task weight variations to model in the dataset. Since, as previously stated, no
recordings were collected with changing task weights during motion, an infinite range of potential
conditions could be artificially simulated with the synthetic dataset. To narrow down the evaluation
(focusing on key situations of interest), and to ensure that the synthetic dataset could be consis-
tently generated using an easily repeatable method, it was decided that three situations would be
evaluated, defined by when the change in task weight occurs in the elbow flexion–extension motion
trajectory. The three weight change conditions (shown in Figure 7.15) were defined as follows:
1. at the midpoint of elbow flexion,
2. at the midpoint of the brief rest period occurring between elbow flexion and extension, and
3. at the midpoint of elbow extension.
With these three points defined, is was then possible to calculate a timestamp for each using the
subject’s video/trigger marker timestamps (previously used as via points for the quintic polynomial
interpolation). As an example, weight change condition one (tW1 ) will be considered.
150
125 tW1
100
75
50
25
0
t0 t1 tW1 t2 t3 t4 t5
Time [s]
Figure 7.16: An example motion path for a fast speed elbow flexion–extension repetition (also
shown in Figure 7.15), labelled with time variables to demonstrate the calculation
of the timestamp for weight change condition one. The time where each via point
occurs has been labelled as t0 (0 s) to t5 (the end of the motion path). The time
when weight change condition one occurs has been labelled tW1 and the midpoint of
the flexion motion (halfway between t2 and t1 ) has been labelled as ∆tW1 .
Assuming the via points are numbered as shown in Figure 7.16, then the duration taken to reach
the midpoint of elbow flexion, ∆tW1 , was found by:

t2 − t1
∆tW1 = , (7.43)
2
where the ceiling function is applied to round up to the nearest integer sample in the case of a
decimal result (units of samples were used for the calculation, as opposed to seconds, to simplify
the programming required in the MATLAB script). Looking at Figure 7.16, it can be seen that
∆tW1 only provides a time duration describing the midpoint time of flexion relative to the start of
the motion, not an absolute timestamp related to the beginning of the overall simulation time. To
shift the calculated midpoint duration back to the global start time, the time when flexion begins
must be added according to the following equation:
tW1 = ∆tW1 + t1 . (7.44)
Generalizing Equations 7.43 and 7.44 to all three weight conditions resulted in the following:
t(i+1) − ti

tWi = ∆tWi + ti = + ti , (7.45)
2
where i is an index denoting the number of the desired weight change condition (1, 2, or 3),
which will shift the via points timestamps accordingly to ensure the correct pair is used during
the calculation.
Using Equation 7.45, the timestamps for each weight change condition were calculated and
could then be used to generate the signal used to control the inertia variations within the simulator.
As discussed in Section 7.3.6.1, the dynamic task weight changes were implemented in Simulink
using a variable transfer function subsystem combined with a Multiport Switch (shown in Figure
7.14); depending on the control signal being sent to the switch, it would send different total
inertia values to the transfer function corresponding with the 0 lbs, 3 lbs, and 5 lbs weights. The
Multiport Switch operates simply, switching to the output port corresponding to the current
integer value of the control signal (signals of 0, 1, and 2 corresponding to inertia values for weights
of 0 lbs, 3 lbs, and 5 lbs, respectively). Thus, all that needed to be done to generate the weight
switch control signal was to create a signal that changed value at the timestamp corresponding to
the current weight change condition (tWi ).
As with the decision of where in the motion trajectory to have the weight change occur, there
was also a decision to be made about what task weight combinations to evaluate (deciding what
the starting weight and end weight should be). To simplify the experiment and reduce the amount
of permeations that required simulation, it was decided to only evaluate two scenarios: starting
at 5 lbs then changing to 0 lbs, and starting at 0 lbs then changing to 5 lbs. The first scenario
presents a use case for the device analogous to the subject dropping a held object during motion
(suddenly changing to a reduced load on the system). The second scenario presents a use case
for the device analogous to the subject reaching for an object and picking it up. Even though
the motion being simulated by the model is highly constrained (only elbow flexion–extension),
which would not realistically be used for a grasping task, the second scenario still provides an
approximation of the change in dynamics such a task would incur (a sudden increase in load),
making it worthwhile to include in the evaluation. Both scenarios were chosen specifically because
they provide an example of common events for which the upper-limb brace will need to compensate
during various activities of daily living. It was decided to only focus on weight changes between
0 lbs and 5 lbs, since this presented the most extreme difference in dynamics, with the thinking
being that this would provide the strictest evaluation conditions for the system and therefore be
the most important to investigate. Future work could look at evaluating weight change scenarios
that incorporate the 3 lb weight (as it is possible this may present different behaviour).
Using the calculated weight change timestamps for the three conditions (changing at flexion
midpoint, changing at the midpoint between flexion/extension, and changing at extension mid-
point), a signal was generated to control the Multiport Switch output for both weight scenarios
(5 lbs to 0 lbs and 0 lbs to 5 lbs). An example of the weight change control signals for each
condition, demonstrating the first task weight scenario (5 lbs to 0 lbs) can be seen in Figure 7.17.
With this task completed, the ability to dynamically vary task weight during motion in the
upper-limb brace simulator was fully realized, accommodating the required functionality necessary
for the evaluation of the TWSC. The developed combined BLDC motor/gearhead actuator model
could then be used to simulate the behaviour of a wearable upper-limb mechatronic device under
multiple use cases, acting a platform to investigate the performance of the TWSC.
7.3.7 Upper-Limb Brace Performance Metrics
Before implementing the TWSC, one more aspect of the wearable upper-limb brace needed to be
considered: how to quantify its performance. The simulator is able to output the current angular
position of the brace for a given input (either voltage, as in the actuator model shown in Figure
7.6, or as a reference motion trajectory to follow, as in the final closed-loop system); however,
5 lbs 5 lbs 5 lbs

Switch Value
Switch Value
Switch Value
0 lbs 0 lbs 0 lbs
t0 tW1 t5 t0 tW2 t5 t0 tW3 t5
Time [s] Time [s] Time [s]
(a) Condition 1 (b) Condition 2 (c) Condition 3
Figure 7.17: Example weight change control signals used to determine the Multiport Switch
inertia output supplied to the variable transfer function subsystem. Weight change
conditions 1 to 3 are shown in Figures (a) to (c), respectively, with all three signals
presenting the first task weight scenario (5 lbs to 0 lbs, dropping a dumbbell during
movement). The weight change occurs at the calculated timestamp for all three
conditions, simulating the desired behaviour (halfway through elbow flexion, at the
midpoint of the motion peak during the change from flexion to extension, and halfway
through elbow extension).
looking at only this output variable qualitatively may not allow for an objective evaluation of
the system performance. To properly investigate the efficacy of the TWSC in a measurable way,
performance metrics must be calculated to provide a quantitative score of the system behaviour.
Toward this end, multiple performance metrics were chosen to act as the main experiment results
of this project, and will be discussed in the subsections below.
7.3.7.1 Root Mean Square Error
The first performance metric chosen for the evaluation addressed one of the most important aspects
of wearable device control: system accuracy. To be a feasible rehabilitation and assistance tool,
wearable mechatronic devices need to be able to reliably move to the user’s desired position. If the
system is unable to properly follow the user’s commands, they will feel frustrated while attempting
to utilize the device and ultimately abandon it as a treatment solution if their level of discomfort
becomes unmanageable. Poor accuracy can also present safety concerns if the device is moving in
a way that could harm the user (e.g., moving beyond the user’s comfortable range of motion). For
these reasons, ensuring that the control system can accurately position the device is of the utmost
importance.
Accuracy can be quantified in many ways, but one common method is to use a measurement
known as error. In a control system, error (err) refers to the difference between the reference input
signal and the system output, given by the equation [295, 348]:
err(t) = θout (t) − θin (t), (7.46)
where θout is the actual output position that the brace moved to and θin is the desired input motion
path that the brace is supposed to be following. Since both the input and output position are
time-domain signals, the calculated error will also be a time-domain signal describing how far from
the desired path the brace was at any given time. Error can be either positive or negative, with the
sign dictating the nature of the discrepancy. Positive error implies that the output overshot the
input, while negative error implies that the output undershot the input. Both types of error are
problematic in their own ways and should to be minimized as much as possible when optimizing
the control system.
Trying to use the error signal directly as an evaluation metric results in a qualitative analysis
(i.e., visually comparing error signals), which can lead to unreliable findings. To attempt to
quantify the analysis of system error it was decided to utilize an accuracy related metric by
calculating the Root Mean Square Error (RMSE) from each error signal. The RMSE will take a
time-domain error signal and “summarize” its behaviour across the entire time range, resulting in
a single “averaged” error value that can be used as a performance score when comparing other
systems. As the name would imply, RMSE is calculated by squaring the error signal, taking the
mean average (by summing all error signal points and dividing by the number of error points
collected), then applying the square root to the result, as seen in the following formula [336, 348–
351]:
v
u
u1 XN
RM SE = t (θout,i − θin,i )2 , (7.47)
N
i=1
where i is the summation index for each collected error signal data point and N is the total number
of error signal data points collected.
In many ways, the RMSE is similar in principle to taking an average of the error signal, and
in fact the Mean Absolute Error (MAE), where the absolute value is applied to the error signal
to compound the effect of both overshooting and undershooting then averaged to get a mean
error score [348, 350, 352], is an accuracy metric that has been used in some works instead of
the RMSE. The choice between MAE and RMSE is still a topic under debate in some circles
[350, 353]; however, it has been shown that RMSE is more sensitive to outlier behaviour through
the squaring of the error signal [348, 350, 352, 353]; therefore, it was chosen as the metric to use
here. The effect of outlier responses is especially important to capture for a rehabilitation device,
since large over/undershoots have the potential to harm the user; thus, the performance metric
used to evaluate the system should capture this result effectively. Using MAE may potentially
cause spikes in error to be lost in the result as it gets “averaged out” of longer duration error
signals (i.e., the impact of the spike is lessened as more examples of small error are added).
7.3.7.2 Normalized Root Mean Square Error
One potential drawback of RMSE is that it can be difficult to use when comparing across different
systems (for example, braces showcased in other studies), or trials of the same system completed
under varied conditions (leading to large differences in the expected range of measured values).
While RMSE is better at capturing outliers than MAE, it can still be affected by the scale of the
error signal in a similar manner. One solution to this is to use a normalized metric instead of
an error metric that retains the units of the system (in this case radians). Scaled to be unitless,
normalized metrics can demonstrate a more objective description of the system accuracy [342, 348,
351], which can allow for easier comparison between different devices (which may have their own
specific sources of “baseline” error or report performance using other units) or trials with a different
value ranges. This is important for this work since the slow speed trials have a much larger duration
and smaller range of observed RMSE values than the fast speed trials, which is to be expected since
slower movements require less power from the actuator and allow for a more forgiving response
time by the system (typically making it easier to achieve a higher accuracy). Thus, the use of a
normalized metric will assist in providing a more objective comparison between trials, because the
results obtained from each will be scored relative to its own baseline performance. The Normalized
Root Mean Square Error (NRMSE) is one such metric and is found by diving the RMSE by the
observed range of the error signal, as shown in the following equation [222, 334, 336, 348, 349, 351]:
RM SE
N RM SE = · 100, (7.48)
errmax − errmin
where errmax is the maximum error value and errmin is the minimum error value of the observed
trial. Multiplication by 100 is performed to provide the final metric in terms of a percentage, as
it is often reported. Note, that the normalization factor used does not necessarily need to be the
error range when calculating NRMSE. Some groups will divide by other measurements, such as
maximum error [348] or the standard deviation of error [342]. Despite the apparent variations,
it appears that all three methods are accepted as standard practice. Thus, it was decided to use
the approach shown in Equation 7.48 for the work presented here, as it seemed, anecdotally, to be
more commonly utilized.
While NRMSE presents arguably a more objective score for accuracy, providing the RMSE
as well is still worthwhile in the evaluation, as it provides an easily interpretable real-world un-
derstanding of the system performance. For example, showing an “8% NRMSE score” may feel
somewhat abstract when considering how the device is actually moving, whereas saying “2 degrees
of RMSE was observed” provides readers with an immediately intuitive result that can be directly
translated to how the device would behave if used (i.e., “if I try to move to a specific position, the
brace may be off by 2 degrees”). For this reason, both RMSE and NRMSE error will be reported
as performance metrics (an approach that is commonly utilized [336, 349, 351]), to ensure the full
behaviour of the system is adequately described.
7.3.7.3 Log Dimensionless Jerk
System accuracy is, of course, a crucial metric to analyse when evaluating wearable mechatronic
devices; however, it is not the only aspect of the brace behaviour that will impact the user.
Comfort plays a large role in the user’s level of satisfaction when operating a wearable device
[7, 14, 29] and can greatly affect wide-spread adoption of such treatment solutions. A device
always positioning itself perfectly with a fast response time will not matter if it applies aggressive
or harsh movements onto the user to achieve this; it is likely that they will abandon the system
regardless of its accuracy since it causes discomfort and is hard to control [11, 29]. In the context
of rehabilitation/assistance tasks, this requirement is even more important, since users recovering
from an injury may be even more sensitive to harsh movements as their limb heals. Comfort can
manifest as a potential safety issue if the user’s movement is constrained due to the nature of their
injury, as such, being able to quantify comfort during operation is an important metric to include in
the TWSC evaluation. To explicitly clarify the terminology, in this context comfort is referring to
the aggressiveness/smoothness of the motion trajectory (how it would feel to have the brace apply
force onto the user’s arm during movement). One may argue that comfort can also be related to
accuracy, since using a system that misses targets and does not properly follow commands would
be frustrating and, hence, uncomfortable; however this aspect of device behaviour has already
been captured by the RMSE metric, hence it will not be addressed by the quantitative comfort
metric. Other aspects of comfort, such as how the brace is attached to the user or the weight of
the system, are of course important as well, but are outside the scope of the evaluation here (as
they relate more to the mechanical design of a prototype, which is not part of this project).
With the importance of evaluating comfort understood, the next step was to choose a metric
to quantify this behaviour. This turns out to be less straight forward than accuracy, since the idea
of comfort can be such a nebulous thing due to its qualitative nature (people may have different
ideas of what constitutes as “comfortable,” and will describe it more as a feeling rather than a
measurable concept). While an absolute comfort metric applicable to the entire global population
will likely never be found, one approach that can be employed is to focus on relative comparisons
(i.e., look at the difference in values between groups, as opposed to trying to interpret the absolute
value directly). This will allow one to claim “Group A is more comfortable than Group B since the
value of the comfort metric is smaller,” even if the actual value itself does not necessarily translate
to an easily understandable real-world measurement.
One metric often used for such evaluations is known as jerk. Jerk is the derivative of acceleration
(or put another way, it is the third derivative of position, or second derivative of velocity) and
provides a description of the rate of change of acceleration [354–357]. Understanding that force
is proportional to acceleration (as given by Newton’s Second Law), then jerk can be thought of
as a measurement of how harshly force is being applied as the rate of acceleration changes, which
can be interpreted as a changing effect on comfort. A high value of jerk indicates a rapid change
of acceleration, which will feel like a application of aggressive force onto the user (likely feeling
uncomfortable). A low jerk value indicates smaller changes in acceleration, leading to smoother
movement for the user (which should feel more comfortable). As with linear motion derivatives,
there is a rotational analogue of jerk known as angular jerk (ζ(t)), which is given by the following
equation:
... dα(t) d2 ω(t) d3 θ(t)

ζ(t) = θ (t) = = = , (7.49)
dt dt2 dt3
where ω and α are angular velocity and acceleration, respectively. The result of Equation 7.49
is a time-domain signal describing how angular jerk varies over time, which can then be used to
assess the smoothness of the motion trajectory. Jerk-based metrics are commonly used in studies
assessing movement smoothness [354, 358]; therefore, it was decided that jerk would be used as
the measurement value to quantify comfort during the TWSC evaluation.
Much like how a direct inspection of the error signal is not sufficient when quantifying system
accuracy, the jerk signal must also be averaged in a way that will provide a measurable score
of comfort level to facilitate analytical comparisons during the evaluation. When choosing a jerk
metric to use, similar considerations that arose during the selection of RMSE/NRMSE also needed
to be taken into account. A metric such as mean absolute jerk would be very sensitive to factors
such as motion amplitude and signal duration [354, 358, 359], making comparisons between different
movement trials difficult. A large/fast movement may show a higher base level of mean jerk (since
the subject is making a drastic motion, bigger changes in acceleration are required); however, that
does not necessarily mean that the motion is not smooth. Likewise, very long recordings may cause
the effect of a short jerky motion to be diminished when calculating an average value (given its
relatively small impact compared to the other jerk measurements in rest of the recording), hiding
the true behaviour of the motion. For this reason, a dimensionless jerk metric was chosen to
quantity comfort during the simulator experiments, as the act of applying a normalization factor
to remove units can mitigate some of these effects [354, 358, 359].
The chosen jerk metric in question is known as Log Dimensionless Jerk (LDLJ) and was
calculated using the following formula [358–360]:
2
!
(t2 − t1 )3 t2
d2 ω(t)
Z
LDLJ = ln dt , (7.50)
ωpeak 2 t1 dt2
where t1 and t2 are the start and end time of the recorded movement, respectively, and ωpeak is the
peak angular velocity achieved during motion. Using the normalization factor of duration cubed
divided by peak velocity squared, the units of the integrated jerk term are cancelled out, making
the metric dimensionless and scaling it to movement duration and amplitude [354, 356, 357, 361].
LDLJ is a popular metric of motion smoothness that has been used in many studies in the literature
[356–361], and, in particular, has shown promise for movement analysis of ADLs [360]. For these
reasons, it was chosen for use in the work here to evaluate comfort in the TWSC experiments.
LDLJ is actually an iteration of the metric dimensionless jerk [354, 355, 359], which has the
same formula as Equation 7.50 without applying the natural logarithm. It has been found, however,
that using the natural logarithm on dimensionless jerk improves its sensitivity to movements in
the physiological range [359], hence LDLJ was chosen for use here instead. There are multiple
ways to calculate dimensionless jerk, with some groups using the third derivative of position,
instead of the second derivative of velocity, in the calculation [354–357]. However, recent work has
discussed that direct measurement of velocity is better at capturing movement smoothness [361],
thus it was used for the calculation here. Likewise, there are examples that use mean velocity in
the normalization term instead of peak velocity [354, 355], but again researchers developing these
metrics have concluded that peak velocity is better at recognizing the desired response [361] (and
the use of peak velocity for normalization seemed, anecdotally, to be overwhelmingly more popular
[356, 358–361]). Therefore, this approach was used in LDLJ calculations implemented in this work.
It should also be noted that in most uses of LDLJ, a negative sign is also included outside of the
natural logarithm in the formula shown in Equation 7.50 to make the metric negative (where
a more negative value indicates a jerkier motion, and a value closer to zero implies a smoother
motion) [356, 358–361]. To simplify reporting, it was decided to exclude this negative sign in
Equation 7.50 to keep the metric positive (as other groups have done as well [357]). For the results
shown here, a larger positive LDLJ value indicates a jerkier motion, and a value closer to zero
implies a smoother motion.
To calculate LDLJ using the simulator outputs, the gearhead velocity was used as the velocity
profile, ω(t), in Equation 7.50. This was chosen because the gearhead shaft is what is actually
applying force onto the user during operation of the brace (whereas the motor rotor is merely
applying internal force onto the gearhead input shaft). Therefore, the logic applied here is that
this measurement of jerk will be closer to what the user would actually feel in regards to their level
of comfort. Looking at how the actuator model was implemented in Simulink (shown in Figure 7.7
and 7.14), it can be seen that the measured output from the simulator is actually motor velocity,
not gearhead velocity. This is because the gear reduction is applied after the integrator converting
motor velocity to position. To rectify this, the gear reduction, n, was applied manually to the
motor velocity signal offline during results analysis to convert it to gearhead velocity, then the
second derivative of the gearhead velocity was calculated using the MATLAB function gradient
to evaluate the numerical derivative twice. It should be noted that a consequence of this approach
is that the jerk metric bypasses the effects of gearhead backlash and the position safety limits on
the motion trajectory (since their blocks occur after the measurement of motor speed). This is an
unfortunate limitation of the simulator; however, it was necessary to be able to properly calculate
LDLJ. Initially, it was attempted to use the final output position to calculate LDLJ by applying
the numerical derivative thrice to acquire a jerk signal, but the appearance of sharp corners in the
position signal (caused by the saturation effects defined by the position safety limits), caused the
results of the numerical derivative to rapidly increase well beyond realistic values, as it interpreted
these corners as near-infinite rates of change [362]. To avoid the effects of these discontinuities on
the derivative, the motor velocity measurement signal was used as the foundation for the LDLJ
calculation instead, with the understanding that the results may differ slightly from the jerk profile
of the actual brace motion (however, it is highly likely that this difference would be very small, if
detectable at all).
Once the performance metrics were chosen and implemented, work on the simulator was finally
complete. The mechatronic upper-limb brace simulator could now be used as an evaluation tool
to quantify the behaviour and performance of the TWSC, to see if it demonstrated feasibility
as a method for improving the adaptability of wearable devices. The next phase of the project
focused on the implementation of a TWSC based on EEG–EMG fusion classifier predictions and
7.4 Implementing the Task Weight Selective Controller 242
its integration within the simulator framework.
7.4 Implementing the Task Weight Selective Controller
The ultimate goal of this project was to develop a controller that can adapt to changes in task
weight, creating a more robust system that improves the user experience during the operation
of a wearable upper-limb mechatronic brace. External disturbances (in this case, the varying
load caused when the user changes the weight they are holding), have been shown to affect system
performance in these devices [6, 11–13, 22, 24, 26, 30, 32–44], thus a controller needs to be designed
to rectify this issue and improve device safety/comfort.
Figure 7.18: A block diagram showing the generic form of a classical closed-loop feedback control
system. In this work, the plant is the mechatronic upper-limb brace (modelled with
the simulator described in Section 7.3) and the controller is the TWSC developed in
this project.
In the highest-level generalization, a closed-loop feedback control system takes the form shown
in Figure 7.18 [26, 295, 298]. The plant represents the process being controlled, which in this
case is the mechatronic upper-limb brace (specifically focusing on controlling its position to follow
a desired motion trajectory). In the previous section, the development of the model used for
the brace was discussed, which resulted in the system block diagram shown in Figure 7.6. This
model was utilized to represent the plant in the closed-loop feedback control system, simulating the
output response of the brace for a given input (essentially, the brace model in Figure 7.6 replaced
the block labelled “Plant” in the control system diagram). The closed-loop feedback path takes a
measurement of the current position (θout ) and subtracts it from the input reference position (θin )
to obtain an error measurement (err, as described in Equation 7.46), which is used by the controller
to determine what the next plant input should be to reduce error. Note that some control system
representations will also include a block in the feedback path to model the behaviour of the sensor
being used to provide the feedback measurement in the real world. For example, the upper-limb
brace might use a rotary encoder to measure joint position/velocity, and the characteristics of the
chosen sensor could be modelled in a similar manner to the DC motor/gearhead actuator (such as,
modelling the sensor rise/fall time as a delay using a first-order transfer function or modelling the
mechanical speed limits of the encoder rotation as a saturation). It was decided for this project
to not include a feedback sensor component into the simulator, for the purposes of simplifying its
implementation, hence unity feedback (as shown in Figure 7.18) was utilized instead.
With the plant model for the upper-limb brace established, the question then becomes what to
use as the controller in the closed-loop feedback control system. As discussed in Section 7.1, there
are many different control system techniques used with wearable robotic devices; however, they
have various drawbacks such as model dependency and high computational complexity. One of
the most ubiquitous controllers used across many different applications and industries is Propor-
tional Integral Derivative (PID) control [24, 34, 298, 299, 301, 307, 311, 312], which demonstrates
both simple implementation and model-free execution. The basis behind PID control is the mul-
tiplication of gain constants with the feedback error signal, the integral of the error signal, and
the derivative of the error signal; with the summation of those terms being used to determine
the new plant input. This process is explained, shown for both time-domain and Laplace-domain
implementations, using the following formulas [289, 295, 298, 299, 307, 311, 312, 363]:
Z
derr(t)
u(t) = Kp err(t) + Ki err(t)dt + Kd , (7.51)
dt
Ki
U (s) = Err(s) Kp + + Kd s , (7.52)
s
where Kp , Ki , and Kd are the gains applied to the proportional, integral, and derivative component
of error signal, respectively, and u is the output of the PID controller (used as the control input
to the plant). Each portion of the PID equation serves a different purpose in determining how to
control the plant to reduce error, with their respective behaviours accounting for different aspects of
the output response [364, 365]. The purpose of the proportional term is to drive the system output
to match the reference signal, with a larger error signal (or larger Kp gain) resulting in a more
aggressive response by the system to push its output towards the goal setpoint. In this sense, Kp is
used to control much of the transient behaviour of the system and will have a large effect on many
aspects of system behaviour, such as rise time and overshoot. The integral term is used to combat
steady-state error: as the integral sums error over time, the response of the integral component
of the PID controller will grow larger when error persists for a significant duration, reducing any
discrepancy between the system output and the reference input. Finally, the derivative term is
used to control fast changes in error (for example quick oscillatory behaviour): since the derivate
provides a measure of the rate of change of the error signal, rapid changes will result in a larger
PID output to combat this system response. Together, all three terms can be used to eliminate (or
at least minimize) error between the system output and reference input, and will improve device
performance when using closed-loop feedback control. The values of the PID gains (Kp , Ki , and
Kd ) can be chosen to balance the contribution of each component and need to be tuned (i.e.,
calibrated) for each system to achieve the desired behaviour/response [365].
PID control has many benefits, leading to its vast popularity across many domains and indus-
tries. For starters, PID control is easy to implement and does not require complex calculations
like many more advanced control methods [39, 307, 311, 312, 362, 365, 366]. This aspect is espe-
cially attractive for wearable devices, where computational capability is often severely limited, as
microcontrollers are mainly used in the place of computers for wearable devices to maintain their
portability and comfort. Another benefit of PID control is that it does not require a detailed model
of the system to tune/implement (and can be thought of in many ways as a model-free approach)
since it can be calibrated experimentally [34, 39, 289, 362, 363, 366, 367]. As discussed in Section
7.1, many advanced control methods used to drive wearable robotic devices require a system model
for their implementation, which is often impossible to obtain properly when considering the human
element of a wearable device introduced by the user. Perfectly modelling all of the complex dynam-
ics introduced by the user’s biology and behaviour is practically impossible, thus the models used
in advanced control methods can potentially suffer from this flaw [11, 12, 23, 26, 27, 27, 32, 34–
38, 40, 41, 43, 45, 46, 290]. Implementing a PID controller only needs the gains to be tuned
such that they obtain the desired system response, and this can be done experimentally without
requiring any system model, if required. This means that the device can be tuned for each person
during setup using a calibration procedure, without having to take any prior measurements of the
user during controller development (some of which may be infeasible/impossible to obtain from a
living human).
Unfortunately, the benefit of a simple setup (requiring only the experimental tuning of PID
gains) also brings with it a drawback: lack of adaptability. The PID gains are often tuned under
a specific set of conditions and, once chosen, will remain fixed during the operation of the device.
As a consequence of this, performance can begin to degrade in cases where the system is operated
under significantly different conditions than those present during tuning, or in situations where
external disturbances cause large changes in system dynamics, such that the previously chosen
PID gains are no longer adequately reducing error [34, 39, 42, 44, 46, 49, 363, 364, 366, 367].
Upper-limb wearable devices can be especially susceptible to this phenomenon [6, 13, 22, 26, 27,
30, 32, 33, 37, 39, 42, 101], since they are intended to provide aid during different activities of daily
living [11–13, 22, 30], which may require different motion profiles/speeds and/or various external
loads to be held by the user that were not tested during the tuning procedure. In a perfect world,
one would hope to find a set of ideal PID gains that result in flawless device performance and are
exceptionally robust to all changing conditions; however, the odds of such gains even existing as
a singular set of values is extremely unlikely (to the point of impossibility). To allow PID control
to be a feasible approach for wearable devices, this technique must be modified to address this
limitation.
7.4.1 Gain Scheduling PID Controllers
One method that has been used in the wider control system domain to address the lack of adapt-
ability in PID controllers (and, in particular, to combat the effect of non-linear systems on PID
control performance) is a technique known as Gain Scheduling. In Gain Scheduling, different
gain values are obtained for each operating condition expected for the plant during the desired
process (often done by separating a non-linear plant model into sections, based on the operating
condition cut-offs, and linearising each smaller section to facilitate easier tuning). Then, while
the system is running, a measurable variable of the operating condition is used to choose which
set of gains to apply at the current instance of the process (referred to as “scheduling” the gains)
[31, 283, 363, 368, 369]. To provide an example, an air plane control system could use Gain
Scheduling to update controller gains based on the current altitude, obtained using readings from
an air-pressure-based altitude sensor [283]. Using PID-based Gain Scheduling, the performance of
the system is improved for each operating condition, since the PID gains have been optimized for
the particular system dynamics observed at that portion of the process. The control system is able
to adapt to the measured changes as they occur, improving the overall output response without
the need for computationally intensive control law calculations [283] or a highly detailed model of
the process [363].
Gain Scheduling has also been demonstrated in wearable mechatronic devices for motion tra-
jectory tracking using joint configurations [31] and movement plane (wrist flexion–extension or
ulnar–radial deviation) [368] as the measured scheduling variable. It has also been used to adapt
treatment levels by varying the amount of torque assistance provided by a wearable rehabilitation
robot based on how much force the patient is applying to the device handle (measured using force
sensors) [370]. In the context of a wearable upper-limb brace specifically, Gain Scheduling could
also be used to allow the brace to adapt to changes in external loads. A set of PID gains could
be found for each task weight planned to be held by the user, and the control system could then
schedule the gains accordingly as task weight changes. The question then becomes, how does
the system measure the weight being held to know when to switch the gains? As discussed in
Chapter 4, there are many complications that conventional force-sensor technologies can intro-
duce into a wearable system, which was the motivation for the development of the EEG–EMG
fusion-based task weight classifiers investigated in the work performed in Chapters 4 to 6. The
task weight prediction outputs of these models can be used as the trigger to switch PID gains in
a Gain-Scheduling-based control system, which is the basis of the novel TWSC introduced here.
The integration of these machine learning models, based on bioelectrical signals, with conventional
control system methods will ideally result in more adaptable wearable devices that operate using
a simpler controller implementation.
While the use of task weight predictions from an EEG–EMG fusion-based classifier as the
scheduling variable in a Gain Scheduling-based control system has not been demonstrated before
in the literature, there are several other adjacent publications with sufficient success to provide a
solid rationale for the feasibility of this new technique. Despite not specifically using the term “Gain
Scheduling” in their descriptions, there exist several studies focused on external load compensation
in upper-limb wearable devices that utilize an approach based around updating controller gains
during system operation: using various advanced control law optimization techniques to calculate
these new values (such as fuzzy logic, to provide one example)[39, 43, 44]. Likewise, there are
also publications where machine-learning models specifically are used to update PID gain values
[363, 364] and controller parameters [292] to improve device performance, as well as an example
of the outputs of an FMG-based payload classifier being used in the torque calculations for an
admittance controller (allowing the system to adapt to changes in external loads) [288]. The
results of these studies provide evidence for the potential feasibility of the TWSC, showing that
both an adaptive approach based on updating controller gains and that using machine learning
to modify these gains in real time can provide successful results. The nature of the TWSC Gain
Scheduling being triggered by a classifier with discrete outputs (as opposed to the typical approach
used in Gain Scheduling, where the scheduling variable is a continuous measurement) causes the
controller to behave analogously to a switched system [371], essentially selecting between different
controllers for each task weight. This aspect of the design is also supported by previous work in
the wearable robotics literature that employed a switching-based control system approach, showing
that changing between various advanced controllers during operation, based on measurements of
different operation conditions, can improve system adaptability and performance [30, 43, 49, 292].
Moving outside of wearable mechatronic device literature, a method similar to the proposed TWSC
(using a classifier output to switch PID gains based on changing external disturbance types) has
been demonstrated previously for a laser-levelling system [372], which further highlights that this
approach has some merits and could be a feasible technique for disturbance rejection in wearable
mechatronic systems as well. These examples provide a justification for an investigation of the
TWSC in upper-limb wearable robotic devices, to see if it can improve system performance in the
presence of changing external loads.
7.4.2 Simulink Implementation
Implementing the actual switching component of the TWSC in Simulink was a relatively straight-
forward procedure, and involved using the developed simulator model (discussed extensively in
Section 7.3) to make a closed-loop feedback control system using a PID Controller block. The PID
Controller block was then modified to accept PID gains as an external input and a Multiport
Switch block was added to control which gains are sent to the PID controller (this operates in
the same manner as the actuator model that was modified to allow for dynamic weight changes
during motion, shown in Figure 7.14). The Simulink implementation of the TWSC can be seen in
Figure 7.19, with a collapsed version of the TWSC (where the actuator model is condensed into
a subsystem) shown in Figure 7.19(a) and the full implementation of the control system shown
in Figure 7.19(b). As with the simulator, a MATLAB script was developed to load in parameter
values and automate the running of the Simulink model for each study participant.
As an aside, one change in the actuator model shown in Figure 7.19(b) (when compared to
the model shown in Figure 7.7 and Figure 7.14) is the removal of the input voltage saturation
block. This is because the PID Controller block was configured internally to use Clamping as
an Anti-Windup method (which can be noticed by the small saturation symbol located on the
PID Controller block output). Windup is a phenomenon that can occur in systems that have
saturation limits on their output, either a physical limit (for example a device not being able to
move due to a mechanical/electrical limit being reached) or an artificial limit (such as a safety limit
placed in software) [289, 311, 312]. In the case of the wearable upper-limb brace simulator, the
saturation in question that caused the windup behaviour is the position safety limits applied to the
brace output. When the system is trying to move, but is stuck at the maximum output position
dictated by the saturation limit, the steady-state error of the system (measured using the integral
component of the PID controller) grows continuously. When the system can finally move in the
other direction, away from the saturation region, it first needs to reduce the erroneously large
steady-state error output being contributed by the integral term of the PID controller before any
motion can begin (as the integral term is still commanding the device to move forward towards
the saturation limit in a failed attempt to remove steady-state error). This can result in what
(a) Collapsed
(b) Expanded
Figure 7.19: The Simulink implementation of the TWSC developed for this project. (a) shows the
control system with the upper-limb brace model collapsed into a subsystem (repre-
senting the system using the traditional closed-loop feedback appearance shown in
Figure 7.18). (b) shows the TWSC integrated into the upper-limb brace simulator
model directly.
appears to be a delay in movement, as the internal calculations of the PID controller (described by
Equations 7.51 and 7.52) work to overcome the overly-large integral component, with motion only
starting once the proportional and derivative error components have finally grown large enough
to counteract the persistence of the integral component [289, 311, 312]. Having device movement
lag behind the reference input signal is undesirable, so Anti-Windup measures were developed to
combat such behaviour. Clamping is a very popular Anti-Windup technique and works using two
approaches [311, 312]. First, the output of the PID controller is limited using saturation, which
is why it replaced the previously implemented voltage saturation block. Since the PID output is
fed directly into the actuator electrical model transfer function block, as seen in Figure 7.19(b), it
was decided that the motor maximum input voltage would be a sufficient choice as the Clamping
limit. The PID output saturation helps to avoid the situation where the controller output grows to
unmanageably high values during windup. The second thing Clamping does is turn off the integral
component (effectively using a Ki value of zero) when the output saturation is reached, preventing
the steady-state error from growing continuously [311, 312]. Together, these methods can be used
to effectively counter windup in the control system, therefore Clamping was implemented for the
TWSC using the internal settings of the PID Controller block.
Once the gain switching mechanism was implemented in Simulink, the next step was to deter-
mine how to best generate a control signal to use with the Multiport Switch block. This is the
place in the control system where the machine learning was actually incorporated, as the idea was
to have the output of a task weight classifier activate the switch and change the PID gains. Several
design decisions had to be made when determining the best way to facilitate this integration.
The first question that had to be answered was which classier to actually use for the TWSC.
Throughout this entire work, many different EEG–EMG fusion-based task weight classifiers, using
various different fusion/classification techniques, were implemented and evaluated. Theoretically,
any of them could be used with the TWSC; however, to simplify its implementation it was decided
to focus on only one to start. In the end, it was decided that the SVM-based WA-50/50 classifier
(discussed in Chapter 4) would be used for this project. This decision was made for two reasons.
Firstly, out of all the EEG–EMG fusion methods evaluated in this work, the WA-50/50 method
had the highest mean accuracy (obtaining an overall accuracy of 83.01 ± 5.21% for a chance level
of 33.33%). It makes sense that a device would be deployed using the best performing model, so it
seemed the most reasonable choice to use this classifier. It should be noted, that this “comparison”
of model accuracies was done purely by looking at their overall accuracy scores and choosing the
highest one; no statistical analysis was performed to formalize this comparison. It is possible that
other EEG–EMG fusion classifiers presented in this work may have had very similar performance
to the WA-50/50 model. Future work could focus on a formal analysis of the difference between
SVM-based and CNN-based EEG–EMG fusion classifiers, and investigate how their particular
behaviour affects the TWSC (e.g., are they more/less sensitive to different sources of error and
how does this affect the control system?), but for this work it was decided to focus the evaluation
on just investigating the TWSC itself.
The second decision made when integrating the classifier into the TWSC was how to incorporate
the actual classification model into Simulink. It would be possible to actually load the trained SVM
classifiers (which were saved as ClassificationECOC objects in MAT files) into the Simulink model
using a custom code block; however, this implementation would be somewhat unwieldy. To obtain
a task weight prediction using a ClassificationECOC object would also require the previously
calculated EEG/EMG features for each subject to be loaded into Simulink (or, if actually trying to
fully recreate the entire process, the raw EEG/EMG signals would need to be loaded into Simulink
and then the features calculated inside the simulator). This process would add a significant amount
of time and complexity to the simulation, with no real benefits. Since the simulation is already
an offline analysis, it was decided to preprocess the WA-50/50 model predictions and use them to
make a prediction control signal that will trigger the Multiport Switch block to the desired PID
gains (in the same manner that the dynamic weight switch control signal was pre-calculated and
merely loaded into the simulator during initialization). This vastly simplified the implementation
of integrating the WA-50/50 classifier into the TWSC and sped up the simulation considerably
by avoiding the need to preload large amounts of data. Future work should consider a real-
time evaluation of an EEG–EMG fusion classifier incorporated into a control system, but for the
purposes of this work it was considered outside the scope of the project.
With these decisions made, the next step in the implementation of the TWSC was the gen-
eration of the prediction control signal using the SVM-based WA-50/50 classifier. The following
subsection will discuss this process, and describe the various methods investigated to achieve this
goal.
7.4.3 Generating the Classifier Prediction Control Signal
As mentioned above, it was decided that the EEG–EMG fusion-based machine-learning classifier
would be incorporated into the TWSC using a pre-generated control signal comprised of the output
task weight predictions. While simple in principle, there are several steps that needed to be taken
to transform the predictions into something more suitable for use with Simulink. One major
challenge comes from the fact that a Simulink simulation is a time-based process, while the SVM
classifier is inherently ignorant of any time series information. At its core, the trained classifier
merely accepts EEG/EMG features and outputs a task weight prediction (i.e., a numerical label
of 0, 1, or 2, which corresponds to a weight of 0 lbs, 3 lbs, or 5 lbs); this output includes no
indication of when this prediction takes place in the motion trajectory. This fact needed to be
rectified, since all Simulink signals require an included set of time measurements to tell the program
how the signal changes with respect to the simulation clock. Luckily, this could be calculated for
the classifier predictions using the previously obtained trigger signal timestamps, combined with
an understanding of how the signal processing procedure segmented the windowed EEG/EMG
features. In general, the procedure followed went thusly: take the EEG/EMG feature matrix and
input it into the WA-50/50 EEG–EMG fusion-based SVM classifier to obtain a list of task weight
predictions, use the trigger signals recorded during data collection to calculate timestamps for
each prediction that synchronizes with the simulator motion trajectory, convert the predictions
and timestamps into a time-domain control signal suitable for Simulink, and, finally, export and
save these signals to be loaded later during the TWSC Simulink experiments using the upper-limb
brace simulator. This process is summarized in Figure 7.20.
In regards to the generated predictions, it was decided to use EEG/EMG features from all
three motion repetitions for this process, to increase the amount of data available for the TWSC
experiments. As mentioned in Chapter 4, the EEG/EMG signals collected during the first and
second elbow flexion–extension repetitions were used as training data to create the SVM classifier,
and the EEG/EMG signals collected during the third elbow flexion–extension repetition were used
as testing data to evaluate classifier performance. Normally, in a machine learning evaluation,
training data would never be used to generate predictions because the model has already been
Figure 7.20: The procedure implemented to generate the classifier prediction control signals used
to update the TWSC. Features were inputted into the WA-50/50 EEG–EMG fusion
classifiers to obtain task weight prediction labels, which were then converted to a
time-domain signal through the calculation of prediction timestamps based on the
trigger markers recoded during the data collection procedure.
optimized to this dataset. The model will be over-fitted to these inputs, which results in artificially
high prediction accuracy that will not reflect the true performance of the classifier. This is why
unseen data that have been totally excluded from the training process should be used to test
performance. The work being done in this project, however, is not a machine learning evaluation
of the WA-50/50 SVM classifier (and in fact this has already been covered extensively in Chapter
4), so it was decided that this phenomenon is of less importance to the control system evaluation
being performed here. It was felt that the potential benefit of additional motion repetitions in the
TWSC experiments out-weighed the taboo of breaching conventional machine learning practices
in this instance. It should be noted that the consequence of this decision is that the obtained
predictions for the first two motion repetitions will be significantly more accurate than the ones
obtained for the third motion (due to the over-fitting present in the input features used for model
training). When considering this effect however, it could result in an interesting opportunity for
evaluating how classifier performance impacts the TWSC. As discussed in Chapter 4, the WA-
50/50 EEG–EMG fusion task weight classifier is still in its initial development stages and requires
further refinement to improve its performance before it could suitably be deployed into a real
wearable mechatronic device. No company would (or perhaps should) sell a system that does
not operate using near-prefect accuracy for fear of potential injury/harm that the device failure
may cause. The predictions obtained using the training dataset presented an opportunity to see
the TWSC operating with a classifier closer to the performance one would expect from an actual
device (i.e., very few misclassifications). The predictions from the third repetition can then be
used to observe the TWSC operating when classifier performance begins to degrade (i.e., more
misclassifications), providing a chance to see how robust the TWSC is in such a situation. Since
many factors can cause the performance of bioelectrical signal-based classifiers to degrade (signal
noise, electrode motion artefacts, muscle fatigue, etc.), this is an important aspect to consider
when looking to integrate machine-learning classification models within device control systems.
7.4.3.1 Calculating Timestamps
After task weight predictions for each motion trial were obtained from the WA-50/50 EEG–EMG
fusion classifier, their respective timestamps needed to be calculated so that a Simulink-compatible
time-domain signal could be generated. Of particular importance during this process was ensuring
that the resulting prediction control signal is synchronized with the input motion trajectories
calculated during the simulator development process (discussed in Section 7.3.5). This prevented
any issues occurring during the simulation that could be caused by mismatched signal lengths
and/or simulated events erroneously occurring at different time points in the two signals.
To calculate predication timestamps (as with many other timestamp calculation procedures
in this work), the previously collected trigger markers indicating the beginning/end of an elbow
flexion–extension repetition were utilized (for a visualization of trigger marker placement, see
Figure 7.11). For the purposes of the explanation here, the portion of the trajectory between
a start/end trigger (i.e., one elbow flexion–extension repetition) will be referred to as a Motion
Region (MR). Recall from Chapter 4 that the task weight classifiers were trained specifically using
only data obtained during movement (meaning the MR portions of the EEG/EMG signals). This
means that, for the purposes of the TWSC, the only times that predictions can be supplied by the
SVM classifier are between the start/end trigger markers. Put another way, this is analogous to
a setup where the classifier is only “on” while the subject is moving. This approach was adopted
purely as a consequence of the training procedure followed in the previous project (since it would
not be feasible to create an entirely new continuous classification model just for use in this work),
and no claims can be made about the superiority of this method opposed to an always-on classifier.
Future work could look at how the rate of supplied predictions affects TWSC performance, and if
a trade-off exists between an intermittent, simplified, classification task verses a continuous, but
more challenging, classification task. That is not to say, however, that the approach of a motion-
only classifier is not a potentially realistic scenario for a mechatronic wearable device control
scheme. Task-specific classifiers that focus on a smaller scope of data tend to perform better
than highly generalized models, so it may be more efficient to make use of multiple smaller-scope
classifiers instead. This is a situation where the motion classification models evaluated in Chapter
3 can provide some benefit. These simpler binary classification models can run continuously to
check that motion has started and, once it has, they can send a signal to turn on the task weight
classifiers. Since the main goal of the work here was to provide an initial evaluation of the TWSC,
it was decided to not include this hypothetical multi-classifier setup into the developed control
system to keep the implementation simple, and to reduce the number of variables included in the
evaluation; however future work could investigate the feasibility of this approach.
The other piece of information used during the calculation of prediction timestamp was knowl-
edge of the windowing procedure used during EEG/EMG feature calculation. As mentioned in
Chapter 4, the EEG/EMG signals were segmented using a sliding window with a length of 250 ms
and an overlap of 50% (125 ms in length). Even though the simulator was performing an offline
analysis of the TWSC, and it was decided that the SVM classier data would be pre-processed
for their inclusion in the controller, an effort was made to still maintain some aspects of realistic
classifier behaviour. Since the model was trained using features calculated using 250 ms windows,
if the classifier was actually implemented in a real device the system, as is, would only be able to
update the classifier prediction every time a new signal window was filled. Due to the 50% overlap
between windows, this means that the predictions would update every 125 ms (since half of the
“old” signal window is included in the overlap, only 125 ms of “new” signal needs to be recorded
before a new feature can be calculated). One exception for this update rate is the very fist window,
which will take the full 250 ms before a prediction can be supplied (since there is no past signal that
overlaps with the first window). Delays like this can have major consequences on control system
performance, so it is important to consider for the TWSC evaluation. It should be noted that this
approach is working under the large assumption that the only source of classification delay is due
to signal windowing (therefore assuming that the task weight prediction is available instantly after
the duration needed to record a full window length), which is not likely to be the case in a real
device. In actuality, there are many other computational processes that need to happen at the end
of every window (filtering the EEG/EMG signals, calculating features, processing features with
the classifier to output a prediction), all of which have an associated execution time cost that will
further increase delay. While this is an important factor to examine, it is outside of the scope of
the offline analysis being performed here, as the time cost of these computations is hard to real-
istically estimate without detailed information of the hardware being used to perform each task.
To simplify implementation for this work, it was decided to only approximate the classifier delay
using the known windowing details and leave a real-time execution/evaluation of the classifier for
future work.
Knowing the prediction timestamps would be based on the trigger markers and windowing
procedure, they could then be calculated using a simple procedure. The first step was to shift
the recorded MR start times (tMRi,start ) so that the first trigger marker (tMR1,start ) will act as
0 s (to align the prediction timestamps with the start time convention used for the calculated
motion trajectories). As discussed above, the first prediction cannot occur until a full window has
passed (250 ms). This means that, to properly calculate the timestamp for the initial prediction
of each MR, 250 ms needed to be added to the shifted trigger marker denoting the start of an
elbow flexion–extension repetition. Executing this will provide the starting prediction timestamp
for each of the three MRs, according to the following equation:
∆tMRi,start = (tMRi,start − tMR1,start ) + 250 ms, (7.53)

where tMRi,start is the motion start trigger marker timestamp for the current elbow flexion–
extension repetition (and i is an index denoting the number of the current MR being evaluated),
tMR1,start is the starting trigger marker timestamp for the first MR, and ∆tMRi,start is the first
prediction timestamp for the current MR (shifted so the start of MR1 happens at 0 s).
After the initial prediction for each MR (which occurs 250 ms after the onset of motion, denoted
by the trigger marker), a subsequent prediction will occur every 125 ms until the end of the MR
(also denoted with a trigger marker). This rate occurs because of the 50% overlap employed with
the 250 ms sliding window (a new full window is available every 125 ms since it uses 50% of the
previous window). To know when to end the 125 ms pattern of addition, the final prediction
timestamp (measured relative to the total motion trajectory timescale) for each MR needed to be
calculated. Initially, it may seem correct to just use the MR ending trigger marker time directly for
this; however, this can result in mismatched timestamps due to how the windowing was performed
during signal segmentation. Since the EEG/EMG signal recording lengths for each subject were
variable (there was no fixed time duration for recording data from the participants, and their trial
lengths would vary depending on how long it took them to execute the protocol), it was highly
unlikely that a round number of windows could be obtained from their signals. In most cases there
would be a small segment of signal leftover after applying the sliding window that was not long
enough to actually fill a 250 ms window. In this situation, the leftover signal portion was merely
discarded and not used during feature calculation; however, it means that the final prediction
from the classifier for each MR does not line up with the ending trigger marker. To find the actual
prediction end time, the total number of obtained predictions for the current MR (NPredict ) was
used instead. Knowing that the initial prediction timestamp is given by Equation 7.53, and that
a new prediction will occur every 125 ms following that, the ending prediction timestamp for each
MR (∆tMRi,end ) was found using the following equation:
∆tMRi,end = (125 ms)(NPredict − 1) + ∆tMRi,start , (7.54)
where i, once again, is an index denoting the number of the current MR being evaluated. Note
that the purpose of subtracting one from NPredict was to correct the number of predictions in the
calculation for the current MR, to consider the fact that the first prediction differs from the “every
125 ms” pattern. Since the initial prediction occurs after a full 250 ms window (which is found
using Equation 7.53) it cannot be included in the 125 ms multiplication, and instead is just added
to the timestamp. This means that the multiplication must then apply to one less than the total
number of predictions obtained for the current MR, since the first prediction time has already
been included in the calculation.
Once the timestamps had been calculated for each task weight classifier prediction, they were
combined with the prediction task weight labels obtained from the WA-50/50 EEG–EMG fu-
sion classifier to form the foundation of the prediction control signal used to activate PID gain
switching in the TWSC. This version of the signal could not be used as is however, and required
further processing to finish the signal generation procedure, which will be discussed in the following
subsections.
7.4.3.2 Initializing the Prediction Signal
After calculating timestamps for the task weight predictions supplied by the classifier, a switching
control signal was made that allows the TWSC to select gains depending on the current load.
However, one edge case still needed to be resolved: how should the TWSC operate in situations
where the classifier cannot yet provide a prediction? This issue would occur in start-up situations
where not enough time had passed yet for the system to record the requisite 250 ms of EEG/EMG
signals needed as inputs to the classification model, forcing the TWSC to essentially operate “blind”
as there is no task weight prediction to draw upon. To reduce the effects that this uncertainty
may cause, a set of rules were established regarding how the prediction control signal should be
initialized. This way, even if the actual task weight is not yet known, the device will still operate
in a consistent manner, which will hopefully prevent adverse events during use.
There were two initialization conditions that needed to be considered for the prediction control
signal. The first occurs at device start up, before any EEG/EMG data have been collected at
all. In the timescale used by the simulator, this would be the prediction value occurring right at
0 s, and will not be able to be updated until after the first 250 ms window of signal collection is
complete. For the purposes of the simulated experiment, each motion trial recording (i.e., set of
three elbow flexion–extension repetitions recorded continuously) is treated as its own self-contained
use of the brace, thus the device start up condition will occur at 0 s for every motion trajectory
evaluated. It was decided for this initialization to always assign a task weight prediction value of
0 lbs (empty handed). Thinking logically, this seemed to be the safest assumption to apply when
making a guess about what weight the user is holding. It seems like a better idea to assume the
user is initially holding nothing and potentially be wrong, which will result in an under-powered
brace that supplies less torque (since it thinks that it does not need to compensate for an external
load), as opposed to assuming that they are holding a weight and supplying more torque than is
necessary (which, in extreme cases, could result in discomfort/injury as the user’s arm is forcefully
moved). Also, speculating further, powering up the device empty handed feels like a more common
use case. Imagining the hypothetical use of the device, it is likely that the user would don the
brace, turn it on, and then begin performing ADL/picking up objects. While it is not impossible
that someone may put on the brace while holding an object in their hand, it does not feel like a
procedure most people would be likely to follow the majority of the time. For these reasons, it
was decided to always initialize the prediction signal at 0 s with a task weight prediction of 0 lbs.
An example prediction signal showcasing this behaviour can be seen in Figure 7.21.
The second initialization condition that needed to be considered was what task weight as-
sumption to use at the start of subsequent motion repetitions in the motion trial. As mentioned
previously, because the EEG–EMG fusion-based SVM classifiers developed in Chapter 4 were only
trained using signal data collected during movement, the classifier used in the TWSC is essentially
“turned off” in between motion repetitions. Just like with the power-up initialization condition,
it takes 250 ms of signal recording to fill the first window of a new motion repetition (meaning no
updated task weight prediction can be supplied to the controller until then). A decision needed to
be made about what task weight assumption to use at the onset of a new movement, until the first
real prediction can be supplied to the TWSC. Unlike the start-up initialization condition however,
this assumption did not need to be a blind guess, since there are some historical prediction data to
draw upon. It was decided for the prediction control signal to use the last task weight prediction
from the previous motion repetition as the initial value for the new repetition. Hence, for exam-
ple, if the final window of the first elbow flexion–extension repetition in the motion trial resulted
in a task weight prediction of 3 lbs, the prediction control signal will continue to supply a 3 lbs
prediction value until after the first 250 ms window of the next elbow flexion–extension repetition
(upon which point the value will change to whatever updated prediction is obtained from the new
window of signal data). A visual example demonstrating this initialization method can be seen in
Figure 7.21. The rationale behind this approach is based on the assumption that if the user was
previously holding a certain load, it is likely that they will have continued holding the same load.
Of course, it is possible that a user will sometimes change loads in between movements, as they
cannot hold the same object forever (an example of weights varying in between motion could be
something like a therapist changing out a patient’s dumbbell in between exercise repetitions during
a rehabilitation therapy session). However, it seemed that using this assumption would at least
provide an informed estimation of which task weight value to initially supply. Instead of blindly
guessing a new value for each motion repetition, or assuming a fixed value for all initializations
(like how the start-up initialization always assumes 0 lbs), this method at least attempts to apply
an educated guess about the task weight based on previous use of the device. While not a perfect
solution, it seems like a reasonable approach given the limitations of the current implementation
of the TWSC.
5 lbs Classifier Predictions

Task Weight Prediction
Assumed Predictions
3 lbs
0 lbs
0 250 375 500 tMR2,start tMR2,start +125 +125
Time [ms]
Figure 7.21: An example prediction control signal highlighting the assumptions made during ini-
tialization. Actual classifier predictions are represented by blue circles, while assumed
prediction values, assigned before the classifier has processed the first window of EEG
and EMG data, are represented by orange squares. At device start-up (before the
first window), the system assumes a task weight of 0 lbs. At the start of a new motion
repetition, the system assumes the value of the last task-weight prediction until a new
window of data has finished being collected.
With the initialization of the prediction signal completed, it could now be used to control
the gain switching in the TWSC. It should be highlighted that the initialization assumptions used
were chosen primarily based on intuition when considering hypothetical usage habits for a wearable
mechatronic upper-limb brace. No claims can be made about these approaches being objectively
superior to other methods (as that was outside the scope of this project) and future work should
focus on a quantitative evaluation of different initialization techniques based on system perfor-
mance and data-driven investigations of user habits/behaviour during the operation of wearable
mechatronic devices. For the purposes of completing an initial evaluation the TWSC, these initial-
ization methods seemed sufficient however, so work continued onto the next steps of implementing
the control system.
7.4.4 Debouncing the Prediction Signal
Before utilizing the classifier predictions in the TWSC, one more step was taken prepare the switch-
ing control signal for the simulator: filtering. As is well known with machine learning systems,
models will occasionally output a misclassification where an incorrect prediction is supplied. Ide-
ally, a classifier would be trained well enough to perform as close to 100% accuracy as realistically
possible; however, it is naive to assume that this will always be the case (or that it will even be
obtainable in many applications). Therefore, it is important to consider methods that can com-
pensate for this error in the TWSC, since it can have consequences on the performance/stability
of the device during use.
Looking at Figure 7.22, an example of a misclassification can be observed. The predictions
remain at a stable value for several windows, before changing suddenly for one window, then
back to the previous prediction. If nothing was known about how the system operates, then one
may need to consider this sudden change as fact; however, knowing how an upper-limb robotic
brace would typically be used can allow a relatively strong educated guess to be made that this
is a misclassification. It is highly unlikely (in fact, borderline impossible) that the user would be
able to completely switch the weight they are holding in 250 ms, then immediately switch back
to the original weight in another 250 ms (all without somehow breaking from their set motion
trajectory). Thus, it can be concluded with relative confidence that this is a misclassification.
Note that the example shown is a very simple scenario (a short value change over one window),
but misclassifications can exhibit more complex behaviour as well (switching between multiple
values and/or switching for longer, before returning to the true stable value); however, they can
still be characterized as unrealistically fast and erratic changes in task weight that are not feasibly
achievable for the user of an upper-limb brace during operation.
5 lbs Raw Predictions

Debounced Predictions
Brief Misclassification
3 lbs
0 lbs
0 250 375 500 625 750
Time [ms]
Figure 7.22: An example prediction control signal showcasing the use of debouncing techniques
to remove a brief misclassification. Raw (un-debounced) predictions supplied from
the classifier are represented using blue circles. The prediction values after being
debounced are shown using green rings. The brief misclassification is represented by
a red “X”. Using debouncing, the misclassification is ignored by the system, as it
is assumed that such a sudden, and short-lasting, change in task weight would not
be possible for the user to do in a realistic scenario, given the indented usage of the
device.
One solution to this problem is to handle the prediction control signal in similar manner to
an electrical switch contact signal in microcontroller programming and implement debouncing
methods. Debouncing is a technique where, if it is known that instability will be present in a
changing signal (for example, fluctuations in voltage readings as a button is being pressed before
full contact with the leads is established), the fluctuations are ignored until the signal has stabilized
(only then considering it to be a true change of state) [373]. An example of this is demonstrated
in Figure 7.22, where the misclassified prediction is ignored by the system and the previous stable
prediction value is assumed instead (since it is understood, due to knowledge of the device use
cases, that such an action is impossible). Adopting this approach will help reduce the number of
misclassifications erroneously utilized by the control system, resulting in better performance. It
also will help avoid situations where the dynamics of the system rapidly oscillate due to impossibly
fast changes in assumed task weight, which can cause issues with the stability of the control system.
Since fluctuations are discarded until the prediction values have stabilized, a more generous window
can be used for the switching frequency of the TWSC (leading to smoother operation). The trade-
off for using debouncing methods is their effect on system response time. Due to the inherent
nature of their implementation (waiting a set number of cycles to ensure a value change is ”real,”
providing time for fluctuations to stabilize) they will introduce a certain amount of delay into
the system, which, if too high, may impact the user’s ability to control the device comfortably.
There is also the risk of missing real task weight change events by assuming them to be incorrect
predictions. Knowing how the device will be used, it is highly unlikely that an extremely fast task
weight oscillation would occur; however, there is always some possibility that it may have been a
true event. The debouncing methods used will need to be implemented in a manner that ensures
that device usage edge-cases are considered and that response time is balanced with accuracy.
After considering these trade-offs, it was decided that debouncing would be implemented into
the TWSC controller to reduce misclassifications of task weight in the EEG–EMG fusion classi-
fier. The approach of utilizing debouncing techniques to filter classifier outputs has been sparsely
demonstrated before in BCI studies focusing on communication and input command detection
[374–376]; however, it does not appear (to the author’s best knowledge) that it has ever been ap-
plied to an application where a classifier is integrated into a device control system. Likewise,
previous demonstrations of prediction debouncing have been implementing using a refractory
period-based approach, meaning that the first observation of a value change is considered true
and then a delay is enacted where no value changes are evaluated until a certain time has passed
[374–376]. This debouncing method does not actually address misclassifications and instead fo-
cuses on reducing the effect of unstable behaviour after a value change, making it unsuitable for
the application outlined in this work. Given the novel use here of classifier debouncing for system
control, and the unique constraints/trade-offs present when utilizing this technique to drive a de-
vice in real time, the decision of how to best implement debouncing in the TWSC was unclear.
Therefore, two new methods were devised for this work to facilitate an evaluation of prediction
control signal debouncing, and testing was performed to compare both techniques (as it was un-
clear which would provide the best result). The following subsections will detail their debouncing
approaches and implementation within the TWSC. For both debouncing methods, it should be
noted that it was decided to always treat the first prediction after the start-up initialization of 0 lbs
as true and use it immediately in the TWSC (essentially skipping debouncing for the first window).
Since the 0 lbs initialization is essentially a guess of the current task weight to use while the first
250 ms window of EEG/EMG signals is being recorded for the classifier, it was assumed that the
first output prediction will likely be a more accurate estimation of weight and therefore should be
used immediately (as opposed to waiting several windows to debounce this initial change). This
increases the risk of using a misclassification at the start-up of the device for control; however,
the 0 lb initialization is also likely a misclassification giving the speculative nature of its use, so it
was felt that the risk is justified. For this reason, the 0 lb initialization is also excluded from the
debouncing calculations, as it is not a true classifier prediction value and could incorrectly bias
the debouncing method. After the first prediction window, debouncing then begins and prediction
values need to be stable (evaluated using the methods discussed below) before the TWSC will
update.
7.4.4.1 Hold-Delay Debouncing Method
The first debouncing technique devised for the TWSC was referred to as the Hold-Delay method.
The idea behind this approach is that the same prediction value needs to be maintained for a
certain number of windows for the TWSC to consider it a true change in task weight. If the
classifier output is fluctuating and does not hold the same value for a certain delay period it is
likely a classification error, therefore the change should be ignored. In this case, the system will
continue to use the last “true” prediction value (until such time that the prediction changes again
and holds a new value for the specified time period). A visual representation demonstrating how
this method operates can be seen in Figure 7.23.
Implementing the Hold-Delay debouncing method was done by comparing the value of each
new prediction to the value obtained during the previous window. When a change was detected,
the prediction from the next window was compared to the new value to see if this change was
maintained. This process would repeat for each successive window until the change was found to

3 lbs
0 lbs
0 250 375 500 625 750 875 1000 1125 1250 1375 1500 1625 1750
Time [ms]
Figure 7.23: A visual representation of the Hold-Delay debouncing method being applied to a pre-
diction switching control signal, showcasing how it operates. Raw (un-debounced)
predictions supplied from the classifier are represented using blue circles. The pre-
diction values after being debounced are shown using green rings. In this example, a
delay value of 3 was used. At 625 ms, the prediction of 5 lbs is not kept because this
value was not maintained for 3 windows. The change in prediction value at 1000 ms is
not applied until 1250 ms, as this is the third repetition of the same prediction value
therefore the Hold-Delay debouncing method considers it to be a true task weight
prediction (this is demonstrated again with the prediction value change at 1500 ms).
be maintained for the specified number of windows, upon which point the value of the debounced
prediction signal was updated to the value of the newly observed change. The concept of a delay in
this method comes from the fact that the debounced prediction signal value does not get updated
until after the hold period, and this change is not retroactive. In a offline analysis, it would be
easy to go back to the previous windows and update their values once it has been established that
the change in value represented a “true” task weight variation (since debouncing was done as a
pre-processing step during prediction control signal generation before running the simulations);
however, this action is not possible to perform in a real-time system. If the TWSC was actually
commanding a real device, it would have already made control decisions based on the predictions
that maintained the old task weight value during the hold-delay windows (while the system is
determining if the change was real or not). Therefore, this behaviour should be reflected in the
approach used in this work to properly evaluate the debouncing methods under realistic conditions.
As discussed previously, debouncing exhibits an inherent trade-off between signal conditioning
and system response time. In this method, that trade-off appears in the choice of which delay
value to use (i.e., for how many windows does the value need to be maintained before the change is
considered to be true?). A longer delay period will result in a smoother prediction signal, since only
very stable value changes will be accepted; however, it will also slow down the ability of the system
to react to changes in task weight (since the prediction value will not be actually updated until it
is held for the entire delay period). As the TWSC is a new approach to wearable device control
systems, and since classifier-output-based debouncing for a robotic control system has not been
demonstrated before in the literature, the decision of which delay value optimally addresses this
trade-off has yet to be investigated. To facilitate this evaluation, multiple delay values were used
so that a comparison of their performance could be performed. As a starting point, it was decided
to use delay values of 2, 3, and 4 in this study. Note that these values specifically correspond to
the number of windows that the prediction must be held before a change is enacted (for example,
using delay of 3 means that the third repetition of a new prediction value is when the system will
consider it to be true and update the value of the debounced prediction signal accordingly). An
example of the difference between delay values can be seen in Figure 7.24.
7.4.4.2 Majority-Vote Debouncing Method
The second debouncing technique devised for the TWSC was referred to as the Majority-Vote
method. This approach worked by sliding a window along the prediction signal and looking at the
values observed inside it at each time point. For every step, the most commonly occurring value
in the voting window was considered to be the true task weight prediction and was used in the
debounced signal. If a tie occurred between values in the voting window, it was assumed that the
classifier outputs were too unstable during that duration so all values in the voting window were
ignored and the previous true prediction value was used in the debounced signal instead. A visual
representation demonstrating how this method operates can be seen in Figure 7.25.
As with the Hold-Delay debouncing method, parameter decisions needed to be made when
implementing the Majority-Vote debouncing method to balance its performance trade-offs. In this
case, the decision to be made was what size of voting window should be used for debouncing (i.e.,
how many prediction values should be included in the vote). A larger window will result in a
smoother signal; however, if the window is too large it may cause shorter events to be missed as
they get out-numbered in the voting process. As a starting point, it was decided to evaluate voting
window sizes of 3, 4, and 5 prediction values.
5 lbs Raw Predictions 5 lbs Raw Predictions


Debounced Predictions Debounced Predictions
3 lbs 3 lbs
0 lbs 0 lbs
0 250 375 500 625 750 0 250 375 500 625 750
Time [ms] Time [ms]
(a) Hold-Delay value of 2 (b) Hold-Delay value of 3

3 lbs
0 lbs
0 250 375 500 625 750
Time [ms]
(c) Hold-Delay value of 4
Figure 7.24: An example demonstrating the effect that changing delay value has on the output
of the Hold-Delay debouncing method. Figures (a), (b), and (c) show the same
prediction control signal being debounced using delay values of 2, 3, and 4, respec-
tively. Raw (un-debounced) predictions supplied from the classifier are represented
using blue circles. The prediction values after being debounced are shown using green
rings. Changing the delay value modifies how many repetitions of the same prediction
value need to be observed before the system will consider it to be a true task weight
prediction. A longer delay value will cause the system to react slower to changes in
task weight, but may result in more stable performance (fewer oscillations between
misclassified task weight predictions).
One challenge presented by this method was what to do at the beginning of a trial before enough
signal windows had been collected to generate the number of classifier predictions needed to fill
the voting window. As mentioned in Section 7.4.3.2, the initialization guess of 0 lbs is skipped in
debouncing, meaning that for a voting window of size 3 it would take 750 ms (three signal windows
of 250 ms) before the classifier could output the required number of predictions. To address this,
the voting window was implemented in a manner where it could be dynamically resized during the
start of a trial to utilize the maximum number of predictions currently available, until such time
that it was possible to use the desired voting window size. Put another way, the voting window

3 lbs
0 lbs
0 250 375 500 625 750 875 1000 1125 1250 1375 1500
Time [ms]
Figure 7.25: A visual representation of the Majority-Vote debouncing method being applied to a
prediction control signal, showcasing how it operates. Raw (un-debounced) predic-
tions supplied from the classifier are represented using blue circles. The prediction
values after being debounced are shown using green rings. The voting window (with
a length of 3 predictions in this example) is represented by the red dashed lines. At
500 ms, the prediction of 5 lbs is not kept because this value loses the vote to the
two predictions with a value of 3 lbs. At 1375 ms, no consensus can be reached since
the three predictions in the voting window have different values; thus, the debounced
value from the previous successful vote is used instead.
always starts out with a size of 1 and grows with each time step until the specified size has been
reached, upon which point it maintains this size and begins moving along the prediction signal.
An example demonstrating how the dynamic start-up window resizing was performed can be seen
in Figure 7.26. Using this approach ensures that the debouncing procedure is only ever performed
on actual classifier outputs (as opposed to filling the voting window with dummy values at the
start), which should result in more consistent behaviour, as the task weight predictions are only
ever influenced by the subject’s EEG/EMG signals, as intended, instead of by bias introduced by
arbitrary guess values used to artificially pad the voting window.
Once prediction signal debouncing had finished, the implementation of the TWSC within the
simulator was complete, and it could begin to control the hypothetical upper-limb robotic brace.
Before experimentation could begin however, one final step remained: optimizing control system
performance by tuning the PID gains. The following subsection will discuss this process and
explain the procedure utilized for this project.


3 lbs 3 lbs
0 lbs 0 lbs
0 250 375 500 625 0 250 375 500 625
Time [ms] Time [ms]
(a) First prediction (b) Second prediction


3 lbs 3 lbs
0 lbs 0 lbs
0 250 375 500 625 0 250 375 500 625
Time [ms] Time [ms]
(c) Third prediction (d) Fourth prediction
Figure 7.26: A demonstration of the voting window dynamically resizing at startup until the full
window length can be used. Figures (a) to (d) show the first four classifier outputs
being added to a prediction control signal in sequence, and debounced using the
Majority-Vote method as each new value is added. Raw (un-debounced) predictions
supplied from the classifier are represented using blue circles. The prediction values
after being debounced are shown using green rings. The voting window (with a length
of 3 predictions in this example) is represented by the red dashed lines. In (a) there are
not enough windows of EEG/EMG data collected to have more than one prediction in
the signal, so the vote window has a length of 1. In (b) a second prediction has been
added, so the vote window grows to a length of 2 to include both classifier output
values. Finally, in (c) a third prediction has been supplied by the classifier, so the
vote window grows to the desired length of 3 and begins sliding across the prediction
control signal as new task weight predictions are added (shown in (d)).
7.4.5 Tuning Task Weight Specific PID Gains
Tuning the gains of a PID controller is a crucial step in implementing the control system and
will likely have an immense effect on its performance. The process of PID gain tuning involves
selecting suitable values for Kp , Ki , and Kd (used as shown in 7.51 and 7.52) to reduce error and
achieve the desired system response. As an aside, it should noted that the PID equation used by
Simulink differs slightly from what is shown in Equation 7.52 and instead uses an implementation
with a filtered derivative term, as follows [362]:
Ki sN
U (s) = Err(s) Kp + + Kd , (7.55)
s s+N
where N is the filter coefficient. The purpose of this term is to first apply a low pass filter to
the error signal, removing high-frequency noise before applying the derivative gain. The coeffi-
cient itself controls the cut-off frequency for the low pass filter. This extra step is taken because
small oscillations caused by high-frequency noise could be misinterpreted by the controller as an
extremely fast change in error, which will prompt an erroneously aggressive response from the
derivative component of the PID control system. To avoid this phenomenon, it is common prac-
tice to use a filtered derivative PID implementation, as Simulink has done, at the cost of adding
a fourth parameter that also requires tuning. Nevertheless, with either implementation method,
proper selection of these values is the main goal when optimizing PID control and can end up
being a deceptively challenging task, depending on the complexity of the driven system.
Regarding tuning methods, many formulaic procedures for PID tuning have been proposed in
the literature. One of the most popular is the heuristic Ziegler–Nichols method, which calculates
gain values based on ratios of the maximum P gain at system instability and the system oscillation
period [39, 298, 311, 363, 365]. This method, while suitable in many applications, can often result
in undesirable behaviour, such as system overshoot (which is not acceptable in a wearable device
for safety reasons) [298, 363, 365]. Since optimal gain values are so dependent on the individual
behaviour of the system being controlled, it is hard to find a “one-size-fits-all” approach to tuning.
This has led to educated “trial-and-error” guessing often being the dominant method used when
selecting gains in many cases. The main drawback to manually tuning PID gains this way is that
it can be a time consuming and tedious procedure, especially if a large number of systems require
optimization. For the 32 subjects used in the upper-limb brace simulator (combined with the three
task weights scenario used during the study, since the use of Gain Scheduling in the TWSC requires
task-weight-specific gain values), it would not be possible to manually tune multiple sets of gains
for each person within any kind of reasonable time frame. Therefore, an automated method for
optimizing the PID gains needed to be developed.
Initially, the use of built-in MATLAB tools was attempted to try and select suitable PID gains
for each subject; however, these methods proved to be unsuccessful. Neither the PID Tuner app
or the Closed-Loop PID Autotuner and Open-Loop PID Autotuner blocks available in Simulink
were able to consistently solve for a set of acceptable PID gains for every subject. The main
issue was that performance of these automated methods varied greatly between subjects: working
fine for some people, but choosing gains that resulted in system instability for others. Likely, the
problem with these approaches was the presence of non-linearities in the system plant (i.e., the
brace actuator model) causing the optimization procedure to fail. These techniques function best
for linear systems and can breakdown if non-linear behaviour is observed. Even after MATLAB
attempted to linearize the system plant, it was still unable to overcome the non-linearities intro-
duced by the physical behaviour of the actuator. Based on these results, it was determined that
another method of tuning the PID gains would need to be developed.
The next approach attempted was to try manually tuning a set of global PID gains (one set
of values to use for all 32 subjects) for each task weight. While a global set of PID gains would
hardly be optimal on a subject-to-subject basis, the hope was that the chosen values would be
good enough to at least allow for a preliminary evaluation of the TWSC. This method entailed
manually choosing PID gains through trial-and-error, then calculating mean RMSE, NRMSE,
and LDLJ averaged across all 32 subjects. New gain values were then chosen and the average
performance metrics would be calculated again to see how global performance varied. This process
continued until the average system error was reduced as much as feasibly possible. The idea behind
this approach was to find gain values that would work for most people, since they were chosen
based on average performance from all subjects. While still requiring some tedious trial-and-
error tuning, using a single set of global PID gains greatly reduced the time spent manually
selecting values (since it only needed to be done once for all subjects simultaneously). To aid
in this procedure, a MATLAB script was developed to semi-automate the process, setting up a
command window interface for inputting PID gain values, running the simulator for all subjects,
and calculating/displaying the average performance metric results. While this method had some
success in finding suitable PID gains, it still suffered from the same issue as the built-in MATLAB
tools: drastic inconsistency between subjects. While the global gains resulted in good performance
for some people, they resulted in unsuitably poor performance for others. The hope was that the
use of average performance metrics would provide a generalized view of system behaviour for all
subjects; however, it was found that some subjects’ performance varied greatly from the group,
which was not properly reflected in the results. Their variable behaviour ended up being obscured
in the performance metrics results used when tuning, since they were averaged out when the mean
was calculated across all 32 subjects. Therefore, it was decided that manually tuning a global
set of PID gains was also not a suitable approach to use in this work, and another method of
optimizing subject-specific PID gains needed to be found.
While manually tuning global PID gains was ultimately unsuccessful, it did provide some in-
sights into the system behaviour that was beneficial when developing a new PID tuning method.
The first observation made was that system performance was mainly driven by the P gain value
(Kp ), which makes sense as it is responsible for pushing the system output towards the desired set-
point. The second observation made was that RMSE and LDLJ present an inversely proportional
relationship. As the P gain was increased, RMSE decayed exponentially towards an asymptote (a
point where increasing P gain no longer resulted in a meaningful decrease in RMSE), while at the
same time LDLJ increased exponentially. An example showcasing this result can be seen in Figure
7.27. This behaviour is intuitive if you consider what device behaviour these metrics represent. A
larger P gain will result in a more aggressive response by the system, allowing it to reduce error
faster by moving quickly to the desired point; however, the trade-off for this is a less smooth (and
therefore less comfortable for the user) trajectory, leading to a higher LDLJ score. The problem
of optimizing the PID gains then becomes a two faceted one, and must balance the trade-off that
occurs when trying to minimize both RMSE/NRMSE and LDLJ. Based on the plots in Figure
7.27, it appeared that an ideal point existed where a P gain value can be chosen to provide the
minimum RMSE achievable by the system while also keeping LDLJ as low as possible. This point
occurs right at the value of Kp where RMSE begins to no longer meaningfully decrease (instead
asymptotically approaching some constant RMSE value), as increasing Kp further beyond this
point will only result in a needless increase in LDLJ (therefore decreasing user comfort) with no
improvement in system error. Thus, using this P gain value should result in ideal system per-
formance, balancing the RMSE–LDLJ trade-off as much as possible. To this end, given the lack
of success with other tuning methods thus far, it was decided manually implement a method to
automate the selection of this optimized subject-specific Kp value. As such, a custom MATLAB
script was developed to achieve this task, with the details of its design outlined in the following
section.
60 35
Slow, 3 lbs
Fast, 3 lbs
50
30
40
RMSE [deg]
25
LDLJ
30
20
20
10 15
Slow, 3 lbs
0 Fast, 3 lbs
10
0 500 1000 1500 2000 0 500 1000 1500 2000
Kp Gain Value Kp Gain Value
(a) RMSE vs. Kp gain (b) LDLJ vs. Kp gain
Figure 7.27: Plots showing the relationship between RMSE (a) and LDLJ (b) as the value of
Kp increases. For both plots, values were taken from a slow and fast motion trial
(with a task weight of 3 lbs), as representative examples. A relationship can be
observed where increasing Kp decreases RMSE exponentially while increasing LDLJ
exponentially. Markers (a circle for slow speed and a square for fast speed) were used
on the plots to indicate the optimal Kp value to minimize RMSE while keeping LDLJ
as low as possible.
7.4.5.1 Optimizing Subject-Specific Proportional Gain
The principle behind the PID gain optimization method developed for the TWSC was simple:
run the simulator using a range of proportional gains and evaluate the RMSE/LDLJ each time
to find the Kp value resulting in the smallest RMSE achievable while keeping LDLJ as small
as possible. This process was repeated for all speed–weight combinations with every subject to
obtain subject-specific tuned gains for each task weight. A summary of the optimization process
can be seen in Figure 7.28. It was decided to focus optimization on only the P gain value,
since it was qualitatively observed during manual tuning attempts that the integral gain (Ki ),
derivative gain (Kd ), and filter coefficient (N ) values had a small impact on system behaviour
when compared to Kp . Constraining the optimization to a one dimensional problem greatly
reduced the computational complexity required to tune the TWSC, and since it was observed that
the P gain dominated the system, it was decided that this simplification was acceptable. For the
duration of the P gain optimization process, the other gains were fixed to the following values:
Ki = 0.01,
Kd = 0.1,
N = 100.
These values were chosen for the other PID parameters based on observations made while experi-
menting with the trial-and-error manual tuning. It was anecdotally found that using these values
resulted in acceptable system behaviour for most subjects, so they were used during P gain opti-
mization. Future work could look at performing a more detailed multi-dimensional optimization
that attempts to select values for all PID parameters simultaneously; however, this work chose to
focus only on P gain value as starting point to test the initial feasibility of the TWSC.
The first step of the optimization procedure was to obtain RMSE and LDLJ scores from the
simulator for every Kp value in the desired range (note only RMSE is actually required for the
Initialize Simulator Initialize Kp Optimization Code
Load Simulated RMSE Results
True All Subjects

Next Subject
Evaluated
True All Subjects

Next Subject
Evaluated
False
All Motion True False

Next Trial
Trials Run
All Motion True

Next Trial
False Trials Run
All Kp False
True
Values Next Kp Value Calculate dRM SE
dKp
Simulated
dRM SE
Where dKp ≥ −T hresh = Kp,T hresh
False
Run Simulator
Save Kp = Kp,T hresh
Calculate RMSE
Export Optimized Kp Results
Save RMSE Result

End of Kp Optimization
Export RMSE Results
Close Simulator
Figure 7.28: The process used to find optimal subject-specific Kp gain values for each motion trial.
The procedure was split into two stages, each using their own MATLAB script. The
first step was to run the upper-limb brace simulator for every subject/trial/Kp value,
and calculate an associated RMSE score to use as the data for the optimization.
Next, these RMSE results were used to find an optimal Kp value by calculating their
derivative and finding when the rate of change of the RMSE, with respect to Kp , drops
below a specified threshold. The two stages were split into two scripts to separate
the time-consuming task of running the simulator from the rest of the optimization
procedure, making it more convenient to work with the optimization code (as it did
not require the simulator to be rerun for every change).
optimization, but LDLJ was calculated as well to verify the procedure). This was done simply
by using a MATLAB script to automate the process of running the simulator and calculating the
metrics. For each subject and all six of their movement trials (combinations of slow/fast speed
with the 0 lbs, 3 lbs, and 5 lbs task weights), a P gain value would be set, the simulator would
run to obtain the output torque/trajectory measurements from the brace, these results would be
used to calculate RMSE and LDLJ (which were saved to an MAT file for use later), then Kp
would be incremented to the next value. This process was repeated until every P gain value in the
specified range was used to run the simulator for all subjects and all speed–weight combinations.
Since RMSE was observed to behave as a decaying exponential as Kp increased, it was decided
to use a variable step size when incrementing the P gain value across the optimization range.
During the initial section of Kp values, a fine resolution is needed to properly capture the quickly
changing steep slope of the RMSE curve; however, as the rate of change slows this resolution is no
longer required. Since running the automated optimization process was time consuming (due to
the large number of times the Simulink-based simulator needed to be run to evaluate all P gain
values for every subject and speed–weight combination), the step size for incrementing Kp was
changed to a larger value for higher P gains in the desired range (reducing the number of times the
simulator would need to run). Both the chosen step sizes, and the desired range of Kp values to
evaluate, were selected based on intuition gained from observing the results of the manual tuning
experiments. A summary of the Kp step sizes used for each section of the optimization range can
be seen in Table 7.5.
Table 7.5: Step sizes and range of P gain values used for the subject-specific proportional gain
optimization procedure. Each step size shown corresponds to the range listed in the
same row (meaning that value was the step size of Kp used during that range of values).
Kp Value Range Step Size

1–300 1
305–750 5
800–2000 50
Following the P gain simulator data collection phase, the saved RMSE scores were then used to
find the optimal Kp values for each subject (allowing each controller to be tuned to that person’s
unique system dynamics). The goal of the optimization procedure was to find the earliest point
on the RMSE curve where the error stopped decreasing substantially, since this would result
in the lowest LDLJ possible for minimal RMSE (since increasing Kp further after this point
does not meaningfully decrease RMSE and only increases LDLJ needlessly). To accomplish this,
the derivative of RMSE, with respect to Kp , ( dRM SE
dKp ) was employed as the data used in the
optimization evaluation. The derivate was calculated using the gradient function available in
MATLAB. Doing this resulted in a curve that measures the RMSE rate of change as the P gain
value increases, which could then be used to find the point at which the RMSE is longer decreasing.
dRM SE
An example plot showing the relationship between dKp and Kp can be seen in Figure 7.29.
0
-0.003
-0.01
-0.02
-0.03
-0.04
Slow, 3 lbs
Fast, 3 lbs
Threshold
-0.05
0 100 200 300 400 500 600 700
Kp Gain Value
Figure 7.29: Plot showing how the derivative of the RMSE (with respect to the P gain) changes as
the value of the P gain increases. It can be seen that the negative rate of change of the
RMSE decreases exponentially toward zero as Kp increases, indicating that the RMSE
is no longer being meaningfully reduced. The plot was made using values taken from
a slow and fast motion trial (with a task weight of 3 lbs), as representative examples.
The threshold value, used as a cut-off point to stop the optimization process, is
indicated with a red dashed line (in this case a threshold value of 0.003 was used).
Markers (a circle for slow speed and a square for fast speed) show the selected P gain
values, located where the derivative of RMSE crosses the threshold. Note, that the
scale of the x and y axis was reduced from the full range to more effectively highlight
the desired behaviour.
In an ideal scenario, RMSE would eventually become constant at a true minimal error when
dRM SE
increasing Kp values and, therefore, all the optimization would need to do is evaluate dKp to
find the first P gain value where the rate of change becomes zero; however, that was not possible
given the results obtained using the simulator. As can be seen in Figure 7.27(a), the RMSE
never becomes constant and instead slowly approaches an asymptote, meaning that the rate of
change of the RMSE will never be exactly zero (as there is always some small error change being
measured in the system). This fact is confirmed when looking at the derivate of RMSE (Figure
7.29), where it can be observed that it asymptotically approaches a non-zero value. To rectify
dRM SE
this, the optimization instead was implemented using a target dKp threshold value (T hresh)
to act as the stopping point for the procedure (represented on Figure 7.29 as the dashed line).
Once the decrease rate of the RMSE went below this specified threshold value, the optimization
procedure would end and the corresponding Kp value at that point (Kp,T hresh ) would be selected.
Put another way, the threshold value specifies how close to zero the rate of RMSE change needs
to be before the optimization procedure no longer considers it worthwhile to keep increasing the
P gain value. The threshold-based evaluation used when optimizing Kp is summarized as follows:
dRM SE
when ≥ −T hresh set Kp = Kp,T hresh . (7.56)
dKp
Note that the use of a negative threshold value, along with a greater-than-or-equal-to sign in
Equation 7.56 was done as a consequence of the derivative of the RMSE being negative (as seen
in Figure 7.29). Since the error is decaying exponentially, the rate of change decreases with each
step, which is reflected in the negative values of the derivate. Using the form of Equation 7.56 as
shown ensured that the comparison of the RMSE derivative and the threshold value was evaluated
correctly.
Using a threshold as the stopping point ensured that the optimization would be able to run
properly and not select an unnecessarily high P gain value as it tries unsuccessfully to drive RMSE
to zero (when such a thing is likely not even possible); however, it does come at a cost. It introduces
yet another parameter that requires tuning: the value of the threshold itself. Selecting a suitable
threshold value is important because it can greatly impact the success of the P gain optimization
process. If too large a value (far away from 0) is used for the threshold, the optimization process
will stop too early and chose a lower Kp value, leading to a higher RMSE. Likewise, if the threshold
value is set too small (close to 0), the optimization will run longer than necessary, selecting a high
Kp value that needlessly increases LDLJ with little-to-no reduction in RMSE. The threshold value
needs to be chosen such that the optimization process stops right at the point where the RMSE is no
longer being improved meaningfully, keeping LDLJ minimal (hopefully reducing user discomfort).
Examples of the effect that different threshold values had on the selection Kp can be seen in Figure
7.30. To clarify, note that the discussion here is referring to the threshold as an absolute value
to make the description more intuitive (remembering that, as per Equation 7.56, the threshold is
actually treated as a negative number in the comparison). To find a suitable threshold to use for P-
gain optimization, several values were tested through trial-and-error to see which provided the best
results. The subject-specific P-gain optimization was run multiple times using threshold values
of 0.01, 0.001, 0.003, 0.005, and 0.0005 and plots of each where compared qualitatively. Based
on observations made during this process, it was decided to use a threshold value of 0.003 in the
final P gain optimization results, as it appeared to offer the best balance between both scenarios.
Future work could focus on developing a more rigorous method of optimizing the threshold value
(implementing something akin to the hyperparameter optimization techniques used in machine
learning); however, it was determined that the basic method outlined here was sufficient for the
purposes of this work.
Once a threshold value was chosen, the P gain optimization procedure was run for all subjects
and for every speed–weight combination (six trials total). This resulted in six subject-specific Kp
values for each participant: one for each speed–weight combination. However, since the EEG–
EMG classifiers, in their current implementation, only output task weight predictions and provide
no speed measurement information, a decision had to be made about which speed-level task weight
gains to use for Gain Scheduling in the TWSC (i.e., the gains obtained from the slow speed trial
or the fast speed trial). While it would be possible to implement the TWSC in such a way that it
switches PID gains based on speed (since the assumption is that the device is measuring speed using
a built in sensor), as well as task weight, through a combined decision approach (such as multiple
cascading switch blocks), it was decided for this work to initially utilize a simpler implementation
that only selected gains based on task weight, to focus the scope of the study on this investigation
as a starting point. Future work could focus on implementing a more complex switching controller
that utilizes multiple sources of information to schedule gains, and could compare its performance
to the TWSC presented here (to see whether using six gain sets, tuned to each speed and task
60
Large Threshold Value
Ideal Threshold Value
50 Small Threshold Value
40
RMSE [deg]
30
20
10
0
0 500 1000 1500 2000
Kp Gain Value
Figure 7.30: A plot of RMSE vs. P gain, highlighting the effect of threshold selection on the P gain
value chosen during subject-specific optimization. If too large of a threshold value
is used (indicated by an upward facing triangle), then the optimization stops early
and a smaller Kp value is chosen, leading to higher RMSE. If the threshold value
used is too small (indicated by a downwards facing triangle), then the optimization
runs longer than required and selects a larger Kp value, leading to needlessly high
LDLJ with no meaningful reduction in RMSE. An ideal threshold value (indicated
by a circle) needs to be found to stop the optimization right as the RMSE stops
decreasing meaningfully, keeping LDLJ as low as possible with minimal RMSE.
weight, provides enough performance improvement to justify the increased system complexity). For
the TWSC presented here, it was decided to use the Kp values for 0 lbs, 3 lbs, and 5 lbs that were
obtained from the fast motion speed trials. The logic applied was that these trajectories required
a more aggressive response, given the increase in torque required to move faster, therefore their
gains should be used so that the device can operate properly in “worst case” scenarios. The slow
motion speed trajectories did not require a very powerful response from the system and seemed to
be easy to follow with little error, therefore; it was assumed that the fast speed Kp values would
be sufficient in these scenarios as well.
7.4.5.2 Reduced Subject Pool
The subject-specific gain tuning approach discussed above was found to be an effective method
for automating the P gain selection process in a consistent manner for all participants; however,
it also had one notable drawback: computation time. Because the devised technique required the
simulator to be re-run for each new Kp value being evaluated (with this repeating for all subjects
and motion trials), the amount of time required to optimize the P gains for all 32 subjects was
not feasible. For this reason, it was decided that the number of subjects used for the TWSC
experiments would be paired down to a smaller number, reducing the computation time necessary
to tune the controllers.
Using a reduced subject pool also had other benefits that pertained to how the results could be
analyzed. Since the performance of the controller/simulator plant model was closely tied to each
individual subject’s parameters and motion trajectories, system results could vary greatly between
participants. As discussed previously in Section 7.4.5, attempting to only look at mean performance
metrics across all subjects caused outlier behaviour to become “lost” in the average, which could
lead to important results being overlooked. The study would benefit from also including a subject-
specific analysis of results (comparing the performance of different controller types for each person
individually) to provide a clear demonstration of how the various TWSC implementations change
the control system behaviour for each person, strengthening the evaluation. That way, even if
a subject has a higher/lower amount of “baseline” error (making them an outlier and causing
their results to be averaged out), it can still be easily observed if a change was demonstrated in
their control system behaviour. While it would be ideal to perform this type of analysis for all
subjects, trying to focus on individual results can quickly become unmanageable for large amounts
of data. Thus, the use of a reduced subject pool (to lower the computational burden of the P
gain optimization method) also facilitated the ability to implement an analysis of subject-specific
results.
To be able to form the reduced subject pool for the TWSC experiments, a selection criteria
needed to be devised to logically chose which subjects to include in the TWSC experiments. After
considering multiple approaches, it was decided to base the selection criteria on each subject’s
EEG–EMG fusion classifier accuracy, and to use a cut-off score that needed to be met for inclusion
in this work. Thus, only subjects whose WA-50/50 classifier mean accuracy, obtained from both
their slow-speed and fast-speed trials, was greater than or equal to 70% were used in the reduced
subject pool. The logic behind this approach was that, in a real-world scenario, a device would
never be deployed to the public with a poorly functioning classification system; development would
simply continue until a suitable accuracy was achieved. To properly implement the TWSC, a
baseline level of competency of the classifier needed to be assumed. Otherwise, the control system
would simply fail, not because the concept of a switching controller in unviable, but because
the classifier is supplying poor information (essentially sabotaging the evaluation of the TWSC).
The purpose of this study was not to evaluate the performance of the EEG–EMG fusion-based
SVM classifiers, as this had already been done exhaustively in Chapter 4, thus it makes sense
to only include the models that demonstrated feasible performance in the TWSC. Therefore,
this selection criteria ultimately serves two purposes: reducing the subject pool and ensuring
reasonable behaviour from the classifiers. The reason that fast and slow speed accuracy were
evaluated separately (as opposed to only looking at overall accuracy, the average of the two) is
that some participants demonstrated highly variable performance between the two speeds. In some
cases, their accuracy score would be high for one speed, but low for the other; however, this effect
was lost when observing the overall accuracy as it gets averaged out. Applying the logic that the
classifiers used in the TWSC should be close to a realistic model deployed in a medical device, then
it makes sense that only classifiers that displayed consistent and robust behaviour across multiple
use-cases would be included. Therefore the accuracy cut-off of 70% used for the selection criteria
when reducing the subject pool was applied to both motion speed accuracies separately.
Using this criteria, 7 subjects from the 32 total were chosen for the reduced subject pool. It was
also decided to include one more subject as a special case: namely the person whose classification
model demonstrated the lowest overall accuracy out of all participants. Even though the purpose
of this study was to evaluate the TWSC with adequately performing classifiers, it was thought that
it would be an interesting analysis to see how the TWSC performs when supplied with poor task
weight predictions. Doing so simulates something akin to a rare, but not impossible, malfunction
scenario where the classifier loses the ability to properly function (for example, if the EEG/EMG
electrodes were to become damaged or fall off the user, resulting in poor/no bioelectrical signal
measurement), which is important to characterize when considering user safety for upper-limb
wearable devices. While the inclusion of one extra subject cannot be considered a formal analysis
of TWSC robustness by any means, it still can provide useful insights and inform the direction
of future studies, which is why it was done here. With this, the total number of subjects in the
reduced dataset was brought up to 8. The mean accuracy scores from the fast and slow motion
speed trials, as well as the overall accuracy score (for comparison), can be seen in Table 7.6. From
this, it can be observed that all subjects (excluding the special “worst-case” scenario) meet the
selection criteria with mean accuracies for both motion speeds above 70%.
Table 7.6: The subjects chosen to be included in reduced subject pool with their WA-50/50 clas-
sifier mean accuracy scores (shown separated by speed, slow and fast, as well as their
overall accuracy score). Only subjects with speed-separated accuracies greater-than-or-
equal-to 70% were selected, with the exception of Subject 25 who was chosen to provide
a “worse case” scenario (as they had the lowest overall accuracy of all 32 subjects).
Subject WA-50/50 Classifier Accuracy

Number Overall Slow Fast
8 85.10 86.08 70.97
11 84.05 85.48 70.77
17 85.81 86.13 81.71
18 85.60 86.95 70.59
28 88.96 90.39 76.81
30 83.98 85.58 71.43
31 79.51 80.58 73.53
25 70.76 73.60 47.37
Using the reduced subject pool, the aforementioned P-gain optimization method was run for
all included participants to obtain a set of subject-specific PID gains for each task weight. The
obtained P-gain values for each subject that were used in the TWSC evaluation experiments can
been seen in Table 7.7. To reiterate, the values of Ki , Kd , and N in the PID controller were fixed
to 0.01, 0.1, and 100, respectively, for all subjects and task weights. Also, the P-gain values used
for 0 lbs, 3 lbs, and 5 lbs in the TWSC simulations were chosen from P-gain optimization results
obtained during fast motion speed trials; however, the P gains tuned using the slow speed motion
7.5 Control System Evaluation Procedure 284
trials are also shown in Table 7.7 for reference/comparison.
Table 7.7: Each subject’s final P-gain values, selected using the subject-specific optimization pro-
cedure. It was decided to only use the gains found for the fast speed motion trials
during the TWSC simulations; however, the P gains tuned using the slow speed motion
trials are also shown here for reference/comparison.
Selected P-Gain Values

Subject
Slow Speed Fast Speed
Number
8 161 165 162 500 465 460
11 192 209 221 485 500 465
17 142 159 163 465 425 480
18 194 200 187 515 485 415
28 193 189 198 495 480 455
30 193 209 218 380 445 450
31 269 247 228 445 470 465
25 191 181 189 485 470 440
Once the PID gains were tuned for the TWSC, the implementation of this novel control system
was finally complete. In conjunction with the developed upper-limb brace simulator model, the
experiments used to evaluate the performance of the TWSC could finally be carried out. The
following section will outline the specific procedure used during the simulation experiments, to
clearly highlight what aspects of the TWSC were analyzed.
7.5 Control System Evaluation Procedure
The goal of this work was to develop a method of integrating an EEG–EMG fusion-based task
weight classifier into a device control system, resulting in the TWSC; however, to fully realize this
objective the TWSC needed to be evaluated to determine the effectiveness of this new solution.
Doing so will also provide important insights into potential areas where the design of the TWSC
could be improved during future work. Since the TWSC controller was developed with the idea in
mind that it can act as an alternative to using basic PID control, one of the primary focuses of the
experiments performed was to compare the TWSC to PID control under different scenarios. The
other objective of the evaluation was to compare different iterations of the TWSC (i.e., different
debouncing methods) to further hone in on optimal design parameters.

To achieve these goals, two different types of experiments were run that are differentiated from
each other by when during the motion trial do the changes to the task weight occur (referred
to here as Experiment 1 and Experiment 2). Within these two groups, there were also different
variations of each experiment, by changing various parameters of the TWSC or the motion task
to evaluate multiple aspects of the TWSC. The following subsections will outline the different
experiments performed using the upper-limb brace simulator to investigate the performance of the
TWSC.
7.5.1 Experiment 1: Task Weight Variation Between Motions
The first set of experiments(labelled Experiment 1) was performed on movement trials where the
task weight was varied between motions. This means that for an entire elbow flexion–extension rep-
etition, the task weight being held by the subject was constant, with changes occurring in between
repetitions when the subject was not moving. An analogous real-world example of this scenario
would be a situation where a patient wearing the brace is performing a series of rehabilitation
exercises and the therapist hands them a new weight in between repetitions.
The purpose of Experiment 1 was to see how the TWSC controller would perform when weight
remains constant as the subject moves. The ideal behaviour of the controller would be to imme-
diately update the gains for the selected task weight as soon as movement begins and then have
those gains remain constant for the rest of the motion repetition; however, it is likely that this
would not always be observed in a real-world system. Misclassifications may cause the TWSC to
incorrectly switch gains while moving, or the TWSC controller may not update immediately at
the start of the motion (especially if a larger delay is introduced by more aggressive debouncing
methods). It is important to know how these situations will affect the performance of the device,
as this will have implications for how successful the user finds their experience with the upper-limb
brace.
Within Experiment 1, multiple iterations of the TWSC were evaluated using different simu-
lations, and these can be further subdivided into two comparison groups that focus on testing
different aspects of the TWSC. The first group of experiments evaluates the TWSC using the var-
ious prediction switching control signals generated to initiate classifier-based gain selection. For
all motion trials (i.e., all speed–weight combinations) from every subject, the simulator was run
multiple times with the debouncing method utilized for the TWSC being modified each iteration.
As discussed in Section 7.4.4, multiple debouncing methods (with multiple settings for each) where
implemented to pre-process the prediction signal generated using the classifier outputs, hence the
purpose of this test was to investigate which method leads to the best device performance. For the
evaluation here, there where three versions of the TWSC using the Hold-Delay debouncing method
(with delay values of 2, 3, and 4), three versions using the Majority-Vote debouncing method (with
vote window sizes of 3, 4, and 5), as well as one version of the TWSC that utilized no debouncing
method, forming the prediction signal from the direct classifier outputs (referred to here as the
raw outputs) to see the effect of debouncing, in general, on performance. This resulted in a total
of seven iterations of the TWSC being used to simulate the control of an upper-limb brace for six
motions trials (three elbow flexion–extension repetitions for each combination of fast/slow speed
with task weights of 0 lbs, 3 lbs, and 5 lbs) from all eight subjects in the reduced pool. For a
summary of the TWSC simulations performed for Experiment 1, see Table 7.8.
The second group of evaluations in Experiment 1 focused on comparing the performance of the
TWSC to conventional PID control, to see if this newly proposed method can act as a suitable
alternative. To achieve this, three more iterations of the control system were utilized within the
simulator to obtain benchmark results for the evaluation. These iterations simulated the use of
static, non-changing gains (essentially just a regular PID controller); specifically, with each using
one of the gains optimized for the three task weight levels (0 lbs, 3 lbs, and 5 lbs). These gains
are the ones obtained from the subject-specific optimization, and the TWSC switches between
them; however, for these experiments no switching is performed and the same gain values are used
for all motion trials. The purpose of this is to obtain results that can act as a control group for
the comparison-focused evaluation, determining if the TWSC actually provides a benefit for the
system, or if it merely introduces unnecessary complexity for little/no improvement over traditional
PID control. This resulted in a total of three iterations of the TWSC being used to simulate the
control of an upper-limb brace for six motions trials (three elbow flexion–extension repetitions for
each combination of fast/slow speed with task weights of 0 lbs, 3 lbs, and 5 lbs) from all eight
Table 7.8: A summary of the TWSC simulations run for Experiment 1. For both testing groups,
every TWSC iteration (with each applicable setting) was used during simulations of all
six motion trials (both speeds with all three weights).
Dataset Test Group Speed Weight TWSC Iteration Settings

Raw -
Delay 2
0 lbs Hold-Delay Delay 3
Slow 3 lbs Delay 4
5 lbs Window 3
Majority-Vote Window 4
Debouncing Window 5
Methods Raw -
Delay 2
0 lbs Hold-Delay Delay 3
Recorded
Fast 3 lbs Delay 4
5 lbs Window 3
Majority-Vote Window 4
Window 5
0 lbs 0 lbs Gain
Static Gain
Slow 3 lbs 3 lbs Gain
(Normal PID)
No Switching 5 lbs 5 lbs Gain
Baseline 0 lbs 0 lbs Gain
Static Gain
Fast 3 lbs 3 lbs Gain
(Normal PID)
5 lbs 5 lbs Gain
subjects in the reduced pool.

In both test groups in Experiment 1, after each simulated motion trial, output measurements
from the simulator where used to calculate the three performance metrics discussed in Section
7.3.7 (RMSE, NRMSE, and LDLJ), which were utilized to facilitate an objective analysis of the
performance of each debouncing method, as well as an analysis of how the TWSC compared to
the benchmark control group. To actually run the simulations, the version of the simulator shown
in Figure 7.7 was utilized, where the Simulink model is configured to use a static inertia value
that remains constant as it runs. It would have been possible to use the version of the simulator
that facilitates a dynamic inertia value while the Simulink model runs (shown in Figure 7.14)
by simply setting the weight change control signal to a constant value; however, since a version
of the simulator that uses a static inertia value already existed, it was decided to utilize that
Simulink model for this experiment instead for simplicity. The performance metric results from
each subject were grouped together and used to generate box plots so that the distributions of each
control method could be compared visually. Due to the high variation in baseline performance
metric values observed between subjects, it was determined that this represented the full scope
of the obtained results better and the findings could be reported more clearly in a succinct way,
rather than using means/standard deviations. Each box plot reports the median (red line), data
range (dashed-line whiskers), outliers (red asterisks), and the interquartile range (25th to 75th
percentile, length of blue box). Outliers were specified as any point that beyond 1.5 times the
length of interquartile range above/below the ends of the box (as is typically done for box plots).
For reasons that will become clear in Section 7.6, no statistical analysis was performed on the
results of this experiment.
7.5.2 Experiment 2: Task Weight Variation During Motion
The second set of experiments (labelled Experiment 2) was performed on movement trials where
the task weight was varied during motion. This means that during an elbow flexion–extension
repetition, the task weight being held by the subject can change suddenly. An analogous real-
world example of this scenario would be a situation where the device user drops the object they
are holding (going from an extra load on the device to none) or the user picks up and object (going
from no weight to an additional load).
The purpose of Experiment 2 was to see how the TWSC controller would perform during
dynamic changes in task weight. The stated purpose for the development of the TWSC was to
make wearable devices more adaptable to changes in external loads, required for completing a
variety of rehabilitation and daily living tasks; therefore, it is only logical to assess the TWSC
performance under such conditions (albeit simplified for a preliminary analysis). It is important
to see how rapid changes in task weight will affect the controller, to ensure that it can respond
appropriately in a manner that maintains ease of use and comfort for the device operator.
For Experiment 2, the TWSC was evaluated using the simulator while three motion trial
parameters were varied: motion speed, start/end weight, and weight change condition. Motion
speed settings were the same before, using either the fast or slow speed; however, for these motion
trials only one repetition of elbow flexion–extension movement is present (due to the use of the
synthetic motion trajectory discussed in Section 7.3.6.2). The start/end weight was varied between
two combinations: 0 lbs to 5 lbs (e.g., picking up and object) and 5 lbs to 0 lbs (dropping an object).
Weight change condition indicated where in the motion trajectory the change in task weight occurs.
Three conditions were considered: at the midpoint of elbow flexion, at the midpoint of the peak
of the motion trajectory (during the brief rest period between flexion and extension), and at the
midpoint of elbow extension. Both the chosen start/end weight combinations and the weight
change conditions are discussed in more detail in Section 7.3.6.2. For all combinations of motion
trial parameters, three versions of the TWSC were evaluated that match the baseline control-
group controller iterations used in the second evaluation test in Experiment 1 (no switching using
static PID gains for each of the three task weights). As well, a fourth version of the TWSC was
evaluated that utilized what is referred to here as manual switching. This method does not use the
classifier outputs to inform the TWSC, and instead manually switches to the correct task weight
at all times. It can basically be thought of as implementing the TWSC with a 100% accurate
classifier that never outputs any prediction errors. The reason for its inclusion was to provide
a best-case example of the TWSC to see how well gain switching-based control of a wearable
mechatronic device could theoretically perform, if not held back by inaccuracies in the machine-
learning model. Since no real-world data was collected for Experiment 2, actually implementing a
classifier-based version would have required the creation of a synthetic classifier prediction dataset
to accompany the synthetic motion trajectories. Before undergoing this process, it was decided
to first test the TWSC under weight varying motion conditions using manual switching to act as
a proof-of-concept to determine if further investigation was warranted. However, after seeing the
results obtained by the manual/no switching TWSC iterations for Experiment 2 it was decided
to forgo an evaluation of a synthetic classifier prediction TWSC, as it was assumed the results
would be similar (see Section 7.6 for these results). These two approaches (manual switching and
static PID gains) were used to evaluate how the TWSC performs with dynamic weight variations,
and to see how this compares to traditional PID controllers. Since the goal of the TWSC is to
provide a more adaptable system, it needs to provide enough of a benefit to device performance
to justify the additional complexity it requires. Thus, a total of four iterations of the TWSC
were used to simulate the control of an upper-limb brace for twelve motions trials: three weight
change conditions for both 0-lbs-to-5-lbs and 5-lbs-to-0-lbs experiments (six trials), with both of
these performed using the fast and slow speed motion trajectory (hence, twelve trials in total).
These trials where run for all eight subjects in the reduced pool. For a summary of the TWSC
simulations performed for Experiment 2, see Table 7.9.
Table 7.9: A summary of the TWSC simulations run for Experiment 2. For both weight level
changes, every TWSC iteration (with each applicable setting) was used during simula-
tions of all three weight change conditions for both speeds.
Weight Level Weight Change

Dataset Speed TWSC Iteration Settings
Change Condition
Manual
-
1 (Flex.) Switching
Slow 2 (Mid.)
0 lbs Gain
3 (Ext.) Static Gain
3 lbs Gain
0 lbs to 5 lbs (Normal PID)
5 lbs Gain
Manual
-
1 (Flex.) Switching
Synthetic Fast 2 (Mid.)
0 lbs Gain
3 lbs Gain
(Normal PID)
5 lbs Gain
Manual
-
1 (Flex.) Switching
Slow 2 (Mid.)
0 lbs Gain
3 lbs Gain
5 lbs to 0 lbs (Normal PID)
5 lbs Gain
Manual
-
1 (Flex.) Switching
Fast 2 (Mid.)
0 lbs Gain
3 lbs Gain
(Normal PID)
5 lbs Gain
Like with the tests in Experiment 1, after each simulated motion trial for Experiment 2 was
completed, the output measurements from the simulator where used to calculate the three per-
7.6 Results 291
formance metrics discussed in Section 7.3.7 (RMSE, NRMSE, and LDLJ), which were used to
facilitate an objective comparison between an ideal (manually switched) TWSC and conventional
PID control. As well, box plots were again utilized to visually represent the distributions of the
performance metric results for all subjects. Each box plot reports the median (red line), data
range (dashed-line whiskers), outliers (red asterisks), and the interquartile range (25th to 75th
percentile, length of blue box). Outliers were specified as any point that beyond 1.5 times the
length of interquartile range above/below the ends of the box (as is typically done for box plots).
In the same vein as Experiment 1, no statistical analysis was performed on the results of Experi-
ment 2 (refer to Section 7.6 for further explanation). To actually run the simulations, the version
of the simulator shown in Figure 7.14 was utilized, where the Simulink model was implemented
to allow a dynamic inertia value that can change as the model is running. The weight change
was controlled using the weight change signals developed in Section 7.3.6.2, and the inputs to the
simulator were the synthetic motion trajectories described in the same section.
7.6 Results
7.6.1 Experiment 1: Task Weight Variation Between Motions
7.6.1.1 Debounced Classifier Switching Results
Once all simulations had been completed for Experiment 1 and 2, and all performance metrics were
calculated, the results were analyzed to evalaute the TWSC. Starting with Experiment 1 (when
task weight varied in between motions), the results showing the comparison of debouncing methods
will be discussed first. The box plots showing the distributions of the RMSE, NRMSE, and LDLJ
results obtained by simulating the implementations of the TWSC that utilized debounced classifier
outputs for gain switching can be seen in Figure 7.31.
Looking at the box plots for the slow speed (Figure 7.31(a)) and fast speed (Figure 7.31(b))
trials, the first observation that can be made is that RMSE is significantly lower during slow
movements, with error values close to zero. This makes sense intuitively, considering that the
forces required by the actuator to move the brace differ for the two speed levels. Moving at a
slower speed requires vastly less torque, making it easier for the actuator to follow the desired
7.6 Results 292
0.5
0.4
RMSE [deg]
0.3
0.2
28
NRMSE [%]
26
24
28
26
LDLJ
24
22
Raw HD2 HD3 HD4 MV3 MV4 MV5 Raw HD2 HD3 HD4 MV3 MV4 MV5 Raw HD2 HD3 HD4 MV3 MV4 MV5
0 lbs, Slow Speed 3 lbs, Slow Speed 5 lbs, Slow Speed
(a) Slow speed results

15
RMSE [deg]
10
20
NRMSE [%]
15
10
22
21
LDLJ
20
19
Raw HD2 HD3 HD4 MV3 MV4 MV5 Raw HD2 HD3 HD4 MV3 MV4 MV5 Raw HD2 HD3 HD4 MV3 MV4 MV5
0 lbs, Fast Speed 3 lbs, Fast Speed 5 lbs, Fast Speed
(b) Fast speed results
Figure 7.31: Box plots showing the RMSE, NRMSE, and LDLJ distributions from simulations that
utilized debounced classifier gain switching for (a) slow and (b) fast speed motion tri-
als where weight varied between motions. TWSC iterations are labelled where “Raw”
indicates no debouncing, “HD” indicates the Hold-Delay method (number specifies
delay value), and “MV” indicates the Majority-Vote method (number specifies vote
window size). Boxes are grouped by task weight.
7.6 Results 293
input motion trajectory (which leads to lower error measurement). Looking at the NRSME, it
appears to actually be slightly higher for the slow speed results, but this is likely a consequence
of the small error range observed during the slow speed motion. Since the error range being used
during normalization is small, even tiny values of RMSE will be a much higher percent of the total
error and lead to a larger NRMSE score (a similar effect can also be seen in the LDLJ result, since
peak velocity is used as a normalizing factor). Comparing the RMSE/NRMSE of the different
debouncing methods, it can be seen that the distributions are very similar for both speed levels
(with only very minor variations occurring in NRMSE for the 3 lb, slow speed trials; however, this
is so small it is likely not a significant difference). For RMSE and NRMSE, the median values,
interquartile range, and data range for each debouncing method are essentially the same, which
implies that using debouncing in the TWSC had little-to-no effect on the accuracy of the wearable
brace.
Comparing the brace output positions measured from the simulator directly further supports
this finding. As can be seen from the representative example taken from a fast speed motion trial
(Figure 7.32), there is almost no visual difference in the output positions from the various simu-
lations performed using different debouncing methods. The position signals are so close together
that the only way to observe any differences is to zoom in on the image significantly, highlighting
how small the changes were between each measurement. This aligns with what was found using
the RMSE/NRMSE performance metrics, that debouncing method had a negligible affect on the
accuracy of the brace.
Focusing on LDLJ, an interesting pattern emerges when comparing the debouncing-based iter-
ations with the raw (direct classifier outputs) TWSC. For all speed–weight combinations (except
fast, 0 lbs), it can be seen that the LDLJ distribution of the raw-prediction controller iteration
trended higher than all debounced-prediction controllers. In a similar vein, as the strictness of the
debouncing method increased (longer delay value or larger vote window) the distributions for the
LDLJ continued to trend lower. It is easier to see for the slow-speed results, but it can still be
observed in the fast-speed box plots as well: the median lines and/or bottom whisker (depending
on which boxes are being looked at) appear lower for the TWSCs that used more aggressively
debounced prediction signals than for their raw/less-debounced counterparts. A good example
7.6 Results 294
150
Output Position [deg]
Raw
100 HD2
HD3
HD4
MV3
MV4
50 MV5
Input
0
0 2 4 6 8 10 12 14 16
Time [s]
(a) Full trial
150
149 Raw
HD2
HD3
148 HD4
MV3
MV4
147 MV5
Input
146
145
15.9 15.92 15.94 15.96 15.98 16 16.02 16.04 16.06
Time [s]
(b) Zoomed in
Figure 7.32: Measurements of the brace output position obtained when using each debouncing
method and setting, taken from an example fast speed, 3 lbs, motion trial. The input
reference trajectory is shown as well, for comparison. The position results from the
full trial are shown in (a), while (b) provides a zoomed-in view of one portion of the
trial to highlight the differences between each measurement.
that clearly highlights this phenomenon can be seen in Figure 7.31(a) for the 5 lbs, slow speed,
trials. The raw median line is higher than both HD2 and MV3, and, when comparing within the
debouncing methods, the median for HD3 is lower than HD2 (and the median for MV4 is lower
than MV3). For fast speed results, this can be observed by looking at the lower bottom whisker
7.6 Results 295
and lower box edge in the 3 lbs box plots for the debouncing methods when compared to the raw
implementation, indicating that the tail of the distributions have shifted down (this is similarly
observed in the 5 lbs, fast speed distributions by looking at the how the bottom whiskers reach
lower LDLJ values). There does seem to be a diminishing return to debouncing, where only the
first or second levels of each method are required to lower LDLJ, and increasing the levels further
does not have as much of an effect (other than potentially increasing system delay by requiring
more time before a prediction switching control signal change is considered true). While far from
a definitive finding, given the overall similarity in the actual LDLJ values for all debouncing meth-
ods, this does show that, in general, the use of debouncing is likely to result in more comfortable
operation of an upper-limb wearable device (the user should experience less jerk). This comes at
no apparent cost to system accuracy, as the RMSE/NRMSE distributions are essentially the same
for the raw and debounced TWSCs.
This effect can be further observed when analysing the simulator outputs qualitatively. Looking
at example plots (Figure 7.33) showing output gearbox velocity measurements (used to calculate
LDLJ, as described in Section 7.3.7.3) taken from representative fast and slow speed motion trials,
it becomes more apparent why the use of prediction debouncing resulted in lower LDLJ scores. In
the velocity profile measured from a TWSC iteration with no debouncing (labelled “Raw”) there
are several moments of brief erratic behaviour, where the velocity response fluctuates rapidly before
returning to the expected smooth profile. This velocity behaviour is present for both the fast and
slow speed motions, and is due to the dynamics of the system rapidly changing due to occasional
misclassifications. When an incorrect estimation of task weight is supplied by the classifier, the
TWSC erroneously updates the gains (since it assumes that it must adapt to a new weight). Given
the brief nature of misclassifications, the classifier quickly begins supplying the correct task weight
prediction again, which causes the TWSC to rapidly return to the previous gain values. This
fast oscillation in controller dynamics is the source of the erratic velocity response observed in the
brace simulator output, which ultimately translates to reduced comfort for the user. However,
when focusing on the velocity results from the simulations where debouncing was used, it can be
observed that this instability is essentially removed. The debouncing methods are successful in
filtering out misclassifications during control system operation, providing better performance for
7.6 Results 296
0.5
Raw
HD3
Gearbox Velocity [rad/s]
MV4
0.25
-0.25
-0.5
0 10 20 30 40 50 60 70 80
Time [s]
(a) Slow speed
4
3
Gearbox Velocity [rad/s]
-1
-2
Raw
-3 HD3
MV4
-4
0 2 4 6 8 10 12 14 16
Time [s]
(b) Fast speed
Figure 7.33: Plots showing example output gearbox velocity measurements taken from a represen-
tative slow (a) and fast (b) speed motion trial. For both speeds, an example of the
simulator output using a TWSC with no debouncing (raw), as well as an example
of a TWSC using each debouncing approach at one setting (HD3 and HV4) are in-
cluded. The velocity results obtained using no debouncing show several moments of
brief erratic behaviour caused by misclassifications, while the simulations that utilized
debouncing did not exhibit this behaviour.
the robotic brace.

7.6 Results 297
7.6.1.2 No Switching Results
For the second test in Experiment 1, TWSC was compared to conventional PID control and
obtained the following results. The box plots showing the distributions of the RMSE, NRMSE,
and LDLJ results obtained by simulating the baseline control-group implementations of the TWSC
that utilized no switching (i.e., normal PID) can be seen in Figure 7.34.
Comparing the results obtained from the static gain simulations against the simulations that
utilized a TWSC, it can be observed that the distributions for all metrics (and all speed–weight
combinations) where extremely similar. The obtained median values were very close to each
other, and the distributions often had a similar shape, which indicates that the small differences
between them are likely not statistically significant. This trend was true for both the debounced
TWSCs, as well as the TWSC that utilized no debouncing, implying that simply improving task
weight classifier accuracy would not result in a meaningful performance increase in this system.
Curiously, even the different static-gain PID control systems exhibited very similar performance to
each other across different speed–weight combinations, even though they were using gains that had
been specifically optimized to only one weight level. This may indicate that the system actuator
model itself was exhibiting relatively robust behaviour, making it insensitive to fluctuating gain
values.
One interesting finding that did come from this experiment is apparent when comparing raw/de-
bounced switching to the static gain that was optimized for the task weight currently being used
in the motion trial. In a sense this demonstrates the performance ceiling one would expect from
a manually switching TWSC; when the task-weight-optimized gain matches the task weight held
for the current trial it essentially mimics the behaviour of a TWSC with a 100% accurate classifier
(since the gain is always “correct” for the duration of the trial because task weight only varies
in between trials in Experiment 1). As discussed above, utilizing debouncing methods resulted
in a trend where slight improvements could be seen in LDLJ when compared against using raw
classifier predictions in the TWSC (which can be seen again in Figure 7.34 for the repeated de-
bouncing method examples). Comparing these to the pseudo-manual switching results obtained
when the optimized static gain is used for its corresponding motion trials (analogous to a 100%
7.6 Results 298
0.5
0.4
RMSE [deg]
0.3
0.2
28
NRMSE [%]
26
24
28
26
LDLJ
24
22
SG0 SG3 SG5 Raw HD3 MV4 SG0 SG3 SG5 Raw HD3 MV4 SG0 SG3 SG5 Raw HD3 MV4
0 lbs, Slow Speed 3 lbs, Slow Speed 5 lbs, Slow Speed

15
RMSE [deg]
10
20
NRMSE [%]
15
10
22
21
LDLJ
20
19
SG0 SG3 SG5 Raw HD3 MV4 SG0 SG3 SG5 Raw HD3 MV4 SG0 SG3 SG5 Raw HD3 MV4
0 lbs, Fast Speed 3 lbs, Fast Speed 5 lbs, Fast Speed
Figure 7.34: Box plots showing the RMSE, NRMSE, and LDLJ distributions from simulations that
utilized no switching for (a) slow and (b) fast speed motion trials where weight varied
between motions. TWSC iterations are labelled where “SG” indicates static gains
(no switching, number corresponds to weight for which the gains were optimized).
Select results from Figure 7.31 are repeated here to facilitate an easier comparison.
Boxes are grouped by task weight.
7.6 Results 299
150 SG0
SG3
SG5
Raw
HD3
100 MV4
Input
50
0
0 2 4 6 8 10 12 14 16
Time [s]
(a) Full trial
150
149
148
SG0
147 SG3
SG5
Raw
146 HD3
MV4
Input
145
15.9 15.92 15.94 15.96 15.98 16 16.02 16.04 16.06
Time [s]
(b) Zoomed in
Figure 7.35: Measurements of the brace output position obtained when using no-switching baseline
controllers optimized for each task weight, taken from an example fast speed, 3 lbs,
motion trial. The input reference trajectory is shown as well, for comparison, along
with position results from simulations that use no debouncing (Raw) and examples of
each debouncing approach (HD3 and MV4). The position results from the full trial
are shown in (a), while (b) provides a zoomed-in view of one portion of the trial to
highlight the differences between each measurement more effectively.
accurate classifier), it can be seen that the LDLJ distributions of the debounced TWSCs match the
pseudo-manual-switching TWSC very closely, while the raw TWSC distributions typically exhibit
a visually different shape. Based on this pattern, it can be seen that incorporating debouncing
7.6 Results 300
techniques as a filtering method for classifier prediction outputs can help compensate for misclas-
sifications present in a non-perfect model and helps move the performance of the TWSC closer to
an “ideal” system with perfect classifier accuracy (improving the overall robustness of the control
method, as a 100% accurate classifier is very unlikely to occur in a real-world device).
7.6.2 Experiment 2: Task Weight Variation During Motion
Moving on to Experiment 2 (when task weight varied during motion), multiple weight change
conditions and weight level changes were simulated to obtain results regarding the performance of
TWSC during more dynamic situations. The box plots showing the distributions of the RMSE,
NRMSE, and LDLJ results obtained by simulating the baseline control-group implementations of
the TWSC that used either manual or no switching (i.e., normal PID) while task weight varied
during motion can be seen in Figure 7.36 (0 lbs to 5 lbs) and 7.37 (5 lbs to 0 lbs). For both Figures,
the slow speed results are shown in (a) and the fast speed results are shown in (b).
Looking at the results obtained from both sets of weight-varying trials (0 lbs to 5 lbs and
5 lbs to 0 lbs), some general trends can be observed for the different weight change conditions
and movement speeds. As in Experiment 1, the RMSE values for the slow speed trials were
significantly lower than the fast speed trials (likely for the same reasons, that slower speed motion
is easier for the actuator achieve). For the weight change conditions, it can be observed that the
LDLJ values for Condition 2 (weight change occurring in the brief rest period at the motion peak
when flexion turns into extension) was much lower than for the other two conditions (changing at
the midpoint of flexion or extension). This finding makes sense when considering the effect that
varying task weight during operation has on the brace. Since the external load is changing while
the brace is not moving in Condition 2, the impact of this disturbance on the system dynamics is
lessened as the actuator is not applying any compensatory torque at the time. When the change
occurs during motion instead, it will happen in the presence of applied torque from the actuator
(that is currently being used to move the brace). The effect of this sudden disturbance means
that the previous amount of torque that the actuator was supplying will no longer be suitable for
moving the brace and the system needs to react quickly to adapt to the new dynamics incurred
by the changing task weight. This may result in more aggressive movements by the device as
7.6 Results 301
0.4
RMSE [deg]
0.35
0.3
0.25
0.2
32
31
NRMSE [%]
30
29
28
30
LDLJ
25
20
Man. SG0 SG3 SG5 Man. SG0 SG3 SG5 Man. SG0 SG3 SG5
0 5 lbs Cond. 1 (Flex.), Slow Speed 0 5 lbs Cond. 2 (Mid.), Slow Speed 0 5 lbs Cond. 3 (Ext.), Slow Speed

20
RMSE [deg]
15
10
28
NRMSE [%]
26
24
22
25
20
LDLJ
15
0 5 lbs Cond. 1 (Flex.), Fast Speed 0 5 lbs Cond. 2 (Mid.), Fast Speed 0 5 lbs Cond. 3 (Ext.), Fast Speed
Figure 7.36: Box plots showing the RMSE, NRMSE, and LDLJ distributions from simulations of
(a) slow and (b) fast speed motion trials where weight varied during movement from
0 lbs to 5 lbs. TWSC iterations are labelled where “SG” indicates static gains (no
switching, number corresponds to the weight for which the gains were optimized) and
“Man.” indicates manual switching. Boxes are grouped by weight change condition:
“Flex.” (flexion), “Mid.” (midpoint), and “Ext.” (extension).
7.6 Results 302
0.4
RMSE [deg]
0.35
0.3
0.25
0.2
31.5
NRMSE [%]
31
30.5
30
29.5
30
LDLJ
25
20
5 0 lbs Cond. 1 (Flex.), Slow Speed 5 0 lbs Cond. 2 (Mid.), Slow Speed 5 0 lbs Cond. 3 (Ext.), Slow Speed

20
RMSE [deg]
15
10
28
NRMSE [%]
26
24
22
25
20
LDLJ
15
5 0 lbs Cond. 1 (Flex.), Fast Speed 5 0 lbs Cond. 2 (Mid.), Fast Speed 5 0 lbs Cond. 3 (Ext.), Fast Speed
Figure 7.37: Box plots showing the RMSE, NRMSE, and LDLJ distributions from simulations of
(a) slow and (b) fast speed motion trials where weight varied during movement from
5 lbs to 0 lbs. TWSC iterations are labelled where “SG” indicates static gains (no
switching, number corresponds to the weight for which the gains were optimized) and
“Man.” indicates manual switching. Boxes are grouped by weight change condition:
“Flex.” (flexion), “Mid.” (midpoint), and “Ext.” (extension).
7.7 Discussion 303
it attempts to compensate, which will result in higher jerk levels and decreased comfort for the
user. Regarding the two weight level variations simulated, there is little difference observed in the
distributions between the two groups. Any minor differences in the median values, or distribution
ranges, for the two groups are very minor and likely not statistically significant. This implies that
the impact of suddenly adding or removing weight had a similar effect on system performance.
The primary result from this experiment, towards the stated goal of evaluating the TWSC, is
the comparison between the manual-switching TWSC and the static gain PID controllers. Similar
to Experiment 1, the results once again showed little difference between using PID control with
static gains and using the TWSC. Looking at the distributions for the results from each controller,
they exhibit extremely similar values for all weight change conditions and weight level variations
simulated. Also, the static gains themselves once again showed a high level of similarity between
their results distributions, again implying that the system itself was not sensitive to variations in
controller gains (even for the more dynamic task simulated here). Again, as with Experiment 1
this is exemplified further when visually inspecting the output position measurements obtained
from the simulator. Looking at the example fast speed, 5 lbs to 0 lbs, motion trial using weight
change condition 1 (midpoint of flexion, as defined in Section 7.3.6.2) shown in Figure 7.38, there
is almost no difference between the output position for the three static PID gains and manual
switching (and any variation that does exist can only be observed if zooming in on the figure
substantially). Ultimately, these findings show that utilizing a TWSC in the modelled system for
tasks where weight changes dynamically during motion does not result in better performance over
using a traditional PID controller, at least in the way that it was simulated. Combined with the
results from Experiment 1, it was found that for either type of task (weight varying in between
motion or during motion) the TWSC failed to demonstrate significantly different performance from
the conventional control method.
7.7 Discussion
The objective of the work presented here was to develop a method of integrating the previously
studied EEG–EMG fusion-based task weight classifiers into the control system of a wearable mecha-
7.7 Discussion 304
150
Man.
SG0
SG3
SG5
100 Input
50
0
0 0.5 1 1.5 2 2.5 3
Time [s]
(a) Full trial
150
Man.
SG0
149 SG3
SG5
Input
148
147
146
145
1.92 1.94 1.96 1.98 2 2.02 2.04 2.06
Time [s]
(b) Zoomed in
Figure 7.38: Measurements of the brace output position obtained when using manual and no
switching control, taken from an example fast speed, 5 lbs to 0 lbs, weight change
condition 1 (midpoint of flexion) motion trial. The input reference trajectory is shown
as well, for comparison. The position results from the full trial are shown in (a), while
(b) provides a zoomed-in view of one portion of the trial to highlight the differences
between each measurement more effectively.
tronic device for the ultimate purpose of improving system adaptability to changes in external load.
To this end, a novel control approach (inspired by the well-established Gain Scheduling technique
and switched-system control approaches) was designed, referred to here as a Task Weight Selective
7.7 Discussion 305
Controller. This method uses task weight predictions supplied by an EEG–EMG fusion classifier
to select between various sets of PID gain values that are optimized specifically to each detected
task weight. To evaluate this control system, an upper-limb mechatronic brace simulator was
developed to model both the robotic device and the human user, allowing for an offline analysis
of performance. The implementation of the simulator, as well as the TWSC, has been outlined
here in detail, contributing a starting point for future development, as well as an initial test of
the methods presented. Continuation of this work will serve to further the development of adapt-
able control methods for wearable mechatronic devices, improving the user’s experience during
activities of daily living and leading to increased adoption of these novel tools.
7.7.1 Simulated TWSC Performance
Regarding the performance of the TWSC, it is obvious from the results obtained during the
simulated experiments that it has yet to conclusively demonstrate an objective improvement over
a traditional PID controller (utilizing static gains). For all tasks and conditions, the distributions
of the performance metrics calculated from all subjects were very similar, to the point where it can
be assumed that there is no significant difference. This pattern was not just present in an analysis
of group performance across all subjects, but was observed at the individual subject level as well.
When looking at the performance metric values calculated for a single subject (excluded from
the results presented here to simplify reporting, due to the large number of values this entails)
there is little difference between the control system iterations that utilized a TWSC and ones
that use a PID controller (hence, reporting all values here provides little benefit, as the results
are almost the same). Calculated metrics aside, even the measured output motion trajectories
from the TWSC and the PID control systems would appear nearly the same when comparing
them visually, indicating that the performance/behaviour of the simulated brace did not change
meaningfully for different control techniques.
Based on these findings, it may lead one to initially assume that the TWSC is overall worse
than traditional PID control methods, and therefore has no merit; however, upon further analysis
of the TWSC results and reflecting on the design of the simulator, this statement is more than
likely an inaccurate oversimplification. Knowing how actual wearable upper-limb mechatronic
7.7 Discussion 306
braces behave in real-world settings, it seems counter-intuitive that simple PID control would be
sufficient to operate the device under changing task weight conditions. It is well documented in the
literature that one of the major challenges currently facing the field of wearable rehabilitation and
assistive robotic devices is a lack of adaptability to changing-use-cases [6, 11–13, 22, 24, 26, 30, 32–
45]. Control system models in such devices often need to be tuned for one specific set of dynamics,
and any attempt to operate the device outside of these parameters will necessitate a re-optimization
procedure (or the utilization of more complex control law approaches). Similarly, operating the
device in the presence of unexpected/unknown external disturbances can result in unstable system
behaviour [6, 13, 22, 26, 27, 30, 32, 33, 37, 39, 42, 101]. This alone presents a huge barrier towards
the widespread adoption of these tools, as users feel that they are too constrained in what tasks
they can perform with the brace (making it not suitable for operation in daily life). Why then
do the results here seem to contradict this fact (by showing that basic PID control can work
adequately across many tasks)? Upon further thought, there is a compelling argument to be made
that this finding is actually a result of the simulator itself, and not based on the merits of gain
switching-based control (or lack thereof).
When experimenting with different values for the model parameters of the simulator, it became
apparent that actuator selection played a largely dominant role in the performance and behaviour
of the system, significantly more so than the impact demonstrated by the modification of controller
gains. Either the chosen actuator used in the model was strong enough and could follow the input
reference trajectory perfectly, or the chosen actuator could not provide enough torque and would
never match the system input. Instead, the weaker actuator would always lag slightly behind the
input reference trajectory, as it would reach its torque and/or current saturation limits and already
be, insufficiently, moving as fast as it could under the current weight conditions. Examples of both
of these situations can be seen in Figure 7.39. Proof of the dominance displayed by actuator
selection in the output behaviour of the simulator model is further demonstrated by the results
obtained from the static gain (i.e., conventional PID) control-group experiments for task weights
varying both in between, and during, motion. For all calculated performance metrics from each
subject, it can be observed that the distributions for the static gain controller simulations are
extremely similar, despite the fact that the proportional gains were optimized to specific weight
7.7 Discussion 307
levels. One would expect that each static gain PID controller utilizing the gains optimized for a
specific task weight would perform best under that external load (and worse for the other two);
however, in reality the results obtained from all three controllers show similar results, regardless of
the current task weight being simulated. While it is possible (and, in fact, almost guaranteed given
its relative simplicity) that the P-gain tuning method was sub-optimal and could be improved via
more complex techniques, one would still expect that different gains would have some meaningful
effect on system performance. It is highly likely that this lack of change in the system observed
when varying gain values is because it is really the actuator model (and its associated strength
parameters) that is ultimately driving the output response of the system.
(a) Weak actuator (b) Strong actuator
Figure 7.39: A screenshot of a Simulink scope that compares the brace output angular position
(yellow solid line) to the input reference trajectory (blue dashed line) for an exam-
ple fast speed, and 3 lb task weight motion trial simulation. (a) shows the output
behaviour of a system using an actuator that is too weak to move the brace at the
required velocity: the output position consistently lags behind the desired trajectory
since the actuator torque limit has been reached and it cannot move any faster. (b)
shows the output behaviour of a system using a strong actuator: the torque limits
are never reached, thus the brace can perfectly follow the reference trajectory (since
it can always move as quickly as is required).
Therefore, accepting this premise, the question then becomes why is it the case that the actuator
parameters dominate system performance? Upon consideration, this is likely a consequence of
simplifications of the model made early on in the development of the upper-limb brace simulator.
To make the task of modelling the system easier, or even possible to implement with the dataset
7.7 Discussion 308
available at the time, it was decided to make the assumption that the device was being used by
a patient who has lost complete control of their arm due to their injury. This means the user is
not applying any force onto the brace actuator themselves, and is instead treated as, essentially, a
static load onto which the brace applies an assistive torque. The logic applied at the time was that
this would provide a “worst case” scenario for the device, as it is expected to move the patient arm
entirely on its own with no support torque being applied by the person’s arm muscles. Thus, when
applying this assumption to the actuator model used for the simulator, the load incurred from the
mass of the subject’s arm, the dumbbell, and the motor/gearhead are combined together into a
single total inertia value, greatly simplifying system dynamics and making the task of implementing
the simulator much easier. To conceptualize this model in its simplest form, the brace and user are
essentially analogous to a one Degree of Freedom (DOF) rotary-joint serial robotic manipulator
(the brace) that functions to position a mass at the end of a link (the user/external load) at
specified angles. A visualization of this concept can be seen in Figure 7.40(a). If analysing the
brace model through this lens of a single rotary joint moving a fixed load, it starts to become more
apparent why actuator selection was the primary indicator of performance and why the system
could function adequately with only a PID controller (that was robust to even small variations in
gain values). From a control perspective, this is an extremely simple task (to control the angular
position of a single motor), therefore it makes sense that basic PID control would be adequate as
long as the actuator can supply enough power to move the load at the speed required to match the
input reference trajectory. Despite the efforts that went into incorporating the various properties
of the human and the device response into the simulator, it still does not capture the complex
interaction behaviours of the user and the wearable mechatronic brace that would be present in
a real-word device, which is ultimately what necessitates the use of an adaptable control method
such as the TWSC in the first place. Thus, while it is true that the results presented in this
work would suggest that the TWSC is not a viable control strategy, it is more likely that the
unsatisfactory results obtained are actually a failure of the upper-limb brace simulator itself and
not the TWSC. In reality, the simulator model should not be considered as a single rotary joint,
but instead as something akin to a two DOF parallel robotic manipulator, where one joint is the
actuator and the other is the torque contribution from the human user’s muscles, acting upon each
7.7 Discussion 309
other to move a load. This concept is visualized in Figure 7.40(b). Using this analogy, it becomes
clear that the “worst case” for the brace is not actually a user with no motor function, but in fact
a user that has partial or full muscle control. This greatly increases the complexity of the control
problem as situations may occur where the user actually works against the brace actuator (moving
in the opposite direction and pushing against the brace with an opposing torque), which would
necessitate even more torque capability from the motor than moving just the weight of the user’s
arm, or there may be instances where erratic behaviour from the user causes unexpectedly changing
dynamics (for example, suddenly changing the amount of torque that they are applying due to
their injured muscle unexpectedly failing from spasm or fatigue). It is also important to consider
that fully modelling the human user is much more complicated than the simplifications applied
here. For this work, the human user was incorporated into the simulator model by considering
how the inertia of their arm acts on the brace during movement. In reality there are many other
physiological factors that affect how the person can move, such as internal friction of their joints
[6, 50, 294, 377] and muscle stiffness/spasticity [37, 45, 99, 101, 294, 377, 378], which should also
be incorporated into the model to achieve the most realistic behaviour possible. While the hope
was that the simpler model employed for the simulator in this work would be enough to provide
a basic proof-of-concept evaluation of the TWSC to act as initial testing data to inform future
development, it is apparent (based on the results obtained) that to be able to truly evaluate the
TWSC, a more complex human–brace model is required.
To achieve this, the control system model of the upper-limb brace needs to be updated to the
design shown in Figure 7.41. Instead of the system plant consisting of only the BLDC motor/gear-
head actuator model (as what was described in the closed-loop feedback control system shown in
Figure 7.18), it instead needs to also incorporate a human model that specifies the torque response
of the user during device operation. When this is combined with the torque contribution of the
actuator, it will provide a more realistic portrayal of the behaviour of the brace. Developing an
accurate model of human body movement is not a simple task by any means, and is an active area
of research in this field. Future work should investigate different methods of modelling human
movement dynamics to see which will provide the best response to use in the simulator. Some
possible areas of focus could be biomechanical physiology-based EMG muscle modelling techniques
7.7 Discussion 310
(a) One DOF serial manipulator
(b) Two DOF parallel manipulator
Figure 7.40: Robotic manipulator configuration diagrams used to conceptually demonstrate the
incorporation of device–human interaction effects in the upper-limb brace simulator.
The concept behind the current design of the simulator is shown in (a), represented
with a one DOF serial manipulator. Since only the human user’s inertia is considered
in the simulation (based on the assumption that they have a complete loss of motor
function) then their effect is only considered in the total mass of the load being
positioned by the manipulator. (b) provides a more realistic concept for the simulator,
where it can be considered as a two DOF parallel manipulator with the actuator and
the human user’s muscle forces acting upon each other while the mass (the user’s
arm) is being positioned.
(such as the Hill model, to provide one example) [99–101] and/or machine learning-based methods
for calculating human motion dynamics using model-free, data-driven, techniques [99]. Properly
implementing these human models will necessitate the collection of a new dataset that measures
a wider range of factors. The dataset used for the work here was chosen mainly as a consequence
7.7 Discussion 311
of the COIVD-19 pandemic (since new data could not be collected during quarantine protocols)
and, unfortunately, it is too limited to be used for developing complex human–robot interaction
models in a meaningful capacity. Future work should include the development of a new data
collection protocol that measures the subject’s movement trajectories (likely through the use of
IMUs, or camera-based motion tracking systems), the applied elbow torque from the human user,
and the torque being supplied by the actuator (to understand the interaction forces between the
subject and the wearable robotic brace). Possible methods of achieving this could be through
the use of a dynamometer/load-cell system that measures elbow-joint force [37, 45] and/or pres-
sure/force sensors on the brace cuffs to measure interaction force as the user pushes against the
device [32, 47, 288]. With this new dataset, an updated version of the mechatronic upper-limb
brace simulator can be developed, which will allow the initial testing of the TWSC to be fully
completed.
Figure 7.41: A block diagram showing the control system improvements required for the simulator
model to demonstrate brace–human interaction effects, allowing for a proper evalua-
tion of the TWSC. The system plant needs to be extended to include a model of the
human user’s behaviour to be included alongside the actuator model.
7.7.2 Classifier Prediction Debouncing Effectiveness
Even though the evaluation performed could not provide a definitive comparison of the TWSC and
conventional PID control (due to limitations in the simulator), this does not mean that the results
obtained here are completely unusable. Beyond highlighting the areas of improvement required
for the simulator (as discussed above), these results also demonstrated an interesting trend with
7.7 Discussion 312
regards to how debouncing classifier predictions in the TWSC affected performance. When looking
at the LDLJ results of Experiment 1 (task weight variation in between motion), a pattern emerged
when comparing the raw-prediction TWSC iterations (no debouncing) with the TWSC iterations
that utilized debouncing. For the simulations were debouncing was used, the distribution of LDLJ
values trended lower, with this pattern being present for all motion trials (although it is slightly
easier to see in the slow speed plots). While far from a conclusive result (given the small differences
in the actual LDLJ values observed), the fact that this trend was present for all trials indicated
that there may be some interesting effect here that is worth analysing further. Based on these
findings, it appears that applying debouncing techniques to classifier predictions as a filtering step
will result in lower amounts of jerk exhibited by the brace during movement, which ultimately
translates to a more comfortable user experience.
When considering the effect that debouncing has on the prediction control signal, and how this
impacts the actions of the TWSC, this finding makes sense. A brief misclassification driving the
TWSC will cause its dynamics to rapidly change between task weight levels, as it adjusts gains
for the “new” task weight only to then quickly switch back after the missclassification is gone.
This action will likely trigger an aggressive response from the motor as it attempts to rapidly
adapt, which will apply uncomfortable forces onto the user. Using debouncing removes/reduces
these scenarios by filtering the prediction control signal in real time each instance the classifier
outputs a new task weight prediction. Looking again at the results of Experiment 1, it can also
be observed that when more strict debouncing (longer delay value or larger vote window) was
used, the LDLJ distributions would trend further down. This is likely an extension of the same
behaviour: as debouncing aggressiveness is increased, even more misclassifications are removed,
leading to fewer situations in which the system dynamics rapidly change. It should be noted
however that, regarding this effect, there does appear to be diminishing returns as the strictness of
debouncing is increased. After the second level of both debouncing techniques (a delay value of 3
and a vote window size of 4) there was no noticeable reduction in the LDLJ distributions, meaning
that there is no reason to continue increasing the levels of either method, as all this accomplishes
is to increase the time it takes for the system to react to changes in task weight (since it will
take more examples of the same prediction value for the system to consider it a true change).
7.7 Discussion 313
Between the two debouncing methods themselves, there did not seem to be any trend indicating a
consensus as to whether one method was better than the other: they both showed similar results,
making any kind of definitive ranking difficult. Future work should continue investigating both
the Hold-Delay and the Majority-Vote debouncing methods to further evaluate the effectiveness
of these approaches.
Comparing the debounced TWSC iterations to the static gain TWSC baseline control group
when used for motion trials used with their respective optimized task weight (analogous to a
100% accurate classifier) provided further evidence of the potential benefits of this approach.
The LDLJ distributions for the simulations that used debouncing matched the pseudo-manual
switching simulation results very closely, while the raw prediction simulations instead appeared to
demonstrate a visually distinct LDLJ distribution. Based on this finding, it can be reasoned that
using debouncing methods to filter the classifier predictions demonstrated the effect of pushing
the TWSC performance closer to that of an ideal system with perfect task weight detection, even
if the classifier actually used in the TWSC is unable to achieve 100% accuracy. It is likely that
this behaviour is due to the debouncing method removing some misclassifications as the filtering is
applied, essentially serving to boost the performance of the classifier post training. Knowing that
these novel debouncing techniques can potentially help compensate for an imperfect classification
model is an important finding given the reality of the performance one can expect from a machine
learning model in the real world. The goal during training is to make a classifier that is as accurate
as possible; however, creating a model that is truly 100% accurate in all circumstances is most
likely unachievable (or at the very least extremely unlikely, given the current understanding of
the limitations of machine learning technology). Therefore, it is vital that the TWSC is robust
enough that it can still be effective when incorporating realistic (i.e., imperfect) classifier models,
and this work demonstrated that the use of debouncing techniques show potential as one tool in
ensuring this property. Incorporating this filtering approach will allow the TWSC to be used with
less-optimized classifiers if needed, helping remove cost/time from the development of wearable
upper-limb mechatronic devices, since the trade-off between model training and resource allotment
can be less strict knowing that debouncing techniques can boost classification accuracy.
Interestingly, these benefits did not seem to come at any cost to brace positioning accuracy: the
7.7 Discussion 314
RMSE distributions for both the raw-prediction and debounced-prediction TWSC simulations were
essentially the same, indicating that accuracy was similar across all iterations. When developing
the concept for the debouncing techniques used in the TWSC, one hypothesis considered was
that using debouncing may introduce too much of a delay in the ability of the system to react
to changes in task weight, which could affect system performance. That does not seem to be
the case here based on the observed results; however, it should be mentioned that the relative
insensitivity of the control system to changing PID gains (discussed previously when outlining the
limitations of the simulator) may be contributing to this behaviour as well. Further evaluation of
the debouncing methods with the updated simulator should be completed in future work to more
definitely quantify their effectiveness; however, these initial results presented here do show promise
for the validation of these techniques.
The evaluation completed for this project showed that debouncing the classifier output, even
when only using a minor amount of filtering, resulted in a pattern of smoother motion from the
simulated brace during movement (leading to a more comfortable experience for the user). This
trend indicates that prediction debouncing may be a beneficial technique to employ when seeking
to integrate machine learning classifier models into a real-time control system. While no strong
claims can be made yet given the relative similarity of the compared values (and the fact that
no statistical significance was tested for in this work), the results obtained do show a promising
potential for this approach and serve as an initial proof-of-concept for the techniques established in
this project. This provides justification for the continued development of these methods in future
work to further investigate the potential benefits they may offer to improve wearable robotic device
control performance.
7.7.3 Future Work
As mentioned above, the immediate next steps for the TWSC project is to develop an updated
version of the upper-limb brace simulator that incorporates a model of more complex human
operator dynamics into the system; however, beyond this there are several other objectives on which
future work should focus to continue the investigation into the performance task weight selective
control. When collecting a new dataset to use for updating the simulator, it would be worthwhile
7.7 Discussion 315
to increase not only the number of measurement signal types collected, but also to diversify the
motion tasks recorded as part of the experiment protocol. The dataset used for this work was fairly
limited with only three task weights, two speed levels, and one motion (elbow flexion–extension)
being recorded. Increasing the number of speed weight combinations, and recording signals during
other movements/tasks, will allow for a stronger evaluation of the TWSC, as it can be tested under
a larger range of conditions. In particular, including trials where weight varies during movement
in the new protocol would allow for Experiment 2 in this work to be redone using actual data
recordings, as opposed to requiring the creation of synthetic data (which will ultimately provide
more realistic evaluation results). At the same time, while updating the simulator, it would also be
worthwhile to consider other tuning methods for selecting optimal PID gain values. The method
used here was somewhat limited and still required some trial-and-error during implementation,
which is obviously not an ideal approach to use if looking to eventually transition to a fully-
deployed commercial device where any required calibration procedures should be fast and easy
for the user to perform. Future work should look into other methods of automatically tuning the
PID gains for each task weight, to improve the usability of the system. Machine-learning based
methods of tuning PID gains (for example, using Reinforcement Learning) have demonstrated
a lot of success in recent years [283, 290, 379] and could potentially be utilized in this work to
improve controller optimization. Finally, it would also be beneficial to compare the TWSC to
popular control system approaches other than static-gain PID. The comparison performed here
was limited to TWSC and PID, mainly due to time-constraints and, given that this evaluation
was meant to act as an initial proof-of-concept for the TWSC, it was felt that this comparison
would be enough to start the investigation. However, it would be of particular interest to compare
the TWSC to other state-of-the-art adaptive and robust control techniques, to see if the TWSC
can provide a similar amount of adaptability to wearable robotic devices during changes in task
weight.
During the updated brace model simulations, it may also be worthwhile to expand the perfor-
mance metrics used for evaluation to extend the analysis to other aspects of brace performance.
This work focused mainly on device output position as the main focus of evaluation (which makes
sense for the initial testing, since the primary objective of the feedback control system is to regulate
7.7 Discussion 316
the angular position of the upper-limb brace); however, there are other measurements taken during
the simulations that can provide interesting information regarding system behaviour. Looking at
the motor torque and current signals directly would allow for an analysis of the power usage and
efficiency of the system, which is of particular interest for wearable devices where available power
may be limited. There may be trade-offs between position accuracy and power usage: for example,
a more accurate iteration of the TWSC might require an aggressive motor response that uses more
current over time. Given that wearable mechatronic systems are typically powered using batteries
with a limited supply, decisions will need to be made about whether accuracy improvements are
worth sacrificing the usage-time of the device. Including secondary analysis objectives into the
evaluation of the TWSC will provide a more comprehensive understanding of its performance,
which will help guide design decisions later when developing new wearable mechatronic systems.
Lastly, future work should also look at ways of incorporating EEG–EMG fusion-based task
weight classifiers into different control techniques other than PID Gain Scheduling. PID control
was chosen as the basis for the TWSC developed here because of its ubiquity across many industries,
its ease of implementation, and the fact that its model-free nature makes it a good candidate for
implementation in wearable robotic devices (where obtaining an accurate model of the human and
device is a very challenging task); however, it is far from the only viable control approach that can
be utilized with wearable upper-limb mechatronic devices. Other control-theory methods focused
on adaptive and robust control may also benefit from the integration of these machine learning
models and could also be used to help improve the ability of wearable upper-limb devices to adapt
to variations in task weight. One potential candidate for this investigation is Active Disturbance
Rejection Control (ADRC). ADRC is a robust control technique that uses an extended observer to
measure all internal and external disturbances (where the internal disturbances are assumed to be
caused by model uncertainties), then combines them into a single component to be removed from
the system output (based on its deviation from the “ideal” input reference) [366, 367, 380, 381].
Treating the errors caused by an unknown plant model as a disturbance removes the problem of
system identification from control law development, as the ARDC method needs to focus only on
disturbance estimation and rejection without needing to consider a model at all [367, 380]. This
provides the main draw of ADRC: like PID, it does not require an accurate model of the system
7.7 Discussion 317
(thanks to the extended observer) and can be tuned experimentally, simplifying its implementa-
tion greatly [366, 367, 380] (unlike other modern non-linear control approaches that do require a
detailed system model to be implemented properly). ADRC is a control technique that takes ad-
vantage of modern state-space control theory, but can be implemented easily using experimentation
in the same manner as classical control approaches (i.e., PID), making it an attractive option for
use in industrial applications. Some groups have proposed ADRC as the replacement for the long-
standing and ubiquitous PID control [366, 367], thus it is likely that it may also benefit wearable
devices. Future work should investigate ways of utilizing the EEG–EMG fusion-based task weight
classifiers with ADRC to see how it compares to the current implementation of the TWSC based
around PID Gain Scheduling. Likewise, future work should also investigate the effect of integrating
EEG–EMG fusion classifiers into other control approaches (not just ADRC). While the focus of
this work was on model-free control methods specifically, it has been well documented (as discussed
in Section 7.1) that modern non-linear adaptive/robust control methods have been successful in
improving wearable upper-limb robotic device performance during the presence of external distur-
bances [38–42] and that the use of machine learning techniques can help to compensate for the
model dependency present in these methods [34, 35, 50, 286, 287, 291, 291, 292]. To completely
ignore this branch of control theory entirely would be foolish; therefore, it is worthwhile to develop
methods of integrating EEG–EMG fusion-based task weight classifiers into these other nonlinear
control laws, to investigate potential performance improvements that it may offer. Given its popu-
larity for disturbance rejection in upper-limb wearable robotic systems [30, 34, 35, 40, 41, 50, 287],
Sliding Mode Control seems a likely candidate for this analysis; however, other adaptive/robust
control methods could be included in the evaluation as well. Further development of task weight
selective control will help improve the adaptability of wearable upper-limb mechatronic devices,
which will ultimately lead to a better user experience and help increase adoption of these novel
tools for assistance and rehabilitation.
7.8 Conclusion 318
7.8 Conclusion
This work proposed a new control system approach to use with wearable mechatronic upper-
limb devices for assistance and rehabilitation, referred to as Task Weight Selective Control. With
the goal of improving device adaptability to varying external loads (referred to as task weights),
the concept behind this technique was a controller that integrates previously developed EEG–
EMG fusion-based task weight classifiers within a well established control system method so that
the output predictions of the classification model can be used to update the controller when
the user changes the weight that they are holding. As the foundation for the TWSC, it was
decided to base its underlying control approach on PID Gain Scheduling, given the ubiquity of
PID control and its ease of implementation as a model-free method. To evaluate the TWSC,
a wearable mechatronic upper-limb brace simulator was developed in Simulink using a model of
the brace actuator (BLDC motor and gearhead) along with subject body measurements from a
previously collected dataset. This chapter describes the implementation procedure of both the
upper-limb brace simulator, as well as the TWSC, to contribute detailed design and methodology
documentation to serve as a starting point for future development. Due to limitations of the
simulator not capturing complex human–device interaction effects, no iterations of the TWSC
were able to demonstrate increased performance over traditional PID control; however, the results
obtained were beneficial for highlighting the simulator improvements required to allow for a full
investigation into the performance of the TWSC. The evaluation did discover an interesting trend
where utilizing debouncing filtering techniques on the classifier output predictions resulted in
smoother motion (i.e., less jerk) and it was observed that debouncing can be used as a method
of improving TWSC robustness to the effects of imperfect classification models (by essentially
boosting classifier accuracy during use through the removal of misclassifications). Future work
should focus on the development of an updated simulator that incorporates a model of the human
user’s movement dynamics and the complex human–actuator interaction effects. Completing this
objective will necessitate the collection of a new dataset, so future work should also look into
expanding the diversity of the movements recorded during execution of the experimental protocol
(more speed–weight combinations, different motion tasks). Doing so will allow for a full evaluation
7.8 Conclusion 319
of the TWSC to objectively determine how beneficial it can be for wearable mechatronic devices.
Once this is established, further development of the TWSC can commence, and an investigation
can be done to discover more novel ways of incorporating EEG–EMG fusion machine learning
classifiers into other control system methodologies (with Active Disturbance Rejection Control
being a good candidate for initial evaluation). Continuing the development and refinement of
the control system approach presented here will ultimately result in wearable mechatronic devices
that are able to adapt to changing loads more effectively during assistance and rehabilitation tasks,
which will serve to improve the user experience during operation and lead to better adoption of
the system.
Chapter 8
Conclusions and Future Work
Wearable robotic devices have shown great potential as a rehabilitation tool for upper-limb im-
pairments; however, there are still several barriers present in the technology that prevent their
widespread use. One of these is the method of user control: the device needs to respond to the
user in an intuitive and comfortable manner, and should be robust enough that it can maintain
consistent performance across a range of daily activities. One method of control that has been
demonstrated extensively in the literature is the utilization bioelectrical signal measurements to
determine operator intent. By decoding the user’s brain (EEG) and muscle (EMG) activity the
wearable mechatronic system can use this information to understand the current task and respond
accordingly.
With modern advancements in computing technology, and the explosion in popularity of
machine-learning solutions, the idea of sensor fusion (combining information from multiple sources
to make decisions) has become a new trend in recent years, and there is evidence that this approach
can also benefit control systems for wearable upper-limb robotic devices. Previous studies have
demonstrated EEG–EMG fusion-based classification for basic motion intention tasks performed
during simple movements, showcasing its feasibility. Therefore, the purpose of this thesis was to
expand the use of EEG–EMG fusion to more complex classification tasks, and to utilize it within
a wearable device control system to improve adaptability: focusing on both the investigation of
new machine-learning methods of EEG–EMG fusion and the development of a new control system
based on EEG–EMG fusion classifier integration.
320
321
An initial analysis of EEG–EMG fusion methods, performed to compare various fusion tech-
niques previously reported in the literature, under dynamic movement scenarios with changing
motion conditions. Through the collection of an EEG/EMG signal dataset, recorded during el-
bow flexion–extension movements at different speeds, while holding various weights, and during
increased levels of muscle fatigue, multiple fusion methods were implemented using trained EEG
and EMG classifiers, and were compared for motion intention detection accuracy.
The next project completed for this thesis focused on expanding the use of EEG–EMG fusion to
a more complicated classification task. Previous EEG–EMG fusion work had primarily utilized this
approach for basic binary motion classification models, and a detailed investigation of EEG–EMG
fusion for other multi-class decisions needed to be demonstrated. To this end, it was decided to
evaluate EEG–EMG fusion for task weight classification (determining the weight a user is holding
during elbow flexion–extension motion), as this information could be employed by a wearable
device control system to adapt to such external disturbances and had not been demonstrated
previously. EEG and EMG classifiers were trained using the collected dataset and methods of
fusing the two signals types for multi-class models were developed and evaluated.
Following the first demonstration of task weight classification using EEG–EMG fusion, it was
decided to investigate the potential of modern deep-learning approaches of machine learning as
a new method of input-level EEG–EMG fusion, namely through the use of Convolutional Neural
Networks. Previously, EEG–EMG fusion was performed mainly with traditional machine-learning
methods (that require the manual calculation/selection of features to use as classifier inputs);
however, deep-learning classifiers are able to automatically extract feature information from the
input data directly (optimizing this as well during training), and it was hypothesized that a
CNN may be able to discover aspects of the EEG–EMG relationship not currently detected from
manual features. To investigate this, CNN task weight classifiers were trained using a combination
of EEG and EMG as inputs, exploring many different ways of representing the signal inputs, and
methods of convolving them inside the CNN, to see how each method of fusing the EEG and
EMG information affected classification accuracy. In a follow up study, multiple methods of image
normalization, used to pre-process the EEG/EMG signals, were also evaluated to further improve
model performance and iterate on the previous designs.
8.1 Contributions 322
With the feasibility of EEG–EMG fusion-based task weight classification well established, the
final project of this thesis focused on the actual control system of a wearable robotic device;
looking at ways to incorporate the task weight classifier into its controller to improve system
adaptability (without the use of complex model-based control law techniques). To that end, a
new control system approach was proposed, referred to as a Task Weight Selective Controller.
The technique utilized a Gain Scheduling-based approach to adaptability, selecting which PID
gains to use depending on the task weight being held, with this information being supplied by
the EEG–EMG fusion task weight classifier. Methods of implementing this system in real time
were considered, and the utilization of classifier prediction filtering, by way of the developed
debouncing techniques, was proposed to improve the stability of the system. To evaluate this new
control method, a wearable upper-limb robotic brace simulator was developed and used to run
offline experiments to measure system performance.
8.1 Contributions
The work presented in this thesis showed that EEG–EMG fusion has many benefits that can be
used to improve wearable mechatronic device control. Through thorough investigation of task
weight and motion intention classification using EEG–EMG fusion, a series of models were devel-
oped and evaluated using a collected dataset. Investigation into the integration of these models
within wearable robotic devices resulted in the proposal of a new control system technique that
incorporates a task weight classifier to improve device adaptability. Ultimately, this work serves
to continue the improvement of wearable mechatronic devices for rehabilitation, through the use
of EEG–EMG fusion, to result in more adaptable systems that are easier for the user to control
and can be used safely across a variety of activities of daily living. The specific contributions of
this work are as follows:
Presented the results of a comparative evaluation of previously demonstrated EEG–EMG

fusion methods for binary motion classification, providing quantitative insight into their rel-
ative performance. The comparison testing undertaken here also evaluated the accuracy of
EEG–EMG fusion methods under dynamic elbow flexion–extension movements with chang-
ing motion conditions, investigating the effect that these factors have on EEG–EMG fusion
accuracy. Previous work on EEG–EMG fusion has mainly evaluated each method separately
[67, 68, 78, 206, 211], and for simple tasks not comparable to the conditions one would expect
when operating an upper-limb wearable mechatronic device [77, 206]; therefore, a more com-
prehensive comparative evaluation of their performance was required to address this gap in
the literature. The results of this study showed that EEG–EMG fusion could provide more
consistent model performance across changing motion conditions (movement speed and load
being held) for subject-independent models. This is a crucial trait to have in wearable device
control, where users will constantly be switching between various activities of daily living
(modifying motion conditions for each different task).
Expanded EEG–EMG fusion techniques to be used with more complex multi-output classi-
fication problems. Previous use of EEG–EMG fusion had been demonstrated primarily for
basic binary motion classification models [75, 78–89], and a detailed investigation of EEG–
EMG fusion for other movement related multi-class decisions needed to be completed. This
work showcased fusion methods (previously utilized mainly for binary classification) adapted
for use with multi-class models, providing an example for decision-level fusion (Weighted
Average method) and input-level fusion (Single Classifier method) for a three-class problem.
The performance of these approaches was then evaluated to determine their feasibility, which
showed that EEG–EMG fusion could outperform the use of EEG or EMG alone.
Successfully demonstrated the ability to classify a new measurement of the human user’s
intention with EEG–EMG fusion: task weight classification (i.e., the weight a user is holding
during elbow flexion–extension motion). This provides task information relevant to the
control system of a wearable robotic device, since it will need to compensate for changes in
external load during operation. Previous studies classifying force levels using EEG or EMG
alone focused on isometric muscle contractions while the subject is not moving [139, 144, 226–
228, 230, 231]; however, this does not provide direct task weight information, as force level
will not correlate directly to external load when a subject is moving at variable speeds
(making task weight a more challenging classification task). This work was able to show that
such a goal is possible, obtaining accuracy scores for three-class task weight classification
models using EEG, EMG, and EEG–EMG fusion above the chance level, and demonstrated
that EEG–EMG fusion-based classifiers out performed the use of only EEG or EMG.
Provided a set of EEG and EMG features that demonstrate potential in extracting task
weight information from the bioelectrical signals. Using a feature selection analysis, a series
of EEG and EMG features were found that were highly chosen by the majority of subjects,
indicating that they may correlate better to task weight changes and should be investigated
further in future development of task weight classification models. Since task weight classifi-
cation had yet to be demonstrated in the literature, the decision of what EEG/EMG features
to extract was unclear, necessitating the investigation performed in this work. At the same
time, this project also evaluated the effect of incorporating movement speed information into
the EEG–EMG fusion task weight classifiers, by way of including a categorical speed label
feature into the input dataset, and comparing it to models containing no speed information;
showing that the inclusion of speed information did result in improved classification accuracy.
Demonstrated the feasibility of utilizing CNNs as a new method of input-level EEG–EMG fu-
sion for movement related tasks. The use of CNNs with EEG-only and EMG-only classifiers
had been extensively reported in the literature [184, 185]; however, the use of this classifica-
tion technique for EEG–EMG fusion was not yet widely adopted. The ability of CNNs to
automaticity extract features from input data, which may be able to recognize interesting
relationships between EEG and EMG, led to the decision to study the performance of this
new EEG–EMG fusion approach through the development of CNN-based EEG–EMG fusion
task weight classifiers. The utilization of CNNs for EEG–EMG fusion resulted in classifiers
with higher accuracy than using EEG or EMG alone, proving this to be a viable method of
fusion.
Investigated the model design of CNN-based EEG–EMG fusion extensively, providing a

quantitative evaluation of different methods of fusing EEG and EMG within the CNN. By
developing models using different input image representations of the bioelectrical signals
(time–frequency domain spectrograms and directly using the time-domain signals), vari-
ous approaches of convolving the signals (single-channel and multi-channel image convolu-
tion, split and 1D signal convolution), and with multiple image normalization pre-processing
methods, this work was able to show which techniques proved most effective for CNN-based
input-level EEG–EMG fusion. The results described here provide important insight into
which methods to continue utilizing/refining during future work seeking to iterate on the
design of this fusion approach.
Introduced a new control system methodology for wearable upper-limb mechatronic devices
aimed at improving their adaptability to changing external loads: Task Weight Selective
Control. By incorporating the previously developed EEG–EMG fusion classifiers, this control
system operates using a Gain Scheduling-based approach where the appropriate set of PID
gains are selected based on the output prediction of the task weight classification model. This
implementation allows the TWSC to be executed in an efficient manner through experimental
tuning, not requiring intensive control law calculations, external load force sensors, or a
detailed system of the model to operate, as many previous control system approaches have
(factors which are particularly large barriers in the context of a wearable robotic device)
[13, 23, 24, 26]. To evaluate this new approach, an upper-limb brace simulator was developed
to facilitate an offline performance analysis, and details of its implementation, along with
the implementation of the TWSC, have been exhaustively documented in the work here to
allow this project to act as a starting point for future work.
Proposed two filtering methods to stabilize classifier output predictions when using control
systems with integrated machine learning models, improving device performance. Through
the use of the debouncing techniques devised for the TWSC, the rate of misclassifications
supplied from the EEG–EMG fusion-based task weight classifiers was reduced during de-
vice operation (post classifier training), as the design of the debouncing methods factored in
the real-time constraints incurred by a wearable mechatronic system. Evaluations of these
classifier prediction filtering approaches within the TWSC, using the upper-limb brace sim-
ulator, showed that their utilization resulted in smoother motion from the device (leading to
increased user comfort) with no loss in accuracy, and highlighted that prediction debouncing
8.2 Future Work 326
can make the control system more robust to imperfectly trained classifiers (loosening the
accuracy constraints required during model development and simplifying implementation).
8.2 Future Work
While the work described in this thesis has provided a thorough investigation into the use of EEG–
EMG fusion-based classification to improve the control systems of wearable upper-limb robotic
devices, there are still many improvements that can be made to this technology and are various
areas of focus that future research should consider as development continues. A detailed discussion
of limitations and potential future work for each project has been provided previously in their
respective chapters; however, a summary of the overall next steps for the work described in this
thesis, as well as a discussion of other future work not previously covered in other chapters, is
provided here. The areas of future work that should be considered for the next steps of this thesis
project are as follows:
As has been discussed, an area of improvement for all projects described in this work is
the collection of more EEG/EMG signal data. Acquiring a larger dataset, with more sub-
jects performing a greater number of motion repetitions, would benefit the development of
new machine learning classifiers greatly, as these models require large amounts of data to
effectively train generalized classifiers. Doing so will allow the development of more subject-
independent models to commence (a more desirable outcome than subject-specific models,
when considering the practicality of implementing these classifiers into a wearable mecha-
tronic system). A new data collection protocol could also improve the variety and detail of
the signals collected: increasing the number of EEG/EMG channels to obtain more infor-
mation regarding the bioelectrical response of the user, and including more combinations of
movement speed and task weight to expand the capability of the EEG–EMG fusion models
(and the TWSC). Finally, the new data collection protocol should consider collecting other
measurements related to human movement, such as motion data (via IMU) and applied el-
bow torque (using force sensors). Not only will this allow better movement speed information
to be incorporated into the EEG–EMG fusion models (which has already shown to improve
8.2 Future Work 327
performance), but it will also provide the necessary data required to develop an updated
version of the upper-limb brace simulator that includes human–robot interaction dynamics
(allowing for a full evaluation of the TWSC to be conducted).
Continue to iterate on the model design of the EEG–EMG fusion methods presented in this
thesis to further improve performance: both for the methods based on traditional machine
learning (decision-level fusion) and the ones that utilize deep learning (input-level fusion).
The work presented here provided an initial feasibility analysis for multiple approaches to
classifying task weight using EEG–EMG fusion, and proved many of them to be effective;
however, there are many aspects of the model designs that could be further optimized to
improve their performance. For the time being, it is recommended to continue work on both
the machine learning approaches that require manual feature extraction (Weighted Average
fusion using SMVs) and that utilize automatic feature extraction (CNN-based fusion). The
results of both projects obtained fairly similar results and, since no formal statistical compar-
ison between them was performed in this work, it is hard to recommend one over the other
with confidence. Therefore, both methods should continue to be improved in future work
and a formal comparison of each approach that takes into account not just model accuracy,
but other key implementation factors of a real wearable device (training time, size of training
dataset required, processing power/time required to operate, etc.), should be completed. For
Weighted Average EEG–EMG fusion, future work should investigate the use of optimized
weights, considering techniques such as subject-specific weightings or dynamic weights that
can adapt to different scenarios (e.g., muscle fatigue, electrode motion artifacts, movement
speed, task being performed, etc.). For CNN-based fusion, future work should continue iter-
ating the model design by investigating different convolution configurations, such as parallel
convolution [277, 281, 282], to see how this approach fuses EEG and EMG inside the CNN.
It should also consider including other deep learning methods into the configuration, such as
LSTMs [75, 171–173, 175] to take advantage of the time varying nature of the EEG/EMG
signals. Finally, it should also experiment with other image representations of the bioelec-
trical signals, such as Wavelet Transforms [162, 177, 241], to see how this affects information
8.2 Future Work 328
extraction and fusion.
The use of sensor fusion to improve wearable device control systems should be extended
beyond using only EEG and EMG during fusion. While EEG–EMG fusion has been shown to
be successful in this work, these bioelectrical signals are far from the only signal types that can
be used to measure human intention to facilitate a human–machine interface. The inclusion
of other non-bioelectrical signals into the sensor fusion paradigm may provide ever better
increases to performance by leveraging the unique information these signals can measure. An
obvious candidate for immediate investigation in the next steps is the fusion of IMU motion
data along with EMG and EEG for a three-way fusion approach, as the results of this work
showed that the incorporation of speed information into EEG–EMG fusion led to improved
accuracy. As well, EMG–IMU fusion has been successfully demonstrated previously in the
literature, which provides further evidence that EEG–EMG–IMU fusion may have potential
benefits [57, 72, 128, 161, 187]. Future work need not stop at merely a three-way fusion with
IMU data however, and other non-bioelectrical measurements of brain/muscle activity could
be utilized as well to see if this augments the information provided by EEG/EMG. Employing
fNIRS has shown potential in measuring brain activity using infrared light to sense blood
oxygen levels [217, 219], and may complement the bioelectrical signal readings from EEG.
FMG has been demonstrated recently as a way to measure muscle activity using force sensors
to detect changes in muscle shape [213, 288], which can provide additional information when
combined with EMG. EEG–FMG fusion has been demonstrated previously [214, 215], so it
stands to reason that EEG–EMG–FMG fusion may have potential as well. Looking into
larger sensor fusion techniques, to expand beyond just EEG–EMG fusion, may provide even
more improvements to wearable robotic systems and should be investigated in future work.
Future investigations will need to evaluate the trade-off between increasing system complexity
(through the addition of extra hardware to measure new signal types) with the improvements
gained, to optimize the design of wearable mechatronic devices.
Due to limitations in the upper-limb brace simulator developed in this work, the full evalua-
tion of the TWSC could not be successfully completed; therefore, future work should focus on
8.2 Future Work 329
the development of an improved simulator to finish the testing started here. To address the
aforementioned limitations, a human muscle force model [99–101] should be included into the
control system plant used in the simulator, to capture the complex human–robot dynamics
present when operating a wearable mechatronic device. This will allow the system response
of the simulator to behave in a more realistic manner, allowing the adaptability benefits of
the TWSC to be showcased more effectively. Following this, future work should perform a
formal comparative evaluation of the TWSC against modern non-linear control methods, to
see how it compares to the current state-of-the-art techniques. Future work should also con-
tinue to investigate methods of incorporating EEG–EMG fusion-based task wight classifiers
into wearable robotic control systems by developing other methods of integrating these mod-
els into different control system approaches. Modern non-linear control techniques may also
benefit from the inclusion of such classification models; therefore, it is worthwhile to evaluate
(with the popularity of Sliding Mode Control in the literature [30, 34, 35, 40, 41, 50, 287]
making it a good initial candidate for this work).
All of the projects completed in this thesis utilized offline analyzes when evaluating EEG–
EMG fusion-based machine learning classifiers and control systems, as is typical of initial
feasibility evaluations; however, this is still a long way from implementing these methods
into an actual device. No offline simulation will ever perfectly capture the real-world be-
haviour of a system; therefore, it is crucial that future work begin the process of translating
the EEG–EMG fusion methods demonstrated here into actual devices to properly evaluate
their performance. This applies to both the machine learning-based projects and the control
system-based project, as both areas will have to face consequences related to the constraints
a wearable system will impose (where computational power is extremely limited and fast
response times are required). Future work will need to investigate methods of feasibility im-
plementing the EEG–EMG fusion classifiers into the embedded system of a wearable mecha-
tronic device, perhaps through the use of cloud-computing to offload the high computational
burden required of machine learning [382] or using Field Programmable Gate Array devices
to highly optimize the process at the hardware level [77]. Significant hardware development
8.2 Future Work 330
is still needed to make EEG/EMG sensing more feasible and cost-effective, which could also
be explored as part of future work. Likewise, the TWSC will need to be implemented into an
embedded system as a discrete-time digital control system, and the processing delays incurred
by the device will need to be factored into its design. Immediate next steps should look into
performing a real-time simulation analysis of the EEG–EMG fusion methods presented here
(to ensure it is feasible to execute them with embedded hardware), then development should
begin on an actual wearable robotic upper-limb brace that is designed to utilize these control
techniques. Following the success of bench-top feasibility/safety testing of this device, it will
need to be evaluated during actual use. Initial analysis should focus first on healthy subjects
(to prove efficacy in a lower-risk environment) then should evaluate the device when used by
actual upper-limb injury patients to see how effective these methods are during treatment
and how the presence of various MSD affects EEG–EMG fusion performance (since it is to
be expected that patients with MSDs may present different EEG/EMG signal behaviour).
EEG–EMG fusion has been shown to be a successful approach to improving human–machine

interaction in wearable robotic devices for assistance/rehabilitation, and presents an exciting op-
portunity to help address current issues in these systems related to user control and adaptability.
However, these methods are still located firmly in the research stage of development, and further
improvements are required before such techniques can be utilized in a real-world device designed
to provide treatment. The future work outlined here begins the process of addressing some of the
limitations holding EEG–EMG fusion back from full integration into upper-limb wearable mecha-
tronic devices, and will help to continue pushing this field of research forward. As work continues,
EEG–EMG fusion can begin to be used within actual devices to improve control and adaptability.
Doing so will lead to a better user experience as they operate these novel rehabilitation/assistance
tools, helping to increase widespread adoption by patients, which will ultimately result in better
treatment outcomes and increases in their quality of life.
References
[1] S. Y. Guan, J. X. Zheng, N. B. Sam, S. Xu, Z. Shuai, and F. Pan, “Global burden and risk
factors of musculoskeletal disorders among adolescents and young adults in 204 countries
and territories, 1990–2019,” Autoimmunity Reviews, vol. 22, no. 8, pp. 103361–1–8, 2023.
[2] V. Azzollini, S. Dalise, and C. Chisari, “How does stroke affect skeletal muscle? State of the
art and rehabilitation perspective,” Frontiers in Neurology, vol. 12, pp. 797559–1–7, 2021.
[3] S. A. Chohan, P. K. Venkatesh, and C. H. How, “Long-term complications of stroke and
secondary prevention: An overview for primary care physicians,” Singapore Medical Journal,
vol. 60, no. 12, pp. 616–620, 2019.
[4] World Health Organization, “Musculoskeletal conditions.” https://www.who.int/
news-room/fact-sheets/detail/musculoskeletal-conditions, Nov 26, 2019. accessed:
Mar 30, 2020.
[5] A. Demont, A. Bourmaud, A. Kechichian, and F. Desmeules, “The impact of direct access
physiotherapy compared to primary care physician led usual care for patients with mus-
culoskeletal disorders: a systematic review of the literature,” Disability and Rehabilitation,
vol. 43, no. 12, pp. 1637–1648, 2021.
[6] T. Desplenter, Y. Zhou, B. P. Edmonds, M. Lidka, A. Goldman, and A. L. Trejos, “Re-
habilitative and assistive wearable mechatronic upper-limb devices: a review,” Journal of
Rehabilitation and Assistive Technologies Engineering, vol. 7, pp. 1–26, 2020.
[7] B. S. Rupal, S. Rafique, A. Singla, E. Singla, M. Isaksson, and G. S. Virk, “Lower-limb
exoskeletons: Research trends and regulatory guidelines in medical and non-medical appli-
cations,” International Journal of Advanced Robotic Systems, vol. 14, pp. 1–27, Dec 2017.
[8] Y. Sun, Y. Tang, J. Zheng, D. Dong, X. Chen, and L. Bai, “From sensing to control of lower
limb exoskeleton: a systematic review,” Annual Reviews in Control, vol. 53, pp. 83–96, 2022.
[9] H. Lee, P. W. Ferguson, and J. Rosen, “Lower limb exoskeleton systems-overview,” in Wear-
able Robotics: Systems and Applications (P. W. Ferguson and J. Rosen, eds.), ch. 11, pp. 207–
229, Academic Press, 2019.
[10] D. Shi, W. Zhang, W. Zhang, and X. Ding, “A review on lower limb rehabilitation exoskeleton
robots,” Chinese Journal of Mechanical Engineering, vol. 32, no. 1, pp. 74–1–11, 2019.
[11] R. J. Varghese, D. Freer, F. Deligianni, J. Liu, and G. Z. Yang, “Wearable robotics for
upper-limb rehabilitation and assistance,” in Wearable Technology in Medicine and Health
Care, pp. 23–69, Elsevier, Jan 2018.
331
REFERENCES 332
[12] M. R. Islam, B. Brahmi, T. Ahmed, M. Assad-Uz-Zaman, and M. H. Rahman, “Exoskeletons

in upper limb rehabilitation: a review to find key challenges to improve functionality,” in
Control Theory in Biomedical Engineering, pp. 235–265, Elsevier, 2020.
[13] T. Proietti, V. Crocher, A. Roby-Brami, and N. Jarrasse, “Upper-limb robotic exoskele-

tons for neurorehabilitation : a review on control strategies,” IEEE Reviews in Biomedical
Engineering, vol. 9, pp. 4–14, 2016.
[14] R. A. Gopura, D. S. Bandara, K. Kiguchi, and G. K. Mann, “Developments in hardware sys-

tems of active upper-limb exoskeleton robots: A review,” Robotics and Autonomous Systems,
vol. 75, pp. 203–220, Jan 2016.
[15] N. Rehmat, J. Zuo, W. Meng, Q. Liu, S. Q. Xie, and H. Liang, “Upper limb rehabilitation
using robotic exoskeleton systems: a systematic review,” International Journal of Intelligent
Robotics and Applications, vol. 2, no. 3, pp. 283–295, 2018.
[16] Y. Huang, W. P. Lai, Q. Qian, X. Hu, E. W. Tam, and Y. Zheng, “Translation of

robot-assisted rehabilitation to clinical service in upper limb rehabilitation,” in Intelligent
Biomechatronics in Neurorehabilitation (X. Hu, ed.), ch. 14, pp. 225–238, Academic Press,
2019.
[17] V. Klamroth-Marganska, J. Blanco, K. Campen, A. Curt, V. Dietz, T. Ettlin, M. Felder,

B. Fellinghauer, M. Guidali, A. Kollmar, A. Luft, T. Nef, C. Schuster-Amft, W. Stahel, and
R. Riener, “Three-dimensional, task-specific robot therapy of the arm after stroke: multi-
centre, parallel-group randomised trial,” The Lancet Neurology, vol. 13, no. 2, pp. 159–166,
2014.
[18] M. Franceschini, S. Mazzoleni, M. Goffredo, S. Pournajaf, D. Galafate, S. Criscuolo,

M. Agosti, and F. Posteraro, “Upper limb robot-assisted rehabilitation versus physical ther-
apy on subacute stroke patients: A follow-up study,” Journal of Bodywork and Movement
Therapies, vol. 24, no. 1, pp. 194–198, 2020.
[19] Y. Iwamoto, T. Imura, T. Suzukawa, H. Fukuyama, T. Ishii, S. Taki, N. Imada,

M. Shibukawa, T. Inagawa, H. Araki, and O. Araki, “Combination of exoskeletal upper limb
robot and occupational therapy improve activities of daily living function in acute stroke
patients,” Journal of Stroke and Cerebrovascular Diseases, vol. 28, no. 7, pp. 2018–2025,
2019.
[20] S. Dehem, M. Gilliaux, G. Stoquart, C. Detrembleur, G. Jacquemin, S. Palumbo, A. Freder-

ick, and T. Lejeune, “Effectiveness of upper-limb robotic-assisted therapy in the early reha-
bilitation phase after stroke: single-blind, randomised, controlled trial,” Annals of Physical
and Rehabilitation Medicine, vol. 62, no. 5, pp. 313–320, 2019.
[21] A. Gupta, A. Singh, V. Verma, A. K. Mondal, and M. K. Gupta, “Developments and clini-
cal evaluations of robotic exoskeleton technology for human upper-limb rehabilitation,” Ad-
vanced Robotics, vol. 34, no. 15, pp. 1023–1040, 2020.
[22] R. Nasiri, H. Aftabi, and M. N. Ahmadabadi, “Human-in-the-loop weight compensation in

upper limb wearable robots towards total muscles’ effort minimization,” IEEE Robotics and
Automation Letters, vol. 7, pp. 3273–3278, Nov 2022.
REFERENCES 333
[23] M. A. Vélez-guerrero, M. Callejas-cuervo, and S. Mazzoleni, “Artificial intelligence-based

wearable robotic exoskeletons for upper limb rehabilitation: A review,” Sensors, vol. 21,
pp. 2146–1–29, Mar 2021.
[24] M. S. H. Bhuiyan, I. A. Choudhury, and M. Dahari, “Development of a control system for
artificially rehabilitated limbs: a review,” Biological Cybernetics, vol. 109, no. 2, pp. 141–162,
2015.
[25] G. Muhammad, F. Alshehri, F. Karray, A. E. Saddik, M. Alsulaiman, and T. H. Falk, “A
comprehensive survey on multimodal medical signals fusion for smart healthcare systems,”
Information Fusion, vol. 76, pp. 355–375, 2021.
[26] J. Wright, V. G. Macefield, A. van Schaik, and J. C. Tapson, “A review of control strategies
in closed-loop neuroprosthetic systems,” Frontiers in Neuroscience, vol. 10, pp. 312–1–13,
2016.
[27] M. A. Gull, S. Bai, and T. Bak, “A review on design of upper limb exoskeletons,” Robotics,
vol. 9, pp. 16–1–35, Mar 2020.
[28] Q. Miao, M. Zhang, J. Cao, and S. Q. Xie, “Reviewing high-level control techniques on
robot-assisted upper-limb rehabilitation,” Advanced Robotics, vol. 32, no. 24, pp. 1253–1268,
2018.
[29] J. Wolff, C. Parker, J. Borisoff, W. B. Mortenson, and J. Mattie, “A survey of stakeholder
perspectives on exoskeleton technology,” Journal of NeuroEngineering and Rehabilitation,
vol. 11, no. 1, pp. 169–1–10, 2014.
[30] H. Seo and S. Lee, “Design of general-purpose assistive exoskeleton robot controller for upper
limbs,” Journal of Mechanical Science and Technology, vol. 33, pp. 3509–3519, Jul 2019.
[31] A. Mauri, J. Lettori, G. Fusi, D. Fausti, M. Mor, F. Braghin, G. Legnani, and L. Roveda,
“Mechanical and control design of an industrial exoskeleton for advanced human empowering
in heavy parts manipulation tasks,” Robotics, vol. 8, pp. 65–1–21, Sep 2019.
[32] L. Hao, Z. Zhao, X. Li, M. Liu, H. Yang, and Y. Sun, “A safe human–robot interactive
control structure with human arm movement detection for an upper-limb wearable robot
used during lifting tasks,” International Journal of Advanced Robotic Systems, vol. 17, no. 5,
pp. 1–15, 2020.
[33] N. Lotti, M. Xiloyannis, G. Durandau, E. Galofaro, V. Sanguineti, L. Masia, and M. Sartori,
“Adaptive model-based myoelectric control for a soft wearable arm exosuit: a new generation
of wearable robot control,” IEEE Robotics and Automation Magazine, vol. 27, pp. 43–53,
Mar 2020.
[34] J. Sun, J. Wang, P. Yang, Y. Zhang, and L. Chen, “Adaptive finite time control for wearable
exoskeletons based on ultra-local model and radial basis function neural network,” Interna-
tional Journal of Control, Automation and Systems, vol. 19, pp. 889–899, Feb 2021.
[35] B. O. Mushage, J. C. Chedjou, and K. Kyamakya, “Fuzzy neural network and observer-based
fault-tolerant adaptive nonlinear control of uncertain 5-DOF upper-limb exoskeleton robot
for passive rehabilitation,” Nonlinear Dynamics, vol. 87, pp. 2021–2037, Feb 2017.
REFERENCES 334
[36] A. M. Khan, D. won Yun, M. A. Ali, K. M. Zuhaib, C. Yuan, J. Iqbal, J. Han, K. Shin, and
C. Han, “Passivity based adaptive control for upper extremity assist exoskeleton,” Interna-
tional Journal of Control, Automation and Systems, vol. 14, pp. 291–300, Feb 2016.
[37] N. Masud, C. Smith, and M. Isaksson, “Disturbance observer based dynamic load torque
compensator for assistive exoskeletons,” Mechatronics, vol. 54, pp. 78–93, Oct 2018.
[38] J. Wang and O. Barry, “Inverse optimal robust adaptive controller for upper limb rehabil-
itation exoskeletons with inertia and load uncertainties,” IEEE Robotics and Automation
Letters, vol. 6, pp. 2171—-2178, Apr 2021.
[39] J. Tang, J. Zheng, and Y. Wang, “Direct force control of upper-limb exoskeleton based on
fuzzy adaptive algorithm,” Journal of Vibroengineering, vol. 20, pp. 636–650, Feb 2018.
[40] A. Riani, T. Madani, A. Benallegue, and K. Djouani, “Adaptive integral terminal slid-
ing mode control for upper-limb rehabilitation exoskeleton,” Control Engineering Practice,
vol. 75, pp. 108–117, Jun 2018.
[41] G. Zhang, J. Wang, P. Yang, and S. Guo, “A learning control scheme for upper-limb ex-
oskeleton via adaptive sliding mode technique,” Mechatronics, vol. 86, pp. 102832–1–12, Oct
2022.
[42] M. Rahmani and M. H. Rahman, “Novel robust control of a 7-DOF exoskeleton robot,”
PLoS ONE, vol. 13, pp. e0203440–1–18, sep 2018.
[43] H. B. Kang and J. H. Wang, “Adaptive robust control of 5 DOF upper-limb exoskeleton
robot,” International Journal of Control, Automation and Systems, vol. 13, pp. 733–741,
Jun 2015.
[44] H. Wang, H. Xu, Y. Tian, and H. Tang, “α-Variable adaptive model free control of iReHave
upper-limb exoskeleton,” Advances in Engineering Software, vol. 148, pp. 102872–1–11, Oct
2020.
[45] B. Brahmi, M. Driscoll, I. K. El Bojairami, M. Saad, and A. Brahmi, “Novel adaptive

impedance control for exoskeleton robot for rehabilitation using a nonlinear time-delay dis-
turbance observer,” ISA Transactions, vol. 108, pp. 381–392, Feb 2021.
[46] A. P. P. A. Majeed, Z. Taha, M. A. Abdullah, K. Zakwan, M. Azmi, and M. A. Zakaria,

“The control of an upper extremity exoskeleton for stroke rehabilitation: an active force
control scheme approach,” Advances in Robotics Research, vol. 2, no. 3, pp. 237–245, 2018.
[47] L. D. da Silva, T. F. Pereira, V. R. Leithardt, L. O. Seman, and C. A. Zeferino, “Hybrid

impedance-admittance control for upper limb exoskeleton using electromyography,” Applied
Sciences (Switzerland), vol. 10, pp. 7146–1–19, Oct 2020.
[48] H. Liu, J. Tao, P. Lyu, and F. Tian, “Human-robot cooperative control based on sEMG for
the upper limb exoskeleton robot,” Robotics and Autonomous Systems, vol. 125, pp. 103350–
1–13, Mar 2020.
REFERENCES 335
[49] R. Sharma, S. Bhasin, P. Gaur, and D. Joshi, “A switching-based collaborative fractional

order fuzzy logic controllers for robotic manipulators,” Applied Mathematical Modelling,
vol. 73, pp. 228–246, Sep 2019.
[50] Q. Wu and Y. Chen, “Adaptive cooperative control of a soft elbow rehabilitation exoskeleton
based on improved joint torque estimation,” Mechanical Systems and Signal Processing,
vol. 184, pp. 109748–1–16, Feb 2023.
[51] D. Camargo-Vargas, M. Callejas-Cuervo, and S. Mazzoleni, “Brain-computer interfaces sys-

tems for upper and lower limb rehabilitation: a systematic review,” Sensors, vol. 21, no. 13,
pp. 4312–1–22, 2021.
[52] J. Lobo-Prat, P. N. Kooren, A. H. Stienen, J. L. Herder, B. F. Koopman, and P. H. Veltink,

“Non-invasive control interfaces for intention detection in active movement-assistive devices,”
Journal of NeuroEngineering and Rehabilitation, vol. 11, pp. 168–1–22, 2014.
[53] L. Bi, A. Feleke, and C. Guan, “A review on EMG-based motor intention prediction of
continuous human upper limb motion for human-robot collaboration,” Biomedical Signal
Processing and Control, vol. 51, pp. 113–127, 2019.
[54] D. Esposito, J. Centracchio, E. Andreozzi, G. D. Gargiulo, G. R. Naik, and P. Bifulco,

“Biosignal-based human–machine interfaces for assistance and rehabilitation: a survey,”
Sensors, vol. 21, no. 20, pp. 6863–1–43, 2021.
[55] N. Nazmi, M. A. A. Rahman, S. I. Yamamoto, S. A. Ahmad, H. Zamzuri, and S. A. Ma-

zlan, “A review of classification techniques of EMG signals during isotonic and isometric
contractions,” Sensors, vol. 16, no. 8, pp. 1304–1–28, 2016.
[56] M. Simao, N. Mendes, O. Gibaru, and P. Neto, “A review on electromyography decoding and
pattern recognition for human-machine interaction,” IEEE Access, vol. 7, pp. 39564–39582,
2019.
[57] C. Fang, B. He, Y. Wang, J. Cao, and S. Gao, “EMG-centered multisensory based technolo-
gies for pattern recognition in rehabilitation: state of the art and challenges,” Biosensors,
vol. 10, no. 8, pp. 85–1–30, 2020.
[58] B. Gudiño-mendoza, B. Gudi, G. Sanchez-ante, and J. M. Antelis, “Detecting the inten-

tion to move upper limbs from electroencephalographic brain signals,” Computational and
Mathematical Methods in Medicine, vol. 2016, pp. 3195373–1–11, 2016.
[59] F. Lotte, “A Tutorial on EEG Signal Processing Techniques for Mental State Recognition in
Brain-Computer Interfaces,” in Guide to Brain-Computer Music Interfacing (E. R. Miranda
and J. Castet, eds.), pp. 131–161, Springer, 2014.
[60] M. Teplan, “Fundamentals of EEG measurement,” Measurement Science Review, vol. 2,

no. 2, pp. 1–11, 2002.
[61] N. Padfield, J. Zabalza, H. Zhao, V. Masero, and J. Ren, “EEG-based brain-computer in-
terfaces using motor-imagery: techniques and challenges,” Sensors, vol. 19, no. 6, pp. 1423–
1–34, 2019.
REFERENCES 336
[62] C. Brambilla, I. Pirovano, R. M. Mira, G. Rizzo, A. Scano, and A. Mastropietro, “Combined

use of EMG and EEG techniques for neuromotor assessment in rehabilitative applications:
a systematic review,” Sensors, vol. 21, no. 21, pp. 7014–1–25, 2021.
[63] H. Yang, J. Wan, Y. Jin, X. Yu, and Y. Fang, “EEG- and EMG-driven poststroke rehabili-
tation: a Review,” IEEE Sensors Journal, vol. 22, pp. 23649–23660, Dec 2022.
[64] N. Hooda, R. Das, and N. Kumar, “Fusion of EEG and EMG signals for classification of
unilateral foot movements,” Biomedical Signal Processing and Control, vol. 60, pp. 101990–
1–8, 2020.
[65] Z. Wang and R. Suppiah, “Upper limb movement recognition utilising EEG and EMG signals
for rehabilitative robotics,” in Future of Information and Communication Conference, (San
Francisco, CA, USA), pp. 676–695, Mar 2–3, 2023.
[66] S. Yang, M. Li, and J. Wang, “Fusing sEMG and EEG to increase the robustness of hand
motion recognition using functional connectivity and GCN,” IEEE Sensors Journal, vol. 22,
no. 24, pp. 24309–24319, 2022.
[67] A. Sarasola-Sanz, N. Irastorza-Landa, E. López-Larraz, C. Bibián, F. Helmhold, D. Broetz,

N. Birbaumer, and A. Ramos-Murguialday, “A hybrid-BMI based on EEG and EMG ac-
tivity for the motor rehabilitation of stroke patients,” in IEEE International Conference on
Rehabilitation Robotics, (London, UK), pp. 895–900, Jul 17–20, 2017.
[68] R. Leeb, H. Sagha, R. Chavarriaga, and J. D. R. Millán, “A hybrid brain-computer interface

based on the fusion of electroencephalographic and electromyographic activities,” Journal of
Neural Engineering, vol. 8, no. 2, pp. 025011–1–5, 2011.
[69] T. Dulantha Lalitharatne, K. Teramoto, Y. Hayashi, and K. Kiguchi, “Towards hybrid EEG-
EMG-based control approaches to be used in biorobotics applications: current status, chal-
lenges and future directions,” PALADYN Journal of Behavioral Robotics, vol. 4, no. 2,
pp. 147–154, 2013.
[70] J. Li and Q. Wang, “Multi-modal bioelectrical signal fusion analysis based on different acqui-
sition devices and scene settings: Overview, challenges, and novel orientation,” Information
Fusion, vol. 79, pp. 229–247, 2022.
[71] K. S. Hong and M. J. Khan, “Hybrid brain-computer interface techniques for improved classi-
fication accuracy and increased number of commands: a review,” Frontiers in Neurorobotics,
vol. 11, pp. 1–27, 2017.
[72] K. Li, J. Zhang, L. Wang, M. Zhang, J. Li, and S. Bao, “A review of the key technologies for
sEMG-based human-robot interaction systems,” Biomedical Signal Processing and Control,
vol. 62, pp. 102074–1–17, 2020.
[73] C. Ahmadizadeh, M. Khoshnam, and C. Menon, “Human machine interfaces in upper-limb

prosthesis control: a survey of techniques for preprocessing and processing of biosignals,”
IEEE Signal Processing Magazine, vol. 38, no. 4, pp. 12–22, 2021.
REFERENCES 337
[74] I. Choi, I. Rhiu, Y. Lee, M. H. Yun, and C. S. Nam, “A systematic review of hybrid
brain-computer interfaces: taxonomy and usability perspectives,” PLoS ONE, vol. 12, no. 4,
pp. e0176674–1–35, 2017.
[75] S. Tortora, L. Tonin, C. Chisari, S. Micera, E. Menegatti, and F. Artoni, “Hybrid human-
machine interface for gait decoding through Bayesian fusion of EEG and EMG classifiers,”
Frontiers in Neurorobotics, vol. 14, pp. 89–1–16, 2020.
[76] R. Leeb, H. Sagha, R. Chavarriaga, and J. D. R. Millán, “Multimodal fusion of muscle and
brain signals for a hybrid-BCI,” in Annual International Conference of the IEEE Engineering
in Medicine and Biology Society, (Buenos Aires, Argentina), pp. 4343–4346, Aug 31–Sep 4,
2010.
[77] H. Wöhrle, M. Tabie, S. K. Kim, F. Kirchner, and E. A. Kirchner, “A hybrid FPGA-based

system for EEG- and EMG-based online movement prediction,” Sensors, vol. 17, no. 7,
pp. 1552–1–41, 2017.
[78] X. Li, O. W. Samuel, X. Zhang, H. Wang, P. Fang, and G. Li, “A motion-classification

strategy based on sEMG-EEG signal combination for upper-limb amputees,” Journal of
NeuroEngineering and Rehabilitation, vol. 14, no. 1, pp. 2–1–13, 2017.
[79] P. Xie, X. Chen, P. Ma, X. Li, and Y. Su, “Identification method of human movement
intention based on the fusion feature of EEG and EMG,” in World Congress on Engineering,
(London, U.K.), pp. 1–5, Jul 3–5, 2013.
[80] F. Adila Ferdiansyah, P. Prajitno, and S. Kusuma Wijaya, “EEG-EMG based bio-robotics
elbow orthotics control,” Journal of Physics: Conference Series, vol. 1528, no. 1, pp. 012033–
1–7, 2020.
[81] E. Loopez-Larraz, N. Birbaumer, and A. Ramos-Murguialday, “A hybrid EEG-EMG BMI

improves the detection of movement intention in cortical stroke patients with complete hand
paralysis,” in International Conference of the IEEE Engineering in Medicine and Biology
Society, (Honolulu, Hawaii), pp. 2000–2003, Institute of Electrical and Electronics Engineers
Inc., Ju1 17–21, 2018.
[82] C. Cui, G. B. Bian, Z. G. Hou, J. Zhao, and H. Zhou, “A multimodal framework based on
integration of cortical and muscular activities for decoding human intentions about lower limb
motions,” IEEE Transactions on Biomedical Circuits and Systems, vol. 11, no. 4, pp. 889–
899, 2017.
[83] A. Chowdhury, H. Raza, A. Dutta, and G. Prasad, “EEG-EMG based hybrid brain computer
interface for triggering hand exoskeleton for neuro-rehabilitation,” in Advances in Robotics,
(New Delhi, India), pp. 1–6, Association for Computing Machinery (ACM), Jun 28–Jul 2,
2017.
[84] H. Kim and S. Choi, “Automatic Sleep Stage Classification using EEG and EMG signal,”
in International Conference on Ubiquitous and Future Networks, (Prague, Czech Republic),
pp. 207–212, IEEE, Jul 3–6, 2018.
REFERENCES 338
[85] H. Aly and S. M. Youssef, “Bio-signal based motion control system using deep learning
models: A deep learning approach for motion classification using EEG and EMG signal
fusion,” Journal of Ambient Intelligence and Humanized Computing, vol. 14, no. 2, pp. 991–
1002, 2021.
[86] N. Sharma, H. S. Ryait, and S. Sharma, “Classification of biological signals and time domain
feature extraction using capsule optimized auto encoder-electroencephalographic and elec-
tromyography,” International Journal of Adaptive Control and Signal Processing, vol. 36,
no. 7, pp. 1670—-1690, 2022.
[87] N. Banluesombatkul, P. Ouppaphan, P. Leelaarporn, P. Lakhan, B. Chaitusaney, N. Jaim-

chariyatam, E. Chuangsuwanich, W. Chen, H. Phan, N. Dilokthanakul, and T. Wilaiprasit-
porn, “MetaSleepLearner: A pilot study on fast adaptation of bio-signals-based sleep stage
classifier to new individual subject using meta-learning,” IEEE Journal of Biomedical and
Health Informatics, vol. 25, pp. 1949–1963, Jun 2021.
[88] S. Y. Gordleeva, S. A. Lobov, N. A. Grigorev, A. O. Savosenkov, M. O. Shamshin, M. V.

Lukoyanov, M. A. Khoruzhko, and V. B. Kazantsev, “Real-time EEG-EMG human-machine
interface-based control system for a lower-limb exoskeleton,” IEEE Access, vol. 8, pp. 84070–
84081, 2020.
[89] F. Sbargoud, M. Djeha, M. Guiatni, and N. Ababou, “WPT-ANN and belief theory based
EEG/EMG data fusion for movement identification,” Traitement du Signal, vol. 36, no. 5,
pp. 383–391, 2019.
[90] T. Desplenter, J. Tryon, E. Farago, T. Stanbury, and A. Luisa Trejos, “Interpreting bio-
electrical signals for control of wearable mechatronic devices,” in Human–Robot Interaction:
Control, Analysis, and Design (D. Zhang and B. Wei, eds.), ch. 5, pp. 93–146, Cambridge
Scholars Publishing, 2020.
[91] I. Campanini, A. Merlo, C. Disselhorst-Klug, L. Mesin, S. Muceli, and R. Merletti, “Funda-

mental concepts of bipolar and high-Density surface EMG understanding and teaching for
clinical, occupational, and sport applications: origin, detection, and main errors,” Sensors,
vol. 22, no. 11, pp. 4150–1–31, 2022.
[92] R. Merletti and S. Mucelib, “Tutorial. Surface EMG detection in space and time: best
practices,” Journal of Electromyography and Kinesiology, vol. 49, pp. 102363–1–16, 2019.
[93] M. Tomasini, F. Benatti, Simone and Casamassima, B. Milosevic, S. Fateh, E. Farella, and
L. Benini, “Digitally controlled feedback for DC offset cancellation in a wearable multichannel
EMG platform,” in Annual International Conference of the IEEE Engineering in Medicine
and Biology Society, (Milan, Italy), pp. 3189–3192, Aug 25–29, 2015.
[94] S. Jaffer and N. H. Ghaeb, “Important features of EMG signal under simple load conditions,”
Polytechnic Journal, vol. 7, no. 1, pp. 33–42, 2017.
[95] C. J. De Luca, “Surface electromyography: Detection and recording,” DelSys Incorporated,

Tech. Rep., pp. 1–10, 2002.
REFERENCES 339
[96] M. Hakonen, H. Piitulainen, and A. Visala, “Current state of digital signal processing in
myoelectric interfaces and related applications,” Biomedical Signal Processing and Control,
vol. 18, pp. 334–359, 2015.
[97] H. Tankisi, D. Burke, L. Cui, M. de Carvalho, S. Kuwabara, S. D. Nandedkar, S. Rutkove,

E. Stålberg, M. J. van Putten, and A. Fuglsang-Frederiksen, “Standards of instrumentation
of EMG,” Clinical Neurophysiology, vol. 131, no. 1, pp. 243–258, 2020.
[98] D. Tkach, H. Huang, and T. A. Kuiken, “Study of stability of time-domain features for
electromyographic pattern recognition,” Journal of NeuroEngineering and Rehabilitation,
vol. 7, no. 1, pp. 21–1–13, 2010.
[99] M. Sartori, D. G. Llyod, and D. Farina, “Neural data-driven musculoskeletal modeling for
personalized neurorehabilitation technologies,” IEEE Transactions on Biomedical Engineer-
ing, vol. 63, pp. 879–893, May 2016.
[100] R. Nasiri, M. Rayati, and M. Nili Ahmadabadi, “Feedback from mono-articular muscles is
sufficient for exoskeleton torque adaptation,” IEEE Transactions on Neural Systems and
Rehabilitation Engineering, vol. 27, pp. 2097–2106, Oct 2019.
[101] T. Desplenter and A. L. Trejos, “Evaluating muscle activation models for elbow motion
estimation,” Sensors, vol. 18, no. 4, pp. 1004–1–19, 2018.
[102] M. Rashid, N. Sulaiman, A. P. P. Abdul Majeed, R. M. Musa, A. F. Ahmad, B. S. Bari, and

S. Khatun, “Current status, challenges, and possible solutions of EEG-based brain-computer
interface: a comprehensive review,” Frontiers in Neurorobotics, vol. 14, pp. 25–1–35, 2020.
[103] S. Vaid, P. Singh, and C. Kaur, “EEG signal analysis for BCI interface: A review,” in Inter-
national Conference on Advanced Computing and Communication Technologies, (Haryana,
India), pp. 143–147, Feb 21–22, 2015.
[104] P. Sawangjai, S. Hompoonsup, P. Leelaarporn, S. Kongwudhikunakorn, and T. Wilaiprasit-

porn, “Consumer grade EEG measuring sensors as research tools: a review,” IEEE Sensors
Journal, vol. 20, pp. 3996–4024, Apr 2020.
[105] M. M. Togha, M. R. Salehi, and E. Abiri, “Improving the performance of the motor imagery-
based brain-computer interfaces using local activities estimation,” Biomedical Signal Process-
ing and Control, vol. 50, pp. 52–61, 2019.
[106] M. Tavakolan, Z. Frehlick, X. Yong, and C. Menon, “Classifying three imaginary states of the
same upper extremity using time-domain features,” PLoS ONE, vol. 12, no. 3, pp. e0174161–
1–18, 2017.
[107] E. Abdalsalam M, M. Z. Yusoff, D. Mahmoud, A. S. Malik, and M. R. Bahloul, “Discrim-

ination of four class simple limb motor imagery movements for brain–computer interface,”
Biomedical Signal Processing and Control, vol. 44, pp. 181–190, 2018.
[108] D. Planelles, E. Hortal, Á. Costa, A. Úbeda, E. Iáñez, and J. M. Azorı́n, “Evaluating clas-
sifiers to detect arm movement intention from EEG signals,” Sensors (Switzerland), vol. 14,
no. 10, pp. 18172–18186, 2014.
REFERENCES 340
[109] V. Mihajlovic, B. Grundlehner, R. Vullers, and J. Penders, “Wearable, wireless EEG so-
lutions in daily life applications: what are we missing?,” IEEE Journal of Biomedical and
Health Informatics, vol. 19, no. 1, pp. 6–21, 2015.
[110] R. Merletti and G. L. Cerone, “Tutorial. Surface EMG detection, conditioning and
pre-processing: best practices,” Journal of Electromyography and Kinesiology, vol. 54,
pp. 102440–1–21, 2020.
[111] J. Wang, L. Tang, and J. E. Bronlund, “Surface EMG signal amplification and filtering,”
International Journal of Computer Applications, vol. 82, no. 1, pp. 15–22, 2013.
[112] Y. Fu, J. Zhao, Y. Dong, and X. Wang, “Dry electrodes for human bioelectrical signal
monitoring,” Sensors, vol. 20, no. 13, pp. 3651–1–30, 2020.
[113] K. E. Mathewson, T. J. L. Harrison, and S. A. D. Kizuk, “High and dry? Comparing active
dry EEG electrodes to active and passive wet electrodes,” Psychophysiology, vol. 54, no. 1,
pp. 74–82, 2017.
[114] V. K. Sarker, M. Jiang, T. N. Gia, A. Anzanpour, A. M. Rahmani, and P. Liljeberg, “Portable

multipurpose bio-signal acquisition and wireless streaming device for wearables,” in IEEE
Sensors Applications Symposium, (Glassboro, New Jersey, USA), pp. 1–6, Mar 13–15, 2017.
[115] E. Stålberg, H. van Dijk, B. Falck, J. Kimura, C. Neuwirth, M. Pitt, S. Podnar, D. I.

Rubin, S. Rutkove, D. B. Sanders, M. Sonoo, H. Tankisi, and M. Zwarts, “Standards for
quantification of EMG and neurography,” Clinical Neurophysiology, vol. 130, no. 9, pp. 1688–
1729, 2019.
[116] M. D. Murphy, D. J. Guggenmos, D. T. Bundy, and R. J. Nudo, “Current challenges facing

the translation of brain computer interfaces from preclinical trials to use in human patients,”
Frontiers in Cellular Neuroscience, vol. 9, no. January, pp. 497–1–14, 2016.
[117] M. Z. Jamal, D.-H. Lee, and D. J. Hyun, “Real time adaptive filter based EMG signal pro-
cessing and instrumentation scheme for robust signal acquisition using dry EMG electrodes,”
in International Conference on Ubiquitous Robots, (Jeju, Korea), pp. 683–688, IEEE, Jun
24–27, 2019.
[118] A. Bertrand, “Distributed signal processing for wireless EEG sensor networks,” IEEE Trans-
actions on Neural Systems and Rehabilitation Engineering, vol. 23, no. 6, pp. 923–935, 2015.
[119] H. J. Hermens, B. Freriks, C. Disselhorst-Klug, and G. Rau, “Development of recommen-

dations for SEMG sensors and sensor placement procedures,” Journal of Electromyography
and Kinesiology, vol. 10, no. 5, pp. 361–374, 2000.
[120] H. Hermens, C. of the European Communities. Biomedical, and H. R. Programme, European

Recommendations for Surface Electromyography: Results of the SENIAM Project. Roessingh
Research and Development, 1999.
[121] C. M. Vidhya, Y. Maithani, and J. P. Singh, “Recent advances and challenges in textile
electrodes for wearable biopotential signal monitoring: a comprehensive review,” Biosensors,
vol. 13, no. 7, pp. 679–1–34, 2023.
REFERENCES 341
[122] P. Tallgren, S. Vanhatalo, K. Kaila, and J. Voipio, “Evaluation of commercially available

electrodes and gels for recording of slow EEG potentials,” Clinical Neurophysiology, vol. 116,
no. 4, pp. 799–806, 2005.
[123] S. L. Kappel, M. L. Rank, H. O. Toft, M. Andersen, and P. Kidmose, “Dry-contact electrode

ear-EEG,” IEEE Transactions on Biomedical Engineering, vol. 66, no. 1, pp. 150–158, 2018.
[124] L. Mesin, “Crosstalk in surface electromyogram: literature review and some insights,” Phys-
ical and Engineering Sciences in Medicine, vol. 43, no. 2, pp. 481–492, 2020.
[125] M. Arvaneh, S. Member, C. Guan, K. K. Ang, and C. Quek, “Optimizing the channel
selection and classification accuracy in EEG-based BCI,” IEEE transactions on bio-medical
engineering, vol. 58, no. 6, pp. 1865–1873, 2012.
[126] T. Alotaiby, F. E. El-Samie, S. A. Alshebeili, and I. Ahmad, “A review of channel selection

algorithms for EEG signal processing,” Eurasip Journal on Advances in Signal Processing,
vol. 2015, no. 1, pp. 1–21, 2015.
[127] M. Mohr, T. Schön, V. von Tscharner, and B. M. Nigg, “Intermuscular coherence between
surface EMG signals is higher for monopolar compared to bipolar electrode configurations,”
Frontiers in Physiology, vol. 9, pp. 566–1–14, 2018.
[128] D. Novak and R. Riener, “A survey of sensor fusion methods in wearable robotics,” Robotics
and Autonomous Systems, vol. 73, pp. 155–170, 2015.
[129] J. Cheng, L. Li, C. Li, Y. Liu, A. Liu, R. Qian, and X. Chen, “Remove diverse artifacts
simultaneously from a single-channel EEG based on SSA and ICA: A semi-simulated study,”
IEEE Access, vol. 7, pp. 60276–60289, 2019.
[130] M. Z. Baig, N. Aslam, and H. P. Shum, “Filtering techniques for channel selection in motor
imagery EEG applications: a survey,” Artificial Intelligence Review, vol. 53, pp. 1207–1232,
2020.
[131] P. Górski, “Common spatial patterns in a few channel BCI interface,” Journal of Theoretical
and Applied Computer Science, vol. 8, no. 4, pp. 56–63, 2014.
[132] N. N. Kumar and A. G. Reddy, “Removal of ECG artifact from EEG data using independent
component analysis and S-transform,” International Journal of Science, Engineering and
Technology Research, vol. 5, no. 3, pp. 712–716, 2016.
[133] Y. Zheng and X. Hu, “Interference removal from electromyography based on independent
component analysis,” IEEE Transactions on Neural Systems and Rehabilitation Engineering,
vol. 27, no. 5, pp. 887–894, 2019.
[134] A. Phinyomark, P. Phukpattaranont, and C. Limsakul, “Feature reduction and selection for
EMG signal classification,” Expert Systems with Applications, vol. 39, no. 8, pp. 7420–7431,
2012.
[135] R. Lahiri, P. Rakshit, and A. Konar, “Evolutionary perspective for optimal selection of EEG
electrodes and features,” Biomedical Signal Processing and Control, vol. 36, pp. 113–137,
2017.
REFERENCES 342
[136] S. Gudmundsson, T. P. Runarsson, S. Sigurdsson, G. Eiriksdottir, and K. Johnsen, “Reliabil-

ity of quantitative EEG features,” Clinical Neurophysiology, vol. 118, no. 10, pp. 2162–2171,
2007.
[137] Z. Tang, K. Zhang, S. Sun, Z. Gao, L. Zhang, and Z. Yang, “An upper-limb power-assist
exoskeleton using proportional myoelectric control,” Sensors (Switzerland), vol. 14, no. 4,
pp. 6677–6694, 2014.
[138] L. H. Smith, L. J. Hargrove, B. A. Lock, and T. A. Kuiken, “Determining the optimal
window length for pattern recognition-based myoelectric control: balancing the competing
effects of classification error and controller delay,” IEEE Transactions on Neural Systems
and Rehabilitation Engineering, vol. 19, no. 2, pp. 186–192, 2011.
[139] C. D. Kuthe, R. V. Uddanwadiker, and A. A. Ramteke, “Surface electromyography based
method for computing muscle strength and fatigue of biceps brachii muscle and its clinical
implementation,” Informatics in Medicine Unlocked, vol. 12, pp. 34–43, 2018.
[140] S. Thongpanja, A. Phinyomark, P. Phukpattaranont, and C. Limsakul, “Mean and median
frequency of EMG signal to determine muscle force based on time dependent power spec-
trum,” Elektronika ir Elektrotechnika, vol. 19, no. 3, pp. 51–56, 2013.
[141] N. Brodu, F. Lotte, and A. Lécuyer, “Comparative study of band-power extraction tech-
niques for Motor Imagery classification,” in IEEE Symposium on Computational Intelligence,
Cognitive Algorithms, Mind, and Brain, (Paris, France), pp. 1–6, Apr 11–15, 2011.
[142] S. Olsen, G. Alder, M. Williams, S. Chambers, M. Jochumsen, N. Signal, U. Rashid, I. K.
Niazi, and D. Taylor, “Electroencephalographic recording of the movement-related cortical
potential in ecologically valid movements: a scoping review,” Frontiers in Neuroscience,
vol. 15, pp. 721387–1–26, 2021.
[143] A. Shakeel, M. S. Navid, M. N. Anwar, S. Mazhar, M. Jochumsen, and I. K. Niazi, “A
review of techniques for detection of movement intention using movement-related cortical
potentials,” Computational and Mathematical Methods in Medicine, vol. 2015, pp. 346217–
1–13, 2015.
[144] M. Jochumsen, I. K. Niazi, N. Mrachacz-Kersting, D. Farina, and K. Dremstrup, “Detection
and classification of movement-related cortical potentials associated with task force and
speed,” Journal of Neural Engineering, vol. 10, no. 5, pp. 056015–1–9, 2013.
[145] S. Bhattacharyya, A. Konar, and D. N. Tibarewala, “Motor imagery and error related poten-
tial induced position control of a robotic arm,” IEEE/CAA Journal of Automatica Sinica,
vol. 4, pp. 639–650, Oct 2017.
[146] P. Gaur, R. B. Pachori, H. Wang, and G. Prasad, “A multi-class EEG-based BCI classi-
fication using multivariate empirical mode decomposition based filtering and Riemannian
geometry,” Expert Systems with Applications, vol. 95, pp. 201–211, 2018.
[147] P. Gaur, R. B. Pachori, H. Wang, and G. Prasad, “An empirical mode decomposition
based filtering method for classification of motor-imagery EEG signals for enhancing brain-
computer interface,” in International Joint Conference on Neural Networks, (Killarney, Ire-
land), pp. 1–7, Ju1 12–17, 2015.
REFERENCES 343
[148] S. Darvishi and A. Al-Ani, “Brain-computer interface analysis using continuous wavelet
transform and adaptive neuro-fuzzy classifier,” in Annual International Conference of the
IEEE Engineering in Medicine and Biology, (Lyon, France), pp. 3220–3223, Jul 17–20, 2007.
[149] M. P. Hosseini, A. Hosseini, and K. Ahi, “A review on machine learning for EEG signal
processing in bioengineering,” IEEE Reviews in Biomedical Engineering, vol. 14, pp. 204–
218, 2021.
[150] S. Briouza, H. Gritli, N. Khraief, S. Belghith, and D. Singh, “A brief overview on machine
learning in rehabilitation of the human arm via an exoskeleton robot,” in International
Conference on Data Analytics for Business and Industry, pp. 129–134, Institute of Electrical
and Electronics Engineers Inc., Oct 25–26, 2021.
[151] N. AlHinai, “Introduction to biomedical signal processing and artificial intelligence,” in

Biomedical signal processing and artificial intelligence in healthcare (W. Zgallai, ed.), ch. 1,
pp. 1–28, Academic Press, 2020.
[152] L. R. Quitadamo, F. Cavrini, L. Sbernini, F. Riillo, L. Bianchi, S. Seri, and G. Sag-

gio, “Support vector machines to detect physiological patterns for EEG and EMG-based
human–computer interaction: a review,” Journal of Neural Engineering, vol. 14, no. 1,
pp. 011001–1–28, 2017.
[153] R. G. Brereton and G. R. Lloyd, “Support vector machines for classification and regression,”
Analyst, vol. 135, no. 2, pp. 230–267, 2010.
[154] R. J. Urbanowicz, M. Meeker, W. La Cava, R. S. Olson, and J. H. Moore, “Relief-based

feature selection: Introduction and review,” Journal of Biomedical Informatics, vol. 85,
pp. 189–203, 2018.
[155] C. Ding and H. Peng, “Minimum redundancy feature selection from microarray gene expres-
sion data,” Journal of Bioinformatics and Computational Biology, vol. 3, no. 2, pp. 185–205,
2005.
[156] X. Yu, P. Chum, and K. B. Sim, “Analysis the effect of PCA for feature reduction in non-
stationary EEG based motor imagery of BCI system,” Optik, vol. 125, no. 3, pp. 1498–1502,
2014.
[157] R. Vega, T. Sajed, K. W. Mathewson, K. Khare, P. M. Pilarski, R. Greiner, G. Sanchez-Ante,

and J. M. Antelis, “Assessment of feature selection and classification methods for recognizing
motor imagery tasks from electroencephalographic signals,” Artificial Intelligence Research,
vol. 6, no. 1, pp. 37–51, 2017.
[158] P. Wei, J. Zhang, F. Tian, and J. Hong, “A comparison of neural networks algorithms for
EEG and sEMG features based gait phases recognition,” Biomedical Signal Processing and
Control, vol. 68, pp. 102587–1–9, 2021.
[159] J. Teuwen and N. Moriakov, “Convolutional neural networks,” in Handbook of Medical Image
Computing and Computer Assisted Intervention, pp. 481–501, Elsevier, Jan 2020.
REFERENCES 344
[160] P. Gavali and J. S. Banu, “Deep convolutional neural network for image classification on
CUDA platform,” in Deep Learning and Parallel Computing Environment for Bioengineering
Systems, pp. 99–122, Elsevier, 2019.
[161] S. Qiu, H. Zhao, N. Jiang, Z. Wang, L. Liu, Y. An, H. Zhao, X. Miao, R. Liu, and G. Fortino,
“Multi-sensor information fusion based on machine learning for real applications in human
activity recognition: state-of-the-art and research challenges,” Information Fusion, vol. 80,
pp. 241—-265, 2021.
[162] S. Chaudhary, S. Taran, V. Bajaj, and A. Sengur, “Convolutional neural network based
approach towards motor imagery tasks EEG signals classification,” IEEE Sensors Journal,
vol. 19, no. 12, pp. 4494–4500, 2019.
[163] M. Dai, D. Zheng, R. Na, S. Wang, and S. Zhang, “EEG classification of motor imagery
using a novel deep learning framework,” Sensors, vol. 19, no. 3, pp. 551–1–16, 2019.
[164] Z. Tayeb, J. Fedjaev, N. Ghaboosi, C. Richter, L. Everding, X. Qu, Y. Wu, G. Cheng,

and J. Conradt, “Validating deep neural networks for online decoding of motor imagery
movements from EEG signals,” Sensors, vol. 19, no. 1, pp. 210–1–17, 2019.
[165] Z. Wang, L. Cao, Z. Zhang, X. Gong, Y. Sun, and H. Wang, “Short time Fourier transfor-
mation and deep neural networks for motor imagery brain computer interface recognition,”
Concurrency and Computation: Practice and Experience, vol. 30, no. 23, pp. e4413–1–9,
2018.
[166] X. Tang, W. Li, X. Li, W. Ma, and X. Dang, “Motor imagery EEG recognition based on
conditional optimization empirical mode decomposition and multi-scale convolutional neural
network,” Expert Systems with Applications, vol. 149, pp. 113285–1–11, 2020.
[167] Y. Li, X. R. Zhang, B. Zhang, M. Y. Lei, W. G. Cui, and Y. Z. Guo, “A channel-projection

mixed-scale convolutional neural network for motor imagery EEG decoding,” IEEE Trans-
actions on Neural Systems and Rehabilitation Engineering, vol. 27, no. 6, pp. 1170–1180,
2019.
[168] S. U. Amin, M. Alsulaiman, G. Muhammad, M. A. Mekhtiche, and M. Shamim Hossain,

“Deep learning for EEG motor imagery classification based on multi-layer CNNs feature
fusion,” Future Generation Computer Systems, vol. 101, pp. 542–554, 2019.
[169] D. Zhao, F. Tang, B. Si, and X. Feng, “Learning joint space–time–frequency features for
EEG decoding on small labeled data,” Neural Networks, vol. 114, pp. 67–77, 2019.
[170] R. T. Schirrmeister, J. T. Springenberg, L. D. J. Fiederer, M. Glasstetter, K. Eggensperger,

M. Tangermann, F. Hutter, W. Burgard, and T. Ball, “Deep learning with convolutional
neural networks for EEG decoding and visualization,” Human Brain Mapping, vol. 38, no. 11,
pp. 5391–5420, 2017.
[171] R. Zhang, Q. Zong, L. Dou, and X. Zhao, “A novel hybrid deep learning scheme for four-class
motor imagery classification,” Journal of Neural Engineering, vol. 16, no. 6, pp. 066004–1–11,
2019.
REFERENCES 345
[172] T. Wilaiprasitporn, A. Ditthapron, K. Matchaparn, T. Tongbuasirilai, N. Banluesombatkul,

and E. Chuangsuwanich, “Affective EEG-based person identification using the deep learning
approach,” IEEE Transactions on Cognitive and Developmental Systems, vol. 12, pp. 486–
496, Sep 2020.
[173] A. Ditthapron, N. Banluesombatkul, S. Ketrat, E. Chuangsuwanich, and T. Wilaiprasit-

porn, “Universal joint feature extraction for P300 EEG classification using multi-task au-
toencoder,” IEEE Access, vol. 7, pp. 68415–68428, 2019.
[174] N. Duan, L. Z. Liu, X. J. Yu, Q. Li, and S. C. Yeh, “Classification of multichannel surface-
electromyography signals based on convolutional neural networks,” Journal of Industrial
Information Integration, vol. 15, pp. 201–206, 2019.
[175] P. Xia, J. Hu, and Y. Peng, “EMG-based estimation of limb movement using deep learning
with recurrent convolutional neural networks,” Artificial Organs, vol. 42, no. 5, pp. E67–E77,
2018.
[176] X. Zhai, B. Jelfs, R. H. Chan, and C. Tin, “Self-recalibrating surface EMG pattern recogni-
tion for neuroprosthesis control based on convolutional neural network,” Frontiers in Neu-
roscience, vol. 11, pp. 379–1–11, 2017.
[177] U. Côté-Allard, C. L. Fall, A. Drouin, A. Campeau-Lecours, C. Gosselin, K. Glette, F. Lavi-

olette, and B. Gosselin, “Deep learning for electromyographic hand gesture signal classifi-
cation using transfer learning,” IEEE Transactions on Neural Systems and Rehabilitation
Engineering, vol. 27, no. 4, pp. 760–771, 2019.
[178] Z. Ding, C. Yang, Z. Tian, C. Yi, Y. Fu, and F. Jiang, “sEMG-based gesture recognition
with convolution neural networks,” Sustainability, vol. 10, no. 6, pp. 1865–1–12, 2018.
[179] A. Ameri, M. A. Akhaee, E. Scheme, and K. Englehart, “Real-time, simultaneous myoelectric

control using a convolutional neural network,” PLoS ONE, vol. 13, no. 9, pp. e0203835–1–13,
2018.
[180] M. Z. Zia ur Rehman, A. Waris, S. O. Gilani, M. Jochumsen, I. K. Niazi, M. Jamil, D. Farina,

and E. N. Kamavuako, “Multiday EMG-Based classification of hand motions with deep
learning techniques,” Sensors, vol. 18, no. 8, pp. 2497–1–16, 2018.
[181] M. Atzori, M. Cognolato, and H. Müller, “Deep learning with convolutional neural net-
works applied to electromyography data: A resource for the classification of movements for
prosthetic hands,” Frontiers in Neurorobotics, vol. 10, pp. 9–1–10, 2016.
[182] H. Chen, Y. Zhang, G. Li, Y. Fang, and H. Liu, “Surface electromyography feature ex-
traction via convolutional neural network,” International Journal of Machine Learning and
Cybernetics, vol. 11, no. 1, pp. 185–196, 2020.
[183] Y. Fang, X. Zhang, D. Zhou, and H. Liu, “Improve inter-day hand gesture recognition
via convolutional neural network based feature fusion,” International Journal of Humanoid
Robotics, vol. 18, no. 2, pp. 2050025–1–22, 2021.
REFERENCES 346
[184] Y. Roy, H. Banville, I. Albuquerque, A. Gramfort, T. H. Falk, and J. Faubert, “Deep

learning-based electroencephalography analysis: A systematic review,” Journal of Neural
Engineering, vol. 16, no. 5, pp. 051001–1–37, 2019.
[185] A. Phinyomark and E. Scheme, “EMG pattern recognition in the era of big data and deep
learning,” Big Data and Cognitive Computing, vol. 2, no. 3, pp. 21–1–27, 2018.
[186] T. Xue, W. Wang, J. Ma, W. Liu, Z. Pan, and M. Han, “Progress and prospects of multimodal
fusion methods in physical human–robot interaction: a review,” IEEE Sensors Journal,
vol. 20, no. 18, pp. 10355–10370, 2020.
[187] J. G. Colli-Alfaro, A. Ibrahim, and A. L. Trejos, “Design of user-independent hand gesture

recognition using Multilayer Perceptron Networks and sensor fusion techniques,” in Interna-
tional Conference on Rehabilitation Robotics, (Toronto, Canada), pp. 1103–1108, Jun 24–28,
2019.
[188] L. Cheng, D. Li, G. Yu, Z. Zhang, and S. Yu, “Robotic arm control system based on brain-
muscle mixed signals,” Biomedical Signal Processing and Control, vol. 77, pp. 103754–1–14,
2022.
[189] Q. Gao, L. Dou, A. N. Belkacem, and C. Chen, “Noninvasive electroencephalogram based

control of a robotic arm for writing task using hybrid BCI system,” BioMed Research Inter-
national, vol. 2017, pp. 8316485–1–9, 2017.
[190] K. Bakshi, R. Pramanik, M. Manjunatha, and C. S. Kumar, “Upper limb prosthesis control:
A hybrid EEG-EMG scheme for motion estimation in transhumeral subjects,” in Annual
International Conference of the IEEE Engineering in Medicine and Biology Society, (Hon-
olulu, Hawaii, USA), pp. 2024–2027, Institute of Electrical and Electronics Engineers Inc.,
Jul 18–21, 2018.
[191] J. Zhang, B. Wang, C. Zhang, Y. Xiao, and M. Y. Wang, “An EEG/EMG/EOG-based

multimodal human-machine interface to real-time control of a soft robot hand,” Frontiers in
Neurorobotics, vol. 13, pp. 7–1–13, 2019.
[192] Z. Li, Y. Yuan, L. Luo, W. Su, K. Zhao, C. Xu, J. Huang, and M. Pi, “Hybrid brain/muscle
signals powered wearable walking exoskeleton enhancing motor ability in climbing stairs
activity,” IEEE Transactions on Medical Robotics and Bionics, vol. 1, no. 4, pp. 218–227,
2019.
[193] K. Leerskov, M. Rehman, I. Niazi, S. Cremoux, and M. Jochumsen, “Investigating the

feasibility of combining EEG and EMG for controlling a hybrid human computer interface
in patients with spinal cord injury,” in IEEE International Conference on Bioinformatics
and Bioengineering, pp. 403–410, Institute of Electrical and Electronics Engineers Inc., Oct
26–28, 2020.
[194] M. S. Al-Quraishi, I. Elamvazuthi, T. B. Tang, M. Al-Qurishi, S. Parasuraman, and A. Bor-

boni, “Multimodal fusion approach based on EEG and EMG signals for lower limb movement
recognition,” IEEE Sensors Journal, vol. 21, no. 24, pp. 27640–27650, 2021.
REFERENCES 347
[195] X. Zhang, H. Li, R. Dong, Z. Lu, and C. Li, “Electroencephalogram and surface electromyo-
gram fusion-based precise detection of lower limb voluntary movement using convolution neu-
ral network-long short-term memory model,” Frontiers in Neuroscience, vol. 16, pp. 954387–
1–21, 2022.
[196] G. Song, R. Huang, Y. Guo, J. Qiu, and H. Cheng, “An EEG-EMG-based motor intention
recognition for walking assistive exoskeletons,” in International Conference on Intelligent
Robotics and Applications, (Harbin, China), pp. 769–781, Aug 1–3, 2022.
[197] F. Davarinia and A. Maleki, “SSVEP-gated EMG-based decoding of elbow angle during goal-
directed reaching movement,” Biomedical Signal Processing and Control, vol. 71, pp. 103222–
1–10, 2022.
[198] K. Kiguchi and Y. Hayashi, “A study of EMG and EEG during perception-assist with an
upper-limb power-assist robot,” in IEEE International Conference on Robotics and Automa-
tion, (Saint Paul, Minnesota, USA), pp. 2711–2716, IEEE, May 14–18, 2012.
[199] J. Gallego, E. Rocon, J. Ibañez, J. Dideriksen, A. Koutsou, R. Paradiso, M. Popovic,

J. Belda-Lois, F. Gianfelici, D. Farina, D. Popovic, M. Manto, T. D’Alessio, and J. Pons,
“A soft wearable robot for tremor assessment and suppression,” in IEEE International Con-
ference on Robotics and Automation, (Shanghai, China), pp. 2249–2254, IEEE, May 9–13,
2011.
[200] G. Cisotto, M. Capuzzo, A. V. Guglielmi, and A. Zanella, “Feature stability and setup
minimization for EEG-EMG-enabled monitoring systems,” EURASIP Journal on Advances
in Signal Processing, vol. 2022, no. 1, pp. 103–1–21, 2022.
[201] E. Nsugbe, O. W. Samuel, M. G. Asogbon, and G. Li, “Phantom motion intent decoding
for transhumeral prosthesis control with fused neuromuscular and brain wave signals,” IET
Cyber-Systems and Robotics, vol. 3, no. 1, pp. 77–88, 2021.
[202] A. Chowdhury, H. Raza, Y. K. Meena, A. Dutta, and G. Prasad, “An EEG-EMG correlation-
based brain-computer interface for hand orthosis supported neuro-rehabilitation,” Journal
of Neuroscience Methods, vol. 312, pp. 1–11, 2019.
[203] P. Aricò, F. Aloise, F. Pichiorri, F. Leotta, S. Salinari, D. Mattia, and F. Cincotti, “FES
controlled by a hybrid BCI system for neurorehabilitation – driven after stroke,” in GNB2012,
(Rome, Italy), pp. 1–2, Jun 26–29, 2012.
[204] F. Cincotti, F. Pichiorri, P. Arico, F. Aloise, F. Leotta, F. De Vico Fallani, J. D. R. Millan,

M. Molinari, and D. Mattia, “EEG-based brain-computer interface to support post-stroke
motor rehabilitation of the upper limb,” in Annual International Conference of the IEEE
Engineering in Medicine and Biology Society, (San Diego, California USA), pp. 4112–4115,
Aug 28–Sep 1, 2012.
[205] F. Grimm, A. Walter, M. Spüler, G. Naros, W. Rosenstiel, and A. Gharabaghi, “Hybrid

neuroprosthesis for the upper limb: combining brain-controlled neuromuscular stimulation
with a multi-joint arm exoskeleton,” Frontiers in Neuroscience, vol. 10, pp. 367–1–11, 2016.
REFERENCES 348
[206] E. A. Kirchner, M. Tabie, and A. Seeland, “Multimodal movement prediction - Towards an

individual assistance of patients,” PLoS ONE, vol. 9, no. 1, pp. 1–9, 2014.
[207] T. D. Lalitharatne, K. Teramoto, Y. Hayashi, and K. Kiguchi, “Evaluation of perception-
assist with an upper-limb power-assist exoskeleton using EMG and EEG signals,” in IEEE
International Conference on Networking, Sensing and Control, (Miami, FL, USA), pp. 524–
529, Apr 7–9, 2014.
[208] V. F. Annese, M. Crepaldi, D. Demarchi, and D. De Venuto, “A digital processor architecture
for combined EEG/EMG falling risk prediction,” in Design, Automation and Test in Europe
Conference and Exhibition, (Dresden, Germany), pp. 714–719, Mar 14–18, 2016.
[209] D. D. Venuto, V. F. Annese, M. D. Tommaso, and E. Vecchio, “Combining EEG and EMG
signals in a wireless system for preventing fall in neurodegenerative diseases,” in Ambient
Assisted Living. Biosystems and Biorobotics, vol. 11, (Catania, Italy), pp. 317–327, 2015.
[210] M. Pritchard, A. I. Weinberg, J. A. R. Williams, F. Campelo, H. Goldingay, and D. R. Faria,
“Dynamic fusion of electromyographic and electroencephalographic data towards use in
robotic prosthesis control,” in Journal of Physics: Conference Series, vol. 1828, pp. 012056–
1–10, 2021.
[211] A. Manolova, G. Tsenov, V. Lazarova, and N. Neshov, “Combined EEG and EMG fatigue
measurement framework with application to hybrid brain-computer interface,” in IEEE In-
ternational Black Sea Conference on Communications and Networking, (Varna, Bulgaria),
pp. 1–5, Jun 6–9, 2016.
[212] V. F. Annese and D. De Venuto, “FPGA based architecture for fall-risk assessment during
gait monitoring by synchronous EEG/EMG,” in IEEE International Workshop on Advances
in Sensors and Interfaces, (Gallipoli, Italy), pp. 116–121, Jun 18–19, 2015.
[213] Z. G. Xiao and C. Menon, “A review of force myography research and development,” Sensors,
vol. 19, no. 20, pp. 4557–1–25, 2019.
[214] M. Connan, E. R. Ramı́rez, B. Vodermayer, and C. Castellini, “Assessment of a wearable
forceand electromyography device and comparison of the related signals for myocontrol,”
Frontiers in Neurorobotics, vol. 10, pp. 17–1–13, 2016.
[215] P. Chen, Z. Li, S. Togo, H. Yokoi, and Y. Jiang, “A Layered sEMG–FMG Hybrid Sensor for
Hand Motion Recognition From Forearm Muscle Activities,” IEEE Transactions on Human-
Machine Systems, pp. 1–10, 2023.
[216] N. Naseer and K. S. Hong, “fNIRS-based brain-computer interfaces: A review,” Frontiers in
Human Neuroscience, vol. 9, pp. 3–1–15, 2015.
[217] X. Yin, B. Xu, C. Jiang, Y. Fu, Z. Wang, H. Li, and G. Shi, “NIRS-based classification
of clench force and speed motor imagery with the use of empirical mode decomposition for
BCI,” Medical Engineering and Physics, vol. 37, no. 3, pp. 280–286, 2015.
[218] K. S. Hong, M. J. Khan, and M. J. Hong, “Feature extraction and classification methods for
hybrid fNIRS-EEG brain-computer interfaces,” Frontiers in Human Neuroscience, vol. 12,
pp. 246–1–25, 2018.
REFERENCES 349
[219] M. J. Khan and K. S. Hong, “Hybrid EEG–fNIRS-based eight-command decoding for BCI:
application to quadcopter control,” Frontiers in Neurorobotics, vol. 11, pp. 6–1–13, 2017.
[220] J. Tryon, E. Friedman, and A. L. Trejos, “Performance evaluation of EEG/EMG fusion meth-
ods for motion classification,” in IEEE International Conference on Rehabilitation Robotics,
(Toronto, Canada), pp. 971–976, Jun 24–28, 2019.
[221] T. Desplenter, A. Kyrylova, T. K. Stanbury, A. Escoto, S. Chinchalkar, and A. L. Trejos, “A

wearable mechatronic brace for arm rehabilitation,” in IEEE RAS & EMBS International
Conference on Biomedical Robotics and Biomechatronics, (São Paulo, Brazil), pp. 491–496,
Aug 12–15, 2014.
[222] K.-H. Park, H.-I. Suk, and S.-W. Lee, “Position-independent decoding of movement inten-
tion for proportional myoelectric interfaces,” IEEE Transactions on Neural Systems and
Rehabilitation Engineering, vol. 24, no. 9, pp. 928–939, 2015.
[223] M. Oskoei and H. H. H. Hu, “Support vector machine-based classification scheme for my-
oelectric control applied to upper limb,” IEEE Transactions on Biomedical Engineering,
vol. 55, no. 8, pp. 1956–1965, 2008.
[224] J. Tryon and A. L. Trejos, “Classification of task weight during dynamic motion using EEG-
EMG fusion,” IEEE Sensors Journal, vol. 21, no. 4, pp. 5012–5021, 2021.
[225] J. T. Gwin and D. P. Ferris, “An EEG-based study of discrete isometric and isotonic human
lower limb muscle contractions,” Journal of NeuroEngineering and Rehabilitation, vol. 9,
no. 1, pp. 35–1–11, 2012.
[226] J. Z. Liu, Q. Yang, B. Yao, R. W. Brown, and G. H. Yue, “Linear correlation between
fractal dimension of EEG signal and handgrip force,” Biological Cybernetics, vol. 93, no. 2,
pp. 131–140, 2005.
[227] A. Abdul-latif, I. Cosic, D. K. Kumar, B. Polus, and C. Da Costa, “Power changes of EEG
signals associated with muscle fatigue: The Root Mean Square analysis of EEG bands,” in
Intelligent Sensors, Sensor Networks and Information Processing Conference, (Melbourne,
Australia), pp. 531–534, Dec 14–17, 2004.
[228] M. Hayashi, S. Tsuchimoto, N. Mizuguchi, M. Miyatake, S. Kasuga, and J. Ushiba, “Two-

stage regression of high-density scalp electroencephalograms visualizes force regulation sig-
naling during muscle contraction,” Journal of Neural Engineering, vol. 16, no. 5, pp. 056020–
1–14, 2019.
[229] R. Roy, D. Sikdar, M. Mahadevappa, and C. S. Kumar, “EEG based motor imagery study of
time domain features for classification of power and precision hand grasps,” in International
IEEE/EMBS Conference on Neural Engineering, (Shanghai, China), pp. 440–443, May 25–
28, 2017.
[230] S. Jitaree and P. Phukpattaranont, “Force classification using surface electromyography from
various object lengths and wrist postures,” Signal, Image and Video Processing, vol. 13, no. 6,
pp. 1183–1190, 2019.
REFERENCES 350
[231] F. S. Ayachi, S. Boudaoud, and C. Marque, “Evaluation of muscle force classification using
shape analysis of the sEMG probability density function: A simulation study,” Medical and
Biological Engineering and Computing, vol. 52, no. 8, pp. 673–684, 2014.
[232] A. M. Oosterwijk, M. K. Nieuwenhuis, C. P. van der Schans, and L. J. Mouton, “Shoulder

and elbow range of motion for the performance of activities of daily living: A systematic
review,” Physiotherapy Theory and Practice, vol. 34, pp. 505–528, Jul 2018.
[233] M. Tröster, U. Schneider, T. Bauernhansl, J. Rasmussen, and M. S. Andersen, “Simulation

framework for active upper limb exoskeleton design optimization based on musculoskeletal
modeling,” in Technische Unterstützungssysteme, die die Menschen wirklich wollen, (Ham-
burg, Germany), pp. 345–353, Dec 11–12 2018.
[234] J. Garrido, W. Yu, and A. Soria, “Modular design and modeling of an upper limb ex-
oskeleton,” in IEEE RAS and EMBS International Conference on Biomedical Robotics and
Biomechatronics, (São Paulo, Brazil), pp. 508–513, IEEE Computer Society, 2014.
[235] P. Castiglioni, “What is wrong in Katz’s method? Comments on: ”A note on fractal dimen-
sions of biomedical waveforms”,” Computers in Biology and Medicine, vol. 40, no. 11-12,
pp. 950–952, 2010.
[236] A. Phinyomark, P. Phukpattaranont, and C. Limsakul, “Fractal analysis features for weak
and single-channel upper-limb EMG signals,” Expert Systems with Applications, vol. 39,
no. 12, pp. 11156–11163, 2012.
[237] W. K. Tam, K. Y. Tong, F. Meng, and S. Gao, “A minimal set of electrodes for motor imagery
BCI to control an assistive device in chronic stroke subjects: A multi-session study,” IEEE
Transactions on Neural Systems and Rehabilitation Engineering, vol. 19, no. 6, pp. 617–627,
2011.
[238] J. Tryon and A. L. Trejos, “Evaluating convolutional neural networks as a method of

EEG–EMG fusion,” Frontiers in Neurorobotics, vol. 15, pp. 692183–1–20, 2021.
[239] P. Leelaarporn, P. Wachiraphan, T. Kaewlee, T. Udsa, R. Chaisaen, T. Choksatchawathi,

R. Laosirirat, P. Lakhan, P. Natnithikarat, K. Thanontip, W. Chen, S. C. Mukhopadhyay,
and T. Wilaiprasitporn, “Sensor-driven achieving of smart living: a review,” IEEE Sensors
Journal, vol. 21, pp. 10369–10391, May 2021.
[240] J. J. Bird, J. Kobylarz, D. R. Faria, A. Ekart, and E. P. Ribeiro, “Cross-domain MLP and
CNN transfer learning for biological signal processing: EEG and EMG,” IEEE Access, vol. 8,
pp. 54789–54801, 2020.
[241] B. Xu, L. Zhang, A. Song, C. Wu, W. Li, D. Zhang, G. Xu, H. Li, and H. Zeng, “Wavelet
transform time-frequency image and convolutional network-based motor imagery EEG clas-
sification,” IEEE Access, vol. 7, pp. 6084–6093, 2019.
[242] Y. Teh and L. J. Hargrove, “The effects of limb position and external load on offline myoelec-
tric pattern recognition control,” in IEEE International Conference on Biomedical Robotics
and Biomechatronics, (New York, USA), pp. 654–659, Nov 29–Dec 1, 2020.
REFERENCES 351
[243] F. Chollet et al., “Keras.” https://keras.io, 2015.
[244] T. O’Malley, E. Bursztein, J. Long, F. Chollet, H. Jin, L. Invernizzi, et al., “Keras Tuner.”
https://github.com/keras-team/keras-tuner, 2019.
[245] C. Luo, X. Li, L. Wang, J. He, D. Li, and J. Zhou, “How does the data set affect CNN-based
image classification performance?,” in International Conference on Systems and Informatics,
(Nanjing, China), pp. 361–366, Nov 10–12, 2018.
[246] K. Greff, R. K. Srivastava, J. Koutnik, B. R. Steunebrink, and J. Schmidhuber, “LSTM:

A search space odyssey,” IEEE Transactions on Neural Networks and Learning Systems,
vol. 28, no. 10, pp. 2222–2232, 2017.
[247] K. S. Sudeep and K. K. Pal, “Preprocessing for image classification by convolutional neural
networks,” in IEEE International Conference on Recent Trends in Electronics, Information
and Communication Technology, (Bangalore, India), pp. 1778–1781, Institute of Electrical
and Electronics Engineers Inc., May 20–21 2016.
[248] X. Wan, “Influence of feature scaling on convergence of gradient iterative algorithm,” Journal
of Physics: Conference Series, vol. 1213, pp. 032021–1–5, Jun 2019.
[249] M. Kociolek, M. Strzelecki, and R. Obuchowicz, “Does image normalization and intensity res-
olution impact texture classification?,” Computerized Medical Imaging and Graphics, vol. 81,
pp. 101716–1–17, Apr 2020.
[250] F. Gao, T. Wu, J. Li, B. Zheng, L. Ruan, D. Shang, and B. Patel, “SD-CNN: A shallow-deep
CNN for improved breast cancer diagnosis,” Computerized Medical Imaging and Graphics,
vol. 70, pp. 53–62, 2018.
[251] Z. Yin, B. Wan, F. Yuan, X. Xia, and J. Shi, “A deep normalization and convolutional neural
network for image smoke detection,” IEEE Access, vol. 5, pp. 18429–18438, Aug 2017.
[252] Z. Yu, X. Jiang, T. Wang, and B. Lei, “Aggregating deep convolutional features for melanoma
recognition in dermoscopy images,” in International Workshop on Machine Learning in Med-
ical Imaging, (Quebec City, Canada), pp. 238–246, Sep 10 2017.
[253] X. Wu, R. He, Z. Sun, and T. Tan, “A light CNN for deep face representation with noisy
labels,” IEEE Transactions on Information Forensics and Security, vol. 13, pp. 2884–2896,
Nov 2018.
[254] S. Saha and M. Baumert, “Intra- and inter-subject variability in EEG-based sensorimo-
tor brain computer interface: a review,” Frontiers in Computational Neuroscience, vol. 13,
pp. 87–1–8, Jan 2020.
[255] A. Burden, “How should we normalize electromyograms obtained from healthy participants?
What we have learned from over 25years of research,” Journal of Electromyography and
Kinesiology, vol. 20, pp. 1023–1035, Dec 2010.
[256] L. Logesparan, E. Rodriguez-Villegas, and A. J. Casson, “The impact of signal normalization

on seizure detection using line length features,” Medical and Biological Engineering and
Computing, vol. 53, pp. 929–942, Oct 2015.
REFERENCES 352
[257] N. Ball and J. Scurr, “Electromyography normalization methods for high-velocity muscle
actions: review and recommendations,” Journal of Applied Biomechanics, vol. 29, no. 5,
pp. 600—-608, 2013.
[258] M. Besomi, P. W. Hodges, E. A. Clancy, J. Van Dieën, F. Hug, M. Lowery, R. Merletti,

K. Søgaard, T. Wrigley, T. Besier, R. G. Carson, C. Disselhorst-Klug, R. M. Enoka, D. Falla,
D. Farina, S. Gandevia, A. Holobar, M. C. Kiernan, K. McGill, E. Perreault, J. C. Rothwell,
and K. Tucker, “Consensus for experimental design in electromyography (CEDE) project:
Amplitude normalization matrix,” Journal of Electromyography and Kinesiology, vol. 53,
pp. 102438–1–17, Aug 2020.
[259] M. J. Islam, S. Ahmad, F. Haque, M. B. I. Reaz, M. A. Bhuiyan, and M. R. Islam, “A novel

signal normalization approach to improve the force invariant myoelectric pattern recognition
of transradial amputees,” IEEE Access, vol. 9, pp. 79853–79868, 2021.
[260] A. Chalard, M. Belle, E. Montané, P. Marque, D. Amarantini, and D. Gasq, “Impact of the
EMG normalization method on muscle activation and the antagonist-agonist co-contraction
index during active elbow extension: Practical implications for post-stroke subjects,” Journal
of Electromyography and Kinesiology, vol. 51, pp. 102403–1–7, Apr 2020.
[261] C. Leon-Urbano and W. Ugarte, “End-to-end electroencephalogram (EEG) motor imagery

classification with Long Short-Term Memory (LSTM) neural networks,” in IEEE Symposium
Series on Computational Intelligence, (Canberra, Australia), pp. 2814–2820, Institute of
Electrical and Electronics Engineers Inc., Dec 1–4 2020.
[262] K. Kawintiranon, Y. Buatong, and P. Vateekul, “Online music emotion prediction on multiple
sessions of EEG data using SVM,” in International Joint Conference on Computer Science
and Software Engineering, (Khon Kaen, Thailand), pp. 1–6, IEEE, Jul 13–15 2016.
[263] X. Zhang, L. Yao, D. Zhang, X. Wang, Q. Z. Sheng, and T. Gu, “Multi-person brain ac-
tivity recognition via comprehensive EEG signal analysis,” in EAI International Conference
on Mobile and Ubiquitous Systems: Computing, Networking and Services, (Melbourne, Aus-
tralia), pp. 28–37, Association for Computing Machinery, Nov 7–10 2017.
[264] O. Akbulut, “Feature normalization effect in emotion classification based on EEG signals,”
Sakarya University Journal of Science, vol. 24, pp. 60–66, Feb 2020.
[265] M. F. Wahid, R. Tafreshi, M. Al-Sowaidi, and R. Langari, “Subject-independent hand gesture

recognition using normalization and machine learning algorithms,” Journal of Computational
Science, vol. 27, pp. 69–76, Jul 2018.
[266] M. F. Wahid, R. Tafreshi, M. Al-Sowaidi, and R. Langari, “An efficient approach to recognize
hand gestures using machine-learning algorithms,” in Middle East Conference on Biomedical
Engineering, (Tunis, Tunisia), pp. 171–176, IEEE Computer Society, Mar 28–30 2018.
[267] E. C. Jeong, S. J. Kim, Y. R. Song, and S. M. Lee, “Comparison of wrist motion classifica-
tion methods using surface electromyogram,” Journal of Central South University, vol. 20,
pp. 960–968, Apr 2013.
REFERENCES 353
[268] Y. R. Tabar and U. Halici, “A novel deep learning approach for classification of EEG motor
imagery signals,” Journal of Neural Engineering, vol. 14, pp. 016003–1–11, Feb 2016.
[269] K. Asai and N. Takase, “Finger motion estimation based on frequency conversion of EMG
signals and image recognition using convolutional neural network,” in International Confer-
ence on Control, Automation and Systems, (Jeju, Korea), pp. 1366–1371, Oct 18–21 2017.
[270] K. T. Kim, S. Park, T. H. Lim, and S. J. Lee, “Upper-limb electromyogram classification of

reaching-to-grasping tasks based on convolutional neural networks for control of a prosthetic
hand,” Frontiers in Neuroscience, vol. 15, pp. 733359–1–10, Oct 2021.
[271] X. Zha, L. Wehbe, R. J. Sclabassi, Z. Mace, Y. V. Liang, A. Yu, J. Leonardo, B. C. Cheng,

T. A. Hillman, D. A. Chen, and C. N. Riviere, “A deep learning model for automated
classification of intraoperative continuous EMG,” IEEE Transactions on Medical Robotics
and Bionics, vol. 3, pp. 44–52, Feb 2021.
[272] T. H. Shovon, Z. Al Nazi, S. Dash, and M. F. Hossain, “Classification of motor imagery EEG
signals with multi-input convolutional neural network by augmenting STFT,” in Interna-
tional Conference on Advances in Electrical Engineering, (Dhaka, Bangladesh), pp. 398–403,
Sep 26–28 2019.
[273] G. S. Jayalakshmi and V. S. Kumar, “Performance analysis of convolutional neural net-

work (CNN) based cancerous skin lesion detection system,” in International Conference on
Computational Intelligence in Data Science, (Gurugram, India), pp. 1–6, IEEE, Sep 6–7
2019.
[274] B. Borhanuddin, N. Jamil, S. Chen, M. Baharuddin, K. Tan, and T. Ooi, “Small-scale deep
network for dct-based images classification,” in International Conference and Workshops on
Recent Advances and Innovations in Engineering, (Kedah, Malaysia), pp. 1–6, Nov 27–29
2019.
[275] H. Dose, J. S. Møller, H. K. Iversen, and S. Puthusserypady, “An end-to-end deep learn-
ing approach to MI-EEG signal classification for BCIs,” Expert Systems with Applications,
vol. 114, pp. 532–542, Dec 2018.
[276] W. Wei, Q. Dai, Y. Wong, Y. Hu, M. Kankanhalli, and W. Geng, “Surface-electromyography-

based gesture recognition by multi-view deep learning,” IEEE Transactions on Biomedical
Engineering, vol. 66, pp. 2964–2973, Oct 2019.
[277] S. Münzner, P. Schmidt, A. Reiss, M. Hanselmann, R. Stiefelhagen, and R. Dürichen, “CNN-

based sensor fusion techniques for multimodal human activity recognition,” in International
Symposium on Wearable Computers, (Maui, Hawaii, USA), pp. 158–165, Association for
Computing Machinery, Sep 11–15 2017.
[278] L. M. Bittar and M. C. F. Castro, “Elbow flexion and extension movements characterization
by means of EMG,” in International Conference on Biomedical Electronics and Devices,
vol. 1, (Funchal, Madeira, Portugal), pp. 147–150, Jan 28–31 2008.
REFERENCES 354
[279] B. Liu, Z. Chen, and Y. Hu, “Lower limb motion recognition by integrating multi-modal
features based on machine learning method,” in International Conference on Computer Sci-
ence and Application Engineering, (Sanya China), pp. 135–1–6, Association for Computing
Machinery, Oct 20–22 2020.
[280] Y. Chang, J. Chen, C. Qu, and T. Pan, “Intelligent fault diagnosis of wind turbines via a
deep learning network using parallel convolution layers with multi-scale kernels,” Renewable
Energy, vol. 153, pp. 205–213, Jun 2020.
[281] N. Patel, A. Choromanska, P. Krishnamurthy, and F. Khorrami, “Sensor modality fusion

with CNNs for UGV autonomous driving in indoor environments,” in IEEE/RSJ Interna-
tional Conference on Intelligent Robots and Systems, (Vancouver, BC, Canada), pp. 1531–
1536, Sep 24–28 2017.
[282] V. Mitra, G. Sivaraman, H. Nam, C. Espy-Wilson, E. Saltzman, and M. Tiede, “Hybrid

convolutional neural networks for articulatory and acoustic information based speech recog-
nition,” Speech Communication, vol. 89, pp. 103–112, May 2017.
[283] P. Swarnkar, S. K. Jain, and R. K. Nema, “Adaptive control schemes for improving the
control system dynamics: a review,” IETE Technical Review, vol. 31, no. 1, pp. 17–33, 2014.
[284] I. R. Petersen and R. Tempo, “Robust control of uncertain systems: classical results and
recent developments,” Automatica, vol. 50, no. 5, pp. 1315–1335, 2014.
[285] B. Wei, “A tutorial on robust control, adaptive control and robust adaptive control-
application to robotic manipulators,” Inventions, vol. 4, pp. 49–1–13, Sep 2019.
[286] W. He, Z. Li, Y. Dong, and T. Zhao, “Design and adaptive control for an upper limb robotic
exoskeleton in presence of input saturation,” IEEE Transactions on Neural Networks and
Learning Systems, vol. 30, pp. 97–108, Jan 2019.
[287] X. Liang, H. Wang, and Y. Zhang, “Adaptive nonsingular terminal sliding mode control for
rehabilitation robots,” Computers and Electrical Engineering, vol. 99, pp. 107718–1–11, Apr
2022.
[288] M. R. Islam and S. Bai, “Payload estimation using forcemyography sensors for control of
upper-body exoskeleton in load carrying assistance,” Modeling, Identification and Control,
vol. 40, no. 4, pp. 189–198, 2019.
[289] W. Yu and J. Rosen, “Neural PID control of robot manipulators with application to an
upper limb exoskeleton,” IEEE Transactions on Cybernetics, vol. 43, pp. 673–684, Apr 2013.
[290] Q. Ai, Z. Liu, W. Meng, Q. Liu, and S. Q. Xie, “Machine learning in robot assisted upper
limb rehabilitation: A focused review,” IEEE Transactions on Cognitive and Developmental
Systems, pp. 1–11, 2021.
[291] M. Benosman, “Model-based vs data-driven adaptive control: an overview,” International

Journal of Adaptive Control and Signal Processing, vol. 32, pp. 753–776, May 2018.
REFERENCES 355
[292] Y. Wen, J. Si, X. Gao, S. Huang, and H. H. Huang, “A new powered lower limb prosthesis
control framework based on adaptive dynamic programming,” IEEE Transactions on Neural
Networks and Learning Systems, vol. 28, pp. 2215–2220, Sep 2017.
[293] J. B. Huang, J. C. Hong, K. Y. Young, and C. H. Ko, “Development of upper-limb exoskele-

ton simulator for passive rehabilitation,” in International Automatic Control Conference,
(Kaohsiung, Taiwan), pp. 335–339, Institute of Electrical and Electronics Engineers Inc.,
Nov 26–28 2014.
[294] D. Winter, Biomechanics and Motor Control of Human Movement. John Wiley & Sons,
Incorporated, 4th ed., 2009.
[295] N. Nise, Control Systems Engineering. John Wiley & Sons, Incorporated, 6th ed., 2011.
® ®
[296] S. Baldursson, “Bldc motor modelling and control - a Matlab /Simulink implementation,”
M.S. thesis, Department of Energy and Environment, Chalmers University of Technology,
Göteborg, Sweden, 2005. [Online]. Available: https://hdl.handle.net/20.500.12380/
10897.
[297] I. Kecskés, E. Burkus, F. Bazsó, and P. Odry, “Model validation of a hexapod walker robot,”
Robotica, vol. 35, pp. 419–462, Feb 2017.
[298] D. O. Aborisade, “DC motor with load coupled by gears speed control using modified Ziegler-
Nichols based PID tunings,” Control Theory and Informatics, vol. 4, no. 5, pp. 58–87, 2014.
[299] V. Sankardoss and P. Geethanjali, “Parameter estimation and speed control of a PMDC
motor used in wheelchair,” Energy Procedia, vol. 117, pp. 345–352, 2017.
[300] A. B. Yildiz, “Electrical equivalent circuit based modeling and analysis of direct current
motors,” Electrical Power and Energy Systems, vol. 43, pp. 1043–1047, Dec 2012.
[301] S. Anatolii, Y. Naung, H. L. Oo, Z. M. Khaing, and K. Z. Ye, “The comparative analysis of
modelling of simscape physical plant system design and armature-controlled system design of
DC motor,” in IIEEE Conference of Russian Young Researchers in Electrical and Electronic
Engineering, (St. Petersburg, Russia), pp. 998–1002, Institute of Electrical and Electronics
Engineers Inc., Feb 1–3 2017.
[302] D. Dresscher, T. J. De Vries, and S. Stramigioli, “Motor-gearbox selection for energy ef-
ficiency,” in IEEE International Conference on Advanced Intelligent Mechatronics, (Banff,
Alberta, Canada), pp. 669–675, Institute of Electrical and Electronics Engineers Inc., Jul
12–15 2016.
[303] B. M. Pillai and J. Suthakorn, “Motion control applications: observer based DC motor
parameters estimation for novices,” International Journal of Power Electronics and Drive
Systems, vol. 10, pp. 195–210, Mar 2019.
[304] P. Boscariol and D. Richiedei, “Energy optimal design of servo-actuated systems: A concur-
rent approach based on scaling rules,” Renewable and Sustainable Energy Reviews, vol. 156,
pp. 111923–1–13, Mar 2022.
REFERENCES 356
[305] N. P. Mahajan and S. B. Deshpande, “Study of nonlinear behavior of dc motor using modeling
and simulation,” International Journal of Scientific and Research Publications, vol. 3, no. 3,
pp. 576–580, 2013.
[306] T. A. Tutunji, “DC motor identification using impulse response data,” in The International
Conference on ”Computer as a Tool”, vol. 2, (Belgrade, Serbia), pp. 1734–1736, IEEE Com-
puter Society, Nov 21–24 2005.
[307] H. Maghfiroh, A. Ataka, O. Wahyunggoro, and A. I. Cahyadi, “Optimal energy control of
DC Motor speed control: Comparative study,” in International Conference on Computer,
Control, Informatics and Its Applications, (Jakarta, Indonesia), pp. 89–93, IEEE Computer
Society, Nov 19–21 2013.
[308] G. Shahgholian and P. Shafaghi, “State space modeling and eigenvalue analysis of the perma-
nent magnet DC motor drive system,” in International Conference on Electronic Computer
Technology, (Kuala Lumpur, Malaysia), pp. 63–67, May 7–10 2010.
[309] Munadi, M. A. Akbar, T. Naniwa, and Y. Taniai, “Model reference adaptive control for
DC motor based on Simulink,” in International Annual Engineering Seminar, (Yogyakarta,
Indonesia), pp. 101–106, Institute of Electrical and Electronics Engineers Inc., Aug 1–3 2017.
[310] W. Wu, “DC motor parameter identification using speed step responses,” Modelling and
Simulation in Engineering, vol. 2012, pp. 189757–1–5, 2012.
[311] S. Kumar and R. Negi, “A comparative study of PID tuning methods using anti-windup con-
troller,” in International Conference on Power, Control and Embedded Systems, (Allahabad,
India), pp. 1–4, Dec 17–19 2012.
[312] M. O. Okelola, D. O. Aborisade, and P. A. Adewuyi, “Performance and configuration analysis
of tracking time anti-windup PID controllers,” Jurnal Ilmiah Teknik Elektro Komputer dan
Informatika, vol. 6, pp. 20–29, Jan 2020.
[313] maxon Group, “EC 90 flat 90 mm, brushless, 260 watt”, 500266 datasheet,
2021. [Online]. Available: https://www.maxongroup.com/medias/sys_master/root/
8882567512094/EN-21-310.pdf.
[314] maxon Group, “Planetary Gearhead GP 52 C 52 mm, 4.0–30.0 Nm”, 223089 datasheet,
8882781421598/EN-21-410-411.pdf.
[315] B. Tibor, V. Fedak, and F. Ďurovský, “Modeling and simulation of the BLDC motor in MAT-
LAB GUI,” in IEEE International Symposium on Industrial Electronics, (Gdańsk, Poland),
pp. 1403–1407, 2011.
[316] A. B. Sajid, A. Marryam, and M. Ali, “Modelling and control of brushless DC Motor.”
EasyChair Preprint no. 5710, EasyChair, 2021. [Online]. Available: https://easychair.
org/publications/preprint/rhM9.
[317] P. Mukherjee and M. Sengupta, “Closed loop speed control of a laboratory fabricated brush-
less DC motor drive prototype using position sensor,” in National Power Electronics Con-
ference, vol. 2018-Janua, (Pune, India), pp. 166–171, Dec 18–20 2017.
REFERENCES 357
[318] K. Krykowski and J. Hetmańczyk, “Constant current models of brushless DC motor,” Elec-
trical, Control and Communication Engineering, vol. 3, pp. 19–24, Sep 2013.
[319] C. Xiang, X. Wang, Y. Ma, and B. Xu, “Practical modeling and comprehensive system iden-
tification of a BLDC motor,” Mathematical Problems in Engineering, vol. 2015, pp. 879581–
1–11, 2015.
[320] C. Huang, F. Lei, X. Han, and Z. Zhang, “Determination of modeling parameters for a
brushless DC motor that satisfies the power performance of an electric vehicle,” Measurement
and Control, vol. 52, pp. 765–774, Sep 2019.
[321] M. Ridwan, M. N. Yuniarto, and Soedibyo, “Electrical equivalent circuit based modeling and
analysis of brushless direct current (BLDC) motor,” in International Seminar on Intelligent
Technology and Its Application, (Mataram, Indonesia), pp. 471–478, Institute of Electrical
and Electronics Engineers Inc., Jul 28–30 2016.
[322] M. Azab, “Comparative study of BLDC motor drives with different approaches: FCS-model
predictive control and hysteresis current control,” World Electric Vehicle Journal, vol. 13,
pp. 112–1–22, Jun 2022.
[323] R. Nalli, K. Subbarao, Ramamoorthi, and M. Kirankumar, “A new integrated AC-DC con-
verter fed sensorless controlling technique for a 3-phase BLDC motor,” Journal of Engineer-
ing Science and Technology, vol. 17, no. 3, pp. 2080–2094, 2022.
[324] G. Sieklucki, “Analysis of the transfer-function models of electric drives with controlled
voltage source,” Przeglad Elektrotechniczny, vol. 88, no. 7a, pp. 250–255, 2012.
[325] maxon Group, “maxon EC motor ironless winding Technology – short and to the point”,
8815461662750/EC-Technology-short-and-to-the-point-14-EN-32-35.pdf.
[326] maxon Group, “maxon DC motor and maxon EC motor Key Information”, 2014. [Online].
Available: https://www.maxongroup.com/medias/sys_master/root/8815460712478/
DC-EC-Key-Information-14-EN-42-50.pdf.
[327] I. Kecskés, E. Burkus, and P. Odry, “Gear efficiency modeling in a simulation model of
a DC gearmotor,” in IEEE International Symposium on Computational Intelligence and
Informatics, (Budapest, Hungary), pp. 65–69, Nov 21–22 2018.
[328] S. S. Ge, T. H. Lee, and S. X. Ren, “Adaptive friction compensation of servo mechanisms,”
International Journal of Systems Science, vol. 32, pp. 523–532, Jan 2001.
[329] A. G. Katsioula, Y. L. Karnavas, and Y. S. Boutalis, “An enhanced simulation model for
DC motor belt drive conveyor system control,” in International Conference on Modern Cir-
cuits and Systems Technologies, (Thessaloniki, Greece), pp. 1–4, Institute of Electrical and
Electronics Engineers Inc., May 7–9 2018.
[330] C. M. Fernandes, R. C. Martins, and J. H. Seabra, “Coefficient of friction equation for gears
based on a modified Hersey parameter,” Tribology International, vol. 101, pp. 204–217, Sep
2016.
REFERENCES 358
[331] F. Beer, E. Johnston, D. Mazurek, P. Cornwell, and E. Eisenberg, Vector Mechanics for
Engineers: Statics and Dynamics. McGraw-Hill Companies, 9th ed., 2010.
[332] K. Kiguchi, T. D. Lalitharatne, and Y. Hayashi, “Estimation of Forearm Supination/Prona-

tion Motion Based on EEG Signals to Control an Artificial Arm,” Journal of Advanced
Mechanical Design, Systems, and Manufacturing, vol. 7, no. 1, pp. 74–81, 2013.
[333] T. D. Lalitharatne, A. Yoshino, Y. Hayashi, K. Teramoto, and K. Kiguchi, “Toward EEG

control of upper limb power-assist exoskeletons: A preliminary study of decoding elbow joint
velocities using EEG signals,” in International Symposium on Micro-NanoMechatronics and
Human Science, (Nagoya, Japan), pp. 421–424, Nov 4–7 2012.
[334] J. H. Kim, F. Bießmann, and S. W. Lee, “Decoding three-dimensional trajectory of executed

and imagined arm movements from electroencephalogram signals,” IEEE Transactions on
Neural Systems and Rehabilitation Engineering, vol. 23, pp. 867–876, Sep 2015.
[335] R. Sosnik and O. Ben Zur, “Reconstruction of hand, elbow and shoulder actual and imagined
trajectories in 3D space using EEG slow cortical potentials,” Journal of Neural Engineering,
vol. 17, no. 1, pp. 016065–1–15, 2020.
[336] H. ElMohandes, S. Eldawlatly, J. M. C. Audı́, R. Ruff, and K.-P. Hoffmann, “A multi-Kalman

filter-based approach for decoding arm kinematics from EMG recordings,” BioMedical En-
gineering OnLine, vol. 21, pp. 60–1–18, Sep 2022.
[337] Z. Lei, “An upper limb movement estimation from electromyography by using BP neural
network,” Biomedical Signal Processing and Control, vol. 49, pp. 434–439, Mar 2019.
[338] F. Xiao, Y. Wang, Y. Gao, Y. Zhu, and J. Zhao, “Continuous estimation of joint angle
from electromyography using multiple time-delayed features and random forests,” Biomedical
Signal Processing and Control, vol. 39, pp. 303–311, Jan 2018.
[339] R. Raj, R. Rejith, and K. Sivanandan, “Real time identification of human forearm kinemat-
ics from surface EMG signal using artificial neural network models,” Procedia Technology,
vol. 25, pp. 44–51, 2016.
[340] Z. Tang, H. Yu, H. Yang, L. Zhang, and L. Zhang, “Effect of velocity and acceleration in
joint angle estimation for an EMG-Based upper-limb exoskeleton control,” Computers in
Biology and Medicine, vol. 141, pp. 105156–1–9, Feb 2022.
[341] J. G. C. Alfaro and A. L. Trejos, “User-independent hand gesture recognition classification

models using sensor fusion,” Sensors, vol. 22, pp. 1321–1–19, Feb 2022.
[342] R. Sharma, A. Dasgupta, R. Cheng, C. Mishra, and V. H. Nagaraja, “Machine Learning for
Musculoskeletal Modeling of Upper Extremity,” IEEE Sensors Journal, pp. 1–13, Aug 2022.
[343] B. Siciliano, L. Sciavicco, L. Villani, and G. Oriolo, Robotics: Modelling, Planning and
Control. Advanced Textbooks in Control and Signal Processing, Springer London, 2009.
[344] S. A. Ali, K. Azha Mohd Annuar, and M. Fahmi Miskon, “Trajectory planning for exoskeleton
robot by using cubic and quintic polynomial equation,” International Journal of Applied
Engineering Research, vol. 11, no. 13, pp. 7943–7946, 2016.
REFERENCES 359
[345] S. Fang, X. Ma, Y. Zhao, Q. Zhang, and Y. Li, “Trajectory planning for seven-DoF robotic
arm based on quintic polynormial,” in International Conference on Intelligent Human-
Machine Systems and Cybernetics, vol. 2, (Hangzhou, China), pp. 198–201, Institute of
Electrical and Electronics Engineers Inc., Aug 24–25 2019.
[346] G. Perumalsamy, P. Visweswaran, J. Jose, S. Joseph Winston, and S. Murugan, “Quin-

tic interpolation joint trajectory for the path planning of a serial two-axis robotic arm for
PFBR steam generator inspection,” in Machines, Mechanism and Robotics (D. Badodkar
and T. Dwarakanath, eds.), pp. 637–648, Springer Singapore, 2019.
[347] C. Nguiadem, M. Raison, and S. Achiche, “Motion planning of upper-limb exoskeleton robots:
A review,” Applied Sciences, vol. 10, pp. 7626–1–21, Nov 2020.
[348] M. V. Shcherbakov, A. Brebels, N. L. Shcherbakova, A. P. Tyukov, T. A. Janovsky, and

V. A. evich Kamaev, “A survey of forecast error measures,” World Applied Sciences Journal,
vol. 24, no. 24, pp. 171–176, 2013.
[349] S. S. Naghibi, A. Fallah, A. Maleki, and F. Ghassemi, “Elbow angle generation during
activities of daily living using a submovement prediction model,” Biological Cybernetics,
vol. 114, pp. 389–402, Jun 2020.
[350] T. Chai and R. R. Draxler, “Root mean square error (RMSE) or mean absolute error (MAE)?
–Arguments against avoiding RMSE in the literature,” Geoscientific Model Development,
vol. 7, pp. 1247–1250, Jun 2014.
[351] C. Wang, W. Guo, H. Zhang, L. Guo, C. Huang, and C. Lin, “sEMG-based continuous
estimation of grasp movements by long-short term memory network,” Biomedical Signal
Processing and Control, vol. 59, pp. 101774–1–14, May 2020.
[352] M. Yu, Y. C. Tham, T. H. Rim, D. S. Ting, T. Y. Wong, and C. Y. Cheng, “Reporting on

deep learning algorithms in health care,” The Lancet Digital Health, vol. 1, pp. e328–e329,
Nov 2019.
[353] G. Brassington, “Mean absolute error and root mean square error: which is the better metric
for assessing model performance?,” in Geophysical Research Abstracts, vol. 19, (Vienna,
Austria), p. 3574, Apr 23–28 2017.
[354] N. Hogan and D. Sternad, “Sensitivity of smoothness measures to movement duration, am-
plitude, and arrests,” Journal of Motor Behavior, vol. 41, no. 6, pp. 529–534, 2009.
[355] E. Kholinne, M. J. Gandhi, A. Adikrishna, H. Hong, H. Kim, J. Hong, and I. H. Jeon, “The
dimensionless squared jerk: an objective parameter that improves assessment of hand motion
analysis during simulated shoulder arthroscopy,” BioMed Research International, vol. 2018,
pp. 7816160–1–8, 2018.
[356] M. Mohr and P. Federolf, “Fatigue-related reductions in movement smoothness during a

lateral shuffle and side-cutting movement,” European Journal of Sport Science, pp. 1–10,
2021.
REFERENCES 360
[357] S. M. Engdahl and D. H. Gates, “Differences in quality of movements made with body-
powered and myoelectric prostheses during activities of daily living,” Clinical Biomechanics,
vol. 84, pp. 105311–1–8, Apr 2021.
[358] S. Balasubramanian, A. Melendez-Calderon, A. Roby-Brami, and E. Burdet, “On the anal-
ysis of movement smoothness,” Journal of NeuroEngineering and Rehabilitation, vol. 12,
no. 1, pp. 112–1–11, 2015.
[359] S. Balasubramanian, A. Melendez-Calderon, and E. Burdet, “A robust and sensitive met-
ric for quantifying movement smoothness,” IEEE Transactions on Biomedical Engineering,
vol. 59, no. 8, pp. 2126–2136, 2012.
[360] P. Gulde and J. Hermsdörfer, “Smoothness metrics in complex movement tasks,” Frontiers
in Neurology, vol. 9, pp. 615–1–7, Sep 2018.
[361] A. Melendez-Calderon, C. Shirota, and S. Balasubramanian, “Estimating movement smooth-
ness from inertial measurement units,” Frontiers in Bioengineering and Biotechnology, vol. 8,
pp. 558771–1–16, Jan 2021.
[362] R. K. Sahu, S. Panda, and S. Padhan, “Optimal gravitational search algorithm for automatic
generation control of interconnected power systems,” Ain Shams Engineering Journal, vol. 5,
no. 3, pp. 721–733, 2014.
[363] A. Majumder, D. Sarkar, S. Chakraborty, A. Singh, S. S. Roy, and A. Arora, “Neural
network-based gain scheduled position control of a pneumatic artificial muscle,” in IEEE In-
ternational Conference on Electronics, Computing and Communication Technologies, (Ban-
galore, India), pp. 1–6, Institute of Electrical and Electronics Engineers Inc., Jul 8–10, 2022.
[364] Z. Wang, Y. Chang, and X. Sui, “Impedance control of upper limb rehabilitation robot
based on neural network,” in International Conference on Industrial Informatics - Comput-
ing Technology, Intelligent Technology, Industrial Information Integration, (Wuhan, China),
pp. 173–176, Institute of Electrical and Electronics Engineers Inc., Dec 2–3, 2017.
[365] R. P. Borase, D. K. Maghade, S. Y. Sondkar, and S. N. Pawar, “A review of PID control,
tuning methods and applications,” International Journal of Dynamics and Control, vol. 9,
pp. 818–827, Jun 2021.
[366] Z. Wu, Z. Gao, D. Li, Y. Q. Chen, and Y. Liu, “On transitioning from PID to ADRC in
thermal power plants,” Control Theory and Technology, vol. 19, pp. 3–18, Feb 2021.
[367] J. Han, “From PID to active disturbance rejection control,” IEEE Transactions on Industrial
Electronics, vol. 56, no. 3, pp. 900–906, 2009.
[368] G. Andrikopoulos, G. Nikolakopoulos, and S. Manesis, “Motion control of a novel robotic
wrist exoskeleton via pneumatic muscle actuators,” in IEEE International Conference on
Emerging Technologies and Factory Automation, (Luxembourg, Luxembourg), pp. 1–8, In-
stitute of Electrical and Electronics Engineers Inc., Sep 8–11, 2015.
[369] H. Boufrioua and B. Boukhezzar, “Gain scheduling: a short review,” in International Con-
ference on Advanced Electrical Engineering, (Constantine, Algeria), pp. 1–6, Institute of
Electrical and Electronics Engineers Inc., Oct 29–31, 2022.
REFERENCES 361
[370] A. Frisoli, E. Sotgiu, C. Procopio, M. Bergamasco, B. Rossi, and C. Chisari, “Design and
implementation of a training strategy in chronic stroke with an arm robotic exoskeleton,” in
IEEE International Conference on Rehabilitation Robotics, (Zurich, Switzerland), pp. 1–8,
Jun 29–Jul 1, 2011.
[371] M. S. Mahmoud, “Switched systems,” in Switched Time-Delay Systems, ch. 4, pp. 75–107,
Springer, Boston, MA, 2010.
[372] H. Chang, W. Q. Ge, H. C. Wang, H. Yuan, and Z. W. Fan, “Laser beam pointing sta-
bilization control through disturbance classification,” Sensors, vol. 21, pp. 1946–1–12, Mar
2021.
[373] R. Yershov, V. Voytenko, and V. Bychko, “Software-based contact debouncing algorithm

with programmable auto-repeat profile feature,” in IEEE International Scientific-Practical
Conference Problems of Infocommunications Science and Technology, (Kyiv, Ukrraine),
pp. 813–818, Institute of Electrical and Electronics Engineers Inc., Oct 8–11, 2019.
[374] J. F. Borisoff, S. G. Mason, A. Bashashati, and G. E. Birch, “Brain-computer interface

design for asynchronous control applications: Improvements to the LF-ASD asynchronous
brain switch,” IEEE Transactions on Biomedical Engineering, vol. 51, pp. 985–992, Jun
2004.
[375] M. Fatourechi, R. K. Ward, and G. E. Birch, “Performance of a self-paced brain computer

interface on data contaminated with eye-movement artifacts and on data recorded in a sub-
sequent session,” Computational Intelligence and Neuroscience, vol. 2008, pp. 749204–1–14,
2008.
[376] B. Awwad Shiekh Hasan and J. Q. Gan, “Unsupervised movement onset detection from
EEG recorded during self-paced real hand movement,” Medical and Biological Engineering
and Computing, vol. 48, pp. 245–253, Mar 2010.
[377] G. Venture, K. Yamane, and Y. Nakamura, “In-vivo estimation of the human elbow joint dy-
namics during passive movements based on the musculo-skeletal kinematics computation,” in
IEEE International Conference on Robotics and Automation, (Orlando, Florida), pp. 2960–
2965, May 15–19, 2006.
[378] M. M. Van Der Krogt, L. Bar-On, T. Kindt, K. Desloovere, and J. Harlaar, “Neuro-
musculoskeletal simulation of instrumented contracture and spasticity assessment in children
with cerebral palsy,” Journal of NeuroEngineering and Rehabilitation, vol. 13, pp. 64–1–11,
Jul 2016.
[379] N. P. Lawrence, M. G. Forbes, P. D. Loewen, D. G. McClement, J. U. Backström, and

R. B. Gopaluni, “Deep reinforcement learning with shallow controllers: an experimental
application to PID tuning,” Control Engineering Practice, vol. 121, pp. 105046–1–14, Apr
2022.
[380] Y. Huang and W. Xue, “Active disturbance rejection control: Methodology and theoretical
analysis,” ISA Transactions, vol. 53, no. 4, pp. 963–976, 2014.
REFERENCES 362
[381] Z. Wu, G. Shi, D. Li, Y. Liu, and Y. Q. Chen, “Active disturbance rejection control design
for high-order integral systems,” ISA Transactions, vol. 125, pp. 560—-570, Jun 2022.
[382] M. Chen, Y. Ma, Y. Li, D. Wu, Y. Zhang, and C. H. Youn, “Wearable 2.0: enabling human-
cloud integration in next generation healthcare systems,” IEEE Communications Magazine,
vol. 55, pp. 54–61, Jan 2017.
[383] T. Verstraten, G. Mathijssen, R. Furnémont, B. Vanderborght, and D. Lefeber, “Modeling

and design of geared DC motors for energy efficiency: Comparison between theory and
experiments,” Mechatronics, vol. 30, pp. 198–213, Sep 2015.
Appendix A
Additional Details for the

Upper-Limb Brace Simulator Model
A.1 DC Motor and Gearhead Model
A.1.1 Modelling the DC Motor
To represent the properties of a DC motor, an equivalent circuit diagram was used to describe
the electrical model (Figure A.1(a)), and an equivalent mechanical system diagram was used to
describe the physical model (Figure A.1(b)). The equivalent circuit models the DC current flowing
through the motor armature (i.e., the internal coils), which generates the magnetic field used to spin
the rotor. The mechanical system diagram models the rotational motion of the rotor, considering
its inertia and internal friction as it spins. This simple method of modelling a DC motor is a
commonly used approach that has been well explored in the literature [293, 295–310].
Focusing first on the electrical model, an equation needed to be found for the system input,
which will be utilized by the controller to drive the actuator. As DC motors are typically voltage
controlled, the armature voltage, v, was defined as the system input. An equation to describe the
input armature voltage with respect to time (v(t)), was solved for using Kirchhoff’s Voltage Law on
the equivalent circuit (Figure A.1(a)), resulting in the following differential equation [295, 296, 298–
301, 303–307, 309, 310]:
363
A.1 DC Motor and Gearhead Model 364
(a) Electrical model
(b) Mechanical model
Figure A.1: An equivalent circuit schematic (a) and equivalent mechanical system diagram (b)
describing the electrical and mechanical properties, respectively, of a DC motor. In the
electrical model (a), v(t) is the input armature voltage, i(t) is the armature current,
e(t) is the back electromotive force voltage (calculated using the motor back emf
constant, Ke , and the rotor angular velocity, θ̇M (t)), R is the armature resistance, and
L is the armature inductance. In the mechanical model (b), τM (t) is the output motor
torque, τRL (t) is the external rotor load torque, JM is the rotor inertia, and bM is the
coefficient of viscous friction of the motor.
di(t)
v(t) = L + Ri(t) + e(t), (A.1)
dt
where L is the armature inductance, R is the armature resistance, i(t) is the armature current
with respect to time, and e(t) is the back electromotive force (back emf) voltage with respect to
time (back emf being the voltage applied backwards into the circuit by the motor, as the spinning
rotor will act like a generator and induce a voltage with the armature coils).
To solve for the back emf, the following equation was utilized [295, 296, 298–301, 303–307, 310]:
dθM (t)
e(t) = Ke = Ke θ̇M (t), (A.2)
dt
where Ke is the back emf constant (a parameter of the motor) and θM (t) is the angular position
of the rotor with respect to time (so subsequently, θ̇M (t) is the angular velocity of the rotor with
respect to time, using Newton’s derivative notation here instead for simplicity).
Combining Equation A.1 with Equation A.2 provided the final differential equation for input
armature voltage:
di(t)
v(t) = L + Ri(t) + Ke θ̇M (t). (A.3)
dt
Looking now at the mechanical model (Figure A.1(b)), a similar procedure was used to obtain
a differential equation representing the physical properties of the motor. Of particular interest
was the output motor torque, τM (t), as this is the actual force that will be applied by the actu-
ator to move the brace (as discussed in Section 7.3.1). Using Newton’s equations of motion and
d’Alembert’s principle, the following differential equation for the mechanical model was obtained
[295, 296, 298–301, 303–307, 309, 310]:
τM (t) − τRL (t) = JM θ̈M (t) + bM θ̇M (t), (A.4)
where τRL (t) is the torque required to move whatever external load is attached to the rotor (which
could be a static load, or a dynamic one that varies with respect to time, depending on the
situation), JM is the inertia of the rotor (which requires some amount of torque to spin), and bM
is the coefficient of viscous friction of the motor (used to model internal friction losses as a viscous
damper, in which lost torque scales proportionally with rotational velocity).
Now that the equations for the mechanical and electrical properties were determined, they
needed to be coupled to complete the model. For DC motors this is done simply with the use of
a coupling equation that defines a proportional relationship between armature current and motor
torque. The coupling equation is defined as follows [295, 296, 298–301, 303–307, 310]:
τM (t) = Kt i(t), (A.5)
where Kt is referred to as the torque constant and is a property of the specific DC motor being used
in the system. With this coupling equation, interactions between the electrical and mechanical
components of the DC motor were defined, allowing the system input of voltage to be related to
the physical movement of the motor.
An interesting property of this value is that when SI units (or another consistent set of units) are
used for all parameters in the DC motor model (as was the case with the work done here), the back
emf constant (Ke ) and the torque constant (Kt ) will be equal [295, 307]. This means that Equation
A.3 could be rewritten using the torque constant, reducing the number of parameter values required
for model calculations. Simplifying the parameters in this manner has been used many times before
in DC motor modelling studies found in the literature [293, 297, 300–302, 305, 307, 308], therefore,
it was decided to employ this method for the DC motor model described here. Thus, the rewritten
equation used in the model was as follows:
di(t)
v(t) = L + Ri(t) + Kt θ̇M (t). (A.6)
dt
The final step in defining the DC motor model was to combine Equations A.6, A.4, and A.5
into a single equation relating the system input to the system output. As mentioned previously,
the system input for this actuator was voltage, as most DC motors are voltage controlled. The
output for this system was angular velocity, as this is the principle by which DC motors operate: as
voltage increases, rotational speed of the rotor increases. External load torque (τRL (t)) is also an
input to the actuator system, but was modelled instead as a disturbance (a non-controlled input)
and therefore is not supplied by the controller. Despite this, it still needs to be included into
the speed response equation for the DC motor to fully simulate its behaviour [308, 310]. Before
rearranging the aforementioned equations, the Laplace Transform was applied to convert them
from time domain equations into Laplace domain equations. This was done to simplify the algebra
required when rearranging the differential equations (as derivative and integral actions can now be
performed by simply multiplying or dividing, respectively, by the Laplace variable s). The result
of taking the Laplace transform, combining, and simplifying Equations A.6, A.4, and A.5 are as
follows:
Kt Ls + R
Θ̇M (s) = 2 · V (s) − · TRL (s). (A.7)
(JM s + bM )(Ls + R) + Kt (JM s + bM )(Ls + R) + Kt 2
Note that the output of Equation A.7 is angular velocity, as this is the actual measurable output of
a DC motor based on its operating principles (i.e., the rotor will spin at different speeds depending
on the input armature voltage). However, this is not an ideal measurement when considering how
the motor will be utilized for control of a wearable upper-limb brace. If the user of the brace is
trying to move their arm to a specific angle, the device will need to travel along a certain motion
path to reach this desired end point, and will need to be able to maintain that static position until
the user wants to move again. To achieve this in a way that feels comfortable to the user, the
mechatronic device will need to follow the desired reference trajectory as accurately as possible;
therefore, it makes sense to use a position control strategy for the system. This approach requires
that rotor position, not velocity, is measured as the output of the DC motor model. Fortunately,
this could be rectified simply by diving the result of Equation A.7 by the Laplace variable(s) to
obtain angular position through the integration of rotor velocity, as shown below [295]:
Θ̇M (s)
ΘM (s) = . (A.8)
s
Following this action, the output of the DC motor model could now be used to describe the motion
of the rotor for a given input voltage.
Another beneficial way of representing a system is through the use of block diagrams. These
diagrams can help provide a more intuitive understanding of system behaviour by showing how
the different subsystems within the model interact, compared to a combined equation (such as
Equation A.7), which may be hard to intuit directly. For this reason, block diagrams are often
used for control system design work. In the case of a DC motor, a block diagram will highlight the
interaction between the electrical and mechanical components of the model by separating them into
different transfer function blocks. An equivalent block diagram representing the model described
by Equation A.7 can be seen in Figure A.2 [293, 299, 301, 303, 305, 306, 309].
Figure A.2: A system block diagram outlining a Laplace domain representation of a DC motor.
The block diagram provides an equivalent description of the model outlined in Equa-
tion A.7. Note that capital letters were used to represent all previously discussed
time-domain variables, indicating a transformation into the Laplace domain (as is
convention).
A.1.2 Modelling the Gearhead
Up until now, the brace actuator was modelled with the assumption that only a single DC motor
was being used to generate motion; however, in an actual prototype this is not likely to be the
case. DC motors, by nature, supply high velocity rotations, but low torque. This means that it is
highly unlikely that a DC motor by itself will be strong enough to move the brace and the user’s
arm. Looking at the previously calculated static torque requirement of 17 Nm, most commercially
available DC motors that would be feasibly attainable using a reasonable budget, and suitable
for wearable applications, will not meet this constraint. To address this limitation, a DC motor
is typically paired with a gearing system built into the motor housing, referred to as a gearhead.
A gearhead uses the mechanical proprieties of gears to reduce the output motor velocity while
concurrently increasing the output motor toque by the same proportion [295, 297, 298, 304]. The
amount that velocity is reduced (and torque is increased) is referred to as the gear ratio, also
known as the as the gear reduction ratio or, often times, as simply the gear reduction, and is
determined by the difference in the number of teeth between the input and output gears. The gear
ratio for a simple two-gear setup (referred to as a single-stage configuration) is given by [295]:
N2
n= , (A.9)
N1
where n is the gear ratio, N2 is the number of teeth on the output gear, and N1 is the number
of teeth on the input gear. Now, most commercially available gearheads will use much more
complex gear-train designs than a single-stage configuration (for example, the previously developed
brace prototype used a planetary gearhead as part of its actuator system [221]); however, most
manufactures will still provide a pre-calculated simplified gear reduction value, often expressed as a
ratio using terminology of n : 1 [314]. This reduced gear ratio can be used to represent a multi-stage
gear system as an equivalent single-stage gearhead [302], simplifying the model calculations. Using
this reported gear ratio, output angular velocity of the gearhead (θ̇G ) is equal to the input motor
velocity (θ̇M ) reduced by the gear ratio (n), according to the following equation [295, 297, 298, 304]:
θ̇M
θ̇G = . (A.10)
n
Note, that Equation A.10 also applies to output position/acceleration as well, since they are
merely output velocity integrated or derived, respectively. In a similar vein, the output torque of
the gearhead (τG ) is increased according to the following equation [295, 297, 298]:
τG = n · τM . (A.11)
Knowing that an actual upper-limb brace would need to include a gearhead along with the
DC motor, modifications needed to be made to the mechanical portion of the actuator model
to incorporate the behaviour of the gearhead into the system [295, 297, 298, 302, 304]. The
updated equivalent mechanical system diagram can be seen in Figure A.3. This model includes
the reduction of the gears, as well as the losses in the system due to the mechanical behaviour
of the gearhead: the inertia of the gears (JG ), which requires torque to move, and the losses due
to friction (represented with the gearhead coefficient of viscous friction, bG ). The model was also
expanded to include the inertia of an external load attached to the gearhead rotor, JL , which
represents the object that the actuator is trying to move (in this case the subject’s arm and the
dumbbell).
The effect of this new mechanical component also needed to be incorporated into the DC Motor
system block diagram (shown in Figure A.2). Firstly, the reduction in output velocity/position,
caused by the gear ratio, needed to be included. This was done simply by adding a gain block to
divide output position by the gear reduction, n. Next, the losses due to internal friction within the
Figure A.3: An equivalent mechanical system diagram (updated from Figure A.1(b)) describing
the physical properties of an actuator comprised of a combined DC motor and a
gearhead. Regarding the new parameters added for the gearhead, N1 and N2 are
the number of teeth on the input and output gear, respectively, JG is inertia of the
gears, bG is the gearhead coefficient of viscous friction, θ̇G is the output velocity of
the gearhead, τG is the output torque of the gearhead, and JL is the inertia of the
external load acting on the gearhead.
actuation system needed to be considered in the mechanical transfer function block. To accomplish
this, it was decided to use a simple approximation, namely, adding the coefficients of friction for
the motor and the gearhead according to the following equation:
bMG = bM + bG , (A.12)
thereby modifying the model to consider all friction losses together as one equivalent value in the
mechanical transfer function block. It should be noted that this method of obtaining an equivalent
coefficient of friction is somewhat simplistic and likely will not capture the full real-world dynamic
response of the gearhead during motion. In an ideal scenario, the individual friction coefficients for
each internal stage of the gearhead would be considered, and an equivalent value (measured with
respect to the motor input shaft of the gearhead) would be calculated by reducing each internal
coefficient by how many gear stages it passes through [295]. However, internal friction specifications
of gearheads (as well as DC motors, for that manner) outlining their construction/behaviour is
rarely provided by actuator manufacturers and can be hard to quantify [297, 303, 327, 383], making
this approach infeasible for many applications. If access to physical hardware is possible, there are
methods to experimentally determine values for damping parameters [302, 303, 383]; however, for
the purposes of this work (a purely simulation-based project), such methods are not practical and
are outside of the defined project scope. Lack of detailed gearhead specifications will often lead
others to simply ignore the damping effect of viscous friction in the gearhead model, with some
studies omitting its coefficient of friction entirely [293, 298, 383]. This is a non-ideal approach
however, as it ignores a significant aspect of the gearhead mechanical behaviour on the model
output and can affect system accuracy [383]. Thus, it was decided for this work that using a simple
addition-based approximation to obtain an equivalent motor–gearhead coefficient of friction would
be a suitable approach, as it, at the very least, makes an attempt to include the gearhead and
motor friction effects into the simulator (as opposed to the alternative of ignoring it entirely).
Finally, the change to system inertias also needed to be considered. The gearhead itself will
have some inertia (JG ) which resists changes in rotation, similar to the rotor inertia of the DC
motor (JM ). If the reported gearhead inertia is measured with respect to the motor input shaft
(as is the case with the actuator chosen for this work), then these inertias can be simply added
together as in Equation A.12 [297, 304, 383]. A more involved discussion arises when considering
how to incorporate the effect of the external load into the actuator model. From the free-body
diagram in Figure 7.2, it is understood that there are actually two loads acting against the brace
actuator: the torque caused by the weight of the user’s arm and the torque caused by the weight
of the dumbbell. Both of these needed to be included in the mechanical model in some way. In
the base system block diagram for just the DC motor (Figure A.2) the external load torque acting
on the rotor (τRL (t)) was modelled as a disturbance that subtracts from the output motor torque
(τM (t)). While this is a suitable method for applying load torque effects, and a second mechanical
function block could be included to calculate the disturbance load torque over time using the load
inertias and a feed-back path from output acceleration [298, 306], there is a simpler method to
achieve the same effect. Instead of requiring a second mechanical transfer function block, the load
inertia (JL ) can instead be reflected through the gearhead and incorporated into the motor transfer
function equation. A property of modelling gears is that inertias acting on the output shaft can
instead be reflected to the input shaft, after considering how the gear reduction ratio will affect
this equivalent inertia [293, 295, 297, 298]. Since a gear train increases torque on the output shaft
of the gearhead, this means that the amount of load inertia “seen” by the input shaft of the motor
rotor (JRL ) will be reduced according to the following equation [293, 295, 297, 298]:
JL
JRL = . (A.13)
n2
Establishing the reflection of external load inertia through the gearhead, the effective inertia
acting on the DC motor could be expressed as follows:
JL
Jtotal = JM + JG + , (A.14)
n2
which was used in the mechanical transfer function block to model the equivalent behaviour of the
rotor, gearhead, and external loads simultaneously (instead of requiring multiple transfer function
blocks). A detailed discussion of how the load inertias were calculated for the model is provided
in Section 7.3.4. Note, that in regards to the effect of the external loads (i.e., the dumbbell and
the subject’s arm) on the motor, it was decided to only include inertia when modelling system
behaviour. Since the dumbbell does not slide across any surfaces during an elbow flexion–extension
repetition (the participant grips the weight tightly in a closed fist), there will be no friction forces
acting on it during motion, making its loading on the actuator purely caused by its inertia (as-
suming air drag is negligible). While it could be argued that there does exist some degree of
friction forces generated by the internal movements of joints/muscles in the user’s elbow, this type
of complex biological interaction would be almost impossible to quantify here in a meaningful way
and could not be modelled using the data available for this work. For these reasons, friction losses
caused by the elbow joint during motion were not considered in the model presented here and only
the inertia of the user’s arm was included in the external load effects (again assuming air drag is
negligible).
Having defined the changes to the mathematical model caused be the inclusion of the gearhead,
the DC motor block diagram (shown in Figure A.2) could then be updated to reflect the new system
properties. The updated equivalent block diagram representing the combined DC motor/gearhead
actuator system can be seen in Figure A.4.
A.2 Dumbbell Inertia Formula Derivation 373
Figure A.4: A system block diagram outlining a Laplace domain representation of a DC motor
combined with a gearhead. This system block diagram was updated from Figure
A.2 to include relevant gearhead parameters. Note that capital letters were used to
represent all previously discussed time-domain variables, indicating a transformation
into the Laplace domain (as is convention).
A.2 Dumbbell Inertia Formula Derivation
Modelling the inertia of the dumbbells required several approximations during formulation, which
will be described here. The first step in calculating the dumbbell inertia was to measure the
geometry of both hexagonal-head weights (since their shape/size will play a large role in their
inertial load). The dimensions of both dumbbells, as labelled in Figure A.5, can be seen in Table
A.1.
Figure A.5: Labels showing the dimension measurements taken for both dumbbells (3 lbs and 5
lbs). The measurement values can be found in Table A.1.
Based on the geometry of the dumbbells, an approximation was employed to simplify the inertia
calculations. It was decided to approximate the shape of the twin heads of the dumbbells as a
cylinder, as opposed to modelling them as hexagonal prisms (which more closely matches their
shape in reality). This was done to greatly simplify the formula used to calculate the inertia of
the heads, since the symmetry provided by the continuous side-face of a cylinder makes inertia
calculation less orientation specific (and since the formula for the moment of inertia of a cylinder
is well known and easy to work with). To ensure the cylinder approximation was still close to
Table A.1: Dimension measurement values taken for the 3 lbs and 5 lbs dumbbells. The dimension
labels correspond to what is shown in Figure A.5.
Dimension [mm]
Label
3 lbs 5 lbs
A 32.2 56.44
B 55 66.14
C 103.22 129.37
D 29.99 27.56
E 59.59 76.16
the shape of the real dumbbell heads, the dimensions of the cylinders were defined by the actual
dumbbell measurements. As shown in Figure A.6, the diameter of the cylinder was chosen to be
the distance between the points of the hexagonal head (label “E” in Figure A.5), to ensure the
cylinder would fully enclose the heads of the dumbbells and would provide a similarly sized load.
The height of the cylinder was defined as the width of the hexagonal head (label “A” in Figure
A.5) for the same reason. While less of an approximation, and more of a simplification, the bar of
the dumbbell that the user holds was also assumed to be a uniform cylinder (which was close to its
actual shape; however, in actuality it may have some slight curves/non-uniformity in its side-face
dimensions along the length of the cylinder). For the handlebar segment, the length of the bar
(label “C” in Figure A.5) was used as the height of the cylinder, and the thickness of the bar (label
“D” in Figure A.5) was used for the diameter.
Figure A.6: A graphical representation of the approximation used to treat the hexagonal dumbbell
heads as a cylinder (to simplify the inertia calculations). The diameter of the cylinder
was chosen as the largest distance between points on the hexagonal face, to ensure
the entire dumbbell head would be encompassed by the cylinder. The diameter label
corresponds to the values shown in Table A.1.
Once the shape of the dumbbell was defined, modelling their moments of inertia could then
begin. To start, it was important to take into account how the dumbbell would rotate with respect
to the elbow joint (the reference point used as the COR for all mathematical modelling in the
simulator). An important aspect to note is that all study participants were instructed to hold the
dumbbell vertically (as shown in Figure A.7).
Figure A.7: A photograph of a study participant holding a dumbbell during the data collection
process. It can be seen that the participant is holding the dumbbell in a vertical
position (with the bar perpendicular to their forearm) to reduce the contribution of
wrist muscles during elbow flexion–extension motion.
This was done to minimize activation of the wrist and forearm muscles during elbow flexion–
extension so that movement was provided solely from the biceps/triceps, allowing for collection
of more isolated EMG signals (it was observed that when holding the dumbbell horizontally,
the participants had a tenancy to curl their wrist inwards when performing elbow flexion). For
the purposes of this simulator, it was assumed that the dumbbells were also being held in this
orientation. Based on this assumption, the rotation of the dumbbell about its own COM and the
COR could then be considered, as outlined in Figure A.8.
From this diagram, it is clear that the parallel axis theorem [294, 331] must be used to translate
the dumbbell moment of inertial about its own COM (J0 ) to the COR (JDb ), according to the
following equation:
JDb = J0 + mDb dDb 2 , (A.15)

Figure A.8: A diagram describing the rotation and inertia of the dumbbell with respect to its own
COM and with respect to the brace. During elbow flexion–extension, the dumbbell
will move along path θ(t) and the components of this rotation occurring around the
dumbbell COM will result in the inertia J0 . This inertia can be related to a rotation
about the COR of the brace using the parallel axis theorem and the distance dDb ,
resulting in the inertia JDb . The dashed lines are used to indicate the axis of rotation
for the brace and the dumbbell, and the dots on both objects are used to indicate
their COM.
where mDb is the mass of the dumbbell (defined by their advertised weight of either 3 lbs or
5 lbs) and dDb is the distance from the COR to the dumbbell COM (previously calculated in
Equation 7.6). The assumption of this model is that the rotation of the user’s arm is a purely
two dimensional motion happening about the elbow joint, so the rotational axis of the dumbbell,
with respect to its own COM, is aligned perpendicularly to the side face of the handlebar cylinder.
This means that the 2D rotation imparted on the dumbbell from a rotation at the elbow joint is
happening solely along the viewing plane (i.e., “along the page” as drawn in Figure A.8). This
factor was important to define, as it determined which formula for the inertia of a cylinder was
used in the model calculations, since moment of inertia is orientation dependent.
To fully resolve Equation A.15, the dumbbell inertia, with respect to its COM, needed to be
determined. To accomplish this task, the dumbbell was split into three sections, each a complete
cylinder. The sections are the two heads on either end of the dumbbell (J1 and J2 ) and the
handlebar (J3 ). Each segment of the dumbbell has its own COM, located at its geometric centre
(using the assumption that the density of the material is uniform and that the cylinder is perfectly
symmetric). The COM of the handlebar section was assumed to be where the COM for the entire
dumbbell was located. The division of the dumbbell into three sections is shown in Figure A.9.
Figure A.9: A graphical representation of the partitions used to split the dumbbell into simplified
sections during the inertia calculations. The dumbbell was split into three smaller
cylinders, each with its own COM (indicated by the dot) and inertia about said COM
(J1 , J2 , and J3 ). The COM of the handlebar (J3 ) is aligned with the dumbbell COM
and COR (indicated by the dashed line). The inertia of the heads (J1 and J2 ) can
be related back to the dumbbell COM and total inertia (J0 ) using the parallel axis
theorem and the distance d1 .
The purpose of this segmentation was to simplify calculations, as the moment of inertia for
each section could be found using the basic inertia formula for a cylinder. Then the inertias of
each section could be added together, according to Equation A.16, to find the total inertia of the
dumbbell.
J0 = J1 ′ + J2 ′ + J3 . (A.16)
Note, as indicated in Figure A.9, the COM of the head segments is located away from the
dumbbell COM (which coincidences with the handlebar COM). This means that the parallel axis
theorem (as described in Equations 7.18 and A.15) is needed to relate the inertias of the head
segments about their own COM (J1 , J2 ) to the dumbbell COM. To simplify notation, the terms
J1 ′ and J2 ′ in Equation A.16 are used to represent the head segment inertias after the parallel axis
theorem has already been applied as follows:
J1 ′ = J1 + m1 d1 2 , (A.17)
where m1 is the mass of just the head segment of the dumbbell and d1 is the distance of the head
COM to the dumbbell COM.
Before completing the calculation, Equation A.16 was further simplified by making the as-
sumption that both dumbbell heads had an identical shape/mass, and had COMs that were the
same distance (d1 ) from the dumbbell COM. This allows the inertial contribution of the dumbbell
heads to be calculated once then simply doubled, as shown in the following equation:
J0 = 2J1 ′ + J3 . (A.18)
Substituting Equation A.17 into into Equation A.18 resulted in the following expanded equation
for the moment of inertia of the dumbbell about its own COM:
J0 = 2 J1 + m1 d1 2 + J3 .

(A.19)
Considering the cylinder shown in Figure A.10, the formula for its moment of inertia, when
rotating about the side face, is as follows [331]:
Figure A.10: Definitions for the dimensions and axes of a cylinder. The label r denotes the radius
of the cylinder, while the label h denotes the height. Dashed lines are used to indicate
internal portions of the cylinder.
1
Jx = Jy = m(3r2 + h2 ), (A.20)
12
where m is mass, r is the radius, and h is the height of the cylinder. Note, that due to the
way the participants held the dumbbell during data collection, the inertia formula for a cylinder
rotating about the z axis is not required. Since all rotations of the dumbbell segments during
elbow flexion–extension occurred about the x /y axes, Equation A.20 is the only cylinder inertia
formula necessary to incorporate into the model. Knowing this, the moment of inertia for the
dumbbell segments could be expanded into the following equations:
1
J1 ′ = m1 3r1 2 + h1 2 + m1 d1 2 ,

(A.21)
12
1
m3 3r3 2 + h3 2 ,

J3 = (A.22)
12
where the subscript 1 is used to indicate the head segment of the dumbbell and the subscript 3 is
used to indicate the handlebar segment of the dumbbell.
Taking Equations A.21 and A.22 and substituting them into Equation A.19 resulted in the
expanded equation for the dumbbell moment of inertia about its own COM, shown below:
1 1
m1 3r1 2 + h1 2 + m1 d1 2 + m3 3r3 2 + h3 2 .

J0 = 2 (A.23)
12 12
This formula was then substituted back into Equation A.15 to obtain the final equation for the
moment of inertia of the dumbbell, with respect to the COR (the elbow joint), as follows:
1 1
m1 3r1 2 + h1 2 + m1 d1 2 + m3 3r3 2 + h3 2 + mDb dDb 2 ,

JDb = 2 (A.24)
12 12
where mDb is the total mass of the dumbbell and dDb is the distance from the COR to the dumbbell
COM (previously solved for in Equation 7.6). Once evaluated, the values obtained from Equation
A.24 for the 3 lbs and 5 lbs dumbbell were then substituted into Equation 7.16 to obtain the total
load inertia values to use with the simulator.
Before this step could be performed however, one final assumption was needed to evaluate
Equation A.24. As it stands, there was no way to feasibly obtain the mass of the two dumbbell
segments (the heads, m1 , and the handlebar, m3 ). To be able to physically measure the mass
of these sections using a scale, the dumbbell would have to be cut into the respective portions,
wastefully destroying it. Instead, it was decided to approximate the mass of each segment by
scaling the total dumbbell mass (mDb ) by the percentage of total volume each segment occupied.
This method assumes that the materials/density of the dumbbell is uniform throughout the object,
implying that mass and volume scale consistently. Knowing that the volume of a cylinder is given
by [331]:
Vcylinder = πhr2 , (A.25)
then the total volume of the dumbbell (VDb ) can be found by adding the volume of each section
together, as follows:
VDb = V1 + V2 + V3 = 2V1 + V3 , (A.26)
where V1 and V2 are the volume of the dumbbell heads (which are assumed to be equal and can
therefore be combined) and V3 is the volume of the dumbbell handlebar. Using the assumption of
uniform density, the mass of each segment was then determined by multiplying the total mass by
the percentage of volume that segment occupies, as follows for both segments:
V1
m1 = mDb · , (A.27)
VDb
V3
m3 = mDb · . (A.28)
VDb
Once the mass of each segment was calculated using Equations A.27 and A.28, these values
were substituted into Equation A.24 to solve for the final dumbbell load inertia to be included in
Equation 7.16.
Appendix B
Permissions and Approvals
The following approvals are presented in this appendix:
1. Ethics approval from the Western University Health Science Research Ethics Board (HSREB)
for the recording of EEG and EMG signals during elbow flexion–extension motions performed
by human volunteers.
2. Permissions for reuse of the published book chapter and papers that constitute portions of
this thesis.
381
382
383
384
385
386
Curriculum Vitae
Jacob Tryon
Education
Sept. 2016 – PhD, Biomedical Engineering, Robotics, University of Western Ontario, Lon-
Present don, Ontario, Canada
Thesis: Evaluating EEG-EMG Fusion-Based Classification as a Method for
Improving Control of Wearable Devices for Upper-Limb Rehabilitation
Supervisor: Dr. Ana Luisa Trejos
Sept. 2011 – BESc, Mechatronic Systems Engineering with Professional Internship, Univer-
Apr. 2016 sity of Western Ontario, London, Ontario, Canada
Honours and Awards

Nov. 2017 WE Bots Battle Bot Fight Night 2017 - First Place
Jul. 2016 International Autonomous Robot Racing Competition 2016 - Third Place
Apr. 2016 Graduated with Distinction
2016 Dean’s Honour List
Jan. 2015 Western Engineering Competition Senior Design - Third Place
Jan. 2013 Charles Yip Memorial 125th Anniversary Alumni Award
Nov. 2012 WE Bots Walking Robot Competition 2012 - Second Place
2011 Western Scholarship of Excellence
Research Experience
Sept. 2016 – Graduate Research Assistant, University of Western Ontario, Wearable
Present Biomechatronics Laboratory (WearMe Lab)
Investigated EEG–EMG fusion techniques for controlling wearable robotic de-
vices, maintained lab 3D printer, worked with industry partner to design bio-
electrical signal measurement circuit.
Jun. 2016 – Undergraduate Research Assistant, University of Western Ontario, Ad-
Aug. 2016 vanced Robotics and Mechatronic Systems Laboratory
Designed and prototyped a circuit board to interface sensors in a Humanetics
crash test dummy with a National Instruments data acquisition system.
CURRICULUM VITAE 388
Industry Experience
May 2014 – Mechanical Engineering Intern, Trudell Medical International, London,
Aug. 2015 Ontario, Canada
Tested design prototypes to support product development, assisted in proto-
typing activities and maintained the 3D printer, completed project focused on
integrating new electronic sensing solutions into existing products.
Teaching Experience
Jan. 2022 – Course Instructor, University of Western Ontario, Department of Mechani-
May 2022 cal and Materials Engineering
MME 4452 Robotics and Manufacturing Engineering Automation.
Sep. 2016 – Graduate Teaching Assistant, University of Western Ontario, School of
Apr. 2021 Biomedical Engineering
MSE 4401 Robotic Manipulators, MSE 2202 Intro. to Mechatronic Design,
MSE 3302 Sensors and Actuators, MME 4487 Mechatronic System Design.
Aug. 2016 – Mechatronics Engineering Summer Academy Instructor, University of
Aug. 2019 Western Ontario, Western Engineering Summer Academy
Week long summer camp where high-school students solder, assemble, and
program a robot kit.
Aug. 2019 Biomedical Engineering Summer Academy Instructor, University of
Western Ontario, Western Engineering Summer Academy
Day long workshop where high-school students solder and assemble an EMG-
activated prosthetic actuated with a shape memory alloy.
Supervisory Experience
Sep. 2019 – Supervision of a high school co-op student project involving the development
Jan. 2020 of a wearable elbow motion tracking brace.
Sep. 2018 – Supervision of a high school co-op student project involving iterating the pro-
Aug. 2019 totype design of a wearable mechatronic brace.
Sep. 2018 – Supervision of an undergraduate student project involving the development of
Apr. 2019 an EEG/EMG controlled mechatronic brace for rehabilitation.
Feb. 2018 – Supervision of a high school co-op student project involving the selection and
Jun. 2018 validation of a wireless EEG data collection headset.
Sep. 2017 – Supervision of an undergraduate student project involving the design of an
Apr. 2018 EEG headset for brainwave interpretation.
Publications
J. Tryon and A. L. Trejos, “Evaluating convolutional neural networks as a method of
EEG–EMG fusion,” Frontiers in Neurorobotics, vol. 15, pp. 692183–1–20, 2021.
J. Tryon and A. L. Trejos, “Classification of task weight during dynamic motion using EEG–
EMG fusion,” IEEE Sensors Journal, vol. 21, no. 4, pp. 5012–5021, 2021.
T. Desplenter, J. Tryon, E. Farago, T. Stanbury, and A. Luisa Trejos, “Interpreting bio-

electrical signals for control of wearable mechatronic devices,” in Human–Robot Interaction:
Control, Analysis, and Design (D. Zhang and B. Wei, eds.), ch. 5, pp. 93–146, Cambridge
Scholars Publishing, 2020.
J. Tryon, E. Friedman, and A. L. Trejos, “Performance evaluation of EEG/EMG fusion meth-

ods for motion classification,” in IEEE International Conference on Rehabilitation Robotics,
(Toronto, Canada), pp. 971–976, Jun 24–28, 2019.
Presentations
J. Tryon, E. Friedman, and A. L. Trejos, “Performance evaluation of EEG/EMG fusion
methods for motion classification,” IEEE International Conference on Rehabilitation Robotics,
Podium Presentation, Toronto, Canada, Jun 24–28, 2019.
J. Tryon and A. L. Trejos, “Evaluating EEG/EMG fusion methods for upper-limb motion
classification,” London Health Research Day, Poster Presentation, London, Canada, Apr 30,
2019.
Volunteer Experience
Oct. 2018 – Lab Outreach Tour Demonstrations, Wearable Biomechatronics Labora-
Jan. 2020 tory (WearMe Lab), University of Western Ontario
Demonstrated EEG and EMG measurement systems during high-school class
tours and university open houses.
Mar. 2017 – Conference Paper Reviewer, Institute of Electrical and Electronics Engi-
Apr. 2019 neers (IEEE RAS and IEEE EMBS)
2019, 2017 International Conference on Rehabilitation Robotics, 2019 Interna-
tional Conference on Systems, Man, and, Cybernetics.
Nov. 2018 Engineering Program Graduate Panel Guest, Faculty of Engineering,
University of Western Ontario
Panellist discussing undergraduate engineering programs to first-year students.
Mar. 2017 – Field Supervisor (In Training), FIRST Robotics Competition, University
Apr. 2017 of Western Ontario
Assisted running weekend-long FIRST robotics competition. Helped assem-
ble/disassemble competition field, led team resetting the field between matches.
Sep. 2013 – Club President, WE Bots Robotics Club, Faculty of Engineering,cvb Uni-
Aug. 2016 versity of Western Ontario
Managed all club activities (weekly meetings, design competitions, social
events), developed beginner Arduino tutorials and robot kits, managed club
inventory and budget, attended relevant departmental meetings.
Sep. 2012 – Mechatronics Executive, WE FIRST Robotics Mentorship Club, Faculty
Mar. 2013 of Engineering, University of Western Ontario
Mentored high-school students during development of a robot for FIRST com-
petition, attended logistics meetings for planning/fundraising.
Professional Memberships
Jan. 2017 – Engineer in Training (EIT), Professional Engineers Ontario (PEO)
Present
Dec. 2016 – Graduate Student Member, Institute of Electrical and Electronics Engineers
Present (IEEE)
Extracurricular Memberships
Sep. 2016 – Graduate Student Advisor, WE Bots Robotics Club
Sep. 2019
Sep. 2013 – Club President, WE Bots Robotics Club
Aug. 2016
Sep. 2012 – Mechatronics Executive, WE FIRST Robotics Mentorship Club
Mar. 2013
Sep. 2011 – Member, WE Bots Robotics Club
Aug. 2013

Evaluating EEG_EMG Fusion for Wearable Upper-Limb Rehabilitation Robots

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Evaluating EEG_EMG Fusion for Wearable Upper-Limb Rehabilitation Robots

Uploaded by

Copyright:

Available Formats

Western University

Electronic Thesis and Dissertation Repository

Evaluating EEG–EMG Fusion-Based Classification as a Method for

Supervisor: Trejos, Ana Luisa, The University of Western Ontario

Follow this and additional works at: https://ir.lib.uwo.ca/etd

Chapter 2. Bioelectrical Signal Processing, Classification, and Fusion

Chapter 3. Comparing EEG–EMG Fusion Methods for Motion Classification

Paper: Performance Evaluation of EEG–EMG Fusion Methods for Motion Classification

Chapter 4. Task Weight Classification Using EEG–EMG Fusion

Chapter 5. Evaluating Convolutional Neural Networks for Input-Level Fusion

Paper: Evaluating Convolutional Neural Networks as a Method of EEG–EMG Fusion

Lay Summary iii

List of Figures xvi

List of Tables xxi

Nomenclature and Acronyms xxiii

2 Bioelectrical Signal Processing, Classification, and Fusion 10

3 Comparing EEG–EMG Fusion Methods for Motion Classification 61

3.2.2 Signal Acquisition and Processing . . . . . . . . . . . . . . . . . . . . . . . 64

4 Task Weight Classification Using EEG–EMG Fusion 76

5 Evaluating Convolutional Neural Networks for Input-Level Fusion 100

5.2.1 Data Collection and Signal Processing . . . . . . . . . . . . . . . . . . . . . 104

6 Investigating the Effect of Image Normalization on CNN-Based Fusion 134

6.2.4.2 Speed-Specific Accuracy . . . . . . . . . . . . . . . . . . . . . . . . 153

7 Development of a Task Weight Selective Control System 174

7.3.7 Upper-Limb Brace Performance Metrics . . . . . . . . . . . . . . . . . . . . 233

8 Conclusions and Future Work 320

A Additional Details for the Upper-Limb Brace Simulator Model 363

B Permissions and Approvals 381

Curriculum Vitae 387

1.1 Photographs of an example wearable upper-limb mechatronic brace. . . . . . . . . 3

2.1 Example of a recorded EMG Signal. . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.1 Concept for EEG–EMG fusion-based task weight classification. . . . . . . . . . . . 79

7.1 Previously developed wearable mechatronic upper-limb brace prototype. . . . . . . 183

2.1 EEG frequency bands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5.1 Literature summary of CNNs that use EEG or EMG. . . . . . . . . . . . . . . . . 103

A.1 Dumbbell dimension measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . 374

ADAM Adaptive Moment Estimation

ADC Analog-to-Digital Converter

AND Logical AND Operation

ANOVA Analysis of Variance

ADRC Active Disturbance Rejection Control

AR1 Autoregressive Coefficient One

AR2 Autoregressive Coefficient Two

AR3 Autoregressive Coefficient Three

AR4 Autoregressive Coefficient Four

BBP Beta Band Power

BCI Brain Computer Interface

BLDC Brushless Direct Current

CAD Computer-Aided Design

CNN Convolutional Neural Network

COM Centre of Mass

COR Centre of Rotation

CSP Common Spatial Pattern

DASDV Difference Absolute Standard Deviation Value

DOF Degree Of Freedom

ERD Event-Related Desynchronization

ERP Event-Related Potential

ERS Event-Related Synchronization

FFT Fast Fourier Transform

FMG Force Myography

fNIRS Functional Near-Infrared Spectroscopy

FPS Frames Per Second

HA Hjorth Parameter: Activity

HC Hjorth Parameter: Complexity