Toward Brain-Computer Interaction in Paralysis A New Approach Based On Visual Evoked Potentials and Depth-of-Field

SPRINGER BRIEFS IN
HUMANCOMPUTER INTERAC TION
Anibal Cotrina
Toward Brain–Computer
Interaction in Paralysis
A New Approach
Based on Visual Evoked
Potentials and
Depth-of-Field
Human–Computer Interaction Series
SpringerBriefs in Human-Computer Interaction
Series editors
Desney Tan, Redmond, USA
Jean Vanderdonckt, Louvain-La-Neuve, Belgium
SpringerBriefs in Human-Computer Interaction presents concise research within
the fast growing, multidisciplinary field of Human-Computer Interaction (HCI).
Designed to complement Springer’s prestigious Human-Computer Interaction
Series this Briefs series provides researchers with a forum to publish cutting-edge
scientific material relating to any emerging HCI research that is not yet mature
enough for a volume in the Human-Computer Interaction Series, but which has
evolved beyond the level of a journal or workshop paper.
SpringerBriefs in Human-Computer Interaction are shorter works of 50–125 pages
in length, allowing researchers to present focused case studies, summaries and
introductions to state-of-the-art research. They are subject to the same rigorous
reviewing processes applied to the Human-Computer Interaction Series but offer
exceptionally fast publication.
Topics covered may include but are not restricted to:
• User Experience and User Interaction Design

• Pervasive and Ubiquitous Computing
• Computer Supported Cooperative Work and Learning (CSCW/CSCL)
• Cultural Computing
• Computational Cognition
• Augmented and Virtual Reality
• End-User Development
• Multimodal Interfaces
• Interactive Surfaces and Devices
• Intelligent Environment Wearable Technology
SpringerBriefs are published as part of Springer’s eBook collection, with

millions of users worldwide and are available for individual print and electronic
purchase. Briefs are characterized by fast, global electronic distribution, standard
publishing contracts, easy-to-use manuscript preparation and formatting guidelines
and have expedited production schedules to help aid researchers disseminate their
research as quickly and efficiently as possible.
More information about this series at http://www.springer.com/series/15580

Anibal Cotrina
Toward Brain–Computer
Interaction in Paralysis
A New Approach Based on Visual Evoked
Potentials and Depth-of-Field
123
Anibal Cotrina
Department of Computer and Electronics
Universidade Federal do Espírito Santo
Sao Mateus, Espírito Santo
Brazil
ISSN 1571-5035
Human–Computer Interaction Series
ISBN 978-3-319-52297-5 ISBN 978-3-319-52298-2 (eBook)
DOI 10.1007/978-3-319-52298-2
Library of Congress Control Number: 2017932115
© The Author(s) 2017

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt from
the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, express or implied, with respect to the material contained herein or
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.
Printed on acid-free paper
This Springer imprint is published by Springer Nature

The registered company is Springer International Publishing AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
To people with disabilities
Foreword
When Dr. Cotrina started his doctoral studies in 2012 under my supervision, I was
not able to figure the such amount of publications we would have together.
The partnership with Dr. Cotrina resulted in 26 publications spread in journals,
conferences, and chapters of book. The final result of his research was the doctoral
thesis entitled “A Brain–computer Interface Based on Steady-state Visual Evoked
Potentials and Depth-of-Field”. And, as an award for his research, Dr. Cotrina got
the first place at the Brazilian BCI Competition 2014.
Thus, this book is a compilation of all experience of Dr. Cotrina in the area of
BCI (brain–computer interface), which include deep aspects of SSVEP
(Steady-State Visual Evoked Potential), Depth-of-field, and stimulation paradigms.
Then, I invite you to enjoy this book, learn about the representation of the visual
system on the retina and the visual cortex, know the typical visual evoked potential
waveforms, and the difference between covert and overt attention, in order to design
a gaze-independent SSVEP-BCI, whose main application is for paralyzed people.
Vitória, Espírito Santo, Brazil Teodiano Freire Bastos Filho

December 2016
vii
Preface
A brain–computer interface is a system that provides a direct connection between

human brain activity and computer-based devices. It generates an alternative
channel of communication between the human and his/her environment bypassing
the normal output pathways. The applications of BCIs are constantly increasing. It
includes communication systems, prosthetics control, rehabilitation, robotics, and
interaction of healthy people with their environment. Nowadays, one of the main
challenges of this technology is to provide practical and suitable applications for
patients with partial or complete paralysis caused by severe neuromuscular disor-
ders. SSVEP-based BCIs are becoming robust systems and are achieving high
performance; however, the traditional assessment is not suitable for employing
paralysis situation because it demands neck, head, or/and eye ball movements.
In order to take advantage of the potential of the SSVEP-BCI, this book intro-
duces and describes a novel approach of setting visual stimuli based on the optical
phenomenon of Depth-of-field. The intention is to offer to users the possibility of
selecting a SSVEP stimulus by shifting focus of the eye, instead of perform neck,
and/or eye ball movements. This approach was inspired in the photography when it
was noticed that any object of a scene can be highlighted by adjusting the focus
control of the camera; and not necessarily moving the camera. Then, if two visual
stimuli are present in the subject field of view, he/she can highlight one of them
(e consequently attenuate the other one) by adjusting the eye focusing. The
advantage of the human eye is that focusing is a reflex mechanism. This book is the
result of the studies and experiments conducted during my Doctoral and
Postdoctoral research at the Intelligent Automation Lab of the Federal University of
Espirito Santo. Its content aims to contribute with the development of technologies
of human–computer interaction in paralysis situation. It starts with a brief definition
of light and how it can modulate the brain signals, and ends showing practical
applications, such as the control of a telepresence robot.
The book is organized as follows; In Chap. 1, the problem is contextualized and
the purpose of the book is established. Also, a hypothesis is formulated. Chapter 2
provides a theoretical background; starting with the definition of the light and
ending with the presentation of a command of a brain–computer interface
ix
x Preface
originated by a light stimulus. Also, a literature review is dealt including recent

relevant works that addressed the application of SSVEP-BCI in patients with
paralysis. In Chap. 3, the new approach of setting the SSVEP stimuli based on
Depth-of-field phenomenon is presented. Concepts related to the optical phe-
nomenon of Depth-of-field are reviewed, such as defocusing and point spread
function. Also, physiological aspects such as the accommodation of the eye
mechanism and retinal blur model are addressed. Chapter 4 introduces a demod-
ulation method for extracting frequential features in the context of the SSVEP-BCI
based on the Depth-of-field. For this aim, the spatial-temporal spectral response
caused by a focused stimulus and a non-focused stimulus was studied. Some
experiments are conducted by evaluating and comparing the spectral power of
focused and non-focused stimuli to verify if the amplitude of the retinal response is
higher for the focused one than for the non-focused one. In Chap. 5, experiments
conducted for evaluating the detection of commands associated to focusing
mechanism are described. The evaluation is conducted in offline mode to find out
the possibilities of employing a SSVEP-BCI based on Depth-of-field in practical
situations. Conventional BCI performance metrics were employed. Chapter 6
presents and describes online practical applications of the SSVEP-BCI based on
Depth-of-field in communication and control. A speller system and a system for
controlling a telepresence robot were implemented. Pilots experiments conducted
for evaluating each system are described. Finally, vantages and limitations of the
proposal and its future directions in patients with paralysis are presented.
Sao Mateus, Brazil Anibal Cotrina

November 2016
Acknowledgements
I am grateful to all the people who supported, collaborated with, and contributed
directly and indirectly to the present work, especially to my doctoral supervisor
Prof. Dr. Teodiano Freire Bastos-Filho for his support and encouragement. Special
thanks to Dr. Alessandro Benevides and Dr. Javier Castillo who contributed with
their great understanding and insightful opinions to the main idea behind this work.
Many thanks to Berthil Borges and Alan Floriano who shared with me many long
days in the LAI lab and contributed directly with this work. Thanks to Prof Dr.
Andre Ferreira and Prof. Dr. Carlos Eduardo Pereira who together with my
supervisor accepted to work with me in the doctoral and postdoctoral research.
Also, I want to acknowledge the CAPES/Brazil agency which provided the the
scholarship and funds for this work and the CNPQ Brazilian Council, FINEP
Funding Brazilian Agency, and FAPES Brazilian Foundation for additional
financial support. This acknowledgment extends to the International Centre of
Theoretical Physics—South American Institute for Fundamental Research; the
Abdus Salam—International Centre of Theoretical Physics, the International Centre
for Genetic Engineering and Biotechnology, and the BCI Society for the grants to
attend international scientific events. My heartfelt thanks go to my parents Priscila
and Saul; to my brothers Saul, David, and Roy; and to my love, the sweet Jane.
Sao Mateus, Brazil Anibal Cotrina

November 2016
xi
Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Fundamentals: From Light to Command . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Light, Eye and Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Image Formation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 Field of Vision and Topographic Map . . . . . . . . . . . . . . . . . . . . . . . 13
2.4 Visual Pathway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.5 Brain Signals and Evoked Potentials . . . . . . . . . . . . . . . . . . . . . . . . 16
2.6 Influence of Refractive Error in VEP . . . . . . . . . . . . . . . . . . . . . . . . 19
2.7 Steady-State VEP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.8 BCI Based on SSVEP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.9 SSVEP-BCI and Gaze Dependence . . . . . . . . . . . . . . . . . . . . . . . . . 25
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3 The Novel Stimulation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1 SSVEP-BCI Stimulation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2 Non-focused Objects and Depth-of-Field . . . . . . . . . . . . . . . . . . . . . 36
3.3 Optical Point Spread Function (PSF) . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4 Retinal Blurry Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5 Accommodation of the Eye . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4 Frequency Demodulation for a SSVEP-BCI Based
on Depth-of-Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.1 Traditional Demodulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.2 Proposed Demodulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.3 Spatio-Temporal Retinal Response for a Flickering Stimulus. . . . . . 52
4.4 Power Spectrum of the Retinal Response. . . . . . . . . . . . . . . . . . . . . 55
xiii
xiv Contents
4.5 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5 Offline Evaluation of Command Detection . . . . . . . . . . . . . . . . . . . . . . 67
5.1 SSVEP Detection Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.2 Experimental Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.4 EOG Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6 Online Applications in Communication and Control . . . . . . . . . . . . . . 79
6.1 Binary Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.2 BCI for Communication—A Spelling System . . . . . . . . . . . . . . . . . 80
6.3 BCI for Control—Control of a Telepresence Robot . . . . . . . . . . . . . 84
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7 Future Directions in Patients with Paralysis . . . . . . . . . . . . . . . . . . . . . 91
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Acronyms
AAN American academy of neurology

ALS Amyotrophic lateral sclerosis
BCI Brain–computer interface
BPF Band-pass filter
CAR Common average reference
CCA Canonical correlation analysis
CNS Central nervous system
CRT Cathode ray tube
DFT Discrete Fourier transform
DMD Duchenne muscular dystrophy
ECoG Electrocorticographic
EEG Electroencephalographic
EMG Electromyography signals
EOG Electrooculography signals
ERD Events related to desynchronization
ERP Event-related potentials
ERS Events related to synchronization
FFT Fast Fourier transform
FIR Finite impulse response
fRMI Functional magnetic resonance imaging
GBS Guillain-Barre syndrome
HMI Human–machine interaction
HPF High-pass filter
ITR Information transfer rate
LASSO Least absolute shrinkage and selection operator
LCD Liquid crystal display
LED Light-emitting diode
LGN Lateral geniculate nucleus
LIS Locked-in state
LPF Low-pass filter
xv
xvi Acronyms
MEG Magnetoencephalographic
NPSF Neural point spread function
PSDA Power spectral density analysis]
PSD Power spectral density
PSF Point spread function
SCP Slow cortical potentials
SMR Sensorimotor rhythms
SNR Signal-noise rate
SSVEP-BCI BCI based on Steady-state visual evoked potentials
SSVEP Steady-state visual evoked potentials
TW Time window
VEP Visual evoked potentials
VE Virtual environment
Chapter 1
Introduction
Nowadays, the interaction between humans beings and machines, or Human-machine

interaction (HMI), has reached a high level of development, as can be observed in
devices with touchscreen interfaces or wireless keyboards. However, many times
this kind of technology is not suitable for people with physical handicaps, such as
quadriplegic individuals. In this cases, alternative interfaces could be proposed by
employing biological signals in order to provide an alternative way to externalize
the user intentions. Biological signals can be recorded by employing Electromyo-
graphy signals (EMG) or Electrooculography signals (EOG) devices, accelerome-
ters, gyroscopes, or eye-tracking devices. For instance, a system with eye-tracking
can be used to control devices, such as robotic wheelchair. But, what about peo-
ple with severe paralysis who cannot control voluntarily their muscles? In recent
years, neuro-degenerative diseases such as Amyotrophic lateral sclerosis (ALS) [1],
Duchenne muscular dystrophy (DMD) [2] or Guillain-Barre syndrome (GBS) [3, 4]
have become relevant. Patients with these diseases loose progressively the control
of their muscles ending in a situation called Locked-in state (LIS), in which people
are almost paralyzed and only conserve a residual control of few muscles. It is a
condition in which patients are aware but cannot move or communicate verbally due
to the paralysis of nearly all voluntary muscles of the body. The ALS is the most
known of these diseases because in 2014, the Ice Bucket Challenge1 that promote
the awareness of this disease, received media attention around the world. It is a pro-
gressive neurodegenerative disease that affects nerve cells in the brain and the spinal
cord. Another disease that has captured the attention of the media is the GBS because
in late 2015 by its possible relation with the Zika virus.2 Notwithstanding, LIS also
might be a result from traumatic brain-injury, hypoxia, stroke, encephalitis, or a brain
tumor [4].
1 Fordetails, please refer to www.alsa.org.

2 In2015, several cases of patients presenting Zika virus was detected in Brazil [5] and an unusual
increase of GBS also was reported in the Northeast of the country; possible associations began to
be investigated (www.who.int).
© The Author(s) 2017 1

A. Cotrina, Toward Brain–Computer Interaction in Paralysis,
SpringerBriefs in Human–Computer Interaction,
DOI 10.1007/978-3-319-52298-2_1
2 1 Introduction
A Brain-computer interface (BCI) is a system that provides an alternative chan-

nel of communication between brain signals and a computer that does not involve
traditional ways as muscle and nerves. It can be used by healthy people or patients
who retain some voluntary muscle control in rehabilitation to command a robotic
wheelchair or a prosthesis. In people with reduced mobility in LIS, BCIs became
a good alternative for communication with their environment or with other people,
such as a nurse or a therapist. Patients could use these systems to express their basic
needs such as thirst, hunger or pain, even for expressing their preferences in terms
of treatment planning, like pain management or end-of-life decision-making [6, 7]
The most common non-invasive method employed in BCI is based on Electroen-
cephalographic (EEG) signals. In 2002, a first patient in LIS regains some measure
of communication through EEG-based BCI, developed by neurological researcher
Niels Birbaumer [8, 9] conducted an analysis of EEG-based BCI for people with
reduced mobility. Also, [10] performed a meta-analysis about BCI in ALS. EEG
based BCIs have attracted the interest of Brazilian researcher groups. For instance,
[11] proposes a BCI to command a robotic wheelchair, [12] developed a BCI based
on Steady-state visual evoked potentials SSVEP with high performance that also
was tested with a robotic wheelchair, [8] proposed a BCI architecture based on
motor mental tasks, [13, 14] evaluated BCIs based on emotional components and,
in [15–17], some preliminary results of the present assessment were reported.
Problem Statement
Currently, SSVEP-BCI systems are presenting good performance, because they
achieve high accuracy rates in the detection of brain activity associated to a BCI
command. In traditional SSVEP-BCI systems, a set of stimuli are presented to users
and they are asked to gaze on one stimulus (Fig. 1.1a). Then, analyzing their EEG
signals, systems detect which stimulus the subjects is gazing on. Finally, a command
that is associated with a target stimulus is sent to computers. However, as shown in
Fig. 1.1b, in SSVEP-BCI systems, users must perform movements redirecting their
neck, head or eyeball to chose another stimulus. When users redirect their gazing, a
(a) (b) f1 f2
Amplitude
f1
f2 d0
f1 f2
d0 d0
Frequency
Fig. 1.1 Setup for conventional SSVEP-BCI system. a A subject with two SSVEP stimuli placed at
the same distance d0 and flickering at frequencies f 1 and f 2 . b Representation of subject and stimuli,
in which dotted lines are the user’s field of view. Frequency versus amplitude display represents the
frequency of the attended stimulus that can be extracted from EEG signals
1 Introduction 3
target stimulus is brought to the center of the field of view and the non-target stimulus
is placed outside the field of view. The main advantage of SSVEP-BCI systems is
given by their high signal-noise rate (SNR) that is attained when the target stimu-
lus is in the center of this view. Paradoxically, it main disadvantage is given by the
dependence of muscular movements. It makes that traditional SSVEP-BCI systems
are not suitable for patients with paralysis. To overcome this muscle dependence,
independent SSVEP-BCI have been proposed, in which the user selects one out of
several superimposed stimuli or stimuli close to each other by selective attention
without redirecting their gaze. This kind of system represents a good option for par-
alyzed people who cannot control with precision their gaze in order to select a target
by performing an exhaustive attention activity instead of voluntary muscular activ-
ity. But this time, the disadvantage is given by the low accuracy rates achieved by
attention-based SSVEP-BCI systems. Also, previous training stages are necessary
because subjects must learn to maintain their covert attention on the target stimulus
that is placed next to the non-attended one, and at the same distance.
Purpose of this Book
In this book, a novel stimulation setting for SSVEP-BCI is described in which two
stimuli are presented together in the center of the user’s field of view but at different
distances from him/her, as shown in Fig. 1.2a. It makes for that if one stimulus is
focused on, the other one is non-focused on, and viceversa.
Considering that (i) when an object is being focused on by the eye, objects behind
and in front of the point of focus are blurred, and the zone where objects are judged
to be in focus is referred to as the phenomenon of Depth-of-field [18, 19]; (ii) the
human eye is able to refocus and sharpen the image in a certain range when the
distance from the object changes, and the accommodation is the mechanism of the
eye to adjust its focal length to focus on objects at different distances [20, 21]; and
(iii) the amplitude of the response in a pattern of visual evoked potentials (VEP) is
dependent on the degree of retinal image focus [22, 23]; then, the novel setup here
(a) (b) f2
d2
f1
Amplitude
f2 d1
f1
f1 f2
d1 d2 Frequency
Fig. 1.2 Setup for the novel SSVEP-BCI paradigm. a A subject with two SSVEP stimuli placed at
the same direction but at diferents distanced (d1 and d2 ) and flickering at frequencies f 1 and f 2 . b
Representation of subject and stimuli, in which dotted lines are the user’s field of view. Frequency
versus amplitude display represents the frequency of the attended stimulus that can be extracted
from EEG signals
4 1 Introduction
described allows users to make a selection by a refocusing mechanism that in turn

does not demand neck, head and/or eyeball movements. A high accuracy rate will
be achieved if the amplitude of the response of focused stimulus due to it is higher
than the amplitude of the response of the non-focused one.
Hence, the following hypothesis can be formulated;
If two stimuli are presented simultaneously in the center of the field of view
of the BCI user, flickering at different frequencies and located at different
distances from the user, then, the focused stimulus will be able to elicit a
distinguishable SSVEP pattern regardless of the presence of the non-focused
stimulus.
Objective
Based on the hypothesis above, this book aims to demonstrate the feasibility of
developing a system for human computer interaction based on SSVEP-BCI and
optical phenomenon of Depth-of-field.
In this sense, an objective is to build and describe the SSVEP-BCI stimulation
setup that allows subjects to attend a target stimulus by focusing mechanism. The
most important aspect of this setup is founded in the distances of stimuli location,
because in a scene the blurriness degree of a non-focused object depends on the
Depth-of-field of a focused object, which in turn depends on the distances between
focused and non-focused objects.
Another objective is to propose a demodulation process suitable for this assess-
ment. It is explored the fact that the amplitude or latency of evoked responses due
to the fact that a light stimulus are affected when there is any refractive error. There
are differences in the projected images onto the retina of focused and non-focused
objects. For instance, in conventional SSVEP-BCI systems, researchers avoid the
presence of refractive errors asking users to move their head, neck, or eyeballs to
maintain the stimulus in focus. In this book, the refractive error is exploited rather
than avoided.
The evaluation of the SSVEP-BCI that employs the stimulation setup based on
Depth-of-field (or SSVEP-BCI based on Depth-of-field) is also contemplated within
the objectives of the book. It is performed in two ways; (i) in offline mode, that
employs common BCI performance metrics, and (ii) in online mode, that address
the use of the proposal in practical applications for communication and control.
References
1. Huggins J, Wren P, Gruis K (2011) What would brain-computer interface users want? opinions
and priorities of potential users with amyotrophic lateral sclerosis. Amyotroph Lateral Scler
12(5):318–4
References 5
2. Bushby K, Finkel R, Birnkrant DJ, Case LE, Clemens PR, Cripe L, Kaul A, Kinnett K, McDon-
ald C, Pandya S, Poysky J, Shapiro F, Tomezsko J, Constantin C (2010) Diagnosis and manage-
ment of duchenne muscular dystrophy, part 1: diagnosis, and pharmacological and psychosocial
management. Lancet Neurol 9(1):77–93
3. Birbaumer N (2005) Breaking the silence: braincomputer interfaces (bci) for communication
and motor control. Psychophy 43(6):517–32
4. Nijboer F, Broermann U (2010) Braincomputer interfaces for communication and control in
locked-in patients. In: Graimann B, Allison B, Pfurtscheller G (eds) Brain-computer interfaces,
revolutionizing humancomputer interaction. Springer, Mineapolis, pp 186–201
5. Zanluca C, Melo VCAd, Mosimann ALP, Santos GIVd, Santos CNDd, Luz K (2015) First
report of autochthonous transmission of zika virus in brazil. Mem Inst Instituto Oswaldo Cruz
110:569–72
6. Laureys S (2005) Death, unconsciousness and the brain. Nat Rev Neurosci 6:899–909
7. Gantner I, Bodart O, Laureys S, Demertzi A (2013) Our rapidly changing understanding of acute
and chronic disorders of consciousness: challenges for neurologists. Futur Neurol 8(1):43–54
8. Benevides A (2013) A brain-computer interface architecture based on motor mental tasks and
music imagery. Federal University of Espirto Santo—Ph.D Thesis, Vitoria ES, Brazil
9. Riccio A, Mattia D, Simione L, Olivetti M, Cincotti F (2012) Eye-gaze independent eeg-based
brain-computer interfaces for communication. J Neural Eng 9(4):1–15
10. Marchetti M, Priftis K (2014) Braincomputer interfaces in amyotrophic lateral sclerosis: a
metanalysis. Clin Neurophysiol 126(6):1255–63
11. Ferreira A (2008) Uma proposta de interface crebro-computador para comando de cadeiras de
rodas. Federal University of Espirto Santo—Ph.D Thesis, Vitoria ES, Brazil
12. Muller S (2010) Interface Cerebro-Computador Baseada em Potenciais Evocados Visuais em
Regime Permanente para Comando de uma Cadeira de Rodas Robotica. Federal University of
Espirto Santo—Ph.D Thesis, Vitoria ES, Brazil
13. Bastos T, Ferreira A, Cotrina A, Arjunan S, Kumar D (2012) Evaluation of feature extraction
techniques in emotional state recognition. In: 4th international conference on intelligent human
computer interaction (IHCI), pp 1–6. IEEE, Kharagpur, India
14. Cotrina A, Benevides AB, Ferreira A, Bastos T (2013) Eeg evaluation during emotional state
elicited by unpleasant sounds to be applied in BCI. In: Proceedings of the 5th international
brain-computer interface meeting, pp 166–167. Asilomar CA, USA
15. Cotrina A, Bastos T, Ferreira A, Benevides A, Castillo J, Rojas D, Benevides A (2014) Towards
a SSVEP-BCI based on depth of field. In: Proceedings of the 6th international BCI conference,
pp 1–6. TU Graz, Graz, Austria
16. Cotrina A, Benevides AB, Castillo J, Ferreira A, Bastos T (2015) Statistical evaluation of a
novel ssvep-bci stimulation setup based on depth-of-field. Res Biomed Eng 31(4):295–306
17. Cotrina A, Benevides A, Pereira CE, Bastos T (2016) Towards a BCI based on vergence eye
movements. In: Proceedings of the 6th international brain-computer interface meeting, p 200.
Asilomar CA, USA
18. Howard I (2012) Perceiving in depth: volume 1 basic mechanisms. Oxford University Press,
New York
19. Pentland AP (1987) A new sense for depth of field. IEEE Trans Pattern Anal Mach Intell
9(4):523–31
20. Ebenholtz SM (2001) Oculomotor systems and perception. Cambridge University Press, Cam-
bridge
21. Gregory RL (1997) Eye and brain, the psychology of seeing, 5th edn. Princeton University
Press, New Jersey
22. Songnian Z, Qi Z, Chang L, Xuemin L, Shousi S, Jun Q (2014) The representation of visual
depth perception based on the plenoptic function in the retina and its neural computation in
visual cortex v1. BMC Neurosci 15(1):1–17
23. Sokol S, Moskowitz A (1981) Effect of retinal blur on the peak latency of the pattern evoked
potential. Vis Res 21(8):1279–86
Chapter 2
Fundamentals: From Light to Command
In this chapter, a theoretical background is provided. It starts by defining the light and
ends by presenting a BCI command due to the light stimulus. Figure 2.1 illustrates
the “path of the light” that is emitted by a visual stimulus and sensed by the human
visual system [1]. The visual information is carried to the brain visual cortex in
which the signal evoked by the object can be measured. A BCI takes the measured
signal and translates it into a command. In this sense, basic concepts about light,
eye and brain are briefly presented. Next, the image formation, the visual pathway,
the topographic representation of the image and the response of visual cortex are
described. Then, EEG and VEP are presented. Transient VEP, steady-state VEP and
how the refraction errors affect the VEP are discussed. Also, SSVEP-BCI and how
the light is translated into commands, are addressed. This theoretical background
Electrode
Visual
stimulus LGN
Scalp
Optic
Object tract
Visual field
Fig. 2.1 Representation of the visual system, together with a stimulus, visual field and single cell
electrode
DOI 10.1007/978-3-319-52298-2_2
8 2 Fundamentals: From Light to Command
is principally based on well-know medical physiology, optics, neural science and

human eye books [1–6]. Finally, a literature review of the more relevant works in the
gaze-independent SSVEP-BCI research is performed.
2.1 Light, Eye and Vision
Light is a form of electromagnetic radiation emitted by the oscillation of materials

electrically charged. Light travels in straight lines unless it encounters an object that
causes it to be reflected or refracted when it turns back in the opposite direction or it
bends traveling at an angle relative to the original path, respectively. Electromagnetic
oscillations have a regular sinusoidal pattern that can be characterized in terms of its
wavelength that is perceived as a color. Figure 2.2 shows the electromagnetic spec-
trum. Visible light includes electromagnetic waves that have wavelengths between
450 and 750 nm, approximately; in which different colors correspond to different
wavelengths within this range [3, 7]. All objects reflect light to different degrees and
the luminance determine their relative contrasts. Vision is based primarily on the
perception of bright-dark contrasts and color contrast enables complex organisms to
distinguish surfaces if they reflect different portions of the visual spectrum.
The eye is a complex optic system in the human body that contains about 125
million neurons specialized to turn light into electrical signals called photoreceptors
[2]. A schematic diagram of a horizontal, sectional view through a human eye is
shown in Fig. 2.3. The cornea and the sclera are the transparent membrane over the
front of the eye and the white membrane around the sides and back of the eyeball,
respectively. The iris, which is the colored part of the eye, controls the aperture of
the pupil regulating the amount of light entering the eye. The pupil is the aperture at
the center of the iris, through which light enters the eye. The crystalline lens of the
eye or lens is a transparent and flexible structure; by changing its curvature through
the contraction or relaxation of the intrinsic muscles of the eye, light coming from
different sources is projected on the back of the eye.
Vision is the faculty or state of being able to see. In the human eye, the light enters
through the cornea and then passes through the pupil. Gazed objects are projected
onto the retinal surface that acts like a movie screen of the eye. Images are projected
sharp when lens focuses on the object. Any image formed on the retina should
cause nerves to fire sending a signal along the optic nerve to be “seen” by the brain.
The retina is the innermost layer of the eye whose function is phototransduction,
converting the light energy into the electrical energy. The retinal surface consists of
neural tissue that contains the photoreceptors, which are the cells that detect the light
waves. Photoreceptors that detect dim light and bright light are named as rods and
2.1 Light, Eye and Vision 9
Frequency (Hz) Wavelength ( )

Gamma-rays 0.01 nm
10 19
0.1 nm
10 18 Blue
400 nm
X-rays
1 nm
10 17
10 nm
10 16
Ultraviolet ~490 nm
100 nm
10 15 Green
Human vision 1000 nm
Near IR 1 μm
10 14 ~570 nm
Infra-red 10 μm
10 13
Thermal IR 100 μm
700 nm
10 12 Red
Far IR
1000 μm
1 mm
10 11
Microwaves 1 cm
10
Frequency (Hz) Wavelength ( )
Fig. 2.2 The electromagnetic spectrum. The numbers indicate wavelength in nanometers (1 nm =
1 × 10−9 m). The band between 400 and 700 nm of visible light is highlighted. It was consider
green color has primarily wavelength in the 500–570 nm range
cones, respectively. The optic nerve consists of the axons of neurons in the retina; it
transmits information from the retina to the brain. The fovea is the central region on
the retina, in which light from the center of the visual field strikes. It is the area of
the retina with the greatest visual acuity. The optic disk is the portion of the retina
where the optic nerve passes through the retina.
Cilliar muscles
Lens
Retina
Pupil
Fovea
Object
Image
Iris
Cornea
Optical nerve
Blind spot
Fig. 2.3 Schematic diagram of the human eye. The image of the external object passes through the
focusing system composed of the lens and the cilliar muscles. The image is projected on the fovea
at the center of the retina; note that it becomes inverted n doing so. Due to the optical nerve, the
blind spot is the region of the retina without photoreceptors
2.2 Image Formation
Light is composed of divergent waves that propagate in all directions from every
point of a visible object. Before a sharp image of the object can be projected onto the
retina, light must be focused on by the human optical system; then, the light of the
projected image reaches light-sensitive receptor cells of the retina; next, signals of
receptors arrive at the brain through the visual pathway. Responses measured at the
visual cortex are related to the topographic association between retinal and cortex
fields [1].
Focusing the Light
The lens is the primary structure responsible for making adjustments for viewing
objects at various distances. The mechanism to adjust the focus of the eye so that we
can concentrate the human attention on an object of interest by altering the shape
of the lens is called accommodation. Depending on the distance of the object, small
muscles attached to the lens, contract or relax, changing its curvature (Fig. 2.4).
These muscles are named ciliary muscles. The amount of light entering the eye is
controlled by the iris and the pupil. Stimulation of sympathetic nerves to the iris
causes these muscles to contract, which then enlarges the pupil, whereas stimulation
of the parasympathetic nerves causes the diameter of the iris to get smaller [4].
2.2 Image Formation 11
Ciliary muscles
Retinal Flattened for weak refraction

image
of far
object
Lens Near object Far object
Retinal Rounded for strong refraction

image
of near
object
Fig. 2.4 Focusing light from distant and near sources. a A relatively flat (weak) lens is sufficient
to converge the light waves reflected from a distant object on the retina. b A rounder (strong) lens
is needed to converge the light waves reflected from a near object on the retina
Imaging the Object

The photoreceptors cones are the cells in which transduction takes place, converting
light energy to electrochemical energy that results in patterns of action potentials
in the optic nerve. In human eyes, two different types of photoreceptors commonly
called rods and cones can be found. Cones are concentrated in the fovea, dispersed
retinal pathways and have high acuity in bright light [1]. It makes the fovea essential
for daytime vision. This is also the region of the retina where the image of an object
of primary interest is being projected. Light-sensitive molecules are activated by
specific ranges of wavelengths in the cones. Rods are not present in the fovea and
are designed to provide some vision in dim light. Light molecules are activated by
a broad range of wavelengths in the rods. Relative distribution of cones and roods
are illustrated in Fig. 2.5. The fovea is the region of the retina with a high density of
photoreceptors that measures about 1.2 mm in diameter. The fovea is at the center of
the inner ring of left and right insets of Fig. 2.5. In this region, cone density increases
almost 200-fold, reaching, at its center, the highest receptor packing density found
-90o 90o
-45o 45o
0o
Fovea Optic disk
Number of photoreceptors
Rods
Cones
-90o -45o 0o 45o 90o

Position in retina (relative to fovea)
Fig. 2.5 Relative distribution of the cones and rods on the retina. The y-axis is the receptor density
and the x-axis is the relative distance from the fovea. Note that the highest density of cone receptors is
located in the fovea and there are no receptors where the optic nerve leaves the eyeball, thus creating
a blind spot. The peripheral vision is primarily due to rods, hence we have minimal abilities to detect
colors in those areas
anywhere in the retina [8]. This high density is achieved by decreasing the diameter
of the cone outer segments such that foveal cones resemble rods in their appearance.
The increased density of cones in the fovea is accompanied by a sharp decline in
the density of rods [1, 8], as illustrated by the contour lines of Fig. 2.6. In this book,
cones are referred as photoreceptors.
2.3 Field of Vision and Topographic Map 13
(a) (b)
~200
> 16
~100
>4
<4
>2
~0
Optic disc Optic disc
20 20
40 40
60 60
Scale: 1000 x cones/mm2 Scale: 1000 x rods/mm2
Fig. 2.6 Contour curves of topographic maps of cones (Left) and rods (Right) density on the retinal
surface. The density of cones on the inner circle of high (exceeds 16000 photoreceptors/mm2 ). The
density of rods is very low on the inner circle. The rings are spaced at intervals of about 20o . The
fovea is at the center of the inner ring
2.3 Field of Vision and Topographic Map
Field of Vision
In Fig. 2.7, the field of vision and its perimeter of the left eye is represented [2]. Field
of vision corresponds to the visual area seen by the eye at a given instant. It is plotted
when the eye is looking toward a central spot directly in front of the eye. Numbers
are in degrees, and the eccentricity angle is the distance by which a target is displaced
from the fovea. A blind spot caused by lack of rods and cones in the retina over the
optic disc is found about 15◦ lateral to the central point of vision.
Topographic Map
The representation of different points in the visual field across a population of cortex
neurons is called a topographic representation or topographic map. As shown in
Fig. 2.8, areas in the primary visual cortex are designated to specific parts of the visual
field, as indicated by the corresponding numbers [1]. Beginning with the ganglion
cells, each level of the visual system projects to the next level in an organized way
so that the map of visual space on the retina is preserved. The area at the center of
the visual field (areas 1–4) that corresponds to the fovea is expanded in the cortex so
that it occupies about half of the entire cortical representation.
2.4 Visual Pathway
The visual pathway is a pathway over which a visual sensation is transmitted from the
eyes to the brain [1]. As illustrated in Fig. 2.1, the pathway starts in a receptive field of
a cells and can be recorded in a single cell of cortex [9]. The visual pathway includes
Left 90 Right
80
70
135 45
60
50
40
30
20
10
180 0
Optic disc
225 315
270
Fig. 2.7 Perimeter chart showing the field of vision for the left eye. White and gray regions indicate
where the light or object can be seen or it cannot, respectively
the retina, optic nerve, the lateral geniculate nucleus (LGN) and visual cortex, in
which (i) the optic nerve is formed by the axons of ganglion cells that are the output
neurons from the retina, which generate action potentials that are transmitted to the
central nervous system (CNS); (ii) LGN that acts as a sensory relay transmitting
information captured by the retina to the visual cortex is composed of six layers;
layers 1 and 2 are called the magnocellular layers, while layers 3, 4, 5 and 6 are
called parvocellular layers; and (iii) the primary visual area of the cerebral cortex,
which is known as striated cortex or cortex V1, is the first stage of cortical processing
of visual information. Cortex V1 contains a complete map of the visual field covered
by the eyes. It receives its main visual input from the LGN and sends its main output
to subsequent cortical visual areas. Cortex V1 is traditionally divided in 6 horizontal
layers, with a characteristic distribution of inputs and outputs across layers. Inputs
from LGN arrive at layer 4. For instance, this layer is divided into sublayers 4A, 4B,
4Cα, and 4Cβ. The main LGN inputs arrive in 4C, magnocellular cells to 4Cα and
parvocellular cells to 4Cβ.
In summary, the visual system possesses parallel processing, in which segregated
pathways transmit different attributes of a stimulus, for example bright and contrast
sensations have different pathways. Figure 2.9 show the Parvocellular (or P-pathway)
2.4 Visual Pathway 15
Left Primary Right

visual
cortex
8 4 2 6
7 5
3 1
Calcarine
fissure
5 7
1 3
2 4
Fovea
6 8
Visual field
Fig. 2.8 Representation of the visual field in the fovea and in the primary visual cortex. This
representation is not proportional, for the neural information obtained by the receptors within the
fovea projects onto a large portion of the visual cortex
and Magnocellular (or M-pathway) pathways. M-pathway has high contrast sensi-
tivity and does not perceive color. P-pathway is color-sensitive and has low contrast
sensitivity. M-pathway (dark gray) starts at the large ganglion cells, projects first
into magnocellular layers of LGN and then into layer 4Cα primary visual cortex.
P-pathway (light gray) starts at small ganglion cells, projects first into parvocellular
layers of LGN and then into layer 4Cβ primary visual cortex [1, 2, 9].
Response at the Visual Cortex
In general, the visual system processes visual features such as motion, color, form,
and depth separately and carries this information in parallel neural pathways [1].
Although information is projected to superior visual, temporal and parietal areas,
activity of Cortex V1 can be measured with a non-invasive EEG electrode. Figure 2.10
shows a simplified flowchart of parallel processing for contrast and luminance infor-
mation, in which the skull effect that is represented by a low-pass filter (LPF) [10] acts
over responses of both layers (4Cα and 4Cβ). EEG signal in visual cortex denoted
by s(t) can be intended as the sum of the response due to parallel pathways with
spontaneous EEG denoted as ξ [11].
2.5 Brain Signals and Evoked Potentials
Electroencephalographic Signals
EEG signals are the neuro-physiologic measurements of the electrical activity of the
brain using electrodes placed on the scalp. The resulting traces are known as the
EEG waves and they represent the electrical activity of a large number of neurons.
The capture of EEG is a non-invasive procedure that reads scalp electrical activity
generated by brain structures and frequently used for diagnostic purpose. The elec-
troencephalographic traces, as shown in Fig. 2.11(a), are defined as electrical activity
recorded from the scalp surface after being picked up by metal electrodes and con-
ductive medium. Only large populations of active neurons can generate electrical
activity recordable on the head surface. Between electrode and neuronal layers, cur-
rent penetrates through skin, skull and several other layers. Weak electrical signals
detected by the scalp electrodes are massively amplified, and then displayed on paper
or stored in computer memory. Due to the capability to reflect both the normal and
abnormal electrical activity of the brain, EEG has been found to be a very power-
ful tool in the field of neurology and clinical neurophysiology. Unfortunately, the
EEG also reflects activation of the head musculature, eye movements, interference
from nearby electric devices, and changing conductivity in the electrodes due to the
movements of the subject or physicochemical reactions at the electrode sites. EEG
corrupted by other signals are called artifacts [12].
To superior brain regions

(MT, V2, V4 or V5 areas)
Skull
LGN
1
EEG electrode
Form and Color 2 &3

4A
Magnocellular Movement 4B
Parvocellular
4C
4C
Retinal ganglion cells Primary Visual Cortex
Fig. 2.9 Functional representation of visual pathways. The magnocellular (dark green) pathway
starts at the ganglion cells, and arrives at the layer 4Cα of cortex V1 after passing through the layers
1 and 2 of LGN. The parvocellular (light green) pathway starts at the ganglion cells, passes through
the layers 3–6 of LGN and arrives at the layer 4Cβ of cortex V1. Then, the visual information flows
to superior brain regions such as V2, V4 or V5; also called MT
2.5 Brain Signals and Evoked Potentials 17
The internationally standardized 10/20 system is usually employed to record the

spontaneous EEG, in which the electrode locations are determined by dividing the
perimeter into 10% and 20% intervals. An alternative to the 10/20 system is the 10/10
system characterized by intervals of 10% that provides a higher channel density.
[13] describe standardization of electrode locations and nomenclature, and evaluate
both position systems. Figure 2.11(b) shows electrode positions according to the
American Electroencephalographic Society. “The electrodes are named by a capital
letter corresponding to the initial of the brain lobe where they are placed (“F”, “C”,
“P”, “O” and “T” for Frontal, Central, Parietal, Occipital and Temporal, respectively),
followed by an even number for the right hemisphere and an odd number for the left
hemisphere. The letter “A” is used for electrodes placed in the ear. For the electrodes
placed in the frontal lobe, near the nasion, the letter “p” is added (Fp = Frontal pole).
For the electrodes in the line connecting the nasion to the inion, the letter “z” is
added”.
Visual Evoked Potentials
An evoked potential is the electrical response recorded from the human nervous
system following presentation of a stimulus that can be detected by EEG and EMG
devices. VEP refer to electrical potentials, initiated by brief visual stimuli, which are
recorded from the scalp overlying the visual cortex [14]. VEP occurs when a subject
observes a visual stimulus, such as a flash of light or a pattern on a monitor. VEP
are used primarily to measure the functional integrity of the visual pathways from
Light Ganglion cells LGN layers Visual Cortex Low-pass Scalp

filter effect signals
Parvo
Bright cellular 4C
P cells
layers layer
s(t)
Magno
4C
Contrast M cells cellular
layer
layers
Other
brain
areas
Object Eye response LGN responses Cortical responses Skull effect
Fig. 2.10 Simplified flowchart of parallel visual pathways. LGN, lateral geniculate nucleus; M,
magnocellular; P, parvocellular. Parvo pathway is composed by P-cell, Parvocellular layers of LGN,
and 4Cα layer of visual cortex. Magno pathway is composed by M-cell, PMagnocellular layers of
LGN, and 4Cβ layer of visual cortex. LPF and ξ represent the effect of the skull and the spontaneous
EEG, respectively
retina via the optic nerves to the visual cortex of the brain [15]. Their waveforms
are usually extracted from the EEG signals by averaging. Peak time is measured
from stimulus onset to the maximum deflection, positive or negative. Latency is the
term employed to indicate the time between stimulus onset and largest amplitude,
for positive or negative deflections. The peak amplitude indicates the integrity of the
neural structures including axons conducting information along the visual pathway
and the latency is related to the time the electrical signal takes to travel from the
retina to the visual cortex. The combination of amplitude and latency is helpful in
determining the health of the visual pathway [16, 17].
Figure 2.12 shows the VEP waveforms for pattern-reversal, pattern onset/offset
and flash stimulation. In the pattern-reversal stimulus, black and white checks change
abruptly and repeatedly to white and black checks, respectively. Due to the fact that
the stimulus is presented as a kind of checkerboard with equal numbers of black
and white checks, there is not significant variation in the luminance. In the pattern
onset/offset, the checkerboard changes abruptly with a diffuse gray background.
Background has the same luminance of the stimulus. In both patterns, size of checks,
stimulation frequency (in reversals per second) and the number of reversals, the mean
of luminance, pattern contrast and field size of the stimulus can be adjusted [16]. In
flash stimulation, it can be used flashing screens, stroboscopic lights, or light-based
portable stimuli. Also, it can be used flickering images in a computer screen, or
flickering light mounted on goggles [18]. In that case, the VEP is elicited by a brief
flash presented in a fairly illuminated environment.
To plot a well-traced curve, a number of visual responses are obtained by repeating
the visual stimulation and them averaging them. Waveforms that allows to measure
latency times and amplitude peaks of the responses are called transient responses of
the visual system. When stimulation is presented periodically, low repetition rate is
required, not faster than two stimuli per second, in order to sensory pathway recovers
(a) Closed eyes (b) Nose

P03 Nz
P1
Fpz
PO7 Fp1 Fp2
P3 AF7
AF3 AFz AF4
AF8
O1 F8
F7
PO8 F5 F3 F1 Fz F2 F4 F6
P2 FT7 FC8
FC5 FC3 FC4 FC6
PO4 FC1 FCz FC2
P4 A1 T7 C5 C3 C1 Cz C2 C4 C6 T8 A2
O2
POZ TP7 CP5 CP3 CP1 CPz CP2 CP4 CP6 TP8
CZ
PZ P7
P5 P3 P1 Pz P2 P4 P6
P8
OZ
A1 Left PO7
PO3 POz PO4
PO8 Right
A2 ear O1
Oz
O2 ear
Iz
Fig. 2.11 a Example of EEG signals of a set of channels with bi-auricular reference and grounded at
forehead. Signals of high amplitude and low frequency are signals recorded when the user was asked
to close his eyes. b Location and nomenclature of the intermediate 10% electrodes, as standardized
by the American Electroencephalographic Society
2.5 Brain Signals and Evoked Potentials 19
Fig. 2.12 Typical VEP P100

10 V-
waveforms for
pattern-reversal (top), pattern
onset/offset (middle) and N75
flash stimulation (bottom). In 0- N135
pattern-reversal responses,
there is two positive peaks at
50 and 100 ms, and a
negative peak at 75 ms (P50, -10-
P100 and N75, respectively). 0 100 300ms
Responses shows little
variation between subjects 10-
Onset/offset pattern response
consists of peaks at 75, 125 C2
and 150 ms (C1, C5 AND 0- C1 C3
~125
C3, respectively). It show ~75 ~150
greater inter-subjects
variability. In flash -10-
responses, peaks can be 0 100 300ms
observed at 30, 90 and 120
ms (N1, N2 and P2, 10-
respectively). These
responses are more variable
than the previous patterns
0- P2
~120
N2
~90
-10-
0 100 300ms
itself (or resets) before a new next stimulus appears. The top inset of Fig. 2.14 shows
the typical VEP response together with the transient response elicited by an pattern-
reversal stimulus of 2 Hz. The gray background is used to indicate the period of
repetition (500 ms) of the stimulation that is higher than the time of the typical
response (≈300 ms).
2.6 Influence of Refractive Error in VEP
It was found that technical and physiological factors such as pupil diameter or refrac-
tive errors affect the VEP [19], because the amplitude of the pattern of an evoked
potential is extremely sensitive to optical blurring. In ophthalmology this effect can
be used as a means of determining refractive error by measuring the amplitude of
the VEP with changes in power of trial lenses [20]. Refractive errors will affect the
interpretation of the VEP results, therefore it is important to take the subject’s visual
acuity into consideration. In the 70s, it was shown that the amplitude of the pattern is
sensitive to the optical blurring [17]. Defocusing causes a degradation of the sharp-
With correction lenses Without correction lenses

Latency
RE
Amplitude LE
BE
AE
tL tL
time time
Fig. 2.13 Comparison between VEP waveforms with (left curves) and without correction lenses
(right curves) of a subject with refraction deficit. From top to bottom, first and second are traces
of right (RE) and left eye (LE), respectively; the third trace corresponds to both eyes (BE); and the
lowest trace is the grand average (AE) of the potentials. t L represents the latency with correction
lenses
ness of the contours of the checkerboard. Figure 2.13 shows an example in which
VEPs are elicited in one subject with and without lenses [21]. It was using lenses
of various dioptric powers to adjust the defocus degree. It can be seen that the VEP
amplitude is hardly influenced by refraction errors or by the degree of visual acuity
of both eyes or in one of them.
2.7 Steady-State VEP
On the other hand, if the stimulation repeated with a fast enough and constant fre-
quency, enough to prevent the evoked neural activity returning to rest (baseline) state,
the elicited response becomes a steady-state response. It is called SSVEP and its fre-
quency is directly associated to the stimulation frequency [22]. Figure 2.14 illustrates
the steady-state responses for three pattern-reversal stimuli with frequencies 6, 12
and 20 Hz. Note, due to the fact that the neural activity does not return to the baseline
state, the waveform of steady-state response (dotted gray curves) are not similar to
the typical VEP waveform (black curves) during the stimulation lapse, indicated by
gray background. Also, the stimulation lapse causes the amplitude of steady-state
responses are different in these three cases. SSVEP waveforms are periodic signals
that, like sinusoidal waveforms, can be measured in terms of its amplitude and phase.
The amplitude is affected by the stimulation lapse given by the stimulus frequency. As
nearly sinusoidal oscillatory waveform, SSVEP contains a fundamental frequency
that is related to the stimulus. In steady-state, the brain response is modulated at
the second harmonic of the stimulation frequency (twice this frequency) for pattern-
reversal stimuli. For flash stimulus, the response is modulated at the fundamental
frequency. In general, SSVEP responses vary with the temporal frequency, spatial
2.7 Steady-State VEP 21
Transient
response
0 300ms 2 Hz
0 ~83 300
Steady-state responses
6 Hz
Stimulation lapse
0 ~41 300 12 Hz
Stimulation lapse
VEP
0 25 300
Stimulus signal of 20Hz
Fig. 2.14 VEP waveform as a function of stimulation frequency. Note that the wave form is
basically modulated at the second harmonic of the stimulus frequency. At the slowest rate (2 Hz)
the components of the transient VEP can be seen
frequency (for patter-reversal and onset/offset pattern), contrast and luminance of

the visual stimulus.
SSVEP are better observed in frequency domain because it has characteristic
peaks caused by the periodic oscillations in steady-state. These peaks arise in the
fundamental and harmonic frequencies of the brain signal. Figure 2.15 shows the
SSVEP response in the frequency domain with normalized amplitudes. The top inset
corresponds the brain response to a light-emitting diode (LED) stimulus flickering
at 6.4 Hz. The curves are the spectrum computed with different signal length or
time window (TW). Peaks at fundamental frequency of stimulus are evident in all
curves. However, it can be seen that the length of signal influence in the amplitude
of peaks. Hence, SNR can be improved by increasing the length of the brain signal
[23]. The bottom inset shows the spectral response for the same subject ad the same
the stimulus but flickering at 5.6 Hz. Peaks in harmonic frequencies can be observed
in both cases.
SSVEP can be elicited by several types of stimuli, such as flickering light given
by a single LED or an array, flickering image given by single graphics rendered on
a computer. The frequency spectrum of the SSVEP due to LED, cathode ray tube
(CRT) or liquid crystal display (LCD) are different from each other, as shown in
[24]. The frequency spectrum was the simplest for the LED, which only contained
the fundamental frequency and harmonics. The spectral responses of CRT contains
additionally peaks caused by the monitor refreshing frequency [25, 26]. And the
frequency spectrum of the response to LCD presents many low-frequency com-
ponents in addition to the fundamental frequency and harmonics. Due to the easy
way of stimulation by employing computer displays and LED arrays, SSVEP tech-
niques are gaining increasing application in BCI research [27, 28]. They are being
used for tagging perceptual and cognitive events, specially in which the employment
Fig. 2.15 SSVEP spectral response computed by using EEG signals of different lengths
(1, 2, 4, 8 s) evoked by visual stimuli flickering at 6.4 Hz (top inset) and 5.6 Hz (bottom inset)
of event related potentials (ERP) are not successful [29]. For example, in the study
of attention, two or more objects (such as letters) are presented in different sides of
computer screen oscillating at different frequencies. Frequencies that are measured
using EEG can show which side is being attended [30].
2.8 BCI Based on SSVEP
Brain-Computer Interfaces
BCIs are systems that could help to restore useful functions to people severely dis-
abled by a wide variety of devastating CNS and neuromuscular disorders, and to
enhance functions in healthy individuals [31]. These systems that measure EEG
activity and convert it into an artificial output also can replace, enhance, supple-
ment, or improve natural CNS output, as shown in Fig. 2.16; and thereby change the
ongoing interactions between users and their external environment [32]. A BCI is
a computer-based system that acquires brain signals, analyzes, and translates them
into commands that are relayed to an output device to carry out a desired action.
Thus, BCIs do not use the brain’s normal output pathways of peripheral nerves and
muscles. This definition strictly limits the term BCI to systems that measure and use
signals produced by the CNS.
2.8 BCI Based on SSVEP 23
Brain-computer interface
Signal Feature Feature

acquisition extraction translation
Commands
Replace
Restore
Enhance
Applications
Supplement
Improve
Fig. 2.16 Design and operation of a BCI system. It records brain signals, extracts specific features
and translates them into commands. BCIs applications improve, supplement, enhance, restore or
replace natural neuromuscular outputs
A BCI records brain signals, extracts particular features from them, and translates
the features into new artificial outputs that act on the environment or on the body itself.
The output could replace natural output that has been lost for injury or disease, and
help to restore lost natural output of someone with a spinal cord injury whose limbs are
paralyzed. Also, it could enhance natural CNS output of someone who is engaged in
a task that needs continuous attention, and supplement natural neuromuscular output
of a subject who is able to control, e.g. a robotic arm. In this sense, recent years have
seen BCI applications as a novel and promising new channel of communication,
control and entertainment not only for people with disabilities but also for healthy
people.
Brain-Computer Interfaces Based on SSVEP
As shown in Figs. 2.14 and 2.15, the waveform of SSVEP is related to the flicker-
ing frequency of a visual stimulus and the spectral response presents peaks in the
fundamental frequency of the stimulus and its harmonics. In BCIs based on SSVEP,
a set of visual stimuli flickering at different frequencies are employed. Each stim-
ulus is associated to a specific command that can be used to control an external
device such as robotic wheelchair. Usually, stimuli are presented to users properly,
in order to avoid interference between two or more stimuli. To send a command to
an external device, users are instructed to attend the stimulus associated with the
specific command. SSVEP-BCIs are developed to capture the EEG signals, extract
the fundamental frequency after processing the signal and translate it into an external
command [33]. Hence, this kind of BCIs allows users to control external devices by
attending a stimulus from a set of flickering stimuli. Currently, the SSVEP-BCI has
some advantages over other EEG-based BCI systems, specially when signals are
recorded over the visual cortex. It includes (i) a high SNR [18]; (ii) a high Informa-
tion transfer rate (ITR) [18]; (iii) a less susceptibility to eye movements and blink
artifacts as well as to EMG artifacts [34]; and (iv) a very little (or no) training stage,
since the VEPs are inherent responses of the brain [18, 35].
Figure 2.17 shows the elements of brain-computer interaction involved in the
controlling of an external device given by a telepresence robot. The BCI captures and
processes the brain signals before extracting features and recognizing the command
associated to the stimulus. Next, a control command is sent to the mobile robot. The
SSVEP response depends of the characteristics of the stimulation source. The color,
intensity, duty cycle and principally the flickering frequency of the stimulus modulate
the response. Software stimulation sources running on a computer consisting of
alternate chessboard patterns or dedicated stimulation sources can be used to control
intensity and waveform of stimulation signals. SSVEP patterns can be automatically
detected through a series of signal processing steps including pre-processing (e.g.,
band-pass filtering), artifact detection/correction, feature extraction (e.g., spectral
content at the stimulation frequencies), and feature classification. BCI performance
is usually assessed in terms of classification accuracy, classification speed, number
of available choices, and bit rate. In SSVEP-BCI systems, the classification accuracy
is primarily influenced by the strength of the evoked response, the SNR, and the
differences in the properties of the stimuli. The classification speed depends on the
time it takes for the SSVEP to be of sufficient strength. Increasing the number of
BCI
Signal Signal Feature Command
acquisition processing extraction recognition
Robotic
Set of visual
Command
Subject device
Brain signals
stimuli
Feedback (movement)
Fig. 2.17 SSVEP-BCI for controlling a robotic device. The BCI captures and processes the brain
signals before extracting features and recognizing the command associated to the stimulus. Next, a
control command is sent to the mobile robot
2.8 BCI Based on SSVEP 25
targets offers a higher number of possible commands but can decrease classification
accuracy and speed [26].
2.9 SSVEP-BCI and Gaze Dependence
Although the traditional SSVEP-BCI systems are becoming robust systems, they
are unsuitable for patients with paralysis who do not have reliable control of eye
movements. Due to the fact that these systems demand muscular activity to attend
a target stimulus, they are called dependent BCI systems [31, 36]. It means that the
features extracted from the EEG signals depends of muscular activity. Consequently,
other assistive technologies could be employed to detect the gaze direction. However,
in recent years strong evidence suggests that people can shift the attention among
stimuli with no gaze movement. This is the basis on the Independent-gaze BCI.
Selective Attention
Visual-spatial attention refers to the ability to selectively process the relevant events
in the visual surroundings and ignore the irrelevant events. Attention can be directed
to visual events in two ways; (i) head and eye movements can be employed to gaze
directly to an object. This is often referred to as an overt shift of attention, and (ii)
alternatively, spatial attention can be directed towards the relevant object or event
without movement of the eyes, often referred to as a covert shift of attention. Covert
spatial attention allows an observer to attend events independent of eye movements.
[37] conclude that covert and overt attention shifts utilize different neural mecha-
nisms. Shift of attention is the mechanism that is employed in SSVEP-BCI to select
one of a set of stimuli. It occurs when directing attention to a stimulus increases the
efficiency of processing decreasing the processing of irrelevant stimulus. Figure 2.18
illustrates this mechanism, in which white circles and points were placed on the
gazed and attended objects, respectively. First figure (left inset) represents an initial
stage, in which an object is gazed and attended. The second figure (middle inset)
illustrates the overt attention, in which the attention and gaze were shifted together.
In the third figure (right inset), only the attention having been shifted, it is called
gaze-independent selection because muscular movements are not required to shift
attention.
Current State of Gaze-Independent SSVEP-BCI Research
BCI research has been focusing on developing communication and control tech-
nologies for people with severe neuro-muscular disorders that could cause partial
or complete paralysis. BCIs are using different technologies as EEG, magnetoen-
cephalographic (MEG) [38], electrocorticographic (ECoG) [39, 40], or intracortical
recordings to measure the brain activity with high spatial resolution. Also, other
signals as functional magnetic resonance imaging (fMRI) are being used to record
information with high spatial resolution.
Attended Gazed Attended Gazed Attended Gazed

stimulus stimulus stimulus stimulus stimulus stimulus
A + B A + B A + B
Fig. 2.18 Covert versus overt attention. Left Starting stage where gazing and attention are in the
plus symbol. Middle Overt attention where both attention and gazing shift from plus symbol to
letter B. Right Covert attention where only the attention shits from plus symbol to letter B; Gazing
directions does not change
In BCI research area, the most common non-invasive method employed in BCI
is based on EEG. In 1929, Hans Berger speculates about the possibility of reading
the thoughts from EEG signals [41]. Then, in 70’s, the term BCI was coined by [42].
Near 1980, Reagan proposed the first SSVEP-BCI [18] and studies demonstrating
control of SCP were published [43]. In 1988, a P300 speller based on ERP was
designed [44]. Late last century, applications of SSVEP-BCI were taken into account
again [45] and the basic principles of events related to desynchronization (ERD) and
events related to synchronization (ERS) were described [46]. [33] develops a non-
invasive SSVEP-BCI with two commands. In 2002, sensorimotor rhythms (SMR)
were used to control a device with a BCI [31]. Figure 2.19 shows a timeline of the
BCI research indicating the year when BCI systems were proposed or developed.
In the past decade, experiments to evaluate the number of people that are able to
operate BCI based on EEG have been conducted. Regarding to BCI based on motor
imagery and ERD, a demographic study conducted by [47] 93% of the subjects were
able to achieve a classification accuracy above 60%. Regarding BCI based on P300,
in an experiment conducted by [48] achieved 72% of the subjects were able to spell
with 100% accuracy. And, regarding to SSVEP-BCI, a demographic experiment
conducted by [49] an accuracy average of 95% was achieved.
Although SSVEP-BCI is one of the systems presenting the best results, the dis-
advantage is given by its muscular dependence because subjects must perform neck,
head and/or eyeball control to redirect their gaze direction making it not unsuitable
for patients with deteriorated brain motor control. Notwithstanding, SSVEP-BCI
systems that do not demand muscular movements are being proposed, for instance,
exploring the covert attention as an alternative of stimulus selection. The representa-
tive studies in SSVEP-BCI that are not dependent on gaze movements are described
briefly below. To see how other BCI systems, such as P300 or motor imagery, are
addressing the problem of gaze dependence please refer the study conducted by
[50]. Also, [51] perform a meta analysis of BCI in ALS. Figure 2.20 shows a time-
line of gaze-independent SSVEP-BCI research, in which systems based on spatial
and non-spatial covert attention; and eye-closed systems are presented.
2.9 SSVEP-BCI and Gaze Dependence 27
Hans Berger Regan purposes and Cheng and Gao Wolpaw et al. use a
speculates about describes the first retake the SMR to control a BCI
the possibility of SSVEP-BCI application of device
reading the SSVEP-BCI
thoughts from EEG
signals Birbaumer et al. Pfurtscheller and
published experiments Lopez da Silva
demonstrating operant describe the basic
controls of SCP principles of ERS
and ERD
1929 1973 1979 1988 1999 2000 2002
Vidal coins the Farwell and Middendorf et al.

term BCI Donching design a develop a non-invasive
P300 speller SSVEP-BCI with two
commands
Fig. 2.19 Time-line of BCI research. It starts at 1929 when Berger speculated about the possibility
of reading thoughts, and indicates when BCI systems based on SCP, ERD, P300, SMR and SSVEP
was proposed or developed
In a pioneer study, [52] showed that the transition from overt to covert attention
in a SSVEP-BCI, allowing a binary decision, resulted in a reduction of classification
accuracy by about 20% on average. In light of this performance decrease, the same
authors redesigned the paradigm modifying the bilaterally displayed stimuli (visual
angle) and obtained an average of binary accuracy of 70.3% [28].
[36] investigated the hypothesis that the superposition of visual stimulation pat-
terns could evoke classifiable changes in SSVEP. They presented the subjects with
two images each oscillating at a different frequency. The oscillating images could
be presented either in a superimposed or separated condition, in order to explore the
role of gaze function on the system performance. In half of the 14 involved healthy
subjects, the overlaid condition induced differences in SSVEP activity elicited by
the visual stimulation patterns that were robust enough to predict an online BCI con-
trol. The authors demonstrated that such SSVEP differences depend on the selective
attention paid to one of two superimposed stimulation patterns.
[53] proposed a covert non-spatial visual selective attention paradigm to operate
a SSVEP-BCI. Two sets of dots with different colors and flickering frequencies were
used to induce the perception of two superimposed transparent surfaces. 18 healthy
subjects were asked to selectively attend to one of the two surfaces in order to control
the BCI system to perform a binary decision task during a three day training program.
An average of accuracy of 72.6% was achieved in the last training session. As reported
in Table 2.1 the system would achieve an ITR of 2.17 bits/min.
Kelly et al. propose Zhang et al. Lim et al. propose Hsu et al.
a SSVEP-BCI propose a SSVEP- a eye-closed evaluate a
based on covert BCI based on SSVEP-BCI SSVEP-BCI
spatial selective covert non-spatial system. with ALS
attention selective attention. patients
2005 2008 2010 2012 2013 2014 2016
Allison et al. Walter et al. use Lesenfants et al.

evaluate the static dots to propose a novel two
superposition of compare the gazed class independent
two stimuli. and covert attention. SSVEP-BCI system.
Fig. 2.20 Time-line of gaze-independent SSVEP-BCI research. Systems based on spatial and non-
spatial covert attention; and eye closed are presented
[54] proposed a novel two-class independent SSVEP-BCI based on covert atten-

tion. The influence of feature extraction algorithms and the number of harmonics
frequencies were studied. Also, a test with online communication on healthy vol-
unteers and patients with LIS was performed. A newly developed portable light
emitting diode-based “interlaced squares” stimulation pattern was employed. Mean
offline and online accuracies on healthy subjects (H) were respectively 85 ± 2% and
74 ± 13%, with eight out of twelve subjects succeeding to communicate efficiently
with 80 ± 9% accuracy. Two out of six LIS patients (end-users) reached an offline
accuracy above the chance level, illustrating a response to a command. One out of
four LIS patients had success in online communication.
[55] compared the modulation of SSVEP amplitudes when subjects directly gazed
at a flickering array of static dots (overt attention) and when they covertly shifted
attention to the dots keeping their eyes at central fixation. A discrimination task
Table 2.1 Comparison of the accuracy and ITR of gaze-independent SSVEPBCI. H: Healthy, End:
End user, on: online, and off: offline
Study Class. Acc. (%) ITR Analysis Popul. Subj. Eye
(bits/min) movement
[28] 2 70.3 0.91 Off H 10 <1o
[36] 2 74 4.18 On H 14
[53] 2 72.6 2.17 On H 18 <1 µV
[55] 2 70.3 0.91 Off H 14 <25 µV
[56] 2 80 10.8 Off/On H/End 11/1
[54] 2 80 Off/On H/End 12/6 Nothing
[57] 2 83 2.00 Off/On End 3
2.9 SSVEP-BCI and Gaze Dependence 29
was performed at the attended location to ensure that subjects shifted attention as
instructed. Horizontal eye movements (allowed in overt attention but to be avoided
in covert attention) were monitored by an electrooculogram. It was demonstrated
that overt and covert attention differ in their effect on SSVEP amplitudes and behav-
ior. The lower amplitude modulation by covert attention was considered for both
development and application of classification procedures in covert SSVEP-BCIs.
An additional study conducted by [56], which was not identified under the search
criteria, performs a classification of binary intentions for individuals with impaired
oculomotor function: eyes-closed SSVEP-BCI, allowing users to express their binary
intentions without needing to open their eyes. A pair of glasses with two light emit-
ting diodes flickering at different frequencies was used to present visual stimuli
to participants with their eyes closed, and recorded EEG patterns were classified in
offline/online experiments conducted with eleven healthy participants and one patient
with severe ALS. Through offline experiments, it was confirmed that SSVEP could
be modulated by visual selective attention to a specific light stimulus penetrating
through the eyelids. The online experiment conducted with the ALS patient showed
a classification accuracy of 80%.
A recent study evaluated the feasibility of using SSVEP-BCI in young, elder and
ALS groups [57]. It was designed for two commands using frontal and occipital
positions. Stimuli were based on arrays of 5 LEDs. Seven ALS patients were consid-
ered in addition to sixteen healthy people. It was reported that four of them were no
capable of completing required tasks. Average of accuracy for healthy people was
higher than 90%. While, the accuracy and the ITR for ALS were 83.3% and 2.00
bits/min, respectively for occipital SSVEP. For frontal electrode the values of accu-
racy and ITR were smaller. Although the focus of this work is not eye movement, it
was considered in the present book because end-users were considered in the study.
Table 2.1 summarizes the results achieved by the studies above that include the
accuracy and ITR. It can be seen that the three more recent studies performed exper-
iments with end-users. All of them tested their systems with two commands and the
highest accuracy rate attained was 80%. The average of the reported ITR was 3.794.
In general, results of this Table provide a true overview of the state-of-the-art con-
cerning gaze-independent SSVEP-BCIs. In traditional SSVEP-BCI systems, high
accuracy rates are achieved; however subjects must perform muscular movements
to redirect their gaze. In SSVEP-BCI systems based on covert attention, subjects do
not perform muscular movements; however the accuracy rates are not high. Further-
more, a previous training stage is necessary because subjects must learn to maintain
their covert attention on the target stimulus. In this context, a novel assessment is
introduced and described in this book, in which subjects select the target stimulus by
shifting of focus that does not demand an exhausted training stage because focusing
is an optical mechanism that is used naturally by humans throughout life. Also, the
focusing mechanism does not demand neck, head or eyeball movements, although
these may or may not happen. Patients with paralysis cannot control muscular move-
ments; however, in patients with neurodegenerative diseases such as ALS where the
sense of sight is not affected, can retain minimal eye movements, including pupil
and accommodation reflex mechanisms; making focus control possible even in the
end stages of these pathologies.
References
1. Kandel E, Schwartz J, Jessell T (1991) Principles of neural science, 4th edn. Mc Graw Hill,
USA
2. Guyton AC, Hall JE (2006) Guyton and hall textbook of medical physiology, 11th edn. Elsevier
Inc., Philadelphia
Press, New Jersey
bridge
5. Atchinson D, Smith G (2000) Optics of the human eye. Elsevier Science, Edinburgh
6. Howard I (2012) Perceiving in depth: volume 1 basic mechanisms. Oxford University Press,
New York
7. Gonzalez RC, Woods RE (2002) Digital image processing. Prentice Hall
8. Packer O, Williams RD (2003) The science of color, 2 edn., chap. Light, the retinal image, and
photoreceptors. Elsevier Science Ltd, Amsterdam, pp. 41–102
9. Livinsgtone M, Hubel D (1988) Segregation of form, color, movement, and depth: anatomy,
physiology, and perception. Science, New Series 240(4853):740–749
10. Huggins JE (2010) Brain-computer interfaces. revolutionizing human-computer interaction,
chap. BCIs based on signals from between the brain and skull. Springer, Mineapolis, pp 221–
239
11. Infantosi A, Lazarev V, De Campos D (2005) Detecting responses to intermittent photic stim-
ulation in the electroencephalogram using the spectral f test. Brazil J Biomed Eng 21(1):25–36
12. Nunez PL, Srinivasan R (2006) Electric fields of the brain. The neurophysics of EEG. Oxford
University Press, New York
13. Jurcak V, Tsuzuki D, Dan I (2007) 10/20, 10/10, and 10/5 systems revisited: their validity as
relative head-surface-based positioning systems. Neuroimage 34(4):1600–1611
14. Spekreijse H (1980) Evoked potentials, chap. Pattern evoked potentials: principles, methodol-
ogy and phenomenology. MTP Press Limited, pp 55–74
15. Holder GE (2004) Electrophysiological assessment of optic nerve disease. Eye 18(11):1133–
1143
16. Odom J, Bach M, Brigell M, Holder G, McCulloch D, Tormene A (2010) Vaegan: Iscev standard
for clinical visual evoked potentials (2009 update). Documenta Ophthalmologica 120(1):111–
119
17. Sokol S (1976) Visually evoked potentials: theory, techniques and clinical applications. Surv
Ophthalmol 21(1):18–44
18. Vialatte FB, Maurice M, Dauwels J, Cichocki A (2010) Steady-state visually evoked potentials:
focus on essential paradigms and future perspectives. Progr Neurobiol 90(4):418–438
19. Kothari R, Bokariya P, Singh S, Narang P, Singh R (2014) Refractive errors and their effects
on visual evoked potentials. J Clin Ophthalmol Res 2(1):3–6
20. Sokol S, Moskowitz A (1981) Effect of retinal blur on the peak latency of the pattern evoked
potential. Vision Res 21(8):1279–1286
21. di Summa A, Fusina S, Bertolasi L, Vicentini S, Perlini S, Bongiovanni L, Polo A (1999)
Mechanism of binocular interaction in refraction errors: study using pattern-reversal visual
evoked potentials. Documenta Ophthalmologica 98(2):139–151
22. Russo FD, Teder-salejarvi WA, Hillyard SA (2003) The cognitive electrophysiology of mind
and brain, chap. Steady-state VEP and attentional visual processing. Academic Press, pp 259–
274
References 31
23. Wang Y, Wang R, Gao X, Hong B, Gao S (2006) A practical vep-based brain-computer interface.
IEEE Trans Neural Syst Rehabil Eng 14(2):234–240
24. Wu Z, Lai Y, Xia Y, Wu D, Yao D (2008) Stimulator selection in ssvep-based bci. Med Eng
Phys 30(8):1079–1088
25. Herrmann CS (2001) Human eeg responses to 1–100 hz flicker: resonance phenomena in visual
cortex and their potential correlation to cognitive phenomena. Exp Brain Res 137(3–4):346–
353
26. Pastor MA, Artieda J, Arbizu J, Valencia M, Masdeu JC (2003) Human cerebral activation
during steady-state visual-evoked responses. J Neurosci 23(37):11621–11627
27. Wang Y, Gao X, Hong B, Gao S (2010) Practical designs of brain-computer interfaces based
on the modulation of eeg rhythms. In: Graimann B, Pfurtscheller G, Allison B (eds) Brain-
computer interfaces: revolutionizing human-computer interaction. Springer, Berlin, pp 137–
154
28. Kelly S, Lalor E, Finucane C, Gary M, Reilly R (2005) Visual spatial attention control in an
independent brain-computer interface. IEEE Trans Biomed Eng 52(9):1588–1596
29. Parkkonena L, Anderssona J, Hamalainen M, Haria R (2008) Early visual brain areas reflect
the percept of an ambiguous scene. Proc Natl Acad Sci105(51):20500–20504
30. Toffanin P, de Jong R, Johnson A, Martens S (2009) Using frequency tagging to quantify
attentional deployment in a visual divided attention task. Int J Psychophysiol 72(3):289–298
31. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain-computer
interfaces for communication and control. Clin Neurophysiol 113(6):767–791
32. He B, Gao S, Yuan H, Wolpaw JR (2013) Brain computer interfaces. In: He B (ed) Neural
engineering, 2nd edn. Springer, US, Mineapolis, pp 87–151
33. Middendorf M, Calhoun G, Jones K (2000) Brain-computer interfaces based on the steady-state
visual-evoked response. IEEE Trans Neural Syst Rehabil Eng 8(2):211–214
34. Lalor EC, Kelly SP, Finucane C, Burke R, Smith R, Reilly RB, McDarby G (2005) Steady-
state vep-based brain-computer interface control in an immersive 3d gaming environment.
EURASIP J Appl Signal Process 19:3156–3164
35. Bin G, Gao X, Wang Y, Hong B, Gao S (2009) Vep-based brain-computer interfaces: time,
frequency, and code modulations. IEEE Comput Intell Mag 4(4):22–26
36. Allison B, McFarland D, Schalk G, Zheng S, Jackson M, Wolpaw J (2008) Towards an indepen-
dent brain-computer interface using steady state visual evoked potentials. Clin Neurophysiol
119(2):399–408
37. Posner MI, Petersen SE (1990) The attention system of the human brain. Ann Rev Neurosci
13(1):25–42
38. Mellinger J, Schalk G, Braun C, Preissl H, Rosenstiel W, Birbaumer N, Kubler A (2007) An
meg-based brain-computer interface (bci). NeuroImage 36(3):581–593
39. Potes C, Gunduz A, Brunner P, Schalk G (2012) Dynamics of electrocorticographic (ecog)
activity in human temporal and frontal cortical areas during music listening. Neuroimage
61(4):841–848
40. Schalk G, Leuthardt E (2011) Brain-computer interfaces using electrocorticographic signals.
IEEE Rev Biomed Eng 4:140–154
41. Birbaumer N (2005) Breaking the silence: Braincomputer interfaces (bci) for communication
and motor control. Psychophy 43(6):517–532
42. Vidal J (1973) Toward direct brain-computer communication. Ann Rev Biophys Bioeng 2:157–
180
43. Birbaumer N, Cohen L (2007) Braincomputer interfaces: communication and restoration of
movement in paralysis. J Physiol 579(3):621–636
44. Farwell L, Donchin E (1988) Talking off the top of your head: toward a mental prosthesis
utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol 70(6):510–523
45. Cheng M, Gao X, Gao S, Xu D (2002) Design and implementation of a brain-computer interface
with high transfer rates. IEEE Trans Biomed Eng 49(10):1181–1186
46. Pfurtscheller G, da Silva FHL (1999) Event-related eeg/meg synchronization and desynchro-
nization: basic principles. Clin Neurophysiol 110:1842–1857
47. Guger C, Edlinger G, Harkam W, Niedermayer I, Pfurtscheller G (2003) How many people are
able to operate an eeg-based brain-computer interface (bci)? IEEE Trans Neural Syst Rehabil
Eng 11(2):145–147
48. Guger C, Daban S, Sellers E, Holzner C, Krausz G, Carabalona R, Gramatica F, Edlinger G
(2009) How many people are able to control a p300-based brain-computer interface (bci)?
Neurosci Lett 462(1):94–98
49. Allison B, Luth T, Valbuena D, Teymourian A, Volosyak I, Graser A (2010) Bci demographics:
how many (and what kinds of) people can use an ssvep bci? IEEE Trans Neural Syst Rehabil
Eng 18(2):107–116
50. Riccio A, Mattia D, Simione L, Olivetti M, Cincotti F (2012) Eye-gaze independent eeg-based
brain-computer interfaces for communication. J Neural Eng 9(4):1–15
52. Kelly S, Lalor E, Finucane C, Reilly R (2004) A comparison of covert and overt attention
as a control option in a steady-state visual evoked potential-based brain computer interface.
In: Proceedings of engineering in medicine and biology society conference, vol 2. IEEE, San
Francisco CA, USA, pp 4725–4728
53. Zhang D, Maye A, Gao X, Hong B, Engel AK, Gao S (2010) An independent brain-computer
interface using covert non-spatial visual selective attention. J Neural Eng 7(1):1–11
54. Lesenfants D, Habbal D, Lugo Z, Lebeau M, Horki P, Amico E, Pokorny C, Gomez F, Soddu A,
Muller-Putz G, Laureys S, Noirhomme Q (2014) An independent ssvep-based brain-computer
interface in locked-in syndrome. J Neural Eng 11(3):1–8
55. Walter S, Quigley C, Andersen SK, Mueller MM (2012) Effects of overt and covert attention
on the steady-state visual evoked potential. Neurosci Lett 519(1):37–41
56. Lim JH, Hwang HJ, Han CH, Jung KY, Im CH (2013) Classification of binary intentions
for individuals with impaired oculomotor function: eyes-closed ssvep-based brain-computer
interface (bci). J Neural Eng 10(2):1–9
57. Hsu HT, Lee IH, Tsai HT, Chang HC, Shyu KK, Hsu CC, Chang HH, Yeh TK, Chang CY, Lee
PL (2016) Evaluate the feasibility of using frontal ssvep to implement an ssvep-based bci in
young, elderly and als groups. IEEE Trans Neural Syst Rehabil Eng 24(5):603–615
Chapter 3
The Novel Stimulation Setup
Depth-of-field of the human eyes is the range of distances near the point of focus
where the eyes perceive the image sharply. Thus, when two visual stimuli are located
properly at different distances from the lens, if the nearest one is being focused on, it
is perceived sharp; while the farthest one (that is non-focused) is perceived blurred.
In the same way, if the farthest stimulus is focused on, the nearest one is perceive
blurry. In the present chapter, the novel SSVEP-BCI stimulation setup is described.
Previously, concepts related to the optical phenomenon of Depth-of-field are defined,
such as defocusing, point spread function and accommodation mechanism. A retinal
blur model is used for computing convenient distances of stimuli placement [1].
Distances must satisfy the condition where if one object is focused, then the other
one is non-focused and vice versa. The mechanism by which the lens can be shaped
in its geometry is therefore known as accommodation of the eye. The lens becomes
more rounded to focus on near objects and more elongated (or stretched) to focus
on objects that are far away. The human eye can refocus an object when its distance
changes, sharpening the projected image on the retinal surface. Hence, focusing does
not require eyeball movement because an accommodation mechanism performs it.
3.1 SSVEP-BCI Stimulation Setup
In traditional SSVEP-BCI systems, a set of stimuli flickering at different frequencies

is used. In these systems, users must redirect their gaze direction which demands
neck, head or eye movements to make a selection of a stimulus. Due to the fact
that a stimulus is associated with a command, muscular movements are demanded
each time that user wants to send a command. In this section, the novel SSVEP-BCI
stimulation setup is described, in which two SSVEP stimuli are presented in field of

DOI 10.1007/978-3-319-52298-2_3
34 3 The Novel Stimulation Setup
Fig. 3.1 The novel stimulation setup based on Depth-of-Field. Stimulation unit composed by two
green LED arrangements that are presented together in the field of view of the user but at different
distances. Stimuli were adjusted to 0.1 lumen for both stimuli, measure at a distance of 15 cm.
Stimulation unit (top-left); A three-dimensional model of stimulation unit and the stimulation box
(top-right); A subject in front of stimulation unit (bottom-left); and a simulated perception when
the farthest stimulus is focused on (bottom-right)
view of the BCI user but at different distances from him. This setup that is based on
the optical phenomenon of Depth-of-field allows users to make a stimulus selection
by shifting their eye focus instead of performing neck, head or eyeball movements.
The setup of the stimulation unit is shown in Fig. 3.1. It is composed of two
green LED arrangements (7 × 5 matrix) connected to two analog signal generators.
The dimension of each arrangement is 18 × 13 mm2 and the luminous flux was
adjusted to 0.1 lumen for both stimuli, measured at a distance of 15 cm. Stimuli
were programmed to flicker at 5.6 and 6.4 Hz with 50% of duty cycle [2, 3]. These
frequencies were chosen to be out of the range of between 15 and 25 Hz that can
provoke photosensitivity epileptic seizures [4]. Both arrangements are separated
2 mm from each other in a vertical plane in order to avoid overlapping into the
users’ field of view. In the top-left inset of Fig. 3.1, it is shown the unit placed into a
stimulation box, which in turn was placed next to the subject’s head, ensuring that
only the two stimuli were in the user’s field of view and avoiding the presence of
other light sources. Also, the interior of the box was painted in black to eliminate
the reflection of lights (top-left inset). A subject in front of the stimulation unit and
a simulation of his perception when the farthest stimulus is focused on, are shown
in the bottom-left and bottom-right insets, respectively.
3.1 SSVEP-BCI Stimulation Setup 35
Nearest Farthest
stimulus stimulus
Ears
13 mm
Head
φh
Ear
2 mm
Head
φv
18 mm
13 mm
18 mm
z n = 30 cm
LED
arrangement
z f = 50 cm
Eyes reference
Fig. 3.2 Distances of placement of stimuli in the novel SSVEP-BCI setup. z n and z f denote the
distances of the nearest ad the farthest stimuli, respectively. φv and φh denote the horizontal and
the vertical visual angles, respectively
Figure 3.2 shows the distances where stimuli are placed. Due to the fact that two
stimuli located at different distances are being used, they are denoted as nearest and
farthest. The nearest stimulus is placed at z n = 30 cm from the user, while the farthest
one is placed z f = 50 cm away. These distances, computed in the same way as [1],
were adopted in this work because they satisfy the condition in that if one stimulus
is focused on, then the other one is non-focused on, and vice versa. Horizontal and
vertical visual angles can be observed, respectively, in the top and bottom insets of
Fig. 3.2. Then, taking into account the distances of both stimuli and the dimension
of the LED arrangements, the horizontal (ϕh ) and the vertical (ϕv ) angles of the
stimulation unit can be computed as
ϕh = 2 tan−1 (w L /2z n ), (3.1)
and
ϕv = tan−1 (h L /z n ) + tan−1 (h L /z f ), (3.2)
where h L = 18 mm and w L = 13 mm are the dimensions of the LED arrangements.

As a result, the angles are given by ϕv = 5.49o and ϕh = 2.48o .
(a)
(b)
Left 90 Right
135 45
40
30
20
10
180 0
Optic disc
225 315
270
Fig. 3.3 Projections of focused and non-focused stimulus on the field of vision. a Perimeter chart
of the field of vision with two gray rectangles that represent the projection area of both stimuli. b
Subjective view focused (nearest) and non-focused (farthest) stimuli with vertical and horizontal
angles
Figure 3.3a shows the perimeter chart of the field of vision of the eye described
in Sect. 2.2, in which the gray rectangles in the center of the chart represent an
approximated area in which both stimuli of the novel setup are projected, considering
the horizontal and vertical angles computed with Eqs. (3.1) and (3.2), respectively.
In Fig. 3.3b it is represented how both stimuli are projected in the subjects’ field of
view when the nearest stimulus is focused on. Also, it is included the vertical and
horizontal angles and the dimension of the LED arrangements. Note that the vertical
and horizontal angles do not exceed the limits of the perimeter of vision, ensuring
that both stimuli will appear together in the field of view. Furthermore, due to the
fact that the angles being very small both stimuli are projected in the region with
high density of photoreceptors of the retina. Hence, the setup here proposed does
not demand neck, head and/or eyeball movement to choice the nearest or the farthest
stimulus, but demands shifting of eye focus.
In the following sections, Depth-of-field, a function that describe blurriness, and
the retinal blurry model are described. Also, the accommodation mechanism that
adjusts the shape of the lens in order to focus on objects is explained.
3.2 Non-focused Objects and Depth-of-Field
An object is imaged sharply by optical systems when it is placed at the point of

focus, then, objects that share the same distance with the point of focus are named as
focused objects and also are imaged sharply. In contrast, objects placed at different
distances are imaged blurry and are called non-focused objects. In general, a natural
3.2 Non-focused Objects and Depth-of-Field 37
Far point Fixation point Near point

(non-focused) (point of focus) (non-focused)
Fig. 3.4 Picture with focused and non-focused objects. Blurred circles on the right represent the
images of points of non-focused objects placed at different distances from point of focus. The
picture was captured with diaphragm aperture, exposure time and ISO speed of 5.6, 1/250 s and
3200, respectively. Picture was corrected adjusting its contrast (−20%), its bright (40%), and its
saturation level (0%)
scene is composed of a set of focused and non focused objects, as shown in Fig. 3.4.
It can be seen that objects behind and in front of the point of focus are perceived
blurry. When an object is out of focus, or non-focused, there is a reduction of contrast
in its projected image and it is perceived blurred [5]. In photography, the blurriness
of the object’s image depends on the distance of the object from the fixation point,
the diameter of aperture and the distance of the lens from the sensor surface. For
instance, the picture of Fig. 3.4 was taken with diaphragm aperture, exposure time
and ISO speed of 5.6, 1/250 s and 3200, respectively. The fixation point was at the
electrode of the center of the image. In human eyes, blurring effect occurs because
the lens converges a bundle of light rays from each point on the focused object to a
discrete point in the retinal image, but it diverges rays from points of the non-focused
object [6, 7]. Blurriness also depends on the distance of the non-focused objects from
the focus point. In the picture, it can be seen that the farthest objects are more blurred
than the objects near it. If the picture was composed of points, non-focused points
could be perceived as blurred circles, as illustrated in the bottom of Fig. 3.4; then, the
diameters of nearest circles are larger than the diameter of farthest circles [8–10].
Notwithstanding, objects that are not placed at the same distance of the focus point
but are placed very near it also can be perceived sharp. Depth-of-field is defined as the
range of distances near the point of focus where the eye imaged an object as sharp.
Then, objects placed out of the Depth-of-field are perceived blurry [5]. It happens
because there is a small region in the sensor surface of visual systems (the retina in
Depth-of-focus Depth-of-field
Lens
Focal Fixation
point point
Circle of Optical axis

confusion
Imaging
Image distance Object distance
plane (retina)
Fig. 3.5 Schematic diagram of Depth-of-field and depth-of-focus. The variation of blurriness is
very slightly and points projected under the circle of confusion are assumed in sharp. The circle of
confusion are related to the lens size, the distance of the lens from the point of focus (or fixation
point), and the distance from the imaging plane. Adapted from [11]
human eyes), in which the variation of blurriness is slight and points projected under
this region are assumed sharp. This region is named the circle of confusion. The
variation may be small enough that is not noticed, and thus there is a small amount of
“tolerance”. As shown in Fig. 3.5, the circle of confusion is related to the lens size,
the distance of the lens from the point of focus (or fixation point), and the distance
from the imaging plane. The range of the tolerance is termed the depth-of-focus [12].
3.3 Optical Point Spread Function (PSF)
Optical PSF, or simply PSF describes the response of an imaging system to a point
source or point object. Ideally, the image of a point source is a point. However, even
in a properly focused system, diffraction and the aberrations cause the illuminance
to spread out around the point image position. It is defined as a relative measure of
the luminous flux at the positions around the ideal point image, which transforms the
image of the point into a spread function that can be approximated to a blur circle
[13]. PSF can be understood as the impulse response of an optical system. Figure 3.6
illustrates this concept, in which the light distribution on the retinal surface (that
responses to a point source) is given by a function named as the Airy disc pattern
[14]. This pattern is formed on the retina by a point spread function caused by the
diffraction at the pupil [14]. Due to the fact that PSF is the distribution of light across
the photoreceptors surface, it can be used to measure the quality of an imaging
system. Then, if it is assumed that an object is composed of a set of points, all points
will be affected by the PSF in the projection image in a visual system. For this reason,
in ophthalmology, PSF is used as a diagnostic tool [8]. Aberrations of human eye
wavefront that cause some distortions of the projected images in the retinal surface,
also can be addressed with this spread function. Aberrations can be caused by defects
3.3 Optical Point Spread Function (PSF) 39
Point spread
Parallel rays
function
from a point
source Retinal
surface
Retinal
surface
Light
Eyeball distribution
Retinal surface
Fig. 3.6 PSF caused by diffraction. The light distribution caused by a point source forms the Airy
disk pattern on the retina due to the diffraction at the pupil
Object
astigmatism defocus astigmatism
trefoil coma coma trefoil
quadrafoil secondary spherical secondary quadrafoil
A set PSF that describes optical aberrations A set of images blurred images
Fig. 3.7 Object, some PSF that describes some optical aberrations and their corresponding simu-
lated images. The circle with “X” represents the convolution operation. Note that the PSF due to
the fact that defocusing is symmetric
of the cornea or the lens, or simply by the defocus [9]. The projected image of an
object can be simulated as the convolution of the true object and the PSF. Figure 3.7
shows some convolution examples in which the effects of optical aberrations in the
projected images are described by some spreading functions [15]. Also, it can be
seen that the PSF due to defocus aberration is concentric and symmetric; in addition,
the image of points of non-focused objects can be modeled as blurred circles, whose
diameters vary with the distance [9].
In optical systems that are limited only by defocusing, the PSF for a specific point
x = (x, y), could be expressed as
ps f (x, ΔD) (3.3)

Non-focused
Focused
object
object
d1 d1
Non-focused Focused
object object
d2 d2
Eye reference
Fig. 3.8 Images of focused and non-focused objects when they are presented together but at dif-
ferent distances. It is assumed that distances d1 and d2 are large enough that the non focused object
is out of Depth-of-Field of the focused one
where ΔD the defocusing degree or the optic power required to bring the object
in focus [16]. Optical power is the degree to which an optical system converges or
diverges light and its unit is given in diopters (m−1 ). The defocus degree of a non-
focused object can be defined as the difference between the current power of the lens
D f ocus , and the power of the lens when this object is in focus Dtarget , as
ΔD = D f ocus − Dtarget . (3.4)
Figure 3.8 illustrates the retinal images of two objects when they are presented
together but at different distances when it was assumed that the distances, d1 and d2 ,
are large enough to ensure that the non-focused object is out of the Depth-of-field of
focused one. Thus, when one of them is focused the other one is non-focused, and
vice versa.
3.4 Retinal Blurry Model
Retinal blurring effect occurs because the lens converges a bundle of light rays from
each point on the focused object to a discrete point in the retinal image, but it diverges
rays from points of the non-focused object. Depth-of-field is the range of distances
near the point of focus where the eye imaged an object as sharp. Hence, as illustrated
in Fig. 3.9, objects behind and in front of the point of focus are imaged blurred.
Blurriness caused by defocusing can be intended as concentric [9]; and the image
of a defocused point can be modeled as a blurred circle, whose diameter varies with
the distance. Thus, the image of a defocused object can be simulated by convolving
the image with a circular average spatial filter. In addition to the point of focus and
the distance from the eye, the diameter of the blurry circle caused by a defocused
point is related to the eye parameters such as pupil size or focal length. For example,
3.4 Retinal Blurry Model 41
Fig. 3.9 Three pictures of a set of electrodes. In the left inset, the label of farthest electrode T5 is
focused on. In the middle inset, the label of the electrode PO7 is focused on and the other electrodes
are non-focused. In the right inset, the label of electrode O1 is focused on; note that the labels
of electrodes T5 and P07 are blurred. Pictures were captured with diaphragm aperture, exposure
time and ISO speed of 5.6, 1/250 s and 3200, respectively. Pictures were corrected adjusting their
saturation level (0%)
in the theoretical model introduced in [17] and adapted by [1], the diameter can be
summarized as
d(z)|z0 = f (D, P, b0 , f a , z 0 , z), (3.5)
where D is the pupil size and P is a pupil factor, b0 is the distance from the lens to
the retina, f a is the focal length of the lens, z 0 is the distance of the fixation point,
and z is the distance of the defocused point. In this simplified model, the image
of defocused points projected on the retinal surface are modeled as blurred circles.
Then, the diameter d of the blurred circle of a non-focused point located at z and
when the fixation point is located at z 0 can be estimated as
D |b − b0 |
d(z)|z0 = , (3.6)
b + (P − 1) f a
where f a is the accommodated focal length; D is the pupil size and P is a pupil
factor, assumed in the original model as 5 mm and 0.9, respectively. The distance
between the retina and the lens is given by b0 ; and b is the distance from the lens
to the place where the non-focused point would be projected sharp, and they can be
computed as
1 1 1 1 1 1
= + and = + .
fa b0 z0 fa b z
(a) Fixation point at z 0 = .3 m (b) Fixation point at z 0 = .5 m
Diameter of blur (mm)

0.3 0.3
0.2 0.2
0.13 0.13
0.1 0.1
0 0
0.3 0.5 1 10 0.3 0.5 1 10
z (m) z (m)
Fig. 3.10 Retinal blur and distance. The fixation point is at distance z 0 and is imaged sharply on
the retina at b0 , but appear blurred for other distances from the eye. The degree of blur is determined
by z 0
Figure 3.10a, b show the curves of the diameter of blur circle d as a function of
distance z computed with Eq. (3.6) for the point of focus placed at z 0 = 0.3 m and
z 0 = 0.5 m, respectively. Note in Fig. 3.10a that the diameter at z = 0.3 m is zero
(white square) while the diameter at z = 0.5 m is around 0.13 mm (black marker).
In the same way, in Fig. 3.10b the diameter at z = 0.5 m (white square) is zero while
the diameter at z = 0.3 m approximately 0.13 mm (black marker).
This model satisfies facts as (i) points placed nearer and farther than fixation points
are imaged blurred; (ii) the diameter of the blurred circle increases as defocused
point approaches the eye; and (iii) the diameter increases asymptotically when the
unfocused point is moving to infinity. Also, in the presence of two objects properly
placed, the model shows the extraordinary situation when one of them is in focus,
the other one is defocused and vice versa.
For example, as represented in Fig. 3.11b, if a point placed at z 0 = 50 cm is
focused on, a blurred circle due to a non-focused point placed at z n = 30 cm is
projected in retinal plane. Figure 3.11a represents the blurred circle caused by the
non-focused point placed at z f , when the focused point was located at z 0 .
The diameter of blurry circle in Fig. 3.11a and 3.11b are 0.129 mm and 0.132 mm,
respectively. The human eye is able to refocus an object when its distance changes,
sharpening the projected image on the retinal surface [6]. Focusing does not require
eyeball movement because it is performed by an accommodation mechanism that
modifies the shape of the lens making it a variable refractive element. The lens
becomes more rounded to focus on near objects and more elongated (or stretched)
to focus on objects that are far away [7].
3.4 Retinal Blurry Model 43
(a)
Ciliary muscle Non-focused Fixation
Lens point point
(nearest) (farthest)
d
Retinal Retinal
image image b zn
of the of the
nearest farthest b0 z0
point point
(b)
Fixation Non-focused
point point
(nearest) (farthest)
d
Retinal Retinal
image image zf
of the of the
nearest farthest b0 z0
point point
Fig. 3.11 Depth-of-field and defocusing. a Retinal blurring effect caused by a non-focused point
placed nearer, z = z n , than the fixation point, z 0 ; d is the diameter of the blurry circle; b0 is the
distance between the lens and the retina; and b is the distance from the lens to the point where the
non-focused point would be imaged sharp. b Retinal blurring effect caused by a non-focused point
placed farther, z = z f , than the fixation point z 0 ; distance b not shown because it is outside the
figure
3.5 Accommodation of the Eye
The human eye, as any other optical instrument, has a limited sensitivity to optical
blur [18]. If there is a little difference between the power and length, then the image
that is formed on the retina will be very slightly out of focus. The defocusing degree
is related to the lens size, pupil aperture and the distance between the fixation point
and the defocused object. As shown in Fig. 3.12, accommodation of the eye is the
mechanism that adjusts the shape of the lens in order to focus an object modifying
the angle of refraction for impinging light rays [7]. It minimizes the defocusing
degree and maximizes image quality. Accommodation for a near target occurs when
the eye accommodates and the lens forms a more spherical shape to bring the near
target into the focus. On the other hand, the lens is elongated (stretched) for focusing
Eyeball Eyeball
Ciliary muscles Ciliary muscles

(relaxed) (contracted)
Suspensory Suspensory
ligaments ligaments
(tense) (slack)
Lens Lens
(thin) (thick)
Cornea Cornea
Fig. 3.12 Accommodation of the lens. Lens are elongated to focus distant objects by cilliary
muscles and suspensory ligaments (left inset). Lens are rounded to focus focus near objects (right
inset)
distant objects. This is because the lens converges the bundle of light rays from each
point on the focused object to a discrete point in the retinal image, but it diverges
rays from a point of non-focused object forming instead of a point, a blurry circle.
Accommodation is a reflex response of the nervous system and appears to be under
negative biofeedback control so as to automatically minimize blur in the retinal
image. When the blur exceeds some threshold a neural signal is sent to the ciliary
muscle to either stimulate or inhibit it, depending on what is needed to correct the
focusing error [6, 7, 19].
References
1. Vishwanath D, Blaser E (2010) Retinal blur and the perception of egocentric distance. J Vis
10(10):1–16
2. Bastos T, Auat Cheein F, Torres Muller S, Cardoso Celeste W, de la Cruz C, Cavalieri D,
Sarcinelli-Filho M, Faria Santos Amaral P, Perez E, Soria C, Carelli R (2014) Towards a new
modality-independent interface for a robotic wheelchair. IEEE Trans Neural Syst Rehabil Eng
22(3):84–567
3. Muller S, Bastos T, Sarcinelli M (2010) Incremental SSVEP analysis for BCI implementa-
tion. In: Engineering in Medicine and Biology Society (EMBC), 2010 Annual International
Conference of the IEEE, pp 3333–6
4. Fisher R, Harding G, Erba G, Barkley G, Wilkins A (2005) Photic- and pattern-induced seizures:
a review for the epilepsy foundation of america working group. Epilepsia 46(9):41–1426
Press, New Jersey
References 45
bridge
8. Dai GM (2008) Wavefront optics for vision correction. SPIE, Washington
9. Mahajan VN (1991) Aberration theory made simple. Spie Optical Engineering Press, Wash-
ington
10. Wilson B, Decker K, Roorda A (2002) Monochromatic aberrations provide an odd-error cue
to focus direction. J Opt Soc Am A 19(5):9–833
11. Songnian Z, Qi Z, Chang L, Xuemin L, Shousi S, Jun Q (2014) The representation of visual
depth perception based on the plenoptic function in the retina and its neural computation in
visual cortex v1. BMC Neurosci 15(1):1–17
12. Wang B, Ciuffreda KJ (2006) Depth-of-focus of the human eye: theory and clinical implications.
Surv Ophthalmol 51(1):75–85
13. Keating MP (2002) Geometric, physical, and visual optics. Butterworth-Heinemann, Woburn
14. Packer O, Williams DR (2003) Light, the retinal image, and photoreceptors. The science of
color, 2nd edn. Elsevier Science Ltd, Amsterdam, pp 41–102
15. Chen L, Singer B, Guirao A, Porter P, Williams DR (2005) Image metrics for predicting
subjective image quality. Optom Vis Sci 82(5):69–358
16. Burge J, Geisler WS (2011) Optimal defocus estimation in individual natural images. Proc Natl
Acad Sci 108(40):16,849–16,854
17. Le Grand Y, El Hage S (1980) Physiological optics. Springer
18. Howard I (2012) Perceiving in depth: Volume 1 basic mechanisms. Oxford University Press,
New York
19. Grusser OJ, Grusser-Cornehls U (1986) Physiology of vision. In: Schmidt RF (ed) Fundamen-
tals of sensory physiology. Springer, Berlin, pp 9–126
Chapter 4
Frequency Demodulation for a SSVEP-BCI
Based on Depth-of-Field
In traditional SSVEP-BCI systems, the analysis and the signal processing usually are
performed considering only the response of the gazed stimulus that is placed on the
center of the field. However, in most SSVEP-BCIs, a set of stimuli are presented to
the users, emerging the possibility that at least one non-gazed stimulus is projected
on the retinal surface. This possibility is not considered in traditional demodulation
processes for extracting the evoked potentials that arise in EEG signals during SSVEP
stimulation. For instance, the demodulation process proposed by [1] only consider
the target stimulus making it not suitable for the novel setup addressed in this book,
in which two stimuli are presented together in the center of the field of view.
In the present Chapter, a suitable demodulation process is proposed for the novel
stimulation setup. The aim of this process is to extract frequency characteristics
evoked by the focused stimulus, instead the non-focused stimulus is also present in
the field of view. This proposal is based on the process Wang’s demodulation [1] that
is based on the power of the signal evoked by the attended stimulus and is described
in Sect. 4.1. Although the power requirement seems obvious for gazed stimulus if
comparing with non-gazed one, it is not for focused stimulus, because the non-
focused one is projected together on the retinal surface. Traditional systems visually
do not evaluate if the power of the focused stimulus is higher than the non-focused one
because the changing of gaze direction ensures that only one stimulus is the center
of the field of view. In this sense, the spatial-temporal retinal response of focused
and non-focused stimuli are evaluated as well as its power spectrum amplitude.
Parameters such as wavelength, stimuli distances or photoreceptors density were
considered in the evaluation. Finally, some experiments are conducted by evaluating
and comparing the spectral power of focused and non focused stimuli in order to
verify if the amplitude of the retinal response is higher for the focused one than for
the non- focused one.

DOI 10.1007/978-3-319-52298-2_4
48 4 Frequency Demodulation for a SSVEP-BCI …
4.1 Traditional Demodulation
In SSVEP-BCI systems, the modulation of brain signals is performed by selective

attention. In these systems, a set of stimuli flickering at different frequencies is
presented and users are asked to choose one of them. A SSVEP pattern that is an
increase of the amplitude at the attended stimulus frequency arises in the EEG signals
recorded at the occipital brain areas [2]. The amplitude of this pattern increases
significantly as the attended stimulus is moved to the central of the visual field [1].
Due to the fact that any stimulus are able to be selected, users’ brain signals can be
modulated with different frequencies. Then, the frequency modulation is the basic
principle of implementing BCIs using SSVEP patterns to detect the users’ gaze
direction.
Figure 4.1 shows a SSVEP-BCI based on the modulation of two frequencies. Gray
rectangles represent two stimuli flickering at different frequencies, dot lines represent
the visual field, and the big and the small circles represent the user’s head and
the occipital electrode, respectively. In modulation of brain signals, the subject can
choose one stimulus changing his gaze direction. Then, a SSVEP pattern that arises
in f1 or f2 can be observed as a peak in the power spectrum of the modulated signal.
Thus, SSVEP-BCI systems are frequency coded as the frequency of each stimulus
is associated to a specific command. In this sense, a computational demodulation
process is needed to decode the gazed direction. The aim of the demodulation process
is to extract embedded information to be converted into a control signal. In this kind
of systems, users could send a sequence of commands by alternating the stimulus
attention.
Fig. 4.1 Traditional SSVEP-BCI modulation/demodulation. Modulation is performed by selective

attention when subject gazes the stimulus flickering at f1 and placed at d0 . Modulated signals was
recorded at occipital electrode and showed at spectral analyzer. Demodulation is performed by
decodifying the attended frequency that is associated to a command
4.1 Traditional Demodulation 49
In Fig. 4.1, stimuli are separated enough to ensure that the non-attended stimulus
is outside of visual field when the attended one is in the center of the visual field. This
information is highlighted and it is worth mentioning it because traditional systems
do not encompass the presence of a non-attended stimulus also in the field of view. For
instance, [1] describes two ways of performing demodulation in traditional SSVEP-
BCI systems based on analytical signals and based on the peak detection.
Analytical Demodulation
In the demodulation based on analytical signals, signals are described as complex
functions defines as
g(t) = s(t) + jŝ(t) = A(t)ejφ(t) , (4.1)
where s(t) is the EEG modulated signal, ŝ(t) is the Hilbert transform of s(t). In the
exponential form, it can be seen that A(t) and φ(t) are the envelope of the signal
and the instantaneous phase, respectively. Then, the instantaneous frequency of the
attended stimulus can be found by deriving the phase as
dφi (t)
ωi = , i = 1, 2...N, (4.2)
dt
where ω is the instantaneous angular frequency, N is the number of visual stimuli
and ωi = 2π fi . Although the demodulation based on analytical signals can be used
to demodulate tasks associated to motor imagery using the A(t) or tasks based on
phase-coded; it is not commonly used to demodulate frequency in SSVEP-BCI.
Demodulation Based on the Spectral Peak Detection
For real systems, [1] adopts the approach of demodulation by detecting the peak value
in the power spectrum for frequency recognition. Due to the fact that the attended
stimulus will induce a peak in the amplitude spectrum at its flickering frequency, the
aim of the demodulation is to search for the peak value in the power spectrum and
determine the corresponding frequency. In this sense, the frequency can be identified
as
ωi = arg max Pss (ω), i = 1, 2...N, (4.3)
ωi
where Pss (ω) represents the power spectrum of the modulated EEG signal s(t), N is
the number of visual stimuli and ωi = 2π fi .
4.2 Proposed Demodulation
In the novel SSVEP-BCI setup proposed and described in this book, the modulation
process is performed by selective attention but based on focused mechanism instead
changing of gaze direction. Figure 4.2 illustrates the novel setup in which gray rec-
tangles represent the two stimuli flickering at different frequencies and dotted lines
Fig. 4.2 Modulation/demodulation for the Novel SSVEP-BCI Setup. Modulation is performed by
selective attention when subject focus on the target stimulus that is flickering at f1 and placed at d1 .
Modulated signals were recorded at occipital electrode. Demodulation is performed by filtering the
modulated signal s(t) in order that s1 (t) only includes the frequency of the focused stimulus, and
s2 (t) only includes the frequency of the non-focused one
delimit the visual field. Stimuli are presented together in the visual field and sepa-
rated conveniently so that if one is in focus the other one is defocused. Rectangle
with border represents the focused stimulus. In the case that two stimuli are present
simultaneously in the central field of view the output will be affected by two stimuli;
then, the EEG signal could be modulated by both frequencies.
Analytical Demodulation
When two stimuli are presented together in the field of view, it can not be assumed
a priori that each stimulus evokes a potential at its flickering frequency. However, in
order to see how each frequency affects the EEG signal, it can be decomposed in two
signals, one that includes only the frequency of the focused stimulus and other one
that only includes the frequency of the non-focused stimulus, as shown in Fig. 4.2.
Hence, a LPF and a High-pass filter (HPF) can be both used with a cut frequency of
ωc = (ω1 + ω2 )/2, in which ωi = 2π fi .
If f1 < f2 , then the LPF and the HPF attenuate the frequencies f2 and f1 , respec-
tively; consequently the analytical signals for f1 and f2 , respectively, are given by
g1 (t) = A1 (t)ejφ1 (t) , (4.4)
and
g2 (t) = A2 (t)ejφ2 (t) ; (4.5)
where Ai (t) is the amplitude and φi (t) is the instantaneous phase. Note that if stimuli
were presented isolated, the instantaneous frequency ωi (for i = 1, 2) could be
4.2 Proposed Demodulation 51
computed by using (4.2). In this sense, a more general expression can be proposed to
find the frequency of the attended stimulus that includes responses of both stimuli as
ωa = a1 ω1 + a2 ω2 , (4.6)
or
dφ1 (t) dφ2 (t)
ωa = a1 + a2 , (4.7)
dt dt
in which ai (for i = 1, 2) is used to find analytically the instantaneous frequency ωa
of the stimulus that is being attended. Then, if the amplitude of focused stimulus is
assumed higher than the amplitude of the non-focused stimulus, the instantaneous
value of ai can be computed as
sgn (A1 (t) − A2 (t))

a1 = + 0.5, (4.8)
2
and
sgn (A2 (t) − A1 (t))
a2 = + 0.5, (4.9)
2
where Ai (t) for i = 1, 2 is the amplitude of the analytical signal. Finally, combining
(4.7), (4.9), and (4.8) the demodulation using analytical signals can be expressed as

sgn (A1 (t) − A2 (t)) dφ1 (t) sgn (A2 (t) − A1 (t)) dφ2 (t)
ωa = + 0.5 + + 0.5 .
2 dt 2 dt
(4.10)
Recall that (4.10) describes analytically the response of the focused stimulus, if
and only if, the amplitude of the power spectrum at its frequency is greater than the
amplitude of the power spectrum at the frequency of the non focused one. Due to the
fact that harmonics frequencies can be also evoked [3], a Band-pass filter (BPF) can
be used before the HPF and LPF filters.
Demodulation based on Spectral Peak Detection
In demodulation based on the peak detection, signal decomposition is not neces-
sary because the frequency values can be compared directly in the spectral domain.
Therefore, according to (4.3), the frequency of the focused stimulus can be found by
ωa = arg max P( ωi | Pss (ω)), (4.11)

ωi
where ωa = 2π fa and Pss (ω) represents the power spectrum of the EEG signal
s(t), which is computed by using Fast Fourier transform (FFT) over a finite TW. In
practical systems, it can be reduced to
ωa = arg max Pss (ωi ). (4.12)

ωi
Analogously to analytical demodulation, in peak based detection, Eq. (4.12) can be

applied with success, if and only if, the amplitude of the power spectrum at the
frequency of focused stimulus is greater than the amplitude of the power spectrum
at the frequency of the non-focused one.
In the following sections, the retinal response is described in order to address
a possible coherent explanation for the relative low power frequency observed for
the non-focused stimulus regarding the focused one. A principle that underlies the
development of the new stimulation setup is described.
4.3 Spatio-Temporal Retinal Response

for a Flickering Stimulus
The idealized retinal image I(x, y) is the projection of an object O(x, y) on the retinal
surface, which does not take into account optical conditions of the visual system
as lens shape or pupil size that affects the projected images. In general, projected
idealized images are associated to an object as
I(x, y) = O(x/M, y/M) (4.13)
where M is the magnification coefficient between object and image planes [4].
Figure 4.3 shows a scene with two objects, in which the bottom is focused and
the top is non-focused. In human eye, the projection of both objects on the retina
are affected by optical conditions of the visual system. The non-focused object are
projected blurred because lens are not accommodated for it. The color of objects
are given by the wavelength, then a more complete way to express a point of the
idealized image is I(x, y, λ) or I(x, λ), in which x = (x, y) is a bidimensional point
and λ is its wavelength.
Non-focused
Sharp object
image
x
Focused
object
Blurred
image
Retinal Surface
Fig. 4.3 Object O(x, y) and its image projection I(x, y) on the retinal surface
4.3 Spatio-Temporal Retinal Response for a Flickering Stimulus 53
The idealized image also can be expressed as I(x, λ) = T (λ, p)O(x, M); where
T is related to the luminous transmittance, p is the area of the pupil, M is the ratio
between the distance from the object (d1 ) and the distance to the projected image
(d2 ), and can be computed from a simplified eye model, M = d2 /d1 [4, 5, 13].
Optical Response
The projected image of a non-focused object, as described in Sect. 3.3, can be
expressed as the convolution of the idealized image and the PSF,
I(x, λ) ∗ psf (x, λ, ΔD). (4.14)
The PSF is computed for each point x and its wavelength λ; ΔD is the power required
to bring in focus a non focused object [6]. Regarding to the focused object, although
its image is projected sharp on the retina, it is also affected by optical conditions of
visual system [4]. In general, all images projected the retinal surface are affected,
for instance, by the pupil aperture and optical variation such as aberrations [7].
A more general form of PSF that also depends on aperture system, the transmit-
tance of the pupil or wave aberration function can be expressed as follows
psf (x, λ, a(z, λ), W (z, λ, ΔD)).
The terms a(z, λ) and W (z, λ, ΔD) specify the effect of the aperture system and
the wave aberration function, correspondingly. The term z is the position in the
plane of aperture. In the human eye, the aperture is characterized by the shape,
size and transmittance of the pupil [6]. The wave aberration function describes the
other aberrations (degradation in image quality) introduced by the lens system not
attributable to diffraction, such as defocusing [8].
Neural Response
Other visual eye condition that affects the retinal response is the non-uniform density
of the photoreceptors at the retinal surface. Thibos and Bradley [9] defined the Neural
point spread function (NPSF) as the neural image formed by an array of neurons in
response to a point of light on the retina. As seen in Figs. 2.5 and 2.6 of Sect. 2.2,
the density of the photoreceptors on the retina varies with its eccentricity; density
is high at the fovea that is the region on the center of the retina and is low in the
peripheral regions. Then, the response of the retinal surface can be expressed as the
convolution of projected image and the NPSF,
I(x, λ) ∗ npsf (x, λ, ε). (4.15)
The NPSF is computed for each point x and its wavelength; ε denotes the eccentricity
of the retinal surface.
The NPSF is also affected by the phenomenon named Stiles-Crawford effect [4].
The light entering the eye near the edge of the pupil produces a lower photoreceptor
response than the light of equal intensity entering near to the center of the pupil.
Then, light reaching a point on the retina through the center of the pupil is more
visually than the same light reaching the same point but through the edge of the
pupil. It makes the retina directionally sensitive and can be described by a parabolic
function as follows,
log η = log ηmax − ρ(d − dmax )2
where η is the sensitivity at d distance from the center of the pupil, and ρ represents
the magnitude of the Stiles-Crawford effect. The effect decreases of the brightness
of a light stimulus with increasing eccentricity of the position of entry of the light
pencil through the pupil.
Spatial Response
Due to the fact that optical and neural characteristics of visual systems affects the
projected image of objects, either focused or non-focused, the spatial response of
the retina can be expressed as the convolution of idealized image and the PSF, and
NPSF;
r(x, λ, ΔD, ε) = I(x, λ) ∗ psf (x, λ, ΔD) ∗ npsf (x, λ, ε), (4.16)
where x, λ, ΔD and ε are a point of the projected image, its wavelength, the optical
power required to bring it focus and their eccentricity, respectively [6, 9].
Spatial-Temporal Response
When the projection of an object is changing in time, the neural response will also
depends on time [10]. Then, the projected image I(x, λ) in a time t can be expressed as
I(x, λ, t). In the particular case, in which the object is given by a flickering stimulus,
all points of a light stimulus is oscillating periodically due to the fact that it is
connected to a function generator. In this case, the projected image can be expressed
as the product of every points of the projected image with sinusoidal function,
I(x, λ, t) = I(x, λ) cos(ω0 t), (4.17)
where ω0 = 2π f0 , and f0 is the flickering frequency of the light stimulus. Then, the
spatio-temporal of the retinal response due to a flickering stimulus can be denoted
as
r(x, λ, ΔD, ε, t, ω0 ) = [I(x, λ) cos(ω0 t)] ∗ psf (.) ∗ npsf (.), (4.18)
where x, λ, ΔD, ε, ω0 and t represents the points of the stimulus, the wavelength,
the power required to bring the object in focus, the eccentricity of the retinal surface,
the flickering frequency and the time, respectively.
EEG Signal at Cortex V1
It was shown in Sect. 2.2 that there are regions at visual cortex related to regions
of the retinal surface. For instance, the activity of the cortex V1 corresponds to the
fovea (Fig. 2.8), after passing through the visual pathway (Fig. 2.9). In this sense, the
response of the visual cortex is associated to x, λ, ΔD, ε, ω0 and t can be expressed
as
4.3 Spatio-Temporal Retinal Response for a Flickering Stimulus 55
rv (x, λ, ΔD, ε, ω0 , t). (4.19)
This is a very relevant expression because the evoked response rv that arise in
response to an external light stimulus, depends on its flickering frequency, its color,
its shape, its blurriness degree and the retinal surface where it is projected. Then, the
EEG signal is composed by an evoked signal and the spontaneous background EEG,
s(t) = rv + ξ, (4.20)
as indicated by [11]. Note that rv does not include the physiological parameters due
to the visual pathway and the skull. Therefore, the temporal response registered at
occipital region can be summarized as
s(t) = s(x, λ, ΔD, ε, ω0 , t, ξ ). (4.21)
4.4 Power Spectrum of the Retinal Response
For an Isolated Stimulus

Figure 4.4a represents a subject attending a stimulus flickering at f1 and placed at
distance d0 ; and Fig. 4.4b illustrates the projection of the stimulus on the retinal sur-
face give by the gray rectangle. Gazed objects are projected onto the fovea, which is
the central region of the retina possesing high density of photoreceptors. Concentric
circles are used to indicate the distance from the center of the surface to the periph-
eral region. The dark gray circle represents the blind point due to the optic nerve
(Fig. 2.7 of Chap. 2). The curve of density of photoreceptors was plotted below the
fovea depiction to indicate that the center of the retinal surface has high density of
photoreceptors compared to the peripheral regions, and there is no photoreceptors in
the blind point (Fig. 2.5 of Chap. 2).
As shown in Fig. 4.4b, an image will excite a high number of photoreceptors when
is projected on the center of the retina, corresponding to a null eccentricity (ε = 0).
Otherwise, when the image is projected out of the center (ε = 0) a low number of
photoreceptors will be excited, then showing the fundamental role of the eccentricity
in the neural response. In addition, when a stimulus is not projected on the fovea, a
change of orientation Δθ is required to bring the stimulus to the center of the field of
view in order to obtain a better neural response. Then, the response s(t) of Eq. (4.20)
can be expressed as
s(t) = s(x, λ, ΔD, ε(Δθ ), ω0 , t, ξ ). (4.22)
Its power spectrum in steady-state can be computed as
Pss (ω) = S(ω)S ∗ (ω), (4.23)

(b)
(a)
Fig. 4.4 Image projection due to an isolated stimulus. a Setup with an stimulus flickering at f0 .
b Retinal projection of the isolated stimulus together with the curve of density of photoreceptors.
Gray rectangle and the central circle represent the stimulus and the fovea, respectively. Note that
the density of photoreceptors is higher in the fovea, and that there is no photoreceptors in the blind
point
where S(ω) is the Fourier transform of s(t) computed with a temporal window w(t);
and S ∗ (w) is the complex conjugate of S(w). Then, the amplitude of the power
spectrum for an isolated stimulus flickering at ω0 is given by
Pss (ω0 ) = Pss (x, λ, ΔD, ε(Δθ ), ξ ), (4.24)
which achieves the maximum amplitude when Δθ = 0.

For Two Stimuli in the Context of Traditional SSVEP Systems
Figure 4.5a shows a conventional SSVEP with two stimuli placed in front of the user
at a same distance (d0 ), with orientations θ1 and θ2 and flickering at frequencies f1 and
f2 , respectively. In that setup, the separation between stimulus plays an important role
because if it is greater than horizontal field of view (θ1 + θ2 > 120o for monocular
vision), the demodulation problem is reduced to the case discussed in Sect. 4.3 for an
isolated stimulus. This occurs because only the attended stimulus will be projected
onto the retinal surface, as shown in Fig. 4.4. Notwithstanding, if the separation is
lower than 120o , the non-attended stimulus is projected on the peripheral region of
the retina, while the attended one will be projected onto the center of the field of
view. Figure 4.5b illustrates this situation for cases in which both stimuli are separated
20o . When the stimulus flickering at f1 is gazed and projected on the center of the
retinal surface (θ = 0), the non-attended one is projected in the peripheral region;
consequently, the number of photoreceptors excited by the gazed stimulus is higher
than the non-gazed one. Therefore, following the Eq. (4.24), the power spectrum of
4.4 Power Spectrum of the Retinal Response 57
(b)
(a)
f1
f2
d0 2 d0
Fig. 4.5 Image projection of two stimuli placed in the traditional setup way when stimulus flickering
at f1 is gazed. a Traditional setup with two stimuli flickering at f1 and f2 frequencies placed at the
same distance (d0 ) of the user and separated θ1 + θ2 degrees. b Retinal projection of both stimuli.
Gazed and non-gazed stimuli are projected in the center and in the peripheral region of the retina,
respectively
(b)
(a)
f1
f2
d0 2 d0
Fig. 4.6 Image projection of two stimuli placed in the traditional setup way when stimulus flickering
at f2 was gazed. a Traditional setup with two stimuli flickering at f1 and f2 , frequencies placed at the
same distance (d0 ) of the user and separated θ1 + θ2 degrees. b Retinal projection of both stimuli.
Gazed and non-gazed stimuli are projected in the center and in the peripheral region of the retina,
respectively
gazed stimulus is given by
Pss (ω1 ) = Pss (x, λ, ΔD, ε(Δθ = 0), ξ ), (4.25)
were ω1 = 2π f1 . And the power spectrum of the non-gazed stimulus is

(b)
(a)
f2
f1
d1 d2
0
Fig. 4.7 Image projection of two stimuli placed in the novel setup way when stimulus flickering at
f1 was focused on. a Novel setup with two stimuli flickering at f1 and f2 frequencies placed at the
same angle (θ0◦ ) and at different distances d1 and d2 . b Retinal projection of both stimuli. Focused
and non-focused stimuli are projected in the center of the retina, sharp and blurred, respectively
(b)
(a)
f2
f1
d1 d2
0
Fig. 4.8 Image projection of two stimuli placed in the novel setup way when stimulus flickering at
f2 was focused on. a Novel setup with two stimuli flickering at f1 and f2 frequencies placed at the
same angle (θ0◦ ) and at different distances d1 and d2 . b Retinal projection of both stimuli. Focused
and non-focused stimuli are projected in the center of the retina, sharp and blurred, respectively
Pss (ω2 ) = Pss (x, λ, ΔD, ε(Δθ = θ1 + θ2 ), ξ ), (4.26)
were ω2 = 2π f2 . Due to the fact that both attended and non-attended stimuli have
the same shape, the same color and are placed at the same distance; x, λ, and ΔD are
the same for Eqs. (4.25) and (4.26). In consequence, the power spectrum principally
depends on the eccentricity; since the maximum value is achieved when ε(0), then,
Pss (x, λ, ΔD, ε(0), ξ ) > Pss (x, λ, ΔD, ε(θ1 + θ2 ), ξ ). (4.27)
4.4 Power Spectrum of the Retinal Response 59
Note the high SNR of the evoked response [1]; then, it is assumed that the sponta-
neous EEG (ξ ) should not affect significantly the measured response. Finally, the
power spectrum of target stimulus flickering at f1 should be higher than the stimulus
flickering at f2
Pss (ω1 ) > Pss (ω2 ), (4.28)
where ω1 = 2π f1 and ω2 = 2π f2 . The same analysis is true when the target stimulus
is flickering at f2 , what is shown in Fig. 4.6a, b.
For Two Stimuli in the Context of the Novel SSVEP Setup
Figure 4.7a shows the novel setup with two stimuli flickering at f1 and f2 and placed in
front of the user so that they are projected together in the center of the retinal surface.
To avoid the dependence of large muscular movements, they are placed conveniently
in the same line of view with the same θ and at different distances from the subject’s
eye (d1 and d2 , respectively). In this setup, the separation between stimuli plays an
important role, because the defocusing degree is related to this distance, as shown in
Sect. 3.4. Figure 4.7b shows the projection of both stimuli on the retinal surface, in
which the the projection of the focused stimulus that is flickering at f1 is represented
sharply and the non-focused one is represented blurry.
As shown in Sect. 3.3, every point of a non-focused stimulus are affected by the
spreading function that project them as a blurry circles; so that the images of non-
focused objects are projected with an expanded area. Nonetheless, although the area
increases, the amount of light covering the photoreceptors of retinal surface remains
the same. Recall that refraction of light causes the change of direction of the photons
[12]. Also, the luminous flux of both stimuli were adjusted to an equal and constant
value of lumens (Fig. 3.1); then, the projected image of the defocused one has lower
intensity. It can be seen in Fig. 4.7b, in which the number of photoreceptors covered
by the focused and non-focused stimuli are different, because the light spreading
would cause variation of the neural response with the eccentricity. Due to the fact
that the optical power required ΔD to bring an object in focus depends on its distance
to the current point of focus, ε(ΔD) can be expressed as ε(Δd).
When the stimulus flickering at f1 is focused on, it is projected sharply in the
center of the retinal surface and the number of photo-receptors excited is higher than
the non-focused stimuli; therefore, the power spectrum of the focused stimulus is
given by
Pss (ω1 ) = Pss (x, λ, Δ1 D, Δθ1 , ε(Δd = 0), ξ ), (4.29)
where ω1 = 2π f1 . And the power spectrum of the non-focused stimulus is
Pss (ω2 ) = Pss (x, λ, Δ2 D, Δθ2 , ε(Δd = d2 − d1 ), ξ ), (4.30)
where ω2 = 2π f2 . Due to the fact that both focused and non-focused stimuli have the
same shape, the same color and are placed at the same line; x, λ, and θ are the same
for Eqs. (4.29) and (4.30). In consequence, the power spectrum principally depends
on the eccentricity; since the maximum value is achieved when ε(0), then,
Pss (x, λ, ΔD, ε(0), ξ ) > Pss (x, λ, ΔD, ε(d2 − d1 ), ξ ). (4.31)
Assuming that the spontaneous EEG (ξ ) should not affect significantly the response
due to high SNR of the evoked response [1]. Finally, the power spectrum of target
stimulus flickering at f1 is higher than the stimulus flickering at f2
Pss (ω1 ) > Pss (ω2 ), (4.32)
where ω1 = 2π f1 and ω2 = 2π f2 . This inequality can be supported by the fact that a

different number of photoreceptors are excited by focused and non-focused stimuli.
In addition, the activity of photoreceptors specialized in contrast detection is lower
because blurred images have poor contrast [13]. The same analysis is true when the
focused stimulus is flickering at f2 , showed in Fig. 4.8a, b.
4.5 Experimental Evaluation
As seen in Eq. (4.32), the amplitude of the power spectrum of the retinal response at
the frequency of the focused stimulus is higher than the amplitude at the non-focused
one. Then, the experimental evaluation of the amplitude of the power spectrum
responses due to the focused and non-focused stimuli in the context of the novel
setup is conducted in this section.
Subjects
Four healthy subjects participated in the study. All of them showed normal or cor-
rected to normal vision. The experiments were undertaken with the understanding
and written consent of the subjects. This study was approved by the Ethics Committee
of the Federal University of Espirito Santo.
Signals
EEG signals were recorded with a device for clinical purposes BrainNet36 together
with a cap of integrated electrodes positioned according the international 10/10 sys-
tem. Passive electrodes at locations P1, P2, P3, P4, Pz, PO3, PO4, PO7, PO8, POz,
O1, O2, and Oz (Fig. 2.11b), and referenced to Cz were employed. The ground elec-
trode was placed on the AFz position. All signals were recorded with a sampling
frequency of 200 Hz.
Protocol
Figure 4.9 shows the protocol of an experiment. Painted rectangles represent stimuli
on, while unpainted rectangles corresponds to stimuli off, f1 and f2 denote the flick-
ering frequencies, and d1 and d2 denote the distances. The black arrow indicates the
4.5 Experimental Evaluation 61
Fig. 4.9 Experimental procedure to evaluate the inequality of visual evoked response due to defo-
cusing. Gray and white rectangles represent stimuli turned on and turned off, respectively. The
arrow indicates which stimuli was being focused
target stimulus. In this protocol both stimuli are turned on sequentially while the user
keeps focusing at just one of them. The first stimulus was turned on and subjects
were asked to focus on it for 7s. Then, the second stimulus was then also turned on,
while subjects maintained their focus on the first one during other 7s. To preserve
the same conditions, signals of responses with isolated and focused stimulus were
recorded in the same trial.
Results
Figures 4.10 and 4.11 show the spectral responses averaged over thirty trials of an
isolated stimulus (black dotted curve) together with the response of two stimuli when
one of them was focused (gray curve). Figure 4.10a shows the electrodes employed
in experiments, and Fig. 4.10b shows responses for each electrode when the target
stimulus was flickering at 6.4 Hz. Black dotted curves, labeled as N6Fx are the
responses obtained from signals recorded when only the nearest stimulus, which
was flickering at 6.4 Hz was turned on, and the farthest one was off, as shown in
Fig. 4.4a. As expected in traditional systems that presents isolated stimulus, it can be
seen strong peaks at 6.4 Hz in all electrodes, which are higher in occipital electrodes
(O1, O2 and Oz) than in parietal electrodes, specifically P1, P2, P3 and P4. On the
other hand, gray curves that were labeled as N6F5 are the responses obtained from
signals recorded when the nearest stimulus flickering at 6.4 Hz was focused on, and
the farthest stimulus flickering at 5.6 Hz was non-focused, as shown in Fig. 4.7a.
Strong peaks can be observed at 6.4 Hz in all electrodes and they are higher at
occipital electrodes (O1, O2 and Oz). Also it can be seen that the amplitude at the
frequency of the non-focused stimulus is small. It can be perceived at electrodes O2,
POz, PO4, PO8, P2 and P4.
(a)
(b)
Normalized Power
P3 P1 Pz P2 P4
0.01 0.01 0.01 0.01 0.01
0.005 0.005 0.005 0.005 0.005
0 0 0 0 0
5.6 6.4 5.6 6.4 5.6 6.4 5.6 6.4 5.6 6.4
Normalized Power
PO7 PO3 POz PO4 PO8

0.01 0.01 0.01 0.01 0.01
0.005 0.005 0.005 0.005 0.005
0 0 0 0 0
5.6 6.4 5.6 6.4 5.6 6.4 5.6 6.4 5.6 6.4
O1 Oz O2
0.01 0.01 0.01 N6F5
0.005 0.005 0.005 N6Fx
0 0 0
5.6 6.4 5.6 6.4 5.6 6.4
frequency Hz frequency Hz frequency Hz
Fig. 4.10 Spectral response of occipital-parietal for NxF5 and N6F5 tasks. a International 10/10
system; black circles are the P1, P2, P3, P4, Pz, PO3, PO4, PO7, PO8, POz, O1, O2 and Oz positions,
and gray circles are the A1, A2, Cz and AFz positions. b Spectral response of occipital-parietal
electrodes for isolated stimulus (black curves) and for focused stimulus (gray curves). Frequency
of nearest and farthest stimuli at 6.4 and 5.6 Hz. N6Fx and N6F5 tasks
Figure 4.11a shows the electrodes employed in experiments, and Fig. 4.11b shows
responses for each electrode when the target stimulus was flickering at 5.6 Hz. Black
dotted curves, labeled as NxF5 are the responses obtained from signals recorded
when only the farthest stimulus that was flickering at 5.6 Hz and the nearest one was
off. As expected, it can be seen strong peaks at 5.6 Hz in all electrodes, which are
higher in occipital and central electrodes (O1, O2, Oz, POz and Pz) than in parietal
electrodes, specially P1 and P3. On the other hand, gray curves that were labeled as
N6F5 are the responses obtained from signals recorded when the farthest stimulus
flickering at 5.6 Hz was focused on, and the nearest stimulus flickering at 6.4 Hz was
non-focused, as shown in Fig. 4.8a. Strong peaks can be observed at 5.6 Hz in all
4.5 Experimental Evaluation 63
(a)
(b)
Normalized Power
P3 P1 Pz P2 P4
0.01 0.01 0.01 0.01 0.01
0.005 0.005 0.005 0.005 0.005
0 0 0 0 0
5.6 6.4 5.6 6.4 5.6 6.4 5.6 6.4 5.6 6.4
Normalized Power
PO7 PO3 POz PO4 PO8

0.01 0.01 0.01 0.01 0.01
0.005 0.005 0.005 0.005 0.005
0 0 0 0 0
5.6 6.4 5.6 6.4 5.6 6.4 5.6 6.4 5.6 6.4
O1 Oz O2
0.02 0.02 0.02
N6F5
0.01 0.01 0.01
NxF5
0 0 0
5.6 6.4 5.6 6.4 5.6 6.4
frequency Hz frequency Hz frequency Hz
Fig. 4.11 a Extended 10/20 international system; black circles are the P1, P2, P3, P4, Pz, PO3,
PO4, PO7, PO8, POz, O1, O2 and Oz positions, and gray circles are the A1, A2, Cz and AFz
positions. b Spectral responses of occipital-parietal electrodes for isolated stimulus (black curves)
and for focused stimulus (gray curves). Frequency of nearest and farthest stimuli at 6.4 and 5.6 Hz
(NxF5 and N6F5 tasks)
electrodes and they are higher at electrodes O1, O2, Oz, PO3, PO4 and POz. Unlike
results showed in Fig. 4.10b, in which peaks at the frequency of the non-focused
stimulus are too small, in this case distinguishable peaks are observed at 6.4 Hz
in all electrodes. Peaks due to the non-focused stimulus are smaller at occipital
electrodes (O1, O2 and Oz) and are higher at electrodes PO3, PO4, POz, and Pz.
Peaks due to the focused stimulus are weak at parietal regions, while peaks due to
the non-focused stimulus are weak in both, occipital and parietal regions. Hence,
evoked amplitudes of focused and non-focused stimulus measured in occipital area
are clearly distinguishable (p < 0.01), whereas evoked amplitudes of focused and
non-focused stimulus measured in parietal area are quite similar (p < 0.05).
4.6 Discussion
Strong peaks were observed at the frequency of the focused stimulus in all electrodes.
Likewise, weak peaks were observed at the frequency of the non-focused stimulus in
some electrodes. Also, it can be seen that the responses at frequencies of non-focused
stimuli were higher when these were placed nearest than farthest, particularly in pari-
etal electrodes. Although in parietal electrodes, responses at frequencies of focused
and non-focused are quite similar (p < 0.05); in other brain regions, specially in the
occipital region, the response of focused was high (p < 0.01).
These results are very important because enable those modulation/demodulation
methods proposed in Eqs. (4.10) and (4.12) to be used in the novel setup. On the other
hand, results endorse the analysis of the retinal response conducted in Sect. 4.4, which
discussed over the difference in amplitude of focused and non-focused stimuli, lead-
ing to a conclusion that the amplitude responses of the attended stimulus should be
higher than the amplitude responses of the non-focused one, as shown in Eqs. (4.28)
and (4.32) for the traditional and the novel setups, respectively. Furthermore, it has
been seen that responses of both conventional and novel setups are associated directly
with the eccentricity of the density of the retinal surface. In conventional SSVEP-
BCI setup the neck, head and/or eyeball movements, characterized by Δθ , bring and
place the target stimulus to the region with maximum density of photoreceptors. On
the other hand, the response in the novel setup depends directly of blurred projection,
which in turn depends on the distance between stimuli. The purpose of the focusing
mechanism is to collimate the light rays of the focused stimulus in direction of the
center of the retinal surface, while the rays of non-focused stimulus are spread, what
is characterized by the defocusing degree ΔD.
The analysis of the retinal response for two stimuli in traditional SSVEP setups
performed in Sect. 4.4 can be extended or generalized for system with a greater
number of stimuli. Due to the fact that density of photo-receptors is high in the
center of the retinal surface, only the gazed stimulus will be projected in this region,
while non-gazed stimuli will be projected out, as show in Figs. 4.5 and 4.6. It can be
one of the reasons why systems with a great number of stimuli are being developed in
the traditional way [14]. In the same manner, the demodulation method described in
Fig. 4.2 of Sect. 4.2 can be used for traditional systems in which more than one stimuli
are present in the field of view. The demodulation based on peak detection showed in
Eq. (4.12) can be employed successfully, since the amplitude of attended stimulus is
higher than the non-attended stimuli. Finally, the demodulation based on analytical
signal, Eq. (4.10), could be used in systems with more than two stimuli, N > 2. In
that case, a more general expression to compute values of ai , for i = 1, . . . , N, can
be proposed.
References 65
References
1. Wang Y, Gao X, Hong B, Gao S (2010) Practical designs of brain-computer interfaces based
on the modulation of eeg rhythms. In: Graimann B, Pfurtscheller G, Allison B (eds) Brain-
computer interfaces: revolutionizing human-computer interaction. Springer, Berlin, pp 137–
154
2. Middendorf M, McMillan G, Calhoun G, Jones K (2000) Brain-computer interfaces based on
the steady-state visual-evoked response. IEEE Trans Neural Syst Rehabil Eng 8(2):211–214
3. Muller-Putz GR, Scherer R, Brauneis C, Pfurtscheller G (2005) Steady-state visual evoked
potential (SSVEP)-based communication: impact of harmonic frequency components. J Neural
Eng 2(4):123
4. Packer O, Williams, DR (2003) The science of color. In: Light, the retinal image, and photore-
ceptors, 2 edn. Elsevier Science Ltd, Amsterdam, pp 41–102
5. Herman IP (2007) Physics of the human body. Springer, Berlin
6. Burge J, Geisler WS (2011) Optimal defocus estimation in individual natural images. Proc Natl
Acad Sci 108(40):16,849–16,854
7. Mahajan VN (1991) Aberration theory made simple. Spie Optical Engineering Press, Wash-
ington
8. Dai GM (2008) Wavefront optics for vision correction. SPIE, Washington
9. Thibos LN, Bradley A (1995) Modeling off-axis vision ii: the effect of spatial filtering and
sampling by retinal neurons. In: Peli E (ed) Vision models for target detection and recognition:
in memory of Arthur Menendez. World Scientific Publishing Co., Singapore
10. Mallot HA (2013) Computational neuroscience: a first course. Springer, Switzerland
11. Infantosi A, Lazarev V, De Campos D (2005) Detecting responses to intermittent photic stim-
ulation in the electroencephalogram using the spectral f test. Braz J Biomed Eng 21(1):25–36
12. Chartier G (2005) Introduction to optics. Springer, Grenoble
14. Yin E, Zhou Z, Jiang J, Yu Y, Hu D (2014) A dynamically optimized SSVEP brain-computer
interface (BCI) speller. IEEE Trans Biomed Eng 62(6):1447–1456
Chapter 5
Offline Evaluation of Command Detection
In Chap. 4, the spatial-temporal spectral response caused by a focused stimulus was

studied, even if a non-focused stimulus is also present in the field of view. Exper-
iments conducted in this direction showed that strong peaks were observed at the
frequency of the focused stimulus, particularly in occipital positions. And, only in
some cases, when focused stimulus was the farthest one, weak peaks were observed
at the frequency of non-focused stimulus in parietal positions.
In the present chapter, the detection of commands associated to focus shifting is
evaluated. Evaluation is conducted in offline mode to find out the possibilities of
employing a SSVEP-BCI based on Depth-of-field in practical situations. Existing
BCI performance metrics were considered in the evaluation [1] such as the accuracy
of the detection, sensitivity and specificity rates, Cohen’s Kappa coefficient, and the
ITR. The command detection was carried out by using well-known SSVEP detection
methods.
5.1 SSVEP Detection Methods
The BCI output is a control signal that is generated by extracting the embedded
information with a demodulation process. In SSVEP-BCI systems, the frequency
modulation of brain signals arises as a reaction of a task-related external stimulus.
In the present chapter the detection of the frequency stimulus denoted by fi (for
i = 1, 2) was performed using EEG signals of three channels (Nc = 3). Detection
methods were computed at fundamental and one harmonic frequencies (Nh = 2).
Three standard detection methods were employed; Canonical correlation analysis
(CCA), Least absolute shrinkage and selection operator (LASSO), and Power spectral
density analysis (PSDA). These methods are briefly described below.

DOI 10.1007/978-3-319-52298-2_5
68 5 Offline Evaluation of Command Detection
Canonical Correlation Analysis

In this method, the SSVEP frequency is identified by finding the maximal correla-
tion between multichannel EEG signals and predefined sinusoidal reference signals
associated with each flickering frequency [2, 3]. It aims to find a maximal correlation
coefficient between two multivariate time series. Hence, given the modulated EEG
signal and the reference signal, expressed as multidimensional random variables X
and Y , respectively, CCA finds the weight vectors Wx and Wy that maximize the
correlation ρ between x and y by solving
E[x T y]
max ρ(x, y) = ,
Wx ,Wy E[x T x]E[yT y]
where x = X T Wx and y = Y T Wy are the linear combination of weight vectors

with the random variables X and Y ; and E[.] is the expected value. The canonical
correlation is given by the maximum value of ρ with respect to Wx and Wy . The
reference signal Y of the i-th stimulus frequency is given by
⎛ ⎞
sin(2π fi t)
⎜ cos(2π fi t) ⎟
Yfi = ⎜ ⎟
⎝ sin(2π Nh fi t) ⎠ .
cos(2π Nh fi t)
Fig. 5.1 Layout for frequency detection by using CCA method. Frequencies of the signal generators
connected to the squares oscillators are used to generate reference sinusoidal waveforms y(t). Then,
reference waveforms are correlated with the EEG signals that contains SSVEP pattern evoked by
the focused stimulus x(t). Finally, the results of correlation blocks are compared
5.1 SSVEP Detection Methods 69
The user’s command is detected by comparing ρ1 and ρ2 that arise as a result of

canonical correlation of X with Yf 1 and Yf 2 , respectively; as follows,
C = sgn(ρ2 − ρ1 ), (5.1)
where sgn denotes the sign function. Frequency detection with CCA method is
summarized in Fig. 5.1.
Least Absolute Shrinkage and Selector Operator
This method based on sparse regression identifies the SSVEP frequency by expressing
the modulated EEG signal X and the set of reference signals Y = [Yf 1 , Yf 2 , . . . , Yfi ]
as a linear regression model X = Y β + ε, in which β represents a noise vector with
zero-mean and unit-variance, and ε represents an additive noise [4]. Thus, β can be
estimated as
β̂ = arg min x − Y β22 + λ β1 ,

β
where .n denotes ln norm, λ is a penalty parameter which controls the sparsity
of solution β̂. The contribution degree (CD) of the i-th stimulus frequency and its
harmonic to the EEG signal can be calculated as
Nc 2Nh j
j=1 h=1 βi,h
CDi = ,
Nc
where Nc = 3 is the number of channels, and Nh is the number of harmonic fre-

quencies. The user’s command corresponding to the maximal contribution degree is
recognized by comparing CD1 and CD2 , as follows
C = sgn(CD2 − CD1 ), (5.2)
where sgn denotes the sign function.

Power Spectral Density Analysis
In this method, the power spectral density is estimated from EEG signals within a
time window by FFT, and its peak is detected to recognize the target stimulus [5].
The SSVEP response is enhanced for detection by computing the SNR, as
nF(f )
SNR(f ) = n/2 n/2 ,
k=s F(f + kΔf ) + k=s F(f − kΔf )
where f is the frequency, F(f ) is the power spectrum and Δf is the frequency step.
The user command is recognized by comparing the sum of the SNR of fundamental
and harmonic frequencies (Nh = 2) of the stimuli, as follows
Nh Nh
C = sgn SNR(hf2 ) − SNR(hf1 ) , (5.3)
h=1 h=1
where sgn denotes the sign function.

Detection Performance Metrics

The performance of the frequency detection was evaluated by employing the accu-
racy of the detection, the sensitivity and specificity rates, and the Cohen’s Kappa
coefficient κ, which takes into account the detections occurring by chance [6]. It can
be computed as
p0 − pe
κ= , (5.4)
1 − pe
where p0 is the proportion of observations in agreement (or accuracy) and pe is the

proportion in agreement due to chance. Furthermore, the ITR that is an objective
measure of BCI performance was computed as follows

1 − p0
ITR = s log2 (N) + p0 log2 (p0 ) + (1 − p0 ) log2 , (5.5)
N −1
where N is the number of commands and s is the commands sent per minute [5, 7].
5.2 Experimental Procedure
Subjects
Eight healthy subjects participated in the study hereinafter labeled from S1 to S8. Four
usually use vision correction glasses (S2, S3, S4, and S8) and two had prior experience
with BCI (S2 and S4). The experiments were undertaken with the understanding and
written consent of the subjects. This study was approved by the Ethics Committee
of the Federal University of Espirito Santo.
Signals
EEG signals were recorded with a device for clinical purposes BrainNet36 together
with a cap of integrated electrodes positioned according the international 10/20 sys-
tem. The EEG was acquired at a sampling rate Fs = 200 Hz, and the grounding
electrode was positioned on the user forehead with bi-auricular reference. Since the
occipital and parietal scalp areas have been demonstrated to contribute most to the
SSVEP pattern recognition [8] only signals from electrodes Pz, O1, and O2 were
used to identify the pattern. Common average reference (CAR) spatial filtering was
performed including signals from P3 and P4 to reduce the correlation between occip-
ital channels originated by external noise. A six order Butterworth band-pass filter
(4–45 Hz) was also applied.
Setup
For offline experiments, the nearest and the farthest stimulus was placed at
d1 ≈ 30 cm and d2 ≈ 50 cm, respectively. Due to the fact that they are presented
together in the subject’s field of view, they can be denoted as (fzn , fzf ), where fzn and
5.2 Experimental Procedure 71
Fig. 5.2 Experimental procedures for evaluating the command detection considering d1 ≈ 30 cm
and d2 ≈ 50 cm and f1 = 5.6 and f2 = 6.4 Hz
fzf denote the flickering frequencies of near and far stimuli, respectively; The target
stimulus can be denoted as f˙ ; then, (f˙zn , fzf ) and (fzn , f˙zf ) indicates which stimuli must
be focused on by the subject. The flickering frequencies were labeled as f1 = 5.6 Hz
and f2 = 6.4 Hz.
Protocol
The experimental procedure was composed by two experiments. In the Experiment
1, the stimuli were set as (fzn , fzf ) = (f1 , f2 ), as shown in left inset of Fig. 5.2. The
experiment was composed of forty 10 s trials; 3 s for resting and 7 s performing a
task. In each task, subjects were asked to focus the nearest stimulus or the farthest
one. The order in which the cue was assigned to the two stimuli was random. The
signal acquisition was performed during four seconds between the 3rd and the 7th
second.
In the Experiment 2, the stimuli were set as (fzn , fzf ) = (f2 , f1 ), as shown in right
inset of Fig. 5.2. The experimental protocol was the same as the Experiment 1.
5.3 Results
Figures 5.3 and 5.4 show the results of Experiments 1 and Experiment 2, respectively.
Figures 5.3a and 5.4a show the average of the normalized spectral response. Curves
were plotted by using signals of ten trials recorded at positions Pz, O1, and O2
for subjects S2, S4 and S7. Curves of the left and right columns are the responses
related to the Task 1 (focus on the nearest stimulus) and Task 2 (focus on the farthest
(b)
(a)
Fig. 5.3 Results of Experiment 1. a Average of normalized spectral response of users S2, S4 and
S7 computed from electrodes Pz, O1, and O2 in ten firsts trials. Responses of task 1 (left) and
response of task 2 (right); and b Accuracy rate of detection SSVEP response for all subjects and
their average by using CCA, LASSO, and PSDA methods for TW = 1, 2, 3, 4 s
stimulus), respectively. Note that a distinguishable SSVEP pattern was elicited by the
target stimulus in the two tasks, regardless both stimuli were present in the subject’s
field of vision. In both experiments, it can be observed that (i) the pattern is observed
clearly in the fundamental frequency for electrodes O1 and O2; (ii) the SSVEP
response when near stimulus was in focus (f˙zn , fzf ) is stronger than the response of
far stimulus in O1 and O2; and (iii) there are responses in the second harmonic of
the frequency of the target stimulus, specially when it is f1 = 6.5 Hz.
Figures 5.3b and 5.4b depict the classification accuracy for all subjects and the
average accuracy obtained by employing the detection methods described above.
Because the length of the analysis window is an important aspect to be considered
when assuming that the background noise is a random additive signal and the potential
is deterministic, the accuracy rates were computed for four TW (1, 2, 3, 4 s). In all
methods, the detection of SSVEP response was computed taking into account the
fundamental and one harmonic frequency (Nh = 2) from 3 channels (Pz, O1, and
O2). In both experiments, it can be seen that (i) for a TW of 1 s and with at least one
detection method subjects, S2, S4 and S7 reached accuracy rates higher than 85%
in Experiments 1 and 2; (ii) Subjects S2 and S4 reached accuracy rates higher than
5.3 Results 73
(b)
(a)
Fig. 5.4 Results of Experiment 2. a Average of normalized spectral response of users S2, S4 and
S7 computed from electrodes Pz, O1, and O2 in ten firsts trials. Responses of task 1 (left) and
response of task 2 (right); and b Accuracy rate of detection SSVEP response for all subjects and
their average by using CCA, LASSO, and PSDA methods for TW = 1, 2, 3, 4 s
95%, respectively in Experiment 1 and 2, with CCA method, for a TW of 4 s; (iii)

Subjects S1, S2, S5, and S7 reached accuracy rates higher than 85% in Experiment
1, with at least one method, for a TW of 4 s; (iv) Subjects S1, S2, S4, and S7 reached
accuracy rates higher than 95% in Experiment 2, also with at least one method, for
a TW of 4 s; (v) Subjects S3, S6 and S8 reached accuracy rates higher than 85% for
CCA method, for TW = 4 s; (vi) Subject S5, who got random accuracy rates in the
first experiment, reached an accuracy rate higher than 75% for the CCA method, for
TW of 4 s.
Tables 5.1 and 5.2 show the accuracy rate, together with sensitivity rate, specificity
rate and ITR obtained in Experiments 1 and 2, respectively, for a TW of 4 s. In
both experiments, all rates including ITR, achieved high values for CCA method.
Higher average in both tables results were highlighted. Statistical Cohen’s Kappa
coefficients, that takes into account the detections occurring by chance, for CCA,
LASSO and PSDA methods are presented in Fig. 5.6 for TW of 4 s. Top and bottom
insets show results of Experiments 1 and 2, respectively. Horizontal lines at 0.6 and
0.8 were plotted to help the interpretation of the results. In both experiments, it can
be seen that the coefficients for subjects S1, S2, S3 and S7 were higher than 0.6 for all
methods. Cohen suggested the Kappa values between 0.6 and 0.8 can be interpreted
as a substantial agreement.
74
Table 5.1 Accuracy rate, sensitivity rate, specificity rate and ITR rate and their average obtained in the Experiment 1 with CCA, LASSO, PSDA methods for
all subjects (TW = 4 s)
Subject Accuracy rate Sensitivity rate Specificity rate ITR (bits/min)
CCA LASSO PSDA CCA LASSO PSDA CCA LASSO PSDA CCA LASSO PSDA
S1 0.90 0.95 0.88 0.90 1.00 0.90 0.90 0.90 0.85 7.965 10.70 6.85
S2 0.95 0.95 0.90 1.00 1.00 1.00 0.90 0.90 0.80 10.71 10.70 7.96
S3 0.90 0.78 0.70 0.95 0.75 0.50 0.85 0.80 0.90 7.97 3.46 1.78
S4 1.00 0.95 0.95 1.00 1.00 1.00 1.00 0.90 0.90 15.00 10.70 10.70
S5 0.55 0.58 0.58 0.75 0.70 0.50 0.35 0.45 0.65 0.11 0.24 0.24
S6 0.85 0.70 0.68 0.95 1.00 0.75 0.75 0.40 0.60 5.85 1.78 1.35
S7 0.93 0.90 0.88 1.00 1.00 0.95 0.85 0.80 0.80 9.24 7.96 6.84
S8 0.68 0.85 0.88 1.00 0.95 1.00 0.35 0.75 0.75 1.35 5.85 6.85
Average 0.84 0.83 0.80 0.94 0.93 0.83 0.74 0.74 0.78 7.27 6.43 5.32
5 Offline Evaluation of Command Detection
5.3 Results
Table 5.2 Accuracy rate, sensitivity rate, specificity rate and ITR rate and their average obtained in the Experiment 2 with CCA, LASSO, PSDA methods for
all subjects (TW = 4 s)
Subject Accuracy rate Sensitivity rate Specificity rate ITR (bits/min)
CCA LASSO PSDA CCA LASSO PSDA CCA LASSO PSDA CCA LASSO PSDA
S1 0.83 0.93 1.00 0.90 1.00 1.00 0.75 0.85 1.00 4.96 9.24 15.00
S2 1.00 1.00 0.93 1.00 1.00 0.85 1.00 1.00 1.00 15.00 15.00 9.23
S3 0.83 0.78 0.80 0.70 0.60 0.80 0.95 0.95 0.80 4.96 3.46 4.17
S4 0.95 0.90 0.90 0.90 0.80 0.80 1.00 1.00 1.00 10.70 7.96 7.96
S5 0.75 0.85 0.70 0.60 0.70 0.70 0.90 1.00 0.70 2.83 5.85 1.78
S6 0.85 0.80 0.53 0.90 0.95 0.85 0.80 0.65 0.20 5.85 4.17 0.27
S7 1.00 0.95 0.93 1.00 0.90 0.95 1.00 1.00 0.90 15.00 10.70 9.23
S8 0.88 0.88 0.78 0.85 0.80 0.85 0.90 0.95 0.70 6.84 6.85 3.46
Average 0.88 0.88 0.82 0.86 0.84 0.85 0.91 0.93 0.79 8.27 7.90 6.36
75
5.4 EOG Measurements
For EOG analysis, four subjects were asked to focus on the nearest stimulus during
7 s and then to shift their focus to the farthest one for 7 s, and viceversa. To avoid the
nose caused by blink artifacts, subjects were instructed to avoid blinking during 2 s,
starting 1 s before shifting stimulus and ending after 1 s. It is illustrated in Fig. 5.5 and
EOG signals were analyzed during this period. The angle of eyeball movement was
computed by interpolation. First, three angles were associated to EOG measurements;
in this sense, users were asked to move their eyeballs across a plane in their front
placed at 1.2 m from a starting point placed at 0◦ to three final points that are separated
0.3, 0.2 and 0.1 m from the starting point. By employing a tangential relation between
distance of the plane and the distances of separation of the final points to the starting
point, the movements correspond to angles 14.03◦ , 9.46◦ and 4.76◦ , respectively.
Then, a linear interpolation between angles and EOG measurements was performed
in order to determine the angle of the eyeball movement during the experiment
described in Fig. 5.5. As a result, it was measured that the average of total movement
of eyeball was 1.98◦ ± 0.8◦ . An analysis of eye movements by employing Kinect
and eye-tracker can be found in [9]. Even if eyeball movements are not required to
focus on the target stimulus, subjects usually perform involuntary movements.
Fig. 5.5 Experimental procedure to evaluate the eyeball movements by using EOG device
5.5 Discussion 77
5.5 Discussion
In this chapter, a novel stimulation setup for SSVEP-BCI based on Depth-of-field

phenomenon was evaluated. Results showed in Figs. 5.3, 5.4, and 5.6, and in
Tables 5.1 and 5.2, indicate that the SSVEP response evoked by the focused stimulus
is not greatly affected by the presence of the non-focused one. Results in Experiments
1 and 2 were quite similar. Even if the amplitude is stronger when the target stimuli is
placed nearer than non-target stimulus, the accuracy of detection of SSVEP response
was not severely affected. The average of accuracy of detection was greater than 75%
for TWs of 2, 3, 4 s. Then, by using the novel BCI stimulation setup here proposed,
the subjects are able to attend a target stimulus by shifting their focus. Due to the fact
that the target stimulus is able to elicit distinguishable evoked potential regardless
the non-focused stimulus that is also present, this assessment can be evaluated in
online applications.
The protocol was structured to fill the three stages of attention described in [10],
given by: disengage, shift, and engage the attention. First the subject was asked to
close his/her eyes to take attention away from the target stimulus, next he/she was
asked to open his/her eyes to shift the attention, and few seconds after, when his/her
attention would be engaged on the new target, EEG signals were recorded. It is worth
noting that three subjects reached accuracy rates greater than 85% for a TW of 1 s.
It is because the signal was recorded three seconds after the stimulation started.
Cohen's Kappa Coefficient

1
0.8
Coefficient
0.6
0.5
1 2 3 4 5 6 7 8
1
0.8
Coefficient
0.6
0.5
Kappa CCA
Kappa LASSO
Kappa PSDA
0
1 2 3 4 5 6 7 8
Subject
Fig. 5.6 Performance of the detection of CCA, LASSO and PSDA methods computed by using the
Cohen’s Kappa coefcient for a TW of 4 s. Coefficients with values above 0.6 and 0.8 are considered
with good and very good agreements, respectively [6]
References
1. Huggins JE, Guger C, Allison B, Anderson CW, Batista A, Brouwer AM, Brunner C, Chavar-
riaga R, Fried-Oken M, Gunduz A, Gupta D, Kbler A, Leeb R, Lotte F, Miller LE, Mller-Putz
G, Rutkowski T, Tangermann M, Thompson DE (2014) Workshops of the fifth international
brain-computer interface meeting: defining the future. Brain-Comput Interfaces 1(1):27–49
2. Lin Z, Zhang C, Wu W, Gao X (2007) Frequency recognition based on canonical correlation
analysis for ssvep-based bcis. IEEE Trans Biomed Eng 54(6):1172–1176
3. Bin G, Gao X, Wang Y, Hong B, Gao S (2009) Vep-based brain-computer interfaces: time,
frequency, and code modulations. IEEE Comput Intell Mag 4(4):22–26
4. Zhang Y, Jin J, Qing X, Wang B, Wang X (2012) Lasso based stimulus frequency recognition
model for ssvep bcis. Biomed Signal Process Control 7(2):104–111
5. Vialatte FB, Maurice M, Dauwels J, Cichocki A (2010) Steady-state visually evoked potentials:
Focus on essential paradigms and future perspectives. Progress Neurobiol 90(4):418–438
6. Japkowicz N, Shah M (2011) Eval Learn Algorithms. Cambridge University Press, New York
7. Wolpaw J, Ramoser H, McFarland D, Pfurtscheller G (1998) Eeg-based communication:
improved accuracy by response verification. IEEE Trans Rehabil Eng 6(3):326–333
8. Muller-Putz G, Eder E, Wriessnegger S, Pfurtscheller G (2008) Comparison of dft and lock-in
amplifier features and search for optimal electrode positions in ssvep-based bci. J Neurosci
Methods 168(1):174–181
9. Cotrina A, Glasgio G, Rivera H, Ferreira A, Bastos T (2015) Evaluation of eye movements
using tracking devices in context of a novel ssvep-bci setup. In: Proceedings of XII Brazilian
symposium of intelligent automation, pp 1–8. UFRN, Natal RN, Brazil
10. Posner MI, Petersen SE (1990) The attention system of the human brain. Annu Rev Neurosci
13(1):25–42
Chapter 6
Online Applications in Communication
and Control
Currently, BCIs are developing alternatives ways for augmentative communication

or control technology for patients with severe neuromuscular disorders [1]. The aim
of this kind of technology is to improve the patients quality of life that can allow them
more independence. In the present chapter, practical online applications for commu-
nication and control of the SSVEP-BCI based on Depth-of-field are presented and
described. Also, results of online pilot experiments are shown. First, the BCI was
employed in a speller system based on binary selection. Spelling is an important
application in human-computer interaction. It is part of a benchmark for commu-
nication application becoming one major challenge in BCI technologies. Next, the
SSVEP-BCI was used for controlling a robotic device for telepresence. A telepres-
ence mobile robot enable users in paralysis situation to perceive, explore or interact
with people as relatives or friends placed in other places. Since, in most of cases
patients with severe degree of paralysis are constrained to remain in bed [2, 3].
6.1 Binary Selection
Due to the fact that both stimuli of the novel stimulation unit proposed in this book are
associated to different commands, the BCI allows to perform a binary selection. BCI
users frequently employs binary selection for answer Yes/No questions. In previous
chapters, it was described and evaluated the novel stimulation setup that consider
two visual stimuli properly placed, on the center of the user’s field of view but at
different distances from him. The selection of the command is performed by focusing
on the associated stimulus, instead of performing gaze movements. As the amplitude
of visual evoked potential decreases while blurriness degree increases, the SSVEP
pattern elicited by a focused stimulus is distinguishable regardless of a non-focused
stimulus is also present.

DOI 10.1007/978-3-319-52298-2_6
80 6 Online Applications in Communication and Control
6.2 BCI for Communication—A Spelling System
Spelling systems enable people to type (or spell) a text on a computer. The speller sys-
tem based on BCI most widely used is the system designed by Farwell and Donchin
using the brain potential P300 [4]. Spelling is an alternative communication chan-
nel for people with severe motor impairment, such as patients with LIS. Spelling
systems that uses mental activity are usually composed of the BCI system, a speller
interface and can include a language prediction module [5]. It can be found spelling
systems that employs different mental activities. For instance, Blankertz and col-
leagues proposed the Hex-o-Speller [6], in which letters of English alphabet and
some punctuation symbols were separated in groups. Both, group and letter selec-
tion was performed by motor imagery. Also, a BCI speller based on traditional
SSVEP-BCI was developed at University Bremen [7].
In the speller system based on BCI and Depth-of-field here described, the letters
of English language and some punctuation symbols were organized in five groups.
In similar fashion of Hex-o-Speller, groups and letters are selected by using specific
brain activity. Figure 6.1 shows the elements of the speller. The stimulation unit
evokes particular features in the brain signals. After signals processing, features are
extracted and translated into commands. Then, the Yes/No commands associated to
both stimuli are used to type a letters and words. In general, a letter of the English
alphabet can be typed by employing the binary selection. For this end, users are
asked for a Yes/No answer when a particular letter or symbol is presented; as shown
Fig. 6.1 Speller system based on SSVEP-BCI and binary selection. Users are able to send two
commands that can be selected by focusing on one of two visual stimuli. Commands recognized by
a BCI system can type a o reject a letter
6.2 BCI for Communication—A Spelling System 81
Fig. 6.2 Relative Frequency of individual English letters

frequencies (percent) of 14
occurrence of individual 12
English letters [8]
10
0
E T A O I N S R H D L U C M F Y W G P B V K X Q J Z
in the block Speller of Fig. 6.1, if answer is No, other letter is presented. If answer
is Yes, the current letter is typed in the computer. This mechanism is repeated until
type a whole word. Letters can be presented by using a computer screen [6, 7].
Notwithstanding, in the case of use the screen for writing the spelled words, letters
can be presented audibly by using the computer speaker. In this case, the letters of
English alphabet and the punctuation symbols were organized in groups. Groups
and letters were presented audibly. The BCI user is able select or reject a group by
focusing the nearest or the farthest flickering stimulus, respectively. And, just like
the Hex-o-Speller, this procedure is repeated until the sentence is written.
Letter Selection Strategy
There are some letters that are widely used. Before grouping the letters, the relative
of frequency of individual English letters were studied. Then, letters were grouped
according to the relative frequency. After creating the groups, particular words were
used to label each group. Finally, both groups and letters were presented to the subject;
from the most frequent to the least frequent. No exact letter frequency distribution
underlies a given language, since all writers write slightly differently. Frequency
analysis is based on the fact that, in any given stretch of written language, certain
letters occur with varying frequencies. Moreover, there is a characteristic distribution
of letters that is roughly the same for almost all samples of a language [8]. For
instance, as showed in Fig. 6.2 the letters E, T, A and O are the most common in the
English language, while Z, Q and X are rare.
Taking into account the relative frequency, the letters of English alphabet were
organized in five groups, which are shown in Table 6.1. Some punctuation symbols
were included in a group, in which “.”, “,”, “_”, and “?” represent the period, the
comma, the space and the question mark, respectively. Each group was named with
particular words with no linguistic meaning. These words, placed in the first column
and labeled as Wi , are easy to pronounce and recall. Letters, also ordered by they
frequency, are in the second column. The labels of the letters are in the third column.
Figure 6.3 shows the groups of letters (circles), the words representation (rectan-
gles connected to circles), and the letters grouped and represented by the word “W2 :
Winsicott”, little rectangles. It is illustrated how a letter can be selected by employ-
ing the binary selection and using this organization. First the word W1 is dictated; if
e, t, a, i,
o, s, h,
l, u, f, n, r,
c, m, p, v, ?,
y, b w d, g ‘_’,
k, j,
x, q,
‘.’, ‘,’ z
WORD SEARCHING
Remaining
BEAUTIFLY WHIMSICOTT GROUDON VILEPLUME letters
LETTER SEARCHING
O S H C M W
Select and type

To search other letter
Fig. 6.3 Representation of mechanisms of word and letter searching. First, groups of letters (labeled
by words) are presented sequentially by dictating. After selecting a group, for instance Whimsicot,
letters are presented. Then, the selected letter is typed in a screen
user does not select it, the next word W2 is dictated. When it is selected, the system
dictates each letter of the group until the user selects the desired letter. After selection
and writing of the letter, the system presents again the sequence of words and letters.
The size of circles are related to the relative of frequency of the letters.
Figure 6.4 presents a simplified flowchart of the letter selection using binary selec-
tion. Note that stop criteria was not considered in Figs. 6.3 and 6.4. The space (_)
character can be used to separate spelled words, and the period can be used to end a
phrase.
Experimental Procedure
The BrainNet36 is a device for clinical purposes that was used in this research for
EEG acquisition together with a cap of integrated electrodes at positions according to
the international 10/20 system. The EEG was acquired at a sampling rate Fs = 200 Hz,
and the grounding electrode was positioned on the user forehead with bi-auricular
Table 6.1 Words; letters and punctuation symbols; and characters labels
Words Letters Labels

W1 : beautifly et ai l u f y b L 11 L 12 L 13 L 14 L 15 L 16 L 17 L 18 L 19

W2 : whimsicott os h cm w L 21 L 22 L 23 L 24 L 25 L 26

W3 : groudon nr d g L 31 L 32 L 33 L 34

W4 : vileplume pv.,_? L 41 L 42 L 43 L 44 L 45 L 46

remaining j x q z L 51 L 52 L 53 L 54
6.2 BCI for Communication—A Spelling System 83
Start
i=1 j=1
Play Wi Play Lij
Is this word Yes Is this letter Yes

selected? selected?
No No
i=i+1 j=j+1
Select and type
Fig. 6.4 Simplified flowchart of the BCI speller for selecting words, and selecting and typing letters
reference. Since the occipital and parietal scalp areas have been demonstrated to
contribute most to the SSVEP pattern recognition, only signals from electrodes Pz,
O1, and O2 were used to identify the pattern. CAR spatial filtering was performed
including signals from P3 and P4 to reduce the correlation between occipital channels
originated by external noise. A six order Butterworth band-pass filter (4–45 Hz)
was also applied. Groups of letters were presented audibly by using the computer
speakers. For selecting a presented letter or word (Yes), the subject was instructed to
focused on the nearest stimulus flickering at 5.6 Hz. In other case, he was instructed
to focused on the farthest stimulus flickering at 6.4 Hz.
Results of a pilot experiment are presented in this section, in which a healthy
subject was asked to spell an arbitrary word. Subject was oriented to include the
punctuation sign period (.) at the end of the word. The spelling system is programed
to synthesize the sound of the word after detecting the period.
Online Results
In this pilot experiment, the subject spelled the word cana (with period). According
to Table 6.1, this word can be represented by its sequence of labels

c a n a . = L 24 L 13 L 31 L 24 L 44 . (6.1)
For instance, fist letter of the sequence (C) labeled as L 24 , is the fourth letter of
the second group. It means that the subject must send correctly two commands to
select the group and four commands to select the word. In this sense, 6 commands
are necessary to write this letter. Consequently, the following equation can be used
Fig. 6.5 Predefined times of Dictation Stimulation Processing

dictation, recording, td tp
stabilization and processing
used in the spelling by using 1.5s 1s 4s .5s
the SSVEP-BCI based on
depth of field in synchronous te tr
mode Stabilization Recording
to compute the number of commands Ncm needed to write a word of 5 characters:

5
Ncm = i k + jk = 28; (6.2)
k=1
where i k and jk describe the positions of the group and the letter, respectively, of the
k-th character.
The BCI was used in synchronous way for sending commands. First, the letters
of groups are dictated; then, the stimulation unit is turned on and the subject is asked
to focus on the target stimulus; then, the brain signals are recorded; and finally, the
stimulation unit is turned off and the recorded signals are processed and translated
into a command. The times associated to these levels are called dictation time, sta-
bilization time, selecting time and processing time and are given by td , te , ts and t p
respectively. Figure 6.5 shows the times considered in this pilot experiment (td = 1.5,
te = 1, ts = 4, and t p = 0.5). Hence, the time per command (tc ) is given by
tc = td + te + ts + t p = 7. (6.3)
The total time is obtained by using the number of commands and the time per
command as follows
Ncm × tc = 28 × 7 = 196. (6.4)
All times was measured in seconds. Finally, after some attempts the subject managed
to write the word cana (with period) in 196 s (3 min 16 s).
6.3 BCI for Control—Control of a Telepresence Robot
A telepresence robot can be understood as an extension of user. It allows people

with severe disabilities to perceive, explore, interact with other environments. Also,
interact with other people placed in other environments. A BCI could be employed
for controlling a telepresence robot by patients with paralysis. Figure 6.6 shows a
BCI system for controlling a telepresence robot. The BCI captures and processes
the brain signals before extracting features and recognizing the command associated
to the stimulus. Next, a control command is transmitted to the mobile robot. The
robot is equipped by a video-camera in order to provide a feedback signal to the
6.3 BCI for Control—Control of a Telepresence Robot 85
Fig. 6.6 BCI for controlling of a telepresence robot. The BCI captures and processes the brain
signals before extracting features and recognizing the command associated to the stimulus. Next, a
control command is sent to the mobile robot. The robot is equipped by a video-camera in order to
provide a feedback signal to the user, that is showed in the screen placed behind the stimulus
user, which is shown in the screen placed behind the stimulation unit. Transmission
of the control signals and the feedback video can be done by using a communication
network.
The feedback information is crucial in the control of the robot navigation. Oper-
ating a robot remotely using a BCI is a complex task due to environments present
obstacles and subject to variations Then, navigation of robots can be performed by
using direct or shared control [2, 3, 9]. In direct control, the user sends the com-
mands to the robot and no autonomous mechanism is used. Hence, in telepresence
robots controlled by using BCI systems, the command originated from the user’s
brain activity is sent to the control system of the robot and it executes the task. The
feedback information is provided by the embedded camera. In shared control, the
tasks execution is the result of the BCI command and an autonomous mechanism
of the robot. The autonomous mechanism uses information of the environment by
using on-board sensors. Due to the fact that the aim of the present section is to
evaluate the feasibility of controlling a telepresence robot by SSVEP-BCI based on
Depth-of-Field, in the experiment here described, the robot was directly controlled.
Control of a Robot in a Virtual Environment
Fig. 6.7a shows the experimental scenario, in which the avatar of a volunteer rep-
resents an end-user, who is constrained to remain in bed. A robotic telepresence
platform given by a mobile robot with sensors and control unit. The robotic platform
is a virtual replication of the robot called TRON (Telepresence Robotic Navigation)
that is composed by a commercial mobile robot (Pioneer 3DX) with two wheeled
(a) (b)
Screen
(feedback)
Camera
Stimulation
unit Screen
Control
User
(avatar) unit and
batteries
Bed
Sensor
laser
Telepresence
robot
Mobile
robot
Fig. 6.7 a Simulated scenario that includes patient in paralysis situation, a telepresence robot and
a SSVEP-BCI based on depth of field. b Telepresence robot
Feedback Volunteer
End user (avatar)
video
Mechanic articulated arm Farthest stimulus Monitor of eye
movements
Nearest stimulus
Fig. 6.8 Subject, stimulation unit and navigation screen with feedback video. Avatar of a end user
in a virtual environment (left inset). Volunteer performing an experiment. An eye tracker device
was used for monitoring eyeball movements (right inset)
differential traction, proximity sensors (laser and ultrasound), a control unity (micro-
controller and a on-boar computer) and batteries [2]; it is shown in Fig. 6.7b. Also,
the platform is equipped with a webcam; a screen and a microphone that provide the
telepresence functionality.
In order to control the robot with two commands of the SSVEP-BCI based
on Depth-of-field, the mobile robot was programmed to perform two movements;
(i) “forward”, in which the robot moves forward for one meter; and (ii) “turn right”,
in which the robot turns 45◦ to its right. For instance, the robot must turn twice
for turning 90◦ to the right, and six times for turning 90◦ to the left. The volunteer
was instructed to control the robot by selecting any of the two commands based on
binary selection. Focusing on the nearest and farthest stimuli are associated to the
forwarding command and turning command, respectively.
Fig. 6.9 Virtual

environment given by a
house with four rooms and
an outdoor area. Gray dotted
lines indicates the requested
travel path. Gray squares
illustrates points distanced 3
m. Circles are points where
robot must change its
direction
The stimulation unit with two flickering stimulus is placed in front of the subject,
and the screen that shows the video recorded by the on-board camera is placed behind
the stimulation unit. Figure 6.8 shows the avatar of a end user into a virtual environ-
ment. This figure also shows a volunteer performing the control of the telepresence
robot. An eye tracker device was used for monitoring eyeball movements during the
experiment. A virtual environment (VE) was developed by using the platform Unity
3D. It is given by a common house with four rooms and an outdoor area surrounded
by a wood fence. It was included natural obstacles such as chairs, tables, doors and
trees. Figure 6.9 shows the VE of the house. The user and the robot are into the user’s
room. The VE was built so that the robot can access all the rooms as well as the
outdoor area.
Experimental Procedure
The experimental procedure described in this section was conducted with one healthy
volunteer. The volunteer was asked to control the robot for traveling by a predefined
path. The path is indicated by the dotted gray lines in Fig. 6.9, in which the starting
point is placed next to the user bed. Gray squares are points spaced by 1 m and
gray circles are point where robot would turn right. Black circle is the only point
where robot should turn to its left. To control the robot with two commands, the user
synchronously select the suitable flickering stimulus by focusing on. When the robot
begins to move, the subject begins to select the followed visual stimulus associated
to the next command. The brain signals are recorded one second after the stimulation
unit was on, during four seconds (TW = 4 s).
Brain signals were recorded by using a customized cap with fourteen electrodes
[10]. These electrodes were placed at positions CPz, P7, P8, Pz, PO3, PO4, PO5,
PO6, PO7, PO8, POz, O1, O2 and Oz; according to international 10/10 system. EEG
signals were acquired with a sample rate of 128 Hz and filtered by using a band-pass
filter (0.2-45 Hz). Also, CAR were performed for spatial filtering. Binary commands
were associated with detected frequencies. The command detection was conducted
during 4 s by executing the CCA detection method. A window of 3 s in length was
displaced 0.25 s four times; then, a command was considered detected if only if the at
least four responses are the same. After the detection, the command was transmitted
to the robot.
Online Results
Results of the pilot experiment are shown in Fig. 6.10; top and bottom insets show the
requested path and the route traveled by the robot, respectively. The starting and the
finish points are placed next to the user’s bed. The subject managed the robot in order
to achieve the finish point. Some wrong commands were sent during the navigation.
They caused the robot to get out of the predefined path. However, the subject made
the robot go back to the right path. Table 6.2 shows the number of commands sent by
the BCI during the navigation and the “ideal”number of commands that makes the
robot follow the route without detours are shown. Considering that it is demanded
one command going 1 m ahead or turning 45◦ to the right; two commands for turning
90◦ right; and six commands for turning 90◦ to the left; the robot could follow the
predefined path with 57 correct commands. However, it is difficult to achieve 100%
of accuracy in a detection system in general. In navigation, a wrong command will
demands two or more right commands to resume the route. Thus, during the execution
of the task, 80 commands were sent by the BCI system, although only 5 mistakes
Fig. 6.10 Requested task

(top inset) and performed
task (bottom inset). The
subject complete the task
with 80 commands. There
were 4 wrong commands.
The task was executed in
7 min 5 s
Table 6.2 Number of commands, expected and sent by the volunteer

Forward Turn Right Turn Left Total
Expected 37 14 6 57
Performed 46 16 18 80
Table 6.3 Number of expected commands, mistakes, time spent for complete the task, and the rate
of commands per minute
Time (s) Commands/min Errors
Expected 285 12 0
Performed 400 11.28 5
were made. Table 6.3 shows the number of mistakes, the expected and the performed
times and the commands that were sent by minute.1
References
1. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain-computer

interfaces for communication and control. Clin Neurophysiol 113(6):767–91
2. Floriano A, Longo B, Baldo G, Cotrina A, Bastos T (2015) Development of an SSVEP-BCI
to command a telepresence robot control. In: Proceedings of XII brazilian symposium of
intelligent automation. UFRN, Natal
3. Leeb R, Tonin L, Rohm M, Desideri L, Carlson T, Millan JDR (2015) Towards independence:
a BCI telepresence robot for people with severe motor disabilities. In: Proceedings of the IEEE
103(6):969–982
4. Farwell L, Donchin E (1988) Talking off the top of your head: toward a mental prosthesis
utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol 70(6):510–23
5. Cecotti H (2010) Spelling with brain-computer interface—current trends and prospects. In:
Proceedings of the 5th French conference on computational neuroscience (Neurocomp10),
pp 215–220
6. Blankertz B, Dornhege G, Krauledat M, Muller KR, Kunzmann V, Losch F, Curio G (2006)
The berlin brain-computer interface: eeg-based communication without subject training. IEEE
Trans Neural Syst Rehabil Eng 14(2):147–152
7. Valbuena D, Sugiarto I, Graser A (2008) Spelling with the bremen brain-computer interface
and the integrated SSVEP stimulator. In: Proceedings of the 4th international brain-computer
interface workshop and training course, pp 291–296
8. Bishop D (2003) Introduction to cryptography with java applets. Jones and Bartlett Publishers
9. Baldo G, Floriano A, Longo B, Bastos T (2015) Shared control strategy applied to command
of an assistive telepresence robot. In: XII Brazilian symposium of intelligent automation, pp
272–6. SBEB, Natal, Brazil
10. Longo B, Floriano A, Castillo J, Bastos T (2015) A multi-proposal mobile eeg system. In:
Proceedings of the XIV Brazilian conference of biomedical engineering. Canal 6, Uberlandia,
Brazil
1 There is a complementary video clip, in which the performance of the experiment is described. This
video is available in https://www.dropbox.com/s/cwriwmow6nscag3/DOFAnibalFinal720.mp4?

dl=0.
Chapter 7
Future Directions in Patients with Paralysis
This book describes a novel approach of human-computer interaction based on

SSVEP-BCI and Depth-of-field.
As shown in the Chap. 5, offline results present strong evidence that EEG signals,
in SSVEP-BCI systems, can be modulated by visual accommodation mechanism
instead of covert attention or eyeball movements to generate binary commands. As
shown in the Chap. 6, online applications in communication and control given by
the speller system and the control of telepresence robot system, respectively, shows
that practical systems can be implemented by using this proposal. Results of pilot
experiments indicate that it is possible to develop a BCI system with high performance
that includes high accuracy rates and high ITR.
Regarding to paralysis situation, it was reported that people with paralysis are
already being trained to use some kind of BCI systems based on P300, slow corti-
cal potential or sensorimotor rhythm [1, 2]. However, although traditional SSVEP
systems are becoming robust, they are limited to patients able to perform eye move-
ments to some degree. To overcome this limitation, SSVEP systems based on covert
attention are being developed. The novel setup based on Depth-of-field phenomenon
here described goes in that direction, as head, neck or eye movements are not need
to select a command, which is based on the accommodation mechanism. It is worth
noting that although accommodation of the lens does not require eyeball movements,
minimal movements were measured when subjects were asked to shift their focus.
The sense of sight is not affected by neurodegenerative diseases as ALS, and patients
retain minimal eye movements, including pupil and accommodation reflex mecha-
nisms; making the focus control possible, even in the end stages of these pathologies.
Then, as shown in this book, the selection by focusing achieves higher accuracy rates
than SSVEP-BCI based on covert attention, and could be employed in alternative or
augmentative communication systems for people with paralysis. Furthermore, a sys-
tem based on shifting of focus does not demand an exhausted training stage because
focusing is an optical mechanism that is used naturally by humans throughout life.
The main limitation of this proposal is given by the minimal eye movement mea-
sured during experiments. Even if the measurement has been associated with involun-
DOI 10.1007/978-3-319-52298-2_7
92 7 Future Directions in Patients with Paralysis
tary movements of the subjects, it was greater than the other assessments, described
in Table 2.1. Another limitation that is shared with the works referred in Table 2.1,
is the quantity of commands. Beside being difficult to implement a control systems,
BCI systems with two commands attain low ITR [3]. The stimulation unit here devel-
oped is small, portable, and does not require a computer screen. It can be consider an
advantage; however, it also can be considered a limitation, due to the both stimuli not
being able to be implemented on computer screens. The Depth-of-field phenomenon
only can be exploited when stimuli are placed at different distances of subjects. In
this sense, an alternative setup that uses the screen to show a stimulus together with
an external stimulus, can be proposed.
Currently, many SSVEP systems use frequencies below 15 Hz [3–9]. However,
frequencies between 5 and 25 Hz could cause fatigue in paralyzed patients [10]. In
order to attain SSVEP systems for real applications that avoid this fatigue limitation,
frequencies above 25 Hz could be used [11]. Furthermore, a brain switch based on
motor imagery can be implemented to activate a SSVEP system only when needed
to send a command and deactivate the stimuli during the resting periods [5]; then,
limiting the stimulation time. In addition, the measuring of the energy of alpha rhythm
of the frontal cortex is proposed to correlate with stress and fatigue, in order to adjust
the frequency and/or luminance of stimuli, or deactivate them [12].
Finally, it is hoped that the evidences presented in this book will enable the reader
to critique, modify and improve them so that practical applications can be proposed
that contribute to improving the quality of life of end users.
References
1. Birbaumer N, Kubler A, Ghanayim N, Hinterberger T, Perelmouter J, Kaiser J, Iversen I,

Kotchoubey B, Neumann N, Flor H (2000) The thought translation device (ttd) for completely
paralyzed patients. IEEE Trans Rehabil Eng 8(2):190–193
3. Middendorf M, McMillan G, Calhoun G, Jones K (2000) Brain-computer interfaces based on
the steady-state visual-evoked response. IEEE Trans Neural Syst Rehabil Eng 8(2):211–214
4. Gao X, Xu D, Cheng M, Gao S (2003) A bci-based environmental controller for the motion-
disabled. IEEE Trans Neural Syst Rehabil Eng 11(2):137–140
5. Pfurtscheller G, Solis-Escalante T, Ortner R, Linortner P, Muller-Putz G (2010) Self-paced
operation of an ssvep-based orthosis with and without an imagery-based brain switch: a feasi-
bility study towards a hybrid bci. IEEE Trans Neural Syst Rehabil Eng 18(4):409–414
6. Nakanashi M, Wag Y, Wang YT, Mitsukura Y, Jung TP (2014) A high-speed brain speller using
steady-state visual evoked potential. Int J Neural Syst 24(6):1–17
7. Zhang Y, Jin J, Qing X, Wang B, Wang X (2012) Lasso based stimulus frequency recognition
model for ssvep bcis. Biomed Signal Process Control 7(2):104–111
8. Hakvoort G, Reuderink B, Obbink M (2011) Comparison of psda and cca detection methods
in a ssvep-based bci-system
9. Cheng M, Gao X, Gao S, Xu D (2002) Design and implementation of a brain-computer interface
with high transfer rates. IEEE Trans Biomed Eng 49(10):1181–1186
References 93
10. Zhu D, Bieger J, Molina GG, Aarts RMR (2010) A survey of stimulation methods used in
ssvep-based bcis. Comput Intell Neurosci 2010:1–12
11. Lin Z, Zhang C, Wu W, Gao X (2007) Frequency recognition based on canonical correlation
analysis for ssvep-based bcis. IEEE Trans Biomed Eng 54(6):1172–1176
12. Cotrina A, Benevides A, Ferreira A, Bastos T, Castillo J, Menezes ML, Pereira CE (2014)
Towards an architecture of a hybrid bci based on ssvep-bci and passive-bci. In: Proceedings
of 36th annual international conference of engineering in medicine and biology society. IEEE,
Chicago IL, USA, pp 1342–1345
Index
A F
Aberrations, 39 Field of vision, 11
Accommodation, 43 Fovea, 8
Accommodation mechanism, 10
Airy disk pattern, 38
G
Amyotrophic lateral sclerosis, 1
Guillain-Barre syndrome, 1
B H
Binary selection, 79 Human-machine interaction, 1
Brain-computer interface, 2
I
Information transfer rate, 70
C International system 10/20, 17
Canonical correlation analysis, 68 Iris, 8
Central nervous system, 13
Circle of confusion, 38
L
Cohen’s Kappa coefficient, 70
Lateral geniculate nucleus, 13
Cones, 10
Least absolute shrinkage and selector oper-
Cornea, 8
ator, 69
Lens, 8
Light, 7
D Light-emitter diode, 21
Demodulation, 49 Locked-in state, 1
Depth-of-focus, 38
Duchenne muscular dystrophy, 1
M
Modulation, 48
M-pathway, 14
E
Electroencephalographic signals, 15
Electromyography signals, 1 N
Electrooculography signals, 1 Neural-point-spread-function, 53
Eye, 8

DOI 10.1007/978-3-319-52298-2
96 Index
O S
Optic disk, 8 Spatial response, 54
Optic nerve, 8 Spatial-temporal response, 54
Optical point-spread function, 38 Speller system, 79
Optical power, 39 Steady-state visual evoked potentials, 19
T
P Telepresence robot, 79
Pepth-of-field, 37
Perimeter of the eye, 11
Power spectral density analysis, 69 V
P-pathway, 14 V1, cortex, 14
Pupil, 8 Vision, 8
Visual evoked potentials, 17
Visual pathway, 13
R
Retina, 8 W
Rods, 10 Wavelength, 7

Toward Brain-Computer Interaction in Paralysis A New Approach Based On Visual Evoked Potentials and Depth-of-Field

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Toward Brain-Computer Interaction in Paralysis A New Approach Based On Visual Evoked Potentials and Depth-of-Field

Uploaded by

Copyright:

Available Formats

SPRINGER BRIEFS IN

HUMANCOMPUTER INTERAC TION

SpringerBriefs in Human-Computer Interaction

• User Experience and User Interaction Design

SpringerBriefs are published as part of Springer’s eBook collection, with

More information about this series at http://www.springer.com/series/15580

© The Author(s) 2017

Printed on acid-free paper

This Springer imprint is published by Springer Nature

Vitória, Espírito Santo, Brazil Teodiano Freire Bastos Filho

A brain–computer interface is a system that provides a direct connection between

originated by a light stimulus. Also, a literature review is dealt including recent

Sao Mateus, Brazil Anibal Cotrina

Sao Mateus, Brazil Anibal Cotrina

4.5 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

AAN American academy of neurology

Nowadays, the interaction between humans beings and machines, or Human-machine

1 Fordetails, please refer to www.alsa.org.

© The Author(s) 2017 1

A Brain-computer interface (BCI) is a system that provides an alternative chan-

described allows users to make a selection by a refocusing mechanism that in turn

is principally based on well-know medical physiology, optics, neural science and

2.1 Light, Eye and Vision

Light is a form of electromagnetic radiation emitted by the oscillation of materials

Frequency (Hz) Wavelength ( )

Frequency (Hz) Wavelength ( )

2.2 Image Formation

Retinal Flattened for weak refraction

Retinal Rounded for strong refraction

Imaging the Object

-90o -45o 0o 45o 90o

Scale: 1000 x cones/mm2 Scale: 1000 x rods/mm2

2.3 Field of Vision and Topographic Map

2.4 Visual Pathway

Left Primary Right

2.5 Brain Signals and Evoked Potentials

To superior brain regions

Form and Color 2 &3

Retinal ganglion cells Primary Visual Cortex

The internationally standardized 10/20 system is usually employed to record the

Light Ganglion cells LGN layers Visual Cortex Low-pass Scalp

Object Eye response LGN responses Cortical responses Skull effect

(a) Closed eyes (b) Nose

Fig. 2.12 Typical VEP P100

2.6 Influence of Refractive Error in VEP

With correction lenses Without correction lenses

2.7 Steady-State VEP

frequency (for patter-reversal and onset/offset pattern), contrast and luminance of

2.8 BCI Based on SSVEP

Signal Feature Feature

2.9 SSVEP-BCI and Gaze Dependence

Attended Gazed Attended Gazed Attended Gazed

1929 1973 1979 1988 1999 2000 2002

Vidal coins the Farwell and Middendorf et al.

2005 2008 2010 2012 2013 2014 2016

Allison et al. Walter et al. use Lesenfants et al.

[54] proposed a novel two-class independent SSVEP-BCI based on covert atten-

3.1 SSVEP-BCI Stimulation Setup

In traditional SSVEP-BCI systems, a set of stimuli flickering at different frequencies

© The Author(s) 2017 33

ϕh = 2 tan−1 (w L /2z n ), (3.1)

ϕv = tan−1 (h L /z n ) + tan−1 (h L /z f ), (3.2)

where h L = 18 mm and w L = 13 mm are the dimensions of the LED arrangements.