Noname manuscript No.

(will be inserted by the editor)

Fault Dependent Modeling of Gimbal Based Robotic Vision System

for Tracking Fast Moving Stimulus in a Dynamic Social Environment

Received: date / Accepted: date

Abstract In a dynamic social environment, robots need high decoupling of eye sensor from the head skele-
ton for tracking fast-moving stimulus. For the independent fast eye movement, high decoupling is achieved
by proposing gimbals system. Gimbal based independent eye movement require stabilization control for
which fault dependent mathematical modeling is also presented. The model is robust against static mass
imbalance and frictional disturbance in the gimbal system that further improve the tracking efficiency.
A detail discussion is also included on how the proposed design mimics to a human eye in physiological
structure, mechanical design, actuation control and feedback response by providing theoretical architec-
ture. The decoupling property of gimbal system makes the proposed design as the best choice for tracking
stimulus in humanoid robots. Simulation results confirm the tracking and regulation control design for
the presented robotic eye. The design and modeling could be a valuable step towards autonomous robotic
eye systems for tracking targets in a dynamics social environment.

Keywords fault dependent modeling, vision system, dynamic environment, robotic eye, decoupling,
stimulus tracking, gimbal system, stabilization control, mass imbalance.

1 Introduction

Fast eyeball movement is much needed in a swarm of robots for communication with other robots or
humans connected in the network in a dynamic environment. Tracking fast moving stimulus is also needed
to perceive more information about the target like human perceive more than 80 percent of information
via eyes [2] and also necessary for the survival of the fittest. Robotic vision systems used camera sensor
instead of eyes for tracking or such communication. Different vision strategies are adopted in the current
literature to improve the acquisition of visual perception information of robots.

For the perception of more information, researchers worked a lot on the design and
also on different types of eye movements in robots. The design of human eye made by Allah
Almighty is perfect that can perform five different types of movements. For pointing, tracking
and recognition targets it uses brain/vision control, extra-ocular muscles and eyeball. Following the natural
human eye tracking system, researchers have developed many human eyes inspired robotic active stimulus
tracking systems in the last 20 years, focusing on the above main three components. Some of the researchers
developed Artificial Intelligence (AI) based brain/vision controller i.e. tracking algorithms [3], [5], scanning
algorithms [19] and recognition algorithms [20]. The AI based vision controllers are able to train for specific
stimulus and can remember it as a human brain. Unfortunately, these algorithms need much time to track
and train due the lengthy process of pointing, tracking, focusing/recognizing and then training. Some of
the robotic communities replaced the extra-ocular muscles or actuators for the eyeball movement with
pneumatic control system [21] or by soft pulleys [22], [23], [24]. These models can only work in slow moving
stimulus due to low decoupling ratio of eyeball to head skeleton so it can’t move fast. As the Listing’s law

Address(es) of author(s) should be given

2

is the basic principle for real human eyeball movements, so the author of [25] implement the Listing’s law
for tendon-driven robot eye. The purpose is to emulate the actual saccade and smooth pursuit movements
performed by human eyes for pointing and tracking respectively. Some researchers in [26], [27], [28], [29]
tried to replace the eyeball using movable cameras but the decoupling ratio with head skeleton is very much
low. Due to the reason, movable camera sensors are unable to work fast for pointing and tracking of fast
stimulus movement. The fast eyeball movement need high decoupling ratio that helps in its synchronization
with quick commands of vision controller and good actuators. The existing models are also unable to work
during fast movement of robot like in walking, riding or sudden change of direction etc in a dynamic
environment. This means that good actuators and strong vision control, both are useless until the eyeball
is synchronized with vision control which is impossible without high decoupling ratio with head skeleton.
In the whole literature, mathematical modeling for a complete robotic vision system which will be the base
for controller design and controlled robotic eyeball movement, is not presented.

For the perception of more information researchers work a lot on the design and also on the movement of
robot eye. The problem is that we have to compromise among the attractive and optimal mechanical design,
fast eyeball movement, quality of pointing, tracking, scanning, recognition, and good actuation control. This
further mean to say that there is no such design that can combine all or some of the properties of human
eyes among the discussed properties. The basic purpose is to follow a fast-moving stimulus in a dynamic
environment. Currently a robotic vision system is needed that can point and track a fast-moving stimulus
for the acquisition of more information in order to scan and recognize it without destroying the beauty of
the physiological and mechanical design. Such a perfect design having all or some of the above-mentioned
characteristics is only possible if we completely decouple the eyeball movement from the head skeleton. The
decoupling will allow the eyeball to move fast according to the commands of vision control or CPU like
an actual human eye tracking system. For this purpose, gimbal system is the best choice for pointing and
tracking, which has the property of decoupling camera sensor from the support on which it is mounted. The
decoupling of eye sensor from robotic head skeleton using gimbal system also help in different
types of movements like bringing the stimulus in center of visual field that is called saccadec
(the fastest eye movement). Furthermore, it helps in tracking the stimulus continuously for
perception of information and recognition that is called smooth pursuit movement of eye.
Researchers work a lot in the current literature of saccadec [6] - [11] and smooth pursuit
movement [12] - [18] of robotic eye.

The first model of saccade movement is given by Young and Stark in 1963. They proposed
sampled data model describing saccade by using discrete rather than continuous control
loops. This model did not take into account the actual brain structures, because so little
was known at that time about the brain stem organization of eye movements. In 1973,
Robinson modified Youngs model by adding the premotor circuitry, which consists of medial
longitudinal fasciculus (MLF) and neural integrator (NI). Both the models were for discrete
not for continuous real time control. Bruske experimentally proved in 1997, that the Superior
Colliculus (SC) and the cerebellum are important to produce accurate saccadic movements
but the model is highly complex. In 2013, Wang designed an active vision system which can
imitate saccade and other eye movements. The saccadic movements are implemented with
open-loop controller, which ensures faster saccade eye movements compared with closed-
loop controller. The model is really an excellent research work but very complex and the
results showed that the maximum speed is 6 degree per 20 millisecond that is much slow.
In 2015, Antonelli proposed a model called Recurrent Architecture (RA). In this model, the
cerebellum is regarded as an adaptive element to learn internal model while brainstem is
regarded as a fixed-inverse model. The experimental results on robot showed that this model
is more accurate than the existing ones but less sensitive.

Smooth pursuit movement is the second type of eye movement for smooth tracking a
moving object in dynamic environment. In the control models, smooth pursuit system act
as a velocity servo controller to rotate the eyes at the same angular rate as the target.
Researchers work a lot on smooth pursuit movement as in [5-11] from different angles. The
smooth pursuit movement with maximum retinal rotational velocity and minimum error,
Robinson worked in 1986. He showed that smooth pursuit is a continuous negative feedback
system responding to the eyeball velocity, different from saccade. He tested his system for
horizontal movement using coil method. The maximum recorded velocity for tracking was 0.6
degrees per 20 milliseconds. In 1989, Deno [32] applied dynamical neural network, which helps
to unify two apparently disparate models of smooth pursuit and clarify dynamical element
organization, to the smooth pursuit system. The dynamical neural network compensated the
Title Suppressed Due to Excessive Length 3

delays from sensory input to motor response but retinal rotational speed reduced to 0.4
degrees per 20 milliseconds. In 2012, Song and Zhang proposed a binocular control model,
which is derived from neural pathway, for smooth pursuit and other eye movements. In their
smooth pursuit experiments, the maximum retinal error is less than 2.2 degrees, which can
keep the target in visual field accurately with the rotational retinal velocity of 0.6 degree per
20 milliseconds. The maximum rotational speed of of in the existing robotic vision systems is
0.6 degree per 20 milliseconds that is much slower than human eye tracking system. Human
eyeball can rotate with 20 degree per 20 milliseconds. The available gimbal system in the
market can rotate at the speed of 90 degree per 20 millisecond that is much faster then both
the real human eye and existing robotic vision systems.

Both the Saccade and Smooth Pursuit movement can be done easily by using the de-
coupling property of gimbal system that helps in rapid jump with faster rotational velocity.
Gimbal based robotic eye is better than these eyes, both in design and two the ztypes of
movements. Firstly, the design of gimbal based robotic eye is much simple than these eyes as
it consists of camera sensor, gimbal system, motors for rotating gimbals and power supply.
Secondly, the maximum angular velocity of these robotic eyes is 6 degree per 20 millisec-
onds while gimbal system can rotate at the speed of 90 degree per 20 millisecond [4]. The
decoupling property make the system highly efficient so widely used in inertial navigation,
rocket engines, missile motions, marine chromocenters, and unmanned air vehicles etc. In
these applications gimbal system have rapid, precise and stabilized response characteristics
against the body motion of the dynamic platform while it is pointing at and tracking a tar-
get. Applications of gimbal system in such fast systems of dynamic environment, motivated
the idea of gimbal based robotic eye for its interaction in dynamic real time human envi-
ronment, which is the main contribution of author. The disturbances like friction, mass imbalance
and vibration in the base or robotic head on which gimbal system is mounted, also affect the efficiency
and tracking performance. Along with these disturbances, a sensor or actuator fault can make the system
useless, so these faults must also be detected timely. If a gimbal based robotic eye is designed, that
can compensate the frictional disturbance and mass imbalances with sensor and actuator
fault detection, then the eye for robots will be the best robotic eye for tracking fast moving
stimulus. That is why these disturbances and faults must be kept in mind and due to the same reason,
fault dependent mathematical modeling for gimbal based robotic eye is presented.

The introduction of ”gimbal” instead of ”eyeball” in a humanoid robotic vision system for decoupling,
the main contribution of the author, has many major advantages over the existing designed models. (i)
Gimbal system has the property of decoupling anything that is in the inner channel from the support on
which it is mounted. Due to this property, high decoupling ratio can be achieved easily. The high decoupling
ratio of gimbal system helps the robotic eye in tracking fast stimulus in a dynamic environment using simple
camera sensors. The decoupling of an eyeball movement from the head skeleton helps a robot to track a
fast-moving stimulus without head movement as in humans. The only eyeball decoupled movement for
tracking reduces the energy consumption as well. Due to the gimbal system, a smaller size of the eye is
also possible with the least cost and high tracking performance which will increase the physiological beauty
of a robot as well. The decoupling property also help in saccade or riped jump of eyeball for
bringing stimulus in center of visual field and smooth pursuit movement for proper tracking
fast moving stimulus. (ii) The proposed theoretical architecture for feedback response, physiological
structure, and mechanical design is more mimic a human vision system than any other existing one. (iii) The
feedback mechanism or neural system response via fast electrical signal between vision control and camera
system is also designed. The proposed vision system is well suitable for combining the existing research on AI
based tracking and recognition (vision control) with research on the fast camera sensor/eyeball movement
due to the high decoupling ratio, in a dynamic social environment. This makes the proposed design more
mimic an actual human vision system. (iv) Another contribution is fault dependent mathematical modeling,
that is robust against the disturbances of static mass imbalance and friction of the gimbal system. The
mathematical modeling provide a base for its stabilization control design.

The paper is divided into the following sections: The coming section 2 is started with the problem
formulation by introducing the Lie algebra of a real human eyeball movement. After that gimbal based
robotic eye is mathematically modeled with the addition of frictional disturbance in the equation of motion.
Sensor fault is also added to the state space equation after linearization. In section 3, theoretical architecture
and tracking mechanism is given in detail that mimics the proposed design to a real human eye. Section 4 is
devoted to explain the experimental implementation procedure and finally the robust stabilization control
design is presented then the paper is concluded.
4

2 Problem Formulation

For the description of complete robotic vision system in mathematical language, we first formulate the
problem. In the robotic vision system stabilization control is accomplished by suspending the sensor in the
inner pitch channel of two-axes gimbal system. For this purpose we first define the reference frames and
notation then using transformation of inertial effect of one channel to other and derived the mathematical
equations using Euler’s angular relation. Frictional disturbance is considered in both channels and after
state space representation fault dependent modeling is also done.

2.1 Saccade and Lie Algebra of Real Human eyes

Fast movement of an eye for bringing a target into the center of the visual field is called saccade. It is well
known that eye position is restricted to a subset of those which are possible automatically. Donder’s law
states that ”the gaze direction of eye determines its orientation uniquely, and the orientation is independent
of eye motion history which leads to gaze direction”. The observed orientation of eyes is the one which can
be reached from primary position via single rotation about an axis which is in a plane perpendicular to
gaze direction. Listing law precisely specifies the allowed space of position.

The three dimension group, G = SO(3) [30] (Special Orthogonal Transformation [31]) of rotation is
used to describe both the actual rotation and unique virtual rotation from primary position. To retain the
eye at a primary position, identity operation is used so that the position is identified with the unit element
of the group e ∈ SO(3). A 3D axis and angle of rotation can parameterize a rotation. By increasing angle,
continuous set of rotations can be generated for each axis. If we take ”n” is a unit axis of rotation, then
exp(ψ.n) is a continuous one-parameter subgroup (SO(2) ⊂ SO(3)) of rotation through angle ψ in the
plane that is perpendicular to n. The calculation of exponent and explicit representation of n as a matrix
can be done using Taylor series expansion. For example, at the one parameter of a subgroup of rotation in
x − y plane, i.e. about z − axis represented of matrix case is
 
0 10
Lz = −1 0 0
0 00

Now the rotation by an angle ψ via direct computation gives

1 1
exp(ψLz ) = I + ψLz + (ψLz )2 + ... + (ψLz )n + .....
2! n!
 
cos ψ sin ψ 0
exp(ψLz ) = − sin ψ cos ψ 0
0 0 1

2.2 Mathematical Modeling

2.2.1 Reference Frames and Notations for Proposed Model

Consider a two-axes, yaw-pitch gimbal system as depicted in figure 1 where yaw gimbal and pitch gimbals
are indicated. Pitch gimbal is denoted with A, and yaw gimbal is denoted with K and the base/body or
the support is denoted with B. The sensor is placed on the pitch or inner gimbal or in fact sensor is the
pitch gimbal [33]. Three reference frames are introduced:

1. A fuselage body-fixed frame B

2. A frame K fixed to the yaw gimbal
3. A frame A fixed to the pitch

Notations for reference frames for yaw gimbal and pitch gimbal are Xk , Yk , Zk and Xa , Ya , Za respectively.
The Xa −axis coincides with the sensor optical axis. The Z-axis, Za -axis and Zk -axis coincide and pointing
Title Suppressed Due to Excessive Length 5

Xa Xk
Camera Sensor
θ
in the inner Gimbal A gle
an Inner Gimbal

Pitch Gimbal
Outer Gimbal
Ya Torque Ty
Yaw Gimbal K

le Xk
ang ψ
Base or
Outer Gimbal
Support B
Yk Torque rotation
Tz
Zk Za Coupling axes

Fig. 1: Two axis gimbal system

downward. The center of rotation is in the frame origin, which is assumed to be the same point for the
three frames. The angles of rotation 1 and 2 are defined in the following way:

1. The body-fixed frame B is carried into coincidence with the yaw gimbal frame K by a positive angle of
rotation ψ about the Zk -axis.
2. The yaw gimbal frame K is carried into coincidence with the pitch gimbal frame A by a positive angle
of rotation θ about they Za -axis.

It is clear that if the angle ψ is zero the body or base B will coincide with outer gimbal K then associated
with these rotations we have the following transformations [31]:
   
cosψ sinψ 0 cosθ 0 sinθ
LKB = −sinψ cosψ 0 , LAK =  0 0 1 
0 0 1 −sinθ 0 cosθ

LKB is the transformation from B to K, i.e., if a vector is expressed by its coordinates in frame B, then
LKB gives the coordinates of the same vector in frame K. Similarly, LAK is the transformation from K to
A. For the angular velocities of frames B, K and A respectively, we introduce:
     
p pk pa
B = q  , K = qk  , A = qa 
r rk ra
where p,q,r are the components in frame B of the inertial angular velocity vector of the frame B itself, and
similarly for the other vectors. Following traditional practice in flight dynamics we use the notations p,q,r
for the roll, pitch, and yaw components, respectively. The inertia matrices of the gimbals are denoted by
pitch gimbal with JA and yaw gimbal with JK respectively:
   
Jax Dxy Dxz Jkx dxy dxz
JA = Dyx Jay Dyz  , JK = dyx Jky dyz 
Dzx Dzy Jaz dzx dzy Jkz

Thus, moments of inertia are denoted by J. The products of inertia, or inertial due to dynamic mass
imbalances i.e. Dxy , Dxz , Dyx , Dyz , Dzx , Dzy and dxy , dxz , dyx , dyz , dzx , dzy are considered zero for the
case.

2.2.2 Equations of Motion

For finding the equations of motion for both gimbals we need to find the coupling of angular velocity
between fixed body and yaw gimbal, between yaw gimbal and pitch gimbal and finally between fixed body
6

(base) and pitch gimbal. Coupling of angular velocities between yaw gimbal and base or body is obtained
by the relation:      
pk p 0
qk  − LKB q  =  0 
rk r ψ
we obtained the following relations.

pk = pcosψ + qsinψ (1)

qk = −psinψ + qcosψ (2)
rk = r + ψ̇ (3)

Similarly we can find coupling of angular velocity between yaw gimbal and pitch gimbal, which is obtained
by the relation:      
pa pk 0
qa  − LAK qk  = θ 
ra rk 0
we got the following relations:

pa = pk cosθ − qk sinθ (4)

qa = qk + θ̇ (5)
ra = −pk sinθ + qk cosθ (6)

The dynamic equations of motion of electronic camera gimbal system are obtained using Euler angular
relationship and the angular momentum equation:

dH̄
T̄ = + w̄ × H̄ (7)
dt
where T̄ is the sum of total external torque, H̄ is the angular momentum and w̄ is the angular velocity.
 
x y z
w̄ × H̄ =  p q r 
H x H y Hz

where total angular momentum is expressed as

H¯x
   
0
H̄ = JK + LT
AK ∗ JA = H¯y  = H¯y 

H̄z H̄z

pitch gimbal rotate about y − axis while yaw gimbal rotate about z − axis so we only need the angular
momentum about these axes.

2.2.3 Equation of Motion for Yaw Gimbal

The equation of of motion of yaw gimbal or outer channel is derived using equation 7 and is given:
1
ψ̈ = [Tz − [Jax Pa cos θ + Jaz ra sin θ + cos θ(Pk cos θ − rk sin θ) − sin θ(P cos ψ...
Jkz + cos2 θ − sin2 θ
+ q sin ψ) − r cos θ]θ̇ − (cos θ sin θ)θ̇ψ̇ − cos θ[sin θ(q cos ψ − P sin ψ)]ψ̇ − (Jkz − sin2 θ)ṙ... (8)
− [Ṗ cos θ sin θ + q̇(sin θ sin ψ) + ṙ cos θ] cos θ] + Pk qk (Jky − Jkx )...
+ Pa qa Jay − Pa qk Jax cos θ − qk ra Jaz sin θ
Title Suppressed Due to Excessive Length 7

2.2.4 Equation of Motion for Pitch Gimbal

Similarly dynamic equation of motion for pitch gimbal or pitch channel is obtained using equation 7 as
below:
1
θ̈ = [Ty + (Jky − Jay )(q̇ cos ψ − ṗ sin ψ) − Jax Pa (P sin θ + r cos θ) − Jaz ra (P cos θ − r sin θ)...
Jay
(9)
1
− Jkx Pk r − Jkz P rk ] − [(Jky − Jay )(P cos ψ − q sin ψ)ψ̇]
Jay

2.3 Frictional Disturbance in Both Channels

One of the most unwanted disturbance that decrease the tracking performance and cause steady state error
is friction. Bing a control engineer. We must always aim to minimize it. Here static friction is considered that
occurs in bearing of both gimbals during its pointing and tracking movement with some angular velocity.
Torque due the static friction causing disturbance in yaw gimbal or outer channel is given below:

Tky = CZ ψ̇

where Tky is total frictional torque at the yaw channel rotating about z − axis and ψ̇ is the angular rate
at which it rotate. Similarly
Tap = Cy θ̇
where Tap is total frictional torque at the pitch channel rotating about y − axis and θ̇ is the angular rate
at which it rotate. The dynamic equations of motion for both gimbals with frictional disturbance are given
as:
1
ψ̈ = [Tz − [Jax Pa cos θ + Jaz ra sin θ + cos θ(Pk cos θ − rk sin θ) − sin θ(P cos ψ + q sin ψ)
Jkz + cos2θ − sin2 θ
− r cos θ]θ̇ − (cos θ sin θ)θ̇ψ̇ − cos θ[sin θ(q cos ψ − P sin ψ)]ψ̇ − (Jkz − sin2 θ)ṙ − [Ṗ cos θ sin θ
+ q̇(sin θ sin ψ) + ṙ cos θ] cos θ] + Pk qk (Jky − Jkx ) + Pa qa Jay − Pa qk Jax cos θ − qk ra Jaz sin θ + CZ ψ̇
(10)
and
1
θ̈ = [Ty + (Jky − Jay )(q̇ cos ψ − ṗ sin ψ) − Jax Pa (P sin θ + r cos θ) − Jaz ra (P cos θ − r sin θ)
Jay
(11)
1
− Jkx Pk r − Jkz P rk ] − [(Jky − Jay )(P cos ψ − q sin ψ)ψ̇] + Cy θ̇
Jay

2.4 Linearization and State Space Representation

Both dynamic nonlinear equations of motion for camera gimbal and outer gimbal given in (10) and (11)
are complex to analyze and control. So converting to linear system via liearization about its origin, which
results in the state space model with state and input matrices. We have four stats that are ψ, ψ̇, θ, θ̇ where
ψ is angle of outer gimbal and θ is the angle of inner gimbal. We also have two actuators for inner gimbal
and outer gimbal respectively ui and uo . where ui is the input to pitch channel that will rotate it, similarly
uo is the input to yaw channel. The nonlinear equations are then linearized across origin via Jacobian,
 df 1 df 1 df 1 df 1   df 1 df 1 
dx1 dx2 dx3 dx4 du1 du2
 df 2 df 2 df 2 df 2   df 2 df 2 
A= dx1 dx2 dx3 dx4  , B = du1 du2 
 df 3 df 3 df 3 df 3   df 3 df 3 
dx1 dx2 dx3 dx4 du1 du2
df 4 df 4 df 4 df 4 df 4 df 4
dx1 dx2 dx3 dx4 du1 du2
8

and we got state matrix A as:

 
0 1 0 0
Jkz (Pa q + q 2 Jay − q(Jky − Jkx )) 0 Jkz [Jkz Pk − q 2 (Jky − Jkz ) + Jaz ra qk ] 0
A=
 
 0 0 0 1
(Jky −Jay )Ṗ (Jky −Jay )P Jax P Pa
Jay − Jkx qr Jay Jay 0

and
0 0
 
 J Tz+1 0     
B=  kz ,C = 0 0 1 0 ,D = 0
0 0 
Ty
0 Jay

we have single output that is the gimbal camera sensor movement θ̇ for pointing and tracking so the final
state space is
˙ = Ax(t) + Bu(t)
x(t)
y = Cx(t) + Du(t)
   
0 1 0 0 0 0
a21 0 a23 0 b21 0 
A=
 0 0 0 1 , B =  0
  
0 
a41 a42 a43 0 0 b42
   
,C = 0 0 1 0 ,D = 0
where a21 , a23 , a41 , a42 , and a43 are the elements of state matrix A and b21 and b42 are the elements of
input matrix B.

The above state space model has four matrices. Matrix A is the state matrix consists of ψ, ψ̇, θ and
θ̇ which are the position and angular velocities of outer gimbal and inner gimbal respectively. Matrix B is
the input matrix or source agency that will rotate the gimbal system means matrix for actuators. Matrix
C is the output matrix which gives us single output that is θ position state of the plant or eyeball. As there
is no direct connection to output so matrix will lead to zero entries of appropriate size.

2.5 Fault Dependent Modeling

Fault analysis is important as it may occur in the gimbal camera sensor or in a process like mechanical
gimbal lock or in actuators. In the presence of these faults tracking of stimulus is impossible. Its detection
is important for safe and fluent tracking of stimulus. Here we consider a gimbal camera sensor fault only.
 
000  
0 0 0 fA
Ff =   , f = f P 
0 0 0
fs
001

where fA is for actuator fault, fP process fault and fs is for gimbal camera sensor faults. we consider only
one gimbal camera sensor where actuator fault and process faults are not considered in the above equation.
The plant state space model with frictional disturbance Ed and gimbal camera sensor fault Ff is given
˙ = Ax(t) + Bu(t) + Ed d(t)
x(t)

y = Cx(t) + Ff f (t)

3 Tracking Mechanism of the Presented Model

The section is specially devoted to discuss the similarities of the robotic vision system with a human
vision system in vision control, feedback mechanism, tracking kinematics, physiological structure mechanical
design and anthropometry. These similarities make the system well suited for humanoid robots.
Title Suppressed Due to Excessive Length 9

3.1 The Human Eye

The human eye tracking system consists of the eyeball, extraocular muscle, and brain. The eyeball has
a camera system that tracks stimulus. The extraocular muscles are used to hold, protect and rotate the
eyeball independent of a head skeleton as shown in figure 2 [32]. The central control or brain helps in
pointing, tracking, scanning, and recognition. These muscles are the main actuators that move the eyeball
according to the central control command, i.e. brain for target tracking.

Neuron are used for signaling between brain and the outside stimulus. Brain take the
target information through neurons for decision and command to the extra ocular muscles.
The actuator muscles actuate to perform five different types of movements that are saccade,
smooth pursuit, vergence, vestibule-ocular reflex (VOR), optokinetic reflex (OKR) and eye-
head coordination. Saccade is to move eyes voluntarily from one point to another by rapid
jumping, while smooth pursuit can be applied to track moving targets. VOR acts to stabilize
retinal images by generating a compensatory eye motion during head turns. OKR can stabilize
retinal images for gazing at rapid moving objects by nystagmus. OKR is driven by retinal
slip while VOR is driven by head velocity signal. Commonly, two or more forms of motion
work simultaneously. Besides, the binocular coordination and eye-head coordination are of
high importance to realize object tracking and gaze control. Our proposed model mimic the
human eye in its high decoupling ratio with head skeleton that helps in two type of eye
movement that are saccade and smooth pursuit movement that are used for pointing and
tracking in dynamic environment. The signaling between target and control is also similar to
human feedback system that is nerural system.

Fig. 2: The Human eye

3.2 Framework of the Novel Humanoid Robotic Eye

The proposed robotic vision system has many similarities with human eye tracking mechanism in many
aspects. The independent movement of gimbal system from the robotic head skeleton is similar to eyeball
movement, feedback mechanism is similar to neural system response and training based vision control
design to the brain. This realism and similarities with human system make the novel humanoid robotic
vision system more efficient than any existing design.

Gimbal system has its basic property of decoupling camera movement from the pivot point or sup-
port. The two-axis electronic gimbal system with its own stabilization control can replace the eyeball and
extraocular muscles. Stabilization control will take its reference command from the vision controller/CPU
as shown in figure [3], where AI based algorithms are implemented that can track and train for a specific
stimulus. The fast-moving gimbal camera sensor based tracking design need synchronization among the
gimbal system, actuation system and AI based vision control. For the synchronization, gimbal camera sen-
10

sor system should have high decoupling from supportive robotic head skeleton, better stabilization control
and AI based algorithms in the CPU/brain/vision controller.

Thanks to gimbal system due to which decoupling of camera sensor from the head skeleton is easily
achieved. Physical disturbances compensation further increase the decoupling ratio for better performance.
Stabilization controller maintain the complete vision system control and stable.

3.3 Theoretical Architecture

Fast pointing and tracking process need a controlled, fast and smooth movement of camera sensor in
a dynamic environment. Stabilization controller takes reference command from vision control/brain and
adjusts the camera sensor tracking the active stimulus as shown in figure [3]. Then the application of
stabilization control in the simulation section verify its regulation and tracking that confirms the preciseness
and quicker pointing and tracking of active stimulus. Position loop and tracking loop both are used for
pointing and tracking the target respectively. The command loop is responsible for the ultimate decision
about the stimulus. A Static frictional disturbance is also compensated in the friction loop.

S
Position Loop
l
C Friction Loop

Command Loop
Gimbal
System

Stabilization Loop
Tracking Loop

Fig. 3: Feedback Mechanism/ Neural Response of Robotic eye

Fig. 4: Kinematic Sketch of the Proposed Eye

Title Suppressed Due to Excessive Length 11

Fig. 5: Experimental Setup for Gimbal Based Robotic Eye

3.4 Feedback Mechanism/ Neural System Response

Like the human neural system, first the camera sensor will point the stimulus and the message will be
delivered to AI based brain control or CPU. AI based algorithms will decide about the stimulus and will
give further command to actuation control for next eye movement. Gimbal system will just help in tracking
the fast stimulus by decoupling the camera sensor. Stabilization controller take the reference command from
vision control or brain and feedback from the active stimulus in order to adjust and stabilize its gimbal
cameras sensor motion according to the dynamic target in the outside environment as shown in figure [3]. A
two-axis rate gyro is usually placed on the inner gimbal measuring the rotational rates in the two directions
of interest. These gyro signals are utilized as feedback to torque motors acting on the gimbals according to
vision control.

In position loop, position sensor determines the position of the target and reference/command signal
from the CPU/Brain. Both the signals are then used for pointing target then controlled command signal is
sent to stabilization loop. Similarly, in tracking loop, rate sensor determines the movement of the camera
sensor and then tracking stimulus with tracking control with respect to a reference/command signal from
CPU/Brain. Both the command signals for pointing and tracking are sent to the stabilization loop. Stabi-
lization controller is finally responsible for the controlled movement of the gimbal camera sensor according
to fast-moving target and commands from the brain.

3.5 Mechanical Design

Anthropometric sketch of the humanoid robotic eye is shown in the figure [4]. Camera sensor in the inner
channel of the gimbal system will adjust its position θ according to target and move around y − axis
while the outer gimbal will move around z − axis. The fixed holders hold the outer gimbal allowing it for
independent movement from the head skeletal system. The stabilization control actuators are responsible
for the controlled movement of the camera sensor. AI based algorithms will give reference command while
the rate sensor is responsible for feedback. The whole mechanical design can be set on the chest or
forehead within 6 inches length, 3 inches width and 3 inches height [4].

The mechanical design of gimbal based robotic eye helps in saccade and smooth pursuit
movements for pointing and tracking. For the range of movement, we know that the available
gimbal system in market have full 360 degree rotation range in both pitch and yaw axis and
can be set in any desired direction. When the gimbal system is fitted in the forehead or chest
then due to the hardware constrain gimbal based eye will be able to see approximately up to
120 degree pitch and yaw movement. The speed of human eyeball movement is beyond 800
degree per second that means it can move at max 20 degree per 20 milli seconds. The speed
is much slower than the rotational speed of gimbal systems. The available gimbal systems in
market can rotate at the speed of 90 degree in 20 milli seconds with milli seconds accuracy
12

that is much better and faster than human eye or any existing robotic eye that can rotate at
6 degree per 20 millisecond at max in tracking process. The rotation speed of gimbal eye will
further depends on the brain algorithms speed but it is sure that its speed will faster then
the 20 degree per 20 milli seconds, the human eyeball rotational speed.

The range of gimbal based robotic eye can cause gimbal lock that is the singularity point
in gimbal system. It occurs when all the channels are align. If we use the gimbal system
as robotic eye then the singularity point or mechanical gimbal lock can not occur due to
hardware constrain as both the channels will not rotate beyond 120 degree but if occur, it
will be removed manually. The singularity point can be treated as process fault or fault in
plant. The range of gimbal based robotic vision system can be set up to different angles.
The inner channel or camera sensor should be set mechanically up to 30 degree superior,
45 degree nasal, 70 degree inferior and 100 degree temporal while the outer channel that is
used to decouple the camera sensor from the head skeleton. The outer channel will rotate
up to 15 degree at max so that it can see like human. These constrains are due to the head
skeleton or may be on the chest.

4 Experimental Implementation

The experimental implementation of the gimbal based robotic eye can be done using servo
motors for pitch and yaw motion. Experimental setup for three axis gimbal system is shown
in figure 5. Power source from any voltage should be converted to 5 volt using buck converter
for sensor MPU6050, arduino and MG996R servo motors. The servo motors will rotate the
gimbal’s channels about respective axis while the arduino will communicate between the
MPU6050 IMU sensor and servo motors. MPU 6050 IMU sensor contain the accelerome-
ter and gyroscope. Gyroscope measure the rotational velocity from where we can find the
angular position just by integrating the angular velocity and accelerometer is used for the
measurement of gravitational acceleration. Both are combined for accurate information about
the sensor orientation [4].

5 Robust Stabilization Control Design

The simulation results for tracking and regulation processes are given in figure 5 and figure
6. For proper eyeball movement and tracking process with its tracking speed can be seen
from the simulation results. Here, cascade control is used for tracking a reference signal or
target. Blue line in the simulation results of figure 5 is the reference signal, target or stimulus
while the red dotted line that is following or tracking the blue line is the eyeball movement.
When target or stimulus move randomly or dynamically, the eyeball tracking it with some
error. The perfect tracking or bringing a target into the center of the visual field by jumping
to the position called saccade is achieved within 4 second of time then the start tracking
target perfectly called smooth pursuit movement. Similarly in figure 6, the output of plant
or robotic eyeball comes to steady state or primary position (that is mentioned in section 2.1
for saccade movement) within 4 seconds. This means that the eyeball will take 4 seconds for
coming to its primary position or horizontal line of sight. The designed control is also robust
against the friction and static mass imbalances that can degrade the tracking performance.
Figure 7 clearly demonstrate the compensation of friction disturbance, if occurs to disturb
the any moment of time.

6 Conclusion

The fault dependent mathematical modeling of gimbal based novel humanoid robot eye is presented which
has a similar mechanism of pointing and tracking as human eyes. The proposed design decoupled the eyeball
Title Suppressed Due to Excessive Length 13

Tracking Reference Input

1.2
Reference Input
Controlled Output
1

0.8

Amplitude
0.6

0.4

0.2

0
0 2 4 6 8 10 12 14 16 18 20
Time (seconds)

Fig. 6: Tracking Reference Input

Plant Output with Compensated Friction

5

3
Output Amplitude

−1

−2
0 2 4 6 8 10 12 14 16 18 20
Time (seconds)

Fig. 7: Regulation Control of the Humanoid eye

Plant Output with Sensor Fault

4

1
Amplitude

−1

−2

−3

−4

−5
0 5 10 15
Time (seconds)

Fig. 8: Frictional Disturbance Rejection

movement from the robotic head skeleton due to the basic property of the gimbal system. Mechanical
design help the eye to rotate freely within desired range while the decoupling property helps in
saccade and smooth pursuit movement for pointing and tracking. Frictional disturbance and static
mass imbalance are compensated for the improvement of tracking performance. Theoretical architecture
and tracking mechanism prove the design more mimic to the human vision system than existing models. The
simulation result of the robust stabilization control design makes the system more suitable for humanoid
robots, especially in tracking fast-moving stimulus in a dynamic social environment.

References

1. D. A. Forsyth and J. Ponce, A modern approach, Computer vision: a modern approach, pp. 88-101, (2003).
2. G. Cannata and E. Grosso, (1999), On perceptual advantages of active robot vision, Journal of Robotic Systems, vol.
16, no. 3, pp. 163-183.
3. W. Jang and Z. Bien, (1991) Feature-based visual servoing of an eye-in-hand robot with improved tracking performance,
in Robotics and Automation, 1991. Proceedings., 1991 IEEE International Conference on, pp. 2254-2260, IEEE.
4. https://howtomechatronics.com/projects/diy-arduino-gimbal-self-stabilizing-platform/
5. A. Manecy, J. Diperi, M. Boyron, N. Marchand, and S. Vi-ollet, (2016) A novel hyperacute gimbal eye to implement
precise hovering and target tracking on a quadrotor, in Robotics and Automation (ICRA), 2016 IEEE International
Conference on, pp. 3212-3218.
14

6. L. R. Young and L. Stark, Variable feedback experiments testing a sampled data model for eye tracking movements,
IEEE Transactions on Human Factors in Electronics, no. 1, pp. 3851, 1963.
7. D. A. Robinson, Models of the saccadic eye movement control system, Kybernetik, vol. 14, no. 2, pp. 7183, 1973
8. C. Quaia, P. Lef‘evre, and L. M. Optican, Model of the control of saccades by superior colliculus and cerebellum, Journal
of Neurophys-iology, vol. 82, no. 2, pp. 9991018, 1999.
9. J. Bruske, M. Hansen, L. Riehn, and G. Sommer, Biologically inspired calibration-free adaptive saccade control of a
binocular camera-head, Biological Cybernetics, vol. 77, no. 6, pp. 433446, 1997.
10. X. Wang, J. van de Weem, and P. Jonker, An advanced active vision system imitating human eye movements, in
Advanced Robotics (ICAR), 2013 16th International Conference on. IEEE, 2013, pp. 16.
11. M. Antonelli, A. J. Duran, E. Chinellato, and A. P. Del Pobil, Adaptive saccade controller inspired by the primates
cerebellum, in Robotics and Automation (ICRA), 2015 IEEE International Conference on. IEEE, 2015, pp. 50485053
12. D. A. Robinson, J. Gordon, and S. Gordon, A model of the smooth pursuit eye movement system, Biological cybernetics,
vol. 55, no. 1, pp. 4357, 1986.
13. C. Brown, Gaze controls with interactions and decays, IEEE Trans-actions on Systems, Man, and Cybernetics, vol.
20, no. 2, pp. 518527, 1990.
14. S. G. Lisberger, E. Morris, and L. Tychsen, Visual motion processing and sensory-motor integration for smooth pursuit
eye movements, Annual review of neuroscience, vol. 10, no. 1, pp. 97129, 1987.
15. D. C. Deno, E. L. Keller, and W. F. Crandall, Dynamical neural network organization of the visual pursuit system,
IEEE Transactions on Biomedical Engineering, vol. 36, no. 1, pp. 8592, 1989.
16. F. Lunghi, S. Lazzari, and G. Magenes, Neural adaptive predictor for visual tracking system, in Engineering in Medicine
and Biology Society, 1998. Proceedings of the 20th Annual International Conference of the IEEE, vol. 3. IEEE, 1998, pp.
13891392.
17. T. Shibata, S. Vijayakumar, J. Conradt, and S. Schaal, Biomimetic oculomotor control, Adaptive Behavior, vol. 9, no.
3-4, pp. 189207, 2001.
18. Y. Song and X. Zhang, An active binocular integrated system for intelligent robot vision, in Intelligence and Security
Informatics (ISI), 2012 IEEE International Conference on. IEEE, 2012, pp. 4853
19. M. D. Kim and J. Ueda, (2018) Dynamics-based motion de-blurring improves the performance of optical character
recognition during fast scanning of a robotic eye, IEEE/ASME Trans-actions on Mechatronics.
20. K. Nickel and R. Stiefelhagen, (2007) Visual recognition of pointing gestures for humanrobot interaction, Image and
Vision Computing, vol. 25, no. 12, pp. 1875-1884.
21. X.-y. Wang, Y. Zhang, X.-j. Fu, and G.-s. Xiang, (2008) Design and kinematic analysis of a novel humanoid robot eye
using pneumatic artificial muscles, Journal of Bionic Engineering, vol. 5, no. 3, pp. 264-270.
22. G. Cannata and M. Maggiali, (2008) Models for the design of bioin-spired robot eyes, IEEE Transactions on Robotics,
vol. 24, no. 1, pp. 27-44.
23. C. F. Martin and L. Schovanec, (1998) Muscle mechanics and dynamics of ocular motion, Journal of Mathematical
Systems Estimation and Control, vol. 8, pp. 233-236.
24. A. D. Polpitiya, W. P. Dayawansa, C. F. Martin, and B. K. Ghosh, (2007) Geometry and control of human eye
movements, IEEE Transactions on Automatic control, vol. 52, no. 2, pp. 170-180.
25. G. Cannata and M. Maggiali, (2006) Implementation of listing’s law for a tendon driven robot eye, in Intelligent Robots
and Systems, 2006 IEEE/RSJ International Conference on, pp. 3940-3945.
26. C. Gielen, S. Gabel, and J. Duysens, (2004) Retinal slip during active head motion and stimulus motion, Experimental
brain research, vol. 155, no. 2, pp. 211-219.
27. S. Schulz, I. Lutkebohle, and S. Wachsmuth, (2012) An affordable, 3d-printable camera eye with two active degrees
of freedom for an anthropomorphic robot, in Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International
Conference on, pp. 764-771, IEEE.
28. D. Pongas, F. Guenter, A. GuIgard, and A. G. Billard, (2004) Development of a miniature pair of eyes with camera for
the humanoid robot robota, in Humanoid Robots, 2004 4th IEEE/RAS International Conference on, vol. 2, pp. 899-911,
IEEE.
29. S. May, B. Werner, H. Surmann, and K. Pervolz, (2006) 3d time-of-flight cameras for mobile robotics, in Intelligent
Robots and Systems, 2006 IEEE/RSJ International Conference on, pp. 790-795, Ieee.
30. R. Gilmore, Lie groups, lie algebras and some of their representations, (1974).
31. A. A. Handzel and T. Flash, (1996) The geometry of eye rotations and listing’s law, in Advances in Neural Information
Pro-cessing Systems, pp. 117-123.
32. B. Ekstrand, (2001) Equations of motion for a two-axes gimbal system, IEEE Transactions on Aerospace and Electronic
Systems, vol. 37, no. 3, pp. 1083-1091.
33. B. Etkin and L. D. Reid, (1996) Dynamics of flight: stability and control, vol. 3. Wiley New York.
34. http://www.eyegougehead.com/civilian/the-human-eye/.
35. https://www.youtube.com/watch?v=ButbHMqUDuA