16th IEEE International Conference on Robot & Human Interactive Communication

August 26
-

29, 2007 / Jeju, Korea

MP-18

Tele-operation of a Mobile Robot Using a Force Reflection Joystick with a Single Hall Sensor
Soonshin Han and JangMyung Lee*, Senior Member, IEEE
Abstract Though the final goal of mobile robot navigation is to be autonomous, operators' intelligent and skillful decisions are necessary when there are many scattered obstacles. There are several limitations even in the camera-based tele-operation of a mobile robot, which is very popular for the mobile robot navigation. For examples, shadowed and curved areas cannot be viewed using a narrow view-angle camera, especially in bad weather such as on snowy or rainy days. Therefore, it is necessary to have other sensory information for reliable tele-operations. In this paper, sixteen ultrasonic sensors are attached around a mobile robot in a ring pattern to measure the distances to obstacles. A collision vector is introduced in this paper as a new tool for obstacle avoidance, which is defined as a normal vector from an obstacle to the mobile robot. Based on this collision vector, a virtual reflection force is generated to avoid the obstacles and then the reflection force is transferred to an operator who is holding a joystick to control the mobile robot. Relying on the reflection force, the operator can control the mobile robot more smoothly and safely. For this bi-directional tele-operation, a master joystick system using a hall sensor was designed to resolve the existence of nonlinear sections, which are usual for a general joystick with two motors and potentiometers. Finally, the efficiency of a force reflection joystick is verified through the comparison of two vision-based tele-operation experiments, with and without force reflection. Key words: Force Feedback, Hall Sensor, Potentiometer, Tele-operation, Collision
I. INTRODUCTION

A PPLICATIONS of robots were popular only in manufacturing factories even a coOOOOOOOuple of years ago. With the development of IT technologies, the technical concept for the robot has been extended to many additional applications, such as, home appliances and recreation. Painted lines or optical tapes are very popular for mobile robots with CCD cameras and the magnetic sensors in the factory applications to follow fixed tracks [1]. These robots have an advantage of precise tracking capability since they are following a fixed track at all times. However, for service robots, surveillance robots, and cleaning robots, installation of fixed tracks is not possible, and flexible and robust localization systems are required [2,3]. Especially for dangerous and unexpected situations where highly intelligent decisions are required dynamically, the autonomous system itself is not reliable any longer. Instead, a tele-operated control system where a human operator directly controls the robot according to his decision, has been utilized so far. In conventional tele-operated systems, the operator
JangMyung Lee is with the Electronics Engineering Department, Pusan National University Korea, is conducting Research Laboratory for Intelligent R obot,, e-mail: jmlee opusan.ac.kr.

relies mostly on visual information to control the slave robot, which has several shortcomings such as narrow view-angles and weather-dependent resolutions [4]. In this research, a bidirectional tele-operation system has been proposed, where a slave robot receives the operator's command through the joystick and sends back the environment information to the operator as a form of feedback force. From the generation of the feedback force corresponding to the environment, a new collision vector is defined to determine the direction of the reflection force to avoid the obstacle. The reflection force value is sent to the joystick with two degrees of freedom to make the operator feel the force and manipulate the joystick according to the force [5]. The collision vector represents the shortest normal vector from an obstacle to the mobile robot and improves the collision avoidance efficiency. The joystick for this research has a hall sensor instead of potentiometers to detect the joystick-bar displacements. Using this new structure, the joystick becomes free from wear that comes from friction between the bar and the housing and degrades position-sensing performance. It also has better performance than the conventional dual hall-sensor structure which suffers from nonlinear characteristics and from mechanical complexities. In section 2, a single hall-sensor structure is introduced, and section 3 describes on the tele-operated control system systematically. The collision vector and virtual impedance algorithms, which are the main contribution of this paper, are introduced in section 4. The experiments of autonomous navigation and tele-operation of a mobile robot are performed, and the results are analyzed to show the superiority of the proposed algorithm in section 5. Section 6 summarizes the contributions of this paper.

II. SINGLE HALL-SENSOR JOYSTICK A single hall-sensor joystick is designed in this paper to overcome the track-wear and nonlinearities in the conventional joysticks where two potentiometers are used to measure the displacement of the joystick-bar. This non-contact electronic joystick is specifically designed for tele-operation experiments of a mobile robot [6]. The mechanical structure of the joystick is shown in Fig. 1. The magnetic flux lines are always forwarding to the center of magnetic-bar rotation axis. When the magnetic-bar is tilted, 0, the horizontal vector of the magnetic flux density, B, which is denoted as Bh and measured by the hall-sensor, can be divide into two components of B, and By along the X-axis

978-1 -4244-1635-6/07/$25.00 2007 IEEE.

206

and Y-axis, respectively.

Fig. 1. Structure of a single hall-sensor joystick.

The voltage outputs of a hall-sensor represent the two components in 900 phase difference. The relation between the magnetic flux density, B, and the corresponding horizontal vector, Bh, can be modeled as

The permanent magnet of the joystick is made of Alnico materials and has an intensity of 1800 2000 Gauss. For the hall sensor, 2SA-10 from Sentron Corp. in Switzerland has been selected for the experiments. The vertical distance between the permanent magnet and hall sensor, D, is 13 mm, and the total length of joystick-bar is 31 mm. The conventional joystick used in this experiment for the comparison has a 1 00KQ potentiometer at each motor axis. The controller for the joystick is implemented with a PIC microprocessor, which controls the DC motors using PWM voltage-output to make the operator feel the reflected force corresponding to the virtual force, (Fl, Fy), sent from the mobile robot. L6203 drivers are used to drive the DC motors. The relation between the rotation angle and two dimensional coordinates is not linear. Therefore, the joystick motions cannot be measured precisely by using the dual potentiometers joystick, especially near the rotation axis. On the contrary, the single hall-sensor joystick represents a clear circular trace.
-

(1) where k(O) incorporates the nonlinearities with sin(O).

When the magnetic flux lines are parallel to the joystick-bar, i(O) equals 1. However the horizontal magnetic field does not increase linearly with the increase of tilt angle, 0 . Actually, it decreases adversely over a certain range of the tilt. The nonlinear characteristics depend on the geometric shape, especially the length of the permanent magnet, L, and the vertical distance between the permanent magnet-bar tip and hall-sensor, D. To compensate the nonlinearities, the nonlinear function, A(O), is defined as a function of 0 as

Bh = A(O)Bsin(O)

III. TELE-OPERATED CONTROL SYSTEM

A. Tele-operated control
Visual Visual

Obstacle Information

Obstacle Information

Position,

Position,

(0)=-1(I I (kO)'
+

(2)

Velocity

Velocity

where n is the linearity relationship between sin(0) and the output of the hall-sensor, and k is a constant which relates to the geometric shape of the magnet-bar and the design specification of the joystick frame. The relation between the outputs of the hall-sensor, (Vx, Vy), and the magnetic flux, (By, By), is linear and is described as follows [7]:

Fig. 2. Overall system block diagram.

V,

cB, cos(a) =cA(O)B cos(a)

cZ(O)Bsin(a)

V = cBy sin(a) =

where the amplification factor, c, depends on the electrical constant of the hall-sensor and amplification factor of the signal transformation circuit, and a is the rotation angle of the joystick bar.

The tele-operated control system proposed in this paper consists of a master joystick, a slave mobile robot, and a Bluetooth and wireless video transceiver for communication (refer to Fig. 2). When the operator moves the joystick, the A/D converter transmits the velocity command to the mobile robot, which corresponds to the hall-sensor output as a proportional value to the joystick displacement. According to the values, the mobile robot drives two DC motors using PWM control signals. While the mobile robot is moving, the ultrasonic sensors attached around the robot gather environmental information in the form of reflection force against obstacles and send the information back to the operator, who can then feel the force. In summary, the mobile robot follows the velocity commands from the operator, detects obstacles, and transmits

207

the obstacle information back to the operator. Therefore, the operator can implement tele-presence through feedback force proportional to the distance and relative velocity between the mobile robot and obstacles [8]. B. Slave mobile robot. To analyze the mobile robot motion, the world and robot coordinates are defined in two dimensional space where a position and orientation vector can be represented as P=[x y 0 ]T. Notice that the motion of a mobile robot can be described by the linear velocity, u, and the angular velocity, 0), and they are functions of the velocities of the left and right wheels.
U=

sensors goes from a minimum of 3 Cm to a maximum of 3 m. To reduce interference among the sensors, each sensor is allowed to work a certain period of time through a decoder and a multiplexer. Encoders are utilized to control the wheel velocities of the mobile robot. Therefore, the position vector of the navigating mobile robot can be calculated at each control period of the wheel. The encoder accuracy, 360 [pulse/rev], is improved four times to 1,440[pulse/rev] by using LS7 166.
IV. REFLECTION OF A VIRTUAL FORCE

1 (VR + VL 2 1 O= (vR -VL). L R

K--

(5)
(6)

The velocity, f, in the world coordinates, is related to the input variable, q, by the Jacobian matrix as

Virtual impedance method is a modification of the general impedance algorithm for mobile robot navigation and collision avoidance of obstacles. As it is illustrated in Fig. 3, the relation between a mobile robot and its goal position/obstacles is modeled by springs and dampers. Based on the virtual impedance model, the feedback force is calculated and transmitted to the joystick to reflect the environmental situations by the force and torque.
Reference Point X(s)

J(p)q&
[u ]T E Rm, and

(7)
Fm
Static Obstaci

where A&= [

E R",

J(P) = sinO 0.
The position vector, P, in the world coordinates can be obtained using the integration of Eq. (7) as

cosO 0

L10

Fod

Obstacle

Dynamic

Realized

Fxl
P=

rxo j

fu(r) cos(O(r))dr fo+u(r) sinQ(O(r))dr

fco(r)dr

(8)

Current Position x(s)

L0- L-0o|

Fig. 3. Virtual impedance method.

Using these geometrical relations, the position of the mobile robot can be calculated at every control cycle, while u and o are calculated using the wheel encoders. The experimental mobile robot consists of a CCD camera on the top, sixteen ultrasonic sensors around the body, and two DC motors/encoders at the bottom. Basically, two differential motors are used to drive the mobile robot, and two passive wheels in front and at the back balance the robot. To keep the wheels on the surface and to minimize the slippage, springs are used in the passive wheels. The motor specifications for the driving wheels are 24V, 40W, 0.8A, 300 rpm (after 10:1 reduction), and 80kgf-cm torque outputs (SPG Inc. S8D40-24A). The encoder attached on the motor is utilized for velocity control and for position calculation. There are sixteen SRF04 ultrasonic sensors attached around the mobile robot, and the detecting range of the

The virtual force to be sent to the operator, F, is calculated to incorporate static obstacles as well as dynamic obstacles as
nos
nod

Fb =

Z '(s)+Z ios =0 iod =0

Fd(S)

(9)

where F0d and Fo are computed using the following virtual impedance model: JF (s) = KS O(XS O (s)) + DS O(AS O (s)) wC Z{KS O {] (O- C )Cunit- Ds' (A C )Cunit } when C <P0 0 , otherwise (10) where po represents a threshold for collision avoidance, the collision vector, (, is defined as a normal vector from an

208

obstacle to the mobile robot, unit is its unit vector,

KS 0 is

a spring coefficient, and DSQ0 is a damper coefficient of the virtual impedance model.

(a) Case 1

it moves very slowly in the middle of obstacles with some chattering. With the aid of this collision vector, now the mobile robot changes its course against the rapid-approaching obstacles and keeps the planned path closely after collision avoidance. In Eq. (10), po should be kept smaller than the sensible range of the ultrasonic sensors, 3m, and also notice that A = [n] - [n -1] Since the ultrasonic sensor has its own sensitivity range, usually about 22.50, the obstacle located more than 300 from the center of sensor beam cannot be detected reliably. There are sixteen ultrasonic sensors around the mobile robot, that is to say, at every 22.5 0 there is a sensor. Therefore, there are only three reliable ultrasonic sensors at a certain instant, which can be used to measure the distance from the mobile robot to the obstacle. Based on this observation, the type of obstacles that can be detected by the mobile robot are classified into three categories: 1. A small obstacle that can be detected by only a single ultrasonic sensor, such as human beings or small trash cans, 2. A slightly larger obstacle detected by two ultrasonic sensors, such as a big box or a table, and 3. A long obstacle detected by three sensors, such as a wall or a long table (refer to Fig. 4(a), (b), and (c)). 1. In the first case, the collision vector can be obtained directly form the measured distance vector as shown in Fig. 4(a), and it is represented as
2. In the second case, the shorter distance vector is denoted
as and the other as L2 (refer to Fig. 4(b)). A normal vector, which is defined as a collision vector, to the center of the mobile robot can be found on the line connecting the two points, P1 and P2, which can be calculated as

C=L.min

(1 1)

(b) Case 2

LI,

(13) where d1 and d2 represent the distances from the mobile robot to the points PI and P2, and 01 and 02 represent the angles
to points

=(xl, y1)= (d cos 01, d1 sin 01) PI =(x2,y2) = (d2 cos02,d2 sin02)

(12)

PI and P2 from the mobile robot, respectively.

(c) Case 3
Fig. 4. Detection of the collision vector for three
different types of obstacles.

Now the collision vector point on the obstacle, PO, can be calculated as
xi

The relation between the mobile robot and obstacles is modeled by springs and dampers to generate the virtual force adopting the general impedance algorithm [9]. However, a collision vector is newly introduced in this paper to define the direction of the virtual spring as well as the virtual damper as the variation of the collision vector, which improves the collision avoidance performance dramatically. When the virtual impedance algorithm has been applied for the obstacle avoidance, the mobile robot stops near at the obstacles and/or

P0

(x0, y0)
V+ 1 V
V+

-Y,
I

(14)

C= PO Ps

(15)

where V= Y2 Y, Ps represents the current position of x2 -x the mobile robot, and 0 represents the angle between the

209

motion direction of the mobile robot, XR, and the beam-center of the ultrasonic sensor. 3. In the last case, the shortest one among the three vectors is denoted as Lmin the left vector is denoted as

L1, and the

last vector as .2. These two, L1 and L2, can be used to defined the collision vector as in the second case. Note that

selected empirically as 6.5 and 2.2, respectively. The feedback force generated by the virtual impedance algorithm, is transmitted to the joystick (operator). And the force/torque at the joystick is measured by a six d.o.f. force/torque sensor to check the error between the feedback force and the real force at the joystick, which becomes 0 in an ideal system.

Le is ignored since its angle with C is bigger than 3 00.

The generated virtual force, Fb, at the slave mobile robot is transmitted to the operator as Tb (S) = sat(Kb * Fb(s)) (16)
where

sat(x) =

lsgn(x) * xmax otherwise

, if X<Xmax

and

Kb represents force feedback gain.

Now the operator can feel the distance to an obstacle through the reflected force, Tb, on the joystick, which enables him to drive the mobile robot safely even under the limited visual information.
V. EXPERIMENTS AND RESULTS
Fig. 5. Trace of the mobile robot in the autonomous navigation.

Experiments were performance in the Intelligent Robot Laboratory of PNU. To emphasize the effects of using the force reflective joystick, the vision information from the CCD camera has been limited by the low illumination. The navigation environment was set with seven static obstacles of triangular, cylindrical and rectangular shapes, which can be detected by one, two and three ultrasonic sensors, respectively. The experimental environment was kept the same for the autonomous and the tele-operated navigations.
A. Autonomous navigation It took 58 seconds for the mobile robot to navigate from an initial position to a target position according to the virtual impedance algorithm with the newly defined collision vector in Eq. (10). The trace of the mobile robot measured by the encoders is illustrated in Fig. 5 where it is noted that the mobile robot navigates to the goal without any collision into

Through experimental analysis, it can be recognized that the error comes from the nonlinearities of the DC motors in the joystick. That is, when the duty ratio ofthe PWM signal to drive the motors has been increased, the current flow in the motors does not show linearity to the PWM input. In the experiments, the error is kept below 500 by compensating the offset of the force/torque sensor and the nonlinearities of motors properly.
B. Tele-operated navigation In the same environment as the autonomous navigation, two individual operators performed the tele-operation to drive the mobile robot to the goal position with and without the reflection force through the joystick. In the first experiment, a CCD camera and a force reflection joystick have been used for tele-operation. Both of the two operators achieved the goal faster than the autonomous navigation by about 10 seconds (Operator A, 48 seconds and Operator B, 46 seconds to the goal). The navigation trace is shown in Fig. 6. Note that the operators are driving the mobile robot in a shorter course to the goal than the autonomous navigation since they are partially utilizing vision information. In the second experiment, the force feedback to the joystick is disconnected to show the effects of the force reflection. Vision information gathered under fairly low illumination is provided to the operator to drive the mobile robot. As it is shown in Fig. 7, the operator cannot find a short course to the goal in the dark environment using only the CCD camera. Since he does not have any information on the distance between the mobile robot and the obstacles, he let

the obstacles. The joystick-bar is stably grasped by a Scorobot ER-1 robot that measures the reflected force from the mobile robot in the autonomous navigation using an ATI - FT3186 force torque sensor attached at the wrist. Since the mobile robot is in autonomous mode, it does not receive any commands from the joystick during the navigation but feeds back the reflection force based on the virtual impedance algorithm to the joystick. This reflected force, F, is measured by the force torque sensor to show the coincidence between the virtual force and the actually reflected force in autonomous navigation. The optimal values of K and D in Eq. (10) are

210

the mobile robot collide with the obstacles several times and followed a long course to the goal. The average tele-operated navigation time is one minute and 20 seconds, which is 22 seconds longer than the autonomous navigation and 33 seconds longer than the tele-operation with force reflection.

feedback, the operator can drive the mobile robot to a goal position much faster and more safely. As a byproduct of this research, a single hall-sensor joystick has been developed, which is highly linear and durable compared to the conventional joystick. Therefore, this single hall-sensor joystick is expected to be popular in the tele-operations of mobile robots and precision machines with haptic feedback. Incorporations of motor nonlinearities and human hand dynamics are left as future research avenues to convey the environmental information to the operator more realistically.
REFERENCES [1] J. Borenstein and Y. Koren, "Tele-autonomous Guidance for Mobile Robots," IEEE Trans. on System, Man and Cyber., vol. 20, no. 6, pp. 1437-1443, 1990. [2] D. Schulz, W. Burgard, D. Fox, S. Thrun and A.B. Cremers, "Web interfaces for mobile robots in public places," IEEE Robotics & Automation Magazine, vol. 7, no. 1, pp. 48-56, 2000. [3] Wagner, C. R and Perrin D.P, "Force Feedback in a Three-Dimension Ultrasound Guided Surgical Task," 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 43-48, 2006. [4] S. S. Lee and J. M. Lee, "Haptic interface design for the tele-surgery," Proc. ofthe CIDAM Workshop on Service Automation and Robotics, pp. 81-91, HongKong, 2000. [5] S. K. An, S. J. Han, H. R. Hur, and J. M. Lee, "Implementation of a Remote Peg-in-Hole Operation Using a Two Degrees of Freedom Force-Reflective Joystick," J of Electrical Engineering and Information Science, vol. 4, no. 3, pp. 315-322, 1999. [6] H. C. Kim, D. H. Kang, and J. M. Lee, "Development of a Non-contact electronic Joystick using Single Hall Sensor," CASS, pp. 68-73, 2005. [7] Lei Xiao and Xiaodial Dong, "The exact transition probability and bit error probability of two-dimensional signaling," IEEE Transactions on Wireless Communications, vol. 54, Issue 4, pp. 1500-1508, 2005. [8] Nohmi. M, Ando. A, and Bock. T "Contact Task by Space Teleoperation Using Force Reflection of Communication Time Delay," IEEE International Symposium on CIRA, pp. 193-198, 2005. [9] Ming ouhyung, Wu-Nan Tasi, and Ming -Chang Tsai, "A Low-Cost Force Feedback Joystick and its use in pc video games," IEEE Transactions on Consumer Electronics, vol. 41, no. 3, 1995.

8(rr)-7.2
6.4 I I

248 -.I.

(mX
ihuocedak

Fig.7

Teeoerto of th moil ooM r

VI. CONCLUSIONS In this paper, a collision vector is newly defined as a normal vector from an obstacle to the mobile robot to

improve the obstacle avoidance performance of the mobile robot under tele-operation as well as autonomous navigation. The reflected force to the operator through the joystick, is determined based on the modified virtual impedance algorithm, which uses the collision vector. To show the effects of the reflected force in tele-operation, the reflected force of the joystick has been measured by a 6 d.o.f sensor installed in the gripper. On account ofthe nonlinearities ofthe motors to the PWM input, the measured value does not exactly match to the virtual force calculated to avoid the obstacles. In the experiments, the error can be kept within 500. The efficiency of obstacle avoidance is proved by comparing the results of two tele-operation experiments: with and without the reflection force in the joystick. To emphasize the necessity of haptic feedback and to weaken usability of the camera, the experimental environment is kept dark. Through analysis, it can be shown that by using the joystick with haptic

211