You are on page 1of 46

BRAINPORT VISION DEVICE

CHAPTER 1
INTRODUCTION
The BrainPort vision device provides information to blind individuals via a neurostimulating array placed on the tongue. This unique device provides immediate benefits
to its blind users in areas of safety, mobility, and recreation and opens a new world of
sensory experience and exploration.
The BrainPort vision device operates by acquiring an image stream from a camera,
similar to a camcorder. Like a camcorder, the moving images are sent to a display,
which, in this case, is the electrode array on the tongue. The image stream is displayed
on the tongue by converting light information to electrical stimulation, which feels like
microscopic bubbles to the user. With minimal training, users learn to interpret the
images on their tongue as information about the scene in front of them.
The BrainPort vision device includes an imaging system capable of working both
indoors and outdoors, with a field of view spanning 3-75 degrees (magnified versus
wide angle views). The tongue array contains 400 electrodes and is connected to the
controller via a flexible cable. The control system is approximately the size of a PDA
and runs for about 3 hours per charge, with swappable batteries.
An artist’s concept of the tongue array (or Intra-Oral Device, IOD) and camera
mounting is shown below alongside the device. The IOD is attached to a flexible boom
by a thin wire. The camera unit is mounted on a pair of eyeglasses frames. The user
controls and the power supply are connected to a belt-worn, pager-style, controller.
A blind woman sits in a chair holding a video camera focused on a scientist sitting in
front of her. She has a device in her mouth, touching her tongue, and there are wires
running from that device to the video camera. The woman has been blind since birth
and doesn't really know what a rubber ball looks like, but the scientist is holding one
and when he suddenly rolls it in her direction, she puts out a hand to stop it. The blind
woman saw the ball through her tongue. Well, not exactly through her tongue, but the
device in her mouth sent visual
DEPT OF ECE,MKITW,Rajampet

Page 1

BRAINPORT VISION DEVICE

2

DEPT OF ECE,MKITW,Rajampet

Page 2

input through her tongue in much the same way that seeing individuals receive visual
input through the eyes. In both cases, the initial sensory input mechanism -- the tongue
or the eyes - - sends the visual data to the brain, where that data is processed and
interpreted to form images. Braille is a typical example of sensory substitution -- in this
case, you're using one sense, touch, to take in information normally intended for
another sense, vision. Electro tactile stimulation is a higher-tech method of receiving
somewhat similar (although more surprising) results, and it's based on the idea that the
brain can interpret sensory information even if it's not provided via the natural channel.

3

Africa (7 million) and China (6 million) Every 5 seconds: One person in our world goes blind 75 million: People will be blind by 2020 (if trends continue) .2.1 Statistics on the Blind: 37 million: People in the world are blind India (9 million).

AI and cybernetics have each gone in and out in the search for machine intelligence So “I Can Read” can be termed as a “Cybernetics System For Disabled (blind)” 4 .2 Cybernetics: Cybernetics is about having a goal and taking action to achieve that goal. Ironically but logically. "Cybernetics" comes from a Greek word meaning "the art of steering“.2.

which is billed as the next best thing to sight The technology has made the dark-dependent world come alive and independent to Craig Lundberg who completely lost his sight after a grenade attack in Iraq. BrainPort could provide vision-impaired people with limited forms of sight. 24. The device which sends visual input through tongue in much the same way that seeing individuals receive visual input through the eyes is called the “BrainPort Vision Device”. which is making the world visible to the ones who lose their sight due to some accidental incidents. is the first British soldier to test the BrainPort system. Craig Lundberg.Chapter 2 What is BrainPort Vision Device? "BrainPort device" is a technology developed in US. The soldier admits that his world has been transformed because of the technology. as he is now able to sense the visuals with his tongue. A Neuro scientist has developed the BrainPort Vision Device that allows the blinds to “see” using their tongue. .

To produce tactile vision.Technically. BrainPort uses a camera to capture visual data. this device is underlying a principle called “electro tactile stimulation for sensory substitution”. 5 .

It represent a symbiosis between instrumentation-for example. Vibrotactile (FINGERTIPS).. a head-mounted camera under the motor control of the neck muscles). and the human user. through a motor system (e. to determine the origin of the information. This can now be extended into other domains with modern technology and the availability of artificial sensory receptors. and (c) the training and psychosocial factors that support the functional demand. and for the brain.Chapter 3 Modes of stimulation and its various forms From the very beginning of the electro tactile stimulation this journey has travelled a lot and the various forms may be described as follows: TVSS-Tactile Vision Substitution Systems. Consistent with the terminology of this issue. Electro Tactile Stimulation for Tongue 3. this is made possible by "instrumental sensory plasticity. it is only necessary to present environmental information from an artificial sensor in a form of energy that can be mediated by the receptors at the human-machine interface.1 Tactile vision substitution system The TVSS may be characterized as a humanistic intelligence system. To constitute such systems then. an artificial sensor array (TV camera)computational equipment." the capacity of the brain to reorganize when there is: (a) functional demand.(b) the sensor technology to fill that demand.g. such as: .

or 5.1. A MEMS technology accelerometer for providing substitute vestibular information for persons with bilateral vestibular loss. Touch and shear-force sensors to provide information for spinal cord injured persons. 6 . 3. 1999). 4. A sensate robotic hand (Bach-y-Rita. A miniature TV camera for blind persons. 2. Instrumented condom for replacing lost sex sensation.

instead of on the skin. cf. & Collins. Kaczmarek. looming and zooming. They learn to make perceptual judgments using visual means of analysis. These have included facial recognition. Kaczmarek. Bach-y-Rita. 1972. Bach-y-Rita. 1995. White. the visual information reaches the perceptual levels for analysis and interpretation via somatosensory pathways and structures.In first sensory substitution project. 1970). 1989. . Webster. and depth judgments (Bach-y-Rita. and complex inspection-assembly tasks. Although the TVSS systems have only had between 100 and 1032 point arrays. Bach-y-Rita. 1995. & Garcia-Lara. 1998. 3.. such as perspective. Saunders. Optical images picked up by a TV camera were transuded into a form of energy (vibratory or direct electrical stimulation) that could be mediated by the skin receptors. Tyler. the low resolution has been sufficient to perform complex perception and "eye"hand coordination tasks. Kaczmarek & Bach-y-Rita. White. 1999. accurate judgment of speed and direction of a rolling ball with over 95% accuracy in batting a ball as it rolls over a table edge. Bach-y-Rita. pp. parallax. The latter were performed on an electronics company assembly line with a 100 point vibrotactile array clipped to the work-bench against which the blind worker pressed the skin of his abdomen. 1998. 187-193). 1995. Scadden. back thigh). our subjects reported experiencing the image in space. Saunders. & Scadden. 1987. In these sensory substitute systems. and through which information from a TV camera (Substituting for the ocular piece of a dissection microscope) was delivered to the human-machine interface (Bach-y-Rita. & Meier. & Crabb. 1996. 1969. Bach-y-Rita. Tompkins. Collins. they developed tactile vision substitution Systems (TVSS) to deliver visual information to the brain via arrays of stimulators in contact with the skin of one of several parts of the body (abdomen.2 Vibrotactile After sufficient training with the TVSS.

7 .

which is obtained by the child's head-mounted camera.Child reproducing perceived image of a teachers hand as displayed on a modified Optacon. LED monitor in foreground is visual representation of active pattern on the tactile display. . The tactile image is picked up with one finger statically placed on the 6 × 24 vibrotactile array.

8 .

In that case. and multimodal and multidimensional stimulation is possible. The electronic system has been described elsewhere (Bach-y-Rita. The results obtained with a small electrotactile array developed for a study of form perception with a finger tip demonstrated that perception with electrical stimulation of the tongue is somewhat better than with finger-tip electrotactile stimulation. Tyler. assures good electrical contact. Aiello (1998a) suggested that the best way to encode intensity information independent of other percept qualities with a multidimensional stimulus waveform was through modulation of the energy delivered by the stimulus. However. and the tongue requires only about 3% of the voltage (5-15 V).. For example. and the frame rate.3 Electrotactile Stimulation for Tongue In the TVSS studies cited above.4-2. the burst interval. The tongue interface overcomes many of these. and may elicit potentially distinct responses. 1998b) has identified six stimulus parameters: the current level. the number of pulses in a burst. the pulse width. Simultaneously. the stimulus arrays presented only black-white information." Aiello (1998a. The presence of an electrolytic solution. Kaczmarek. . in principle.3. was minimal (gradient mode of stimulation). in a study of electrical stimulation of the skin of the abdomen. the tongue electrotactile system does present gray-scaled pattern information.0 mA). its implementation could be included within a microelectronic package for signal treatment. and much less current (0. than the finger-tip. the sensory receptors are close to the surface. Although the gradient mode of stimulation requires a real-time fulfilment of mathematical constraints among all the parameters. saliva. we have also modelled the electrotactile stimulation parameter space to determine how we might elicit tactile "colors. the interval between pulses. the energy was varied in such a way that the displacement in the parameter space. et al. Since it is in the protected environment of the mouth. corresponding to a given transition between energy levels. The tongue is very sensitive and highly mobile. be varied independently within certain ranges. All six parameters in the waveforms can. without gray scale.

1998). 9 .

Electrotactile stimuli are delivered to the dorsum of the tongue via flexible electrode arrays (Figure 3) placed in the mouth. The electrotactile stimulus consists of 40-μs pulses delivered sequentially to each of the active electrodes in the pattern. The tongue electrode array and cable are made of a thin (100 μm) strip of polyester material (Mylar®) onto which a rectangular matrix of gold-plated copper circular electrodes has been deposited by a photolithographic process similar to that used to make printed circuit boards. This structure was shown previously to yield strong. Positive pulses are used because they yield lower thresholds and a superior stimulus quality on the fingertips. . Bursts of three pulses each are delivered at a rate of 50 Hz with a 200 Hz pulse rate within a burst. comfortable electrotactile precepts. with connection to the stimulator apparatus (TDU) via a flat cable passing out of the mouth.

Close-up of 144-point (12 x 12) "virtual ground" electrotactile tongue display 10 .

0 mV and a 5 : 1 signal-to-noise ratio (SNR).and negative-polarity stimulation pulses of ten different widths: 20. electrotactile entrainment currents (iEN) were determined by adjusting the stimulation current from near zero to the minimal value resulting in one AP for each stimulation pulse. producing an output pulse whenever the recorded signal entered a predefined amplitude-time window. 70. Following pre amplification and band pass filtering (200-10 000 Hz).1 to 1. These currents exceeded the absolute thresholds (the currents causing occasional AP's) by approximately5%. The width sequence was reversed during the second run on each of the three fibers. 200. 40. 50.Action potentials (AP's) thus recorded had amplitudes from 0. a differential amplitude detector identified AP's. . In the first experiment. delivered at a rate of 10 pulses/s. 150.A circular electrode surrounding the recording site served as the ground reference. 300. 100. 30. and 500 _s. The entrainment current was determined twice for positive.

Relative timing between simultaneous mechanical and electrotactile stimulation. 11 . The top trace represents the sinusoidal. 30-Hz. 50-100-_m (0-P) mechanical displacement.

controlled. The idea is to communicate non-tactile via electrical stimulation of the sense of touch." For a blind person. it would turn into spatio-temporal nerve patterns of impulse along the optic nerve fibers. Indeed. it means the encoding of the electrical Pattern essentially attempts to mimic the input that would normally be received by the non-functioning sense – vision. the optical image entering the eyes does not go beyond the retina. for instance) applies small. painless currents (some subjects report it feeling something like soda bubbles) to the skin at precise locations according to an encoded pattern. ears and skin those carry sensory information to the brain are set up in a similar manner to perform similar activities.Chapter 4 When a human looks at an object. In practice. the big challenge to the scientists is how to correctly encode the nerve signals for the sensory event and send them to the brain through the alternate channel as the brain appears to the flexible when it comes to interpreting sensory input. The concepts at work behind electro tactile stimulation for sensory substitution are complex. the channels such as eyes. the brain recreates the images. To substitute one sensory input channel for another. By analysing the impulse patterns. Instead. . this typically means that "an array of electrodes receiving input from a nontactile information source (a camera.

the parietal lobe in the brain receives touch information.So patterns of light picked up by a camera to form an image are replacing the perception of the eyes and converted into electrical pulses that represent those patterns of light. while the occipital lobe receives vision information. In other words. When the nerve fibers forward the image-encoded touch signals to the 12 . Under normal circumstances. the skin is actually receiving image data which would be then sent to the brain in the forms of impulse. when the encoded pulses are applied to the skin.

parietal lobe. "the electric field thus generated in subcutaneous tissue directly excites the afferent nerve fibres responsible for touch sensations". arrays of electrodes can be used to communicate non-touch information through pathways to the brain normally used for the touch related impulses. The breakthrough of the BrainPort technology is to use the tongue as the substitute sensory channel. . Within the system.

13 .

Chapter 5 THE STRUCTURE OF THE BRAINPORT DEVICE The figure below shows the structure of the BrainPort Vision Device. .

"The encoded signal represents differences in pixel data as differences in pulse characteristics such as frequency. 14 . the CPU runs a program that turns the camera's electrical information into a spatially encoded signal. Each set of pixels in the camera's light sensor corresponds to an electrode in the array. amplitude and duration. After that.The optical information that would normally hit the retina is picked up by the digital camera in digital form. It uses radio signals to send the ones and zeros to the CPU for encoding.

" Then. pulse duration. a base unit. black pixels as no stimulation. HOW DOES THE BRAINPORT VISION DEVICE WORK? The BrainPort vision system consists of a postage-stamp-size electrode array for the top surface of the tongue (the tongue array). Users often report the sensation as pictures that are painted on the tongue with Champagne bubbles. a digital video camera. At last. intervals between pulses and the number of pulses in a burst. The base unit translates the visual information into an stimulation pattern that is displayed on the tongue. . with the ability to invert contrast when appropriate. and gray levels as medium levels of stimulation. The tactile image is created by presenting white pixels from the camera as strong stimulation. the brain interprets and uses the information coming from the tongue as if it were coming from the eyes. and a handheld controller for zoom and contrast inversion. among other parameters. the electrode array (shown in Fig) receives the resulting signal via the stimulation circuitry and applies it to the tongue. Visual information is collected from the user-adjustable head-mounted camera (FOV range 3–90 degrees) and sent to the BrainPort base unit.Multidimensional image information takes the form of variances in pulse current or voltage.

15 .

.

.

1 6 .

study participants have been able to recognize high-contrast objects.With the current system (arrays containing 100 to 600+ electrodes). software. it requires less voltage to stimulate nerve fibres in the tongue (5 to 15 volts) compared to areas like the fingertips or abdomen (40 to 500 volts). the tongue is the best choice for conveying tactile-based data to the brain until this moment . Also. Last but not least. there is no stratum conium (an outer layer of dead skin cells) on the tongue which act as an insulator. and usability. their location. movement. and on overall device miniaturization. Therefore. we are currently working on improvements to the tongue display hardware. With these characteristics. Trained blind participants use information from the tongue display to augment understanding of the environment. the tongue skin area is the most sensitive one because there are more nerve fibres and they are much closer to the surface. since the tongue is surrounded by saliva which contains electrolytes. As a result. the area of the cerebral cortex that interprets touch data from the tongue is larger than the areas serving other body parts. it would help to maintain the current flow between the electrode and skin tissues. WHY TONGUE IS USED IN BRAINPORT TECHNOLOGY Compare to all other skin areas. Moreover. and some aspects of perspective and depth. Our ongoing research with the BrainPort vision device demonstrates the great potential of tactile vision augmentation and we believe that these findings warrant further exploration.

17 .

at home. such as parking lots.Chapter 6 ADVANTAGES: Blind users can use the Brain Port vision device independently . school and in business. traffic circles. These benefits enable greater independence at home. at work. SAFETY Navigating difficult environments. Secondary benefits include applying the technology toward specific hobbies and recreational situations. mobility and object recognition. complex intersections Recognizing quiet moving objects like hybrid cars or bicycle . greatly improving quality of life. and in public spaces indoors and out as a tool for improved safety.

18 . Finding continuous sidewalks. lobby or restaurant in an office or hotel. hall intersections.MOBILITY Finding doorways. sidewalk intersections and curbs.

. coffee mug. cane.Chapter 7 OBJECT RECREATION Locating people Locating known objects such as shoes. The BrainPort electrodes would receive input from a sonar device to provide not only directional cues but also a visual sense of obstacles and terrain. keys APPLICATIONS Just a few of the current or foreseeable applications include providing elements of sight for the visually impaired in the medical field.

BrainPort may also provide expanded information for military pilots. The surgeon could wear electro tactile gloves to receive tactile input from robotic probes inside someone's chest cavity. 19 . BrainPort applications include robotic surgery. such as a pulse on the tongue to indicate approaching aircraft or to indicate that they must take immediate action.

.

.

2 0 .

They are looking into underwater applications that could provide the Navy Seals with navigation information and orientation signals in dark. .Chapter 8 CURRENT AND POTENTIAL APPLICATIONS While the full spectrum of BrainPort Vision Technology applications has yet to realized. BrainPort may also provide expanded information for military pilots. scientists have been exploring potential military uses with a grant from the Defence Advanced Research Projects Agency (DARPA). murky water. such as a pulse on the tongue to indicate approaching aircraft or to indicate that they must take immediate action. The BrainPort electrodes would receive input from a sonar device to provide not only directional cues but also a visual sense of obstacles and terrain. Just a few of the current or foreseeable applications include providing elements of sight for the visually impaired in the medical field. the device has the potential to lessen an array of sensory limitations and to alleviate the symptoms of a variety of disorders. Beyond medical applications.

the surgeon could feel what he's doing as he controls the robotic equipment.Other potential BrainPort applications include robotic surgery. 21 . In this way. The surgeon could wear electro tactile gloves to receive tactile input from robotic probes inside someone's chest cavity. and gamers might use electro tactile feedback gloves or their controllers to feel what they're doing in a video game. Race car drivers might use a version of BrainPort to train brains for faster reaction times.

it has just picked up in this decade due to the miniaturization of electronics and dramatic improvement of the computer's speed. Already more streamlined than any previous setup using electro tactile stimulation for sensory substitution.we might be seeing a camera with the size of a grain of rice embedded in people's foreheads. It's not exactly a system on a chip. In the case of the BrainPort Vision Device the electronics might be completely embedded in a pair of glasses along with a tiny camera and radio transmitter.Chapter 9 CONCLUSION Even though this is a field of scientific study that has been around for nearly a century. and the mouthpiece would house a radio receiver to receive encoded signals from the glasses. but gives it 20 years -. BrainPort envisions itself even smaller and less obtrusive in the future. .

22 .

com/printarticle.pdf http://wistechnology. http://howstuffworks.suite101.cfm/brainport_vision_device http://www.telegraph.scientificamerican.uk/health/healthnews/7350218/British-soldierblinded-in-Iraq-trials-new-technology-to-see-using-his-tongue.net/2009/08/16/brainport-vision-device-lets-blinds-seewith-tougues/ http://www.gizmowatch.com/article.php?id=2319 http://www.com http://india.cfm?id=device-lets-blind-see-withtongues http://topnews.net.com/article.berkeley.cchem.com/entry/brainport-vision-device-for-thevisually-impaired-to-feel-images/ http://accessibletravel.edu/~reimer/courses/present/samples/sample 2. Paul et al.co." Journal of Rehabilitation Research and Development. "Form perception with a 49-point electro tactile stimulus array on the tongue: A technical note.nz/content/22766-brainport-device-transforms- world-blinds http://www.itechnews.REFERENCES Bach-y-Rita.html .

23 .