An Attempt to Extend Human Sensory Capabilities by Means of Implant Technology

Kevin Warwick, Mark Gasson, Benjamin Hutt and Iain Goodhew Department of Cybernetics, School of Systems Engineering, University of Reading, Whiteknights, Reading, RG6 6AY, UK.
( K.Warwick@rdg.ac.uk, M.N.Gasson@rdg.ac.uk, B.D.Hutt@rdg.ac.uk & I.C.B.Goodhew@rdg.ac.uk ) Abstract - In this paper an attempt is described to increase the range of human sensory capabilities by means of implant technology. The key aim is to create an additional sense by feeding signals directly to the human brain, via the nervous system rather than via a presently operable human sense. Neural implant technology was used to directly interface a human nervous system with a computer in a one off trial. The output from active ultrasonic sensors was then employed to directly stimulate the human nervous system. An experimental laboratory set up was used as a test bed to assess the usefulness of this sensory addition. Keywords: Implant Technology, Micro Electrode Array, Sensory Enhancement, Neural Interface. body, wearable computers require signal conversion to take place to interface the technology with the human sensory receptors. Of considerably greater interest are studies in which a direct link is formed between the nervous system and technology. In this respect, numerous animal studies have been carried out. In one of these, the extracted brain of a lamprey was used to control the movement of a smallwheeled robot to which it was attached [8]. This procedure utilised the fact that the innate response of a lamprey is to position itself in water by detecting and reacting to light playing on the water surface. When surrounded by a ring of lights this unconditioned behaviour was decoded into motion commands to move the robot around an arena towards the active light source. A number of relevant studies have also been carried out involving rats as the subjects. In the first of these [9], an environment was created such that when rats pulled a lever they received a liquid reward. By chronically implanting electrodes in the motor cortex of the rats brains and decoding the neuronal population activity, signals could be extracted directly when each rat thought about pulling the lever, but before any physical movement occurred. This information was used to directly release the reward before a rat pulled the lever. Over the period of a few days, four of the six rats involved learned that they need not initiate the action in order to obtain the reward; merely thinking about it was sufficient. In another rat study [10], direct brain stimulation of rats was employed in order to teach them to solve a maze problem. Reinforcement learning was used with pleasurable stimuli evoked when a rat moved in the correct direction. For humans, studies looking at the integration of technology with the human central nervous system have varied from diagnostic [11], to the amelioration of symptoms [12], [13], [14], to the augmentation of existing senses [15], [16]. But the most widely reported research involving human subjects is that based on the development of an artificial retina [17]. In this work, small arrays have been successfully attached to a functioning optic nerve, but

Introduction

Research is currently being carried out (e.g. [1], [2]) in which biological signals are directly measured, operated upon by appropriate signal processing techniques and then used to either control a device or utilised as an input to some type of feedback mechanism. In the vast majority of cases the signals are measured externally to the body which presents innumerable communication and measurement problems [3], [4]. Whatever measurement system is used, losses and errors occur due to the signal attenuation through the body and compounded noise issues associated with small signal detection. Problems also arise when translating electrical energy from the computer to the electrical energy that is apparent on the nervous system. When only external stimulation is apparent, for example mechanical vibration or thermal variation, then it is not possible to select unique sensory receptor channels due to the general nature of the stimulation procedure. As far as human studies are concerned, wearable computers provide one possible route for linking humans and technology together. Items such as shoes and glasses have been augmented with microcomputers [5] [6], and indications of stress and alertness can be monitored, with the state of the device altered to affect the wearer. In a widely reported application a miniature computer screen was fitted onto an otherwise standard pair of glasses [7]. In this way the wearer was given remote vision, in which additional information about a remote scene could be relayed. However, although positioned adjacent to the

where the person has no vision. By means of direct stimulation of the nerve with appropriate signal sequences this has resulted in the user perceiving simple shapes and even letters. Electronic neural stimulation has proved to be extremely successful in other areas, with applications ranging from cochlea implants to the treatment of Parkinsons disease symptoms. The most relevant to this study is the use of a brain implant, which enables a brainstem stroke victim to control the movement of a cursor on a computer screen [18], [19]. Initially functional magnetic resonance imaging of the subjects brain was made. When the subject was asked to think about moving his hand the fMRI scanner was used to localise where the activity was most pronounced. A hollow glass electrode cone containing two gold wires (Neurotrophic Electrode) was then implanted into the motor cortex, being positioned in the area of maximum-recorded activity. With the electrode in place, when the patient thought about moving his hand the output from the electrode was amplified and transmitted by a radio link to a computer where the signals were translated into control signals to bring about movement of the cursor. The subject successfully learnt to move the cursor around by thinking about different movements. The Neurotrophic Electrode uses tropic factors to encourage nerve growth in the brain. Over the period that the implant was in place, no rejection of the implant was observed; indeed the neurons grew into the electrode allowing stable long-term recordings. In each of the cases in which human subjects are involved, the aim is to either bring about some restorative functions when an individual has a physical problem of some kind, e.g. they are blind, or conversely it is to give a new ability to an individual who has very limited abilities of any kind due to a major malfunction in their brain or nervous system. In this paper we are concerned with neither of these situations. Rather we consider here the possibility of giving an extra capability to an individual human, to enable them to achieve a broader range of successful sensory input. In the following section we describe the MicroElectrode Array (MEA), which was implanted into a human nervous system to act as the silicon/biological interface. Subsequently, a pilot study is described in which the output from an ultrasonic sensory arrangement is employed to directly stimulate the human nervous system, via the implant, such that ultrasonic signals are proportional, over a restricted yet respectable range, to the extent of nervous system stimulation. It should be emphasised that what is described here is an actual pilot application study involving a human test case rather than a computer simulation.

Figure 1. A 100 electrode, 4x4mm Micro Electrode Array, shown on a UK one pence piece for scale.

Microelectrode Array

When a direct connection to the human nervous system is required, there are, in general, two approaches for peripheral nerve interfaces, extraneural and intraneural. The cuff electrode is the most common extraneural device, and this fits tightly around the nerve trunk, allowing for the recording of the sum of the single fibre action potentials, known as the compound action potential (CAP). This can be used for recording or for crudely selective neural stimulation of a large region of the nerve trunk. In some cases the cuff can contain a second or more electrodes, thereby allowing for an approximate measurement of signal speed travelling along the nerve fibres. For applications which require a much finer granularity of neural signal for both selective monitoring and stimulation however an intraneural interface such as a MicroElectrode Array (MEA) is particularly well suited. The MEA employed in this work contains a total of 100 electrodes which, when implanted, become distributed within the nerve fascicle. In this way, it is possible to gain direct access to muscle spindles, motor neural signals to particular motor units and sensory receptors. Essentially such a device allows a bi-directional link between the human nervous system and a computer [20][21][22]. On 14 March 2002, during a 2 hour procedure at the Radcliffe Infirmary, Oxford, an MEA was surgically implanted, into the median nerve fibres of the left arm of the first named author (KW). The array measured 4mm x 4mm with each of the electrodes being 1.5mm in length. With the median nerve fascicle estimated to be 4mm in diameter, the electrodes penetrated well into the fascicle. The array was pneumatically inserted into the median nerve such that the body of the array sat adjacent to the nerve fibres with the electrodes penetrating into the fascicle [22].

The arrangements described remained permanently in place for 96 days, until 18th June 2002, at which time the implant was successfully removed.

Correctly Identified Pulses (%)

The array was positioned just below the wrist, following a 4cm long incision. A further incision, 2 cm long, was made 16cm proximal to the wrist. The two incisions were connected by a tunnelling procedure such that wires from the array ran up the inside of the left arm where they exited and connected onto an electrical terminal pad which remained external.

possible via the radio link from the local PC or the remote PC via the encrypted tunnel.
100 90 80 70 60 50 40 30 20 10 0 0 20 40 60 80 100 Stimulation Pulse No Stimulation Pulse Stimulation Current (A)

Nerve Stimulation
Figure 2. Effect of stimulation amplitude on the number of correctly identified pulses and absence of pulses (over 100 trials)

The array, once in position, acted as a neural interface. Signals could be transmitted directly from a computer, by means of either a hard wire connection or through a radio transmitter/receiver unit, to the array and thence to directly bring about a stimulation of the nervous system. It was found during experimentation that typical activity in the median nerve fibers occurred around a centroid frequency of approximately 1 KHz and it was noted that signals of interest occurred below 3.5 KHz. The authors wish to point out here however that significant, relevant, measurable signals were witnessed well above a frequency of 1 KHz, which has to this time been considered as the upper limit frequency of interest [23]. The authors suspect that such a limit may have been inadvertently placed historically because of the type of recording/measuring device (e.g. glass micropipette) widely employed. However noise was a distinct problem due to inductive pickup on the wires. This noise needed to be severely reduced and to this end a fifth order band limited Butterworth filter was used with corner frequencies of flow= 250 Hz and fhigh = 7.5 KHz. To help reduce the noise pickup from the umbilical wires between the implanted electrode array and the bank of filters, and to allow freedom of movement, a small wearable radio device was developed to be worn on a gauntlet around the wrist. This custom hardware consisted of a 20 way multiplexer, two independent filters, two 10 bit A/D converters, a microcontroller and an FM radio transceiver module. Either 1 or 2 electrodes from the array could be quasi-statically selected, digitized and their output transmitted over the radio link to a corresponding receiver connected to a PC. At this point they could either be recorded or transmitted further in order to operate networked technology, as described in the following section. Onward transmission of the signal was via an encrypted TCP/IP tunnel, over the local area network, or wider internet. Remote configuration of various configuration parameters on the wearable device was also

Figure 3. Voltage profile during one bi-phasic stimulation pulse cycle with a constant current of 80A. Stimulation of the nervous system by means of the array was especially problematic due to the extremely limited nature of existing results using this type of interface. Published work is restricted largely to a respectably thorough but short-term study into the stimulation of the sciatic nerve in cats [21]. Much experimental time was therefore required, on a trial and error basis to ascertain what voltage/current relationships would produce a reasonable (i.e. perceivable but not painful) level of nerve stimulation. Further factors which may well emerge to be relevant, but were not possible to predict in this experimental session were: (a) The plastic, adaptable nature of the human nervous system and the brain even over relatively short periods. (b) The effects of movement of the array in relation to the nerve fibres, hence the connection and associated input

impedance of the nervous system was not completely stable. After extensive experimentation it was found that injecting currents below 80A onto the median nerve fibers had little perceivable effect. Between 80A and 100A all the functional electrodes were able to produce a recognizable stimulation, with an applied voltage of 40 to 50 volts, dependant on the series electrode impedance. Increasing the current above 100A had no apparent additional effect; the stimulation switching mechanisms in the median nerve fascicle exhibited a non-linear thresholding characteristic. During this experimental phase, it was pseudo randomly decided whether a stimulation pulse was applied or not. The user, wearing a blindfold, was unaware of whether a pulse had been applied or not, other than by means of its effect in terms of neural stimulation. The users accuracy in distinguishing between an actual pulse and no pulse at a range of amplitudes is shown in Figure 2. In all subsequent successful trials, the current was applied as a bi-phasic signal with pulse duration of 200 sec and an inter-phase delay of 100 sec. A typical stimulation waveform of constant current being applied to one of the MEAs implanted electrodes is shown in Fig 3. It was therefore possible to create alternative sensations via this new input route to the nervous system. It must be reported however that it took 6 weeks for the recipients brain to repetitively recognize the stimulating signals accurately. This time period can be due to a number of contributing factors: (a) The team had to learn which signals (what amplitude, frequency etc.) would be best in order to bring about a recognizable stimulation. (b) The recipients brain had to learn to recognize the new signals it was receiving. (c) The bond between the recipients nervous system and the implant was physically changing (becoming stronger) Figure 4. Experimentation and testing of the ultrasonic baseball cap. Ultrasonic sensors were fitted to the rim of a baseball cap (see Figure 4) and the output from these sensors, in the form of a proportional count, was employed to bring about a direct stimulation of the nervous system. Hence when no objects were in the vicinity of the sensors, no stimulation occurred, and as an object moved close by so the rate of stimulation pulses being applied increased in a linear fashion up to a pre-selected maximum rate. No increase in stimulation occurred when an object moved closer than 30cm to the sensors. The ultrasonic sensors were open type piezoelectric ceramic transducers with conical metal resonators and operated at 40 KHz. These were used in a pair, one for transmit and one for receive, to give maximum sensitivity for small and distant objects. The most useful range for the experimentation was found to be 2 3m, this being also dependent on the size of object. A simple microcontroller was programmed to perform the echo ranging on the pair of transducers, and provide the range to the first detectable object only. This was translated into a stimulation pulse train, which operated on a single pin of the electrode array. Pins on the array had been tested for their suitability for stimulation by the earlier experimentation in which the recipient identified the presence or absence of stimulation pulse trains at various amplitudes and repetition frequencies. It was found that very little learning was required for the new ultrasonic sense to be used effectively and successfully merely a matter of 5/6 minutes. This said it must be remembered that it had already taken several weeks for the recipients brain to successfully, accurately recognize the current signals being injected. As a result, in a witnessed experiment, the recipient, whilst wearing a blindfold, was able to move around successfully within a cluttered laboratory environment, albeit at a slower than normal walking pace. The sensory input was felt as a new form of sensory input (not as

Sensory Experiment

An experiment was set up to determine if the human brain is able to understand and successfully operate with sensory information to which it had not previously been exposed. Whilst it is quite possible to feed in such sensory information via a normal human sensory route, e.g. electromagnetic radar or infra-red signals are converted to visual, what we were interested in was feeding such signals directly onto the human nervous system, thereby bypassing the normal human sensory input.

touch or movement) in the sense that the brain made a direct link between the signals being witnessed and the fact that these corresponded in a linear fashion to a nearby object. The extent of this perception was also such that when an object was rapidly brought into the recipients line of (ultrasonic) sight, this had the effect of frightening the recipient.

On the positive side however, a further aspect of the research was to investigate the human bodys acceptance or rejection of such an implant. No infection whatsoever was witnessed during the course of the experiment and during extraction it was observed that body tissue had grown around the array holding it in its original place. The authors are aware that this is a one off pilot study based on only one recipient. It may well be that other recipients react in other ways. However the study can be taken as practical evidence of the concept. Before any large-scale conclusions can be drawn on the usefulness and effect of such sensory input, many further trials will be necessary though. The team are now involved in such studies and further implant trials will occur in due course. It is recognised that as far as human extrasensory input is concerned, what has been achieved is an extremely primitive first step. Indeed it may well be the case that implants of the type used here are not ultimately those selected for a good link between a computer and the human brain. Nevertheless the results obtained were extremely encouraging and lead one to hope that a rich form of alternative sensory input will be possible in the future.

Discussion

One aspect of this study is the potential use of these types of implants to help those with a lesion in their nervous system. The aim being to bring about some, otherwise missing movement; to return control of body functions to the bodys owner [23]; or to allow a recipient to operate technology around them [24], generally with the target of servicing the communication and control needs of people with severe motor disabilities. The particular line of research described in this paper however could have an immediate impact for those who are blind, not in any way to repair their blindness, but rather to give them an alternative sense. This would allow them to move around in the world with an ultrasonic knowledge of any nearby objects. Certainly the success of our one off study indicates that it may be possible for a person who is blind to be fitted in the same way with an implant and make relatively rapid use of it. A big question remains however as to how different brains adapt to such a new input stream. One shortfall of the study described was the potential impact of other sensory information in carrying out the trials. By means of the blindfold only the visual sense was removed, in particular audio was retained. The new ultrasonic sense was then operated in coordination with the normal audio input. We cannot be clear therefore how much of a role the audio input played in allowing for successful maneuvering around the laboratory. Another shortfall was the single location of the ultrasonic sensors on the baseball cap rim. In this position they could be regarded as substitute eyes, thereby giving the brain a relatively easy task of replacing input from the blindfolded eyes with ultrasonic information from the same geographical neighborhood. It would have been interesting to investigate alternative positions for the sensors, particularly behind the recipient (quite simply to turn the cap round). Whether the human brain would then have been immediately able to recognize objects positioned behind the body would have been a completely different question that remains to be answered. Indeed via an internet connection the ultrasonic sensors could have been positioned separately from the body, possibly even in another country. How the brain, in these circumstances, would understand the signals being received remains as a research item of interest.

Acknowledgements
Ethical approval for this research to proceed was obtained from the Ethics and Research Committee at the University of Reading and, in particular with regard to the involved neurosurgery, was given by the Oxfordshire National Health Trust Board overseeing the Radcliffe Infirmary, Oxford, UK. Our thanks go to Mr. Peter Teddy and Mr. Amjad Shad who performed the neurosurgery at the Radcliffe Infirmary and ensured the medical success of the project. Our gratitude is also extended to NSIC, Stoke Mandeville and to the David Tolkien Trust for their support. We also wish to thank The Science Museum, Naples, Italy who now have the actual microelectrode array and associated technology employed in this experiment on permanent display.

References
[1] W. Penny, S. Roberts, E. Curran and M. Stokes, EEG-based communication: A pattern recognition approach, IEEE Transactions on Rehabilitation Engineering, Vol. 8, Issue.2, pp. 214-215, 2000. [2] S. Roberts, W. Penny and I. Rezek, Temporal and spatial complexity measures for electroencephalogram based brain-computer interfacing, Medical and Biological Engineering and Computing, Vol.37, Issue.1, pp.93-98, 1999.

[3] J. Wolpaw, D. McFarland, G. Neat and C. Forneris, An EEG based brain-computer interface for cursor control, Electroencephalography and Clinical Neurophysiology, Vol. 78, Issue.3, pp. 252-259, 1991. [4] A. Kubler, B. Kotchoubey, T. Hinterberger, N. Ghanayim, J. Perelmouter, M. Schauer, C. Fritsch, E. Taub and N. Birbaumer, The Thought Translation device: a neurophysiological approach to communication in total motor paralysis, Experimental Brain Research, Vol. 124, Issue.2, pp. 223-232, 1999. [5] J. A. Paradiso, K. Hsiao, A. Y. Benbasat and Z. Teegarden, Design and implementation of expressive footwear, IBM Systems Journal, Vol 39, Issue.3/4, pp. 511529, 2000. [6] E. Thorp, The invention of the first wearable computer, In Proceedings of the Second IEEE International Symposium on Wearable Computers, pp.48, Pittsburgh, October 1998. [7] S. Mann, Wearable Computing: A first step towards personal imaging, Computer, Vol. 30, Issue.2, pp. 25-32, 1997. [8] B. Reger, K. Fleming, V. Sanguineti, S. Simon Alford, and F. Mussa-Ivaldi, Connecting Brains to Robots: an artificial body for studying computational properties of neural tissues, Artificial life, Vol.6, Issue.4, pp.307-324, 2000. [9] J. Chapin, R. Markowitz, K. Moxon and M. Nicolelis, Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neuroscience, Vol.2, Issue.7, pp. 664-670, 1999. [10] S. Talwar, S. Xu, E. Hawley, S. Weiss, K. Moxon and J. Chapin, Rat navigation guided by remote control. Nature, Vol. 417, pp. 37-38, 2002. [11] M. Denislic and D. Meh, Neurophysiological assessment of peripheral neuropathy in primary Sjgrens syndrome, Clinical Investigator, Vol. 72, Issue.11, pp.822-829, 1994. [12] M. S. Poboroniuc, T. Fuhr, R. Riener and N. Donaldson, Closed-Loop Control for FES-Supported Standing Up and Sitting Down, Proc. 7th Conf. of the IFESS, Ljubljana, Slovenia, pp. 307-309, 2002. [13] M.R. Popovic, T. Keller, M. Moran and V. Dietz, Neural prosthesis for spinal cord injured subjects, Journal Bioworld, Vol. 1, Issue.1, pp. 6-9, 1998. [14] N. Yu, J. Chen and M. Ju, Closed-Loop Control of

Quadriceps/Hamstring activation for FES-Induced Standing-Up Movement of Paraplegics, Journal of Musculoskeletal Research, Vol. 5, No.3, pp.173-184, 2001. [15] M. Cohen, J. Herder and W. Martens, Cyberspatial Audio Technology, JAESJ, J. Acoustical Society of Japan (English), Vol. 20, No. 6, pp. 389-395, November, 1999. [16] A. Butz, T. Hollerer, S. Feiner, B. McIntyre and C. Beshers, Enveloping users and computers in a collaborative 3D augmented reality, IWAR99, San Francisco, pp. 35-44, October 20-21, 1999. [17] H. Kanda, T. Yogi, Y. Ito, S. Tanaka, M. Watanabe and Y. Uchikawa, Efficient stimulation inducing neural activity in a retinal implant, Proc. IEEE International Conference on Systems, Man and Cybernetics, Vol 4, pp 409-413, 1999. [18] P. Kennedy, R. Bakay, M. Moore, K. Adams and J. Goldwaithe, Direct control of a computer from the human central nervous system, IEEE Transactions on Rehabilitation Engineering, Vol. 8, Issue.2, pp. 198-202, 2000. [19] P. Kennedy, D. Andreasen, P. Ehirim, B. King, T. Kirby, H. Mao and M. Moore, Using human extra-cortical local field potentials to control a switch, Journal of Neural Engineering, Vol.1, Issue.2, pp.72-77, 2004. [20] M. Gasson, B. Hutt, I. Goodhew, P. Kyberd and K. Warwick, Bi-directional human machine interface via direct neural connection, Proc. IEEE Workshop on Robot and Human Interactive Communication, Berlin, German, pp. 265-270, Sept 2002. [21] A. Branner, R. B. Stein and E. A. Normann, Selective Stimulation of a Cat Sciatic Nerve Using an Array of Varying-Length Micro electrodes, Journal of Neurophysiology, Vol. 54, No. 4, pp. 1585-1594, 2001 [22] K. Warwick, M. Gasson, B. Hutt, I. Goodhew, P. Kyberd, B. Andrews, P. Teddy and A. Shad, The Application of Implant Technology for Cybernetic Systems, Archives of Neurology, Vol.60, No.10, pp. 13691373, 2003. [23] W. Finn and P. LoPresti, Handbook Neuroprosthetic methods, CRC Press, 2003. of

[24] E. Leuthardt, G. Schalk, J. Wolpaw, J. Ojemann and D. Moran, A brain-computer interface using electrocorticographic signals in humans, Journal of Neural Engineering, Vol.1, Issue.2, pp.63-71, 2004.