You are on page 1of 19

Generic visual perception processor

While computing technology is growing in leaps and bounds, the human brain continues to be the world's fastest computer. Combine brain-power with seeing power, and you have the fastest, cheapest, most extra ordinary processor ever-the human eye. Little wonder, research labs the world over are striving to produce a nearperfect electronic eye. The 'generic visual perception processor !"##$' has been developed after %& long years of scientific effort. !eneric "isual #erception #rocessor !"##$ can automatically detect ob'ects and trac( their movement in real-time . The !"##, which crunches )& billion instructions per second *+#,$, models the human perceptual process at the hardware level by mimic(ing the separate temporal and spatial functions of the eye-to-brain system. The processor sees its environment as a stream of histograms regarding the location and velocity of ob'ects. !"## has been demonstrated as capable of learning-in-place to solve a variety of pattern recognition problems. +t boasts automatic normali-ation for varying ob'ect si-e, orientation and lighting conditions, and can function in daylight or dar(ness.

This electronic .eye. on a chip can now handle most tas(s that a normal human eye can. That includes driving safely, selecting ripe fruits, reading and recogni-ing things. ,adly, though modeled on the visual perception capabilities of the human brain, the chip is not really a medical marvel, poised to cure the blind.


College of applied sciences, Mavelikkara

each addressing a particular parameter extracted from the video signal. Mavelikkara . . its si-e or its orientation.T0. such as its shape. +t is desirable to provide devices including combined data processing units of a similar nature. +n particular. and are particularly useful for image processing.uch devices can be termed an electronic spatio-temporal neuron. si-e and orientation of the ob'ect. The present invention also provides a method for perception of an ob'ect using characteristics. each processing a 12T2 2$. 2 College of applied sciences. but may also be used for other signals. by a function in order to generate individually an output value. or electronic spatio-temporal neuron . using a device composed of a set of histogram calculation units. such as audio signals.Generic visual perception processor BAC GROUND O! T"# IN$#NTION The invention relates generally to methods and devices for automatic visual perception. it is desirable to provide devices including multiple units for calculating histograms. The techni/ues of the present invention are also particularly useful for trac(ing one or more ob'ects in real time. and more particularly to methods and devices for processing image signals using two or more histogram calculation units to locali-e one or more ob'ects in an image signal using one or more characteristics an ob'ect such as the shape.

luminance and saturation values in a specific shape. +t would be relatively simple for a C56.perceive. chip to implement in hardware the separate contributions of temporal and spatial processing in the brain. . and then trac( up to eight user-specified ob'ects in a video stream based on hue.ob'ect. spatial orientation. relative position etc. ma'or performance strength over current-day vision systems is its adaptation to varying light conditions. speed and direction of motion. adapt to real time changes in lighting without recalibration. The brain-eye system uses layers of parallel-processing neurons that pass the signal through a series of preprocessing steps. with the bac(ground.. then inside this outline. saturation. as when an ob'ect gets closer to the viewer or moves farther away. Today7s vision systems dictate uniform shadow less illumination . The result is an inexpensive device that can autonomously . defined as a certain set of hue. day or light. designed to wor( under 8normal9 lighting conditions. elements that are characteri-ed by their tone. color. can be used only dawn to dus(. luminance. %OTI#NTIA& 'IG"T#D The !"## was invented in 1992. The !"## trac(s an . are determined. by *4" founder #atric #irim .differences. a general outline of a moving ob'ect is determined with respect to a relatively stable bac(ground.Generic visual perception processor 3sing the techni/ues of the present invention. The !"## on the other hand. The !"##7. resulting in real-time trac(ing of multiple moving ob'ects within a visual scene. #irim created a chip architecture that mimic(ed the wor( of the neurons. with the help of multiplexing and memory. Mavelikkara . 3 College of applied sciences. from frame to frame in a video stream by anticipating where it7s leading and trailing edges ma(e . That means it can trac( an ob'ect through varying light sources or changes in si-e.and even next generation prototype systems.

matches and process patterns. Thus the signal ripples in the neural networ(. This is a crucial difference. Thus !"## becomes an efficient tool for applications li(e the pattern matching and recognition.Generic visual perception processor :or many decades the field of computing has been trapped by the limitations of the traditional processors. *ut there are many situations where there is an inexistence of an algorithm or inability of a human to understand the algorithm. 4ven if pattern is not recogni-able to a human programmer in input the neural networ(. it will dig it out from the input. 2 large number of such neurons interconnected form a neural networ(. +t won7t abruptly fail to wor(. *y their design even if a group of neurons get. from traditional processors as they fail to wor( even if a few components are damaged. 4ach neuron is capable of implementing a simple function. +t can solve a problem with its neural learning function. every time changing the weighted values associated with each input of every neuron. 5any futuristic technologies have been bound by limitations of these processors . 0eural networ(s are extremely fault tolerant. Mavelikkara . !"## recogni-es stores . 4ven in these extreme cases !"## performs well. There are large number of input lines and an output line to a neuron. "O( IT (OR ' *asically the chip is made of neural networ( modeled resembling the structure of human brain. the neural networ( only suffers a smooth degradation of the performance. These changes in the ripples will naturally direct the weights to modify 4 College of applied sciences. +t ta(es the weighted sum of its inputs and produces an output that is fed into the next layer. The weights assigned to each input are a variable /uantity. The basic element here is a neuron. Traditional processors wor( by slicing each and every complex program into simple tas(s that a processor could execute. 4very input that is given to the neural networ( gets transmitted over entire networ( via direct connections called synaptic connections and feed bac( paths. This re/uires an existence of an algorithm for solution of the particular problem.These limitations stemmed from the basic architecture of these processors.

The tra'ectory ends when the system reaches e/uilibrium. then follows them by trac(ing their color through the scene. the networ( activity burrows a tra'ectory in this state space. The chip emulates the eye. wherein each frame includes a succession of pixels whose assembly forms a space. +f the pattern was meant to be matched. the chip identifies ob'ects of interest. +nput to the electronic eye can come from video. *y mimic(ing the eye and the visual regions of the brain. Where as in pattern recognition. When we draw the state diagram of a neural networ(. the rest are red and green$ and %=& million monochromatic rods that are >< times more sensitive than the cones through two processing steps-tonic and phasic.Generic visual perception processor into those values that will become stable .That is. determines each ob'ect's speed and direction.eal-time outputs perceive.o instead of capturing frames of pixels. the !"## puts together the salient features necessary for recognition. 2t this point the information about the signal is stored as the weighted values of inputs in the neural networ(. . only %< per cent of which see . recogni-e and analy-e both static images and time-varying patterns for specific ob'ects. which has < million cones sensitive to color. The tra'ectory begins with a computation problem.Tonic processing auto-scales according to ambient light conditions. The chip mimics the human eye7s two processing steps. speed. The weights associated with inputs gradually change to learn new pattern information. 5ost of the tra'ectory consists of transient behavior or computations. The problem specifies initial conditions which define the beginning of tra'ectory in the state This is the final state of the neural networ(. those values does not change. the final neuronal state represents the pattern that is closest match to the input pattern. infrared or radar signals. for example an image for a twodimensional space. the pattern to be recogni-ed defines the initial conditions. tonic and phasic . Mavelikkara . their heading. 2 neural networ( geometri-es computation. shading and color differences. the pattern to be learned defines the initial conditions. is composed of a succession of frames. "ideo signal . . +n pattern learning. enabling it to adapt to a 5 College of applied sciences.

#irim is wor(ing on a .2s light7s edges pass over the cones and rods and cones of the eye. 2fter all this processing the visual signal is then sent to the brain for higher-level observation and recognition tas(s. +n addition. . these neurons deduce the leading and trailing edge of the ob'ect and mar( with registers 6 College of applied sciences.Generic visual perception processor range of luminosity. The faster 'itter accounts for the eye-sensitivity to the smallest detectable feature. which is an edge moving between ad'acent rods or cones. *ecause only detected movements and color along with the shape and contour of ob'ects is sent to the brain. :or the future. T"# %ROC#''ING D#TAI&' The electronic eye follows exactly the same theoretical processing steps of the real eye with hard-wired silicon circuitry around each pixel in its sensor array. 4ach pixel is read by the vision chip with hardware that determines and scales luminescence. rather than sharp contrast changes. each pixel has two auxiliary neurons that define the -one in which an ob'ect is located-that is. recalls the direction of previous movement. 2 sensor array is a set of several sensors that an information gathering device uses to gather information usually directional in nature$ that cannot be gathered from a single source for a central processing unit. rather than raw pixels.visual. remembers movement in the previous moment. mouse for hand-gesture interface to computers that ta(e advantage of that high compression ratio.. The eye typically sweeps a scene about two to three times a second as well as ma(ing vibratory movements at about %&& ?. these local feedbac( loops detect contrast changes caused by ob'ects moving through the scene :or detecting smooth contours. Mavelikkara . from the direction in which an ob'ect is moving. trac(s color. the average compression ratio information is about %=<. and then deduces the speed of the various detected ob'ects from parallel phasic and tonic neural circuitry. the eye adds ocular movement. #hasic processing determines movement by using local variable in the feedbac( loops loops. each parameter has an associated neuron that handles its processing tas(s in parallel. *asically.

these tiny application programs are loaded directly into the !"##'s internal . a few registers. 2pplication programs themselves can use CDD.25.. complex stereoscopic vision applications. the company said. The chip is priced at @AB&. The engineer needs no (nowledge of the internal wor(ings of the !"##.ince processing in each module on the !"## runs in parallel out of its own memory space. 'O!T(AR# A'%#CT' 6n the software side. When set in master-slave mode. multiple !"## chips can be cascaded to expand the number of ob'ects that can be recogni-ed and trac(ed. 2 card with a soc(eted !"## and 64 Cbytes of :lash . DI$ID# AND CON)U#R . *4" dubs the neural-learning capability of its development environment . The modules measure =& s/uare mm. *4"'s director of research and development.500. have %&& pins and can handle )&-5?video signals.6nce debugged. because of its ease of use. an adder and a comparator. said .programming by seeing and doing.. with an input amplifier with programmable gain auto scaling the signal.teve . and then testing the results to gauge the application's success. The 7 College of applied sciences. . the chip accommodates analogue-input line levels for video input. (ernel within the on-chip microprocessor. any number of !"## chips can divide and con/uer. a host operating system running on an external #C communicates with the !"##'s evaluation board via an 6. only application-specific domain (nowledge. .25 comes at @1.65.Generic visual perception processor associated with the first leading-edge$ and last trailing-edge$ pixel belonging to the ob'ect. which ma(es calls to a library of assembly language algorithms for visual perception and trac(ing of ob'ects. Mavelikkara . for instance.upplied as a %&&-pin module..#rogramming the !"## is as simple as setting a few registers.owe. . 4ach of these silicon neurons is built with .

which is about to mandate that cars sound an . saturation between )& and )< percent. . *ased on the neural networ( learning functions. R#COGNITION TA' +n applications. the chip allows application software to be developed /uic(ly through a combination of simple operational commands and immediate feedbac(. 2nother way is to EtrainEthe neural networ( by feeding it teaching patterns and letting it change its weights according to some learning rule. The ability of a neural networ( to learn from the data is perhaps its greatest ability. saturation. to loo( for a closed eyelid only within the rectangle bordered by the corners of the eye. luminance. direction of motion and spatial orientation. .alarm.imple applications such as detecting and trac(ing a single moving ob'ect can be programmed in less than a day. This learning of neural networ( can be described as its configuration such that the application of a set of inputs produces the desired set of outputs.ince some applications may also re/uire multiple levels of recognition. The !"## further subcategori-es pixels by ranges. when drowsy drivers begin to nod off F the !"## is first programmed to detect the driver's 8 College of applied sciences. Mavelikkara . speed. the !"## has software hoo(s to pass along the recognition tas( from level to level. each pixel may be described with respect to any of the six domains of information available to it F hue. "arious methods to set the strengths of the connections exist. :or instance. 4ven complex applications such as detecting a driver falling sleep can be programmed in a month. using a priori (nowledge. 6ne way is to set the synaptic connection explicitly. to detect when a driver is falling asleep F a capability that could find use in California. for instance luminance within %& percent and B< percent. hue of blue.Generic visual perception processor system's modular approach permits the developer to create a hierarchy of application building bloc(s that simplify problems with inheritable software characteristics. and moving upward in scene. 2 set of second-level pattern recognition commands permits the !"## to search for different ob'ects in different parts of the scene F for instance.

+n executing up to )& *+#. G$%% ARC"IT#CTUR# The chip houses 23 neural bloc(s. normally indicative of blin(ing. they are interpreted as the driver nodding off. The !"## multiplexes this neural hardware with off-chip scratchpad memory to simulate as many as %&&. the histogram maps Luminance. Technically. Mavelikkara . histogram to calculate the speed and direction of the motion. both temporal and spatial. 9 College of applied sciences. +t helps us analy-e. for which it creates histograms of head movement. which searches only within the eye area rectangles. 4ach of these synapses can be changed through the on-chip microprocessor for a combined processing total of over B.Generic visual perception processor head. ?igh-speed movement there. correct the contrast of the image. connections. is discounted. but when blin(s become slower than a predetermined level. which is defined from the way the human eye perceives the brightness of different colors.&&& synaptic connections per neuron. :or example. each consisting of )& hardware input and output . The microprocessor reads these histograms to identify the area for the eye. Then the recognition tas( passes to the next level. Luminance weighs the effect of this to indicate the actual perceived brightness of the image pixels due to the color components. The spatial-processing system analy-es the resulting . to analy-e successive frames of a video stream. ?istogram is a bar chart of the count of pixels of every tone of gray that occurs in the image. and trigger an alarm.difference.synaptic. and more importantly.) billion synaptic connectionsGsecond. our eyes are most sensitive to greenH we see green as being brighter than we see blue. the temporal neurons identify pixels that have changed over time and generate a >-bit value indicative of the magnitude of that change.

Generic visual perception processor :ig A. direction of motion and spatial orientation. speed. based on a model. and then the microprocessor directs the vision processor to search within the corner points of a rectangular area in which the nose of the driver would be expected to be located.% . and is able to perceive and trac( ob'ects based on combinations of hue. spatial processing and histogram processing. and to select pixels having characteristics of the shadows of nostrils. +n applications. luminance. the processor core detects movement of the eyelids. :irst. +n a case of detection of a driver falling sleep. the driver is identified. The results are calculated to the end of 10 College of applied sciences. Mavelikkara . flash memory and peripherals.epresentation of !"## MU&TI%&# %#RC#%TION' The vision perception processor chip processes motion images in real-time. the vision processor called generic visual perception processor !"##$ is used with a microprocessor. The chip has three functionsI temporal processing.

and the internal three functions play the following roles. the histogram processor perceives ob'ects in the images that have undergone significant change over time and the magnitude of the changes.patial processing involves the processing of pixels within a locali-ed area of the images to determine the speed and direction of movement of each pixel. Then.$. the microprocessor is able to determine when the driver is falling asleep and triggers an alarm accordingly. The highly parallel internal design of the vision processor allows it to perform over )& billion instructions per second *+#.Generic visual perception processor the image frame being analy-ed. and the microprocessor analy-es the resulting histograms to determine characteristics indicative of nostrils. The vision processor includes a total of )> spatio-temporal neural bloc(s. Mavelikkara . such as moving ob'ect detection. Temporal processing involves the processing of successive frames of an image to smooth the pixels of the image over time to prevent interference affecting ob'ect detection. the microprocessor directs the vision processor to isolate a rectangular area in which the eyes of the driver are expected to be located. and to select pixels having characteristics of high-speed movement normally indicative of blin(ing. 4ach connection can be controlled through the on-chip microprocessor. :rom the blin( duration and interval information. The vision processor provides a mechanism to reconfigure the spatio-temporal neural bloc(s to perform a wide variety of vision processing. . each having approximately )& different input and output connections. based on the model. such as the spacing and shaping and shape of the nostrils. The chip is able to detect and trac( eight ob'ects per frame. . meaning )=& ob'ects per second in a >& framesGs systems such as T" or high definition T"s ?1T"$.eceiving the information about speed and direction of each pixel from spatial and temporal processors. 11 College of applied sciences. The microprocessor analy-es over time the histograms ta(en of the eye area to determine the duration of each blin( and the interval between blin(s.

epresentation of histogram calculation unit 12 College of applied sciences.Generic visual perception processor :ig %&. Mavelikkara .% .

6. and can function in daylight or dar(ness. The engineer needs no (nowledge of the internal wor(ings of the !"##. multiple !"## chips can be cascaded to expand the number of ob'ects that can be recogni-ed and trac(ed. . 7.imple applications can be /uic(ly prototyped in a few days. compared to a mere millions handled by #entium-class processors of today. 5. Mavelikkara . and then trac( up to eight user-specified ob'ects in a video stream. !"## has been demonstrated as capable of learning-in-place to solve a variety of pattern recognition problems.+t is an inexpensive device that can autonomously . the company said. 2. The chip could be useful across a wide variety of industries where visual trac(ing is important 13 College of applied sciences.perceive.Generic visual perception processor AD$ANTAG#' AND DI'AD$ANTAG#' AD$ANTAG#' 1. with medium-si-e applications ta(ing a few wee(s and even big applications only a couple of months 8. the !"## can handle some )& billion instructions per second.ince processing in each module on the !"## runs in parallel out of its own memory space. orientation. +t has automatic normali-ation for varying ob'ect si-e. only application-specific domain (nowledge. and lighting conditions. . +mitating the human eye's neural networ(s and the brain. 4. 3. .

on the other hand. though modeled on the visual perception capabilities of the human brain. Today's vision systems dictate uniform. poised to cure the blind.normal. adapts to real-time changes in lighting without recalibration. shadow less illumination.Generic visual perception processor The !"##'s ma'or performance strength over current-day vision systems is its automatic adaptation to varying lighting conditions. its inventors are confident that the !"## will /uic(ly blossom into a multibillion dollar business. *ut with do-ens of potential applications in a myriad of industries. the chip is not really a medical marvel. and even next-generation prototype systems. invention and is not aimed at con/uering human blindness. DI'AD$ANTAG#' . lighting conditions. can be used only from dawn to dus(.adly. Mavelikkara . The chip is an industrial. 14 College of applied sciences. day or night. The !"##. designed to wor( under . rather than medical.

15 College of applied sciences. as well as in security access using facial. 2lso with transportation. automatic cruise control. signaling the driver when the vehicle deviates from the proscribed lane. Mavelikkara . ground reconnaissance and other military applications. /uality control and assemblyH automotive systems such as intelligent air bags that monitor passenger si-e and traffic congestion monitorsH pedestrian detection. !"## does so by monitoring the left and right lanes of the road.Generic visual perception processor A%%&ICATION' *4" lists possible applications for the !"## in process monitoring. . !"## have applications in robotics. !"## could be used in developing systems for collision avoidance. iris. smart air bag systems. particularly for dirty and dangerous 'obs such as feeding hot parts to forging presses. and spraying toxic coatings on aircraft parts. automatic inspectionH and medical uses including disease identification. fingerprint. cleaning up ha-ardous waste. measurement of traffic flow.otive industr?ere. The chip could also prove useful in unmanned air vehicles. 2nother automotive application is warning erratic drivers. !"##-type devices could (eep .watch. 6r act as a monitor for the car's progress to ma(e sure it does not get too close to the curb. par(ing management and the inspection of crac(s in rails and tunnels. license plate recognition. electronic toll collection. miniature smart weapons. license plate recognition.+ Ro/otics +n manufacturing. and trigger off an alarm if the driver were to nod off to sleep at the wheel. automatic par(ing management. or height and gait identification *+ Auto. automatic cargo trac(ing. electronic toll collection.

as well. bar code reading. cloud identification. and !"##-based systems could be inexpensively developed to detect intruders and fires.Generic visual perception processor 0+ Agriculture and fis1eries +n agriculture and fisheries engineering. 16 College of applied sciences. and many other fields. tra'ectory correction. ripeness detection and yield identification. ground reconnaissance and surveillance. automatic target detection. trademar( database indexing. expression understanding. +n addition to the above applications. such as military target ac/uisition and fire control. construction of virtual reality environment models. human motion analysis. blood analy-ers. harvest control. cardiac monitoring. 5ilitary applications include unmanned air vehicles. "ision systems are useful in home security. !"## should be able to wor( as medical scanners. Mavelikkara . !"## could help with tas(s that traditionally have been labor intensive such as disease and parasite identification. seal and signature verification. 2+ Militar.. ban( chec(s.applications The chip can be used in other important pattern-recognition applications.

visuali-ation +nternal 6.25.)K 5?.)>).i# %. 6riented edges. .lave 1ata . C language programmation #C+. lines.aturation. curves.).ate max$I =& 5egapixel per second 1ynamic ..C format -=&. >. Mavelikkara . . !"##.> Watt. #. 5otion orientation. 17 College of applied sciences. 0T.I C-56. ?ue.erial :lash 5emory.>"olts for #2LG0T. % 5bits .DJ< C Complete application in one . +)C.Cristal.angeI %&-bits.emi-!raphic interface visuali-ation !3+ with mouse #2L. #W5 &. +mager. velocity$. > channels #arametersI Luminance.x%.C. "!2.Generic visual perception processor PRELIMINARY INFORMATION OF GVPP-7B Generic Visual Perception Processor • • • • • • • • • • • • • • • • • • 2rray :ormat max$I J&&?xB&&" :rame rateI &-%&& "!2 frames per second progressive-scan +nterface 5odeI 5asterG. %&5bits ".T0 bloc(s 5ulti-scales possibilities optional$ 5ulti-chips connections capabilities . corners ComputationI B= .

The chip could be useful across a wide swathe of industries where visual trac(ing is important.Generic visual perception processor !UTUR# 'CO%#' :or future. which provides tremendous applications. the !"## is a single chip that can detect ob'ects in a motion video signal and then to locate and trac( them in real time far more dependably than competing systems. This is a generic chip. scientists are wor(ing on a 8visual mouse9 for handgesture interface to computers that ta(e advantage of that high compression ratio. Mavelikkara . according to company scientists. 5odeled on the visual perception capabilities of the human brain. and can manage most tas(s performed by the eye. :uture studies also involve using this processor as an eye of the robots. CONC&U'ION' +mitating the human eye's neural networ(s and the brain. the generic visual perception processor can handle about )& billion instructions per second. which cost far more. 18 College of applied sciences. and we've already identified more than %&& applications in ten or more industries.

pixelin(.eduGcl.comGsupport L<MhttpIGGgvpp. Mavelikkara .ppt.Generic visual perception processor R#!#RCNC#' L1M httpIGGwww.. 2ddison Wesley L=MhttpIGGwww.res.patentstorm. 0eural networ(s and fu--y L3M *art Cos(o.htm 19 College of applied L2M httpIGGwww.techweb.