| SCIENTIFIC an Article from | AMERICAN JUNE, 1986 VOL. 254 NO. 6The Perception of Apparent Motion When the motion of an intermittently seen object is ambiguous, the visual system resolves confusion by applying some tricks that reflect a built-in knowledge of properties of the physical world by Vilayanur S. Ramachandran and Stuart M. Anstis Vision programs and even neon signs have long banked on the fact that human beings have a quirk in their visual system. When it Is con- fronted with @ rapid series of stil im- ages, the mind can “fill in” the gaps between “frames” and imagine that it sees an object in continuous motion. For instance, a series of neon arrows lighted up in succession are perceived asbeing single arrow moving through space. The illusion of continuous mo- tion is called apparent motion to dis- tinguish it from “real” motion, which is perceived when an object’ moves continuously across a viewer's visual field, When Sir Laurence Olivier ap- pears to be fencing in a film, he is in apparent motion, whereas a person walking across the theater in front of the screen isin real motion. In the century or 50 since the mo- tion picture was invented, filmmakers and television workers have learned to create many compelling illusions of motion, but their progress has been furthered mainly by rule-of-thumb empiricism. Psychological research is only now beginning to describe the mechanisms by which the visual sys- tem—the retina and the brain—per ceives apparent motion. Pes of motion pictures, tele~ "Ts, zarting point of our own in- vestigations was the premise, set forth by Bela Julesz of the AT&T Bell Laboratories and Oliver J. Braddick of the University of Cambridge, that to perceive an intermittently visible object as being in continuous motion the visual system must above all detect what is called correspondence. That is, it must determine which parts of suc: cessive images reflect a single object in motion. If each picture differs only slightly from the one before it, the visual system can perceive motion; if successive pictures differ greatly, the illusion of motion will be destroyed 102 Our main question, then, was: How does the visual system go about detect- ing correspondence? One popular view holds that the brain does so by acting like a computer. When an image stim- tulates the retina, the eye transmits the image (o the brain as an array of tiny points of varying brightness. The brain then compares each point to every point in succeeding frames. By means of complex computations the brain fi- nally discerns the one set of matched points composing a single object that has changed its position—has moved Attempts to build machines that “sec” are generally based on this principle. ‘The scheme seems logical enough when a simple, unambiguous display is presented. For instance, if a small dot is shown in one frame and is followed by an identical dot placed slightly to the right, the visual system will readily identify the dot in the first frame as an object and find it again—displaced—in the second frame [see top illustration on age 104} The scheme becomes problematical, however, when correspondence is to be detected in more intricate displays For example, suppose two identical dots are shown in vertical alignment on a computer or television screen and are then replaced by congruent dots shifted to the right. In theory the visual system is now confronted with wo possible correspondences: the dots, in the first frame could be seen to jump horizontally along parallel paths to the right, or they could be seen to jump diagonally, in which case they would have to cross paths. In practice viewers always see the dots moving in parallel, never crossing, In another display a computer-gen- crated random-dot pattern forms the first image; then a square region is cut out of the middle and shifted horizon- tally to create the second image [see bottom illustration on page 104]. To the unaided eye the second image appears tobe identical with the firstand to have ‘no separate central square. Now the images are superposed and then alter- nated rapidly so that the outer dots are in perfect register, or correlate, and so appear to be immobile. The middle re- gion, where the dots are out of regis- ter, appears to move: a well-delineated square is perceived to be oscillating from side {0 side. To produce these two illusions by ‘means of point-to-point matchings the brain would somehow need to inva: ‘date hundreds of potential matches, deeming them to be false. While it is possible that the brain laboriously matches all the points and then sub jects the matches to a series of elimina tion tests, our investigation suggests an entirely different approach to detect- ing correspondence: the visual system applies strategies that limit the number of matches the brain needs to consider and thereby avoids the need for com- plex point-to-point comparisons. WW believe perception of apparent ‘motion is controlled in the early stage of visual processing by what isin effect a bag of tricks, one the human visual system has acquired through natural selection during millions of years of evolution. Natural selection is inherently opportunistic, It is likely that the visual system adopted the pro- posed visual short cuts not for their mathematical elegance or aesthetic ap: peal, as some would suggest, but sim- ply because they worked. (We call this idea the utilitarian theory of percep- tion.) In the real world anything that moves is a potential predator or prey. Hence being able to quickly detect mo: tion and determine what moved, and in what way, is crucial to survival. For example, the ability to see apparent motion between widely separated im- ages may be particularly important when detecting the motion of animals that are seen intermittently, as whenthey move behind a screen of foliage or a tree trunk. ‘One trick of the visual system is to extract salient features, such as clus ters of dots rather than individual dots), from a complex display and then search for just those features in succes: sive images. This significantly reduces the number of potential matches and thus speeds the perceptual process; af- ter all, the probability that two chunks of a visual scene will be similar is much smaller than the probability that two points of brightness wil be similar, Among the features the visual sys tem might attempt to extract from im- fages are sharp outlines and edges or blotches of brightness and darkness; the latter are technically called areas fof low spatial frequency. We have evaluated each of these and found that ‘motion when still images such as these are flashed, the vist tem must above elect corresponds it must object in motion. 103tection of correspondence by a viewer, Clearly the mechanism for percei ing apparent motion can accept var ious inputs for detecting correspon- dence. We have found a preference for seeing low spatial frequencies and tex- tures; other investigators, such as Shi: mon Ullman of the Massachusetts In- stitute of Technology, have found that under certain circumstances line ter- minations and sharp edges also serve as cues. Perhaps the visual system per- ceives motion cues hierarchically, first scanning for coarse features before homing in on finer features, rather like fan anatomist who first looks through ‘a microscope set at low power before switching to higher magnification. One bit of evidence supporting this view is that subjects do indeed sometimes see the white square in the experiment ited above move toward the outline square, but only when the images are presented slowly and there is time to scrutinize the image 1 addition to extracting salient fea. tures a second trick of the visual system is to limit the matches it will consider to those yielding perceptions of motion that are sensible, or could ‘occur in the real, three-dimensional ‘world. In other words, as David Marr of M.LT. first suggested, the visual sys tem assumes the physical world is not a chaotic and amorphous mess, and it capitalizes on the world’s predictable physical properties. For instance, if the pairs of jumping dots described above were actually rocks, they would col lide if they moved diagonally in the same depth plane and so would fail to reach opposite corners; the only log- ical perception of the dots’ motion is therefore that the two dots in the first frame move in parallel to their positions in the second frame. Sure enough, when these dots are viewed through a stereoscope (a double-lens viewer) and seem to be in separate planes, observers do see them cross; in the real world, objects in different planes—such as airplanes at different altitudes—can indeed cross each other without colliding. In order to examine the notion that the visual system assumes the world has order, we presented subjects with various motion displays that could be interpreted in more than one way and observed how subjects resolved the ambiguity. We found that one rule ap. plied by the visual system is reminis. eat of Isaac Newton's first law of mo- tion, namely that objects in motion tend to continue their motion along a straight path, The visual system per ceives linear motion in preference to FEATURES OF OBJECTS that might be extracted to detect correspondence are com pared in this experiment. A solld square (cen) s Shown against a dark background and is ‘then replaced with an outline square on the left and a solid circle on the right. The viewer ‘who is confronted with these images usually ees the square move toward the circle rather ‘than toward the outlined square, suggesting that ri ightness (low s tial frequencies) are mote likely to be detected initially than sharp edges oF fine outlines, aioe TEXTURED DISPLAYS shown here are generated by computer. When they are super= posed and alters serve as a cue for detecting r dilfer from the outer regions in texture, or inating the possibilty of detecting correspor the dots in the right-hand nating the possibility of detecting correspondence by point-to-point matching. Therefore the fact that viewers see an inner square osellate horizontally when the images are alternat- cd can only be explained by the abiliy of the visual system to detect changes in texture perceiving abrupt changes of direction. We demonstrated the power of this rule with an illusion that incorporated a “bistable (dual state) quartet”: two dots brielly presented at diagonal cor- ners of a square and then replaced by identical dots at the other two cor- ners. A bistable quartet can be per- ceived in two ways, somewhat like the familiar Necker cube, which viewers see oscillating between two perspec- tives. With approximately equal fre- quency observers of a bistable quartet see two dots oscillating horizontally or two dots oscillating vertically Tit bistable quartet was embedded in the center part of two horizon- tal rows of dots that appeared to be streaming in opposite directions [see bottom illustration on next pagel. Only fone dot in cach row was visible at a time. When the streaming dots reached the center of the screen, the bistable quartet became visible. At that point viewers could in theory see the dots continue in a horizontal path or could see them make a 90-degree turn fol- Towed by a second 90-degree turn, to produce two U-shaped trajectories. In practice observers invariably saw hor izontal streaming, indicating that the tendency to see linear motion over came the ability to see the dots in the quartet move vertically. The U-shaped motion was seen only when the paral Tel rows were brought very close to each other; then Newton's law came in conflict with a competing tendency to see motion between the closest iden- tical points. The proximity principle gains increasing power as objects are moved closer to each other. A second rule that limits the pos sibilities for correspondence is that objects are assumed to be rigid; that is, 105the same feature in second fumes rom side ose. Siig, Epo one anal Wits! move mipelrcey wate eens {inching cach kort. ag wld en We ‘eed all parts of te leopurd-more uous ston ach tat sen the [Ropar soc ean Sy of i all dees” he visa yer Slee ae een cone ‘Misr pare agen {ecbae hove srace ears move Ax arinat tat denonscues Reve hele ergy voed feo "A fate eperinen al don nce Pando dot paters ve stated te pana of cape lcrmiecinacontmnouecescupae moter, anapericuy he iy of ‘Gpenck picture for hl’ cond ce fow pul eqn eect Sch NECKERCUME awed ore nnn Pitino pe a) View” Cte We sperm bled er DARENCP cet eset ste sw random income! ftom, al bre of fw coat ins ad ‘ickdeeawomcnnepepaa habit “aon” onan tad ck Sn al se raves ot route hee oand abuting Fandom de pers oe Yona sua one moving te ae ax race a nag Tse of eas co ‘leepurdiapineromatranchof one the pain nthe mugs towhich thy ysconehepcen The mangas. irctontrany ina Aicrdee ore ed rte inade tothe ced ihe dein be Seay Pick ou ay tae fue othe bded tole at sesse at te ft ns thie sholowe Kepar ach ax aban ape (or tusin of te ew npr dd ot" TREpheoancton of captured 6 cient sathe lightning low sigh a the mages vee again onook sabes wep how ts Spataltcgoen.ofssenah sheds alerted, lt main speed that en ilaingsioecon sete _ i‘uate here ebro ‘ition a pane ft) Proponent of te {Shor Gu would esa iner de ‘bjt wil roaresvely cover and alsands square below ae present. Sd dpm replace by Sno dy otra fe ey tua ‘oupur One man espace Sabie omar te onde, ‘ete cidern twinge tomar Gd fame On he other hand wins rota Once ete ual ‘em ended 1 peteie the meton Is on expe Ione sera ple SE SycaaTe h/tel," c fein em ch STs Glunadunicn) swwed ver: tamooed Subject cul! tr ey SEMIS ye ce ye {ORHED LOW-CONTRAST HARS copa ale rave patie ‘Seseeperpantona nanan aq ares ncaa nea ‘Sit mere nee te sen vinet ttt cpu ua a tee ‘Scie ether bak Cap se Uy Sosa 10‘Shed nani prt el, One rene seen er wp ate Sev Nc ing ee at Sable quartet, we sithed of the di iy isu bl snd tlre SS BE hy ere te nding tendency to perce sar Inve inergteted the fou sissy tt Sue eet moving eh Sate Tesely ett Yed {Ror tarer one of ar (kamnehsn Sr redo ‘tna Se shed tone et be lover ilmaton at re Neate ‘ued coo he dv th send Snot ex To sn aparetscise, Soyer ne comet ‘ely Sin te sar they {he sna held. (he pretence of the ‘Sold eae he un, Onsieesenan nes sem rap extract act eas Srpomtsorant mathioe an 9 Slower mch debe oeion ‘Ete he dete tn eplemo/maten eure somes i , os ‘chao ket one Tet tatiana Sot rca ns oi lp {jpenta ore bcd te yh uch tee eine he moon me el Images at seed to api oa Renta dg ti, Ahecarly ager of nina proces Steerer avr hs ‘age anlsion Newrbioosel ev ‘ce has braced hepa Sea Brome anda ‘Motte of Tecnnoony ad by Semit ‘Zao Uinhssty Colege Londo, ‘hey hae aol monte ak (Besant ter sharp faarer Tie e ‘of obi sring on sat rege ster than pao obs ‘Sarin sharp feat ant soaps ‘etree ey may i Sef atrayoce he ere ‘none fry meh artery, at ‘Rin th pewrgel proch {ito e pce wh Situcian of motion ctecing von rcs ig nae vein 108