co» United States Patent Browy et al. USO10S80213B2 (10) Patent No.: (45) Date of Patent: US 10,580,213 B2 Mar. 3, 2020 (54) SYSTEMS AND METHODS FOR SIGN LANGUAGE RECOGNITION (71) Applicant: Magie Leap, Ine. Plantation, FL. (US) (72) Inventors: Erie Browy, Coral Springs, FL (US Michael Janusz Woods, Mouatsin View, CA (US): Andrew Rabinovich, San Francisco, CA (US) (73) Assignee: Magie Leap, Ine. Plantation, FL (US) (4). Notice: Suibject to any disclaimer, the tem of this patent is extended or adjusted under 33 USC. 184(b) by 0 days, (21) Appl Nos 187702,812 (22) Filed: Sep. 12,2017 65) Prior Publication Data US 201810075659 AL Mar. 15, 2018, Related US. Application Data (60) Provisional application No, 62/394,013, filed on Sep 13, 2016, provisional application No. 62/440,320, filed on Des. 29, 2016 (51) In ch Gust 19700 @or1o1) Gus 1728 (200501) (Continned) (2) [Link] cre G6T 19/006 (2013.01); GO2B 27017 (2013.01); G02B 2770172 (2013.01); GO6F 1/163 (2013 01), GO6E 1/1686 (2013.01) GOOF 1/1688 (2013.01), GO6F 3011 (2013.01); GU6E 8/013 (201301): GO6E OTF (2013.01); GO6F 30304 (2013.01), {GOGF 177289 (2013.01); GO6K 9/0355 (2013.01); Ga6K 9700671 (2013.01); G6K 9/78 (2013.01), GO2B 27/0179 (2013.01), {G02B 2027/014 (2013.01) (Continued) (68) Field of Classification Search None See application file for complete search history. 66) References Cited US. PATENT DOCUMENTS 850221 BL sua0390 BD 105 Tickle AQ0L2 Heomandez-Rebolar (Continued) FOREIGN PATENT DOCUMENTS WO WORISOS01 F201 (OTHER PUBLICATIONS, Intemational Search Repor and Write Opinion fr PCT Applica tion No, PCT/US20170S1178, dated Nov. 16,2017 (Continued) Primary Examiner (74) Auorney, Agent or Firm & Bear, LLP Jefey J Chow Kaobbe, Martens, son on ABSTRACT A sensory eyewear system for a mixed reality device ean facilitate user interactions with the other people or with the environment. As one example, the sensory eyewear system can recognize and interpre a sign language and present the teanslated information toa user ofthe mixed reality device The wearable system can also recognize text in the User's cavironment, modify the text (e- by changing the content tor display characteristics of the text), and render the modi fied text to osclude the orginal text. 20 Claims, 29 Drawing SheetsUS 10,580,213 B2 Page? oy mck eee Ap eure useeslai (ult cei 97a om) eae at aac eee ae oat Sor’ Gaonor) eens eet Geacsoo Goon) cama ietlceta feat ine —Goocot muy Meat Gea iicccee tines lttie aa ar sas Goon Bisset nat eee ¥ (2o06.01) dolsuoierst AL ‘La0ld. Mtoe sal GO6T 770 (2017.01), DOLSOL03306 Al 42015 Kaji et al. Usce muscrns am a 2027/0141 (2013.01), Go2B 2027/0178 (2013.01), G02B 2027/0187 2013.01}, GO6E 3/16 (2013.01), Ga6K 2209/01 (2013.01); (GO6T 7/70 (2017.01); GO6T 2207/20084 (201301) 66) References Cited US. PATENT DOCUMENTS 849274 BY 72013 Agra Sososer B2 22018 Maconmaca Sooners 8272015 Andrade et S.is208 B2 123018 Miler 531089 B2 ‘42016 Maceamaa S348 A? S016 Gao e a Disso7 §°— G.a016 Nate 917s? 282016. Schowengert et 54470906 182102016 Kae a Sian B2 ‘12017 Gao wa Droiss s ews tal 9.720.008 2 Gao 971:300 2 Schowengeit ea. DeDg7M S 122017 Fisker et SsLse8 AZ 122017 Gaove a SSS7SOL BD 12018 Welch otal 95H 309 2ouarsisas 2nuvs01938 aor 3017 oat 20120035007 AL 22012 Lobsaa ta aoiaoiasers Burdeo ta doizo1za62 2 BanZew eta 20120213820 bo1sioos2922 donvnsos? soioxs Raghoctardayal Sorssomwee Glnborg ea soisonnees Al soisonnzwss Al dotsonssals Al soisoxo2ss? Al aoisoxiso9s AL aoLsonleoss Al aoisoues™ Al ‘0346490 Al doisonaeass Al doreootta9 Al soeone2ss Al soleousides AL 0270686 Al £2015. Walch 2015 Cheng 52015. Schowenged et. 102018 Miler ea 102015 Oserhout T2018. Sopaya el T1015 Publier et a 122018. Tekolste a 12018 Welch ea 12016. Gan 13016 Bask ta. 22016 Iyer eta 92018 Ste ea Sovponseaso Al® $2017 hung Go6r 3018 F043. 20170270103 A1* 92017 Golan Gato 15105 (OTHER PUBLICATIONS ogi. S. etal, “Sign Language Recognition using Neural Ne works, TEM Joural, vol. 34), Now. 2014, in 6 pages. URL [Link], awn, So a, "Neural Nework based Hand Gesture Recog- nition, Iatermtiona Jounal of Emerging Reseach in Management tnd Technology. Mar 2013, in $ poses. Hang. Jt al, “Sign Language Recognition Using 3D Convol ional Neural Networks": University of Selene and Technology of (China, fun. 2015, in 6 pape. Tango, C., "Real Time Sign Langage Recognition Using Artic Seah Neural Networks, Sula Unv- Babes-Bola, informatica, OL VEC, Sop. 2011, in 10 pages “Mekal P< al, “Real-time Sign Language Recognition fused on ‘Neural Network Architecture" IEEE, Mar 201, im 5 pages, Intrtinal Preliminary Rept on Patctbity fr PCT Applica tion No, PCTUS2O170S117% dated Mar. 19, 2019, * cited by examinerU.S. Patent Mar. 3, 2020 Sheet 1 of 29 US 10,580,213 B2 FIG. 1U.S. Patent Mar. 3, 2020 Sheet 2 of 29 US 10,580,213 B2 REPOSITO! = if PROCESSING i 1 DATA it J FIG. 2AUS 10,580,213 B2 Sheet 3 of 29 Mar. 3, 2020 U.S. PatentU.S. Patent —Mar.3,2020 Sheet 4 of 29 US 10,580,213 B2 FIG. 3US 10,580,213 B2 Sheet 5 of 29 Mar. 3, 2020 U.S. Patent war coe D 9h D 859 oer A ep ot oy com orUS 10,580,213 B2 Sheet 6 of 29 Mar. 3, 2020 U.S. Patent $320, 410 < FIG. 5U.S. Patent —Mar.3,2020 Sheet 7 of 29 US 10,580,213 B2 DRIVE ELECTRONICS Y T UGHT/ IMAGE SOURCE eto FIG. 6 6220 648 gygUS 10,580,213 B2 Sheet 8 of 29 Mar. 3, 2020 U.S. Patent 90L OL zou ANd! AHOSNES Anant AMYNOLLYLS Anan wnsiA Z°Sld asvaviva dv uBZINDOO3Y 419380 UAZINOOORY 193780 901 802 ‘@INAISAS BaSN zk WINBISAS ¥3SN voz 7 NwU.S. Patent Mar. 3, 2020 Sheet 9 of 29 US 10,580,213 B2 a 810 USE CAMERAS AND VARIOUS INPUT ELEMENTS TO COLLECT IMAGES, POINTS, AND DATA 820 DETERMINE SPARSE POINTS BASED ON INPUT ELEMENTS: 830 CRAWL TRHOUGH POINTS TO RECOGNIZE ONE OR MORE. OBJECTS USING A MAP DATABASE 40 ‘TRANSMIT THE MAPPING INFORMATION AND RECOGNIZED ‘OBJECTS TO THE LOCAL WEARABLE SYSTEM 850 DISPLAY A DESIRED VIRTUAL SCENE TO THE USER IN RELATION TO RECOGNIZED OBJECTS FIG. 8U.S. Patent Mar. 3, 2020 Sheet 10 of 29 US 10,580,213 B2 900 ~ TOTEM EYE SPEECH TRACKING TRACKING TRACKING GAME GESTURE / POSE EYE CAMERAS (X2) rene TRACKING MIXED REALITY 960 HAND GESTURES / TOTEM 950, (STEREO CAMERA) DEPTH MAP/ IMAGES a FUSION IRACTIVE 20 PROJECTORS (WIDE FOV CAMERAS) IMAGES (02) Mu FIG. 9U.S. Patent Mar. 3, 2020 Sheet 11 of 29 US 10,580,213 B2 100 DETECT AND CAPTURE USER'S INTERACTION WITH A. TOTEM 1020 DETECT A POSITION, ORIENTATION, ANDIOR MOVEMENT OF THE TOTEM WITH RESPECT TO REFERENCE FRAME 4030 MAP USER'S INTERACTION WITH POSITION, ORIENTATION, AND/OR MOVEMENT OF TOTEM 4040 DETERMINE USER INPUT BASED ON THE MAPPING FIG. 10U.S. Patent Mar. 3, 2020 Sheet 12 of 29 US 10,580,213 B2 1100 ™~ 4110 IDENTIFY A VIRTUAL Ul 1120 GENERATE DATA FOR THE VIRTUAL Ul 1130 ‘SEND THE DATA TO A USER DEVICE 1140 DISPLAY THE VIRTUAL UI BASED ON THE SENT DATA, 1150 WAIT FOR A COMMAND FROM A USER, RECOGNIZE GESTURE OR OTHER COMMAND? YES 1170 GENERATE VIRTUAL CONTENTIPERFORM ACTION BASED (ON THE COMMAND FIG. 11US 10,580,213 B2 Sheet 13 of 29 Mar. 3, 2020 U.S. Patent soezt ZL Old TROL N¥3ZIND093¥ 193780. Tet 30¥aINI RIONUN beds zngow Ayouisodsy Geeaigad oniss300ud aLOWaY eT WALSAS ONLLNANOO 3L0NaY cod 8 ¥3ZINDOORY 193780 wer vl THON ovatU.S. Patent —Mar.3,2020 Sheet 14 of 29 US 10,580,213 B2 1300- f YOU~ HOW ARE YOU? 1388 1348 1350 FIG. 13AU.S. Patent Mar. 3, 2020 Sheet 15 of 29 US 10,580,213 B2 1982 1360 ™ WHERE Is THE PTO? 1365 PTO: PATENT AND TRADEMARK OFFICE FIG. 13BU.S. Patent Mar. 3, 2020 Sheet 16 of 29 US 10,580,213 B2 13700 FIG. 13CUSS. Patent Mar. 3, 2020 Sheet 17 of 29 ‘US 10,580,213 B2 FIG. 13DU.S. Patent Mar. 3, 2020 Sheet 18 of 29 US 10,580,213 B2 1400. ™ ‘CAPTURE IMAGE INFORMATION 1404 IN AN ENVIRONMENT DETECT PRESENCE OF SIGN LANGUAGE IN IMAGE INFORMATION 141, ‘SIGN LANGUAGE DETECTED? 0— Yes CONTINUE») 1428 PROCESSING, 140e~ DETERMINE A SOURCE OF THE SIGN 14 LANGUAGE NO 1420. ee YES s4—| GOTO 1448 FIG. 14AU.S. Patent Mar. 3, 2020 Sheet 19 of 29 US 10,580,213 B2 1440 ~*~ MONITOR REQUEST FOR AUXILIARY INFORMATION 464 Fever necEnen RETRIEVE AUXILIARY INFORMATION DISPLAY AUXILIARY INFORMATION DETECT A CONDITION FOR DISMISSING ‘CONVERTED TEXT ANDIOR AUXILIARY INFORMATION INFORMATION FIG. 14BU.S. Patent Mar. 3, 2020 Sheet 20 of 29 US 10,580,213 B2 1490— Ny ‘1482 PROCESSING YES ¥. | RECEIVE TEXT DISPLAY TEXT 1460 MONITOR REQUEST FOR AUXILIARY INFORMATION DISPLAY AUXILIARY INFORMATION CONVERTED TEXT ANDIOR AUXILIARY INFORMATION DISMISS CONVERTED TEXT ANDIOR AUXILIARY INFORMATION FIG. 14CU.S. Patent Mar. 3, 2020 Sheet 21 of 29 US 10,580,213 B2 0 4510\) IDENTIFY AND RECOGNIZE SPEECH IN AN | ENVIRONMENT 1620, IDENTIFY A TARGET LANGUAGE 18 CONVERT THE SPEECH INTO THE TARGET LANGUAGE ia DETERMINE AUDIO-VISUAL CONTENT #0 ASSOCIATED WITH THE CONVERTED SPEECH FOR PRESENTATION TO A USER TO AHEAD-MOUNTED DISPLAY FOR PRESENTATION FIG. 15U.S. Patent Mar. 3, 2020 Sheet 22 of 29 US 10,580,213 B2 1602 1608 Ks) 1610 1614 1618 FIG. 16A 1622 210U.S. Patent Mar. 3, 2020 Sheet 23 of 29 US 10,580,213 B2 1602 1606 KE) 1610 1614 1618 1634 FIG. 16B 210U.S. Patent Mar. 3, 2020 Sheet 24 of 29 US 10,580,213 B2 1602 &) 1614 1618 FIG. 16C 210U.S. Patent Mar. 3, 2020 Sheet 25 of 29 US 10,580,213 B2 1602 1628 1614 1630 1618 FIG. 16D 210U.S. Patent Mar. 3, 2020 Sheet 26 of 29 US 10,580,213 B2 1602 1614 1618 FIG. 16E 210U.S. Patent Mar. 3, 2020 Sheet 27 of 29 US 10,580,213 B2 1700. ™ +704 JRECEIVE OPTICAL SIGNAL THROUGH ONE OR| MORE CAMERAS 4708 IDENTIFY AN IMAGE FROM THE SIGNAL ari IMAGE COMPRISES LETTER: (OR CHARACTERS? A716 ‘CONVERT THE LETTERS OR CHARACTERS IN ‘THE IMAGE INTO TEXT 1720 [INSTRUCT A DISPLAY TO RENDER TEXT TO APPEAR AT A FIRST DEPTH OF A PLURLAITY OF DEPTHS FIG. 17US 10,580,213 B2 Sheet 28 of 29 Mar. 3, 2020 U.S. Patent oat 2001 8bL Old UBLNIO ‘ONIddOHS uvLSa109 a 018 Gey uvLsd10o oy ntU.S. Patent Mar. 3, 2020 Sheet 29 of 29 US 10,580,213 B2 1900 ™ 4910 RECEIVE IMAGES OF A USER'S ENVIRONMENT 1920 |ANALYZE THE IMAGES TO IDENTIFY SIGNAGE IN THE USER'S ENVIRONMENT RECOGNIZE TEXT ON THE SIGNAGE | CONVERT AT LEAST A PORTION OF THE TEXT| TO A TARGET LANGUAGE 1950 Hi poe - 1 MODIFY DISPLAY CHARACTERISTICS 1 ‘ASSOCIATED WITH THE TEXT 1960 \ CAUSE THE TEXT TO BE RENDERED IN THE | TARGET LANGUAGE BY A MIXED REALITY i DEVICE FIG. 19US 10,580,213 B2 1 SYSTEMS AND METHODS FOR SIGN LANGUAGE RECOGNITION (CROSS-REFERENCE TO RELATED APPLICATIONS, ‘This application claims the benefit of priority under 33 USCC. § 119(0) 0 US, Provisional Application No, 62394, 013, filed on Sep. 13, 2016, entitled “SENSORY EYE” WEAR." and U.S. Provisional Application No. 62/440,320 fled on Dee. 29, 2016, ented “SYSTEMS AND METH- ‘ODS FOR AUGMENTED REALITY." the disclosures of ‘which are hereby incorporated by reference herein in their FIELD ‘The present disclosure relates to virtual reality and aug- mented reality imaging and visualization systems and more particularly 10 recognizing sign language or text in an 2 ‘environment and tendering virtual content based on the recognized sign language o¢ text. BACKGROUND, Modern computing and display technologies have fac tated the development of systems for so called “viral reality”, “augmented reality”, or "mixed reality” experi- ‘ences, wherein digitally reproduced images or portions thereof are presented to a user in @ manner wherein they scom to bo, or may be peresived as, real. virtual reality, oF “VR™, scenario typically involves presentation of digital or virtual image information without transparency 40 other actual real-world visual input; an augmented reality, or “AR”, scenario typically involves presentation of digital or Virtua image information as an sugmentation to visualize tion ofthe aetual world around the user, @ mixed reality, oF “MR, related to merging real and viral worlds to produce new environments where physical and viral objects co- ‘exist and interact in real Gime, As it tras out, the human Visual perception system is very complex, and producing a VR, AR, of MR technology that facilitates a comfortable, nanial-eeling rich presentation of viral image elements amongst other vital or real-work! imagery elements is challenging. Systems and methods disclosed herein address ‘Various challenges related to VR, AR and MR technology. SUMMARY \ariows embodiments of a mixed reality system for rec- ‘ognizing sign language and text in an environment are disclosed. These embodiments advantageously may pemait ‘grater interaction among differenty-abled persons. ‘A sensory eyewear system for a mixed reality device can facilitate users interactions with the other people or With the ‘environment. As one example, the sensory eyewear system ‘can recognize and interpreta sign language, and present the translated information to a user of the mixed reality device The wearable system can also recognize text in the user's ‘environment, modify the text (e., by changing the content ‘or display characteristics of the text), and render the modi fied text to oechude the orginal tex. Details of one or more implementations of the subject matter deseribed in this specification are set forth in the ‘accompanying drawings and the description below. Other features, aspects, and advantages wll hecome appareat from the description, the drawings, and the claims. Neither this 2 ‘summary nor the following detiled description purports 10 efine or limit the scope of the inventive subject matter. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 depits an ilusteation ofa mixed reality scenario with certain virtual realty objets, and certain physical objects viewed by a person FIG, 2A schematically ilstrates an example of a wear able system which ean implement a sensory eyewear sy ‘em. FIG. 28 schematically illustrates various example com: ponents ofa wearable system, FIG. 3 schematically illustrates aspects of an approach for simulating three-dimensional imagery using multiple depth planes. FIG. 4 schematically ilustrates an example of a wave- juide stack for outputting image information to user IG. 8 shows example exit hams that may be outputted ya waveguide, FIG. 6s a schematie diagram showing an optical system including 8 wavegtide apparatus, an optical coupler subsys- tem to optically couple light to or fom the waveguide apparatus, and eontral subsystem, used in the generation of ‘8 muli-focal volumetric display, image, or light field. FIG. 7 is @ block diagram of an example of a wearable system. FIG, 8 is a process How diagram of an example of a method of rendering virtual content in relation to rocognized objects. FIG. 9 is o block divgram of another example of a wearable system that includes a sensory eyewear system FIG. 10 is a process flow diagram of an example of a ‘method for determining user input to a wearable system. FIG. 11 is @ process How diagram of an example of @ method for interacting with a virtual user interface FIG. 12 schematically illustrates an overall system view depicting multiple wearable systems interacting with each other FIG. 134 shows an example user experience ofa sensory eyewear system where the sensory eyewear system can interpret a sign language (eg. gestured by a signer) FIG, 13B shows another example user experieace of a sensory eyewear system, where target speech and auxiliary {information are both presented, FIG. 13C shows an example user experience ofa sensory eyewear system in a telepresence session. TIG. 12D illustrates an example virial user interface for inerpeting a sign language. TIGS, IMA and 14D sllsiate example processes for facilitating interpersonal communications with «sensory eyewear system, FIG. 14C isa process lo diagram of an example method for determining siliary information and presenting the ‘auxiliary information associated with converted tex TIG. 18 illstrates another example process for fo ing interpersonal communications with a sensory eyewear system. FIGS. 16A-168 illustrate example user experiences for a sensory eyewear system which is configured to recognize ‘ext in the environment, moxiy the display charatersties oF the text, and render the modified text. FIG. 17 illustrates an example process of a sensory eyewear for fucilitaing a user's interactions with the envi