Scribd Logo
Upload

Welcome to Scribd!

  • US11876679

    Uploaded by

    Sanjay Sharma
    23 views84 pages
    New US Patent 2024 on

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    New US Patent 2024 on

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    23 views84 pages

    US11876679

    Uploaded by

    Sanjay Sharma
    New US Patent 2024 on

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    of 84
    2) United States Patent 6a amy m 03) en @y ws) @) 6) «2 Koponen et al. METHOD AND APPARATUS FOR INTERACTING WITH A NETWORK INFORMATION BASE IN A DISTRIBUTED. NETWORK CONTROL SYSTEM WITH. MULTIPLE CONTROLLER INSTANCES Applicant: Nicra, Ine. Pulo Alto, CA (US) Inventors: Teemu Koponen, San Fraciseo, CA (US); Martin Casado, Portola Valley CA (US); Jeremy Stribling, San Francisco, CA (US); Natasha Gude Palo Alto, CA (US) Assignee: Nicira, Ine, Palo Alto, CA (US) Notice: Subject to any disclaimer, the tem ofthis pateat is extended of adjusted under 35 USC. 1540) by 33 days, Appl. Nos 17/848:529 Filed: Dees 11, 2021 Prior Publication Data 20220173968 AL Jun, 2, 2022 Related U.S. Application Data Continuation of aplication No. 15/181,685, filed on Jun, 14, 2016, now Pat, No. 11,223,531, whieh is a (Continued) Int. ch, HOWL 41/0893 (2022.01) Hos 49/154 (202201) (Continued) v, CPC. MOdL 41/0893 (2013.01); GOOF 18/17312 (2013.01); HOMd. 12/4633 (2013.01: (Continued) USOII876679B2 & (0) Patent No: US 11,876,679 B2 (45) Date of Patent: Jan, 16, 2024 (58) Field of Classification Search None ‘ce application file for complete search history 66) References Cited US. PATENT DOCUMENTS. 470413 A 611988 Bagge 4961/1398 101990 Hong ta. (Coutinved) FOREIGN PATENT DOCUMENTS =P 13423 AL 82004 (Continved) (OTHER PUBLICATIONS ‘Shimowishi, Hideyuki, otal, “Visualized Network Infiastuctue ‘sing Open low,” 2010 IEEETEIP Network Operations and Man sgement Symposium Workshops, Ape 19,2010, 6 pages, IEEE (Coutinsed) Primary Examiner — Hong Sha (74) Attorney, Agent, or Firm — Quarles & Brady LLP 6 ABSTRACT A control system including several controllers for managing several switching elemens. A fist controller registers 9 second controller for receiving a notification when © data tuple changes in a network infomation base (NIB) storage of the first controller that stores data for managing a set of Switching elements. The fist controller changes the data tuple i the NIB. The Hirst controller sends the notification to the second controller ofthe change tothe data tuple in the NIB, The first and second eontrollers operate on to dll cent computing. devices, Each controller receives logical control plane data for specifying logial datapath sets and fonverts the logical contol plane data to physical control plane data for enabling the switching elements to implement the logical datapath ses, 17 Claims, 40 Drawing Sheets US 11,876,679 B2 Page 2 (60) on (2) 66) Related U.S. Application Data ‘continuation of application No. 13/219,557, fled on Aug. 26, 2011, now Pat. No. 9,391,928, and a con- tinuation of application No, 13/177,529, filed on Je 6.2011, now Pat. No. 8.743,889, and a continuation: incpat of application No, 13/177.536, fled on Ju. 6 2011, now Pat. No. 8.959.215, anda continuation-in- part of application No. 18/177,538, filed on Jul. 6, 2011, now Pat, No. 8890,823, Provisional application No. 61/505,100, filed on Jul 6,2011, provisional application No. 61/305,102, fled con Jul. 6, 2011, provisional application’ No. 61/$08,103,_ filed ‘on Jul. 6, 2011, provisional application No. 61/501,785, filed on Jun. 28, 2011 provisional aplication No, 61/501, 745, filed on Jun 27, 2011, provisional application No. 61/482.615, filed on May 4, 2011, provisional application No. 64/482,616, fled_on’ May 4, 2011, provisional pplication No. 61/482,208, filed on May 3, 2011, provisional application No. 61/466.483, fled oa Mar 22, 2011, provisional application No. 61/429,783, fled on Jan. 4, 2011, provisional application No 61/429.754, filed on Jan. 4, 2011, provisional ‘pplication No. 61/361.913, filed on Tul. 6, 2010, provisional application No. 61/361,012, fled on Ju 6, 2010, Int. C1 HOM 45/586 (2022.01) most 390 (202201), Gor 15173 (2006.01) Hod 246 (2006.01) ‘HOM 41783 (2022.01) Hos 4500 (2022.01) Host 4170896 (202201) ‘Host 61/5007 (202201) Mos 410816 (2022.01) Ost 4170853 (202201) Most Wer (202201) ‘Hos, 4502 (202201), GOOF 11/07 (200601), us. C1 CPC 1. Hod 41/0816 (2013.01), Hod 41/0883 (2013.01); HOE. 41/0896 (2013.01); Hod. 48/00 (2013.01); HOdL 45/02 (2013.01): HL 45/586 (2013.01), HOML 477783 2013.01): HodL 49200 (2013.01); Hoa, 49/1846 (2013.01); HOSE 49/3063 (2013.01); Hol 49/70 (2013.01); HOSE 61/5007 (2022.08), G06 11/07 (2013.01) Hal 2101622 202205) References Cited US. PATENT DOCUMENTS. 5080873 A 911901. Robins et a Sdoso02 A LLD9S Solway ea Seoa9a1 A ‘41006 Dev eal S5S0816 A 811906 Hardwick eta Shaves A 1908 Charwam tal S9si967 A $1008 Rasb aa S780 A 61908 Kuno Sr00996 ARID Wale ota E80s701 A 911908 Grossman et a §882222 A 11/1908 Deiados Sss4905 A 121998 Van Loo 5926463 A Sows A 6006.75 § ss 8 ‘esas ‘6002 21 & ‘10099 ‘108700 8 ‘6208/16 BL 6219899 BL A14986 BL 6812745 BL sian saisa2s ‘om 934 G07 38 n3s02 rats (668.740 rss 03 ‘888792 04985 ona ‘604187 ‘6959002 (6.063.885 ‘6088 887 ‘5996 056 ‘6000484 Fo0291 Faso 2088.171 296.228 saevans 7x69 Taw Tao2's1s 733891 Ta3o.678 Tani 73x10 390.971 7370.108 706.401 7490598 740048 Jasns7 Jamis 7asv370 7830 781274 755.002 7587492 7590.50 759334 1593 "42000 73000 73000 2000 2000 M00 ‘42001 112001 12002 2002 002 ‘4002 7002 003 Acar ta ‘Takahashi Picazo ea Vincent Bums ea eset fa. Holener eta aock ta. Mahalia ea. MeCloghne eta Yage! etl Esposio etl Mackene tal Nisbet Taguehs ta Shih et Chin Brokat et Chiles MeCrosky tal Posiman eal. Mea tl Gronke Ederman ta Zadikian oa Balakrishnan ea Wynne etal Ponace et. Gibson ta: Foster et Crump Stet Abe ea Martin Thelier ea Bova a Kaki tal. Giirer ea Yang etal Ting tl. Satkinen tl Marsan Lowy eta Matt Ravina etal ‘shi fomin eal. Coates ‘sen ta Sich eta. Gp ea Testa etal Ghent Naik eal Das etal Mercer Jovgenscs Goldene al Ot eal Bangi cal ‘Arn tal Oguchi ea. Bugengen el Post US 11,876,679 B2 Page 3 66) References Cited 8345558 BP U/2013. Nicholon ta S351MIs B2 12013 Zhao eal USS. PATENT DOCUMENTS. 352941 BL L2013 Protopopov ea Sa18216 B2 4013 Huphere a 7630358 BI 122009 Lakioni eal. 842230 B2 42013 Nakajima ‘Toaheiee 2 ‘12010. Khanna ca RasGoxt BP 62013 Ranganan ota 7680851 B2—U/2010 Takashige et SSOU7I8 B2 $2013 Wang ea. 7707309 B2 412010 Shanbhag eta. SS6S108 BL 102013 Mahal et a ‘Tun B2 S201) Balakrishnan al NS6SS07 2100013 Zheng 7768509 B2 72010 Dei eal 83578\008 2013 Brandsine et. 7709.057 2—-KIO10 oot, It a S590083 BR? 1L2013. Sees ea 7778906 B2 82010 Bauer 8612027 BL 122013 Brandwine TISS856 282010 Hashimoto et 621.088 B2 122013. Fswaran ea 92010. Vohe ea Seakiss BL '22014 Brande oa 92010 Wang Ros0'sis B2 22014 Asai etal 92010. Kitnura Seo 220l4 Ravan eta 9010. Verbeke ea S70S313 B2 4014 Merwe et a 7808929 B2_ 102010 Wong etal 8717895 BP $2014 Koponcn eta. TRIS4U BD 102010 Gry ca S7igom RP $2014 Roponen ea. TSISAS2 B2 102010 Mathews eta STANSAK B2 62014 Catado tal 7820482 BI 11/2010 Minet ce S74389 BP 62014 Koponen ta TSB0847 BY 112010 Nadeus et ool TN46733 B2 112010. ats a 3 2014 7847801 B2 122010 Oi soi ZSS0549 BZ 122010 Wheeler Tota TR7NG80 BL ‘T2011 Chapman ta rota 7881.281 B) 22011 Bugenbagen eal 82014. Roponen eta 7885276 BL 22011 Lin 82014 Catado etal TROIS BD 32011 Royalty S02) 299014 Koponun ta 7912955 BL $2011 Machina ta 883749) B2 92014 Catado tal 7925661 B2 42011 Broussard et KS42'67 829014 Koponcn ct al Tox67 BL $2011 Enum ea. S670 B2 102014 Mania eal Toss BL $2011 Millr ea SSS0.468 2112014 Koponen eta. Zpassi7 BL S011. Syvain Soisas) b2 122014 Pet a Tpasese BL S201 N 958292 8222015 Fulton et ‘Toasox BL S201 ROS021S RE 22015 Koponen ea Zpssses BI $2011 960328 B2 22015 Casndo eal 7958255 BL 63011 Rot30s B2 22018 Casado etal. Tomar? R620 Rose038 2 22018 Casndo eal Zprs272 B2 T2011 Chang et 966040 B2 22015 Koponen eta. 7991859 BI -RIOIL Miller ea 991186 B2 42015 Kogan a ‘Toosass BL R20LL Bayar oa S079 B2 42013. Roponen sta ‘SOOK998 B2 —N20L1 Levy et S.008:087 B2 42013 Lambath ea ‘$010.69 82 —N20L1 Sankaran eta Stoves B2 42015 ashame S023 BL 92011 Porolant et al 908482 BY S201 Fallon USL@33 B2 102011 Bueno et Sous BL 2018 Cola eta ‘SUR2899 B2 102011 Archer et al 9.088,183 B2_— 62015 Casa eal. Bode486 BI 102011 Miler ee SOTT64 B2 72018 Casado et a Rosas? BL TL20LL Shika ta 9.088609 B2 72018 Casado et al S0ss745 B2 102011 Aur 9.106.587 B2 2018. Koponen tak 088799 B2 112011 Richardson ea. SUIDSIL B2 82015 Casado tal 08032 B2 12011 Riddle ea. 911721663 B2 102018 Casado et ‘860,799 B2 120L1 Beadle otal 91270880 BL "22016 ‘Abraham ta ‘$060875 Bi 11/2011 Lambeth 91368210 B2 2016 Lamba ota Sow.87L B2 12012 Tole ea. 72016. Koponen etal 000608 B2 12012 Blingsty ot al $2016 Pandey ea: ‘8098686 B2 12012 Lapa et a 122016 Koponen ea 33012. Kwan eal 102018 Koponen eal RELA? BI 42012 Miler eal "62019. Koponen et 3b 8148630 BL 32012 On [0.326156 B2 6 2019 Koponen eta 10737 B2 42019 Metke ot M2988 B> 112022. Koponen ea SUSS028 B2 42012 Avellamich eta 800/864 B2 112022 Canad etal Sioits2 B2 42019 Opiceki cea aooiopeistt AL 11/2001 Viswanathan ea. $166201 B2 42012. Richanon eal aon20034i89 Al '¥002 Hadlock et 82012 Kagan et a 20020003082 AL 72002 Gonds (62012 See ea 20020161867 AL 102002 Cochran & a $3012 Atlan a 20020198369 AI 123002 Rawlins oa Reena: rota icuniei a al Sooxoo09ss2 AL ‘1/2003. Benfld stl OMT BI 72012 Miler et doosoodit7 A 23003. Stak 82300050 BL 72012. Brandine & 20030088850 AL 32003, Rangarjan et al 820872 BI 82013 Brandwvine et al 20030060072 AL 42003 Yoshimura etl 8330655 B28 2012 Goggin tal an0s0188086 AL 72003 O'Reilly 268071 B 92012 Sindhu ta 20030204768 AL 102003 Fee 268075 B2 92012 Pandey 20040084773 AL 32004 ays ota 281067 B2 102012 Stlowite ‘20040087286 AL 32004 Laren tal 831209 BI 112012 Miler e a 20040087300 AL 32008 Enomoto t 8320386 B2 112012 Louat tal ‘2n040DS4580 AL 32004 Kelley al 33309 BI 122012 Moisand «al 20040081799 AL $2001 Coleman 8339964 B2 122012 Gaanaseharan ta 20040073659 AL 4/2004. Rajsc ta US 11,876,679 B2 Page 4 66) 20080098505 doogo193677| 20040210889 2nown07%66 aoosoner7 20080018669 2008021683 2nnsoo77ss donsons0n7? 20080083953 20080120160 2nnso13204 00s 0201308 bn0s0n28952 donsoar8971 20080232230 290026225 2npan2s999 donsons707s 0030092940 2nos0093976 2npw10s7e 20060174087 | 200801 Tass 2donso1s2033 anpso182037| anos 1865s aonworsi93? 20080187908 | 2oos0190243 sonni93266 aops0221961 2oos0248109 2nne0282805 do0028239 2070083860 20070050763 aon 00ss789 20070064673 anor 0140138 so0701s6019 aooror7as 20070192395 aooron013 doovontrae 20090220089 2oononn0sse aonpon i> aon 079989 anor onao160 bonnion4so42 20070250608 aovrons07a. aonronee33 aonyoowstss aooporsasat aooronsTa8 doowon02s9 00s 000268 2oowon0ss soowontaaa9 ops 0040467 2oowons0U4 aopwo0s0621 2n0w0052006 20080039556 20080071900 aonworei20? aooso1ss704 2oowo189769 20080196100 20080212963 20080225780 20080225853 References Cited USS. PATENT DOCUMENTS. al A al 2004 Sanna 102004 2004 aun 100s Ws 205 005 005 200s sams 5005 ro.200s roanns ro200s 22006 2006 2006 Shove S006 Sa 82006 82006 2006 ‘82006 82006 2006 $2006 82006 2006 10.2006 112006 12000 22007 32007 3007 $2007 2007 S007 72007 2007 82007 92007 S007 52007 102007 fo2007 ro2na7 toa0o7 ro2007 2007 2007 1bauo? 12007 12007 3008 1 oe ¥ioos 3008 008 7008 82008 ‘82008 9 2008 9008 908 (Clemmensen Dare (Childress Carollo eta tite ‘Arad etal Newton a: Fauci etal Chan ea May Pluie a Guingo ta Naiket Mayhew al Sanne Nagar ea. Cama Takobashol ta Fratturn eal. ‘sms eta hina et al Krishnaswamy... HOAL.$8/22 “assina Hashimoto ot al Habibe Chen eta Chen oa Riel Abels ta. Shimano eal Bevin eta Siddha et al Basso a a Shor eta Rents otal Dyck eal Pas Kagan etal Clase etal Banta ea linker eta Pat ta Maceatert ea Moms Nebara l Mane Cnet a Goodie Surat etal Hole ea Paterson Jones a Margols et Wat Boe a Moore kaon et al ‘Octavian et Bose ea Cosboln et Drow et Chand al is et Mendirata ea Brisene ta MeGuire Edwards eal ‘Greenspan eal. Heer ea Renmasn ea Maret Canada et Madhavan eta Fomine a Mconmik et Melman et 20090240129 AL anugnDsiseo Al gnigormi06 Al 2n0s02919%0 Al Soowox01303 AL animonsiost AL 2ooo0083823 AL Sonounso2t AL aonoomeags AL 2oo 0099625 AL aonoonars9s AL 2oooor308 AL goooor22910 Al 2nvoot3ss7 Al dono014i220 AL aononrsosat Al 20090180527 AL aonooietsa7 AL gnionn20M AL 20090238325 AL aononrs047s Al aonoursta0 Al gninorrest Al 2nvo0r79s86 Al dono0r928s8 AL gnionr96926 AL 20090303880 AL soioop46sst AL aninpsi2e AL 20100061251 AL sornnx2709 AL dorowiision Al go1n0r2s66? Al ao1u0t3i6s6 Al So1wo138s30 AL soinnnaeao Al 2o1001s3sst Al son0162036 AL aoinntesso? Al 2o10018945) Al aoinoteoder Al aoinnteoss? AL ao1oo1si6.2 Al 2100191846 AL doiwoi92207 AL gomnni92296 AL 2o1w0r0849 At so1w02149# AL aoinmsi7et Al 2o100rsisss AL aoinrsi2 Al 20100775017 AL ani00r7si99 At 2100288400 AL 20100290385 AL sowor0293s Al 20100322255 AL Sotuinooisr AL So1v 0004913 At dotuooi0s7s AL doit oo1s2ts At 201410026521 AL 201110032840 AL do1voos2Kss 1 20110078552 AL or oorss6 1 2o1v007S6r4 AL o1100ssss7 1 SOIL 085589 AL doi o103239 1 domwotons91 At doiwo197ss AL dowwoL4oa AL 20110182053 AL 20110194567 AL doi o22s167 AL 20110261825 AL 2o1u0771007 AL 02008 02008 2008 Sox 123008 12009 22000 2200 20m 008 400 00 $200 Shoe 6200 ‘e200 ‘52008 ‘62009 S009 008 102009 03009 2009 112009 a0 122009 22010 ou 32010 #010 2010 $2010 Show 2010 ‘2010 62010 ‘62010 53010 72010 72010 72010 72010 Tho 72010 73010 2010 £2010 52010 102010 to2010 102010 102010 2010 2010 2010 12010 vou 2011 13011 voit 2201 220 220u1 3201 32001 3201 20 aon, ‘2011 Soon S201 6201 62011 82011 520uL 02011 2011 2009 De Richardson Challener ea Frey eta Tadhnt ta Matsuoka Clemivensen Mode ea. Ohno ea. Canes Kannappan ea Palabra Ran eta Barco ea. (Cea oa Fong ota Tia Tripathi ile ea. Drow eta Dimiros a Cohn Kove oa hit a Unban eal Lambeth ta. Saivly ta Mate ea ovat ta Piney el armas ot Detiaan eta ain etal Soundararajan| Soret Ass Kagan etal ‘Amschu & inden ta Sha etal ‘llr a Shae ‘Aut ta Raleigh Raleish Raleih Mucta ‘Akt ta Naryanan Martin ta Zhang el Hao & a. Dro Nevarja et Dominguez «al Wang Gamage ta. Merve etal Patel ot Mitsamoe amiath eta taal, Goantsckaran oa Chang ea. aybay «al Davis eal Edvard ta Merwe etal Merwe eta Shea Bhattacharjee Iehino Wing eta US 11,876,679 B2 Page 5 (66) References Cited wo 2002082019 42 4/2009 wo, 010103009 At 93010, USS. PATENT DOCUMENTS, wo ani0its0s0 A> 102010, wo, dotagesaae At ‘73012 2oivo2s2e49 at? 122011 041.4770 S708 soooeoss OTHER PUBLICATIONS aor 0290534, pee ees ‘Nox Published Commonly Owned 17S. Appl No, 17989864 led dor os108%9 Ala Now 18, 2022, 154 pages, Nc, ne avons Kemet a Anderson, Thomas, tal, "Overcoming the Teme passe oh oun ostsont Brasil imation” Ape 2005, page IEEE Comer Society aor200186 Xiong ta Anal Fabicne, tal, "Analysis and Evan of XEN based 2otz 09619 Tanaka ti ital Router” Sep. 3008, 63 pages, Unite de recherche INRA aorzoneoit Davis et ae eee hove-Alpes, Montbonnet Sinlamir, France. saiaeies one Aner, Mukaramad Bia tal “Buldng & Fas, Vitalie Data Soro Mulan 7 s = _. ae) 2 |e 2 eee le N <>} 105 > US 11,876,679 B2 Sheet 2 of 40 Jan. 16, 2024 U.S. Patent 25mg Zaundiy ws, }, soz pains SON caams yous | rains sz xz oz ove sz id ove US 11,876,679 B2 Sheet 3 of 40 Jan. 16, 2024 U.S. Patent § aan C401 Tap SON be on { su uojeoyjddy US 11,876,679 B2 Sheet 4 of 40 Jan. 16, 2024 U.S. Patent paundiy Tana Tang Er) su US 11,876,679 B2 Sheet 5 of 40 Jan. 16, 2024 U.S. Patent § andy PSOr 2s01 Caso c_Ps0t ioe! SON dq 1 ‘spu099y IN, au sae coo cr acta. be—e| [or] [= ] [= ] saat ns ses 7 $ 3 ae ses 7 ors J ses J oss U.S. Patent Jan. 16, 2024 Sheet 6 of 40 US 11,876,679 B2 coe 8 Ne x OC) XS B ' 7 65 Ft 600 Al B Al Al BY eee Figure 6 i; B J A oi) oe Application(s) NBS t 05 > 610 5 1 4 { { tf A] B Al B A] B ooh 6 ok ok Figure 7 US 11,876,679 B2 Sheet 7 of 40 Jan. 16, 2024 U.S. Patent 8 aundhy Tape Fane ) U.S. Patent Jan. 16, 2024 Sheet 8 of 40 US 11,876,679 B2 900 Figure 9 aX) eo* A XX) ee Od U.S. Patent Jan. 16, 2024 Sheet 9 of 40 US 11,876,679 B2 Global NIB ios cms cine Control Control Virtualization Virtualization NOS} Nos} Figure 10 1020 ~ Global NIB__ 1025 |) < Global NIB | | 4 | Control Control Control Control Figure 11 US 11,876,679 B2 Sheet 10 of 40 Jan. 16, 2024 U.S. Patent E1 aanByy sist Ee wopeariddy jon | T asn U.S. Patent Jan. 16, 2024 Sheet 11 of 40 US 11,876,679 B2 Figure 14 NIB U.S. Patent Jan. 16, 2024 Sheet 12 of 40 US 11,876,679 B2 1500 1510 > cso | Switch 123 C1530 Host 456 — Link479 oc ml 1 3 © 1520 10 T = Figure 15 1610 1650 Forwarding Engine Key Yate Vane wo | Swieni3d Tos Se on On Pore Poa 1640 Key Value Value Key > Tak Pore On Tinks Sane Tink ‘On Tinks Key D Value Tinks 79 On Figure 16 US 11,876,679 B2 Sheet 13 of 40 Jan. 16, 2024 U.S. Patent LL andy ocLI >) coll od ls--->] 2pON le} roman, ” oot F Covel yury oyqe, Supassuog Fede] outug Supsessog 80H] I osu J out J out J U.S. Patent Jan. 16, 2024 Sheet 14 of 40 US 11,876,679 B2 1810 > 1820 > Node Class Port Cass Attributes > Attributes, + Pons Statistics Adreses Nove ports bound ame of Node Port Numer Statsies Logical Por Number Type Identities suas Name of Por (alee Adresses Link Class ‘stebute hI Type Identifier ser Por Attached link ad Port Conta Figs 18405) sits Queeotestonsof PY” queyeCaeon pe] Class Aeributes speed Queues Weight 1850 > Qs without numbers Queue Cl Usage Attribut Ports bound to this ae ae it Te © Type ldentier Qe ahove and below thisone ‘Number Quene Mapping 4 Transmit pockets Sree aaa this one 1 Received bytes 4 Transmit Emons ‘Type Identifier Figure 18 U.S. Patent Jan, 16, 2024 Sheet 15 of 40 US 11,876,679 B2 1910 > 1920 > 1930 > 1940 > ccnasats cass [L| rorwaraing engine [| vorwaraing able Network Class Attributes PP] Class Attributes P| ‘Chass Attributes Attributes Forwarding Engines Forranling Toles ow Fos Newwork Type Management Comaton Sate Engine ble fs bound ‘ype Kener Managemen Connection S 7 pe 5 ‘Chassis Engine ls Tocal unique able Connection Ste sss Ei unig | Datapath 1D ‘x Packs 1950 1960 ‘Bored 7 1 Engine Capability Host 1 T User Flags. Engine Action Fags Use Host Contsiner ‘ype Menifee Figure 19 2000 Common NIB Funetions Query Create Destroy ess Attributes Register for Notifications Synchronize Configure Pull entity into the NIB Figure 20 U.S. Patent Jan. 16, 2024 Sheet 16 of 40 US 11,876,679 B2 Figure 21 aX) eee . Od) e°* Od) US 11,876,679 B2 Sheet 17 of 40 Jan. 16, 2024 U.S. Patent ostz shez tz aan US 11,876,679 B2 Sheet 18 of 40 Jan. 16, 2024 U.S. Patent € aandyy aus we sr 98ury LGN © ose US 11,876,679 B2 Sheet 19 of 40 Jan. 16, 2024 U.S. Patent oste oore a 2) 1 ana PC And Ly oor 1D Sauna we | GA sr aay & sovz U.S. Patent Jan. 16, 2024 Sheet 20 of 40 US 11,876,679 B2 2500 + 2570 2565 Ptp Tiigaer Lis 2505 > | C2525 C2545 cMtocm PTD Tieser DHT Range Interface Tracker Taenttier 2530 2510 ~ Z C2550 NIBio PTD Ordering Interface Raed Mose 2575 > 2515 7 2535_~) | 2555 NIB CM tostance a Master Taker NOS Treker M Ka Uist 2540 2560 Switel MasterList cM Insiance List Nos. Instance List Figure 25 U.S. Patent Jan, 16, 2024 Sheet 21 of 40 US 11,876,679 B2 | 2600 | pxtp }———+] Application interface 2 a 2 | 645 608 _ i Controller Tash Table3@24 | 207? cio | x 2610 2620 | = Notification Request List ae ‘Transfer Modules 2630 2615 Sich Contoler cM 2635 DAT 705 ‘Triggers CM Interface [} 202 |] DHT Instance PID Triggers oa sich Inerice PTD 2640 | ess Figure 26 U.S. Patent Jan. 16, 2024 Sheet 22 of 40 US 11,876,679 B2 2700 2710 v Register application's request for callbacks on the NIB data record ‘NIB data record changed? Does any application callback get triggered? No requests? 2750 2 Send Application(s) notification(s) from the NIB (=) Figure 27 U.S. Patent Jan. 16, 2024 Sheet 23 of 40 US 11,876,679 B2 a (sm) : 2805 Figure 28 1 Receive notification oF ‘change within NIB 2810 Do any secondary siorages Isthere a new nei tobe object inthe NIB? notiied? 2815 Does now Selet secondary storage structure to Doss ne No notify created in secondary storage ayer 2 ge ly Notify secondary storage structure ‘Yes ¥ Z Select secondary storage structure fk ‘Any other secondary storage structure need to be Yes written to? 2800 Y ‘Create record in selected secondary storaye structure for new object 2860 > 2825 y No to NIB if necessary 2830 ¥ ‘Add secondary storage to list to notify of NIB record change 2835 U.S. Patent Jan. 16, 2024 IDET TeNICeRITN PID Resort? CI Calbsck Sheet 24 of 40 US 11,876,679 B2 CM's PTD Trigger List =a Reson? cans Coaullers 23 PTD Recon SP CMECaTIbase PID Recon —[ CMI Callbck: Pr Resend SP CMI alk 3030 DHT DTT DHT Recon DHT Resort DHT Record 2930 Reset? Rett Resort Controllers C20 Coolie] Conller 2955 Figure 29 3040 3050 Hash Record I Triggers Hoshicdex | Data Contes 12,4 Figure 30 U.S. Patent Jan. 16,2024 Sheet 25 of 40 Receive notification from secondary storage of change to record 3105 need to be No imported into NIB? 3115 2 Query the secondary storage structure for the value of changed record, and include registration for subsequent notification if necessary i Import change into NIB Yes Figure 31 US 11,876,679 B2 3100 U.S. Patent Jan. 16, 2024 Sheet 26 of 40 US 11,876,679 B2 C3200 cm NIB 7 303 NIB New NIB Envy 326 Seoondary ‘Secondary ‘Ssondary Storage Laver Storage Laver Storage Laver 7 306 C305 NIB NIB impor ] | : 1 [spa] [irpon Movute | [state || ' 1 Loa onary ‘Secondary ‘Secondary Storage Laver Storage Laver Storage Laser Figure 32 U.S. Patent Jan. 16, 2024 Sheet 27 of 40 US 11,876,679 B2 Instance 1 Each Controller Instance: Updater 3320 c ia 330 Reseive a change to request c 7 Fe ‘Change request list (change Reena j--P| Retrieve Request 3330 End Isthe fnstance maste? of NIB portion 7x, fe changey Yes pais a Call NIB Updater | ~ ‘Waite change in request list «0 mastered area of NIB, ss ¥ Remove change from request i 3300 v a Update response list (propagate through) Figure 33 nd US 11,876,679 B2 Sheet 28 of 40 Jan. 16, 2024 U.S. Patent Ogre rE aanBiy 4098090 fury LHa sof, d sive i ! © oe S141 USPH +—>| sosuodsoy, safeunyy Auand) = souand AC cove U seve sowoy, { 2Inpopy ons Lua a1000Y © owe \ core US 11,876,679 B2 Sheet 29 of 40 Jan. 16, 2024 U.S. Patent ogse oo CL_ soap eH ozse SE any © oise © oose U.S. Patent Jan, 16,2024 Sheet 30 of 40 US 11,876,679 B2 3620-5 3660 > Hash 3000 >) = Gover LU 3665 3660 367 Soto 5 3680 J C3657 3630 3685 >) on = pur k= ‘Continmation ‘Manager > Range List 3660 © 3015 = 3680 > ae se 3095 iS = 3660 > = \ 3670 — Trigger Hash, Cae] reser Tabe T 3650 een co = Cseos Figure 36 U.S. Patent Jan. 16, 2024 Sheet 31 of 40 US 11,876,679 B2 3700 >, 3720 > el Hash key Generator Tesh C3765 x0 5375s 5 3765 3730 Caro c cy put a5 > Miser ange List 7 Cans 3701 >) NOS Instance A i NOS instance 8 L a 3706 J 3760_> mth 3765 ad Kh 3770 sotnion 3795 rH oie aris OA ates s100 5 3160 > vate = ey te ae insane ns |] tance ms woleon _ ‘ry Mangger F———] as abe Taras] Piscenor 370 J weit 3750 190 J C3760 U ass Figure 37 U.S. Patent Jan. 16, 2024 Sheet 32 of 40 US 11,876,679 B2 Start 3800 c Y rc x10 sess request Y 3820 Generate hash if’ necessary v 3830 Check DHT range list 3840 3850 Can access request be processed locally’ Yes Execute access request No Y 3880 Y 3860 Send access request to are nee Receive triggers if any Y 3885 y (~ 3870 Handle trigger Receive confirmation * notifications if any y 3890 Send confirmation [_g——_______] Figure 38 US 11,876,679 B2 Sheet 33 of 40 Jan. 16, 2024 U.S. Patent 66 aan3y ox6e_>y oF nun, sono ae | 20u2001 WD >] — see T oe ove J IN J ‘oe F did oe os6e >) owes > 28019 = shod sousond uuadlo = suo 11° 198 vow LL = anueuor jr U.S. Patent Jan. 16, 2024 Sheet 34 of 40 US 11,876,679 B2 C4000 NIB to PTD Layer Propagation C4010 1 p= PTD Layer to PTD Layer Replication [pe PTD Layer to NIB Propagation Figure 40 U.S. Patent Jan, 16,2024 Start Receive update request for PTD. 4120 Sheet 35 of 40 Y Cc 410 US 11,876,679 B2 CT Aloo C4170 Is this instance a PTD master? Perform master update process. C4130 ‘Transmit PTD update to PTD master. Received update command from PTD master? 4140 Update PTD, End Figure 41 U.S. Patent Jan. 16,2024 Sheet 36 of 40 Start os - Receive PTD update request Te Direct all slave instances to update their PTDs and request acknowledgment of completion of update from slave instances Received ‘acknowledgement from all slave instances? US 11,876,679 B2 c 2 Call error handler Record PTD update in master PTD. J 260 Y Send NIB update commands to all instances. and request acknowledgment of completion of update from instances am 4280 Received acknowledgement from all instances? Call error wat [-—*( ts Figure 42 U.S. Patent Jan. 16, 2024 Sheet 37 of 40 US 11,876,679 B2 430s 2 4361 sm 1 369 me nu ale a rs wate TO frac E4364 | te a3si |” =a TH (4365 ‘| oy T 4360 J i 4360 J woes eee | ss Cece Las) a 4370 FT 4380 7 4370 J 4380 J ais 5 ‘Lissnsfoe F 4364 at a | 4304 Bef 1364 C m__| feat | BCS 4350 J 4360 J 4350 J 4360 J 4370 J 4380 J 4370 J 4380 J Figure 434, Figure 43A |Figure 43 Figure 43B “To Figure 4325 U.S. Patent Jan. 16, 2024 Sheet 38 of 40 US 11,876,679 B2 ans 5 305 ee 7s | ree |e a Interface, Juteriace uiecfave’ 4396 Joteriace_} Cos qe | lees ‘> ee 4350 J 4360 J 4350 J 4360 J tee Luss tauriee if MY 396 | liaaees 4370 7 4380 7 370 7 4380 7 4340 >) ae pancae) = Leda ee | thy i, 7 mi ou | ass} a im FF tc eT sai | a (ee 4350 7 4360 J 4350 J 4360 7 _ 497 NIB, 4 NB 4 NIB. E 4362 c y— 4362 2 i a as 4370 J 3x0 7 4370 7 4380 7 Figure 43B Figure 43A Figure 43: Se U.S. Patent Jan. 16, 2024 Sheet 39 of 40 US 11,876,679 B2 C400 cso NIB to DHT instance Propagation Tp Record change in DHT Tp Send updates to each NIB import module on trigger list End Figure 44 US 11,876,679 B2 Sheet 40 of 40 Jan. 16, 2024 U.S. Patent Sp aang orsr 3IOMION, sou JOssao01g soo1aaq A1OWayA, ose nding waists sbsr US 11,876,679 B2 1 METHOD AND APPARATUS FOR INTERACTING WITH A NETWORK INFORMATION BASE IN A DISTRIBUTED NETWORK CONTROL SYSTEM WITH. MULTIPLE CONTROLLER INSTANCES, CLAIM OF BENEFIT ‘TO PRIOR APPLICATIONS This application is » continuation application of US. patent application See. No. 15/181,685, filed on Jun. 14, 2016, now published as US. Patent Publication 2016) (0294627, US. patent application Ser, No. 15/181,685 is a ‘continuation application of U.S. patent application Ser. No, 13/219,557, filed on Aug. 26, 2011, now issued as US. Pat No. 9.391.928. US. pateat application Ser. No. 13219,557 js 2 continuation application of U.S. patent application Ser No. 13/177,529. filed on Jul 6, 2011, now issued as US. Pa. No. 8.743.889. U'S. patent application Ser. No. 13/177,529 ‘luims the benelit of US. Provisional Patent Application 61361,912, filed on Jul. 6, 2010; U.S. Provisional Patent Application 61/361,913, filed on Jul. 6, 2010; US, Provie sional Patent Application 61/429,753, filed on Jan. 4, 2011 US. Provisional Patent Application 61/429,754, filed on Jan. 4, 2011; US. Provisional Patent Application 61/466, > “453, fled on Mar, 22,2011; U.S, Provisional Patent Appi cation 61/482,205, filed on May’ 3, 2011; US. Provisional Patent Application 61/482,615, fled on May 4, 2011; US. Provisional Patent Application 61/482,616, fled on May 4, 2011; US. Provisional Patent Application 61/501,743, filed ‘on Jun. 27, 2011; and U.S. Provisional Patent Application (61/501 785, filed on Jun. 28, 2011. US. patent application Ser. No. 13/219,557 isa continuation-in-part application of US. patent application Ser, No, 13/177,338, filed on Ju. 6, 2011, now issued as US. Pat, No. 8,830,893. US. patent application Ser. No. 13/219,557 is also a comtinuation-in- prt application of U.S. patent application Ser. No. 13/177, 536, filed on ul, 6, 2011, now issued as US, Pat. No. 8.950.215. U.S. patent application Ser. Nos. 13/177,536 and 19/177,538 claim the benefit of US. Provisional Patent Application 61/361,912, filed on Jul. 6, 2010: US. Prov ‘sional Patent Application 61/361,913, filed on Tul 6, 2010; US. Provisional Patent Application 61/420.753, filed on Jan, 4, 2011; US, Provisional Patent Application 61/429, 754, ied on Jan. 4, 2011; US. Provisional Patent Applica: tion 61/466,453, fled oa Mar. 22, 2011; U.S. Provisional Patent Application 61/482,205, fled on May 3, 2011; US. Provisional Patent Application 61/482,615, fled on May 4, 2011; US. Provisional Patent Application 61/482,616, fled ‘on May 4, 2011; US. Provisional Patent Application 61/501, 743, fled on Jun. 27, 2011; and US. Provisional Patent Application 61/501,785, fled on Jun. 28, 2011. U.S. patent ‘application See. No, 13/219.587 claims the benelt of US. Provisional Patent Applicaton 61/482,205, fled on May 3, 2011; US. Provisional Patent Application 61/482,615, fled ‘on May 4, 2011; US. Provisional Patent Applicaton 61/482, 616, fled on May 4, 2011; U.S. Provisional Patent Appli ‘ation 61/501,743, filed on Jun. 27, 2011; US. Provisional Patent Application 61/501,785, ied on Jun, 28, 2011; US. Provisional Patent Application 61/$08,100, fled on Jul. 6, 2011; US. Provisional Patent Application 61/505,108, filed ‘on Jul. 6, 2011; and US. Provisional Patent Application, 61/505,102, filed on Jl. 6, 2011. U.S, patent application Sor. Nos. 15/181,688, 19/219,5S7, 13/177,520, 13/177,536 and 19/177,538 ‘and US. Provisional "Patent Applications 61/361,012, 61/361,913, 61/420,753, 61/420,84, 61/466, 453, 61/482.205, 61/482,615, 61/482,616, 61/501,783, o 2 61/501,785, 61/505,100, 61/505,102, and 61/505,103 are incorporated herein by reference. BACKGROUND ‘Many current enterprises have large and sophisticated networks comprising switches, hubs, routers, servers, Work- stations and other networked devices, which suppor a variety of connections, applications ‘and. systems. The ‘increased sophistication of computer networking, including Virtual machine migration, dynamie workloads, muli-ten- fancy, and customer specific quality of service and security configurations require a better paradigm for network contr Networks have traditionally been managed through low level configuration of individual components, Network con- figurations often depend on the underlying. network: for ‘blocking @ User's access with an access contol list entry requires knowing the user's cutrent IP address. More complicated tasks require more extensive retwork knowledge? forcing guest users” port 80 trai to traverse an HTTP proxy requires Knowing the current net- work topology and the location of each guest. This process is of increased dificlty where the network switching ele- sents are shared acrost enuliple ers. In response there is a growing movement, driven by both industry and academia, towards a new network control paradigm called Saftware-Defined Networking (SDN). In the SDN paradigm, a network controller, running on one or more servers in a network, controls, maintains, and imple- ‘ments contro logic that governs the forwarding behavior Of shared network switching elements on a per user bass. Making network management decisions often requires Knowledge of the network state, To facilitate management decision making, the network contoller ereaes and main- tains a view ofthe nenwork state and provides an application programming interface upon which management applica- tions may access a view of the network sate. Three of the many ehallenges of large networks (includ {ng datacenters and the enterprise) are scalability, mobility. ‘and mult-tenaney and often the approaches taken to address ‘one hamper the other. For instance, one can easily provide rework mobility for vital machines (VMs) withia an L2 domain, but L2 domains cannot scale to large sizes. Aso, retaining, tenant isolation greatly, complicates mobility Despite the high-level interest in SDN, no existing products have been able to satisfy all of these requirements. BRIEF SUMMARY. ‘Some embodiments of the invention provide system that allows several diferent logical datapath sets to be specified for several different users through one or more shared network infasiructre switching elements (elered to as “switching elements” below). In some embodiments, the system includes a set of software tools that allows the sytem to accept logical datapath sets from users and to confignre the switching elements to implement these logical datapath sets. These software tools allow the system to virtualize control oF the shared stitching elements and the network that is defined by the connections between these shared ‘switching elements, in a manner that prevents the different ‘sere from viewing or controlling each other’ logical data- path sets (ie, each other's switching logie) while sharing fhe same switching elements In some embodiments, one of the software tools that allows the system to virtualize contro ofa set of switching clements (ie, © allow several users to share the same US 11,876,679 B2 3 switching elements without viewing or controling each ‘lher's logical datapath ses) isan intermediate data storage structure that (1) stores the state oF the network, (2) receives and records modifications to differen parts of the network from different users, and (3), in some embodiments, pro- vides different views ofthe state of the network to diferent users. For instance, in some embodiments, the intermediate data storage siicure is network information hase (NIB) data structure that stores the state of the network that is ‘fined by one or more switching elements, The system uses this NIB data structure as an intermediate storage structure Tor reading the state of the network and writing modifiea- tions to the state of the network. In some embodiments, the NIBalso stores the logical configuration and the logical state foreach user specifi logical datapath set. In these embodi- ‘ments, the information in the NIB that represents the slate of the actual switehing elements accounts for only a subset of the total information stored in the MB. In some embodiments, the system bas (1) network ‘operating system (NOS) to erate and maintain the NIB. > Sorage structure, and 2) one or more applications that ran ‘on top of the NOS to specify loge for reading values fom fand writing values tothe MB, When the NIB is modified in ‘order (0 elfetuate a change in the switching logic of a ‘witching element, the NOS of some emboxliments also propauates the modification to the svtehing element The system of different embodiments uses the NIB dif= ‘erently to virtualize acess tothe shared switching elements and network. In some embodiments, the system provides different views of the NIB to different users in order 10 fensure that diffrent users do not have direct view and control over each other's switching logic. For instance, in ome embodiments, the MB is hierarchical data strctare that represents different attibutes of dilfereat switching elements as elements (¢.. different nodes) ina hierarchy. ‘The NIB in some of these embodiments is @ multilayer hierarchical data stricture, with each layer having a hierar chical structure and one or more elements (@., nodes)

    You might also like