Se ere ee oe ed standing human learning and inductive reasoning eran eek ne ed peer ees See ee eee ete ees See een pee oe en er ee ee eta ee eee poe ea ser cere need ie pert eu et Sem ene nee een Seana eereet ne Sens Wid i ore Pes a ERR OO Re nc . Nira ges ao OC Rod AeA AU tO SETS] Te ea SANJEEV KULKARNI cee als GaW EY\WILEY SERIES RC PROBABILITY AND STATISTICS tals by WALTER A. SHEWHART and SAMUEL S. WILKS. Edt: David, ling. Noe A C. Ces, Gare Mf. Pomaric Haves Gols, ain lost, Gor Melenbergs, Dovid. Sat, titan FM, Smith Ray Tay, Sanford Weiter tors EmcitYc Boro J Sh ante, Josaph Kade Jaf Tegels ‘compet ist fhe tis a hi ss apa at eed ois vane, An Elementary Introduction to Statistical Learning Theory SSANJEEV KULKARNI Depa School of Eginsing amd Apple See 9 of lect aging GILBERT HARMAN Deparment of Pilosohy Pinson Univesity WILEY |A JOHN WILEY & SONS, INC, PUBLICATION‘oil 201 te Wy Sn ne A es an tine ey Se Han, ee pus pat mye pl eto pc nay vip ma cknaa maha, opens ay vi, rca te Se tr Cae Ses Cr ia ee pl ‘Shs as Pe ate enh pn eo ay (Sse Coy Cee ino Remone be ures 4 OO) THN SeuTe sat wee eae’ it eS ‘ue, ran Toa) ae | Se yc Lino incr Wir: Wk he ptr mb ne ae ‘eta ay eo eins ears hoe ey ot ‘fin ets a rn a yess tan ny ig! woes ‘rie fs ym ec se ‘hte ft ae yh of ple yer mers amg han ‘tn nmap tn omen ao eee fein ot ie lit ln ln a Seer oe EN Se ‘toy pte in cet: oma or mv fren son Wey ed vio hwy of ans a Pain Da ‘ie ey icin in ig ay Sas Kir ier Ha eer yep SSeS etna nee wae Contents Prete 1 Introdeton: Cheatin, Learning, Featre, an ‘ppliations LL Scope 1 12 Why Machine eaming? 2 13 Some Appleton, 3 13 Inge Resopnion, 3 132 Sposh Recopitien, 3 133. Mec! Dingpori, 134 Seated Ariage 4 14 Mownsteens, ests, of eae Veco 4 15) The Nex fr Probie, 5 16 Supers Leajg. 5 17 Senay. 6 18 Append Inaton, 6 18 Questions, 7 110 References, 8 2. Probably 2.1 Probably of Some Base Evens, 10 122 robbie of Componnd Eve, 12 23 Contos Probab, 13 24 Drawing WitoutRepecenent, 14 25 Alans Buty Prdtem, 15 26 Rando Vasa, 15 apace Ye, 16 wnn 2b Vaan, 17 29° Summary. 19 210 Appel: Imerpretios of Pobbity, 19 211 Questions, 29 212 References 21 13 Probabiity Denses 31 An Example a Two Dimensions, 23 32° Random Numer ia (1), 23 33 Density Functions, 24 34 Proablty Dense ie Higher Dimensions, 27 45 Ising nd Contos! Dente, 27 36 Expected Valu and Virace, 28 37 Laws of Large Numbers, 29 38 Summary, 30 39° Appentt: Measuabiliy, 30 3.10 Qustins, 32 BIL Referens, 32 4Te Pattern Recognition Problem 1A Simple Example, 34 42° Decivon Rules, 33 43 Suess Ceo, 37 4A The Bex Gasifer Bayes Decision Rue, 37 445° Cominanor Fetes and Dewi 38 86 Sommary. 39 447) Agpwa Unsoamably May. 39 48 Questons, 0 49° References, 41 {5 The Opsinal Bayes Decision Rule St Bayes Theme, 52 Bays Decision Rule, 44 ‘53 Optimality and See Commens, 45 54 Anxanple, 17 55. Bayes Toren snd Decision Role with Dense, 8 56 Summary, #9 57 Apps: Defining Conical Pobbility, 50 ‘CONTENTS 58 Question, 50 59 Referees, 53 earning from Hsamples ss 61 Lack of Keowee of Distbuions, 55 62 Tsinng Da, 36 63. Assumptions onthe Ting Duta, 57 GA A Bev Force Apprsch to Lesrig, 92 65 Curse of Dinmnsonaty, Indusive Bi, ed No Fre Lach, 60 66 Suman, 6 {57 Appenti: Wht Sort of Leasing? 62 53 Questions 89 References 6 ‘The Nearet Neighbor ule 6 TA The Nes Neighbor Rae, 6S 72 enformune ofthe Newest Neighbor Ral, 66 73 nan and Prof Sketch of romance, 57 18 Using more Neigh, 73 Summary, 70 16 pent: When People use Nearest Neighbor Ressning, 70 161 Who i a Buchel? 70 162 Legal Rescaing, TL 163 Moral Reasoning, 71 17 Questions, 72 78 Refences, 13° ‘Kernel Rates ™ 1 Matatin, 7 82 A Vast on Neat Nein Rules, 25 83 Koma Rules, 76 A Universal Costensy of Keel ales, 79 5 Pete Fupcions, 80 86 Move Genet Keres 8 87 Suoary, 82 88. Apenis: Kerol, Sina, and Fears, $2 89° Question, 8 10 Referens,le vi ‘CONTENTS Neural Networks: Porxprons 6 9.1 Mullye Feedforward Neon, 6 92 Neunl Newerks for Leasing and Clasistion, 87 93 Pecepns, 39 931 Tresld, 0 94 Leming Rae for Pecepbos, 0 9 Repesoations! Capa f Peeepuos, $2 95 Summary. 94 97 Appeals: Model of Mind, 95 98 Question, 96 99° References, 97 10. Malayer Networks » 104 Repeseation Cabin of Maitayer 102 Leaning and Sigal ups 101 103 Thing Ere and Weak Space, 108 104 nor Minimization by Gradient Descent 108 105 Backqropagtion, 106 1.6 Deron of Beckpropagation Equtions, 109 106.1 Derivation fr «Single Uni, 110 1062 Derivation fora Nework U1 107 Summary. 113 108 Appendix: Gratien Descont and Reasoning toward Reece aulinaen, 13 109 Questions, 114 1010 References, 115 11 PAC Learning us 111 Clas af Decision Rue, 117 112 Bes Rule om @ Cs 118 13 Probably Apposintly Comet Citron 1 14 PAC Leal, 120 15 Semmiry, 2 - 116 Appendix: KetisingIadscemibls, 122 117 Questions, 128 A References, 123, contents, & 2 Ve Dineasion ms « 121 Appoxiaton aod Estiaton Boe, 125, 122 Stating 126 123 VCDimewsion, 127 124 Lewin Res 128 125 Some Examples, 139 126 Appliction to Newal New, 152 127 Summary, 15 128 Amend VC Dimension a Popper Dincesio, 133 129 Quen, 134 1210, Referens, 135, Infinite VC Dimension ro 131A Herat of Cae and Made PAC Ceteon, 138 132 Misi Ves Complexity Ta-Of, 138 133 Leaning Ress 139 154 Tncive Bie aad Simpliciy, 140 135 Summary. 141 136 Appendix: Uniform Convergnce and Univeral Cassone, 141 137 Questions, 142 19S References, 15 ‘The Function Fsimaton Probem, wt 141 Bama, 144 142 Soc us 143 Best Ett Rearesion Faction, 146 144 Leaning in Fonction Estimation, 146 143 Sommay, 197 186 Appnin: Repression Toward he Mean, 147 147 Queson, 148 14S Retennce, 149 earning Fanci Estimation 150 15.1 Review of he Fanon ExinatonRegeson Problemy, 150, 152 Nears Meighor Rules, 15t 153 Kemal Metids, 15 154 New Nework Leung, 152 155 main with «Fixed Cass of Functions, 15316 156 Sharing, Peed Dimension, and Leaming, 154 187 Coectaion, 156 188 Append Accuracy, Precis, Bian, and Varance in ‘eeimaion, 156 159° uetins, 157 1310, Refeece, 158 Simply 16 Simplicity in See, 160 GLI Explicit Appeal Sinpliciy, 160 16:12 tbe Word Sipe? 161 1613 Maen Appelt Simply, 151 16.14 Inplict Apps wo Simply, 161 162 Ondering Hype, 162 1621 Two Kids of Simplicity Ordering, 162 163, Two Examples, 16 163.1 Cue iting, 163 1632 Enumentive Indotion, 16 164 Simplicity a Simplicity of Representation. 165 1641 Fic ona Paras System of Representation? 166 1642 Ave Fewer Paarots Singer? 167 165 Pragmitio Thor of Simply, 167 166 Simptisty sd Gib Ladera, 168 167 Summay. 169 16 Appendix Rai Science an Stasical Laing Theory, 169 169 Questions, 170 1610, References, 110 Support Vector Machines 17 Mapping te Teatre eon 173 172 Maximising the Magn, 175 173 Opinizaton sn Suppor Vectors. 177 174 pletenton ad Connection o Kernel Meas, 17 175 Denis ofthe Opinion Polem, 180 171 Resting Separation Conditions, 180 1752. gta fo Margin 1753 Shak Varbles for Nonepable Examples, 18 17544 Reforma and Sohn of Optiminasion, 182 175 Summary, 188, 160 m ‘CONTENTS “ wa na m9 Append: Canute, 134 uesions, 185 References, 186 18 Booting a ie 182 183 ry iss 6 187 188 9 Wes Laing Rules, 187 Combining Clin 188 Disebaion oa te Ting Bumps, 189 ‘he AdabosAlgothn, 190 Perfomance om Tining Ds, 191 (Generation Peformance, 192 Surmay, 194 ‘Appendix: Eosenble Meds, 194 ‘Questions, 198 110, Retereees, 196 i i z ;Preface “Tic bok of toad and aces noduton to te slave new fd ‘fsa omming or, lld har has emerged Fo enineing tes of tle engntn end machine leasing. dovlepnenss in noaparameic tatsties, ‘Segue sceace, be stay of ane eaning in nui, dveopnenal and ‘iene pehalogy, fe pulovopbi robin of ied, and the pion ‘Th bok iste probe! of ery suceesflnsodutry course on “Lesing -Theary and Epistemology” ht we have been aching jl ie tical en tecting and pose at Princeton Universi. The eure ope a ll sens ‘dha a pie preeiteseher tas Soe aja sls nd init ‘rout. Aoegh ch of th teal ohn we ve Fond tat te min sar oth ascearible and ppeisted by a tage of tales In ach ‘Shy, dts ve iladd freshmen roa esos Wh mas fom te ‘sions ongnering, humans, ad socal sence ‘The engnering tay of pete cognition 'cooened with dveoying ao nated cst dcimialeberwen varus inputs 2 well way. Hw can ‘he pat oes develop sts wo Sean and somal on he ass of awe ‘ndrnes? How ean 4 maufctre cain «computed sys to ani ‘rly omnes? Can comps Be ed aye medal mages 0 make ‘Gagmes? Moc leaning povdes an ein way to approach Some pater eg tion robles. is pil to tan ater esis bande ip cds. ‘Atoms ystems cn inert WE west ea oper spoch eos tion. Acompuis nigh uo chin leaning to velop ays that can sale neil ages ne ay at exper do. ‘Mache ling td pater renpiton seaso cca ith the gen sl pinpes nvidia eanig syste Rate than develop algriins fm ert and aad Boe manner for nck ew aplication sstemute methods ‘xy canbe extesly uel, tis a impr fo have tiga fr eels {te peroornce of leasing stem. Knowing wha Is chicwale end what iPREFACE po provie Resch and fen migrate ehni for pti ean ales “These gusious ae also related pilonpical questions tat aie inept ‘mology. Wat can we leo sd ow can we lea” What can we lean sos ‘ther ins and he external wos? What can we etn Cough indo? “Tae placa rodlem of ndaion sls how hs oust my ing onthe ai of ndtv reson, given hat eth the ress of inde fhe resi dacs nt guarantee tah of ie conelson, Ter sno singe ‘ofuon to this problem, not boss thee sno slum, ot Hea three ‘many. dependiag en what outs ernng. as bok, We cela hw vais ‘ottion depend on he way te elem of inductions formule, “Tus, we hope tis Dok wl serve ss an mss nehton ttl Jeaing thar for s bod satince, Forte terested amore indepth ses of eaning tour or pet algo, we hope ebook wil pre 3b Sting pst Fer thse iestd ia epistenaogyo& pkg in gene, we ope the book wl lp dn eomectons imo ver eleva eas rm te ei ‘And for ies, we hope ebook wil el vie an uderstuoding 0 some deep ‘el funduretl igh rom stl aang thn hat ae a the be of vances nari eligece an shes ight on eet ond is of ering "Ne acknowledge wah uke a Culm Development Gat fn e250 Amiverary ond for inovaton in Underage Eavcaion fom Pinel Unt ‘ers. Rajeev Kulkami gave ws exeely useful comments ono whole book, ich ha realy improved the et. Joel Pre and Miya Gupta povided “alos cont on varios pars. We hae alo enefied frm cael tng by Joss Has We are alo peel wo our eachng ata oer th ar tnd de many sans who te douse th coment ofthe cane with ‘Thakst CHAPTER | Introduction: Classification, Learning, Features, and Applications 14 score tn dhs took we ate concer mainly wih tem clasifcaion—clasiying en aft chet to one of several eateries onthe bs of seven observations oc mea Suenens of ths eles The snp ese Is chsileaton ofan aj no on of ‘wo cargoes, bua more general cate lows fray nite suber of etegor=s. ‘hsesnd hvely sod tank seston of +l umber ht ply ban ised Wo some ppt of he bck. Abi clasibtin sevrleberains. So cr meatuemeat of the objet we sale bd or estimate sued on se ‘eration, Mos of ear iscusion concer sss fing abut heft sk, clasifisaon ‘But weocesinal sy someting abo the onda estationIn eter ease, se are tte irae fo eating objet or estimating vale, ven cert Mor peal, we ae intestate earings for clsifaoho estiaton "We das sone conte examples fare below. Fr now, think abate Jing ecognne hain sharaters faces o ber jee from visual tn (f, thik abou the protien of coping spoken wend. Whe hema ae ety good ates ges of Ciena amy ntl eating, {tig dealt design automated gots foros ak wh efmance nd obsess aire nea hse of buna vente oe hn nlf etry of efor in els suchas elect engine ing, atiemats, computer scene, sas, philosophy, and ogni sees, tun cansfil fr ouefon the bn machine lesan gett have ve ee declpe. That i cnorows popes has ben madi ein ery ‘Bema cag att orig Ty, os Et, ‘Sera ier a ‘SUB Sey an isa 1b na Wey Sn,2 TRODUCTION slewing, and spplcatons, Rees inti re rs despa practical and ae ‘eleram toa range of ccpins such s tose we have tuned above. Many A he ha det acsearle tro ues, Howevey, ns ae ‘his mater fa advanced Ive, regi ter echnical background and experi ‘Cura in his book w provide an scsi nrsctiont thi eld either 1 ints far hse wishing wo pure te subj so more dete fo those ‘kesring tread undestandg fhe si eas Feros fu book we oso ‘he probom af to-laspatem clasifetn. Ths ble aie in tanya ‘pleas and sfc cht exp many ofthe bee Inthe Al, Se Temoves Some uanecery complication. Althogh may import eps faring arene coveted by this del. we provide many god teers for ‘ore pt, genera, a ther mele. We he ths Bok wil sve at ‘ale ey post 12 WHY MACHINE LEARNING? Algor for eopszng pte woul be wsefl ina wile ange of probes ‘Tha aby i one spect of “arial nteligeos” But on might eon as try we med to densa metodo leering ood le for classic on, as oppo oj ing at wha 4 god ae ar given sptoton td impemeatng "The main aso is bt in many applicatos, he only way we cand a good rue i toa dt ofan one Por cumple is very hd to dese exay What costes» fice ian imape, and there cand ce op ith ¢ ‘lasieaton reo decide whther cra = given imsge contains fas Bi, ‘Sve a good alos slgpeshn. we ait be able wo rset the agri with ‘muy expe of images of face and many exanpcs OF mgges wba fe, nd then lee alert cone up ih = good rl for ecopaing wth tt face reset. Thee ae eter hese of bing» eangsgoi s Welsch 5 fbwsiess wo err ia astumpons or modeling, eed eed for ‘rpc promaming bd alain fo caning conn genera fora eiaeteton problem, we nt nei 0 which of sexes “anogss the objet belongs onthe Ras some mearremens of te abet Teor a good le, we we das coms of many examples of bets wth i ‘rect chsifcato, The fling gusts inet ie 1. Wha do we mean by an bjs” and “easement” of teat? 2. de cusiteaton problem, what ae the caries 10 whieh we asin secs? 4. Inthe extimation poem, wha ar he valet me tg to inate? 4 How do we mesure the euity of lansfaton or eimation ni and ‘wha ete best le we oad ope fr? rr aa SOME APPLICATIONS 3 5, What ifomition ie waibie ta for ing? (6, How do we go shot ening god clasifeaon ora exinuton ne? We deride the answer to he in th qustons in hs cape. To ase the temininggistons some backgroud maton prays peovicd (Chops 2 nd 3. Wits ts Beckum, he anser fo te fourth gustion Gncused in Chaps 4 and. The aawer tthe ff geo fs esd in ‘Chap 6 The et of the Bok eevee to varius spt of nd approaches © thet qoesion 13 SOME APPLICATIONS Before dicing fetes dts ma be pu to have some ences examples in mind. Thre ase a wide rage of applies for eaning,clasiton, a ‘natin, Here we ection ot» fem 13 Image Recognition “Tet te any applets in which he obj to be casi ie Sita age ‘Tre "easement in this ese might dessibe the ua of each of he pil Inte ge In the ce of Hak and wht image the ines of each pil fever a one meinen fhe nage ae NN plas, hc he tal aber ff pics (and hence arene) dees fa cole una, each piel ‘abe conierel providing Unce measrenenscoesponing othe ees, ‘eth he clo compen ay RGH wah. Hence, fran 3 eal Image, here ue 3 menses. Depeting on he span, the ar any lasifieaion tasks based on sng ‘tes mestrmats: Face deteon orecopiton a common a seal pp ‘ston ethic, the “etoile” let be face ver fae proven re It be separ omgery fo exch perm in» date of nial ‘ferent appcton i eharacter secopaion Ia hs case, te Weng can tbe segmented in smaller ingest each conan single chart, and the ‘eres nigh coat of he 28 letersof he pet (2 eter i appr aad lower ce leer re dings the 10 ean posi some special ‘Saracen (id, qnton mar, comms one) In yet anche appctin, he ages ight be ffl parts andthe cae ston tak ie to dace weber he cumet pr deve ne 182. Speech Recoglon In aeech recog, we ae inte in rcogicing te words atered by a sealer. The measuremans In this aplcsion might be» w= of ranbers a represent the spssh sgn he signal peal seen into pons {hu contain det wards or phonemes fe each segment the speech ig4 INTRODUCTION be rps ina vasivy of way. Fr etal th signa can be represented by he mente fcr) in eet te ogy bands Aldough he dels fhe signal representa ae abide the scope ofthis ook, he signal can be ‘epeinted ltimay by 2 set of eal vals. Te smpet as, ecxegors igh te a spl ing wheter the ‘tern i" vrs no A igh ore completed ai might be odie ‘which oth 10 agi utr, Or tte mg ew etgey for cach word ur, {Tle diciomry of aeceple wads alte tsk ight e to dei which if ny, of ts lage ember of words Ps Bon ere 133, Medica Diagnosis In mac agoss, we ae intrest in whetber or nother is seas ese (Gnd which dita). There sn sepa category for each of he ies under ‘Sumer and one cegoy for i ee reno nee i resent "The mesraremot nh pcan recall thereto catia medial tess (eos pret temperature, and vous Noo ts) o mec! dag- ‘mss ich meal nage), pesenceldseceines of Vion salons, ‘nd ome ase ps informaton stow te patie ge, welt) ‘On the hao feral of he mearremens, we wr et die wah Geese GF ay is preset 134 Staical Arbitrage unc, ssc abage refers fo wwomsted tang Stes hat we op feully of & very ator term and imolve a lnge number of scales. I th Stalls, ne tie lo dein stag sii forest of sects 09 the ‘als ques sch as Mtl colons aang the lage sto secre, ce motemeas over eet te boos ad gael economia Ya tie. Tose cn be thought ofa the "menses ad he pein can Be ‘Start clasieton or eimai ptt th at clan te x ‘utes gt be buy “all” "do nthing”foreah eco, Ine eimai ‘se one Mg ny wo pede expected eum ofeach sect er same ure time horizon. Ih coe on pil ee see ss ofthe expected ‘et aes aingdaision (ey lle) 14 MEASUREMENTS, FEATURES, AND FEATURE VECTORS. ‘As we acme in Sections 1. an 1.3 in laying an best, we we observe oe about he objec in ore Yo make er decsn or exam, when mans wi clas as ebet they might lok a Be ojo pik apf ten 10 tok Or hey might woe seme ise 0 mesure thr pope the jet sch as ie, weight and temper SUPERVISED LEARNING s Simi, her designing 4 mative 10 atamatcaly cs (or lam © dus bet, we ssae hl he machine hs access 9 measurements a ‘a popertes ofthe objet, These mestrements come from senor ta apie ‘Sow seal varisler of iret, betes ofthe ojos For simpy, ahs book we mol each marmot (or fete) a ing ‘aptret bya sgl eal mambo Altegh in Some spins, cra ets fay not be vey aul reese by a nomber, his esumpon allows sss ‘on of the mos common lei technigucs that wre wef i be wos eee ‘pola. "Wessun ta alt relent ad ava apt of dhe: ts can be ‘aptred in sft numer of wetsrenenfeaues, These fine nimber of fens can be pu lout fom a fete vector. Sappse there ated far tes with he value of the fees ven by 1,004. Te etre veto Pe ana sot) Tis fete vs ct be agit of «po 8 Wer ia aimenonl space RY, wich we eal heer sce. Fah componet of the fentie voc idcating the vue of the coreponing ett the aloe tong presi meron af he fee space Teche cue of age recogion with mV» igs, te namber of ees 15 fora lok and white Emage and 39° fo acer mags In sch eegnion, the name of eres i ou the mabe of al ales ts epeset te speech segment tobe case |S THENEED FoR PROBABILITY 1 most splictins, th caegry ofthe objet is ot nigel ad efnively sxenine by the value te Fete wee. Ther are soe Tusaenal reson. foe ti Fi, itugh wo be ce fhe mesure Testes epee he ope of the objec inert forcast, hs i aly eo the ce. "The messred fenes might o eae some ingoran deals. This shold te ctearin the exams eh above Stcond, depending on ie apbeation andthe pci menses, he are ‘ales may bonny. Tats ee may hesome ines unc oars inthe Greed vale fhe fears to ht even th stn bjt igh ee {diffe vals on iflerst cron, Forde vets i ep ee tol from probability to formate the ote precy and gid the soln. In Capers 2 ad 3, we review some of {he bak tol fom poi tat me need for tae sof Be Bock 1.6 SUPERVISED LEARNING ‘Mr roving the necenry bkground roe probably a Caper 4 we for ln the ptersrcogiion rblem. Inthe ed and osal) cas, whee the de se ensue ie gw, te oon tothe cea thie dita ome a?‘ vrRODUCTION problem i wel known and aero fom setice Thi i dcased a Caper 5 weve, ne much more typical as in epplcatons th undying rob sSigy dtitetons ae oo Known. nhs casey We RY to overcome th ck of kavlege by rating 1 abd exalt me cas im Cher 6. Th Teamieg potion, ss fornia a Chap 6, js one pe of mle arin ublen nova by vas ems sucha ear om examples, experi ar Ina, site! poem clasfcaton, sata! pater recogiion an sisal learn. “The tem “sper” ang arcs fom he fac tat eles we ame tat we ave aces tae ppt abel by a “sper” or “ech” Ths commas With “unspersedIarnag” i which many examples of objects a ‘rail, bathe css to which the bjs belong ae unkown, The are alo ‘te formlatons of machine ering poles sch speed Hearing nl weitrcomont Jeng, well many oer aed problem in tate, pte scence, an ther Be. Bul thi ook oe foarenclsvely ont (eof super aro 1 SUMMARY a his chapter, we deste he general poles of lsicon an eimaticn sd dus sve conrte ed important pplicatos. We the ined the ‘erminology of feaues, fee vectra etre space. The need fo uc, ‘obit nd ring as desrbed "We have mentored th cation and ection, We fat mainly ox clase nit book, wih sme dacsion of extension oeximaion- Th the nest two chaps, ve review sme popes of pobay that a Jimocamt for aspects Gscused ate esto he book. Aer ta, we formal tie clsietion (or perm recopiton) problem and dias grea tes advan, 18 APPENDDX: INDUCTION “Th append athe ent of cach chap icy ics etn ide ees, perhaps of poopie natu. Ins book, we ae concermed prirly with inductive fearing eather thn dducve Jeng, Dedoctve ning consi in dtving tne cous foe ‘remics whose ath warner the ith ofthe cocason For exam, 30 igh Fear Gat the aes of parla is equal ots tas ines fs Right by docu ths fom what you aleaiy snow about ecagple and stout ho the ‘af paeora eld to te aes oa estangle with the ie at ed QuEsTIONS 7 eight You might theo Kant the 6 of rag i sto it hte ies a seg by ducing so he fac ha ay tangle exactly al of 2 cerain prety, Induce laraag coms in aching a conlson from evidence thet does pot gerne te tah of he cotciion. Fer exe, you might ner foe he ‘achat ral assests Bee dled before noon om Sass UP ‘st ow to th conten that mall wl be delivered before noon set Say. ‘his isan inductive inference, becuse te dln do not guarace he th of the conelson Somctim, the cnclason fan ndtv neces alse even, ‘tough he "pees of he infeeoe all to Plaga ln "the picnphal pele of suction” aa-how one can teed eleving inductive colo rom ee peice. Cay, is nt pam 0 rove dslucvely dat any such infucoveconcsion Is Wu is promi we, Soe rp indie nfereeces donot provide sock a guarantee. Even You ae ‘sii intel in shihing th yout mal wl be delved before oan et Suny, is compat with yor cine tht our mal nt tro re ‘on nett Sane leton tnt & spec cse of efecto, Tigh be sugges ht inducon has alot aay edo te consions inthe pasts it seseomble fo conceal slit alvaye fad ote feociaos inthe fare The objetion fo thi sypgeian i a hs scl ‘tong we reasoning tat nd i jutied modern seth ndction josie! ‘On the oer it pose oof noel jeifcaton of ada? ‘Wont any suc jostication take he fer «ded agua 30 so Teil emerge tt tai ering theory poids pai dtne mabe ail jtifentons fortran shod sven cn acmpions. 19 Questions 1. What is sft apace? What doth dimensions of ish space represen? ‘What a vectar? What a fete seca? 2. we want owe the vals of F sitet ates in ere te tasty bc, ster ach feature con have any of ifn vale, wa i the nto te fee pce? 23. Fora 12% 12 grpsae image 256 grayscale level), ow any dimensions te the for he eae Yee? How nny difleat pose fate ects Isclsicaton api cae of estan? What ifeences aes between ‘ype cases of elasicaton and ype! asso exile?5. Aout the pole of into (a) Whats he problem of indtion? (b) How does the reliably of ndetion compare with he ebiblty of de (6) How night sisi! ening ory sy someting a te iy | ‘ndcson? 1.10, REFERENCES Stic paren geo i lds a ce wo hf a nace, sous eile ebb adh ih gk ‘ach fran ar Sts ptr megs oa lea) ep of he ev ea of mucin eng and near Space ah oe mah ee ‘iy, min skal ens cpu sorte, span cee nomen andthe noon a od sev ais Mil (197 i ae ncn nasa machine ring grey ikon (aol) oeupto- de dissin te poi fc Tet erent ee at ery ss td aon ep Bone Panen rep Wisin (BC) Spt Bk 27, Deer PR, Ker Pose spon itch. Ealeoad Cas (Nh ret A 2 Desay, yet I, Lp A pcb any par spn Now Yak Senger ey oe ‘sa HO, HP Pen clas sn sn ni, New Yor Wi 17, ‘Dts RO, Ha Surk DG. Pty Sarin 2a New Yr Wie, 20 he nici ail ato eon Sn Depo cs, tn Sd ot. New ek Sng 208, YE. AO pen dio rine en al any, or SK Lage G, ni Leung pte aie A many IEEE Tae Int Tse 986 217-255 cal Mt ing Bono A): Nein 197 ‘ison NL Levi ies, Ne Yr McCraw "Wily on : ‘Tea, KoarobesK Pate ego. Anda Accs “om ‘Theda 5, Pris A, Kosrnns K Coes Dew per nop ib ech Amhara Pe 0. REFERENCES. ’ on VAC The sate tial aig hey, New Yr: Sing 19 ‘tes Te re finds n Tae Stee Eneyap of Fit: 2010, pin sedate len. ‘Ytmte MS. Koowing nd esi New Yo Wey: 188CHAPTER 2 Probability In this ad th net chapter, we exlin some ofthe clemenary mathematics of proba. This rode the maleate Tondo or ein teeny ‘nd fos the bss or tsa! eaming hor. In paral, we = nts In eaning wea thre i uncer in the waedyng bjs eat vec), “the abel ning the leo which the objets ele), ad the oh Tewecn the cs ofthe object and he fete vector, Tis uncetasy Wi todd rab, 1 ths caper, we expla some of the bases of dice probaly. Tha Js, we fas One esse with « Oe cube of outcomes of res. Ite oat chaper, we bey uss he comtnns case, in whi robabiy dws ‘ecoes important In the appendix we discuss sme possible eres of bai, 21. PROBABILITY OF SOME BASIC EVENTS ‘Suppose iat we have an ou bo Sontining balls of ais clr. ball i ‘ade drawn fren he bow clo tea hen i place ack tbe {ow The potbiy of going sb parca oer the fon ft tale inthe owt tae Ba a "Toe ssunpon tat al du mudomiy means th te potty of eoving any pata al og othe rab of eovig ny ter Ral ‘Toa isan ssanpon soot coe or of eae. We do nt ese he sete plication ofthe pipe of inference, which says ‘hati ll we knw is {Gat bow ae Wit posses thn tbe pablty of any one age {he proba of ay a. One prblen wih tis pce tat er an ‘Steret wage of seaming pout. For xanple i we tuto co, sed ‘we sy ae af thos poses te Beds, to als, and oe of ec? OF Oro ‘a et rig hey. Fe ‘Sen ene ne 201 by eb ley ne PROBABILITY OF SOME BASIC EVENTS u sl: he coin ae same vers dif? Or four posits: Mee, ena aes and itll ‘Store ta tere ae Nfl ne bow, of them ae Bic, W oft ao “aes hat noe is Bl sk ad we (For exemple, ne fhe als as ‘ack and white spon) Then te grobabity of geting = ack bal BA ad the prota of gting white als WI ‘Notation: Let sy dene the calor of te mh al avn. So, a | : reas hatte ft all awn i ack andr is white” means at the seer {ha rae i white, Foran event 3, PS) deo te probaiy ofS, So, Pty [wit dents te prota tha the mal wie “hen we Have PCs, lick) = 0/2 aod Psy whe) = WN. ‘We ae ing tat ofall i th ck and wt, we ae asuming a ‘nis iposuble dat the ih ball don is bod Back and whe. In ou aoton, ‘ne we sung tht he owing Imps: (yi acy) ‘Gone te pobbilty hat the nh all de ish Dac and white, Gees Buck whe). Since Done of he Bal is. Bosh ck and te he ftin of the Re stare bth Sack and wii . So, we Bae iG is bck i wie) =. "pura: The pray ofan imposible event i 0 {isc at als are bh ack ad white, Wha the probity tater snk alr hie al ed? That pot the fan of als hat “ev cter Black or white. Tht faction ithe som of the faction of als ae Wisc pis the faoon of tle at are wie, B/N FW. = (+ WYN. So, ing 7" mca" Pele Bac) (x wit) = PE i bck) + Pi wie. In pou: The probably that oe of eer ncompabeeent wile it ‘he on of i proabiies ‘Sopp every ball sehr back o white. Then the facto of als a ar cite ck or wie 0 = PAG i ack wit) Ia gear An vent hat garand cc has the pebadiy J | ain tha the nth ll dow il be et lack or ot lick Let ws we tig lack 9 sun hat he al ra in he th swing mo Bak. Then weve Ge is ck) Vr sk) Since a all cannot be oth ck and no Mack, we ave Plas lak) + Petes eas)2 PROBABILITY Stating the fs tm rom oth des te equation is Py bb) = Play i th, hsm The bie ee ins th proliant 9) Pe. 22 PROBABILITIES OF COMPOUND EVENTS Suppose tt we draw vo tal, aershe it, pting back be a al a troy mating he als efor ring to std. Ti lle "aig vith rpm) What ie potblty of peting a Black bal es ns And wie hall be cond ine? Weve clue bow many posits ete ain al and bow any of thos pots a es swith fit sack an the oor wae ‘Since we areca et any of te als te ot tne snd abo equal ely gt my of be Hale mt bol he Sod ne thea > ogy ily combats in al How many ef hee we ees tn ih esa ck so the secon a wt? We know tay of i 2 ck tal sou tte he ime. Fr cach of se ay nyo Be ite ls cul be rn the seco te. Shwe eB > Wag a Wi (we might daw sack alist and we bal second. a Wan wit) = SM Node at Bw (ay (w mw GW) (x)= 1s sera she proaiy of ser independ evn ie prot of i ‘nabs, ve inp cnr one beso aft he reba theater nth et roa feta 2 tal the second dn nots yet we fs sk a oe {het rw ne a ving wih open) ‘Wit the ea of etn xy Bc al Ft i eds (wi eleceen? The ae Wx Noe No N= 9 posiir Thre eB Bx Wx Wx W = BOWS way opting to lk als allowed by ies wie bas Tht te mn mmer f ef et any we eects eter it at a lan star, whe Wisi) Te we 54 ~ 10 sich wgrncepars $e pea a {ting eal to ck bale the Rafe rams (wih eplcean co lay tac) Pls wi), ‘CONDITIONAL PROBABILITY 3 ‘We cam me his lls If thr a fe posse postions fore mo lc bls thn thre are ve coi for the re Huck all And ons ecb ie ston fa ack bal the tee se only fow pele pono fo the ‘feond ak all Tus, tee ar 5 #—20 posible placement fo Blackball ‘and ick tll 2 But is cout the poston of the hack balls wie. That {stosay, the ptr white bskiteonbie Mack could be abtind with ak tall 1m poson 2 ard Wak al 2 in poston 5, o wits lak Yall 1 In psiton nd bic hl 2 in poiton 2 ther pony woul pine the patter hit: nk-wite-nie Hac. Th ok cat of he ale cortig, we Gi te conn: by 2. Ts gener, he number way of sting rts rom ase of et ie ner HH en a iam tn — Don 2) 0) a ad tei” Toe ithe renoing st ha a te special ae we cued shove. I arial if we se ios met fos hae encod or Be int tm, m1 ies oe scone, soon, with =r + ees ‘tetera em egies nn Denar tore d= But as before, we ned ak care of maple couning Now, wth es stead i dbs couing, we hve ted each angen! tesa 0 ed 92 Side he count by whic sve Eguion 21). 23, CONDITIONAL PRORABILITY Suppose tht Gv tls ae dew wt eplcement What she probabil hat exact Mack all re ra given Da rei at est pe Back Ba? We ‘wt thie cones! probably a P(A. whee A iste even of daving ‘racy to lak alan isthe event of dang a et one Bick al "To comput his comtonal proba, we coapare he mb of poses in whic at enon ick aldo Wi he namber of posites in which "Te mmber of posabites in which tes ope ack bl i awn at be ‘quo he tl mute of posites mais i mumbo of posts ie wach ‘aly white alse dav, Since tere are W ways to da only whe bal. the ne N° — Woy to gt test one Hak all. "We aady cele tat ce me TOW ways to daw exc to Blk ae whan ve ala drown wit elseeat So, the condone roby7 PROBABILITY of deaving eaty te tak aly ven thi I oe ck hal deen ot ve cnn it element 1ostws Stppoe We als st Back o white and aso plaice sls. So thre ae fo ‘egos, lack and pac lak a ls, wt ad platy aed white nd ‘ls Soppoe we know te aumber fl ofc of hes caepe, NUE), ‘NUMG), NUMIWP) and NUMWG). From Bt infrmstion we a ae eat how many bsnl thee are, bow aay se ac, hw many ew bom ‘may ae gs and how many are psi A bal andl seed Geel ‘ta infomain, what che endtonal probaly PCG) ea al lack, svc th sgl? I a be the faction of he ae bas tat ae lk ‘er wey um) O10 Naas 1fwe ave the neat nthe deamination byte member ‘tbat Ns we et hich ete se Peso, PCG) ‘Ase gem ral, tong as PUD 0, we ave pean = ARB) PB) ‘Tis at eqaton sometimes akn to be he dition of contin pobiy eat fe the ein which PCB) 0 24 DRAWING WITHOUT REPLACEMENT. ‘Sopp tht we done ps back the tls fer hey re ra nse, we cote ‘andy fom he all renin In he bol. Then be probabies change ae ‘sch ding as is whitey is wie) = [RANDOM VARIABLES 1s ws Pais wei al) 5, Wat ste bait of ering four vite al ia sow, deawing wien sepacoent? ASsueng Wat thee a 2 Hens our Whe bal ey W 2 8), We te 25. A CLASSIC BIKTHDAY PROBLEM ‘Wat tbe pobbiy uti a oom with 23 people cus at random, at east two wil have te same fthy? Assume thal no oe ine room i bam February 29a tat person is aly iy to be brn on al te oe es. "Te ier ie minus the potty that 99 two of them have th ne tidy. “Sse ht the people inthe rom canbe ordered in someway an conse the probly ht any ven pon des oc ave te same Bihay 48 any of Ihe eater people inthe ove "Tare ae 364 pou das fo the second pron’ Hine if et be iret tom the fist pesos tray, othe roby ta the second does het hve he sue thy os ests 36/868, By te sameresoiag, suing tat he iets donot ave he sn thy, he pea ha he hid dees tot hve th samira si of thm 363/368. noe of th fist W 1 opi ve the same ay the he bi tha he Wh doesnt have the ‘me bithdy sary previo persone (6 — N)/34S So the probly htm woo he 2 have the sae iy it | $6 3 x ayy he symbol means “i pproximly equal". The, te pbb ata Teast two ve te se bitay © 0 226 RANDOM VARIABLES: ‘athe previous examples (he fous of his cape) these ite rb of ‘zones of abdom exper Te etomea tom xperinent al ‘random vriabes Ofen «endo vrable Ie Seed Wy X. wie the capi«Teer sed to dings som vale fromthe lower ise, which i

0. Then, © and a ae wo stint pois in) hen Payee) — 2p ahewan Ih yas ae distin porns ia (0.1), tet POP a) =p. But te 2 ge ‘soca ¥, weave Np> 1, whe pobbites rue ob bee and «Seth eotbliy of slong any putea unter sO yeti pb of -Slesing some auber in he ow consider he ee mht with a robaty of 13, we ge so mer tesesn Os 1/2 with all te number the teva [0.2] Ping ly Hes, ‘Wit proba of 2, we gets number cen 1 a wih i ames in the ftv (2 also ng egal ely Using an arnt similar abo, he obey oF chsring ny paar ‘mbt 0S in wht sens atthe uber in th eval (72 1] es ly 1 Shose in he art [0/2 To answe hs queso it takes moe ete ‘oes he prot of ss of pass rae fn the posaly of scone Inv! pit One a od ihe sto beens uncon Wich we sate in te ex section 83 DENSITY FUNCTIONS, ‘A very ef yt wok wth probabil inte conosco we dey {fesctons.& Senco px) tat ivaen net sa with wea tice ps) eee ‘1 valid density fneton. Suity speaking, we nes ps) to sas tbr ‘ue enc eondon called nearby, Webel seas the son ot estab ments anon ne ppb dled ction 's beyond the sap of his Bok. We assume hn lhe estos an et ne da ithe measurable, Givens densty function ps), the probe ofan itv A ingen by te swe ener he fenton 7s) overeat A, Ts apis mae cel sae 5 well. Tht i ay, for a gener (neal) at A, thoy of ted A) is jst eae ade a) ove le eau, ise te gr 32, Doi ni “ategrl of pc) ove A ad ie writen as [oe ow Pay “ised in Fg 3.2 "sate wo sn ea ronan) to bea val dey function ae PO)20 falls 02 ad [rms - 6D “The Sat condtonpaatas hat PA) =O fray St A he sod codon susie a the probity fhe wl eal line th bay cling Some real nub) Thee conon toga sre ht or st, we fave Oz PCA) = 1 woud fr te proty oft ‘Retin ote exaples dice Secon 32, cor the cae where 8 ‘users sees om thee 01] with ll be erbor Sng geal Hy. ‘What his eas is ta gen any sal inevl coined (0) he pobbiity of the tava te sane prs ofthe eat poo, Ts ci fe epee% PROBABILITY DENSITIES ne scood ample with «probaly of 1/3 we gta ouner betneen 0 12a witha probably of 28 we aes ber between td Ta cach al nr, che wumbers are eqoly key. The can be repose by te ses ancon Bi osesia moa fais 12<221 Jo rere. “These wo dnt nto ae shown in Fg 33, Not th th aloe of th sty fiction atthe pots of sony ate ne ipetn,That frp), te cou uve taken (0) = p(t) =O, nd Hkowite fr p(n). And we cd have taken po (/2)='4/3.Actly, ven ser sles wold nx hav changed hing. The ale of density at ny Bie nombr of on sot trp ce is doesnot eange the intl fren under 9) Ne any St ao es ok | ehnge te prota Lhe ae) dee oot epee th baby of the pin AS we have aealy meetin, i the commons cae the estab of ny nial pst is grarly zr, wile for rany ois we may have pix) >. Alsons) “ton as ers ‘ception ofp), sgh we can thnk ofthe “probably ma” as eine spend ot ove the ral ine Wil ps) sepeseating ow cence ‘musi at. More peiey, et B) dente te ine eet). AD ‘saroritoo Wo the deny ais pnea by PCB )}/Qe) At we Ie cet ‘a, we et et approximations op) In hs a we ge sin PB) te pu) 69 min pen M33. Mo ity pay, JOINT AND CONDITIONAL DENSITIES ” ‘4 PRODABILITY DENSITIES IN HIGHER DIMENSIONS Retnig tthe mensional ae of Ais soi a sgt 7 denote a ‘wobbly density ncn hat capers where on he ret A's sha ly {ofall AS in dimension, p) needs sat he covatons pe) = 0. fora z 65 [oan os ‘Given dnsiyfunson CF), the probaly PS) hat Ace's shot fin erin ea S on he re given by = fom en ‘The ntpeation ofp) a density ls iar tthe I-tensona case ‘That ty fc point Xo he ge ote BE, «) denote alo eds « with Ft is cent, We lel the probly PCB) chat Ales sot wl [omewaere in th ul and thn vid tht probaly by he se fhe tall, Swhich eg by ae Ee) =e Tan at one meson ce obama be pit equals ny PtBE.0) haeiaa 09 Likewise, we an onsier probability desi fain ia RE wel The evel ‘pent is compet amlogea 1 the ?mensonl ase In pata, Egstons (G.9.3, (7, and Bho withthe onchange being ta? irelac with [and hat he ea ofthe BL Be eplaed wh the vole of be all co) is eee oreo rere meee elon mi, wry Ly natans miata ain tee era Smee la pee® beats on = [frouina as 8 we won expe.