Chapter 2 One-Dimensional Optimization 2.1 Introduction In this chapter we shall discuss three distinct approaches of generating approximation schemes {r,}, convergent to a local minimizer of the function J: [a,U] — R. At the outset, we shall assume that f is continuous and unimodal on [a,b] - it has only one local extremizer. Further, the initial interval [a,0] is known. The function f will be referred to as the objective function. The three approaches to locate the local minimizer are * Function Comparison Methods * Polynomial Interpolation Methods © Iterative Methods 2.1.1 Function Comparison Methods Recall that the objective function under consideration is continuous and unimodal. We evaluate the function value at two intermediate points 2, and aq such that a < x; < 2 < b. Based on the relative magnitude of f(x) and J(z2), we can reduce the interval of uncertainty — the interval that contains the S(x1) < f(x2): Let the vertical lines in Fig. 2.1.1 denote the function values al those points. Because of the assumption of unimodality, the minimum should lie to the left of x2 in part (a) of this figure: it will lie either in the interval [a,x] or in [1,22]. It cannot lie in the interval [22,8] which, as a result, can be eliminated from future searches. If we can know this situation @ priori, we would make the length of the interval (x2, }] as large as possible.sles Se Sar) > Shea): Docume of the aes jm ‘tui fe to the right fs and ane te taterval fey] ex be ein re seateli,se () tm ig, 21 S21) = sles): Vass of the assumption of animodaity, the winiom SENT ein eyrash the itera lesz and [eb] ea be linn, se (2) i Fig: 201. Hoe Hs covered sn the other too position f(z) © fl2a), Ses) < fis) e mom e (a) sezs)< sas) ves the right interval tobe elite e momo (0) Flay) > Ses) results in the It interval tobe lina e a) (6) los) = fen) resent ight and the eft interval tobe eliminated Figure 21.1: Reglons tobe eliminated under diferent conditions order to reduce the interval of uncertainty; we wou ike to maximize che eng of interval fo] im () fo] in (8) The only way’ to masiise ‘both of them ie to1make them equal That Is, 2, and 2) should be placed One-Dimensional Optimization 5 symmetialy Inthe iter [ab] such that aan au Here represents the length ofthe interval fa 2] We wll explore different way arian and xy sai te above condition. In the following discus, She aon deat wil eer to the proces of ecocting new pons, nding the {lnclton values at thee pointe nnd bimimeting a part ofthe region, The efiacy tach method is dood «+ Number of times the funtion value sto be evaluated at evry iteration 1 Sie ofthe terval elininated as a feation ofthe starting interval ize “Theve aro many ways of chasing #1 and zy, satoying Ba. (211). These wil te discuad i dealin See. 22 2.1.2 Polynomial Interpolation Methods We interpolate the objective fanction J on (a. bya polynomial that has the tine Eretion alec or deritive vale of a mambo of points on oy "The predicted minimizer s then usd to replace on of the provious pints 0 obtain anew st of points inerpolting the objective funtion J ove stl ote which atl brackets the mini ‘Ae we need deal wes interpolates, Hie Hustructive to havo & rit Ascussion on the tieory of polynomial interpolation. Lat zy)--- 39 be (WT) distinct pots on fo) We shall ny that a polyol po degree m interpolates f(z) at 20 <4 1 with Some numbers in the Fibonacel sone ae gen Table 223,Opsiniation Dubie 22.8 Siene tamer boas seguense ] Figwe 22.7 Fiboosec earch with uncertainty terval = La et the search interval [0] be of length Li. Choose 24 sudh that each poist Fussy, 28) ig 227. Note that the condition tn éctance from the ead a shown in Fi Bp it) casted bp this choice. With this la saint dntcrya ater ome teation = Ly igune 22.8 The length of ue wing Bq, (227). After the fiat ‘Sminated rom the sarc rion. In ether ence, the new interval of wc Bef tength Ls, which trom Eq, (229) becomes t= Bt ‘Stoation in (0) of Fig. 21-1. Ax ately [eb be ‘certain: Examine what happen fw ib place te 51> 25 symmetiealy lide the interval [2ys8] at datas off pve by Fret! ean) from either end, se ig. 228. The claim ts that 1 coinciden with 2, Thc rsOptimi hacked by dtet computation. The leg of interval a2] is given by ~ Pty Bete, ‘ Fea, , Bo 2 = Ft BL, wing Bn 2210) ing Ea. (22.7) (222) Ba. (22-10). This sequal tothe eng ofthe itera f,23-I fllows Uo coincides with. Thus there ie ted to evaluate /(cs) a0 9) dy avalable. The fanetion value Ha to be evluated oly a ove pot, inthis caro 2. The Interval in which the sare has fob done the eration i ive by e243) the length of interval containing the mL, aod the length ef the interior pint Gom the ed poi, f ate tye Fine hn au) Fae? * Fv ear Iegth ofthe ltr conning the imam hs been rece to th mile rae of and the Heras an be stoped. The Uy, 1 Heaton ae sed, The Bi artnet fonction ian he eon cc Ta the ttl aber of ancton ton. conc metod i NDepend othe dae cacti te trea Sng the unity, the mae ef rsa has tobe eealoed point tha il remain tobe eed. In (N'—1}th the intron pints areata distance offs gien by Anat Rumi (One-Dimensional Optimization 6 ‘Rigorithon 32.3 (Fibonacel Searcy Isippata, 28 ahs 1 2 Choose N suck tat Fy > Fi Kesh fe= Han) f foringtoW Ide Fea, tym PEO, ith s frten a te be enayeays sot Belly h — Fatt = fa “This imps tha in he last terton, the nein ve atc al te ‘Tone om thc enpeintiing tht they etn The poe nly Secnus wo wan theta pote be dies fom ech others male booting te taerr pens tbe as dita of (=) hom nc eter ‘Th oy eclarecan be sua Alga 223 2.2.9 Solve the problem presented in Example 224 by Fibonacl meio. “The Flbonacsalariton giver Algurithn 223i applied. Table 224 od Fig 22.10 show de consgence of he eth to the lation To Feduce the lngih of uneriiny to O0OL Urn the tang Heth we teed Fy > 1000, Frm Table 224 we code hat = Ws We tas need 10 erations is Optimization ne Dimenizst gion = ® 4 af rly rh i a a m te 6 4 Gn Tigre 2218 mci sc. Gomrer a teri br ini a ; EEE TT REA Res Set G s whnain e "ana ereaer erin) E = cs Ter Te T 7 z pas ——~ozza01 “2 ra 3 ozo 0 se07e 3 dawor 3) sors a ssao0e 2 aawor 270m = “ojua0s— —oseore 5 bale amuse osm “oseean. & aera arise “oaea) —o.reont {ages 2am © Cossig— -oasiaut saree avers asusee2 © oasis 8 assis nanere — Osbusle 0.84883 to 2eroug 267s oases 08748 25rd Tageten — oseiess 1 broet Toasts osteo 0 2enaT Coasiena — ossiso0 Mo 2st Coasiese — osie09 1s 2srera “oasis “0389 2.2.8 Golden Section Search, 1 Fibonace method, one pos the inlerie pols with diferent ration ofthe tou dotance at every Heaton, For exe, in Wersion Ty the poit #2 fs pact at E#=¥ faction ofthe total distance and inthe wecond Herston, Ee FA action oft In gon ection appro his atl remins the mame for every se, Sin ‘istic uf Fy. (ry © 1) hon the ould wait 7 tobe at distance of) Ieee asa ftir cae ‘Ths we et ity met igs 2°21 Sogo ints fl eat Wr try mn he hat itt ily.aeet'y, 2218 mays (t 2 artim hom the shore equations, we ge “sloth in cra vote an Ana tie tod bese Gehan a iterations, this lengli is (I~ r)! ini Prot oot Wea dow Example 2.24 Save the quem guesente sn Enanple (Gage 87) Uy the wet ot pen metion ware mth the ft certainty of tu teOptimization aa Ga dat eae ros bain ¥— riba) 6 fi = Seen. f= Ses) 7 while > edo Sith sf then Bemis metro h fm ayn zac abo rb—a). f 2 endif IS end while Slea) = flea) then b Lone ten those of the starting ough, 0001 (e210) wen ea) ‘We conclude that we need 21 iertions. As we hve to erluate the Tucton tie in the fe eration ad one each in the rest we ed 22 function evalvations: ‘The converges of the iarates tote rin ‘Nee fom the valves reported in"Table 225 and in Fig 2212. or more on algeithiic approach to eptimizaion, refer to Polak [5 2.6 Case Study Revisited ow toe the eave stay an sercondactr manulctring. presented ia "22 Gpmge $5). Reel ht the faci f-repesting Ue total number Food IC produced er ovr sen by 1 = 2= H+ 5200 eaid ered Optimizat oo Tible 2.2.5 Coben Section Sean Algorithm - Convergence of erates for nization of fla} = (= Wee Miz 3) i 3) E a rm ey re T 7 hse — ase 2 ure 3. 0mm aaanrTT 3 2p 3 Coser aasnoes 4 2236 ans osunes — “ossorr7 5 iets 2e1s asso “os © saras Simei osseats — “o.s7eoat 7 Bears aame ogee “oso 3 2szres . bance namo “0.seun17 8 seu | 249609 assum —o;adss Wo sro 238600 Moseusse © —o.saro Haran ste Loastaon— o.sse884 i2 Boros asinua oasis “0 aso 13 bore 2ssoce “osama “0.84800 1 bereen2suoce “Osseo “0 seas00 1s bomen aero © -oaseo” “0 aso 1 asreea borer “ose osean. ito asiniy asreor “oa ase ie 25rnr 2st Coss ose is asi? ase “bad oa 257m ass osm © _0asao 2 257mg ase osm © —0aea9 25rma ase “ose “ose Figuse 22.12 Golan Section Seve Alperthn - Convergence of erates for tization of (2) = f= 1) aM 3) inf) where & represents the thickness of the wafer i micron mites. We need To"thexiiz this Fanclion Ar meaioned cli, we solve the equivalentional Optienization for Semiconductor Case Stuy ‘ubie 226, Two Feit Hg Heal Sor Semator Cas Stay stable 2.27; Haeton Sear T US ain} Pte pst stunt 7 Tie. Us ae 208 Tg “amas ‘ ed ies 3k ts Tasso : ‘ecee ae 113 Cioae sane “om? $1 fore —abone2 anor. ms 6 1S Nyooas —aaTa4 974.16 Vass 1st Uma “terest “aprast Leen Tear 2 130008 [Soses caprasr —rasr ss ora28 3 MSs Tomes cree | ~aoree tase Cam Cama 10 ees gms maorde | abrae Host Cistta coats “prs hae “erie nie Cieasr ister nae Cigrst eer Sener “brs a une “Set 5 lass wore “iors 2 be “rn stats cima “brie etn “irish 1Saus wis “araet ed Cine! “arse te Tate ee ed Siri neat u ideae ret : oe Teresi “area, te “rte a eer Sinst “rte = ae eo SSael Care ‘Sse 3 Gen Pore 2 Bee Tees Tava ee rough i 5H A 0.0025 m2 ‘The new point 3, which is to replace one of the three original pots a, b 4 Sieh born ‘plot of Jz) = P(e) eres the deviation that it produces ix shows so Pig 23:1 2.3.2 Quadratic Interpolation ‘Wo berin with iterpolting pots a ‘tsdratic p(s) interpolating f(z 0, given by with fo0] athe initial interval of Uncertainty and ara inert point fo. By Ba. (216) (Page 82), the 2) 4 He 4 flat — 0) 2) + flea) + flo) ‘The objective functon f Is evated nt $* and one ofthe orginal winks is TRplace depeoing om both = relive function values ad also the position of BS, four dotnet eases are tte qrepbicllyn Pg 2.13 02.18, ‘Til inmadistely goes we Alpin 231 "Wer now show that the order of convergence for quadtieiterpulathon steps 13 Using Newton’ or the quadratic potyoeia eeplting, Ble) at sasen-y 9 aya (eler By (211) ge 82) ben by ale) = Fea) + Seay z4cale = 24) + Deuszu-suealls~ aa) ~ 1.4) “Theses ofthe derivative ofthis polyoma gn (23s) De pilensad = Sensual + Slseztarozt-alaues 4 a4]b Smooth 1009 wal me) , v ips 29.18: Deformation of a ent de loading J teticiently dieeninble, He whi convex combination of 420-1 fet sc ach Ut prasnals Llane terme ase ‘he teal pot af Je) ves wee ats inner of em fet Bact) 24a sae zedfa se) oy Figure 23.16: Case 2:5 4s fre > fs Ones sislousl Optimization n Ke Figure 23:18: Cue 4: > eS < fn ‘Rigorthim 23:1 (Quadratic Toterpolation Algorith) =10. fe 5 faut Paes td evne ob) py «ena [Rese eae ete ese We i then oe Ih otteife! Sean fy fe then wena doka he en keBL = stec 5 gemma ncn se 0 Fey = sensilla ened) on LO eae mans) ae =) 7s Bi +62 = ale = al 0.2} | That is -o.4 | fe tnterpolation Algorithm - Convergence of teats tha} om fev Pen 424-9) Fee 23:19 Quad intiaton of fete 22-3) + Deere seat berate Bxample 2.3.1. Sslve the prob (ones 7) Uy qundraic interpolation method. ‘The quadrai Interpolation algorihoe given in Agorithan 23.1 {e'loeat the minima at 257795 seth a accuracy of 001%. The _Sutarctng Be. (2:6) fron Ba. (23.7) wo gt = 6g" — ee Mle texas tease stun o¢ eral require es th ‘ise Tea) nan Fir 319) tant alles 4) tse wee 2.3.8 Cubic Interpolation beg with tt Intro estat Jairo J | 4 ta Sting that bc ftv a i erent ie 4 ad bom Ba. (219) (page 59), we mt Un cae put) ene Sahat bien by ee RESTS Sates rng ror ea vd by Pale) = Fe} + Phe) + Flevn die ol? : etal eters 239 4 Tesh tie a}"e 0) i Lee age. We et ‘sh the estimated new pin 3° gic | “This dteence uation bs nsaton of he fon y= oh wha + sin we reas) —shd= 109 Borate 39) Sacra, nl be ean at 3° ad mo he oe eae ern oh a tacos sene Dimension San Eablet - Co : a Er ee ne ae re ee ee SES eb me aes SHSe iam pct as 3 feces goes fees ume we ame came i Miueiahat Bes cme: ime = 8 aot eaaee eee | tcl ener eens clan 2573s 267m 297795 0380 + £0) 0 0 f(t") > S10) + Me) > Oor 4G") > O and 58") < 0) This gives Algorithm 232. In Fig. 23:20, 2.21 and 23:22, thre dierent ‘examples are iven for the oeéarence of mininnin in cubic interpolation ‘ethno Example 2.8.2 Solve the problem presented in Beanple 22.1 (age 87) by cubic iterpolation method ‘he cubic iterpolation algorithm given i Algorithm 2.3.2 is applied to Uecat the mininum at 2577 with em secrecy of OOO1%E. The number ‘trations required ls 3.‘ rents are reported in Table 23.11 nd tn Fy. 23.9, he mlnimam stained at 257735 : ‘As we have dove fora quadratic interpolation, ane ean show thet the order of ‘amergence for cubic interplation algorithm fs 2 Optimisation me) Fique 2320: Beample of ocitene of minimum in xb interpolation fee) Figure 2.21: Example of ecartence of minimann in cubic tery 23.4 Case Study Revisited We now salve the probe of be lauding, pret bn See 2.4 (pane 10) ‘The load funtion under consideration is ven Uy baer. sires i Raa ERIEJe Inerplation, Algo: ) Convergence of Hert se emayge = 2yee= 3) Im 3 Figure 23:23 Cub rstmiaton £2) : ‘Table 23.11: Cuble Interpolation, Algoritinn:, Comwermenes “of itevates for TAS dation ofa) = mee ~ 212 ~ 9) 8 HS) = he 3 i 2 as 3 0385 0 Lo 2 Sas 3 cass 0 210% 2 Figure 23.22: Example of occurrence of asim in abi itepolaton input a be 2 fon M0) b= 10) ‘1s maxima lad of 18,585 Newtons before the beat fis. deviation atthe {ine of failure ie O.tmor 10cm. 2.4 Iterative Methods 1 Case Study 3: Optimal Design of a Pi ae SEY a Jon = 86 ‘mimi power i delivered to Lat the pres etput actons I be deo ty Pe Then, an gee aa) — Te sean to id the maximum val ‘cout me misimee J = Teries to the maser shone 23.1 and 2.32. vclade tht we ca ep chee = represents the depacemeot We TAG folion at winch the beam fll AS = SUS 23'2 and 23:15 hoe the comversence of cere uncon under coneraton by Alco eine minimum aff a2 13,888. THE we coos = Optimization! One Dimens a an a 3 0 0.125, 02 o =18249.1 -6801.5, x po ms » n 0 0.106073 0.106087 0 135848 -13584.8 Fesesieng, oe og tape siamese ies Fare Je $ Sam heme f cise pie Hn atin Ts te Te te a. ae ‘Re? + Im? i (24.4) To Sagat eee ae ee _ VRe* + Tn? Sa (ta ~by~ be)? +(b4ar + y~ tyr) (any 10.and b= 1, the objective function econ Le Ql -wn ae Otto y yey GasOptimization on 60 400) sghe a Pen) aod & nti suc e Opin sation =~ 2a a a Proof, Weta Ge") = 0. Sinte Gs) coins, har exists a aera Figure 2425: Function to be minnie In Piet Insound 2 such iat [O{a)| = Uff In view af Corley 211 (fags 8) fly tat the Harton sone {hie point wf. By Tao's ‘Thisis a wo dimeaslonal mininiztion probing x oof he parameters, "becomes a one eimensional initiation protiem. Suppose we Sea y= 8 = Gt") Clays) 27+ (16-52)? 4 eee) 4 Eee 2 = gan 4 272 “The graph of J given in Fig. 24.25, soerttntimnee AE Benin Manoa ane forte at his method, the minimisr is generat though th fixed point iteration seme for Un funtion o)n2-#2 Fe) hee ote) = £2). J) the fnetio {ond the ideal pine 2 of (2) aCe") aterto) ae) we assume that "(2 # 0, we have ote) 0M ate [As O"() le bounta (s ly 081, we hae fess Sele? tobe nimi. G°() sein "Pret #0 and geen ty Brample 2.41 Solve the bio sete (page 57) ly Newton's method, i ow) Ai akan ee Lites) ean 2.4.3-Secant Method { tes) AVéncaut mod appre ta dilive ofthe ne re SECO Eee snes | vio 3 te he a el ~~ PTS baat en cant aay vine gto) Bete im sec Te convene toa" Parther ths caneergene sql2 1 ae ca ‘ *opustsi onesie SecDimenional pinion ag ‘Rit 2414 Neve Agni Canepa tr REPT E teri Arai yt (214) ee 2) we ee gH Hann) ene > pe intl sien) 3 Ran Teche nate ay ; mes 3 Figure 24.26: Newton's Algo - Converge of arses fr minimization oF) = Me ae 3) a 3) om ead ca he tase oss te bei, some Game a8 convex combination of zua4-s an 2* So, we ge. ea) iis nets ‘We bove the owing thor spiro Then 2.4. ving coneresicn Cae oy Ba 48) tte Dasma © a 2). Te oer aca that p16 = nd ee or lag kw have sn BA Let J. > bly tC wth 2 a «cite i eee Meer El ig” Then tre at en intr fo coed = Yel = deel A lat the tarts (aa) ted as im Be Fak) ence to With = lee ee ioe eqatin Corerpnas of order oes See eee | eee iis fo 2 ne ca thee of Newton's etbod. FOr the robe io Se 5 ull ime am Serr Sy ce) : Bry a 24s ence or anes Ey interpolating g(2) bya linear plynomial at at he lrg rot of which 16.20 pimensional Optimization s £00 yey” of 6 ann 4 & until resale ue : eon -0.4 “Table 2.015: Sect Method = Comerenc af eats for mininistion of -0.d fey=(e Nie 23) igure 24.37; Secant Method - Convergence of iterates for minnizaton 7 z yee Re tal 2 feat asso ‘ible 24.16: Newton's Alri 3 ios “oar é on aie 2 291481 58.0741 é isms a8 1 2 Bar To388 [s_2snsi_ 58.0741 J Example 2.42. Sole the problem preened in Bxsiople 221 "Tble 24.27: Scant Algorithm (age 37) by secant mod, ‘Poe secant method given in Algorithm 242 it applied to leat the Tessa ac 2877 he numberof erations requted 7. "The res tre repoted in Table 2418 and i Pig. 2427 ' sc 773 — toa | 1 tea 2 asiaei ssora {3_zsus1_ssorat J 2.4.4 Case Study Revisited characterized by the folowing equations: “We now tove the cage stay on eet design, presented in See 241 (page 8). JEST ac faad n= 1b, the objective uneion to be minmind i ven by Je) = Mi = M8 +72 tables 24.16 and 2437 and Fig, 2438 and 24.29 give the minima of che ke aed lores Did sd 242. We nd that eaximum power soe Aiered when 2 = 282. In eter word, the optinal parameters arePianta i Optimization Fire 24.28; Conergcace of computed vale forthe ease sty though Figure 24.20: Convergence ofcompate values for theca study though cat mathed 2.5 Exercises 1. Let onal interval method be appli to obtain one ofthe ca maximises ofthe funeion f(z) = =e-* an [64 Obtain fret four interval of ‘enraity, bachting he Toe! mass Which masimam wil hi ‘netbod betcha and why? 2. Weis desired to obtain the maximum of the fanetion J(2) = 2943043043 fn [22] by equa interval lgthn. Find he bracketing Interval tbe (td of ath eration 3. Comput the ater of uncertainty afer the fourth eration to obtatn the toca inimie fhe function f(z) = =°—Se+1 on 0.2] by Pleonact and {plden section method. Ie eavued thatthe el neva of weertanty One-Dimensional Optimization 93 So gt 10° et the bisection method be applied to compute the minimizer of the SGneton f(=) == sing on [$n 4. Fla the bracketing interval tthe fd of 12th itraion. Which pint wlth bractating interval caput? Do the enna for finding the masimizer ofthe function f(a) = 2 1 oe on tnt. 5. Compare the interne of uncertainty ofthe function ) for obtaining the los! maxinier on [1,2] by Fibanael, golden section ‘and bisection methods atthe ed of the tid Keaton. Assume tat te fra terval of uncertainty 10 ~ nie 2Ke-9) ‘Obtain the local mininizer and masimizr ofthe function Me) =2 523 — 2004 on [3,0] and 0.3] by sing qodratic and cube intepoltin neo, 7. Use Newton and secant mnths for the above problem in the give tetera 8 Pind the minimum of the fnction Ba) = 065— 228, — gota ‘wing Newton and secant methods (Give » fed point eration acme to compute the extemizer of the function (a) sins on (m/e a/2) fie nce uc uae theo mee ens following functions. - : 1e)= 6" tsins-tonz +2 on Bea) =efine conse on se)=2+ Sle) = xn? 2+ inf onReferences 1M] 8.D. Conte an Ce Door. Blementry numeri te ‘porch: McGraw Hill Book Cu, Wek apie cs BB} 7B. agar ana DM tn RE ranean, Optinzation of Chemis! Processes Tl Book Cotapany. New York 108, 18] MC. Joa and RK, Bose © 4a and RK: Bowe, Some Fes in Honea action Anle M1 DA Pitre, Optimisation Theory wth Appiston dh Wie & Son 11 Fook. Optimization: Alors and ott presinations Singer (6) 5.5. Tao. _Brginering ptimiation, Uheory and prstice Thternaional Pet Tad 1088 mie (7) UB Seales Intraduction to nontnearoptincation Mecriton, 965. [8] S.M, See, editor. VEST Tecknlogy. McGraw Hil New Yok, 188. Chapter 3 Unconstrained Gradient Based Optimization Methods 3.1 Introduction In this chapter we ale concer wth the pole ofthe fog i je PEED En) Wie aaumod that By (2.1.1) fs 9 lca tsi thoughot ths chapter Chop Kate (2h coma BED a 4 ltt) wire 309 te step ho Bia hese TO sg al wanes,