yo122021 (Clase_01.jpynb - Calaboratory » Session 1: Principles of numerical mathematics In this first session we are going to explain the basis of the numerical mathematics. Pilars of a cathedral They are like the pilars of a cathedral, maybe they are not the most attractive part of it, but everything will collapse without them. > Well-posedness and condition number of a problem We are going to start defining a classification of the problems that we might find. Consider the following expression: F(x,d) =0 in which we call x the unknown, d data and Fis the relation between x and d. if F and d are known, finding z will be called the “direct problem’. If F and a: are known, finding d will be called the "inverse problem’. If x and d are known, finding F will be called the "identification problem’. In this course we will, mostly, study the direct problem. Example: Let us suppose we have this expression: Ad — 6 hitps:icolab research. google com/crivel16JOVigacg2tHPcp-IC)SOkeViSKAXkgMscroifo=eB9seIXYr2ou8priniMods=Irue 19yo122021 ‘Clase_01.jpynb - Calaboratory We can think about it as a linear system [5 AT) [ in which the unknown is the @ and then the direct problem will be solving the linear system, or as [El t}- [1 in which calculating the product would be the direct problem. a matrix product Sometimes you define what is direct and what is inverse, but often, the history of the problem will tell which is which. Now, we give a definition that tells us whether or not a problem is well-posed (Jacques Hadamard), that is, if it is a reasonable problem from the mathematical perspective Definition: We say that a problem is well-posed if it admits a unique solution « which depends with continuity on the data. Definition: We say that a problem is ill-posed if it is not well-posed. Definition: We say that x depends with continuity on the data if a little change éd in the data produces a small change in the solution dx. Mathematically: if F(d + 6d, a + da) = 0 then: Yn > 0,3K(m 4) : ||Sd\| <1 > ||da\) < K(n, d)|\ddl where K is a constant that depends on 7) and d. Example: The solution of the equation 3x + d = 0, where dis the data, is x = d/3 whichis a continous function on d and is unique, so the problem is well-posed. Example: Find the number of roots of the polynomial p(x) = 4 — (2a — 1)a? + a(a — 1) (a is the data of the problem). |s easy to check that we have four real roots if a > 1, two is a & (0, 1) and no real roots if a < 0. This is an ill posed problem because the solution does not depend continuously from the data. hitps:icolab research. google com/crivel16JOVigacg2tHPcp-IC)SOkeViSKAXkgMscroifo=eB9seIXYr2ou8priniMods=Irue 29yo122021 ‘Clase_01.jpynb - Calaboratory # roots Nn a Notice that the problem is not calculating the roots, but the number of roots as a function of the parameter a. Example: Another kind of example of an il: posed problem is given by the equation y’ = $y!/> with y(0) = 0. Since the solution is y(t) = +¢°/?, the solution is not unique (it could be plus t3/? or it could be minus €*/2). As uniqueness is required for well-posedness, the problem is ill- posed, ; [=] 2 [= ] 0 0.5 1 1.5 nN hitps:icolab research. google com/crivel16JOVigacg2tHPcp-IC)SOkeViSKAXkgMscroifo=eB9seIXYr2ou8priniMods=Irueyo122021 ‘Clase_01.jpynb - Calaboratory Most problems are not so clearly ill-posed as the previous examples, that is, there is a range of problems that even if they are well posed they will be incresingly hard to solve. To quantify how well or ill posed is a problem we define the condition numbers: Definition: Relative condition number K (a) = sup 2d iaeb léd/ Definition: Absolute condition number \ae| Kans(d) = su od) = su al Disa neighborhood of the origin that denotes the admissible perturbations of the data. Definition: We say a problem is “ill-conditioned” if Kis "big* where the definition of big depends on the problem. It is important to understand that the conditioning of a problem does not depend on the algorithm used to solve it. You can develop stable and unstable algorithms for well posed problems. The concept of stability for algorithms will be defined later on Having a "big" or even infinite condition number does not imply that the problem is ill-posed. Some ill-posed problems can be reformulated as an equivalent problem (that is, one that has the same solution) which are well-posed. If a problem admits a unique solution, then there exist a mapping G,, called the resolvent, between the data and the solutions sets such that: z = G(d), that is, F(G(d),d) =0 According to this, and assuming G is differentiable in d (G' (d) exist), the Taylor expansion of G is G(d + 6d) — G(d) = G'(d) id + o(||5d\|) for dd > 0 This let us redefine the condition numbers in terms of the resolvent G: lal G(d)|l Exercise: Prove that the expressions of the contition numbers using the resolvent are equivalent K(D) = ||G'(d)|| and Kats © ||G'(d)| to the original ones. Example of ill-conditioning: Algebraic second degree equation. We want to calculate the solutions of x? — 2px + 1 with p > 1. Obviously z.=pt/p 1 hitps:icolab research. google com/crivel16JOVigacg2tHPcp-IC)SOkeViSKAXkgMscroifo=eB9seIXYr2ou8priniMods=Irue a9so122001 Ctase_01 yn -Colaboratry We can formulate this problem as F(a, p) = x” — 2px + 1 where p is the data and x4 = (x,,a_) the solution. The resolvent G(p) = (p + \/p? — 1,p — y/p? — 1) andits derivative G'(p) = (1 + p//p? — 1,1 — p/p? — 1) Then: iid) _ (4p* = 2)/(p? - 1)” K(d) * ||6'(@)| =a le@i (4p? = 2) Kas(d) © ||G'(@)|| = v2—2 Vp-i If p >> 1 then the problem is well-conditioned (two distinct roots). If p = 1 (one double root), then G is not differentiable but in the limit p > 1~ the problem is ill conditioned as lim ||@'(P)|| =e. ipl = >— However, the problem is not il-posed. We can reformulate it as. F(x,t) =a? —((1+#)/t)x +1 witht = p+ \/p?— 1. Inthis case x, = tand a_ =1/t are the same fort = 1,and K(t) ~1VteR » Stability of numerical methods Let's assume the the problem F(x, d) = 0 is well-posed. Then, a numerical method to approximate its solution will consist, in general, of a sequence of approximate problems F,(tqsdq)=0 n>1 We would expect that «,, —> «. For that itis necessary that d,, —> d and that F,, approximates F’ when n eS Definition: We say that F;,(an,d,) = 0 is consistent if F (2,4) = Fa(a,d) ~ F(x,d) > 0 where «is the solution of F(x, d) = 0 for the datum d. Definition: We say that a method is strongly consistent if F,(z,d)=0 Wn. In some cases when iterative methods are used, we can write them as F(@n5%n-1)°°*y2n-qrdn) = 0 where aq, n—1,°**;2n_1 are given. In this case the property of strong consistency becomes F,(x,2,---,2,d)=0 Vn >q Examples: Fen) * Newton's method: x, = 2n 1 — FG 7 is strongly consistent hitps:icolab research. google com/crivel16JOVigacg2tHPcp-IC)SOkeViSKAXkgMscroifo=eB9seIXYr2ou8priniMods=Iruetet 0122021 Case_01jpyn -Colaboratory + Composite midpoint rule: fa =f? f(t)dt, 2, = HOR, (AGE) n> 1 with H = (b—a)/nandt, = a+ (k—1)H. This method to calculate the integral is consistent, but only strongly consistent if f is a piecewise linear polynomial In general, numerical methods obtained from the mathematical problem by truncation of limit operations (like integrals, derivatives, series,...) are not strongly consistent. Definition: We say that a numerical method F), (z¢,,, d,) = 0 is well-posed (or stable) if for any fixed n there exists a unique solution 2, corresponding to the datum d,,, that the computation of 2, as a function of d,, is unique, and that x, depends continuously on the data, i.e: Yn > 0,5Kn(m dn) = ||6dn|| <7 = ||dx — nl] < Kn(n, dn)|/bdn| We can also define: [sen l/l Ise K,(d,) = sup ————— and) Katsn(dn) = su; (0) = 5 Tad ton te) = MP Td and from these: K™" (dy) = lim, sup Kn(dn) and Ki"(dy) = Jim sup Kets (dn) T0 n>k TRI dk K num is the relative asymptotic condition number and K7}"” is the absolute asymptotic condition number of the numerical method corresponding to the datum d,, The numerical method is said to be well-conditioned if the condition number K™” is "small" for any admissible datum d,, and ill-conditioned otherwise. We can also define the resolvent G,,, for the numerical method: 2 = G(d,), that is F(Gn (dn), dn) =0 Assuming itis differentiable: IIdn| Kn(dn) * ||Gn(dn)|| and Kats © ||Gh(dn)|l Gn (dn)Il Examples: * Sum and subtraction. The sum defined as f:ROR (ab) ath has derivative f’(a,b) = (1, 1)", and thus, its condition number K((a,b)) ~ “+! ~ 1 The subtraction defined as. ja+b) > :R OR f aye 6 has derivative f’(a,b) = (1, —1), and thus, its condition number K((a,b)) ~ which can be very big if a = b. hitps:icolab research. google com/crivel16JOVigacg2tHPcp-IC)SOkeViSKAXkgMscroifo=eB9seIXYr2ou8priniMods=Iruesor021 Case_01jpyn -Colaboratory ‘+ Finding the roots of x? — 2px + 1 = 0 is well-conditioned, but we can develop an unstable algorithm: x = p — y/p® — 1 because this formula is subject to errors due to numerical cancellation of digits in the subtraction. The Newton's method could be a stable algorithm to solve this problem: 22 — 2panatl En = Eq — 1 2rn1 — 2p ‘el len—Pl if the algorithm converges, then x, —> a+. or x_, therefore, |zn — p| > yp? — 1 and A ka(p) > KR" (p) = (p) @)* TA problem. Then, if p + 1 the problem is ill-conditioned. The method's condition number is K’,(p) To compute K;"”"(p) we notice that which is similar to the condition number of the exact Definition: We say that the numerical method F’,(x,,, d,,) = 0 is convergent if and only if Ve > 0 Sno(e), 35(n9,€) : Vn > no, /||ida | < 5(n9,€) > llx(d) — 2, (d+ dd,) || <€ where d is an admissible datum, x(d) the corresponding solution, and «(d+ dd,) is the solution of the numerical problem (F),(z,,,d,)) with datum d + dd). Definition: Absolute and relative errors: E(&n) = |x — tn| Definition: Error by component: Era (®n) Relations between stability and convergence The concepts of stability and convergence are strongly connected. If a (numerical) problem is well-posed, stability is a necessary condition for convergence. Moreover, if the numerical problem is consistent, stability is a sufficient condition for convergence. This is known as "equivalence" or "Lax-Richtmyer" theorem: For a consistent numerical method, stability is equivalent to convergence. ~ Sources of errors in computational models Whenever the numerical problem (NP) is an approximation of a mathematical problem (MP) and this latter is in turn a model of a physical problem (PP), we say that NP (F,(a¢n, dn) = 0) is a computational model for PP. The global error € = |2rp, — 2,| can be interpreted as the sum of the MP error ey, and the computational problem error e, = @ — a (e = em + €c) hitps:icolab research. google com/crivel16JOVigacg2tHPcp-IC)SOkeViSKAXkgMscroifo=eB9seIXYr2ou8priniMods=Irue 79yo122021 ‘Clase_01.jpynb - Calaboratory €q: Error induced by the numerical algorithm and the rounding errors. In general, we can enumerate the following sources of err 1. Errors due to the model, that can be reduced by using a proper model 2. Errors due to data, that can be reduced improving the measurement's accuracy. 3. Truncation errors, arising from the approximation (truncation) of limit operations (integrals, derivatives,...). 4, Rounding errors. Type 3 and 4 errors give rise to the computational error, A numerical method will be convergent if this error can be made arbitrarily small increasing the computational effort. Although convergence is the primary goal of a numerical method, there are also the accuracy, the reliability and the efficiency. Accuracy means that the errors are small with respect to a fixed tolerance. It is usually quantified by the infinitesimal order of the error e,, with respect to the discretization characteristic parameter (for example the largest grid spacing) Note: Machine precision does not limit, theoretically, the accuracy, Reliability means that itis very likely that the global error is below a certain tolerance Efficiency means that the computational (effort) complexity needed to control the error (number of operations and memory) is as small as possible. hitps:icolab research. google com/crivel16JOVigacg2tHPcp-IC)SOkeViSKAXkgMscroifo=eB9seIXYr2ou8priniMods=Irue 89sor021 Case_01jpyn -Colaboratory Definition: Algorithm is a directive that indicates, through elementary operations, all the passages needed to solve a problem. It should finish after a finite number of steps, and as a consequence the executor (man or machine) must find within the algorithm itself all the instructions to completely solve the problem Complexity of an algorithm is a measure of its executing time. Complexity of a problem is the complexity of the algorithm with smallest complexity capable of solving the problem. Interesting links: + Well and illposed problems + Condition number (Wikipedia) hitps:icolab research. google com/crivel16JOVigacg2tHPcp-IC)SOkeViSKAXkgMscroifo=eB9seIXYr2ou8priniMods=Irue