407 views

Uploaded by k3ndr0

save

- Civil Servicr Main 2 STATSTICS_II
- Section13.pdf
- Panel Data I
- Hamilton
- Low Cost Carrier in China
- ase10book
- Coefficient of Determination
- Patterns and Determinants of Organized Dairy Industry Growth a Panel Study
- Princeton Research Work
- 59
- Regression Interpolation
- The Dynamic Relationship Between Stock Returns, Trading Volume and Volatility:
- Forecasting Inflation With ANN Models - BCP - PortalGuarani
- Determinants of Water Price, Contract Choice and Crop Production Inefficiency in Groundwater Irrigation Markets in Bangladesh
- c12
- ece
- 6.899-presLecture3
- FDI Economic.growth GCC 2009
- Matrrix
- Full Text 4
- phpNURdrk
- Du Msc Math Entrance Paper2013
- Sub-Riemannian Geometry and Optimal Transport
- Economic Shocks and Civil Conflict: An Instrumental Variables Approach
- SCRA Eligibility
- The ANNALS of the American Academy of Political and Social Science-2001-Rao-85-104.pdf
- 11 Dynamic Regression
- Education and development
- Admission.vit.Ac.in Vitmee2012 Syllabus MCA
- Regression Analysis
- The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution
- Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future
- Dispatches from Pluto: Lost and Found in the Mississippi Delta
- Yes Please
- The Unwinding: An Inner History of the New America
- Sapiens: A Brief History of Humankind
- The Prize: The Epic Quest for Oil, Money & Power
- The Emperor of All Maladies: A Biography of Cancer
- Grand Pursuit: The Story of Economic Genius
- This Changes Everything: Capitalism vs. The Climate
- A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story
- Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America
- John Adams
- The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers
- Smart People Should Build Things: How to Restore Our Culture of Achievement, Build a Path for Entrepreneurs, and Create New Jobs in America
- Rise of ISIS: A Threat We Can't Ignore
- The World Is Flat 3.0: A Brief History of the Twenty-first Century
- The New Confessions of an Economic Hit Man
- Team of Rivals: The Political Genius of Abraham Lincoln
- Bad Feminist: Essays
- Angela's Ashes: A Memoir
- Steve Jobs
- How To Win Friends and Influence People
- The Incarnations: A Novel
- You Too Can Have a Body Like Mine: A Novel
- Extremely Loud and Incredibly Close: A Novel
- The Sympathizer: A Novel (Pulitzer Prize for Fiction)
- The Silver Linings Playbook: A Novel
- Leaving Berlin: A Novel
- The Light Between Oceans: A Novel
- Brooklyn: A Novel
- The Flamethrowers: A Novel
- The Blazing World: A Novel
- The Rosie Project: A Novel
- The First Bad Man: A Novel
- We Are Not Ourselves: A Novel
- The Master
- Bel Canto
- A Man Called Ove: A Novel
- The Love Affairs of Nathaniel P.: A Novel
- Life of Pi
- The Cider House Rules
- A Prayer for Owen Meany: A Novel
- My Sister's Keeper: A Novel
- Lovers at the Chameleon Club, Paris 1932: A Novel
- The Bonfire of the Vanities: A Novel
- The Perks of Being a Wallflower
- The Kitchen House: A Novel
- Beautiful Ruins: A Novel
- Interpreter of Maladies
- The Wallcreeper
- The Art of Racing in the Rain: A Novel
- Wolf Hall: A Novel
- Contagem Padr o Em Placas Para Mes Filos Aer Bios e Coliformes a 35 c 928255147
- MRTP R6026L
- 2012 Operating Experience With the Pfeiffer MVR Mill and MultiDrive
- Security Infrastructure10
- DIAPOSITIVAS DE QUIMICA.pptx
- AD1 - 2013.2 - Educação Especial - Licenciaturas
- 112260316-tesis-tijeral
- Cnil Rapport Garder La Main Web
- Ejemplos Para Desarrollar Un Guion
- Plan de acción eu
- Acelerómetros
- MB009.pdf
- Valoración de Elementos No Estructurales PDF
- Boundary Value Problems_2
- 9. Labores culturales
- 2.2 R y L mejorada
- BOVINOSCap5
- Blindness .ppt
- Compte rendu Mallarmé
- chapter2_answers.pdf
- Aula de Agroecologia Ifapa Chipiona_ Sanidad Vegetal
- Monografia de Nucleotidos y Nucleósidos
- Coisas Legais Pra Fazer Com Sua Placa de TV
- AISI 410
- Project Delay and Cost Overrun
- Evaluacion de La Estructura Familiar
- The Discipline of the Anointing
- AWG052WP
- Carta de Instrucciones Romina 170105
- Acoplamientos Vulkan

You are on page 1of 6

by levmar

Manolis I. A. Lourakis

Institute of Computer Science

Foundation for Research and Technology - Hellas (FORTH)

Vassilika Vouton, P.O. Box 1385, GR 711 10

Heraklion, Crete, GREECE

February 11, 2005

Abstract

The Levenberg-Marquardt (LM) algorithm is an iterative technique that

locates the minimum of a function that is expressed as the sum of squares

of nonlinear functions. It has become a standard technique for nonlinear

least-squares problems and can be thought of as a combination of steepest

descent and the Gauss-Newton method. This document briefly describes the

mathematics behind levmar, a free LM C/C++ implementation that can be

found at http://www.ics.forth.gr/˜lourakis/levmar.

Introduction

The Levenberg-Marquardt (LM) algorithm is an iterative technique that locates

the minimum of a multivariate function that is expressed as the sum of squares

of non-linear real-valued functions [4, 6]. It has become a standard technique

for non-linear least-squares problems [7], widely adopted in a broad spectrum of

disciplines. LM can be thought of as a combination of steepest descent and the

Gauss-Newton method. When the current solution is far from the correct one,

the algorithm behaves like a steepest descent method: slow, but guaranteed to

1

converge. When the current solution is close to the correct solution, it becomes a

Gauss-Newton method. Next, a short description of the LM algorithm based on

the material in [5] is supplied. Note, however, that a detailed analysis of the LM

algorithm is beyond the scope of this report and the interested reader is referred to

[5, 8, 9, 2, 10] for more comprehensive treatments.

In the following, vectors

and arrays

appear in boldface and is used to denote

transposition. Also, and denote the 2 and infinity norms respectively.

Let be an assumed functional relation which maps a parameter vector

to an estimated measurement vector

. An initial parameter es-

timate

and a measured vector are provided and it is desired to find the vector

that best satisfies the functional relation , i.e. minimizes the squared distance

! !

with ! "# . The basis of the LM%$'algorithm &(

is a linear approximation to

in the neighborhood of

. For a small , a Taylor series expansion leads to

the approximation $+& $4&

*) -,. /

01)32 (1)

&:9

&

where 2 is the Jacobian matrix 5+687 . Like all non-linear optimization methods,

5

LM is iterative:8Initiated 8

at the starting point

0 , the method produces a series of

vectors

<;+=

?>8 that converge towards a local minimizer

$'&

for . Hence, at

each

$+&

step,

it is required $+to&Bfind

the $+&(

that

minimizes $'the&

quantity @# /

A)

# # ! #

,

2 2 . The sought is thus $C&

the solution to a

# !

linear least-squares problem: the minimum is $4attained &

when 2 is orthogonal

$'&

# ! ED

to the column space of 2 . This leads to 2 /2 , which yields as the

solution of the so-called normal equations [1]:

$4&

!

2 2 2 (2)

The matrix 2 2 in the left hand side of Eq. (2) is the approximate Hessian, i.e. an

approximation to the matrix of second order derivatives. The LM method actually

solves a slight variation of Eq. (2), known as the augmented normal equations

F $4&

!

2 (3)

F

where the off-diagonal elements of are identical to the corresponding elements

FHGG I GG

of 2 2 and the diagonal elements are given by )KJL2 2NM for some

IPORQ

. The strategy of altering the diagonal elements of 2 2 is called damping $C&

I

and $4&is referred to as the damping term. If the updated parameter vector

)

with computed from Eq. (3) leads to a reduction in the error ! , the update is

accepted and the process repeats with a decreased damping term. Otherwise, the

damping term is increased, the augmented $8&

normal equations are solved again and

the process iterates until a value of that decreases error is found. The process

of repeatedly solving Eq. (3) for different values of the damping term until an

acceptable update to the parameter vector is found corresponds to one iteration of

the LM algorithm.

In LM, the damping term is adjusted at each iteration to assure a reduction in

F

the error ! . If the damping is set $'to

&

a large value, matrix in Eq. (3) is nearly

diagonal and the LM update

$'&

step is near the steepest descent direction. More-

over, the magnitude of is reduced in this case. Damping also handles situations

where the Jacobian is rank deficient and 2 2 is therefore singular [3]. In this way,

LM can defensively navigate a region of the parameter space in which the model

is highly nonlinear. If the damping is small, the LM step approximates the exact

quadratic step appropriate for a fully linear problem. LM is adaptive because it

controls its own damping: it raises the damping if a step fails to reduce ! ; oth-

erwise it reduces the damping. In this way LM is capable to alternate between a

slow descent approach when being far from the minimum and a fast convergence

when being at the minimum’s neighborhood [3]. The LM algorithm terminates

when at least one of the following conditions is met:

S ! ! !

The magnitude of the gradient of , i.e. 2 in the right hand side of

Eq. (2), drops below a threshold TU;

$8&

S

The relative change in the magnitude of drops below a threshold TV>

S

The error ! !

drops below a threshold TXW

S

A maximum number of iterations Y is completed

0Z\[

If a covariance matrix ]_^ for the measured vector is available, it can be in-

; ;

corporated into the LM algorithm by minimizing the squared ]a^ ` -norm ! ]^ ` !

instead of the Euclidean ! ! . Accordingly, the minimum is found by solving a

weighted least squares problem defined by the weighted normal equations

; $4& ; !

2 ] ^` 2 2 ] ^` (4)

The rest of the algorithm remains unchanged. The complete LM algorithm is

shown in pseudocode in Fig. 1. It is derived by slight modification of algorithm

3.16 in page 27 of [5]; more details regarding the LM algorithm can be found

W

there. Indicative values for the user-defined parameters are b dcCQ ` , Te; Tf>

EcCQ ;/g EcCQhQ

TfW ` ,Y . levmar is a free C/C++ implementation of this LM al-

0Z\[

gorithm that can be found at http://www.ics.forth.gr/˜lourakis/levmar.

References

[1] G.H. Golub and C.F. Van Loan. Matrix Computations. Johns Hopkins Uni-

versity Press, 3rd edition, 1996.

[2] C.T. Kelley. Iterative Methods for Optimization. SIAM Press, Philadelphia,

1999.

Marquardt Nonlinear Least-Squares Method. Computers in Physics Journal,

11(1):110–115, Jan./Feb. 1997.

Least Squares. Quarterly of Applied Mathematics, 2(2):164–168, Jul. 1944.

for Non-Linear Least Squares Problems. Technical Uni-

versity of Denmark, 2004. Lecture notes, available at

http://www.imm.dtu.dk/courses/02611/nllsq.pdf.

ear Parameters. SIAM Journal of Applied Mathematics, 11(2):431–441, Jun.

1963.

http://plato.asu.edu/topics/problems/nlolsq.html,

Jul. 2004. [Accessed on 4 Aug. 2004.].

IMM-REP-1999-05, Technical University of Denmark, 1999. Available at

http://www.imm.dtu.dk/˜hbn.

[9] J. Nocedal and S.J. Wright. Numerical Optimization. Springer, New York,

1999.

Input: A vector function :
ji
with kmlon , a measurement vector

H
and an initial parameters estimate

0p
.

%r

Output: A vector

H
minimizing q# 0/

0 .

Algorithm:

uQ

Yts ; vts x& w ;

ys

; &

z ! P# !

s 2 2 ; s 0/

0 ; {s 2 ;

%}| G
GG

I

stop:=( { Te; ); s b~f ;%%% ;

while (not stop) and ( YtY )

c Z\[

Y*s Y) ;

repeat $4&

z IB

Solve %$4&|

)

{ ;

if Tf>

stop:=true;

else $+&

s

&( >

*) ;

> $ & $+&

8/ ! # q# I

s /

){1 ;

OQ 8/

if

; & &

z 8 ! uq# !

s 2 2 ; s

; {s 2 ;

%}| &( > |

!

stop:=( { Te; ) or ( TfW );

I I ; c# wf_#Pc W }w

s ~a:( W ; vts ;

else

I I }w

s ~v ; vts ~v ;

endif

endif

until (OQ ) or (stop)

endwhile

s

;

[5, 8] for details.

[10] W.H. Press, S.A. Teukolsky, A.W.T. Vetterling, and B.P. Flannery. Numer-

ical Recipes in C: The Art of Scientific Computing. Cambridge University

Press, New York, 1992.

- Civil Servicr Main 2 STATSTICS_IIUploaded byshubham kharwal
- Section13.pdfUploaded byMASHIAT MUTMAINNAH
- Panel Data IUploaded byalaa_h1
- HamiltonUploaded byMayté Ysique
- Low Cost Carrier in ChinaUploaded byyan
- ase10bookUploaded byJohn BonJovi
- Coefficient of DeterminationUploaded byleekiangyen79
- Patterns and Determinants of Organized Dairy Industry Growth a Panel StudyUploaded byshiv raj singh Rathore
- Princeton Research WorkUploaded byKhuslen Amgalan
- 59Uploaded byChipo Chip
- Regression InterpolationUploaded byZlatan Telečki
- The Dynamic Relationship Between Stock Returns, Trading Volume and Volatility:Uploaded byrahulagarwals
- Forecasting Inflation With ANN Models - BCP - PortalGuaraniUploaded byportalguarani
- Determinants of Water Price, Contract Choice and Crop Production Inefficiency in Groundwater Irrigation Markets in BangladeshUploaded byirri_social_sciences
- c12Uploaded byRadian Sigit
- eceUploaded bybunny_589318117
- 6.899-presLecture3Uploaded byleopoup
- FDI Economic.growth GCC 2009Uploaded byjohnalis22
- MatrrixUploaded byYHTRTR
- Full Text 4Uploaded byazhangx
- phpNURdrkUploaded byContinuing Education at the University of Vermont
- Du Msc Math Entrance Paper2013Uploaded byAnkitSisodia
- Sub-Riemannian Geometry and Optimal TransportUploaded byraul_carrascov
- Economic Shocks and Civil Conflict: An Instrumental Variables ApproachUploaded byMau Bau
- SCRA EligibilityUploaded bySHAILESH KUMAR
- The ANNALS of the American Academy of Political and Social Science-2001-Rao-85-104.pdfUploaded byTina Janežič
- 11 Dynamic RegressionUploaded byibinmajedatyahoo
- Education and developmentUploaded byPragyan Thapa
- Admission.vit.Ac.in Vitmee2012 Syllabus MCAUploaded byRahul Dev
- Regression AnalysisUploaded byJenningsJingjingXu

- Contagem Padr o Em Placas Para Mes Filos Aer Bios e Coliformes a 35 c 928255147Uploaded byRobson Martins da Costa
- MRTP R6026LUploaded bysmcraft
- 2012 Operating Experience With the Pfeiffer MVR Mill and MultiDriveUploaded byamk2009
- Security Infrastructure10Uploaded byJonny Nuklz
- DIAPOSITIVAS DE QUIMICA.pptxUploaded byJose Vera Cieza
- AD1 - 2013.2 - Educação Especial - LicenciaturasUploaded bylhonoratobr
- 112260316-tesis-tijeralUploaded byAlonso Romero Sumi
- Cnil Rapport Garder La Main WebUploaded byKevin Victoire
- Ejemplos Para Desarrollar Un GuionUploaded byRicardo Esteban Vieitez
- Plan de acción euUploaded byEnrique Sanjuán Muñoz
- AcelerómetrosUploaded byLuis Antonio Escobar
- MB009.pdfUploaded bySelvakumar R
- Valoración de Elementos No Estructurales PDFUploaded byjorgeRH1502
- Boundary Value Problems_2Uploaded byKartika Nugraheni
- 9. Labores culturalesUploaded byDiieGviito Qviñones Melgarejo
- 2.2 R y L mejoradaUploaded byLuis Castillo
- BOVINOSCap5Uploaded byMatt Randolf
- Blindness .pptUploaded byMegawati Abubakar
- Compte rendu MallarméUploaded bySombre Arcane Zine
- chapter2_answers.pdfUploaded byLalu Suhaimi
- Aula de Agroecologia Ifapa Chipiona_ Sanidad VegetalUploaded byRenato1982
- Monografia de Nucleotidos y NucleósidosUploaded byPamelita Liñan
- Coisas Legais Pra Fazer Com Sua Placa de TVUploaded bypaulocljr
- AISI 410Uploaded byLinus Loo
- Project Delay and Cost OverrunUploaded byEngr. Amjad Ali Ikram,PMP,MPM,FIE(Pak),MCMAA(USA),MAACEI(USA)
- Evaluacion de La Estructura FamiliarUploaded byFredis A Ventura A
- The Discipline of the AnointingUploaded byJoseph Samuel King
- AWG052WPUploaded bySeba Lopez
- Carta de Instrucciones Romina 170105Uploaded byJoel Luis Quispe Rosales
- Acoplamientos VulkanUploaded byrenzoblade