Nonlinear Optimization using Newton’s Method

Newton's method for optimization is similar to Newton's method for root finding.  The primary difference between the two is that instead of finding zero values for the system of equations it finds zero values for the system of first derivatives of the equations. 

For this algorithm:

1. Start with a set of guess values for the x's.
2. Either:
   1. Calculate the Hessian using either analytical or numerical second order derivatives.  Invert the Hessian to get the Hessian inverse.
   2. Calculate the Hessian inverse with out calculating the Hessian using either analytical or numerical methods.
3. Calculate the objective function's gradient using analytical or numerical first derivatives.
4. Calculate the change in the x's by multiplying the negative of the Hessian inverse by the gradient.
5. Calculate the new x's.
6. Go to step 2) until either the optimization algorithm converges or diverges.

Example

For this example, there is a local minimum at x₁ = 1 and x₂ = 2, with an objective function value of 0.

Initial Guess Values:

x₁ = 3

x₂ = 3

Objective function value = 27

Iteration 1

Gradient

Hessian

Inverse Hessian

Delta X

New X

Objective function value = 1.801

Iteration 2

Gradient

Hessian

Inverse Hessian

Delta X

New X

Objective function value = 0.060

Iteration 3

Gradient

Hessian

Inverse Hessian

Delta X

New X

Objective function value = 0.000185

Iteration 4

Gradient

Hessian

Inverse Hessian

Delta X

New X

 Objective function value = 2.79E-9