Penalty Function Algorithm

As can be seen in the previous example, incorporating penalty functions in the objective function allows
the use of the unconstrained minimization algorithms.

However, when the problems become complex, good values of the penalty terms may not be readily
available.

The following algorithm gradually adjust the value of the penalty terms to ensure that the search does
not stop prematurely.

67
 Penalty Function Algorithm

P x, R   f  x   R, g j ( x), hk ( x) 
j  1,2,  J k  1,2,  K

Input: x(0) R(0)

Calculate P(0)

t=0

Minimize P to find x(t+1)

Result is P(t+1)

t=t+1
Yes
Px(t  1), R(t )   Px(t ), R(t )   1 ? Stop

No

R(t  1)  R(t )  R

In case of failure, use:

R(t  1)  R(t )  R

68
Example 7.11:

Use the Penalty Function Algorithm to:

Minimize 𝑓(𝑥) = 𝑥 2

Subject to, 𝑔 = 1 − 𝑥 ≤ 0

It is clear that the solution for this problem is, x=1.

Let,

𝜖1 = 0.001
x(0)=10

t=0

R(0)=2

R=1

Preparation:

After adding the penalty term, the problem becomes,

Minimize, 𝑃 = 𝑥 2 + 𝑅〈(1 − 𝑥)〉2

In this example, we will used the closed form mathematical solution. Using single variable or
multivariable search would be more suitable for realistic problems.

Mathematical Solution:
𝑑𝑃
= 2𝑥 + 2𝑅〈(1 − 𝑥)〉(−1) = 0
𝑑𝑥
Solution:
𝑅
𝑥∗ = 1−𝑥 ≤ 0
𝑅+1
𝑥∗ = 0 1−𝑥 >0

69
𝑃 = 𝑥 2 + 𝑅(0)〈(1 − 𝑥)〉2

Solution:
𝑅(0) 2
𝑥(1) = =
𝑅(0) + 1 3

𝑃(𝑥(0), 𝑅(0)) = 100

4 2 2 2
𝑃(𝑥(1), 𝑅(0)) = + 2 (1 − ) = = 0.67
9 3 3
|𝑃(𝑥(1), 𝑅(0)) − 𝑃(𝑥(0), 𝑅(0))| ≤ 𝜖1 ? No

R(1)=R(0)+R=3

t=t+1=1

70
𝑃 = 𝑥 2 + 𝑅(1)〈(1 − 𝑥)〉2

Solution:
𝑅(1) 3
𝑥(2) = =
𝑅(1) + 1 4

4 2 2 7
𝑃(𝑥(1), 𝑅(1)) = + 3 (1 − ) = = 0.78
9 3 9
9 3 2 12
𝑃(𝑥(2), 𝑅(1)) = + 3 (1 − ) = = 0.75
16 4 16
|𝑃(𝑥(2), 𝑅(1)) − 𝑃(𝑥(1), 𝑅(1))| ≤ 𝜖1 ? No

R(2)=R(1)+R=4

t=t+1=2

71
𝑃 = 𝑥 2 + 𝑅(2)〈(1 − 𝑥)〉2

Solution:
𝑅(2) 4
𝑥(3) = =
𝑅(2) + 1 5

9 3 2 13
𝑃(𝑥(2), 𝑅(2)) = + 4 (1 − ) = = 0.81
16 4 16
16 4 2 20
𝑃(𝑥(3), 𝑅(2)) = + 4 (1 − ) = = 0.80
25 5 25
|𝑃(𝑥(3), 𝑅(2)) − 𝑃(𝑥(2), 𝑅(2))| ≤ 𝜖1 ? No

R(3)=R(2)+R=5

t=t+1=3

The progression of the solution is shown below.

t R x
0 2 10
1 3 0.67
2 4 0.75
3 5 0.80

Is this interior or exterior penalty?

Homework:
Use the Penalty Function Algorithm solve Problems 7.31 and 7.34. Document your solution and make
case that your solution is correct.
Note: Use any of the codes you wrote in Chapter 6.

72
Method of Multipliers (MOM)

While the previous method focused on updating penalty parameters, this algorithm attempts to update
the penalty terms instead.

To better understand the algorithm, let’s assume that:

 Inequality and equality constraints are expressed in terms of bracket and parabolic forms
respectively and,
 All constraints are of the same order such that one penalty parameter can be used.

The optimization problem will change from the general form of,

Minimize,
𝑚 𝑝

𝑃 = 𝑓(𝑋) + ∑ 𝛾𝑗 𝐺𝑗 [𝑔𝑗 (𝑋)] + ∑ 𝜉𝑘 𝐻𝑘 [ℎ𝑘 (𝑋)]

𝑗=1 𝑘=1

In the new form the objective function is expressed as,

Minimize,
𝑚 𝑝
2 2
𝑃 = 𝑓(𝑋) + 𝛾 ∑(〈𝑔𝑗 (𝑋) + 𝜎𝑗 (𝑡)〉2 − 〈𝜎𝑗 (𝑡)〉2 ) + 𝜉 ∑ ((ℎ𝑘 (𝑋) + 𝜏𝑘 (𝑡)) − (𝜏𝑘 (𝑡)) )
𝑗=1 𝑘=1

As you may have noticed, we introduces two new vectors:  and , which are labeled bias

t is the iteration counter.

The terms  and  are updated according to the following equations:
𝜎𝑗 (𝑡 + 1) = 〈𝑔𝑗 (𝑋) + 𝜎𝑗 (𝑡)〉

𝜏𝑘 (𝑡 + 1) = (ℎ𝑘 (𝑋) + 𝜏𝑘 (𝑡))

In the proposed algorithm, we typically start with .

A possible flowchart is shown below.

73
 Input X(0) (0) (0)
Calculate f(X(0))
t=1

Minimize the Multiplier Update Function to

determine X(t)
Calculate f(X(t))

|f(X(t))-f(X(t-1))|<1 Yes Stop

No

Calculate (t) (t)

t=t+1

74
Example 7.12:

Use the Method of Multipliers Algorithm to:

Minimize 𝑓(𝑥) = 𝑥 2

Subject to, ℎ = 𝑥 − 1 = 0

It is clear that the solution for this problem is, x=1.

Let,

𝜖1 = 0.001
x(0)=10

(0)=0

Preparation:
2 2
𝑃 = 𝑥 2 + 𝑅 ((𝑥 − 1 + 𝜏(𝑡)) − (𝜏(𝑡)) )

Rearranging,

𝑃 = 𝑥 2 + 𝑅((𝑥 − 1)2 + 2𝜏(𝑡)(𝑥 − 1))

The minimum is at:

𝑑𝑃
= 2𝑥 + 2𝑅((𝑥 − 1) + 𝜏(𝑡)) = 0
𝑑𝑥
Or,

𝑅(1 − 𝜏(𝑡))
𝑥∗ =
𝑅+1

75
Let, R=1

x(0)=10

(0)=0

f(x(0))=100

t=1

𝑅(1 − 𝜏(0)) 1(1 − 0)

𝑥(1) = = = 0.5
𝑅+1 1+1
f(x(1))=0.25

|f(x(1))-f(x(0))|<1? No

𝜏(1) = (ℎ(𝑥(1)) + 𝜏(0)) = ((𝑥(1) − 1) + 𝜏(0)) = −0.5 + 0 = −0.5

76
t=2

𝑅(1 − 𝜏(1)) 1(1 − (−0.5))

𝑥(2) = = = 0.75
𝑅+1 1+1
f(x(2))=0.5625

|f(x(2))-f(x(1))|<1? No

𝜏(2) = (ℎ(𝑥(2)) + 𝜏(1)) = ((𝑥(2) − 1) + 𝜏(1)) = −0.25 − 0.5 = −0.75

t=3

𝑅(1 − 𝜏(2)) 1(1 − (−0.75))

𝑥(3) = = = 0.875
𝑅+1 1+1
f(x(3))=0.767

|f(x(3))-f(x(2))|<1? No

𝜏(3) = (ℎ(𝑥(3)) + 𝜏(2)) = ((𝑥(3) − 1) + 𝜏(2)) = −0.125 − 0.75 = −0.875

The progression of the solution is shown below.

t x 
0 0 0
1 0.5 -0.5
2 0.75 -0.75
3 0.875 -0.875

Homework:
Use the Method of Multiplier to solve Problems 7.31 and 7.34. Document your solution and make case
that your solution is correct.
Note: Use one of the codes you write in Chapter 6.

77
Checking the Convergence of Constrained Optimization Problems
These are several ways to assess the convergence of search algorithms.
1. Assess the neighborhood of the final solutions by moving ±∆𝑥𝑖 in all variables. If testing fails to
find a point with lower function value, it may be safe that the search is terminated at a point
that is either local or global minimum.
2. Test Kuhn-Tucker conditions while monitoring if the final solution is associated with any active
constraints.

78
Constrained Optimization Test Problems
As with the previous chapters, researchers have identified set of problems to test new algorithms and
assess their effectiveness. The text has several ones. We will look at two in particular.
Minimum Cost of Compressed Air Storage Tank

Lo and Papalambros (1990)1 solved this problem using a deterministic method for calculating global
minimum presented this problem. A compressed air storage tank is to be designed for a working pressure
of 300 psi. The tank volume is 750 ft3. The vessel, which is made of carbon steel, is cylindrical with two
hemispherical ends. This cylinder is made of two identical halves that are longitudinally welded. The heads
are formed and welded to the cylinder. The objective is to minimize the total cost, which is the material
cost and welding cost. The problem variables are, x1 (outside cylinder radius), x2 (head thickness), x3 (shell
thickness), and x4 (shell length). The problem can be modeled as,

Minimize,

f x   0.622x1 x3 x4  3.544 x12 x2  1.58x3 2 x4  19.84x1 x3 2

Subject to,
g1 ( x)  x 2  0.00954x1  0
g 2 ( x)  x3  0.0193x1  0
 4x13 
g 3 ( x)   x12 x 4   1296000 0
 3 

0.10  x1  200.0
0.10  x 2  10.0
0.10  x3  10.0
0.10  x 4  235.0

A value of R equal to 108 is used in this search.

Constraint g3 is divided by 106 to make it of the same order as other constraints.

The initial guess is, (60, 1.0, 1.5, 150)T.

The global minimum of this problem is at (38.0, 0.362, 0.734, 235)T.

1
C. Lo and P. Papalambros, "A Deterministic Global Design Optimization Method for Nonconvex Generalized Polynomial
Problems," Advances in Design Automation, pp. 41-49, ASME, New York, 1990.

79
Welded Beam Design Optimization
This problem, which was proposed by Rekalitis et al. (1983) 2describes a beam that is welded to a rigid
column. A concentrated load is applied to the end of the beam.
It is required to minimize the total cost of production, which combines to the labor costs (a function of
the weld dimensions) plus the cost of the weld and beam material.
The design variables the weld and beam dimensions.

x1 x2

x3

L F

x4

Welded Cantilever Beam

The constraints include limits on the shear stress, bending stress, buckling load and end deflection, and
several size constraints.
The problem can be described as,

Minimize,

f x   1  c3  x12 x2  c4 x3 x4 L  x2 

Subject to,

2
V. Rekalitis, A. Ravindaran, and K. M. Ragsdell, "Engineering Optimization: Methods and Applications," Wiley-Interscience, New York,
2006.

80
g1 ( x )   d   ( x )  0
g 2 ( x)   d   ( x)  0
g 3 ( x)  x 4  x1  0
g 4 ( x)  Pc ( x)  F  0
g 5 ( x)  0.25   ( x)  0
0.125  x1  10.0
0.10  x 2  10.0
0.10  x3  10.0
0.10  x 4  10.0

where,

c3=0.10471 $/in3 c4=0.04811 $/in3


 x    ' 2 2 ' " cos   "2 
0 .5

F
 '
2 x1 x 2
MR
 "
J
 x 
M  F L  2 
 2 
0 .5
 x 2  x  x 2 
R   2   3 1  
 4  2  

  x 2  x  x 2 
 
J  2 0.707x1 x 2  2   3 1
 
  12  2   
  
cos  x 2 / 2 R
6 FL
 ( x) 
x 4 x3 2

4.013 EI  x EI 
Pc ( x)  1 3
2  2 L  
L 
x 4 x3 3
I
12
Gx3 x 4 3

3

4 FL3
 ( x) 
Ex4 x33

The material characteristics are,

E=30X106 psi,

G=12X106 psi,

d=30,000 psi, and

81
d=13,600 psi.
The beam length, L, is equal to 14 inch while the load, F, is equal to 6000 pound.

A value of R, Equation (19), equal to 100 is used in this search. Constraints g1, g2, and g4 are divided by
104 to make them of the same order as other constraints.

The initial guess is, (1, 7, 4, 2)T.

The minimum value is, (0.2444, 6.2187, 8.2915, 0.2444)T. The final function value is 2.3810. The method
of multipliers (MOM) used 1173 objective function and 1177 constraint evaluations respectively,
Rekalitis et al. (1983).

82