Jour of Adv Research in Dynamical & Control Systems, Vol.

12, Issue-02, 2020

An Efficient Spectral Conjugate Gradient

Parameter with Descent Condition for
Unconstrained Optimization
Mustafa Mamat1, Ibrahim Mohammed Sulaiman1, *, Malik Maulana2, Sukono3, Zahrahtul Amani Zakaria1

1
Faculty of Informatics and Computing, Universiti Sultan Zainal Abidin, Terengganu, Malaysia
2
Department of Mathematics, Faculty of Mathematics and Natural Science, Universitas Indonesia, Indonesia
3
Department of Mathematics, Faculty of Mathematics and Natural Sciences, UniversitasPadjadjaran, Indonesia

Abstract:The spectral gradient parameters and conjugate gradient algorithms are among the most efficient algorithms
for solving large-scale unconstrained optimization problems. This is due to efficient numerical performance and
simplicity of their algorithms. Numerous studies have been done recently to improve these methods. In this paper, we
proposed an efficient spectral conjugate gradient algorithm by combining the spectral gradient parameter and conjugate
gradient coefficient. The modified spectral conjugate gradient method satisfies the sufficient descent condition
independent of line search procedure. An interesting feature of the proposed method is that it can be applied to large-
scale unconstrained optimization problems. Preliminary numerical results are presented under strong Wolfe line search
to illustrate the efficiency of the proposed method.

Keywords:Conjugate gradient methods, spectral CG method, unconstrained optimization, convergence analysis, line
search procedure.

Introduction

Nonlinear conjugate gradient (CG) methods are among the efficient numerical methods for solving large-scale
optimization problems due to their low memory requirements, global convergence properties in addition to the
simplicity of their iteration. The nonlinear CG algorithm is designed to solve an unconstrained optimization problem of
the form:

, ∈ (1)

where is a continuously differentiable nonlinear function whose gradient ∇ = is available. The

method computes it iterates using the following recurrence formula.

= + , ≥ 0. (2)

where α is the step size computed along the search direction . The search direction is always the Steepest
descent direction for = 0 [1]. However, subsequent directions are computed as follows

=− + , ≥ 0. (3)

where the coefficient is known as the conjugate gradient coefficient that differentiate various CG methods.
Some well-known formulas for are Fletcher-Reeves (FR) method [2], Polak-Ribiere-Polyak (PRP) method [3, 4],
Hestenes-Stiefel (HS) method [5] and defined by:
! ! !
∇ ∇ " "
∇ ∇ −∇ #$
∇ ∇ −∇
= ! , = ! , = !
∇ ∇ ∇ ∇ ∇ −∇
Also, recent modifications of the CG method are Rivaie et al. (RMIL) [6], Sulaiman et al., (SMARZ) [7] with
formulas as follows:
! ‖. ‖
)! ) − ) , − ‖. / ‖ 2
&'( $&* + /01
% = ! , = !
− ) − )

2487
DOI: 10.5373/JARDCS/V12I2/S20201296
*Corresponding Author: Ibrahim Mohammed Sulaiman, E-mail: sulaimanib@unisza.edu.my
Article History: Received: Feb 20, 2020, Accepted: May 18, 2020
 An Efficient Spectral Conjugate Gradient Parameter with Descent Condition for Unconstrained Optimization

Numerous studies have been done on the convergence of these CG methods [23]. This includes Zoutendijk [8], who
proved the global convergence of FR method using exact line search and the result was extend by Wolfe [9, 10] to PRP
and HS under the standard Wolfe line conditions. More so, the convergence RMIL and SMARZ has been student under
various line searches by Rivaie et al. [6], Sulaiman et al. [7].
Recently, Bergin and Martínez [11] proposed a new spectral conjugate gradient method by combing the spectral
gradient parameter and the conjugate gradient method as follows.

3/01 4/01 5/01 6 ./

= 6
7/01 4/01
, (4)

where 8 = − and 9 = − . For : ≡ 1, (4) reduces to a modified CG method proposed by

Perry [12]. However, the method would reduce to the well-known method of PRP if : ≡ 1 and the exact line search is
used. More so, if the successive gradients are orthogonal and : ≡ 1, then, (4) would reduce to the FR method. The main
disadvantage of this method is that may not be a descent direction during the execution process. Under the standard
Wolfe condition, the researcher established the convergence of this method. This work was further extended to an
alternative spectral parameter as follows:

5/01 6 5/01
: = 6 4 (5)
5/01 /01

Numerical results illustrate the efficiency of the methods compare to the PRP method [3], FR method [2], and Perry
method [12] under various line searches. Recent research on the spectral CG method focus on memoryless BFGS
updates approach [13-16]. Zhang et al., [17], describe another efficient spectral FR (sFR) conjugate gradient method
with spectral parameter : and CG coefficient as follows:

7/01 6 4/01 ‖./ ‖=

: = ‖./01 ‖=
, = = ‖. = (6)
/01 ‖

The outstanding feature of this method is that satisfies the sufficient descent condition independent of any line
search procedures. The sFR method would also reduce to classical FR method under exact line search [18]. The
researchers proved the convergence of the sFR under modified Armijo line search. Numerical results obtained show that
the method performed better than the classical PRP method. More recently, Liu et al. [18] extended the work of Birgin
and Martinez [11] and Zhang et al., [17] to proposed spectral parameter as follows:

./01 6 7/01 ./ 6 7/01

: =− ‖./01 ‖=
+ ‖./ ‖=
, ≥ 1. (7)

An interesting feature of this method is that it can be applied to general CG methods. More so, the method reduces
to sFR method if = . The global convergence of this method was established under some mild conditions.
Recently, Mahmoud et al. [24] proposed a new spectral parameter and established its convergence prove under strong
Wolfe line search. An application of a new spectral parameter can also be referred to Sulaiman et al. [22].
Motivated by the descent properties of the spectral CG method in addition to efficient numerical results, we
proposed a modification of the spectral CG coefficient (sSMARZ) under the strong Wolfe line search. The attractive
feature of this method is the descent direction it generates independent of any line search.

Modified Spectral Conjugate Gradient Algorithm

In this section, based on above discussion on the spectral and CG parameters, we propose a modified spectral CG
method that satisfy the sufficient descent condition. The iterative sequence > ? of this proposed algorithm is obtained
using (2) where the search direction is computed as follows:

− = 0B
=@ $&* + , (8)
−: g + ≥1

$&* +
where = and the spectral parameter : is computed by,
6
./ 7/01
: = C1 − 6 7
./01
D (9)
/01

2488
Jour of Adv Research in Dynamical & Control Systems, Vol. 12, Issue-02, 2020

The algorithms of the of the modified spectral SMARZ is described below.

Algorithm 1. Modified Spectral sSMARZ.

Step 1. Given an initial E , E = − E , set ≔ 0. If ‖ ‖ ≤ H, stop. Else,
Step 2. Compute using $&* +
Step 3. Compute based on (8)
Step4. Compute : by (9).
Step 5. Determine satisfying (10) and (11)
!
+ ≤ +I (10)
! !
≥J (11)

Step 6. Update the new point based on (2).

Step 7. Check if ‖ ‖ = 0, then stop. Else, go to step 2 with = + 1.

For the convergence analysis of the modified sSMARZ spectral method, we need to know that from Sulaiman et al.,
[7, 21], we have

6 KL/ K KL/ K 6
./ C./ 7 D ‖./ ‖= . 7
$&* + KL/01 K /01 KL/01 K / /01
= 6 = 6 )
‖7/01 ‖= 7/01
, (12)
7/01 7/01 )/ /

after some simplification, they have

$&* + ‖./ ‖=
0≤ ≤ ‖7 = . (13)
/01 ‖

Equation (13) plays an important role in the convergence analysis of the modified spectral method.

The following assumptions are often required in the convergence analysis of the CG algorithms.

Assumption A. is bounded from below on the level set Ω = > ∈ / ≤ E ?.

Assumption B. In some neighborhood O of Ω, issmooth and is Lipchitz continuous in O, such that, there
exist P > 0(constant) satisfying;

‖ − 9 ‖ ≤ P‖ − 9‖ ∀ , 9 ∈ O. (14)

Some interesting properties of sSMARZ method is presented as follows.

Convergence Analysis

This section discussed the convergence analysis of the proposed spectral CG method. Apart from the numerical
performance, all conjugate gradient algorithms are expected to satisfy the sufficient decent property defined as:

g! ≤ −S‖g ‖T UV ≥0 (15)

The following Theorem would be used to show that the proposed method satisfies the descent condition (10).

Theorem 1.Consider any CG method of the form (2), if the search direction is computed by (8), (9), where =
$&* +
, then condition 14 holds for all ≥ 0.

Proof. The prove of this Theorem is by induction. Since

! T
E E = −‖ E‖ (16)

2489
DOI: 10.5373/JARDCS/V12I2/S20201296
*Corresponding Author: Ibrahim Mohammed Sulaiman, E-mail: sulaimanib@unisza.edu.my
Article History: Received: Feb 20, 2020, Accepted: May 18, 2020
 An Efficient Spectral Conjugate Gradient Parameter with Descent Condition for Unconstrained Optimization

then,it is obvious that(14)holds for = 0.We also need to show that (14) hold truefor ≥ 0.
!
Multiplying through(8), by , we have
! 5$&* + !
= −: ‖ ‖T + (17)

From (9) and (17), we have

6
! ./ 7/01 5$&* + !
= − C1 − 6 7 D‖ ‖T + (18)
./01 /01

!
From the conjugacy conditions, we have = 0. Thus, (17) reduces to
! ‖T
= −‖ (19)

Hence, (15) hold true for + 1 and this completes the prove. □

The global convergence of 5$&* + would be discussed in this section.

Consider the assumptions A and B, then, the following lemma based on Zoutendjik [8] condition is very crucial in
the convergence studies of most CG algorithm.

Lemma 1. Let Assumptions A and B holds true. For any CG iterative algorithm defined by (2), where is defined by
(3), and the step-size Y satisfies (10) and (11). Then,

6
./ 7/ =
∑]^E < + ∞. (20)
‖7/ ‖=

The proof of this lemma follows from [8]. From (13) and by Lemma 1, we have the following convergent theorem.

Theorem 2. Suppose Assumptions A and B holds true. For any CG method of the form (2) and (3), where the step-
size is computed by (10) and (11). Also, suppose Assumptions A and B and the (15) holds true. Then,

lim ‖ ‖=0
→]

Proof. The prove of this theorem would be by contradiction. We suppose the theorem is not true, then, ∃ d > 0 (d
is a constant) such that

‖ ‖T ≥ d. (21)

From (8), we have

$&* +
+: g = (22)
! T
squaring all side of (22) and dividing by will give

$&* + T !
‖ ‖T ‖ ‖T 2: :T‖ ‖T
! T
= ! T
− ! T
− ! T

! ‖T , then, we have
But, = −‖
$&* + T !
‖ ‖T 2: :T‖ ‖T
= ! − −
‖ ‖T ‖ ! ‖T ‖ ! ‖T
which reduce to

2490
Jour of Adv Research in Dynamical & Control Systems, Vol. 12, Issue-02, 2020
fgh i = ‖7 =
/01 ‖
= = − : T − 2: ‖./ ‖=
(23)
K./6 7/ K

$ &'( ‖./ ‖=
However, from (13), we have ≤ ‖7 = . Substituting in (23) and completing the square will give,
/01 ‖
T
‖ ‖T ‖ ‖T ‖ ‖T 1
! =j k ! + : T − 2: + 1 − 1
T ‖ ‖ T T ‖ ‖T
T
‖ ‖T ‖ ‖T : −1 T
1
=j k ! − +
‖ ‖T T ‖ ‖T ‖ ‖T
‖ ‖l ‖ ‖T 1
= ! +
‖ ‖l T ‖ ‖T

≤ ‖7 = + ‖. = . (24)
/01 ‖ /‖

But‖7 = = ‖. = , hence, (24) reduces to

m‖ m‖

‖ ‖T 1 1
! = +
T ‖ E‖
T ‖ ‖T

which implies

‖ ‖T 1
! ≤n
T ‖ o ‖T
o^E
6
./ 7/ = p=
‖7/ ‖=
≥ . (25)

From (21) and (25), it follows

] ! T
n = + ∞.
‖ ‖T
^E

However, it contradicts our assertion in Lemma 1, and thus, completes the prove. □

Numerical Results

ThePreliminary results presented illustrates the performance of the proposed sSMARZ algorithm compare to the
classical algorithms ofRMIL and SMARZ methods under the strong Wolfe line search. Most of the test problems
considered in Table 1 are from Andrei [19], and each are computed using four ranging initial points. The termination
criterion is set as ‖ ‖ ≤ H, where the value ofH = 10 q . All algorithms are written on MATLAB (2015a) and run on a
Corei5 processor. The computational results are analysed based on number of iterations and CPU time using
performance profiles introduced by Dolan and More [20].
Figure 1 and figure 2 presents the numerical performance of the proposed sSMARZ method compared to the
existing methods of RMIL and SMARZ. The result show that the proposed method outperformed the existing methods
in terms of number of iterations and CPU time under the strong Wolfe line search.

2491
DOI: 10.5373/JARDCS/V12I2/S20201296
*Corresponding Author: Ibrahim Mohammed Sulaiman, E-mail: sulaimanib@unisza.edu.my
Article History: Received: Feb 20, 2020, Accepted: May 18, 2020
 An Efficient Spectral Conjugate Gradient Parameter with Descent Condition for Unconstrained Optimization

Table 1: List of test function.

NO Function Dim Initial point
1 Six Hump 2 (0.5,0.5), (8,8), (40,40)
2 Three Hump 2 (-1,1), (1, -1), (2, -2)
3 Quadratic Qf1 2 (3,3), (5,5), (10,10)
4 Diagonal 2 2 (1,1), (5,5), (15,15)
5 Leon 2 (2,2), (4,4), (8,8)
6 Extended Wood 4 (5,…,5),(20,..,20),(30,..,30)
7 Cloville 4 (2,..,2),(4,..,4),(10,..,10)
8 Hager 2 (7,7), (15,15), (20,20)
9 Extended Penalty 2 (40,40), (80,80), (100,100)
10 Dixon & Price 2, 4 (6,6), (30,30), (125,125),(30,..,30),(125,..,125)
11 Arwhead 2, 10 (8,8), (24,24), (48,48), (24,..,24),(48,..,48)
12 Quartc 2, 10 (8,8), (16,16),(30,30), (16,..,16),(30,..,30)
13 Quadratic QF2 2, 10 (4,4),(40,40),(80,80),(40,..,40),(80,..,80)
14 Generlized Tridiagonal 1 2, 10 (3,3),(21,21),(90,90),(21,..,21),(90,..,90)
15 Generlized Tridiagonal 2 10, 100 (15,.,15),(30,.,30),(150,..,150)
16 Rosenbrock 2, 10,100, 1000 (3,3),(15,15),(75,75),(3,..,3),(75,..,75)
17 Shallow 2, 10,100,1000 (-2,-2),(12,12),(200,200),(200,..,200)
18 Extended White & Holst 2, 10,100,1000 (-3,-3),(6,6),(10,10), (-3,..,-3),(6,..,6),(10,..,10)
19 Extended Tridiagonal 1 2, 10,100,1000 (25,25) ,(50,50), (75,75), (25,..,25) ,(50,..,50),
20 Generralized Quartic 2, 10,100,1000 (7,7),(70,70),(140,140),(7,..,7),(140,..,140)
21 Sum Squares 2, 10,100,1000 (1,1),(5,5),(10,10), (1,..,1),(5,..,5),(10,..,10)

Figure 1: Performance profile based on number of Figure 2: Performance profile base on CPU time
iterations

Conclusion
In this paper, a modified spectral conjugate gradient algorithm is proposed for unconstrained optimization problems.
This method satisfies the sufficient descent condition independent of the line search. An interesting feature of the
proposed method is that it can be applied to large-scale problems. Numerical computations are carried out using some
standard benchmark problems under the strong Wolfe line search. Preliminary results obtained show that the proposed
method is efficient and promising. Thus, can be used as alternative approach for large-scale unconstrained optimization
problems. Also, future research can focus on proving the convergence of this method under different line search
methods.

Acknowledgment
The authors would like to thank the anonymous reviewers for their valuable comments. This work was funded by
Malaysian government under the grant number (FRGS/1/2017/STGO6/UniSZA/01/1).

2492
Jour of Adv Research in Dynamical & Control Systems, Vol. 12, Issue-02, 2020

References

[1] Sulaiman, I. M., Supian, S.,and Mamat, M. New class of hybrid conjugate gradient coefficients with
guaranteed descent and efficient line search. In IOP Conference Series: Materials Science and Engineering,
vol. 621, no. 1, 2019, art. Id. 012021.
[2] Fletcher, R., and Reeves, C. M. Function minimization by conjugate gradients. The computer journal, vol. 7,
no. 2, 1964, pp. 149-154.
[3] Polak E and Ribiere G. Note sur la convergence de directions conjug´ees, Rev. Francaise Informat Recherche
Opertionelle, 3e ann´ee, vol 16, 1969, pp. 35–43.
[4] Polyak, B. T. The conjugate gradient method in extremal problems. USSR Computational Mathematics and
Mathematical Physics, vol. 9, no. 4, 1969, pp. 94-112.
[5] Hestenes M R and Stiefel E L. Methods of conjugate gradients for solving linear systems, J. Research Nat.
Bur. Standards, vol. 49, 1952, pp. 409–436.
[6] Rivaie, M., Mamat, M., Leong, W. J., and Ismail, M. A New conjugate gradient coefficient for large scale
nonlinear unconstrained optimization. Int. Journal of Math. Analysis, vol. 6, no. 23, 2012, pp. 1131-1146.
[7] Sulaiman I M, Mamat M, Abashar A, Rivaie M. The global convergence properties of an improved conjugate
gradient method Applied Mathematical Science, vol 9, no. 38, 2015, pp. 1857-1868.
[8] Zoutendijk, G. Nonlinear programming, computational methods. Integer and nonlinear programming, 1970,
pp. 37-86.
[9] Wolfe, P. Convergence conditions for ascent methods. SIAM review, vol. 11, no. 2, 1969, pp. 226-235.
[10] Wolfe, P. Convergence conditions for ascent methods. II: Some corrections. SIAM review, vol. 13, no. 2,
1971, pp. 185-188.
[11] Birgin, E. G., and Martinez, J. M. A spectral conjugate gradient method for unconstrained
optimization. Applied Mathematics and optimization, vol. 43, no. 2, 2001, pp. 117-128.
[12] Perry, A. A modified conjugate gradient algorithm. Operations Research, vol. 26, no. 6, 1978, pp. 1073-
1078.
[13] Babaie-Kafaki, S. Two modified scaled nonlinear conjugate gradient methods. Journal of Computational and
Applied Mathematics, vol. 261, 2014, pp. 172-182.
[14] Aini, N. Rivaie, M. Mamat, M. and Sulaiman, I. M. A Hybrid of Quasi-Newton Method with CG Method for
Unconstrained Optimization. J. Phys.: Conf. Ser. vol. 1366, 2019, art. id. 012079.
[15] Andrei, N. A scaled BFGS preconditioned conjugate gradient algorithm for unconstrained
optimization. Applied Mathematics Letters, vol. 20, no. 6, 2007, pp. 645-650.
[16] Wu X. A new spectral Polak- Ribière -Polak conjugate gradient method, ScienceAsia. vol. .41, 2015, pp. 345-
349.
[17] Andrei, N. A Dai-Yuan conjugate gradient algorithm with sufficient descent and conjugacy conditions for
unconstained optimization, Appl. Math. Lett. vol. 21, 2008, pp. 165–171.
[18] Liu, J. K., Feng, Y. M., and Zou, L. M. A spectral conjugate gradient method for solving large-scale
unconstrained optimization. Computers & Mathematics with Applications, vol. 77, no. 3, 2019, pp.731-739.
[19] Andrei, N., An unconstrained optimization test functions collection, Advanced Modelling and optimization,
vol. 10, no. 1,2008, pp. 147-161.
[20] Dolan, E. D. and Moré, J. J., Benchmarking optimization software with performance profiles. Math.
Program. vol. 91, no. 2, 2002, pp. 201–213.
[21] Sulaiman I M, Mamat M, Abdelrahman A, Rivaie M, Fatma S M, and Zabidin S. A Modified Nonlinear
Conjugate Gradient Method for Unconstrained Optimization, Applied Mathematical Sciences vol. 9, no. 54,
2015, pp. 2671 – 2682.
[22] Sulaiman, I. M., Yakubu, A. U., and Mamat, M. (2020). Application of Spectral Conjugate Gradient
Methods for Solving Unconstrained Optimization Problems. An International Journal of Optimization and
Control: Theories & Applications. (Accepted).
[23] Yakubu, U.A., Sulaiman, I.M., Mamat, M., Ghazali, P., Khalid, K. The global convergence properties of a
descent conjugate gradient method. Journal of Advanced Research in Dynamical and Control Systems. vol.
vol. 12, no. 2, 2020, pp. 1011-1016.
[24] Mahmoud, D., Sulaiman, I. M., Rivaie, M., and Mamat, M. A New Spectral Conjugate Gradient Method with
Strong Wolfe Line Search. International Journal of Emerging Trends in Engineering Research. vol. 8, no. 2,
2020, pp. 391-397.

2493
DOI: 10.5373/JARDCS/V12I2/S20201296
*Corresponding Author: Ibrahim Mohammed Sulaiman, E-mail: sulaimanib@unisza.edu.my
Article History: Received: Feb 20, 2020, Accepted: May 18, 2020