1

DCG: Distributed Conjugate Gradient for Efficient

Linear Equations Solving
Haodi Ping, Yongcai Wang, and Deying Li

Abstract—Distributed algorithms to solve linear equations in

50 50
multi-agent networks have attracted great research attention and 1
1
many iteration-based distributed algorithms have been developed. 10 10
11 11
The convergence speed is a key factor to be considered for 25
9 13
25
9 13
distributed algorithms, and it is shown dependent on the spectral 2 2
arXiv:2107.13814v1 [cs.DC] 29 Jul 2021

5 5
radius of the iteration matrix. However, the iteration matrix is 0 12 0 12
determined by the network structure and is hardly pre-tuned,
8 3 8 3
making the iterative-based distributed algorithms may converge
-25 -25
very slowly when the spectral radius is close to 1. In contrast, in 7 7
centralized optimization, the Conjugate Gradient (CG) is a widely
4 4
adopted idea to speed up the convergence of the centralized -50 6 -50 6
solvers, which can guarantee convergence in fixed steps. In -50 -25 0 25 50 -50 -25 0 25 50
this paper, we propose a general distributed implementation of
CG, called DCG. DCG only needs local communication and (a) The Richardson Iteration (b) DCG
local computation, while inheriting the characteristic of fast Fig. 1. The convergence trails in R2 .
The marker ‘+’ represents the start a
convergence. DCG guarantees to converge in 4Hn rounds, where trail. The marker ‘∗’ and ‘◦’ represent the converged estimation x̂i and the
H is the maximum hop number of the network and n is the ground truth xi , respectively.
number of nodes. We present the applications of DCG in solving
the least square problem and network localization problem. The
results show the convergence speed of DCG is three orders 40
DCG
of magnitude faster than the widely used Richardson iteration Richardson
method. 30

Index Terms—distributed algorithm, conjugate gradient, linear 20

equations, network localization, least square problem
10

I. I NTRODUCTION
In many multi-agent applications, the underlying problem
can be reduced to solving a system of linear equations[1]. Be-
cause the autonomous networked agents are usually discretely
deployed, each agent only has the access to communicate with
direct neighbors. Moreover, in certain scenarios, each agent
only desires its own state. Such characteristics consequently
give rise to distributed solvers for linear systems. Different
from the centralized solvers, distributed solvers usually adopt
iterative manners. The key feature of iterative approaches is
linear iterations, where each agent receives states of direct
neighbors; updates and sends its own state. The barycentric
linear localization algorithm[2], is a typical iteration-based
method.
The convergence speed is a crucial factor of the iteration-
based distributed algorithms, which determines whether the
distributed algorithm can be used when the application re-
quires a fast response. However, the convergence rate of many
iterative methods, e.g., Jacobi iteration, Gauss-Seidel iteration,
and Richardson iteration[3], are characterized by the spectral
radius of the iteration matrix. However, the spectral radius is
The authors are with School of Information, Renmin University of China,
Beijing, P.R.China, 100872.
E-mail: {haodi.ping, ycw, deyingli}@ruc.edu.cn
0
100 200 300 400 500 ... 2
Iteration (t)
Fig. 2. The mean square error w.r.t. iteration rounds.
determined by the network topology and is difficult to be pre-
tuned. Thus the convergence speed is highly uncertain and
may be very slow in some network states.
To guarantee fast convergence of distributed algorithms
in solving linear equations, this study explores the idea of
Conjugate Gradient (CG)[4]. CG has two desired properties
for solving the linear systems: 1) CG converges to the exact
solution after a finite number of iterations, which is not larger
than the size of the system matrix; 2) CG is suited for solving
linear systems with large and sparse system matrices. But CG
is essentially a centralized solver. In pursuit of distributed
implementation, we design an efficient protocol to synchronize
the necessary vectors to update the residual. Our distributed
CG, i.e., DCG remarkably speeds up convergence by paying
limited neighborhood communication costs.
The rest of the paper is organized as follows. We formulate
the problem and present related algorithms in Section II. DCG
is proposed in Section III. Applications of DCG are discussed
in Section IV. DCG is evaluated in Section V. The paper is
concluded with further discussions in Section VI.
 2

Notations: Throughout the paper, let AX = b denote a sys-

tem of linear equations, where A ∈ Rn×n is the coefficient
matrix, b ∈ Rn×d is the right-hand side vector, and X ∈ Rn×d
is the vector of unknowns. n is the number of variables and
d is the spatial dimension of a vector, d ∈ {2, 3}. The vector
Ai,: = [Ai1 · · · Ain ] denotes the ith row of A, i ∈ {1 · · · n}.
A = D − L − U. (7)
D is the diagonal component of A. −L and −U are the upper
triangle component and the lower triangle component of A,
respectively. Then, the Jacobi iteration is specified as:

II. P RELIMINARIES X̂(t + 1) = D−1 (L + U)X̂(t) + D−1 b. (8)

A. Problem Formulation From the local behavior of an individual agent vi , the state
Consider a network of n agents V = {v1 , · · · , vn }, each update is:
agent vi is capable of communicating with the agents within  
n
its reception range R. Let E be the set of edges and (i, j) ∈ E 1  X
if the distance between vi and vj is not larger than R. Then the x̂i (t + 1) = bi − Aij x̂j (t) (9)
Aii
multi-agent network can be represented as a graph G = (V, E). j=1,j6=i

The neighbors of vi is denoted by Ni , j ∈ Ni if (i, j) ∈ E. In Gauss-Seidel iteration, the iteration function is designed
Problem: Assume that A is non-singular. Let X∗ denote the as:
unique solution satisfying AX = b. Suppose each agent vi
holds a state vector x̂i ∈ Rd . Initially, vi knows Ai,: and bi,: .
The problem is to devise a local rule for each agent to update X̂(t + 1) = (D − L)−1 UX̂(t) + (D − L)−1 b. (10)
its state x̂i leveraging the local communication with agents in
Ni so that x̂i (t) converges to x∗i within finite t. The local update of agent vi ’s state is:
 
i−1 n
1  X X
B. Related Work x̂i (t + 1) = bi − Aij x̂j (t + 1) − Aij x̂j (t)
Aii j=1 j=i+1
The basic idea of the iterative methods is as follows. Given
(11)
an initialization x̂(0), generate an iteration sequence {x̂(t)}∞
t=0
Another brief iteration method is the Richardson iteration
in a certain manner, so that:
when A is symmetric positive definite. The iteration function
is:
lim x̂(t) = x∗ , A−1 b. (1)
t→∞

Generally, the state update of an iterative method can be X̂(t + 1) = X̂(t) + ω(b − AX̂(t)). (12)
represented as:
The state of agent vi is updated as:
x̂(t + 1) = φk (x̂(t), x̂(t − 1) · · · , x̂(0), A, b), (2)
 
where x̂(0) = φ0 (A, b) or x̂(0) is selected manually. φk is n
X
called the iteration function. The specific design of iterative x̂i (t + 1) = x̂i (t) + ω bi − Aij x̂j (t) , (13)
functions are based on the matrix splitting. j=1

Definition 1 (Matrix Splitting). Suppose a non-singular matrix
A ∈ Rn×n , a split of matrix A is defined as A = M − N,
where M is also non-singular.
Consider a general linear system:
AX = b,
where A is non-singular. (3) can be transformed as follows
through matrix splitting A = M − N.
MX = NX + b.
Then, we can construct an iteration function as:
MX̂(t + 1) = NX̂(t) + b,
which is equivalent to:
X̂(t + 1) = M−1 NX̂(t) + M−1 b , GX̂(t) + g.
G = M−1 b is called the iteration matrix. It is straightforward
that different iterative methods can be constructed by varying
M.
In Jacobi iteration, A is splitted as:
where ω is a non-negative scalar and is suggested to be
2
λmax +λmin . Similar methods include the Successive Over-
Relaxation (SOR) iteration, the Symmetric SOR (SSOR) it-
eration, the Accelerated OR (AOR) iteration, the Symmetric
AOR (SAOR) etc [5].
(3)
C. Convergence and Convergence Rate
Since the aforementioned methods are iterative, a crucial
(4) issue is to guarantee iteration convergence. For a general
iteration function as in (6), its convergence is guaranteed by
the following theorem.
(5) Theorem 1. The iterates formulated by X̂(t+1) = GX̂(t)+b
converges for any X̂(0), if and only if ρ(G) < 1[6].
ρ(G) is the spectral radius of the iteration matrix G. See
(6)
Theorem 4.1 of [6] for the proof.
Apart from knowing when the iteration converges, it is also
desirable to explore how fast it converges. Saad [6] presented
that the convergence rate τ is the natural logarithm of the
inverse of the spectral radius:
 3

• Update Direction (Line 12). di needs another vector

1 r(t − 1) to be updated. It is obtained by Synchro-
τ = ln = − ln ρ(G). (14)
ρ(G) nize Vector(ri (t − 1)). Then, r(t − 1)T r(t − 1) =
P n 2
It can be seen that these iterative methods converge slowly i=1 ri (t − 1) and di can be calculated locally.
when G has unstable eigenvalues. • Update Step Size (Line 14). Represent the numerator and
denominator of αi as numer = d(t)T r(t) and denom =
d(t)T Ad(t), respectively. The direction vector d(t) is
III. DCG: A G ENERAL D ISTRIBUTED C ONJUGATE
G RADIENT I MPLEMENTATION
obtained by Synchronize Vector(dPn i (t)). So the numerator
can be known as numer = i=1 di (t)ri (t).
The most attractive feature of CG is that it converges within An intermediate vector T(t) = [T1 (t) · · · Tn (t)]T is
a fixed round of iterations. But CG is essentially a centralized introduced to calculate denom. By communicating with
P
gradient-based solver for linear equations. See Section 6 of [6] neighbors, vi calculates Ti (t) = j∈Ni A ij j (t).
d
for the original CG algorithm. Although parallel or distributed The complete T(t) is obtained P by Synchro-
n
CG algorithms have been reported for allocating computational nize Vector(Ti (t)). So denom = i=1 di (t)Ti (t).
loads in cloud computing[7] and estimating spectrum in sensor Then αi = −numer/denom.
networks[8], the general distributed CG has not been well • Update State (Line 16). The state X bi (t) updates a step
touched. αi (t)di (t) along its last state.
The key difficulty in implementing Distributed CG (DCG) is Overall, all of the above DCG procedures are completely
that the state X bi is updated based on several vectors, however,
distributed and implemented through neighborhood message
vi only knows the ith element of these vectors. Specifically, passing.
Xbi (t) = Xbi (t − 1) + αi (t)di (t). αi (t) and di (t) are calculated
using the direction vector d(t) and the residual vector r(t).
But vi only knows di and ri . Algorithm 1: Distributed Conjugate Gradient
(DCG) of vi
Input: Ω(i,:) ; bi ;
A. Vector Synchronization Output: x̂i ;
To supplement the necessary information, we design the 2 Xbi (0) ← 0; di (0) ← 0; ri (0) ← −bi // Initialize
Synchronize Vector(zi ) protocol, through which vi can gather /* Update X bi as in Section III-B */
any complete vector z by constantly exchanging respective 4 for iterations t ∈ {1, · · · , tmax } do
elements with neighbors (Line 21-31 of Algorithm 1). H is 6 b − 1))i − bi ; // residual
ri (t) = (ΩX(t
the largest number of hops throughout the network. So the 8
T
if r(t) r(t) < ε then
synchronization can be finished in H rounds. H can be set 10 break // meets accuracy requirement
to n if it is not given at network deployment. The operation T
r(t) r(t)
‘merge’ (Line 29) means selecting all non-zero elements of 12 di (t)=−ri (t)+ r(t−1) T r(t−1) di (t − 1) //direction
T
the input vectors and stacking them as a new vector while 14
d(t) r(t)
αi (t) = − d(t) T Ωd(t) ; // step size
maintaining their original indexes.
16 Xi (t) = Xi (t − 1) + αi (t)di (t); // state
b b
18 return X bi (t).
B. Distributed Conjugate Gradient
19 Function Synchronize Vector(zi )
DCG is detailed as Function DCG (Line 2-18 of Algo-
21 initialize zi (0) ← [0(i−1)×1 zi 0(n−i)×1 ];
rithm 1). At initialization (Line 2), the state X bi (0) is set to
23 for iterations k ∈ {0, · · · , H} do
0. The direction vector component di and the residual vector
25 exchange zi (k) with Ni ;
component ri are set to 0 and −bi , respectively. At each
27 for each vj ∈ Ni do
iteration t, the local behavior of a node vi is as follows:
29 zi (k + 1) ← merge(zi (k), zj (k));
• Update Residual (Line 6). The residual ri is updated by
ri (t) =P b i (t − 1) − bi , which is realized by ri (t) =
(AX) 31 return zi .
−bi + j∈Ni Aij X bj (t − 1). A(i,:) and bi are known to
vi at the begining. Xbj (t − 1) is the state of neighbor vj
obtained by local communication.
• Check Residual (Line 8). CG theoretically completes C. Analysis of DCG
after n iterations [3]. However, due to the accumulated
floating point rounding off errors, the residual and the In each round, four vectors are obtained by synchronization,
direction gradually lose accuracy. Thus, DCG terminates so it converges within 4Hn rounds when A is non-singular.
by checking whether r(t)T r(t) < ε. The threshold ε In actual applications, there may exist measurement noise so
is an empirical value based on accuracy requirement. that A and b are influenced and the constructed linear system
r(t) is obtained by: r(t) = Synchronize Vector(ri (t)). may be unsolvable[9]. Let A and b denote the noisy matrices
as:
After knowing the synchronized Pn r, the squared residual
is calculated as r(t)T r(t) = i=1 ri (t)2 . A = A + ∆A, b = b + ∆b, (15)

where ∆A and ∆b are error matrices implying the noise. If
the noisy matrix A is still non-singular, DCG still converges
to the neighborhood of X∗ .
Lemma 1. For a noisy linear system AX = b, DCG
converges if the error matrix satisfies:
||∆A|| < λmin (A). (16)
Proof. From (15), we can obtain:
A = A(I + A−1 ∆A). (17)
Since ||A−1 ∆A|| ≤ ||A−1 || ||∆A|| = λ||∆A|| , the matrix A
min (A)
must be non-singular if (16) holds. Thus DCG can converge
∗ −1
to the neighborhood of X∗ shown as: X = A b.
IV. A PPLICATIONS OF DCG
In this section, we investigate applying DCG to two actual
scenarios.
A. The Least Square Problem
Linear equations arising from the era of engineering are
usually over-determined. A typical scenario is distributed
parameter estimation, where the observation equations are as:
Ai x i = bi + δ i , (18)
where xi is the desired parameter to be calculated and δi is the
component implying the measurement noise of bi . Due to the
measurement noise, such linear equations usually do not have
a solution that exactly meets the constraints. This problem can
be formulated as follows. Suppose there is no solution to the
linear system AX = b and AAT is non-singular. Each agent
vi only knows the ith row of A and b, which are denoted
by Ai,: and bi,: , respectively. Design a distributed rule for
each agent to update its state xi so that xi (t) converges to the
unique solution of:
AT AX = AT b. (19)
Let Ω = AT A and β = AT b. To solve the least square
problem as in (19), each agent should be aware of the ith row
of Ω and β using Ai,: and bi,: .
Initially, vi maintains Ai,: .Then, vi transmits Ai,j to Ni and
receives Aj,i from Ni , then it also knows the nonzero elements
of the ith column A:,i , i.e., ATi,: . Thus the nonzero elements
of Ωi,: are calculated distributively as:
Xn
Ωij = ATik Akj , ∀vj ∈ Ni . (20)
k=1
Similarly, β i,: can be calculated as:
Xn
βij = ATik bkj , j ∈ {1, · · · , d}. (21)
k=1
Finally, the problem can be solved as xi = of the ith row Mi,: by Mi,j = −Ai,j if vj ∈ Ni . Then, vi
DCG(Ωi,: , β i,: ). Therefore x solves the least squares
problem in (19).
4
B. The Network Localization Problem
The geographical locations of nodes are fundamental in-
formation for many multi-agent applications[10], [11], [12].
Network localization techniques are usually adopted for cal-
culating node locations in infrastructure-less scenarios[13],
[14], which are formulated as follows. For a network of
m + n agents in Rd , let V = A ∪ F denote the entire node
set. Nodes in A = {v1 , · · · , vm } are called anchor agents,
whose locations PA = {p1 , · · · , pm } are known. Nodes
in F = {vm+1 , · · · , vm+n } are called free agents, whose
locations PF = {pm+1 , · · · , pm+n } are unknown. Each
agent vi can only sense the relative distance dij between vi and
any neighbor vj ∈ Ni . Each agent can exchange its estimated
location p̂i and distance measurements with neighbors. The
network localization problem is to design a distributed protocol
for each agent to update its location p̂i so that it converges to
pi .
To solve the network localization problem, we transform it
to a linear system. First, each location pi is represented as a
linear combination of locations of neighbors:
X
pi = aij pj , (22)
vj ∈Ni
where aij are called barycentric coordinates. The calculation
of barycentric coordinates involve only local distance measure-
ments and the specific process is introduced by Diao et al. [2]
in R2 and by Han et al. [15] in R3 . Then after calculating the
barycentric coordinates for each node, the agent locations can
form a linear system:
    
PA I 0 PA
= . (23)
PF B C PF
 
I 0
A = ∈ R(m+n)×(m+n) is constructed with
B C
barycentric coordinates, i.e., the ith row of A is the barycentric
coordinate of vi w.r.t. its neighbors Ni . Then, the localization
problem can be transformed into solving the following linear
system:
(I − C)PF = BPA . (24)
Writing I − C as M, (24) can be reformulated as:
MPF = BPA . (25)
Considering that CG is used when the system matrix is positive
definite [3]. Thus, we multiply MT to both sides of (25):
MT MPF = MT BPA . (26)
Then, MT M is positive definite. To apply DCG to distributed
localization, the localization model in (26) is reformulated to
ΩPF = β, where Ω = MT M ∈ Rn×n and β = MT BPA ∈
Rn×d .
Algorithm 2 shows the routine of DCG-Loc. For initializa-
tion, vi needs to know the ith row of Ω and β to invoke
the general DCG. After constructing the local linear model
by (22), vi maintains Ai,: , so it knows the nonzero elements
transmits Mij to Ni and receives Mji from Ni (Line 4-6), so
it also knows the nonzero elements of the ith column M:,i ,

i.e., MTi,: . Thus the nonzero elements of Ωi,: are calculated
distributively as:
Xn
T
Ωij = Mik Mkj , ∀vj ∈ Ni . (27)
k=1
To calculate β (i,:) , an intermediate vector µ ∈ Rn×d implying
BPA is introduced. vi locally calculates the ith row of µ:
X
µi,: = Aia pa , (28)
va ∈Ni ∩A
where Ni ∩A represents neighboring anchors. µi,: = [0, 0] if no
anchor is found in Ni . Then, vi sends µi,: to Ni and receives
µj,: of each vj ∈ Ni∗ (Line 4-6). Thus, β i,: is calculated as:
X
T
β i,: = Mi,j µj,: ,
vj ∈Ni ∩F
where Ni ∩ F means the non-anchor barycentric neighbors.
For convenience, each location p̂i ∈ Rd×1 is decomposed to
[p̂1i , · · · , p̂di ]. β (i,:) is decomposed to [βi1 , · · · , βid ]. The ele-
ment p̂ji is calculated by DCG(Ai,: , βij ) (Line 10). Therefore,
from the GBLL model in (26), p̂i is calculated leveraging
DCG-Loc, where communications only involve message pass-
ing with neighbors.
Algorithm 2: Distributed Conjugate Gradient Lo-
calization (DCG-Loc) of vi
Input: neighbors: Ni ; barycentric coordinates: Ai,: ;
Output: location: p̂i ;
2 Mij ← −Aij ; calculate µi,: as (28);
4 transmit Mij and µi,: to vj ∈ Ni ;
6 receive Mji and µj,: from vj ∈ Ni ;
8 calculate Ωi,: as (27), calculate β i,: as (29);
1 d
10 return p̂i ← [DCG(Ωi,: , βi ),· · · , DCG(Ωi,: , βi )].
V. E VALUATION
In this section, we evaluate the convergence speed between
our proposed DCG algorithm and the representative Richard-
son iteration by counting the iteration rounds. Simulations are
conducted using MATLAB R2020b in both R2 and R3 . A
network denoted by G = {V, E} is deployed. The number of
anchors is set to d + 1, which is the minimum number of
anchors required to uniquely localize a network in Rd .
Fig. 1 and Fig. 2 shows the results in R2 . DCG and
Richardson iteration are adopted to solve the linear localization
problem modeled by the network in Fig. 1. It is shown that
both DCG and Richardson successfully converge to the ground
truth within finite rounds of iterations. However, Fig. 2 shows
that the Richardson iteration consumes 5 × 105 rounds while
DCG only needs 550 rounds. From Fig. 3 and Fig. 4, similar
results can be obtained in R3 . Overall, DCG-Loc is shown to
be faster than Richardson iteration about 1,000 times.
VI. C ONCLUSION
In this paper, we proposed DCG to enable a network of
n agents to solve linear equations like AX = b in fixed
rounds of iterations. The DCG algorithm is presented with
5
(a) The Richardson Iteration (b) DCG
Fig. 3. The convergence trails in R3 .
(29) 60
DCG
Richardson
40
20
0 ... 2
100 200 300 400
Iteration (t)
Fig. 4. The mean square error w.r.t. iteration rounds.
property analysis and two applications. Compared with tradi-
tional Richardson iteration, DCG shows 3 magnitudes faster
convergence speed. In future work, we will consider reducing
the communication burden that DCG requires.
R EFERENCES
[1] Shaoshuai Mou, A Stephen Morse, Zhiyun Lin, Lili Wang, and Daniel
Fullmer. A distributed algorithm for efficiently solving linear equations.
In 2015 54th IEEE Conference on Decision and Control (CDC), pages
6791–6796. IEEE, 2015.
[2] Y. Diao, Z. Lin, and M. Fu. A barycentric coordinate based distributed
localization algorithm for sensor networks. IEEE Transactions on Signal
Processing, 62(18):4760–4771, 2014.
[3] Wolfgang Hackbusch. Iterative Solution of Large Sparse Systems of
Equations, chapter 10.2, page 234. Applied Mathematical Sciences.
Springer International Publishing, 2 edition, 2016.
[4] Charu C Aggarwal. Linear Algebra and Optimization for Machine
Learning. Springer, 2020.
[5] Richard Barrett, Michael Berry, Tony F Chan, James Demmel, June
Donato, Jack Dongarra, Victor Eijkhout, Roldan Pozo, Charles Romine,
and Henk Van der Vorst. Templates for the solution of linear systems:
building blocks for iterative methods. SIAM, 1994.
[6] Yousef Saad. Iterative methods for sparse linear systems. SIAM, 2003.
[7] Leila Ismail and Rajeev Barua. Implementation and performance eval-
uation of a distributed conjugate gradient method in a cloud computing
environment. Software: Practice and Experience, 43(3):281–304, 2013.
[8] Songcen Xu, Rodrigo C De Lamare, and H Vincent Poor. Distributed
estimation over sensor networks based on distributed conjugate gradient
strategies. IET Signal Processing, 10(3):291–301, 2016.
[9] Xu Fang, Xiaolei Li, and Lihua Xie. 3-d distributed localization
with mixed local relative measurements. IEEE Transactions on Signal
Processing, 68:5869–5881, 2020.
[10] Thien-Minh Nguyen, Zhirong Qiu, Thien Hoang Nguyen, Muqing
Cao, and Lihua Xie. Persistently excited adaptive relative localization
and time-varying formation of robot swarms. IEEE Transactions on
Robotics, 36(2):553–560, 2020.
[11] T. Sun, Y. Wang, D. Li, Z. Gu, and J. Xu. Wcs: Weighted component
stitching for sparse network localization. IEEE/ACM Transactions on
Networking, 26(5):2242–2253, 2018.
 6

[12] Y. Wang, T. Sun, G. Rao, and D. Li. Formation tracking in sparse

airborne networks. IEEE Journal on Selected Areas in Communications,
36(9):2000–2014, 2018.
[13] H. Ping, Y. Wang, D. Li, and T. Sun. Flipping free conditions and their
application in sparse network localization. IEEE Transactions on Mobile
Computing, pages 1–1, 2020.
[14] Haodi Ping, Yongcai Wang, and Deying Li. Hgo: Hierarchical graph
optimization for accurate, efficient, and robust network localization.
In Proceedings of the 29th International Conference on Computer
Communications and Networks, ICCCN, pages 1–9, 2020.
[15] T. Han, Z. Lin, R. Zheng, Z. Han, and H. Zhang. A barycentric coor-
dinate based approach to three-dimensional distributed localization for
wireless sensor networks. In 2017 13th IEEE International Conference
on Control Automation (ICCA), pages 600–605, 2017.