Received April 13, 2018, accepted May 12, 2018, date of publication May 18, 2018, date of current

version June 19, 2018.

Digital Object Identifier 10.1109/ACCESS.2018.2838259

Link Prediction on Directed Networks

Based on AUC Optimization
BOLUN CHEN 1,2 , YONG HUA1 , YAN YUAN1 , AND YING JIN1
1 Collegeof Computer Engineering, Huaiyin Institute of Technology, Huaian 233003, China
2 Department of Physics, University of Fribourg, CH-1700 Fribourg, Switzerland

Corresponding author: Yong Hua (huayong1994@126.com)

This work was supported in part by the Chinese National Natural Science Foundation under Grant 61602202, in part by the Natural
Science Foundation of Jiangsu Province under Contract BK20160428 and Contract BK20161302, in part by the Foundation of Jiangsu
Provincial Science and Technology Department under Contract BY2016061-25, and in part by the China Scholarship Council.

ABSTRACT Complex networks have become high-dimensional, sparse, and redundant due to the rapid
expansion of the Internet. Effective link prediction techniques are needed to obtain the most relevant and
important information for Internet users. A new link prediction algorithm based on the area under the receiver
operating characteristic curve (AUC) as the evaluation metric is proposed in this paper. In the proposed
method, the AUC is treated as the objective function and the link prediction problem is transformed into an
optimization problem. A group of topological features is defined for each ordered pair of nodes. By using
those features as the attributes of the node pairs, link prediction can be treated as a binary classification, where
the class label of each node pair is determined by the existence of a directed link between the node pair. Then,
the binary classification problem can be solved by the AUC optimization. According to the empirical results,
high-quality predictions can be achieved by our algorithm.

INDEX TERMS Link prediction, AUC metric, complex networks, target function.

I. INTRODUCTION between people can be found through the analysis of social

As Internet technology rapidly develops, the world is becom-
ing increasingly complicated with frequent networking. In the
real world, various information, biological, and social sys-
tems, from interpersonal relationships to colony structures,
from transportation to the online world, from ecosystems to
the nervous system, can be considered as networks in which
vertices represent interactions between vertices or links and
entities denote relations. Undoubtedly, some potential links
remain undetected, and redundant links and errors occur in
complex networks due to space and time limitations and
the experimental conditions. Moreover, we need to forecast
potential and missing links based on known network infor-
mation. This is the objective of the forecast challenge related
to networks as a result of the changing interactions in complex
networks [1], [2].
In fact, link prediction methods can be utilized as auxil-
iary means to study the structure of social networks. Indeed,
it is essential to forecast future potential links among peo-
ple. The link prediction problem has an extensive scope of
practical applications; for instance, users’ potential friends
can be identified by link prediction and they can even be
introduced to others in online networks [3]. Potential links
relations [4], [5]. Furthermore, to forecast potential coop-
erators and article types, link prediction could be utilized
in academic networks [6]. The links between nodes show
interactive relations in biological networks, such disease-
gene and metabolic networks and protein-protein interaction
networks [7]. The research on link prediction has significant
theoretical importance and an extensive scope of practical
value [8], [9]. For instance, the research can offer a unified
and convenient platform to fairly compare the mechanisms
of network development and to help to theoretically compre-
hend the mechanisms of complex network development to
study complex network evolution models.
The rest of this article is organized as follows.
Section 2 introduces the related work. Section 3 reviews
the problem of link prediction and presents the AUC score
for evaluating the results. Section 4 describes the features
of node pairs for link prediction. Section 5 proposes the
AUC_LP algorithm and describes the details of implementing
the algorithm. The experimental results of the AUC_LP are
presented and analyzed in Section 6, and the performance is
compared with that of similar approaches. Finally, a summary
is provided in Section 7.

2169-3536
2018 IEEE. Translations and content mining are permitted for academic research only.
28122 Personal use is also permitted, but republication/redistribution requires IEEE permission. VOLUME 6, 2018
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization
II. RELATED WORK
According to a majority of research on link prediction,
network links are not directed. In real-world applications,
numerous systems can be represented by directed networks
consisting of directed hyperlinks. For instance, followers
form links to opinion leaders on microblogs and food webs
are composed directed linkages from predators to preys. The
asymmetric nature of links is an aspect of the direct net-
work. As direct networks, modeling links introduce compli-
cations but provide important advantages for analysis. Two
states exist between a pair of nodes when a link is symmet-
ric: absent or present. Additionally, four possibilities exist
between two nodes v and u when the links are asymmetric:
the absence of a link between v and u, u and v are connected
mutually, v links to u, and u links to v. Because v is more
essential to u than u is to v, v has a status or power edge over
u if there is a directed link from u to v.
Some early work in the field of directed networks anal-
ysis aimed to uncover the principles of structural design in
ecology, neurobiology, biochemistry, social and engineering
networks. An algorithm was developed by Milo et al. [10] to
detect ‘‘network motifs’’, i.e., important, recurring patterns
of internal connections. They provided insight into 13 types
of 3-node connected sub-graphs in 19 real directed networks.
Link formation of directed networks has received more atten-
tion than that of undirected networks in studies on online
social networks, especially microblogging networks, where
ties are asymmetric. Golder and Yardi [11] conducted a web-
based experiment on Twitter and found that mutuality and
transitivity are important indices of the desire to build new
connections. Information and social networks were combined
by Yin et al. [12] who also studied the distribution of rela-
tionship types for new links between nodes that are two hops
apart. Garlaschelli and Loffredo [13] focused on link reci-
procity, and based on the actual level of correlation between
mutual links, they proposed a novel measure of reciprocity to
enable network ordering.
According to Zhu et al. [14], in the process of information
spreading, non-reciprocal links are less essential than recip-
rocal ones. They also presented some explanations from the
perspective of network efficiency and connectivity in direct
online social networks. Methodology and formalization were
proposed by Romero and Kleinberg [15] to study two-step
direct closure and to prove its essential function in link for-
mation on Twitter. Schall [16] studied triad patterns to predict
links among nodes in directed networks and proposed an
algorithm called triadic closeness. Moreover, a hypothesis for
the potential theory of link formation in a general directed
network was proposed by Zhang et al. [17].
Furthermore, link prediction was used to verify the deduc-
tion. However, to the best of our knowledge, most research on
link prediction in directed networks focuses on predicting the
directions of existing or potential links; very few quantitative
approaches output the probability of the occurrence of a link
between each node pair.
VOLUME 6, 2018
In addition, several learning approaches related to predict-
ing relations have been proposed on the basis of the index
provided by relation predicting metrics, external informa-
tion and internal attributes. The above approaches based on
learning consist of matrix factorization, probabilistic graph
models and feature-based classification. For instance, the ver-
tex collocation profile metric is considered to be a member
of specific characteristic that represents a portions of the
topology information in detail [18]. Additionally, the appli-
cation of attributes of nodes and relations, including users’
interests, ages, friends and characteristics, has been reported
to be a good way to enhance the relation predicting ability.
Li and Chen [19] proposed a learning approach on the basis
of graph kernels and applied various characteristics, including
the introduction, book title, age, education level and key-
words, to predict user-item relations in a bipartite network.
Scellato et al. [20] believed that global features, social fea-
tures and location features are beneficial characteristics to
predict relations in social networks with a supervised learning
frame.
Enhancing the ability to predict is one of the superiorities
of non-topological characteristics. Nevertheless, most of the
time these characteristics are either unavailable or hard to
obtain. Many non-topological characteristics are critically
interrelated to the domain. To identify and exploit these
characteristics, we need to fully learn the domain. Hence,
generic characteristics, such as topological features, net-
works and nodes, are typically considered in normal relation
prediction applications. Additionally, non-topological char-
acteristics must be considered when the goal is a real relation
prediction learning model. Finally, the main objective in rela-
tion prediction is how to select features in the same way that
other information is learned.
However, fortunately, we can apply many characteris-
tic selection methods in machine learning. For instance,
Scripps et al. [21] proposed learning technology that can
automatically determine the most predictive attributes and
topological features by aligning the adjacency matrix of a net-
work. Pujari and Kanawati [22] proposed a novel topological
method that uses a supervised rank gathering approach. They
applied data to determine weights related to every calculated
characteristic on the basis of the competence of each attribute
to predict known relations. Chiang et al. [23] presented a
weighted machine learning approach for relation prediction
that applies characteristics from long cycles in a network.
The tactics of machine learning can be applied to network
link prediction. Bliss et al. [24] developed a method to predict
future links through the use of covariance matrix adapta-
tion evolution strategy (CMA-ES) to optimize the weights
applied in the linear integration of node similarity indices
and sixteen neighborhoods. The notion of path analysis and
probabilistic link prediction was proposed by Sarukkai [25],
who used Markov chains. Bhawsar et al. [26] used various
features of social networks and applied genetic algorithms
to predict links; they built chromosomes via feature selec-
28123
 B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization
tion in the genetic algorithm. To address the issue of link
prediction, a non-negative matrix factorization model based
on neighborhoods was proposed by Zhao et al. [27]. The
model adopts the factors of characteristics from the topolog-
ical structure that combine with the local neighboring struc-
ture of the potential network. A rank aggregation approach
to predict links in complex networks was proposed by
Pujari and Kanawati [28]. Complete Bayesian inference was
performed by Ermis and Cemgil [29] through variational
Bayesian (VB) methods on the coupled and single-tensor
factorization models. The approach can be performed in large
models. Cai et al. [30] presented a quick comparison on the
basis of a method used to predict the relations connected to
a specific node. They used a random walk method to build a
road to the relations of a specific node. Lv et al. [31] presented
a common structural similarity index that independent of
previous knowledge of the network in terms of perturbation of
the adjacency matrix. Li et al. [32] added the determination
of a definition to the benefits of their own final roads and
the roads related to the final nodes and proposed a new
consistency index called Scop.
Recently, many scholars found that a modular network
structure is capable of disclosing future links. Some scholars
have used local partitioning of a network to predict future
links. For instance, Zhang et al. [33] found that information
provided online is both misleading and redundant. Due to the
‘‘less can be more’’ characteristics, some algorithms can be
designed to improve prediction performance by eliminating
links from the original networks. Clemente and Grassi [34]
focused on the problem of evaluating the local clustering in
complex networks. This measure can be defined in various
ways for cases of networks with weighted edges.
However, less attention has been given to directed and
weighted networks. The authors provided a new local clus-
tering coefficient for these types of networks, beginning
with those existing in previous studies for the undirected
and weighted case. Moreover, four specific components are
extracted to separately consider different types of triangle
patterns.
Moreover, an empirical application on several real net-
works was presented, and the performance of coefficients
from previous studies was compared with the performance of
our coefficients. Agreste et al. [35] assessed the community
detection algorithms in terms of scalability and accuracy
and proposed the infomap algorithm, which can be regarded
as a promising tool for the purposes of web data analysis,
to illustrate the trade-off between computational performance
and accuracy. Santos et al. [36] proposed a consensus graph
that clusters algorithms in terms of directed networks and
achieves superior performance with small-sized clusters and
a high mixture index.
Nevertheless, many algorithms based on similarity con-
sider only the benefits of roads related to two nodes and
ignore the effects the two nodes when considering their com-
parability. Hence, the inaccurate outcomes cannot be fully
trusted.
28124
AUC is the most common measurement to evaluate the
quality of predictions results. As a crucial measure of per-
formance, AUC has been applied to various tasks, including
class-imbalance learning, cost-sensitive learning, instances
ranking, and information retrieval. For link prediction in
networks, regardless of the method we use, our final goal is to
optimize the AUC. Since AUC is a measurement of the link
prediction result quality, we can optimize it directly instead of
calculating alternative similarity indexes or optimizing other
parameters, which is required in other models describing the
topological features of the network. Inspired by this fact,
we present a link prediction method that directly optimizes
the AUC. In the method, AUC is treated as the objective
function, and link prediction is transformed into an optimiza-
tion problem. A group of topological features is defined for
each ordered pair of nodes. By using those features as the
attributes of the node pairs, link prediction can be treated as
a binary classification problem in which the class label of
each node pair is determined by whether a directed link exists
between the node pair. Thus, the link prediction problem
can be reduced to determining which edges belong to the
positive class, where the edges exist, and which belong to the
negative class, where the edges are absent. Then, the binary
classification problem can be solved by AUC optimization.
According to the experimental findings on actual networks,
accurate results and high speed are achieved by the algorithm
in comparison with other methods.
III. LINK PREDICTION AND AUC SCORE
A network represented by a directed graph G = (V , E) is
discussed in this paper. In the graph, E is the set of links
and V is the set of nodes. G does not allow self-connections
and multiple links. When N = |V | is the number of nodes, U
is used to define the set that contains all the possible directed
links of N (N − 1). When aij = 1 and A = [aij ] is the
neighboring matrix of network G, there is a linkage between
nodes j and i; otherwise, aij = 0. The same derivation can
be applied to the weighted links, in which the A entries refer
to constant or ordinal values. The link prediction task can be
viewed as links that will show up in the future or missing links
in the series of links U − E that do not exist.
Our approach is to give a score, S(x, y), to every pair
of ordered nodes (x, y) ∈ U . The current possibility of a
direct link between nodes is indicated by the score. The larger
S(x, y) is, the higher the probability that there is a link from
node x to node y for a pair of nodes (x, y) in U \E.
To evaluate the link prediction results, links in the network
are divided into two parts: the probe set E P represents the
unknown information for forecasting and the training set
E T represents the known information. E T ∪ E P = E and
E T ∩ E P = ∅. An ordered list of all non-observed links (i.e.,
U −E T ) is provided by the link prediction algorithm, or every
non-observed link is provided. Therefore, for (x, y) ∈ U −
E T , the score Sxy quantifies the current probability.
AUC is the most widely adopted evaluation metric to
quantify the accuracy of the prediction approaches to assess
VOLUME 6, 2018
B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization
the results of link prediction. As a significant measure of
performance that is commonly utilized in various tasks,
including class-imbalance learning, cost-sensitive learning,
information retrieval and rank learning, AUC can be utilized
in datasets with regular stands. For example, because AUC
is blind to the class distribution, recall and accuracy are
inadequate. A score is assigned by the similarity-oriented link
prediction approaches to every non-existing and existing edge
within the network, and an ROC (receiver operating charac-
teristic) curve is constructed based on the scores assigned to
the ordered node pairs. Generally, a larger AUC indicates bet-
ter performance. The AUC of a randomly selected predictor
is 0.5 while the AUC value of a perfect predictor is 1.0.
Assume n = n1 + n2 and that there are n2 non-exiting
links and n1 existing links in the network. Let si be the score
assigned to ei , and let en1 +1 , en2 +1 , . . . , en and e1 , e2 , . . . , en1
be the sets of non-existing and existing links, respectively.
The ROC for link prediction can be obtained as follows:
(1) First, sort the links in descending order of score and let
the sorted sequence of the edges be e01 , e02 , . . . , e0n1 .
(2) Generate a coordinating system of the ROC curve,
where the ordinate indicates the current edges and the
abscissa represents the non-existing edges. The abscissa
consists of n2 steps, and the length of each step
is 1/n2 . The ordinate consists of n1 steps, and the length of
each step is 1/n2 . Thus, a unit-side-length square is formed
by the ROC coordinate system.
(3) The ROC curve begins at the origin of the coordinate
system and moves rightwards toward the square’s upper-
right corner. In each step, the direction of the movement is
determined by the sorted sequence of e01 , e02 , . . . , e0n1 . The
curve in the i − th step moves upward one step if e01 is an
existing edge. e0i+1 is an existing edge, and s0i = s0i+1 if e01 is
a non-existing edge. Then, the curve moves one step along
the diagonal; or it moves one step rightwards otherwise. The
curve forms the ROC curve when it reaches the upper-right
corner of the square after n steps.
(4) The area under the ROC curve is the AUC value of link
prediction results. The process of generating an ROC curve is
illustrated with the following examples.
Example 1: The forecasting results of the AUC are esti-
mated in example 1. Three non-existing links and 4 existing
links are included in this example. Therefore, the scores of the
pairs of non-existing and existing links should be compared.
The existing links among the 12 pairs have higher scores than
those of the non-existing links in the 7 pairs: (a, e), (a, f ),
(a, g), (b, f ), (b, g), (c, g) and (d, g). The score equals the
score of a non-existing link, and e can only be found in the
existing link b. Thus, AUC = (7 + 0.5)/12 = 0.625, n0 = 7,
and n00 = 1.
How much better the algorithm is than pure chance is
indicated by the degree to which the value exceeds 0.5. In fact,
the value of AUC should be approximately 0.5 if the scores
are obtained from an identical and independent distribution.
For link prediction in networks, regardless of the method
implemented, the AUC is the most common measure for eval-
VOLUME 6, 2018
FIGURE 1. ROC curve of Example 1.
uating the quality of prediction results. In essence, the goal
link prediction methods is to optimize the AUC. Inspired by
this fact, we present a link prediction method that directly
optimizes the AUC. In this method, AUC is used to construct
the objective function, and link prediction is transformed into
an optimization problem.
Example 2: Assume the links a, b, c, d, e, f , and g are in
a network, where e, f , and g are non-existing links and a,
b, c, and d are existing links. The prediction result gives the
scores sa , sb , sc , sd , se , sf , and sg to the related links such
that sa > se = sb > sf > sc > sd > sg . This study first
ranks the links in descending order of their scores to obtain
the sequence a, e, b, f , c, d, and g to generate the ROC curve.
Then, as shown in Figure 1, the coordinate system of the ROC
curve can be determine. Herein, the ordinate consists of 4
steps and the abscissa consists of 3 steps. As the first link
in the sequence is a, the curve initially moves upwards one
step, starting from the origin of the coordinate system. Then,
as the next two links in the sequence are the existing link b
and the non-existing link e, se = sb , the curve moves one
step along the diagonal. The next link is f , a non-existing link.
Therefore, the curve moves one step rightwards. According to
Figure 1, the ROC curve is completed by repeating the move-
ment until the upper-right corner of the square is reached.
The shaded portion under the curve is the AUC, which is
7.5/12 = 0.625.
The ROC curve does not have to be drawn to calculate
the AUC, which can be seen as the possibility that a ran-
domly selected non-existing link is given a lower score than
a randomly selected existing link. A non-existing link and an
existing link are randomly selected to compare their similarity
scores. Among n independent comparisons, the existing link
has the same score n00 times and it has a higher score n0 times;
therefore, the AUC is as below:
AUC = (n0 + 0.5n00 )/n. (1)
The following examples are utilized to explain the AUC
calculation based on (1).
IV. FEATURES OF NODE PAIRS FOR LINK PREDICTION
Link prediction can be treated as a classification of node pairs
based on their structural properties reflected by the features
of the node pairs. If there is a link between nodes j and i
28125
 B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization

or aij = 0, then A = [aij ] is the neighboring matrix of the

network G, in which aij = 1. The same derivation can be
applied to the weighted links, in which the entries of A are
constant or ordinal values. Link prediction aims to identify a
similarity matrix S = [sij ], where sij is the probability that
link (i, j) exists. We first construct topological features for
each node pair ek = (i, j) to form a feature vector xk . The
research is intended to identify a function h that maps xk to FIGURE 2. Four types of triangles involving node vi . (a) OI type.
the target sij = h(xk ) so that sij = h(xk ). A systematic way (b) OO type. (c) IO type. (d) II type.
to incorporate the additional data is to add the information as
covariates. Network entities provide additional information 3. In degree of node vj .
The in degree of node vj is denoted as djin = ni=1 aij and
P
in real applications. In terms of simplicity, our discussion is
limited to only the relational statistics; however, the model is the number of edges pointing to node vj .
can be extended to address covariates. 4. PageRank of node vi .
Constructing the mapping function h, like generalizing the The PageRank algorithm is as follows:
inherent topological structures underlying the relationships of X Pt−1 (vj ) 1 − c
entities, is the key to a successful forecast. The relationships Pt (vi ) = c + , (3)
among nodes, for instance, common paths, neighbors and djout n
(vj ,vi )∈E
node properties, such as centrality and degree, depend on the
structure of the whole or an important portion of the node cen- where Pt (vi ) defines the probability of visiting node vi by
trality and network structure. Moreover, international knowl- a random walker in time step t. The damping factor c is
edge is usually required by the shortest distance between commonly set to 0.85. The random walker moves along the
nodes. For large complex networks, international network links of the network with probability c, conforming to the
knowledge is not available, and the whole network is not first term on the right-hand side, while jumping to a randomly
practical. It has become essential to identify the existence of chosen node with probability 1 − c, conforming to the second
linkages between nodes via local information only. Therefore, term.
to identify the closeness of two nodes among other node-pair 5. Leader-Follower Ranking for node vi .
characteristics, this study extends the concepts of clustering, This study aims to build an optimal ordering of nodes so
distribution sequence, neighborhood, and assortativity. The that a link is likely to point to a node with a higher ranking
node-pair characteristics can be compiled as a feature vector and originate from a node with a lower ranking to predict
for node pairs with known connectivity. Then, to predict the the direction of an edge between two nodes in a directed
existence of a connection from a particular feature vector, network. This ranking is called the Leader-Follower rank.
a model is constructed by AUC optimization. An algorithm is applied to the directed graph to calculate
For an ordered node pair (vi , vj ), which represents the the Leader-Follower rank of a node. The algorithm is slightly
potential directed link eij = (vi , vj ), we use the node-pair modified to allow two nodes to share an index in the list of
properties and node properties to construct a feature vector ordered rankings. Moreover, a highly ranked node receives a
with the following features: low score.
1. Out degree of node vi . 6. The clustering coefficient of node vi .
The out degree of node vi , denoted as diout , is simply the The ratio between the number of triangles that could have
number of edges starting from node vi . To calculate this been formed and the number of triangles node vi belongs to
value, we use the elements of the adjacent centmatrix: diout = can be used to define the clustering coefficient of node vi .
P n Since the edges of the triangles have directions, 4 types
j=1 aij .
2. Node betweenness for node vi . of triangles involving node vi are considered, as shown in
Proposed by the famous sociologist Linton Freeman, node Figure 2, where the 4 types of triangles are denoted OI, OO,
betweenness is a measure of the centrality of the node within IO and II.
the network [55]. Defined as B(vi ), the betweenness of node We define the 4 types of clustering coefficients of node vi
vi is the fraction of all the shortest paths between all the pairs as:
of nodes that penetrate node vi : NOI
C OI (vi ) = out (4)
X ni di × diin
uw NOO
B(vi ) = (2)
guw C OO (vi ) = out (5)
u6=vi =w di × diout
NII
Here, guw is the total number of directed shortest paths C II (vi ) = in (6)
from node u to w and niuw is the number of directed di × diin
shortest paths from node u to node w that pass through NIO
C IO (vi ) = in (7)
node vi . di × diout

28126 VOLUME 6, 2018

B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization

Here, NOI , NOO , NIO and NII are, respectively, the numbers
of triangles involving node vi of type OI , OO, IO and II ,
corresponding to Figure 2(a), (b), (c) and (d), respectively.
7. The clustering coefficient of ordered node pair (vi , vj ).
The ratio between the two nodes and the number of trian-
gles can be used to define the clustering coefficient between
an ordered node pair (vi , vj ). Since the edges of the triangles
have directions, 4 types of triangles, OI, OO, IO and II, are
considered, as shown in Figure 3.
Equation (12) as shown at the bottom of this page, yields
a coefficient that is the mean of all the links. All the calcula-
tions of diout + djin can be dropped to approximate the local
assortativity between the node pair (vi, vj) for M . M refers to
the number of links in the network. The degree of correlation
rijOI of node pair (vi, vj) is defined as follows:
4diout djin − diout − djin
rijOI = . (13)
2(diout )2 + 2(djin )2 − diout − djin

From (13), we can see that rijOI ranges in 0 < rijOI < 1.
Similarly, we can define the degrees of correlation rijOO , rijII
and rijIO of node pair (vi , vj ) as follows:

4diout djout − diout − djout

rijOO = (14)
2(diout )2 + 2(djout )2 − diout − djout
4diin djin − diin − djin
FIGURE 3. Four types of triangles of clustering coefficients for ordered rijII = (15)
node pair (vi , vj ). (a) OI type. (b) OO type. (c) II type. (d) IO type. 2(diin )2 + 2(djin )2 − diin − djin
We define the 4 types of clustering coefficients of ordered 4diin djout − diin − djout
node pair (vi , vj ) as: rijIO = (16)
2(diin )2 + 2(djout )2 − diin − djout
P
OI h6=i,j aih ahj Calculating the local assortativity requires individual hope
Cij = (8)
diout × djin visibility in the network only, and evaluating rij presupposes
the existence of a link.
P
OO h6=i,j aih ajh
Cij = out (9) 9. Common neighbors.
di × djout
P The neighbors between the node pairs should considered
II h6=i,j ahi ahj in the degree of correlation and clustering. More informa-
Cij = (10)
diin × djin tion can be extracted by observing shared nodes in the
P neighborhood of each node while accounting for the non-
h6=i,j ahi ajh
IO
Cij = , (11) immediate neighbors in the network with a visibility scope
diin × djout greater than one hop. For an ordered node pair (vi, vj) in a
corresponding to Figure 3(a), (b), (c) and (d), respectively. directed network, we consider the directed paths from v to
The coefficient of node-pair clustering is the rate of the u and define the out-neighborhood of node v as 0out (v) =
common neighbors of nodes vj and vi in different directions, (w|w ∈ V , (v, w) ∈ E) and the in-neighborhood of node u as
as shown in Equation(8-11). The clustering coefficient can be 0in (u) = (w|w ∈ V , (w, u) ∈ E). The common neighbor of
determined with one hop visibility within the network only. ordered node pair (v, u) is defined as the set 0out (v) ∩ 0in (u).
8. Degree of Correlation. We call the set 0out
1 = 0 v the first-order out-neighbors of
out
The degree of correlation between two nodes is another node v, and define the i-th-order out-neighbors of node v,
property of node pairs. A majority of networks are considered denoted by 0out i (v), as the set of out-neighbors of the nodes

to show a mixture of disassortative or assortative pairs. In in 0out (v) that do not belong to any of the previous neighbor-
i−1

a network, the Pearson correlation coefficient of two nodes hoods, namely:

represents the assortativity of a node pair. Since there are Similarly, we can define 0in i (u) as the i-th-order in-

two types of degree for each node, we define 4 types of neighbors of node u.
degrees of correlation for a directed network, rijOI , rijOO , rijII We define CN 2 (v, u) = 0out (v) ∩ 0in (u) = 0out
1 (v) ∩ 0 1 (u)
in
and rijIO , corresponding to different combinations of the types as the 2n d-order common neighbors of ordered node pair
(v, u), and i
of degrees of the two nodes. For instance, degree correlation h the i-th-order icommon neighbors as CN (u, v) =
j
j+k=i 0out (v) ∩ 0in (u) . For instance, the 3 d-order
S k r
r OI is defined as:

(vi ,vj )∈E di dj /M − [ (vi ,vj )∈E 2 (di

out in 1 out
+ djin )/M ]2
P P
r OI
= P out 2 in P 1 out in
. (12)
i [(di ) + (dj )]/2M − [ i 2 (din + dj )/M ]
n o
0out
i
(v) = u|u ∈ V and ∃w ∈ 0vi−1 , (u, w) ∈ E and u ∈
/ ∪i−1 0 k
k=1 out (v) . (17)

VOLUME 6, 2018 28127

 B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization

common neighbors of ordered node pair (v, u), CN 3 (v, u), vj from vi , which is:
is the combination of the intersection of 0out 1 (v) and 0 2 (u)
 in
and that of 0out (v) and 0in (u). We use CN i (v, u) (i =
2 1 X 1
sRA
ij = (21)

dzin · dzout
 2,i. . . , n)
1,  as the features of ordered node pair (vi , vj ). Z ∈0(vi )∩0(vj )
CN (v, u) can be calculated via adjacent matrix A. For
instance, the 3r d-order common neighbors of ordered node The RA index assigns greater weight to less-connected
pair (v, u), CN 3 (v, u), can be calculated as follows: neighbors to refine the counting of common neighbors.
  X Of the above features, 1 to 8 are based on node properties,
ahw awk ,
 3
CN (v, u) = (18)

and the rest are based on ordered-node-pair properties. Except
h,k,w6=v,u,avh =aku =1 the Leader-Follower ranking, every feature has a similar ver-
sion in undirected networks. Therefore, our method can be
10. Jaccard coefficient.
applied to both directed and undirected networks.
The data are utilized to compare the diversity and similarity
of the neighboring sets. For an ordered node pair (vi , vj ),
V. THE LINK PREDICTION ALGORITHM
the Jaccard coefficient is the size of the intersection divided
The main idea of our algorithm is that we use the AUC
by the size of the neighbor sets 0in (vj ) and 0out (vi ), as shown
metric as the target function of optimization to obtain high-
in the formula below, and it is used to evaluate the similarity
quality link prediction results. AUC is a good measure of
of the neighboring sets of the two nodes.
performance. Several algorithms have been devoted to the
0out (vi ) ∩ 0in (vj ) optimization of AUC, mainly by reducing the surrogate con-
 
Jaccard
sij = (19) vex loss on a set of training data.
0out (vi ) ∪ 0in (vj )

11. Preferential attachment index (PA).
Preferential attachment can be utilized to create scale-free
networks. The probability of a new link pointing to node vj is
proportional to djin , and the probability of a new link originat-
ing from node vi is proportional to diout . A similar mechanism
can produce scale-free networks without any development,
in which a new link is created and an old link is eliminated in
each time step. Tvj is proportional to diout × djin , and the new
link is connected to vi . The related similarity index motivated
by this mechanism can be obtained as below:
sPA out
ij = di × djin .
This formula can be used to quantify the importance of
the links subject to different dynamics based on networks,
for instance, transportation, synchronization, and percolation.
This feature does not require information about the neighbor-
hood of every node; therefore, the computation is simple.
12. Resource allocation index (RA).
The dynamics of resource allocation on the complex net-
works motivate this index. For an ordered node pair (vi , vj )
that is not directly connected, node vi can send resources to vj
with its neighbors as transmitters. The simple case assumes
that each transmitter has a resource unit and will distribute
the information to all its neighbors. The RA index between
vj and vi can be defined as the quantity of resources sent to
n
1 X X
AUC =
M · (P − M )
k=1 vi ∈0(vk ) vj ∈0(v k)
n
1 X X X
AUC =
M · (P − M )
k=1 vi ∈0(vk ) vj ∈0(v k)
28128
A. THE OBJECTIVE FUNCTION
We treat link prediction as a binary classification problem
that is solved by maximizing the AUC. We use the tuple
(xij , yij ) to represent an ordered node pair (vi , vj ), where
xij = (xij1 , xij2 , . . . , xijm ) is a vector consisting of m features
of the node pair. The feature vectors of all the nodes form an
m ∗ n feature matrix X = [xij ]. yij is the label indicating the
existence of a directed link between ordered node pair (vi , vj ).
If there is a directed link between the node pair, then yi = 1;
otherwise, yi = −1. We set our prediction model as:
(20) h(xij ) = sij = wi xijT , (22)
where wi = (wi1 , wi2 , . . . , wim ) is the weight vector, and wij
reflects the importance of the j − th feature on the prediction
results. Our goal is to find the optimal weight vector w∗i such
that sij and yij are as close as possible. Let W = [wij ] be an
n ∗ n weight matrix, and let S = [sij ] be the similarity matrix.
The goal is then to identify a function h that maps A to the
target S such that S = h(X ) = WX T . Namely formula (23)
and (24).
Here, we use P = N (N − 1)/2 to denote the number of
links in the universal set containing all possible links. The
number of links in the network is M . I (b) is an indicator
function that returns 1 if the argument is 0 and true, and the
set of neighbors of node vk can be denoted as 0(vk ), with
X 1
[I (ski > skj ) + I (ski = skj )] (23)
2
/
1
[I (h(xki ) > h(xkj )) + I (h(xki ) = h(xkj ))]. (24)
2
/
VOLUME 6, 2018
B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization

0(vk ) = v|(vk , v) ∈ E. Algorithm AUC_LP (Link Prediction Based on AUC)

( Input: A: Adjacency matrix of the network;
1 b = true
I (b) =
0 b = flase
Since the direct optimization of AUC can be cast as inte-
grated optimization, it usually results in an NP-hard problem.
A convex optimization problem that minimizes the objective
functions, as below, can approximate Formula (26), as shown
at the bottom of this page.
Here, w = [wij ] is the weight matrix, k·kF is the Frobenius
norm, ι is the loss function and λ is the regular coefficient.
In our algorithm, we use the squared loss function ι(x) =
(1 − x)2 , and the term ι[h(xki ) − h(xkj )] is as follows: [1 −
wk (xki − xkj )T ]2 .
Therefore, the loss function L(w) can be rewritten as For-
mula (27), as shown at the bottom of this page.
We define (Formula (28)) as, as shown at the bottom of this
page.
Then:
Xn
L(w) = Lk (wk ). (29)
k=1
The aim of our algorithm is to obtain the optimal weight
vector w∗i by minimizing Lk (wk ) for k = 1, 2, . . . , n. We use
gradient descent to minimize Lk (wk ), and we initially obtain
the gradient as Formula (30), as shown at the bottom of this
page.
(0)
After setting the initial values of wk , we can update the
(k)
value of wk by the following formula:
∂Lk
wt+1 = wtk − ηt · . (31)
k
∂wk
Such iterations can be conducted until convergence or until
the number of iterations exceeds a constant integer N .
B. THE AUC _LP ALGORITHM’S FRAMEWORK FOR LINK
PREDICTION
In this section, we present the AUC_LP algorithm.
VI. EXPERIMENTAL RESULTS AND ANALYSIS
This section discusses the four sets of benchmark data repre-
senting networks from disparate areas: the US airport network
(25) Output: S: Scoring matrix between nodes;
Begin:
1. Matrix A regularization;
2. Parameter initialization;
3. Construct the feature vectors for each node pair, store in
a matrix X ;
4. Set the initial values of parameters W ;
5. While not termination condition do;
6. For every node vk in V do;
7. update wk according to formula (31);
8. End;
9. End While;
10. Output the result S = W · X T ;
End
TABLE 1. The topological characteristics of the largest components of the
four network samples.
(USAir), political blogs (PB), a protein-protein interaction
network (PPI) and the Internet (INT). This research tests the
largest connected component in each dataset. The topological
characteristics of the largest components of the networks are
shown in Table 1. NUMC is the number of associated com-
ponents in the network and the size of the largest one. M and
N are the total numbers of links and nodes, respectively. For
instance, 1222/2 in dataset PB indicates that the network has
2 associated components and the largest one has 1222 nodes.
e is the efficiency of the network, K is the mean degree of
the network, and r and C are the assortative coefficient and
clustering coefficient, respectively.
We begin by studying the reliance of the forecasting per-
formance on parameters λ ∈ (0, 1] and η ∈ (0, 1] in each
dataset. The results are shown as heatmaps in Figure 4.

n
λ 1 X X X
L(w) = kwk2F + ι[h(xki ) − h(xkj )]. (26)
2 M · (P − M )
/
k=1 vi ∈0(vk ) vj ∈0(vk)
n
λ 1 X X X
L(w) = kwk2F + [1 − wk (xki − xkj )T ]2 . (27)
2 M · (P − M )
/
k=1 vi ∈0(vk ) vj ∈0(vk)

λ 1 X X
Lk (wk ) = kwk k2F + [1 − wk (xki − xkj )]2 . (28)
2 M · (P − M )
/
vi ∈0(uk ) vj ∈0(u k)

∂Lk 1 X X
= λwk − {2(xki − xkj ) − 2wk (xki − xkj )(xki − xkj )T }. (30)
∂wk M · (P − M )
/
vi ∈0(uk ) vj ∈0(u k)

VOLUME 6, 2018 28129

 B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization
FIGURE 4. The dependence of the AUC on parameters λ and η in different datasets.
TABLE 2. Comparison of the accuracy of algorithms quantified by AUC.
In Figure 4, we find that a maximum AUC still exists when
λ and η are not simultaneously small, and the optimal AUC
under all possible values of λ and η occurs in the region where
they are greater than 0.3. Specifically, the optimal parameters
are λ=0.6 and η=0.3 in USAir, λ = 0.5 and η = 0.4 in PB,
λ = 0.35 and η = 0.75 in PPI, and λ = 0.9 and η = 0.4 in INT.
In addition, the highest accuracy is achieved by the algo-
rithm when using AUC as the metric under the optimal
parameters λ and η. The mean AUC for the various algorithms
in the 10-fold cross-validation (CV) are shown in Table 2. The
highest AUC score of the 11 algorithms for each dataset is
highlighted in bold.
As shown in Table 2, AUC_LP index has the best overall
performance among the local indices. For example, compared
with those of CN, the average AUC is improved by 1.12%
for USAir, 4.64% for PB, 7.48% for PPI, and 51.54% for
INT. When compared against Jaccard, the average AUC is
improved by 5.28% for USAir, 12.09% for PB, 7.01% for
PPI, and 49.92% for INT. When the comparison is against PA,
the improvement is even greater: 30.35% for USAir, 28.60%
for PB, 8.13% for PPI, and 51.54% for INT. Therefore,
AUC_LP achieves the best accuracy, as measured by AUC.
Besides AUC, precision is another important metric. The
mean precision of the algorithms in the 10-fold CV is shown
in Table 3. The highest precision achieved by the algorithms
in each dataset is shown in bold.
28130
TABLE 3. Comparison of the accuracy of algorithms in terms of precision.
The results from the 11 methods show that the AUC_LP
algorithm outperforms the other ten algorithms in terms of
precision. Specifically, compared with those of CN, the pre-
cision in improved by 13.27% for USAir, 18.19% for PB,
164.04% for PPI and 1104.80% for INT. When compared
against Salton, Jaccard and LHN_I, the improvement is even
greater. AUC_LP’s performance for dissimilar networks is
especially strong: it is either perfect or almost perfect. By con-
trast, no other algorithms are successful.
In addition, we also compare our method with some
representative methods (mainly based on community detec-
tion, clustering coefficient, and time window) to demon-
strate the performance of our method. Figure 5 presents
the prediction results four networks obtained with seven
research approaches (S-Rank, CMA-ES, VB, Node-LP, WT,
ConClus-LP, and CD-LP). The study is performed with 90%
of the data collected from the networks used as the training
set and the rest used for the test set.
In addition, the size of the training dataset was increased
from 50% to 90%. The results of the different methods with
different training dataset sizes are shown in Figure 6 and
Figure 7.
According to the test results, the efficiency of the
approaches can be improved by selecting appropriate network
characteristics. Moreover, the prediction accuracy is unre-
lated to the size of the network. Specifically, the prediction
VOLUME 6, 2018
B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization
FIGURE 5. The comparison of the accuracy of algorithms according to AUC and Precision in different
datasets.
FIGURE 6. Comparison of the average AUC of different algorithms with different training dataset sizes.
accuracy can be improved if the study targets a set of net-
works.
Extensive experiments have been performed to observe
networks and to verify the validity of the AUC_LP algorithm
compared with other link prediction algorithms. The results
of this study can be used to assess other network-prediction
algorithms. Moreover, this study confirms previous research
results and simultaneously provides a reference for the
study of information screening technology and relevant
professionals.
A. ROBUSTNESS ANALYSIS OF THE ALGORITHM
Robustness refers to the capacity to endure perturbations and
failures and serves as the key attribute to many complex
mechanisms, including complex networks.
Studies of the robustness of complex mechanisms are of
great importance in many fields [37]. Robustness serves as an
important attribute of ecosystems in the field of ecology and
VOLUME 6, 2018
can provide insight into reactions to disturbances, such as the
extinction of a species [38]. Network robustness is helpful
for biologists to explore mutations and diseases and also
for the recovery of mutations [39]. The network robustness
principles help to promote the understanding of the risks and
stability of banking systems [40]. Additionally, in engineer-
ing, network robustness is helpful to assess the resilience of
infrastructure mechanisms, for example, power grids or the
Internet [41].
Most networks are not random networks. Although the
number of nodes is small, there can be a massive num-
ber of connections. The majority of networks are quite
small, in general, consistent with the law promoted by
Zipf [42]. A complex network with a power law dis-
tribution and degree distribution is called a scale-free
network.
Scale-free networks are seldom impacted by random fail-
ures. Usually, the small nodes are randomly selected. Even
28131
 B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization

FIGURE 7. Comparison of the average precision of different algorithms with different training dataset sizes.

FIGURE 8. The relationship between the degrees of nodes and the number of nodes.

when these nodes are randomly removed, the basic connectiv-
ity is maintained. A scale-free network is highly vulnerable to
deliberate attacks compared with random networks because
of the non-uniformity, fault tolerance and survivability. The
nodes of a scale-free network do not have to be removed if
the strongest central nodes with the maximum connectivity in
the system maintain a strong influence on the connectivity of
the whole network and satisfy the critical point. Afterward,
the network is separated into isolated islands that are inca-
pable of contacting each other.
Thus, many studies have proposed robust evaluation
indexes and designs to prevent intentional and robust attacks.
For instance, De Meo et al. [43] found that graph robust-
ness can be predicted rapidly via the Randic index. Alenazi
and Sterbenz [44] claimed that adding links to maintain the
balance of the link-betweenness would lead to the best net-
work resilience against the aforementioned attacks among
the robustness metrics studied. Liu et al. [45] created a new
MA-CR, namely, the memetic algorithm, through a two-level
learning strategy to promote network community robustness
while maintaining the community structure and degree distri-
bution with the goal of decreasing two-level targeted attacks.
Research on real-world networks and scale-free networks
has assessed the stability and effectiveness of the proposed
algorithms relative to other high-end algorithms. Baig and
Akoglu [46] conducted the first large-scale empirical corre-
lation analysis of attack strategies, i.e., the node importance
metrics they adopt for the graph robustness.
This task is approached in three ways and illustrates that
some computationally complex strategies can be closely pre-
dicted by simpler strategies. Moreover, a few strategies can
be adopted to serve as a close proxy for consensus.
The original dataset is analyzed first during the robustness
analysis. Additionally, the relationship between the degrees
of the nodes and the number of nodes in the original network
must be analyzed. Figure 8 presents the experimental results.

28132 VOLUME 6, 2018

B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization
FIGURE 9. Analysis of the nodes’ degree in the training set and probe set.
FIGURE 10. The error of W between adjacent time periods.
Figure 8 shows an inverse proportional relationship
between the node degree and number of nodes. Therefore,
in these networks, when the node degree is small, there is
a greater number of occurrences. By contrast, a large node
degree is associated with a smaller number of occurrences.
The distribution of the four network degrees generally
follows the Zipf law. Therefore, the four network degrees
mentioned above can be regarded as scale-free networks.
Malliaros et al. [37] considered scale-free networks under
deliberate and random attacks. Furthermore, the robustness
of the network connectivity with the two class node removal
strategies was compared. The results showed that scale-free
networks are robust to random node failures, so the robustness
of these four networks can be analyzed preliminarily.
Moreover, because connections between these objects may
exist in both the test set and the training set, we need to
consider the number of node links in the training set, which
serves as the abscissa, and the node links in the test set, which
VOLUME 6, 2018
serves as the ordinate. Thus, the partitioning of the dataset can
be seen as a network attack. The ratio of training set is set to
50%, and correlation of the node degree is obtained randomly.
Figure 9 (a-d) presents the results of these four datasets. The
scatter plots in panels (a) to (d) demonstrate the node degree
in the NP, that is, the training set, and NT, that is, the probe
set, when the dataset is randomly classified. Panel (e) and
panel (h) present the relationship between the changes in the
ratio of the training set and the nodes’ degrees in the probe
set and training set. The red line corresponding with the sets
represents random classification.
Figure 9 shows that NP and NT have an almost linear
relationship because the links are selected as training set and
probe set with equal probability. Additionally, C–Node is
approximately 1 in the datasets divided randomly as training
set ratio changes.
By comparing the network topology properties of the orig-
inal dataset and the training set, we can conclude that the
28133
 B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization
FIGURE 11. The change in the AUC of the AUC_LP algorithm with NC .
scheme used to divide datasets in a deliberate attack does
not affect the original network degree distribution. The basic
characteristics of the original network can be maintained
while achieving high robustness.
B. CONVERGENCE ANALYSIS OF THE ALGORITHM
We also study the convergence conditions of the algorithm.
Formula (28) is convex because it is a quadratic loss function.
Therefore, a gradient descent strategy (31) can be used to test
the convergence of the extreme points of Formula (28).
The rate of convergence depends on the step size µt . If the
step size is too large, convergence to the optimal solution
may be difficult, but a small step size may lead to slow
convergence. To achieve a better convergence rate, we use a
variable step size determined by the Armijo rule [47].
In the process of the iteration of weight matrix W , we cal-
culate the error of W between
adjacent time
2periods. The error the findings in this article are beneficial because they can
formula is: Error (t) =
W (t) − W (t−1)
F . Then, we plot help Internet retailers to better utilize the current forecasting
the relation between the error and the iteration number of
the algorithm for different datasets. The results are shown
in Figure 10.
Figure 10 shows that in the iteration of weight matrix W ,
the error trend presents a sharp decrease and then rapidly
reaches the state of convergence. Therefore, the iteration time
complexity of this step is O(NC), which is equivalent to O(1).
Finally, we study how the AUC of the AUC_LP algorithm
under optimal parameters λ and η changes with the number
of iterations NC for different datasets. The results are shown
in Figure 11.
From Figure 11, we can see that the AUC of the AUC_LP
algorithm shows an increasing trend and always maintains a
high value as NC increases in the four datasets. For example,
the AUC value remains constant at approximately 0.935 when
the number of iterations is greater than 80 for the USAir
dataset. For the PB dataset, the AUC is maintained at approxi-
mately 0.940 when the number of iterations is greater than 75.
28134
Therefore, the AUC_LP algorithm not only achieves higher
AUC values on real networks than those of other algorithms
but is also robust.
VII. CONCLUSION
Various areas, including anthropology, sociology, computer
science, and information science, have recently paid great
attention to the issue of link prediction due to the current
large number of network statistics in electronic form. A novel
framework is proposed to assess the performance of a link
prediction algorithm in this article. In the algorithm, the score
matrix is effectively decomposed with the weight matrix and
the adjacency matrix. We use the AUC metric as the target
function of optimization and obtain the optimal weight matrix
via the gradient descent algorithm. According to the empirical
findings, the AUC_LP algorithm yields high-quality forecast-
ing results compared with those of other algorithms. Thus,
approaches.
REFERENCES
[1] B. Barzel and A.-L. Barabási, ‘‘Network link prediction by global silencing
of indirect correlations,’’ Nature Biotechnol., vol. 31, no. 8, pp. 720–725,
2013.
[2] L. Zhu, D. Guo, J. Yin, G. Ver Steeg, and A. Galstyan, ‘‘Scalable temporal
latent space inference for link prediction in dynamic social networks,’’
IEEE Trans. Knowl. Data Eng., vol. 28, no. 10, pp. 2765–2777, Oct. 2016.
[3] A. Papadimitriou, P. Symeonidis, and Y. Manolopoulos, ‘‘Fast and accurate
link prediction in social networking systems,’’ J. Syst. Softw., vol. 95, no. 9,
pp. 2119–2132, 2012.
[4] J. Tang, S. Chang, C. Aggarwal, and H. Liu, ‘‘Negative link prediction in
social media,’’ in Proc. 8th ACM Int. Conf. Web Search Data Mining, 2015,
pp. 87–96.
[5] K. Jahanbakhsh, V. King, and G. C. Shoja, ‘‘Predicting missing contacts
in mobile social networks,’’ Pervasive Mobile Comput., vol. 8, no. 5,
pp. 698–716, 2012.
[6] W. Liang, X. He, D. Tang, and X. Zhang, ‘‘S-Rank: A supervised rank-
ing framework for relationship prediction in heterogeneous information
networks,’’ in Proc. Int. Conf. Ind., Eng. Other Appl. Appl. Intell. Syst.,
Springer, 2016, pp. 305–319.
VOLUME 6, 2018
B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization
[7] H. Hwang, D. Petrey, and B. Honig, ‘‘A hybrid method for protein–protein
interface prediction,’’ Protein Sci., vol. 25, no. 1, pp. 159–165, 2016.
[8] P. Wang, B. Xu, Y. Wu, and X. Zhou, ‘‘Link prediction in social networks:
The state-of-the-art,’’ Sci. China Inf. Sci., vol. 58, no. 1, pp. 1–38, 2015.
[9] M. E. J. Newman, ‘‘Clustering and preferential attachment in growing
networks,’’ Phys. Rev. E, Stat. Phys. Plasmas Fluids Relat. Interdiscip.
Top., vol. 64, no. 2, p. 025102, 2001.
[10] R. Milo, S. Shen-Orr, S. Itzkovitz, N. Kashtan, D. Chklovskii, and U. Alon,
‘‘Network motifs: Simple building blocks of complex networks,’’ Science,
vol. 298, no. 5594, pp. 824–827, 2002.
[11] S. A. Golder and S. Yardi, ‘‘Structural predictors of tie formation in twitter:
Transitivity and mutuality,’’ in Proc. IEEE 2nd Int. Conf. Social Comput.
(SocialCom), Aug. 2010, pp. 88–95.
[12] D. Yin, L. Hong, X. Xiong, and Brian D. Davison, ‘‘Link formation
analysis in microblogs,’’ in Proc. 34th Int. ACM SIGIR Conf. Res. Develop.
Inf. Retr., 2011, pp. 1235–1236.
[13] D. Garlaschelli and M. I. Loffredo, ‘‘Patterns of link reciprocity in directed
networks,’’ Phys. Rev. Lett., vol. 93, no. 26, p. 268701, 2004.
[14] Y.-X. Zhu, X.-G. Zhang, G.-Q. Sun, M. Tang, T. Zhou, and Z.-K. Zhang,
‘‘Influence of reciprocal links in social networks,’’ PLoS ONE, vol. 9, no. 7,
p. e103007, 2014.
[15] D. M. Romero and J. M. Kleinberg, ‘‘The directed closure process in hybrid
social-information networks, with an analysis of link formation on twitter,’’
in Proc. ICWSM, 2010, pp. 138–145.
[16] D. Schall, ‘‘Link prediction in directed social networks,’’ Social Netw.
Anal. Mining, vol. 94, no. 1, p. 157, 2014.
[17] Q.-M. Zhang, L. Lü, W.-Q. Wang, Y. Xiao, and T. Zhou, ‘‘Potential theory
for directed networks,’’ PLoS ONE, vol. 8, no. 2, p. e55437, 2013.
[18] R. N. Lichtenwalter and N. V. Chawla, ‘‘Vertex collocation profiles: Sub-
graph counting for link analysis and prediction,’’ in Proc. 21st Int. Conf.
World Wide Web, 2012, pp. 1019–1028.
[19] X. Li and H. Chen, ‘‘Recommendation as link prediction in bipartite
graphs: A graph kernel-based machine learning approach,’’ Decision Sup-
port Syst., vol. 54, no. 2, pp. 880–890, 2013.
[20] S. Scellato, A. Noulas, and C. Mascolo, ‘‘Exploiting place features in
link prediction on location-based social networks,’’ in Proc. 17th ACM
SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2011, pp. 1046–1054.
[21] J. Scripps, P.-N. Tan, F. Chen, and A.-H. Esfahanian, ‘‘A matrix alignment
approach for link prediction,’’ in Proc. 19th Int. Conf. Pattern Recognit.
(ICPR), Dec. 2008, pp. 1–4.
[22] M. Pujari and R. Kanawati, ‘‘Link prediction in complex networks by
supervised rank aggregation,’’ in Proc. IEEE 24th Int. Conf. Tools Artif.
Intell. (ICTAI), Nov. 2012, pp. 782–789.
[23] K.-Y. Chiang, N. Natarajan, A. Tewari, and I. S. Dhillon, ‘‘Exploiting
longer cycles for link prediction in signed networks,’’ in Proc. 20th ACM
Int. Conf. Inf. Knowl. Manage., 2011, pp. 1157–1162.
[24] C. A. Bliss, M. R. Frank, C. M. Danforth, and P. S. Dodds, ‘‘An evolution-
ary algorithm approach to link prediction in dynamic social networks,’’
J. Comput. Sci., vol. 5, no. 5, pp. 750–764, 2014.
[25] R. R. Sarukkai, ‘‘Link prediction and path analysis using Markov chains,’’
Comput. Netw., vol. 33, nos. 1–6, pp. 377–386, 2000.
[26] Y. Bhawsar, G. S. Thakur, and R. S. Thakur, ‘‘Model for link prediction in
social network by genetic algorithm approach,’’ Data Mining Knowl. Eng.,
vol. 7, no. 5, pp. 191–196, 2015.
[27] Y. Zhao, S. Li, C. Zhao, and W. Jiang, ‘‘Link prediction via a neighborhood-
based nonnegative matrix factorization model,’’ in Proc. 3rd Int. Conf.
Commun., Signal Process., Syst., 2015, pp. 603–611.
[28] M. Pujari and R. Kanawati, ‘‘Supervised rank aggregation approach for
link prediction in complex networks,’’ in Proc. 21st Int. Conf. World Wide
Web, 2012, pp. 1189–1196.
[29] B. Ermis and A. T. Cemgil. (2014). ‘‘A Bayesian tensor factorization
model via variational inference for link prediction.’’ [Online]. Available:
https://arxiv.org/abs/1409.8276
[30] C. Dai, L. Chen, and B. Li, ‘‘Link prediction based on sampling in complex
networks,’’ Appl. Intell., vol. 47, no. 1, pp. 1–12, 2017.
[31] L. Linyuan, P. Liming, Z. Tao, Z. Yi-Cheng, and H. E. Stanley, ‘‘Toward
link predictability of complex networks,’’ Proc. Nat. Acad. Sci. USA,
vol. 112, no. 8, pp. 2325–2330, 2014.
[32] L. Li, L. Qian, J. Cheng, M. Ma, and X. Chen, ‘‘Accurate similarity index
based on the contributions of paths and end nodes for link prediction,’’
J. Inf. Sci., vol. 41, no. 2, pp. 167–177, 2015.
VOLUME 6, 2018
[33] Q.-M. Zhang, A. Zeng, and M.-S. Shang, ‘‘Extracting the information
backbone in online system,’’ PLoS ONE, vol. 8, no. 5, p. e62624, 2013.
[34] G. P. Clemente and R. Grassi, ‘‘Directed clustering in weighted net-
works: A new perspective,’’ Chaos, Solitons Fractals, vol. 107, pp. 26–38,
Feb. 2018.
[35] S. Agreste et al., ‘‘An empirical comparison of algorithms to find commu-
nities in directed graphs and their application in Web data analytics,’’ IEEE
Trans. Big Data, vol. 3, no. 3, pp. 289–306, Sep. 2017.
[36] C. P. Santos, D. M. Carvalho, and M. C. V. Nascimento, ‘‘A consensus
graph clustering algorithm for directed networks,’’ Expert Syst. Appl.,
vol. 54, pp. 121–135, Jul. 2016.
[37] F. D. Malliaros, V. Megalooikonomou, and C. Faloutsos, ‘‘Estimating
robustness in large social graphs,’’ Knowl. Inf. Syst., vol. 45, no. 2,
pp. 645–678, 2015.
[38] R. V. Solé and M. Montoya, ‘‘Complexity and fragility in ecological
networks,’’ Proc. Roy. Soc. London B, Biol. Sci., vol. 268, no. 1480,
pp. 2039–2045, 2001.
[39] A. E. Motter, N. Gulbahce, E. Almaas, and A.-L. Barabási, ‘‘Predicting
synthetic rescues in metabolic networks,’’ Mol. Syst. Biol., vol. 4, no. 1,
p. 168, 2008.
[40] A. G. Haldane and R. M. May, ‘‘Systemic risk in banking ecosystems,’’
Nature, vol. 469, no. 7330, pp. 351–355, 2011.
[41] R. Albert, I. Albert, and G. L. Nakarado, ‘‘Structural vulnerability of the
North American power grid,’’ Phys. Rev. E, Stat. Phys. Plasmas Fluids
Relat. Interdiscip. Top., vol. 69, no. 2, p. 025103, 2004.
[42] M. E. J. Newman, ‘‘Power laws, Pareto distributions and Zipf’s law,’’
Contemp. Phys., vol. 46, no. 5, pp. 323–351, 2005.
[43] P. De Meo et al., ‘‘Estimating graph robustness through the Randic index,’’
IEEE Trans. Cybern., vol. 10, pp. 1–14, 2017.
[44] M. J. F. Alenazi and J. P. G. Sterbenz, ‘‘Evaluation and comparison of
several graph robustness metrics to improve network resilience,’’ in Proc.
7th Int. Workshop IEEE Reliable Netw. Des. Modeling (RNDM), Oct. 2015,
pp. 7–13.
[45] W. Liu, M. Gong, S. Wang, and L. Ma, ‘‘A two-level learning strategy based
memetic algorithm for enhancing community robustness of networks,’’ Inf.
Sci., vol. 422, pp. 290–304, Jan. 2018.
[46] M. B. Baig and L. Akoglu, ‘‘Correlation of node importance measures: An
empirical study through graph robustness,’’ in Proc. 24th Int. Conf. World
Wide Web, 2015, pp. 275–281.
[47] L. Armijo, ‘‘Minimization of functions having Lipschitz continuous first
partial derivatives,’’ Pacific J. Math., vol. 16, pp. 1–3, Nov. 1966.
BOLUN CHEN received the Ph.D. degree from the
Nanjing University of Aeronautics and Astronau-
tics, Nanjing, China, in 2016. He was a Visiting
Scholar with the University of Fribourg, Fribourg,
Switzerland. He is a Professor with the Huaiyin
Institute of Technology, Lianyungang, China. His
research interests include link prediction, recom-
mender systems, and data mining. He is currently
a reviewer of the Journal of Information Science,
World Wide Web Journal, and Reliability Engi-
neering & System Safety.
YONG HUA received the M.S. degree from the
Huaiyin Institute of Technology, Lianyungang,
China. His research interests include maximiza-
tion of influence of complex network and recom-
mender systems.
28135
 B. Chen et al.: Link Prediction on Directed Networks Based on AUC Optimization

YAN YUAN received the M.S. degree from the YING JIN is currently a Professor with the Huaiyin
Huaiyin Institute of Technology, Lianyungang, Institute of Technology, Lianyungang, China. Her
China. Her research interests include complex net- research interests include deep learning, recom-
work, data analysis, and machine learning. mender systems, and data analysis.

28136 VOLUME 6, 2018