Received October 5, 2019, accepted October 18, 2019, date of publication October 24, 2019, date of current version

November 7, 2019.
Digital Object Identifier 10.1109/ACCESS.2019.2949465

Social Network-Oriented Learning Agent for

Improving Group Intelligence Coordination
JINGTAO QI , LIANG BAI, AND YANDONG XIAO
College of System Engineering, National University of Defense Technology, Changsha 410073, China
Corresponding author: Yandong Xiao (xiaoyandong08@gmail.com)
This work was supported by the National Natural Science Foundation of China under Grant 61902418.

ABSTRACT Group coordination is embedded in social networks, which aims to reach a consensus or solve
conflicts among social individuals. Recently, improving performance has become a challenging task and
drawn considerable interest in this field. To eliminate barriers of group coordination, we abstract the problem
into a networked color coordination game and introduce learning agents encoded with Q-learning to collect
local information and learn local individual behaviors. We first show that learning agents can effectively
improve the group coordination performance. By properly selecting parameters, we find that learning agents
acting with low greedy parameter levels and placed in central locations can drastically accelerate group
coordination. Greedy parameters and positions are vital to the problem. Finally, we indicate that learning
agents have a direct effect on neighbor individuals and an indirect effect on non-neighbor individuals to
act on the coordination network. Moreover, we propose a conflict relationship index which is the average
rounds required for solving conflicts and indicate learning agents solve conflicts that cannot be solved by
an individual. Hence, learning agents create further benefits to group coordination in these complex social
networks. This paper provides a detailed analysis of the learning agents in a networked color coordination
game and shows that artificial intelligence provides a solution to the group coordination problem.

INDEX TERMS Group coordination, learning agents, reinforcement learning.

I. INTRODUCTION and their group, and the inability of individuals to effectively

With the rapid development of network technology, social
network platforms are being widely used as the carrier of
competition and cooperation [1]. Therefore, improving com-
petition and cooperation among intelligent individuals to
satisfy the needs of society represents a challenging and fasci-
nating focus area [2], [3]. Because group cooperation requires
the sacrifice of individual interests to achieve a higher overall
interest [4]–[7], it is often hard to achieve the maximum col-
lective societal interest [8]. Additionally, individual ‘‘blind’’
phenomena, for example, decisions completed with limited
information, make it difficult for individuals to weigh the
relationship between individual interests and collective inter-
ests [7], [9], namely, the social dilemma [10], [11]. However,
even when the social dilemma can be solved, there remains
another obstacle–group coordination. Groups often have dif-
ficulty achieving the maximum collective interest. The pos-
sible reasons for this problem include conflicting interests
among individuals, conflicting interests between individuals
The associate editor coordinating the review of this manuscript and
approving it for publication was Lin Wang . tion and collective behavior for promoting the cooperation
coordinate their actions globally [12]. Even if all individu-
als effectively coordinate with their neighbors, this may not
result in the maximum collective interest for the group, which
is a sub-optimization problem [11], [13]–[15]. However, with
the development of technology and the constantly deepening
understanding of social systems, the complex group coordi-
nation problem can become manageable in society [16], [17].
To solve the group coordination problem and satisfy the
needs of society, experts in various fields (including biolo-
gists, physicists, and psychologists) are focusing on this prob-
lem [18]. There exists many original and influential works in
this field from the interdisciplines among statistical physics,
network science, game theory and so on. Perc et al. [19]
reviewed the evolutionary dynamics of group interactions
on structured populations. Later, Perc et al. [20] reviewed
the statistical physics research of human cooperation, focus-
ing on spatial pattern formation, the spatiotemporal dynam-
ics of observed solutions, and self-organization that may
either promote or hinder socially favorable states. Besides,
Wang et al. [21] highlighted the importance of pattern forma-

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/
156526 VOLUME 7, 2019
J. Qi et al.: Social Network-Oriented Learning Agent for Improving Group Intelligence Coordination
under adverse conditions, as well as the synergy effects.
In the application of human lives, Helbing et al. [22] showed
that complexity science can mitigate the undesirable cascade
effects and accelerate the emergence of favorable kinds of
self-organization in the system. Within the field of control
science and engineering, many works have emerged. Con-
sidering network structures, Enemark et al. [23] explained
why the negative effect of constraint outweighs the posi-
tive effect of communication in complex coordination tasks.
Rand et al. [24] provided an experimental demonstration of
the power of a static network structure in stabilizing human
cooperation, which is consistent with a quantitative evolu-
tionary game theoretical prediction. In a more recent work,
Enemark et al. [25] showed that connections can influence
coordination in two different ways, wherein connections that
reduce the number of solutions to a problem play a nega-
tive role in coordination and connections facilitate coordina-
tion. To improve this problem, Peysakhovich and Lerer [26]
constructed consequentialist conditional cooperation in a
large class of games that can construct good strategies by
conditioning one’s behavior solely based on outcomes using
deep reinforcement learning techniques and demonstrated
that the method is effective in social dilemmas beyond sim-
ple matrix games. Shirado and Christakis [12] researched
the group coordination problem and discover a surprising
solution, namely, that locally noisy agents improve global
human coordination and help the group achieve the max-
imum collective interests. In the field of computer tech-
nology, this problem is often abstracted as a multi-agent
problem in research. Salazar et al. [27] presented a dis-
tributed approach to coalition emergence in large-scale multi-
agent systems which achieves full cooperation over differ-
ent complex networks. Peleteiro et al. [28] presented a new
mechanism based on three main pillars (indirect reciprocity,
coalitions and rewiring) to improve cooperation among
self-interested agents placed in a reputation-based game.
Shibusawa et al. [29] proposed a behavioral strategy called
the expectation of cooperation, which is simple and easy to
implement but, nevertheless, can spread over and maintain
cooperation in agent networks under certain conditions. How-
ever, these studies are not applicable to solving the social
dilemma of coordination in some agent networks, or they are
not applicable to actual social systems.
According to the above research, if locally noisy agents
improve global human coordination and help the group
achieve the maximum collective interests, learning agents
can bring about a better outcome. Artificial intelligence has
recently arisen as a widespread theory, and it applies to many
problems. In this paper, we introduce learning agents based
on local information into a networked color coordination
game and provide a solution to the group coordination prob-
lem. The major contributions of this paper can be summarized
as follows.
1) We abstract the group coordination problem into a
networked color coordination game [30]. Based on this
model, we introduce learning agents (q-bots) encoded
VOLUME 7, 2019
with Q-learning based on local information into the
network in exchange for the same number of con-
ventional agents. In addition, we design clear action
rules for q-bots and conventional agents. We classify
the conflicts into resolvable conflicts and unresolvable
conflicts according to the action rules of conventional
agents.
2) We carry out comparison experiments on the network
with and without q-bots. By analyzing the comparison
results, we find that the main cause of the group coor-
dination problem is that conventional agents cannot
reach the collective objective by eliminating only their
local conflicts in the individual perception. In addition,
we prove the effectiveness of q-bots in solving the
group coordination problem.
3) To evaluate the effect of parameters on q-bots in solving
the group coordination problem, we evaluate 10 condi-
tions: 1 control condition without q-bots and 9 condi-
tions that combine the greedy parameters and network
locations of q-bots. In this experiment, we find that
the greedy parameter and network location play an
important role in solving the problem. We show that
q-bots acting with low levels of greedy parameters
placed in central locations meaningfully improve the
group coordination by more than 80%.
4) To better understand the mechanism of q-bots in solv-
ing the group coordination problem, we carry out
2 experiments. We calculate the degree of node color
change for every 100 generations in one experiment.
Based on this experiment, we show that q-bots act
on the coordination network by affecting not only
neighbor individuals but also non-neighbor individuals.
Then, we propose a conflict relationship index in which
the average rounds required for solving conflicts and
indicate q-bots eliminate conflicts that cannot be solved
by individuals.
The remainder of this paper is organized as follows.
In Section II, a networked color coordination game model
is presented to characterize group coordination. Based on
this, in Section III, we introduce q-bots into the model and
formulate the action rules of conventional agents and q-bots.
To verify and explore the effectiveness of q-bots in solving
the group coordination problem, numerical experiments are
performed on a networked color coordination game case, and
the results are presented in Section IV. Our conclusions and
final discussion are provided in Section V.
II. GROUP COORDINATION MODEL
The group coordination problem is complex and unique
in social networks. Therefore, to solve this problem,
the abstracted and representative model is critical [31]. In this
section, a networked color coordination game model is pre-
sented to characterize this problem.
Group coordination can be characterized by a networked
color coordination game. The color coordination game is an
156527
 J. Qi et al.: Social Network-Oriented Learning Agent for Improving Group Intelligence Coordination
undirected network G = (V , E, C, S) [32], where V =
Vc1 (r) ∪ Vc2 (r) ∪ Vc3 (r) ∪ · · · ∪ Vck (r) = {v1 , v2 , v3 , . . . , vz }
denotes the node set, E = {e1 , e2 , e3 , . . . , em } ⊆ V ×
V represents information flows between agents, set C =
{c1 , c2 , c3 , . . . , ck } shows the alternative colors and the sorted
sequence S(r) = (s1 (r), s2 (r), s3 (r), . . . , sz (r)) records the
colors of the nodes in round r. All agents are divided into
k sets Vcj (r)(j = 1, 2, 3, . . . , k) according to their color in
round r. For example, if the color of one node vi is cj in
round r, si (r) = cj , this node belongs to set Vcj (r). Here,
z = |V | = |S|, m = |E| and k = |C| express the numbers of
nodes, edges, and colors in the networked color coordination
game, respectively. In the model, edges that connect nodes
with the same color are considered conflicts, while other
edges are considered conflict-free. As noted, agents act using
local information in the game. Specifically, in this model,
agents can see only the colors of neighbors with whom they
share the same edge in addition to their own color, which is
also called the ‘‘blind’’ phenomenon.
Based on social network characteristics, we build the
model network according to scale-free networks [33],
namely, the network structure is created from the beginning
by randomly attaching new nodes (each with l links) to
existing nodes based on the degree of existing nodes [12].
The topology of networks is fixed during the whole color
coordination game. In our network, the adjacency index is
( 3:
1 if vi and vj are connected
d(vi , vj ) = (1)
0 otherwise.
Before the game, we colorize each node by randomly
selecting a color with a uniform distribution in the set C.
For example, we randomly select one color cx to colorize one
node vi , that is,
 10: end if
cx ∈ U (C)

si (0) = cx (2)

Vcx (0) = Vcx (0) ∪ {vi }.

During the game process, each agent acts simultaneously
according to action rules, which are introduced in Section III,
in each round. For example, the color of one node vi is cx in
round r, and then this node changes its color to cy based on
the action rules in round r + 1. This process is
 
si (r) = cx
 si (r + 1) = cy
 selected color with a uniform distribution in set C 0 . By estab-
vi ∈ Vcx (r) → Vcx (r + 1) = Vcx (r) − {vi } (3)


/ Vcy (r)
vi ∈


Vcy (r + 1) = Vcy (r) ∪ {vi }.
When the game is finished, the condition that should be met
is the existence of no conflicts in the network. Specifically,
any two nodes that share the same edge have different colors.
This condition is
∀ cx ∈ C, ∀ vi , vj ∈ Vcx (r), d(vi , vj ) 6 = 1. (4)
As the condition of the game finishing, it describes the
objective of group coordination that achieves the maximum
156528
collective interest of the group, which is characterized as the
state in which there is no conflict in the game. In addition,
the rounds of the game are considered as a problem-solving
evaluation index, and we set an upper limit u for the rounds
of the game in our experiment.
III. THE ACTION RULES FOR AGENTS
To mimic the individuals in real-life group coordination,
we formulate the action rules for conventional agents and
q-bots.
A. THE CONVENTIONAL AGENTS ACTION RULE
We formulate the rule for conventional agents as follows. For
one conventional agent connected with greater than or equal
to one conflict edge in round r, this agent randomly selects
from the whole color set if the random number is less than the
random parameter of conventional agents and the colors of its
neighbors occupy all color in set C; otherwise, the agent ran-
domly selects from the set including the color not occupied
by its neighbors. This agent will not change its color if it is
connected without conflict edges.
Algorithm 1 The Conventional Agent Action Rule
1: if ∃vj ∈ Nvi , si (r) = sj (r)(j 6 = i) then
Nv
2: if Pc ∈ U [0, 1] ≤ P or ∀cj ∈ C, Vcj i (r) 6= ∅ then
C0 = C
4: else
Nv
5: C 0 = {cj |Vcj i (r) = ∅, cj ∈ C}
6: end if
7: si (r + 1) ∈ U (C 0 )
8: else
9: si (r + 1) = si (r)
Algorithm 1 shows the above-mentioned conventional
agent action rule, where set Nvi includes the neighbors of one
node vi and vi , Pc ∈ U [0, 1] represents a random number Pc
generated with a uniform distribution in the interval [0, 1],
the constant P represents the random parameter of conven-
tional agents (characterized by the uncertainty of individuals
Nv
in a practical problem), set Vcj i (r) denotes nodes colorized
with cj that belong to set Nvi and U (C 0 ) denotes a randomly
lishing this rule, we consider the action of the individual in
reality, such as the individual uncertainty and the individual
expectation of coordinating with neighbors.
When the condition meets the requirement of Pc ∈
U [0, 1] ≤ P, Fig. 1(a) and (b) show examples of this action
rule. In Fig. 1(a), the center node changes its color to pur-
ple for the next round because it meets the conditions that
Nv Nv
Vcj i (r) 6= ∅(cj = green, orange) and Vcj i (r) = ∅(cj =
purple) for the current round. In Fig. 1(b), the center node
changes to the other color generated with a uniform distribu-
tion of the set C = {purple, green, orange} for the next round
VOLUME 7, 2019
J. Qi et al.: Social Network-Oriented Learning Agent for Improving Group Intelligence Coordination
FIGURE 1. The examples of action rules for conventional agents and the
rule-induced conflicts. In this example, K = 3, C = {purple, green,
orange}, black (or red) edges denote conflict-free (or conflict) cases.
(a), (b) Examples for action rules for conventional agents. (c) Example for
resolvable conflicts. (d) Example for unresolvable conflicts.
Nv
because the node meets the condition that Vcj i (r) 6 = ∅(cj =
green, orange, purple) for the current round.
In addition, conflicts are classified as resolvable and unre-
solvable conflicts based on this rule [12]. When the condition
meets the requirement of Pc ∈ U [0, 1] ≤ P, the resolvable
conflicts can be eliminated when either node selects from the
set composed of the color not occupied by neighbors, but
the unresolvable conflicts cannot. Figure. 1(c) and (d) show
examples of resolvable conflicts and unresolvable conflicts,
respectively. In Fig. 1(c), the 2 center nodes change their
color to purple and orange based on this rule for the next
round. By doing so, this conflict is eliminated; therefore, it is
considered a resolvable conflict. In Fig. 1(d), the 2 center
nodes change their color to purple based on this rule for the
next round. However, this conflict sinks into infinite loop;
therefore, it is considered an unresolvable conflict. In the
following experiments, conflicts are regarded as unsolvable
conflicts when two agents have only one color to select
according to conventional agent rules; otherwise, they are
regarded as resolvable conflicts.
B. THE LEARNING AGENT ACTION RULE
To improve the group coordination problem, we introduce
q-bots into the color coordination game in exchange for the
same number of conventional agents. We encode q-bots with
Q-learning, which is a model-free reinforcement learning
algorithm [34]. The Q-learning algorithm includes an agent,
VOLUME 7, 2019
a set of states W , a set of actions A, a reward b and a table T
that records the expected reward of an action taken in a given
state. The core algorithm is as following:
T 0 (wr , ar ) ← (1−α)T (wr , ar )+α[br +γ max T (wr+1 , a)],
a
(5)
where T 0 denotes the updated table, br is the reward received
when moving from state wr to state wr+1 , α is the learning
rate and γ is the discount factor. To select an optimal action
for any given state, (5) is used to iteratively update T during
the learning process.
In our model, we use a multi-agent Q-learning method
[34], [35], which exchanges h q-bots for the same number of
conventional agents in our model. One q-bot located at node
vi corresponds to one table Tvi . For multi-agent Q-learning,
the state of one q-bot is a sorted sequence S Nvi (r) that records
the color of the q-bot and its neighbors. This state records the
location information of neighbors corresponding to the q-bot
and the color information of neighbors. The action set of one
q-bot is set C, which denotes the colors that can be selected by
the q-bots. When the network composed of one q-bot and its
neighbors is conflict-free, the corresponding reward b equals
1; otherwise, it equals 0.
Algorithm 2 The Learning Agent Action Rule
1: if ∃vj ∈ Nvi , si (r) = sj (r)(j 6 = i) then
2: if εc ∈ U [0, 1] ≥ ε then
3: si (r + 1) = argmax c Tx (S Nvi (r), c)
4: else
Nv
5: if Pc ∈ U [0, 1] ≤ P or ∀cj ∈ C, Vcj i (r) 6= ∅ then
6: C0 = C
7: else
Nv
8: C 0 = {cj |Vcj i (r) = ∅, cj ∈ C}
9: end if
10: si (r + 1) ∈ U (C 0 )
11: end if
12: else
13: si (r + 1) = si (r)
14: end if
Based on the multi-agent Q-learning method, q-bots act
with a greedy strategy [36], and we set the greedy parameter
(ε). We clarify the action rules of the q-bots as follows. One
q-bot qi connected with greater than or equal to one conflict
edge selects the maximum expected reward of a color in a
given state if the random number is greater than ε; other-
wise, it selects based on the conventional agent action rule.
This q-bot does not change its color if it is not connected
with a conflict edge. Algorithm 2 shows these action rules.
Because each agent acts simultaneously according to action
rules, we can get the next state of one q-bot and update the
corresponding table Tvi after all agents choose actions.
Independently of the above-mentioned ε, we also manipu-
late the q-bot network location, including its random, central
and peripheral attributes. A peripheral location means that
156529
 J. Qi et al.: Social Network-Oriented Learning Agent for Improving Group Intelligence Coordination
q-bots are assigned to locations that have the lowest network
degree. Likewise, a central location means that q-bots are
assigned to a location with the largest network degree. A ran-
dom location means that q-bots are randomly assigned to the
network.
IV. CASE STUDY
To eliminate the barriers of group coordination, we introduce
q-bots into the networked color coordination game and per-
form a simulation experiment. In the experiment, where the
number of edges added each time during building a network
l = 2, the number of colors k = 3 (chromatic number
representing the minimum number of colors which is neces-
sary to color the networks to meet the conflict-free condition,
because there are no four interconnected nodes in the model
network constructed with l = 2), the random parameter of
conventional agents P = 0.1, the discount factor γ = 0.9 and
the learning rate α = 0.5. The networks constructed in the
following experiment are globally solvable. First, we analyze
the key cause of the group coordination problem and evaluate
the effectiveness of q-bots in solving this problem. Then,
we analyze the effect of key parameters on q-bots in the solu-
tion. Finally, we investigate the mechanism of q-bots in depth
to solve this problem, which includes the mode of q-bots
acting on coordination networks and conflict relationships.
A. EFFECTIVENESS ANALYSIS OF LEARNING AGENTS
In this experiment, by contrasting the performance with and
without q-bots in one game, we find the key cause of the
group coordination problem and present a hypothesis that
q-bots can effectively solve the group coordination prob-
lem. Additionally, we give a simple example to illustrate the
advantages of q-bots. Then, by contrasting the performance of
the network with and without q-bots in 10 different networks
with 1000 nodes, we validate this hypothesis. The q-bots are
assigned to center locations, and ε = 0.1 in this section. The
experimental results are presented in Fig. 2.
As shown in Fig. 2(a), for the game without q-bots,
although conventional agents aim to eliminate all conflicts
based on the rule, most of the conflicts remain in the network,
and the game rounds are too long. Therefore, we consider
the key cause of the group coordination problem as being
that conventional agents cannot achieve the collective goal by
eliminating only their local conflicts individually, called the
‘‘blind’’ phenomenon. In contrast, for the game with q-bots,
the game rounds are much shorter, and there are considerably
fewer conflicts. Based on the above analysis, we hypothe-
size that q-bots can effectively solve the group coordination
problem.
Compared with conventional agents, the advantage of
q-bots is that they can choose the action with the highest
reward according to the learning information, so that the
games with q-bots can achieve a conflict-free state with a
higher probability, and the disadvantage of learning agents
is that iterative learning takes time. To prove this point,
we present an example for illustration in Fig. 2(b). Games
156530
without q-bots always get stuck into an infinite loop of unre-
solvable conflicts. By contrast, games with q-bots can help to
avoid previous pitfall and easily achieve a conflict-free state
because they can induce other nodes to break the infinite loop
of unsolvable conflicts by choosing the color with the highest
reward obtained from iterative learning.
To validate the above-mentioned hypothesis, we perform
an experiment and show the result in Fig. 2(c). Because
games without q-bots often sink into an infinite loop of
unresolvable conflicts, games without q-bots rarely complete
within the upper limit, and the number of rounds in games
without q-bots does not change with increasing generations.
In contrast, the number of rounds in games with q-bots
decreases with increasing generations, and more and more
games complete within the upper limit. This result supports
our hypothesis that q-bots help to improve the group coordi-
nation problem.
B. ANALYSIS OF KEY LEARNING AGENT PARAMETERS
There are many learning agent parameters, but our model
is based on networks, and q-bots are based on the learn-
ing algorithm; therefore, we consider only the key param-
eters, including network location and ε. In this section,
we manipulate the q-bot network location and ε to evaluate
10 conditions, including 1 control condition without q-bots
and 9 condition combinations including 3 types of network
locations (random, central and peripheral) and 3 levels of ε
(0.1, 0.3 and 0.5).
Figure. 3 shows the results of this experiment as the
relationship between rounds and the survival proportion
in 20 games under the 10 conditions. With decreasing ε,
the survival proportion obviously decreases, which indicates
that q-bots with a smaller ε solve the group coordination
problem better. There is a clear distinction between the three
types of network locations. The central location case is the
best, and the peripheral location case is the worst. The ran-
dom location case lies between them. The q-bots with cen-
tral locations have many more neighbors than those with
the other two types of locations; therefore, we assume that
q-bots act on coordination networks by affecting not only
individuals to whom they are connected directly but also
other individuals. This assumption is proven in Section IV-C.
Additionally, the performance of the game with q-bots with
different ε levels and locations is either better or worse. For
example, when q-bots act with ε = 0.1 and are placed in
central location, games have the lowest proportion of survival
in 20 games, and q-bots improving the game by more than
80%. This result supports our conclusion in Section IV-A
that q-bots effectively solve the group coordination problem.
However, when q-bots act with ε = 0.5 and are placed
in a peripheral location, games have the maximum survival
proportion, which demonstrate q-bots even play a negative
role. Therefore, ε and location are vital to q-bots in solving
group coordination. By properly selecting the parameters,
q-bots can solve the problem with the best effect. Otherwise,
q-bots play a negative role in group coordination.
VOLUME 7, 2019
J. Qi et al.: Social Network-Oriented Learning Agent for Improving Group Intelligence Coordination

FIGURE 2. The experiments to compare game performance without q-bots or with q-bots. We provide two animations for the game–one with
q-bots and one without q-bots (see Supplementary Videos). (a) Bar chart of the number of conflicts and 6 snapshots of colored networks that
compare the games without q-bots and with q-bots in one game (z = 20, h = 3 and u = 200). The upper and lower transverse axes show the
results without q-bots and with q-bots, respectively. In the bar chart, dark blue and dark red bars show the number of unresolvable conflicts, and
light blue and red edges show the number of resolvable conflicts. In the 6 snapshots, circle nodes show conventional agents, square nodes show
q-bots, red edges and blue edges show conflicts, and black edges represent the absence of conflict. (b) Examples for the analysis of action rules
for conventional agents and q-bots. (c) The error bar for the rounds that compare the games without q-bots and with q-bots in 10 different
networks (each network was tested with 500 generations, z = 1000, h = 50 and u = 2000). The blue error bar shows the case without q-bots, and
the red error bar shows the case with q-bots.

C. ACTION MECHANISM OF LEARNING AGENTS ON q-bots acting on coordination networks. We count the number
COORDINATION NETWORKS of node color changes for every 100 generations in the process
To study the way q-bots act on group coordination in depth, of iterative learning and normalize these data by dividing
we carry out an experiment to analyze the action mode of their maximum. The processed data show the degree of color

VOLUME 7, 2019 156531

 J. Qi et al.: Social Network-Oriented Learning Agent for Improving Group Intelligence Coordination
FIGURE 3. Survival curves of the game according to ε and the locations
of the q-bots. Each of the conditions is conducted over 20 different
networks (each network was tested once with z = 20, h = 3 and u = 200).
The curves show the percentages of games unsolved within the upper
limit. Dark blue curves show the results of the games with q-bots
according to the ε level (horizontal axis) and network location (vertical
axis). Light blue curves show the results of the control condition of games
without q-bots.
change in the nodes. In this section, q-bots are placed in a
central location, and the results are shown in Fig. 4.
For games without q-bots, there is no clear distinction
between the different intervals of generations, and the degree
of color change is too high. For cases with q-bots, the degree
of color change decreases with decreasing ε. This result
matches the result in Section IV-B indicating that q-bots with
smaller ε levels influence nodes more strongly. The degree of
color change in the nodes decreases with the process of learn-
ing, but there is no clear distinction between the generation
of intervals [100, 200] and [200, 300], which shows that the
effect of the q-bots on group coordination will be saturated
with the process of learning.
It is apparent that q-bots influence their neighbors. How-
ever, if the q-bot cannot influence other nodes, the group
coordination with q-bots has almost no improvement. There-
fore, we are more confident in the assumption given in
Section IV-B. By comparing the results of games with and
without q-bots, we find that the degree of color change for
nodes connected with q-bots and other nodes decreases with
the decrement of ε levels, but the degree of decrease for
nodes connected with q-bots is obviously higher than that
for other nodes with the decrement of ε levels. Therefore,
we validate the assumption mentioned in Section IV-B stating
that q-bots have a direct effect on the neighbor individuals
and an indirect effect on the non-neighbor individuals to act
on the coordination network, but ‘‘information’’ is lost in the
dissemination. Thus, q-bots act on the coordination network.
156532
FIGURE 4. Effect of q-bots on the behavior of other nodes. The color of
the nodes denotes the number of color changes each 100 generations in
one game (z = 20, h = 3 and u = 200). The coloring network is based on
the interval of generations (horizontal axis). Circle nodes show the
conventional agents, and square nodes show the q-bots. (a) The games
with q-bots by ε level (vertical axis). (b) The games without q-bots.
D. ACTION MECHANISM OF LEARNING AGENTS ON
CONFLICT RELATIONSHIPS
To study the way q-bots act on group coordination in depth,
we carry out an experiment to analyze the action mode of
q-bots acting on conflict relationships among individuals.
We propose the average rounds index for solving conflicts.
For a conflict with one edge in one game, the definition of
the average rounds required for solving conflict is
CLi
CSi = , (6)
CNi
where CLi shows the number of rounds in which edge ei
sinks into this type of conflict and CNi shows the number
of intervals in which edge ei sinks into this type of conflict.
For many edges or many games, we simply weight the mean
of this index. This index is used to measure the efficiency
of conflict solving. Based on this index, we analyze the
action mode of q-bots acting on conflict edges in 10 different
networks with 1000 nodes. In this section, q-bots are placed in
a central location, and edges are classified as edges connected
to q-bots and edges not connected to q-bots based on their
relationship with q-bots.
Figure. 5 shows the results of this experiment. For games
without q-bots, there is no clear distinction in the average
rounds required for solving conflict with increasing gener-
ations. For games with q-bots, the average rounds required
for solving conflict decreases with increasing generations.
The q-bots with a smaller ε level have better performance.
VOLUME 7, 2019
J. Qi et al.: Social Network-Oriented Learning Agent for Improving Group Intelligence Coordination

FIGURE 5. Effect of q-bots on the edges. The curves show changes in the average rounds required for solving conflict with increasing generations. Each
case is performed in 10 different networks (each network was tested with 500 generations, z = 1000, h = 50 and u = 2000). Blue curves show the cases
of edges connected to q-bots, and orange curves show the cases of edges not connected to q-bots. From left to right, the cases are in sequence of the
index of conflicts, resolvable conflicts, and unresolvable conflicts. (a) The cases with q-bots by ε level (vertical axis). (b) The cases without q-bots.

Additionally, the improvement of this index for edges con-
nected to q-bots is obviously higher than that for edges not
connected to q-bots, which supports the results that q-bots
solve the problem of coordination effectively and have an
effect on all individuals (the effect on neighbors is larger than
that on non-neighbors) in the above experiment.
In addition, we find that the improvement magnitude of
unresolvable conflicts is much higher than that of resolv-
able conflicts in games with q-bots. Therefore, we ignore
the improvement of resolvable conflicts. By contrasting the
games with and without q-bots, we conclude that q-bots
eliminate the conflicts that cannot be solved by individuals.
This result matches the example in Fig.2 (b). Thus, q-bots
act on the conflict relationship among individuals in group
coordination.
V. CONCLUSION
Group coordination often uses social networks, but improving
group coordination represents a challenging and fascinating
problem. In the coordination process, individuals often have
difficulty reaching the collective objective because of the
personal interest and ‘‘blind’’ phenomenon in the social
networks. To satisfy the needs of society and solve the
group coordination problem, we propose a solution on the
basis of artificial intelligence. We use a networked color
coordination game model to characterize group coordination
and introduce q-bots into this model. The learning agent
function for solving the group coordination problem is as
follows.
1) The group coordination problems caused by individ-
uals unable to reach the collective objective by elim-
inating only their local conflicts from an individual
standpoint are improved by q-bots.
2) By properly selecting the ε and network location,
q-bots drastically improve group coordination. The
q-bots acting with low levels of ε and placed in cen-
tral locations improve the group coordination by more
than 80%.

VOLUME 7, 2019 156533

 J. Qi et al.: Social Network-Oriented Learning Agent for Improving Group Intelligence Coordination
3) In acting on the coordination network, q-bots work by
having not only a direct effect on neighbor individuals
but also an indirect effect on non-neighbor individuals.
To act on the conflict relationship, q-bots eliminate
conflicts that cannot be solved by an individual. In the
above two ways, q-bots improve the group coordination
problem to help groups achieve collective interests.
As a result, the finding in this paper provides q-bots to
improve group coordination by solving the sub-optimization
problem arising from the individual ‘‘blind’’ phenomenon.
For example, we set q-bots in positions (decision mak-
ers) that have the most neighbors in practice, which can
improve the auxiliary function and provide certain decision-
making support. However, beyond this point, this work on
group coordination applies not only to human populations
but also to groups composed solely of robots attempting
to coordinate their actions [12]. From a broad perspective,
this work can be applied in more realistic or complex
coordination problems, such as those involving military
robots or autonomous moving vehicles. The code that sup-
port the findings of this study are available from GitHub
(http://github.com/JingtaoQi/color-game).
Although this paper proves our idea, we consider only
some basic factors, including the ε level and network location
of the learning agents [37]. Further work is required, includ-
ing considering dynamic networks in which the connections
between nodes may change [38], the effect of historical infor-
mation on individual decision making [39], ‘‘prosocial pref-
erences’’ in which individuals imitate the majority behavior
[40], the effect of the social environment on individual deci-
sion making [41], [42] and the effect of individual reputations
on cooperative behaviors (individuals can maximize their rep-
utation by cooperating with surrounding high-reputation indi-
viduals) [43], [44]. Furthermore, because of the particularity
of group coordination in society, how to control complex
group coordination is also a promising research topic.
REFERENCES
[1] E. A. Smith, ‘‘Communication and collective action: Language and the
evolution of human cooperation,’’ Evol. Hum. Behav., vol. 31, no. 4,
pp. 231–245, 2010.
[2] E. Fehr and U. Fischbacher, ‘‘Social norms and human cooperation,’’
Trends Cognit. Sci., vol. 8, no. 4, pp. 185–190, 2004.
[3] M. Scatà, A. Di Stefano, A. La Corte, P. Liò, E. Catania, E. Guardo, and
S. Pagano, ‘‘Combining evolutionary game theory and network theory to
analyze human cooperation patterns,’’ Chaos, Solitons Fractals, vol. 91,
pp. 17–24, Oct. 2016.
[4] D. G. Rand and M. A. Nowak, ‘‘Human cooperation,’’ Trends Cogn. Sci.,
vol. 17, no. 8, pp. 413–425, Aug. 2013.
[5] R. Axelrod and W. D. Hamilton, ‘‘The evolution of cooperation,’’ Science,
vol. 211, no. 4489, pp. 1390–1396, 1981.
[6] S. Strang and S. Q. Park, ‘‘Human cooperation and its underlying mecha-
nisms,’’ in Social Behavior From Rodents to Humans. Cham, Switzerland:
Springer, 2016, pp. 223–239.
[7] M. Tomasello and A. Vaish, ‘‘Origins of human cooperation and morality,’’
Annu. Rev. Psychol., vol. 64, pp. 231–255, Jan. 2013.
[8] D. D. P. Johnson, P. Stopka, and S. Knights, ‘‘The puzzle of human
cooperation,’’ Nature, vol. 421, no. 6926, pp. 911–912, 2003.
[9] N. Miller, S. Garnier, A. T. Hartnett, and I. D. Couzin, ‘‘Both information
and social cohesion determine collective decisions in animal groups,’’
Proc. Nat. Acad. Sci. USA, vol. 110, no. 13, pp. 5263–5268, 2013.
156534
[10] A. Lerer and A. Peysakhovich, ‘‘Maintaining cooperation in
complex social dilemmas using deep reinforcement learning,’’
2017, arXiv:1707.01068. [Online]. Available: https://arxiv.org/
abs/1707.01068
[11] R. M. Dawes, ‘‘Social dilemmas,’’ Annu. Rev. Psychol., vol. 31, no. 1,
pp. 169–193, 1980.
[12] H. Shirado and N. A. Christakis, ‘‘Locally noisy autonomous agents
improve global human coordination in network experiments,’’ Nature,
vol. 545, no. 7654, pp. 370–374, 2017.
[13] R. L. Calvert, ‘‘Leadership and its basis in problems of social coordina-
tion,’’ Int. Political Sci. Rev., vol. 13, no. 1, pp. 7–24, 1992.
[14] J. R. Dyer, A. Johansson, D. Helbing, I. D. Couzin, and J. Krause, ‘‘Lead-
ership, consensus decision making and collective behaviour in humans,’’
Philos. Trans. Roy. Soc. B, Biol. Sci., vol. 364, no. 1518, pp. 781–789, 2008.
[15] J. B. Van Huyck, R. C. Battalio, and R. O. Beil, ‘‘Tacit coordination games,
strategic uncertainty, and coordination failure,’’ Amer. Econ. Rev., vol. 80,
no. 1, pp. 234–248, 1990.
[16] Y.-Y. Liu, J.-J. Slotine, and A.-L. Barabási, ‘‘Controllability of complex
networks,’’ Nature, vol. 473, no. 7346, pp. 167–173, 2011.
[17] Y.-D. Xiao, S.-Y. Lao, L.-L. Hou, and L. Bai, ‘‘Edge orientation for
optimizing controllability of complex networks,’’ Phys. Rev. E, Stat.
Phys. Plasmas Fluids Relat. Interdiscip. Top., vol. 90, no. 4, 2014,
Art. no. 042804.
[18] V. Capraro, ‘‘A model of human cooperation in social dilemmas,’’ PLoS
One, vol. 8, no. 8, 2013, Art. no. e72427.
[19] M. Perc, J. Gomez-Gardenes, A. Szolnoki, L. M. Floría, and Y. Moreno,
‘‘Evolutionary dynamics of group interactions on structured populations:
A review,’’ J. Roy. Soc. Interface, vol. 10, no. 80, 2013, Art. no. 20120997.
[20] M. Perc, J. J. Jordan, D. G. Rand, Z. Wang, S. Boccaletti, and A. Szolnoki,
‘‘Statistical physics of human cooperation,’’ Phys. Rep., vol. 687, pp. 1–51,
May 2017.
[21] Z. Wang, L. Wang, A. Szolnoki, and M. Perc, ‘‘Evolutionary games on
multilayer networks: A colloquium,’’ Eur. J. Phys. B, vol. 88, p. 124,
May 2015.
[22] D. Helbing, D. Brockmann, T. Chadefaux, K. Donnay, U. Blanke,
O. Woolley-Meza, M. Moussaid, A. Johansson, J. Krause, S. Schutte, and
M. Perc, ‘‘Saving human lives: What complexity science and information
systems can contribute,’’ J. Stat. Phys., vol. 158, no. 3, pp. 735–781,
Feb. 2015.
[23] D. P. Enemark, M. D. McCubbins, R. Paturi, and N. Weller, ‘‘Does more
connectivity help groups to solve social problems,’’ in Proc. 12th ACM
Conf. Electron. Commerce, Jun. 2011, pp. 21–26.
[24] D. G. Rand, M. A. Nowak, J. H. Fowler, and N. A. Christakis, ‘‘Static
network structure can stabilize human cooperation,’’ Proc. Nat. Acad. Sci.
USA, vol. 111, no. 48, pp. 17093–17098, 2014.
[25] D. P. Enemark, M. D. McCubbins, R. Paturi, and N. Weller, ‘‘Can we
think locally, act globally? Understanding when local information can
facilitate global coordination,’’ PSN, Lab. Exp., Topic, Nov. 2010. doi:
10.2139/ssrn.1711582.
[26] A. Peysakhovich and A. Lerer, ‘‘Consequentialist conditional cooperation
in social dilemmas with imperfect information (short workshop version),’’
in Proc. 32nd Workshops AAAI Conf. Artif. Intell., 2018, pp. 310–316.
[27] N. Salazar, J. A. Rodriguez-Aguilar, J. L. Arcos, A. Peleteiro, and
J. C. Burguillo-Rial, ‘‘Emerging cooperation on complex networks,’’
in Proc. 10th Int. Conf. Auto. Agents Multiagent Syst., vol. 2, 2011,
pp. 669–676.
[28] A. Peleteiro, J. C. Burguillo, and S. Y. Chong, ‘‘Exploring indirect reci-
procity in complex networks using coalitions and rewiring,’’ in Proc. Int.
Conf. Auton. Agents Multi-Agent Syst., 2014, pp. 669–676.
[29] R. Shibusawa, T. Otsuka, and T. Sugawara, ‘‘Spread of cooperation in
complex agent networks based on expectation of cooperation,’’ in Proc.
Int. Conf. Princ. Pract. Multi-Agent Syst. Cham, Switzerland: Springer,
2016, pp. 76–91.
[30] M. Kearns, S. Suri, and N. Montfort, ‘‘An experimental study of the
coloring problem on human subject networks,’’ Science, vol. 313, no. 5788,
pp. 824–827, 2006.
[31] Y. Xiao, M. T. Angulo, J. Friedman, M. K. Waldor, S. T. Weiss, and
Y.-Y. Liu, ‘‘Mapping the ecological networks of microbial communities,’’
Nature Commun., vol. 8, no. 1, 2017, Art. no. 2042.
[32] I. Sghir, J.-K. Hao, I. B. Jaafar, and K. Ghédira, ‘‘A distributed hybrid
algorithm for the graph coloring problem,’’ in Proc. Int. Conf. Artif. Evol.
(Evol. Artificielle). Cham, Switzerland: Springer, 2015, pp. 205–218.
[33] A.-L. Barabási and R. Albert, ‘‘Emergence of scaling in random net-
works,’’ Science, vol. 286, no. 5439, pp. 509–512, 1999.
VOLUME 7, 2019
J. Qi et al.: Social Network-Oriented Learning Agent for Improving Group Intelligence Coordination

[34] C. J. C. H. Watkins and P. Dayan, ‘‘Q-learning,’’ Mach. Learn., vol. 8, LIANG BAI received the B.E. and B.M. degrees
nos. 3–4, pp. 279–292, 1992. from Xi’an Jiao Tong University, in 2002, and
[35] L. Waltman and U. Kaymak, ‘‘A theoretical analysis of cooperative behav- the M.E. and Ph.D. degrees from the National
ior in multi-agent q-learning,’’ in Proc. IEEE Int. Symp. Approx. Dyn. University of Defense Technology, in 2005 and
Program. Reinforcement Learn., Apr. 2007, pp. 84–91. 2008, respectively.
[36] E. R. Gomes and R. Kowalczyk, ‘‘Dynamic analysis of multiagent Q- He is currently a Professor with the School
learning with ε-greedy exploration,’’ in Proc. 26th Annu. Int. Conf. Mach. of System Engineering, National University
Learn., 2009, pp. 369–376.
of Defense Technology. His research interests
[37] A. Sánchez, ‘‘Physics of human cooperation: Experimental evidence and
include multimedia content analysis and access,
theoretical models,’’ J. Stat. Mech., Theory Exp., vol. 2018, no. 2, 2018,
Art. no. 024001. particularly for video and images, big multimedia
[38] D. G. Rand, S. Arbesman, and N. A. Christakis, ‘‘Dynamic social networks data, complex system modeling, and network analysis.
promote cooperation in experiments with humans,’’ Proc. Nat. Acad. Sci. Dr. Bai is a member of the Chinese Computer Federation.
USA, vol. 108, no. 48, pp. 19193–19198, 2011.
[39] M. Cebrian, R. Paturi, and D. Ricketts, ‘‘Experimental study of the impact
of historical information in human coordination,’’ 2012, arXiv:1202.2503.
[Online]. Available: https://arxiv.org/abs/1202.2503
[40] M. N. Burton-Chellew and S. A. West, ‘‘Prosocial preferences do not
explain human cooperation in public-goods games,’’ Proc. Nat. Acad. Sci.
USA, vol. 110, no. 1, pp. 216–221, 2013.
[41] K. A. Cronin, D. J. Acheson, P. Hernández, and A. Sánchez, ‘‘Hierar-
chy is detrimental for human cooperation,’’ Sci. Rep., vol. 5, Dec. 2015,
Art. no. 18634.
[42] P. Van Den Berg, L. Molleman, J. Junikka, M. Puurtinen, and F. J. Weissing,
‘‘Human cooperation in groups: Variation begets variation,’’ Sci. Rep.,
vol. 5, Nov. 2015, Art. no. 16144.
[43] Y. Dong, S. Sun, C. Xia, and M. Perc, ‘‘Second-order reputation promotes
cooperation in the spatial prisoner’s dilemma game,’’ IEEE Access, vol. 7,
pp. 82532–82540, 2019.
[44] M.-H. Chen, L. Wang, S.-W. Sun, J. Wang, and C.-Y. Xia, ‘‘Evolu-
tion of cooperation in the spatial public goods game with adaptive
reputation assortment,’’ Phys. Lett. A, vol. 380, nos. 1–2, pp. 40–47,
YANDONG XIAO received the B.E. degree in
Jan. 2016.
information system engineering and the Ph.D.
JINGTAO QI received the B.E. degree in simula- degree in system engineering from the National
tion engineering from the National University of University of Defense Technology, Changsha,
Defense Technology, Changsha, China, in 2018, China, in 2012 and 2018, respectively, and the
where he is currently pursuing the M.E. degree in Ph.D. degree from Harvard University, Boston,
control science and engineering. MA, USA, where he was also a Research
His current research interests include studying Trainee.
complex systems with a combination of theoretical He is currently a Lecturer with the School
tools and data analysis, mathematical modeling of System Engineering, National University of
of heterogeneous information networks, applying Defense Technology. His research interests include complex system mod-
network methodologies to analyze the develop- eling, artificial intelligence, swarm intelligence, interdisciplinary principles
ment of complex collection intelligence, and the data-driven study of col- for community ecology, and social networks.
lective behavior.

VOLUME 7, 2019 156535