Journal of the Operational Research Society

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/tjor20

Dynamic subgroup-quality-based consensus in

managing consistency, nearness, and evenness
quality indices for large-scale group decision
making under hesitant environment

Ming Tang, Huchang Liao, Xiaomei Mi, Xuanhua Xu & Francisco Herrera

To cite this article: Ming Tang, Huchang Liao, Xiaomei Mi, Xuanhua Xu & Francisco Herrera
(2021) Dynamic subgroup-quality-based consensus in managing consistency, nearness, and
evenness quality indices for large-scale group decision making under hesitant environment, Journal
of the Operational Research Society, 72:4, 865-878, DOI: 10.1080/01605682.2019.1708823

To link to this article: https://doi.org/10.1080/01605682.2019.1708823

Published online: 13 May 2020.

Submit your article to this journal

Article views: 94

View related articles

View Crossmark data

Citing articles: 2 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=tjor20
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY
2021, VOL. 72, NO. 4, 865–878
https://doi.org/10.1080/01605682.2019.1708823

ORIGINAL ARTICLE

Dynamic subgroup-quality-based consensus in managing consistency,

nearness, and evenness quality indices for large-scale group decision
making under hesitant environment
Ming Tanga , Huchang Liaoa,b , Xiaomei Mia , Xuanhua Xuc and Francisco Herrerab,d
a
Sichuan University, Chengdu, China; bUniversity of Granada, Granada, Spain; cCentral South University, Changsha, China; dKing
Abdulaziz University, Jeddah, Saudi Arabia

ABSTRACT ARTICLE HISTORY

With the increasing complexity of decision-making environment and the development of Received 16 April 2019
societal demands, large-scale group decision making (LSGDM) has become a hot topic in Accepted 17 December 2019
recent years. Due to the demand of the current decision-making environment, up to now,
KEYWORDS
studies regarding LSGDM methods focussing on different approaches have been published.
Large-scale group decision
However, there are few studies that focus on evaluating the results of LSGDM problems. making; adaptive consensus
This study proposes three subgroup quality indices, namely, consistency degree, nearness reaching process; subgroup
degree and evenness degree, to measure the effectiveness of LSGDM methods. These three quality indices; hesitant
quality indices not only can evaluate the effect of a subgroup on the global group, but also fuzzy set
can measure the effect of final decision results. Furthermore, the consistency measure is
used to design the weights of subgroups, based on which an adaptive dynamic consensus
reaching process is introduced. An illustrative example is given to verify the applicability and
effectiveness of our proposed model.

1. Introduction result, the uncertainty in decision making is becom-

Group decision making (GDM) can be viewed as a
situation where a group of experts express their
opinions regarding candidate alternatives, and then
select the best one. Generally, a traditional GDM
problem contains a small number of experts (Wu &
Xu, 2018). However, with the increasing complexity
of social environments, there is a trend that a large
number of experts are involved in decision-making
problems. Therefore, large-scale GDM (LSGDM)
becomes a much-talked-about research topic and
attracts a lot of attention from scientific scholars
(Ding, Wang, Shang, & Herrera, 2019; Ma, Zhu,
Ponnambalam, & Zhang, 2019; Palomares, Mart
ınez,
& Herrera, 2014; Song & Li, 2019).
An LSGDM problem has the following three
features (Ma et al., 2019): 1) The number of experts
varies from dozens to hundreds; 2) Individual
experts in the group can provide preferences in rela-
tively different places and at relatively different
times; 3) Preferences provided by experts tend to be
uncertain due to the decision-making environment.
The third point is especially a challenging issue in
LSGDM. In LSGDM problems, large-scale experts
may have various backgrounds, expertise levels and
personal interests. Meanwhile, there are also hesitant
and uncertain nature in human cognition. As a
ing increasingly inevitable. Fortunately, the proposal
of hesitant fuzzy set (HFS) (Torra, 2010) provides
an efficient tool to deal with hesitation situations.
There is a necessity to use HFSs to express experts’
uncertain information when they are hesitant among
several numerical values.
Up to now, there are four main topics in the field
of LSGDM:
1) Sub-group clustering (Liu, Shen, Chen, Chen,
& Wang, 2014; Liu, Zhou, Ding, Palomares, &
Herrera, 2019; Wu & Xu, 2018);
2) Consensus reaching process (CRP) in LSGDM
(Rodr
ıguez, Labella, De Tr
e, & Mart
ınez, 2018;
Tang et al., 2019);
3) LSGDM methods (Liu, Fan, & Zhang, 2016);
4) Behaviour management (Li, Dong, & Herrera,
2019; Palomares et al., 2014; Xu, Du, &
Chen, 2015).
Subgroup clustering is essential in LSGDM and
generally has two effects (Ma et al., 2019): one is to
reduce the size of the decision-making problem,
based on which the cost and complexity can be
reduced; the other is to find common opinion pat-
terns such as subgroups with high similar judgment

CONTACT Huchang Liao liaohuchang@163.com Business School, Sichuan University, Chengdu 610064, China
ß The Operational Research Society 2020
866 M. TANG ET AL.
information, in which one member of the group can
be identified as a spokesperson. After the clustering
process, we need to find a solution based on all sub-
groups’ opinions. However, some subgroups may
think their opinions have not been fully considered
in the final solution and thus refuse to accept the
group decision result. Therefore, it is important to
apply the CRP in LSGDM to gain all subgroups’
support. The CRP usually needs iterations to achieve
a minimum threshold. To reach this goal, all sub-
groups need to make their contributions. Owing to
various perceptions, motivations and experience lev-
els, different subgroups may have different contribu-
tions and exert different influences on the global
group opinion. It seems of interest to measure the
subgroup quality, i.e., a subgroup’s effect on the
CRP, based on which, we can objectively determine
the weights of subgroups or adjust their weights
dynamically.
LSGDM has been researched from different per-
spectives. However, studies from other aspects have
barely been studied, such as measuring and evaluat-
ing the effect of LSGDM results, or how to evaluate
the quality of a subgroup’s opinion in the final col-
lective opinion. Currently, many approaches aiming
at solving LSGDM problems have been developed.
When we use different approaches to solve the same
problem, we may obtain different final rankings of
alternatives, in other words, inconsistent decision
results are derived with respect to different LSGDM
methods (Wu & Xu, 2018). The results can also be
influenced by the clustering methods. Therefore, it
seems of interest to construct a quality index system
to evaluate the effect of LSGDM results, to know
which LSGDM method would be better from
an aspect.
As aforementioned, we can design subgroups’
dynamic weights in the CRP that are suitable for
practical decision-making problems. The dynamic
consensus provides a tool to deal with dynamic
decision-making problems by allowing some param-
eters to be changed, as determined by external or
subjective factors (Herrera-Viedma, Cabrerizo,
Kacprzyk, & Pedrycz, 2014). In the decision-making
process, it seems of interest to analyse the environ-
ment characteristics and conditions of the handled
problem. In current situations, decision-making
problems occur in a dynamic environment (P
erez
et al., 2018). So far, the dynamic consensus has been
extended into different aspects. Among these
aspects, the adaptive consensus (P
erez, Cabrerizo,
Alonso, & Herrera-Viedma, 2014) is an emerging
and efficient method. This type of consensus can
generate recommendations according to some
parameters such as the level of consensus and the
weights of experts. P
erez et al. (2014) proposed a
consensus model with a feedback mechanism that
can adjust the amount of advice, depending on the
importance level of the expert. However, the weight
of each expert in their model remains static. In add-
ition, existing LSGDM models did not consider the
CRP adapting to the weights of subgroups.
Therefore, it is an interesting topic to design an
adaptive consensus model with dynamic weights of
experts (subgroups in LSGDM), which we called the
adaptive dynamics CRP.
According to the above-mentioned facts, this
study first proposes three subgroup quality indices
to evaluate the effect of LSGDM methods. These
three subgroup indices are defined from three per-
spectives: the first one is measuring the similarity
degree between priorities of subgroup’s alternatives
with regard to that of the global collective group,
which is called the consistency degree; the second
one is the size of divorced values, which is called
the nearness degree; the last one is the uniform dis-
tribution degree of total differences, which is called
the evenness degree. Based on the judgment infor-
mation of subgroups, we first measure the opinion
quality of each subgroup on the global group opin-
ion, and then introduce a method to evaluate effects
of decision results of LSGDM problems. Next, a
subgroup quality index named the consistency
degree is used to design the adaptive CRP in
LSGDM. In each iterative round, the weights of sub-
groups are adjusted dynamically. To reduce the
supervision cost, an adaptive mechanism with three
strategies adapting to the weights of subgroups is
put forward. The contributions of this study are
highlighted as follows:
1) We propose three subgroup quality indices, i.e.,
consistency degree, nearness degree and even-
ness degree, to evaluate the influence of each
subgroup and the effect of LSGDM decision
results. It will be helpful to know the quality of
decision results and select an LSGDM method
with good performance.
2) Based on the consistency degree, we introduce
a dynamic subgroup-quality-based CRP to
solve LSGDM problems. The weight of each
subgroup is determined by two factors: the
size and the consistency degree of the opin-
ions. Then, the weights of subgroups are
adjusted in the CRP dynamically. Three strat-
egies adapting different weights of subgroups
are introduced.
The rest of this paper is organised as follows.
Section 2 introduces the preliminaries used in this
study. In Section 3, we put forward three subgroup
quality indices, i.e., consistency degree, nearness
 JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 867

degree and evenness degree, to evaluate the results When evaluating alternatives, the preference rela-
of LSGDM problems. Section 4 presents details of tion based on pairwise comparisons is a popular
the adaptive dynamic CRP. In Section 5, an illustra- tool for experts to express their opinions. The con-
tive example and some comparisons are provided. cept of hesitant comparison matrix (HCM) com-
Conclusions are drawn in Section 6. posed of hesitant pairwise comparisons was
introduced (Xia & Xu, 2013). An HCM on X is rep-
resented as a matrix H ¼ ðhij Þnn  X  X, where
2. Preliminaries
hij ¼ fclij jl ¼ 1, 2, :::, #hij g (#hij is the number of ele-
This section introduces some preliminaries used in the ments in hij ) is an HFE, indicating all possible pref-
rest of this study, including the description of LSGDM
erence values to which alternative xi is preferred to
problems and the hesitant judgment information. rðlÞ rð#h lþ1Þ
xj : hij satisfies: cij þ cji ij ¼ 1, hii ¼ f0:5g,
rðlÞ
2.1. Description of large-scale group decision #hij ¼ #hji , for all i, j ¼ 1, 2, :::, n, where cij is the
making problems lth largest element in hij :
One problem in the decision-making process
Current societal and technological demands lead a
with hesitant judgment information is that the car-
large number of participants to take part in the
dinality of HFEs may be different. We adopt the
decision-making process (Tang et al., 2019). As a
parameter g proposed by Xu and Zhang (2013) to
result, LSGDM has become a hot research topic in
add elements to hij ði < jÞ, and 1g to add elements
recent years. The main elements of an LSGDM
to hji ði > jÞ: Then, a normalised HCM (NHCM)
problem are shown as follows:
(Zhu & Xu, 2014) H  ¼ ðh  ij Þ
nn can be obtained,
 ij ¼ maxf#h  ij ji, j ¼ 1, 2, :::, ng,
(1) A discrete set of alternatives X ¼ fx1 , x2 , which satisfies: #h
:::, xn g ðn  2Þ, which represent possible where #h  ij is the number of elements in hij :
solutions to the LSGDM problem; The score function of an HFE h is defined as
(2) A set of experts E ¼ fe1 , e2 , :::, em g, who are Farhadinia (2013):
invited to evaluate alternatives. There are two
X#h X #h
main principles for the value of m: One is sðhÞ ¼ dðlÞcl dðlÞ (2)
m  20 (Wu & Xu, 2018); the other is m  11 l¼1 l¼1
(Xu, Du, et al., 2015). In this study, we use #h is a positive-valued monotonically
where fdðlÞgl¼1
the former principle.
increasing sequence of the index l: Based on
The CRP in GDM or LSGDM is usually an itera- Equation (2), we can obtain a preference relation
tive process containing some discussion rounds. composed of score values. Then, the score value of
These discussion rounds need to be coordinated by each alternative can be derived as:
X
n 
the moderator, who plays a critical role in the CRP
sðxi Þ ¼ sðhij Þ n (3)
(Mata, Mart
ınez, & Herrera-Viedma, 2009). The j¼1
tasks of the moderator mainly include (Palomares
et al., 2014): 1) calculating the consensus level of the To obtain the collective HCM for a group, the
group; 2) generating recommendations for experts weighted operator proposed by Xia and Xu (2013) is
to reach group consensus. used:
m
HFWAðh1 , h2 , :::, hm Þ ¼ 丣 ðwk hk Þ
k¼1
2.2. Hesitant judgment information X
m 
¼ [ wk crðsÞ
m
One feature of LSGDM problems is that the prefer- rðsÞ rðsÞ
c1 2h1 , c2 2h2 , :::, cm 2hm
rðsÞ
k¼1
ences provided by experts tend to be uncertain (Ma (4)
et al., 2019). Hence, we use the concept of HFS
T
(Torra, 2010) to depict uncertain and hesitant infor- where w ¼ ðw1 , w2 , :::, wm Þ is the weight vector of
P
mation. Let X be a reference set, an HFS on X can hk with wk 2 ½0, 1 and m k¼1 wk ¼ 1:
be represented in the form of
A ¼ fhx, hA ðxÞijx 2 Xg (1) 3. Subgroup quality indices for LSGDM:
where hA ðxÞ is a set of values in ½0, 1, denoting the
Consistency degree, nearness degree and
possible membership degrees of the element x 2 X
evenness degree
to the set A: hA ðxÞ is called the hesitant fuzzy elem- In this section, we introduce three quality sub-
ent (HFE) (Xia & Xu, 2013). group indices, i.e., consistency degree, nearness
868 M. TANG ET AL.

degree, and evenness degree, to evaluate the Through the normalisation process, we can obtain
results of LSGDM problems. NHCMs. To save space, four NHCMs are not illus-
trated. The collective HCM HG is obtained as:

8 9
>
> f0:5g f0:35, 0:4g f0:575, 0:625g >
f0:725g
>
>
> >
>
< f0:6, 0:65g f0:5g f0:625, 0:65, 0:675g f0:775, 0:85g =
HG ¼
>
> f0:375, 0:425g f0:325, 0:35, 0:375g f0:5g f0:65g > >
>
> >
>
: f0:275g f0:15, 0:225g f0:65g f0:5g ;

3.1. Consistency degree The score values of the alternatives corresponding

to each subgroup’s HCM and the collective HCM
An HCM composed of pairwise comparisons reflects
are shown in Table 1.
different preferences between alternatives. The con-
sistency degree of a subgroup’s HCM with respect
Based on Table 1, we can obtain the dominance
to the collective HCM can reflect the effect of the
class of each alternative regarding subgroup
subgroup on the global group opinion to some
extent. Inspired by Pang and Liang (2012), we use G1 :½x1   
s1 ¼ fx1 g, ½x2 s1 ¼ fx1 ,x2 g, ½x3 s1 ¼ fx1 ,x2 ,x3 g,
the dominance relation to measure the priorities ½x4 
s1 ¼ fx1 ,x2 ,x3 ,x4 g: Thus, the family set of domin-
between alternatives in this section. ance classes derived by the HCM HG1 is: X=R
s1 ¼
For a given HCM with nðn1Þ=2 pairwise com-
f½x1    
s1 ,½x2 s1 ,½x3 s1 ,½x3 s1 g ¼ ffx1 g,fx1 ,x2 g, fx1 ,x2 ,x3 g,
parisons among n alternatives, we say that alterna-
fx1 , x2 ,x3 ,x4 gg:
tive xi dominances xj if sðxi Þ  sðxj Þ, denoted as
Similarly, we can obtain the dominance class of
xi Rs xj : Then, the dominance class with regard to
each alternative derived by the collective HCM HG :
alternative xi is ½xi  s ¼ fxj 2 Xjsðxj Þ  sðxi Þg: Let ½x1   
sG ¼ fx1 , x2 g, ½x2 sG ¼ fx2 g, ½x3 sG ¼ fx1 , x2 , x3 g,
X=Rs ¼ f½x1 s , ½x2 s , :::, ½xn 
  
s g denote the family set ½x4 
sG ¼ fx1 , x2 , x3 , x4 g, and the family set of domin-
of dominance classes. The dominance classes in 
X=R ance classes derived by HG is X=R
sG ¼ f½x1 sG ,
s do not form a partition of X in general. We
use an example below to explain this concept. ½x2   
sG , ½x3 sG , ½x3 sG g ¼ ffx1 , x2 g, fx2 g, fx1 , x2 , x3 g, fx1 ,
x2 , x3 , x4 gg:
Example 1.1. Suppose that there is a solar power We use the first iterative round of the CRP to
company that needs to select a supplier from four explain the consistency index. Below we define the
possible alternatives: fx1 , x2 , x3 , x4 g: 20 experts are consistency degree of a subgroup HCM with regard
invited to evaluate these four suppliers. For the con- to the collective HCM.
venience of calculation, this example uses four sub- Definition 1. In an LSGDM problem, the consist-
groups ðG1 , G2 , G3 , G4 Þ with the same number of ency degree of an HCM HGh given by the subgroup
experts and the same weight. Four subgroups’ Gh with regard to the collective HCM HG is defined
HCMs are collected as follows: as

8 9 8 9
>
> f0:5g f0:6g f0:7, 0:8g f0:8g > > >
> f0:5g f0:2, 0:4g f0:6g f0:8g > >
>
> >
> >
> >
>
< f0:4g f0:5g f0:6g f0:6, 0:7g = < f0:6, 0:8g f0:5g f0:7g f0:8, 0:9g =
HG1 ¼ , HG2 ¼
> f0:2, 0:3g
> f0:4g f0:5g f0:6g > > >
> f0:4g f0:3g f0:5g f0:6g > >
>
> >
> >
> >
>
: f0:2g f0:3, 0:4g f0:4g f0:5g ; : f0:2g f0:1, 0:2g f0:4g f0:5g ;
8 9 8 9
>
> f0:5g f0:3g f 0:4, 0:5 g f0:6g > > >
> f0:5g f0:3g f0:6g f0:7g >
>
>
> >
> >
> >
>
< f0:7g f0:5g f0:6g f0:8, 0:9g = < f0:7g f0:5g f0:6, 0:7, 0:8g f0:9g =
HG3 ¼ , HG 4 ¼
> f0:5, 0:6g
> f0:4g f0:5g f0:7g > > >
> f0:4g f0:2, 0:3, 0:4g f0:5g f0:7g >
>
>
> >
> >
> >
>
: f0:4g f0:1, 0:2g f0:3g f0:5g ; : f0:3g f0:1g f0:3g f0:5g ;
 JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 869

sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
Table 1. Score values of the alternatives corresponding to Pn 2
different HCMs. i¼1 ðsh ðxi ÞsG ðxi ÞÞ
Neah ¼ 1 (6)
x1 x2 x3 x4 n
G1 0.667 0.542 0.442 0.367
G2 0.558 0.700 0.450 0.317
G3 0.467 0.667 0.542 0.342 Obviously, 0 < Neah  1: The closer Neah is to 1,
G4 0.525 0.613 0.438 0.300 the higher the closeness of the subgroup Gh with
G 0.554 0.654 0.479 0.331
respect to the global group. As a result, this sub-
n j x  \ x j group’s opinion has a larger contribution to the cur-
1X ½ i sh ½ i sG
Conh ¼ (5) rent group opinion. In other words, this subgroup
n i¼1 j½xi 
s
j has a higher impact on the current iteration.
h

Example 1.3 (Continuation of Example 1.1). Based

where N is the number of subgroups. ½xi  
sh 2 X=Rsh on Definition 2, we can obtain the nearness degree of
denotes the dominance class of alternative xi each subgroup HCM with respect to the collect-
regarding the score values derived from the HCM HGh ive HCM.

sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
ð0:6670:554Þ2 þ ð0:5420:654Þ2 þ ð0:4420:479Þ2 þ ð0:3670:331Þ2
Nea1 ¼ 1 ¼ 0:928
4

provided by the subgroup Gh : ½xi  

sG 2 X=RsG denotes
Similarly, Nea2 ¼ 0:972, Nea3 ¼ 0:946, Nea4 ¼
the dominance class of alternative xi regarding the 0:964: Based on the former consistency index,
collective values derived from the collective HCM HG : subgroups G2 and G4 have equal effects. If two
subgroups have the same consistency degree, then
Example 1.2 (Continuation of Example 1.1). The the subgroup with the higher nearness degree has
consistency degree of HG1 provided by the subgroup a higher impact on the group results, as the
G1 with respect to the collective HCM HG is: opinion of the subgroup is closer to the
group opinion. Therefore, the subgroup G2 is
4 j x  \ x j   more effective.
1X ½ i sh ½ i sG 1 1 1 3 4
Con1 ¼ ¼ þ þ þ
4 i¼1 j½xi 
s
j 4 1 2 3 4
h

¼ 0:875 3.3. Evenness degree

In this section, the concept of the evenness degree is
introduced based on the entropy measure. This
Similarly, the other three subgroups’ consistency
index measures the uniform distribution degree of
degrees are calculated: Con2 ¼ 1, Con3 ¼ 0:917,
total differences among all alternatives.
Con4 ¼ 1: Therefore, in this round, subgroups G2
and G4 have the largest effect on the group decision Definition 3. In an LSGDM problem, the evenness
under the consistency index. degree of the HCM HGh given by the subgroup Gh
Obviously, Conh 2 ð0, 1: The closer Conh is to 1, with regard to the global collective HCM HG is
the higher the consistency degree of the subgroup defined as:
Gh with respect to the global group. As a result, this
subgroup’s opinion has a larger contribution to the X
n
js ðx ÞsG ðxi Þj
Eveh ¼ K Pn h i
current group opinion. i¼1 i¼1 jsh ðxi Þ  sG ðxi Þj
(7)
jsh ðxi ÞsG ðxi Þj
ln Pn
i¼1 jsh ðxi Þ  sG ðxi Þj
3.2. Nearness degree
This section proposes another index to measure where K is the Boltzman’s constant (Nos
e, 1984),
subgroups’ effects in the LSGDM from the perspec- which equals ln1N such that 0 < Eveh  1:
tive of divorced values. Example 1.4 (Continuation of Example 1.1). Based
Definition 2. In an LSGDM problem, the nearness on Definition 3, we can obtain the evenness degree
degree of an HCM HGh given by the subgroup Gh of each subgroup’s HCM with regard to the global
with regard to the collective HCM HG is defined as collective HCM.
870 M. TANG ET AL.

1 0:113 0:113 0:112 0:112
Eve1 ¼  ln þ ln
ln 40:298 0:298 0:298 0:298

0:037 0:037 0:036 0:036
þ ln þ ln
0:298 0:298 0:298 0:298
¼ 0:902
Similarly, the other three subgroups’ evenness degrees
are: Eve2 ¼ 0:818, Eve3 ¼ 0:781, Eve4 ¼ 0:991:
Obviously, 0 < Eveh  1: The closer Eveh is to 1,
the higher the evenness degree of the distribution of
difference between a subgroup’s score values and
the collective score values is.
Based on these three indices, we can evaluate the
effect of each subgroup in an LSGDM problem. In
an LSGDM problem, a subgroup with higher con-
sistency is closer to the global collective decision
results. If two subgroups have the same consistency
degree, then the subgroup with the higher nearness
degree is closer to the global collective decision
results. If two subgroups have the same nearness
degree, then the subgroup with the higher evenness
degree is closer to the global collective deci-
sion results.
3.4. Evaluating the results of an LSGDM problem
In the following we introduce a method to evaluate
the results of an LSGDM problem, from which we
can find out the degree of rationality and validity of
decision-making results.
Definition 4. In an LSGDM problem, the total con-
sistency degree C of the results is defined as
X
N back mechanism.
C¼ vh Con h (8)
h¼1
where vh denotes the weight of the subgroup Gh
and Con h denotes the consistency degree of the ori-
ginal HCM HGh given by subgroup Gh with respect
^ G:
to the final global collective HCM H
The weight of a subgroup can be determined by
its size (the number of experts). Most existing litera-
ture uses this weight determination method. In this
study, we propose another method, which combines
the size and consistency degree to assign weight to
a subgroup.
Definition 5. In an LSGDM problem, the total
nearness degree P of the results is defined as
X
N
P¼ vh Nea h (9)
h¼1
where vh denotes the weight of the subgroup Gh
and Nea h denotes the nearness degree of the
original HCM HGh given by the subgroup Gh with
^ G:
respect to the final global collective HCM H
Definition 6. In an LSGDM problem, the total
evenness degree E of the results is defined as
X
N
E¼ vh Eve h (10)
h¼1
where vh denotes the weight of the subgroup Gh
and Eveh denotes the evenness degree of the original
HCM HGh given by the subgroup Gh with respect to
the final global collective HCM H ~ G:
Obviously, all these three indices satisfy: C 2
ð0, 1, P 2 ð0, 1 and E 2 ð0, 1: Using these three
indices, we can evaluate the results of an LSGDM
problem. First, for an LSGDM problem, if we use
different methods, then the method with the highest
total consistency degree indicates that all subgroups’
opinions are closer to the final decision results. If
two methods have the same consistency degree,
then the method with higher total nearness degree
indicates that all subgroups’ opinions are closer to
the global decision results. If two methods have the
same nearness degree, then the method with the
highest evenness degree indicates that all subgroups’
opinions are closer to the final decision results.
4. Subgroup quality-based dynamic CRP
In this section, we propose a framework of the sub-
group quality-based CRP. This framework includes
four phases: 1) clustering process, 2) computing the
consensus level, 3) consensus control and feed-
4.1. Clustering process
The aim of a clustering algorithm is to classify data
from a data set into finite and discrete sets accord-
ing to the similarity degrees between them.
Elements in the same set should be more similar
than those in other sets. The clustering process is
essential for solving LSGDM problems.
In 2014, Rodr
ıguez and Laio (2014) proposed a
density-based clustering algorithm in Science,
named “fast search and find of density peaks
(FSFDP).” This algorithm introduced a new
method to select clustering centroids. The authors
declared that a clustering centroid has the charac-
teristics of high local density and a larger relative
distance from other centroids. This algorithm is
simple and efficient. In this study, the FSFDP algo-
rithm is applied to classify HCMs Hk , k ¼
1, 2, :::, m: The procedure of the HCM-FSFDP algo-
rithm is presented as follows:

Algorithm 1. (The HCM-FSFDP algorithm)
Input: HCMs Hk ¼ ðhij, k Þnn , k ¼ 1, 2, :::, m and the
number of sub-groups NðN  2Þ:
Output: Subgroups G1 , G2 , :::, GN :
Step 1. Each HCM Hk is transformed into its nor-
malised form, i.e., the NHCM.
Step 2. Calculate the distances between all NHCMs.
Here, we use the Euclidean distance (Xu & Xia,
2011) shown below to calculate the distances:
0 0 111=2
X
n #hxi
X
1 1
dðH1 , H2 Þ ¼ @ @ jh ðxi Þ  h2 ðxi Þj AA
rðlÞ rðlÞ 2
n i¼1 #hxi l¼1 1
Step 3. Sort the distances dij in an ascending order
and select a truncation distance dc :
Step 4. Compute the density qi of all nodes accord-
ing to the function:
X
qi ¼ vðdij dc Þ (12)
j subgroups. The normalised forms of these N HCMs
where vðxÞ is a 0-1 function such that: vðxÞ ¼ 1
when x < 0 and vðxÞ ¼ 0 when x  0: qi denotes
the number of nodes whose distance from Hi is
smaller than dc :
Step 5. For each node Hi , find the node Hj that has
larger density than Hi and select the smallest dij ,
denoted as di : To select appropriate centroids, we
set a threshold for the minimum value of qi and di :
Step 6. Assign the remaining nodes to the same
cluster as its nearest neighbour of high density.
Step 7. Output clusters G1 , G2 , :::, GN and end
the algorithm.
4.2. Dynamic weights based on quality indices
After classifying the experts into several subgroups,
the next objective is to determine each subgroup’s
weight. In the first round of aggregating the HCMs
of subgroups into the collective HCM, we use the
size of a subgroup as a principle to determine its
weight. After that, we need to use the effect of each
subgroup to update its weight.
Let #Gh be the number of experts in subgroup Gh
and Conh be the consistency degree of Gh with
respect to the global group. The size and consistency
degree are aggregated as:
uðGh Þ ¼ ð1 þ #Gh ÞðConh minðConÞÞb
where minðConÞ is the minimum consistency degree
of all subgroups and b is a parameter that controls
the impact of the consistency degree in the calculat-
ing process. To highlight the importance of the sub-
group’s quality instead of size in the weight
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 871
determination process, we put the subgroup’s effect
in exponent in Equation (13).
4.3. Consensus control
In the process of GDM, the most critical issue is to
find a consensus result being supported by all
experts. Strictly speaking, consensus means a full
agreement among all experts regarding all alterna-
tives. However, this concept is hard to achieve in
real-world scenarios. Then, Saint and Lawson (1994)
softened this concept and described consensus as a
state of a mutual agreement among experts and all
opinions have been conveyed to the satisfaction of
(11) them. A dynamic CRP can be defined as the process
of consensus changing with decision-making envir-
onment and specific parameters (P
erez et al., 2018).
These environmental factors and parameters
include alternatives, experts, experts’ importance
and criteria.
Let fHG1 , HG2 , :::, HGN g be N HCMs given by N
are fH  G1 , H
 G2 , :::, H
 GN g: The collective matrix by
aggregating N NHCMs is Hc ¼ ðhcij Þij : Then, the
consensus level of a subgroup is computed by
 h Hc Þ
CLðGh Þ ¼ 1dðH (14)
The global consensus level is obtained by
1X N
GCL ¼ CLðGh Þ (15)
N h¼1
The consensus degree on a position can be
defined as
cphij ¼ 1jhhij hcij j (16)
The consensus degree on an alternative is
Pn
j¼1, j6¼i cpij
cai ¼ (17)
n1
A predefined consensus threshold d should be set
in advance. If GCL  d, the consensus process fin-
ishes and the selection process is applied to obtain
the final decision results. Otherwise, the feedback
strategy with another iterative discussion round
should be adopted to provide suggestions to experts.
Next, we describe the details of the adaptive strat-
egy. All subgroups are classified into three levels: 1)
Glow , low importance subgroups; 2)Gmed , medium
importance subgroups and 3) Ghigh , high import-
(13)
ance subgroups. This classification is implemented
by a fuzzy matching mechanism whose parameters
are determined according to specific decision-mak-
ing problems. We define three feedback strategies to
increase the consensus level. There are two phases:
preference searching and advice generating.
872 M. TANG ET AL.
a) Preference searching for low-importance sub-
groups. If a subgroup is of low importance, accord-
ing to our weight determination method, then this
subgroup has a low consistency degree with the glo-
bal group. In other words, the opinion of this sub-
group is far from the global group. It is reasonable
to require many modifications for these subgroups.
All experts in low-importance subgroups are
required to make modifications on all pairs of alter-
natives that have low agreement levels. The search-
ing strategy for Glow acts as follows:
i) For a subgroup Gh 2 Glow , the alternatives
with a consensus degree lower than a thresh-
old, c1 , are identified by
ALT1 ¼ fxi jcai < c1 ÙGh 2 Glow g
The value of c1 can be set in advance or be
determined according to the specific consensus
level in each discussion round. A high value of
c1 needs a large amount of changes while a
low value of c1 may increase iterative rounds.
Therefore, it is important to select an appro-
priate c1 : Here, we use the average degree of
consensus at the alternative level as c1 , such
P
that c1 ¼ m i¼1 cai =m:
ii) The pairs of alternatives that need to be modi-
fied are identified by
POA1 ¼ fði, jÞjcai < c1 Ùxi 2 ALT1 g
Generation of advices
Once the subgroups and pairs of alternatives that
need to make modifications are identified, the mod-
erator provides direction rules to give suggestions.
The directions rules are recommended as follows:
DR. 11: If sðhhij Þ < sðhGij Þ, then all experts in the
subgroup Gh should increase the assessment
associated with the pair of alternatives ðxi , xj Þ:
DR. 12: If sðhhij Þ > sðhGij Þ, then all experts in the
subgroup Gh should decrease the assessment
associated with the pair of alternatives ðxi , xj Þ:
b) Preference searching for medium-importance
subgroups. In this situation, the importance of a
subgroup is higher than that of a low-importance
subgroup. It is reasonable to reduce the number of
changes in preference values. In the previous situ-
ation, all pairs of alternatives that have a low agree-
ment level are required to be modified. In this
situation, the alternatives that have preference values
with low agreement are selected. An alternative
P
proximity threshold ca  ¼ #Gh cahr =#G is used
i r¼1 i h
here, where cahr
i is the consensus level of alternative
xi associated with expert er in Gh : Another import-
ant difference is the number of experts involved in
the change of preferences. The moderator only pro-
vides modified suggestions to those experts whose
consensus level is lower than the average consensus
degree of all experts.
i) For a subgroup Gh 2 Gmed , the experts whose
consensus degree is lower than that of this
subgroup as compared to the global group are
identified by
EXP 2 ¼ fehr jCLhr < CLðGh ÞjGh 2 Gmed , ehr 2 Gh g
(20)
ii) The alternatives with a consensus degree lower
than the threshold c2 , are identified by
ALT2 ¼ fxi jcai < c2 g (21)
(18) where c2 can be calculated analogous to c1 :
iii) The pairs of alternatives that need to be modi-
fied are identified by
POA2 ¼ fði, jÞjcai < c2 Ùxi 2 ALT2 g (23)
iv) The set of preference values selected to make
modifications is identified by
2 ¼ fði, jÞ 2 POAjcai  ca
PREhr hr
 ig (24)
Note that it is not necessary for all experts identi-
fied in Equation (20) to make changes. If an expert
does not have preference values that should be
(19)
changed as identified by Equation (24), then this
expert does not need to make changes.
Generation of advices
The directions rules are recommended as follows:
DR. 21: If sðhhr G
ij Þ < sðhij Þ, then the expert ehr in the
subgroup Gh should increase the assessment
associated with the pair of alternatives ðxi , xj Þ:
G
DR. 22: If sðhhr
ij Þ > sðhij Þ, then the expert ehr in the
subgroup Gh should decrease the assessment
associated with the pair of alternatives ðxi , xj Þ:
c.1) Preference searching for high-importance sub-
groups. In such case, fewer changes than the two afore-
mentioned strategies can be made. The former situation
requires changes for alternatives. Nevertheless, this situ-
ation only requires changes to be made to the pairs of
alternatives whose consensus degrees are smaller than a
P#Gh hr
specific proximity threshold cp  ij ¼ r¼1 cpij =#Gh at
the level of pairs of alternatives.
i) For a subgroup Gh 2 Ghigh , the experts whose
consensus degree is lower than that of this
subgroup to the global are identified by
EXP 3 ¼ fehr jCLhr < CLðGh ÞjGh 2 Ghigh , ehr 2 Gh g
(25)
 JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 873

Figure 1. Framework of the proposed model.

ii) For any subgroup Gh 2 Glow , the alternatives 5.1. On the LSGDM evaluation
with a consensus degree lower than threshold
This part selects two typical illustrative examples
c3 are identified by
from Wu and Xu (2018) and Xu, Zhong, et al.
ALT3 ¼ fxi jcai < c3 g (26) (2015) to verify the validity of the index system for
LSGDM problems. These two papers used the same
where c3 can be calculated in analogous to c1 : data set to deal with an emergency large-scale prob-
lem composed of 20 experts.
iii) The pairs of alternatives that need to be modi- Wu and Xu (2018) proposed a consensus model
fied are identified by for LSGDM with changeable clusters based on the
possibility distribution of an HFS. Those 20 experts
POA3 ¼ fði, jÞjcai < c3 Ùxi 2 ALT3 g (27) were divided into 3 clusters with the k-means clus-
tering algorithm. Next, we use our proposed method
iv) The set of preference values selected to make to evaluate their approach. Note that the original
modifications is identified by judgment information is given as fuzzy preference
relations. The results are presented in Table 2.
3 ¼ fði, jÞ 2 POAjcai  
PREhr hr
c a i Ùcphr c p ij g
ij   Xu, Zhong, et al. (2015) introduced an exit-
(28) delegation mechanism to solve LSGDM problems.
They first separated the experts into several clusters,
which are treated as decision units. Then, if the
Generation of advices nearness degree of a cluster is lower than the con-
This situation has the same direction rules as the sensus threshold, this cluster is advised to exit the
preference searching for medium import- decision-making process. However, the “exited”
ance subgroups. cluster can allocate its trust weight to other clusters
A summarisation graph of the proposed model is and delegate them to make decisions. Therefore, the
presented in Figure 1. “exited” cluster still influences the following

Table 2. Evaluation results for the method proposed by Wu

5. Illustrative example and Xu (2018).
In this section, we use two parts to illustrate the Evaluation index

effectiveness and practicality of the two proposed Subgroups Consistency Nearness Evenness
methods. One is the LSGDM evaluation framework, G1 1.000 0.895 0.969
G2 0.875 0.940 0.885
and the other is the adaptive dynamic large- G3 0.833 0.889 0.994
scale CRP. Global group 0.881 0.908 0.951
874 M. TANG ET AL.

Table 3. Evaluation results for the method proposed by Xu, clustering method, the number of clusters and the
Zhong, et al. (2015). weight-determining method for clusters.
Evaluation index
Subgroups Consistency Nearness Evenness
G1 0.708 0.818 0.741
5.2. On the adaptive dynamic consensus
G2 0.833 0.942 0.926 reaching process
G3 0.875 0.906 0.786
G4 0.792 0.902 0.970 Sichuan Province is located in the transition zone
G5 0.833 0.869 0.812
G6 0.854 0.930 0.901
between Qinghai-Tibet Plateau and Sichuan Basin,
Global group 0.815 0.894 0.858 and is frequently affected by topography, geomorph-
ology, rainstorms, earthquakes and other induced fac-
discussions. Based on their model, 20 experts are tors, which make it a geological hazard-prone area. In
classified into 6 clusters. In the following discussion the flood season, Sichuan suffered from multiple
rounds, the first cluster exited the decision process. cycles of heavy rainfall, which caused landslides, col-
Next, we use our proposed method to evaluate their lapses and other geological disasters. To strengthen
approach. The results are shown in Table 3. the prevention of geological hazards in flood season,
From the results of Tables 2 and 3, we can find that Sichuan launched a round of investigation concerning
Wu and Xu (2018)’s method performed better than the hidden dangers of geological hazards with the help
that of Xu, Zhong, et al. (2015). The nearness degree of new technologies such as remote sensing,
of the former study has a slight advantage, while the unmanned aerial vehicles and three-dimensional laser
consistency degree and the evenness degree both have scanning. Suppose that there are four regions
clear advantages. Xu, Zhong, et al. (2015)’s exit-dele- fx1 , x2 , x3 , x4 g in Ya’an City to be selected as candidate
gate mechanism would let a cluster’s experts exit the alternatives. To scientifically assess the disaster risk
decision-making process. Liao, Xu, Zeng, and Xu and take preventive measures in order of priority, 20
(2016) also investigated this issue in the environment experts, represented by E ¼ fe1 , e2 , :::, e20 g, are
of small-scale GDM. They pointed out that the invited to make decisions on these four alternatives.
method that removes experts whose opinions were far Next, we use our proposed model to solve this prob-
from the group opinion had two drawbacks: one was lem. To make comparisons with the previous two
the loss of original judgment information; the other methods, this study uses the same data as Wu and Xu
was attacking the participation motivation of experts. (2018) and Xu, Zhong, et al. (2015). 20 experts pro-
Furthermore, many other factors can also influence vided their evaluation information using fuzzy prefer-
the results of a LSGDM problem, such as the ence relations, which are presented below:

0 1 0 1 0 1 0 1
0:5 0:4 0:4 0:6 0:5 0:3 0:6 0:4 0:5 0:6 0:2 0:3 0:5 0:9 0:7 0:8
B C B C B C B C
B 0:6 0:5 0:3 0:4 C B 0:7 0:5 0:6 0:6 C B 0:4 0:5 0:4 0:3 C B 0:1 0:5 0:8 0:7 C
P1 ¼ B C B
B 0:6 0:7 0:5 0:7 C, P2 ¼ B 0:4
C, P3 ¼ B C, P4 ¼ B C,
@ A @ 0:4 0:5 0:6 C
A
B 0:8
@ 0:6 0:5 0:4 C
A
B 0:3
@ 0:2 0:5 0:1 C
A
0:4 0:6 0:3 0:5 0:6 0:4 0:4 0:5 0:7 0:7 0:6 0:5 0:2 0:3 0:9 0:5
0 1 0 1 0 1 0 1
0:5 0:7 0:4 0:5 0:5 0:9 0:9 0:8 0:5 0:3 0:7 0:8 0:5 0:1 0:6 0:6
B C B C B C B C
B 0:3 0:5 0:1 0:2 C B 0:1 0:5 0:7 0:8 C B 0:7 0:5 0:3 0:6 C B 0:9 0:5 0:6 0:4 C
P5 ¼ B C B
B 0:6 0:9 0:5 0:4 C, P6 ¼ B 0:1
C, P7 ¼ B C, P8 ¼ B C,
@ A @ 0:3 0:5 0:4 C
A
B 0:3
@ 0:7 0:5 0:3 C
A
B 0:4
@ 0:4 0:5 0:3 C
A
0:5 0:8 0:6 0:5 0:2 0:2 0:6 0:5 0:2 0:4 0:7 0:5 0:4 0:6 0:7 0:5
0 1 0 1 0 1 0 1
0:5 0:1 0:8 0:4 0:5 0:4 0:4 0:4 0:5 0:4 0:4 0:2 0:5 0:4 0:4 0:1
B C B C B C B C
B 0:9 0:5 0:6 0:6 C B 0:6 0:5 0:1 0:5 C B 0:6 0:5 0:1 0:2 C B 0:6 0:5 0:5 0:4 C
P9 ¼ B C B
B 0:2 0:4 0:5 0:8 C, P10 ¼ B 0:6
C, P11 ¼ B C, P12 ¼ B C,
@ A @ 0:9 0:5 0:4 C
A
B 0:6
@ 0:9 0:5 0:4 C
A
B 0:6
@ 0:5 0:5 0:7 C
A
0:6 0:4 0:2 0:5 0:6 0:5 0:6 0:5 0:8 0:8 0:6 0:5 0:9 0:6 0:3 0:5
0 1 0 1 0 1 0 1
0:5 0:6 0:4 0:2 0:5 0:6 0:2 0:3 0:5 0:6 0:4 0:1 0:5 0:3 0:6 0:7
B C B C B C B C
B 0:4 0:5 0:3 0:7 C B 0:4 0:5 0:4 0:3 C B 0:4 0:5 0:3 0:4 C B 0:7 0:5 0:8 0:8 C
P13 ¼ B C B
B 0:6 0:7 0:5 0:6 C, P14 ¼ B 0:8
C, P15 ¼ B C, P16 ¼ B C,
@ A @ 0:6 0:5 0:4 C
A
B 0:6
@ 0:7 0:5 0:7 C
A
B 0:4
@ 0:2 0:5 0:9 C
A
0:8 0:3 0:4 0:5 0:7 0:7 0:6 0:5 0:9 0:6 0:3 0:5 0:3 0:2 0:1 0:5
0 1 0 1 0 1 0 1
0:5 0:2 0:6 0:6 0:5 0:4 0:7 0:6 0:5 0:6 0:6 0:7 0:5 0:7 0:6 0:8
B C B C B C B C
B 0:8 0:5 0:8 0:8 C B 0:6 0:5 0:4 0:8 C B 0:4 0:5 0:9 0:9 C B 0:3 0:5 0:6 0:7 C
P17 ¼ B C B C B
B 0:4 0:2 0:5 0:6 C, P18 ¼ B 0:3 0:6 0:5 0:7 C, P19 ¼ B 0:4
C, P20 ¼ B C:
@ A @ A @ 0:1 0:5 0:9 C
A
B 0:4
@ 0:4 0:5 0:9 C
A
0:4 0:2 0:4 0:5 0:4 0:2 0:3 0:5 0:3 0:1 0:1 0:5 0:2 0:3 0:1 0:5

Figure 2. The point distribution.
Table 4. The structure of clusters.
Subgroup Gh Number of experts Experts
G1 6 e1 , e2 , e8 , e9 , e12 , e17
G2 7 e3 , e5 , e10 , e11 , e13 , e14 , e15
G3 7 e4 , e6 , e7 , e16 , e18 , e19 , e20
Table 5. The weight of each sub-group.
Subgroup Gh #Gh Conh
G1 6 0.938
G2 7 0.729
G3 7 0.792
We set d ¼ 0:85, b ¼ 1: Note that this study
assumes that all experts agree to make modifications
according to the suggestions provided by
the moderator.
First, we use the Euclidean distance to calculate
the distances between these 20 preference relations.
There are 190 distances altogether. We sort all these
distances dij in ascending order and select the trun-
cation distance dc to be 0.1958. Figure 2 displays the
decision graph with the horizontal axis being
the density of points qi and the vertical axis being
the parameter di : Here, according to the Step 5 of
Algorithm 1, we set the threshold of qi and di be 5
and 0.25, respectively. Based on Figure 2, we can
select the appropriate clustering centroids. In this
example, we select experts e2 , e14 , e18 to be three
clustering centroids. Note that expert e3 and e14 are
repeated centroids. We select one of them randomly.
The structures of three subgroups are shown in
Table 4.
After calculating, we obtain the consensus level
0:845 < 0:85, and thus the CRP should be applied.
We compute the consistency degree of these three
subgroups and update their weights. The results are
presented in Table 5.
Round 1
Next, the consensus control process is applied to
improve the agreement level. To generate adaptive
recommendations, three subgroups are divided into
three subsets: Glow ¼ fG2 g, Gmed ¼ fG3 g, Ghigh ¼
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 875
fG1 g: The consensus degree at the alternative level
for three subgroups is presented in Figure 3.
a) Preference searching for low-importance
subgroup. For G2 , all experts are required to make
modifications. According to Equations (18) and
(19), alternatives and pairs of alternatives that
should be modified are fx2 , x3 g and fð2, 3Þ, ð2, 4Þ,
ð3, 1Þ, ð3, 2Þg, respectively.
Generation of advices
All experts in subgroup G2 should increase their assess-
ments for positions ð2, 3Þ and ð2, 4Þ, and decrease the
assessments for positions ð3, 1Þ and ð3, 2Þ:
b) Preference searching medium-importance sub-
group. For G3 , according to Equations (20)–(22), the
experts, alternatives and pairs of alternatives are
fe4 , e6 , e7 , e16 , e19 , e20 g, fx1 , x2 g and fð1, 4Þ, ð2, 3Þ,
ð2, 4Þg, respectively. According to Equation (23),
experts required to modify the selected pairs of alterna-
tives are identified:
Weight
PRE32, 4 ¼ fð1, 4Þ, ð2, 3Þ, ð2, 4Þg,
0.412
0.275 PRE32, 6 ¼ fð1, 4Þ, ð2, 3Þ, ð2, 4Þg,
0.313
PRE32, 7 ¼ fð1, 4Þ, ð2, 3Þ, ð2, 4Þg,
PRE32, 16 ¼ fð1, 4Þ, ð2, 3Þ, ð2, 4Þg,
PRE32, 19 ¼ fð1, 4Þ, ð2, 3Þ, ð2, 4Þg, PRE32, 20 ¼ fð1, 4Þg:
Generation of advices
The experts in subgroup G3 are required to increase
the preference value PRE32, 7 ¼ fð2, 3Þg;
The experts in subgroup G3 are required to
decrease the following preference values: PRE32, 4 ¼
fð1, 4Þ, ð2, 3Þ, ð2, 4Þg, PRE32, 6 ¼ fð1, 4Þ, ð2, 3Þ, ð2, 4Þg,
PRE23,7 ¼fð1,4Þ,ð2,4Þg, PRE23,16 ¼fð1,4Þ,ð2,3Þ,ð2,4Þg,
PRE23,19 ¼fð1,4Þ,ð2,3Þ,ð2,4Þg, PRE23,20 ¼fð1,4Þg:
c) Preference searching high-importance subgroup.
For G1 , according to Equations (24)–(26), experts,
alternatives and pairs of alternatives are
fe1 , e8 , e9 , e12 , e17 g, fx1 , x2 g and fð1, 2Þ, ð2, 1Þg,
respectively. According to Equation (27), the experts
who are required to modify the selected pairs of
alternatives are identified:
PRE13, 8 ¼ fð1, 2Þ, ð2, 1Þg, PRE13, 9
¼ fð1, 2Þ, ð2, 1Þg, PRE13, 17 ¼ fð1, 2Þ, ð2, 1Þg,
Generation of advices
The experts in subgroup G1 are required to increase
the following preference values: PRE13, 8 ¼ fð1, 2Þg,
PRE13, 9 ¼ fð1, 2Þg, PRE13, 17 ¼ fð1, 2Þg:
The experts in subgroup G3 are required to
decrease the following preference values: PRE13, 8 ¼
fð2, 1Þg, PRE13, 9 ¼ fð2, 1Þg, PRE13, 17 ¼ fð2, 1Þg:
876 M. TANG ET AL.

Round 2 5.3. Comparison with previous two methods

Suppose that the experts receive the modified sug-
To evaluate the effect of our proposed method, we
gestions and provide the following new preferences:
compute three indices: consistency degree, nearness
For experts in subgroup G1 2 Ghigh ,
degree and evenness degree. The results are pre-
sented in Table 6. Furthermore, Table 7 shows some
h812 ¼ 0:3, h821 ¼ 0:7, h912 ¼ 0:2, h921 ¼ 0:8, h17
12
comparisons concerning three methods (our pro-
¼ 0:4, h17
21 ¼ 0:6; posed method; Wu & Xu, 2018; Xu, Zhong,
et al., 2015).
For experts in subgroup G2 2 Glow , The previous two studies used the emergency
problem as an illustrative example. Their consensus
h323 ¼ 0:5, h324 ¼ 0:4, h331 ¼ 0:7, h332 ¼ 0:5; h523 ¼ 0:2, threshold is 0.8. Our illustrative example has a
h524 ¼ 0:3, h531 ¼ 0:5, h532 ¼ 0:8; h10
23 ¼ 0:2, higher consensus threshold of 0.85. Furthermore,
the proposed method used two rounds of iterations
24 ¼ 0:6, h31 ¼ 0:5, h32 ¼ 0:8, h23 ¼ 0:2, h24 ¼ 0:3,
h10 10 10 11 11

and achieved a consensus level of 0.883. From this

31 ¼ 0:5, h32 ¼ 0:8; h23 ¼ 0:4, h24 ¼ 0:7,
h11 11 13 13
point of view, our method performed better, since
31 ¼ 0:5, h32 ¼ 0:8; h23 ¼ 0:5, h24 ¼ 0:4, h31 ¼ 0:7,
h13 13 14 14 14
the final consensus level of the other two methods
32 ¼ 0:5; h23 ¼ 0:4, h24 ¼ 0:5, h31 ¼ 0:5,
h14 15 15 15
was about 0.8. This can occur because of
two reasons:
32 ¼ 0:6;
h15

1. One is the clustering algorithm. Wu and Xu

For experts in subgroup G3 2 Gmed , (2018) used the k-means clustering algorithm
and Xu, Zhong, et al. (2015) applied the vector
h414 ¼ 0:7, h423 ¼ 0:7, h424 ¼ 0:6, h614 ¼ 0:7, h623 ¼ 0:6,
space based clustering algorithm. These two
h624 ¼ 0:7, h714 ¼ 0:7, h723 ¼ 0:4, h724 ¼ 0:5, algorithms belong to the partitioning method.
14 ¼ 0:6, h23 ¼ 0:7, h24 ¼ 0:7, h14 ¼ 0:6, h23 ¼ 0:8,
h16 16 16 19 19 The FSFDP algorithm used in this study is
based on density.
24 ¼ 0:8, h14 ¼ 0:7:
h19 20
2. Another important difference is the weight
determination method. This study took the size
The consensus levels for the three subgroups are: of a subgroup and the consistency degree into
CLðG1 Þ ¼ 0:921, CLðG2 Þ ¼ 0:865, CLðG3 Þ ¼ 0:863: consideration when computing the weight of a
The global consensus level is GCL ¼ 0:883 > 0:85: subgroup. The consistency degree can reflect
Thus, the consensus threshold is achieved. the agreement level in alternative ranking.
Therefore, our proposed method has the best
performance regarding the nearness index. As a
result, the original judgment information can be
reserved as much as possible. A subgroup with
higher consistency degree will be assigned a
larger weight. This can accelerate the CRP.
Furthermore, this integrated weight-determin-
ation method combing two indices is more
comprehensive than single index-based meth-
Figure 3. The consensus degree at the alternative level. ods. For instance, if we only use the scale index,
then the subgroups with the same number of
Table 6. The evaluation results regarding the example.
experts but possessing different inner character-
Evaluation index
istics will have the same weights.
Subgroups Consistency Nearness Evenness
G1 0.938 0.955 0.684
G2 0.729 0.891 0.986
We should note that different studies have their
G3 0.792 0.893 0.987 own emphasis. Wu and Xu (2018) focus on change-
Global group 0.835 0.918 0.863 able clusters. Xu, Zhong, et al. (2015) proposed an

Table 7. The evaluation results of three methods.

Evaluation index
Consensus Number of Final consensus
Methods threshold iterations level Consistency Nearness Evenness
The proposed method 0.85 2 0.883 0.835 0.918 0.863
Wu and Xu (2018) 0.8 3 0.802 0.881 0.908 0.951
Xu, Zhong, et al. (2015) 0.8 3 0.797 0.815 0.894 0.858

exit-delegation mechanism. Our proposed model
paid attention to the adaptive CRP with hybrid
strategies for subgroups with different importance.
It is hard to say which method is the best.
6. Conclusions
Up to now, LSGDM methods developed greatly and
have been widely researched. However, studies from
some other perspectives, such as how to evaluate the
effects of different LSGDM methods, or how to
measure the decision results and how to find out
each subgroup’s contribution and quality in the
decision process, are relatively deficient.
In this study, we proposed three quality subgroup
indices to measure each subgroup’s influence on the
decision-making process. Next, based on this infor-
mation, we introduced a method to evaluate the
results of an LSGDM problem. Furthermore, effects
of different LSGDM methods can also be obtained
by comparing their performance under a specific
index. Another important contribution was the pro-
posal of the adaptive dynamic CRP. The consistency
degree of each subgroup was adjusted into the CRP
dynamically. Then, we divided the subgroups into
three categories according to their importance.
Three feedback strategies adapting to their weights
were introduced to improve the consensus level.
The effectiveness and practicality of our proposed
model was illustrated by an example of geological
hazard risk assessment.
Based on the comparison work, we found that
our consensus model performed better in conver-
gence speed. The consensus threshold of the pro-
posed model was 0.85 while the other two papers
reported a threshold of 0.8. Furthermore, our model
reached a consensus level of 0.883 after two rounds
of iterations. The other two studies reached a con-
sensus level of about 0.8 after three rounds of itera-
tions. Several factors can result in this, such as the
weight-determination method for subgroups, clus-
tering method or number of clusters.
In the near future, we will investigate other spe-
cific factors that may exert influence on LSGDM
results, such as the clustering algorithms and the
weight-determining methods. In addition, non-
cooperative behaviour detection and management is
also an important research topic in LSGDM. In the
CRP, experts may refuse to accept suggestions or
only partially make adjustments. In such case, we
can refer to Lahtinen, H€am€al€ainen, and Jenytin
(2019)’s method to design a preference elicitation
process such that the effects of biases counteract
each other. In this way, experts are not forced to
change their behaviours and make adjustments
on judgments.
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 877
Disclosure statement
No potential conflict of interest was reported by
the authors.
Funding
The work was supported by the National Natural Science
Foundation of China (71771156, 71971145).
ORCID
Ming Tang http://orcid.org/0000-0002-4338-699X
Huchang Liao http://orcid.org/0000-0001-8278-3384
Xiaomei Mi http://orcid.org/0000-0001-8651-0033
References
Ding, R. X., Wang, X. Q., Shang, K., & Herrera, F.
(2019). Social network analysis-based conflict relation-
ship investigation and conflict degree-based consensus
reaching process for large scale decision making using
sparse representation. Information Fusion, 50, 251–272.
doi:10.1016/j.inffus.2019.02.004
Farhadinia, B. (2013). A novel method of ranking hesitant
fuzzy values for multiple attribute decision-making
problems. International Journal of Intelligent Systems,
28(8), 752–767. doi:10.1002/int.21600
Herrera-Viedma, E., Cabrerizo, F. J., Kacprzyk, J., &
Pedrycz, W. (2014). A review of soft consensus models
in a fuzzy environment. Information Fusion, 17, 4–13.
doi:10.1016/j.inffus.2013.04.002
Lahtinen, T. J., H€am€al€ainen, R. P., & Jenytin, C. (2019).
On preference elicitation processes which mitigate the
accumulation of biases in multi-criteria decision ana-
lysis. European Journal of Operational Research, 282(1),
201–210. doi:10.1016/j.ejor.2019.09.004
Li, C. C., Dong, Y. C., & Herrera, F. (2019). A consensus
model for large-scale linguistic group decision making
with a feedback recommendation based on clustered
personalized individual semantics and opposing con-
sensus groups. IEEE Transactions on Fuzzy Systems,
27(2), 221–233. doi:10.1109/TFUZZ.2018.2857720
Liao, H. C., Xu, Z. S., Zeng, X. J., & Xu, D. L. (2016). An
enhanced consensus reaching process in group decision
making with intuitionistic fuzzy preference relations.
Information Sciences, 329, 274–286. doi:10.1016/j.ins.
2015.09.024
Liu, B. S., Shen, Y. H., Chen, X. H., Chen, Y., & Wang,
X. Q. (2014). A partial binary tree DEA-DA cyclic clas-
sification model for decision makers in complex multi-
attribute large-group interval-valued intuitionistic fuzzy
decision-making problems. Information Fusion, 18,
119–130. doi:10.1016/j.inffus.2013.06.004
Liu, B. S., Zhou, Q., Ding, R. X., Palomares, I., &
Herrera, F. (2019). Large-scale group decision making
model based on social network analysis: Trust relation-
ship-based conflict detection and elimination. European
Journal of Operational Research, 275(2), 737–754. doi:
10.1016/j.ejor.2018.11.075
Liu, X., Xu, Y. J., Montes, R., Ding, R. X., & Herrera, F.
(2019). Alternative ranking-based clustering and reli-
ability index-based consensus reaching process for hesi-
tant fuzzy large scale group decision making. IEEE
878 M. TANG ET AL.
Transactions on Fuzzy Systems, 27(1), 159–171. doi:10.
1109/TFUZZ.2018.2876655
Liu, Y., Fan, Z. P., & Zhang, X. (2016). A method for large
group decision-making based on evaluation information
provided by participators from multiple groups. Information
Fusion, 29, 132–141. doi:10.1016/j.inffus.2015.08.002
Ma, Z. Z., Zhu, J. J., Ponnambalam, K., & Zhang, S. T.
(2019). A clustering method for large-scale group deci-
sion-making with multi-stage hesitant fuzzy linguistic
terms. Information Fusion, 50, 231–250. doi:10.1016/j.
inffus.2019.02.001
Mata, F., Mart
ınez, L., & Herrera-Viedma, E. (2009). An
adaptive consensus support model for group decision-
making problems in a multigranular fuzzy linguistic
context. IEEE Transactions on Fuzzy Systems, 17(2),
279–290. doi:10.1109/TFUZZ.2005.856561
Nos
e, S. (1984). A unified formulation of the constant tem-
perature molecular dynamics methods. The Journal of
Chemical Physics, 81(1), 511–519. doi:10.1063/1.447334
Palomares, I., Mart
ınez, L., & Herrera, F. (2014). A con-
sensus model to detect and manage noncooperative
behaviors in large-scale group decision making. IEEE
Transactions on Fuzzy Systems, 22(3), 516–530. doi:10.
1109/TFUZZ.2013.2262769
Pang, J. F., & Liang, J. Y. (2012). Evaluation of the results
of multi-attribute group decision-making with linguistic
information. Omega, 40(3), 294–301. doi:10.1016/j.
omega.2011.07.006
P
erez, I. J., Cabrerizo, F. J., Alonso, S., Dong, Y. C., Chiclana,
F., & Herrera-Viedma, E. (2018). On dynamic consensus
processes in group decision making problems. Information
Sciences, 459, 20–35. doi:10.1016/j.ins.2018.05.017
P
erez, I. J., Cabrerizo, F. J., Alonso, S., & Herrera-Viedma,
E. (2014). A new consensus model for group decision
making problems with non-homogeneous experts. IEEE
Transactions on Systems, Man, and Cybernetics: Systems,
44(4), 494–498. doi:10.1109/TSMC.2013.2259155
Rodr
ıguez, A., & Laio, A. (2014). Machine learning.
Clustering by fast search and find of density peaks.
Science (New York, N.Y.), 344(6191), 1492–1496. doi:10.
1126/science.1242072
Rodr
ıguez, R. M., Labella, A.,
De Tr
e, G., & Mart
ınez, L.
(2018). A large scale consensus reaching process man-
aging group hesitation. Knowledge-Based Systems,
159(2018), 86–97. doi:10.1016/j.knosys.2018.06.009
Saint, S., & Lawson, J. R. (1994). Rules for reaching con-
sensus: A modern approach to decision making. San
Francisco: Jossey-Bass.
Song, Y. M., & Li, G. X. (2019). A large-scale group decision-
making with incomplete multi-granular probabilistic lin-
guistic term sets and its application in sustainable supplier
selection. Journal of the Operational Research Society,
70(5), 827–841. doi:10.1080/01605682.2018.1458017
Tang, M., Zhou, X., Liao, H., Xu, J., Fujita, H., &
Herrera, F. (2019). Ordinal consensus measure with
objective threshold for heterogeneous large-scale group
decision making. Knowledge-Based Systems, 180, 62–74.
doi:10.1016/j.knosys.2019.05.019
Torra, V. (2010). Hesitant fuzzy sets. International
Journal of Intelligent Systems, 25(6), 529–539. doi:10.
1002/int.20418
Wu, Z. B., & Xu, J. P. (2018). A consensus model for
large-scale group decision making with hesitant fuzzy
information and changeable clusters. Information
Fusion, 41, 217–231. doi:10.1016/j.inffus.2017.09.011
Xia, M. M., & Xu, Z. S. (2013). Managing hesitant infor-
mation in GDM problems under fuzzy and multiplica-
tive preference relations. International Journal of
Uncertainty, Fuzziness and Knowledge-Based Systems,
21(06), 865–897. doi:10.1142/S0218488513500402
Xu, X. H., Du, Z. J., & Chen, X. H. (2015). Consensus
model for multi-criteria large-group emergency deci-
sion making considering non-cooperative behaviors
and minority opinions. Decision Support Systems, 79,
150–160. doi:10.1016/j.dss.2015.08.009
Xu, X. H., Zhong, X. Y., Chen, X. H., & Zhou, Y. J.
(2015). A dynamical consensus method based on exit–-
delegation mechanism for large group emergency deci-
sion making. Knowledge-Based Systems, 86, 237–249.
doi:10.1016/j.knosys.2015.06.006
Xu, Y. J., Wen, X. W., & Zhang, W. C. (2018). A two-
stage consensus method for large-scale multi-attribute
group decision making with an application to earth-
quake shelter selection. Computers & Industrial
Engineering, 116, 113–129. doi:10.1016/j.cie.2017.11.025
Xu, Z. S., & Xia, M. M. (2011). Distance and similarity
measures for hesitant fuzzy sets. Information Sciences,
181(11), 2128–2138. doi:10.1016/j.ins.2011.01.028
Xu, Z. S., & Zhang, X. L. (2013). Hesitant fuzzy multi-
attribute decision making based on TOPSIS with
incomplete weight information. Knowledge-Based
Systems, 52, 53–64. doi:10.1016/j.knosys.2013.05.011
Zhu, B., & Xu, Z. S. (2014). Analytic hierarchy process-
hesitant group decision making. European Journal of
Operational Research, 239(3), 794–801. doi: 10.1016/j.
ejor.2014.06.019.