Visual Informatics 5 (2021) 26–33

Contents lists available at ScienceDirect

Visual Informatics
journal homepage: www.elsevier.com/locate/visinf

Evaluating user cognition of network diagrams

∗
Xiaojiao Chen a , , Xiaoteng Tang a,b , Zijing Luo c , Jiayi Zhang a,b
a
School of Art and Archaeology, Zhejiang University, Hangzhou, china
b
College of Computer Science and Technology, Zhejiang University, Hang zhou, China
c
School of Mechanical Engineering, Southeast University, Nanjing, China

article info a b s t r a c t

Article history: Edges crossing and nodes overlapping have a significant effect on the users’ recognition and compre-
Received 25 November 2021 hension of network diagrams. In this study, we propose a visual evaluation method for users’ cognition
Received in revised form 7 December 2021 of network diagrams. First, this method carries out a set of cognitive experiments to collect the user’s
Accepted 7 December 2021
cognitive performance that affects the variables, including accuracy and response time. The user’s pupil
Available online 11 December 2021
diameter is measured through an eye tracker to reflect their cognitive load. Second, the significance test
Keywords: points out the visual features as independent variables and then establishes an evaluation regression
Network diagram model. The experimental results show that the number of edges, edge length, node visual interference,
Shortest path and edge occlusion contribute to the evaluation models of response time, and edge occlusion and
Eye tracking the number of node connections contribute to the accuracy model. Finally, these evaluation models
Cognitive evaluation
demonstrate good predictability when assessing users’ cognition of network diagrams and provide
practical recommendations for their use.
© 2021 The Authors. Published by Elsevier B.V. on behalf of Zhejiang University and Zhejiang University
Press Co. Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction subjectively evaluate whether the visualization results meet cog-

Network diagrams with various interaction modes display net-
work data (Gardony et al., 2018). Many efficient visualization
methods and techniques facilitate data mining, further analy-
sis, as well as visual graphics rendering. In particular, layout
algorithms are primary visualization tools, which automatically
render a graph’s networks nodes and connections within several
seconds (Sun et al., 2010). Several studies have explored network
diagram algorithms (Huang et al., 2006; Purchase, 2002). For
instance, the layout rendering algorithm depicts the original data
(Tunkelang, 1994), the visual compression algorithm is suitable
for large-scale data (Zhang et al., 2019), and the force-directed
algorithm (FDA) is used for network diagram visualization. Cru-
cially, despite the continuous improvement of visualization tech-
niques, many urgent problems lead to different cognitive perfor-
mances in users, such as edge crossing and edge occlusion. In
most cases, researchers place emphasis on visual methods rather
than the user’s perception of topology. Thus, how to improve user
cognition of network diagrams is a neglected research area.
Researchers select appropriate algorithms to visualize data
based on experience and visualization requirements and then
∗ Corresponding author.
E-mail addresses: chenxiaojiao@zju.edu.cn (X. Chen),
tangxiaoteng@zju.edu.cn (X. Tang), 18795866350@163.com (Z. Luo),
12121188@zju.edu.cn (J. Zhang).
nitive requirements and the needs of visualization tasks. For a
more intuitive evaluation method, cognitive experiments can be
viewed as an effective usability testing method for the readability
evaluation model in this paper, which provides an objective and
physiological measurement of users’ cognition of node-link rela-
tionships. Therefore, the purpose of this study is to scientifically
evaluate and optimize the function of topology as well as filter
the best way to visualize data. Most importantly, it will actually
improve the readability of network diagrams for users. Previ-
ous research has described some user experiments to evaluate
network diagrams. For example, a study assessing visualization
techniques evaluated the effectiveness of visualizations by using
EEG and cognitive load (Anderson et al., 2011). Lace also proposed
a validated method to evaluate working memory during visual-
ization tasks using pupillometry (Padilla et al., 2020). This type
of user experiment only evaluates a specific network graph at a
time. Thus, if the cognition of a large number of network diagrams
is to be assessed, then a large number of repetitive experiments
have to be carried out. However, our work has been devoted to
solving this problem. Generally, the paper has these following
contributions:
(1) We design a task to seek the shortest path in a network
diagram that can be used to evaluate users’ cognitive per-
formance in finding nodes and edges.
(2) We filter specific visual features of network diagrams that
are significantly related to searching speed and accuracy,
which contributes to various task demands.

https://doi.org/10.1016/j.visinf.2021.12.004
2468-502X/© 2021 The Authors. Published by Elsevier B.V. on behalf of Zhejiang University and Zhejiang University Press Co. Ltd. This is an open access article under the
CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
X. Chen, X. Tang, Z. Luo et al.
(3) We propose a cognitive evaluation method by which the
prediction model avoids a large number of repetitive user
experiments and improves evaluation efficiency.
2. Related work
2.1. Visual evaluation metrics
In addition to the well-studied evaluation methods mentioned
above (the measurement of accuracy, response time, and pupil
diameter), visual metrics have been proven to reflect the cog-
nitive performance, which actually affects users’ cognition of
network diagrams. Purchase first proposed the assessment of
network diagrams, along with seven aesthetic evaluation indexes
for network node connection graphs (Purchase, 2002). These in-
dexes contain minimizing edge crossings, minimizing edge bends,
maximizing symmetry, maximizing the minimum angle between
edges leaving a node, maximizing edge orthogonality, maximiz-
ing node orthogonality, and maximizing consistent flow direction.
However, in practical tests, some conflicts exist among differ-
ent metrics. Moreover, there may be some absolute antagonism
making it more challenging to evaluate the visual layout (Ware
et al., 2002). Therefore, some researchers combined the geometric
feature calculations of nodes and connecting edges to propose
a visual readability evaluation index for network diagrams from
the user’s perception (Kobourov et al., 2014). The readability
evaluation index can be judged in different experiments for users’
particular requirements to verify whether it can truly reflect
the observation effect on the network diagram (Mutzel, 2006).
For example, user diagram recognition tasks were designed to
compare and evaluate force-directed layouts, orthogonal layouts,
and hierarchical layouts (Pohl et al., 2009). The experimental
results suggested that the readability index filtered the layouts
that met users’ demand of cognition.
Furthermore, Dunne considered multiple aspects of user cog-
nition of network visualizations and proposed node-node overlap,
edge crossing angle, angular resolution, group overlap, and vi-
sualization coverage (Dunne et al., 2015). Then Dunne further
proposed the node readability index, side length readability in-
dex, and overall layout readability index to point out the lay-
out’s position with poor readability of the topology structure and
measure the readability difference of different designs. Besides,
various readability evaluation indexes differ from their principal
utilities. Huang explored three readability indexes for seeking the
shortest path between network nodes to improve network layout
(Huang et al., 2006). They were the node’s minimum adjacency
angle, the average adjacency angle of the node, and the standard
deviation of the node’s adjacency angle. Experiments showed that
measuring the user’s time and accuracy in seeking the shortest
path with the minimum adjacency angle was most effective. And
the increased crossing on small and large graphs with varying
densities has been studied (Kobourov et al., 2014). The latest
research claims that the readability index is related to the user’s
visual perception of diagrams. However, there are some problems
with the evaluation of the readability indexes. Accordingly, users’
cognition of network diagrams could be affected by multiple
factors. Therefore, a single index that assesses user cognition
effect might reduce the reliability of the evaluation.
2.2. The evaluation method of cognitive performance
According to Kosara et al. user studies can provide scientific
advice and then measure a network’s performance (Kosara et al.,
2003). Similarly, user studies can answer specific data questions
in different ways (William and Robert, 1983). In addition to sub-
jective evaluation, working memory plays a vital role in cognitive
27
Visual Informatics 5 (2021) 26–33
activities, and it is responsible for the processing of task-related
information, including reasoning and comprehension (Anderson
et al., 2011). Besides, EEG (Klimesch, 1999) measured that work-
ing memory is reflected by changes of the alpha and theta. Sim-
ilarly, Wang explored the neural circuit basis for spatial working
memory (Constantinidis and Wang, 2004). Previous research has
also revealed that cognitive performance tasks are essential ways
to evaluate cognitive ability. Specifically, cognitive performance
involves two main factors: accuracy and response time. They are
applied to assess the efficiency of visualization (Szafir et al., 2018;
Kyllonen and Zu, 2016; Cronbach and WG, 1951), especially in
distinguishing whether a design assists users in understanding
data or not. Kobourov et al. (2014) measured accuracy and com-
pletion time to assess the task performance for automatically
generated layouts for graphs. Amende also suggested that effec-
tiveness and efficiency, measured by task correctness, were the
most widely used endogenous variables (Amende, 2010). Though
many researchers have claimed that both speed and accuracy can
evaluate the effect of visualization, numerous cognitive scientists
(Kyllonen and Zu, 2016; Just et al., 2003) still have questions
about the precision of these two metrics. This may be attributed
to one subject responding more quickly and correctly than the
other one. Another reason comes from a shortage of time. Thus,
speed (reaction time) and accuracy are insufficient for evaluating
cognition of network diagrams.
Consequently, task requirements and eye-tracking results have
been used to improve the accuracy of cognitive assessment.
According to Padilla et al. (2020), dual-task experimental designs
evaluate the working memory in visualization. They concluded
that task completion times and pupillometry could reflect task
difficulty and cognitive load. Eye-tracking has also been used to
compare graph layouts’ readability. Nuamah showed that subjec-
tive performance (NASA task load index) and cognitive perfor-
mance (response time and accuracy) could evaluate the cognitive
fit (Nuamah, 2020). In Afridi’s research, the same NASA-TLX test
was conducted (Afridi, 2020). And then, visual search demands
and memory load through pupil dilation were used to provide
supplemental data (Stolte et al., 2020). In terms of rationality
and accuracy, Chen et al. (2017) concluded that eye-tracking
equipment could obtain values precisely and then evaluate digital
interfaces in an effective way. Therefore, this paper considers ac-
curacy and response time as the two main criteria for evaluating
a user’s cognitive performance in visualization. By collecting and
analyzing pupil diameter data from the eye tracker, the accuracy
of assessing users’ cognitive performance is greatly enhanced.
3. Method
We conduct an empirical cognitive experiment to build a real-
time model for assessing the visual cognitive performance of the
network diagrams. This paper collects the Les Misérables and
dolphin data set and generates 60 sets of visual experiment ma-
terials using the Fruchterman–Reingold (FR) algorithm. Then, the
participants are requested to search the shortest path in network
diagrams using eye-tracking technology. The experimental pro-
gram records cognitive performance data (accuracy and response
time) and pupil diameter data. In these data, 30 sets of cognitive
results in S2 are used as test data sets. Then the visual features
that affect topological cognition in visualization are quantified
into evaluation index variables. Furthermore, regression model-
ing analysis is performed on the index variables and experimental
cognitive performance to generate an objective evaluation model.
Finally, the fitted data set S1 is used to verify the effectiveness of
the model (see Figs. 7 and 8).
X. Chen, X. Tang, Z. Luo et al. Visual Informatics 5 (2021) 26–33

Table 1
Visual characteristics of nodes and edges.
Visual metrics Graphical description

Number of node connections

Edges crossing

Node visual interference

Fig. 1. The network of characters in Les Misérables network generated by FR

force layout algorithm.
Edge occlusion

Edges length

3.1. Cognitive experiment

3.1.1. Experiment design

This paper carries out the task of searching the shortest path;
the accuracy and reaction time are recorded to evaluate cognitive
performance. Besides, the pupil diameter is used as the evaluation
criterion of cognitive load. The shortest path in a network dia-
gram possesses the least number of connections among the two
nodes’ paths in a network diagram. The shortest path perception
reflects the visual characteristics of nodes and edges. According to
the previous research (Huang et al., 2006; Dunne et al., 2015), the
main visual features, including the number of path edges, edges
crossing, edges length, node visual interference, edge occlusion,
the number of node connections, are selected as the independent
variables. Table 1 describes these visual characteristics. The red
nodes and edges represent their visual features. The experiment
task is designed carefully. At the beginning of the experiment,
the starting point and the target node are randomly marked by
red color. Then, the participants identify the shortest path and
remember the number of shortest paths. In the end, they are
required to record it on the computer.
3.1.2. Stimuli
The Dolphin data set with 62 nodes and 256 edges and the net-
work in Les Misérables (Dickinson et al., 2013) with 77 nodes and
508 edges are used as the experimental stimulus material (Knuth,
2000). The FR force layout algorithm is selected to generate the
network diagrams. Automatically arranging the nodes and edges
in network diagrams has always been an important research area
in visualization, and the layout algorithm based on the force-
directed model is the most widely used (Chao et al., 2015). In our
work, FR force layout algorithms were used to conduct a cognitive
experiment. As shown in Fig. 1, the colors, shapes, and sizes of
nodes and edges in the visual layout are the same. Geometric
properties of layout among each node and edge are different,
such as position, edge length, and edge angle. Sixty experimental
stimuli are divided into two groups: the fitted experimental data
set S1 and the tested experimental data set S2.
Many features seriously affect the readability of node-link
relationships as stated earlier. When facing the network structure
of the network diagrams, some people observe the macroscopic
organization of the network, while purposeful users additionally
have nodes relationship searching included in the cognitive be-
havior. The shortest path cognitive decision task is preferable,
28
Fig. 2. An example of calculating the shortest path metrics for network
diagrams.
corresponding with the high difficulty decision selecting behav-
ior. We choose the shortest path distinction mission, namely,
choice reaction time task, as the representative of all cognitive ex-
periment searching tasks. In a network diagram, the shortest path
is to find a path between two vertices or nodes such that the sum
of the weights of its constituent edges is minimized. Visual char-
acteristics of nodes and edges affect the shortest path cognition.
In order to evaluate these characteristics efficiently, these visual
features are quantified and accumulated. The visibility of the data
sequence is reflected by the relationship between the nodes (Shi
and Xu, 2020). And those evaluation metrics are implemented in
a JavaScript API (application programming interface) to generate
visual graphic samples, just like the method used (Dunne et al.,
2015). Fig. 2 shows that the shortest path in this graph passes
through the nodes q1 , q2 , q3 , and the passed edges are l1 , l2 . For
example, the shortest path node edge can be calculated by the
following formula: ξec = ℵec (q1 ) + ℵec (q2 ) + ℵec (q3 ). Similarly,
the connection angle metrics of the shortest path formula is ξL =
L (l1 ) + L (l2 ) + L(l3 ).
With a given undirected and weightless network diagram G,
nodes vi and vj , the number of edges passed by the shortest path
p, and the set of edges on the shortest path is {e_1,e_2. . . ep}. Due
to complex searching tasks and various visual factors, seven visual
features, which affect the shortest path cognition, are quantified
as evaluation metrics. The specific calculation methods are shown
below.
(1) Number of shortest path edges ξn
Obviously, the greater the number of shortest path edges
between nodes, the more difficult it is for users to find them,
and the performance of finding will decrease accordingly. The
calculation method is:
ξn (vi , vj ) = p (1)
X. Chen, X. Tang, Z. Luo et al.
Fig. 3. Examples of the misreading of the shortest path caused by the cover of
the shortest path.
(2) Shortest path with edges crossing ξec
The intersection angle of the shortest path between nodes and
intersection numbers may impact users searching for the shortest
path. Fewer edge crossings and a 70–90 degree crossing angle will
make the layout more available to users. The calculation method
is:
∑ p
ξec (vi , vj ) = Cei
i=1
∑m
i=1 (1−t·cosθi )
Ceij = m′
(2) Shortest path with edges length ξel
m’ = m-k_j-k_i+1, m’ is the maximum number of crosses
within a certain edge and other edges, j and k respectively rep-
resent two end nodes of the connected edge. t is the crossover
parameter of two sides.
If the total edge between nodes vi and vj are too long in the
shortest path, it may lead to users to spend more observation
time to distinguish. Thus, the entire length of the shortest path
should be controlled within a certain range to facilitate observing
quickly. Therefore, this metric accumulates all the edge length
metrics on the path:
p
∑ 3.1.4. Apparatus and procedures
ξel (vi , vj ) = L(ei )
i=1
(4) Shortest path with node visual interference ξnvi
The edges of the nodes on the shortest path intersect with
other irrelevant edges, which will cause visual interference for
users to search for the path. In order to quantify and calculate
the visual metrics, the visual interference of all nodes should
be calculated. After corresponding visual indexes are quantified
and calculated, the calculation results of all nodes need to be
accumulated. The calculation method is:
p
∑ pre-experiments were carried out to design experimental details
ξec (vi , vj ) = ℵsvi (vi )
i=0
(5) Shortest path with edge occlusion ξeo
If the connecting edges in the path are covered, it will probably
cause users’ misunderstanding on the shortest number of paths.
For example, the dashed line in Fig. 3-a is one of the correct
shortest paths that can be counted up to five in total. However,
the red connecting edge and the red node are covered, which
may interfere with users to consider that there is a connection
between the nodes, and the number of shortest path connections
is 3 (dotted lines), as Fig. 3-b was shown.
For the edge ei in the given shortest path, the distance d from
all nodes is calculated. If d is less than the radius r of the node, it
Visual Informatics 5 (2021) 26–33
is marked as 1. Otherwise, the metric value is recorded as 0, and
then accumulated, as shown in Eq. (6).
p
∑
ξeo (vi , vj ) = E(ei ) (6)
i=0
E(ei ) is the covering coefficient of connecting edges in the path,
and the calculation method is shown in Eq. (7).
1,
{
d≤r
Eei = (7)
0, d>r
(6) Number of shortest path node connections ξnc
The number of node connections in the path increases, and
the difficulty for users to find the path rises up, which directly
affects the user’s searching performance. Therefore, the number
of connected edges of all nodes on the shortest path is calculated
and shown in Eq. (8).
p
∑
ξnc (vi , vj ) = d(vi ) (8)
i=0
In addition, the adjacent angles of the shortest path edges are
also taken into account. Particularly, the angles of the shortest
(2) path edges should be clearly identifiable. Then the calculation
method is similar to that of the adjacent edges of the nodes.
Since the metric of the shortest path task is an accumulated
(3)
value, the larger the index value, the greater the degree of visual
interference.
3.1.3. Participants
Thirty undergraduate and postgraduate students with 12 males
and 18 females volunteered for this experiment. They ranged in
age from 21 to 27, with normal or corrected vision (minimum
score of 20/40 by a Snellen near acuity test) and no color blind-
ness or color vision deficiency. Their demographics, including
age, gender, occupation, and visual conditions, were recorded.
Besides, the ethics committee of the University approved the
investigation, and all participants gave consent to the experiment.
(4) The experiments were conducted in the laboratory that sim-
ulated the general working environment. Experimental stimuli
were presented on a 21 inches monitor with a brightness of 93
cd/m2. All information was displayed with a resolution of 1280
×1024 pixels and a refresh rate of 60 Hz. We set the viewing
distance range from 550 mm to 680 mm and used Tobiix2-300
eye tracker to collect use pupil diameter.
Tobii studio (Tobii x2-300 eye tracker supporting software)
was used to present 60 network diagrams randomly. And the
diagram stimulus was marked with two red nodes by a random
algorithm. All 20 participants identified the number of the short-
est path between two red nodes without a time limit. A set of
(5) and select the stimuli. In a sequence, the instructions in the center
of the screen explained the experimental process and operation
steps. Then the participants could press any key on the keyboard
to start the experiment.
In the experiment process, the screen shows blank for 1000 ms
which assisted in eliminating the visual residue. Then, the symbol
‘‘+’’ appeared 1500 ms to indicate the location of the fixation
point. Afterward, the network diagrams were displayed with a
time limit. The participant made judgments according to the
generated graphs. If the participant confirmed the answer, they
would press the corresponding number button. After completing
25 material tests, the participants were free to rest for 2 min. The
entire experiment, including pre-experiment, formal experiment,
and break, took 20∼30 min to complete (see Fig. 4).
29
X. Chen, X. Tang, Z. Luo et al.
Table 2
Linear regression model coefficient table of response time and correct rate.
User performance Metrics Correlation
coefficient R
Number of edges ξn edge
occlusion ξeo node visual
Response time 0.874
interference ξnv i edges
length ξel
edge occlusion ξeo Number
Accuracy 0.737
of node connections ξnc
Fig. 4. Experimental process.
3.2. Establishing the shortest path cognitive assessment model
The pre-experiment results showed a significant linear rela-
tionship between visual features and user cognitive performance
(reaction time and accuracy). The specific correlation level could
be obtained by correlation test. The progressive regression anal-
ysis of SPSS (Neokleous and Schizas, 2011) was used to carry out
multiple linear regression modeling between the index variables
of fitted data set S1 and cognitive performance. The P-value was
used to exclude the evaluation index variables with P ≥ 0.1, and
the regression model with the highest correlation coefficient R
was selected. According to Nadine’s review of information visu-
alization (Amende, 2010), regression analysis or variance analysis
could be used to analyze the reliability and validity of the empir-
ical studies. Particularly, the general form of the multiple linear
regression model is shown in Eq. (9), while K is the number
of evaluation variables, βi (i = 1, 2, . . . , k) called regression
coefficient. βi (i = 1, 2, . . . , k) is called regression coefficient.
Yi = β0 + β1 X1 + β2 X2 + · · · + βi Xi + · · · + βn Xn (9)
A multiple linear regression cognitive evaluation model is
established by significant evaluation variables and cognitive per-
formance, which is used to predict users’ cognitive effect on
network diagrams.
4. Results
4.1. Correlation analysis between visual features and response time
The scatterplot of visual features and reaction time is estab-
lished. Seven visual features in the fitted data set (S1) are used
as independent variables for scatterplot analysis and plotting.
The x-axis represents various feature variables, and the y-axis
represents the response time that completes the task. As shown
in Fig. 5, the number of shortest path edges, the node visual inter-
ference, and the path length is positively correlated with reaction
30
Visual Informatics 5 (2021) 26–33
Constant Regression P
coefficient B
2.757 0.000
0.989 0.045
−5.082
2.637 0.011
2.272 0.043
−0.238 0.000
1.014
−0.187 0.014
Fig. 5. Correlation analysis between linearly distributed evaluation indicators
and response time.
time. And there is no significant linear correlation between other
indicators and reaction time.
Furthermore, Pearson correlation is used to analyze these
three visual features, and the detailed correlation features are
shown in Table 2. First, a significant correlation exists between
the path length and reaction time (P = 0.013, P < 0.05), and
the correlation coefficient r is 0.455, which is between 0.4 and
0.6. Thus, a positive and moderate correlation exists between the
shortest path edge length and reaction time. Second, there is a
significant correlation between the node visual interference and
the reaction time (P = 0.008, P < 0.05), and the correlation
coefficient r is 0.483, which depicts a positive and moderate
correlation. Finally, the correlation results between the number of
node connections and reaction time show a significant correlation
(P = 0.045, P < 0.05), R = 0.374, which presents a positive and
weak correlation.
(2) Correlation analysis between visual feature and accuracy
The scatterplot of visual feature and accuracy illustrates that
the accuracy was related to the number of node connections,
where p = 0.001, P < 0.05, r = −.0570. According to the
analysis of this data, node connections are negatively and moder-
ately correlated with accuracy. The scatterplot of other indicators
shows that there was no apparent linear relationship between the
metrics and the accuracy, and the correlation analysis results are
also verified.
4.2. Eye-tracking results
Generally, pupil diameter is 1.5–3 mm if there is no stimula-
tion, while it will become more significant after being stimulated.
X. Chen, X. Tang, Z. Luo et al.
Fig. 6. Comparison of pupil diameters under different edge occlusion.
Hence, the change of pupil diameter can be used to infer the
cognitive load (Yang and Wang, 2018). The relationship between
pupil diameter and edge occlusion (the value of edge occlusion
index is 0 or 1) is analyzed. As shown in Fig. 6, the x-axis and
y-axis represent the cumulative sum of pupil diameter and edge
occlusion, respectively. In addition, the pupil diameter suggests
that under the same number of paths, the increase in edge cov-
erage will dilate the pupil diameter, leading to an increase in
cognitive load. With the same edge occlusion, the rise in the
number of paths also causes users’ cognitive load to increase.
Therefore, edge occlusion and the number of paths are highly
correlated with users’ cognitive load, and these two variables
should be considered in the modeling.
4.3. Establishment and validation of shortest path cognitive evalua-
tion model
(1) Establishment of the evaluation model
Stepwise & multiple linear regression analysis is used to filter
the variables with significance factor (P ≥0.05), and then the
model with the highest correlation coefficient r is selected. The
coefficient data of the response time model and accuracy model
are shown in Table 2.
In the construction of the regression model, the response time
model is composed of four positively correlated features: the
number of edges, edges length, node visual interference, and edge
occlusion. The correlation R-value indicates that the degree of
linear correlation in the reaction time model is 87.4%. The degree
of linear correlation in the correct rate model is 73.7%. Therefore,
both models mentioned above have good predictability. Substi-
tute the correlation coefficient into Eq. (8) and then form into
Eqs. (9) (10). The regression model of reaction time and accuracy
is established. Also, the results of variance analysis demonstrate
that the interaction between variables is not significant, and it has
no effect on the accuracy of the model.
Srt = 2.757ξn + 0.989ξeo + 2.637ξnv i
(10)
+2.272ξ el − 5.082
Sa = −0.238ξeo − 0.187ξnc + 1.014 (11)
(2) Validation of evaluation model
The comparisons of the fitted value and the experimental
value are shown in Figs. 7 and 8. The experimental results (solid
line) of both two data sets and the model fitted results (dotted
line) indicate a higher fitted degree in the reaction time and accu-
racy, having a similar prediction trend. According to Table 3, the
two correlation coefficients both have high values in correlation
coefficient verification that indicates a high correlation with the
experimental cognitive results of users of the model. Therefore,
the shortest path cognitive model can be regarded as a valuable
predictor for the cognitive evaluation of network diagrams.
31
Visual Informatics 5 (2021) 26–33
Fig. 7. Comparison of the fitted data set Y1 and the model result Y1’.
Fig. 8. Comparison of test data set Y2 and model result Y2’.
Table 3
Correlation coefficients between model calculation results of different data sets
and experimental evaluation results.
Data Set Cognitive Spearman’s rank Kendall’s tau-b
performance correlation correlation
coefficient coefficient
Response time 0.975 0.743
Fitted data set
accuracy 0.884 0.725
Response time 0.925 0.798
Tested data set
accuracy 0.812 0.654
5. Discussion
This work proposes an evaluation model of user cognition
of network diagrams and reduces users’ cognitive barriers. We
explore the correlation between diagram features and cognitive
performance, and the results show that four visual features are
significantly related to response time. Furthermore, two visual
features are directly associated with accuracy. Then an evaluation
model is established and verified. The verification results indicate
that the shortest path cognitive model had good predictabil-
ity for the feature evaluation of network diagrams. Therefore,
it is important to evaluate and filter the appropriate network
algorithm and improve the efficiency of users’ recognition and
comprehension of diagrams.
A linear relationship is described between response time and
four visual features, including the number of edges, edge length,
node visual interference, and edge occlusion. These four features
are used as the independent variables to build the regression
model of response time. The results identify a positive but weak
correlation between the number of edges and the response time.
The scatterplot illustrates that the linear relationship is signif-
icant when the number of sides of the shortest path is small.
Nevertheless, there will be no considerable linearity with more
edges. Therefore, to a certain extent, the increase in the number
of shortest path node connections may cause an increase in
cognitive difficulty and response time. A positive and moderate
correlation is seen between accuracy and two visual features (the
edge occlusion and the number of node connections). This is
X. Chen, X. Tang, Z. Luo et al.
because as the edge occlusion increases, users’ search efficiency
for the shortest paths in the network diagrams decreases simul-
taneously. Similarly, the more visual interferences a search task
has, the longer it takes to complete. A similar result was reported
by Lam that too many interactions or interferences might create
a cognitive overload, specifically when interactions could cause
a simultaneous change (Lam et al., 2012). And in terms of node
connection numbers, Zhang et al. practiced the principle of ‘‘total
variable length to be the shortest’’ and defined the average length
of the optimal edge as an index to evaluate the algorithm’s layout
when studying a multi-layer network structure (Zhang et al.,
2019). There is also a positive and weak correlation between
the shortest path with edge occlusion and the response time.
Specifically, the more serious edge occlusion a network diagram
has, the greater it impacts on user experience and the longer
the search time. In fact, the influence of edge occlusion on the
complexity of network diagrams has been widely studied, and
most of the research problems are about alleviating the factors
that affect user cognition. The response problems are consistent
with the results of this research. For example, in 1995, Purchase
(2002) initially proposed a network diagram visual evaluation.
Kobourov carried out a set of controlled experiments to explore
how edge crossing affects the comprehensibility of a graphic lay-
out and found that the number of edges crossing was an essential
feature in evaluating network diagrams (Kobourov et al., 2014).
For large and dense graphs, an increase in the number of edge
crossings corresponded with no statistically significant decrease
in performance time. For network layouts, many researchers have
designed a variety of algorithms to reduce the space occupied by
nodes around the problem of edge concealment (Misuse et al.,
1995; Li et al., 2005). Particularly, many force-oriented layout
algorithms have been applied to solve these problems (Heer et al.,
2005). However, force-oriented methods usually cannot guaran-
tee that all overlaps have been eliminated. Edge occlusion is still
an important factor affecting visual cognition. Sun et al. (2019)
proposed that excessive edges can make the network visually
cluttered, making it hard to explore. Therefore, edge occlusion
should be used as an important evaluation index in a cognitive
model.
The edge occlusion and the number of node connections are
viewed as independent variable indicators for building the accu-
racy regression model. The correlation analysis result between
the experimental metrics and the correct rate shows that R =
−.0570, R<0, thus, the number of shortest path node connections
and the accuracy are negatively correlated to a moderate degree.
When the number of nodes increases, the complexity of path-
searching increases, leading to a decrease in users’ accuracy.
In terms of edge occlusion, Huang found that edges crossing
has a significant impact on user preferences and task perfor-
mance (Huang et al., 2006). Similarly, Kobourov suggested that
increasing the number of edges crossing may cause a statistically
significant decline in accuracy (Kobourov et al., 2014). In the
analysis of visualization, excessive nodes or node details will
impose a cognitive burden on users to understand the provided
information (Tory, 2004). The number of node connections is
transferred to evaluation metrics to evaluate the accuracy. For in-
stance, Dunne suggested that node overlap resulted in a negative
impact on the accurate calculation of the number of individ-
ual nodes (Dunne et al., 2015). Huang et al. found that nodes’
positions and angle resolutions play an important role in user
perception (Huang et al., 2006).
6. Conclusions
A network diagram is the most commonly used schematic to
depict the connections between nodes in a computer network.
32
Visual Informatics 5 (2021) 26–33
However, little is known about the various cognitive problems in-
volved in users identifying and understanding network diagrams.
In this study, we propose a visual evaluation method for users’
cognition of network diagrams. Cognitive research holds that ac-
curacy and response time can be viewed as the two main criteria
to evaluate users’ cognition. Along with the factors mentioned
above, eye-tracking technology can obtain pupil diameter, which
reflects cognitive load. We carried out two cognitive experiments
to measure cognitive performance and obtain pupil diameter,
and performed regression analyses on the results. An evaluation
model of regression analysis between significant visual features
and cognitive performance was established to evaluate the di-
agram layout. In particular, edge occlusion and the number of
edges are considered independent variables to build the accuracy
regression model. The number of edges, node visual interference,
edge length, and edge occlusion are the independent evaluation
variables of the response time regression model. The comparisons
of the fitted value set and the experiment value set assume
a higher matching degree in response time and accuracy. The
correlation coefficient verification also proved the predictability
of the model.
There are two main limitations of this study. First, all the
participants were students and relatively young, meaning the
sample is not particularly representative. Thus, future experimen-
tal work should recruit participants from all walks of life and of a
wider range of ages. Second, this study focuses on a typical visual
cognitive task (the task of seeking the shortest path). More types
of tasks and more in-depth analyses need to be carried out in
assessing the evaluation of network diagrams.
CRediT authorship contribution statement
Xiaojiao Chen: Conceptualization, Methodology, Validation,
Formal analysis, Writing – review & editing, Funding acquisition.
Xiaoteng Tang: Investigation, Data curation, Writing – original
draft, Writing – review & editing. Zijing Luo: Methodology, Val-
idation, Formal analysis, Resource, Data curation. Jiayi Zhang:
Writing – review & editing.
Declaration of competing interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared
to influence the work reported in this paper.
Acknowledgments
The work was supported by the Provincial Key Social Science
Foundation of Zhejiang, China (Grant No. 22JCXK03Z) and the
Fundamental Research Funds for the Central Universities, China.
We would like to thank the reviewers for their constructive
comments and all the participants for actively participating in our
experiment.
Ethical approval
The experiment was approved by the ethics committee of the
Zhejiang University, and all participants signed consent before
they took part in the study.
X. Chen, X. Tang, Z. Luo et al.
References
Afridi, A.H., 2020. NASA-TLX–Based workload assessment for academic resource
recommender system. Pers Ubiquit Comput..
Amende, N., 2010. A Structured Review of Information Visualization Success
Measurement, presented at the Proceedings of HCI 2010.
Anderson, E.W., Potter, K.C., Matzen, L.E., et al., 2011. A user study of visualiza-
tion effectiveness using EEG and cognitive load. Comput. Graph. Forum 30
(3), 791–800.
Chao, S., Cheng, T., Hui, L., et al., 2015. Survey on automatic network layouts
based on force-directed model. Comput. Eng. Sci. 37 (3), 457.
Chen, X., Xue, C., Chen, M., Tian, J., 2017. Quality assessment model of digital
interface-based on eye-tracking experiments. J. Southeast Univ. (Nat. Sci. Ed.)
47 (1), 38–72.
Constantinidis, C., Wang, X.-J., 2004. A neural circuit basis for spatial working
memory. Neuroscientist 10 (6), 553–565.
Cronbach, L.J., WG, W., 1951. Time-limit tests: Estimating their reliability and
degree of speeding | SpringerLink. Psychometrika 16, 167–188.
Dickinson, B., Valyou, B., Hu, W., 2013. A genetic algorithm for identifying
overlapping communities in social networks using an optimized search
space. Soc. Netw. 2 (4), 193–201.
Dunne, C., Ross, S.I., Shneiderman, B., Martino, M., 2015. Readability metric
feedback for aiding node-link visualization designers. IBM J. Res. Dev. 59
(2/3), 14:1–14:16.
Gardony, Aaron L., Martis, Shaina B., Taylor, Holly A., Brunyé, Tad T., 2018. Inter-
action strategies for effective augmented reality geo-visualization: Insights
from spatial cognition. Human-Computer Interaction.
Heer, J., Card, S., Landay, J., 2005. Prefuse: A toolkit for interactive informa-
tion visualization. In: Proceedings of SIGCHI Human Factors in Computing
Systems. pp. 421–430.
Huang, W., Hong, S.-H., Eades, P., 2006. Layout effects on sociogram perception.
Graph Drawing 3843, 262–273.
Just, M.A., Carpenter, P.A., Miyake, A., 2003. Neuroindices of cognitive workload:
Neuroimaging, pupillometric and event-related potential studies of brain
work. Theoretical Issues Ergon. Sci. 4 (1–2), 56–88.
Klimesch, W., 1999. EEG alpha and theta oscillations reflect cognitive and
memory performance: a review and analysis. Brain Res. Rev. 29 (2–3),
169–195.
Knuth, D.E., 2000. The Stanford GraphBase: A Platform for Combinatorial
Computing Donald, Vol. 3. pp. 1–4.
Kobourov, S.G., Pupyrev, S., Saket, B., 2014. Are crossings important for drawing
large graphs? In: Progress in Pattern Recognition, Image Analysis, Computer
Vision, and Applications, Vol. 8827. pp. 234–245.
Kosara, R., Healey, C.G., Interrante, V., et al., 2003. Thoughts on user studies:
Why, how, and when. IEEE Comput. Graph. Appl. 23, 22–25.
Kyllonen, P., Zu, J., 2016. Use of response time for measuring cognitive ability.
J. Intell. 4 (4), 14.
33
Visual Informatics 5 (2021) 26–33
Lam, H., Bertini, E., Isenberg, P., et al., 2012. Empirical studies in information
visualization: Seven scenarios. IEEE Trans. Vis. Comput. Graphics 18 (9),
1520–1536.
Li, W., Eades, P., Nikolov, N., 2005. Using spring algorithms to remove node
overlapping. In: APVis ’05: proceedings of the 2005 Asia-Pacific symposium
on Information visualisation, Vol. 45. pp. 31–140.
Misuse, K., Eades, P., Lai, W., Sugiyama, K., 1995. Layout adjustment and the
mental map. J. Vis. Lang. Comput. 6 (2), 183–210.
Mutzel, P., 2006. An alternative method to crossing minimization on hierarchical
graphs. SIAM J. Optim. 11 (4), 16.
Neokleous, K.C., Schizas, C.N., 2011. Computational modeling of visual selective
attention. Procedia Comput. Sci. 7, 244–245.
Nuamah, J.K., 2020. Evaluating effectiveness of information visualizations using
cognitive fit theory: A neuroergonomics approach. Applied Ergon. 88, 9.
Padilla, L.M.K., Castro, S.C., Quinan, P.S., et al., 2020. Toward objective evaluation
of working memory in visualizations: A case study using pupillometry and
a dual-task paradigm. IEEE Trans. Visual. Comput. Graph. 26 (1), 332–342.
Pohl, M., Schmitt, M., Diehl, S., 2009. Comparing the Readability of Graph Layouts
using Eyetracking and Task-Oriented Analysis. The Eurographics Association.
Purchase, H.C., 2002. Metrics for graph drawing aesthetics. J. Vis. Lang. Comput.
13 (5), 1501–5166.
Shi, H., Xu, M., 2020. A multiple-attribute decision-making approach to
reinforcement learning. IEEE Trans. Cogn. Dev. Syst 12 (4), 695–708.
Stolte, M., Gollan, B., Ansorge, U., 2020. Tracking visual search demands and
memory load through pupil dilation. J. Vision 20 (6), 21.
Sun, Y., Jiang, Y., Zhao, X., 2010. Survey on the research of network visualization.
Comput. Sci. 37 (2), 12–30.
Sun, D., et al., 2019. Planningvis: A visual analytics approach to production
planning in smart factories. IEEE Trans. Visual. Comput. Graphics 26 (1),
579–589.
Szafir, D.A., good, The., bad, the., 2018. And the biased: five ways visualizations
can mislead (and how to fix them). Interactions 25 (4), 26–33.
Tory, M., 2004. Human factors in visualization research. IEEE Trans. Vis. Comput.
Graphics 10 (1), 72–84.
Tunkelang, D., 1994. A practical approach to drawing undirected graphs. Comput.
Sci. 30.
Ware, C., Purchase, H., Colpoys, L., McGill, M., 2002. Cognitive measurements of
graph aesthetics. Inf. Vis. 1 (2), 103–110.
William, S.C., Robert, M., 1983. Graphical perception: theory, experimentation,
and application to the development of graphical methods. J. Amer. Statist.
Assoc. 79 (387), 531–554.
Yang, X., Wang, Q., 2018. The correlation analysis of eye movement indicators,
controler’s workload and aircraft dynamics. Sci. Technol. Eng. 18 (15).
Zhang, X., Wu, L., Yu, S., et al., 2019. A multi-force directed layout algorithm for
multilayer networks visualization. J. Comput.-Aided Des. Comput. Graph. 31
(4), 639.