This article has been accepted for inclusion in a future issue of this journal.
Content is final as presented, with the exception of pagination.
IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS
Deep Reinforcement Learning Enabled Multi-UAV
Scheduling for Disaster Data Collection
With Time-Varying Value
Pengfu Wan , Gangyan Xu , Member, IEEE, Jiawei Chen , and Yaoming Zhou
Abstract— The congestion and disruption of information
infrastructures frequently happen during disasters, which would
hinder the understanding of disaster scenarios, and thus impede
rapid response activities. With the advantages of high flexibility
and efficiency, this paper proposes to use UAVs as temporary and
mobile relays for disaster data collection. However, different from
many existing data collection scenarios in industrial sectors, the
disaster data value varies with UAV arrival time and service time
in terms of their importance for disaster response, which makes
the scheduling of UAVs challenging. To address such a problem,
this paper proposes an attention-based Deep Reinforcement
Learning (DRL) method for multi-UAV scheduling considering
time-varying data value. Specifically, the problem is modeled as
a specific team orienteering problem with time-varying value.
Then the relationships between UAV route selection and service
time at each node are analyzed, based on which the computing
efficiency for solution algorithms can be improved. After that,
an attention-based DRL method is developed, with a calibrated
attention model and decoding method. Finally, systematic compu-
tational experiments are conducted to evaluate the performance
of the proposed method, which demonstrates its superiority over
popular methods in UAV scheduling, especially for large-scale
and complex scenarios.
Index Terms— Disaster response, unmanned aerial vehicle,
multi-UAV scheduling, data collection, deep reinforcement learn-
ing, time-varying value.
I. I NTRODUCTION
T IMELY disaster data is essential for efficient responses,
thus vital for saving lives and preventing economic
losses [1], [2], [3]. Nevertheless, information infrastructures
are always vulnerable that would be congested or even
destroyed by disasters, which may impede the disaster data
Manuscript received 17 April 2023; revised 10 October
2023 and 9 December 2023; accepted 16 December 2023. This work
was supported in part by the National Natural Science Foundation of China
under Grant 72174042 and Grant 72101223, in part by the Natural Science
Foundation of Guangdong Province under Grant 2023A1515011402, in part
by the Natural Science Foundation of Shenzhen Municipality under Grant
JCYJ20230807140406013, and in part by the Startup Fund of The Hong
Kong Polytechnic University. The Associate Editor for this article was J. Li.
(Corresponding author: Gangyan Xu.)
Pengfu Wan, Gangyan Xu, and Jiawei Chen are with the Depart-
ment of Aeronautical and Aviation Engineering, The Hong Kong Poly-
technic University, Hong Kong (e-mail: pengfu.wan@connect.polyu.hk;
gangyan.xu@polyu.edu.hk; superlaser-jw.chen@connect.polyu.hk).
Yaoming Zhou is with the Department of Industrial Engineering and
Management, Shanghai Jiao Tong University, Shanghai 200240, China
(e-mail: iezhou@sjtu.edu.cn).
Digital Object Identifier 10.1109/TITS.2023.3345280
1558-0016 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
1
collection processes and bring many blind spots in disas-
ter areas [4]. To address such problems, many alternative
approaches have been proposed, e.g., data crowd-sourcing [5],
social media mining [6], and deploying mobile communication
units [7]. Recently, with the technological advancement of
Unmanned Aerial Vehicle (UAV), it has been proposed to use
UAVs as temporary and mobile relays for disaster data collec-
tion, with many pilot applications in 2015 Nepal earthquake,
2021 Henan Flash Flood, 2022 Luding earthquake, etc.
UAV-based data collection is emerging in various scenarios,
e.g., large-scale wireless sensor networks [8], communication
networks [9], [10], [11], infrastructure inspection [12], and
construction process monitoring [13], among which UAV
demonstrates many advantages of high flexibility and speed,
economic efficiency, and easy-to-deploy. Meanwhile, many
works have been conducted to improve its collection quality
and efficiency [14], [15], [16], in which UAV route planning
and scheduling has attracted much attention, with objectives
on minimizing UAV energy consumption [17], [18], flight
time [19], task completion time [20], data packet losses [21],
etc. These works provide extensive knowledge and valuable
insights about UAV-based data collection. However, they can-
not be directly adopted in disaster data collection, which has
four distinct features that make the problem more complex
and challenging.
Firstly, the value of disaster data is highly dependent on
collection time and varies with time. On the one hand, since
the timeliness of data is vital in disasters, the data value will
decrease dramatically with time. On the other hand, disaster
data value peaks at the time collection starts and decreases dra-
matically afterward. This is because the most critical data (e.g.,
rescue and relief demand data) usually be sent by victims at the
very beginning, then they tend to use the temporal relay (UAV)
for non-critical or even disaster-irrelevant communications
thereafter. Incorporating such time-varying characteristics of
data value greatly increases the complexity of the problem
and makes efficient scheduling of UAVs more challenging.
Secondly, the disaster data volumes or values usually dif-
fer from regions regarding their populations and degrees of
damage. It makes the disaster area contain heterogeneous data
points (regions), which are different from previous works in
industrial scenarios. Such heterogeneity further affects the
decisions on UAVs’ service time at different points and makes
the scheduling problem more complex.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
2 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS
Thirdly, due to the scarcity of UAVs in disasters, it cannot
cover all affected regions but only parts of them, as in the case
of 2021 Henan Flood. It differs our work from previous ones in
industrial data collection where the problems can be modeled
and solved based on the variants of Vehicle Routing Problems
(VRPs). In this work, besides efficient routing decisions, the
sub-sets of regions to be covered should also be decided.
Fourthly, decisions in emergency situations should be made
efficiently. Nevertheless, considering the complexities of the
problem discussed above, and its relatively large scale in terms
of the number of demand regions and UAVs, it is a challenge
to design efficient decision-making algorithms.
Taking the above features and challenges into consideration,
this work develops a Deep Reinforcement Learning (DRL)
based multi-UAV scheduling method to maximize the value
of disaster data collected, and finally support efficient and
effective disaster responses. The contributions of this work
lie in the following four aspects:
• An integrated mathematical model is developed that could
well capture the features of time-varying data value and
partial coverage of demand regions for multiple UAVs
scheduling in disaster data collection.
• Through analyzing the interaction between UAV arrival
time and service time on potential disaster data value
collected, an approximate analytical solution is developed
to accelerate the UAV scheduling process.
• Through embedding the interdependent decision pro-
cesses of UAV routes and service time at each region,
a new attention-based DRL framework is developed that
can support real-time decision-making for multi-UAV
scheduling in disaster data collection.
• Systematic experimental case studies are conducted that
verify the advantages of the proposed method over
existing learning-based and heuristics-based methods in
different scenarios. Results can also be adopted as bench-
marks for future research.
The rest of the paper is structured as follows. Related works
are reviewed in Section II. The mathematical model of the
multi-UAV based disaster data collection problem is presented
and analyzed in Section III. Section IV discusses the DRL-
based solution method. Then experimental case studies and
results analysis are given in Section V. Finally Section VI
concludes the whole paper and points out the future works.
II. R ELATED W ORK
In this section, the relevant works are reviewed from three
streams: UAV-assisted data collection, team orienteering prob-
lems with time-varying value, and reinforcement learning for
combinatorial optimization.
A. UAV-Assisted Data Collection
UAV-assisted data collection has attracted a lot of attention
in recent years, and extensive work has been done across dif-
ferent areas [14], [22]. However, in practical applications, there
are some challenges faced by UAV-assisted data collection,
including the selection of data collection mode [23], sensors
deployment [24], UAV speed control [25], etc. In particular,
Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
as an important issue in this field, UAV scheduling has
attracted much attention with different performance measures,
such as energy consumption, coverage ranges, collection cost,
and efficiency [26]. For example, considering the high energy
consumption of sensors in data transmission, Baek et al. [27]
proposed an energy-efficient UAV routing by maximizing
the minimum residual energy of sensors. Li et al. [28]
considered the cooperation among vehicles and UAVs in
6G-based IoT networks, and designed data collection routes
to improve the coverage ratio and reduce collection costs.
And Wang et al. [29] took multiple objectives of UAV data
collection into account, and proposed two schemes for flight
cycle minimization and energy efficiency maximization.
There are also different techniques developed for solv-
ing the UAV scheduling problems in data collection, e.g.,
graph-theory-based, optimization-based, and learning-based
methods [14]. Specifically, graph-theory-based methods can
convert the geographical space into graphs and generate data
collection routes based on graph analysis [30], which are
widely applied in space division problems [31]. Optimization-
based methods can get the optimal or near-optimal route
by optimization related techniques, such as branch and
bound [32], dynamic programming [33], and successive con-
vex optimization [34]. However, they cannot cope with
complex and large-scale problems, thus inappropriate for
disaster scenarios that require efficient decision-making [14].
Recently, learning-based methods that adopt supervised learn-
ing [35] and reinforcement learning [36] techniques are
emerging, which could deal with dynamic and uncertain
environments, as well as large-scale problems. There are
currently two mainstream reinforcement learning frameworks
for UAV-based data collection. One adopts grid world models
for depicting data collection scenarios and using value-based
methods [37], [38], [39]. The other applies policy-based meth-
ods, taking into account more flexible scenario settings and
UAV action spaces [40], [41].
However, previous works focus on trajectory design, and
UAV route scheduling issues are still not well covered.
Besides, existing works mainly consider the amount of data
collected, while in a disaster scenario, both the value and
amount of data collected should be considered, which is more
complex and challenging.
B. Team Orienteering Problem With Time-Varying Value
Team Orienteering Problem (TOP) [42] refers to the prob-
lem that given a set of nodes with different values and a fixed
amount of time, each team member decides its path to visit
these nodes such that the total value of all paths is maximized.
Different from VRPs [43], members in TOP only need to
visit a subset of the nodes, which makes it more appropriate
for scenarios with insufficient resources, such as healthcare
logistics [44], disaster response [45], etc.
Driving by various practical cases, especially in emergency
response systems, TOP with time-varying value has also been
studied [46], [47]. Generally, the time-varying properties can
be classified into three groups. The first is the arrival-time-
dependent value, where the value varies with the arrival
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

WAN et al.: DRL ENABLED MULTI-UAV SCHEDULING FOR DISASTER DATA COLLECTION 3

time of members at each node. For example, Erkut and

Zhang [48] created a competitive scenario for salespeople in
which sales potentials decreased linearly with time, and Ekici
and Retharekar [49] extended similar models in multiple-agent
problems. The second is the service-time-dependent value,
which means the value attained is a function of service time.
For example, Erdoğan and Laporte [50] provided a model
that vehicles can continue collecting the remaining value by
making several passes, in which the total value is affected
by different service time. The third group is compound-time- Fig. 1. Multi-UAV scheduling for disaster data collection.
dependent value, which is a combination of arrival-time and
service-time dependent values. Such a scenario was considered decided in multi-UAV scheduling for disaster data collection
in [47] within a disaster rescue scenario, where the rescue problems. In addition, as service time is a continuous variable,
success rate is related to ambulance arrival time and rescue it will be difficult to determine directly in RL methods.
time. Yu et al. [51] further extended it to robust models where
service time is uncertain. In these works, it is also assumed that
III. P ROBLEM M ODELLING
accumulated data value decreases linearly with arrival time.
It is worth noting that the compound-time-dependent value In this section, the model of multi-UAV scheduling problem
is the most complicated one, and cases related to emergency for disaster data collection is developed first, then the inter-
responses and disaster data collection always fall into this actions between UAV route design and service time decision
type [52]. are analyzed.
Existing works mainly solved TOP with time-varying
value using exact [47] and various heuristic algorithms [51]. A. Mathematical Model
Although it performs well in small and medium sized problems After disasters strike, many regions within the affected
and uncertain environments, it cannot well cope with large- area may lose their communication networks, which hinders
scale problems efficiently. In addition, compared to existing victims inside from sending their rescue/relief demand data
works in humanitarian logistics, the time-varying models of out. The amount and value of these disaster-related data
disaster data collection are built based on instant data value differed from regions, regarding different sizes, populations,
rather than accumulated data value, and have not been well and damages in these regions, as illustrated in Fig. 1. A fleet of
investigated. UAVs will then be dispatched from a central depot to visit each
region, hovering over as temporary relays, to collect disaster
C. Reinforcement Learning for Combinatorial Optimization data sent from victims.
Given the set of disaster regions as C and let n = |C|, the set
RL methods have been widely adopted in many CO prob- of all nodes with the affected region can be represented as N =
lems, including the maximum cut problem [53], [54], bin {0}∪C ∪{n+1}, where nodes {0} and {n+1} represent the same
packing problem [55], [56], minimum vertex cover prob- depot where UAVs depart from and return to. A mixed-integer
lem [53], [57], and etc. In addition, route scheduling problems, nonlinear programming model for this problem is formulated
such as Traveling Salesman Problem (TSP), VRP, and TOP, (1) to (9) and the notations adopted in the model are shown
that have been widely studied and well solved using exact in Table I.
or heuristics methods [58], [59], [60], [61], have received
increasing attention from RL research community to cope XXX
with dynamic scenarios and large-scale problems [62], [63], max xi jk f (a jk , s j ) (1)
[64]. For instance, Lin et al. [65] considered time windows i∈N j∈C k∈V
constraints and battery charging demand for VRP and built an n
X n+1
X
end-to-end DRL framework. Li et al. [66] developed a DRL s.t. xi,m,k − xm, j,k = 0 ∀m ∈ C, k ∈ V
method based on a vehicle selection decoder to deal with VRP i=0,i̸ =m j=1, j̸ =m
with heterogeneous capacities. (2)
RL methods for CO consist mainly of value-based and X n
X
policy-based approaches. Value-based approaches learn a value xi,m,k ≤ 1 ∀m ∈ C (3)
function to evaluate the value of possible states and actions, k∈V i=0,i̸ =m
then construct solutions based on evaluations, such as Deep n+1
X
Q Learning (DQN) [53], [67]. In contrast, policy-based x0,m,k ≤ 1 ∀k ∈ V (4)
approaches can directly determine the next action given the m=1
current state, including REINFORCE [63], [68] and Proximal ai,k + ti, j + si − a j,k ≤ M(1 − xi, j,k ) ∀i, j ∈ N , k ∈ V
Policy Optimization (PPO) [54], [69].
(5)
Although RL methods for CO problems are efficient and
relatively mature, most applications only consider one vari- a0,k = 0 ∀k ∈ V (6)
able, while both traveling routes and service time have to be an+1,k ≤ Tmax ∀k ∈ V (7)

Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
4 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS
N OTATION TABLE
si ≥ 0 ∀i ∈ N
xi, j,k ∈ {0, 1} ∀i, j ∈ N , k ∈ V
The objective function (1) maximizes the data value col-
lected by all UAVs, which depends on the routes of UAVs
and data collection time (service time) at each region. Con-
straint (2) means every UAV should leave the disaster region
m ∈ C it visited, which guarantees flow conservation of the
problem. Constraint (3) ensures every disaster region is visited
at most once. Inequality (4) limits the maximum number of
UAVs depart from the depot. The visiting sequence within each
route is specified in constraint (5), where M is a large positive
constant to linearize the inequality. If xi, j,k = 1, it means
that UAV k would visit the region j after visiting the region
i, and the arrival time at region j must be greater than or
equal to the sum of the arrival time at region i, traveling
time from region i to region j, and the service time at region
i. if xi, j,k = 0, inequality (4) is always satisfied since M
is large enough. Constraint (6) initializes the departure time,
while constraint (7) is the endurance limitation for UAVs.
Constraints (8) and (9) are the ranges of decision variables
si and xi jk .
B. Time-Varying Data Value
According to the analysis in Section I and Section II, the
disaster data value vik collected by UAV k at node i is a
function of aik and si , denoted as vik = f (aik , si ).
1) Arrival-Time-Dependent Value: Due to the timeliness
requirement in disasters, the earlier the disaster data are
collected, the more value they may contain. According to the
investigations on the survival rate with rescue time [70], the
instant data value of node i (with origin instant data value of
βi ) is considered to decrease linearly with aik , denoted as bik :
bik = βi − α · aik (α > 0)
where the decreasing rate of data value is α.
2) Service-Time-Dependent Value: Different from service-
time-depended value models adopted in emergency rescue
scenarios, the data value decreasing process over si contains
three stages: (i) During the initial stage right after UAV arrives,
a large amount of informative and critical disaster data will
Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
TABLE I
(8)
(9)
Fig. 2. Data value with UAV arrival time and service time.
be quickly collected. Since this stage is usually very short,
the data value is assumed to be constant with bik . (ii) After a
certain time, the data value begins to decrease at an increasing
rate as new disaster data is generated at a decreasing rate while
non-critical data begin to show up [71]. (iii) When most critical
disaster data are collected, the instant data value will stay at
a very low level. With the above analysis, the logistic curve
f ik is appropriate for approximating such a process. In this
work, the midpoint of the logistic curve is proportional to βi
with the ratio g. The reason is that if one node contains more
data value, the UAV should take more time to collect its data.
Integrating with (10), the instant data value f ik after the data
collection starts can be depicted as (11).
bik
f ik = ( g > 0, x ∈ [0, si ] ) (11)
1 + e x−gβi
Based on Equation (10) and (11), the collected data value
vik can be obtained by integrating the logistic function over
service time si as (12) and illustrated in Figure 2.
βi − α · aik
Z si Z si
vik = f ik dx = dx (12)
0 0 1 + e x−gβi
(10) C. Model Analysis
The TOP with time-varying data value is more complex
than the basic TOP (which is already an NP-Hard problem),
that not only traveling routes with arrival time at each node
should be determined, but also the service time of UAV
at each node. It can hardly be solved by heuristic based
methods within a short time, let alone exact optimal solutions.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
WAN et al.: DRL ENABLED MULTI-UAV SCHEDULING FOR DISASTER DATA COLLECTION
Fig. 3. Decision-making process for service time.
Fig. 4. Data value comparison between two adjacent nodes.
Besides, according to our preliminary works, it makes DRL-
based methods difficult to converge, especially in large-scale
problems. Therefore, this part will further analyze the model
to make it easy to solve.
Consider one UAV that just arrives at a node in its route, let
the UAV choose the next node to be visited first, then decide
the service time of the current node. After finishing the data
collection process at the current node, this UAV will fly to the
next chosen node and update the arrival time of that node. This
process will be repeated until the UAV returns to the depot,
as illustrated in Figure 3.
Specifically, the service time of the UAV at the current node
is decided upon the selection of the next node. According to
the feature of time-varying data value, the time for the UAV to
leave the current node depends on the expectation of collected
value in the future. To make it simple for analysis, we first
ignore the impact of other nodes in the route, and then the
decision on service time at the current node can be made by
comparing the potential value gained at the current node with
the next node.
Figure 4 shows the detailed comparison process. Take UAV
k as an example, given the current node and next node as
node 1 and node 2, the service time s1 has to be decided.
Based on constraint (5), a1k + s1 + t12 is the arrival time of
node 2. Based on formula (10), the decayed data value in terms
of the arrival time of these two nodes can be represented as
follows:
b1k = β1 − α · a1k
b2k = β2 − α · (a1k + s1 + t12 )
The instant data value at the time UAV leaves node 1 is
derived based on formula (11) and represented as:
b1k
f 1k (x = s1 ) = (15)
1 + es1 −gβ1
Assuming s2 remains unchanged and using formulas (12)
and (14), the extra instant data value obtained by the UAV k
at node 2 is formulated as (16):
∂ f 2k ∂b2k s2
Z Z s2
1 1
− =− dx = α dx
∂s1 ∂s1 0 1 + e x−gβ2
0 1+e
x−gβ2
(16)
Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
5
Furthermore, since the service time at node 2 is unchanged,
the data value collected from node 2 only depends on the
arrival time at node 2. Based on the formula (12), earlier arrival
time would improve collection efficiency, thus the formula (16)
expresses the additional data value collected per unit of arrival
time advancement.
Then the optimal service time s1∗ can be obtained by solving
equation (17), where s2 is the largest service time at node
2 limited by Tmax :
Z s2
b1k 1
s −gβ
=α x−gβ2
dx (17)
1+e 1 1
0 1+e
If service time s1 is smaller than s1∗ , it means that UAV
can get more data value at the current node, so it should not
leave. If s1 is larger than s1∗ , staying at the current node will
not be a rational idea since more data value can be collected
at the next node. Thus s1∗ is the optimal solution when only
considering balancing the collected data value between two
adjacent nodes.
The above analysis can be extended to a multi-node situ-
ation that covers the entire route. Assume there are m nodes
1, 2, . . . , i, . . . , m − 1, m in one UAV route, and the current
node is i. Different from the previous scenario that only
considers the next node i + 1, the UAV needs to consider all
subsequent nodes i +2, . . . , m. According to equation (17), the
reason that UAV chooses to leave the current node is to obtain
more data value in the next node. In a multi-node situation,
the UAV will always collect no fewer data values than in a
double-node situation. Considering that more data value may
be obtained at future nodes, the optimal service time of the
current node i in the multi-node sequence should be less than
or equal to si∗ . Hence, it can be derived that the former result
si∗ is a mathematical upper bound for the optimal solution
of service time given the route, and the UAV should leave
the current node on or before time aik + si∗ . Since the time
allocation of multiple UAVs is independent of each other and
only depends on their own routes, the above analysis can be
easily adopted in multi-UAV scenarios.
IV. M ETHODOLOGY
With the above analysis, this section develops a DRL-
based solution method. In real-world scenarios, models that
match the disaster scenarios are pre-trained, enabling efficient
decisions to be made quickly when a disaster occurs. The
overall model and framework of our DRL-based method will
(13)
be introduced first, then the attention model and decoding
(14) process will be discussed.
A. Overview of the DRL-Based Method
To solve the problem of multi-UAV scheduling problem
using the DRL-based method, the Markov Decision Process
(MDP) model for the problem is built first. Here, the problem
is modeled as a finite MDP M =< S, A, P, R >. S is a
finite set of states, including the information of the set of
nodes N and UAVs V . The node information contains the
locations and the original data value β of nodes, and UAV
information includes the current positions and the used time
of UAVs. A is a finite set of actions consisting of the node
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
6 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS
set N . Unavailable nodes that cannot fulfill constraints (3)
and (7) are masked when choosing actions. P is a transition
function that denotes the possibility of the next state given the
current state and action. R is the reward function vik , which is
determined by arrival time aik and service time si according
to the formula (12). The optimization objective of the MDP
model is consistent with formula (1).
To reduce variance and improve training efficiency, the boot-
strap method is widely applied in developing DRL algorithms,
such as Actor-Critic. However, the problem of delayed rewards
arises since the service time of the current node cannot be
determined until the next node is selected, which poses a chal-
lenge for the adoption of the bootstrap method. Therefore, the
REINFORCE algorithm is chosen, which generates complete
sequences during sampling and returns the total rewards in the
end. Denote the policy of one UAV as π with parameter θ,
and in a problem instance s, all actions chosen by the UAV
under π(θ ) are a = (a1 , a2 , . . . , an ). Then, the whole policy
π is decomposed into:
Yn
π(a|s, θ) = π(at |s, a1:t−1 , θ) (18)
t=1
The purpose of the algorithm is to optimize parameter θ to
maximize final rewards. Denote G(a) as the total data value
obtained by the solution a. Based on Monte Carlo policy
gradient, J (θ ) can be obtained as the performance measure
of policy π(θ ):
∇ J (θ ) = Eπ [G(a)∇logπ(a|s, θ)]
In addition, one efficient way to reduce gradient variance
and increase convergence speed is to use a baseline b(s). Since
the baseline does not vary with action a, adding a baseline will
not affect the validity of the equation (19). Then the policy
gradient with baseline can be formulated as:
∇ J (θ ) = Eπ [(G(a) − b(s))∇logπ(a|s, θ)]
Specifically, the exponential moving average baseline with
a low computational cost is developed. In each iteration, the
baseline b(s) will be updated by a decay factor γ . In the
training process, it is impossible to get all possible actions
and expected rewards, thus batch gradient descent and Adam
are adopted. The interaction between the agent (UAV) and the
environment is proposed and illustrated in Figure 5. In the
following, the attention model and decoding method will be
further discussed to introduce our end-to-end framework.
B. The Attention Model
Although the global environment information is embedded
as inputs when making decisions, agents should automatically
extract the most crucial information and ignore noises from
useless ones, thus the attention model is adopted. It consists
of three main parts: graph embedding, context embedding, and
decoding mechanism.
1) Graph Embedding: Graph information here includes
the location information of nodes X and their maximum
instant data value β when disaster happens. As all the graph
information is static, the graph embedding only needs to be
calculated once in one instance. For each node in the graph,
Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
Fig. 5. Agent-environment interaction mode.
node information is mapped into a high dimensional vector
space h with parameters W and l. The embedding of node i
is represented as:
h i = Wi [X i ; βi ] + li (21)
Single-node embedding cannot reflect the relationships
among nodes, and an attention mechanism is applied to encode
node embedding further. Denote the weights of key, query, and
value as W k , W q , and W v , respectively. Then for node i, its
(19)
key ki , query qi , and value vi could be computed as:
ki = W k h i , qi = W q h i , vi = W v h i (22)
For any other node j in the graph, the attention weight
of node j on node i is calculated by taking a dot product
between qi and k j . Denote dk as the dimension of the key,
(20) then the attention-based node embedding h i′ is formulated as
a weighted value based on softmax attention scores:
X qiT k j
h i′ = softmax( √ )v j (23)
j
dk
In addition, to depict different feature spaces of graph
information, the input embedding is divided into m parts and
multi-head attention (MHA) is adopted. To each part, the sub-
input embedding forms a head based on the formula (23).
Denote W o as the parameter matrix for weights of subspaces,
the multi-head embedding of node i is formulated as:
MHAi = W o Concat(h i1
′
, h i2
′
, . . . , h im
′
) (24)
Each attention layer consists of two sublayers: a MHA
layer and a feed-forward layer, and each sublayer includes
a skip connection. The final node embedding is the output
of encoding layers, which are composed of multiple stacked
attention layers. The graph embedding is the mean of the final
node embeddings.
2) Context Embedding: Context embedding consists of
three parts: graph embedding, current state embedding, and
depot embedding. In the current state embedding, the remain-
ing flight time of the UAV is added to the current state.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

WAN et al.: DRL ENABLED MULTI-UAV SCHEDULING FOR DISASTER DATA COLLECTION 7

As UAVs should return to the depot finally equation (2), the Algorithm 1 Reinforce With Exponential Moving
depot embedding is included in context embedding. Similar to Average Baseline
graph embedding, both the current state embedding and depot Input: number of epochs N , steps per epoch T , batch
embedding are encoded by MHA. With context embedding size B, decay factor γ
h c and query weight W q , query for context embedding qc is Initialize network weights of encoder and decoder;
formulated as: for epoch = 1, 2, . . . , N do
qc = W q h c (25) for step = 1, 2, . . . , T do
generate B instances X 1 , X 2 , . . . , X B ;
3) Decoding Mechanism: The possibilities of the next for i = 1, 2, . . . , B do
action are obtained by the decoding mechanism. Since the while termination condition is not satisfied
context embedding includes information about the current do
state, qc is used to query ki of other nodes and calculate select actions through current policy;
attention weight between them. Note that not all nodes are if previous node is not the depot then
available since visited nodes cannot be revisited (equation (3)), calculate service time of previous
and some nodes that make UAV exceed its maximum flight node;
time are prohibited (equation (7)). The set of unavailable nodes end
forms a mask, and attention weights of unavailable nodes are update the environment and observe
masked. Then the attention weight u i of node i is: current states;
 T end
 q√c ki if node i is available record actions a1 , a2 , . . . , a B and states
ui = dk (26) s1 , s2 , . . . , s B ;
compute total rewards G 1 , G 2 , . . . , G B ;

−∞ otherwise
end
To reduce the varying range of u i , the tanh function is used to
if no baseline is generated before then
clip the result. Possibilities of all actions are obtained through
b ← avg(G 1 , G 2 , . . . , G B );
the softmax function of attention weights.
else
b ← γ avg(G 1 , G 2 , . . . , G B ) + (1 − γ )b;
C. Decoding Method end
PB
Different from single-agent problems, multiple UAVs with ∇θ ← i=1 (G i − b)∇θ logπ(ai |si , θ );
different states are considered in this work. One feasible end
method is to compare the actions of different UAVs and choose end
the most appropriate decision. Unlike generating joint actions,
UAVs in the competitive mechanism have to compete with
each other, and only one action of one UAV is chosen each
the total value is obtained according to the formula (1), and the
time. The main advantage of this method is avoiding conflicts
policy is updated according to the REINFORCE framework.
when UAVs tend to select the same node. As the current
j The pseudo-code of REINFORCE with an exponential moving
states of UAVs are different, context embedding h c have to
j average baseline is presented in the Algorithm 1.
be calculated for each UAV j, then query qc and the attention
j
weight u i between UAV j and node i are calculated as:
j j
V. E XPERIMENTAL C ASE S TUDY
qc = W q h c (27)

jT A. Experiment Setting
 qc k i

j √ if node i is available for UAV j In this work, a square disaster area with size [0, 1] × [0, 1]
ui = dk (28)
 is built for experimental case study. A set of disaster regions
 −∞ otherwise
with data to be collected are randomly generated within the
To all attention weights calculated above, the possibility for area. Their instant data values β are also randomly generated
action ai j (choosing UAV j and node i) is in the range of [20, 30]. In the experiments, the data value
j
decreasing rate α is set to 0.005, implying that data value
eu i will decrease to 0 after 200 minutes, which is consistent
p(ai j ) = P P (29)
j with the understanding of golden window (around 3 hours)
j i eu i
in emergencies. Meanwhile, g is set as 0.5 in formula (11).
Greedy decoding is developed and the action with the largest Besides, a fixed airspeed of 0.05 unit distance per minute
possibility is chosen. After getting the new action, the service is set for UAVs. To verify the performance of the proposed
time of the previous node is calculated if the previous node is method in different scales of problems, the number of disaster
not the depot. Then the environment information, including regions Nregion is set from 5 to 500 while the number of UAVs
arrival time and UAV positions, will be updated based on NUAV ranges from 1 to 20. In the following, each scenario is
constraint (5), and new states of UAVs will be observed. numbered, as ’U 10− N 50’ denotes the scenario with 10 UAVs
Repeating such interactions until all UAVs return to the depot, and 50 disaster regions.

Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

8 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

TABLE II TABLE III

H YPERPARAMETER TABLE C OMPARISON IN S MALL -S CALE P ROBLEMS

The hyperparameters adopted in the training process are

presented in Table II. It is worth mentioning that the proposed
method requires relatively a small number of parameters and
has low computational complexity. Specifically, for all the sce-
narios, the number of parameters remains constant at 99,328.
And for scenarios such as ’U 20− N 150’, the FLOPs (Floating
Point Operations) of the proposed method is 39.882G, which
is relatively small. It should be noted that, when dealing with
large-scale problems, the batch size can be substituted with
either 256 or 128. All experiments are conducted on a Dell
Precision Tower 5820 server with a TITAN RTX GPU. The
implementation of all algorithms is performed using Python.

B. Benchmark Methods Fig. 6. Training returns of episodes.

To verify the performance of the proposed DRL method,

five algorithms are taken for comparison, as listed below. All
algorithms share the same service time calculation process as
our proposed DRL method, only different in routing decisions.
• SCIP: Solving Constraint Integer Programs (SCIP) is
widely adopted for mixed-integer nonlinear programming
model [47], [72], which can get exact optimal results.
It will be adopted to verify whether the proposed model
could generate near-optimal solutions.
• Greedy Algorithm: Greedy Algorithm follows a greedy
strategy that chooses UAVs and nodes based on the
“optimal matching” principle [73]. It is widely adopted
in literature as a benchmark method.
• Tabu Search: Tabu Search uses the tabu table to avoid the
local optimization problem, thus works well in TOP [51],
VRP [74], and data collection problems [75].
• Genetic Algorithm: Genetic Algorithm is a metaheuristic
method that proved to be effective in TOP [76] and multi-
UAV data collection framework [77].
• DRL: A state-of-the-art DRL method [64], which com-
bines transformer and pointer network model is adopted
for comparison, which is proved effective in orienteering
problems.
C. Computational Results
In the experiments, each scenario is running 20 times with
randomly generated seeds, and their average performance will
be taken for comparison.
1) Ablation Experiments for Warm Start and Attention
Mechanism: Considering the difficulty of directly training
medium and large-scale instances, the trained network of
small-scale instances is used as the initial network, which
is represented as warm start in Figure 6. Here, compared to
directly training ’U 5 − N 20’ instances, using ’U 2 − N 10’ as
the initial network could provide a faster convergence curve.
In addition, to demonstrate the necessity of the attention mech-
anism in our algorithm, the result of the ablation experiment is
also shown in Figure 6. Compared to our proposed algorithm,
the algorithm without a multi-head attention mechanism can
hardly learn anything, resulting in a curve with little upward
trend.
2) Comparison in Small-Scale Problems: For small-scale
problems, SCIP is adopted for comparison to verify whether
the proposed method could generate near-optimal solutions.
Besides, considering the timeliness requirement in disasters,
the maximum computing time for SCIP is set as 7200s.
Table III shows the results. For ’U 1 − N 5’ and ’U 2 − N 5’,
SCIP performs better and the gap between two methods is
pretty small. For ’U 1 − N 8’ and ’U 2 − N 8’, our DRL
method performs better than SCIP. Results show that proposed
method can get approximately optimal solutions. In addition,
proposed DRL method can immediately generate approximate
optimal solutions for the problem, unlike SCIP which requires
a relatively long time for the results.
3) Comparison in Medium-Scale and Large-Scale Prob-
lems: The experiment results for medium-scale and large-scale
problems are shown in Table IV. In general, the proposed
method performs better than Greedy Algorithm and DRL.
For some medium-scale instances (e.g., the first 4 rows
in Table IV), the results of the Tabu Search and Genetic
Algorithm are close to or even better than those of the
proposed method. However, the performance of these heuristic
algorithms worsens as the scale of scenarios increases. As the
number of disaster regions enlarged to no less than 200 while

Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

WAN et al.: DRL ENABLED MULTI-UAV SCHEDULING FOR DISASTER DATA COLLECTION 9

TABLE IV
C OMPARISON IN M EDIUM -S CALE AND L ARGE -S CALE P ROBLEMS

TABLE V
I MPACT OF D IFFERENT ROUTES S EQUENCE L ENGTH ON E XPERIMENTAL P ERFORMANCE

TABLE VI
I MPACT OF D IFFERENT P ROBLEM S CALES ON E XPERIMENTAL P ERFORMANCE

fixing the number of UAVs at 20 (see the lower part of
Table IV), we can find that the superiority of the proposed
method becomes more dominant. In addition, Figure 7 is
employed to visualize the performance of algorithms in large-
scale scenarios.
4) Advantages of Proposed Method in Long-Sequence and
Large-Scale Problems: Table V examines the impact of route
sequence length, which is determined by the ratio of Nregion
to NUAV , on algorithmic efficiency. Ratio N /U means the
average number of disaster regions allocated to each UAV in
different scenarios. Conventionally, the higher the ratio, the
more disaster regions visited by each UAV. Table V shows
the performance of different algorithms by using the results
of Greedy Algorithm as the benchmark. According to Table V
and Figure 7, it can be observed that the higher the ratio, the
better the performance of both learning-based methods, which
shows that they have advantages over heuristic methods when
dealing with long route sequence problems. In comparison,

Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
10
Fig. 7. Comparative trend curves across different problem scales
under 20 UAVs.
there is no evident growing trend in the performance of these
heuristic methods.
In Table VI, the product of the number of disaster
regions and number of UAVs, N·U, is regarded as an indi-
cator to measure the scale of each problem, which varies
from 500 to 10000. The percentage results of the four methods
listed in Table VI are still benchmarked against the Greedy
Algorithm. As the scale of the instances increases, the per-
formance gap between our proposed method and compared
algorithms becomes increasingly larger (as shown in the last
column of Table VI). This phenomenon illustrates that in
some small-scale and medium-scale problems, other meth-
ods, especially heuristic methods, can quickly traverse the
neighborhood and continuously update their optimal solutions.
However, in large-scale problems, the neighborhood of the cur-
rent solution grows exponentially, which dramatically reduces
their iterative efficiency. By contrast, the efficiency of the
proposed DRL method is minimally affected by the scale of
the problem, and its performance on large-scale problems is
also very good.
D. Discussions
According to the above experimental results, several advan-
tages and managerial implications can be concluded.
Firstly, the proposed method is effective in realizing efficient
and high-quality scheduling of UAVs for disaster data collec-
tion, no matter small-scale problems or large-scale problems.
In practice, the proposed method can be adopted for real-time
decision-making in disasters.
Secondly, the proposed method performs better in long-
sequence problems where the UAV will visit many regions in
one trip. In future, with the improved capacity of UAVs and
their prolonged endurance, the long-sequence feature will be
more prominent as UAVs are capable of visiting more nodes in
one trip. Therefore, the proposed method will be more popular
in many scenarios with the development of UAV technologies.
Thirdly, in real-life applications, given the number of
regions in an area (e.g., the area of responsibility) and the
number of UAVs, the multi-UAV scheduling model can be pre-
trained using a simulation-based environment and then directly
applied in practice. Our experimental results have shown that
Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS
the performance of the trained model is superior to heuristics
methods and the state-of-the-art DRL method when facing
different cases with the same scenario settings.
VI. C ONCLUSION
To address the problem of UAV-assisted disaster data col-
lection, this paper proposed an efficient multi-UAV scheduling
method considering the feature of time-varying data value.
First of all, a TOP-based mathematical model is developed to
maximize the data value collected by all UAVs. Besides, the
features of the time-varying data value in terms of both UAV
arrival time and service time are analyzed and modeled. Mean-
while, to accelerate the solution algorithms, the relationships
between UAV route selection and service time are analyzed.
An attention-based DRL method is then proposed, which can
obtain high-quality solutions in near real-time. Several typical
scenarios are simulated and different algorithms are tested,
which verify the advantages of the proposed method in both
computing efficiency and solution quality.
This work can be extended from the following directions.
Firstly, the proposed mathematical model with time-varying
value can be further developed to incorporate more com-
plex scenarios, such as multiple disaster types, dynamic and
uncertain environments, and non-uniform spatial distribution
of disasters. Secondly, the dynamics of the scenarios can be
considered with regard to the development of disasters and
the changes of UAV numbers. Thirdly, the method proposed
in this paper can be further improved with higher robustness
to different scenarios. Fourthly, the multi-agent reinforcement
learning frameworks can be studied to deal with decentralized
scenarios.
R EFERENCES
[1] I. Nourbakhsh, R. Sargent, A. Wright, K. Cramer, B. McClendon,
and M. Jones, “Mapping disaster zones,” Nature, vol. 439, no. 7078,
pp. 787–788, Feb. 2006.
[2] M. Zook, M. Graham, T. Shelton, and S. Gorman, “Volunteered geo-
graphic information and crowdsourcing disaster relief: A case study
of the Haitian earthquake,” World Med. Health Policy, vol. 2, no. 2,
pp. 7–33, Jul. 2010.
[3] M. Morton and J. L. Levy, “Challenges in disaster data collection during
recent disasters,” Prehospital Disaster Med., vol. 26, no. 3, pp. 196–201,
Jun. 2011.
[4] I. Junglas and B. Ives, “Recovering it in a disaster: Lessons from
Hurricane Katrina,” MIS Quart. Executive, vol. 6, no. 1, pp. 39–51,
2007.
[5] H. To, S. H. Kim, and C. Shahabi, “Effectively crowdsourcing the
acquisition and analysis of visual data for disaster response,” in Proc.
IEEE Int. Conf. Big Data, Oct. 2015, pp. 697–706.
[6] P. R. Spence, K. A. Lachlan, and A. M. Rainear, “Social media and
crisis research: Data collection and directions,” Comput. Hum. Behav.,
vol. 54, pp. 667–672, Jan. 2016.
[7] T. Sakano et al., “Disaster-resilient networking: A new vision based on
movable and deployable resource units,” IEEE Netw., vol. 27, no. 4,
pp. 40–46, Jul. 2013.
[8] S. Wang, Y. Long, Y. Zhou, and G. Xu, “Multi-UAV route planning
for data collection from heterogeneous IoT devices,” in Proc. IEEE Int.
Conf. Ind. Eng. Eng. Manag. (IEEM), Dec. 2022, pp. 1556–1560.
[9] X. Pang, M. Sheng, N. Zhao, J. Tang, D. Niyato, and K.-K. Wong,
“When UAV meets IRS: Expanding air-ground networks via passive
reflection,” IEEE Wireless Commun., vol. 28, no. 5, pp. 164–170,
Oct. 2021.
[10] T. Ma et al., “UAV-LEO integrated backbone: A ubiquitous data collec-
tion approach for B5G Internet of Remote Things networks,” IEEE J.
Sel. Areas Commun., vol. 39, no. 11, pp. 3491–3505, Nov. 2021.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
WAN et al.: DRL ENABLED MULTI-UAV SCHEDULING FOR DISASTER DATA COLLECTION
[11] X. Pang, W. Mei, N. Zhao, and R. Zhang, “Intelligent reflecting
surface assisted interference mitigation for cellular-connected UAV,”
IEEE Wireless Commun. Lett., vol. 11, no. 8, pp. 1708–1712, Aug. 2022.
[12] Y. Tan, S. Li, H. Liu, P. Chen, and Z. Zhou, “Automatic inspection
data collection of building surface based on BIM and UAV,” Autom.
Construct., vol. 131, Nov. 2021, Art. no. 103881.
[13] K. Asadi et al., “An integrated UGV-UAV system for construc-
tion site data collection,” Autom. Construct., vol. 112, Apr. 2020,
Art. no. 103068.
[14] Z. Wei et al., “UAV-assisted data collection for Internet of Things:
A survey,” IEEE Internet Things J., vol. 9, no. 17, pp. 15460–15483,
Sep. 2022.
[15] I. Jawhar, N. Mohamed, J. Al-Jaroodi, D. P. Agrawal, and S. Zhang,
“Communication and networking of UAV-based systems: Classifica-
tion and associated architectures,” J. Netw. Comput. Appl., vol. 84,
pp. 93–108, Apr. 2017.
[16] D. Liu et al., “Opportunistic UAV utilization in wireless networks: Moti-
vations, applications, and challenges,” IEEE Commun. Mag., vol. 58,
no. 5, pp. 62–68, May 2020.
[17] W. Ejaz, A. Ahmed, A. Mushtaq, and M. Ibnkahla, “Energy-efficient task
scheduling and physiological assessment in disaster management using
UAV-assisted networks,” Comput. Commun., vol. 155, pp. 150–157,
Apr. 2020.
[18] X. Pang, J. Tang, N. Zhao, X. Zhang, and Y. Qian, “Energy-efficient
design for mmWave-enabled NOMA-UAV networks,” Sci. China Inf.
Sci., vol. 64, no. 4, Apr. 2021, Art. no. 140303.
[19] J. Gong, T.-H. Chang, C. Shen, and X. Chen, “Flight time minimization
of UAV for data collection over wireless sensor networks,” IEEE J. Sel.
Areas Commun., vol. 36, no. 9, pp. 1942–1954, Sep. 2018.
[20] Z. Wang, G. Zhang, Q. Wang, K. Wang, and K. Yang, “Completion time
minimization in wireless-powered UAV-assisted data collection system,”
IEEE Commun. Lett., vol. 25, no. 6, pp. 1954–1958, Jun. 2021.
[21] Y. Emami, B. Wei, K. Li, W. Ni, and E. Tovar, “Deep Q-networks for
aerial data collection in multi-UAV-assisted wireless sensor networks,”
in Proc. Int. Wireless Commun. Mobile Comput. (IWCMC), Jun. 2021,
pp. 669–674.
[22] M. T. Nguyen et al., “UAV-assisted data collection in wireless sensor net-
works: A comprehensive survey,” Electronics, vol. 10, no. 21, p. 2603,
Oct. 2021.
[23] S. R. Yeduri, N. S. Chilamkurthy, O. J. Pandey, and L. R. Cenkeramaddi,
“Energy and throughput management in delay-constrained small-
world UAV-IoT network,” IEEE Internet Things J., vol. 10, no. 9,
pp. 7922–7935, May 2023.
[24] Q. Wu, P. Sun, and A. Boukerche, “Unmanned aerial vehicle-assisted
energy-efficient data collection scheme for sustainable wireless sensor
networks,” Comput. Netw., vol. 165, Dec. 2019, Art. no. 106927.
[25] X. Li, J. Tan, A. Liu, P. Vijayakumar, N. Kumar, and M. Alazab,
“A novel UAV-enabled data collection scheme for intelligent transporta-
tion system through UAV speed control,” IEEE Trans. Intell. Transp.
Syst., vol. 22, no. 4, pp. 2100–2110, Apr. 2021.
[26] B. Alzahrani, O. S. Oubbati, A. Barnawi, M. Atiquzzaman, and
D. Alghazzawi, “UAV assistance paradigm: State-of-the-art in appli-
cations and challenges,” J. Netw. Comput. Appl., vol. 166, Sep. 2020,
Art. no. 102706.
[27] J. Baek, S. I. Han, and Y. Han, “Energy-efficient UAV routing for
wireless sensor networks,” IEEE Trans. Veh. Technol., vol. 69, no. 2,
pp. 1741–1750, Feb. 2020.
[28] T. Li, W. Liu, Z. Zeng, and N. N. Xiong, “DRLR: A deep-reinforcement-
learning-based recruitment scheme for massive data collections in
6G-based IoT networks,” IEEE Internet Things J., vol. 9, no. 16,
pp. 14595–14609, Aug. 2022.
[29] T. Wang, X. Pang, J. Tang, N. Zhao, X. Zhang, and X. Wang, “Time and
energy efficient data collection via UAV,” Sci. China Inf. Sci., vol. 65,
no. 8, Aug. 2022, Art. no. 182302.
[30] R. Penicka, J. Faigl, and M. Saska, “Physical orienteering problem for
unmanned aerial vehicle data collection planning in environments with
obstacles,” IEEE Robot. Autom. Lett., vol. 4, no. 3, pp. 3005–3012,
Jul. 2019.
[31] S. Aggarwal and N. Kumar, “Path planning techniques for unmanned
aerial vehicles: A review, solutions, and challenges,” Comput. Commun.,
vol. 149, pp. 270–299, Jan. 2020.
[32] M. Samir, S. Sharafeddine, C. M. Assi, T. M. Nguyen, and A. Ghrayeb,
“UAV trajectory planning for data collection from time-constrained IoT
devices,” IEEE Trans. Wireless Commun., vol. 19, no. 1, pp. 34–46,
Jan. 2020.
Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
11
[33] H. Hu, K. Xiong, G. Qu, Q. Ni, P. Fan, and K. B. Letaief, “AoI-minimal
trajectory planning and data collection in UAV-assisted wireless powered
IoT networks,” IEEE Internet Things J., vol. 8, no. 2, pp. 1211–1223,
Jan. 2021.
[34] W. Chen, S. Zhao, Q. Shi, and R. Zhang, “Resonant beam charging-
powered UAV-assisted sensing data collection,” IEEE Trans. Veh.
Technol., vol. 69, no. 1, pp. 1086–1090, Jan. 2020.
[35] J. Chen et al., “Efficient data collection in large-scale UAV-aided
wireless sensor networks,” in Proc. 11th Int. Conf. Wireless Commun.
Signal Process. (WCSP), Oct. 2019, pp. 1–5.
[36] L. Liu, K. Xiong, J. Cao, Y. Lu, P. Fan, and K. B. Letaief, “Average AoI
minimization in UAV-assisted data collection with RF wireless power
transfer: A deep reinforcement learning scheme,” IEEE Internet Things
J., vol. 9, no. 7, pp. 5216–5228, Apr. 2022.
[37] S. Fu et al., “Energy-efficient UAV-enabled data collection via wireless
charging: A reinforcement learning approach,” IEEE Internet Things J.,
vol. 8, no. 12, pp. 10209–10219, Jun. 2021.
[38] P. Tong, J. Liu, X. Wang, B. Bai, and H. Dai, “Deep reinforcement
learning for efficient data collection in UAV-aided Internet of Things,” in
Proc. IEEE Int. Conf. Commun. Workshops (ICC Workshops), Jun. 2020,
pp. 1–6.
[39] K. K. Nguyen, T. Q. Duong, T. Do-Duy, H. Claussen, and L. Hanzo,
“3D UAV trajectory and data collection optimisation via deep reinforce-
ment learning,” IEEE Trans. Commun., vol. 70, no. 4, pp. 2358–2371,
Apr. 2022.
[40] M. Sun, X. Xu, X. Qin, and P. Zhang, “AoI-energy-aware UAV-
assisted data collection for IoT networks: A deep reinforcement learning
method,” IEEE Internet Things J., vol. 8, no. 24, pp. 17275–17289,
Dec. 2021.
[41] Y. Wang et al., “Trajectory design for UAV-based Internet of Things
data collection: A deep reinforcement learning approach,” IEEE Internet
Things J., vol. 9, no. 5, pp. 3899–3912, Mar. 2022.
[42] I.-M. Chao, B. L. Golden, and E. A. Wasil, “The team orienteering
problem,” Eur. J. Oper. Res., vol. 88, no. 3, pp. 464–474, Feb. 1996.
[43] H. Qin, X. Su, T. Ren, and Z. Luo, “A review on the electric vehicle
routing problems: Variants and algorithms,” Frontiers Eng. Manag.,
vol. 8, no. 3, pp. 370–389, Sep. 2021.
[44] R. Aringhieri, S. Bigharaz, D. Duma, and A. Guastalla, “Novel appli-
cations of the team orienteering problem in health care logistics,” in
Optimization in Artificial Intelligence and Data Sciences. Rome, Italy:
Springer, 2022, pp. 235–245.
[45] S. Saeedvand, H. S. Aghdasi, and J. Baltes, “Novel hybrid algorithm for
team orienteering problem with time windows for rescue applications,”
Appl. Soft Comput., vol. 96, Nov. 2020, Art. no. 106700.
[46] V. F. Yu, P. Jewpanya, S.-W. Lin, and A. N. P. Redi, “Team orienteering
problem with time windows and time-dependent scores,” Comput. Ind.
Eng., vol. 127, pp. 213–224, Jan. 2019.
[47] Q. Yu, Y. Adulyasak, L.-M. Rousseau, N. Zhu, and S. Ma, “Team
orienteering with time-varying profit,” Informs J. Comput., vol. 34, no. 1,
pp. 262–280, Jan. 2022.
[48] E. Erkut and J. Zhang, “The maximum collection problem with time-
dependent rewards,” Nav. Res. Logistics, vol. 43, no. 5, pp. 749–763,
Aug. 1996.
[49] A. Ekici and A. Retharekar, “Multiple agents maximum collection
problem with time dependent rewards,” Comput. Ind. Eng., vol. 64, no. 4,
pp. 1009–1018, Apr. 2013.
[50] G. Erdosan and G. Laporte, “The orienteering problem with variable
profits,” Networks, vol. 61, no. 2, pp. 104–116, Mar. 2013.
[51] Q. Yu, C. Cheng, and N. Zhu, “Robust team orienteering prob-
lem with decreasing profits,” INFORMS J. Comput., vol. 34, no. 6,
pp. 3215–3233, Nov. 2022.
[52] S. A. Shah, D. Z. Seker, S. Hameed, and D. Draheim, “The rising role
of big data analytics and IoT in disaster management: Recent advances,
taxonomy and prospects,” IEEE Access, vol. 7, pp. 54595–54614, 2019.
[53] E. Khalil, H. Dai, Y. Zhang, B. Dilkina, and L. Song, “Learning
combinatorial optimization algorithms over graphs,” in Proc. Adv. Neural
Inf. Process. Syst., vol. 30, 2017, pp. 1–11.
[54] Q. Cappart, T. Moisan, L.-M. Rousseau, I. Prémont-Schwarz, and
A. A. Cire, “Combining reinforcement learning and constraint program-
ming for combinatorial optimization,” in Proc. AAAI Conf. Artif. Intell.,
2021, vol. 35, no. 5, pp. 3677–3687.
[55] H. Hu, X. Zhang, X. Yan, L. Wang, and Y. Xu, “Solving a new 3D
bin packing problem with deep reinforcement learning method,” 2017,
arXiv:1708.05930.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
12
[56] Q. Cai, W. Hang, A. Mirhoseini, G. Tucker, J. Wang, and
W. Wei, “Reinforcement learning driven heuristic optimization,” 2019,
arXiv:1906.06639.
[57] S. Manchanda, A. Mittal, A. Dhawan, S. Medya, S. Ranu, and A. Singh,
“GCOMB: Learning budget-constrained combinatorial algorithms over
billion-sized graphs,” in Proc. Adv. Neural Inf. Process. Syst., vol. 33,
2020, pp. 20000–20011.
[58] J. Li, M. Zhou, Q. Sun, X. Dai, and X. Yu, “Colored traveling sales-
man problem,” IEEE Trans. Cybern., vol. 45, no. 11, pp. 2390–2401,
Nov. 2015.
[59] X. Meng, J. Li, X. Dai, and J. Dou, “Variable neighborhood search for a
colored traveling salesman problem,” IEEE Trans. Intell. Transp. Syst.,
vol. 19, no. 4, pp. 1018–1026, Apr. 2018.
[60] X. Xu, J. Li, and M. Zhou, “Delaunay-triangulation-based variable
neighborhood search to solve large-scale general colored traveling
salesman problems,” IEEE Trans. Intell. Transp. Syst., vol. 22, no. 3,
pp. 1583–1593, Mar. 2021.
[61] X. Xu, J. Li, M. Zhou, and X. Yu, “Precedence-constrained col-
ored traveling salesman problem: An augmented variable neighborhood
search approach,” IEEE Trans. Cybern., vol. 52, no. 9, pp. 9797–9808,
Sep. 2022.
[62] I. Bello, H. Pham, Q. V. Le, M. Norouzi, and S. Bengio, “Neu-
ral combinatorial optimization with reinforcement learning,” 2016,
arXiv:1611.09940.
[63] W. Kool, H. Van Hoof, and M. Welling, “Attention, learn to solve routing
problems!” 2018, arXiv:1803.08475.
[64] R. Gama and H. L. Fernandes, “A reinforcement learning approach to the
orienteering problem with time windows,” Comput. Oper. Res., vol. 133,
Sep. 2021, Art. no. 105357.
[65] B. Lin, B. Ghaddar, and J. Nathwani, “Deep reinforcement learning for
the electric vehicle routing problem with time windows,” IEEE Trans.
Intell. Transp. Syst., vol. 23, no. 8, pp. 11528–11538, Aug. 2022.
[66] J. Li et al., “Deep reinforcement learning for solving the heterogeneous
capacitated vehicle routing problem,” IEEE Trans. Cybern., vol. 52,
no. 12, pp. 13572–13585, Dec. 2022.
[67] T. Barrett, W. Clements, J. Foerster, and A. Lvovsky, “Exploratory
combinatorial optimization with reinforcement learning,” in Proc. AAAI
Conf. Artif. Intell., 2020, vol. 34, no. 4, pp. 3243–3250.
[68] M. Nazari, A. Oroojlooy, L. Snyder, and M. Takác, “Reinforcement
learning for solving the vehicle routing problem,” in Proc. Adv. Neural
Inf. Process. Syst., vol. 31, 2018, pp. 1–11.
[69] L. Duan et al., “A multi-task selected learning approach for solving 3D
flexible bin packing problem,” 2018, arXiv:1804.06896.
[70] Z.-C. Li and Q. Liu, “Optimal deployment of emergency rescue stations
in an urban transportation corridor,” Transportation, vol. 47, no. 1,
pp. 445–473, Feb. 2020.
[71] L. Zhuang, J. He, Z. Yong, X. Deng, and D. Xu, “Disaster information
acquisition by residents of China’s earthquake-stricken areas,” Int. J.
Disaster Risk Reduction, vol. 51, Dec. 2020, Art. no. 101908.
[72] S. Vigerske and A. Gleixner, “SCIP: Global optimization of mixed-
integer nonlinear programs in a branch-and-cut framework,” Optim.
Methods Softw., vol. 33, no. 3, pp. 563–593, May 2018.
[73] Y. Pang, Y. Zhang, Y. Gu, M. Pan, Z. Han, and P. Li, “Efficient data col-
lection for wireless rechargeable sensor clusters in harsh terrains using
UAVs,” in Proc. IEEE Global Commun. Conf., Dec. 2014, pp. 234–239.
[74] Y. Long, G. Xu, J. Zhao, B. Xie, and M. Fang, “Dynamic Truck–UAV
collaboration and integrated route planning for resilient urban emergency
response,” IEEE Trans. Eng. Manag., pp. 1–13, 2023.
[75] O. Ghdiri, W. Jaafar, S. Alfattani, J. B. Abderrazak, and
H. Yanikomeroglu, “Energy-efficient multi-UAV data collection
for IoT networks with time deadlines,” in Proc. IEEE Global Commun.
Conf., Dec. 2020, pp. 1–6.
[76] F. S. Moosavi Heris, S. F. Ghannadpour, M. Bagheri, and F. Zandieh,
“A new accessibility based team orienteering approach for urban tourism
routes optimization (a real life case),” Comput. Oper. Res., vol. 138,
Feb. 2022, Art. no. 105620.
[77] S. Alfattani, W. Jaafar, H. Yanikomeroglu, and A. Yongacoglu, “Multi-
UAV data collection framework for wireless sensor networks,” in Proc.
IEEE Global Commun. Conf. (GLOBECOM), Dec. 2019, pp. 1–6.
Authorized licensed use limited to: BLDEA's College of Eng & Tech - Vijayaura. Downloaded on February 09,2024 at 07:08:39 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS
Pengfu Wan received the B.S. degree in indus-
try engineering from Nanjing University, Nanjing,
China, in 2021, and the M.Sc. degree in engineer-
ing enterprise management from The Hong Kong
University of Science and Technology, Hong Kong,
in 2022. He is currently pursuing the Ph.D. degree
with the Department of Aeronautical and Aviation
Engineering, The Hong Kong Polytechnic Univer-
sity, Hong Kong.
His research interests include data-driven opti-
mization and control, reinforcement learning, and
emergency management.
Gangyan Xu (Member, IEEE) received the B.S.
degree in automation and the M.E. degree in sys-
tems engineering from the Huazhong University of
Science and Technology, Wuhan, China, in 2009 and
2012, respectively, and the Ph.D. degree in systems
engineering from The University of Hong Kong,
Hong Kong, in 2016.
He is currently an Assistant Professor with The
Hong Kong Polytechnic University, Hong Kong.
Prior to that, he was an Assistant Professor with the
Harbin Institute of Technology, Shenzhen, China; a
Research Fellow with Nanyang Technological University, Singapore; and a
Research Assistant with the City University of Hong Kong, Hong Kong.
His research interests include data-driven optimization and control, intelligent
transportation systems, resilient engineering, and emergency management.
Dr. Xu is an Editorial Board Member of Advanced Engineering Informatics
and a Special Corresponding Expert of Frontiers of Engineering Management.
Jiawei Chen received the B.S. degree in automation
from the Harbin Institute of Technology, Shenzhen,
China, in 2022. She is currently pursuing the Ph.D.
degree with the Department of Aeronautical and
Aviation Engineering, The Hong Kong Polytechnic
University, Hong Kong.
Her research interests include data-driven opti-
mization and control, intelligent transportation sys-
tems, and reinforcement learning.
Yaoming Zhou received the B.Eng. degree in
mechatronics from Zhejiang University, Hangzhou,
China, in 2014, and the Ph.D. degree in operations
research from The University of Hong Kong, Hong
Kong, in 2018.
From 2018 to 2019, he was a Senior Algorithm
Engineer with Alibaba. Since 2019, he has been
an Associate Professor with the Department of
Industrial Engineering and Management, Shanghai
Jiao Tong University, Shanghai, China. His research
interests include the modeling and analysis of trans-
portation systems, and the integration of operations research, data analytics,
and artificial intelligence and their application to transportation.