sensors

Article
A Multiple Sensors Platform Method for Power Line
Inspection Based on a Large Unmanned Helicopter
Xiaowei Xie 1 , Zhengjun Liu 1, *, Caijun Xu 2 and Yongzhen Zhang 1
1 Institute of Photogrammetry and Remote Sensing, Chinese Academy of Surveying and Mapping,
Beijing 100830, China; 2014102140006@whu.edu.cn (X.X.); yz_zhang87@163.com (Y.Z.)
2 School of Geodesy and Geomatics, Wuhan University, Wuhan 430079, China; cjxu@sgg.whu.edu.cn
* Correspondence: zjliu@casm.ac.cn; Tel.: +86-010-6388-0536

Academic Editors: Cheng Wang, Julian Smit, Ayman F. Habib and Michael Ying Yang
Received: 31 March 2017; Accepted: 22 May 2017; Published: 26 May 2017

Abstract: Many theoretical and experimental studies have been carried out in order to improve the
efficiency and reduce labor for power line inspection, but problems related to stability, efficiency, and
comprehensiveness still exist. This paper presents a multiple sensors platform method for overhead
power line inspection based on the use of a large unmanned helicopter. Compared with the existing
methods, multiple sensors can realize synchronized inspection on all power line components and
surrounding objects within one sortie. Flight safety of unmanned helicopter, scheduling of sensors
and exact tracking on power line components are very important aspects when using the proposed
multiple sensors platform, therefore this paper introduces in detail the planning method for the flight
path of the unmanned helicopter and tasks of the sensors before inspecting power lines, and the
method used for tracking power lines and insulators automatically during the inspection process.
To validate the method, experiments on a transmission line at Qingyuan in Guangdong Province
were carried out, the results show that the proposed method is effective for power line inspection.

Keywords: unmanned helicopter; power line inspection; automatic target tracking

1. Introduction
With the development of the global economy and population, the power demand keeps rising,
leading to an increasing scale of expansion of overhead transmission lines, of which a major part
needs to cross regions with relatively adverse natural environment and complex terrain, such as
rivers, mountains, forests and even depopulated zones, etc. Because the power lines and towers suffer
from long-term exposure to the environmental conditions in the field, combining the influences of
material aging, thunder strikes, and interference from trees, it is highly possible for malfunctions of
electric power equipment to happen, including broken conductors, damaged or overheating insulators,
tilted towers, and discharges caused by trees [1,2]. Therefore electric power departments need to
conduct regular inspection on electric transmission lines, to detect potential abnormal conditions and
to conduct timely maintenance.
The current inspection on electric transmission lines mainly depends on manpower inspection
and manned helicopter inspection. However the labor intensity is quite high, the working conditions
are poor and the inspection efficiency concerning manpower inspection is low; whereas in the case
of manned helicopter inspections, the cost is quite high, and the operation is risky. In recent years,
a variety of schemes have been proposed to replace the manual inspection and manned helicopter
inspection, which can be divided into three categories. The first category is the use of various kinds of
robots for power line inspection [3–9], for example, the “LineScout” developed by the Hydro-Québec
research institution (Varennes, QC, Canada), the “Expliner” developed by the Tokyo Institute of
Technology (Tokyo, Japan), the “TI” developed by Electric Power Research Institute (Palo Alto, CA,

Sensors 2017, 17, 1222; doi:10.3390/s17061222 www.mdpi.com/journal/sensors

Sensors 2017, 17, 1222 2 of 17

USA), etc. These robots are characterized by high safety and low cost, can be used in continuous
inspection for fault diagnosis of power lines, the deficiency is that they cannot support inspection on
other power line components and surrounding objects. The second category is inspection methods
using cameras or video cassette recorders (VCRs) based on unmanned aerial vehicles (UAVs); examples
can be seen in [10–19]. Power line components and surrounding objects can be inspected by using
different sensors in different sorties. However, limited by the short flight time of UAVs, it is unfeasible
to conduct long-distance inspection on power lines. The third category is inspection methods using
typical photogrammetric or LiDAR techniques based on manned aerial vehicles, which are more
effective and can be used for long distance inspection, but the details of the power line components
cannot be obtained by these methods [20–23].
In this paper, we propose a multiple sensors platform method for power line inspection.
Compared to the schemes mentioned above, the use of multiple sensors can support inspection
of all power line components and surrounding objects within one sortie. A large unmanned helicopter
is used here to meet the requirements for carrying the multiple sensors platform and to support the
power line inspection over a long distance. Therefore, it will be more effective and more comprehensive
for power line inspection.
When the multiple sensors platform is used for power line inspection, the flight safety of the
unmanned helicopter, scheduling of sensors and exact tracking of power line components are very
important aspects. This paper present the method of using a multiple sensor platform, which includes
two steps, the first is to plan the flight path of the unmanned helicopter and the coordinates needed to
perform the tasks for each sensor; the second is to complete all tasks automatically based on an exact
target tracking algorithm, through which we can acquire different data about power line components
or surrounding objects efficiently and accurately with different sensors during the inspection process.
This paper is organized as follows: Section 2 introduces the components of the multiple sensors
platform. Section 3 introduces the planning for the flight paths of the unmanned helicopter and the
tasks of each sensor before inspecting the power lines. Section 4 introduces the automatic tracking
method used during the power line inspection process. Section 5 validates the dependability of the
method presented in this paper on the basis of experiments, followed by conclusions in Section 6.

2. The Composition of the Multiple Sensors Platform

The multiple sensors platform includes two parts, inertial stabilized platform and the sensor pod,
as shown in Figure 1a. A two-axis inertial stabilized platform is adopted here [24], which is used
to adjust the attitude of the sensor pod so that the sensors’ line of sight (LOS) can track the targets
accurately in real time. The sensor pod is the unit where all the sensors are placed, which integrates
six sensors in total including the position and orientation system (POS), light detection and ranging
(LiDAR), thermal camera, ultraviolet camera, a long-focus camera and a short-focus camera as shown
in Figure 1b. Among all the sensors, the lines-of-sight of the infrared camera and the ultraviolet camera,
the long-focus camera and the short-focus camera are parallel to each other.
POS can obtain the real-time attitude and coordinates of the sensor pod, and it is used on
real-time target tracking during the power line inspection and on the data analysis after the inspection.
The high-precision point cloud acquired by LiDAR can be used on monitoring the distance between
the electric transmission lines and the surroundings (trees in particular). The thermal infrared data
collected by the thermal camera can be used on monitoring the heat status of facilities including electric
power fittings, power lines and insulators, etc. The ultraviolet videos acquired by the ultraviolet
camera can be used for monitoring abnormal discharges of electric power fittings, power lines and
insulators. Images acquired by the short-focus camera can be used on inspecting whether there are
broken conductors in the shield wires and titled towers. High-resolution images acquired by the
long-focus camera can be used on diagnosing the anomalies including missing pins, burst insulators
and corroded electric power fittings. Parameters of the sensors are shown in Table 1, including sensor
name, product model, focal length, field of view (FOV), and data type.
Sensors 2017, 17, 1222 3 of 17

Sensors 2017, 17, 1222 3 of 17

(a) (b)
Figure 1. Appearance of the multiple sensors platform: (a) Overall picture of the multiple sensors
Figure 1. Appearance of the multiple sensors platform: (a) Overall picture of the multiple sensors
platform; (b) Appearance of the sensor pod.
platform; (b) Appearance of the sensor pod.
Table 1. Parameters of the sensors.
Table 1. Parameters of the sensors.
Sensor Name Product Model Focal Length (mm) FOV ( × ) Data Type
LiDAR Rigel VZ400 - - Point cloud
Sensor Name Product Model Focal Length (mm) FOV (H × V) Data Type
Thermal camera customized 100 9° × 6° Video
° °
LiDAR
Ultraviolet camera Rigel VZ400
customized -- 9 × 6.75 - VideoPoint cloud
◦ ° ◦
Thermal camera camera Canon
The short-focus customized
5D Mark II 100
35 54° × 938× 6 Image Video
◦ ◦
The long-focus
Ultraviolet camera camera Canon 5D Mark II
customized 180- 11° × ×° 6.75 Image Video
9 7.6
◦ ◦
The short-focus camera Canon 5D Mark II 35 54 × 38 Image
◦ ◦
3. Planning
The of camera
long-focus Flight PathsCanon
and Tasks of theII Sensors
5D Mark 180 11 × 7.6 Image
The planning of the flight paths of the unmanned helicopter and tasks of the sensors includes
3. Planning of Flight
two parts: Paths
the first is to and Tasks
design the of thepaths
flight Sensors
of the unmanned helicopter while inspecting the
electric transmission lines; and the second is to plan the tasks of all sensors while the unmanned
The planning of the flight paths of the unmanned helicopter and tasks of the sensors includes two
helicopter is flying along the flight paths. The current UAV power lines inspection generally adopts
parts:onethesensor,
first isthe
to flight
design theplanning
path flight paths
mostlyofinvolves
the unmanned
connecting helicopter
and smoothingwhilea inspecting the electric
series of flight-path
transmission
segments referring to the towers of power lines, which fails to consider other influencing factors, and is
lines; and the second is to plan the tasks of all sensors while the unmanned helicopter
flyingthealong themostly
sensor flightcontinue
paths. The current
working UAV power
on time-lapse, lines is
so there inspection
no need togenerally
plan sensoradopts one sensor,
tasks [25–27].
the flight pathfrom
Different planning mostly
the current involves
flight connecting
paths planning, and smoothing
the flight control system a series of flight-path
of the unmanned segments
helicopter
presented
referring to the in this paper
towers can automatically
of power lines, whichstart failsthe smoothingother
to consider function according
influencing to the waypoints,
factors, and the sensor
mostly so continue
we only need to provide
working waypoints with
on time-lapse, so no smoothing.
there is no needIn addition,
to planwhilesensorplanning the waypoints,
tasks [25–27]. Different
from the current flight paths planning, the flight control system of the unmanned helicopterpower
we also take into account the influence of complex terrain (such as mountains) and non-target presented
lines. On the other hand, using multiple sensors can support inspection for all power line components
in this paper can automatically start the smoothing function according to the waypoints, so we only
and surrounding objects within one sortie, so scheduling of sensors is very important aspect.
need Therefore,
to providethis waypoints with no smoothing. In addition, while planning the waypoints, we also
paper presents the planning method for the tasks of all sensors during the inspection
take into account
process. the influence of complex terrain (such as mountains) and non-target power lines.
On the other hand,
Before usingthe
planning multiple sensors
flight paths andcan support
tasks inspection
of the sensors, it is for all power
assumed linehigh-precision
that the components and
surrounding
3D pointobjects
cloud ofwithin oneelectric
the entire sortie,transmission
so scheduling linesofhassensors is very(the
been acquired important
LiDAR on aspect. Therefore,
the multiple
this paper presents the planning method for the tasks of all sensors during the inspection process.
sensors platform can be used to acquire the point clouds), thus the precise 3D coordinates of the
towers, insulators and power lines can be obtained, as well as the digital surface
Before planning the flight paths and tasks of the sensors, it is assumed that the high-precision model (DSM) data
of the entire transmission line corridor, of which the resolution and elevation
3D point cloud of the entire electric transmission lines has been acquired (the LiDAR on the multiple accuracy are 15 m and
3 m, respectively.
sensors platform can be used to acquire the point clouds), thus the precise 3D coordinates of the
towers, insulators and power lines can be obtained, as well as the digital surface model (DSM) data
3.1. Waypoints Planning of the Unmanned Helicopter
of the entire transmission line corridor, of which the resolution and elevation accuracy are 15 m and
There are three reasons for planning the waypoints of the unmanned helicopter. First, the
3 m, respectively.
unmanned helicopters have a maximum flight time which is limited by their fuel carrying capacity.
Second, the
3.1. Waypoints distanceofbetween
Planning the unmanned
the Unmanned helicopter and the power lines must be larger than a
Helicopter
safe distance. Finally, the closer the unmanned helicopter is to the electric transmission lines, the
There aredata
better the three reasons
collected by for planning
the sensors thetowaypoints
is, and of data
better acquire the from
unmanned helicopter.
insulators First,
on each tower, it the
unmanned helicopters have a maximum flight time which is limited by their fuel carrying capacity.
Second, the distance between the unmanned helicopter and the power lines must be larger than a safe
distance. Finally, the closer the unmanned helicopter is to the electric transmission lines, the better the
Sensors 2017, 17, 1222 4 of 17

data collected
Sensors 2017,by the sensors is, and to better acquire data from insulators on each tower, it is4necessary
17, 1222 of 17

to slow down at each tower. Therefore, it is necessary to plan the waypoints in a way that can not
is necessary to slow down at each tower. Therefore, it is necessary to plan the waypoints in a way
only guarantee a safe and effective inspection, but also ensure data quality good. The planning of the
that can not only guarantee a safe and effective inspection, but also ensure data quality good. The
waypoints of using the unmanned helicopter on inspecting electric transmission lines includes three
planning of the waypoints of using the unmanned helicopter on inspecting electric transmission lines
steps.includes
The first step
three is toThe
steps. create the referential
first step waypoints
is to create the according
referential waypointstoaccording
coordinates of the towers
to coordinates of of
the electric transmission
the towers lines
of the electric that need lines
transmission to bethat
inspected. The
need to be second step
inspected. is to optimize
The second step is tothe referential
optimize
waypoints based on
the referential the DSM
waypoints dataonand
based the coordinates
DSM data andofcoordinates
non-targetofelectric transmission
non-target lines. Finally,
electric transmission
basedlines. Finally,
on the based on
waypoints ofthe waypointshelicopter,
unmanned of unmanned helicopter,
operating operatingof
parameters parameters
LiDAR can of LiDAR can
be calculated.
be calculated.
3.1.1. Referential Waypoints Planning
3.1.1. Referential Waypoints Planning
Assuming that D1 is the horizontal safe distance between the unmanned helicopter and the
power lines, Assuming
D2 is the thatvertical
is the horizontal
safe distancesafe distance
between between
the unmanned the unmanned
helicopterhelicopter
and the and power the lines.
power lines, is the vertical safe distance between the unmanned helicopter and the power lines.
As shown in Figure 2, T( x, y, z, w) representing the n towers on the electric transmission lines, ( x, y, z, w)
As shown in Figure 2, T , , , representing the n towers on the electric transmission lines,
representing
, , ,
for the centric coordinates at the top and the width of the towers, these data can be
representing for the centric coordinates at the top and the width of the towers, these data
acquired
can be from the 3Dfrom
acquired pointthecloud data of
3D point the electric
cloud data of transmission lines. We begin
the electric transmission lines. by
Weconnecting
begin by the
points of every two towers to obtain a series of line segments
connecting the points of every two towers to obtain a series of1 line T T , T T ,
2 segments
2 3 · · · , T T . Secondly,
n−1, n , ⋯ , .move
all theSecondly,
line segmentsmove all with 1 + segments
theDline w/2 horizontally
with ⁄2 horizontally
+to obtain a series oftocross
obtainpoints C =of{cross
a series 2 , · · · , Cn },
C1 , Cpoints
then Cmove
= , with
C , ⋯ , D2, vertically
then move to C obtain
with the temporary
vertically waypoints,
to obtain R0 = { R1 ,which
which iswaypoints,
the temporary R2 , · · is
· , R n }.
Finally, =
R extend , the , ⋯ ,distance
. Finally, extend
S along R2the to obtainS point
R1 distance along R0 as theto obtain point andasextend
inlet point, the inletthe
point,
distance
andRextend
S along n−1 Rn the distance
to obtain S along
point to obtain point
Rn+1 as the appearance point, the asRthe= {appearance
R0 , R1 , · · · point,
, Rn , Rthe
n +1 }R are
= the
, , ⋯ , , are the final waypoints,
final waypoints, S is set according to the practical situation. S is set according to the practical situation.

Figure
Figure 2. 2.The
Theprinciple
principle of
of referential
referentialwaypoints
waypointsplanning.
planning.

3.1.2. Optimization of Waypoints

3.1.2. Optimization of Waypoints
Unavoidably, except for target electric transmission lines that need to be inspected, sometimes
Unavoidably, except for
there are also non-target target electric
transmission linestransmission lines
in the same area. that need
Therefore, wetoalso
be need
inspected, sometimes
to ensure the
theresafety
are also non-target transmission lines in the same area. Therefore, we also
distance between the unmanned helicopter and the non-target transmission lines. In addition,need to ensure the
safetygiven
distance between
the complex the unmanned
environment helicopter and
of the transmission line the non-target
corridor, it is alsotransmission
need to keep alines. In addition,
safe distance
givenbetween
the complex environment
the unmanned of theand
helicopter transmission line corridor,
ground to guarantee it is also
the safety of theneed to keephelicopter.
unmanned a safe distance
We assume
between that
the unmanned ishelicopter
the safe distance betweento
and ground theguarantee
unmannedthe helicopter
safety of andthe
theunmanned
ground. helicopter.
The optimization of waypoints is to check and handle security
We assume that D3 is the safe distance between the unmanned helicopter and the ground.risks in referential waypoints,
based on the DSM data and coordinates of non-target transmission lines. The detailed steps of
The optimization of waypoints is to check and handle security risks in referential waypoints, based
on the DSM data and coordinates of non-target transmission lines. The detailed steps of optimizing the
Sensors 2017, 17, 1222 5 of 17

waypoints are 17,

Sensors 2017, as 1222
follows. For the auxiliary waypoints obtained in the following steps, the unmanned 5 of 17
helicopter does not slow down at these waypoints, therefore here each flight-path segment Ri Ri+1 is
deemsoptimizing
as a whole, the waypoints
which theare as follows.
auxiliary For the auxiliary
waypoints waypoints
are included in, asobtained
Ri · · · Rinp ·the
··Rfollowing
i +1 :
steps,
the unmanned helicopter does not slow down at these waypoints, therefore here each flight-path
(a) For any one segment
segment Ri Ri+as1 , disperse
is deems a whole, itwhich
into n the
temporary
auxiliarypoints according
waypoints are to a step length.
included in, as
(b) Staring
⋯ ⋯from: Ri , we take each temporary point p as the center and D3 as the radius to draw
a circle.
(a) Based
For any oneon the DSM data,
segment estimate
, disperse whether
it into the distance
n temporary points between
according thetocoordinates
a step length. of point p
and the ground within the circle is greater
(b) Staring from , we take each temporary point p as3 the center and than D is estimated. If it is greater, the
as the radius to draw a following
stepscircle.
should Based be on
continued;
the DSM data, if it isestimate
less, the auxiliary
whether waypoint
the distance R p is added
between at point of
the coordinates p.point p
The height
of Rand
p should
the enable
ground the
within distance
the circlebetween
is greater R p and
than all ground
is points
estimated. If within
it is the
greater, circle
the to be greater
following
steps should be continued; if it is less, the auxiliary waypoint
than D3 . When point p is too close to a certain waypoint, this waypoint can be heightened to is added at point p. The height
ensure should
of that the enable
heightthe of distance meets the and
point p between all ground points
requirement, without within
the the circle toto
necessity beadding
greater new
auxiliary waypoints. p
than . When point is too close to a certain waypoint, this waypoint can be heightened to
ensure that the height of point p meets the requirement, without the necessity to adding new
(c) Taking point p as the center and taking D1 as the radius, re-draw a circle to estimate whether
auxiliary waypoints.
there are non-target power lines falling into the new circle, as shown in Figure 3.
(c) Taking point p as the center and taking as the radius, re-draw a circle to estimate whether
(d) If there
therearearenon-target
non-target powerpowerlines lines fall into
falling intothe thenewnew circle
circle, as and
shown cross with 3.
in Figure axle b, the distance
between point p and the power lines is r 0 , it need to add an auxiliary waypoint R acquired by
(d) If there are non-target power lines fall into the new circle and cross with axle b, thepdistance
moving p outwardly
between pointp and with D1 − rlines
the power 0. is r′, it need to add an auxiliary waypoint acquired by
(e) moving p
If there are power outwardly
lines fallwith this
into − ′.circle and cross with axle a, it is necessary to estimate whether
(e) If there distance
the vertical are power lines fall
between into pthis
point andcircle
the powerand crosslinewith axle a,than
is greater it is D necessary to estimate
2 . If it is greater, nothing
whether the vertical distance between point p and the power line is greater than . If it is
needs to be done; if it is less, it is necessary to add the auxiliary waypoint R p , of which the
greater, nothing needs to be done; if it is less, it is necessary to add the auxiliary waypoint , of
( x, y) can be acquired by intersection point between the power lines and axle a, and z is the height
which the , can be acquired by intersection point between the power lines and axle a, and
of thez ispower
the heightlinesofadded
the power with D2 .added
lines Similarly,
with when point p
. Similarly, is too
when close
point to a certain waypoint, it is
p is too close to a certain
feasible
waypoint, it is feasible to directly heighten this waypoint to ensure that the safe between
to directly heighten this waypoint to ensure that the safe distance point p and
distance between
the lines
point can p andmeetthethe requirement.
lines can meet the requirement.
(f) (f)
RepeatRepeat
Steps Stepsb–eb–e
to to completethe
complete the optimization
optimization ofofallall waypoints.
waypoints. It needs
It needsto beto noted that, once
be noted that,a once
new auxiliary waypoint was added in steps b, d, and e, it is
a new auxiliary waypoint was added in steps b, d, and e, it is necessary to recalculate the necessary to recalculate the dispersed
temporary
dispersed points from
temporary pointsthe from
last waypoint, then repeatthen
the last waypoint, Stepsrepeat
b–e. Steps b–e.

(a)

(b)
Figure 3. The interference of non-target power lines: (a) Power lines in parallel; (b) Power lines cross
Figure 3. The interference of non-target power lines: (a) Power lines in parallel; (b) Power lines
over.
cross over.
Sensors 2017, 17, 1222 6 of 17

Sensors 2017, 17, 1222 6 of 17

3.1.3. Operating Parameters of LiDAR
In order to ensure the data acquired by LiDAR meet the requirement, reasonable operating
3.1.3. Operating Parameters of LiDAR
parameters need to be calculated before power line inspection. The longitudinal and lateral distances
In order
between laser to ensure the
footprints ,
( data acquired by LiDARaccording
) can be calculated meet thetorequirement,
the followingreasonable
formulas: operating
parameters need to be calculated before power line inspection. The longitudinal and lateral distances
= /2 (1)
between laser footprints (dlon , dlat ) can be calculated according to the following formulas:
= 4ℎ tan /2 / (2)
dlon = v/2 f s (1)

= f s tan+
dlat =ℎ 4h (θ/2)/F (2)
(3)
q
h = D12 + D22 (3)
= tan − / + tan + + ′ / (4)
θ = tan−1 (( B − D1 )/H ) + tan−1 B + D1 + w0 /H



(4)
= ⁄2 + tan + /2
/
− tan + + ′ /
(5)
θs = π/2 + tan−1 D1 + w0 /2 /D2 − tan−1 B + D1 + w0 /H


(5)

θe = +θ
= θs + (6)
(6)
In the
In theformulas
formulasabove,
above, represents
v represents flightflight
speedspeed
of theof the unmanned
unmanned helicopter,
helicopter, represents
f s represents scanning
scanning
speed, speed, ℎ distance
h represents represents distance
between LiDARbetween LiDARlines,
and power andF power lines,
represents pulse represents pulseθ
repetition rate,
repetition rate, represents scanning angle, represents the distance need to be covered
represents scanning angle, B represents the distance need to be covered on each side of the power lines, on each
side of the power
w0 represents the average represents
lines, width the average
of the power width of the
lines, H represents the power
averagelines, represents
flying height relativethe to
average flying height relative to the ground, represents starting scanning
the ground, θs represents starting scanning angle, and θe represents terminational scanning angle, angle, and as
represents terminational scanning
to theangle, as shown of din Figure 4. According to the requirement of
shown in Figure 4. According requirement lon and dlat for power line inspection, ( f s , F, θs , θe )
and for power line inspection,
can be determined.
, , , can be determined.

Figure 4. The scanning angle of LiDAR.

3.2. Planning of Tasks of the

the Sensors
Sensors
Task planning
Task planningisisthetherealization
realizationofof
thethe scheduling
scheduling of the
of the sensors.
sensors. TheThe LiDAR,
LiDAR, thermal
thermal camera
camera and
and ultraviolet
ultraviolet camera
camera can start
can start workwork directly
directly whenwhen the unmanned
the unmanned helicopter
helicopter reaches reaches the starting
the starting point
pointcan
and and canwhile
stop stop while it reaches
it reaches the ending
the ending point; point;
it onlyitneeds
only needs
to set to setpoints
task task points
at the at the starting
starting point
pointending
and and ending point
point for for starting
starting and stopping
and stopping data collection.
data collection. Therefore Therefore
the majorthe major
part of thepart of the
planning
planning
of tasks ofofthe
tasks of theissensors
sensors to planisthe
to plan
taskthe taskfor
points points
the for the short-focus
short-focus cameracamera to the
to track track the power
power lines
lines and the long-focus camera to track the insulators. Task points represent the
and the long-focus camera to track the insulators. Task points represent the initial position for the initial position for
the cameras
cameras to carry
to carry out out
the the photo-taking
photo-taking tasks;tasks;
eacheach
tasktask
pointpoint corresponds
corresponds to a to a target
target position.
position. HereHere
the
the infrared camera and the ultraviolet camera are not taken into consideration, because
infrared camera and the ultraviolet camera are not taken into consideration, because the lines-of-sight the lines-of-
sight
of the of the thermal
thermal camera,camera, ultraviolet
ultraviolet camera, camera, short-focus
short-focus cameras andcameras and long-focus
long-focus cameras are cameras
parallelare
to
parallel
each to each other.
other.
Sensors 2017, 17, 1222 7 of 17
Sensors 2017, 17, 1222 7 of 17

As shown
As shown in Figure
in Figure5a,5a,assuming
assuming thatthat Ti Ti+1 isis aasegment
segmentof of power
power lines
lines composed
composed of two
of two
neighboring
neighboring towers,
towers, andandthe
thelength TL. The
length isis TL. The corresponding
correspondingflight segment
flight segment of the unmanned
of the unmanned
helicopter
helicopter is Ris , and
i Ri +1 , and thethe lengthisis RL,
length RL, Rc isisaatemporary
temporary point closetoto Ri+. 1 .
pointclose

(a)

(b)

Figure
Figure 5. The
5. The principle
principle of task
of task planning:(a)
planning: (a)The
Thelayout
layout of
of task
task points
pointsfor
fortwo
twocameras;
cameras;(b)(b)
The order
The order to
to shoot insulators of the long-focus
shoot insulators of the long-focus camera.camera.

To ensure the short-focus camera and the long-focus camera can accomplish all tasks when the
To ensure helicopter
unmanned the short-focus camera
is flying throughandthethe long-focus
flight segment, camera canfor
task points accomplish all tasks
the short-focus camerawhen
to the
unmanned helicopter is
track the power lines offlying through the
are set at flight segment, task points for the short-focus camera
and task points for the long-focus camera to track the to
trackinsulators
the power the of T
on lines Ti+1 are
itowers areset at R i Rc and
setat task
. The pointssteps
detailed for the long-focus
are as follows: camera to track the
insulators on the Ti+1 towers are set at Rc Ri+1 . The detailed steps are as follows:
(a) According to the insulators’ number of towers on the right side, we can get the
(a) According
number M ofto thetheinsulators’
task pointsnumber of Ti+1 towers
for the long-focus on the
camera. The right side,
centric we can get
coordinates theinsulators
of the number M of
are used
the task as target
points positions,
for the and the
long-focus shooting
camera. Theorder is from
centric bottom to of
coordinates topthe
andinsulators
from left toareright
used as
as positions,
target shown in Figure and the5b.shooting
The insulators
order on the other
is from bottomsidetooftop
theand
towers
fromareleft
inspected
to rightthrough
as shown in
bilateral
Figure 5b. Theflying.
insulators on the other side of the towers are inspected through bilateral flying.
(b) To ensure that there is enough time for the implementation, set the minimum distance between
(b) To ensure that there is enough time for the implementation, set the minimum distance between
every two task points as ts, multiplied by M, we can get = × which is the flight
every two task points as ts, multiplied by M, we can get RL2 = tsis×tracking
distance of the unmanned helicopter when the long-focus camera
M which is the flight distance
the insulators. The
of the unmanned helicopter when the long-focus camera is tracking
start point of the task points of the long-focus camera can be obtained based on the insulators., andThethe start
point R
spacing
c of the task points of the long-focus
between the two task points is ts. camera can be obtained based on RL 2 and the
,
spacing
(c) between
Calculate the two
the image taskW
width points is ts. to FOV of the short-focus camera and the distance
according
(c) between
Calculate thecamera
image and power
width Wlines, the distance
according to FOV hereofadopts + ⁄2, which
the short-focus camera is mentioned in
and the distance
Section 3.1. Starting from , calculate the times the short-focus camera
between camera and power lines, the distance here adopts D + w/2, which is mentioned in need to take photos
1
Section 3.1. Starting from Ti , calculate the times the short-focus camera need to take photos
between Ti Ti+1 in the horizontal direction as the overlap is q, N = TL/((1 − q) × W ) + 1.
The centric position of each image can be used as the target position for tracking.
Sensors 2017, 17, 1222 8 of 17

Sensors 2017, 17, 1222 8 of 17

(d) Then to plan uniformly-spaced task points for the short-focus camera according to N on the flight
segment of Ri Rc , theinlength
between of Ri Rcdirection
the horizontal is RL1 . as However,
the overlapwhen , N =< (⁄N +
is RL 1 −M ) ×
× ts, +
all1.the
Thetask
pointscentric
of theposition
short-focused camera
of each image need
can be used to be
ascompleted starting
the target position forfrom Ri based on a separation
tracking.
(d) Then
distance of to
ts.plan
The uniformly-spaced
task points of thetask points for
long-focus the short-focus
camera are put offcamera
orderly,according N on thetask
and thetoexceeded
pointsflight segment
are all the position
set in of of Ri+of1 . Meanwhile
, the length is . the
However,
unmanned RL <
when helicopter + is set
× to, all the
hovering
task points of the short-focused camera need to be completed starting
at the position of Ri+1 , and the hovering duration is determined according to the amount ofa the from based on
separation distance of ts. The task points of the long-focus camera are put off orderly, and the
exceeded tasks and the minimum duration needed to implement the tasks.
exceeded task points are all set in the position of . Meanwhile the unmanned helicopter is
set toto
According hovering at above,
the steps the position
we can
of obtain
, and
thethe hovering duration
coordinates of all taskispoints
determined
in theaccording to
flight segment
the amount of the exceeded
of Ri Ri+1 and the corresponding targets. tasks and the minimum duration needed to implement the tasks.
According to the steps above, we can obtain the coordinates of all task points in the flight
segmentTarget
4. Automatic of and the corresponding targets.
Tracking
This paper adopts
4. Automatic TargettheTracking
double closed loop control method to achieve the automatic target tracking.
The outer loop adopts the distance control method, enabling the camera to start aiming and tracking
This paper adopts the double closed loop control method to achieve the automatic target
when the helicopter is close enough to the task point, to ensure the orderly implementation of
tracking. The outer loop adopts the distance control method, enabling the camera to start aiming and
the inspection tasks.theThe
tracking when inner loop
helicopter is closeadopts
enough the
to attitude control
the task point, method,
to ensure the which
orderly uses the real-time
implementation
coordinates
of the of the projection
inspection center
tasks. The innerof loop
the camera obtained
adopts the attitudefrom POSmethod,
control and thewhich
coordinates
uses theofreal-time
the targets
to be tracked
coordinates of the projection center of the camera obtained from POS and the coordinates adjusting
to calculate the pitch and heading, adjust the posture of camera by gradually of the
the stabilized
targets toplatform,
be tracked which meets the
to calculate the requirements whenadjust
pitch and heading, the difference
the posturebetween
of camera posture of camera
by gradually
and theadjusting
pitch and theheading
stabilizedwe platform, whichismeets
calculated smallthe requirements
enough. whenDGPS
Real-time the difference between
is adopted hereposture
to ensure
of camera
the precision of and the pitch and of
the coordinates heading we calculated is small enough. Real-time DGPS is adopted here
the camera.
to ensure
The the precision
principle of algorithm of theiscoordinates
as shown of inthe camera.
Figure 6. The first step is to obtain the coordinates of
The principle of algorithm is as shown in Figure 6. The first step is to obtain the coordinates of
the current task points through initializing the task list. Second is to check whether enter into the
the current task points through initializing the task list. Second is to check whether enter into the
current task in real time according to the distance control method. Third is to calculate the heading
current task in real time according to the distance control method. Third is to calculate the heading
and pitch
andangle of theoftarget
pitch angle through
the target the coordinates
through the coordinatesof of
thethe
target points
target pointsand
andthethe centric coordinates of
centric coordinates
the camera
of the camera which are corrected by installation errors, and send the angles to the platformto
which are corrected by installation errors, and send the angles to the platform toadjust
adjust the
posturethe
ofposture
camera. ofLast
camera.is toLast
estimate in realintime
is to estimate real whether
time whetherthe current attitude
the current attitudeofof
the
thecamera
camerameets
meets the
requirements for photo-taking,
the requirements and send
for photo-taking, andthe instruction
send for take
the instruction photo
for take to the
photo camera
to the cameraandandcontinue
continue the
the if
next task next task if the
it meets it meets the requirements.
requirements.

Current POS
posture

Current POS Installation Centric coordinates

coordinates error correction of Camera

Initialize Get current Distance Entered into Calculate heading

Y Aimed at target?
tasks list task point control the task? and pitch

N
N
Time over? N
Y
Y

Y
End N More task? Take photo

Figure 6. The
Figure principle
6. The ofofautomatic
principle automatictarget tracking.
target tracking.

One important
One important thingthing to note
to note aboutabout this method
this method is that
is that there
there is aistime
a time limit
limit forfor
thethe executionofofeach
execution
each task because the distance between each two task points is fixed based on the tasks planning.
task because the distance between each two task points is fixed based on the tasks planning. Therefore,
Therefore, once the task has not been completed within the specified time, it will be given up and the
once the task has not been completed within the specified time, it will be given up and the next task
will be started. There are three critical parts in the entire task implementation, which are real-time
correction of installation errors, distance control and attitude control, the details in Sections 4.1–4.3.
Sensors 2017, 17, 1222 9 of 17

next task will be started. There are three critical parts in the entire task implementation, which are
Sensors 2017, 17, 1222 9 of 17
real-time correction of installation errors, distance control and attitude control, the details in Sections
4.1–4.3.
4.1. Real-time Correction of Installation Errors
4.1. Real-time Correction of Installation Errors
During the inspection process on power lines, real-time coordinates of the phase center of the GPS
antennaDuring the by
output inspection
POS are process
obtainedon power To
directly. lines, real-time
ensure coordinates
the precision of theofautomatic
the phase tracking,
center of itthe
is
GPS antenna output by POS are obtained directly. To ensure the precision
necessary to transfer the real-time coordinates of the phase center of the GPS antenna to the real-timeof the automatic tracking,
it is necessary
coordinates of to
thetransfer the real-time
photographing center coordinates
of the camerasof thebyphase centerthe
correcting of the GPS antenna
installation to the real-
errors:
time coordinates of the photographing center of the cameras by correcting the installation errors:
∆x∆w
       
xce ∆xwc xe
∆
g
e g
( L, B) R,nl ( R, P, H, ) R, lg (α, β, γ, ),  ∆y∆w + Rc ( ϕ,, ω,, k)∆ ∆ycw + +  ye  (7)
 e 
 yc  = Rn = (7)
    
g +
zec ∆z∆wg ∆z
∆ w
c z e

e
In the
the equation
equationabove,above,Rn ( L, B , ) is isthe
therotation
rotationmatrix matrix from
from thethe
navigation
navigation coordinate
coordinate system
system to the
to
n
Rl ( R, P, H, ) ,is the
geocentric
the geocentriccoordinate
coordinate system.system. isrotation matrix
the rotation from the
matrix from IMU
thecoordinate
IMU coordinatesystemsystem to the
navigation coordinate
to the navigation coordinate Rlg (α, β, γ) is, the
system.system. , rotation matrix from
is the rotation matrix the reference
from coordinate
the reference system
coordinate
g
of the stabilized
system platformplatform
of the stabilized to the IMUto the coordinate system. system.
IMU coordinate Rc ( ϕ, ω, k) is ,the, rotation
is thematrix
rotation from
matrix the
camera x e , ye , ze ) is
from thecoordinate system to the
camera coordinate reference
system to thecoordinate
reference system coordinate of the stabilized
system of the platform.
stabilized
 ( platform.
c c c
the ,3D ,coordinates
is the 3D coordinates
of the of thecenter.
photographing photographing
( xe , ye , ze )center. , is the ∆x
is the GPS ,coordinates. w , ∆y
GPS w , ∆zw is
g coordinates.
g g
∆ , ∆ , ∆ is the eccentric component between the center of
the eccentric component between the center of the stabilized platform and the phase center of the GPSthe stabilized platform and the
c c cantenna. ∆ , ∆ , ∆
antenna. (∆xw , ∆yw , ∆zw ) is the eccentric component from the photographing center of the camera to
phase center of the GPS is the eccentric component from the photographing
center
the of the
center camera
of the to the platform.
stabilized center of the stabilized platform.

4.2. Distance Control

As shown
shown in in Figure
Figure7,7,the theunmanned
unmannedhelicopter
helicopterhashas toto
pass
passthethe
task points
task points planned
planned in Section
in Section3.2
in
3.2ideal conditions.
in ideal conditions. However,
However, influenced
influenced bybyvarious
variousfactors,
factors,thetheunmanned
unmanned helicopter
helicopter cannot
cannot fly
strictly according to the planned flight paths, causing difficulties in estimating directly whether the
position
position ofofthe theunmanned
unmanned helicopter
helicopter satisfies the conditions
satisfies the conditions to implement the tasks.
to implement theThe usual
tasks. solution
The usual
solution is to set a distance threshold . When the distance between the unmanned helicopter task
is to set a distance threshold D. When the distance between the unmanned helicopter and the and
point
the task is point
less than D,than
is less it meets , it the
meets requirement, but the
the requirement, but value of D of
the value here is notiseasy
here to set,
not easy to when the
set, when
value of D of
the value is tooissmall and the
too small and unmanned
the unmannedhelicopter is too deviated
helicopter from the from
is too deviated planned theflight
plannedpaths, the
flight
possibility of failing to
paths, the possibility ofimplement the tasks isthe
failing to implement quite
taskshigh; and high;
is quite whenand the when
value the of Dvalue
is tooofgreat,is the
too
tracking
great, theeffect will be
tracking influenced.
effect will be This paper present
influenced. a method
This paper presentcombining
a method distance
combining and angle to check
distance and
angle to check
whether whether
it is satisfied it isthe
with satisfied with the
requirement to requirement
perform taskstoor perform
not. tasks or not.
It is assumed that the current position of the unmanned unmanned helicopter is is p,, the location of the last
task point is C1 , and the location of the next task will be carried out is C2 .. Firstly a greater distance
threshold D isis
threshold setset
andand we ensure
we ensure thatdistance
that the the distance
between between
p and C2 will and willmeet
certainly certainly meet the
the threshold D.
Secondly, . Secondly,
threshold calculate ∠ pC2 C calculate
1 when the ∠ distancewhen the distance
between p and C2between and D, From
is smaller than is smaller
Figurethan ,
6 it can
From
be seenFigure
that when6 it can be 2seen
∠ pC that when
C1 reaches 90◦ , ∠ reachesflies
the unmanned 90°,over
the unmanned flies over
the cross section of Cthe cross
exactly. section
Lastly,
◦ °
 ◦2
with θthe
< 3requirement C1  ≤ θ,

of angle
an exactly.
thresholdLastly,
is setanθangle
< 3 , threshold is set
it is satisfied , it is satisfied
whenwith 90 − ∠ pC2when
the requirement
p meets
° ◦
pC2 C |90
meets
∠ 1 cannot− ∠ reach |90 ≤ exactly
, ∠ duecannot
to datareach 90° exactly due to data errors.
errors.

Figure 7. The principle of distance control.

Figure 7. The principle of distance control.
Sensors 2017, 17, 1222 10 of 17

4.3. Attitude Control

After the position of the unmanned helicopter meets the requirements for implementing the
current task, we calculate the heading and the pitch angle between the photographing center of the
camera and the target, and send adjustment instruction to the stabilized platform. When the difference
between the posture of camera and the calculated angle smaller than angle threshold, we perform the
current task. The steps of the attitude control are as follows:

(a) Motion compensation: If the time-delay of the camera shooting is t, before calculating the heading
and the pitch angle between the photographing center of the camera and the target, it needs
to take the distance of the unmanned helicopter flying forward within t into consideration.
The current coordinates of the photographing center of the cameras are ( x0 , y0 , z0 ), the current


speeds provided by POS are v x , vy , vz , and after the time t, the photographing center of the
cameras reach the coordinates of ( x1 , y1 , z1 ). According to Equation (8), ( x1 , y1 , z1 ) can be obtained:
       
x1 vx t ∆x x0
c 0 0 0 
 y1  = Rl α , β , γ  vy t  +  ∆y  +  y0 


(8)
      
z1 vz t ∆z z0

In the formula above, Rcl (α0 , β0 , γ0 ) is the rotation matrix from the IMU coordinate system to the
coordinate system of the camera. (∆x, ∆y, ∆z) is the eccentric component between the IMU center
and the center of the camera.
(b) Angle calculation: Coordinates of the photographing center of the camera are ( x1 , y1 , z1 );
coordinates of the target are ( x2 , y2 , z2 ). We can calculate the heading and pitch at according to
Equations (9) and (10):


 π/2 − ε ( x2 > x1 , y2 6= y1 )

 3π/2 − ε x < x , y 6= y
( 2 1 2 1)
heading = (9)

 0 ( x 2 = x ,
1 2y > y 1)


π ( x2 = x1 , y2 < y1 )

pitch = sin−1 ((z1 − z2 )/dist) (10)

k = ( y2 − y1 ) / ( x2 − x1 ) (11)

ε = tan−1 (k) (12)

q
dist = ( x2 − x1 )2 + ( y2 − y1 )2 + ( z1 − z2 )2 (13)

(c) Attitude control: To send angle adjustment instructions to the stabilized platform. Setting an
angle threshold to conduct real-time estimation on whether the attitude of the platform meets the
requirement for the threshold value. If it meets the requirement, photographing instructions are
sent to the cameras; if it does not meet the requirement, we repeat Steps a-c until the requirement
is met.

5. Experiments and Analyses

A 110 kV electric transmission line at Qingyuan in Guangdong Province was selected for
experiments to validate the feasibility of the method presented in this paper. The total length of this line
is 4.2 km; it has 13 towers that are numbered from 316 to 328, and has 78 insulators. The geographical
environment of the power line passages is complex with quite great topographic relieves, as shown in
Figure 8. The 3D point cloud data of this electric transmission line was obtained by APLIS. We adopted
the method introduced in Part 3 to plan the waypoints of unmanned helicopter and the tasks of sensors.
Sensors 2017, 17, 1222 11 of 17

The operating Parameters of LiDAR are shown in Table 2, and the parameters adopted in the planning
are as 2017,
Sensors shown in Table 3.
17, 1222 11 of 17

2m 168 m
Figure
Figure 8.
8. Appearance
Appearance of
of power
power line
line for
for the
the experiments.
experiments. The
The elevation
elevation in
in this
this area
area is color-coded.
is color-coded.

Table 2. The
Table 2. The operating
operating Parameters
Parameters of
of LiDAR.
LiDAR.

Parameter Value
Parameter
Scanning speed 40 line/s Value
Pulse repetition
Scanning speed rate 300 kHz 40 line/s
Pulse repetition
Starting rate angle
scanning 40° 300 kHz
◦
Starting scanning
Terminational angle angle
scanning 105° 40
◦
Terminational scanning angle 105
Table 3. The parameters adopted in the planning flight path and task points.
Table 3. The parameters adopted in the planning flight path and task points.
Parameter Value
Safe distance to power line in horizontal direction 30 m
Parameter Value
Safe distance to power line in vertical direction 40 m
Safe distance to distance
Safe power line
of in horizontal
unmanned direction
helicopter 5030mm
Safe distance to power line in vertical
The distance of S direction 5040mm
Safe distance of unmanned helicopter 50 m
The overlay of image 30%
The distance of S 50 m
The overlay
The minimum distance for task
of image 5 30%
m
The minimum time
The minimum distance for task for task 25s m
The minimum time for task 2s
Figure 9a shows the planned waypoints of the unmanned helicopter. There are 30 waypoints, of
which the green ones are starting points and the purple ones are ending points. After taking off from
Figure 9a shows the planned waypoints of the unmanned helicopter. There are 30 waypoints, of
around tower No. 328, the unmanned helicopter firstly flew along the right side of the power lines
which the green ones are starting points and the purple ones are ending points. After taking off from
from tower No. 328 to tower No. 316, then it flew along the left side of the power lines from tower
around tower No. 328, the unmanned helicopter firstly flew along the right side of the power lines
No. 316 to tower No. 328. At a place near tower No. 323, there is a crossing between a non-target
from tower No. 328 to tower No. 316, then it flew along the left side of the power lines from tower
power line and the experimental power line, so we increase the height of the corresponding
No. 316 to tower No. 328. At a place near tower No. 323, there is a crossing between a non-target
waypoint. In addition, we add an auxiliary waypoint between the two waypoints corresponding the
power line and the experimental power line, so we increase the height of the corresponding waypoint.
tower No. 324 and tower No. 325, for the large terrain variation. Figure 9b presents the task points of
In addition, we add an auxiliary waypoint between the two waypoints corresponding the tower
the cameras. For the short-focus camera, there are 163 task points which are displayed in light blue,
No. 324 and tower No. 325, for the large terrain variation. Figure 9b presents the task points of the
and the average distance between each two task points is about 30 m. For the long-focus camera,
cameras. For the short-focus camera, there are 163 task points which are displayed in light blue, and
there are 78 task points which are displayed in pink, and the distance between each two task points
the average distance between each two task points is about 30 m. For the long-focus camera, there
is 5 m. The distance between every two towers is long enough, therefore the unmanned helicopter
are 78 task points which are displayed in pink, and the distance between each two task points is 5 m.
does not need to hover at any one of towers to extend the time to take photos of the insulators. Figure
9c shows the task points at tower No. 320. Figure 9d presents the comparison between the actual
flight paths and the planned flight paths of the unmanned helicopter, the deep green represents for
the actual waypoints, and it can be seen that the unmanned helicopter cannot strictly fly according
to the planned flight paths. The line-pressing accuracy in this experiment is about ±5 m.
Sensors 2017, 17, 1222 12 of 17

The distance between every two towers is long enough, therefore the unmanned helicopter does not
need to hover at any one of towers to extend the time to take photos of the insulators. Figure 9c shows
the task points at tower No. 320. Figure 9d presents the comparison between the actual flight paths
and the planned flight paths of the unmanned helicopter, the deep green represents for the actual
waypoints, and it can be seen that the unmanned helicopter cannot strictly fly according to the planned
Sensors 2017, 17, 1222 12 of 17
flight paths. The line-pressing accuracy in this experiment is about ±5 m.

(a) (b)

(c) (d)

2m 168 m

Figure9.9.The
Figure Theresult
resultof
ofplanned
plannedflight
flightpath
pathand
andtasks,
tasks,and
andtrue
trueflight
flightpath,
path,the
theelevation
elevationin
inthis
thisarea
areaisis
color-coded.
color-coded. (a) The planned flight path; (b) The planned task points; (c) The zoomed image oftask
(a) The planned flight path; (b) The planned task points; (c) The zoomed image of task
points
pointsat
attower
towerNo.
No.320;
320;(d)
(d)The
Thetrue
trueflight
flightpath
pathduring
duringpower
powerline
lineinspection.
inspection.

Dataacquired
Data acquiredin inthis
thisexperiment
experimentincludes
includes956 956M Mlaser-point
laser-pointcloud
clouddata
dataofofthe
theentire
entirepassages
passagesof of
thepower
the powerlines,
lines,15.8
15.8G GHDHDinfrared
infraredvideos
videosof ofthe
theelectric
electrictransmission
transmissionline,
line,51.4
51.4G Gultraviolet
ultravioletvideos
videos
of the electric transmission line, 163 images of the power lines taken by the short-focus
of the electric transmission line, 163 images of the power lines taken by the short-focus camera, and camera, and
73images
73 imagesof ofthe
theinsulators
insulatorstaken
takenbybythe
thelong-focus
long-focuscamera.
camera.
The scheduling
The scheduling method
method (task
(task planning
planning of of sensors)
sensors) ofof multiple
multiple sensors
sensors can
can be
beevaluated
evaluated bybythe
the
success rates of the tasks. Compared with the planned task points, it can be found
success rates of the tasks. Compared with the planned task points, it can be found that the success that the success
rateof
rate oftasks
tasksforforthe
theshort-focus
short-focus camera
camera and
and the
the long-focus
long-focus camera
camera are 100%
100% and 93.5% respectively,
respectively,
as shown
as shown in Table 4. From FromthetheSection
Section4,4,weweknow
know that thethe
that distance
distancebetween
betweeneach twotwo
each tasktask
points will
points
limit the time for task execution, therefore there are several failed
will limit the time for task execution, therefore there are several failed tasks for tasks for the long-focus camera
long-focus camera
affectedby
affected bysome
somefactors
factors(adjusting
(adjustingspeed
speedof ofthe
thestable
stableplatform,
platform, response
responsetime
timeof ofcamera,
camera,and andflight
flight
speed of unmanned helicopter, etc.), while the success rate of tasks for the short-focus camera is
higher than the long-focus camera because the distance between each two task points of which is
longer. However, longer distance between each two task points for the long-focus camera will affect
the image resolution and viewing angle, so we should take a balance between data quality and task
execution time based on actual power lines information when conducting power line inspection.
Sensors 2017, 17, 1222 13 of 17

speed of unmanned helicopter, etc.), while the success rate of tasks for the short-focus camera is higher
than the long-focus camera because the distance between each two task points of which is longer.
However, longer distance between each two task points for the long-focus camera will affect the image
resolution and viewing angle, so we should take a balance between data quality and task execution
time based on actual power lines information when conducting power line inspection.

Table 4. The success rate of tasks for two camera.

Sensor Planned Task Number Image Number Success Rate

The short-focus camera 163 163 100%
The long-focus camera 78 73 93.5%
Total 241 236 98%

The effect of automatic target tracking can be evaluated through analyzing and comparing the
offset 2017,
Sensors distance between the photo center and the target object on the images. Because the FOV13ofofthe
17, 1222 17
long-focus camera is smaller than the short-focus camera and it is more difficult to tacking target for
the long-focus
the long-focus camera,
camera, we
we use
use the
the result
result of
of the
the long-focus
long-focus camera
camera to to evaluate
evaluate the
the effect
effect of
of automatic
automatic
target tracking.
target tracking. Figure
Figure 10
10 shows
shows thethe horizontal
horizontal and
and vertical
vertical offsets
offsets between
between thethe photo
photo center
center and
and the
the
target object
target object onon the images respectively, the mean values are −
the images 0.05m
−0.05 m and
and 0.04
0.04 m,
m, and
and the
the standard
standard
deviation are
deviation are 0.71
0.71 mm and
and 0.69
0.69 m,
m, as
as shown
shown in in Table
Table 5. However,
However, the the image
image coverage
coverage is
is 6.7
6.7 mm in
in the
the
horizontaldirection
horizontal directionandand1010mmininthethe vertical
vertical direction,
direction, greater
greater thanthan
the the offset
offset distance,
distance, so wesocan
wefind
can
find targets in all images. Thus we can conclude that the automatic target tracking
targets in all images. Thus we can conclude that the automatic target tracking method perform quite method perform
quite well.
well.

(a) (b)
Figure 10. Offset between image center and the target. (a) Offset in horizontal direction; (b) Offset in
Figure 10. Offset between image center and the target. (a) Offset in horizontal direction; (b) Offset in
vertical
vertical direction.
direction.

Table 4. The success rate of tasks for two camera.

Table 5. The mean and standard deviation of offset in two direction.
Sensor Planned Task Number Image Number Success Rate
Direction Mean (m) Standard Deviation (m)
The short-focus camera 163 163 100%
Horizontal −0.05 0.71
The long-focus camera 78 73 93.5%
Vertical 0.04 0.69
Total 241 236 98%

Figure 11 shows the data acquired by multiple sensors platform. Figure 11a shows the point cloud
Table 5. The mean and standard deviation of offset in two direction.
data of power lines acquired by LiDAR, the compass on the left displays elevation, while the compass
Direction
on the right displays azimuth; Figure 11bMean (m)theStandard
shows Deviation
thermal infrared (m)acquired by thermal camera,
data
Horizontal −0.05 0.71
Vertical 0.04 0.69

Figure 11 shows the data acquired by multiple sensors platform. Figure 11a shows the point
cloud data of power lines acquired by LiDAR, the compass on the left displays elevation, while the
Sensors 2017, 17, 1222 14 of 17

H, L, M and C represent the maximum temperature, minimum temperature, temperature at image

center, and temperature at crosshair position respectively; Figure 11c shows the photons acquired
by ultraviolet camera, which are superimposed on the optical image; Figure 11d shows the image of
tower acquired by short-focus camera; Figure 11d shows a zoomed image of an insulator acquired by
the long-focus camera.
Sensors 2017, 17, 1222 14 of 17

(a)

(b) (c)

(d) (e)

Figure 11. The data acquired by sensors during power line inspection. (a) The LiDAR point cloud, the
Figure 11. The data acquired by sensors during power line inspection. (a) The LiDAR point cloud,
compass on the left displays elevation, while the compass on the right displays azimuth; (b) The
the compass oninfrared
thermal the leftdata,
displays elevation,
H, L, M while the
and C represent themaximum
compasstemperature,
on the rightminimum
displaystemperature,
azimuth; (b) The
thermaltemperature
infrared data, H, L,center,
at image M and andC temperature
represent the maximum
at crosshair temperature,
position minimum
respectively; temperature,
(c) The photons
acquired
temperature at by ultraviolet
image camera,
center, and superimposed
temperatureonatthe optical image;
crosshair (d) The
position image of tower
respectively; (c)acquired
The photons
acquiredbyby
theultraviolet
short-focus camera,
camera; (e) The zoomed image
superimposed of the
on the insulator
optical acquired
image; (d) by
Thetheimage
long-focus camera.
of tower acquired
by the short-focus camera; (e) The zoomed image of the insulator acquired by the long-focus camera.
 Sensors 2017, 17, 1222 15 of 17
Sensors 2017, 17, 1222 15 of 17

Differentdata
Different data acquired
acquired by
bysensors
sensorscancanbe be
synchronized
synchronizedbased on POS
based on data.
POS After
data. manual analysis
After manual
analysis of the data, two anomalies were found, the first is a broken insulator on tower No. is319,
of the data, two anomalies were found, the first is a broken insulator on tower No. 319, another that
one of the
another torsional
is that one ofdampers had dampers
the torsional fallen off had
at tower
fallenNo.
off327, as shown
at tower in Figure
No. 327, 12. in Figure 12.
as shown

(a) (b)
Figure
Figure12.
12.Two
Twoanomalies
anomaliesononthe
thepower
powerline.
line.(a)
(a)Broken
Brokeninsulator
insulatoronontower
towerNo.
No.319;
319;(b)
(b)AAtorsional
torsional
dampers fallen off at tower No. 327.
dampers fallen off at tower No. 327.

6. Conclusions
6. Conclusions
In this paper, a multiple sensors platform method based on the use of a large unmanned
In this paper, a multiple sensors platform method based on the use of a large unmanned helicopter
helicopter was proposed for power line inspection. LiDAR, thermal camera, ultraviolet camera, and
was proposed for power line inspection. LiDAR, thermal camera, ultraviolet camera, and two cameras
two cameras with different focal lengths are used to acquire information about power line
with different focal lengths are used to acquire information about power line components and
components and surrounding objects. Scheduling of sensors, exact tracking on power line
surrounding objects. Scheduling of sensors, exact tracking on power line components and flight
components and flight safety when the multiple sensors platform is used to power line inspection
safety when the multiple sensors platform is used to power line inspection were introduced in detail.
were introduced in detail. Experiments were carried out on an 110 kV electric transmission line at
Experiments were carried out on an 110 kV electric transmission line at Qingyuan in Guangdong
Qingyuan in Guangdong Province, and the results show that the proposed method is effective for
Province, and the results show that the proposed method is effective for power line inspection. In the
power line inspection. In the future, the method needs to be optimized to further improve the success
future, the method needs to be optimized to further improve the success rate and accuracy of the task
rate and accuracy of the task execution. In addition, we will also start research on the automatic
execution. In addition, we will also start research on the automatic processing and analysis of data
processing and analysis of data acquired by multiple sensors platform for power line inspection.
acquired by multiple sensors platform for power line inspection.
Acknowledgments:
Acknowledgments:We Weare
aregrateful
gratefulfor
forthe
thetwo
twoanonymous
anonymousreviewers
reviewersfor
fortheir
theirinsightful
insightfulcomments,
comments,which
whichled
led
usustotoimprove
improve the manuscript. This work was supported by the National Natural Science Foundation ofChina
the manuscript. This work was supported by the National Natural Science Foundation of China
(No.41431069)
(No. 41431069) and
and thethe State’s
State’s KeyKey Project
Project of Research
of Research andand Development
Development PlanPlan
(Grant (Grant No. 2016YFC0803105).
No. 2016YFC0803105). We
We thank
thank Guangdong
Guangdong Electric Electric PowerResearch
Power Science Science Institute,
ResearchBeijing
Institute, BeijingofUniversity
University AeronauticsofandAeronautics and
Astronautics,
Astronautics, and 60th Institute of General Staff of the Chinese People’s Liberation Army for providing help in the
and 60th Institute of General Staff of the Chinese People’s Liberation Army for providing help in the course of
course of the experiment.
the experiment.
Author Contributions: Xiaowei Xie wrote the majority of the paper; Xiaowei Xie, Zhengjun Liu, Caijun Xu, and
Author
YongzhenContributions:
Zhang providedXiaowei Xie wrote
the ideas the majority
and verified of the paper;
the methods; Xiaowei
Xiaowei Xie, Zhengjun
Xie, Zhengjun Liu,Caijun
Liu and CaijunXu
Xu, and
revised
the manuscript.
Yongzhen Zhang provided the ideas and verified the methods; Xiaowei Xie, Zhengjun Liu and Caijun Xu revised
the manuscript.
Conflicts of Interest: The authors declare no conflict of interest.
Conflicts of Interest: The authors declare no conflict of interest.
References
References
1. Bjarnadottir, S.; Li, Y. Risk-based economic assessment of mitigation strategies for power distribution poles
1. subjected toS.;
Bjarnadottir, hurricanes. Struct. economic
Li, Y. Risk-based Infrastruct.assessment
Eng. 2014, of
10,mitigation
740–752. [CrossRef]
strategies for power distribution poles
subjected to hurricanes. Struct. Infrastruct. Eng. 2014, 10, 740–752, doi:10.1080/15732479.2012.759240.
Sensors 2017, 17, 1222 16 of 17

2. França, G.B.; Oliveira, A.N. A Fire-Risk-Breakdown System for Electrical Power Lines in the North of Brazil.
J. Appl. Meteorol. Climatol. 2014, 53, 813–823. [CrossRef]
3. Pouliot, N.; Montambault, S. LineScout Technology: From Inspection to Robotic Maintenance on Live
Transmission Power Lines. In Proceedings of the 2009 IEEE International Conference on Robotics and
Automation, Kobe, Japan, 12–17 May 2009.
4. Richard, P.L.; Pouliot, N. Introduction of a LIDAR-based obstacle detection system on the LineScout power
line robot. In Proceedings of the IEEE International Conference on Advanced Intelligent Mechatronics (AIM),
Besancon, France, 8–11 July 2014.
5. Debenest, P.; Guarnieri, M. Expliner—Robot for inspection of transmission lines. In Proceedings of the IEEE
International Conference on Robotics and Automation, Pasadena, CA, USA, 19–23 May 2008.
6. Debenest, P.; Guarnieri, M. Field experiences using LineScout Technology on large BC transmission crossings.
In Proceedings of the 1st International Conference on Applied Robotics for the Power Industry, Montreal,
QC, Canada, 5–7 October 2010.
7. Phillips, A.; Engdahl, E. Autonomous Overhead Transmission Line Inspection Robot (TI) Development and
Demonstration. In Proceedings of the 2nd International Conference on Applied Robotics for the Power
Industry, ETH Zurich, Zurich, Switzerland, 11–13 September 2012.
8. Byambasuren, B.E.; Kim, D. Inspection Robot Based Mobile Sensing and Power Line Tracking for Smart Grid.
Sensors 2016, 16, 250. [CrossRef] [PubMed]
9. Seok, K.H.; Kim, Y.S. A State of the art of Power Transmission Line Maintenance Robots. J. Electr. Eng. Technol.
2016, 9, 1412–1422. [CrossRef]
10. Jones, D. Aerial inspection of overhead power lines using video: Estimation of image blurring due to vehicle
and camera motion. IEE Proc. Vision Image Signal Process. 2000, 147, 157–166. [CrossRef]
11. Whitworth, C.C.; Duller, A.W.G. Aerial video inspection of overhead power lines. Power Eng. J. 2001, 15,
25–32. [CrossRef]
12. Wang, B.; Chen, X. Power line inspection with a flying robot. In Proceedings of the 1st International
Conference on Applied Robotics for the Power Industry, Montreal, QC, Canada, 5–7 October 2010.
13. Dai, L.; Qi, J. Camera selection for unmanned helicopter power line inspection. In Proceedings of the 3rd
IEEE PES Innovative Smart Grid Technologies—Asia (ISGT Asia), Tianjin, China, 21–24 May 2012.
14. Li, H.M.; Wang, B. The design and application of SmartCopter: An unmanned helicopter based robot
for transmission line inspection. In Proceedings of the Chinese Automation Congress, Changsha, China,
7–8 November 2013.
15. Luque-Vega, L.F.; Castillo-Toledo, B. Power line inspection via an unmanned aerial system based on the
quadrotor helicopter. In Proceedings of the 17th IEEE Mediterranean Electrotechnical Conference, Beirut,
Lebanon, 13–16 April 2014.
16. Rafique, S.F.; Bodla, M.K. Design and implementation of a UAV for power system utility inspection.
In Proceedings of the 16th International Power Electronics and Motion Control Conference and Exposition
(PEMC), Antalya, Turkey, 21–24 September 2014.
17. Mulero-Pázmány, M.; Negro, J.J. A low cost way for assessing bird risk hazards in power lines: Fixed-wing
small unmanned aircraft systems. J. Unmanned Veh. Syst. 2014, 2, 5–15. [CrossRef]
18. Liu, C.A.; Dong, R. A 3D Laboratory Test-platform for Overhead Power Line Inspection. Int. J. Adv.
Robot. Syst. 2016, 13, 72. [CrossRef]
19. Zhang, Y.; Yuan, X. UAV Low Altitude Photogrammetry for Power Line Inspection. ISPRS Int. J. Geo-Inf.
2017, 6, 14. [CrossRef]
20. McLaughlin, R.A. Extracting transmission lines from airborne LIDAR data. IEEE Geosci. Remote Sens. Lett.
2006, 3, 222–226. [CrossRef]
21. Yoonseok, J.; Gunho, S. A Piecewise Catenary Curve Model Growing for 3D Power Line Reconstruction.
Photogramm. Eng. Remote Sens. 2012, 78, 1227–1240. [CrossRef]
22. Connie, K.O.; Remmel, T.K.; Sohn, G. Mapping tree genera using discrete LiDAR and geometric tree metrics.
Bosque 2012, 33, 313–319. [CrossRef]
23. Kim, H.B.; Sohn, G. Point-based Classification of Power Line Corridor Scene Using Random Forests.
Photogramm. Eng. Remote Sens. 2013, 79, 821–833. [CrossRef]
Sensors 2017, 17, 1222 17 of 17

24. Zhou, X.Y.; Jia, Y. Experimental Validation of a Compound Control Scheme for a Two-Axis Inertially
Stabilized Platform with Multi-Sensors in an Unmanned Helicopter-Based Airborne Power Line Inspection
System. Sensors 2016, 16, 366. [CrossRef] [PubMed]
25. Shanmugavel, M.; Tsourdos, A. Co-operative path planning of multiple UAVs using Dubins paths with
clothoid arcs. Control Eng. Pract. 2010, 18, 1084–1092. [CrossRef]
26. Lin, Y.; Saripalli, S. Path planning using 3D Dubins Curve for Unmanned Aerial Vehicles. In Proceedings of
the International Conference on Unmanned Aircraft Systems (ICUAS), Orlando, FL, USA, 27–30 May 2014.
27. Wang, Y.; Zhu, X.P. Path Planning Based on PH Curves for Unmanned Aerial Vehicles. Comput. Simul. 2013,
30, 76–79.

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).