6th IFAC Conference on Sensing, Control and Automation for

6th IFAC Conference on Sensing, Control and Automation for

Agriculture
6th
6th IFAC Conference
IFAC Conference onon Sensing,
Sensing, Control
Control and
and Automation for
for
Agriculture AvailableAutomation
online at www.sciencedirect.com
6th IFAC Conference
December
Agriculture
Agriculture on Sensing,
4-6, 2019. Sydney, Control and Automation for
Australia
December 4-6, 2019. Sydney, Australia
Agriculture
December
December 4-6,
4-6, 2019.
2019. Sydney,
Sydney, Australia
Australia
December 4-6, 2019. Sydney, Australia
ScienceDirect
IFAC PapersOnLine 52-30 (2019) 342–348
An
An Evaluation
Evaluation of
of an
an Apple
Apple Canopy
Canopy Density
Density
An
An Evaluation
Mapping of an
System
Evaluation Apple
for
of an Apple a Canopy
Canopy Density
Variable-rate
Density
Mapping System for a Variable-rate
Mapping System for a⋆⋆ Variable-rate
Mapping SystemSprayer
for a⋆ Variable-rate
Sprayer
Sprayer
Sprayer ⋆
Mengying Hu ∗∗ Mark Whitty ∗∗
Mengying Hu ∗∗ Mark Whitty ∗∗
∗∗
Mengying Hu
Mengying Hu Mark Mark WhittyWhitty ∗∗
Mengying Hu ∗
Mark Whitty ∗∗
∗ UNSW Sydney, NSW 2052, Australia (e-mail:
∗
UNSW
∗ UNSW Sydney,
Sydney, NSW 2052, Australia (e-mail:
∗
UNSW NSW 2052, 2052, Australia
mengying.hu@unsw.edu.au).
Sydney, NSW Australia (e-mail:
(e-mail:
∗∗UNSW Sydney,
∗ mengying.hu@unsw.edu.au).
NSW 2052, Australia (e-mail:
∗∗ UNSW mengying.hu@unsw.edu.au).
UNSW Sydney, NSW 2052, Australia (e-mail:
Sydney, NSW 2052,
mengying.hu@unsw.edu.au).
∗∗ UNSW mengying.hu@unsw.edu.au).
Australia (e-mail:
∗∗
Sydney, NSW
NSW 2052, Australia (e-mail:
m.whitty@unsw.edu.au)2052, Australia
∗∗ UNSW Sydney, m.whitty@unsw.edu.au) (e-mail:
UNSW Sydney, NSW
m.whitty@unsw.edu.au)
m.whitty@unsw.edu.au)2052, Australia (e-mail:
m.whitty@unsw.edu.au)
Abstract: This
Abstract: This paper
paper proposes
proposes methods
methods for for evaluating
evaluating an an apple
apple canopy
canopy density
density mapping
mapping systemsystem
Abstract:
as an
Abstract: input This
for
This paper
a proposes methods
variable-rate
paper proposes methods
sprayer for evaluating
both an apple
apple canopy
trellis-structured
evaluating an canopy(2D) density
and
density mapping system
standalone
mapping system
(3D)
as an
Abstract: input for
This a variable-rate
paper proposes sprayer
methods for both trellis-structured
evaluating an apple canopy(2D) and
density standalone
mapping (3D)
system
as
apple
as an
an input
orchards.
input for
for a
The
a variable-rate
mobile
variable-rate sprayer
terrestrial
sprayer for
system
for both
used
both trellis-structured
in this paper
trellis-structured (2D)
consists
(2D) and
of
anda 2Dstandalone
LiDAR
standalone (3D)
(Light
(3D)
apple
as an orchards.
input for The
a mobile
variable-rate terrestrial
sprayer system
for used
both in this paper
trellis-structured consists
(2D) of
anda 2D LiDAR
standalone (Light
(3D)
apple
Detection
apple orchards.
and
orchards. The
Ranging),
The mobile
mobile terrestrial
three RGB-D
terrestrial system
cameras
system used
used in
and
in this
a
this paper
GPS-RTK
paper consists
module.
consists of
of a
a A2D
2D 3D LiDAR
point
LiDAR (Light
cloud
(Light
Detection
apple and Ranging),
orchards. The mobile three RGB-D
terrestrial cameras
system used and
in a GPS-RTK
this paper module.
consists of a A2D 3D point (Light
LiDAR cloud
Detection
was
Detection and
generated
and forRanging),
for 2D
Ranging), or three
3D
threetree RGB-D
row, cameras
then convertedand a
to GPS-RTK
a 2D matrix module.
with A 3D
density point cloud
distribution
was generated
Detection and 2D or 3D
Ranging), threetreeRGB-D
row, then cameras
convertedand to a GPS-RTK
a 2D matrix module. A 3D distribution
with density point cloud
was generated
generated
information
was forfor
the
for 2D
2D or 3D3D
variable-rate
or treeRGB-D
tree row, then
sprayer.
row, cameras
then converted
Quad and to
frames
converted awere
to GPS-RTK
aa 2D
2Dplaced
matrix
matrix module.
in with
the
with A 3D
density
trees
densityto pointground
cloud
distribution
gain
distribution
information
was generated for the
for 2D variable-rate
or 3D tree sprayer.
row, then Quad frames
converted towerea 2D placed
matrix in the
with trees
densityto gain ground
distribution
information
truth data for the variable-rate sprayer. Quad frames were placed in the trees to gain ground
truth data for
information
information forforGPS
for
the validation
GPS variable-rate
the validation
variable-rate
and canopy
andsprayer.
canopy Quad
sprayer.
density.
density.
Quad
They
frames
They were
frames
were
were
extracted
were placed
extracted
placed
in the
in the
from
from the
trees 3D
the to
trees 3Dgain
to
point cloud
pointground
gain cloud
ground
truth
by
truth data
intensity
data for
for GPS
GPS validation
thresholding
validationand and
RANSAC
and canopy
canopy density.
along with
density. They
their
They were extracted
locations
were and
extracted from the
timestamps.
from the 3D
3D point
point cloud
cloud
by
truthintensity
data forthresholding
GPS validationand andRANSAC canopy along withThey
density. theirwere locations and from
extracted timestamps.
the 3D point cloud
by intensity
Five
by intensity
evaluationthresholding
methods
thresholding anddiscussed
are
and RANSAC
RANSAC in along
this
along with to
paper
with their locations
validate
their locationsthe and timestamps.
timestamps.
robustness
and and repeatability
Five
by evaluation
intensity methods
thresholding are
and discussed
RANSAC in this
along paper
with to validate
their locationsthe robustness
and and
timestamps. repeatability
Five
of the evaluation
canopy methods are discussed
discussed inQuad
this paper
paper to validate
validate the robustness and repeatability
Five
of
Five canopy density
theevaluation
evaluation
methods
density
methods
mapping
are
mapping
are overall
system:
system:in
discussed
this
inQuad
locations
to
locations
this paper
will
will be
to validate
used
be the
the
to
to evaluate
usedrobustness
evaluate
robustness
and
and
GPS
GPS accuracy;
repeatability
accuracy;
repeatability
of
Quad
of the
the canopy
density
canopy density
deviation
density mapping
and
mapping system:
system: Quad
deviation
Quad locations
in multiple
locations will
will be
passes
be used
will
used to evaluate
evaluate
to evaluate theGPS
GPS accuracy;
repeatability.
accuracy;
Quad
of the density
canopy deviation
density and
mapping overall
system:deviation
Quad in multiple
locations willpasses
be will
used evaluate
to evaluate theGPSrepeatability.
accuracy;
Quad
Then density
manual
Quad manual deviation
classified
density deviation and
quad
and overall
density
overall deviation
will
deviation be in multiple
compared
in multiple with passes
with
passes will
automatic evaluate
will evaluate densitythe repeatability.
extraction
the extraction
repeatability. to
Then
Quad density classified
deviation quad
and density
overall will
deviation be compared automatic density to
Then manual
evaluate
Then manual classified
correlation;
classified The quad
quad density
misalignment
density will
of
will be in
the
be multiple
compared
point
compared cloud passes
with
from
with will
automatic
both
automatic evaluate
sides density
of
densitythe
a repeatability.
extraction
tree row
extraction to
exists
to
evaluate
Then correlation;
manual classified The misalignment of the point cloud from both sides of a tree row exists
evaluate
but this
evaluate correlation;
paper compares
correlation; Thequad
The between density
misalignment
misalignment two will
sides
of be
of the
the
of compared
point
the
pointsamecloud
cloud with
row. from
from automatic
The both
result
both sides density
of aa tree
indicates
sides of extraction
tree
that row
rowit mightto
exists
exists
but this paper
evaluate compares
correlation; The between
misalignment two sides of of the
the pointsamecloud row. fromThebothresult indicates
sides of a that
tree rowit might
exists
but
be this paper
butsufficient to compares
scan between two sides of the same row. The result indicates that it might
be
but
this paper
sufficient
this paper scan from
to compares
compares
one
one side
frombetween
between sidetwoonly
onlysides
two
to
to halve
sides
of thethe
halve thethe
ofmaking
field
same
fieldrow.
same
work.
work. The result indicates that it might
be
be sufficient
The
The proposed
sufficient
proposed
to scan
scan from
tosystem
system will
from
will
one
help
one
help
side
the
side
the
only
only to
to halve
decision
decision halve
making
the
the field
in
field
in aarow.
work. The resultsprayer.
variable-rate
work.
variable-rate
indicates that
sprayer. The
it might
The evaluation
evaluation
be
Thesufficient
methods
The proposed
are
proposed to scan from
system
practical,
system will
easy
will one
help
to
help side
be
the only
theapplied to in
decision
decision halve
makingthe approaches
similar
making field
in aa work.
in variable-rate
to
variable-rate map sprayer.
canopy
sprayer. The evaluation
evaluation
density.
The
methods
The are practical,
proposed system will easyhelp to be theapplied
decision in making
similar approaches
in a variable-rate to mapsprayer.canopy density.
The evaluation
methods
methods are practical,
are practical, easy to
easy to bebe applied
applied in similar
in similar approaches
approaches to map
to map canopycanopy density.
density.
© 2019, IFAC
methods are(International
practical, easy Federation
to beof applied
Automatic in Control) Hosting by Elsevier
similar approaches to mapLtd. All rights
canopy reserved.
density.
Keywords: LiDAR,
Keywords: LiDAR, mapping,
mapping, canopy canopy density,
density, evaluation,
evaluation, variable-rate
variable-rate spraying
spraying
Keywords:
Keywords: LiDAR,LiDAR, mapping,
mapping, canopy canopy density,
density, evaluation,
evaluation, variable-rate
variable-rate spraying
spraying
Keywords: LiDAR, mapping, canopy density, evaluation, variable-rate spraying
1. INTRODUCTION
1. INTRODUCTION LiDAR
LiDAR is is short
short forfor Light
Light Detection
Detection and and Ranging,
Ranging, a a laser
laser
1.
1. INTRODUCTION
INTRODUCTION LiDAR
scanner.
LiDAR is short
short for
isTumbo et
for Light
al. (2002)
Light Detection
compare
Detection and
and Ranging,
ultrasonic
Ranging, and a
a laser
1. INTRODUCTION scanner.
LiDAR isTumbo
short et
for al. (2002)
Light compare
Detection andultrasonic
Ranging, and a laser
laser
scanner. Tumbo
measurements Tumboofet et al.
al. (2002)
(2002) compare ultrasonic and mea-
Mapping canopy
Mapping canopy density
density isis helpful
helpful for for the
the farmers
farmers to
to better
better
scanner.
measurements
scanner. Tumbo ofetcitrus
citrus
al.
canopy
canopy
(2002)
volume
compare
compare
with
with manual
volumeultrasonic manual
ultrasonic
and
and
laser
mea-
laser
measurements
surement methods of citrus
citrus
for canopy volume with manual mea-
Mapping
manage
Mappingthe
manage
canopy
the
canopy density
orchards.
orchards.
is
is helpful
For
densityFor instance,
helpful
instance,
for
for theit farmers
the
it
to
to better
helps accurately
farmers
helps better measurements
accurately surement
measurements methods of for fifteen
of citrus
canopy
fifteen
canopy
trees.
volume
trees.
volume
The
The withresult
with
shows
resultmanual
shows mea-
manual
laser
laser
mea-
Mapping
manage
characterise
manage canopy
the
the the densityFor
orchards.
average
orchards. is helpful
tree
For instance,
volume
instance,fortothe
it
it farmers
helps
calculate
helps to better
accurately
the most
accurately surement
could
surement providemethods
methodshigher for
for fifteen
resolution,
fifteen trees.
trees. The
faster
The result
data
result shows
acquisition,
shows laser
as
laser
characterise the average tree volume to calculate the most could
surement providemethodshigher for resolution,
fifteen faster
trees. The data acquisition,
result shows as
laser
manage
characterise
appropriate
characterise the orchards.
the
spray
the average
volume
average For
tree
for
tree instance,
volume
all
volumespray to
to it helps
calculate
applications
calculate accurately
the
the inmost
that
most could
well asprovide
couldasprovidethe higher
ability to resolution,
deal
highertoresolution, with faster
faster data
defoliated acquisition,
trees
data acquisition,or smallas
as
appropriate spray volume for all spray applications in that well
could the
provide ability
higher deal
resolution,with defoliated
faster data trees or
acquisition, smallas
characterise
appropriate
block. It
appropriate the
spray
requires
spray average
volume tree
for
repeatability
volume for volume
all
and
all spray to
high
spray calculate
applications
accuracy
applications the in
for
inmost
that
long
that well
well as
replants.
as the
the ability
ability to
to deal
deal with
with defoliated
defoliated trees
trees or
or small
small
block. It requires repeatability and high accuracy for long replants.
well as the ability to deal with defoliated trees or small
appropriate
block.
term
block. It
use. spray repeatability
It requires
requires volume for alland
repeatability andspray
high
high applications
accuracy
accuracy for in long
for that replants.
long replants.
term
block.
term
term It requires repeatability and high accuracy for long Evaluation
use.
use.
use.
replants. methods
Evaluation methods need need to to bebe established
established to to estimate
estimate
For
term canopy
use. density mapping in orchards, a range of sensing Evaluation
canopy
Evaluation methods
density.
methods Colao need
needet to
al.
to be
(2017)
be established
use
established to estimate
convex-hull
to estimate
and
For canopy density mapping in orchards, a canopy density. Colao et al. (2017) use convex-hull and
For canopy
devices
For have
canopy density
been
density mapping
used such
mapping in
inasorchards,
2D LiDARs,
orchards, aa range
range
range3D
of
of sensing
sensing
LiDARs,
of sensing
Evaluation
canopy
alpha-shape
canopy
alpha-shape
methods
density.
density.to
to
Colao
measure
Colao
measure
needet
the
et
the
tocanopy
al.
al. be established
(2017)
(2017)
canopy
use
volume.
use
volume.
to estimate
convex-hull
The
convex-hull
The
and
convex-
and
convex-
devices
For
devices have
canopy
have been
density
been used
used such
mapping
such inas
as 2D
2D LiDARs,
orchards,
LiDARs, a range3D
3D LiDARs,
of sensing
LiDARs, canopy
alpha-shape
hull will density.to Colao
measure
overestimate et
thetheal. (2017)
canopy
structure use
volume.
of theconvex-hull
The
tree; and
convex-
however,
ultrasonic
devices have sensors,
been RGB
used cameras,
such as 2D depth
LiDARs, cameras,3D infrared
LiDARs, alpha-shape
hull to
will overestimate measure the
thethe canopy
structure volume.
of the tree; The convex-
however,
ultrasonic
devices
ultrasonic have sensors,
been
sensors, RGB
used
RGB cameras,
such as
cameras, 2D depth
LiDARs,
depth cameras,
cameras,3D infrared
LiDARs,
infrared alpha-shape
hull
the will to measure
overestimate
alpha-shape will the
create canopy
structure
voids volume.
of
insidethe The
tree;
the convex-
however,
tree. Both
sensors,
ultrasonic hyperspectral
sensors, RGB cameras,
cameras, and stereo
depth cameras.
cameras, Some the
infrared hull alpha-shape
will overestimate the structure
will create of thethe
voids inside tree;tree.
however,
Both
sensors,
ultrasonic hyperspectral
sensors,combine
RGB cameras,
cameras, and stereo
depth cameras.
cameras, Some hull will overestimate the structure of the tree;tree.
however,
sensors,
other
sensors,
other
hyperspectral
researchers
hyperspectral
researchers
cameras,
theseand
cameras,
combine these
stereo
sensors
and stereo
sensors to gain aainfrared
tocameras.
cameras.
gain
Some
Some the
better
better
the alpha-shape
methods
methods
alpha-shape
are
are
will influenced
greatly
will
greatly
create
create
influenced
voids
voidsby
by
inside
the
inside
the
the
noises
the
noises
tree.
in
in
Both
LiDAR
Both
LiDAR
sensors,
other
vision
other of hyperspectral
researchers
canopy
researchers cameras,
combine
density.
combine these
(Underwood
these and stereo
sensors
sensors et to
al.
to cameras.
gain
(2016),
gain Some the
aa Shalal
better
better methods
scans.
methodsalpha-shape
They are
are will influenced
greatly
just compared
greatly createtwo
influenced voidsby
by inside
methodsthe
the theshowed
noises
and
noises tree.
in
in Both
LiDAR
sim-
LiDAR
vision of canopy density. (Underwood et al. (2016), Shalal scans.
methods They are just compared
greatly influencedtwo methods
by the and showed
noises in sim-
LiDAR
other
vision
et al.
vision researchers
of canopy
(2013),
of canopy combine
density.
Shalal et
density. these
(Underwood
al. sensors
(2015),
(Underwood et
Auat
et to
al.
al. gain
(2016),
Cheein
(2016), a better
Shalal
et
Shalalal. scans.
ilar
scans. They
performances,
They just
just compared
not
compared giving two
two methods
ground
methods truthand
andforshowed
showed sim-
evaluation.
sim-
et al. (2013), Shalal et al. (2015), Auat Cheein et al. ilar performances,
scans. They just not giving
compared twoground
methods truthandforshowed
evaluation.
sim-
vision
et al.
(2011),
et al. of canopy
(2013),
Dong
(2013), and density.
Shalal
Isler
Shalal et
et (Underwood
al.
(2018),
al. (2015),
Nielsen
(2015), et
Auat
et
Auat al. (2016),
Cheein
al. (2012),
Cheein Shalal
et
et Benal.
al. ilar
Hu
ilar performances,
et al. (2012)
performances, not
did
not giving
not
givinggive ground
any
ground truth
experiment
truth for
for evaluation.
or evalua-
evaluation.
(2011), Dong and Isler (2018), Nielsen et al. (2012), Ben Hu
ilar et al. (2012)
performances, did
not not
givinggive any
ground experiment
truth for or evalua-
evaluation.
et al. (2013), Shalal
Isleret(2018),
al. (2015), Auat al.Cheein
(2012),etBen al. Hu Hu et
tion, et al.al. (2012)
(2012) did not(2017)
give any
any experiment or evalua-
evalua-
(2011),
Grocholsky
(2011),
Grocholsky
Dong
Dong et and
et al.
and
al.
Isler
(2011))
(2011)) (2018), Nielsen
Nielsen et
et al. (2012), Ben tion,eteither.
either. Escol
Escoldid et
et al.
not
al. give
(2017) slice the
the point
sliceexperimentpoint cloud
or
cloud into
into
(2011),
Grocholsky
Grocholsky Dong et
et and
al.
al. Isler (2018), Nielsen et al. (2012), Ben Hu
(2011))
(2011)) tion,
prisms
tion, al.
either.
and
either. (2012)
Escol
measure
Escol did
et
etthenot
al.
al. give any
(2017)
canopy
(2017) sliceexperiment
volume.
slice the
the point
Visually
point or
cloudevalua-
measured
cloud into
into
Among them, some researchers choose ultrasonic sensors prisms
tion, and
either. measure
Escol ettheal. canopy
(2017) volume.
slice theVisually
point measured
cloud into
Grocholsky
Among them, et al.
some(2011))
researchers choose ultrasonic sensors prisms
canopy
prisms and
volume
and measure
with
measure the
tree
the canopy
width
canopy volume.
was Visually
compared
volume. Visuallywith measured
LiDAR
measured
Among
to them,
calculate some
some researchers
density(Hu et al.
al. (2012),choose
(2012), Tumbo ultrasonic
et al. sensors
al. (2002)).
(2002)).
sensors prisms canopy volume with tree width was compared with LiDAR
Among
to them,
calculate density(Hu researchers
et chooseTumbo ultrasonic
et canopyand
measurement
canopy volume
volume measure
towith
show
with the
tree canopy
treecorrelation
width was
width volume.
was ranging Visually
compared
compared with
from
with measured
r LiDAR
= 0.56
LiDAR
Among
to
to calculate
The them,
system
calculate some researchers
density(Hu
developed
density(Hu et
by al.
al. (2012),
Hu
et Hu choose
et al.
al.
(2012), Tumbo
(2012)
Tumbo ultrasonic
was et al.
made
et made sensors
al. (2002)).
up of
(2002)). of to measurement
canopy volume towith
show tree correlation
width wasranging
compared from = 0.56
withrrr LiDAR
The system developed by et (2012) was up measurement
0.82
measurement depending to show
to showon correlation
the algorithms
correlation ranging
rangingused. from
However,
from = 0.56
= 0.56
for
to
The
PC, calculate
system
ultrasonic
The ultrasonic density(Hu
system developed et
by
by Hu
sensor, single
developed al.
single (2012),
et
et al.
chip,
Huchip, Tumbo
(2012)
flow
al.flow
(2012) was et
was made
control al. (2002)).
mechanism
made up
up ofof to to 0.82
measurement depending to show on the algorithms
correlation rangingused. However,
from = 0.56
rmeasure for
PC, sensor, control mechanism 2D
to 0.82
trees,
0.82 depending
it
dependingwill beon
on the
more
the algorithms
difficult
algorithms toused. However,
manual
used. However, for
for
The
PC,
and
PC, system
ultrasonic
nozzles.
ultrasonic developed
sensor,
sensor, by Hu
single
single et
chip,
chip,al. (2012)
flow
flow was
control
control made
mechanism
mechanism up of 2D
to trees,
0.82 it
dependingwill beon more
the difficult
algorithms to manual
used. However,measure for
and nozzles. 2D
canopy
2D trees,
trees, it
volume.
it will
will be
Prism
be more
slicing
more difficult
tends
difficult to
to manual
overestimate
manual measure
measurethe
PC,
and
and canopy volume.
ultrasonic sensor, single chip, flow control mechanism 2D trees, it will be more difficult to manual measure
nozzles.
nozzles. Prism slicing tends to overestimate the
canopy
volume
canopy volume.
and cannot
volume. Prism
solve
Prism slicing
the
slicing tends
canopy
tends porosity
to overestimate
problem.
overestimate the
The
the
⋆
and
⋆ nozzles.
Sponsor and financial support by Horticulture Innovation Aus- volume
canopy and cannot
volume. solveslicing
Prism the canopy tendsporosity problem. The
to overestimate the
volume and
existing and cannot
methods solve the
mentioned the canopy
canopy
above porosity
might notproblem.
be The
be suitable
⋆ Sponsor and financial support by Horticulture Innovation Aus- volume cannot solve porosity problem. The
⋆
tralia Limited
Sponsor
Sponsor and through
financial
and through project
support
financial project AP16005,
by
support AP16005, ”Developing
Horticulture
by Horticulture Innovation
InnovationAgri-Tech
Aus-
Aus- existing
volume methods
and cannot mentioned
solve the above
canopy might
porosity notproblem. suitable
The
tralia Limited ”Developing Agri-Tech existing
for 2D methods mentioned above might not be suitable
⋆ Sponsor
Solutions
tralia
tralia and
for
Limited
Limited financial
thethrough support
Australian
for thethrough
Apple
project
project by Horticulture
Industry”.
AP16005,
AP16005, Innovation
”Developing
”Developing Aus-
Agri-Tech
Agri-Tech for 2D trees.
existing
existing
methods
trees.
methods
Unlike
Unlike 3D
3D trees,
mentioned
mentioned
above
trees,
above
2D trees
2D might as
as it
trees not itbeis shown
issuitable
shown
Solutions
tralia
Solutions
SolutionsLimited
Australian
thethrough
for the
for
Apple
project
Australian
Australian Apple
Apple
Industry”.
AP16005,
Industry”.
Industry”. ”Developing Agri-Tech for
for 2D2D trees.
trees. Unlike
Unlike 3D
3D trees,
trees, 2D might
2D trees not
trees as
as ititbeis
issuitable
shown
shown
Solutions for the Australian Apple Industry”. for 2D trees. Unlike 3D trees, 2D trees as it is shown
2405-8963 ©
Copyright © 2019
2019,IFAC
IFAC (International Federation of Automatic Control)
342 Hosting by Elsevier Ltd. All rights reserved.
Copyright
Peer review© under
2019 IFAC 342 Control.
responsibility of International Federation of Automatic
Copyright
Copyright © 2019 IFAC
© 2019 IFAC 342
342
10.1016/j.ifacol.2019.12.563
Copyright © 2019 IFAC 342
IFAC AGRICONTROL 2019

December 4-6, 2019. Sydney, Australia Mengying Hu et al. / IFAC PapersOnLine 52-30 (2019) 342–348 343

Fig. 1. 3D tree (left) vs 2D tree (right)

in Fig. 1 are not well distinguishable and through-holes

influence a lot in canopy volume. Therefore, neither convex
hull nor alpha-shape is an ideal interpretation. Moreover,
the existing methods lack in evaluation, especially in terms
of repeatability and robustness.
Fig. 2. Hardware system design
Underwood et al. (2016) present a mobile terrestrial scan-
ning system for standalone almond orchards. The authors
scan the orchard with around 580 trees at peak bloom,
fruit-set and just before harvest for two subsequent years.
Data was collected from onboard LiDAR and camera
sensors, then post-processed with an automated software
pipeline. The 3D point cloud model of each tree is ob-
tained separately to calculate canopy volume and assess
its relationship to yield. Each point in the 3D point cloud
is considered as a cubic voxel of edge 0.1m. The canopy
volume is calculated by the sum of two halves from both
sides. The overlap will certainly overestimate the volume.
Strong agreement is seen for the consistency of LiDAR
volume measurements over two years, however, no results
are shown for trellis structured orchards and no ground
truth. Fig. 3. Examples for quads in the orchard

This paper presents a canopy density mapping system. The
system uses a 2D LiDAR for the purpose of designing
a variable-rate sprayer for orchards. This paper seeks
to evaluate the full mapping system for robustness and
repeatability. A major advantage is the generated density
maps are easy to store and visualize when compared
with 3D point clouds and form a suitable input for a
farmer’s variable rate spraying decision support system.
The evaluation methods are suitable to apply to other
similar approaches to map canopy density.
2. SYSTEM DESIGN
2.1 Hardware
A 2D LiDAR SICK LMS151-10100 was set up for main
data collection cause. Three RealSense D435 RGB-D cam-
eras were set-up vertically as shown in Fig. 2 to capture
both RGB images and depth data of apple trees from the
side view. In addition to this, a dual GPS-RTK module
was used to collect location information. The coordinate
of the sensors in the truck was measured for coordinate
transformation. The system was designed and built by
SwarmFarm.
Data was collected by SwarmFarm in apple orchards in
Warragul, VIC, from flowering season (October, 2018)
to fruitlet season (January, 2019), from sparse foliage to
heavy foliage. Two 2D rows and one 3D row were scanned
from both sides.
A quad mentioned in this paper was a metal frame of 0.8
m x 0.8 m. A QR code was setup in lower right to indicate
the position. It was placed in the canopy to help with the
ground truth. Highlighted with retro-reflective orange tape
as in Fig. 3, it was easy to be extracted from LiDAR data
because it returned high-intensity points.
Ten quads were setup in the 3D row - six on one side and
four on the other side, While eighteen in two 2D rows - six
on one side and three on the other side.
2.2 Software
Data collection was carried out by using Robot Operating
System Kinetic Kame (ROS). Google Cloud Platform was
used for data storage for its large storage and accessibility.
Data then were post-processed by using Python 2.7.12. Tiff
files were generated for visualization in MATLAB R2017b.
A Tiff image for a tree row discussed in this paper is less

343
IFAC AGRICONTROL 2019
344
December 4-6, 2019. Sydney, Australia Mengying Hu et al. / IFAC PapersOnLine 52-30 (2019) 342–348
Fig. 4. Validation procedure
Fig. 5. Interpolation of scan jumps
than 20 KB. This file format largely reduces the need for
storage space.
3. EXPERIMENTAL APPROACH
The experimental approach goes as indicated in Fig. 4.
We built the canopy density map and extracted quads
for ground truth. Then we evaluated the system in five
aspects.
3.1 Build a canopy density map for a row
The procedure went as follows: Lines of points from LiDAR
were projected to 3D space by using GPS, and LiDAR
offset for the GPS mounting in relation to the LiDAR
mounting;
Start and end location of each row are recorded during
the data collection. 3D point cloud from above step can
be trimmed to remain data consistency. The ground was
removed by height. The netting was removed by statistical
noise removal as well;
Scan jumps happens for unstable speed or GPS missing.
Interpolation was carried out in 3D point cloud where
the distance between two scans is over 0.01m as shown
in Fig. 5;
Downsampling was implemented by a voxel grid filter with
leaf size 0.5m;
344
Then from the side view, the canopy was cropped into
cells with 0.5 x 0.5m. The number of points in each cell
was calculated as the canopy density;
A map of canopy density from cells could be generated.
In general, the range of the number of points is from 0 to
500. So all the maps are visualized by MATLAB using a
range (0,500) . The canopy density map of a 3D row and
the original point cloud from side view is shown in Fig. 6.
Retro-reflective points were extracted as potential quad
points. A RANSAC algorithm shown in Algorithm 1 was
used to fit a square of 0.8 x 0.8 m to the candidates from
the side view. Every time a quad was detected, the points
of the quad were removed from the points until there were
less than 100 points in the candidate pool. Duplicate fitted
quads then could be combined according to their locations.
Quad IDs were calculated from GPS location and the point
density within each quad was calculated.
Algorithm 1. Using RANSAC to fit a square in a point cloud to
extract the quads
Data: P, max iter, threshold, min inlier
Result: Extract one quad from retro-reflective points
Quad=zeros(1,3) //Quad location
nBest=0
threshold=constant
while iter <max iter do
P3=SelectRandomSubset(P)
Quad candidate = ComputeQuad(P3)
nInliers = ComputeInliers(Quad)
Cost = ComputeCost(P3, nInliers)
if nInliers >nBest and Cost <threshold then
Quad = Quad candidate
nBest = nInliers
end
end
With GPS and LiDAR data, two rows of trellis-structured
and standalone trees are built. It can be seen in Fig. 7 that
the quads are extracted and moved above using intensity
thresholding and RANSAC. The blue points returned from
the quads covered with retro-reflective tapes.
In addition, each quad in each scan has been associated
with the exact image by the timestamp and GPS location.
In this way, manually classified quad density is enabled to
evaluate the method.
With the above information, we will be able to evaluate
the repeatability and robustness of the canopy density
mapping method from five aspects.
3.2 GPS validation
Quads were placed in the canopy and scanned multiple
times during multiple passes. We were be able to extract
quad utm positions, which can be used to evaluate the
accuracy of GPS position. In this way, the standard
deviation of quad position horizontally and vertically were
calculated for twelve quads in a 2D row and a 3D row. The
2D row was scanned for nine times and the 3D row was
scanned for fifteen times within one week.
IFAC AGRICONTROL 2019

December 4-6, 2019. Sydney, Australia Mengying Hu et al. / IFAC PapersOnLine 52-30 (2019) 342–348 345

Fig. 6. An example for canopy density map for 3D trees

3.5 Consistency for the whole row

One 2D row and one 3D row have been processed for 10

times. Data were collected within one week. The number
of points are calculated. Density in each cell is calculated
by the number of points in each cell, of course, after
downsampling. The maps are compared for consistency.

3.6 Consistency from both sides

Misalignment of the point cloud from both sides of a tree

Fig. 7. Extracted quads in trellis-structured tree row
3.3 Consistency of quads density
For 2D tree row, one quad has been scanned for nine
times, while 3D tree row has been scanned for fifteen times
within one week. The point cloud inside one quad was
cropped out. After applying voxel grid filter with a certain
leaf size, the number of the point N was calculated and
divided by the area of the quad to get the density, as it is
shown in (1).
row happens due to many factors including time synchro-
nization, bad road condition, and wind. Underwood et al.
(2016) use the average of the tree canopy volume from
both sides to calculate individual tree canopy volume. For
a solid object with smooth surface, like a cylinder post, the
method is practical. While for a tree with through holes
and complicated surface, points from the opposite side of
the tree can still be detected so it will greatly overestimate
the canopy volume.
So we explored the possibility that scanning a tree row
from one side might be sufficient to characterize the canopy
density. We scanned 2D and 3D tree rows from both sides
and analyzed the relationship.
4. RESULTS AND DISCUSSION

N ∗ leaf size2 4.1 GPS accuracy

density = (1)
quad edge2
There are many components that contribute to GPS error:
The density of each quad has been taken into account to the timing of the GPS signal, time synchronization with
derive the standard deviation. computer and LiDAR, the motion, the data rate...
In this paper, we focus on the overall error rather than
specific error sources. As it is can be seen in Table. 1,
3.4 Manual and automatic measurements comparison the position of the same quad in 2D tree row deviates 30
mm horizontally,33mm vertically. For 3D trees, the quad
position deviates 74 mm horizontally, 32mm vertically.
Because a late data collection, the 2D tree rows were all
faced with an almost full canopy condition. As we did not Table 1. GPS deviation of the center of twelve
foresee this problem and put the frames in around 1 meter quads over multiple passes
height, where the canopy varied little. The variation of
quads in 2D tree row is fairly small. Therefore, in this Row σh (mm) σv (mm) Row σh (mm) σv (mm)
subsection, only manual classified quads in 3D tree rows 2D 39 35 3D 73 30
are discussed. Ten quads were manually classified their 2D 34 40 3D 69 28
2D 28 30 3D 78 33
density from 0-10 during fifteen passes. The fifteen passes
2D 29 37 3D 36 34
were collected within one week. 2D 29 27 3D 85 39
Manual measurements and automatic measurements by 2D 24 26 3D 104 27
(1) have been compared to see the relationship. Average 30 32 Average 74 32

345
IFAC AGRICONTROL 2019
346
December 4-6, 2019. Sydney, Australia Mengying Hu et al. / IFAC PapersOnLine 52-30 (2019) 342–348

Fig. 8. R2 between manual classification and automatic Fig. 9. Canopy density map for 3D tree row and the dif-
calculation of quad density. Canopy density in each ference map. The last image is the average difference
quad is manually classified as a number in range map for 5 rows above.
0-10 (10 means full canopy), while automatic mea-
surements give a percentage from 0-1 (1 means full Table 2. Correlation coefficient of
canopy). canopy density
Row R2 Row R2
2D 0.98 3D 0.99
GPS performed pretty well. In the 2D tree row, the 2D 0.98 3D 0.98
deviation is within 4 cm horizontally and vertically. In 2D 0.98 3D 0.98
the 3D tree row, the horizontal deviation is also within 10 2D 0.98 3D 0.98
cm. Though it is still bigger than the radius of flowers or 2D 0.98 3D 0.99
branches, it should be enough for variable-rate sprayers. 2D 0.98 3D 0.98
2D 0.98 3D 0.99
2D 0.90 3D 0.97
2D 0.94 3D 0.98
4.2 Result for consistency of quads density Average 0.97 Average 0.98

The standard deviation of the density from twenty-eight

4.5 Result for consistency from both sides
quads on average is 3%, showing robustness during multi-
ple scans for both 2D and 3D trees.
As it can be seen from Table. 3, the average correlation
coefficient is R2 = 0.89 for 2D tree rows, and R2 = 0.90
for the 3D tree row.
4.3 Result for manual and automatic measurements Compared with consistency for one side of a row during
comparison multiple passes, the difference is a little bit higher, but
still within tolerable range. The difference map and the
Automatically calculated density in above session shows calculated standard deviation show the consistency of two
robustness. The correlation coefficient between manual sides of the same row is good. The data we collected from
and automatic measurement for 3D trees is R2 = 0.82. this system give us the confidence to tell that it is enough
Due to small sample size and limited variation in canopy to scan only one side of a row to halve the workload.
density, this figure is not very high. However, compared Table 3. Correlation coefficient of
with the result from Escol et al. (2017), which showed canopy density from both sides
r = 0.52 to 0.82, our experiment shows much better
correlation between manual and automatic canopy density Row R2 Row R2
measurements. 2D 0.91 3D 0.92
2D 0.91 3D 0.85
2D 0.90 3D 0.91
2D 0.89 3D 0.91
4.4 Result for consistency for the whole row 2D 0.90 3D 0.92
2D 0.90 3D 0.92
2D 0.91 3D 0.90
As the numbers of datasets were chronological, we put 2D 0.90 3D 0.92
ten datasets for visualization for 2D and 3D trees. Two 2D 0.81 3D 0.90
difference maps are also given in Fig. 9 and Fig. 10. Average 0.89 Average 0.90

The correlation efficient is calculated for both 2D and 3D

rows in Table. 2. The result shows R2 is 0.96 for the 2D tree
row, 0.98 for 3D tree row. Taking into account the natural
growing of the leaves and the GPS/IMU information that
has not been fused in LiDAR data, the result is quite
promising.
5. CONCLUSION
This paper proposes several evaluation methods for a
canopy density mapping system. The system consists of
a 2D LiDAR, RGBD cameras and GPS module. Canopy

346
IFAC AGRICONTROL 2019
December 4-6, 2019. Sydney, Australia Mengying Hu et al. / IFAC PapersOnLine 52-30 (2019) 342–348 347
Fig. 10. Canopy density map for 3D tree row and the difference map. The last image is the average difference map for
5 rows above.
Fig. 11. Canopy density map for 3D tree row from both sides and the difference map. The last image is the difference
map for both sides of the same row.
Fig. 12. Canopy density map for 2D tree row from both
sides and the difference map. The last image is the
difference map for both sides of the same row.
density maps were obtained from side view as a matrix,
which can be visualized as a Tiff image. The advantage
for the format is the small storage and accessibility. The
ground truth for evaluation comes from the quads placed
in the canopy.
The GPS accuracy has been assessed firstly by the quad
positions to show the precision within 80 mm. Then
the consistency and robustness of quad density has been
shown. Manual and automatic measurements of quad den-
sity for 3D tree row has shown good correlation. The
consistency of the whole row during multiple passes have
a correlation R2 = 0.97. In conclusion, the system is val-
idated with robustness and repeatability. The evaluation
methods also have high-adaptability to be utilized in other
similar circumstances. Moreover, the paper studies the
canopy density maps from both sides of the same row.
The maps show strong correlation of R2 = 0.89, which
means the trees are symmetric to some degree and can be
characterized from only one side. This gives a new insight
to simplify the problem of misalignment of point cloud
from both sides of tree rows.
Overall, this work seeks to evaluate a canopy mapping
system for robustness and repeatability. The evaluation
methods have the potential to be applied in other agri-
cultural circumstances, such as irrigation, pruning, flower
and fruitlet thinning. Future work will try to study canopy
elements - flowers, leaves and branches - on individual trees
to gain an insight into how apple trees grow throughout
the whole year, and how growth varies in an orchard. Then,
with these understandings, a better orchard management
system for farmers can be built.
347
ACKNOWLEDGEMENTS
The authors would like to thank Tom Wyatt, William
McCarthy, Angus Ross, Angus Hogan and other people
from SwarmFarm for building the hardware and collect-
ing ground truth data. Thank you to Julie Tang, Hi-
ranya Jayakody, Annie Xu Wang for helping build the
Google Cloud Platform and Google Compute Engine.
Many thanks to the financial support of Horticulture
Innovation Australia Limited through project AP16005,
“Developing Agri-Tech Solutions for the Australian Apple
Industry”. Mengying Hu is also funded by China Scholar-
ship Council (CSC), and partially funded by UNSW for
her PhD study.
REFERENCES
Auat Cheein, F., Steiner, G., Perez Paina, G., and Carelli,
R. (2011). Optimized EIF-SLAM algorithm for precision
agriculture mapping based on stems detection. Comput-
ers and Electronics in Agriculture, 78(2), 195–207.
Ben Grocholsky, Stephen Nuske, Matt Aasted, Supreeth
Achar, and Terry Bates (2011). A Camera and Laser
System for Automatic Vine Balance Assessment. Amer-
ican Society of Agricultural and Biological Engineers.
Colao, A.F., Trevisan, R.G., Molin, J.P., Rosell-Polo, J.R.,
and Escol, A. (2017). A Method to Obtain Orange Crop
Geometry Information Using a Mobile Terrestrial Laser
Scanner and 3d Modeling. Remote Sensing, 9(8), 763.
Dong, W. and Isler, V. (2018). Tree Morphology for
Phenotyping from Semantics-Based Mapping in Or-
chard Environments. arXiv:1804.05905 [cs]. ArXiv:
1804.05905.
Escol, A., Martnez-Casasnovas, J.A., Rufat, J., Arn, J.,
Arbons, A., Seb, F., Pascual, M., Gregorio, E., and
Rosell-Polo, J.R. (2017). Mobile terrestrial laser scanner
applications in precision fruticulture/horticulture and
tools to extract information from canopy point clouds.
Precision Agriculture, 18(1), 111–132.
Hu, K., Fu, Z., Ji, R., Wang, J., and Qi, L. (2012). Design
and Development of Variable Rate Spraying System
Based on Canopy Volume Measurement. In D. Li and
Y. Chen (eds.), Computer and Computing Technologies
IFAC AGRICONTROL 2019
348
December 4-6, 2019. Sydney, Australia Mengying Hu et al. / IFAC PapersOnLine 52-30 (2019) 342–348

in Agriculture V, IFIP Advances in Information and

Communication Technology, 402–413. Springer Berlin
Heidelberg.
Nielsen, M., Slaughter, D., Gliever, C., and Upadhyaya,
S. (2012). Orchard and tree mapping and description
using stereo vision and lidar.
Shalal, N., Low, T., McCarthy, C., and Hancock, N.
(2013). A preliminary evaluation of vision and laser
sensing for tree trunk detection and orchard mapping.
In J. Katupitiya, J. Guivant, and R. Eaton (eds.), Pro-
ceedings of the Australasian Conference on Robotics and
Automation (ACRA 2013), 1–10. Australasian Robotics
and Automation Association, Sydney, Australia.
Shalal, N., Low, T., McCarthy, C., and Hancock, N.
(2015). Orchard mapping and mobile robot localisation
using on-board camera and laser scanner data fusion
Part A: Tree detection. Computers and Electronics in
Agriculture, 119, 254–266.
Tumbo, S.D., Salyani, M., Whitney, J.D., Wheaton, T.A.,
and Miller, W.M. (2002). Investigation of laser and
ultrasonic ranging sensors for measurements of citrus
canopy volume.
Underwood, J.P., Hung, C., Whelan, B., and Sukkarieh,
S. (2016). Mapping almond orchard canopy volume,
flowers, fruit and yield using lidar and vision sensors.
Computers and Electronics in Agriculture, 130, 83–96.

348