5th IFAC Conference on Sensing, Control and Automation for

5th IFAC Conference on Sensing, Control and Automation for

Agriculture
5th IFAC
5th IFAC Conference
Conference onon Sensing,
Sensing, Control
Control and
and Automation
Automation for
for
Agriculture
5th IFAC14-17,
Conference on Sensing, Available
Control online at www.sciencedirect.com
and Automation for
August
Agriculture
Agriculture 2016. Seattle, Washington, USA
August 14-17, 2016. Seattle, Washington, USA
Agriculture
August
August 14-17, 2016. Seattle, Washington, USA
August 14-17,
14-17, 2016.
2016. Seattle,
Seattle, Washington,
Washington, USA
USA
ScienceDirect
IFAC-PapersOnLine 49-16 (2016) 161–165
Dual-arm
Dual-arm Robot
Robot Design
Design and
and Testing
Testing for
for Harvesting
Harvesting Tomato
Tomato in
in Greenhouse
Greenhouse
Dual-arm
Dual-arm Robot Design and Testing for Harvesting Tomato in Greenhouse
Robot Design and Testing for Harvesting Tomato in Greenhouse
Yuanshen,
Yuanshen, Zhao*.
Zhao*. Liang
Liang Gong**.
Gong**. Chengliang
Chengliang Liu***.
Liu***. Yixiang
Yixiang Huang****
Huang****
Yuanshen,
Yuanshen, Zhao*. Liang Gong**. 
Chengliang Liu***. Yixiang Huang****
Huang****
Yuanshen, Zhao*. Liang Gong**. Chengliang Liu***. Yixiang
Zhao*. Liang Gong**. Chengliang
 Liu***. Yixiang Huang****


*School
*School of
of Mechanical
Mechanical Engineering,
Engineering, Shanghai
Shanghai Jiao
Jiao Tong
Tong University,
University, Shanghai,
Shanghai, CO
CO 200240
200240
*School
*School of
of Mechanical
China
Mechanical
*School of Mechanical Engineering,
(Tel:
Engineering,
Engineering, Shanghai
021-34206075;
Shanghai
Shanghai Jiao
e-mail:
Jiao Tong
Tong University, Shanghai,
zhaoyuanshen@126.com).
Tong University, Shanghai, CO
University, Shanghai,
Jiaozhaoyuanshen@126.com). CO 200240
CO 200240
200240
China (Tel: 021-34206075; e-mail:
** School of China (Tel:
Mechanical 021-34206075;
Engineering, e-mail:
Shanghai zhaoyuanshen@126.com).
Jiao Tong
China (Tel:
China (Tel:
** School of Mechanical 021-34206075;
021-34206075;
Engineering, e-mail:
e-mail:
Shanghai Tong University, Shanghai, CO 200240
University, Shanghai,
zhaoyuanshen@126.com).
zhaoyuanshen@126.com).
Jiao CO 200240
** School
** School of
of Mechanical
Mechanical Engineering,
China Shanghai Jiao
Jiao Tong
Tong University,
University, Shanghai,
Shanghai, CO
CO 200240
** School of Mechanical China (e-mail:
Engineering,
Engineering, gongliang_mi@sjtu.edu.cn)
(e-mail:Shanghai
Shanghai Jiao Tong University,
gongliang_mi@sjtu.edu.cn) Shanghai, CO 200240
200240
*** China (e-mail: gongliang_mi@sjtu.edu.cn)
*** School
School of
of Mechanical Engineering,
Mechanical China
China (e-mail:Shanghai
(e-mail:
Engineering, Jiao
Jiao Tong
Tong University,
University, Shanghai,
gongliang_mi@sjtu.edu.cn)
gongliang_mi@sjtu.edu.cn)
Shanghai Shanghai, CO
CO 200240
200240
***
*** School
School of
of Mechanical
Mechanical Engineering,
China, Shanghai Jiao
Jiao Tong
Tong University,
University, Shanghai,
Shanghai, CO
CO 200240
*** School of Mechanical China, (e-mail:
Engineering,
Engineering, chlliu@sjtu.edu.cn)
Shanghai
Shanghai
(e-mail: Jiao Tong University,
chlliu@sjtu.edu.cn) Shanghai, CO 200240
200240
****School China, (e-mail: chlliu@sjtu.edu.cn)
****School of
of Mechanical
Mechanical Engineering,
China, Shanghai
China, (e-mail:
(e-mail:
Engineering, Jiao
Jiao Tong
Tong University,
University, Shanghai,
chlliu@sjtu.edu.cn)
chlliu@sjtu.edu.cn)
Shanghai Shanghai, CO
CO 200240
200240
****School
****School of
****School of Mechanical Engineering,
MechanicalChina
of Mechanical (e-mail:
Engineering, Shanghai Jiao Tong University,
huang.yixiang@sjtu.edu.cn) Shanghai, CO 200240
(e-mail:Shanghai
Engineering,
China Shanghai Jiao
Jiao Tong
Tong University,
University, Shanghai,
huang.yixiang@sjtu.edu.cn) Shanghai, CO
CO 200240
200240
China
China (e-mail:
(e-mail: huang.yixiang@sjtu.edu.cn)
huang.yixiang@sjtu.edu.cn)
China (e-mail: huang.yixiang@sjtu.edu.cn)

Abstract: In
Abstract: In order
order toto improve
improve the the efficiency
efficiency of of robotic
robotic harvesting
harvesting in in unstructured
unstructured environment,
environment, aa
Abstract:
modular
Abstract: In
concept
In order
of
order
In order to
to improve
dual-arm
improve
to improve the
robot
the
thefor efficiency
for harvesting
efficiency
efficiency of
of robotic
tomatoes
robotic
of tomatoes harvesting
is proposed
harvesting
robotic harvesting in
inin unstructured
this paper.
unstructured environment,
The objective to
environment, toaaa
modular concept
Abstract: of dual-arm robot harvesting is proposedininunstructured environment,
this paper. The objective
modular
develop
modular
modular concept
a modular
concept of
of dual-arm
robotic
dual-arm robot
system
robot for
which
for harvesting
works
harvesting tomatoes
based on
tomatoes is proposed
human-robot
is proposed in this paper.
collaboration
in this is
paper. The objective
reached.
The Due
objective to
to
develop aconceptmodularofrobotic
dual-arm robotwhich
system for harvesting
works basedtomatoes is proposedcollaboration
on human-robot in this paper.is The objective
reached. Due to
develop
develop
develop aaa modular
the complexity
complexity modular
modular robotic
of working
working
robotic system
system which
environment,
system
robotic environment, which
whichan anworks
works
works based
artificial
based
based on
on human-robot
recognition
on approachcollaboration
human-robot
human-robot conducted by
collaboration
collaboration is
by
is reached.
operator
reached.
is operator Due
Due to
through
Due
reached.through to
to
the of artificial recognition approach conducted
the
the
the complexity
marking the
complexity
complexity of
tomato
of working
object
working
of working environment,
on the graphic
environment,
environment, an
an artificial
user
artificial recognition
interface is used
recognition approach
for tomato
approach conducted
recognition
conducted by
and
by operator through
localization.
operator throughA
marking the tomato object on the graphican artificial
user recognition
interface is used for approach conducted by
tomato recognition andoperator through
localization. A
marking
dual-arm
marking the
frametomato object
equipped on
with the
twographic
3 DoF user interface
manipulators is
andused
two for tomato
different recognition
type and
end-effectorslocalization.
used to A
pick
dual-armthe
marking the
frametomato
tomato object on
on the
objectwith
equipped twographic
the 3 DoFuser
graphic user interface
interface is
manipulators used
isand twofor
used tomato
fordifferent recognition
tomato type
recognition and
and localization.
end-effectorslocalization.
used to pickA
A
dual-arm
tomatoes frame
dual-arm
dual-arm frame
frame equipped
is designed
designed
equipped
equippedand with
tested
with
with two
two 333 DoF
DoF manipulators
respectively.
DoF
tworespectively.manipulators
manipulators and
and two
The cooperation
cooperation
and two
two ofdifferent
of
different
different type
type end-effectors
two end-effectors
end-effectors
type end-effectors
end-effectors used
used to
could improve
used to pick
improve
to the
pick
pick
tomatoes is and tested The two could the
tomatoes
harvestingis
tomatoes
tomatoes is designed
efficiency
designed
designed and
isefficiency and
and tested
significantly.
tested respectively.
The EtherCAT
tested respectively.
respectively. The
EtherCAT
The cooperation
bus based
The cooperationbased of
cooperation of two
control
two end-effectors could
and communication
communication
two end-effectors
ofcontrol end-effectors could improve the
systemthe
could improve
improve is
the
harvesting significantly. The bus and system is
harvesting
adopted
harvesting
harvesting to efficiency
increases
efficiency significantly.
the reliability
significantly. The
and
The EtherCAT
speed of
EtherCAT bus
motion
bus based
control
based control
and
control and
data
and communication
communication.
communication system
Concerning
system is
is
adopted to efficiency
increases the significantly.
reliability and The speed
EtherCAT bus based
of motion controlcontrol
and data andcommunication.
communicationConcerning
system is
adopted to
control software,
software,increases the reliability and speed of motion control and data communication. Concerning
adopted
adopted
control to
to increasesaa graphic
increases graphic
the user interface
interface
the reliability
reliability
user and was designed
and speed
speed
was designed
of
of motion
motionto exchange
to exchange
control
control and the data
and
the operator’s
data communication.
communication.
operator’s commands and display
display
commands Concerning
Concerning
and
control
the state
control
control software,
information
software, a
a graphic
of
graphic user
robot.
user interface
The
interface was
was designed
performances of
designed to
field
to exchange
test
exchange the
showed
the operator’s
the commands
efficiency
operator’s of
commands theand display
developed
and display
the statesoftware,
informationa graphic user interface
of robot. was designed
The performances to exchange
of field test showedthe operator’s
the efficiency of theand
commands display
developed
the
the
the state
robot system,
state
state information
information
information of
of robot.
of robot. The
some shortcomings
shortcomings
robot. of the
The
The performances
the were of
robot were
performances
performances alsofield
of
of found
field
field test
forshowed
test showed
testfor
showed the
the efficiency
future work.
work.
the efficiency of
efficiency of the
of the developed
the developed
developed
robot system, some of robot also found future
robot
robot system,
system, some
some shortcomings
shortcomings of
of the
of the robot
robot were
were also
also found
found for
for future
future work.
work.
© 2016,
robot IFAC
system,
Keywords: (International
some Federation
shortcomings theofrobot
Automatic
were Control)
also found Hosting
forField by tests
future Elsevier
work. Ltd. All rights reserved.
Keywords: Dual-arm robot Tomato harvesting Stereo camera Field tests Human-robot
Dual-arm robot Tomato harvesting Stereo camera Human-robot collaboration
collaboration
Keywords: Dual-arm
Keywords: Dual-arm
Keywords: Dual-arm robotrobot Tomato
robot Tomato harvesting
Tomato harvesting Stereo
harvesting Stereo camera
Stereo camera
camera Field Field tests
Field tests Human-robot
tests Human-robot collaboration
Human-robot collaboration
collaboration


 robot
1.  robot prototype
prototype was was designed
designed and and built
built for for aa novel
novel artificial
artificial
1. INTRODUCTION
INTRODUCTION robot prototype
harvesting strategywas in designed
unstructuredand built for
environments, a novel artificial
involving
1.
1. INTRODUCTION
INTRODUCTION
robot prototype
robot prototype
harvesting strategywas designed
wasindesigned
unstructuredand built
built for a novel
for
and environments, a novel artificialaa
artificial
involving
1. INTRODUCTION harvesting
human-machine
harvesting strategy
strategy taskin
in unstructured
distribution
unstructured environments,
(Ceres
environments,et al., involving
1998).
involving The a
As
As oneone of of thethe most
most important
important processes
processes in in agriculture
agriculture harvesting
human-machine strategy taskin unstructured environments,
distribution (Ceres involving
et al., 1998). Theaa
human-machine
operator performed task
task the distribution
detection (Ceres
and et al., 1998). The
As
As
As one
production,
one of
of
one of the
production,
the
the
the most
harvesting
most
theharvesting important
work
important
most important is processes
not only
processes
not only a in
work is processes a in agriculture
intiresome task
agriculture
agriculture
tiresome task
human-machine
human-machine
operator performed task distribution
distribution
the and precise
detection(Ceres
(Ceres et
preciseal., location
et al., 1998).
1998). The
location of
The
of
operator
fruits by performed
means the detection and precise location of
production,
but also
production,
production, the
requiring
the harvesting
a large
harvesting
the harvesting
but also requiring a large amount
work
amount
work
work of is
of
is not
skilled
not
is skilled
not onlyonly
only a
labour
a tiresome
in
tiresome
a tiresome
labour a task
certain
task
task
in a certain
operator
operator
fruits means of
performed
by performed of theaa laser
the range
detection
detection
laser range and finder.
and
finder. Ji
Ji et
precise
precise al.
al. (2014)
location
et location
(2014)of
of
but
period
but also
also requiring
Moreover,
requiring a
a large
the
large amount
tomato
amount of
trees
of skilled
in
skilled labour
greenhouse
labour in
in a
a certain
are tall
certain fruits
fruits by
introduced
fruits by means
by means an
means of
of a laser
assistant-mark
of aa laser range
laser rangeapproach
range finder.
finder. to
finder.toJi Ji et
Jirecognizeal.
recognize
et
et al. (2014)
and
al. (2014)
(2014)
but alsoMoreover,
period requiring a the largetomato
amounttrees of skilled labour in aare
in greenhouse certain
tall introduced an assistant-mark approach and
introduced
locate the an assistant-mark approach to recognize and
period
which
period Moreover,
increases
Moreover,
period increases
which Moreover, the the
risk
the
thethe
tomato
of injury
tomato
risktomato
of injury
trees
for
trees
trees in
in greenhouse
farmers. Projecting
greenhouse
in greenhouse
for farmers. Projecting
are
are tall
into
tall
are into
tall the picking-point
introduced
introduced
locate an
an assistant-mark
assistant-mark
picking-point for
for robotic
approach
approach
robotic harvesting.
to
to recognize
harvesting. Oren
Oren et
recognize al.
etand
and
al.
which
the
which increases
future, the
increases the
issue
the risk
of
risk of
labour
of injury
injury for
shortage
for farmers.
is expected
farmers. Projecting
to
Projecting into
become
into locate
(2012)
locate
locate the
also
the picking-point
defined
picking-point and for
for robotic
implemented
robotic harvesting.
a Oren
collaboration
harvesting. Oren et
of
Oren etofal.
et al.
al.aa
which
the increases
future, the risk
the issue of injury
of labour for farmers.
shortage is expectedProjecting
to become into (2012) the
alsopicking-point
defined and for robotic harvesting.
implemented a collaboration
the future, the (2012)
human
(2012) also
operator
also defined
and
defined and
robot
and implemented
applied
implemented to a collaboration
target
a fruit
collaboration of
of aaa
detection.
of
aathe
the huge
future,
future,
huge the issue
challenge
the
challenge to
issue
issue of
of labour
of labour shortage
to agricultural
labour
agricultural industry
shortage
industryis
shortage is expected
is(Gongal
expectedet
expected
(Gongal
to become
toal.,
etto 2015).
become
become
al., 2015). (2012)
human also
human
operator
operator
Experimental
defined
and and
and
results
robotimplemented
robot
applied to target
applied
indicated that to
a collaboration
target
the
fruit detection.
fruit
fruit detection.
recognition
aThehuge challenge to agricultural industry (Gongal et al., human operator and robot applied to target fruit detection.
aThe
a huge
huge harvesting
challenge robot
challengerobot
harvesting to was
to agricultural
agriculturalfirstly
was firstly proposed
industry
industry
proposed (Gongal
(Gongal by et
by Schertz al., 2015).
Schertz
et al., and
2015).
2015).
and human operatorresults
Experimental
Experimental
and robot
results
applied
indicated
indicated
thatto the
that
targetfruitfruitrecognition
detection.
The
Brown
The
The
Brown
harvesting
(1968)
harvesting asrobot
(1968) asrobot
harvesting a
robot was
potential
was firstly
way
was firstly
a potential firstly
way to
proposed
to combat
proposed
combat by
proposed by
the
by Schertz
agricultural
the Schertz
Schertz
agricultural
and
and
and
system
system based
Experimental
Experimental on
on human-robot
based results
results indicated
human-robot that the
indicated collaboration
that the fruit
the
collaboration
recognition
fruitincreased
fruit recognition
recognition
increased the
the
Brown
labour (1968)
crisis. as a
Harvestingpotential way
robots to
are combat
designed the toagricultural
sense system
success
system based
detection
based on
on human-robot
rate
human-robotto collaboration
94% and
collaboration reduce increased
the
increased the
time
the
Brown (1968)
Brown crisis.
labour as a
(1968) Harvesting potential
as a potential way
way are
robots to combat
to combat
designed the sense the
agricultural
thetoagricultural the system based
success
success
consumption
detection
detection
by
on human-robot
20%.
rate to 94%
rate to
collaboration
94%
and reduce
and reduce
increased
the time
the
the
time
labour
complex crisis. Harvesting
agricultural robots
environment are
using designed
various to sense the success
success detection by 20%.rate rate to to 94%94% and and reduce
reduce the the time
labour
labour crisis.
complex crisis. Harvesting
Harvesting
agricultural robots
robots are
environment are
using designed
variousto
designed tosensors
sense and
sense
sensors the
the
and consumption
consumption
detection
by
time
use
use
complex
that
complex
complex agricultural
information,
agricultural environment
together
environment
agricultural environment
that information, with
together withusing
using
a
using
a goal,
various
goal, to
various
various
to perform
sensors
perform
sensors and
some
sensorssome and
and
consumption
consumption
Another by 20%.
by 20%.
20%. goal of efficient robotic harvesting is
use
actions that information, together with aaa goal,
goal, to perform some Another way
way toward
toward the the goal of efficient robotic harvesting is
use
use that
actions that(Edan
(Edan et
et al.,
al., 1994).
information,
information, together
together
1994). These
These with
withactions
actions are toto
areto
goal, move
move the
to perform
perform tool
thesome
some
tool Another
to buildway toward
aa toward
multi-arm the goal of
robotic efficient
harvester. robotic A harvesting
number is
actions (Edan et al., 1994). These actions are to move the Another
to buildway
Another way toward
multi-arm the
the goal
goal of
of efficient
robotic efficient
harvester. robotic
robotic number of
A harvesting
harvesting is
is
of
of
actions
actions
of an end-effector
(Edan
an end-effector et al., to grasp
1994).
(Edan et al.,to1994). or
These
graspThese pick
or pick actions
actionsa fruit
are
a fruit object.
to
areobject.move
to moveAlthoughthe tool
Although
the tool
tool to build
manipulators
to build a
a multi-arm
are mounted
multi-arm robotic
on
robotic a harvester.
mobile
harvester. robot A number
platform, of
and
of
of an end-effector
harvesting
an robots
end-effector to
hold
to grasp
ample
grasp or pick
promise
or pick a
a fruit
for the
fruit object.
future,
object. Although
currently
Although to build a multi-arm
manipulators are mountedrobotic harvester.
on a mobile robotAplatform,
number and of
of an end-effector
harvesting robots hold to grasp
ampleor pick afor
promise fruit
theobject.
future, Although
currently manipulators
each manipulator are mounted on aa mobile robot platform, and
harvesting robots hold manipulators
each manipulator are isis assigned
assignedonspecific
mounted mobile
specific fruit to
to harvest.
fruitrobot platform,
harvest. Zion
Zionand et
et
the overall
harvesting
overallrobots
harvesting
the robots hold ample
performance
hold
performance ampleof
ample promise
ofpromise
promise for
for the
harvesting
for
harvesting the future,
future, currently
the robot
future,
robot is
is often
currently
currently
often each
al., manipulator
(2014) from isIsrael
is assigned
assignedhas specific
designed
specific fruit
a
fruit to harvest.
harvest. Zion
multi-arm
to Zion
melons et
et
the overall
insufficient to performance
compete with of harvesting
manual operation robot (Griftis often each(2014)
al., manipulator
from isIsrael
assignedhas specific
designedfruit to harvest. Zion
a multi-arm melons et
the
the overall
overall toperformance
insufficient compete withof
performance harvesting
ofmanual
harvestingoperation robot
robot(Grift is et
is al.
often
often
et al. al., (2014)
al., (2014) robot
harvesting from which
from Israel has
Israel has
enableddesigned
the
designed a multi-arm
multi-arm
maximum
a melons
number
melons of
insufficient
2008).
insufficientThe to
to compete
bottlenecks
compete with
with to manual
promote
manual operation
the
operation (Grift
commercial
(Grift et
et al. al., (2014) robot
harvesting from which
Israel has enableddesigned a multi-arm
the maximum melons
number of
insufficient
2008). Theto bottlenecks
compete withtomanual promote operation
the (Griftcommercialet al.
al. harvesting
melons to robot which According
enabled the the maximum number of of
2008).
application
2008). The
The of bottlenecks
harvesting
bottlenecks to
robots
to promote
are
promote the the
low
the commercial
efficiency
commercial and to be
harvesting
harvesting
melons be harvested.
robot
robot which
which
harvested. enabled
enabled
According to
to the
the the idea
idea of
maximum
maximum of multi-robot
number
number
multi-robot of
2008).
application Theof harvesting
bottlenecksrobots to promote
are the low theefficiency
commercial and melons to
cooperation
melons toto bebe harvested.
for
befor fruit
harvested. According
harvesting, to
Noguchi
AccordingNoguchi
to thethe idea
et
the idea
idea of
al. multi-robot
(2004)
of (2004)
multi-robotalso
application
high of harvesting robots are the low efficiency and melons
cooperation harvested. According
fruit harvesting, to et al. of multi-robotalso
high cost
cost (Edan
application
application of
(Edan et
et al.,
al., 1999).
of harvesting
harvesting 1999). robots
robots are are the the lowlow efficiency
efficiency and and cooperation
proposed for fruit
fruit harvesting,
aa master-slave harvesting,
robot Noguchi et al.
al. (2004)
(2004) alsoalso
high
high cost
cost (Edan
(Edan et
et al.,
al., 1999).
1999).
cooperation
cooperation
proposed for
for
master-slave robot system
fruit harvesting, Noguchi
Noguchi
system for
for field
et
et
field al.operations.
(2004) also
operations. In
In
high cost
Combining (Edan et al., 1999). proposed
this
proposed a master-slave
multiple
a master-slave
robot system,robot
robota system
high
systemlevel for of
for field operations.
autonomy
field operations. to In
the
In
Combining human human workersworkers and and robots
robots synergistically
synergistically is is aa proposed
this
this
robot
a master-slave
multiple
multiple
was
robot system,
robot system,
achieved system,
to
robot
allow
system
a high
a them
level
high level
level
to
for of
field operations.
autonomy
of autonomy
cope autonomy
with
In
to the
to the
unexpected the
Combining human workers and robots synergistically is this multiple robot a high of to
viable
Combining
Combining
viable approach
human to
humantoworkers
approach increase
workers
increaseandandthe success
robots
therobots
success rate of
synergistically
synergistically
rate of robotic is aaa
robotic
is this
robotmultiple
robot was
events was
and
robot system,
was achieved
achieved
obstacles.
to allowa them
to allow
allow
Modular
high level
to cope
themdesign
to cope
of autonomy
copeis
with unexpected
with
to the
unexpected
aa practical and
viable approach
harvesting. This to increase the success rate of robotic robot achieved to them to with unexpected
viable
viable This approach
approach
approach
harvesting. to
to increase
increase
approach for
for thethe
the robotic
robotic harvesting
the success
success rate
rate of
harvesting is
is to
of robotic
robotic
to robot
events was
andachieved
events and
feasible and
way
obstacles.
obstacles.
to reduce
to allow
Modular
Modular
the
them to
design
design of
investment
copeis with unexpected
practical
is harvesting
a practical
practicalrobot
and
and
harvesting.
separate the This
fruit approach
approach for the robotic harvesting is to events obstacles. Modular design is a and
harvesting.
harvesting.
separate fruit recognition
This
the This approach
recognition for
forstage
the from
the
stage robotic
robotic
from the harvest
harvest stage
harvesting
the harvesting is by
is
stage to
to
by events and
feasible wayobstacles.
feasible and
(Hwang wayKim,
to reduce Modular
to reduce
reduce
2003).
design of
the investment
the investment
investment of
is harvesting
a practicalrobot
of harvesting
harvesting robot
and
robot
separate
marking the
the fruit
targetrecognition
fruit a stage
priori. from
In the the harvest
Agribot stage
project, by feasible way to the
separate
separate the
marking the fruit
the fruit
targetrecognition
recognition stage
stage In
fruit a priori. from
from thethe
the harvest
harvestproject,
Agribot stage
stage bybyaa feasible and
(Hwang
(Hwang and
wayKim,
and Kim,
to reduce
2003). the investment of harvesting robot
Kim, 2003).
2003).
marking the target fruit (Hwang
marking
marking the the target fruit aaa priori.
target fruit priori.
priori. In In
In thethe Agribot
the Agribot project,
project, aaa
Agribot project, (Hwang and Kim, 2003).
2405-8963 ©
Copyright © 2016, IFAC (International Federation of Automatic Control)
2016 IFAC 161Hosting by Elsevier Ltd. All rights reserved.
Copyright
Peer review© 2016 responsibility
under IFAC 161Control.
of International Federation of Automatic
Copyright
Copyright © 2016
© 2016 IFAC
2016 IFAC
IFAC 161
161
Copyright ©
10.1016/j.ifacol.2016.10.030 161
IFAC AGRICONTROL 2016
162 Yuanshen Zhao et al. / IFAC-PapersOnLine 49-16 (2016) 161–165
August 14-17, 2016. Seattle, Washington, USA

In this paper, a dual-arm robot for harvesting tomato in
greenhouse scene was modular designed and field testing.
Firstly, the overview and workflow of the developed dual-
arm harvesting robot were described in the paper. Then, 5
major functional modules of the harvesting robot were
introduced, and some shortcomings of the developed system
were summarized according to the field tests. Finally, the
remainder of this article present a summary about the study
of dual-arm harvesting robot and outlook of this study.
2. OVERVIEW OF THE ROBOT
mentioned above, a human-robot collaboration strategy was
introduced into the framework of proposed system. The fruit
recognition task was implemented by operator through
pointing out the target tomato on the screen interface which
displayed the working site. Then, the spatial position of the
marked tomato was obtained by stereo camera. Excepting
fruit recognition, the order of carrier vehicle driving was also
given by operator through user interface. The other
operations were conducted by robot itself. All the task
sequence of picking operation for the dual-arm robot was
shown in Fig. 2.

2.1 The Structure of Prototype Robot Calculate the 3D

Initialise system position of the tomato
object
The proposed robot system was designed to fit Venlo type
greenhouse, in which the heating pipes was used as guided Move vehicle to next
position on the rail Inverse kinematics of
rail. The platform vehicle designed as a carrier was driven on each manipulator
the heating pipes. Except these auxiliary units, the dual-arm
harvesting robot was composed of 5 major functional Acquire and transmit Manipulator move to
modules which are dual-arm type robotic manipulator, the images of working the picking point
exchangeable modular type end-effectors, vision system field
equipped with stereoscope camera, EtherCAT based
communication and control system, and graphic user Cut the stem and grasp
Detect the ripe tomato the tomato object by
interface. As shown in Fig. 1, a stereo camera mounted on the by operator through two type end-effectors
top of the robot platform was used to acquire the image of graphic user interface
working scenes and provide 3D position of tomato object.
Two-arm manipulators
The robotic dual-arm constituted by two 3 DoF manipulators moving to the original
was used to position the end-effector to the picking-point. A N location
Ripe tomato
saw cutting device was designed as end-effector which could
existing?
cut the stem of tomato. The other end-effector was used to
grasp the tomato for avoiding tomato shaking caused by the Y N
cutting operation. The robot system adopted tele-operative Vehicle at
Mark the tomato object end of rail?
concept through developing wireless remote data and signal
on the 3D image
communication and graphic user interface. The machine displayed unit Y
control system of each manipulator was built based on direct
Harvest end
current (DC) servo motors and EtherCAT bus.

Stereo camera
Dual-arm
Fig. 2. The workflow of tomato harvesting robot.

Heating pipes 3. THE MODULAR ROBOT SYSTEM

End-effector
User interface
Carrier vehicle
Fig. 1. Integrated dual-arm robotic system for harvesting
tomato.
2.2 The Workflow of Tomato Harvesting Robot
Considering unreliability of autonomous fruit detection for
various fruit harvesting robots in unstructured environment,
the fruit recognition is the most difficult challenge to improve
the effectively of robotic harvester. For solving the obstacles
3.1 Dual-arm Manipulator
Fig. 3(a) shows the developed dual-arm robotic manipulator
which was composed of two 3 degrees of freedom (DoF)
Cartesian type robot manipulators. The 3 DoF Cartesian type
robot manipulators were owned one prismatic joint and two
rotational joints to ensure the proper workspace. Fig. 3 (b)
shows the simulation results of reachable workspaces of the
dual-arm robot manipulators. Totally 6000 red and blue dots
were used to display the possible picking-point of each arm
respectively. The common area of red and blue dots is the
workspaces. According to the width of the simulation results,
the distance between each picking point on heating pipes was
designed as 400mm. The frame of dual-arm manipulators
also can rotate 180 degrees according to the central axis.
Overall geometrical and kinematic parameters of the
proposed dual-arm robot manipulators were given in Table 1.

162
 IFAC AGRICONTROL 2016
August 14-17, 2016. Seattle, Washington, USA
Yuanshen Zhao et al. / IFAC-PapersOnLine 49-16 (2016) 161–165 163

Table 1. Overall parameters of dual-arm robot manipulators to 45 degree with respect to the Y direction. The Y direction
adjusting was also driven by an actuator controlled by the
Parameter Design value
MCU system.
Length Forearm, mm 250
Length Upper arm, mm 330 The pneumatic type end-effector which is composed of
Distance between two arms, mm 400 vacuum suction device and plastic socket is showed in Fig. 4
Range of vertical slide, mm [0 300] (b). The plastic socket moved to the centre point of tomato
θ1 , deg [-45° 45°] object firstly, and then the vacuum suction cup gripped it to
θ2 , deg [-45° 45°] prevent shaking. The pneumatic type end-effector could
approach 20mm to 50mm offset point along the stem axis of
Vertical slide
ripe tomato from the centre. The diameter of cup was 100mm
θ1 and vacuum power was set up to handle 100gf to 1kgf weight.
Stem cutting was performed repeatedly until pneumatic type
Forearm
Upper arm
gripper gripped the target tomato.
Z Y
X

θ2

Central axis Platform

(a) Saw cutting type end-effector (b) Pneumatic type gripper

(a) The 3D model of the dual-arm robot manipulator Fig. 4. Two kinds of the end-effectors for the dual-arm robot.

3.3 Vision System

500

yz x A colour stereo camera (Bumblebee2, Point Grey, Vancouver,

yz Canada) with a resolution of 480(H) ×640(L) at 48FPS was
Z / mm

x
0
2

1
arm1
2 used as the visual system of robot. The stereo vision system
arm2
-500
1 consisted of two identical colour CCD sensors was attached
500 to the top of dual-arm frame to ensure the proper view field,
500
0
200
300
400 and the image process was performed by the host computer
100
-500 -100
0 (Lenovo, Beijing, China, Inter(R) Core(TM) i3-370 CPU,
Y / mm X / mm
Random Access Memory (RAM) 4.0GB). The task of ripe
(b) The simulation results of workspace for the dual-arm tomato detection and localization was completed at two
robot manipulators different stages. Firstly, ripe tomato was detected and marked
by operator in the 3D reconstructive image through graphic
Fig. 3. The structure of dual-arm manipulator: (a) 3D model user interface. Then, the three dimensions position P(x, y, z)
of manipulator, (b) the possible workspace of the dual-arm of tomato object was calculated through the principle of
robot manipulators. triangulation (Sun et al., 2011) using the two images captured
by the stereo camera. Since adopting human-robot
3.2 End-effectors collaboration approach to detect and locate ripe tomato, the
robustness and efficiency of robotic harvesting increased
End-effectors were developed as modular type in order to obviously.
ensure two robotic arms working cooperatively. The end-
effector was designed to be lightweight and simple
exchangeable. As shown in Fig. 4, two different kinds of end- P(x, y, z)
effectors such as cutting device for fruit separation and
vacuum cup for grasping target tomato were installed on the
end of each manipulator respectively. Fig. 4(a) shows the
schematic drawing of saw cutting type end-effector. The
Fig. 5. The principle of 3d position measurement of tomato
actuator of saw cutting type end-effector was driven by belt using stereo camera.
pulley, and the other side of the pulley was connected to the
motor’s output shaft which was controlled by the micro
control unit. The pose of cutter was also adjusted in two 3.4 Motion Control and Communication System
directions: X and Y. The X direction adjusting was conducted
by four-bar linkage, and adjusting range was from -45 degree

163
IFAC AGRICONTROL 2016
164 Yuanshen Zhao et al. / IFAC-PapersOnLine 49-16 (2016) 161–165
August 14-17, 2016. Seattle, Washington, USA
The purpose of motion control was to ensure the developed
robot to implement the picking operation accurately and
rapidly. The motion control and communication system
transmits the operator’s command to control unit of the
machinery and also receives the signal state of the task
execution of machine. As shown in Fig. 6, the core unit of the
control and communication system was the host computer
installed with the graphic user interface and other controlling
software modules. The host computer transmitted operator’s
command to the G-MAS through Modbus for distributing the
kinematic parameter of each joint motor. Then, the control
signal was transmitted through EtherCAT bus. The output
section included seven DC servo motors and supporting
digital drivers (G-Solo Whistle). The main characteristic of
EtherCAT bus is parallel transmission, which effectively
increases the reliability and speed of motion control and data
communication.
Modbus
TCP
G-MAS
Computer
EtherCAT
G-Solo Whistle
Digital Driver
DC servo motor
Fig. 6. Schematic diagram of controller and communication
system.
3.5 Graphic User Interface
Fig. 7. The graphic user interface.
The graphic user interface is a visualisation control screen
which is used to transmit the control command from operator
and displayed the current state of robot. The graphic user
interface consisted of four units: 3D reconstructed image
display unit, robot motion simulating unit, basic command
interface unit, and control command interface unit. As shown
in Fig. 7, the top left of screen is the 3D reconstructed image
164
displayed the robot working scene. The tomato object could
be marked in this screen unit by operator using a small red
circle, and the centre of the circle replaced the position of
tomato object. The top right of the screen shows the position
of the robot in virtual environment which is completely
computer-generated three dimensional graphics for the
simulation of human-computer interaction. The bottom left of
screen is the basic command interface unit which is used to
set system initialisation. The bottom right of screen is the
control command interface unit which is designed to set some
motion parameters of the robot. Visual C++ was selected as
programming development tool for designing the graphic
user interface software. Details about the developed user
interface software were described by Liu et al. (2014).
4. FIELD TESTS AND ANALYSIS
4.1 Filed Tests
To further verify the reliability and adaptability of the dual-
arm harvesting robot, the final integrated system was tested
in the Shanghai Morden Agriculture garden during May 2015
as shown in Fig. 8. In practice, once a tomato object was
marked in the 3D reconstructed image 2by operator, the robot
was able to pick the marked tomato autonomously. If all the
ripe tomatoes were harvested, the robot moved to the next
working position controlled by operator through control
command interface unit. A few shortcomings of the robot
system were revealed through the field tests.
Fig. 8. Dual-arm tomato harvesting robot testing in the
greenhouse
4.2 Shortcomings of Developed Robot System
The 3D image reconstruction is built on the basis of image
matching. However, the complex of the natural environment
caused the failure of image matching in some special
conditions. As shown in Fig. 7, the black area in display unit
of user interface was the result of matching failure. The
varying illumination is the main factor to influence the
performance of image match. On the other hand, the time
consumption of 3D image reconstructing is high which
directly affected to the picking cycle time.
 IFAC AGRICONTROL 2016
August 14-17, 2016. Seattle, Washington, USA
Yuanshen Zhao et al. / IFAC-PapersOnLine 49-16 (2016) 161–165 165
The control strategy applied in the developed harvesting
robot which is a visual open-loop control mode based on
detecting the accurate position of the fruit object in 3D
workspace was developed. The stereo camera can measure
the space distance between target fruit and the end effector
through triangulation theory. Following precision distance
measurement, the movement parameter of the manipulators
can be planned through calculating the kinematics equation.
So, the precisions of kinematic model and calibration of
vision system have great influence on the control accuracy of
manipulators. Because of robot movement and working site
change, the errors of vision system and manipulators might
cause harvest failure.
5. CONCLUSIONS AND OUTLOOK
This paper reports on a modular concept of a dual-arm robot
for harvesting tomatoes in the greenhouse scene. The
objective to build a system which is able to human-robot
cooperatively harvesting fruits was reached. Due to the
complexity of working environment and limited visibility of
the tomato objects, an artificial recognition approach which
the tomato object was marked in the human interface by
operator was used for tomato recognition and localization.
One of the major parts of the robot system is the dual-arm
manipulators with 3 DoF respectively. Two types of end-
effectors to detach tomato have been developed and tested.
The cooperation of two different type end-effectors could
significantly improve the harvesting efficiency. The
EtherCAT bus based communication system could
effectively enhance the reliability of communication system
and transmission speed. Concerning software, the human
interface was designed to exchange the control orders and
state information of robot.
According to the performance of field tests, future work is to
analysis the cause of the failure and to improve the
performance of the robot in terms of harvest success rate,
cycle time and causes of failures. The dual-arm robot
collision free motion planning and mission control are also
need to research in the future.
ACKNOWLEDGEMENTS
This work was mainly supported by a grant from the National
High-tech R&D Program of China (863 Program
No.2013AA102307), and partially supported by the National
Science and Technology Support Project of China
(No.2014BAD08B01) and Shanghai Science Project
(No.14111104603). The authors would like to thank all the
other partners of the projects.
REFERENCES
Ceres, R., Pons, J.L., Jimenez, A.R., Martin, J.M., Calderon,
L. (1998). Design and implementation of an aided fruit-
harvesting robot. Ind. Robot 25(5), 337-346.
Edan, Y. and Gaines, E. (1994). Systems engineering of
agricultural robot design. IEEE T. Syst. Man. Cy. 24(8),
1259-1265.
165
Edan, Y. (1999). Food and agriculture robotic. Handbook of
Industrial Robotic, Second edition. Chapter 60, pp.
1143-1155
Gongal, A., Amatya, S., Karkee, M., Zhang, Q., and Lewis,
K. (2015). Sensors and systems for fruit detection and
localization: A review. Computers and Electronics in
Agriculture. 116, 8-19.
Grift, T., Zhang, Q., Kondo, N., and Ting, K.C. (2008). A
review of automation and robotics for the bio-industry. J.
Biomech. Eng. 1 (1), 37-54.
Hwang, H. and Kim, S., (2003). Development of multi-
functional tele-operative modular system for greenhouse
watermelon. IEEE/ASME International Conference on
Advanced Intelligent Mechatronics, 2003, 2, pp: 1344-
1349.
Ji, C., Zhang, J., Yuan, T., and Li, W. (2014). Research on
key technology of truss tomato harvesting robot in
greenhouse. AMM. 442, 480-486.
Liu, B.X., Gong, L., Chen, Q.L., Zhao, Y.S., and Liu, C.L.
(2014). A stereo Visual interface for manipulating the
grasping operation of a fruit-harvesting robot.
International Robotics and Applications (7th ICIRA),
8918, pp: 239-244.
Noguchi, N., Will, J., Reid, J., and Zhang Q. (2004).
Development of a master-slave robot system for farm
operations. Comput. Electr. Agr. 44, 1-19.
Oren, Y., Bechar, A., and Edan, Y. (2012). Performance
analysis of a human-robot collaborative target
recognition system. Robotica 30, 813-826.
Schertz, C.E. and Brown, G.K. (1968). Basic considerations
in mechanizing citrus harvest. Transaction of ASAE.
343-346.
Sun, J., Lu, B., and Mao, H.P. (2011). Fruits recognition in
complex background using binocular stereovision. J.
Jiangsu U. Nat. Sci. Ed. 32(4), 423-427.
Zhao, D.A., Lv, J.D., Ji, W., Zhang, Y., and Chen, Y. (2011).
Design and control of an apple harvesting robot.
Biosystems Engineering. 110, 112-122.
Zion, B., Mann, M., Levin, D., Shilo, A., Rubinstein, D., and
Shmulevich, I. (2014). Harvested-order planning for a
multiarm robotic harvester. Comput. Electr. Agric. 103,
75-81.