https://doi.org/10.20965/jrm.2018.

p0180
Yoshida, T., Fukao, T., and Hasegawa, T.

Paper:

Fast Detection of Tomato Peduncle

Using Point Cloud with a Harvesting Robot
Takeshi Yoshida∗ , Takanori Fukao∗ , and Takaomi Hasegawa∗∗
∗ Ritsumeikan University

1-1-1 Noji-higashi, Kusatsu, Shiga 525-8577, Japan

E-mail: {yoshi-da, tfukao}@fc.ritsumei.ac.jp
∗∗ Denso Corporation

1-1 Syowa-cho, Kariya, Aichi 448-8661, Japan

E-mail: takaomi hasegawa@denso.co.jp
[Received September 19, 2017; accepted February 1, 2018]

This paper proposes a fast method for detecting
tomato peduncles by a harvesting robot. The main ob-
jective of this study is to develop automated harvest-
ing with a robot. The harvesting robot is equipped
with an RGB-D camera to detect peduncles, and an
end effector to harvest tomatoes. It is necessary for
robots to detect where to cut a plant for harvesting.
The proposed method detects peduncles using a point
cloud created by the RGB-D camera. Pre-processing
is performed with voxelization in two resolutions to re-
duce the computational time needed to calculate the
positional relationship between voxels. Finally, an en-
ergy function is defined based on three conditions of
a peduncle, and this function is minimized to identify
the cutting point on each peduncle. To experimentally
demonstrate the effectiveness of our approach, a robot
was used to identify the peduncles of target tomato
plants and harvest the tomatoes at a real farm. Using
the proposed method, the harvesting robot achieved
peduncle detection of the tomatoes, and harvested
tomatoes successfully by cutting the peduncles.
Keywords: harvesting robot, peduncle detection, point
cloud processing, voxel processing
1. Introduction
Agriculture in Japan has recently been faced with a
shortage of workers. This serious problem is due to the
aging population, and a decrease in the number of agri-
cultural workers. However, this presents an opportunity
for technology development in the agricultural industry.
With advances in smart agriculture such as information
and communication technology, or robotics technology,
farmers can earn greater profits with less labor.
There are busy agricultural periods in which more
workers are required than usual. During the harvest sea-
son in particular, it is necessary for farmers to secure ad-
ditional workers because the manpower required is the
greatest during this time. Because harvesting is a seasonal
job, it is difficult to employ workers stably. However, if
farmers cannot secure enough workers, they are not able
to harvest their crops at the proper time. Therefore, the
main objective of this study is to develop a robot for auto-
mated harvesting to be used as a temporary worker during
busy times.
Various studies have dealt with the application of agri-
cultural robots. Irie et al. proposed an asparagus harvest-
ing robot [1, 2]. Their robot measures whether asparagus
is tall enough for harvesting with a 3D sensor. They also
proposed a mechanism for a robotic arm and an end effec-
tor to grasp and cut the asparagus. Bachche et al. proposed
a design and model of a gripper and cutting system for a
robotic arm with 5 degrees of freedom to harvest sweet
peppers in horticultural greenhouses [3]. In the proposed
design, both the gripping and cutting operations were per-
formed using a single servo motor to avoid complications
in the system. Kondo et al. proposed components, manip-
ulators, end effectors, and visual sensors for fruit harvest-
ing robots adapted for use with tomatoes, cherry toma-
toes, strawberries, and grapes [4]. Chen et al. proposed a
harvesting humanoid robot system to pick tomatoes, and
a visual cognition approach that enables the robot to har-
vest tomatoes [5]. Yaguchi et al. proposed the design and
development of an autonomous tomato harvesting robot
equipped with a rotational plucking gripper, which was
robust in estimating errors [6]. Hayashi et al. proposed
a strawberry-harvesting robot composed of a cylindrical
manipulator, end-effector, machine vision unit, storage
unit, and traveling unit [7, 8]. Tokunaga et al. proposed
an algorithm and designed an integrated circuit for recog-
nition of circular patterns in binary image based template
matching with a modified matching degree [9]. Their pro-
posed system is part of the vision system for a watermelon
harvesting robot. Yongsheng et al. proposed a mechani-
cal vision system to design a robot that can automatically
recognize and locate apples for harvesting [10]. Arima
et al. proposed a cucumber harvesting robot using a vi-
sual sensor, manipulator, end effector, and traveling de-
vice [11]. Monta et al. proposed a robotic vision method
using cameras, and a 3D vision system using a laser sen-
sor for a tomato harvesting robot and a cucumber harvest-

180 Journal of Robotics and Mechatronics Vol.30 No.2, 2018

© Fuji Technology Press Ltd. Creative Commons CC BY-ND: This is an Open Access article distributed under the terms of
the Creative Commons Attribution-NoDerivatives 4.0 International License (http://creativecommons.org/licenses/by-nd/4.0/).
 Fast Detection of Tomato Peduncle with a Harvesting Robot

ing robot [12]. van Henten et al. proposed a concept for a

modular cucumber harvesting robot equipped with a ther-
mal cutting device and tested in a green house [13–15].
It is necessary for robots to detect where to cut plants
for harvesting. Therefore, this study focuses on peduncle
detection to improve autonomous harvesting. For pedun- RGB-D camera
cle detection, there are two broad types of methods: the
3D point cloud based method, and the 2D image based
method.
End effector
Sa et al. proposed a method to detect peduncles from
a 3D model reconstructed from images of detected sweet
peppers using a robot equipped with a robotic arm and an
RGB-D camera [16]. In this approach, dense point clouds Fig. 1. RGB-D camera and end effector.
of sweet peppers were reconstructed from multiple views
using Kinect Fusion [17]. They combined color infor-
mation to distinguish red sweet peppers from their green Peduncle
peduncles, and they used geometric features to discrimi-
nate between peduncles and green peppers. The surface
normals of the point cloud were calculated to generate a
Point Feature Histograms (PFH) [18]. The PFH is one
of the descriptors that can capture geometric characteris-
tics of the 3D point cloud. The PFH features were then
concatenated with HSV color information to create a fea-
ture vector used for classification. Peduncle detection was
performed using a Support Vector Machine (SVM) with
an RBF kernel. As the PFH feature requires a dense 3D
point cloud, this method involves a high computational
time to detect peduncles. Fig. 2. A target tomato with the peduncle identified.
Luo et al. proposed a method to detect peduncles of
grapes with one side of 2D stereo images taken by a
stereo camera [19]. In this approach, a grape cluster A point cloud acquisition with RGB-D camera
is first identified using color-based image segmentation
based on the H-I component. After image segmentation Voxelization with two resolutions
and noise elimination in the image, contour information Sparse voxels
for the grape clusters is extracted, including the barycen-
ter, the top point, and the bounding rectangle of the con- Classification of tomato regions
tour. The region of interest (ROI) of the grape peduncle
is determined based on the barycenter, the top point co- Clustering of tomato regions
Dense voxels
ordinates, and the maximum Euclidean distance between
of the left and right points of the contour. Line detec- Tomato regions
tion is performed using a Hough transform on a binary Identifying the cutting points
ROI with a bilateral filter applied. Then, the line close to
Fig. 3. Flowchart of the proposed method.
the barycenter is identified as the peduncle. However, as
this method relies on 2D images that do not contain depth
information, it is difficult to detect peduncles in environ-
ments with many ridges. growing, the proposed method extracts tomato regions in
We propose a fast method to detect peduncles with a 3D the visual scene. Finally, an energy function is defined by
point cloud that includes depth information. Fig. 1 shows three conditions and then minimized to identify the appro-
part of the developed tomato harvesting robot. Fig. 2 priate cutting points on the peduncles.
shows a target tomato used in our experiments. This robot
is equipped with an RGB-D camera to detect the peduncle
of the tomato plant, and an end effector to harvest toma- 2. Peduncle Detection Algorithm
toes. The proposed method uses only one frame of the
point cloud data taken by the RGB-D camera without us- This section describes our method for identifying cut-
ing any images. In addition, voxelization is applied to re- ting points. The proposed method comprises four steps:
duce the amount of data, and the computational time for voxelization of a point cloud, classification of the tomato
calculating the positional relationship between voxels is region, clustering of the tomato region, and identification
reduced as a consequence. Through classification with a of the cutting point. Fig. 3 shows a flowchart of the pro-
support vector machine and clustering with voxel region posed method.

Journal of Robotics and Mechatronics Vol.30 No.2, 2018 181

Yoshida, T., Fukao, T., and Hasegawa, T.

300
Back Ground
250 Tomato
Support Vector
200

Blue
150

100

50

(a) Point Cloud (b) Voxel 0

300
200 300
Fig. 4. Point cloud and corresponding voxels created by 100
00 50 100 150 200 250
Green Red
voxelization.
Fig. 6. RGB data used for learning with SVM.

(a) Selection of a (b) Judging whether (c) New seed action

random seed neighbours belong following the same
to a region or not rules
(a) Sparse resolution (b) Dense resolution
Fig. 7. Transition of seeds.
Fig. 5. Two different voxel resolutions.

2.3. Clustering of Tomato Regions

2.1. Voxelization of Point Cloud
The point cloud is converted into a voxel grid by com-
bining close points. Fig. 4 shows point clouds and the
corresponding voxels created by voxelization.
The voxel grid comprises three types of information:
coordinates of each of the voxels, what type of points
are contained by each voxel, and what types of voxels
are next to each other. Using this adjacency information,
voxelization has not only the effect of down sampling but
also the effect of reducing the calculation time required to
search neighboring points. This study uses voxelization
in two resolutions: sparse resolution is used for classifi-
cation of tomato regions and clustering of tomato regions
(Fig. 5(a)), while a dense resolution is used for identify-
ing the cutting points (Fig. 5(b)).
2.2. Classification of the Tomato Regions
The next step is to divide voxels into a tomato region
and an “other” region. Here, a Support Vector Machine
(SVM) is used to easily classify the tomato regions be-
cause there are very few objects on the tomato farm that
are similar in color to a tomato, and an SVM runs fast
even on a standard computer [20]. Even if parts of the
tomato region are not classified as tomatoes, these are
reintegrated in the next region growing process. The hy-
perplane of the SVM is learned from the RGB data of
several images. Fig. 6 shows the RGB data of the tomato
region and of the other region in the images. Fig. 6 also
shows the support vector used by our model, which pre-
dicts whether each voxel is parts of a tomato region.
In the previous step, a point cloud was divided into the
tomato region and the other region. The next step is to di-
vide the tomato region into tomato segments. To accom-
plish this, Region Growing is applied as follows. First, a
voxel is selected randomly as the seed for Region Grow-
ing (Fig. 7(a)). Then, voxels adjacent to the seed are clas-
sified as belonging to the tomato region or not (Fig. 7(b)).
Voxels adjacent to the seed that are determined to belong
to the tomato region are added into a single segment con-
taining the seed. They are treated as new seeds and the
voxels adjacent to these new seeds are judged according
to the same rules (Fig. 7(c)).
The algorithm returns to the first step if no voxels are
added to the region. A tomato region does not mix with
other regions because this algorithm is based on an as-
sumption that the tomato regions are placed far apart at
the cultivation stage.
2.4. Identifying the Cutting Points
To identify a cutting point on a peduncle, one voxel re-
gion is divided into layers in the vertical direction (Fig. 8).
This equals a clustering of voxels located at the same
height. This method is based on an assumption that each
tomato is vertically cultivated. Thus, each layer is evalu-
ated to determine if it includes a tomato peduncle based
on three conditions.
The first condition is that the target layer is small. This
is because the diameter of a layer including both the pe-
duncle and the fruit of tomato is much larger than that of
a layer containing only a peduncle.

182 Journal of Robotics and Mechatronics Vol.30 No.2, 2018


Layer t+2
Layer t+1
Layer t
Fig. 8. Layers used for the division of voxels.
The second condition is that the target layer is close to
the end effector. This is because the end effector can not
cut the peduncle if the identified cutting point is far away
from it because the end effector may catch on an obstacle.
The third condition is that the target layer is close to
a vertical orientation. The direction of the approach of
the end effector should be as direct as possible relative to
the peduncle, because the end effector can only move in a
limited range of angles.
Next, an energy function to determine if a target layer
satisfies these three conditions reasonably is defined as
in Eq. (1), where l is the layer index, V (l) is the set of
voxels which belong to layer l, and λ and γ are weighting
coefficients.
  λ
E(l) = n V (l) +   ∑ vv1
n V (l) v ∈V (l)
 3. Experiments
1 − ŷy · (w
w − v ) if v joins w
+γ min
v ∈V (l), w ∈V (l+1) ∞ otherwise.
. . . . . . . . . . . . . . . . . . . .
The first term in Eq. (1) presents the first condition re-
garding the size of a target layer, where n() denotes the
number of elements in a set. For rapid calculation, this al-
gorithm uses the number of elements to calculate the size
of a target layer.
The second term in Eq. (1) presents the second condi-
tion on the distance between a target layer and the end
effector, where v are voxels that belong to the set V (l)
and  1 is the L1 norm. The L1 norm is used to quickly
calculate the distance between a target layer and the end
effector.
The third term in Eq. (1) presents the third condition
with regard to the direction of the layer, where ŷy is a ver-
tically upward unit vector. Fig. 9 shows the directions
between voxels that belong to two different layers. The
algorithm selects the direction between voxels that is clos-
est to the vertically upward direction. For example, in the
case of Fig. 9, the algorithm selects direction b.
The energy function is digitized with the introduction
of the layers in this algorithm. The minimum of the value
calculated for all layers is considered the minimum value
of the energy function.
Journal of Robotics and Mechatronics Vol.30 No.2, 2018
Fast Detection of Tomato Peduncle with a Harvesting Robot
a b c d
Fig. 9. Directions between voxels that belong to two differ-
ent layers.
Rail
End effector
Robot arm
Fig. 10. Proposed harvesting robot and representative field
used for experiment.
For the experiments, the robot was tasked with iden-
tifying the cutting points of target tomato peduncles to
(1)
harvest target tomatoes at a real farm. Fig. 10 shows
our experimental field and the proposed harvesting robot
equipped with a robot arm and an end effector. Most of
the tomato peduncles are on the rails as shown in Fig. 10.
The robot faces one side of a row of tomatoes while mov-
ing on the rails between the rows.
First, the robot was placed in front of a target tomato
as the initial position. The robot then observes a tomato
with the RGB-D camera, and subsequently identifies the
cutting point of the tomato as described above. Finally,
the robot inserts its end effector to the cutting point with-
out dropping the tomato.
Figures 11 and 12 show the results of this experiment.
Figs. 11(a) and 12(a) show the input point cloud data for
one frame as captured by the RGB-D camera. Figs. 11(b)
and 12(b) show output voxels based on the point clouds in
Figs. 11(a) and 12(a). They also show cutting points iden-
tified by the proposed method. Both cutting points were
identified as points belonging to the peduncles that could
be approached by the end effector. However, Fig. 12
shows two tomato bunches, but only one cutting point.
This is because the algorithm outputs the point that is
closest to the end effector. Figs. 11(c) and 12(c) show
183
Yoshida, T., Fukao, T., and Hasegawa, T.

(a) Input point cloud (a) Input point cloud

Cutting point

Cutting point

(b) Output voxels and a cutting point (b) Output voxels and a cutting point

(c) Energies of the search layers (c) Energies of the search layers

Fig. 11. Case 1 of the experimental results. Fig. 12. Case 2 of the experimental results.

184 Journal of Robotics and Mechatronics Vol.30 No.2, 2018


Table 1. Comparison of calculation times.
Input points Calculation times [s]
Case 1 54,598 0.41
Case 2 52,899 0.42
Sa et al. [16] 6,094 40
Table 2. Results of continuous harvest trial.
Trial No. 1 2 3 4 5 6
Number of tomatoes 4 4 4 4 2 2
Number of successes 4 4 4 4 2 2
the energies of three conditions and the sum of energies.
These search layers begin with the layer at the top of the
tomato region.
Table 1 presents a comparison of the calculation times
required for our method and the method proposed by Sa
et al. [16]. Sa et al. used an Intel Core i7 processor.
Comparatively, our robot uses an Intel quad core Xeon
E3-1245v5 (3.5 GHz). The proposed peduncle detec-
tion uses only one core because the other cores are used
for other tasks. Although the target crops are different,
the proposed method is very fast compared with alterna-
tive methods, even though the proposed method includes
more input point clouds. The method proposed by Sa
et al. requires a dense point cloud to use Point Feature
Histograms [18] for peduncle detection. This leads to a
high computational time for the construction of a dense
point cloud based on the RGB-D data used by Kinect Fu-
sion [17]. On the contrary, the proposed method requires
only one frame to detect peduncles. In addition, the vox-
elization step in the proposed method helps to reduce the
calculation time by down sampling and pre-calculating
the adjacencies of voxels.
When the RGB-D camera can capture the peduncle
clearly, our method can detect the peduncle almost com-
pletely. However, if the RGB-D camera can only capture
the object unclearly due to occlusion or noise, our method
cannot accurately detect the peduncle. Table 2 gives the
results of the continuous harvest trial. This table lists the
number of successful harvests for each trial. The success
rate was approximately 95% across these trials.
Finally, Fig. 13 shows an example of a harvesting robot
programmed according to the proposed method for suc-
cessfully harvesting a tomato.
4. Conclusion
This paper proposed a fast method for identifying cut-
ting points on tomato peduncles for a harvesting robot. In
this approach, an energy function is defined by three con-
ditions, and is minimized to identify appropriate cutting
points on peduncles based on the voxels. The experimen-
tal results confirmed that the tomato peduncle detection
Journal of Robotics and Mechatronics Vol.30 No.2, 2018
Fast Detection of Tomato Peduncle with a Harvesting Robot
7
2
1
Fig. 13. An example of the harvesting process.
and the cutting point identification properly directed the
harvesting robot to harvest tomatoes.
References:
[1] N. Irie, N. Taguchi, T. Horie, and T. Ishimatsu, “Development
of Asparagus Harvester Coordinated with 3-d Vision Sensor,” J.
Robot. Mechatron., Vol.21, No.5, pp. 583-589, 2009.
[2] N. Irie and N. Taguchi, “Asparagus Harvesting Robot,” J. Robot.
Mechatron., Vol.26, No.2, pp. 267-268, 2014.
[3] S. Bachche and K. Oka, “Design, Modeling and Performance Test-
ing of End-Effector for Sweet Pepper Harvesting Robot Hand,” J.
Robot. Mechatron., Vol.25, No.4, pp. 705-717, 2013.
[4] N. Kondo and M. Monta, “Fruit Harvesting Robotics,” J. Robot.
Mechatron., Vol.11, No.4, pp. 321-268, 1999.
[5] X. Chen, K. Chaudhary, Y. Tanaka, K. Nagahama, H. Yaguchi,
K. Okada, and M. Inaba, “Reasoning-Based Vision Recognition for
Agricultural Humanoid Robot Toward Tomato Harvesting,” 2015
IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS),
pp. 6487-6494, 2015.
[6] H. Yaguchi, K. Nagahama, T. Hasegawa, and M. Inaba, “Develop-
ment of an Autonomous Tomato Harvesting Robot with Rotational
Plucking Gripper,” 2016 IEEE/RSJ Int. Conf. on Intelligent Robots
and Systems (IROS), pp. 652-657, 2016.
[7] S. Hayashi, K. Shigematsu, S. Yamamoto, K. Kobayashi, Y. Kohno,
J. Kamata, and M. Kurita, “Evaluation of a Strawberry-Harvesting
Robot in a Field Test,” Biosystems Engineering, Vol.105, No.2,
pp. 160-171, 2010.
[8] S. Hayashi, K. Takahashi, S. Yamamoto, S. Saito, and T. Komeda,
“Gentle Handling of Strawberries Using a Suction Device,” Biosys-
tems Engineering, Vol.109, No.4, pp 348-356, 2011.
[9] Y. Tokunaga, T. Hakukawa, and T. Inoue, “Algorithm and Design
of an Intelligent Digital Integrated Circuit for a Watermelon Har-
vesting Robot,” J. Robot. Mechatron., Vol.11, No.3, pp. 173-182,
1999.
[10] S. Yongsheng, L. Gang, and F. Juan, “Location of Apples in Trees
Using Stereoscopic Vision,” Computers and Electronics in Agricul-
ture, Vol.112 (Supplement C), pp. 68-74, 2015.
[11] S. Arima and N. Kondo, “Cucumber Harvesting Robot and Plant
Training System,” J. Robot. Mechatron., Vol.11, No.3, pp. 208-212,
1999.
[12] M. Monta, N. Kondo, S. Arima, and K. Namba, “Robotic Vision
for Bioproduction Systems,” J. Robot. Mechatron., Vol.15, No.3,
pp. 341-348, 2003.
[13] E. J. van Henten, J. Hemming, B. A. J. van Tuijl, J. G. Kornet,
J. Meuleman, J. Bontsema, and E. A. van Os, “An Autonomous
Robot for Harvesting Cucumbers in Greenhouses,” Autonomous
Robots, Vol.13, No.3, pp. 241-258, 2002.
[14] E. J. Van Henten, B. A. J. Van Tuijl, J. Hemming, J. G. Kornet,
J. Bontsema, and E. A. Van Os, “Field Test of an Autonomous
Cucumber Picking Robot,” Biosystems Engineering, Vol.86, No.3,
pp. 305-313, 2003.
185
 Yoshida, T., Fukao, T., and Hasegawa, T.

[15] E. J. Van Henten, B. A. J. Van Tuijl, G.-J. Hoogakker, M. J. Van Der

Weerd, J. Hemming, J. G. Kornet, and J. Bontsema, “An Au-
tonomous Robot for De-leafing Cucumber Plants grown in a High-
wire Cultivation System,” Biosystems Engineering, Vol.94, No.3,
pp. 317-323, 2006.
[16] I. Sa, C. Lehnert, A. English, C. McCool, F. Dayoub, B. Upcroft,
and T. Perez, “Peduncle Detection of Sweet Pepper for Autonomous
Crop Harvesting – Combined Color and 3-D Information,” IEEE
Robotics and Automation Letters, Vol.2, No.2, pp. 765-772, 2017.
[17] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim,
A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon,
“Kinect fusion: Real-Time Dense Surface Mapping and Tracking,”
2011 10th IEEE Int. Symposium on Mixed and Augmented Reality,
pp. 127-136, 2011.
[18] R. B. Rusu, N. Blodow, and M. Beetz, “Fast Point Feature His-
tograms (FPFH) for 3D Registration,” 2009 IEEE Int. Conf. on
Robotics and Automation, pp. 3212-3217, 2009.
[19] L. Luo, Y. Tang, X. Zou, M. Ye, W. Feng, and G. Li, “Vision-Based
Extraction of Spatial Information in Grape Clusters for Harvesting
Robots,” Biosystems Engineering, Vol.151, pp. 90-104, 2016.
[20] C. Chang and C. Lin, “LIBSVM: A Library for Support Vector Ma-
chines,” ACM Trans. on Intelligent Systems and Technology, Vol.2,
No.3, pp. 1-27, 2011.
Name:
Takanori Fukao
Affiliation:
Professor, Department of Electrical and Elec-
tronic Engineering, Ritsumeikan University
Address:
1-1-1 Noji-higashi, Kusatsu, Shiga 525-8577, Japan
Brief Biographical History:
1996- Assistant Professor, Kyoto University
2001- Visiting Scientist, Carnegie Mellon University
2004- Associate Professor, Kobe University
2015- Professor, Ritsumeikan University
Main Works:
• robotics, autonomous vehicles
Membership in Academic Societies:
• The Japan Society of Mechanical Engineers (JSME)
• The Institute of Electrical and Electronics Engineers (IEEE)

Name:
Takeshi Yoshida

Affiliation:
Assistant Professor, Research Organization of Name:
Science and Technology, Ritsumeikan Univer- Takaomi Hasegawa
sity
Affiliation:
Project Assistant Manager, Robotics System De-
velopment Department, Denso Corporation
Address:
1-1-1 Noji-higashi, Kusatsu, Shiga 525-8577, Japan
Brief Biographical History:
2015- Assistant Professor, Aoyama Gakuin University
2017- Assistant Professor, Ritsumeikan University Address:
Main Works: 1-1 Showa-cho, Kariya, Aichi 448-8661, Japan
• robotics, computer vision Brief Biographical History:
Membership in Academic Societies: 2006- Denso Corporation
• The Robotics Society of Japan (RSJ) Main Works:
• The Society of Instrument and Control Engineers (SICE) • robotics
• Information Processing Society of Japan (IPSJ)

186 Journal of Robotics and Mechatronics Vol.30 No.2, 2018

Powered by TCPDF (www.tcpdf.org)