DESIGN AND FIELD EVALUATION OF A

ROBOTIC APPLE HARVESTING SYSTEM WITH A

3D-PRINTED SOFT-ROBOTIC END-EFFECTOR
C. J. Hohimer, H. Wang, S. Bhusal, J. Miller, C. Mo, M. Karkee

ABSTRACT. Fresh market apple harvesting is a difficult task that relies entirely on manual labor. Much research has been
done on the development of mechanical harvesting techniques. Several selective harvesting robots have been developed for
research studies, but there are no commercially available robotic systems. This article discusses the design and development
of a novel pneumatic 3D-printed soft-robotic end-effector to facilitate apple separation. The end-effector was integrated
into a robotic system with five degrees of freedom that was designed to simplify the picking sequence and reduce costs
compared to previous versions. Apples were successfully harvested using the low-cost robotic system in a commercial or-
chard during the fall 2017 harvest. A detachment success rate on attempted apples of 67% was achieved, with an average
time of 7.3 s per fruit from separation to storage bin. By conducting this study in an orchard where problematic apples were
not removed to increase the detachment success rate, current pruning and thinning practices were assessed to help lay the
foundation for future studies and develop strategies for successfully harvesting apples that are difficult to detach.
Keywords. Apple catching, Apples, Automated harvesting, Field experimentation, Harvesting robot, Soft-robotic gripper.

F
or the past several decades, researchers have been
trying to develop systems for autonomous fresh
market fruit harvesting, with the goal of reducing
labor cost and the overall dependence on labor. The
success and failure of such systems have been discussed in
detail in several review articles in the last several years (Bac
et al., 2014; Peterson, 2005; Zhang et al., 2016a). Of specific
interest to this work are harvesting systems for fresh market
apples. Despite several projects dating back as far as 1987,
researchers have been unable to bridge the gap from R&D to
commercialization or even to large-scale studies (Zhang et
al., 2016a). Several projects are underway to create commer-
cial systems, but even the most advanced of these projects is
still in the early startup stages (e.g., FFRobotics, Israel;
Abundant Robotics Inc., California).
This gap between R&D and commercialization stems
from several complex problems with autonomous harvest-
ing, and any proposed system must balance each component
Submitted for review in June 2018 as manuscript number MS 12986;
approved for publication by the Machinery Systems Community of ASABE
in December 2018.
The authors are Cameron J. Hohimer, Graduate Student, School of
Mechanical and Materials Engineering, and Heng Wang, Graduate
Student, School of Engineering and Applied Sciences, Washington State
University, Richland, Washington; Santosh Bhusal, Graduate Student,
Center for Precision and Automated Agricultural Systems (CPAAS),
Washington State University, Prosser, Washington; John Miller, Professor,
School of Engineering and Applied Sciences, and Changki Mo, Professor,
School of Mechanical and Materials Engineering, Washington State
University, Richland, Washington; Manoj Karkee, Professor, Center for
Precision and Automated Agricultural Systems (CPAAS), Washington
State University, Prosser, Washington. Corresponding author: Cameron
Hohimer, 2710 Crimson Way, Richland, WA 99352; phone: 509-372-7356;
e-mail: cameron.hohimer@wsu.edu.
and design choice with performance factors as well as eco-
nomic metrics. For a system to be commercially viable, it
must add value to the current method for harvesting a spe-
cific crop. In the apple industry, this threshold seems to be
relatively low, as harvesting labor amounts to almost 30% of
the variable annual cost, and this seasonal labor force is in
increasingly short supply (Gallardo et al., 2010).
One of the major hurdles faced by researchers in this field
is the complex, unstructured environment in which a robot
must operate. Unlike typical industrial robots that perform
tasks repeatedly, harvesting robots must operate in a space
with poorly defined positions and without a clear path from
point A to point B within the workspace. Often, a robot is
required to operate without complete knowledge of the ob-
jects in the workspace that need to be avoided.
Over the last several decades, fruit growers have made
significant progress in orchard design to maximize crop
yield and reduce the labor required to harvest and maintain
fruit trees. With this effort, highly structured, high-yield or-
chards are being developed and planted across the industry.
Several studies have been conducted to assess and develop
tree structures to improve reachability in fruit trees. Bloch et
al. (2018) used environmental modeling to propose optimal
robot configurations and designs to improve performance in
reachability based on the tree structure. Similarly, reachabil-
ity simulations in high-density trellised pear trees were con-
ducted to assess pickable fruit using only linear motion; the
authors concluded that up to 93% of the fruit could be
reached by linear motion only (Arikapudi and Vougioukas,
2017; Vougioukas et al., 2016). These results provide insight
into optimal tree architectures, but little work has been done
to understand individual fruit locations in relation to other
fruit, tree branches, and trellis wires and how these factors

Transactions of the ASABE

Vol. 62(2): 405-414 © 2019 American Society of Agricultural and Biological Engineers ISSN 2151-0032 https://doi.org/10.13031/trans.12986 405
relate to robotic harvesting success rates.
It has been stated that growers are willing to prune their
trees to facilitate robotic harvesting, but thus far no study has
been conducted to determine how apples might need to be
thinned to improve a robot’s harvesting success rate. Addi-
tionally, the lack of detailed reporting within robotic harvest-
ing research makes it difficult to assess the possible modifi-
cations to the field environment, such as thinning, that have
been done to improve performance (Bac et al., 2014). In
sweet pepper harvesting, the robotic harvesting success rate
(harvested fruit vs. total fruit in canopy) can be increased by
more than ten times by removing clusters and occluding
leaves before harvesting (Bac et al., 2017).
The research reported in this article addresses this issue
by studying a grasping type end-effector and how harvesta-
bility is affected when apples have not been thinned before
harvesting. This study is expected to provide essential infor-
mation for optimizing both crop yields and robotic harvest-
ing success rates. Several conditions that are not conducive
to robotic harvesting are expected with this approach. Previ-
ously, apples that were too close to branches or trellis wires,
in clusters of two or more, or on long pendulum-like
branches were removed before robotic harvesting. The re-
sults of this thinning yielded, on average, only a possible
3.5 fruits per cycle and fewer than 2.5 fruits successfully
picked per cycle (Silwal et al., 2017). Without information
on the number of fruits thinned or removed before harvest-
ing, it is impossible to assess the true efficiency of a robot
because the tree was altered to improve performance. For
this field trial, the objectives were to:
 Develop a pneumatic soft-robotic end-effector to re-
duce the risk of damage to both the tree and robot due
to collisions.
 Assess the performance of the harvesting system in an
orchard that has not been thinned to remove problem-
atic apples and thereby artificially improve harvesting
efficiency and performance.
Figure 1. Robotic harvester preparing to harvest ‘Jazz’ apples in 2017.
406
BACKGROUND
Over the past several years, Washington State University
has been developing an autonomous robotic system for har-
vesting fresh market apples. The first prototype was tested
in the fall of 2015 using a robot with seven degrees of free-
dom (DOF) and a tendon-driven three-fingered end-effector
with little to no lateral reach, i.e., to the robot’s left or right
(Davidson and Mo, 2015; Silwal et al., 2017). This initial
field trial was largely successful, but it had two areas for pos-
sible improvement: (1) expand the workspace to increase the
number of pickable apples, and (2) implement a fruit con-
veyance system (previously, picked apples were simply
dropped to the ground).
In 2016, the robot workspace was expanded by adding an
additional DOF (8-DOF total), so the robot had wider lateral
movement. Attempts were made to address fruit conveyance
by adding a secondary robot to catch and store apples after
the harvesting arm had separated them from the tree. This
work is described in detail by Davidson et al. (2016a, 2017).
In the 2016 field trial, a major shortcoming was a failure to
detach fruit from the tree, as well as a high tendency for the
catching robot to collide with low-hanging branches. Be-
cause of these difficulties, the performance of the catching
robot was not assessed.
Excessive movement prevented the robot arm from con-
sistently separating apples from the tree. The first field trial
in 2015 was largely successful because the robot was con-
strained to ensure that it pulled apples away from the tree
along a vector normal to the end-effector palm, something
that was not true for the 2016 kinematics. These observations
showed a need to further refine many of the design changes
from the 2015 and 2016 field trials. These changes and the
results of the 2017 field trials are discussed in this article. A
goal of this study was to assess the performance of the ro-
botic harvester in an environment that had not been artifi-
cially thinned to increase the robot’s efficiency. Figure 1
shows the harvesting robot staged to pick ‘Jazz’ apples dur-
ing the fall 2017 harvest.
TRANSACTIONS OF THE ASABE
ROBOTIC APPLE HARVESTER DESIGN
SOFT-ROBOTIC PNEUMATIC END-EFFECTOR
Fruit grasping with robotic end-effectors spans a wide
range of applications, and many different types have been
implemented with various degrees of success (Blanes et al.,
2011). Many of these end-effectors were designed for use in
a more controlled setting, such as a processing facility, and
not in an unpredictable field environment (Ceccarelli et al.,
2000; Davis et al., 2008; Morales et al., 2014). Some were
designed for field or greenhouse applications but have been
limited to laboratory testing (Dimeas et al., 2015). Most of
these end-effectors rely on compliant underactuated designs
using a combination of rigid components linked together by
flexible hinges that allow them to adapt to slight changes in
object shape and size. These systems work well in controlled
settings, but a more compliant approach is required when po-
sition error and obstacle interference are common.
A goal of this study was to assess the performance of the
robotic harvester in an orchard that had not been artificially
cleared of problematic apples (e.g., removal of clusters, pen-
dulum fruit, and fruit close to or behind trellis wires or
branches) or otherwise altered to enhance the robot’s perfor-
mance. While normal pruning and thinning occur throughout
the year to improve crop yield, based on traditional growing
practices, these operations are not done with robotic harvest-
ing in mind. In this study, the intent was to allow the robot
Figure 2. (a) CAD model of soft-robotic end-effector and (b) diagram of actuator with dimensions in millimeters.
Figure 3. Soft-robotic end-effector grasping an ‘Envy’ apple: (a) unactuated side view, (b) actuated side view, and (c) actuated top view.
62(2): 405-414
to experience what a human picker would see at the start of
the harvest, i.e., some apples in contact with trellis wires and
branches as well as the trunks of the trees, making collisions
inevitable.
To accomplish this without risking damage to the tree or
robot, a soft-robotic end-effector was developed. While sim-
ilar end-effector prototypes have been previously developed
for different applications, few have moved beyond labora-
tory testing (Amend et al., 2012; Rateni et al., 2015; Udupa
et al., 2014; Wang et al., 2017; Yap et al., 2016). The end-
effector consists of three highly compliant, pneumatic actu-
ators mounted around a soft, flexible palm designed to sup-
port an apple against the actuators, as shown in figure 2a.
Figure 2b shows the dimensions of the 3D-printed actuators.
Figure 3 shows the functional prototype gripping an apple
before harvesting.
Each actuator is symmetrically positioned about a circle,
120° apart. The upper two actuators are shorter (95.25 mm)
and are designed to stabilize the apple within the grasp of the
end-effector. The lower actuator is longer (152.4 mm) and is
designed to wrap almost fully around the apple. This config-
uration reduces potential interference between the actuators
when grasping a particularly small apple.
The actuators were fully 3D-printed without any post-
processing required. Each actuator was fabricated in a single
step and then put into service. For this study, a custom three-
407
axis gantry system was built and fitted with a Titan Extruder
with a V6 HotEnd (E3D-Online, Chalgrove, U.K.) suitable
for printing flexible filaments to fabricate the actuators. Sim-
ilar to other actuators fabricated with fused deposition mod-
eling (FDM), the actuators used in this study are comprised
of bellows that are inflated, causing the actuator to bend
(Yap et al., 2016). The speed, motion, and direction are con-
trolled by a combination of the actuator geometry and the
input air pressure (Mosadegh et al., 2014; Sun et al., 2013).
For this study, the actuators were printed with a larger cross-
sectional area and without the constraint of a specific print-
ing orientation, resulting in a significant increase in the pull-
ing force for a given input pressure.
Previously, such devices have been formed out of soft sil-
icone using a mold and casting process that involves multi-
ple steps (Ilievski et al., 2011). Fully 3D-printed actuators
have the advantage of being fabricated out of a much more
durable thermoplastic polyurethane (TPU) with a Shore
hardness of 85 A, compared to silicone with a Shore hard-
ness of 00 to 30 A. This durability is necessary in an orchard
environment where collisions between the tree and end-ef-
fector are common.
To determine their maximum pulling force, the actuators
were clamped into a custom tensile loading machine and ac-
tuated at various pressures while pulling against a ring at-
tached to a load cell (MLP 750, Transducer Techniques,
Temecula, Cal.). Table 1 summarizes the pulling force at
various internal pressures. The actuators tested in this study
required a minimum of 103 kPa (15 psi) for complete actua-
tion. At this pressure, the force exerted by the actuator was
determined to be 49.53 6.3 N. The time required for actua-
tion is also given in table 1. A simple benchtop timing device
was fabricated to record the delay between the pressure ap-
plied and a pushbutton pressed by the actuator. Each tested
pressure was repeated ten times and averaged with standard
deviations, also reported in table 1. The actuation time rep-
resents a significant improvement over tendon-driven fin-
gers, which required 1.5 s to close and 1.0 s to release (Silwal
et al., 2017).
Because of the many factors (e.g., stem location and
length, fruiting branch length, fruit maturity, fruit diameter)
involved in the force required to separate fruit from a tree
(Erdoǧan et al., 2003; He et al., 2017), it is difficult to di-
rectly correlate an actuator’s pulling force to the force re-
quired to separate a given apple from a tree. Several hand-
picking studies conducted on different apple varieties rec-
orded the pulling force normal to the grasping fingers. Da-
vidson et al. (2016b) found peak forces of 11 to 30 N de-
pending on apple variety, while Li et al. (2016) reported
grasping forces of 5 to 15 N. Both studies looked at a variety
of grasping techniques using a three-finger grasp. Li et al.
(2016) focused on a more traditional picking technique by
applying a bending moment to the stem, while Davidson et
Table 1. Pulling force and activation time of soft-robotic actuator from
benchtop tests. Force and time values are means of ten tests.
Pressure (kPa) Pulling Force (N) Actuation Time (ms)
103.4 49.54 6.37 82.5 1.95
206.8 73.28 3.67 74.5 2.27
275.8 96.12 1.70 65.1 0.74
344.8 100.28 4.73 60.2 1.14
408
al. (2016b) focused on straight-line pulls similar to the sep-
aration technique used in the present study. Those previous
results suggested that the soft actuators used in this study
have sufficient grasping strength for fruit separation. For
field testing, 103.4 kPa (15 psi) was set on the air supply
regulator, and the compressor was allowed to recover fully
between harvesting sequences.
HARVESTING ARM KINEMATICS
The harvesting arm is a 5-DOF system with two prismatic
(P) joints and three revolute (R) joints. The system first cap-
tures an image to identify apples and determine their loca-
tions to generate individual fruit coordinates, as discussed
later. Branches and other possible obstacles are not identi-
fied, and obstacle avoidance was not included in this study.
The apple coordinates are processed one at a time by the con-
trol software and inverse kinematics solver to calculate the
joint positions and sequence for the picking process. Apples
are prioritized from right to left starting with the smallest
y-coordinate, i.e., the apple farthest to the right in the image
frame. This minimizes movement of the y-axis translation
(q2) and eliminates backtracking. Further optimization could
be done to maximize the speed or minimize the travel mo-
tion. As tested, this picking process was a design simplifica-
tion chosen to reduce backtracking and translation.
Figure 4a shows the kinematic model of the robot in its
PPRRR configuration. Within this PPRRR configuration,
the prismatic joints are used to translate the robot in the x-y
plane. The prismatic joint along the y-axis (q2) is used to po-
sition the arm in line with an apple’s y-coordinate. The pris-
matic joint along the x-axis (q1) is used to approach the fruit.
The arm is first staged along the y-axis, 15 cm in front of the
fruit using the three revolute joints and then translated in the
x-direction using the prismatic joint. Figure 4b is a graphical
representation of this staging and translation in the x-z plane.
Fruit detachment is the reverse of this movement. Table 2
lists the Denavit-Hartenberg parameters of the system.
Simulations were conducted to compare the reachability
of the 5-DOF robot to that of the 7-DOF robot used in the
2015 field tests and the 8-DOF robot used in 2016 (Davidson
et al., 2017; Silwal et al., 2017). Fruit coordinates were ex-
tracted from the collected images and used to test the perfor-
mance through simulation. The coordinates used in the sim-
ulation were for apples located on the third and fourth trellis
wires, which was the target workspace for all three robots.
The results of the simulated reachability are shown in
table 3. From this table, it is clear that the 5-DOF robot lim-
ited the number of reachable fruit. Further analysis of the
simulation results showed that many of the missed fruit were
located too close to the harvesting arm. There were two rea-
sons for this. First, the arm is limited in its motion on the
prismatic joint in the x-direction. If a solution cannot be
found inside the limits of the revolute joints, the arm does
not adjust along the prismatic joint.
The second cause for the missed fruit is that the distance
between the transport vehicle (and therefore the robotic arm)
and the fruit wall is not fixed. As the vehicle (driven by a
human operator) moves along an orchard row, this gap be-
comes wider or narrower. This is an unavoidable conse-
quence of the field environment, and the robot does not cur-
TRANSACTIONS OF THE ASABE
 a)

Figure 4. (a) Kinemetric model of the 5-DOF arm and (b) 2D visual description of picking sequence.

Table 2. Denavit-Hartenberg parameters of 5-DOF harvesting robot.

r 2  l32  l42
Joint a (m)  (°) d (m)  (°)   cos1 (5)
1 0 -90 d1 0 2  l3  r
2 0 90 d2 -90
3 0.27 90 0.19 3 where Xt and Zt are the coordinates of the apple, l3 and l4 are
4 0.25 90 0 4 the lengths from the Denavit-Hartenberg parameters for
5 0.2 90 0 5
joints 3 and 4, and  is the desired end-effector pitch, which
Table 3. Simulation results of three robots tested from 2015 to 2017. was fixed at 0° in this study. Using an analytical solution
Robot Reachable Apples greatly reduces the computational time of the solver. Simu-
5-DOF 234 of 335 (69.9%) lated results showed that the 5-DOF solver is 10,000 times
7-DOF 260 of 335 (77.6%) faster than the numerical 8-DOF solver. While this does not
8-DOF 274 of 335 (81.8%)
amount to noticeable saving in time per apple picked, it is an
important step toward commercialization, when the control
rently account for this in any way. Ideally, the robot work- software will be implemented with a microcontroller.
space would completely overlap the fruit space at every stop Control was carried about using a Linux OS system with
of the arm. An area of active research is to find ways to allow custom C++ code. Dynamixel servomotors (Robotis Inc., Ir-
the robot to adapt to this dynamic environment (Wang et al., vine, Cal.) were used for the three revolute joints, and
2018). NEMA 23 stepper motors and a belt drive system were used
While the reduction in the number of DOF comes at a cost for the two prismatic joints. These actuators offered a low-
(table 3), it also has some benefits, the most significant of cost but sufficiently stable and responsive system. The Dy-
which is a great reduction in overall system cost. One of the namixel servomotors have internal feedback control and po-
main objectives of this project is to optimize the cost and sition sensors. The stepper motors used an open-loop soft-
speed of a robotic harvester to improve its commercial via- ware step-count to maintain positional accuracy.
bility. By removing three motors from the kinematic chain, For the 2017 harvest, the system was constrained such
the total cost of the hardware was reduced by almost 50% that the pitch of the end-effector was fixed at 0°. In previous
from previous versions. This is a significant cost saving con- harvest trials with this robot, the pitch was predefined based
sidering that one of the easiest ways to make robots faster is on the height of the apple. Handpicking data showed that
to employ several of them. there is little benefit to an inclined pull versus a horizontal
Another major advantage is that the inverse kinematics pull (a pitch equal to 0°) for separating fruit (Davidson et al.,
algorithm has an analytical solution. The three revolute joint 2016b). In this study, the pitch was fixed at 0° for all apples,
angles are determined using the following equations: regardless of height. By fixing the pitch of the end-effector,
3  atan2  Z t , X t    (1) the approach angle and grasping were held constant for all
attempted apples. This allowed better assessment of the
4     (2) grasping efficiency of the end-effector, particularly when the
stem location and surrounding branch locations were un-
5    1  2 (3) known. Fixing the pitch in such a way limited the robot
workspace and may have prevented some fruit from being
l32  l42  r 2 reachable (assumed to be a very few). However, it was not
  cos 1 (4) possible to allow an arbitrary pitch, as this would not con-
2  l3  l4
sistently separate fruit from the tree and would cause a

62(2): 405-414 409

 Figure 5. Apples identified by vision system for automated harvesting.
greater number of unnecessary collisions.
APPLE DETECTION
In this study, a relatively inexpensive stereo vision sys-
tem was used for apple detection. A BumbleBee XB3 indus-
trial stereo vision imaging sensor (FLIR Integrated Imaging
Solutions Inc., Richmond, BC, Canada) was used as a single
global camera system. The imaging sensor was mounted be-
hind the robot’s prismatic gantry. The sensor was placed
nearly 1.5 m away from the trees. Stereo image pairs were
captured with five different exposure values to overcome is-
sues related to the variable lighting conditions (Silwal et al.,
2017). Each of the five images was fused to form a single
image that minimized under- and overexposed regions in the
images. The intrinsic and extrinsic parameters for the camera
were computed using checkerboard calibration. These pa-
rameters were used in stereo rectification of the fused image,
making them epipolar. The camera calibration process for
this study was similar to that of Silwal et al. (2017). The ste-
reo rectified image pairs were used to compute disparity
maps, which gave the depth information of each object in the
camera’s field of view.
Apple detection was performed using a supervised classi-
fication technique. Nearly 1400 images from the BumbleBee
XB3 sensor with different exposure levels were used for
training the detectors. Two apple detectors were formed us-
ing a boosted cascade of Haar-like features (Viola and Jones,
2001) and local binary patterns (Ahonen et al., 2006) using
OpenCV-310. The two detectors were combined at the deci-
sion level using the 50% overlap criteria defined by Evering-
ham et al. (2010) to form the final apple detector. Combining
the two detectors minimized the number of false detections
and thus increased the precision of the system (up to 90%).
Figure 5 shows a typical image after apple detection, with
eleven detected apples within the robot’s field of view. High
precision was desired in the robotic harvesting of apples to
increase the speed of picking while minimizing the risk of
damaging the tree. Finally, the 3D coordinates of all detected
apples within the robotic arm’s workspace were identified
and transferred to the robotic arm’s controller.
FIELD RESULTS
Field testing was performed in a commercial apple or-
chard in Prosser, Washington, over four days (October 20
and 26, and November 1 and 3) in the fall of 2017. The first
two days of the experiment were conducted with ‘Jazz ‘ap-
410
ples, while days three and four were conducted with ‘Envy’
apples. As previously stated, a goal of this study was to as-
sess the performance of the robotic harvester in a commer-
cial orchard without altering the environment; therefore, no
thinning and pruning was performed to improve the picking
efficiency. The only exception was the removal of apples di-
rectly behind a trunk or large branch that were identified by
the vision system; attempts to pick these apples would cause
damage to the robot or tree. Given that these occluded apples
were rarely identified by the vision algorithm, only a handful
of apples were removed. Unfortunately, the exact number
was not recorded; however, a conservative estimate is that
approximately 15 apples were removed under this criterion.
Incorporating obstacle detection into the vision system, so
that these occluded apples could be ignored by the robot and
manually picked later, is an area of active research. In a com-
mercial orchard, pruning and thinning are common through-
out the year. These management operations were performed
outside the control of this study and were not done to im-
prove robotic harvesting efficiency but were based on best
practices of the growers to increase overall yield. Every ef-
fort was made to present the robot with a harvesting environ-
ment that a human picker would experience.
Simple assessment of the robot’s performance showed
that the robot successfully harvested 116 apples, or ~67%,
of 172 attempted picks. Given the overall small number of
harvesting robots reported in the literature, this is not outside
the range (42% to 77%) of success rates reported by Bac et
al. (2014) for other autonomous apple harvesting studies.
Several of these previous studies were performed in orchard
environments that had been modified specifically to improve
the robot’s harvesting efficiency, e.g., removal of fruit clus-
ters, pendulum fruit, and fruit close to or behind trellis wires
or branches (Silwal et al., 2017; Zhang et al., 2016b). This
reported success rate does not account for identified fruit that
were omitted because they were outside the robot’s work-
space. This is significant when compared with table 3, where
69.9% of detected fruit were reachable, or inside the robot’s
workspace. Thus, approximately 46.8% of the detected ap-
ples were harvested. Multiple harvesting arms could be used
to improve this success rate. Because of the space limitations
between orchard rows, it is difficult to design a robot capable
of harvesting fruit over the entire vertical space of a tree. In
this study, many of the missed fruit were outside the robot’s
workspace because of the height limitation but could have
been picked by a robot positioned to pick fruit higher in the
canopy. Active research is underway to optimize the robot
location to maximize the overall success of a single arm
TRANSACTIONS OF THE ASABE
(Wang et al., 2018).
Missed apples were classified into five categories: clus-
ters, position error, pseudo pendulum, failure to detach, and
obstructed grasp. Table 4 lists the counts for each category.
Apple clusters, apples obscured from view, and those that
were too close to obstacles and were removed prior to har-
vesting would pose no problems for a human picker. The av-
erage time per fruit was 7.3 0.78 s. This time accounts for
the entire picking process, from apple identification to stor-
age bin. It does not include setup time and includes missed
apples in the averages. The control software recorded the cy-
cle time, and the total cycle time was then divided by the
number of apples attempted.
The most significant cause for missed apples was clus-
tered fruit, at 37% of the total missed apples. Most of these
misses occurred because the end-effector collided with a sec-
ond apple in the cluster during the approach. The collision
caused a displacement of the targeted apple, resulting in the
end-effector’s inability to grasp the apple. Additionally, the
collision occasionally caused the second apple to fall from
the tree, resulting in a failure to pick the second apple be-
cause it was no longer on the tree. This fallen apple was also
considered a clustered fruit miss.
Another major source of missed apples (29%) was posi-
tion errors between the vision system and the robotic arm.
Most of the time, these errors resulted from collisions be-
tween the end-effector and the tree or trellis wire. While the
end-effector was designed to be compliant and unaffected by
collisions, the prismatic joints use an open-loop control
scheme. Collisions tended to result in small position errors
that accumulated over multiple attempted apples. A feed-
back control scheme should be implemented on the prismatic
joints to avoid this type of failure.
The third largest source of missed apples (20%) was the
Table 4. Failure rates by category of missed apples.
Category Total Missed Total Attempted
Clustered fruit 21 of 56 (37%) 21 of 172 (12%)
Position error 16 of 56 (29%) 16 of 172 (9%)
Pseudo pendulum 11 of 56 (20%) 11 of 172 (6%)
Failure to detach 5 of 56 (9%) 5 of 172 (3%)
Obstructed grasp 3 of 56 (5%) 3 of 172 (2%)
Figure 6. Sample images of apple cluster identification: (a) a cluster of three apples that could be picked if properly sequenced, (b) occlusion of
an apple cluster misidentified as a single apple, and (c) a cluster of three apples misidentified as two apples.
62(2): 405-414
pseudo pendulum effect. In these misses, rather than apply-
ing force directly to the stem as the robot pulled on an apple,
the pulling effort caused the fruiting branch to swing away
from the tree, and the robot let go before separation occurred.
Finally, the remaining missed apples were caused by failure
to detach (9%) or an obstructed grasp (5%). Apples that
failed to detach slipped out of the end-effector. Inconsistent
airflow was a factor for these apples, as the supply pressure
from the portable air compressor decreased after several ap-
ples in a picking sequence. Obstructed apples occurred when
one or more of the actuators closed around an adjacent
branch or wire, resulting in incomplete grasp of the fruit.
DISCUSSION
At present, there is no guidance on how commercial
growers can revise their pruning and thinning strategies to
maximize robotic harvesting efficiency and minimize the is-
sues currently faced by robotic harvesting systems. There-
fore, a robotic system that addresses the performance limita-
tions observed in this study due to fruit clusters, pseudo-pen-
dulums, and other issues would improve the commercial vi-
ability of such a harvesting system. Several observations
were made during the harvesting tests and are discussed here
as ways to improve robotic harvesting efficiency without al-
tering the orchard environment (e.g., by thinning clustered
apples).
Of the apples successfully picked, 36 (approx. 31%) were
classified as clustered. The robot’s success in picking a par-
ticular cluster of apples was observed to be largely random,
but observations were made as to why certain apples were
picked successfully. The biggest factor between success and
failure when picking a cluster was the apples’ relative posi-
tions within the cluster. Figure 6 shows sample images of
apple cluster identification. Apples in the same y-z plane
(i.e., side by side) as the target apple (fig. 6a) were more
likely to be pushed aside by the approaching end-effector, in
contrast to apples with an offset from each other in the x-
direction (i.e., depth) (fig. 6b).
In the cluster of three apples shown in figure 6a, the upper
411
two apples were most difficult to pick. As the end-effector
tried to grasp one of these two apples, the actuators collided
with the other apple and fail to grasp either of them. The
lowest apple of the cluster was successfully picked during
the harvesting test, but the upper two apples were left on the
tree because the end-effector failed to grasp either of them
successfully. This occurred on several occasions with simi-
lar fruit clusters. With higher-level logic for the picking se-
quence and end-effector approach vectors, all three apples
could be picked, and the cluster picking performance could
be improved to acceptable levels. With this proper sequenc-
ing, the lowest apple would be picked first, and the end-ef-
fector would then approach the other two apples from the
right and left, respectively. However, additional degrees of
freedom would be required to approach the fruit with a yaw
angle other than zero, which would negate the cost saving
gained from the reducing the system to 5-DOF.
Figure 6c shows another common difficulty in cluster
harvesting: an apple cluster has been misidentified, and three
apples are identified as just two. In this case, the robot is un-
likely to pick any of the three apples successfully because
the misidentified apples are not well aligned to any of the
actual apples. Even with improvements to the motion plan-
ning and picking sequence, these types of clusters will be
very difficult to harvest with a grasping end-effector because
of the partial occlusion of the apples, which makes identifi-
cation difficult. Full occlusions are less of a problem because
the system only sees the apple in the foreground, as in fig-
ure 6b. In this case, reimaging of the tree is necessary; oth-
erwise, the robot will harvest only the identified apple, while
the remaining occluded fruit are left on the tree.
Pendulum apples also posed a significant problem in suc-
cessfully separating them from the tree. Figure 7 shows two
pendulum apples with their long fruiting branches outlined
with boxes. An attempted solution to this problem was to
pitch the end-effector down from 0° to -90° before releasing
the apple. By doing this, the direction of the applied load on
the stem was altered, which could detach the apple. How-
Figure 7. Apples with long fruiting branches classified as pendulums.
412
ever, in practice, this method was not as effective as ex-
pected. Currently, the system does not identify apples that
might be pendulums, nor does it acquire any information
about the stem location or fruiting branch. Continuing re-
search on branch and obstacle detection could be helpful in
addressing this problem. Other possible solutions to this
problem are: (1) modifications to current pruning practices
to minimize the presence of long branches in tree canopies,
and (2) incorporating sensors into the end-effector with feed-
back control to help determine when an apple has been sep-
arated from the tree, to help the robot identify failures, and
to allow second attempts at missed apples. While the first
solution seems the simplest to implement, it may increase
labor costs, as time and labor are already spent on training
and pruning the trees along trellis wires. In contrast, embed-
ded end-effector sensors with real-time feedback have po-
tential to improve the robot’s efficiency beyond what could
be achieved with horticultural practices, including the ability
to sense picking failures, collisions, apple proximity, false
positives, etc.
CONCLUSION
Apples were successfully harvested using a low-cost ro-
botic system in a commercial orchard during the fall 2017
harvest. A detachment success rate on attempted apples of
67% was achieved, with an average time of 7.3 s per fruit
from separation to storage bin. By conducting this study in
an environment in which problematic apples had not been
removed to increase the detachment success rate, the current
pruning and thinning practices were also assessed to help lay
the foundation for future studies and develop strategies for
successfully harvesting apples that are difficult to detach.
The soft-robotic end-effector offers several improve-
ments over previous grasping end-effectors. The compliant
actuators were unharmed by collisions with the tree canopy
and trellis wires and offered a significant improvement in
grasping speed over previous tendon-driven fingers, closing
in several orders of magnitude less time (milliseconds versus
seconds) compared to the previous design. Clustered apples
caused some problems for both the vision system and the ro-
botic arm; however, future studies with high-level sequenc-
ing logic may improve the performance in picking apple
clusters.
ACKNOWLEDGEMENTS
This research was supported in part by USDA Hatch and
Multistate Project Funds (Accession Nos. 1005756 and
1001246), a USDA National Institute for Food and Agricul-
ture competitive grant (Accession No. 1000339), the Wash-
ington State University (WSU) Agricultural Research Cen-
ter (ARC), and the National Science Foundation (NSF)
Graduate Research Fellowship Program (NSF Grant No.
121477-005). Any opinions, findings, and conclusions ex-
pressed in this publication are those of the authors and do not
necessarily reflect the views of the USDA, NSF, or Wash-
ington State University. The authors would also like to
acknowledge Allan Brothers, Inc., for their support.
TRANSACTIONS OF THE ASABE
 challenge. Intl. J. Comput. Vision, 88(2), 303-338.
REFERENCES https://doi.org/10.1007/s11263-009-0275-4
Ahonen, T., Hadid, A., & Pietikainen, M. (2006). Face description
Gallardo, R. K., Taylor, M., & Hinman, H. (2010). 2009 Cost
with local binary patterns: Application to face recognition. IEEE
estimates of establishing and producing Gala apples in
Trans. Pattern Anal. Mach. Intel., 28(12), 2037-2041.
Washington. Factsheet FS005E. Pullman, WA: Washington
https://doi.org/10.1109/TPAMI.2006.244
State University Extension.
Amend, J. R., Brown, E., Rodenberg, N., Jaeger, H. M., & Lipson,
He, L., Fu, H., Karkee, M., & Zhang, Q. (2017). Effect of fruit
H. (2012). A positive-pressure universal gripper based on the
location on apple detachment with mechanical shaking. Biosyst.
jamming of granular material. IEEE Trans. Robotics, 28(2), 341-
Eng., 157, 63-71.
350. https://doi.org/10.1109/TRO.2011.2171093
https://doi.org/10.1016/j.biosystemseng.2017.02.009
Arikapudi, R., & Vougioukas, S. G. (2017). Linear reachability of
Ilievski, F., Mazzeo, A. D., Shepherd, R. F., Chen, X., &
fruits in orchard trees with actuator constraints. ASABE Paper
Whitesides, G. M. (2011). Soft robotics for chemists.
No. 1701421. St. Joseph, MI: ASABE.
Angewandte Chemie Intl. Ed., 50(8), 1890-1895.
https://doi.org/10.13031/aim.201701421
https://doi.org/10.1002/anie.201006464
Bac, C. W., Hemming, J., van Tuijl, B. A., Barth, R., Wais, E., &
Li, J., Karkee, M., Zhang, Q., Xiao, K., & Feng, T. (2016).
van Henten, E. J. (2017). Performance evaluation of a harvesting
Characterizing apple picking patterns for robotic harvesting.
robot for sweet pepper. J. Field Robotics, 34(6), 1123-1139.
Comput. Electron. Agric., 127, 633-640.
https://doi.org/10.1002/rob.21709
https://doi.org/10.1016/j.compag.2016.07.024
Bac, C. W., van Henten, E. J., Hemming, J., & Edan, Y. (2014).
Morales, R., Badesa, F. J., Garcia-Aracil, N., Sabater, J. M., &
Harvesting robots for high-value crops: State-of-the-art review
Zollo, L. (2014). Soft robotic manipulation of onions and
and challenges ahead. J. Field Robotics, 31(6), 888-911.
artichokes in the food industry. Adv. Mech. Eng., 6, article
https://doi.org/10.1002/rob.21525
345291. https://doi.org/10.1155/2014/345291
Blanes, C., Mellado, M., Ortiz, C., & Valera, A. (2011). Review:
Mosadegh, B., Polygerinos, P., Keplinger, C., Wennstedt, S.,
Technologies for robot grippers in pick and place operations for
Shepherd, R. F., Gupta, U., ... Whitesides, G. M. (2014).
fresh fruits and vegetables. Spanish J. Agric. Res., 9(4), 1130-
Pneumatic networks for soft robotics that actuate rapidly. Adv.
1141. https://doi.org/10.5424/sjar/20110904-501-10
Funct. Mater., 24(15), 2163-2170.
Bloch, V., Degani, A., & Bechar, A. (2018). A methodology of
https://doi.org/10.1002/adfm.201303288
orchard architecture design for an optimal harvesting robot.
Peterson, D. L. (2005). Harvest mechanization progress and
Biosyst. Eng., 166, 126-137.
prospects for fresh market quality deciduous tree fruits.
https://doi.org/10.1016/j.biosystemseng.2017.11.006
HortTech., 15(1), 72-75.
Brown, P., Rasch, M., & Dietrich, C. (2014). Vacuum apple
https://doi.org/10.21273/horttech.15.1.0072
harvester. Conklin, MI: Phil Brown Welding. Retrieved from
Rateni, G., Cianchetti, M., Ciuti, G., Menciassi, A., & Laschi, C.
http://www.philbrownwelding.com/index.php/new-products
(2015). Design and development of a soft robotic gripper for
Ceccarelli, M., Figliolini, G., Ottaviano, E., Mata, A. S., & Criado,
manipulation in minimally invasive surgery: A proof of concept.
E. J. (2000). Designing a robotic gripper for harvesting
Meccanica, 50(11), 2855-2863. https://doi.org/10.1007/s11012-
horticulture products. Robotica, 18(1), 105-111.
015-0261-6
https://doi.org/10.1017/S026357479900226X
Silwal, A., Davidson, J. R., Karkee, M., Mo, C., Zhang, Q., &
Davidson, J. R., & Mo, C. (2015). Mechanical design and initial
Lewis, K. (2017). Design, integration, and field evaluation of a
performance testing of an apple-picking end-effector. Proc.
robotic apple harvester. J. Field Robotics, 34(6), 1140-1159.
ASME Intl. Mechanical Engineering Congress and Exposition.
https://doi.org/10.1002/rob.21715
New York, NY: ASME. https://doi.org/10.1115/IMECE2015-
Sun, Y., Song, Y. S., & Paik, J. (2013). Characterization of silicone
50482
rubber based soft pneumatic actuators. Proc. IEEE/RSJ Intl.
Davidson, J. R., Hohimer, C. J., & Mo, C. (2016a). Preliminary
Conf. on Intel. Robots and Systems, (pp. 4446-4453).
design of a robotic system for catching and storing fresh market
Piscataway, NJ: IEEE.
apples. IFAC-PapersOnLine, 49(16), 149-154.
https://doi.org/10.1109/IROS.2013.6696995
https://doi.org/10.1016/j.ifacol.2016.10.028
Udupa, G., Sreedharan, P., Sai Dinesh, P., & Kim, D. (2014).
Davidson, J. R., Hohimer, C. J., Mo, C., & Karkee, M. (2017). Dual
Asymmetric bellow flexible pneumatic actuator for miniature
robot coordination for apple harvesting. ASABE Paper No.
robotic soft gripper. J. Robotics, 2014, article 902625.
1700567. St. Joseph, MI: ASABE.
https://doi.org/10.1155/2014/902625
https://doi.org/10.13031/aim.201700567
Viola, P., & Jones, M. (2001). Rapid object detection using a
Davidson, J., Silwal, A., Karkee, M., Mo, C., & Zhang, Q. (2016b).
boosted cascade of simple features. Proc. IEEE Computer
Hand-picking dynamic analysis for undersensed robotic apple
Society Conf. on Computer Vision and Pattern Recognition.
harvesting. Trans. ASABE, 59(4), 745-758.
Piscataway, NJ: IEEE.
https://doi.org/10.13031/trans.59.11669
https://doi.org/10.1109/CVPR.2001.990517
Davis, S., Gray, J. O., & Caldwell, D. G. (2008). An end effector
Vougioukas, S. G., Arikapudi, R., & Munic, J. (2016). A study of
based on the Bernoulli principle for handling sliced fruit and
fruit reachability in orchard trees by linear-only motion. IFAC-
vegetables. Robot. Comput. Integ, Manuf., 24(2), 249-257.
PapersOnLine, 49(16), 277-280.
https://doi.org/10.1016/j.rcim.2006.11.002
https://doi.org/10.1016/j.ifacol.2016.10.051
Dimeas, F., Sako, D. V., Moulianitis, V. C., & Aspragathos, N. A.
Wang, H., Hohimer, C. J., Bhusal, S., Karkee, M., Mo, C., &
(2015). Design and fuzzy control of a robotic gripper for
Miller, J. H. (2018). Simulation as a tool in designing and
efficient strawberry harvesting. Robotica, 33(5), 1085-1098.
evaluating a robotic apple harvesting system. IFAC-
https://doi.org/10.1017/S0263574714001155
PapersOnLine, 51(17), 135-140.
Erdoğan, D., Güner, M., Dursun, E., & Gezer, I. (2003). Mechanical
https://doi.org/10.1016/j.ifacol.2018.08.076
harvesting of apricots. Biosyst. Eng., 85(1), 19-28.
Wang, Z., Torigoe, Y., & Hirai, S. (2017). A prestressed soft
https://doi.org/10.1016/S1537-5110(03)00024-2
gripper: Design, modeling, fabrication, and tests for food
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., &
handling. IEEE Robot. Automat. Letters, 2(4), 1909-1916.
Zisserman, A. (2010). The Pascal visual object classes (VOC)

62(2): 405-414 413

 https://doi.org/10.1109/LRA.2017.2714141
Yap, H. K., Ng, H. Y., & Yeow, C.-H. (2016). High-force soft
printable pneumatics for soft robotic applications. Soft Robot.,
3(3), 144-158. https://doi.org/10.1089/soro.2016.0030
Zhang, Z., Heinemann, P. H., Liu, J., Baugher, T. A., & Schupp, J.
R. (2016a). The development of mechanical apple harvesting
414
technology: A review. Trans. ASABE, 59(5), 1165-1180.
https://doi.org/10.13031/trans.59.11737
Zhang, Z., Heinemann, P. H., Liu, J., Schupp, J. R., & Baugher, T.
A. (2016b). Design and field test of a low-cost apple harvest-
assist unit. Trans. ASABE, 59(5), 1149-1156.
https://doi.org/10.13031/trans.59.11708
TRANSACTIONS OF THE ASABE