USO09554512B2

(12) United States Patent (10) Patent No.: US 9,554,512 B2

Davidson et al. (45) Date of Patent: Jan. 31, 2017

(54) ROBOTIC SYSTEMS, METHODS, AND (52) U.S. Cl.

END-EFFECTORS FOR HARVESTING CPC ................ A0ID 46/30 (2013.01); B25J 9/104
PRODUCE (2013.01); B25J 15/0009 (2013.01)
(58) Field of Classification Search
(71) Applicant: Washington State University, Pullman, CPC ...... A01D 46/24: A01D 46/30: A01D 46/005;
WA (US) B41J9/04; B41J9/46; B41J 19/021; B41J
19/023; B41J 5/007; B41J 15/10; B41J
(72) Inventors: Joseph Ryan Davidson, Somerville, USPC 99; 5.0s.
MA (US); Changki Mo, Richland, WA See application file for complete search history.
(US); Qin Zhang, Richland, WA (US);
Abhisesh Silwal, Prosser, WA (US); (56) References Cited
Manoj Karkee, Richland, WA (US) U.S. PATENT DOCUMENTS
(73) Assignee: Washington State University, Pullman, 4,519, 193 A * 5/1985 Yoshida ................. AO1D 46.24
WA (US) 348.89
4,663,925 A * 5/1987 Terada ................... AO1D 46.24
(*) Notice: Subject to any disclaimer, the term of this ck 382,153
patent is extended or adjusted under 35 4,718,223. A 1/1988 Suzuki ................... AopA.
U.S.C. 154(b) by 0 days. 2005, 0126144 A1* 6/2005 Koselka ................. AO1D 46/30
56,102 R
(21) Appl. No.: 14/849,729
* cited by examiner
(22) Filed: Sep. 10, 2015 Primary Examiner — Robert Pezzuto
O O (74) Attorney, Agent, or Firm — Whitham, Curtis &
(65) Prior Publication Data Cook, P.C.
US 2016/OO73584 A1 Mar. 17, 2016 (57) ABSTRACT
Robotic systems and specialized end-effectors provide for
O O automated harvesting of produce Such as fresh market
Related U.S. Application Data apples. An underactuated design using tendons and flexure
(60) Provisional application No. 62/050,048, filed on Sep. joints with passive compliance increases robustness to posi
12, 2014. tion error, overcoming a significant limitation of previous
fruit harvesting end-effectors. Some devices use open-loop
(51) Int. Cl. control, provide a shape-adaptive grasp, and produce contact
AOID 46/24 (2006.01) forces similar to those used during optimal hand picking
AOID 46/30 (2006.01) patterns. Other benefits include relatively low weight, low

B25, 5/00 (2006.01) 7 Claims, 11 Drawing Sheets

O)

MACHN O3
CBCOFiOON
WSON.SYSEM SYSTEM

MANIPULAOR
TS18
101

END-EFFECTOR 107

POWER SORAGE
O4
U.S. Patent Jan. 31, 2017 Sheet 1 of 11 US 9,554,512 B2

103
MACHINE OCOMOON
WSION SYSTEM SYSTEM

102
MANPULATOR 106
101

105 ENO-EFFECTOR to

POWER SORAGE
O4.

Figure 1
202

-(- 10

Figure 2
U.S. Patent Jan. 31, 2017 Sheet 2 of 11 US 9,554,512 B2

304 303 302 306

U.S. Patent Jan. 31, 2017 Sheet 3 of 11 US 9,554,512 B2

3O6

503
Figure 5
U.S. Patent Jan. 31, 2017 Sheet 4 of 11 US 9,554,512 B2

Figure 6B Figure 6A
702

10

Figure 7
U.S. Patent Jan. 31, 2017 Sheet S of 11 US 9,554,512 B2

Figure 8
U.S. Patent Jan. 31, 2017 Sheet 6 of 11 US 9,554,512 B2

900

MACH NEWSON SYSTEM DEERMINES

OCAON OF PECE OF PRODUCE

MANPUATOR GUIDES END-EFFECTOR

902 O ARGE LOCATON

END-EFFECTOR POSTONING
903 TERMINAED

PRIMARY FNGER SECONOARY

904 CLOSURE ABOU FENGER CLOSURE 905
PRODUCE ABOUT SEM

MANPUAOR ROTATION AND

906 RERACING FORCE FROM OCAON

907 SORAGE OF PECE OF PRODUCE

Figure 9
U.S. Patent Jan. 31, 2017 Sheet 7 of 11 US 9,554,512 B2

1OOO

Acquire image & localize 1001

fruit

Motion planning algorithm Send arm to picking position

Find nearest apple

Apple in <
t- reachable x

*...*s -

Approach fruit along azimuth

angle

Pick & drop fruit - 1008

..Y
- s
* \ 1009
^ Any
- unpicked ^
apples *
Y remaining? --

Figure 10
U.S. Patent Jan. 31, 2017 Sheet 8 of 11 US 9,554,512 B2

88:

888

88:

88:
$33. . . . . . . . - * * * * * * * * *. . . . . . . . . . . . . . . . . . . . . .‘. . . . . . . . . . . . . . . . . . . . . creww.ew > - - - - - - - - - - . . . . . . . . . .. . . . . . . . . . .* . . . . . . . . .
8.88, 383 -8.8: 383 & 88: 888 8.83 8.3: 888. 8:8

Figure 11
U.S. Patent Jan. 31, 2017 Sheet 9 of 11 US 9,554,512 B2

$$$

3 &x=4
* *x

&# #
3*8.9-
U.S. Patent Jan. 31, 2017 Sheet 10 of 11 US 9,554,512 B2

8s.

&
4.

SY.
U.S. Patent Jan. 31, 2017 Sheet 11 of 11 US 9,554,512 B2

•
>

&

e«…*. ¿

wer"
(N) a.o.
 US 9,554,512 B2
1. 2
ROBOTIC SYSTEMS, METHODS, AND level. Despite numerous attempts to transfer industrial
END-EFFECTORS FOR HARVESTING robotic technology directly to field based, biologically
PRODUCE driven environments, the mechanization of specialty crop
harvesting has achieved only limited Success primarily due
CROSS-REFERENCE TO RELATED to inadequate accuracy, speed, and robustness.
APPLICATIONS Fruit in a single crop possess a high level of variability.
For example, tree fruit vary in position, shape, size, and
This application claims the benefit of U.S. Provisional growing orientation. Even for the same apple cultivar,
Patent Application No. 62/050,048, filed Sep. 12, 2014, the 10
parameters such as size and stem length vary widely within
complete contents of which are herein incorporated by a single tree. There also exists a year-to-year variability in
reference. these parameters. Fruit removal technique is usually the
STATEMENT REGARDING FEDERALLY
largest cause of fruit injury. Insufficient automated devices
SPONSORED RESEARCH OR DEVELOPMENT exist for fruit harvesting which are able to accommodate
15 these requirements.
This invention was made with government Support under Because of rising labor costs, a high workplace injury rate
Contract No. 2013-67021-20942 awarded by the United due to ladder use, and increasing uncertainty about the
States Department of Agriculture-National Institute of Food availability of farm labor, the lack of mechanical harvesting
and Agriculture (USDA-NIFA) through the National Robot is a critical problem receiving much attention from both
ics Initiative (NRI). The government has certain rights in the federal agencies (e.g., United States Department of Agricul
invention. ture) and state and local organizations (e.g., Washington
Tree Fruit Research Commission).
FIELD OF THE INVENTION The basic functional requirements of an apple picking
end-effector are to approach and reach for the fruit and then
The invention generally relates to robotic produce har 25 detach an unblemished apple from the tree. In addition to
vesting and, more particularly, end effectors and Supporting being efficient, productive and economically feasible, it is
systems for robotically harvesting apples and other produce important that the system not damage the picked fruit,
while preferably maintaining harvest quality sufficient for adjacent fruit, or the tree. The end-effector can damage the
fresh market sales. apple by applying excessive force during picking or by
30 employing inappropriate stem separation techniques. Some
BACKGROUND different techniques have been investigated for end-effector
designs.
In the U.S. Pacific Northwest, a large, seasonal labor Bulanon and Kataoka (Bulanon & Kataoka, 2010)
force is required for the production of tree fruit crops like designed an end-effector that used a peduncle holder to
fresh market apples, cherries, and pears. The most time and 35 apply pressure against the peduncle before removing the
labor-intensive task in fruit crop production is harvesting. In fruit with a lifting and twisting motion. Though this tech
Washington State alone the apple and pear harvest requires nique minimized damage to the fruit, the system was con
the employment of approximately 30,000 additional workers strained in that the end-effector had to approach the apple
with an estimated harvest cost of $1,100 to $2,100 USD per horizontally.
acre per year. To reduce harvesting costs and dependence on 40 Baeten et al. (Baeten, Donne, Boedrij. Beckers, &
seasonal labor, researchers have developed shake-and-catch Claesen, 2008) developed a novel gripper consisting of a
systems for the mass harvesting of fruits such as berries, flexible silicon funnel that used vacuum suction to activate
cherries, and citrus. These techniques, which apply vibration the gripping function. During field tests the average harvest
to the trunk or branch of the tree in order to separate the fruit, ing time was approximately nine seconds, but stem pulls
are typically used to harvest fruit destined for the processing 45 occurred with approximately 30% of the harvested apples. It
market where there are established tolerances for fruit was also important to sequence apple selection so that
bruising and external defects. There have been some adjoining apples in a cluster would not interfere with the
attempts to develop mass harvesting systems for fresh picking process.
market citrus, cherries, and apples, but the systems demon Zhao et al. (Zhao, Lu, Ji, Zhang, & Chen, 2011) proposed
strated marginal rates of fruit detachment, were only effi 50 a cutting end-effector utilizing multiple sensors that dem
cient with compatible tree-training systems, or frequently onstrated impressive fruit detachment rates during field
harvested fruit without stems. tests. Although cutting minimizes the likelihood of stem
The use of robotics technology is another approach pulls, it usually requires more complex control require
researchers have tried for the harvesting of tree fruit. For ments, which can lead to higher costs.
economic reasons related to changing labor conditions, 55
Scientists and engineers started to actively work on research SUMMARY
and development of fruit-picking robots in the 1980s. These
earlier research efforts defined the basic functional require According to an aspect of the invention, Solutions are
ments of a fruit-picking robot as the following: i) locate the provided to address one or more of the following objectives:
fruit on the tree in 3D dimensions; ii) approach and reach for 60 1. Achieve average harvesting time, which is defined as
the fruit; and iii) detach an undamaged fruit from the tree the time from fruit localization to storage in a container,
and deposit it in a container. In order for a fruit-picking of six seconds or less.
robotic system to be commercially viable, it has to be 2. Replicate the kinematics and dynamics of human
economically feasible and provide harvesting rates (e.g. picking to minimize fruit damage and stem pulls.
fruit/second) comparable to those obtained through manual 65 3. Minimize the volume of the end-effector workspace in
harvesting. Additionally, the system should minimize dam order to reduce the likelihood of collisions with adja
age to both the plant and the harvested fruit to a tolerable cent fruit and branches.
 US 9,554,512 B2
3 4
4. A system not constrained to approaching the fruit from forces, which allow the stem to break away from the tree
a single direction. branch without damage to the branch or the fruit.
5. Adaptable to harvesting of multiple apple cultivars with
variable, distinct geometries. BRIEF DESCRIPTION OF THE DRAWINGS
6. Lightweight, simple, low-cost, and robust to an agri
cultural environment. FIG. 1 is a block diagram of a robotic harvesting system.
According to an aspect of the invention, underactuated FIG. 2 is a manipulator and end-effector of a robotic
end-effectors are presented that are made for the robotic harvesting system.
harvesting of produce, especially tree fruit Such as apples. FIG. 3 is a robotic end-effector with the end-effector in an
The device is optimized for speed, low complexity, suitabil 10 open configuration.
ity for a highly variable field environment, and the replica FIG. 4 is a top view of the robotic end-effector of FIG. 3.
tion of manual hand picking so as to minimize fruit damage. FIG. 5 is sectional view of a primary finger of the
end-effector in FIG. 3.
In some embodiments, the end-effector produces a spherical FIGS. 6A and 6B show, respectively, an example optimal
power grasp with a normal force distribution and picking 15 grasp configuration with an end-effector and a grasp con
sequence replicating selected human patterns. figuration used during manual apple picking.
According to another aspect of the invention, an under FIG. 7 is the robotic end-effector of FIG. 3 with the
actuated, tendon-driven end-effector with compliant flexure end-effector in a closed configuration grasping an object.
joints is provided to improve system performance in the FIG. 8 shows the kinematic structure of an exemplary
presence of position errors as well as enhance robustness to manipulator.
variable fruit size, shape, and orientation. In some embodi FIG. 9 is a method for harvesting produce with a robotic
ments, the end-effector has few or no sensors (e.g., for harvesting system.
detecting angular positions or points of contact). An exem FIG. 10 is a process for harvesting from a tree with a
plary end-effector incorporates open-loop control to reduce robotic harvesting system.
complexity and improve picking speed. Examples are pre 25 FIG. 11 shows a simulated example grasp configuration
sented which determine the normal forces developed during from a Matlab solver used to determine angles of link
grasping of the apple. Results indicate that open-loop, rotation and points of normal contact for various fruit
feedforward control can be used to produce optimal normal positions.
force patterns. FIG. 12 shows the normalized proximal force that devel
Whereas advanced robotic hands designed to perform 30 ops at static-equilibrium for a single finger. The X-y coor
manipulation tasks with high dexterity are quite complex, dinate represents the position of the center of a circle with
a radius of 40 mm.
underactuated hands are much simpler devices that can FIG. 13 shows proximal and distal normal forces com
perform a human-like grasp compliant to the object without pared at five different actuator loads for each finger. The data
requiring independent actuation of each joint. An underac 35 points represent the mean values of three different iterations
tuated device is beneficial because only grasping of the fruit and the error bars present standard deviation.
is needed the end-effector does not require the capability FIG. 14 shows the proximal link normal forces that
to perform dexterous manipulation. A tendon-driven device develop during a power grasp of a sphere with a diameter of
gives a transmission system that is lightweight, relatively 80 mm. The normal forces are plotted versus the actuator
simple, and enables an adaptive grasp of multiple apple 40 load.
cultivars.
According to another aspect of the invention, a robotic DETAILED DESCRIPTION
system integrates a manipulator, end-effector, and machine
vision system during fruit harvesting. The manipulator may FIG. 1 is a block diagram of a robotic harvesting system
be a serial link manipulator, for example. The manipulator 45 100 for use in orchards, for example. In some embodiments,
and end-effector have a low-cost design. The sequence of the robotic harvesting system 100 is for apple harvesting.
picking motions used by the robot replicate the dynamics of The primary moving components include an end-effector
human apple picking. 101, for grasping apples and removing them from the tree,
According to another aspect of the invention, a method of and a manipulator 102, for moving the end-effector as a
autonomous robotic harvesting of fruit is described which 50 whole through space to reach different parts of a tree. In
includes approaching a piece of fruit that is nearest to and addition, an exemplary system 100 includes a machine
reachable by a robotic end-effector, the approach being vision system 103 for determining target locations of apples
made along an azimuth angle that provides a direct to which the manipulator 102 can then move the end-effector
approach: grasping the piece of fruit with a first set offingers 101. The system 100 also generally includes support ele
(i.e., primary fingers, described below) and, separately,
55 ments such as a storage container 104 in which harvested
apples are collected, a power source 105, a locomotion
grasping the stem of the piece of fruit with a second set of system 106 for moving the entire system 100 from tree to
fingers (i.e., secondary fingers, described below); and pick tree, and other Support elements generally shown by block
ing and dropping the piece of fruit. The first set of fingers 107, including, for example, a control interface, wireless
use, for example, a power grasp. The second set of fingers 60 antenna for remote control, and the like.
use, for example, a pinch grasp. Significantly, to reduce or FIG. 2 shows an example manipulator 102 and end
eliminate bruising of the fruit and/or damaging the tree effector 101. The manipulator 102 has a kinematic frame
branch, it is advantageous to rotate the piece of fruit (e.g., work flexible enough to accommodate the crop environ
through a pendulum motion). In some embodiments, this is ment. Agriculture fields and orchards present an
performed simultaneously with retracting the robotic end 65 environment that is considerably more unstructured than
effector away from the branch to which the piece of fruit is other areas of automation like manufacturing plants. Envi
attached. The rotation bends the fruit stem, causing shear ronmental factors include variable outdoor conditions, com
 US 9,554,512 B2
5 6
plex plant structures, inconsistency in product shape and FIG. 5 shows an enlarged sectional view of a primary
size, and delicate products. With respect to apple picking, a finger 302 based on a bisection along the finger's plane of
primary source of variation with which the manipulator is symmetry. The primary finger 302 is illustrated in an open
configured to accommodate is the highly irregular and configuration. In the open configuration, a non-Zero angle
unstructured apple tree. 5 exists between the proximal and distal links. In each of the
To improve obstacle avoidance (e.g., branches, tree primary fingers 302 a tendon 309 is attached (e.g., tied) at
trunks, other pieces of fruit) during harvesting, a manipu the distal tip. Leading away from the distal tip, the tendon
lator 102 preferably has six degrees of freedom (DOF). In 309 follows a hollow channel through the distal link303 and
Some exemplary embodiments, the manipulator 102 is an then through the proximal link304. The tendon 309 is routed
open chain, serial link manipulator with revolute joints. This 10 in Such a way the friction is minimalized. For example, the
configuration is one that advantageously offers a spherical tendon 309 may be routed over two small dowel pins 502 in
workspace. Different embodiments may have different the proximal link 304 in order to reduce friction. There is a
maximum reach. For example, a suitable reach is at least 0.5 single, free-spinning pulley 503 at the base of each primary
meters (e.g., 0.5, 0.6, 0.7, 0.8, 0.9, or 1.0 meters). The finger 302 that guides the tendon 309 to its termination point
manipulator 102 includes actuators 202 such as modular 15 on a disc differential 310 where the tendon is secured (e.g.,
Dynamixel Pro actuators (Robotic Inc., Irvine, Calif.). The with a nut). The disc differential 310 is attached with a
frame 203 can be fabricated from, for example, aluminum tendon to a horn of the actuator 313. In a preferred embodi
sheetmetal, which is lightweight, relatively inexpensive, and ment, all of the primary fingers 302 are controlled with the
sufficiently durable for fruit orchard conditions. single actuator 313. Actuator 313 together with the disc
FIGS. 3 and 4 show enlarged depictions of an end-effector differential 310 actuates all of the primary fingers 302
101. An end-effector 101 is a tendon-driven, underactuated together. As an alternative, one or more additional actuators
gripping device. An individual end-effector 101 includes a may be provided to allow independent actuation of one or a
plurality of fingers (also referred to interchangeably as subset of the primary fingers 302. However, this generally
“digits). The exemplary embodiment illustrated in the adds undesirable cost and complexity without meaningful
figures includes three primary fingers 302 each with two 25 gains in overall functionality and performance of the end
links (or phalanges) connected by flexure joints. The distal effector 101.
link 303 connects to the proximal link 304 with a distal Underactuation of the end-effector 101 is supported by the
flexure joint 305, and the proximal link 304 additionally disc differential 310. The disc differential 310 is a variant of
connects to the palm 306 with a proximal flexure joint 307. a seesaw mechanism and is essentially a circular disc made
Underactuation and the passive compliance provided by the 30 of for example, a thin plastic. Each of the three tendons 309
flexure joints provide several advantages in the unstructured is secured at the edge of the disc differential 310 of the
orchard environment. For example, underactuation between differential, e.g., with a nut. With the primary fingers 302 are
the finger links 303 and 304 helps to ensure a shape-adaptive configured symmetrically around the palm 306, the termi
grasp of fruits with variable shapes, sizes, and orientations. nation points of the tendons are arranged symmetrically
The passive compliance of the flexure joints 305 and 307 35 around the disc 310. In the event that one primary finger 302
also increases robustness to positioning errors. In the case of contacts the fruit before the other primary fingers 302, the
unintended collisions which are expected during harvesting, disc differential 310 rotates and enables further displacement
the flexure joints 305 and 307 can sustain out-of-plane of the two tendons of the remaining two primary fingers 302
deflection and large deflections without damage. that have not yet made contact with the piece of fruit.
The palm 306 is defined as a base to which to the fingers 40 In some embodiments, an end-effector 101 includes one
302 are attached. In some embodiments, the palm serves to or more (e.g., at least two) secondary fingers 311 for
fix the proximal end of each primary finger 302 to a specific applying pressure against the stem during fruit detachment.
location with respect to the proximal ends of the other The secondary fingers 311 are controlled by a separate
primary fingers 302. According to an exemplary configura actuator 314 from the primary fingers. In contrast to the
tion, the primary fingers 302 are arranged symmetrically 45 primary fingers 302, the secondary fingers 311 preferably
around the palm 306. For example, for an end-effector 101 each have only a single link. Each secondary finger link
with three primary fingers 302, each primary finger is spaced includes a padding 308 (e.g., a soft rubber pad) for making
apart from the other two primary fingers by 60 degrees (e.g., contact with the fruit stem. In example embodiments, the
as measured using the center of palm 306 as a center point). secondary fingers 311 do not make contact with the fruit. A
Alternative embodiments may have more than three primary 50 pair of secondary fingers actuate in a direction of one
fingers (e.g., 4 or 5 primary fingers); however, three primary another but not in a direction of the palm's center. In an open
fingers are preferred for providing satisfactory performance position, a non-Zero angle formed between the secondary
without undue cost and complexity. The primary fingers 302 fingers (30 degrees, for example, as illustrated in FIGS. 3
are spaced Such that when grasping a sphere of average and 4). The secondary fingers 311 are in the open position
apple diameter (e.g., 80 mm), the proximal links 304 make 55 when the manipulator 102 is positioning the end-effector
contact with the fruit on its equator. Alternative embodi 101 around a piece of fruit. The secondary fingers 311 switch
ments intended for harvesting produce of other sizes (e.g., to the closed position in parallel with the closing of the
tomatoes, lemons, grapefruits, oranges, etc.) may be con primary fingers 302. After the secondary fingers 311 are
structed with a similar configuration to end-effector 101 but moved into the closed position they are in contact with and
with component sizes scaled larger or Smaller based on the 60 put pressure on the fruit stem. The change from the open to
intended produce to be harvested. An exemplary length of a closed configurations of a pair of secondary fingers 311 is
single primary finger 302 is roughly equivalent to the length characterizable as a pinch grasp, not unlike the motion of a
of an adult male human index finger. Each link of each finger human pinch using the forefinger and thumb (particularly
includes a padding 308 (e.g., a soft rubber pad) for contact when the forefinger and thumb are each kept straight). A
ing the piece of fruit. The paddings 308 increase friction and 65 floating pulley 316 provides underactuation between the
tangential forces between the end-effector 101 and fruit secondary fingers 311, each of which is connected to the
Surface. palm 306 with a flexure joint 315. Each secondary finger 311
 US 9,554,512 B2
7 8
has a single tendon 312. The tendon 312 in each secondary have multiple contact points with the gripped element; each
finger 311 is secured at the link tips and passes over a link or phalange has at least one contact point).
floating pulley. The floating pulley 316 is attached with a FIG. 6A shows an example of an optimal robotic pattern
tendon to a horn of the actuator 314. In an example embodi of grasping an object Such as an apple. The pattern illustrated
ment, the actuator 314 is the only actuator provided for 5
is an example configuration optimized for an end-effector
actuating all of the secondary fingers 311. with three primary fingers as studied during field experi
In some embodiments, the palm 306 is circular and may ments. This pattern minimized grasping normal forces and
also be concave. A feature of the palm 306 is the provision reduced bruising compared to other tested patterns. Note
of a padding 401 (e.g., a soft rubber insert) arranged thereon. 10 that this example pattern does not include force being
The padding 401 serves as an allowable point of contact applied against the stem. For comparison, FIG. 6B shows a
during a grasping task. The manipulator 102 moves the typical grasp pattern used by human laborers during manual
end-effector 101 to a piece of fruit (e.g., an apple) with an apple picking. The harvesting end-effector is the only system
open configuration. Ideally, the end-effector is Switched to
the closed configuration after contact is made between the component that makes physical contact with the fruit. In
15
padding 401 of the palm 306 and the piece of fruit. This order to minimize damage and improve detachment Success
helps ensure the piece of fruit is centrally positioned among rates, some embodiments include a design based on apple
the primary fingers 302. In some embodiments, the palm 306 growth habits and the human hands manipulation methods
may include a pressure sensor 402 (shown Schematically in during fruit picking. Based on qualitative observations, the
FIG. 4) for detecting when contact is made between the typical pattern used by professional apple pickers is a
padding 401 and the piece of fruit. In other embodiments, a two-finger grasp with opposing fingertips placed along the
grasp does not require contact between the palm 306 and the fruits equator. To separate the fruit from the branch, the
piece of fruit. The grasp can be performed and the detach hand moves the fruit in a pendulum motion while the index
ment of the fruit completed without palm contact. finger pushes the stem, as illustrated by FIG. 6B. Using force
An exemplary robotic system 100 meets the following 25 sensors, a field experiment was performed to track and
five performance criteria: i) detachment success of at least quantify normal forces during three other grasp configura
90% (detachment success is defined as the number of tions achievable by an end-effector with three opposing
successfully harvested ripe fruit per total number of local primary fingers. The best grasp configuration of the three
ized ripe fruit present in the workspace of a manipulator tested is illustrated in FIG. 6A. None of the three grasps
30 included the application of pressure against the stem.
102); ii) picking time of 6 seconds or less (picking time is
defined as the time required to pick and store one piece of Results from these configurations were then compared with
fruit excluding the time required for ripeness determination the manual picking pattern. Pulling while simultaneously
and fruit localization); iii) damage rate to the fruit at 10% or rotating the fruit (e.g., see arrow 601 in FIG. 6A), and
less; iv) usable to harvest multiple apple cultivars; and v) 35
thereby bending the stem, produces a combined pulling and
relatively lightweight, simple, and cost effective. pendulum motion. This induces shear forces in the stem. In
An end-effector 101 accounts for the variability from one Summary, experimental results show that the application of
piece of fruit to the next. The configuration of the end only a pulling force along the axis of the stem produces a
effector 101 takes into consideration the fruits physical purely tensile force in the stem and that the normal grasping
properties such as friction, firmness, and tensile strength of 40 forces required to detach the fruit usually exceed bruising
the stem. Performance of an end-effector 101 can be mea thresholds. To break the stem-abscission joint with lower
sured according to the above-identified criteria. grasping forces that minimize bruising, bending the stem is
According to some embodiments, an end-effector 101 generally required so as to induce shear forces.
replicates the manipulation methods of the human hand FIG. 7 shows an end-effector 101 grasping a substantially
during apple picking and addresses the constraints imposed 45 spherical object 702 such as an apple. In some embodiments
by fruit growth habits. An optimal pattern of fruit removal the grasp of the end-effector 101 is characterizable as a
with the human hand is to grasp the piece of fruit (e.g., a passive, adaptive grasp. An adaptive grasp describes the
single apple) with the thumb and middle finger at opposite process whereby the fingers conform to the shape of the
points on the equator, place the forefinger against the base of object being grasped. For the end-effector 101 this process
the stem, and rotate the fruit against the orientation of the 50 is passive because shape conformity during grasping does
stem. Compared to pulling the fruit away from the tree, this not require additional actuating components that provide
method required less force to break the stem-abscission joint energy to the system. Dexterous manipulation of the fruit is
and reduced the likelihood of fruit damage. Though the not required in Some embodiments. Embodiments without
human grasp is a usually a fingertip grasp, an exemplary 55
dexterous manipulation may provide a normal force distri
end-effector 101 provides an enveloping power grasp of the bution replicating human patterns at the proximal point of
fruit. The power grasp may be characterized as being COntact.
spherical. A power grasp advantageously enhances robust Visual sensing is an essential and primary task for an
ness to position error caused by the machine vision system autonomous robotic harvesting system. However, vision is
103, for example. In a fingertip grasp, contact is made 60 often considered a bottleneck for developing commercially
between the gripping element (e.g., a human hand or robotic applicable robotic harvesting systems. Variable lighting con
end-effector) and the gripped element (e.g., a piece of fruit, ditions, fruit clustering, and occlusion are some of the
an apple) only at the most distal link or phalange, and often significant challenges that limit the performance of the
more specifically the most distal tip thereof. In contrast, a machine vision system in an orchard environment. Prior to
power grasp includes contact between the gripping element 65 harvesting, the robotic system needs to identify and accu
and the gripped element at multiple links or phalanges of rately locate the fruit. According to an example configura
each of one or more fingers (e.g., Some or all of the fingers tion for a robotic harvesting system 100 (FIG. 1), the
 US 9,554,512 B2
9 10
machine vision algorithm developed by Silwal et al. (Silwal, TABLE 1.
A., Gongal, A., Karkee, M., 2014. Apple Identification in
Kinematic configuration of the 6-DOF manipulator.
Field Environment with Over-the-Row Machine Vision Sys
tem. Agricultural Engineering International: Agric Eng Int’l Link Length Twist Angle Offset Length Rotation Angle
(CIGR Journal), 16(4): 66-75) is used by the machine vision Joint a (m) a (degrees) d (m) 0 (degrees)
system 103 to identify the apples. In brief, this algorithm 1 O 90 177 0.
uses Circular Hough transformation (CHT) to identify 2
3
O
O
90
90
O
270
0.
0.
clearly visible fruit, as well as individual apples in clusters, 4 O 90 O 0.
and blob analysis (BA) to identify partially visible fruit. It 10 5 O 90 219 05
has been previously tested in an orchard environment with 6 186 90 O 0.
90% fruit identification accuracy. The physical hardware of
the machine visions system 103 includes a global camera Multiple options for the fabrication of a robotics system
system which comprises, for example, a single CCD and end-effectors according to the invention exist and will
15
(Charged Couple Device) color camera (Prosilica occur to those of skill in the art based on the teachings
GC 1290C, Allied Vision Technologies, Exton, Pa.) mounted herein. For Small scale production purposes, end-effector
on top of a time-of-flight based three-dimensional (3D) components can be manufactured with 3D printing technol
camera (Camcube 3.0, PMD Technologies, Sigen, Ger ogy (see Example 2 below).
many). The purpose of this configuration is to acquire color FIG. 9 is a flowchart for a method 900 for harvesting
images with the CCD camera to identify the apples and then produce (e.g., apples) with a system (e.g., system 100) and
obtain their 3D coordinates from the PMD camera to local end-effector (e.g., end-effector 101) according to the inven
ize their position in space. Unlike other fruit harvesting tion. The method 900 is an example for picking an individual
vision systems that attach a camera to the manipulator or piece of produce and is repeated iteratively for picking
end-effector, the use of a single set of global cameras doesn’t 25 multiple pieces of produce from a single tree.
require computationally expensive visual servoing tech The machine vision system first determines a location of
niques that may constrain manipulation speeds. The a piece of produce to be picked (block 901). This location is
supplied to the manipulator which guides the end-effector to
machine vision system 103 is required only once, namely at this location (block 902). The end-effector is maintained in
the beginning of each harvest cycle to identify and localize 30 an open configuration while being moved into position
the apples, thereby saving time and increasing the efficiency around the piece of produce. The end-effector positioning is
of the entire harvesting system 100. terminated after the produce is centrally positioned within
FIG. 8 and Table 1 show the kinematic structure of a the end-effector (block 903). This may include the palm of
manipulator 102 according to an example embodiment. The the end-effector making contact with the piece of produce.
figure and table also show the kinematic structure's associ 35 In Some systems a pressure sensor in the palm of the
ated geometric parameters. The recursion formulas devel end-effector determines when this contact is made. At this
oped by Wang and Ravani (L. T. Wang and B. Ravani, point the primary fingers of the end-effector close about the
“Recursive Computations of Kinematic and Dynamic Equa piece of produce (block 904) and, for a system that includes
tions for Mechanical Manipulators.” IEEE Journal of Robot secondary fingers, these are closed to grip and apply pres
40 sure to the stem of the produce (block 905). The manipulator
ics and Automation, Vols. RA-1, no. 3, pp. 124-131, 1985) then pulls the end-effector away from the initial target
are used for forward kinematics computations. Numerical location to remove the grasped produce from the plant
Solutions to the inverse kinematics problem are determined (block 906). For apples, for example, this action may be a
with the combined optimization method first proposed by rotation and retraction from the initial produce location. The
Wang and Chen (L. T. Wang and C. C. Chen, “A Combined 45 manipulator then moves the end-effector to the storage
Optimization Method for Solving the Inverse Kinematics container where the piece of produce is released and stored
Problem of Mechanical Manipulators.” IEEE Transactions (block 907). The procedure 900 is then repeated for another
on Robotics and Automation, vol. 7, no. 4, pp. 489-499, piece of produce.
August 1991). This algorithm was selected because it has FIG. 10 shows a flow diagram that is an example process
been shown to be computationally efficient and does not
50 1000 for harvesting/picking a single tree in an orchard with
require matrix inversion. The convergence tolerance was set an end-effector 101. The end-effector balances optimal pick
ing dynamics with simplicity and reduced cost. After the
at 1E-6, and the joint limits of the manipulator were used as machine vision system 103 localizes the position of every
boundary constraints. The inverse kinematics algorithm has apple within its field of view (block 1001), the manipulator
been developed in Matlab (Mathworks Inc., Natick, Mass.), 55 102 guides the end-effector to an approach point a fixed
compiled into a C++ shared library, and integrated with the distance (e.g., 10 cm) away from the nearest fruit (block
manipulator's controller in the Microsoft Visual Studio 1003) in conjunction with a motion planning algorithm
development environment. In some embodiments, a (block 1002). When multiple apples are in close proximity
manipulator's planned trajectory is executed using a simple, (e.g., in a cluster), the system 103 directs the end-effector to
open loop look-and-move approach. A manipulator 102 is 60 the closest remaining apple first (block 1004). A decision is
preferably configured for use with traditional orchards (e.g., made whether the apple is in a reachable workspace (block
with naturally shaped apple trees) as well as simple, narrow, 1005). If not, then the system finds the next nearest apple. If
reachable, then, following a specific azimuth angle, the
accessible, and productive (SNAP) canopies. A SNAP end-effector 101 makes a horizontal approach along a direct
canopy is a two-dimensional, planer canopy Supported by a 65 path to the fruits position (block 1006). The azimuth angle
wire and posts trellis system whereby most of the branches is the angle that the manipulator must rotate horizontally for
and fruit are visible and accessible to machines. the end-effector to approach the fruit along a straight line.
 US 9,554,512 B2
11 12
So, for a fruit located directly in front of the manipulator the represents the normal contact force on the proximal and
azimuth angle is zero. When the end-effector reaches the distal links, and f, is the actuator force. For this two-link
predetermined fruit position, the primary fingers and sec mechanism the contact Jacobian J is
ondary fingers simultaneously close around the fruit and
stem, respectively (block 1007). After a pause (e.g., approxi 5
mately one second), the end-effector is then rotated through f ( b (3)
a predetermined angle (e.g., 30 degrees, counterclockwise) T b + lycos0, b,
and retracted along the same approach path to a fixed
distance (e.g., 13 cm) away from the tree. The end-effector 10 where b is the proximal force location, b is the distal force
is then opened and the fruit is dropped into a storage location, and 1 is the proximal link length. The normal
container (block 1008). The manipulator then moves the forces can then be found by
end-effector to the next fruit approach point and repeats the
picking process until all fruit are removed from the tree
(decision block 1009). 15 In most grasps the proximal normal force will be located on
Example 1 or about the fruits equator.
The actuation force that produces the normal forces of
To reduce design complexity and enhance speed of har human picking patterns is determined. In reality, this is a
vesting, an example end-effector is provided which has no complicated process because the normal forces are highly
pressure sensors and utilizes open-loop, feedforward con dependent upon the final kinematic configuration of the
trol. An environmental model is provided for characterizing underactuated finger. During harvesting operations position
the actuation torque required to produce the desired link errors and variation in fruit shape and size will lead to
normal forces. Each end-effector finger is a single-acting numerous end-effector grasp configurations. Likewise, in
cable-driven system with two links and two flexure joints. Some configurations negative normal forces may develop,
The flexures are modeled as simple pin joints with rotational 25 which indicates loss of contact of the respective link. The
stiffness. For this example, which does not use sensors to actuation force input provided with open-loop control
detect angular positions or points of contact, modeling the should ensure that in all possible configurations the final
flexures as pin joints with torsion springs is sufficient. grasp does not damage the fruit. A Matlab simulation
However, if desired, a more accurate model of flexure (Mathworks Inc., Natick, Mass.) was conducted to examine
bending is given Odhner and Dollar (L.U. Odhner and A. M. 30 the effect of position error on the proximal normal force. The
Dollar, “The Smooth Curvature Model: An Efficient Repre center of a circle with diameter of 80 mm was placed at
sentation of Euler-Bernoulli Flexures as Robot Joints,’ different positions in the x-y plane having an error ranging
from -10 to +10 mm in the X-direction and 0 to 10 mm in
IEEE Transactions on Robotics, vol. 28, no. 4, pp. 761-772,
2012) and can be used to estimate beam deflection. Rotation the y-direction. The grid included a total of 400 different
of the proximal and distal links is coupled until the proximal 35 fruit position points. At each point it was assumed that the
link makes contact with the object at which point the distal fruit was constrained by the stein/branch system. The simu
link will continue to independently rotate until the system is lation used a solver to estimate the configuration where the
constrained. For this cable-driven finger the kinematics of finger links were tangent to the circle and then determined
the links are coupled and may be expressed as the change in link positions A0 and the points of contact b
40 and b assuming a single point of contact on each link. An
rA6=JA0 (1) example of an equilibrium grasp configuration calculated by
the solver is shown in FIG. 11. The resulting proximal
where r is the radius of the actuator pulley, A0 is the normal force was then determined with eqn. (4). Tangential
angular displacement of the actuator pulley, AO represents forces due to friction were not considered in this example.
the configuration change of the links, and J. (r. ra) is the 45 The presence of tangential forces should increase the pullout
actuator Jacobian of the finger. The pulley radii are repre force required to remove the fruit from the end-effector's
sented by r and r. The quasi-static equation of equilibrium, grasp. The results of the simulation are shown in FIG. 12.
which can be found with analytical mechanics and the The force data, which is scaled with unity-based normaliza
principle of virtual work, is tion, shows that for a constant actuation force the proximal
50 normal force is greater for increasing A0. As shown in FIG.
10, the X-position of the finger base is located at -0.04 m.
where For increasing values of X the fruit moves away from the
finger base and the proximal link must rotate further in order
to contact the fruit. By Superimposing force data from two
55 planar fingers in an opposed grasp, it is possible to create
vector fields for the total resultant force on an object. By
Superimposing the data from each of the three fingers the
greatest resultant force can be expected near the center of the
represents the joint stiffness of the flexures, Je R* is the end-effector at a small distance away from the palm.
contact Jacobian that maps between contact forces acting on 60
the finger and the joint torques, Example 2
An example end-effector was used to characterize the
relationship between actuation force and normal contact
I-C) 65 forces. Normal forces were measured during a grasp of a
plastic sphere with radius of 40 mm. The sphere was located
symmetrically with respect to the end-effector, meaning the
 US 9,554,512 B2
13 14
centerline of the end-effector was coincident with the center DC motor, reduction gearhead, PID controller, motor driver,
of the sphere resting on the palm. Three piezoresistive force and data network. These particular models have stall torques
sensors (TekScan Inc., Boston, Mass.) were attached to the of 7.3 N-m (MX-64AR) and 2.5 N-m (MX-28AR). Because
contact locations on the proximal links. To complete a power the operating voltage of the end-effector servos is 12 VDC,
grasp the end-effector's actuator, which is nonbackdrive it used a separate power Supply than the manipulator, which
able, was operated in torque mode and driven to its stall operates at 24 VDC. The servos are controlled in the
point. The normal force was then measured at static equi Microsoft Visual Studio C++ development environment
librium. This measurement was repeated for increasing using the software development kit (SDK) provided by the
actuator loads. The experimental set-up corresponds with manufacturer.
the configuration illustrated in FIG. 7. The same test was 10
The mass of the assembled end-effector in Example 2 was
repeated for the distal normal forces. The results for the 0.4 kg. Based on the average mass of the apples harvested,
normal force measurements at five different actuator loads
are shown in FIG. 13. As expected, for increasing tendon the total payload of the end-effector and harvested fruit is
tension the proximal normal force is significantly higher generally less than 0.7 kg. Some key geometric and physical
than the distal force. While the change in proximal force is 15
parameters of the end-effector fingers are provided in Table
relatively linear for increasing tendon tension, the change in 2. The orthogonal distance from the midline of the flexure to
distal force is highly irregular and shows sharp jumps the tendon entry point is used for the equivalent pulley radii.
interspersed with horizontal slopes. The proximal links The stiffness of the flexure joints was experimentally deter
remain static once contact with the Surface is made. The mined. The joints were modeled as simple torsion springs,
and their rotational stiffness was measured with a load cell.
distal links, however, often adjusted their equilibrium con The stiffness ratio k/k of two between the joints plays an
figurations at new actuator loads. FIG. 14 shows a compari influential role in the nature of the coupled motion between
son of the proximal normal forces measured for the three the links. The arrangement of the fingers is designed to
fingers simultaneously at ten different actuator loads. The provide a spherical power grasp fully encompassing the
results present the mean values from six different iterations. fruit. In this grasping sequence the proximal link makes
For this particular grasp approximately 10% of the actua 25
contact with the object first before the distal link flexes to
tor's maximum torque value was required to produce proxi cage the fruit. In order to ensure this two-phased motion, the
mal normal forces representative of those developed during distal flexure joint is stiffer than the proximal flexure joint.
manual picking of apples, which is approximately 7 N. Individual features or multiple features from the Example 2
Based on the simulation results presented in FIG. 12, the end-effector may be incorporated into various alternative
contact forces developed with this actuation torque for 30
embodiments in the practice of the invention. This Exam
asymmetric grasp configurations should remain below the ple's end-effector is illustrative only and is not necessarily
11 N force threshold that caused bruising during field tests intended to be limiting.
of hand-picking patterns. Though the normal force distribu
tion was similar for each of the fingers, the grasp was not TABLE 2
force-isotropic as would be expected for a symmetric grasp. 35
Because normal forces are highly configuration dependent, Physical and geometric parameters
slight variations in the sensor placement can significantly of the end-effector of Example 2.
impact the final results. Also, when increasing tendon ten Length Joint Stiffness Pulley Resting
sion the point of proximal contact sometimes slightly Link I (m) k (N-mrad) Radius r (m) Angle 0 ()
adjusted. The change in position of normal contact relative 40
to the sensor may be a source of Some of the variance 1
2
O.O7
O.042
0.055
O. 111
O.OO956
O.OO716
45
35
between the proximal force values.
The components of the end-effector in Example 2 were
fabricated from solid models by a Replicator 2X printer While the descriptions herein have largely referred to the
(MakerBot Industries, New York). Additive manufacturing 45 harvest of fruit and especially apples, the descriptions pre
minimizes the fasteners required for assembly and lessens sented are of non-limiting examples. Embodiments of the
the total weight of the device. The fingers were printed as invention may be used for harvesting any of a variety of
monolithic parts of ABS plastic. Molds for the finger pads types of produce, be it vegetables or fruit. Exemplary
were included in the solid parts as thin shells. A soft urethane end-effectors according to the teachings herein are espe
rubber (Vytaflex 30, Smooth-On, Inc.) was poured into both 50 cially well suited for substantially spherical produce such as
the finger pad cavities and the palm. After the elastomers oranges and grapefruit. Pears, lemons, limes, and other fruit
cured, the shells of the finger pad molds were cut away. or vegetables may also be harvested with variations of the
Dovetail joints included on the fingers and palm hold the example embodiments disclosed herein.
rubber pads in place. The flexures for the primary and While exemplary embodiments of the present invention
secondary fingers joints were printed with flexible filament 55 have been disclosed herein, one skilled in the art will
(Ninjaflex, Fenner Drives, Inc.) and inserted directly into recognize that various changes and modifications may be
cavities in the links. Multiple high-strength low-friction made without departing from the scope of the invention as
filaments will occur to those of skill in the art for use as the defined by the following claims.
tendons. As an inexpensive option, high strength fishing line
(e.g., 100-lb) was used in the Example. The actuator pulleys, 60 Symbols
floating pulley, and differential were also printed parts. The
actuators selected for the primary and secondary fingers are 1, 1 Length of proximal and distal links
the Dynamixel MX-64AR (Robotis Inc., Irvine, Calif.) and k. k. Stiffness of proximal and distal flexures
MX-28AR, respectively. It is advantageous to employ a K Stiffness matrix
servos providing the capability of position, speed, and 65 r, r Equivalent pulley radii of proximal and distal joints
torque-based commands. Dynamixel servos include a 4.096 r. Actuator pulley radius
step absolute encoder after gear reduction and integrate a A0, Angular displacement of the actuator pulley
 US 9,554,512 B2
15 16
A0 Configuration change of the links 3. The system of claim 1, wherein said end-effector has a
J Actuator Jacobian matrix of the finger passive, adaptive grasp.
f Vector of normal contact forces 4. The system of claim 1, wherein the end-effector further
J. Contact Jacobian matrix that maps between link contact comprises a single actuator and a disc differential for simul
forces and joint torques taneously actuating all of the primary fingers together.
b1, b2 Location of proximal and distal normal forces 5. The system of claim 1, wherein the end-effector utilizes
We claim:
1. A robotic produce harvesting system, comprising: open-loop, feed forward control.
an end-effector comprising 6. The system of claim 1, wherein the primary fingers are
a palm; and 10
arranged symmetrically about the palm.
at least three primary fingers which are tendon-driven, 7. A method of autonomous robotic harvesting of fruit,
wherein said end-effector is underactuated, and comprising
wherein the robotic end-effector is configured to grasp approaching a piece of fruit that is nearest to and reach
a piece of produce to be harvested with a power able by a robotic end-effector, said approach being
grasp. 15
made along an azimuth angle;
a manipulator for positioning the end-effector in three grasping the piece of fruit with a first set of fingers; and
dimensional space; and picking and dropping the piece of fruit,
a machine vision system for providing a location of the wherein said approaching step includes determining,
piece of produce to be harvested. using a machine vision system, a location of the piece
2. The system of claim 1, wherein planned trajectories of 20 of fruit that is nearest to and reachable by the robotic
the manipulator are executed using a look-and-move end-effector.
approach.