Fish deboning requires

knowledge of anatomy
W.E. Friedrich , N.R. Spooner and P.K. Lim
Introduction In automated systems, a knowledge- sensor as it enters the camera
Natural products, such as food based approach can provide exper- enclosure. In order to keep the
products derived from animals and ience-based interpretation of pre- moisture level low the camera cabinet
plants, exhibit a high degree of processed sensor data. A combination is purged with air. The image is filtered
variation. Processing of these products of a sensor-equipped robot and a and processed using a priori
requires a high degree of system knowledge-based controller provides, knowledge and relevant information is
flexibility in order to cope with the therefore, a suitable basis for the extracted. The knowledge-based
object diversity. The required construction of a system capable of processing system receives simplified
flexibility may be provided by sensor- processing variable products. and reduced pixel-based information
equipped robotic systems coupled with This article describes the dev- and examines possible pin bone
effective processing capabilities. Key elopment of such a sensor-equipped locations in relation with its knowledge
factors for successful automation of robot system using knowledge-based base to generate the proposed cutting
these processes are: information processing techniques. co-ordinates (Figure 2). A set of robot
The prototype application describes the instructions is produced from these co-
• effective; removal of the lateral pin bones from ordinates which are transmitted to the
• sensing; fish fillets by means of water-jet robot controller in time for the fillet to
• interpretation; and cutting guided by a gantry robot. be cut. The pin bones are cut during
System details are presented describing transport via conveyor tracking. A
• control. some of the problems during develop- water-jet cutter is used as an end
Sensing of natural products is often ment and implementation. Initial effector of the gantry-based robot
difficult and involves complex proces- results are discussed followed by con- (Plate 1).
sing surface and lighting conditions siderations for shopfloor production, A PLC system running in the
with undesirable reflections and low including optimization of cycle times background is part of the motion
contrast. In automated systems there is and waste reduction. control system and co-ordinates
limited potential for enhancement camera and robot activation by
using traditional filtering techniques monitoring the detection sensors.
and pattern recognition methods. In the Implementation
process of eliminating vast amounts of The automation of fish deboning
irrelevant information there is the requires individuals cuts because of the Machine Software
vision stages
danger of eliminating useful data. In high variation in object size and station
many situations the remaining infor- geometry, particularly deviations of
mation is insufficient to determine shape and length of the pin bone line. Camera enclosure
required actions for the robotic system. The necessary tasks include sensing
CCD camera Robot station
Manual processing of these products individual fillets, data interpretation
often involves a high degree of tactile and cutting. These requirements are UV lighting Conveyor
and visual sensitivity combined with met by the prototype system using a Fish fillet
well-developed dexterity, skill and modular approach including sub-
experience. The human operator systems for sensing, knowledge-based Water jet
combines sensory inputs such as sight processing and motion control.
and touch in a continuous manner with Figure 1. Schematic system overview
his/her acquired articulated skills – Overview
thereby using the ability to interpret The task is to remove the lateral pine
sensory data in a qualitative manner, to bones from Tarakihi fillets, a species in
recognize patterns and to evaluate the Southern Pacific. The set-up of the Sensing Knowledge- Motion
module based control
these with previously acquired automated system consists of three module module
knowledge and experience. major subsystems: a sensing system, a
knowledge-based processing system
and a robot system. Fish fillets are
transported via industry-typical wire
This work was funded by the NZ
mesh conveyor through the complete
Foundation for Research Science and Camera and Operator Robot and
Technology. The authors wish to thank the processing system including imaging preprocessing experience process
Crop and Food Research Institute and station and cutting station. Figure 1 control
Sanfords Limited Seafood Processors and shows the principles.
Exporters for the use of their facilities Opto-electric sensors trigger the Figure 2. Software modules and their
during trials and for their helpful advice. capture of an image from the fillet correlation

18 Industrial Robot Vol. 22 No. 5, 1995, pp. 18-21 © MCB University Press, 0143-991X

Plate 1. System set up
Safety is assured by completely
isolating emergency stops from the
operation. In the event of actuation a
complete reset or cold start is required.
The system is PC-based with
separate interfaces for image
processing and motion control. Initial
development took advantage of the
independent parts. For each subsystem
a file format has been developed so that
results could be transferred via file
from one module to the next during the
development stage. This format has
later been used as a basis for the design
of the communication protocol. The
knowledge-based part is programmed
in Prolog while the remaining modules
are implemented using C. Interfacing
between the principally different
languages proved to be a challenge.
Sensor system
The sensor system has to extract
relevant information required by the
knowledge-based control system. The
machine vision part processes informa-
tion relevant to the determination of the
cutting path for a particular fillet. This
requires the capture of an image of the
fillet with its background followed by
the extraction of the pin bone co-
ordinates, fillet length, fillet height,
and hand (left or right side) of the fillet.
Fillets may originate from either the
left or right side of the fish.
A monochrome CCD camera with
shutter is employed. The fillet is
illuminated by ultraviolet light causing
the pin bones to fluoresce and
consequently increases the contrast
between bones and flesh. Separation of
fillet body and background is
reasonably straightforward. The raw
image is 512 × 512 pixels reduced after
filtering and area selection to 256 ×
256.
The outline of the fillet is identified
using a boundary detection algorithm
avoiding the thin lines of the stainless
steel conveyor mesh which are shown
in grey scale values similar to the fillet
body. The boundary parameters are
used to calculate the length, height,
side of the fillet[1] and to select the
area for pin bone extraction. A reduced
area where in bones are expected is
selected within the outline of the fillet.
This restriction of the image area
reduces processing time significantly,
and simplifies the identification of
pixels likely to represent pin bones,
while eliminating pixels unlikely to
represent pin bones. The area is
reduced to around 25 per cent of the
raw fillet area by this method.
Subsequent image processing is aimed
at finding distinct groups of pixels
representing expected pin bones.
A “fuzzy” Hough transform[2] is
used to find pixel groups within a
tolerance band about a straight line
pointing towards the tail end of the
fillet. It is known that the line of pin
bones in a fillet exhibit this character-
istic. The final stage of pin bone
location is to identify clusters of pixels
representing pin bones thereby
eliminating pixel groups which are
unlikely to be representing pin bones
for example those very close to the
fillet boundary. On completion of the
machine vision system operations,
perceived pin bone co-ordinates, fillet
length, fillet height and fillet side are
passed on to the knowledge-based
system part.
Knowledge-based processing system
The knowledge-based control system
comprises two main functions, namely
to use a priori knowledge of fillet
characteristics to enhance incomplete
knowledge and to determine the
cutting co-ordinates of the proposed
cutting path. The knowledge-based
control system examines actual fillet
information in the context of known
features. Compensation is made for
information missed by the sensor
system or information eliminated
during the data reduction process. In
our example application, rules have
been compiled from statistics gathered
from a large number of fillets of the
species being processed.
Pin bone information may be
incomplete towards the tail end or head
end of the bone line or between
observed bones. To achieve a
satisfactory cutting result is is impor-
tant to determine where the complete
bone line location is expected before
determining the cutting path. Rules
from the knowledge base determine
whether to extend the perceived bone
line or to add missing bone positions
between observed bones, taking into
consideration the fish species and
detected pin bone positions.
Each pin bone position is assigned a
certainty factor indicating the degree of
Plate 2. Typical processed image with esti-
mated pin bone locations and cutting line
uncertainty about the precise pin bone
position. Correctly perceived bones
have a full (100 per cent) certainty and
added bones have lower certainties. Pin
bone co-ordinates associated with
lesser certainty result in a wider cut
(Plate 2). Insufficient pin bone
information or information not
adequately fitting the acquired
knowledge of the particular species
results in a rejection of the fillet.
The cutting path is calculated by
adding predetermined offsets to the pin
bone co-ordinates. Top and bottom
offsets vary according to the perceived
location certainty, and to compensate
for the bone angle in the fillet flesh
distinguishing between right- and left-
hand fillets. The results of these
operations is a series of co-ordinates
representing a cutting path for the
robotic water-jet cutter. A typical path
is shown by the light line on Plate 1.
Note, that the path is wider towards the
tail end of the fish owing to greater
uncertainty of pin bone locations at this
part of the path.
Motion control system
The robot system is based on a 2D
gantry design driven by DC servo-
motors via ball screws. The robot has a
working envelope of 800mm ×
500mm. Despite its small size and light
loads (a water-jet cutting head imposes
a total load of less than 0.5kg) the
system provides a static load capacity
of 100kg. This primarily results from
high stiffness requirements for the
processing task but also results from a
need for a high durability and the
possibility of other higher load tasks in
the future. The robot motion is
synchronized with the conveyor
motion by tracking via shaft encoder.
The water-jet nozzle is the end effector
of the gantry robot (Plate 3).
In the robot motion module, cutting
path information from the expert
system is converted into a set of
instructions. The result of the
completed co-ordinate transformation
is passed to the robot control system in
the form of motion instructions based
IR Vol. 22 No. 5, 1995 19

Plate 3. Cutting in progress
on Cartesian co-ordinates. The
transport system is a conveyor con-
sisting of a stainless steel mesh belt
typically used in water-jet cutting
operations in the industry.
Operational factors
Alignment
In order to achieve good cutting results
three critical factors have emerged.
One of the main factors is the
mechanical alignment of camera,
transport system and robot cutter.
System tolerances are very critical and
affect subsequent processes. For
example, with the viewing area of
300mm width the machine vision
system provides a resolution of 1.67
pixels per mm requiring the camera to
be positioned within 0.2 degrees.
To avoid image distortion the
camera plane must be parallel to the
conveyor plane with similar stringent
requirements. An incorrectly adjusted
camera affects directly the path
location relative to the fish fillet.
The trigger sensor must be aligned
correctly to the conveyor belt. Any
tracking error during transport directly
offsets the proposed cutting path
vertically. Similarly, deviations in the
conveyor/robot synchronization affect
the path proportional to the distance
from the reference position of the
gantry robot. Also the absolute position
accuracy as well as the repeatability of
the robot is of prime importance. Our
aim was to keep the overall position
accuracy well below ±1mm. But even
tolerances in the order of 1mm present
a significant deviation considering a
typical cutting distance of 6-8mm.
Image quality
The second major factor is the quality
of the sensor information and its
consistency. A specific type of UV
light was used to reveal pin bones
which fluoresce in this light. Vital
factors to enhance the contrast include
considerations on illumination,
optimized lens filtering, camera
sensitivity in the UV light band and
20 Industrial Robot
appropriate shutter speed. Precise and
consistent algorithms are required to
preserve as much valid data as possible
and eliminate light reflections owing to
wet surface area.
Appropriate rules
Another factor is the quality of the
knowledge base. The bone line
estimation is based on average
statistical data from a large number of
fillets excluding extremes. A high
uncertainty is associated with the
length of the bone line and individual
bone positions since pin bone
information is often missing towards
the tail end of the bone line. This
uncertainty is compensated for by
cutting a wider than optimum area
from the fillet. Additional information
could be utilized to reduce waste.
Trial results
Initial cutting trials with more than 150
fillets resulted in 83 per cent correctly
positioned cuts, including all pin bones
in the cut out (Plate 4). Of these, 20 per
cent were acceptable, but the cut
portion may not have been clearly
separated from the fillet. Owing to
unacceptable variation in the cutting
path 12 per cent were rejected and 5 per
cent of the fillets were rejected by the
deboning system and not processed
because of uncertain bone locations.
The deviation of the cutting path is due
to expansion variation of the conveyor
mesh, misalignment between camera
and robotic cutter, inconsistent
registering of the passing fillet,
misintepretation of object features and
object movement during cutting. The
average weight lost by automatic
deboning amounted to around 12 per
cent compared to approximately 8 per
cent with manual operation. The
overall cycle time in our test series was
approximately seven seconds with 1.5
to 2 seconds for the image processing
operation, approximately 1 second for
the rules-based interpretation with an
average cutting time of 2.7 seconds
whereby 0.8 seconds were required to
Plate 4. Deboning result
reach the fillet and return to the
reference position.
Local sources estimated the current
cost for manual deboning to NZ
$500per tonne of fish. With a turnover
of a medium to large fish processing
plant of 200 tonnes per year, the
operating costs are in the order of NZ
$100,000 p.a. Assuming an investment
pay back of two years, the cost of such
a system should not exceed NZ
$200,000.
Considerations for
commercial development
The demonstration system operates at a
rate of around eight fillets per minute
with an 83 per cent success rate. Many
opportunities exist to improve cycle
time, cutting process and flexibility.
Cycle time optimization
The cutting operation is completely
independent from the image processing
operation. A second image may be
processed while the cutting operation
of the previous fillet is in progress.
This effectively halves the processing
cycle time. A further improvement
would be to introduce two cameras and
to feed fillets in two lanes, one for
right-hand sides and one for left-hand
sides. Positioned alternatively and
placed the same way up the pin bone
lines would face the centre.
Accordingly, the cutting station can
process fillets from a centre position
thereby reducing the positioning and
return time of the cutter head (Figure
3).
These improvements can be
achieved with the existing hardware
set-up. Once a cutting path is down-
loaded the motion controller completes
the cut independently. Employing a
faster processor for the image
processing operation significantly
reduces the processing time even
further and shorter fillet following
distances can be achieved. With the full
use of the water-jet cutting equipment,
fillets can be processed at a rate of
approximately 20 fillets per minute.
Process improvements
Minimizing waste is an important
factor for any commercial develop-
ments. In Tarakihi, the pin bones are
typically curved. In order to avoid
cutting through bones with a vertically-
oriented cutter a wider cut is necessary.
however, by inclining the cutter to a

Camera 1
Fillet
Camera 2
Moring
conveyor Frame
Figure 3. Proposed twin line deboning
predetermined angle a narrow path
closer to the pin bones can be followed.
The realization of this would involve
either a fixed angle arrangement or a
rotational axis.
Species extension
A feasibility study[3] conducted prior
to this project demonstrated that the
principle idea can be applied to a
variety of different fish species. This
flexibility can be accomplished with
the existing system by adapting to the
rules in the knowledge base, by
adjusting lighting and optimizing
sensor processing algorithms. The
modularity of the subsystems enables
the interchange of components for the
processing of different fish species. A
library of variety specific rules can be
created along with the required
processing pattern. These modules are
Gantry robot
Robot cutting path
loaded from a database to allow
multiple production runs with various
species.
Application flexibility
The system also provides the flexibility
in task organization permitting the
system to perform other processing
tasks such as portioning, removal of fat
or surface defects on meat and fish.
The application change would require
modifications to all modules but could
provide a user option.
Conclusion
A modular concept for the flexible
processing of individually shaped
objects has been introduced. A sensor-
equipped robot system has been
successfully applied to the deboning of
Tarakihi fillets. The system is con-
trolled by knowledge-based processing
techniques. The main modules are
based on sensing, knowledge-based
interpretation and motion control.
Experimental results using a robot
manipulated water-jet cutter are
encouraging. This application has
demonstrated that the system is a
practical solution for the processing of
naturally varying products. The system
configuration is flexible and offers
interchangeable modules and control
strategies. The generic concept
introduced applies to many intelligent
automation situations. The potential
for alternative applications are
outlined.
References
1. Batchelor, B.G., Hill, D.A. and
Hodgson, D.C., Automated Visual
Inspection, IFS Publications, Bedford,
1985.
2. Duda, R.O. and Hart, P.E., “Use of the
Hough transform to detect lines and
curves in pictures”, Communication of
the ACM, Vol. 15, 1972, pp. 11-5.
3. Nummonda, T., Carlson, M., Palmer, T.
and Andrews, M., “Using machine
vision in fish processing”, Proceedings
of the 5th New Zealand Image
W.E. Friedrich, N.R. Spooner and P.K. Lim
are based at Industrial Research Ltd,
Brooke House, 24 Balfour Road, Parnell,
PO Box 222S, Auckland, New Zealand.
Tel: +64 9 303 4116; Fax: +64 9 307 0618.
IR Vol. 22 No. 5, 1995 21