Overview of Image-based 3D Vision Systems for
Agricultural Applications
Abhipray Paturkar, Gourab Sen Gupta, and Donald Bailey
School of Engineering and Advanced Technology,
Massey University, New Zealand
Correspondence: {A.Paturkar, G.SenGupta, D.G.Bailey}@massey.ac.nz
Abstract—Various agricultural robots exist which are intended
to help farmers to make decisions. Vision systems are an integral
and essential part of these agricultural robots to perceive and
evaluate the work space, inspect crops and detect objects. The
use of 3D vision systems can provide a 3D reconstructed model
which will supply detailed information including depth estimation
about a crop or an object or environmental structures. However,
these vision systems have some drawbacks which are limiting
their use in an agricultural area. The limiting factors are dealing
with lighting conditions, navigating unstructured environment,
harvesting speed, occlusion and overlapping of fruits, accuracy
of the system etc. This paper investigates the limitations and
challenges of the state-of-the-art of 3D image reconstruction tech-
niques in agricultural applications. Active and passive approaches
have been reviewed. Factors to be considered while designing the
vision system have been identified.
Index Terms—Computer vision; 3D vision system; Agricultural
automation; Precision Agriculture; Reconstruction techniques
I. I NTRODUCTION
Precision agriculture is all about collecting information
about the crop and its environmental conditions, including
soil to help farmers in the process of decision making [1].
As agriculture is becoming increasingly modernised, the ap-
plication of vision based robots and intelligent machines is
helping farmers and researchers in many ways. These robots
have helped to reduce production and labour costs, and at the
same time, it has enhanced environmental control and helped
to improve the quality of agricultural products [2]. Perceptive
agricultural robots are designed to conduct various agricultural
operations such as cultivating, transplanting, harvesting, spray-
ing, and trimming. Vision based robots are intended to sense
the unstructured agricultural environment by different sensors
e.g. cameras, lidar etc. and use all this information to perform
the required operations [3]. Surely, vision based systems have
a phenomenal future but currently, the use of an agricultural
robot is inadequate to completely emulate manual operation
due to some limitations of their vision systems. The success
of agricultural robotics depends heavily on the performance
of the vision systems.
Vision systems exploit visual information to help an agri-
cultural robot to roam autonomously in the uncertain, unstruc-
tured and changing environment. Vision systems for agricul-
tural robots are implemented to resolve some of the challenges
such as:
978-1-5386-4276-4/17/$31.00
2017
c IEEE
• Handle natural objects with a huge variety in shape,
colour, size, texture, and solidity.
• Navigate a complicated and barely structured workspace
with substantial variations in illumination and a high
degree of occlusion.
• Identify randomly located object or fruit which demands
operating in three-dimensions
Vision based systems are an appealing approach to fulfill
these challenges. One-dimensional (1D) and two-dimensional
(2D) vision systems are already successful in the food produc-
tion chain [4]. It is strongly believed that machine vision is at
a cusp of making deep inroads into agricultural automation.
Enhanced technology combined with low sensor prices have
influenced researchers to move into three-dimensional (3D)
approaches. In the last decade, tremendous work has been done
in agricultural 3D vision systems. There are two 3D modelling
techniques, rule based and image based [5], but most of
the work has been done in image based techniques because
of its realistic input (capturing images) and fitting operation
property. As reconstruction techniques and algorithms are
developing, state-of-the-art techniques are evolving rapidly.
In this paper, we review a range of image-based 3D vision
systems in the context of agriculture for indoor and outdoor
applications. After reviewing these systems, we investigate the
limitations and challenges present in these image-based vision
systems. We compare active and passive approaches. We also
examine which methods are useful to reduce the challenges
identified above. The rest of the paper is organized as follows;
Section II discusses image-based 3D vision systems along with
their various applications in an agricultural context and their
limitations. Section III discusses challenges and limitations
of existing techniques. Future directions to overcome the
limitations are outlined in Section IV.
II. 3D VISION SYSTEMS
Image-based 3D vision systems broadly fall into two cat-
egories. The various techniques and their classification are
depicted in Fig. 1. Image-based techniques use real world
data to model the complete 3D structure of plants or objects,
mostly based on methods developed by computer vision re-
searchers. While image-based techniques have made tremen-
dous progress in achieving accurate models, constructing an
exact representation is still a challenging research problem.
 Fig. 1. Classification of 3D vision system
Image based vision system is mainly divided into active and
passive approaches:
A. Active Approaches
Lidar is basically an extension of the principles employed in
radar technology. It calculates the distance between the scanner
and the target object by illuminating the object using a laser
and measuring the time taken for the reflected light to return
[6] . There are two main ways in which lidar is used. Firstly,
airborne lidar in which a scanning system is attached to an
airplane and secondly, terrestrial laser scanning, which is used
for ground based vehicles.
James Underwood et al. [7] presented a terrestrial laser
scanning system to gather information for almond orchard
mapping and per tree yield calculation. 580 trees were mea-
sured at three different times during the harvest season over
approximately two years. The data collection process in the
first year was affected by lens-flare and saturation as a result
of sun-light (illumination). In the second year, they tried to
keep the sun always behind the camera, which led to a time
consuming data collection process as they needed to scan half
of the row in the morning heading west and remaining half
row in the afternoon facing east. During pre-harvest and once
the foliage developed, the fruits became occluded and many
almonds were hidden while scanning. Since, each side of a
tree is scanned individually; misalignment occurs in between
two halves due to GPS drift. The variation in fruit size was
one of the main challenges during this experiment as the small
size almonds are more difficult to discern than larger almonds.
During data processing, the image processing part took around
two days including human and computer time as the dataset
was collected for 2.3 hectare orchard.The lidar processing took
less than 5 minutes to segment all the trees in the 2.3 hectares.
Use of structured light is an alternative method for depth
estimation. In this method, the light source (either visible or
near-infrared) is offset a known distance from an imaging
device. The light from the emitter is reflected into the camera
by the target scene. Knowledge of the light pattern enables
the depth to be inferred through triangulation. The Microsoft
Kinect is possibly the most common example of a structured
light device which transmits a near-infrared dot pattern [8].
This sensor has an advantage that its software (Microsoft’s
KinectFusion) permits depth information from multiple views
to be fused into a single image model. Unfortunately, Kinect
sensors go through certain limitations when applied in outdoor
scenarios. It can be hard to use in day light. Greatly reflective
leaf surfaces could also work as mirrors i.e. reflecting a heavy
proportion of the emitted pattern elsewhere from the imaging
device which making it difficult to detect.
Zhang et al. [9] proposed a conveyor based quality grading
system mainly used for apples. This system detects the defects
in apples by considering its stem-end and calyx because it is
difficult to differentiate between calyx and defect. Sometimes
it is quite confusing for human eyes to differentiate between
calyx and a defect in just one glance while selecting an apple.
Hence, it is important to have a system which will differentiate
between calyx and defect easily. This grading system has a
limitation of poor calibration in between projector and camera
which leads to false depth information and because of this the
accuracy of the system is low. This system takes on an average
around 80ms for each image by using industrial computer. The
false acceptance rate is 3.8% and in contrast, the false rejection
 TABLE I
C OMPARATIVE ANALYSIS OF ACTIVE APPROACHES
Advantages
-Able to get depth information in dim light.
Lidar -Robust against interference.
-Robust against sunlight.
-Excellent accuracy.
-Low cost system.
Structured light -High speed of 3D capture, up to 30 fps.
-Useful in animal inspection applications.
-Directly measure in 3D at very high spatial
resolution.
rate is 0.95%Ṁoreover, this system is unable to work for real
time fruit grading system in industries.
Zhang et al. [10] in their apple grading system on a con-
veyor belt, used a single CCD camera combined with a near-
infrared linear-array structured light for 3D reconstruction.
To identify the apple stem and calyx they only considered
the upper surface. For structured light systems, the distance
between the CCD camera and laser projector should be fixed
(or at least known). This system finding the stem-end and calyx
depends only on the detection of a concavity. The limitation
of this system is, it can only be used to monitor the upper
half of the apple which should be visible at the time of
inspection. This system has overcome several restrictions and
disadvantages of existing technologies with excellent accuracy
of about 97.5% and robustness.
Rosell-Polo et al. [11] describes the use of a Kinect sen-
sor applied to two different areas: precision agriculture and
precision livestock farming. They have experimented with the
sensors in different seasonal conditions. They found that the
Kinect sensor is not efficient at detecting small or complex
targets in daylight conditions and for this reason they have con-
ducted their outdoor experiments during sunset and evening.
Some experiments for detection of trunks and branches re-
ported the minimal noticeable branch diameter of around 6
mm and this limitation is responsible for floating fruits or
leaves in the resultant 3D model. For livestock farming, they
carried experiments in indoor environments where the lights
are structured. The animals are moving during inspection and
this movement has been considered as it is difficult to derive
an accurate 3D model if there is any movement. To monitor
the growth and body condition of the animals, they conducted
experiments for four months. This study shows that structured
light based Kinect sensor is efficient for indoor applications
only.
Overall, lidar provides better results for variation in illumi-
nation and has good accuracy for 3D reconstruction as com-
pared to the structured light system. A detailed comparative
analysis of active approaches is given in Table I.
B. Passive Approaches
In spite of the fact that lidar could be powerful, it needs
costly equipment which makes the overall system expensive.
Disadvantages
-Bulky with moving parts.
-Costly for setup.
-Problems under adverse weather conditions (rain, dust).
-Weak performance in edge detection due to the spacing
between the light beams.
-Highly sensitive to natural light.
-Small field-of-view.
-Difficulty to deal with shiny surfaces
-Limited sensing range
Passive approaches are therefore getting tremendous attention,
as these approaches only need a standard off-the-shelf digital
camera for capturing overlapped images which are then pro-
cessed by computer to calculate depth or 3D structure. Passive
approaches use radiation present in the scene. Stereo vision
and structure-from-motion (SFM) are the most commonly used
approaches.
Stereo vision has three main processing stages: camera
calibration, feature extraction, and correspondence matching.
A stereo camera captures a pair of images (right and left).
By using this stereo pair, the disparity can be calculated to
get the camera coordinates of matching points in the scene.
Structure-from-Motion follows a similar process to stereo
vision. Nevertheless, whereas stereo vision captures two or
more images concurrently, SFM captures images serially from
different view points as the camera moves. 3D data is then
estimated either by matching pairs of images or globally
matching features between all the images [12].
A mobile platform might not have knowledge of its en-
vironment; it needs to perform operations in an unstructured
surrounding, which is a potential problem. Francisco et al. [13]
fused the output from a stereo vision system and navigation
system for precision agriculture. This system consists of
a stereo camera, GPS sensor and inertial calculation unit.
For each stereo pair, the GPS sensor calculates the global
coordinates of the position of the camera. As there are a
number of GPS satellites it prevents fusing of images with
wrong coordinates. Image noise is the biggest challenge while
creating local maps as the quality of local maps can be affected
by the disparity value. If there is any mismatch between two
images, for example as a result of limited texture within the
scene, occlusion, or poor illumination it might end up with
determining wrong coordinates. These unfiltered images are
pixels with erroneous stereo data which leads to meaningless
locations. Synchronization between the moving speed of the
mobile platform and the processing speed of the system is
very critical. If the processing is not fast enough, data may be
lost.
Xiang et al. [14] investigated recognition of clustered fruit
on trees using stereo vision. In this system, depth information
of clustered tomatoes was obtained from the stereo vision. For
a cluster, most of the back tomatoes are occluded by the front
 TABLE II
C OMPARATIVE ANALYSIS OF PASSIVE APPROACHES

Stereo vision
Structure-from-motion
Advantages
-Off-the-shelf cameras could be used.
-Provides efficient RGB stream.
-Robust for controlled environment.
-Cost effective.
-Easy implementation.
-Hand held cameras could be used.
-Suitable for any application.
-Useful in animal inspection applications.
-Cost effective.
Disadvantages
-Correspondence problem because of low texture.
-Depth range is depends upon the baseline distance.
-Not effective in sunlight because of reflection.
-Computationally costly.
-Correspondence problem due to poor illumination
-Generation of point cloud is time consuming.
-Image overlapping is an issue.
-Difficulty to deal with shiny surfaces
-Intrinsic and extrinsic camera parameters are needed
to calculate.

tomatoes and front tomatoes prevent the manipulator from
reaching the back tomatoes because of incorrect recognition
of tomatoes in this cluster. Noise in the depth map is another
issue as a few pixels with unusual depth values because of false
matching leads to wrong recognition. Although this method is
useful for recognition of clustered tomatoes, the accuracy of
this method is not adequate if the occlusion is serious. There
is still scope for improvement of the accuracy if a fruit is
occluded by leaves.
Si et al. [15] propose a vision system which is able to
autonomously locate apples in trees by using a stereo camera.
Colour and shape based analysis are two important methods
for fruit recognition. Colour based analysis fails if there is a
shadow. Edge extraction should be performed before feature
extraction; once an apple is identified then branches and leaves
are recognised to create unwanted noise points. So in this case
it is difficult to handle a high degree of occlusion. Feature
extraction of an apple is mostly based on the contour. A Hough
transform is commonly used for this but it is time consuming.
The accuracy of the proposed system was not satisfactory and
there is further space for improvement.
Kaczmarek [16], [17] presents a method to improve the
calculation of distance between an autonomous harvesting
robot and a tree with ripe fruits using five cameras. This system
called equal baseline multiple camera set (EBMCS). For this
set of five cameras, calibration and rectification is a difficult
task to do because every side camera has to be calibrated
with the reference (centre) camera. This has the potential to
create confusion and introduce errors in the system. Ground
truth is prepared for every set of images. Ground truth points
correspond to points of the central image and are important
in this scenario because in the neighbourhood of every edge
of the centre image there must be an area which is not
visible in one of the side images. Parameters such as ground
truth, camera calibration and rectification are important aspects
while considering stereo vision system using multiple cameras.
Exceptions Excluding Merging Method (EEMM) is responsi-
ble to enhance the quality of disparity maps by combining
maps obtained from individual stereo cameras incorporated in
EBMCS. There are several limiting factors in this system e.g.
wide range of real disparities in the set of images and selecting
size of windows to used in the stereo matching algorithm. The
result of this method has shown a significant reduction in error
distance estimation by 26.55% in comparison with a single
stereo camera.
Quan et al. [18] and [19] proposed a semi-automatic method
for modelling the plants completely from images. This method
is based on structure-from-motion. Although this method is
easy to use it has some limitations. Due to a large amount
of leaves on plants and trees, it is difficult to reconstruct
each leaf perfectly. In such case the user has to manually
manipulate data to obtain a realistic 3D model. Moreover,
sometimes it is not possible to scan a plant or tree in 360
degrees because of neighbouring trees and this might lead to an
improper 3D model. Furthermore, this method requires more
effort to reconstruct unusual structural modifications because
of disease. There is more loss of data due to occlusion which
provides difficulties while creating the 3D model. So, there
are still some problems with this method which need to be
solved. This method has provided some really good results
which are motivating its use in other applications like 3D face
reconstruction.
Jay et al. [20] proposed a method which builds a 3D
model of a crop row to get the plant structural parameters.
This 3D model is acquired using structure-from-motion with
the help of colour images captured by translating a single
camera along the row. In this method, they have considered
the plants are in a linear row structure to avoid complexities
of the process. In the actual scenario, some parameters make
some difference like ground geometry, illumination, occlusion
and most importantly scanning speed. These parameters must
be considered for implementation. In addition, the scanning
process of plants is totally manual so there might be some error
while scanning as well. These limitations are common in these
types of systems and should be considered when developing
an error-free system.
In consequence, passive techniques generally perform well
in spite of some challenges. The detailed comparative analysis
of passive approaches is given in Table II.
III. CURRENT CHALLENGES AND RESEARCH
DIRECTIONS
From the literature, passive approaches appear to be the
most promising approach for 3D vision because of the lower
 TABLE III
LIST OF CHALLENGES IN STEREO VISION AND STRUCTURE - FROM - MOTION

Stereo vision
Structure-from-motion
Challenges
- Field of view is limited.
-Struggles with reflections.
-Occlusion handling.
-Extra light is required for dim light implementations.
-Shadows are responsible for poor depth estimation.
-Baseline should be maintained.
-Camera position is important.
-Parallax and correspondence problem.
-Necessity of camera calibration.
-Occlusion handling.
-Object movement during capture process.
-High computation is required.
-Image overlapping is an issue.
-Poor in dim light.

cost. However, there are a few challenges which need to be techniques.

addressed such as:
1) Illumination effects: For a stereo vision, the matching
problem is complicated by differences in illumination
from different view points, and poorly textured surfaces
[14]. Because of variation in lighting conditions, it is
difficult to get good quality results.
2) Poor hand-eye coordination: Lack of robust image
processing algorithms is responsible for weak visual in-
formation which is important for the robotic manipulator
to reach the fruit.
3) On-board processing: Current vision systems work
off-line as the system collects images from a camera
for processing later. This limits commercialisation of
agricultural robotics.
4) Fruit or obstacle recognition: The accuracy of fruit
recognition is an important aspect. Current vision sys-
tems use inefficient fruit recognition algorithms which
are responsible for inaccurate results.
5) Time consuming algorithms: Current techniques use
time consuming algorithms which take approximately
20-30 mins to process the data and if the data is huge,
it might take around a day
6) Occlusion handling: Because of the colour similarity
between fruit and leaves and because of the clustering
of fruit, it is difficult to fetch the properly ripened fruit.
This is a daunting task and a much-studied aspect of
automated harvesting, but still a long way to go to get
proper efficiency to handle occlusion.
7) Human-Computer combination: Most of the methods
in vision system are semi-automatic which needs user
involvement to manipulate data to get the proper result.
Although all these aforementioned techniques are challeng-
ing in outdoor conditions, experiments in controlled conditions
have given some promising results. Nonetheless, all these
challenges are provide future research directions to resolve
the issues with efficient solutions. Table III shows some
more challenges in stereo vision and structure-frommotion
IV. CONCLUSION
Computer vision has been studied for more than 50 years,
and in the last 5 years, tremendous work has been focused in
the context of agriculture. Passive approaches have gained a
lot of attention despite their limitations as these approaches are
reliable and less costly than active approaches. In this paper,
we have investigated some of the different approaches used
for image-based 3D vision. It is clear that, although image-
based approaches are able to produce good results, they still
need human interaction. A fully automated system is clearly
a necessity. In addition, parameters like occlusion, scanning
speed, identification of fruit, location of an obstacle or fruit
make a difference in real time implementations and these must
be considered while designing a system. Illumination problems
can be minimised if the system uses lidar, as the review
shows that lidar systems are robust against sunlight and able
to retrieve depth data in dim light as well. Occlusion handling
requires more efficient and effective algorithms. Efficient
algorithms, advanced computing and reduction in hardware
costs are needed to attract more researchers to this area and to
achieve breakthrough. So, until then semi-automatic systems
must be used. We have reviewed active and passive approaches
and have identified the various challenges and limitations of
image based 3D vision in an agricultural context. While much
progress has been made, there is clearly scope for significant
improvement.
R EFERENCES
[1] S. Konam, “Agricultural aid for mango cutting AAM,” in International
Conference on Advances in Computing, Communications and Informat-
ics (ICACCI), 2014, pp. 1520–1524.
[2] M. Vázquez-Arellano, H. W. Griepentrog, D. Reiser, and D. S. Paraforos,
“3-D imaging systems for agricultural applications a review,” Sensors,
vol. 16, no. 5, p. 618, 2016.
[3] N. dos Santos Rosa, P. E. Cruvinel, and J. de Mendonça Naime,
“Prerequisite study for the development of embedded instrumentation
for plant phenotyping using computational vision,” in 11th International
Conference on Semantic Computing (ICSC), 2017, pp. 445–451.
 [4] M. Bergerman, E. Van Henten, J. Billingsley, J. Reid, and D. Mingcong,
“IEEE robotics and automation society technical committee on agricul-
tural robotics and automation,” IEEE Robotics & Automation Magazine,
vol. 20, no. 2, pp. 20–23, 2013.
[5] F. Remondino and S. El-Hakim, “Critical overview of image-based
3D modeling,” in International Workshop on Recording, Modeling and
Visualization of Cultural Heritage:. CRC Press, 2005, p. 299.
[6] C. A. Northend, “Lidar, a laser radar for meteorological studies,”
Naturwissenschaften, vol. 54, no. 4, pp. 77–80, 1967.
[7] J. P. Underwood, C. Hung, B. Whelan, and S. Sukkarieh, “Mapping
almond orchard canopy volume, flowers, fruit and yield using lidar and
vision sensors,” Computers and Electronics in Agriculture, vol. 130, pp.
83–96, 2016.
[8] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J.
Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinect-
fusion: Real-time dense surface mapping and tracking,” in 10th IEEE
International Symposium on Mixed and Augmented Reality (ISMAR),
2011, pp. 127–136.
[9] C. Zhang, L. Chen, W. Huang, Z. Guo, and Q. Wang, “Apple stem-
end/calyx identification using a speckle-array encoding pattern,” in 11th
International Conference on Signal Processing (ICSP), vol. 2, 2012, pp.
1110–1114.
[10] B. Zhang, W. Huang, C. Wang, L. Gong, C. Zhao, C. Liu, and D. Huang,
“Computer vision recognition of stem and calyx in apples using near-
infrared linear-array structured light and 3D reconstruction,” Biosystems
Engineering, vol. 139, pp. 25–34, 2015.
[11] J. R. Rosell-Polo, F. A. Cheein, E. Gregorio, D. Andujar, L. Puig-
domènech, J. Masip, and A. Escolà, “Advances in structured light
sensors applications in precision agriculture and livestock farming,”
Advances in Agronomy, vol. 133, pp. 71–112, 2015.
[12] T. Jebara, A. Azarbayejani, and A. Pentland, “3D structure from 2D
motion,” IEEE Signal Processing Magazine, vol. 16, no. 3, pp. 66–84,
1999.
[13] F. Rovira-Más, Q. Zhang, and J. F. Reid, “Stereo vision three-
dimensional terrain maps for precision agriculture,” Computers and
Electronics in Agriculture, vol. 60, no. 2, pp. 133–143, 2008.
[14] R. Xiang, H. Jiang, and Y. Ying, “Recognition of clustered tomatoes
based on binocular stereo vision,” Computers and Electronics in Agri-
culture, vol. 106, pp. 75–90, 2014.
[15] Y. Si, G. Liu, and J. Feng, “Location of apples in trees using stereoscopic
vision,” Computers and Electronics in Agriculture, vol. 112, pp. 68–74,
2015.
[16] A. L. Kaczmarek, “Improving depth maps of plants by using a set of
five cameras,” Journal of Electronic Imaging, vol. 24, no. 2, p. 023018,
2015.
[17] A. Kaczmarek, “Stereo vision with equal baseline multiple camera
set (EBMCS) for obtaining depth maps of plants,” Computers and
Electronics in Agriculture, vol. 135, pp. 23–37, 2017.
[18] L. Quan, P. Tan, G. Zeng, L. Yuan, J. Wang, and S. B. Kang, “Image-
based plant modeling,” in ACM Transactions on Graphics, vol. 25, no. 3,
2006, pp. 599–604.
[19] P. Tan, G. Zeng, J. Wang, S. B. Kang, and L. Quan, “Image-based tree
modeling,” in ACM Transactions on Graphics, vol. 26, no. 3, 2007,
p. 87.
[20] S. Jay, G. Rabatel, X. Hadoux, D. Moura, and N. Gorretta, “In-field
crop row phenotyping from 3D modeling performed using structure from
motion,” Computers and Electronics in Agriculture, vol. 110, pp. 70–77,
2015.