Row Detection Using a Machine Learning Approach
for Autonomous Agriculture Vehicles
Niko Anthony Simon
Department of Electrical Engineering
University of St. Thomas
2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC) | 978-1-6654-1490-6/21/$31.00 ©2021 IEEE | DOI: 10.1109/CCWC51732.2021.9375999
St. Paul MN, United States
Abstract— Autonomous agriculture vehicles require robust
vision systems to navigate farm fields within rows. Commonly,
image processing methods are used for row detection in
applications like forestry and crop production to collect growth
information and yield data based on location. Often these methods
are implemented at a bird’s eye view by images obtained via
drones to determine distinct patterns, but this does not offer close-
up and specific information. Autonomous agriculture vehicles can
offer a ground-level view of crops and are capable of relaying more
nuanced data to the user. The proposed analysis focuses on
determining the effectiveness of various features for a machine
learning based image classification of crops at a ground-level view.
Due to the similarities in the color and patterns observed in the
images, classification for autonomous navigation is a difficult
problem. The proposed solution is a machine learning based model
constructed from color and texture features to predict the
occurrence of the crop. The input image data is divided into
pieces, and this prediction is performed on a segment by segment
basis to construct a coordinate map of distinct areas of interest in
the input image. A combination of color features produced by K-
means clustering and texture features produced by Haralick
textures was found to deliver 95% accuracy. Autonomous
navigation is a real-time application, and computational efficiency
is important. Therefore we also target to reduce processing time
while achieving high accuracy for real-time navigation.
Keywords---K-means, Haralick, Machine Learning, Autonomous
Agriculture Vehicles, Row Detection
I. INTRODUCTION
Several aspects must be considered when analyzing
agricultural image data. First, distinct regions of interest must
be identified. For an autonomous vehicle on a farm, this means
that the image data contains crops (in this case corn), separated
into rows, grass in between the crop rows, and the sky.
Associated conditions such as weather and camera orientation
will cause variations. For example, if the sky is overcast in one
image and sunny in an identical setting, it is possible for the
color regions which separate distinct areas to be muddled. If the
autonomous vehicle is close to crops, close-up views of plants
need to be considered, which may be highly detailed and could
be miss-classified. Various lighting conditions also affect
classification results. If all lighting conditions are known
beforehand, it may be possible to construct a series of color
ranges to represent every input condition. An example input
image is shown in Fig. 1.
If an unexpected situation were to occur, such as grass being
darker than usual or corn stalks being the same color as grass,
then confusion could be caused when trying to distinguish
different regions. This means that if image pixel colors are the
978-0-7381-4394-1/21/$31.00 ©2021 IEEE
Authorized licensed use limited to: East Carolina University. Downloaded on June 18,2021 at 23:54:29 UTC from IEEE Xplore. Restrictions apply.
Cheol-Hong Min
Department of Electrical Engineering
University of St. Thomas
St. Paul MN, United States
Figure 1. Example image taken from a farm vehicle showing (corn)
crops, grass and sky during the daytime with shadows on the grass.
main input to a system, it may still have some dependence on
lighting even if results generally seem dependable, as shown in
[1,3,5,9,10, and 13]. On the other hand, texture identification
depends entirely on patterns. Unlike color-based identification,
which relies on determining pixel values and comparing them
against a pre-set range, the texture depends on identifying
patterns that the pixels create.
II. PROPOSED METHOD
For our application, a machine learning classification
approach is examined, specifically identifying the best
performing features. With the features, the proposed system will
be implemented as a real-time application. Therefore, certain
aspects regarding simulation versus processing efficiency and
feature choice will be highly important. The proposed method
seeks to use a minimal amount of features while producing a
high enough degree of prediction accuracy. A basic diagram for
this procedure is shown in Fig. 2. If the majority of predictions
are correct, misidentifications will stand out as obvious outliers
in the output and can be treated as noise in post processing.
Using a minimal amount of features is desirable because
features take time to be processed. In a real time application,
depending on how the feature is processed, it will take a certain
amount of processing time to execute. This increased
processing time is the trade off for extra feature information that
may not be absolutely necessary for classification purposes.
Color and texture features are the most widely used over most
machine learning classification based applications however,
other types of features are also examined in a comprehensive
feature analysis for this data to measure feature effectiveness.
Figure 2: Flow diagram of the proposed method
0273
This method also relies on a pool of training data in order to
construct a model.
The training images must be an accurate representation of
system inputs. If there is a significant pattern variation within a
particular region, then more training images may be required to
develop an accurate model. An example of this would be corn
in different stages of the growing season. Depending on the time
of year and environmental conditions, there exists a variety of
different textures and colors that can be observed from the same
location.
The proposed method also relies on a preprocessing stage
that takes the input image and breaks it down into segments. The
segments are then fed into the model, and predictions are plotted
as coordinates to represent distinct regions of interest. This is
different than other methods which process the input image as a
whole rather than process it gradually in segments.
Because this type of agricultural data does not include
distinct objects, but rather areas of a particular occurrence, like
crops or grass, segmenting the data is desirable because a higher
resolution can be achieved at the boundaries where different
areas exist. The most critical boundary is between the crop and
the grass because this is where the autonomous vehicle travels,
and it would be undesirable for the vehicle to drive into the crop.
The second most important boundary is between the crop and
the sky because it helps to establish crop height, the horizon line,
and other factors relating to the spatial orientation of the
autonomous vehicle. Some of this data can be beneficial to
supplement already existing sensor data for the autonomous
vehicle, as suggested by [1, 10 and 12].
III. FEATURES AND TRAINING IMAGES
A. Features
Color data is based on image pixel values. Ideally, if all of
the colors for a given area were known, a range of colors could
be developed to represent that area. However, if the lighting is
slightly different, those color ranges would be invalidated. This
is specifically relevant for cases of shadow.
Checking every pixel value is expensive in terms of
processing. Determining arbitrary color ranges and checking
pixel values in this systematic manner may give good resolution,
but only if the color ranges remain constant. In the case of
agricultural data, several different conditions could influence the
color. Evaluating colors in this level of detail could introduce
unwanted problems. For this reason, only the highest occurring
colors in a given training image are recorded and used as
features.
Determining the most frequently occurring colors is
achieved using K-means clustering, which is used to examine all
colors present in a segment. The detected colors are then labeled
and divided into groups, which can be measured in terms of
quantity of occurrence. Selecting an arbitrary n most occurring
colors must be implemented with care because, in some cases,
there may not exist n different colors in a given area, as
suggested by Yadav in [3]. This effect was observed in the sky
region where on a clear cloudless day, there may be only two or
three distinct colors in a given segment, while a segment of grass
may contain twenty distinct colors.
Authorized licensed use limited to: East Carolina University. Downloaded on June 18,2021 at 23:54:29 UTC from IEEE Xplore. Restrictions apply.
There are a variety of ways to determine texture information
but one method was ultimately chosen for its ease of execution.
Texture features are calculated using Haralick textures, which
essentially looks for patterns in adjacent pixels and outputs a set
of coefficients to represent those patterns. Haralick textures are
often used in biology and medical applications to identify
organics. This is somewhat similar to the complex textures
found in most plant life, as shown by Bekkari in [8]. Texture
identification can be performed in dim or changing lighting
conditions. This could be useful if the autonomous agriculture
vehicle were to be active in the evenings.
Another benefit is that the number of coefficients produced
by the Haralick texture operation is relatively small. Such
characteristics are relevant for the same reason discussed for
color features where processing time is increased. Gabor filters
were considered as texture features for this implementation as
well. However, they require several more coefficients and
multiple filter instances to perform effectively.
Moments are also explored as a possible feature. Image
moments are a weighted average of pixel values and are
commonly used for shape detection. Specifically, the Hu
Moment is tested as a possible feature because it does not require
certain phase-related inputs that other types of moments require
for calculations, and the output is in the form of only seven
coefficients, as shown by Fu [7].
B. Training Images
Eight hundred reference images, each for the crop, sky, and
grass are used to train the model. These images were created by
segmenting several farm images from the surrounding location,
extracting sample areas, and manually separating them into the
three categories of interest. To develop an accurate model, it is
vital to use several examples of the training textures that are
expected to be observed in the input image. If the training
images are too different in perspective, size, or pattern from the
input image, the quality of the predictions will suffer. Once the
model is developed, it can be reused without having to compile
again. One possible benefit to this design is that multiple models
could be created and loaded if the surroundings change
depending on the specific type of crop or season of operation.
For each segment, the model outputs a percent confidence
value for the predicted region, either the crop, grass or the sky.
This means that there will be a dominant percentage and
secondary as well as tertiary percentages that predict whether
any given segment shows any given region. In a segment with
mixed regions, however, percentages may be more skewed. The
goal is to distinguish the different textures and colors, but if a
segment is very large, the greater the chance an individual
segment will contain more than one region.
In order to address such limitations, an alternative
preprocessing technique is considered. Rather than simply
splitting the input into segments in a grid-like fashion, a sliding
window can be applied to sweep across the input and create
segments at a rate specified by the user. At a certain point,
though, optimizations like this start to become a hardware
design question. GPU selection and other hardware choices are
outside the scope of this analysis, but they are major
considerations in terms of field implementation. Associated
factors are a consideration in [4,6, and 11].
0274
 Figure 3. Sample Segments of Input Image in Two Different Sizes
360x240 Pixels (Left) 90x60 Pixels (Right)
IV. EXPERIMENTAL RESULTS
A. Effects of Segment Size
During classification, the original image is segmented as part
of processing the input, but the size of the segment affects
accuracy and precision. Classification increases in difficulty
depending on which specific area of the input is observed.
Because corn stalks have a variety of textures in the husk and
leaves, it is even possible that parts could even be confused with
the texture of the grass. When segments decrease in size, small
areas of shadow and other small pockets of unexpected patterns
become more apparent, which can be seen in Fig. 3. Increasing
the number of segments does make the output more precise by
increasing the number of predictions. However, the
interpretation of predictions produced by the model must also
become more accurate as a tradeoff. For the proposed method,
segments of 90x60 pixels are used.
When segments decrease in size, small areas of shadow and
other small pockets of unexpected patterns become more
apparent. Increasing the number of segments does make the
coordinate map more precise by increasing the number of
coordinates. However, the interpretation of predictions
produced by the model must also become more complex to
distinguish variations among individual textures and colors. At
most for this application, segments could be reduced to about
45x30 pixels while maintaining an identifiable representation of
a given region. Any dimensions smaller than this may introduce
inconsistency. For classification, a Random Forrest classifier is
used to construct the machine learning model. Several other
classifier types were tested, and most performed similarly to
each other in terms of output accuracy. Specific classifier type
is not a critical point of the proposed method and can be changed
if a different classifier is found to process data more efficiently
for this application.
B. Prediction Tolerance and Environmental Factors
To maintain the original aspect ratio of the input image,
horizontal points are scaled by a factor of two-thirds. Once all
Figure 4. KNN Decision boundary, note yellow, blue and green colors
have no special meaning and simply show divisions between the crop,
sky and grass, K=7
Authorized licensed use limited to: East Carolina University. Downloaded on June 18,2021 at 23:54:29 UTC from IEEE Xplore. Restrictions apply.
Figure 5: Close up view of corn stalks and cooresponding KNN decision
boundary where K=7
of the image segments have been mapped to X and Y
coordinates, they are fed into a KNN classifier to determine a
decision boundary. KNN is a supervised machine learning
algorithm commonly used for classification where coordinates
are classified based on the labels of neighboring data. The value
K represents the number of neighboring data that is considered
during the classification, and there are tradeoffs to choosing the
appropriate value for K. If K is too small, a bias could be
introduced, creating noise in the decision boundary. If the value
for K is too large, increased processing time can become an
issue, especially for a real-time application. The KNN classifier
serves as a post-processing stage if portions of the output have
low confidence in predictions.
In cases where there is irregular or overlapping clustering of
coordinates, the value of K becomes more relevant, but in this
case, it can be kept low without having negative effects, which
is demonstrated by Guo in [2]. The KNN classifier groups the
voids in data in the grass regions of the image map caused by
the shadows, which can be seen in Fig. 4 and Fig. 5. In the
formation of the decision boundary, the voids created by the
shadows in the grass are appropriately filled in based on the
clustering of surrounding coordinates.
C. Feature Effectiveness
If the input image itself has a close view of crop textures and
colors, it has the potential to affect row identification predictions
if not expected. In the case of Fig. 5, the corn is still in season,
but the distribution of distinct regions is more complicated. The
sky is intermingled with the corn stalks, and the grass can be
seen on the other side of the furrow. This is a likely scenario for
an autonomous device where it would be an undesirable
situation for it to drive into crops, which means there needs to
be a clear distinction between regions even if one region is
taking up most of the image.
Table 1 shows features that were evaluated using the training
data pool of images. Each feature is measured against every
other feature for its contribution to the aggregate accuracy.
Figure 6: Output predictions with no percentage threshold or KNN
clustering showing clear division between the crop, grass, and sky
0275
 TABLE I. TABLE OF FEATURE EFFECTIVENESS
Feature, Number of Coefficients, and Effectiveness
Between 3
Color (K-Means) Effective
and 10
4 Groupings
Haralick Texture Effective
of 13
Hu Moment 7
Not
Effective
Not
Variance 16200
Effective
Median 16200 Effective
Gaussian (Sigma=7) 16200 Effective
Color, texture, and Gaussian features were the overall top-
performing features. A Gaussian filter essentially applies a
normal distribution and changing the value of sigma adjusts
variance at the mean value of the bell curve. Even though
Gaussian and some time-domain features (like median) have
reasonable contributions, including all of them would increase
the calculation time per segment. However, adding additional
features would be possible. Different implementations of color
and texture-based features could be added as long as the
calculation is not overly intensive.
Segment size and environmental factors are not the only
important considerations. The logic which interprets model
predictions can be adjusted to omit predictions below a
minimum threshold. Corn stalks are correctly identified as the
major region in the foreground in Fig. 5, even when the camera
orientation is adjusted to a close-up view of the crop. This is a
good indication that no biases have been introduced into the
feature matrix during the construction of the model. In the case
of Fig. 6, prediction thresholds are removed to observe the raw
output of the model. Most of the misclassification occurs in the
crop region, and overall accuracy of about 95% is achieved. In
general, the KNN decision boundary does not need to be used as
a post-processing step unless certain environmental factors are
present that have not been considered.
V. CONCLUSIONS
Under certain lighting conditions, color features perform
significantly better, especially in cases of minimal shadow. If
there is a significant amount of shadow caused by natural light,
texture features will offer more valuable feature data for
classification. This is one of the benefits of including different
feature types. Other types of features that are commonly used
for classification in one-dimensional data, like median and
Gaussian filters, calculate the segment as a whole and produce
thousands of coefficients upon each instance. Overall, the
tradeoff of increased processing is not worth the benefit that
these features offer. A purely signal processing based approach
can be very effective but has some disadvantages when
compared to a machine learning approach. This is evident just
based on the flexibility of design. Certain signal processing
implementations may require the re-design of filter parameters,
whereas a machine learning approach could simply swap out a
model.
The most challenging aspect of working with this type of
data is the complexity and similarities between the crop and
grass image data. Furthermore, applications that involve
Authorized licensed use limited to: East Carolina University. Downloaded on June 18,2021 at 23:54:29 UTC from IEEE Xplore. Restrictions apply.
classifying plant life or other organic materials offer valuable
comparisons for determining effective classification features in
this application. The proposed method adequately solves the
classification problem. However, there is an inherent issue
associated with this type of machine learning based approach.
Because features must be manually chosen, the performance of
the model is directly related to that choice. Further work on this
application will include video input and investigation of a neural
network-based classification solution.
REFERENCES
[1] M. Kaur and C. Min, "Automatic Crop Furrow Detection for Precision
Agriculture," 2018 IEEE 61st International Midwest Symposium on
Circuits and Systems (MWSCAS), Windsor, ON, Canada, 2018, pp. 520-
523, doi: 10.1109/MWSCAS.2018.8623906.
[2] J. Guo and X. Wang, "Image Classification Based on SURF and
KNN," 2019 IEEE/ACIS 18th International Conference on Computer and
Information Science (ICIS), Beijing, China, 2019, pp. 356-359, doi:
10.1109/ICIS46139.2019.8940198.
[3] H. Yadav, P. Bansal and R. Kumar Sunkaria, "Color dependent K-means
clustering for color image segmentation of colored medical images," 2015
1st International Conference on Next Generation Computing
Technologies (NGCT), Dehradun, 2015, pp. 858-862, doi:
10.1109/NGCT.2015.7375241.
[4] U. Shruthi, V. Nagaveni and B. K. Raghavendra, "A Review on Machine
Learning Classification Techniques for Plant Disease Detection," 2019
5th International Conference on Advanced Computing & Communication
Systems (ICACCS), Coimbatore, India, 2019, pp. 281-284, doi:
10.1109/ICACCS.2019.8728415.
[5] W. Winterhalter, F. V. Fleckenstein, C. Dornhege and W. Burgard, "Crop
Row Detection on Tiny Plants With the Pattern Hough Transform,"
in IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3394-3401,
Oct. 2018, doi: 10.1109/LRA.2018.2852841.
[6] K. Aparna and P. Supriya, "Precision Agriculture in Maize Fields," 2018
Second International Conference on Intelligent Computing and Control
Systems (ICICCS), Madurai, India, 2018, pp. 1407-1410, doi:
10.1109/ICCONS.2018.8662936.
[7] M. Fu, F. Liu, Y. Yang and M. Wang, "Background pixels mutation
detection and Hu invariant moments based traffic signs detection on
autonomous vehicles," Proceedings of the 33rd Chinese Control
Conference, Nanjing, 2014, pp. 670-674, doi:
10.1109/ChiCC.2014.6896705.
[8] A. Bekkari, S. Idbraim, D. Mammass and M. E. Yassa, "Exploiting
spectral and space information in classification of high resolution urban
satellites images using Haralick features and SVM," 2011 International
Conference on Multimedia Computing and Systems, Ouarzazate, 2011,
pp. 1-4, doi: 10.1109/ICMCS.2011.5945611.
[9] X. Jinlin and J. Weiping, "Vision-Based Guidance Line Detection in Row
Crop Fields," 2010 International Conference on Intelligent Computation
Technology and Automation, Changsha, 2010, pp. 1140-1143, doi:
10.1109/ICICTA.2010.400.
[10] C. Tu, B. J. van Wyk, K. Djouani, Y. Hamam and S. Du, "An efficient
crop row detection method for agriculture robots," 2014 7th International
Congress on Image and Signal Processing, Dalian, 2014, pp. 655-659,
doi: 10.1109/CISP.2014.7003860.
[11] L. Zheng and J. Xu, "Multi-crop-row detection based on strip
analysis," 2014 International Conference on Machine Learning and
Cybernetics, Lanzhou, 2014, pp. 611-614, doi:
10.1109/ICMLC.2014.7009678.
[12] G. Jiang, C. Zhao and Y. Si, "A machine vision based crop rows detection
for agricultural robots," 2010 International Conference on Wavelet
Analysis and Pattern Recognition, Qingdao, 2010, pp. 114-118, doi:
10.1109/ICWAPR.2010.5576422.
[13] N. H. Chehade, J. Boureau, C. Vidal and J. Zerubia, "Multi-class SVM
for forestry classification," 2009 16th IEEE International Conference on
Image Processing (ICIP), Cairo, 2009, pp. 1673-1676, doi:
10.1109/ICIP.2009.5413395.
0276