53

Crop and weed identification in sugarcane

fields using rgb uav imagery

Inacio Henrique Yano

Embrapa

Nelson Felipe Oliveros Mesa

Unicamp

Barbara Teruel
Unicamp

'10.37885/220910047
ABSTRACT

The presence of weeds in the sugarcane crop can affect its production. Usually, farmers use
herbicides to control weeds. Precision agriculture is an alternative to avoid damages to the
environment, and it is a way to save economic resources. Once, herbicides will be applied only
in the appropriate places and correct dosages. Precision agriculture requires georeferenced
information on the location of the infestation spots. In this sense, UAVs can fly at low altitu-
des to play an important role. This permit takes images with the required spatial resolution
for a good classification of the weeds present in the crop. The UAV also can fly on demand,
solving the problem of temporal resolution. This work proposes using RGB cameras assem-
bled in UAVs to capture images to classify weeds in sugarcane crops. The RGB camera is
a solution affordable for a large number of producers. In tests realized in an experimental
field with sugarcane, four weeds’ species, and soil, using Artificial Neural Networks classifier,
was obtained 72.33% to 77.33% of overall accuracy and Kappa coefficient of 0.668 to 0.724.

Keywords: Drone, Herbicide, Machine Learning.

 INTRODUCTION

Brazil is the largest sugarcane producer. This large production is mostly used for sugar
and alcohol production [1]. The weeds without control can cause yield losses of more than
90% in sugarcane farming due to the competition for water, sun, and nutrients [2]. Because
sugarcane occupies large areas, weed control is usually chemical [3].
To reduce environmental impact and improve economic returns. It increased the interest
in precision agriculture in the post-emergence application of herbicides. The aim of precision
agriculture is weed identification presence. And also the level of its infestation. For the use of
the herbicides only when necessary and with adequate dosage. For this herbicide’s precise
application, precision agriculture needs georeferenced information [4].
The use of satellite images for weeds identification is not feasible for objects smaller
than 0.20 m due to its low spatial resolution [5]. For this reason, it is only possible to identify
weeds’ spots in situations of high-level infestation [6]. An alternative is the use of UAV Imagery.
The UAV has the advantage of flight in low-altitude and in on-demand, which provides the
spatial and temporal resolution necessary for a good classification of crops and weeds [7].
In [8], there is an example of crop and weed classification using images taken from
an RGB camera mounted in a UAV, which achieved 77% in weed classification. [8] used
the Object-based image analysis (OBIA), but this method worked only to classify the weeds
between the crop rows. For this reason, in this work, the classification process was done
using statistical descriptors of sub-images for weed classification in any inch of the field [9].
Another approach is the deep learning technique [10], but this method needs large samples
amounts. And hard work of annotating objects of interest in the images.
In this work, we tested three classifiers. Classification and Regression Trees (CART)
because it requires less effort for data preparation during pre-processing [11], Random Forest
(RF) used in several remote sensing works [12], and Artificial Neural Network (ANN) because
of its ability to deal with noisy data [13] and non-linear problems [14].
The RGB imagery was chosen because of its low-cost and therefore is accessible to a
large number of producers. In preliminary tests, ANN was the best classifier, which achieved
72.33 % to 77.33% of overall accuracy classifying sugarcane, soil, and four weed species
and 0.668 to 0.724 of Kappa coefficient.

MATERIALS AND METHODS

The previous section presented the weeds presence problem in sugarcane production,
the advantages of precision agriculture to control the weeds, and the possibility of using UAV

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
763
for image acquisition for crop and weed classification. This section presents the details of
using a UAV and a Weed Classification System (WCS).
Figure 1 shows the diagram with the main activities of the Weed Classification System
(WCS). The first process of WCS is the image acquisition from a camera mounted in a UAV.
The Imagery Acquisition subsection describes the details of this process. Three groups
divided the image dataset for a Cross-validation used for Classification Modelling and test.
The WCS uses the Weka Software for crop and weed classification. Weka is a Data
Mining with Open Source Machine Learning Software, developed by the University of Waikato,
Hamilton, New Zealand. The input data for the Weka Software were 24 statistical descriptors,
eight statistical descriptors for each channel of the RGB system. The Crop and Weed Data
subsection describes the statistical descriptor extraction.
The Weka Software generates the models in the Classification Modelling process. The
Model Evaluation process tested the models, reporting the models’ accuracy. In this work, we
tested CART, RF, and ANN classifiers. We used the overall performance index and Kappa
coefficient to choose the best model. Figure 6 shows another result of the WCS, where each
plant has a color code for better seeing.

Figure 1. Weed Classification System (WCS).

Source: The authors themselves.

Imagery Acquisition

The images were taken from a UAV System formed by GoPro Hero3 10 MP Silver
Edition mounted in a DJI Phantom (Figure 2). For this work, the sugarcane and the four weed
species were planted at the experimental field at FEAGRI/Unicamp, latitude 22º 82’ south
and longitude 47º 06’ west. Several studies [15,16] report there are no significant losses in
the crop when weed controlling starts with a height below 15 cm. Because of this, the image
dataset only contains weeds with a height of less than 15 cm.

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
764
 Figure 2. UAV System formed by a DJI Phantom with a GoPro Hero3 Silver Edition.

Source: The authors themselves.

The drone flew at approximately 3 meters in height. At this altitude, the resolution is
about 59 pixels/cm2 because, in higher flights, the imagery resolution was not enough for
weeds classification. The resolution calculation used an acrylic box as a reference, 14.1
cm long by 12.4 cm wide. The dimension of the square side length of 112 cm also used the
acrylic box measures as a reference (Figure 3).

Figure 3. Acrylic box dimensions and crop and weeds area.

Source: The authors themselves.

Figure 4 shows examples of the five plants to be classified in this work: (a) sugarcane
or Saccharum sp and four well-known weeds in the sugarcane field: (b) Panicum maximum,
(c) Ipomoea hederifolia, (d) Euphorbia heterophylla, and (e) Brachiaria decumbens.

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
765
Figure 4. Images of the five species classified in this work: (a) sugarcane (Saccharum spp), (b) Panicum maximum, (c)
Ipomoea hederifolia, (d) Euphorbia heterophylla, and (e) Brachiaria decumbens.

Source: The authors themselves.

Crop and weeds data

Three groups divided the images taken from the UAV System for a Cross-validation
test. A manual activity extracted samples of the five plants and soil from these three groups.
Figure 5 shows the images with the samples extracted for Group 1 [17] corresponding to
the Crop and Weeds Sampling process of Figure 1. The imagery of groups 2 and 3 is in
Appendix A (Figures A1 and A2). The class soil joins non-plant objects like soil, straw,
and acrylic box.

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
766
Figure 5. Images of the five species classified in this work with samples extracted: (a) sugarcane (Saccharum spp), (b)
Panicum maximum, (c) Ipomoea hederifolia, (d) Euphorbia heterophylla, and (e) Brachiaria decumbens.

Source: The authors themselves.

All the samples had the identification of which plant species belonged. The unit for the
classifier were sub-images of 25 pixels2. These units are the division of each manually ex-
tracted sample by a square of a side length of 5 pixels [18]. Each sub-image had 24 statistical
descriptors calculated, eight for each color of the RGB system. The statistical descriptors
chosen for this work were mean, average deviation, standard deviation, variance, kurtosis,
skewness, and maximum and minimum values [19]. These eight descriptors describe the
dispersion around the mean, the shape, and the sharpness of the sub-image histogram [20].
These 24 statistical descriptors were the input data used for the Classification Modelling
and Model Evaluation processes. The size of the sub-image depends on the leaf occlusions
and image resolution. In larger sub-images, there is a higher probability of occurrence, in
the same sub-image, of plants of different species. The same problem could happen to soil
and straw, making the classification process more difficult. On the other hand, too small
sub-images have less differentiation in the 24 statistical descriptors. And also can make the
classification process more difficult.
The summary of the images, samples, and sub-images used in the Classification
Modelling and Model Evaluation processes can be seen in Tables 1 and 2, respectively. To in-
crease the sampling and turn possible balance the classes. We used the oversampling
technique [21]. For a good training process, the generation of models used 300 sub-images

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
767
per class. The Model Evaluation process used only the first 50 sub-images to avoid the in-
terference of unbalanced classes in the Kappa coefficient.

Table 1. 3-fold Cross-validation data.

Group 1 Group 2 Group 3

Image Sub- Image Sub- Image Sub-
Classes Samples Samples Samples
Number images Number images Number images
Sugarcane 3905 19 77 3915 8 76 3899 19 50
Panicum
3911 13 61 3920 13 50 3899 6 54
maximum
Ipomoea
4026 9 50 3959 6 73 3899 5 55
hederifolia
Euphorbia
4024 4 68 3920 5 61 3899 2 62
heterophylla
Brachiaria
3957 7 51 3975 7 75 3899 4 59
decumbens
Several Several Several
Soil 7 252 4 185 4 109
images images images
Total 59 559 43 520 40 389
Source: The authors themselves.

RESULTS

Table 2 shows the results of the 3-fold Cross-validation test for ANN, RF, and CART.
The first column of Table 2 shows the Training set formed by two groups. The following co-
lumns show the third group used for evaluating the model, the overall accuracy index, and
the Kappa coefficient for ANN, RF, and CART classifiers. Appendix B shows the confusion
matrix of these results.

Table 2. Results from 3-fold Cross-validation.

ANN RF CART

Training Validation DG (%) Kappa DG (%) Kappa DG (%) Kappa

Group 1 and 2 Group 3 72.33 0.668 66.00 0.592 62.00 0.544
Group 1 and 3 Group 2 77.33 0.728 67.33 0.608 64.33 0.572
Group 2 and 3 Group 1 77.33 0.728 70.00 0.640 70.67 0.648
Source: The authors themselves.

Figure 6 shows the image selected to show the result of the Model Evaluation process.
Figure 6a is the original image, the Figure 6b is the original image with each plant species
having a color code: red for sugarcane, blue for Panicum maximum, yellow for Ipomoea he-
derifolia, orange for Euphorbia heterophylla, and violet for Brachiaria decumbens.

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
768
Figure 6. Image of five species of plants: (a) original image with samples extracted, (b) image with five species of plants
identified by a color-coding: red for sugarcane, blue for Panicum maximum, yellow for Ipomoea hederifolia, orange for
Euphorbia heterophylla, and violet for Brachiaria decumbens.

Source: The authors themselves.

DISCUSSION

The ANN classifier had the best Kappa results of 0.668 to 0.728 and overall accuracy
of 72.33% to 77.33%. Accordingly, with [22], the ANN classifier achieved good agreement for
all results, and the RF and CART classifiers had some good and some moderate agreement
results (Table 2). These results confirm the best performance for ANN, also achieved by [9].
Regarding the visual result with the species of plants colored, in general terms, the
species had correct identification according to their color code. However, there was a pro-
blem in identifying Ipomoea hederifolia, which were misidentified with Brachiaria decumbens
highlighted in Figure 7 with a green rectangle. And, there was also misidentification between
Panicum maximum and Brachiaria decumbens highlighted in pink (Figure 7).
Despite the errors identified, we should highlight that the other four classes of plants had
correct identification in Figure 6b. The incorrectly identified parts of Figure 7 had the leaves
turned to the side and not upwards, having lighting problems.

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
769
Figure 7. Identification errors Ipomoea hederifolia by Brachiaria decumbens in green and Panicum maximum by Brachiaria
decumbens in pink.

Source: The authors themselves

CONCLUSION

Considering that, the results for ANN were with a Kappa coefficient above 0.61. Thus
achieved a good agreement result, accordingly, with [22]. Therefore, the WCS can become
a weed classification product. And possibly also be adapted to identify weed groups such
as broadleaf and narrow-leaf or to map planting failures. These needs will be the target
of future studies.

Acknowledgments

The authors thank to Agricultural Engineering Faculty (FEAGRI) of Unicamp for the
concession of the experimental area.

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
770
Appendix A – Images of the Cross-validation of groups 2 and 3

Figure A1. Images of the group2 of five species classified in this work: (a) sugarcane (Saccharum spp), (b) Panicum
maximum, (c) Ipomoea hederifolia, (d) Euphorbia heterophylla, and (e) Brachiaria decumbens, and their samples extracted:
(a’) sugarcane (Saccharum spp), (b’) Panicum maximum, (c’) Ipomoea hederifolia, (d’) Euphorbia heterophylla, and (e’)
Brachiaria decumbens.

Source: The authors themselves.

Figure A2. Images of the group3 of five species classified in this work: (a) sugarcane (Saccharum spp), Panicum maximum,
Ipomoea hederifolia, Euphorbia heterophylla, and Brachiaria decumbens and their samples extracted: (b) sugarcane
(Saccharum spp), Panicum maximum, Ipomoea hederifolia, Euphorbia heterophylla, and Brachiaria decumbens.

Source: The authors themselves.

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
771
Appendix B – Confusion Matrixes of Cross-Validation Tests

=== Confusion Matrix ANN Group 1 Validation ===

a b c d e f <-- classified as
47 2 0 0 1 0 | a = sugarcane
16 34 0 0 0 0 | b = Panicum maximum
5 0 45 0 0 0 | c = Ipomoea hederifolia
0 0 1 44 5 0 | d = Ipomoea hederifolia
17 3 15 3 12 0 | e = Brachiaria decumbens
0 0 0 0 50 | f = soil

=== Confusion Matrix ANN Group 2 Validation ===

a b c d e f <-- classified as
21 24 0 0 5 0 | a = sugarcane
0 50 0 0 0 0 | b = Panicum maximum
0 0 45 3 2 0 | c = Ipomoea hederifolia
0 1 0 40 9 0 | d = Ipomoea hederifolia
4 10 4 1 31 0 | e = Brachiaria decumbens
1 4 0 0 0 45 | f = soil

=== Confusion Matrix ANN Group 3 Validation ===

a b c d e f <-- classified as
28 5 2 0 15 0 | a = sugarcane
1 26 0 0 23 0 | b = Panicum maximum
0 0 33 0 17 0 | c = Ipomoea hederifolia
0 0 4 46 0 0 | d = Ipomoea hederifolia
0 2 5 9 34 0 | e = Brachiaria decumbens
0 0 0 0 50 | f = soil

=== Confusion Matrix RF Group 1 Validation ===

a b c d e f <-- classified as
47 2 1 0 0 0 | a = sugarcane
22 25 0 0 0 3 | b = Panicum maximum
11 0 33 1 5 0 | c = Ipomoea hederifolia
0 0 0 44 6 0 | d = Ipomoea hederifolia
19 6 12 2 11 0 | e = Brachiaria decumbens
0 0 0 0 0 50 | f = soil

=== Confusion Matrix RF Group 2 Validation ===

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
772
 a b c d e f <-- classified as
32 11 0 0 2 5 | a = sugarcane
2 44 0 0 3 1 | b = Panicum maximum
0 1 42 2 5 0 | c = Ipomoea hederifolia
0 14 13 17 6 0 | d = Ipomoea hederifolia
2 7 20 1 20 0 | e = Brachiaria decumbens
3 0 0 0 0 47 | f = soil

=== Confusion Matrix RF Group 3 Validation ===

a b c d e f <-- classified as
27 6 8 0 9 0 | a = sugarcane
8 23 1 0 18 0 | b = Panicum maximum
1 0 39 0 10 0 | c = Ipomoea hederifolia
0 0 11 36 2 1 | d = Ipomoea hederifolia
2 2 4 17 25 0 | e = Brachiaria decumbens
2 0 0 0 0 48 | f = soil

=== Confusion Matrix CART Group 1 Validation ===

a b c d e f <-- classified as
45 0 2 0 3 0 | a = sugarcane
13 30 2 0 1 4 | b = Panicum maximum
8 0 32 0 9 1 | c = Ipomoea hederifolia
0 0 1 43 6 0 | d = Ipomoea hederifolia
17 4 9 1 19 0 | e = Brachiaria decumbens
1 0 0 0 6 43 | f = soil

=== Confusion Matrix CART Group 2 Validation ===

a b c d e f <-- classified as
24 16 3 0 2 5 | a = sugarcane
3 40 2 0 3 2 | b = Panicum maximum
0 0 41 4 5 0 | c = Ipomoea hederifolia
4 12 11 17 6 0 | d = Ipomoea hederifolia
3 12 7 1 27 0 | e = Brachiaria decumbens
6 0 0 0 0 44 | f = soil

=== Confusion Matrix CART Group 3 Validation ===

a b c d e f <-- classified as
27 6 8 0 8 1 | a = sugarcane
5 28 3 0 14 0 | b = Panicum maximum

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
773
 0 2 30 1 17 0 | c = Ipomoea hederifolia
2 0 8 30 4 6 | d = Ipomoea hederifolia
2 2 13 9 23 1 | e = Brachiaria decumbens
1 1 0 0 0 48 | f = soil

REFERENCES
1. Martins, H. H., Shikida, P. F., Sesso, U. A., Gimenes, R. M., & Calvo, J. C. (2017).
Employment generation and structural decompositon effects of the sugar and alcohol
industries in Brazil. Revista ESPACIOS, 38(11).

2. Firehun, Y., & Tamado, T. (2006). Weed flora in the Rift Valley sugarcane plantations
of Ethiopia as influenced by soil types and agronomic practises. Weed biology and
management, 6(3), 139-150.

3. Cerdeira, A. L., Paraíba, L. C., Queiroz, S. C. N. D., Matallo, M. B., & Ferracini, V. L.
(2015). Estimation of herbicide bioconcentration in sugarcane (Saccharum officinarum
L.). Ciência Rural, 45(4), 591-597.

4. Verruma, A. A., Martinelli, P. R. P., Rabello, L. M., Inamasu, R. Y., Santos, K. E. L.,
Bettiol, G. M., & Bernardi, A. D. C. (2017). SOIL AND WEED OCCURRENCE MAPPING
AND ESTIMATES OF SUGARCANE PRODUCTION COST/MAPEAMENTO DO SOLO
E OCORRÊNCIA DE PLANTAS DANINHAS E ESTIMATIVAS DO CUSTO DE PRO-
DUÇÃO DE CANA-DE-AÇÚCAR. Revista Brasileira de Engenharia de Biossistemas,
11(1), 68-78.

5. López-Granados, F., Torres-Sánchez, J., Serrano-Pérez, A., de Castro, A. I., Mesas-

-Carrascosa, F. J., & Peña, J. M. (2016). Early season weed mapping in sunflower
using UAV technology: variability of herbicide treatment maps against weed thresholds.
Precision agriculture, 17(2), 183-199.

6. Holben, B. N. (1986). Characteristics of maximum-value composite images from tem-

poral AVHRR data. International journal of remote sensing, 7(11), 1417-1434.

7. Xiang, H., & Tian, L. (2011). Development of a low-cost agricultural remote sensing
system based on an autonomous unmanned aerial vehicle (UAV). Biosystems engine-
ering, 108(2), 174-190.

8. Peña, J. M., Torres-Sánchez, J., de Castro, A. I., Kelly, M., & López-Granados, F. (2013).
Weed mapping in early-season maize fields using object-based analysis of unmanned
aerial vehicle (UAV) pictures. PLoS One, 8(10), e77151.

9. Yano, I. H., Santiago, W. E., Alves, J. R., Mota, L. T. M., & Teruel, B. (2017). CHOO-
SING CLASSIFIER FOR WEED IDENTIFICATION IN SUGARCANE FIELDS THROUGH
IMAGES TAKEN BY UAV. Bulgarian Journal of Agricultural Science, 23(3), 491-497.

10. Osorio, Kavir et al. A deep learning approach for weed detection in lettuce crops using
multispectral images. AgriEngineering, v. 2, n. 3, p. 471-488, 2020.

11. Sheffield, Kathryn; Dugdale, Tony. Supporting Urban Weed Biosecurity Programs with
Remote Sensing. Remote Sensing, v. 12, n. 12, p. 2007, 2020.

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
774
12. Yuba, Norio et al. Discriminating Pennisetum alopecuoides plants in a grazed pasture
from unmanned aerial vehicles using object‐based image analysis and random forest
classifier. Grassland Science, v. 67, n. 1, p. 73-82, 2021.

13. Jindal, N., & Kumar, V. (2013). Enhanced face recognition algorithm using pca with
artificial neural networks. International Journal of Advanced Research in Computer
Science and Software Engineering, 3(6).

14. Gonçalves, R. M., Coelho, L. D. S., Krueger, C. P., & Heck, B. (2010). Modelagem
preditiva de linha de costa utilizando redes neurais artificiais. Boletim de Ciências
Geodésicas, 16(3).

15. Knezevic, Stevan Z. et al. Critical period for weed control: the concept and data analysis.
Weed science, v. 50, n. 6, p. 773-786, 2002.

16. Gower, Steven A. et al. Effect of planting date, residual herbicide, and postemergence
application timing on weed control and grain yield in glyphosate-tolerant corn (Zea
mays). Weed Technology, v. 16, n. 3, p. 488-494, 2002.

17. Rahman, M., Blackwell, B., Banerjee, N., & Saraswat, D. (2015). Smartphone-based
hierarchical crowdsourcing for weed identification. Computers and Electronics in Agri-
culture, 113, 14-23.

18. Christensen, S., Søgaard, H. T., Kudsk, P., Nørremark, M., Lund, I., Nadimi, E. S., &
Jørgensen, R. (2009). Site‐specific weed control technologies. Weed Research, 49(3),
233-241.

19. Yano, I. H., Mesa, N. F. O., Santiago, W. E., Aguiar, R. H., & Teruel, B. (2017). Weed
identification in sugarcane plantation through images taken from remotely piloted aircraft
(RPA) and KNN classifier. J. Food Nutr. Sci, 5, 211.

20. Dixit, A., Hegde, N., & Hiremath, P. S. (2016). Cluster Analysis of Satellite (LISS-III)
Pictures of Earth surface.

21. Hussain, Lal et al. Detecting congestive heart failure by extracting multimodal features
with synthetic minority oversampling technique (SMOTE) for imbalanced data using
robust machine learning techniques. Waves in Random and Complex Media, p. 1-24,
2020.

22. Zhang, J., Yang, C., Song, H., Hoffmann, W. C., Zhang, D., & Zhang, G. (2016). Evalua-
tion of an airborne remote sensing platform consisting of two consumer-grade cameras
for crop identification. Remote Sensing, 8(3), 257.

Open Science Research VI - ISBN 978-65-5360-212-0 - Volume 6 - Ano 2022 - Editora Científica Digital - www.editoracientifica.com.br
775