Computers and Electronics in Agriculture 183 (2021) 106064

Contents lists available at ScienceDirect

Computers and Electronics in Agriculture

journal homepage: www.elsevier.com/locate/compag

Detecting soybean leaf disease from synthetic image using multi-feature

fusion faster R-CNN
Keke Zhang a, Qiufeng Wu b, *, Yiping Chen a
a
College of Engineering, Northeast Agricultural University, Harbin 150030, China
b
College of Arts and Sciences, Northeast Agricultural University, Harbin 150030, China

A R T I C L E I N F O A B S T R A C T

Keywords: An accurate detection of soybean leaf disease in soybean field is essential for soybean quality and the agricultural
Soybean leaf disease detection economy. Though many works have been done in identifying soybean leaf disease, because of the insufficient
Synthetic image dataset and technical difficulties, the tasks about detecting soybean leaf disease in complex scene are little
Complex scene
dressed. This paper develops a synthetic soybean leaf disease image dataset to tackle the problem of insufficient
Multi-feature fusion
dataset at first. Further, detecting soybean leaf disease in complex scene requires the detection model to be able
to precisely discriminate various features, such as features of healthy leaves and diseased leaves, features of
leaves with different diseases and so on. Thus, this paper designs a multi-feature fusion Faster R-CNN (MF3 R-
CNN) to address the above intractable problem. We obtain the optimal mean average precision with 83.34% in
real test dataset. Moreover, the experimental results indicate that the MF3 R-CNN trained only by synthetic
dataset is effective in detecting soybean leaf disease in complex scene and superior to the state-of-the-art.

1. Introduction themselves. Besides, the studies on soybean leaf disease identification

Soybean is a widely cultivated plant that provides protein and oil for
people. In addition, the demand of soybean is increasing with the growth
of population. Therefore, it is essential to enlarge the yield of soybean.
However, soybean leaf disease will cause considerable yield losses and
pose a threat to food security simultaneously. Thus, accurate and real-
time detection of soybean leaf disease is urgently needed.
In the field of soybean leaf disease identification based on computer
vision, the early traditional work usually consists of two independent
processes, i.e., feature extraction and disease classification. For example,
researchers extracted canopy color feature (Guan et al., 2016) or
diseased spots feature (Qi et al., 2006; Ma et al., 2017; Li et al., 2019)
first, then fed the extracted features to classifier to recognize soybean
leaf disease. However, the divergence of one disease in different stage is
obvious, and the symptom of some different diseases presents similar.
The above factors lead to the traditional recognition method has poor
universality when solving the problem of soybean leaf disease identifi­
cation. With the development of computer vision, the identification of
soybean leaf disease via deep learning becomes research hotspot. Since
the public dataset Plant Village (Hughes et al., 2015) does not collect
soybean leaf image, most researchers collect soybean leaf disease image
are relatively small.
The studies on soybean leaf disease identification based on deep
learning can be divided into two types according to the utilized soybean
leaf image. The first one is that the collected images contain only one
diseased soybean leaf (Wu et al., 2019; Gui et al., 2019). In this condi­
tion, authors identified soybean leaf disease by the improved Convolu­
tional Neural Network (CNN) directly. The second one is that the
collected images include multiple soybean leaves (Tetila et al., 2019;
Jiang et al., 2019; Xiong et al., 2020). In this situation, authors utilized
segmentation algorithm to segment the original image into images
containing only a single soybean leaf (Tetila et al., 2019; Xiong et al.,
2020), or images containing diseased soybean leaf region (Jiang et al.,
2019) at first. Then used a specially designed CNN to recognize the
soybean leaf disease category. In addition, some researchers are devoted
to identify soybean pests image via CNN (Sun et al., 2020), or estimate
soybean leaf defoliation using CNN (Silval et al., 2019).
To our knowledge, there is no work for detecting soybean leaf disease
in complex scene. The “complex scene” in our work refers to the soybean
leaf disease image that containing multiple diseased leaves, multiple
soybean plants, weeds, soil, massive healthy leaves and so on. Further­
more, the target diseased leaves only occupy relatively small regions

* Corresponding author.
E-mail address: qfwu@neau.edu.cn (Q. Wu).

https://doi.org/10.1016/j.compag.2021.106064
Received 21 June 2020; Received in revised form 21 September 2020; Accepted 23 February 2021
Available online 5 March 2021
0168-1699/© 2021 Elsevier B.V. All rights reserved.
K. Zhang et al.
Fig. 1. The overview of detecting soybean leaf disease in complex scene via MF3 R-CNN and synthetic image.
over the whole image, and it is common for leaves to cover each other. In
the real soybean field, monitoring soybean disease usually by utilizing
fixed camera or smart phone, the collected images are mostly with
complex scene. Thus, detecting soybean leaf disease in complex scene is
essential and a detection model suitable for image with complex scene is
urgently needed. There are two reasons for the lack of work on soybean
leaf disease detection in complex scene. The first one is that the
complexity of detecting soybean leaf disease is higher than recognizing
soybean leaf disease only, since the detection task should not only
recognize the disease category of all diseased soybean leaves, but also
locate them. The second aspect is that the soybean leaf disease image in
complex scene is difficult to collect. On the one hand, the image
collection will consume massive manpower, material resources and
time. On the other hand, the soybean leaf disease image in complex
scene needs to be carefully collected, if the occlusion between the
diseased soybean leaves in collected image is serious, the detection
model will be difficult to learn the comprehensive features of each
disease.
To tackle the above problems, first, the synthetic soybean leaf disease
images were generated to address the problem of insufficient dataset.
Second, the multi-feature fusion Faster R-CNN (MF3 R-CNN) was pro­
posed to detect soybean leaf disease in complex scene, and the MF3 R-
CNN model was trained only by synthetic image generated from the first
step. Third, the trained detection model was evaluated by the real soy­
bean leaf disease image collected from the soybean field and internet.
The overview of detecting soybean leaf disease in complex scene via MF3
R-CNN and synthetic image is illustrated in Fig. 1.
As we all know, deep leaning based methods require large dataset.
However, the collection of a sufficient image dataset is expensive and
time-consuming. To tackle the problem of insufficient image dataset,
researchers tend to use synthetic image to train deep models. Image
synthesis methods can be divided into two types according to whether
the synthesis technology utilizes real image. The first one is that the
process of synthesis technology contains real image. For example, Silval
et al. (2019) used synthetic image to estimate soybean leaf defoliation. A
synthetic defoliation image is generated by removing leaf-belonging
pixels in the real image in three different manners. Galbally et al.
(2015) applied synthetic image to recognize on-line signatures. They
generated the synthetic signature by combining real signature image
and pen-up trajectory. Gupta et al. (2016) utilized synthetic image on
2
Computers and Electronics in Agriculture 183 (2021) 106064
locating text in natural images. They designed the synthetic image by
automatically fitting text to natural images.
The other is that the real image is not involved in the synthesis
process. In more detail, the synthesis technology of not using real image
consists of two approaches, that is, relying on modern graphic engines
and designing special procedures. For instance, Hattori et al. (2015)
established a scene-specific pedestrian detector by synthetic image.
They created simulation scene by combining pedestrian renderings and
scene geometry. Mancini et al. (2016) trained an obstacle detector on
synthetic image. They used Unreal Engine 4 with Urban City pack to
create synthetic scenarios. Richter et al. (2016) used synthetic image to
train semantic segmentation system. They explored the use of modern
games such as Grand Theft Auto to generate large pixel-wise ground
truth images. Barth et al. (2018) applied synthetic image in segmenting
sweet pepper images. They used PlantFactory 2015 Studio to create
virtual sweet pepper plants. Wang et al. (2019) estimated head pose
from synthetic image. The commercial software FaceGen was used to
generate synthetic head pose image. Even though synthesizing image by
graphic software is convenient, some researchers prefer to synthesize
image by designing specific procedures. For example, Cicco et al. (2017)
utilized synthetic image to detect crop and weeds. They procedurally
generated large synthetic image by randomizing crop, weed species, soil
and light condition. Rahnemoonfar and Sheppard (2017) adopted syn­
thetic image to build a fruit counting model. They designed a procedure
to create variable-sized tomatoes in synthetic image. Režnáková et al.
(2017) utilized synthetic image in recognizing on-line handwritten
gesture. They designed a synthetic image generation system to create
virtual handwritten gesture. The key idea of image synthesis methods is
consistent, that is, to create synthetic image for a certain task, but the
implementation is different. The aforementioned works use synthetic
image to train their models and achieve acceptable results, which proves
the rationality and effectiveness of using synthetic image in deep
learning based tasks. Thus, in this work, the real soybean field complex
background image and the soybean leaf disease image in simple scene
were used to generate the synthetic soybean leaf disease image.
In this study, our main contributions can be summarized as follows:
(1) The MF3 R-CNN model is designed to identify soybean leaf disease in
complex scene. The MF3 R-CNN model is trained only by synthetic
soybean leaf disease image and evaluated on real image. Our experi­
mental results show that the performance of the proposed MF3 R-CNN
K. Zhang et al.
Fig. 2. The real image dataset.
model is superior to the state-of-the-art. (2) The synthetic soybean leaf
disease image dataset was generated, and it has been experimentally
shown that the synthetic method proposed in this work is effective. (3)
Five novel feature fusion methods in MF3 R-CNN are presented.
The remainder of this paper is as follows. Section 2 introduces the
details of the synthetic soybean leaf disease image and the construction
of the MF3 R-CNN model. Section 3 presents the massive experiments
and results. Section 4 concludes the paper.
2. Materials and methods
2.1. Synthetic image generation
In this section, we present the proposed method of synthesizing
soybean leaf disease image in this work.
Fig. 3. Diagram of the generation of the synthetic soybean diseases image.
3
Computers and Electronics in Agriculture 183 (2021) 106064
2.1.1. Description of the real image dataset
By consulting plant protection experts and conducting field research
in soybean field, three common soybean leaf diseases of the season, that
is, virus disease, frogeye leaf spot and bacterial spot, were studied in this
work. From July to September 2019, the soybean leaf disease image
dataset in simple scene and soybean field complex background image
dataset were collected from the soybean experimental base of the
Northeast Agricultural University and Xiangyang farm, Harbin, China.
The real image dataset is shown in Fig. 2.
The first column of Fig. 2 displays the soybean bacterial spot images.
The symptoms of soybean bacterial spot on the leaves are as follows: the
spot is a small chlorotic irregular spot at the beginning, it is water-
stained, and it expands in the later stage, a yellow halo appears on the
edge of the spot, and the middle of the lesion is dark brown. The disease
spots are mostly irregular, and leaf edge disease spots often appear in
pieces, which can cause local leaf death. The second column of Fig. 2
shows the soybean frogeye leaf spot images. The frogeye leaf spot is
K. Zhang et al. Computers and Electronics in Agriculture 183 (2021) 106064

Table 1 2.1.3. Data augmentation

The details of training dataset. Data augmentation technologies were used on the synthetic soybean
Disease in image Synthetic Augmented Total leaf disease image dataset, which not only enlarge the scale of the
dataset dataset training dataset, but also enhance the robustness of the detection model
Virus disease 50 500 550 (Krizhevsky et al., 2012; Ma et al., 2018; Cruz et al., 2019; KC et al.,
Frogeye leaf spot 50 500 550 2019; Liang et al., 2019). In this work, data augmentation techniques
Bacterial spot 50 500 550 such as reflection, rotation and color adjustment are applied (Krizhevsky
Frogeye leaf spot and Bacterial 50 500 550 et al., 2012; Zhang et al., 2016; Jia et al., 2017; Cruz et al., 2019; Liang
spot
et al., 2019; Zhang et al., 2019). The training dataset is given in Table 1.
The first column in Table 1 indicates the category of diseased soybean
mostly round, oval or irregular spots in the leaves. The center of the spot leaves contained in the image. The number of the total target objects in
is grayish white, the edge of the spot is reddish brown, the center of the training dataset is 2200.
late spot is gray, and the edge is dark brown. The third column of Fig. 2 In this work, the soybean leaf disease detection model is trained on
shows the soybean virus disease images. The symptoms of soybean virus synthetic image, but the testing process is conducted on the real image.
disease are systemic shrinkage of leaves. The leaves are distorted and The test dataset is real soybean leaf disease image dataset collected from
deformed. The mesophylls have dark green sore-like protrusions, and soybean field (25 images) and the internet (5 images). In addition, some
the leaf margins curl downward. The last two columns of Fig. 2 display images in test dataset are displayed in Fig. 4.
the soybean field complex background images. As shown in the image,
the soybean field complex background images contain multiple soybean
2.2. Multi-feature fusion Faster R-CNN (MF3 R-CNN)
plants, weeds, soil, massive healthy leaves and so on.

2.2.1. Motivations
2.1.2. Details of the synthesis process
In the real soybean field, monitoring soybean disease usually by
The pipeline of generating synthetic soybean leaf disease image can
utilizing fixed camera or smart phone, the collected images are mostly
be summarized as follows (see also Fig. 3). Firstly, the coordinates of the
with complex scene, that is, images that containing multiple diseased
midpoint of the major veins of the leaves with less severe occlusion in
leaves, multiple soybean plants, weeds, soil, massive healthy leaves and
background image were obtained, that is, the interactive program was
so on. Thus, detecting soybean leaf disease in complex scene is critical
provided to the agricultural experts, and they were requested to deter­
and a detection model suitable for image with complex scene is urgently
mine which leaves should be selected and then the corresponding co­
needed.
ordinates of the midpoint of the major veins of the leaves will be
Detecting soybean leaf disease in complex scene requires the detec­
returned and saved automatically. Then, one or multiple coordinates
tion model to be able to study comprehensive features and precisely
were randomly selected and marked in the background image (see the
discriminate various features, for example, features of healthy leaves
red circle in the marked image in Fig. 3). Next, the annotated back­
and diseased leaves, features of leaves with different diseases, features of
ground images were saved in different folders, and then soybean leaf
background such as soybean plants, weeds and so on. In addition, in the
disease images in simple scene were randomly put into a folder with a
field of object detection, Faster R-CNN (Ren et al., 2015) is a strong
marked background image. Finally, the diseased leaf image was merged
detector and has been successfully applied in various tasks (Xue et al.,
in the location of the annotated leaf region in the background image.
2018; Yang et al., 2018; Wang et al., 2018; Zheng et al., 2018; Li et al.,
The process framed by the blue box is programmatic and implemented
2019). Faster R-CNN is a two-stage network, which identifies region
on Matlab software. The process of merging images is manual and
proposals in the initial stage and classifies the objects within the region
implemented on Photoshop software.
proposals in the second stage. One-stage detector, representative YOLO

Fig. 4. The real soybean leaf disease test image dataset.

4
K. Zhang et al. Computers and Electronics in Agriculture 183 (2021) 106064

Fig. 5. Operating mechanism of the MF3 R-CNN.

Fig. 6. All detectors in this work: (a) baseline, (b) early skip, (c) halfway skip 1, (d) halfway skip 2, (e) late skip, (f) fusion skip.

5
K. Zhang et al.
Fig. 7. The process of determining anchor boxes.
(Redmon et al., 2016; Redmon and Farhadi, 2017) and SSD (Liu et al.,
2016) skip the second stage of Faster R-CNN and directly regresses the
default anchors into detection boxes (Liu et al., 2018). One-stage de­
tector usually run faster than two-stage detector, but they may not reach
the same level of accuracy, especially for the situation that the differ­
ences between different categories are small. Further, multi-scale
feature representations and contextual are essential for accurate visual
recognition (Bell et al., 2016). Features at different layers focus on
different information, that is, lower layers contain finer visual details,
such as location and counter, and higher layers include stronger se­
mantic information (Liu et al., 2018). Thus, this paper designed a multi-
feature fusion Faster R-CNN (MF3 R-CNN) to detect soybean leaf disease
in complex scene.
2.2.2. Implementation details and architecture visualization
In this work, we improved the Faster R-CNN model by improving its
feature extraction network. For feature extraction network, the Faster R-
CNN model use VGG-16 (Simonyan and Zisserman, 2015) to extract
features from input images. In this work, first, we selected the ResNet-50
(He et al., 2016) as the feature extraction network, and then optimized
the structure of ResNet-50. The reasons for choosing ResNet-50 as
feature extraction network are that, on the one hand, ResNet-50 is an
excellent backbone network and performs strongly in feature represen­
tation. Moreover, ResNet-50 is widely used as the backbone network in
massive object detection tasks (Lin et al., 2017; Liu et al., 2018; Lin et al.,
2018; Li et al., 2018). On the other hand, to address the dilemma be­
tween detection performance and computational complexity, ResNet-50
is a better choice than ResNet-101 or ResNet-152. In the following,
ResNet-50 will be denoted as ResNet.
In order to implement “multi-feature fusion” in MF3 R-CNN, first,
skip connection was introduced into the structure of ResNet, then the
generated fusion feature maps were sent to the region proposal network
(RPN) and RoI pooling layer. The operating mechanism of MF3 R-CNN is
given in Fig. 5.
Five skip connection methods are explored in this work, named as
early skip (see Fig. 6(b)), halfway skip 1 (see Fig. 6(c)), halfway skip 2
(See Fig. 6(d)), late skip (see Fig. 6(e)) and fusion skip (see Fig. 6(f)). It
should be noted that, for the clarity of the image, the same network
structure of different detectors was omitted and drew the part with
different connections only. The difference between the early skip,
halfway skip 1, halfway skip 2 and late skip is the location of the layer
connected to the feature extraction layer defined in baseline detector.
The difference between the fusion skip and other skip methods is the
complexity of the skip connection.
In this work, the detector using the original ResNet as feature
6
Computers and Electronics in Agriculture 183 (2021) 106064
extraction network was denoted as baseline, and the architecture of
baseline detector is shown in Fig. 6(a). The feature extraction layer in
using ResNet as feature extraction network is the final convolutional
layer of the 4-th stage of ResNet (He et al., 2017). Considering that the
number of layers of ResNet is large and inconvenient to draw all layers,
each building block is simplified to a rose red rectangle with a mark such
as “res3b”, “res3c” and so on. Meanwhile, the same network structure of
different detectors was omitted and only the different parts between all
detectors are drew. The implementation details of the five skip
connection methods are as follows:
(1) Early skip. First, the “res3b” and “res3d” are concatenated in
depth, and then an average pooling layer is added to reduce
dimension. Next, the “res4f” layer concatenates the average
pooling layer in depth. Then, the fusion feature maps are fed into
the RPN and the RoI pooling layer. In addition, the “res5a”,
“res5b” and “res5c” building blocks are deleted for improving
computational efficiency.
(2) Halfway skip 1. The “res4a” layer concatenates the “res4f” layer
in depth. Then, the fusion feature maps are fed into the RPN and
the RoI pooling layer.
(3) Halfway skip 2. The “res4c” layer concatenates the “res4f” layer
in depth. Then, the fusion feature maps are shared by the RPN
and the RoI pooling layer.
(4) Late skip. The “res4e” layer concatenates the “res4f” layer in
depth, and then the fusion feature maps are sent to the RPN and
the RoI pooling layer.
(5) Fusion skip. The fusion skip mode is a combination of the early
skip method and the halfway skip 2 method.
3. Experiments and results
3.1. Implementation details
The proposed MF3 R-CNN is implemented under Matlab software
with the NVIDIA TITAN Xp GPU. The training parameters are as follows:
the optimization method is stochastic gradient descent (SGD) algorithm
with a momentum of 0.9. The initial learning rate is 0.001 and divided
by 2 every 2 epochs, and the number of epochs is 10.
In this work, the number and scale of the anchor boxes are deter­
mined by clustering algorithm as in Redmon and Farhadi (2017). The
ground truth box distribution of the training dataset was visualized (see
Fig. 7(a)) to understand the range of object sizes better. Fig. 7(a) shows
that the size and shape of the most groups of objects are similar. Fig. 7(b)
displays the mean IoU (Intersection-over-Union) versus number of
K. Zhang et al. Computers and Electronics in Agriculture 183 (2021) 106064

Table 2 which exceeds the baseline model by 18.17%. The mAP of the early skip
Detection performance comparison between the baseline and MF3 R-CNN. model with the lowest performance in MF3 R-CNN is 69.69%, which still
Detection Average precision (AP) (%) mean AP (mAP) exceeds the baseline model by 4.52%. Fig. 8 shows the comparison
models (%) detection results on a real soybean leaf disease image, which contains
Virus Frogeye leaf Bacterial
disease spot spot five diseased leaves and two kinds of diseases: frogeye leaf spot (two
objects) and virus disease (three objects). The ground-truth image with
Baseline 75.34 66.94 53.22 65.17
Early skip 84.32 67.48 57.26 69.69
annotations and the detection results of the baseline model and MF3 R-
Halfway skip 1 89.66 71.80 61.05 74.17 CNN are displayed. It should be noted that, in the annotations of each
Halfway skip 2 91.80 72.27 85.96 83.34 image, the class label of each disease is simplified to the first word of the
Late skip 91.72 72.11 67.90 77.24 disease for convenience. For example, the “virus disease” is simplified to
Fusion skip 88.44 70.87 51.60 70.30
“virus”, the “frogeye leaf spot” is simplified to “frogeye” and so on. It can
be seen that the baseline model fail to detect the biggest leaf infected by
anchor boxes. Combining the Fig. 7(a) and Fig. 7(b), the number of virus disease, and each MF3 R-CNN model can detect all diseased soy­
anchor boxes is set to 4. The size of the 4 anchor boxes is determined by bean leaves.
the results of the clustering. It has been experimentally shown that the proposed MF3 R-CNN
trained only by synthetic image dataset can effectively detect the soy­
bean leaf disease in complex scene. In addition, the designed halfway
3.2. Experiments between baseline and MF3 R-CNN skip 2 model achieves the highest mAP of 83.34% among all models, and
exceeds the baseline model by 18.17%. Further, the followed sections
In this work, 30 test images collected from soybean field (25 images) will be advanced based on the halfway skip 2 model.
and the internet (5 images) were utilized to evaluate the baseline model
and MF3 R-CNN introduced in Section 2.2.2. Mean Average Precision
(mAP) is selected as the primary evaluation metric in comparison ex­
periments, and the reason given in Ren et al. (2015) is that the mAP is
the actual metric for object detection rather than focusing on object Table 3
proposal proxy metrics. Detection performance comparison between the MF3 R-CNN and state-of-the-
art.
The detection performance comparison between the baseline model
and MF3 R-CNN is presented in Table 2. In addition, a visualization of Model AP(%) mAP (%)
detection results between the baseline model and MF3 R-CNN is dis­ Virus disease Frogeye leaf spot Bacterial spot
played in Fig. 8. In Table 2, the first column gives all detection models in Faster R-CNN 75.04 52.55 30.91 52.83
this work, the rest columns present the average precision of the three YOLO v2 44.58 13.71 19.05 25.78
diseases and the mAP of each detection model. As can be seen from MF3 R-CNN 91.80 72.27 85.96 83.34
Table 2, the halfway skip 2 model obtains the highest mAP of 83.34%,

Fig. 8. Visualization of the detection results between the baseline and MF3 R-CNN.

7
K. Zhang et al. Computers and Electronics in Agriculture 183 (2021) 106064

Fig. 9. Visualization of the comparison results with other approaches.

Fig. 10. Visualization of test results.

3.3. Comparison with state-of-the-art
The comparison experiments with the two state-of-the-art detection
models, Faster R-CNN and YOLO v2 are conducted. We train all models
by the synthetic soybean leaf disease image and test on the real image.
Detection results are presented in Table 3. It should be noted that the
“MF3 R-CNN” in the first column refers to the halfway skip 2 model.
In Table 3, the mAP of MF3 R-CNN, Faster R-CNN and YOLO v2 are
83.34%, 52.83% and 25.78% respectively. Meanwhile, the visualization
of comparison results is given in Fig. 9. It should be noted that the “MF3
R-CNN” in the Fig. 9 refer to the halfway skip 2 model. The ground-truth
image in the first row is the same as the ground-truth image in Fig. 8, and
was collected from soybean field. Five diseased leaves and two kinds of
diseases: frogeye leaf spot (two objects) and virus disease (three objects)

8
K. Zhang et al. Computers and Electronics in Agriculture 183 (2021) 106064

Table 4 In Table 4, the recall rate and precision rate of virus is 93.10% and
Classification confusion matrix for soybean leaf diseases detection in complex 96.43% respectively, which can be obtained that the classification per­
scene. formance of MF3 R-CNN in identifying virus disease is great. However,
Disease Disease category Null Recall rate the classification performance of frogeye leaf spot and bacterial spot still
category
Virus Frogeye leaf Bacterial
(%) needs to be improved. The reason for the failure situations will be dis­
disease spot spot cussed combined with the results visualization subsequently.
For the frogeye leaf spot and the bacterial spot, the precision rate still
Virus disease 27 0 1 1 93.10
Frogeye leaf 0 43 5 4 87.76 needs to be improved. The main reason for confusing the frogeye leaf
spot spot and bacterial spot is that it is hard to distinguish which disease
Bacterial spot 0 0 21 1 95.45 occupies the major composition when the two diseases appear in one
Null 1 6 5 – – leaf simultaneously. Given an example in Fig. 11, the disease category
Precision rate 96.43 87.76 65.63 – –
(%)
corresponding to the enlarged partial image is frogeye leaf spot, but the
detector classified the disease to bacterial spot with a confidence prob­
ability of 0.5925. In the enlarged image, there are many tiny gray-white
are contained in the ground-truth image. The MF3 R-CNN model pre­ spots (belonging to frogeye leaf spot) and two obvious irregular brown
cisely detected all diseased soybean leaves, however, the Faster R-CNN lesions (belonging to bacterial spot). However, the symptom belongs to
model mistakenly detected a healthy leaf as a leaf with frogeye leaf spot, the frogeye leaf spot occupies more region in leaf, meanwhile, according
and the YOLO v2 model only detected two leaves with virus disease. The to the identification of plant protect experts, the disease of the enlarged
ground-truth image in the second row was collected from the internet,
and includes only one leaf with bacterial spot. The MF3 R-CNN precisely
detected the diseased soybean leaf, but the detection results of the Faster
R-CNN and YOLO v2 models are both wrong. Experimental results show
that, the performance of the proposed MF3 R-CNN in detecting soybean
leaf disease in complex scene is superior to the state-of-the-art.

3.4. Visualization of the detection results

The visualization of the detection results of the optimal MF3 R-CNN

was conducted (see Fig. 10). As can be seen in Fig. 10, the MF3 R-CNN
can effectively identify and locate the virus disease, frogeye leaf spot and
bacterial spot of soybean leaf diseases in natural soybean field. In
Fig. 10, most images contain multiple soybean plants and multiple
diseased leaves, and the MF3 R-CNN can recognize and locate the soy­
bean leaf diseases accurately. In addition, for the distant objects and
some partially occluded objects, the MF3 R-CNN can also achieve a
satisfactory performance.

3.5. Evaluation of the multi-classification performance of MF3 R-CNN

For quantitatively analysis the multi-classification performance of

the MF3 R-CNN, the confusion matrix of the test results is presented in
Table 4. Moreover, the precision rate and recall rate of each disease
category are given simultaneously. The “Null” in column represents the
number of missing detection targets, and the “Null” in row indicates the
Fig. 12. Sample image of a missing detection situation.
number of detected objects that are not included in ground-truth image.

Fig. 11. Sample image of a wrong classification situation.

9
K. Zhang et al.
leaf should be the frogeye leaf spot. A missing detection situation is
shown in Fig. 12, the image on the upper right is a leaf with frogeye leaf
spot, which is partially occluded and the spot area is very small. The
image on the lower right is a leaf with bacterial spot, which is partially
occluded and appears in the image in a side posture. Thus, the detection
model did not detect the two diseased leaves.
3.6. Discussion
Due to the insufficient dataset and technical difficulties, the research
on the soybean leaf disease detection in complex scene is scarce.
Therefore, this work propose to use synthetic image dataset to train the
MF3 R-CNN model. The comparison experiment between the baseline
model and MF3 R-CNN indicates that the MF3 R-CNN beats the baseline
model by a large margin (See Table 2). In addition, the comparison
experiment between the MF3 R-CNN and state-of-the-art shows that the
MF3 R-CNN is superior to the state-of-the-art (See Table 3 and Fig. 9).
The MF3 R-CNN trained only by the synthetic image dataset can perform
well on the real test dataset, which not only proves the effectiveness of
the designed model and synthetic image, but also shows that the model
has generalization ability. However, the classification performance of
the MF3 R-CNN needs to be further improved, especially for the iden­
tification of the bacterial spot. For the situations that multiple diseases
appear in one leaf or the diseased leaves are seriously occluded, the MF3
R-CNN may get into trouble.
4. Conclusion
In this work, we target the problem of detecting soybean leaf disease
in complex scene and make improvements in dataset and model. First,
we generate synthetic soybean leaf disease image to tackle the insuffi­
cient dataset problem. Second, the novel MF3 R-CNN model is designed
which mixes multi-features by connecting different layers in the feature
extraction network in a skipping manner. Third, the MF3 R-CNN model
is trained by synthetic image dataset and test on the real image dataset.
The optimal mAP obtained by MF3 R-CNN is 83.34%, and the experi­
mental results indicate that the proposed MF3 R-CNN trained only by
synthetic dataset can effectively detect the soybean leaf disease in
complex scene and superior to the state-of-the-art. We hope the key idea
in this work can trigger some inspiration to other computer vision tasks.
In the future work, other diseases in the soybean field will be added,
besides, the designed model will be packaged and embedded in the
smart phone application.
CRediT authorship contribution statement
Keke Zhang: Conceptualization, Methodology, Software, Writing -
original draft. Qiufeng Wu: Supervision, Writing - review & editing,
Project administration, Funding acquisition. Yiping Chen: Data cura­
tion, Validation, Visualization.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
Acknowledgements
This work was supported by National Key Application Research and
Development Program in China under Grant 2018YFD0300105-2 and
Harbin Applied Technology Research and Development Program under
Grant 2017RAQXJ096.
10
Computers and Electronics in Agriculture 183 (2021) 106064
References
Barth, R., Ijsselmuiden, J., Hemming, J., et al., 2018. Data synthesis methods for
semantic segmentation in agriculture: A Capsicum annuum dataset. Comput.
Electron. Agric. 44, 284–296.
Bell, S., Zitnick, C.L., Bala, K., et al., 2016. Inside-outside net: detecting objects in context
with skip pooling and recurrent neural networks. In: IEEE Conference on Computer
Vision and Pattern Recognition, pp. 2874–2883.
Cicco, M.D., Potena, C., Grisetti, G., et al., 2017. Automatic model based dataset
generation for fast and accurate crop and weeds detection. In: IEEE/RSJ
International Conference on Intelligent Robots and Systems, pp. 5188–5195.
Cruz, A., Ampatzidis, Y., Pierro, R., et al., 2019. Detection of grapevine yellows
symptoms in Vitis vinifera L. with artificial intelligence. Comput. Electron. Agric.
157, 63–76.
Galbally, J., Diaz-Cabrera, M., Ferrer, M.A., et al., 2015. On-line signature recognition
through the combination of real dynamic data and synthetically generated static
data. Pattern Recogn. 48, 2921–2934.
Guan, H., Li, J., Ma, X., et al., 2016. Recognition of soybean nutrient deficiency based on
color characteristics of canopy. J. Northeast A&F Uni. (Natural Science Edition). 44
(12), 136–142.
Gui, J., Wu, Z., Li, K., 2019. Hyperspectral imaging for early detection of soybean mosaic
disease based on convolutional neural network model. J. Zhejiang Univ. (Agric. &
Life Sci.) 45 (2), 256–262.
Gupta, A., Vedaldi, A., Zisserman, A., 2016. Synthetic data for text localisation in natural
images. In: IEEE Conference on Computer Vision and Pattern Recognition,
pp. 2315–2324.
Hattori, H., Boddeti, V.N., Kitani, K., et al., 2015. Learning scene-specific pedestrian
detectors without real data. In: IEEE Conference on Computer Vision and Pattern
Recognition, pp. 3819–3827.
He, K., Gkioxari, G., Dollar, P., et al., 2017. Mask r-cnn. In: IEEE International Conference
on Computer Vision, pp. 2980–2988.
He, K., Zhang, X., Ren, S., et al., 2016. Deep residual learning for image recognition. In:
IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.
Hughes, D., Salathé, M., et al., 2015. An open access repository of images on plant health
to enable the development of mobile disease diagnostics. arXiv preprint arXiv:
1511.08060.
Jia, S., Wang, P., Jia, P., et al., 2017. Research on data augmentation for image
classification based on convolution neural network. In: Chinese Automation
Congress. pp. 4165–4170.
Jiang, F., Li, Y., Yu, D., et al., 2019. Soybean disease detection system based on
convolutional neural network under Caffe framework. Acta Agriculturae
Zhejiangensis. 31 (7), 1177–1183.
KC, K., Yin, Z., Wu M., et al., 2019. Depthwise separable convolution architectures for
plant disease classification. Comput. Electron. Agric. 165, 104948.
Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. ImageNet classification with deep
convolutional neural networks. In: International Conference on Neural Information
Processing Systems, pp. 1097–1105.
Li, C., Song, D., Tong, R., et al., 2019a. Illumination-aware faster R-CNN for robust
multispectral pedestrian detection. Pattern Recogn. 85, 161–171.
Li, J., Shi, C., Shan, Q., et al., 2019b. Visual Identification System of Soybean Frogeye
Leaf Spot Based on SURF Feature Extraction. Soybean Science. 38 (1), 90–96.
Li, Z., Pao, C., Yu, G., et al., 2018. DetNet: A Backbone network for Object Detection.
arXiv:1804.06215v2.
Liang, Q., Xiang, S., Hu, Y., et al., 2019. PD2SE-Net: Computer-assisted plant disease
diagnosis and severity estimation network. Comput. Electron. Agric. 157, 518–529.
Lin, T.Y., Dollár, P., Girshick, R., et al., 2017. Feature Pyramid Networks for Object
Detection. arXiv:1612.03144v2.
Lin, T.Y., Goyal, P., Girshick, R., et al., 2018. Focal loss for dense object detection. arXiv:
1708.02002v2.
Liu, W., Anguelov, D., Erhan, D., et al., 2016. Ssd: Single shot multibox detector. In:
European Conference on Computer Vision, pp. 21–37.
Liu, W., Liao, S., Hu, W., et al., 2018. Learning efficient single-stage pedestrian detectors
by asymptotic localization fitting. European Conference on Computer Vision.
Ma, J., Du, K., Zheng, F., et al., 2018. A recognition method for cucumber diseases using
leaf symptom images based on deep convolutional neural network. Comput.
Electron. Agric. 154, 18–24.
Ma, X., Guan, H., Qi, G., et al., 2017. Diagnosis model of soybean leaf diseases based on
improved cascade neural network. Trans. Chinese Soc. Agric. Mach. 48 (1), 163–168.
Mancini, M., Costante, G., Valigi, P., et al., 2016. Fast robust monocular depth estimation
for obstacle detection with fully convolutional networks. In: IEEE/RSJ International
Conference on Intelligent Robots and Systems, pp. 4296–4303.
Qi, G., Ma, X., Guan, H., 2006. Extraction of the image of soybean target leaf spot based
on improved genetic algorithm. Trans. Chinese Soc. Agric. Eng. 25 (05), 142–145.
Rahnemoonfar, M., Sheppard, C., 2017. Deep count: Fruit counting based on deep
simulated learning. Sensors. 17, 905.
Redmon, J., Divvala, S., Girshick, R., et al., 2016. You only look once: Unified, real-time
object detection. In: IEEE Conference on Computer Vision and Pattern Recognition,
pp. 779–788.
Redmon, J., Farhadi, A., 2017. Yolo9000: Better, faster, stronger. In: IEEE Conference on
Computer Vision and Pattern Recognition, pp. 6517–6525.
Ren, S., He, K., Girshick, R., et al., 2015. Faster r-cnn: Towards real-time object detection
with region proposal networks. In: Advances in Neural Information Processing
Systems. pp, 91–99.
Režnáková, M., Tencer, L., Plamondon, R., et al., 2017. Forgetting of unused classes in
missing data environment using automatically generated data: Application to on-line
handwritten gesture command recognition. Pattern Recogn. 72, 355–367.
K. Zhang et al.
Richter, S.R., Vineet, V., Roth, S., et al., 2016. Playing for data: ground truth from
computer games. In: European Conference on Computer Vision, pp. 102–118.
Silval, A., Bressan, P.O., Goncalves, D.N., et al., 2019. Estimating soybean leaf defoliation
using convolutional neural networks and synthetic images. Comput. Electron. Agric.
156, 360–368.
Simonyan, K., Zisserman, A., 2015. Very deep convolutional networks for large-scale
image recognition. International Conference on Learning Representations.
Sun, P., Chen, G., Cao, L., 2020. Image recognition of soybean pests based on attention
convolutional neural network. J. Chinese Agric. Mechanization. 41 (2), 171–176.
Tetila, E.C., Machado, B.B., Menezes, G.K., et al., 2019. Automatic recognition of
soybean leaf diseases using UAV images and deep convolutional neural networks.
IEEE Geosci. Remote Sens. Lett. 99, 1–5.
Wang, D., Tang, J., Zhu, W., et al., 2018. Dairy goat detection based on Faster R-CNN
from surveillance video. Comput. Electron. Agric. 154, 443–449.
Wang, Y., Liang, W., Shen, J., et al., 2019. A deep Coarse-to-Fine network for head pose
estimation from synthetic data. Pattern Recogn. 94, 196–206.
11
Computers and Electronics in Agriculture 183 (2021) 106064
Wu, Q., Zhang, K., Meng, J., 2019. Identification of soybean leaf diseases via deep
learning. J. Inst. Eng. (India): Series A. 100 (4), 659–666.
Xiong, J., Dai, S., Ou, J., et al., 2020. Leaf deficiency symptoms detection method of
soybean based on dep learning. Trans. Chinese Soc. Agric. Mach. 51 (1), 195–202.
Xue, Y., Zhu, X., Zheng, C., et al., 2018. Lactating sow postures recognition from depth
image of videos based on improved Faster R-CNN. Trans. Chinese Soc. Agric. Eng. 9
(34), 189–196.
Yang, Q., Xiao, D., Lin, S., 2018. Feeding behavior recognition for group-housed pigs
with the Faster R-CNN. Comput. Electron. Agric. 155, 453–460.
Zhang, L., Yang, F., Zhang, Y.D., et al., 2016. Road crack detection using deep
convolutional neural network. In: International Conference on Image Processing,
pp. 3708–3712.
Zhang, L., Mohamed, A.A., Chai, R., et al., 2019. Automated deep learning method for
whole-breast segmentation in diffusion-weighted breast MRI. J. Magn. Reson.
Imaging 51 (2).
Zheng, C., Zhu, X., Yang, X., et al., 2018. Automatic recognition of lactating sow postures
from depth images by deep learning detector. Comput. Electron. Agric. 147, 51–63.