Professional Documents
Culture Documents
Smart Monitoring of Crops Using Generative Adversarial Networks
Smart Monitoring of Crops Using Generative Adversarial Networks
net/publication/335361618
CITATION READS
1 283
4 authors:
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Hamideh Kerdegari on 07 December 2020.
1 Introduction
Weed infestation is a major challenge for the agriculture sector. Early detection
and removal of weed can greatly improve crop yield. Traditional methods of weed
removal are time consuming: they require farmers to physically survey, identify
and treat infested areas. UAV are used by farmers with cameras (RGB, multi
or hyper spectral) to obtain a better view of their farm and identify specific
weed infestations. Ultimately, the goal is to implement more effective treatment
measures and reduce if not eliminate entirely weed infestation [1].
RGB cameras and multispectral sensors flown by UAV have proven to be
useful in early weed detection. Figure 1 illustrates an example with spectral
images of a farm. Captured data is then analyzed by computer vision algorithms
to detect the presence of weed. Studies that utilize RGB cameras mainly apply
2 H. Kerdegari et al.
feature extraction techniques for the detection and classification of weed from
crop. Hung et al. [2] proposed a feature learning based approach with a bank of
image filters to draw image statistics and feed them to a linear classifier to be able
to detect presence of weed from images captured by a RGB camera mounted on
a UAV. In [3], the authors used a commercial camera that operates in the visible
spectrum for a ultra-high resolution image acquisition over wheat fields. Their
study calculated six different vegetation indices based on the RGB spectrum
and achieved a high accuracy of above 85% in the detection and classification
of vegetation. Although RGB image analysis methods can be successfully used
for weed identification and classification [4], RGB images captured by UAV have
few limitations in crop-weed disambiguation. Results in [5] showed that better
performance for weed detection via RGB images was obtained for larger size
of weed plants. To achieve high accuracy feature detection and learning, an
expensive high resolution camera is necessary to capture sufficient details of the
crop and weed. Further, RGB images capture less information at higher altitudes
and it was observed that vegetation indices decrease as altitude increases [6].
Due to the limitations of the visible spectrum (RGB band), use of additional
spectral bands, in the infrared side of the spectrum, have shown to provide
detailed information that enables accurate calculation of vegetation indices and
crop-weed classification [7]. Therefore, multispectral cameras are increasingly
used for crop growth monitoring. Health analysis can be implemented through
extraction of vegetation indices, such as the normalized difference vegetation
index (NDVI), green normalized difference vegetation index (GNDVI) and soil
adjusted vegetation index (SAVI) [8]. These indices are computed from different
spectral bands captured by the multispectral camera and can be further utilized
to analyze vegetation conditions.
Studies employing multispectral cameras apply different approaches for weed
detection. For instance, in [9], an object detection based image analysis method
Intelligent Crop Monitoring using Semi-supervised GAN 3
was proposed that identifies objects in crop rows and applies classification tech-
niques to discriminate crop and weed from the spectral images. Spectral index
variation is an approach that considers spectral reflectance variation to discrim-
inate between crop and weed [6]. Various statistical methods for weed detec-
tion from multispectral images have been proposed and include approaches such
as Mahalanobis distance computation between vegetation rates [10] and partial
least squares discriminant analysis classification models [11] that have shown de-
tection accuracy in range over 80%. Machine learning approaches such as support
vector machine (SVM) method was found to show better accuracy performance
[12], compared to the decision tree (DT) method [13] where multispectral images
were classified along with the use of NDVI thresholding.
More recently, deep learning methods have been explored for crop-weed dis-
ambiguation. In particular, the basic convolutional neural network (CNN) has
gained ground in the analysis and classification of remote sensing data such as
multispectral images [14]. For instance, Sa et al. [15] applied cascaded CNN, Seg-
net, on multispectral image datasets for classifying sugar beet crop from weed.
They trained six models on different spectral channels and achieved a classifica-
tion F1 score of 0.8. The study was further extended to include a sliding window
approach on orthomosaic maps of the farm to apply a deep neural network and
achieved improved performance accuracy [16]. Similarly, several other studies
applied CNN based methods for weed classification with images captured from
both UAV and ground based vehicles and achieved a high performance accuracy
[17], [18].
However, a large amount of training data for learning is an inherent require-
ment of deep learning methods. The lack of large corpora of specific labeled
data is a challenge in general and for multi-spectral data in particular. Further-
more, collecting large corpora of multispectral image data with UAV platforms
for crop-weed classification system is time consuming and expensive. To address
this challenge, this paper utilizes a semi-supervised version of the generative ad-
versarial networks (GAN) [19]. This method generates photo-realistic crop-weed
images and can be employed to augment training data. In the presented GAN
based semi-supervised classification method, a generator creates large realistic
images, in turn, forcing a discriminator to learn better features for more accurate
pixel classification. To the best of our knowledge, application of GAN methods
for multispectral image classification is not well explored and our work addresses
this research gap. The main contributions of this paper are:
– first application of semi-supervised GAN for classification of multispectral
images acquired by UAV,
– investigation of limited annotated data for multispectral image classification
task.
Section 2 presents the proposed approach by providing a brief background of
GAN and semi-supervised GAN, then the design and structure of the proposed
model for semi-supervised learning is described in system overview. Section 3
deals with experimental results, where results on the weedNet dataset [15] are
presented, finally section 4 concludes the paper.
4 H. Kerdegari et al.
2 Proposed Approach
This section presents a brief background about GAN, semi-supervised GAN and
then describes the proposed network architecture for semi-supervised pixel-wise
classification of multispectral crop/weed imaging data.
Fig. 2. The semi-supervised GAN architecture. Random noise is used by the Generator
to generate an image. The Discriminator uses generated data, unlabeled data and
labeled data to learn class confidences and produces confidence maps for each class as
well as a label for a fake data.
Fig. 3. The network architecture of our semi-supervised GAN. The noise is a vector
of size 100 sampled from a uniform distribution and is used as input to the generator.
The number of feature maps in the four different convolutional layers, respectively, are
256, 128, 64, 32 and 1 (Here 1 shows the number of channels).
classes, and accordingly, the label with the highest probability is assigned to the
pixel. Figure 4 shows some generated images in different channels. Interestingly,
these images indicate that the semi-supervised GAN framework is able to learn
spatial object patterns, for example, crop shape and weed shape.
Fig. 4. Images generated by the generator of the semi-supervised GAN on the weedNet
dataset. Interestingly, patterns related to crops and weeds from NDVI, Red and NIR
channel can be observed that highlights the effectiveness of the approach.
3 Experimental Results
N IR − Red
N DV I = (2)
N IR + Red
is extracted indicating the difference between soil and plant. Therefore, each
training/test image consists of the 790nm NIR channel, the 660nm Red chan-
nel, and NDVI imagery. The dataset contains only crop, or weed, or crop-weed
combination along with their corresponding pixel-wise annotated data. For semi-
supervised training, different percentages of pixel-wise annotated images (such
as 50%, 40% and 30%) are used as labeled data to the discriminator and the
rest of images are without pixel-wise annotations. As metric, F1 score measure
that is a harmonic average of precision and recall is employed:
precision ∗ recall
F1 = 2 ∗ (3)
precision + recall
Where precision is T PT+F
P TP
P , recall is T P +F N , TP, FP and FN indicate the
number of true positive, false positive and false negative, respectively. Quanti-
tative results of our method on weedNet are shown in Table 1. F1 measure with
a varying number of input channels and different amount of labelled data are
used as evaluation metric in this experiment. Considering the difficulty of the
dataset, all models (including different channels + different amount of labeled
data) perform reasonably well (about 80% for all classes). As shown in Table 1,
two input channels (Red and NIR) yield higher performance compared to single
channels as they contain more useful features to be used by the semi-supervised
GAN network. However, using 3 channels (NDVI + Red + NIR) did not improve
performance as NDVI depends on NIR and Red channels rather than captur-
ing new information. Furthermore, the network was evaluated by reducing the
Table 1. Results on the weedNet dataset using 50%, 40% and 30% of labeled data
with different number of channels for semi-supervised GAN, and cascaded CNN [15]
with fully labeled data. Higher F1 values indicate better classification performance.
amount of labeled data starting at 50% and then reducing by step 10 to 30% to
find out how it affects the classification performance. It is expected that higher
amount of labeled data result in better performance. It can be seen by comparing
the results of the 50%, 40% and 30% in Table 1.
Results of decoder-encoder cascaded CNN on weedNet dataset [15] is shown
in the last column of Table 1. As compared with our semi-supervised GAN,
it achieved higher accuracy using fully labeled data. However, we showed that
semi-supervised GAN with limited training data ables to achieve a good accuracy
about 80%.
Qualitative results on some sample images are depicted in Figure 5. As it
is shown, each row contains original Red channel, NIR channel, NDVI imagery,
semi-supervised GAN probability output and the corresponding ground truth.
The probability of each class is mapped to the red, green and black color rep-
resenting weed, crop and background, respectively. There are some noticeable
weed and crop misclassification areas in the images that occur mostly when crop
and weed are surrounded by each other. This misclassification shows that net-
work can capture high-level features such as shape and texture in addition to
the low-level features.
Fig. 5. Qualitative results of some sample images from the weedNet test set. The first
three columns are input data to the semi-supervised GAN, the fourth is the results of
semi-supervised GAN using 30% of labeled data and the last column is ground truth.
Intelligent Crop Monitoring using Semi-supervised GAN 9
4 Conclusion
References
1. Lottes, P., Khanna, R., Pfeifer, J., Siegwart, R. and Stachniss, C.: UAV-based crop
and weed classification for smart farming. In: IEEE International Conference on
Robotics and Automation (ICRA), pp.3024-3031 (2017)
2. Hung, C., Xu, Z. and Sukkarieh, S.: Feature learning based approach for weed
classification using high resolution aerial images from a digital camera mounted on
a UAV. In: Remote Sensing, 6(12), pp.12037-12054 (2014)
3. Torres-Snchez, J., Pea, J.M., de Castro, A.I. and Lpez-Granados, F.: Multi-temporal
mapping of the vegetation fraction in early-season wheat fields using images from
UAV. In: Computers and Electronics in Agriculture, 103, pp.104-113 (2014)
4. Herrera, P.J., Dorado, J. and Ribeiro, A.: A novel approach for weed type classifi-
cation based on shape descriptors and a fuzzy decision-making method. In: Sensors,
14(8), pp.15304-15324 (2014)
5. Pea, J.M., Torres-Snchez, J., Serrano-Prez, A., de Castro, A.I. and Lpez-Granados,
F.: Quantifying efficacy and limits of unmanned aerial vehicle (UAV) technology for
10 H. Kerdegari et al.
weed seedling detection as affected by sensor resolution. In: Sensors, 15(3), pp.5609-
5626 (2015)
6. Samseemoung, G., Soni, P., Jayasuriya, H.P. and Salokhe, V.M.: Application of
low altitude remote sensing (LARS) platform for monitoring crop growth and weed
infestation in a soybean plantation. In: Precision Agriculture, 13(6), pp.611-627
(2012)
7. Lpez-Granados, F., Torres-Snchez, J., Serrano-Prez, A., de Castro, A.I., Mesas-
Carrascosa, F.J. and Pea, J.M.: Early season weed mapping in sunflower using
UAV technology: variability of herbicide treatment maps against weed thresholds.
In: Precision Agriculture, 17(2), pp.183-199 (2016)
8. Bannari, A., Morin, D., Bonn, F. and Huete, A.R.: A review of vegetation indices.
In: Remote sensing reviews, 13(1-2), pp.95-120 (1995)
9. Pena, J.M., Torres-Snchez, J., de Castro, A.I., Kelly, M. and Lpez-Granados, F.:
Weed mapping in early-season maize fields using object-based analysis of unmanned
aerial vehicle (UAV) images. In: PloS one, 8(10), p.e77151 (2013)
10. Louargant, M., Villette, S., Jones, G., Vigneau, N., Paoli, J.N. and Ge, C.: Weed
detection by UAV: Simulation of the impact of spectral mixing in multispectral
images. In: Precision Agriculture, 18(6), pp.932-951 (2017)
11. Herrmann, I., Shapira, U., Kinast, S., Karnieli, A. and Bonfil, D.J.: Ground-level
hyperspectral imagery for detecting weeds in wheat fields. In: Precision agriculture,
14(6), pp.637-659 (2013)
12. Ishida, T., Kurihara, J., Viray, F.A., Namuco, S.B., Paringit, E.C., Perez, G.J.,
Takahashi, Y. and Marciano, J.J.: A novel approach for vegetation classification
using UAV-based hyperspectral imaging. In: Computers and Electronics in Agricul-
ture, 144, pp.80-85 (2019)
13. Natividade, J., Prado, J. and Marques, L.: Low-cost multi-spectral vegetation clas-
sification using an Unmanned Aerial Vehicle. In: IEEE International Conference on
Autonomous Robot Systems and Competitions (ICARSC), pp. 336-342 (2017)
14. Mahdianpari, M., Salehi, B., Rezaee, M., Mohammadimanesh, F. and Zhang, Y.:
Very deep convolutional neural networks for complex land cover mapping using
multispectral remote sensing imagery. Remote Sensing, 10(7), p.1119 (2018)
15. Sa, I., Chen, Z., Popovi, M., Khanna, R., Liebisch, F., Nieto, J. and Siegwart, R.:
weedNet: Dense Semantic Weed Classification Using Multispectral Images and MAV
for Smart Farming. In: IEEE Robotics and Automation Letters, 3(1), pp.588-595
(2016)
16. Sa, I., Popovi, M., Khanna, R., Chen, Z., Lottes, P., Liebisch, F., Nieto, J., Stach-
niss, C., Walter, A. and Siegwart, R.: WeedMap: A large-scale semantic weed map-
ping framework using aerial multispectral imaging and deep neural network for
precision farming. In: Remote Sensing, 10(9), p.1423 (2018)
17. Bah, M.D., Dericquebourg, E., Hafiane, A. and Canals, R.: Deep learning based
classification system for identifying weeds using high-resolution UAV imagery. In:
Science and Information Conference, Springer, pp.176-187 (2018)
18. Lottes, P., Behley, J., Milioto, A. and Stachniss, C.: Fully Convolutional Networks
with Sequential Information for Robust Crop and Weed Detection in Precision Farm-
ing. In: arXiv preprint, arXiv:1806.03412 (2018)
19. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S.,
Courville, A. and Bengio, Y.: Generative adversarial nets. In: Advances in neural
information processing systems, pp. 2672-2680 (2014)
20. Radford, A., Metz, L. and Chintala, S.: Unsupervised representation learn-
ing with deep convolutional generative adversarial networks. In: arXiv preprint,
arXiv:1511.06434 (2015)