Professional Documents
Culture Documents
Zefri Et Al - 2023 - Advanced Classification of Failure-Related Patterns On Solar Photovoltaic Farms
Zefri Et Al - 2023 - Advanced Classification of Failure-Related Patterns On Solar Photovoltaic Farms
Abstract— Here, we propose an approach that relies on temperatures, the intrusion of moisture, diverse precipitation
digital photogrammetry and deep learning to classify thermal types, and the UV sun radiation [3], [4]. Across the literature,
infrared patterns sheltering potential failures within solar the reported degradation rates vary differently, depending on
panels from aerial imagery collected by drones. We collect the panels’ location, along with the dominant degradation
images from a solar plant using a rotary-wing drone equipped factors [5]–[8]. In [5], the reported rates range from −0.8% to
with an onboard thermal camera. The captured images are −4.9% per year. The affected PV panels have degraded
processed using a photogrammetric pipeline that stitches the performance, because their individual components cease to
images together producing a georeferenced thermal
function properly; as a consequence, the overall performance
orthomosaic. The solar panels are digitized, extracted from the
orthomosaic, labeled into 4 classes, augmented using
is reduced. PV panels are arranged within arrays of multiple
transformations acting on their geometry and radiometry then layouts. When only one panel is degraded due to the presence
utilized to constitute a dataset to train from scratch and validate of either one or multiple failures, the whole PV array is
a developed deep learning classifier. The latter consists of a affected, because the output of the array is only as strong as
convolutional neural network architecture comprising two core its weakest component. If one panel is not performing
blocks: (1) a convolutional block that produces multi-level optimally, the entire array will be penalized [9], [10]. This is
feature maps from the images, followed by (2) a multi-layer why it is important to monitor PV panels and detect failures
perceptron block that classifies the constructed feature maps early to perform timely maintenance interventions [1], [11].
according to the considered categories. The final developed
model scores an F1-score of 98.2% on our validation sub-
Digital imagery, based especially on the thermal infrared
dataset, which confirms both its high performance and spectrum, has proven itself as an efficient and reliable way to
generalizability on additional data. The proposed approach diagnose PV panels. Panels with failures tend to overheat at
elaborates an efficient, comprehensive and cost-effective the affected parts, which makes from the long-wave thermal
framework to monitor solar farms through the use of drone- infrared spectrum ([8 − 14𝜇𝑚]) a suitable way to spot any
based thermal sensing, photogrammetry and deep learning, unusual excessive temperature patterns [11], [12]. Parallelly,
alongside addressing the drawbacks related to the use of classic the use of drones in monitoring is becoming increasingly
techniques. popular in a variety of industries [13]–[16], among which the
energy industry occupies a prominent place [17]–[19]. For PV
Keywords— Inspection; Solar farm; Deep learning; Drone; inspection, drones can be used to inspect hard-to-reach PV
Thermal sensing; failure arrays; they can also provide a time-saving and cost-effective
way to overfly large-scale farms collecting images using
I. INTRODUCTION lightweight cameras [20]–[22]. After image collection, failure
During their operating lifetime, solar photovoltaic (PV) detection can be carried out by analyzing the images, for color,
panels are affected by multiple failures because they are made texture or shape changes that are likely to shelter failures.
up of many assembled components, such as toughened glass,
aluminum framing, ethylene vinyl acetate encapsulation, Various works have been conducted on the use of aerial
wiring, and other manufacturing materials [1], [2]. These thermal sensing for failure detection on solar PV panels. In
components are subject to being damaged or degraded over [23], the authors performed a flight campaign of a utility-scale
time due to environmental factors, such as extreme ambient PV solar farm through aerial thermography. Post-flight image
analysis has been carried out to spot PV panels having failures.
Authorized licensed use limited to: MIT Libraries. Downloaded on May 28,2024 at 15:43:50 UTC from IEEE Xplore. Restrictions apply.
The use of aerial imagery was concluded to provide faster which makes the use of a GPS/GNSS-based system to
results; however, the study did not propose any automatic navigate the PV farm in search of panels with failures not
approach nor models for failure detection on images. The practical.
panels were browsed manually by an operator in search of
overheating components. In [24], the authors proposed a An analysis of the previous works highlights three main
Convolutional Neural Network (CNN) that classifies aerial issues summarized as follows:
captures into images containing one or multiple panels having • Automatic failure detection: some approaches rely on
failures, and images with entirely normal panels. The eyesight-based examination of the images, which is a
collected dataset comprised 3336 thermal images, with 811 tiring and error-prone process, especially when it
samples corresponding to the first class (images having no comes to big data collected from large-scale farms.
panel with a failure), on which several augmentations have
been applied to enhance the representativeness. The • The geolocalization issue, which relates to finding a
developed model scored an F1-score of 75%; however, the panel with a failure on-field based on its source aerial
global proposed approach has a major drawback, which is image: while the majority of works have gone into
providing no cues on the location of the affected panels, and failure detection on images, the posterior
this makes any on-field intervention a difficult task. geolocalization issue has received very little interest.
In [25], the authors adopted an object detection approach, • The multiplicity of failure-related thermal patterns:
where they used a YOLOv5-based architecture to extract PV hotspots are the predominant thermal pattern on PV
panels along with hotspot failures through drawing bounding models [29], which explains the reasons why their
boxes around. The dataset they have prepared comprised 5600 detection has received the major interest in the
aerial images with a total of 124000 modules approximately, literature. However, other failure-related patterns also
and has been captured from publicly available videos. The exist, and they also have to be addressed to perform a
final obtained model achieved a mean average precision of comprehensive diagnosis of any solar farm.
98.1%; however, the study presents two major drawbacks. This work addresses the aforementioned issues through
The first is related to the constitution of the dataset, where an developing an approach that relies on a deep neural network-
over duplication is observed due to the high overlap between based algorithm for the automatic detection of failure-related
the captured images. The second relates to the impossibility of PV thermal patterns. An exhaustive diagnosis is provided
retrieving a PV panel from its source image, since the through considering four classes. The approach uses as a
geolocalization information has been discarded along the starting point an orthomosaic generated through a
process of image capturing from the videos. A similar photogrammetry-based stitching process. The latter enables a
approach is proposed in [26] (the object detection approach). uniform imaging of the whole farm, along with making it
After extracting the PV panels, the authors used this time a possible and straightforward to locate the occurred failures
Region-based CNN (R-CNN) to extract hotspots from the conveniently on-field for posterior on-site interventions. The
images of panels. The approach was developed starting from remainder of this work is structured as follows: Section 2
a dataset that was collected from six PV farms, with a total describes the materials and methods used to carry out the
around 9000 panels. The obtained detector scored an accuracy work. Section 3 presents the obtained results that are discussed
of 96%; however, while the approach enables to locate PV in Section 4 prior to a general conclusion with
panels having failures through the use of orthomosaics, only recommendations.
hotspots were considered in this study. This makes from the
study not sufficiently extensive, since PV panels are subject to II. METHODOLOGY
other failure-related thermal patterns such as pointed heating,
entirely overheated panels and aligned overheating cells. A. Aerial image collection
Another study that relies anew on object detection through an A PV solar farm was selected in order to conduct our on-
R-CNN model was presented in [27]. At a first level, the field image collection. The site choice was determined by two
authors performed an extraction of the PV arrays, then at a criteria. The first is the possibility to conduct an acquisition
second level, they extracted the hotspots contained within mission, following permission from the farm owners. The
each array. While the study tackled the geolocalization issue second is comprising the highest possible number of panels
through relying on the exterior orientation parameters that are with diverse thermal failures so as to construct a representative
embedded within each captured image, it was tailored dataset for the classifier to develop. A prior flight planning has
according to a PV farm with a specific and rare panel been performed following three steps:
arrangement, which calls into question its generalizability to
other farms. In addition, only one failure pattern was • Delimitation of the interest area;
addressed (hotspots), while practically several other pattern • Determination of the flight height (30m) along with the
types are found within real-world scenarios. overlaps (85% for the end and side laps);
• Generation of the flight lines and saving the project
In [28], the authors proposed an approach where the issue plan as a KML vector file for on-field use.
of locating PV panels on field has been taken into account. For
that, they have generated a panorama of the stitched drone Aerial images were collected with a hexa-copter drone
images based on feature detection and matching, then type DJI M210 (Fig. 1). On it, was mounted a thermal infrared
performed image segmentation to extract the PV panels. The camera type Zenmuse XT2 of focal length 9mm, a pixel pitch
latter were fed to a CNN model that was trained and validated of 17μm and a resolution of 640×512 pixels. The panels were
to classify seven thermal failures. While the classifier overflown at early morning to avoid the apparition of sun
achieved an accuracy of 97.52%, the prepared dataset was too reflections with the camera set a nadiral viewing position.
small to be considered as representative. Also, the generated
panorama was deprived from a geographic reference system,
Authorized licensed use limited to: MIT Libraries. Downloaded on May 28,2024 at 15:43:50 UTC from IEEE Xplore. Restrictions apply.
Fig. 1. The used drone with the onboard camera (DJI Matrice 210).
Authorized licensed use limited to: MIT Libraries. Downloaded on May 28,2024 at 15:43:50 UTC from IEEE Xplore. Restrictions apply.
into account false positives and false negatives at the same
time.
For every training/validation iteration, the learning curves
(where the loss function along with the F1-score are plotted as
a function of epochs) were inspected in search of underfitting
or overfitting pattern. The complexity and hyperparameters of
the model were adjusted accordingly until obtaining a model
that has been trained under an optimal regime. In underfitting
cases, the model was mainly simplified through removing
fully-connected layers or reducing the number of neurons per
layer; reduction of convolutional/pooling layers and the filters
they contain was also performed. For overfitting cases,
opposite actions were carried out to obtain a less complex
model, along with using L2 regularization that penalizes large Fig. 4. A clipped region from the generated thermal orthomosaic,
showcasing various observed thermal patterns related to failures on the
model weights and dropout that randomly drops neurons at a overflown PV farm.
fixed rate during training.
Data preparation and model development were performed B. Dataset preparation and model development
using the Python programming language and the deep learning Table I summarizes the number of the obtained images for
library TensorFlow (V2.8). each class after panel extraction, then after image
augmentation. As presented, the final obtained dataset has a
III. RESULTS AND DISCUSSION high representativeness regarding all the considered classes.
A. Aerial image collection and post-flight image processing
TABLE I. SUMMARY OF THE PREPARED DATASET
In total, 325 images have been collected from the PV farm,
which contained polycrystalline panels. A sample image is Class Initial dataset Augmented dataset
0 177 885
illustrated in Fig. 3. Images have a resolution of 640x512 1 146 730
pixels each. At the chosen flight height (30m), a ground 2 69 345
sampling distance (GSD) around 5.6cm/pixel was obtained. 3 72 360
As shown in Fig. 3, the GSD enables spotting the overheating Total 464 2320
parts of the panels conveniently. The images also feature
significant radial lens distortions, since a wide-angle camera
has been used, with a focal length of 9mm. The distortions will The final adopted model is presented in Fig. 5 and consists
be corrected after the photogrammetric post-flight processing. of two blocks:
A convolution/pooling layer block, which contains:
• A 2D convolution layer (Conv2D_1) containing 8
filters of size 3 × 3, implemented with a stride of 1 and
a ''same'' padding, with ReLU activation ;
• A 2 × 2 2D MaxPooling layer (MaxPooling2D_1) ;
• A 2D convolution layer (Conv2D_2) containing 6
filters of size 3 × 3, implemented with a stride of 1 and
a padding of type ''same'', with ReLU activation ;
• A 2 × 2 2D MaxPooling layer (MaxPooling2D_2);
• A 2D convolution layer (Conv2D_3) containing 6
filters of size 3 × 3, implemented with a stride of 1 and
a padding of type ''same'', with ReLU activation ;
• A 2 × 2 MaxPooling 2D layer (MaxPooling2D_3) ;
• A Flatten layer, which transforms the feature elements
extracted by the convolution layers into a single one-
Fig. 3. A sample collected image from the dataset, featuring radial lens dimensional vector (𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 ).
distortions.
A multi-layer perceptron block, which contains:
Fig. 4 shows a clipped part from the generated • A dense layer (Dense_1) of 64 neurons, with ReLU
orthomosaic. The latter suppresses lens distortions and activation and ridge regression (L2);
stitches the images in a way that produces a uniform scene • A dropout layer (Dropout_1) with a ratio of 0.5;
across the whole site. Various thermal patterns are observed • A dense layer (Dense_2) of 64 neurons, with ReLU
on the panels, which makes from the orthomosaic a suitable activation and ridge regression (L2);
basis to constitute a representative dataset for the addressed • A dropout layer (Dropout_2) with a ratio of 0.5;
four-class classification task. • A dense layer (Dense_3) of 4 neurons, with Softmax
activation, at which the network gives as output the
final predicted class.
Overall, the final adopted model comprised a total of
11092 trainable parameters.
Authorized licensed use limited to: MIT Libraries. Downloaded on May 28,2024 at 15:43:50 UTC from IEEE Xplore. Restrictions apply.
plateau from epoch 70. The curves converge around 98%
without a generalization gap, which confirms that model
learning has been performed within an optimal regime leading
to a high performance metric. The final obtained F1-score
value on the validation dataset was 98.2%.
Authorized licensed use limited to: MIT Libraries. Downloaded on May 28,2024 at 15:43:50 UTC from IEEE Xplore. Restrictions apply.
satisfactory for its effective deployment in real-world Soc. Environ., vol. 26, no. February, p. 100712, 2022, doi:
maintenance missions. In the meantime, there are several 10.1016/j.rsase.2022.100712.
recommendations that we suggest for future work: [15] C. Gomez and H. Purdie, “UAV- based Photogrammetry and
Geocomputing for Hazards and Disaster Risk Monitoring – A Review,”
• Constituting a dataset that features other PV panel Geoenvironmental Disasters, vol. 3, no. 1, 2016, doi: 10.1186/s40677-
016-0060-y.
types namely monocrystalline and thin-film;
[16] Y. Sun et al., “UAV and IoT-Based Systems for the Monitoring of
• Extending the classification task to additional classes; Industrial Facilities Using Digital Twins: Methodology, Reliability
• Developing a classifier that also addresses the Models, and Application,” Sensors, 2022, doi: 10.3390/s22176444.
classification of failures that are detectable using the [17] S. Asadzadeh, W. J. de Oliveira, and C. R. de Souza Filho, “UAV-
RGB spectrum. based remote sensing for the petroleum industry and environmental
monitoring: State-of-the-art and perspectives,” J. Pet. Sci. Eng., vol.
ACKNOWLEDGMENT 208, no. July 2021, 2022, doi: 10.1016/j.petrol.2021.109633.
[18] T. R. Wanasinghe, R. G. Gosine, O. De Silva, G. K. I. Mann, L. A.
This work was supported by the Research Institute for James, and P. Warrian, “Unmanned aerial systems for the oil and gas
Solar Energy and New Energies (IRESEN) in Rabat industry: Overview, applications, and challenges,” IEEE Access, vol.
(Morocco) [Reference: Green INNO-PROJECT-09- 8, pp. 166980–166997, 2020, doi: 10.1109/ACCESS.2020.3020593.
SmartDrone4PV]. [19] M. H. Nordin, S. Sharma, A. Khan, M. Gianni, S. Rajendran, and R.
Sutton, “Collaborative Unmanned Vehicles for Inspection,
REFERENCES Maintenance, and Repairs of Offshore Wind Turbines,” Drones, vol. 6,
no. 6, pp. 1–26, 2022, doi: 10.3390/drones6060137.
[20] A. Kirsten Vidal de Oliveira, M. Aghaei, and R. Rüther, “Aerial
[1] L. Koester, S. Lindig, A. Louwen, A. Astigarraga, G. Manzolini, and
infrared thermography for low-cost and fast fault detection in utility-
D. Moser, “Review of photovoltaic module degradation , field
scale PV power plants,” Sol. Energy, vol. 211, no. November, pp. 712–
inspection techniques and techno-economic assessment,” Renew.
724, 2020, doi: 10.1016/j.solener.2020.09.066.
Sustain. Energy Rev., vol. 165, no. November 2021, p. 112616, 2022,
doi: 10.1016/j.rser.2022.112616. [21] S. Gallardo-saavedra, L. Hernández-callejo, and O. Duque-perez,
“Technological review of the instrumentation used in aerial
[2] L. Hernández-Callejo, S. Gallardo-Saavedra, and V. Alonso-Gómez,
thermographic inspection of photovoltaic plants,” Renew. Sustain.
“A review of photovoltaic systems: Design, operation and
Energy Rev., vol. 93, no. May, pp. 566–579, 2018, doi:
maintenance,” Sol. Energy , 2019, doi: 10.1016/j.solener.2019.06.017.
10.1016/j.rser.2018.05.027.
[3] M. Aghaei et al., “Review of degradation and failure phenomena in
[22] N. M. Kumar, K. Sudhakar, M. Samykano, and V. Jayaseelan, “On the
photovoltaic modules,” Renew. Sustain. Energy Rev., vol. 159, no. July
technologies empowering drones for intelligent monitoring of solar
2021, p. 112160, 2022, doi: 10.1016/j.rser.2022.112160.
photovoltaic power plants,” Procedia Comput. Sci., vol. 133, pp. 585–
[4] C. Ferrara and D. Philipp, “Energy Procedia Why Do PV Modules 593, 2018, doi: 10.1016/j.procs.2018.07.087.
Fail ?,” no. 2011, 2012, doi: 10.1016/j.egypro.2012.02.046.
[23] A. Kirsten Vidal de Oliveira, M. Aghaei, and R. Rüther, “Aerial
[5] J. Kim, M. Rabelo, S. P. Padi, H. Yousuf, E. Cho, and J. Yi, “A Review infrared thermography for low-cost and fast fault detection in utility-
of the Degradation of Photovoltaic Modules for Life Expectancy,” scale PV power plants,” Sol. Energy, vol. 211, no. February 2021, pp.
Energies, vol. 14, pp. 1–21, 2021, doi: 712–724, 2020, doi: 10.1016/j.solener.2020.09.066.
https://doi.org/10.3390/en14144278.
[24] R. Pierdicca, E. S. Malinverni, F. Piccinini, M. Paolanti, A. Felicetti,
[6] S. W. Adler, M. S. Wiig, A. Skomedal, H. Haug, and E. S. Marstein, and P. Zingaretti, “Deep convolutional neural network for automatic
“Degradation Analysis of Utility-Scale PV Plants in Different Climate detection of damaged photovoltaic cells,” in International Archives of
Zones,” IEEE J. Photovoltaics, vol. 11, no. 2, pp. 513–518, 2021, doi: the Photogrammetry, Remote Sensing and Spatial Information
10.1109/JPHOTOV.2020.3043120. Sciences - ISPRS Archives, 2018, vol. 42, no. 2, pp. 893–900. doi:
[7] L. Lillo-Sánchez, G. López-Lara, J. Vera-Medina, E. Pérez-Aparicio, 10.5194/isprs-archives-XLII-2-893-2018.
and I. Lillo-Bravo, “Degradation analysis of photovoltaic modules after [25] Q. Zheng, J. Ma, M. Liu, Y. Liu, Y. Li, and G. Shi, “Lightweight Hot-
operating for 22 years. A case study with comparisons,” Sol. Energy, Spot Fault Detection Model of Photovoltaic,” Sensors (Switzerland),
vol. 222, no. May, pp. 84–94, 2021, doi: vol. 22, 2022, doi: https://doi.org/10.3390/s22124617.
10.1016/j.solener.2021.04.026.
[26] M. Vlaminck, R. Heidbuchel, W. Philips, and H. Luong, “Region-
[8] B. Aboagye, S. Gyamfi, E. A. Ofosu, and S. Djordjevic, “Degradation Based CNN for Anomaly Detection in PV Power Plants Using Aerial
analysis of installed solar photovoltaic (PV) modules under outdoor Imagery,” Sensors, vol. 22, pp. 1–18, 2022, doi:
conditions in Ghana,” Energy Reports, vol. 7, pp. 6921–6931, 2021, https://doi.org/10.3390/s22031244.
doi: 10.1016/j.egyr.2021.10.046.
[27] Á. Huerta Herraiz, A. Pliego Marugán, and F. P. García Márquez,
[9] L. Casula, G. D’Amico, G. Masala, and F. Petroni, “Performance “Photovoltaic plant condition monitoring using thermal images
estimation of photovoltaic energy production,” Lett. Spat. Resour. Sci., analysis by convolutional neural network-based structure,” Renew.
vol. 13, no. 3, pp. 267–285, 2020, doi: 10.1007/s12076-020-00258-x. Energy, vol. 153, pp. 334–348, 2020, doi:
[10] J. M. Álvarez et al., “Analytical modeling of current-voltage 10.1016/j.renene.2020.01.148.
photovoltaic performance: an easy approach to solar panel behavior,” [28] C. J. Kuo, S. Chen, and C. Huang, “Automatic detection , classification
Appl. Sci., vol. 11, no. 9, 2021, doi: 10.3390/app11094250. and localization of defects in large photovoltaic plants using unmanned
[11] I. Høiaas, K. Grujic, A. Gerd, and I. Burud, “Inspection and Condition aerial vehicles ( UAV ) based infrared ( IR ) and RGB imaging,”
Monitoring of Large-Scale Photovoltaic Power Plants : A Review of Energy Convers. Manag., vol. 276, no. September 2022, p. 116495,
Imaging Technologies,” Renew. Sustain. Energy Rev., vol. 161, no. 2023, doi: 10.1016/j.enconman.2022.116495.
February, p. 112353, 2022, doi: 10.1016/j.rser.2022.112353. [29] M. Dhimish and A. M. Tyrrell, “Power loss and hotspot analysis for
[12] S. A. Rahaman, T. Urmee, and D. A. Parlevliet, “PV system defects photovoltaic modules affected by potential induced degradation,” npj
identification using Remotely Piloted Aircraft (RPA) based infrared Mater. Degrad., 2022, doi: 10.1038/s41529-022-00221-9.
(IR) imaging: A review,” Sol. Energy, vol. 206, no. November 2019, [30] M. M. Najafabadi, F. Villanustre, T. M. Khoshgoftaar, N. Seliya, R.
pp. 579–595, 2020, doi: 10.1016/j.solener.2020.06.014. Wald, and E. Muharemagic, “Deep learning applications and
[13] J. G. A. Barbedo, “A review on the use of unmanned aerial vehicles challenges in big data analytics,” J. Big Data, vol. 2, no. 1, pp. 1–21,
and imaging sensors for monitoring and assessing plant stresses,” 2015, doi: 10.1186/s40537-014-0007-7.
Drones, vol. 3, no. 2, pp. 1–27, 2019, doi: 10.3390/drones3020040. [31] S. Hao, Y. Zhou, and Y. Guo, “A Brief Survey on Semantic
[14] N. Amarasingam, A. S. Ashan Salgadoe, K. Powell, L. F. Gonzalez, Segmentation with Deep Learning,” Neurocomputing, vol. 406, pp.
and S. Natarajan, “A review of UAV platforms, sensors, and 302–321, 2020, doi: 10.1016/j.neucom.2019.11.118.
applications for monitoring of sugarcane crops,” Remote Sens. Appl.
Authorized licensed use limited to: MIT Libraries. Downloaded on May 28,2024 at 15:43:50 UTC from IEEE Xplore. Restrictions apply.