A Deep Learning Approach for Automated Fault
Detection on Solar Modules Using Image Composites
Anne Gerd Imenes1,2, Nadia Saad Noori1, Ole Andreas Nesvåg Uthaug2, Robert Kröni3, Filippo Bianchi1,4,
and Nabil Belbachir1
2021 IEEE 48th Photovoltaic Specialists Conference (PVSC) | 978-1-6654-1922-2/21/$31.00 ©2021 IEEE | DOI: 10.1109/PVSC43889.2021.9518540
1
NORCE Technology, Norwegian Research Centre AS, 4879 Grimstad, Norway
2
Department of Engineering Sciences, University of Agder, 4879 Grimstad, Norway
3
Jendra Power AG, 8635 Dürnten, Switzerland
4
Department of Mathematics and Statistics, UiT the Arctic University of Norway, 9019 Tromsø, Norway
Abstract—Aerial inspection of solar modules is becoming
increasingly popular in automatizing operations and maintenance
in large-scale photovoltaic power plants. Current practices are
typically time-consuming as they make use of manual acquisitions
and analysis of thousands of images to scan for faults and
anomalies in the modules. In this paper, we explore and evaluate
the use of computer vision and deep learning methods for
automating the analysis of fault detection and classification in
large scale photovoltaic module installations. We use
convolutional neural networks to analyze thermal and visible color
images acquired by cameras mounted on unmanned aerial
vehicles. We generate composite images by overlaying the thermal
and visible images to investigate improvements in detection
accuracy of faint features related to faults on modules. Our main
goal is to evaluate whether image processing with multi-
wavelength composite images can improve both the detection and
the classification performance compared to using thermal images
alone. The hypothesis is that fusion of images acquired at different
wavelengths (i.e., thermal infrared, red, green, and blue visible
ranges) would enhance the multi-wavelength representation of
faults and thus their histogram feature signatures. The results
showed a successful detection and localization of faint fault
features using composite images. However, the classification of the
fault categories did not show significant improvements and needs
continued investigation. This research represents a step towards
the design of robust automated methods to improve fault detection
from airborne images. Further work is still necessary to reach a
classification accuracy comparable with the performance of
human experts.
Keywords—classification, composite image, deep learning,
detection, fault, infrared, module, photovoltaic, visible
I. INTRODUCTION
Efficient monitoring and fault detection are required to
maintain high performance and avoid economic loss throughout
the lifetime of a photovoltaic (PV) power plant. This has led to
a need for efficient fault detection methods for PV modules in
large-scale power plants. Aerial inspection by drones is
becoming essential to efficiently cover large areas and inspect
PV modules by carrying multiple imaging devices. Mellit et al.
[1] and Pillai et al. [2] provided a comprehensive review of
various types of PV faults and means for detection. Machine
learning has been adopted recently for automatizing detection
and classification in several applications including detecting
978-1-6654-1922-2/21/$31.00 ©2021 IEEE
faults in PV plants, by using electrical and non-electrical data
(e.g., visible or infrared imaging) [1][3][4][5][6]. In power
plants, operation and maintenance systems typically integrate
monitoring and inspection components to provide timely
updates about the PV plant condition and initiate necessary
actions, such as raising alarms or scheduling maintenance
cycles. The large number of images collected during aerial
inspection of large PV power plants makes the analysis
demanding. While traditional practice relies on manual labor to
analyze thousands of images to correctly identify and classify
faults, machine learning (ML) techniques based on deep
learning (DL) can significantly speed up and automize the
analysis. In this work, composite images are created from visible
(VIS) and infrared thermal (IRT) images, with metadata
extracted from both image sources and global positioning
system (GPS) coordinates collected by aerial drone imaging of
a large-scale PV power plant. Different strategies for image pre-
processing (i.e., creating composite images, scaling, de-
warping) and object labeling for automated fault detection using
supervised ML algorithms are investigated. Our main goal is to
evaluate whether image processing such as multi-wavelength
composite images can improve both the detection and the
classification performance with improved selectivity of faults
compared to using thermal images only.
II. METHOD
We frame the problem as a supervised object/pattern
detection and region proposal and use the “You Only Look
Once” (YOLO) DL architecture [8]. For the work in this paper,
we chose YOLOv3, which offers a good trade-off between
computational complexity and performance in terms of
detection and localization accuracy [9]. This paper focuses on
creating three-layered composite images from infrared and
visible images. The methodology can be divided into two main
parts: i) Cleaning the dataset, creating composite images, and
annotating the images by drawing a bounding box around each
category of fault. ii) Training the object detection model
YOLOv3, and finally testing the model on new images.
Cleaning and preparation of the raw images is here performed
manually by filtering away unusable images from the dataset
and removing fisheye lens distortion from the images. However,
work is ongoing to make these processes automated based on
calibration parameters for data collection including drone
1925
sensors, camera intrinsic parameters, and PV system metadata.
The methodology stages are further described below.
A. Creating composite images
Composite images created by merging optical images with
other types of detectable wavelengths is sometimes called false
color images. This approach has been used for example for
remote sensing to identify and classify agricultural sites,
geological data, or detect shadows [10][11]. The most common
color model used by computers is the additive color model, also
called the RGB model, using the three primary colors red (R),
blue (B) and green (G). To represent all colors, a computer uses
three layers of matrices, each representing the intensity of one
of the three RGB colors as a number between 0 and 255, giving
images a size of MxNx3. If an image contains extra information
as in MxNxY layers, the additional information for Y>3 cannot
be viewed using the RGB format and for visible display on a
computer screen this information will be lost.
The method considered for creating composite images in this
work was to select and shuffle combinations of layers from a
given pair of IRT and VIS images, and to use average grayscale
of one image as a replacement layer in the other image. For
instance, an image described through the RGB format may be
using a fully blue image (B, 0, 0) and shuffling this layer to (0,
0, B) means having a fully red image as represented by the
originally blue matrix with all pixel intensities 0-255. When
creating composite images for ML and DL algorithms, it is
essential to identify the layers holding the most relevant
information for fault detection. One aspect is the typical colors
representing the faults to be detected. For instance, white, gray,
and brown may be representative of bird droppings, sand is
represented by more red and green color than blue, while
vegetation overgrowth is most likely green.
The IRT and VIS images typically have different distortion,
resolution, and field of view (FoV) parameters. However, the
imaging sampling rate can be the same for both image types, but
even if the two cameras are synchronized there will typically be
a short time delay between the two image acquisitions. To create
the composite images a modified template matching (MTM)
method was applied that provided the rotation, scale, location,
and accuracy of the matching. The MTM resulted in satisfactory
composite images through a process that can be semi-automated.
Manual input is needed to make masks on a per-flight basis (for
a given PV plant configuration), but the method could possibly
be fully automated for large datasets using similar images. To
correct for lens distortion effects, the Warp Perspective method
that is part of the OpenCV module for Python was used. This
method relies on four points in space within an image to warp
the image to a new perspective using four new points [7].
Fig. 1 illustrates the process of combining VIS and IRT
images into a composite image. Stretch, scale, and rotation are
adjusted according to MTM, and the IRT image is zero-padded
(seen as black areas in the left image of Fig.1) before moving
the images into the matching location detected through MTM,
with visual validation of the final result. Following this, the two
images are stitched together into a composite image using layer
combinations as described above. Fig. 2 shows an example of a
composite image using the green and blue layers of the VIS
image, combined with one IRT layer as follows: (i) (VIS-blue,
978-1-6654-1922-2/21/$31.00 ©2021 IEEE
VIS-green, IRT-grayscale), (ii) (VIS-blue, VIS-green, IRT-
green), and (iii) (VIS-blue, VIS-green, (IRT-green/2+IRT-
blue/2)). The first combination contains all the information of
the IRT image within the grayscale, which is a channel available
in raw IRT images. The last two combinations are tested for a
clearer visualization of hot areas using the colored IRT images.
The original IRT and VIS images in this case study have
sizes of 640x512 and 4000x3000 pixels, respectively. YOLOv3
can take different width/height ratio of images as input data, but
the more the ratio differs from 1, the worse it detects. Since the
VIS images acquired from the fisheye lens are distorted and
have an aspect ratio of 1.33, which differs from the IRT camera
aspect ratio of 1.25, distortion had to be removed and both VIS
and IRT images had to be rescaled. This was done using a
method that estimated the stretch needed for MTM to function.
The resulting composite images are 512x512 pixels, which
were generated by cropping the sides of the images as shown in
Fig. 1. Next, to generate composite images, masks were created
using a simple hue, saturation, value (HSV) conversion method,
which was improved by applying blurring and sharpening of the
images (as in histogram equalization where contrast is improved
by changing the intensity level of the pixels based on the
intensity distribution of the input image) [12][13]. As shown in
Fig. 2, composite images were matched for time-synchronized
VIS and IRT images (i.e., image pairs captured at the same time
or with a small fixed-time delay between the images).
Fig. 1. Process of combining visible and infrared images: (Right) IRT image
padded to fit MTM detected location; (middle) stretch, rotation and scaling of
VIS image; (left) visual validation of MTM.
Fig. 2. Resulting composite image using two VIS layers (blue and green)
combined with one IRT layer as follows: (Right) Average blue and green IRT
layer; (middle) green IRT layer; (left) grayscale IRT layer.
B. Fault classification
Since the YOLO architecture is trained by optimizing a
supervised loss function, it is necessary to prepare the training
dataset by annotating images, i.e., by drawing a bounding box
around each fault and by assigning to each bounding box a class
label, which specifies the fault type. In Table I, we describe a list
of the fault categories considered in this study and we explain
how they appear in the VIS and IRT channels. The aim is to
detect fault patterns appearing in VIS and IRT images. The fault
categories were chosen with particular attention to separating
1926
faults that require different type of operator actions based on
their cause (such as internal cell defects versus soiling instances)
and severity (i.e., loss of power production or safety hazards).
The ID3 category in Table I is chosen as a combination class for
IRT fault patterns including both single and multiple hot cells
within the module. This class represents faults of different
internal origin (cell damages and mismatch, short-circuit partial
defect, cold soldering causing poor electrical contact to metal
fingers, broken or short-circuited interconnects) that may appear
as similar IRT patterns. Faults in this category do not require
immediate action but will need follow-up or module
replacement over time. Classification of potential induced
degradation (PID) has not been prioritized, since it is mainly
affecting older modules fabricated during a limited period of
years. This fault is not critical for safety but can have severe Fig. 3. LabelImg interface: Example showing two labelled junction box defects
impact on power and would here fall into the ID3 category. PID (ID4) and one cell defect combination class (ID3). Most of the modules have
warm, but not hot, junction boxes and are not labelled as faults.
requires inspection of IRT images from several or all modules
in a string to be correctly classified. Classification of shading
D. Fault detection and classification
from surrounding objects has also not been prioritized in this
case study. but can be detected from both VIS and IRT images Object detection and classification is performed with
(hot spot heating) and has severe impact on power. YOLOv3, a deep learning convolutional neural network (CNN).
Disconnected strings are normally detected by electrical The YOLOv3 algorithm is known for its speed and
monitoring. generalization capability. Due to the many possible variations of
PV plant configurations, this DL model is suitable for detecting
TABLE I. FAULT CLASSIFICATION EXAMPLES USED IN THIS WORK and localizing defects in IRT and composite images. Fault
detection and classification accuracy is tested by comparing
Fault code Fault identification Example
results for composite images versus IRT images alone.
ID1 - Broken modules, IRT: Warm or hot single cells
glass breakage or in a patchwork format. III. RESULTS
VIS: Visibly broken glass.
This case study evaluates aerial images collected in pairs of
ID2 - Partial defect IRT: One or more warmer
visible and infrared images taken at the same time at an altitude
substrings within a module. of 20 m using an XT2 camera with pinhole lens for IRT and
VIS: Not detectable. fisheye lens for the VIS camera. As the PV modules at this site
were recently cleaned, few examples of soiling were available.
ID3 - Cell defect, IRT: Hot cells, hot spots, or Also, no broken modules were present in the current dataset.
combination class patchwork of hot cells within a
(cell damage, short- module. A. Challenges for the DL model from raw image data
circuit partial defect, VIS: Not detectable. Table II describes challenges that may reduce detection
cold soldering,
interconnect defects) accuracy in the object detection model and how some of these
ID4 - Junction box IRT: Localized warm or hot challenges could be mitigated. To reduce errors in the detection,
defect area near edge of module and to improve the detection performance in general, it is useful
where junction box is mounted. to provide more training data. With a sufficiently large number
VIS: Not detectable. of images containing examples of the given challenging
ID5 – Dirt, soiling IRT: Warmer spots or cell area behavior, such as natural temperature gradients or reflection
where dirt is present, typically from surroundings, the DL model can learn how to handle
near edges of module. difficult cases.
VIS: Reflection, color changes.
TABLE II. CHALLENGES FOR OBJECT DETECTION USING MACHNINE
LEARNING AND POSSIBLE MITIGATION STRATEGIES
C. Image annotation Challenge Mitigation techniques
As previously discussed, each image is annotated with a list Reflection Reflection from the surroundings, such as clouds or
of bounding boxes framing each fault. For PV power plants there from nearby objects, and from the sun disc itself, can change
may be multiple regions containing faults in each image, with surroundings the thermal and optical signatures seen in images. This
challenge is mitigated by flight path planning that avoids
one class (fault type) per region. Suitable methods include a such unwanted reflection in images, or by penalizing DL
labelling toolbox from MATLAB or a Python software called models that detect reflection signal.
“LabelImg” [14]. The interface of LabelImg is shown in Fig. 3. High IRT Apparent high temperature readings from surrounding
The example image contains three bounding boxes, two reading from regions may confuse the DL network to search for defects
associated with class ID4 (junction box defect, JBD) and one of surroundings in irrelevant areas. While some DL models can learn to
ignore irrelevant areas, filtering methods and color-
class ID3 (cell defect combination class, CD-CC). thresholding can help to exclude information outside the
module frames.

978-1-6654-1922-2/21/$31.00 ©2021 IEEE 1927

 Changing IRT images may show different temperature scales on an correctly classify different faults that may appear similar in the
temperature image-to-image basis, if not forced constant by the camera IRT images, such as dirt, cell defects, and broken module,
scale settings. Neural networks can learn to focus on shape,
color gradients and texture, rather than the color value without additional information.
itself. For the composite images, a smaller dataset of 283 images
Natural Natural temperature gradients may be observed in IRT has been tested, divided into 159 for training and 124 for
temperature images. Intra-module temperature variations are typically testing. Examples of results are shown in Fig. 5. Here, an
gradients 3-5 K but may reach up to 8 K due to convective heat
transfer, depending on installation [15],[16]. DL models overall mAP of 0.75 was achieved. These results indicate that
can use the raw data from the thermal imaging camera to using the composite image approach may allow us to localize
learn the different temperature representations. faults that are not detected in IRT images alone.
Warm Junction boxes naturally heat up during operation and The detection accuracies were similar when using IRT
junction box appear as warm regions in IRT images. The IRT signature
can vary from none (cold module in morning) to bright
images alone versus the composite images of the type IRT
(warm module in afternoon). It may be difficult to set an grayscale combined with two visible layers for training, and it
exact limit to discriminate normal versus faulty operation. can therefore be assumed that enough essential information
As above, DL models can use the raw data from the from the IRT image is kept within the composite image for
thermal imaging camera to learn the different temperature
representations. The output from the DL detection model
accurate fault detection. Hence, the two visible layers could
can also be combined with electrical data to confirm if the possibly also be used to represent other information from
defect belongs to the “warm” or “hot” category. selected relevant wavelength ranges, which could provide
Partial Partial shading of a PV module or array will cause time- valuable information added to the IRT layer.
shading varying thermal signatures seen in IRT images, as well as
corresponding optical signatures in VIS images. The daily
While the detection accuracy is good, in both cases of using
movement of the shade is a unique feature of this fault. IRT images only or using the composite images, the YOLOv3
Shading can cause severe power reduction and hot spot classification accuracy is unsatisfactory, i.e., the different ID-
development, which, in turn, can accelerate ageing and classes are confused. This could be solved by using more
cause permanent damage. Labels for shade detection can
be provided to the DL model.
complex, yet less computationally efficient DL models, such as
Disturbances Images taken during drone flight may be disturbed by region based convolutional neural network (R-CNN) to
during drone movement, and drones tend to twist during flight due to improve the classification accuracy. Also, importantly, the
flight disturbances in the magnetic field. Poor-quality images current available dataset did not contain a sufficiently large
may be automatically filtered out or manually removed number of images, and also did not contain all the different
both from training and predictions, to improve the
accuracy of fault detection and classification. types of faults to be classified, for robust DL detection and
Imaging The distance of the images in the training and test set classification of all fault types. A new image collection
distance and should match, however, this is not always possible. In campaign has therefore been initiated to provide larger datasets
resolution addition, selecting the optimal flight distance for for training and further testing of the model.
maximum performance of the DL model is not always
possible. Flying high above the modules will include
larger sections of the power plant in each image and
reduce inspection time, but small faults will be difficult to
detect and classify especially for VIS images. Flying very
low will increase flying time and costs, increase
reflections and fisheye distortion, increase risk of
shadowing from the drone, and limit the overview needed
to detect large-scale effects. A trade-off is usually chosen
based on costs and image resolution, where the latter will
impact the DL model performance.

B. Detection results
The prediction output is given in mean average precision
(mAP) for each class. A mAP of 1 for a given class indicates
that the predicted bounding boxes for that class completely
overlap with the true bounding boxes, while a mAP equal to
zero indicates that the predicted bounding boxes for that class
are always wrong.
The results show that YOLOv3 performed well in detecting
a fault but struggled to correctly determine the type of fault
category. The fault detection accuracy was first tested by using
the IRT images alone, see example in Fig. 4. From a total
dataset of 880 annotated IRT images, the set (b)
was divided into 760 for training and 120 for validation and
testing, respectively. An overall mAP fault Fig. 4. (a) Illustration of the deep learning detection process using IRT images
detection/classification accuracy above 0.5 was achieved, only. (b) Example of results from tests of the YOLOv3 algorithm using IRT
which is considered satisfactory. Indeed, it is difficult to images only, displaying the successful detection of junction box JBD (ID4) and
cell defect combination class CD_CC (ID3, label is hidden behind JBD label).

978-1-6654-1922-2/21/$31.00 ©2021 IEEE 1928


(b)
Fig. 5. (a) Illustration of the process for creating composite VIS and IRT
images, which are then forwarded to the deep learning detection network. (b)
Example of detected multiple faults in a single composite image, classified as
dirt (ID5), which should have been hot junction box (ID4) if correctly
classified, and cell defect combination class CD_CC (ID3), correctly classified.
IV. SUMMARY AND MAIN FINDINGS
This research work is aimed at investigating the potential for
enhanced automated identification and classification of faults in
aerial images from PV power plants. In this paper we have
explored the method of creating composite images fusing
thermal infrared and visible color wavelengths for more robust
fault analysis in automated machine learning algorithms. The
integration of images acquired at different wavelengths (i.e.,
thermal infrared, red, green, and blue visible ranges) may
enhance the multi-wavelength representation of faults and thus
their histogram feature signatures. We adopted standard
methods for matching and combining the visual and thermal
infrared images. In particular, we relied on manual adjustment
to match the visual and thermal images to create a composite
image. Ongoing work continues to investigate methods of
automating the image processing based on input parameters
from the drone and camera sensors and the PV power plant
metadata.
Fault detection and classification accuracy have been tested
using the YOLOv3 deep learning model, comparing results
when using composite images versus thermal images alone. The
results are preliminary but show successful detection of fault
patterns and indicate that composite images can improve fault
detection. However, it is also apparent that the YOLOv3
algorithm is not ideal for classification of faults. Applying
further image processing techniques are necessary to enhance
classification accuracy, such as color thresholding,
978-1-6654-1922-2/21/$31.00 ©2021 IEEE
segmentation, and filtering. Using a region-based automated
machine learning algorithm such as R-CNN, could improve
classification of the faults identified by YOLOv3. As the
classification accuracy can be dependent on the representative
fault patterns in the dataset, these aspects will be investigated in
future works using image datasets with a larger representation
of different faults.
ACKNOWLEDGEMENT
This work is supported by the project ALSPIN co-funded by
the research council of Norway and Equinor ASA under the
EnergiX grant number 296216.
REFERENCES
[1] A. Mellit, G. M. Tina, and S. A. Kalogirou, «Fault detection and diagnosis
methods for photovoltaic systems: A review”. Renewable and Sustainable
Energy Reviews, 91, pp. 1-17, 2018.
[2] D. S. Pillai and N. Rajasekar, “A comprehensive review on protection
challenges and fault diagnosis in PV systems”, Renewable and
Sustainable Energy Reviews, 91, pp. 18-40, 2018.
[3] A. Livera, M. Theristis, G. Makrides, and G.E. Georghiou, “Recent
advances in failure diagnosis techniques based on performance data
analysis for grid-connected photovoltaic systems”. Renewable Energy,
133, pp. 126-143, 2019.
[4] R. Pierdicca, M. Paolanti, A. Felicetti, F. Piccinini, and P. Zingaretti,
“Automatic Faults Detection of Photovoltaic Farms: solAIr, a Deep
Learning-Based System for Thermal Images”. Energies, 13, no. 24, p.
6496, Dec. 2020.
[5] X. Li, Q. Yang, Z. Lou, and W. Yan, “Deep Learning Based Module
Defect Analysis for Large-Scale Photovoltaic Farms”. IEEE Transactions
on Energy Conversion, 34, no. 1, pp. 520–529, 2018.
[6] A. Mellit and S. Kalogirou, “Artificial intelligence and internet of things
to improve efficacy of diagnosis and remote sensing of solar photovoltaic
systems: Challenges, recommendations and future directions”.
Renewable and Sustainable Energy Reviews, 143, 2021.
[7] OpenCV [Online], “Geometric Image Transformations”, URL:
https://docs.opencv.org/3.4/da/d54/group__imgproc__transform.html
(accessed 25.01.2021).
[8] J. Redmon, S. Divvala, R. Girshick and A. Farhadi, “You only look once:
Unified, real-time object detection”. In: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 27-30 June
2016, Las Vegas, USA, pp. 779-788 (2016).
[9] J. Redmon and F. Ali, "Yolov3: An incremental improvement", arXiv
preprint arXiv:1804.02767, 2018.
[10] J.C. Mars, “Mineral and Lithologic Mapping Capability of WorldView 3
Data at Mountain Pass, California, Using True- and False-Color
Composite Images, Band Ratios, and Logical Operator Algorithms”.
Economic Geology, 113, no. 7, pp. 1587–1601, Nov. 2018.
[11] M. Teke, E. Baseski, A.Ö. Ok, B. Yüksel, and Ç. Senaras, «Multi-spectral
False Color Shadow Detection”. In: U. Stilla, F. Rottensteiner, H. Mayer,
B. Jutzi, and M. Butenuth (eds.), Photogrammetric Image Analysis,
Springer, Berlin Heidelberg, pp. 109–119, 2011.
[12] V. Peter, M.A. Khan, and H. Luo, “Automatic Orange Fruit Disease
Identification Using Visible Range Images”. In: International Symposium
on Intelligence Computation and Applications, Springer, Singapore, pp.
341-359, Nov. 2019.
[13] H. Ibrahim and N.S.P. Kong, , “Image sharpening using sub-regions
histogram equalization”. IEEE Transactions on Consumer Electronics,
55, no. 2, pp. 891-895, 2009.
[14] Tzutalin [Online], “LabelImg”, 2020 github repository, URL:
https://github.com/tzutalin/labelImg (accessed 01.02.2020).
[15] U. Jahn et al., “Review on Infrared and Electroluminescence Imaging for
PV Field Applications”, Report IEA-PVPS T13-10, Mar. 2018.
[16] B. Herteleer, “Outdoor thermal and electrical characterisation of
photovoltaic modules and systems”, PhD thesis, KU Leuven, Belgium,
Feb. 2016.
1929
978-1-6654-1922-2/21/$31.00 ©2021 IEEE 1930