2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC 2019)
Application Research of Artificial Intelligent
Technology in Substation Inspection Tour
Jianxiang Li1, Haipeng Wang1, Yabo Zhao1, Rui Huang2, *Shangwei Yang1
1. Shandong Luneng Intelligence Technology CO., LTD.
2. State Grid Shandong Electric Power Company
Jinan, China
*979893971@qq.com
Abstract—At present, the management and behavior control
of substation field operating persons rely entirely on manual
verification and supervision. There are hidden dangers of
inadequate site control. And the fire, foreign body floating and
other abnormal situations, unexpected defects in the scene cannot
be automatically identified through the substation video
monitoring system. Aiming at the problems existing in substation
inspection tour, this paper applies artificial intelligent technology
to it. Through the research of intelligent identification
monitoring and image intelligent discrimination technology, it
can realize the intelligent control of field personnel and operation
behavior, automatic confirmation and alarm of abnormal
situation such as foreign body, smoke and flame in station area,
realize the automatic layout function of electronic fence, and
automatically identify and warn the situation of intrusion,
crossing fence, mistakenly entering live area, etc. Improve the
quality of site safety control and inspection efficiency.
Keywords—video monitoring system; artificial intelligent
technology; automatically identify
I. INTRODUCTION
Intelligent video analysis technology abroad is mostly used
in the field of security. Most of the algorithms and technologies
are in the hands of companies of United States and Israel.
Many domestic businesses and research institutes are also
introducing foreign advanced technology, while striving to
carry out independent development. However, intelligent video
monitor technology is not widely used especially in domestic
substations.
The concept of deep learning originates from the study of
artificial neural networks. Since Frank proposed the perceptron
model in 1958, the research of neural network is beginning to
rise. However, due to the limitation of computer performance
at that time, multilayer neural network training cannot be
realized. The most influential breakthrough of deep learning in
computer vision field occurred in 2012, when Hinton's team
won the Image Net Classification Competition with deep
learning, this achievement has produced a tremendous shock in
the field of computer vision and triggered an upsurge in deep
learning [1].
Deep learning can also be summed up as a sub-field of
machine learning. Its motivation is to build and simulate a
978-1-5386-8178-7/19/$31.00 ©2019 IEEE 1097
Authorized licensed use limited to: Centro Universitário Fei. Downloaded on January 04,2022 at 20:14:47 UTC from IEEE Xplore. Restrictions apply.
neural network for analysis and learning of human brain, which
imitates the mechanism of human brain to interpret data, such
as images, sounds and text. Traditional pattern recognition
system includes two parts: feature and classifier. The
optimization of feature and classifier is separated. In the
framework of deep neural network, the joint optimization of
feature representation and classifier can bring into full play the
possibility of their cooperation. For target detection technology,
the good features of each image play a key role in the accuracy
of detection results. Deep learning can express image features
as simple linear relations through multi-layer nonlinear
mapping, and integrate them with classification process to
achieve end-to-end learning [1~2].
II. IMPLEMENTATION PLAN
In this paper, the key technologies of intelligent video
surveillance are studied by sorting out the requirements of
video surveillance business and monitoring technology in
substations. The image classification algorithm based on
convolution neural network in deep learning is studied to
realize the target classification in substation monitoring image.
Based on the FasterR-CNN deep learning framework, the
location and recognition of moving objects in video are
realized, and the tracking and behavior analysis of monitoring
objects are realized by combining traditional video tracking
technology and image processing technology.
Through the performance characteristics of flame, smoke
and foreign body in video and image, color, texture and feature
extraction based on time and space, different deep learning
network discriminant models are established to realize
automatic detection and alarm of smoke, flame and foreign
body in station area. Through the research of electronic fence
technology of video analysis, the security fence with signs in
the station is designed, and the electronic fence based on
intelligent video technology in working area and current
carrying areas is automatically deployed for defense or
evacuation [2~4]. By judging the area boundary and target
position, the automatic identification of intrusion, crossing
fence and mistakenly breaking through current carrying areas is
realized. The overall technical route is shown in Fig. 1.
 Demand model
Intrusion Personnel Fire Foreign body environmental
detection control recognition detection monitoring

Artificial intelligent technology

Deep learning Image recognition GPU smart
technology technology chip

Algorithm implementation
Foreign object Personnel
Flame
Target detection behavior
Target Smoke feature Design of
classification based on analysis
detection detection extraction electronic
based on spatio- based on
based on based on algorithm fence based
convolutional temporal confidence-
FastR- drift based on on target
neural feature based deep
CNN analysis frequency recognition
network analysis of learning
domain
video model

Realization effect
Automatic Analysis Automatic
Automatic intrusion Automatic intrusion
of Personnel protection of
alarm alarm
Behavior electronic fence

Fig. 1. Technical roadmap

intelligent identification and tracking technology realizes

III. TECHNICAL ROUTE
A. General Scheme
(1) Application research of deep learning in substation
environment monitoring field
This paper mainly focuses on the design of online real-time
video intelligent analysis technology framework for Substation
Video Surveillance. It mainly includes image acquisition and
intelligent analysis technology, high efficiency and precision
video processing technology, foreign body intrusion and
abnormal state intelligent identification and tracking
technology. Image acquisition and intelligent analysis
technology is mainly to design a high-definition monitoring
device based on GPU image processing chip to realize
intelligent video analysis at the front end, which includes
visible image acquisition, video image coding and decoding,
wired and wireless data transmission, target recognition and
tracking shooting, etc., Which achieves the integration of
image acquisition and intelligent analysis. High efficiency and
precision video processing technology meets the requirements
of intelligent processing and improves image processing effect,
including image pre-processing such as enhancement and de-
noising, key frame extraction, target detection and feature
extraction. Foreign body intrusion and abnormal state
automatic identification of foreign body intrusion, illegal work
of personnel, fire and foreign body legacy in substation
monitoring area. This part mainly includes data conversion,
feature extraction and verification, and the construction of
recognition framework [3~4].
This paper classifies the image attributes of equipment and
other objects in video images, and proposes an image
classification model based on convolutional neural networks
(CNN). Establish discriminant models for different objects, and
return the images categories after network model calculation.
Different models are enriched with the increase of video
database samples. In CNN, convolution layer, domestication
layer, pooling layer and inner layer are used to accurately
acquire the images semantic information. The deep neural
network structure has a very good effect on massive image
classification tasks. After iteration, error back propagation
algorithm is used to update the node parameters in each layer.
Because the traditional convolution neural network algorithm
is only used to classify single label images, the result is only
the label with the highest probability. However, this paper
needs to accomplish the attribute recognition of multi-label
images, so the traditional algorithm cannot accomplish this task.
In this paper, we will change the last layer of Softmax into
Sigmoid layer, when the probability of an attribute is greater

1098

Authorized licensed use limited to: Centro Universitário Fei. Downloaded on January 04,2022 at 20:14:47 UTC from IEEE Xplore. Restrictions apply.
than a set experiment value, the corresponding attribute value
in the predicted attribute table is to be 1, otherwise it is 0. This
problem can be solved with the help of the Sigmoid function
characteristics [4].
(2) Personnel detection and automatic behavior analysis
based on intelligent video analysis algorithm.
Personnel detection and analysis technology based on video
surveillance refers to the use of computer super computing
power to analyze the collected video, extract the moving target
information, and continuously track the target. Based on the
pre-trained abnormal behavior model, when unexpected events
and abnormal behavior are found, alarm signals are
Specifically, an online learning feature classifier is used to
quantify the feature vector extracted from the current checked
area, and the area with the best classification output is regarded
as the target area. This algorithm can update the classifier
continuously by using the feature vector extracted from the
optimal classification region of the current frame. This self-
feedback mechanism greatly improves the adaptability of the
algorithm to the change of the target shape. The process of
moving target tracking based on online learning classification
is shown in Fig. 3.
Local search
strategy
Online learning

automatically generated to remind supervisors to make timely Video

sequence Scanning window image set Feature extraction Online classification mode
Classification
measures. The main technical points involved include the image results

detection of moving objects and the tracking of moving objects.

Moving object detection in video surveillance refers to the Fig. 3 Online moving target tracking flowchart
separation of the parts that do not belong to the static
background from the image, and the extraction of the size, For abnormal behavior detection, the method framework is
location and number of objects of the target area, which is the shown in Fig.4. In the algorithm, image block training method
basis of advanced visual processing such as image recognition based on deep learning is adopted. Firstly, normal behavior
and speech recognition. Generally, the target can be separated training, key area modeling and regional association are carried
from the background by the motion of the target relative to the out. When detecting, video image blocks are extracted from the
scene, so as to realize the detection of the moving target. tested video, and the block trajectory is optimized to delete the
Because the background object is relatively static or moving wrong data, so as to obtain more accurate video image blocks.
slowly, and the foreground object is moving relatively, object Then the extracted video image blocks are trained and
detection is regarded as a classification problem to determine classified by using deep learning network according to the key
whether the pixels belong to the foreground or the background regions characteristics [6~7]. Finally, the abnormal behavior is
[4~7]. judged and tagged according to the information.
At present, ViBe algorithm is a common effective moving Candidate
Trajectory
Optimization of
Abnormal Recognition
Video image video image video image Marking
target detection algorithm. In this algorithm, the background Test video
block blocks optimization
of video
block classification
based on video
result
abnormal
model establishes a sample set at the each pixel position. By extraction
image blocks image block
behavior
Normal model
comparing the new pixel value with the sample set, we can based on video
determine whether the current pixel belongs to the background Training
video
image block
Key area modeling
Normal behavior
or not [5~7]. Let v (x, y) be the pixel value at (x, y) position in training
and regional
association
an image frame, Mt(x,y)={v_1,v_2,⋯ ,v_n } as a background
samples set for time t at (x, y)( n is sample set size), let SR (v Video block training based on deep learning framework
(x, y)) be a region with (x, y) as the center and R as the radius,
Fig. 4 Video local anomaly detection framework
if Mt(x,y)[SR(v(x,y))]>#min （ threshold ） , it can be
determined that the current pixel is the background point. The (3) Research on substation fire and foreign object detection
detection effect of this algorithm is shown in Fig. 2. technology based on deep learning and motion characteristics.
Smoke is produced when an object burns. The color and
movement direction of smoke can be used as initial recognition
features. The basic information of flame includes color, size
change, edge contour, geometric characteristics and inclination.
The irregular flame scintillation will be caused by the change
of airflow, and its spectrum and energy information can be
used as the flame characteristics [7]. In addition, the block
connected area of flame is also its basic characteristic. Fully
exploiting and utilizing these basic characteristics can be used
Fig. 2 Moving target detection effect. as a favorable condition to distinguish the flame area, detect it
well and alarm.
Aiming at the video analysis requirement of substation
scene, this paper adopts classification-based moving target In this paper, a fire monitoring algorithm based on motion
tracking technology. The key of this algorithm is to find a features is designed. Firstly, candidate areas of flame and
boundary between the scene background and the moving object smoke are detected by using deep learning, and the detection
image, and to separate the foreground object from the local range is determined. Then, the fire and smoke are identified by
background. the change different frames, and the accurate location of the
fire is achieved. For detected flame candidate regions, 20

1099

Authorized licensed use limited to: Centro Universitário Fei. Downloaded on January 04,2022 at 20:14:47 UTC from IEEE Xplore. Restrictions apply.
frames can be taken for motion detection and color model
detection. The pixels satisfying the conditions are obtained, and
the R-channel values of 20 consecutive frames are extracted
from the sequential images to form a one-dimensional vector,
which is transformed by a wavelet filter. Two high frequency
sub-bands containing the frequency fluctuation information of
flame pixels are obtained from the transformation [7~9].
Therefore, it can be used as a basis for judging whether the
pixels are flame. Because the smoke density is lower than the
air density, and there is certain directivity in the diffusion, the
movement direction is generally from below, so the analysis of
the smoke fluidity can effectively distinguish the smoke
moving area of from others. For drifting, discrete direction
search can be used to calculate the move direction of
rectangular block, and each direction angle corresponds to a
direction coding. By searching two consecutive frames in
discrete motion direction, the motion direction of rectangular
block is determined by absolute error and minimum rule, and
the candidate area is judged as smoke area according to the
motion direction [8~10].
Fig. 5 Effect chart of automatic flame and smoke detection
(4) Automatic layout and abnormal behavior analysis of
electronic fence based on identification image recognition.
In this paper, an electronic fence with color identification is
designed to realize the automatic layout of different attribute
areas in substation. As one of the most widely used visual
features in image recognition, color information has less
dependence on image size, direction and perspective, and has
high separability and robustness. For the division of different
areas such as safety area, working area, live area and prohibited
area in substation in substation, it is necessary to select
appropriate color information to describe different attributes. In
this paper, a method combining color space and centroid
detection of target area is proposed to realize the automatic
generation of fence area [9~11].
Firstly, the fence logo is designed with easy-to-recognize
color such as red, yellow, blue, and green and so on. When the
surveillance camera captures the logo, it recognizes the logo
based on the color information and extracts the logo area. The
region centroid is calculated as the electronic fence "setting
point" in the video image. Different closed or semi-closed
regions are automatically generated by combining different
"setting point" with the region color information. The working
area is arranged by the field staff, and the automatic
1100
Authorized licensed use limited to: Centro Universitário Fei. Downloaded on January 04,2022 at 20:14:47 UTC from IEEE Xplore. Restrictions apply.
deployment and evacuation can be realized. Combining the
previous staff detection and target tracking algorithms,
detecting the target location and the temporal and spatial
characteristics of the fence boundary area, different criteria are
defined for intrusion, crossing the fence, and mistakenly
entering the live area. Once the pre-set warning information is
issued, the alarm information is output and the abnormal
behavior is warned by the alarm device [10~12].
IV. CONCLUSIONS
Through the application research of substation video
surveillance and image intelligent analysis technology based on
artificial intelligence, intelligent management and control of
field personnel and operation behavior are realized. Realize
automatic alarm of substation security and fire protection;
improve the automation and intelligence level of substation
equipment status, environmental monitoring and personnel
control. The quality of patrol inspection is greatly improved,
and the substation inspection efficiency is improved.
ACKNOWLEDGMENT
The research of this paper is supported by STATE GRID
Corporation of China science and technology project: the
research and application of key technologies of automatic
inspection. Project number: 520600180003.
REFERENCES
[1] HUANG Siqian. Research and Application of Intelligent Video
Analysis Algorithms in Substations [D]. Sichuan: Chengdu, Southwest
Jiaotong University, 2012.
[2] Tang, M.W., Dai, L.H., Lin, C.H., Wang, F.D., and Song, F.G.,
"Application of unmanned aerial vehicle in inspecting transmission
lines," Electric Power, 46(3), 35-38(2013).
[3] Von Gioi, R.G., Jakubowicz,J., Morel, J.M., and Randall,G., "LSD: A
fast line segment detector with a false detection control," IEEE
Transaction on Pattern Analysis and Machine Intelligence, 32(4), 722-
732(2010).
[4] LI Dan. Design and Implementation of Substations Based on intelligent
Video Analysis [D]. Henan: Zhengzhou, Zhengzhou University, 2016.
[5] Akinlar, C. and Topal, C., "EDlines: real-time line segment detection by
edge drawing," Proc of IEEE International Conference on Image
Processing, 2837-2840(2011).
[6] Girshick, R., Donahue, J., Darrell, T., et al. "Region-Based
Convolutional Networks for Accurate Object Detection and
Segmentation, " IEEE Transaction on Pattern Analysis & Machine
Intelligence, 38(1), 1-1(2015).
[7] Lu Yanbin. Study on Moving object Detection and Tracking in Video
Surveillance [D]. Shanghai, Shanghai Jiao Tong University, 2007.
[6] Fu, L., Lu, S.T., "Obstacle detection algorithms for aviation," IEEE
International Conference on Computer Science & Automation
Engineering, 710-714(2011).
[9] Girshick, R., "Fast R-CNN," 2015 IEEE International Conference on
Computer Vision. 1440-1448(2015).
[10] Ren, S.Q., He, K.M., Girshick, R., Sun, J., "Faster R-CNN: Towards
Real-time Object Detection with Region Proposal Network, 1-14(2015).
[11] Zhang, X., Gao, J., Wang, W.G., Liu, L., Zhang, J.J., "Image mosaic
approach of transmission tower based on saliency map," Journal of
Computer Applications, 35(4), 1133-1136(2015).
[12] Li Li, Wang, B. H., Peng, X., Wang, W. G., Li, Y.B., "The application
of image based visual servo control system for smart guard, " IEEE
International Conference on Control and Automation, 1342-1345(2013).