Fast image recognition of transmission tower based on big data

Abstract

Big data technology is more and more widely used in modern power systems. Efficient collection of big data such as equipment status, maintenance and grid operation in power systems, and data mining are the important research topics for big data application in smart grid. In this paper, the application of big data technology in fast image recognition of transmission towers which are obtained using fixed-wing unmanned aerial vehicle (UAV) by large range tilt photography are researched. A method that using fast region-based convolutional neural networks (Rcnn) convolutional architecture for fast feature embedding (Caffe) to get deep learning of the massive transmission tower image, extract the image characteristics of the tower, train the tower model, and quickly recognize transmission tower image to generate power lines is proposed. The case study shows that this method can be used in tree barrier modeling of transmission lines, which can replace artificial identification of transmission tower, to reduce the time required for tower identification and generating power line, and improve the efficiency of tree barrier modeling by around 14.2%.

1 Introduction

Big data technology has become a hot topic in recent years. It refers to the technical system or technology framework of extracting, discovering and analyzing data from large numbers of different types and sources, and extracting their value using economic methods [1,2,3,4]. The technology contained in big data is not only for collecting massive data but also for deep mining, processing and analyzing data, in order to achieve the maximum value of the data.

Modern power system is one of the important fields in the application of big data technology [5,6,7,8,9,10]. The ultimate goal of smart grid is to build a panoramic real-time system covering the whole production process of the power system, including power generation, transmission, transformation, distribution, consumption and dispatching. Moreover, supporting the safe, self-healing, green, strong and reliable operation of smart grid requires real-time data acquisition, transmission and storage, and the rapid analysis for massive multi-source data [11, 12].

Deep learning can accomplish artificial intelligence tasks that require highly abstract features, such as speech recognition, image recognition, and natural language understanding [13, 14]. The deep layers model is an artificial neural network containing multiple hidden layers, and the multilayer nonlinear structure that makes it has strong expressive power and modeling ability for complex tasks [15]. Training deep layers model is a long-term research problem. In recent years, hierarchical and layer after layer initialization methods have brought hope for training deep layers models, and have been applied successfully in many application fields [16, 17].

The main steps of deep learning include building the network architecture, identifying learning goals, and starting learning [18]. Deep learning for a neural network with many hidden layers is a continuous process. First, a standard parameter is defined, and then it is constantly corrected to get the weight between each node in the graph [19]. Machine deep learning can also be applied to image recognition by first making training sets using massive pictures, and then constantly training to get the weight of the neural network and finally forming the model. After the model is built, the machine can identify whether a picture has a target image when a new picture is entered into the machine.

Image recognition technology is widely used in power system. In the field of power distribution, image recognition technology has been applied to identify poles and wires on the maps of distribution facilities [20, 21]. The safe and stable operation of substations plays a vital role in the stability of power grid, and image processing technology is applied to automatic acquisition of power equipment and achieving real-time observation and analysis of substation equipment [22, 23]. In terms of power transmission, image recognition can be used to measure the icing thickness of transmission lines where the intuitive observation of ice coating on transmission lines can be realized by the image measuring method of ice thickness on remote online systems [24, 25].

By the end of 2016, the length of transmission lines at 220 kV and above in China was about 642 thousand kilometers [26], and continued increasing rapidly. In 2017, the length of newly added transmission lines at 220 kV and above was 41,459 km in China [27]. Tree barrier modeling of transmission line is a cumbersome task as it has to recognize transmission towers to generate power lines to measure the distances between wires and trees. This paper proposed a method of fast image recognition of transmission towers based on big data, which can greatly improve the efficiency of large-scale tree barrier modeling.

The rest of the paper is organized as follows. Section 2 proposes the use of fast region-based convolutional neural networks (Rcnn) convolutional architecture for fast feature embedding (Caffe) to get deep learning of the massive transmission tower images, extract the image characteristics of the tower, train the tower model, and quickly recognize transmission tower images. In Section 3, the process of tree barrier modeling is introduced and the method of fast image recognition of transmission towers is used to replace the traditional artificial identification to improve the efficiency of tree barrier modeling. Through field test case analysis, the feasibility of the proposed method and the efficiency improvement of tree barrier modeling of transmission lines are verified in Section 4. Finally, conclusion is drawn in Section 6.

2 Method of image recognition of transmission tower

2.1 Training tools of image recognition

Caffe is a deep learning framework widely used in video and image processing. The deep learning of image recognition can be achieved by applying Caffe in Windows environment. The comparison of the Caffe deep learning framework and other mainstream frameworks is shown in Table 1.

Table 1 Deep learning framework comparison

As can be seen from Table 1, there are two main differences between Caffe and other deep learning development tools.

• (1) The code of Caffe is written by pure and efficient C ++ language, which has sophisticated modular coding, pure and independent network definition, and can be easily transplanted. Furthermore, Caffe has no hardware and platform limitations, and is suitable for commercial development and scientific research.

• (2) Caffe offers a complete toolkit for training, testing, fine-tuning and developing models. Through fine-tuning and trained models, the new applications can be developed quickly and efficiently.

CPU and GPU are the two operation modes of Caffe, thought faster computing speed can be achieved with GPU mode. More than 40 million images can be identified by K40 or Titan GPU per day, which means that the speed of identification for a picture is only 2.5 ms, and more complex models can be processed quickly. Using fast Rcnn Caffe, the deep learning of numerous tower images of transmission line is trained, and the image features of the transmission tower is extracted. Finally, the model for transmission tower recognition is formed to realize fast image recognition of transmission tower by the machine.

2.2 Image recognition model

To model image recognition of transmission tower, the neuronal network architecture needs to be determined first. The computing process of a neuron in the neuronal network structure is shown as:

$$ z\kern0.5em =\kern0.5em {n}_1{w}_1+\dots +{n}_k{w}_k+\dots +{n}_K{w}_K+b $$

(1)

$$ f(z)=a $$

(2)

where n ₁ ...n _K are parameters, and w ₁ ...w _K are the weights for each node in the numerous pictures of the transmission towers, b is the deviation and f(z) is an incentive function. The computing result of a neuron in the neuronal network is added to the deviation which is then assigned to an incentive function to form the calculation result of the neuron. Neurons are calculated once per layer, and Fig. 1 shows the results of the neurons after multi-level operation.

Fig. 1

Neuronal network architecture. It shows the result after multi-level operation of a neuronal network with hidden layers

Multi-layer computing of neuron is convolution neural network and the dimension is multi-layer convolution. Compared with two-dimensional convolution, the number of channel is increased and each channel is still calculated according to two-dimensional convolution. Multiple channels and multiple convolution kernels perform two-dimensional convolution respectively to obtain multi-channel output, and the formula is expressed as:

$$ {\displaystyle \begin{array}{l}{\mathrm{Y}}^{\mathrm{m}}\left(l,k\right)={\mathrm{X}}^{\mathrm{n}}\left(l,k\right)\star {H}^{nm}\left(l,k\right)\\ {}\kern3.5em =\sum \limits_{n=0}^{N-1}\sum \limits_{i=0}^{I-1}\sum \limits_{j=0}^{J-1}{x}^n\left(l+i,k+j\right){H}^{nm}\left(i,j\right)\end{array}} $$

(3)

It is assumed that the convolutional layer has M output channels and N input channels, as shown in Fig. 1, and MN convolution kernels are needed to achieve the conversion of the channel numbers. As seen in Fig. 1 and (3), X _n is the two-dimensional feature map of the n^th (0 ≤ n < N) input channel, Y _m is the two-dimensional feature map of the m^th(0 ≤ m < M)output channel, H _nm is the two-dimensional convolution kernel in the m^th column and n^th row. Assuming the convolution kernel is I·J and the size of the transmission tower characteristic map of each output channel is L·K, the convolutional computation of the layer forward propagation for each sample is given as:

$$ Calculations(MAC)=I\cdot J\cdot L\cdot K\cdot N\cdot M $$

(4)

2.3 The training process of image recognition

As shown in Fig. 2, the training algorithm of image recognition of transmission towers consists two phases of the forward propagation phase and backward propagation phase.

• Step 1: stage of forward propagation

Fig. 2

Calculation process of forward propagation (left) and backward propagation (right). Forward propagetion calculates from the input unit to the output unit, backward propagation does the opposite and corrects the intermediate gradients

From the input unit to the first hidden layer H1 it is calculated as follows.

For each unit j in hidden layer H1, y _j  = f(z _j ), z _j  = ∑ _i w _ij x _i , where i traverses all input layer nodes and Z _j is the weighted sum of all nodes in the previous layer. The nonlinear function f transforms Z _j in the network to obtain the layer output y _j .

From H1 to H2 the calculations are as follows.

For each unit k in hidden layer H2, y _k  = f(z _k ), z _k  = ∑ _j w _jk y _j , where j traverses all nodes in hidden layer H1.

From H2 to the output layer, it is calculated as follows.

For each unit l in the output layer, y _l  = f(z _l ), z _l  = ∑ _k w _kl y _k , where k traverses all nodes in hidden layer H2.

• Step 2: stage of backward propagation

In the backward propagating process, each layer needs to calculate the error gradient of the output node of this layer, that is, the weighted sum of all error gradients from the input nodes of the latter layer. Then, the error gradient is transmitted to the input nodes of this layer using chain rule, and the error gradient reaches the first layer through the backward propagation path layer by layer. All the weight layers are then updated together after the backward propagation. The error gradient of output units is derived from the loss function. Considering the output layer unit l corresponding to the loss function E, from the output unit to the second hidden layer H2 it is calculated as follows.

For each unit k in hidden layer H2, the error gradient is $\frac{\partial E}{\partial {y}_k}={\sum}_l\frac{\partial E}{\partial {z}_l}\cdot \frac{\partial {z}_l}{\partial {y}_k}={\sum}_l{w}_{kl}\frac{\partial E}{\partial {z}_l}$, where l traverses all output layer nodes.

Similarly, the error gradient of hidden layer H1 can be drawn as $\frac{\partial E}{\partial {y}_j}={\sum}_k\frac{\partial E}{\partial {z}_k}\cdot \frac{\partial {z}_k}{\partial {y}_j}={\sum}_k\frac{\partial E}{\partial {y}_k}\cdot \frac{\partial {y}_k}{\partial {z}_k}{w}_{jk}$, where k traverses all nodes in hidden layer H2.

The error gradient of input layer is $\frac{\partial E}{\partial {x}_i}={\sum}_j\frac{\partial E}{\partial {y}_j}\cdot \frac{\partial {y}_j}{\partial {z}_j}\cdot \frac{\partial {z}_j}{\partial {x}_i}={\sum}_j\frac{\partial E}{\partial {y}_j}\cdot \frac{\partial {y}_j}{\partial {z}_j}{w}_{ij}$, where j traverses all nodes in hidden layer H1.

According to the equations, the backward propagation can be applied repeatedly, and the gradients are obtained from the top level through all the modules to the input level. After all the intermediate gradients have been obtained, the gradient of the loss function relative to the internal weight of each module is calculated.

2.4 The training model and recognition accuracy of transmission tower with fast Rcnn

Intelligent image recognition of transmission tower from transmission line inspection photos needs train machine learning. After selecting transmission line inspection photos for making the data set and subsequent training of the data set, the model will be generated. Through detecting the generated model, changing the model and iteration times and data ratio, and then conducting data training, the optimal model and data ratio are obtained after comparing the test results.

Making data sets needs a large number of transmission line inspection photos containing transmission towers. As can be seen from Table 2, each data set has 1886 transmission line inspection photos. Training machine to extract the feature values of the massive photos in the data sets, a model for identifying transmission tower is formed.

Table 2 The experimental data of fast Rcnn and experimental results

The experimental data of fast Rcnn and the experimental results are shown in Table 2. It can be seen that the model VGG16 with data ratio 6–4 is the best choice for training transmission tower recognition with fast Rcnn, and the recognition accuracy from transmission line inspection photos reaches 86%.

3 Large-scale tree barrier modeling of transmission line

3.1 Process of tree barrier modeling

For modeling tree barrier of power transmission lines, it needs take tilt photography photos of the transmission lines with fixed-wing UAV first, and then identifies the transmission towers from the photos and generates the power lines after obtaining the position of the towers. After generating power lines and point cloud classification, the distance between trees and wires is measured. The process of tree barrier modeling is shown in Fig. 3.

Fig. 3

Process of traditional tree barrier modeling. The process includes five steps, they are taking tilt photography photos of transmission line, artificial identification of transmission tower, generating power lines, point cloud classification, measuring the distance between trees and wires

3.2 Defect classification of tree barrier of high-voltage transmission lines

The horizontal and vertical distances between trees and high-voltage transmission lines affect the safe operation of the transmission lines, and different distances between trees and different voltage levels of the transmission lines lead to different levels of tree barrier defect. According to the grading principle and management and control requirements of tree barrier defect of Guangdong Power Grid, the defect classification of tree barrier for different voltage levels of transmission lines is shown in Table 3, where d _v is the vertical distance and d _h is the horizontal distance.

Table 3 Defect classification of tree barrier [29]

3.3 Improving the efficiency of tree barrier modeling by image recognition of transmission tower based on big data

In 2017, the length of transmission lines at 220 kV and above in China was about 68.35 million kilometers, and the daily tree barrier modeling of transmission lines becomes a heavy workload. Image recognition of transmission tower is a key step in the process of tree barrier modeling, which takes significant time as shown in Table 4. The time of traditional tree barrier modeling is calculated as:

$$ {T}_t={T}_1+{T}_2+{T}_3+{T}_4+{T}_5 $$

(5)

where T₁ is the time of taking tilt photography photos of transmission lines, T₂ is the time of artificial identification of transmission tower, T₃ is the time of generating power line, T₄ is the time of point cloud classification and T₅ is the time of measuring the distance between trees and wires.

Table 4 Time comparison of process of traditional tree barrier modeling and tree barrier modeling proposed in this paper

Using the method of fast image recognition of transmission tower based on big data, the time of tree barrier modeling is calculated as:

$$ {T}_b={T}_1+{T_2}^{\hbox{'}}+{T_3}^{\hbox{'}}+{T}_4+{T}_5 $$

(6)

where T^’₂ is the time of image recognition of transmission tower and T’₃ is the time of generating power line by using the method proposed in this paper.

In Table 4, times of each step of traditional tree barrier modeling of Guangdong Power Grid including tilt photography are illustrated. According to Table 4, T₁ and T₄ are calculated as:

$$ {T}_1=\frac{\left({L}_{ls}\times 2\right)}{28\times 60} $$

(7)

$$ {T}_4=\frac{\left({L}_{ls}\times 35\right)}{400} $$

(8)

where L _ls is the line span. It can be seen that the artificial identification of transmission tower takes more time, making up a larger proportion of the total time of traditional tree modeling. The method proposed in this paper can quickly recognize the transmission tower, so as to significantly reduce the times of image recognition of transmission tower and generating power lines, and improve the efficiency of tree barrier modeling. The increase of efficiency of tree barrier modeling is calculated as:

$$ \eta =\frac{T_t-{T}_b\ }{T_t} $$

(9)

4 Case study

In 2017, the flight mileage of fixed-wing UAV was over 200,000 km for tree barrier modeling with tilt photography in Guangdong Power Grid, and 80,000 km of effective transmission line inspection was completed [28]. From these data, 20 transmission lines are selected randomly for case analysis.

4.1 Taking tilt photography photos of transmission lines

For tree barrier modeling, tilt photography photos of the transmission lines need to be taken using fixed-wing UAV to identify the transmission towers from the photos. Figure 4 is a photograph with transmission towers and Fig. 5 shows a recognized transmission tower. When selecting the tilt photography photos of transmission lines, the number of control points should be at least 5 in a transmission line with even distribution, and the corresponding two photos in two respective routes should be selected as far as possible.

Fig. 4

A photograph with transmission towers. The yellow points represent transmission towers in this tilt photography photo of transmission line with fixed-wing UAV

Fig. 5

A recognized transmission tower. The yellow point represents the transmission tower recognized by the proposed method

4.2 Generating power lines after image recognition of transmission tower

As shown in Fig. 6, after image recognition of the transmission towers, they are identified as the control points, and adding 5 control points evenly can generate the power lines. Using appropriate method of selected points for point cloud classification, the vegetation on the surface and houses of the visible points are classified as “ground”, and all these points are marked. The distances between trees and wires can then be measured, as shown in Fig. 7.

Fig. 6

Adding control points and generating power lines. The purple quadrangles represent control points and the red line represents generated line

Fig. 7

Point cloud classification. The green part represents the vegetation on the surface and houses of the visible points, then the distances between trees and wires can be measured

4.3 Tree barrier modeling of transmission lines

Twenty transmission lines are selected randomly for tree barrier modeling, which contain voltage levels of 110 kV, 220 kV and 500 kV. These twenty transmission lines are generated according to the method proposed in this paper, and the distances between the trees and wires are measured. The measurement results are shown in Table 5.

Table 5 Results of tree barrier modeling

5 Results and discussions

After tree barrier modeling analysis of the 20 randomly selected transmission lines, it can be seen from Table 5 that the numbers of emergency defects, major defects, general defects and concerns are 3, 1, 10 and 6, respectively, as further illustrated in Figs. 8 and 9. The concern cases refer to situations where the trees are close to the transmission lines but not to reach the general defect level.

Fig. 8

Results of tree barrier modeling in case study. The numbers of emergency defects, major defects, general defects and concerns are 3, 1, 10 and 6, respectively

Fig. 9

Times of traditional and proposed tree barrier modeling methods. X axial is the number of the case study and y axial is the time it uses

According to Table 4, the times of traditional tree barrier modeling and tree barrier modeling based on fast image recognition of the twenty transmission lines in the case study are calculated using (5) and (6) respectively, and the results are compared in Table 6. According to (9), the improved efficiency of tree barrier modeling can also be calculated.

Table 6 Comparison of traditional tree barrier modeling and tree barrier modeling based on fast image recognition of transmission towers

It can be seen that the proposed fast image recognition based on big data can reduce the time of image recognition of transmission tower and generating power lines, leading to significantly improved efficiency of tree barrier modeling. In the studied cases, the average improvement of efficiency of tree barrier modeling is around 14.2%.

6 Conclusions

Tree barrier modeling of transmission line is a demanding task, whereas identification of transmission tower is one of the necessary and time-consuming steps. In this paper, the application of big data technology in fast image recognition of transmission towers is researched. The use of fast Rcnn Caffe is proposed for image deep learning to train the transmission tower model for fast identification of transmission tower images. Although the artificial identification method is more accurate in tree barrier modeling than the proposed method, the case study shows that the proposed method of fast image recognition of transmission towers can replace the artificial identification with greatly improved efficiency of tree barrier modeling and can be applied to large-scale tree barrier modeling of transmission lines.