2023 - Pipeline Inspection and Health Monitoring Technology Hongfang

Hongfang Lu · Zhao-Dong Xu ·
Tom Iseley · Haoyan Peng · Lingdi Fu
Pipeline
Inspection
and Health
Monitoring
Technology
The Key to Integrity Management
Pipeline Inspection and Health Monitoring
Technology
Hongfang Lu · Zhao-Dong Xu · Tom Iseley ·
Haoyan Peng · Lingdi Fu
Pipeline Inspection
and Health Monitoring
Technology
The Key to Integrity Management
Hongfang Lu Zhao-Dong Xu
China-Pakistan Belt and Road Joint China-Pakistan Belt and Road Joint
Laboratory on Smart Disaster Prevention Laboratory on Smart Disaster Prevention
of Major Infrastructures of Major Infrastructures
School of Civil Engineering School of Civil Engineering
Southeast University Southeast University
Nanjing, China Nanjing, China
Tom Iseley Haoyan Peng

Construction and Engineering Management China-Pakistan Belt and Road Joint
Purdue University Laboratory on Smart Disaster Prevention
West Lafayette, IN, USA of Major Infrastructures
School of Civil Engineering
Lingdi Fu Southeast University
Safety, Environment and Technology Nanjing, China
Supervision Research Institute
PetroChina Southwest Oil and Gasfield
Company
Chengdu, China
ISBN 978-981-19-6797-9 ISBN 978-981-19-6798-6 (eBook)

https://doi.org/10.1007/978-981-19-6798-6
© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature
Singapore Pte Ltd. 2023
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse
of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
The pipeline is a crucial lifeline project that can transport oil, gas, water, and other
resources. It is often called the blood vessel of the energy system. Thus, pipeline
engineering is essential in ensuring national energy security, promoting economic
development, and ensuring social stability.
After many years of service, the pipeline gradually enters the aging stage. This
will inevitably lead to accidents and serious economic losses. In addition, accidents
in the city may lead to casualties, traffic paralysis, and other consequences. Using
advanced technology to understand the health status of pipelines is a very challenging
task in pipeline engineering.
In recent decades, many inspection and monitoring technologies have emerged
worldwide to assess pipelines’ conditions accurately. These technologies involve
acoustics, optics, and electromagnetism. Not only that, in the context of big data and
artificial intelligence, people use advanced computer and information techniques to
cooperate with inspection and monitoring to solve the problems encountered.
This book deals with interdisciplinary knowledge. It can let readers know about
the existing pipeline inspection and monitoring methods. This book is written for
managers, technicians, and researchers engaged in pipeline safety and can also
provide a reference for some graduate students engaged in relevant research. In
this book, a reader who wants to understand the background and health status of
global pipelines should read Chap. 1. In Chap. 2, pipeline inspection techniques
are described in detail, including visual, electromagnetic, acoustic, optical, and
chemical inspections, and the applicability of each method is indicated. Chapter 3
presents distributed fiber-optic and signal-based monitoring techniques for pipelines.
In Chaps. 4 and 5, some artificial intelligence-based methods and data processing
methods are presented. These contents mainly provide a reference for the post-
processing part of inspection or monitoring. In addition, we set up Chapter 6. By
providing several practical engineering cases, readers can have a more in-depth
understanding of the application of related technologies.
The first author, Dr. Hongfang Lu, would like to express his sincere gratitude to
his research advisor, Dr. Tom Iseley, for his support over the years. He would like
to thank his family members for their love and support. Moreover, he also would
v
vi Preface
like to thank his friends for their friendship and support. The second author, Dr.
Zhao-Dong Xu, would like to thank the Major Project of Fundamental Research
on Frontier Leading Technology of Jiangsu Province (Grant No. BK20222006) and
the National Key Research and Development Program of China (2019YFE0121900)
for their funding. The third author, Dr. Tom Iseley, would like to thank the Buried
Asset Management Institute-International (BAMI-I) for their support during the work
on this book. The fourth author, Mr. Haoyan Peng, would like to thank his family
members for their support. The fifth author, Ms. Lingdi Fu, would like to thank her
family members for their support.
The authors of this book would like to thank the editors for their effective
cooperation and great care in making possible the publication of this book.
Nanjing, China Hongfang Lu

Nanjing, China Zhao-Dong Xu
West Lafayette, USA Tom Iseley
Nanjing, China Haoyan Peng
Chengdu, China Lingdi Fu
Acknowledgements
This book was financially supported by the Major Project of Fundamental Research
on Frontier Leading Technology of Jiangsu Province (Grant No. BK20222006);
Natural Science Foundation of Jiangsu Province (Grant No. BK20220848), the
National Key Research and Development Program of China (2019YFE0121900); the
2022 Open Project of Failure Mechanics and Engineering Disaster Prevention, Key
Lab of Sichuan Province (Grant No. FMEDP202212); and the Zhejiang Engineering
Research Center of Intelligent Urban Infrastructure (Grant No. IUI2022-YB-02). We
also would like to thank the following people for their help:
Chapter 1:
John C. Matthews, Trenchless Technology Center, Louisiana Tech University,
Ruston, LA 71270, United States
Saleh Behbahani, Construction Engineering and Management, Purdue University,
West Lafayette, IN 47907, United States
Mohammadamin Azimi, GeoEngineers, Inc., San Diego, CA 92128, United States
Chapter 2:
Chapter 3:
Dongmin Xi, China-Pakistan Belt and Road Joint Laboratory on Smart Disaster
Prevention of Major Infrastructures, School of Civil Engineering, Southeast Univer-
sity, Nanjing, China
Chapter 4:
Chapter 6:
Houming Ni, Asoe Hose Manufacturing Inc., Taizhou 225319, China
Jing Li, Safety, Environment and Technology Supervision Research Institute of
PetroChina Southwest Oil and Gasfield Company, Chengdu, Sichuan, 610041, China
vii
viii Acknowledgements
Dong Lin, Safety, Environment and Technology Supervision Research Institute of

Jian Gao, Safety, Environment and Technology Supervision Research Institute of
Chaolang Li, Safety, Environment and Technology Supervision Research Institute
of PetroChina Southwest Oil and Gasfield Company, Chengdu, Sichuan, 610041,
China
Kebing Shi, Asoe Hose Manufacturing Inc., Taizhou 225319, China
Contents
1 Background and Health Problems of Pipelines . . . . . . . . . . . . . . . . . . . . 1

1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Pipeline Classification and Construction Status . . . . . . . . . . . . . . . . . 1
1.3 Pipeline Health Status Globally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Pipeline Inspection Technology System . . . . . . . . . . . . . . . . . . . . . . . . 10
1.5 Technical System of Pipeline Health Monitoring . . . . . . . . . . . . . . . . 10
1.6 Global Pipeline Inspection and Monitoring Standards
and Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2 Pipeline Inspection Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2 Visual Inspection Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3 Electromagnetic Inspection Technology . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.1 Magnetic Flux Leakage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.2 Remote Field Eddy Current . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.3.3 Broadband Electromagnetic . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.3.4 Pulsed Eddy Current System . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.3.5 Ground Penetrating Radar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.4 Acoustic Inspection Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4.1 Acoustic Emission Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4.2 Ultrasonic Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.4.3 Ultrasonic Guided Wave Method . . . . . . . . . . . . . . . . . . . . . . . 41
2.4.4 Echo Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.4.5 SmartBall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.4.6 Sonar System Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.4.7 Leakfinder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.4.8 Sahara . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.5 Optical Inspection Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.5.1 Lidar System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.5.2 Diode Laser Absorption Method . . . . . . . . . . . . . . . . . . . . . . . 56
ix
x Contents
2.5.3 Thermal Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

2.5.4 Spectral Imaging Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.6 Chemical Composition Analysis-Based Method . . . . . . . . . . . . . . . . . 60
2.6.1 Sniffer Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.6.2 Vapor Sampling Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.7 Technology Selection Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . 61
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3 Pipeline Health Monitoring Technology . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.2 Optical Fiber Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.2.1 Optical Time Domain Reflection (OTDR) . . . . . . . . . . . . . . . 83
3.2.2 Fiber Bragg Grating (FBG) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.2.3 Interferometric Optical Fiber Sensor . . . . . . . . . . . . . . . . . . . . 89
3.3 Signal-Based Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.3.1 Volume/Mass Balance Method . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.3.2 Negative Pressure Wave Method . . . . . . . . . . . . . . . . . . . . . . . 94
3.3.3 GPS Time Label Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
3.3.4 Pressure Point Analysis Method . . . . . . . . . . . . . . . . . . . . . . . . 96
3.3.5 Cross Correlation Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
3.3.6 Transient Test-Based Technique . . . . . . . . . . . . . . . . . . . . . . . . 98
3.3.7 State Estimation Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
3.4 Technology Selection Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . 106
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4 Health Monitoring Technology Based on Artificial Intelligence . . . . . 117
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.2 Classic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.2.1 Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.2.2 Naive Bayes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
4.2.3 Artificial Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
4.2.4 Kernel-Based Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
4.2.5 Decision Tree Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
4.2.6 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
4.3 Optimizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
4.3.1 Fruit Fly Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
4.3.2 Grey Wolf Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
4.3.3 Whale Optimization Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 133
4.3.4 Nondominated Sorting Genetic Algorithm II . . . . . . . . . . . . . 134
4.3.5 Multi-objective Grey Wolf Optimizer . . . . . . . . . . . . . . . . . . . 136
4.3.6 Multi-objective Salp Swarm Algorithm . . . . . . . . . . . . . . . . . . 136
4.4 Application Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
4.4.1 Fault Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
4.4.2 Risk Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.4.3 Condition-Related Parameter Prediction . . . . . . . . . . . . . . . . . 140
Contents xi
4.4.4 Visual Defect Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

4.5 Application Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
4.5.1 Model Category . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
4.5.2 Model Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
4.5.3 Data Size and Data Division . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
4.5.4 Input Variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
4.5.5 Error (Accuracy) Indicator . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
4.5.6 Real-World Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
4.6 Specific Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
4.6.1 Burst Pressure Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
4.6.2 Pullback Force Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
5 Data Preprocessing Technology in Pipeline Health Monitoring . . . . . . 213
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
5.2 Advantages of Big Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
5.3 Data Correlation Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
5.3.1 Chi-Square Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
5.3.2 Information Gain and Information Gain Ratio . . . . . . . . . . . . 219
5.3.3 Covariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
5.3.4 Correlation Coefficient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
5.4 Data Dimensionality Reduction Method . . . . . . . . . . . . . . . . . . . . . . . 222
5.4.1 Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 222
5.4.2 Linear Discriminant Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 222
5.4.3 Locally Linear Embedding . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
5.4.4 Laplacian Eigenmaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
5.4.5 High Correlation Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
5.4.6 Factor Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
5.4.7 Independent Component Analysis . . . . . . . . . . . . . . . . . . . . . . 230
5.5 Data Noise Reduction Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
5.5.1 Wavelet Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
5.5.2 Empirical Mode Decomposition . . . . . . . . . . . . . . . . . . . . . . . . 233
5.5.3 Variational Mode Decomposition . . . . . . . . . . . . . . . . . . . . . . . 237
5.5.4 Singular Spectrum Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
5.6 Data Exception Elimination and Missing Supplement Methods . . . . 242
5.6.1 K-nearest Neighbor Substitution Method . . . . . . . . . . . . . . . . 242
5.6.2 Regression Filling Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
5.7 Multi-source Heterogeneous Data Fusion Method . . . . . . . . . . . . . . . 245
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
6 Application and Cases of Pipeline Inspection and Monitoring . . . . . . . 257
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
6.2 Closed-Circuit Television (CCTV) Inspection . . . . . . . . . . . . . . . . . . 257
6.2.1 Considerations Before an Inspection . . . . . . . . . . . . . . . . . . . . 257
6.2.2 Project Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
6.2.3 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
xii Contents
6.3 Magnetic Flux Leakage Inspection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260

6.3.2 Inspector Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
6.3.3 Inspection Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
6.4 Remote Field Eddy Current . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
6.4.2 Inspector Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
6.4.3 Inspection Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
6.4.4 Inspection Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
6.5 Pipeline Monitoring in the Landslide Section . . . . . . . . . . . . . . . . . . . 282
6.5.2 Monitoring Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
6.5.3 Monitoring Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
Chapter 1
Background and Health Problems
of Pipelines
1.1 Introduction
Pipelines are essential lifeline projects and blood vessels for energy delivery. In the
past, people paid attention to the construction of pipelines [1]. However, with the
increase in pipeline accidents in recent years, an increasing number of managers
have focused on the safety of pipelines [2]. Therefore, pipeline inspection and moni-
toring technology has received extensive attention [3]. This chapter introduces the
construction and health status of the pipeline, leading to the current inspection and
monitoring technology system.
1.2 Pipeline Classification and Construction Status
The pipeline is equipped with power devices (pumps, compressors), valves, and
other accessories for conveying liquids and gases [4]. Its materials can be metallic
and non-metallic, such as steel, concrete, and polyethylene [5]. Pipeline diameters
can be as small as 5 cm and as large as 9 m, depending on the transport scale [6].
According to the transmission requirements, the pipeline can be pressure-free or as
high as more than 10 MPa (such as oil and gas pipelines) [7].
Oil, gas, and water pipelines are the most common types of pipelines. Most
pipelines are buried underground (some are laid overhead, see Fig. 1.1), so their
essential contribution to the economy is often ignored. In fact, almost all the water
transported from the treatment plant to the individual, the natural gas transported
from the city gate station to the household, or the oil transported from the sea to the
land refinery are transported through pipelines [8].
The pipeline is not the only way to transport oil, gas, and water. Especially for oil
and gas resources, it is common to transport them by road, railway, and waterway
[9]. Their advantages and disadvantages are shown in Fig. 1.2. Compared with trucks
and railways, pipelines have always been the preferred transportation mode for liquid
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 1
H. Lu et al., Pipeline Inspection and Health Monitoring Technology,
https://doi.org/10.1007/978-981-19-6798-6_1
2 1 Background and Health Problems of Pipelines
Fig. 1.1 Conventional pipe

laying methods a buried
method (the picture shows
the excavation of buried
pipelines); b overhead laying
method
(a)
(b)
and gas due to its economic, safe, and reliable characteristics. Therefore, the use of
pipelines is pervasive worldwide. As shown in Fig. 1.3, as of 2017, a total of 120
countries worldwide have built pipelines with a mileage of about 3.5 million km.
Among them, the United States has 2,225,032 km, accounting for 64% of the total
mileage in the world. Russia ranks second, with nearly 260,000 km of pipelines.
China ranks fourth in pipeline mileage, about 87,000 km, and ranks first in Asia.
As of 2017, there are approximately 3800 transmission (long-distance) oil and gas
pipelines worldwide, with a total length of approximately 1.95 million km [9, 11].
Global oil and gas pipelines are mainly distributed in Asia Pacific, Russia, Central
1.2 Pipeline Classification and Construction Status 3
Advantage Disadvantage
The transportation volume is large and The one-time investment of pipeline

the transportation cost is low. facilities is large.
Pipelines
Not affected by weather and traffic. The requirements for gas source and
destination are high, and they need to be
No special railway facilities need to be close to the railway.
built.
Railway
Transportation costs are high.

Not limited by source and destination.
Vulnerable to weather and traffic
There is no need to invest in the conditions.
construction of transportation facilities.
Road Fuel and labor costs are high.
Good economy. Transportation The temperature and pressure control

technology is mature. requirements of the transport equipment
are high.
Waterway
Fig. 1.2 Advantages and disadvantages of the four modes of transportation [10]
Asia, Europe, North America, Latin America, the Middle East, and Africa. As shown
in Fig. 1.4, the total length of oil and gas pipelines in North America accounts for
about 43% of the world [1].
Due to global oil and gas prices and economic recession, investment in oil and
gas pipeline construction has declined since 2016, from 166 billion dollars in 2016
to 106 billion dollars in 2018 [11]. The new pipeline is mainly concentrated in gas
and submarine pipelines, with the largest investment in North America and the Asia
Pacific, followed by the Middle East and Latin America. For example, in the United
States (as shown in Fig. 1.5), most of the gas pipeline construction was concentrated in
the 1950s and 1960s, while oil pipeline construction was concentrated in the 1940s–
1960s [12]. Table 1.1 lists large-scale oil and gas long-distance pipeline projects in
recent years.
As shown in Fig. 1.6, from oil (gas) fields to users, oil and gas pipelines can be
divided into gathering pipelines, transmission pipelines, and distribution pipelines
2225032
2.5×106
2.0×106
Mileage (km)
1.5×106
1.0×106
259913
5.0×105
100000
86921
45597
39850
39778
38906
37008
35676
34612
34335
29642
27468
26963
23345
23190
21704
20877
16349
0.0
Brazil
United Kingdom
China
Russia
Canada
Iran
Mexico
India
Italy
Ukraine
Germany
Algeria
France
Poland
Hungary
United States
Argentina
Australia
Kazakhstan
Indonesia
Fig. 1.3 20 countries with the longest pipeline mileage in the world. Data source https://www.wor
ldatlas.com/articles/top-20-countries-by-length-of-pipeline.html
Fig. 1.4 Global transmission oil and gas pipeline length

1.3 Pipeline Health Status Globally 5
Fig. 1.5 Oil and gas pipelines built in different years in the United States
Table 1.1 Large oil and gas long-distance pipeline projects in recent years
Project Medium Length (km) Diameter (mm) Transport capacity
Central Asia–China gas Gas 1833 1067 55 billion m3 /a
pipeline
Nord Stream Gas 1222 1220 55 billion m3 /a
Polarled gas pipeline Gas 482 914 70 million m3 /d
TurkStream Gas 930 813 31.5 billion m3 /a
Sino-Myanmar gas Gas 793 1016 12 billion m3 /a
pipeline
Sino-Myanmar crude oil Crude oil 771 813 12 million t/a
pipeline
(distribution pipelines are only applicable to the gas system) [13]. Their functions
and features are shown in Table 1.2.
1.3 Pipeline Health Status Globally
During the long-run process, oil and gas pipelines will fail due to corrosion, weld
defects, third-party damage, and other reasons [14]. Figures 1.7 and 1.8 show two
conventional corrosion defects. Table 1.3 lists the major oil and gas pipeline accidents
Oil (or gas) Gathering pipelines Processing plant

wells (oil or gas) (oil or gas)
Transmission
Oil refinery pipelines (oil or gas)
Users
Gas distribution
station
Fig. 1.6 Oil and gas systems
in the world in recent years. Many scholars have concluded that the failure modes of
the gas pipeline are cracking and perforation, and the primary failure form of the oil
pipeline is perforation. Tables 1.4 and 1.5 show the trend of failure rates of oil and
gas pipelines based on the statistics of different countries or institutions [15–23].
In the United States, some pipeline accident statistics can be found in Pipeline
and Hazardous Materials Safety Administration (PHMSA), as shown in Figs. 1.9
and 1.10. In general, the incident rate of gas pipelines showed a downward trend,
and the peak period of accidents was from 2004 to 2009.
In the field of reliability, from the time of commissioning to scrapping, the failure
probability usually follows the bathtub curve (Fig. 1.11). For oil and gas pipelines,
the failure probability curve is divided into three stages: (1) Initial stage of pipeline
production: due to design, construction, welding, and pipe material problems. The
probability of accidents is high, the process usually lasts 0.5–2 years, and the number
Table 1.2 Characteristics of different types of pipes

Pipe type Gathering Transmission Distribution
Function Transport fluid from the Transport fluid over long Deliver gas to the user
wells to the processing distances across states,
plant or storage tank countries, and continents
Diameter (mm) Under 450 for gas, Usually 500–1200 Under 900 for main
50–200 for crude oil pipelines, less than 50
for service pipelines
Length (m) Approximately 200 Up to thousands –
Medium Natural gas, crude oil, Natural gas, crude oil, Natural gas
natural gas liquids natural gas liquids, and
refined products
Pressure (MPa) Under 5 for gas 1.5–8.5 Up to 1.5 for main
pipelines, around 0.05
for service pipelines
Material Steel Steel Steel, cast iron, plastic,
and copper
Fig. 1.7 The old pipeline

with external corrosion
defects
Fig. 1.8 The old pipeline

with corrosion perforation
of pipeline failures per 1000 km is about 5 times; (2) Stable operation stage of
pipelines: mainly due to corrosion and third-party damage. The process usually lasts
15–20 years, and the number of pipeline failures per 1000 km is about 2 times; (3)
Aging stage of pipelines: due to increased corrosion and wear. The probability of
failure increases significantly, and the annual number of pipeline failures per 1000 km
is usually greater than 2 times, so repairs at this stage are complicated [33].
From the perspective of accident cause, the European Gas pipeline Incident data
Group’s statistics [35] show that 25% of natural gas pipelines’ failure is caused by
corrosion, second only to external interference, as shown in Fig. 1.12. According
to PHMSA (see Fig. 1.13), excavation damage is the leading cause of gas pipeline
failure, followed by equipment failure. For oil pipelines, equipment failure is the
primary cause of failure, followed by corrosion.
Oil and gas pipeline leakage is one of the common types of accidents and is
also one of the leading causes of heavy losses. The leakage accident affects the
regular pipeline operation and threatens the environment and personal safety. In many
countries, the aging of pipelines is serious, and the condition of pipeline facilities
is not optimistic. For example, according to the 2021 U.S. wastewater infrastructure
condition assessed by the American Society of Civil Engineers (ASCE) (Fig. 1.14),
although the data for some states are not available, it is not difficult to see that the
Table 1.3 Major accidents in oil and gas pipelines in recent years
Time Location Pipeline name Loss Cause
November Qingdao, Donghuang oil 62 people were The oil pipeline
2013 Shandong, China pipeline [24] killed and 136 were leaked (because of
injured. The direct corrosion) and
economic loss is exploded during
750 million Chinese the rush repairs
Yuan (124.9 million
U.S. dollars)
November Near Amherst, Keystone crude 9700 barrels of oil Construction
2017 South Dakota, oil pipeline [25] leaked damage
USA
September Kenya Nairobi pipeline About 100 people Pipe leakage
2011 [26] were killed in the
fire, and at least 116
people were
hospitalized for
different degrees of
burns
May 2014 Fort McMurray, Northwestern Estimated A bend is fractured
Alberta, Canada Minnesota gas 16.5 million m3 of
pipeline [27] gas were released
July 2014 Kaohsiung, Urban gas Several big The pressure of the
Taiwan, China pipeline [28] explosions, 32 propylene pipe is
people were killed abnormal
and 321 were injured
condition of wastewater infrastructure in most states (94.28% of the 35 states with

a score) is C (mediocre, requires attention) or D (poor, at risk). The overall rating of
wastewater facilities in the United States is D+. Looking back at the rating of C in
1988, it shows that the condition of the infrastructure has declined dramatically in
about 30 years.
Inspection and monitoring of oil and gas pipelines are conducive to reducing
carbon emissions and environmental pollution [36], and is also the key to pipeline
integrity management [37, 38]. Currently, the inspection techniques for pipeline leaks
are mainly for water pipelines, and most of them are based on the principle of acoustic
inspection, such as SmartBall and Sahara [39]. Relative to the water pipelines, the
particularity of oil and gas pipelines is mainly reflected in the following aspects [40]:
(1) the pressure of oil and gas pipelines is much higher than that of water pipelines;
(2) the medium in oil and gas pipelines has high risk.
Table 1.4 The trend of failure rates of oil pipelines based on statistical data
Country/Region Institution Statistical Cause of failure Failure rate Reference
time range (top three) trend
China PetroChina 2006–2015 Third-party Descend [29]
Natural Gas damage,
& Pipeline manufacturing
Company defects,
construction
quality
Canada AER 1990–2012 Internal Descend [30]
corrosion,
third-party
damage,
external
corrosion
United Kingdom UKOPA 1962–2016 External Descend [32]
corrosion,
external
interference,
weld defects
United States PHMSA 2004–2020 Corrosion, weld Descend [20]
failure,
third-party
damage
AER Alberta Energy Regulator; UKOPA United Kingdom Onshore Pipeline Operators’ Association;
PHMSA Pipeline and Hazardous Materials Safety Administration
1.4 Pipeline Inspection Technology System
Different scholars have different classifications of inspection technologies.

According to the degree of automation, inspection techniques can be divided into
automatic, semi-automatic, and manual inspection [41]. According to the intuitive
degree of inspection data, inspection technology can be divided into direct and
indirect inspection [42, 43]. Moreover, some scholars have divided the inspection
methods into optical and non-optical methods [44, 45]. The most common classifi-
cation method is based on inspection technology characteristics and can be divided
into hardware-based and software-based methods [46, 47]. Table 1.6 lists the system
of pipeline inspection technology.
1.5 Technical System of Pipeline Health Monitoring
Unlike pipeline inspection, monitoring is a long-term process. It is an automatic

system for condition monitoring, feature recognition, and condition evaluation to
meet the needs of long-term service safety of pipelines, and to provide decision
1.6 Global Pipeline Inspection and Monitoring Standards and Specifications 11
Table 1.5 The trend of failure rates of gas pipelines based on statistical data
Country/Region Institution Statistical Cause of failure Failure rate Reference
time range (top three) trend
China PetroChina 2006–2015 Third party Descend [29]
Natural Gas & damage,
Pipeline manufacturing
Company defects,
construction
quality
Canada AER 1990–2012 Internal Descend [30]
corrosion, third
party damage,
external
corrosion
Europe EGIG 1970–2013 External Descend [31]
interference,
corrosion,
construction
defects/material
failure
United Kingdom UKOPA 1962–2016 External Descend [32]
corrosion,
external
interference,
weld defects
United States PHMSA 2004–2020 Corrosion, weld Descend [20]
failure, third
party damage
EGIG European Gas Pipeline Incident Data Group
support for the management and maintenance of pipelines. Therefore, pipeline moni-
toring often requires sensing devices to collect data and employ models to build a
system that provides decision-making. Pipeline health monitoring technologies can
be distinguished by principle, as shown in Table 1.7.
1.6 Global Pipeline Inspection and Monitoring Standards

and Specifications
In addition to developing techniques for detecting leakages in oil and gas pipelines,
it is vital to establish relevant standards or specifications. According to the literature
survey, the current common standards are shown in Table 1.8, totaling 20 stan-
dards. These standards come from the National Development and Reform Commis-
sion (China), American Petroleum Institute (API), American Society of Mechan-
ical Engineers (ASME), International Organization for Standardization (ISO), Det
300 16
Mileage
Incident rate
Incident rate (times·10 -5·mile -1·yr -1)

250
Mileage (104 miles) 14
200
12
150
100 10
50
8
0
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Fig. 1.9 Incident statistics of gas pipeline in the United States (gas pipeline safety events are called
incidents according to 49 CFR 191.3)
Mileage
16
Accident rate
Incident rate (times·10-3·mile-1·yr-1)

3.0
14
12
Mileage (104 miles)
10 2.5
6
2.0
4
0 1.5
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
Fig. 1.10 Accident statistics of oil pipelines in the United States (oil pipeline safety events are
called accidents according to 49 CFR 195.50)
Norske Veritas (DNV), and so on. These standards can be classified according to their
primary functions: operating procedure (used to guide the operation and preparation
requirements of inspection in the field), technical issues related (introduce the prin-
ciple of related technology), and management related (requirements for inspection
management). Based on these 20 standards, the statistic is made by the country (or
Probability of failure
Initial Stable
stage operation Aging stage
stage
1-2 years 35-40 years Time
Fig. 1.11 Pipeline failure probability curve (bathtub curve) [34]
10.1%
28.4%
14.9%
External interference
Corrosion
Construction defect/Material failure
Hot tap made by error
3.8% Ground movement
Other and unknown
17.8%
25%
Fig. 1.12 Statistics of failure causes for natural gas pipelines (EGIG)
organization) that established the standard. It can be seen from Fig. 1.15 that most
of the standards are issued by China and USA.
18.3% 11.6%
Corrosion
5.6% 19.3% Equipment failure
Excavation damage
Incorrect operation
7.5% Material failure
Natural force damage
9.4% Other incident cause
6.9% 21.5%
Other outside force damage
(a)
2%
4.5% 1.9%
5.4% 22.5%
Corrosion
Equipment failure
15.6%
Excavation damage
Incorrect operation
Material failure
3.6% Natural force damage
Other accident cause
44.4% Other outside force damage
(b)
Fig. 1.13 Statistics of failure causes for oil and gas pipelines (PHMSA). a Natural gas pipelines;
b oil pipelines
Fig. 1.14 The condition of wastewater infrastructure in the United States

Table 1.6 Classification of

Classification of inspection Inspection technique
pipeline inspection
techniques (by principle)
technology
Visual inspection technology /
Electromagnetic inspection Magnetic flux leakage
technology Remote field eddy current
Broadband electromagnetic
Pulsed eddy current system
Ground penetrating radar
Acoustic inspection technology Acoustic emission method
Ultrasonic method
Ultrasonic guided wave
method
Echo impact
Sonar system method
Leakfinder
Sahara
Optical inspection technology Lidar system
Diode laser absorption
Thermal imaging
Spectral imaging
Chemical component inspection Sniffer method
technology Vapor sampling method
Table 1.7 Classification of

Classification of inspection Monitoring technique
pipeline monitoring
techniques (by principle)
technology
Optical fiber sensing Optical time domain reflection
Fiber Bragg grating
Interferometric optical fiber
sensor
Signal-based method Volume/mass balance method
Negative pressure wave
method
GPS time tag method
Pressure point analysis method
Cross correlation analysis
Transient test-based techniques
State estimation method
Table 1.8 Related standards for pipeline inspection

Standard code Name Issuance department Latest version
SY/T 4109 [48] Nondestructive testing National Development 2020
standard of oil and gas and Reform Commission
steel pipeline (China)
CJJ 181 [49] Technical specification Housing and urban–rural 2012
for inspection and development of the
evaluation of urban People’s Republic of
sewer China
DVGW G 465-4 [50] Gas leak detection and Deutscher Verein des 2001
gas measuring devices Gas- und Wasserfaches
for supervision of gas
pipeline systems
CJJ 61 [51] Technical specification Housing and urban–rural 2017
for urban underground development of the
pipeline detection and People’s Republic of
survey China
API RP 1130 [52] Computational pipeline API 2007
monitoring for liquids
GB/T 27699 [53] Steel pipeline in-line Standardization 2011
inspection technical Administration of the
specification People’s Republic of
China
API 1175 [54] Pipeline leak detection API 2015
program management
T/CIRA 14 [55] X-ray digital imaging China Isotope and 2020
inspection method for Radiation Association
pipeline welds (CIRA)
API TR 1149 [56] Pipeline variable API 2015
uncertainties and their
effects on leak
detectability
API PUBL 346 [57] Results of range-finding API 1998
testing of leak detection
and leak location
technologies for
underground pipelines
API PUBL 4716 [58] Buried pressurized API 2002
piping systems leak
detection guide
API RP 1175 [59] Recommended practice API 2017
for pipeline leak
detection—program
management, and
companion guide bundle
(continued)
Table 1.8 (continued)

Standard code Name Issuance department Latest version
ISO 20486 [60] Non-destructive ISO 2017
testing—leak
testing—calibration of
reference leaks for gases
testing—leak
testing—tracer gas
method
testing—leak
testing—vocabulary
testing—acoustic
emission testing
(AT)—leak detection by
means of acoustic
emission
ASTM E432 [63] Standard guide for ASTM 2017
selection of a leak
testing method
ASTM E479 [64] Standard guide for ASTM 2006
preparation of a leak
testing specification
SY/T 6889 [65] In-line inspection of National Energy 2012
pipelines Administration (China)
DNVGL-RP-F302 [66] Offshore leak detection DNV 2019
6
4
China
USA
ISO
Europe
Fig. 1.15 Statistics of relevant standards for pipeline inspection released in different countries (or
organizations)
References
1. Lu, H., Behbahani, S., Azimi, M., Matthews, J. C., Han, S., & Iseley, T. (2020). Trench-
less construction technologies for oil and gas pipelines: State-of-the-art review. Journal of
Construction Engineering and Management, 146(6), 03120001.
2. Lu, H., Xu, Z. D., Iseley, T., & Matthews, J. C. (2021). Novel data-driven framework for
predicting residual strength of corroded pipelines. Journal of Pipeline Systems Engineering
and Practice, 12(4), 04021045.
3. Lu, H., Iseley, T., Behbahani, S., & Fu, L. (2020). Leakage detection techniques for oil and gas
pipelines: State-of-the-art. Tunnelling and Underground Space Technology, 98, 103249.
4. Gan, Y., El-Houjeiri, H. M., Badahdah, A., Lu, Z., Cai, H., Przesmitzki, S., & Wang, M. (2020).
Carbon footprint of global natural gas supplies to China. Nature Communications, 11(1), 1–9.
5. Erickson, P., & Lazarus, M. (2014). Impact of the Keystone XL pipeline on global oil markets
and greenhouse gas emissions. Nature Climate Change, 4(9), 778–781.
6. Jin, H. (2010). Design and construction of a large-diameter crude oil pipeline in Northeastern
China: A special issue on permafrost pipeline. Cold Regions Science and Technology, 64(3),
209–212.
7. Allouti, M., Schmitt, C., Pluvinage, G., Gilgert, J., & Hariri, S. (2012). Study of the influence
of dent depth on the critical pressure of pipeline. Engineering Failure Analysis, 21, 40–51.
8. Shin, S., Lee, G., Ahmed, U., Lee, Y., Na, J., & Han, C. (2018). Risk-based underground
pipeline safety management considering corrosion effect. Journal of Hazardous Materials,
342, 279–289.
9. Yuan, M., Zhang, H., Wang, B., Shen, R., Long, Y., & Liang, Y. (2019). Future scenario
of China’s downstream oil supply chain: An energy, economy and environment analysis for
impacts of pipeline network reform. Journal of Cleaner Production, 232, 1513–1528.
10. Lu, H., Ma, X., Huang, K., Fu, L., & Azimi, M. (2020). Carbon dioxide transport via pipelines:
A systematic review. Journal of Cleaner Production, 266, 121994.
11. Zhu, Q. Z., Wu, C., Li, Q. Y., Zhang, X. Q., Zeng, L. B., & Gao, S. B. (2017). Development
status and trend of global oil and gas pipelines. Oil & Gas Storage and Transportation, 36(4),
375–380.
12. PricewaterhouseCoopers, L. L. P. (2013). Economic Impacts of the Oil and Natural Gas Industry
on the US Economy in 2011.
13. United States Government Accountability (USGA) Office. (2014). Oil and gas transportation:
Department of transportation is taking actions to address rail safety, but additional actions are
needed to improve pipeline safety. Neuroscience & Behavioral Physiology, 27(4), 347.
14. Cheng, F. (2016). Monitor safety of aged fuel pipelines. Nature, 529(7585), 156.
15. Lam, C., & Zhou, W. (2016). Statistical analyses of incidents on onshore gas transmission
pipelines based on PHMSA database. International Journal of Pressure Vessels and Piping,
145, 29–40.
16. Bianchini, A., Guzzini, A., Pellegrini, M., & Saccani, C. (2018). Natural gas distribution system:
A statistical analysis of accidents data. International Journal of Pressure Vessels and Piping,
168, 24–38.
17. Girgin, S., & Krausmann, E. (2016). Historical analysis of US onshore hazardous liquid pipeline
accidents triggered by natural hazards. Journal of Loss Prevention in the Process Industries,
40, 578–590.
18. Bubbico, R. (2018). A statistical analysis of causes and consequences of the release of hazardous
materials from pipelines. The influence of layout. Journal of Loss Prevention in the Process
Industries, 56, 458–466.
19. Vetter, C. P., Kuebel, L. A., Natarajan, D., & Mentzer, R. A. (2019). Review of failure trends
in the US natural gas pipeline industry: An in-depth analysis of transmission and distribution
system incidents. Journal of Loss Prevention in the Process Industries, 60, 317–333.
20. Biezma, M. V., Andrés, M. A., Agudo, D., & Briz, E. (2020). Most fatal oil & gas pipeline
accidents through history: A lessons learned approach. Engineering Failure Analysis, 110,
104446.
References 19
21. Restrepo, C. E., Simonoff, J. S., & Zimmerman, R. (2009). Causes, cost consequences, and risk
implications of accidents in US hazardous liquid pipeline infrastructure. International Journal
of Critical Infrastructure Protection, 2(1–2), 38–50.
22. Siler-Evans, K., Hanson, A., Sunday, C., Leonard, N., & Tumminello, M. (2014). Analysis of
pipeline accidents in the United States from 1968 to 2009. International Journal of Critical
Infrastructure Protection, 7(4), 257–269.
23. Belvederesi, C., Thompson, M. S., & Komers, P. E. (2018). Statistical analysis of environmental
consequences of hazardous liquid pipeline accidents. Heliyon, 4(11), e00901.
24. Zhu, Y., Qian, X. M., Liu, Z. Y., Huang, P., & Yuan, M. Q. (2015). Analysis and assessment of
the Qingdao crude oil vapor explosion accident: Lessons learnt. Journal of Loss Prevention in
the Process Industries, 33, 289–303.
25. CNN. (2017). Keystone pipeline leaks 210,000 gallons of oil in South Dakota. https://www.
cnn.com/2017/11/16/us/keystone-pipeline-leak/index.html
26. Tudorica, D. (2014). A comparative analysis of various methods of gas, crude oil and oil
derivatives transportation. International Journal of Sustainable Economies Management, 3(1),
16–25.
27. Pioneer Press. (2014). Northwestern Minnesota gas pipeline explosion: ‘It was just hell
on earth’. https://www.twincities.com/2014/05/25/northwestern-minnesota-gas-pipeline-exp
losion-it-was-just-hell-on-earth/
28. Liaw, H. J. (2016). Lessons in process safety management learned in the Kaohsiung gas
explosion accident in Taiwan. Process Safety Progress, 35(3), 228–232.
29. Wang, T., Wang, X., Li, Z., Xue, L., Gao, Z., & Wang, Y. (2017). Comparison on failures of
long-distance oil & gas pipelines at home and abroad. Oil & Gas Storage and Transportation,
36(11), 1258–1264.
30. Alberta Energy Regulator, 2017. 2016/17 Annual Report, Alberta, Canada.
31. European Gas Pipeline Incident Data Group. (2015). EGIG gas pipeline incident. 9th Report
of the European Gas Pipeline Incident Data Group (period 1970–2013). http://www.egig.eu/
uploads/bestanden/ba6dfd62-4044-4a4d-933c-07bf56b82383. Accessed December 17, 2015.
32. Goodfellow, G. D., Lyons, C. J., & Haswell, J. V. (2018). UKOPA Pipeline Product Loss
Incidents and Faults Report (1962–2016). Ref: UKOPA/17/002, Derbyshire, UK.
33. St. Clair, A. M., & Sinha, S. (2014). Development of a standard data structure for predicting
the remaining physical life and consequence of failure of water pipes. Journal of Performance
of Constructed Facilities, 28(1), 191–203.
34. Farshad, M. (2011). Plastic pipe systems: Failure investigation and diagnosis. Elsevier.
35. Wu, J., Li, H., Wang, L., Xu, B., Wang, H., Cai, Y., Liu, S., & Tao, Z. (2017). Performance
evaluation indices of pipeline leak monitoring systems. Oil & Gas Storage and Transportation,
36(2), 209–213.
36. Kishawy, H. A., & Gabbar, H. A. (2010). Review of pipeline integrity management practices.
International Journal of Pressure Vessels and Piping, 87(7), 373–380.
37. Xie, M., & Tian, Z. (2018). A review on pipeline integrity management utilizing in-line
inspection data. Engineering Failure Analysis, 92, 222–239.
38. Mora, R. G., Hopkins, P., Cote, E. I., & Shie, T. (2016). Pipeline integrity management systems:
A practical approach. ASME Press.
39. Liu, Z., & Kleiner, Y. (2013). State of the art review of inspection technologies for condition
assessment of water pipes. Measurement, 46(1), 1–15.
40. Dong, Y., & Yu, D. (2005). Estimation of failure probability of oil and gas transmission pipelines
by fuzzy fault tree analysis. Journal of Loss Prevention in the Process Industries, 18(2), 83–88.
41. Murvay, P. S., & Silea, I. (2012). A survey on gas leak detection and localization techniques.
Journal of Loss Prevention in the Process Industries, 25(6), 966–973.
42. Folga, S. M. (2007). Natural gas pipeline technology overview. Technical Report, Argonne
National Laboratory.
43. Liu, A. E. (2008). Overview: Pipeline Accounting and Leak Detection by Mass Balance, Theory
and Hardware Implementation. Quantum Dynamics.
44. Batzias, F. A., Siontorou, C. G., & Spanidis, P. M. (2011). Designing a reliable leak bio-detection
system for natural gas pipelines. Journal of Hazardous Materials, 186(1), 35–58.
45. Sivathanu, Y. (2003). Natural gas leak detection in pipelines. Technology Status Report,
En’Urga Inc., West Lafayette, IN.
46. Scott, S. L., & Barrufet, M. A. (2003). Worldwide assessment of industry leak detection
capabilities for single & multiphase pipelines. Offshore Technology Research Center.
47. Bie, Q., Zheng, Y., Fu, M., & Song, X. (2007). Leakage detection technology of oil and gas
transmission pipelines and its development trend. Petroleum Engineering Construction, 33(3),
19–23.
48. Development, N., & Commission, R. (2013). SY/T 4109: Nondestructive testing standard of
oil and gas steel pipeline. Petroleum Industry Press.
49. Housing and Urban-Rural Development of the People’s Republic of China. (2012). CJJ 181:
Technical specification for inspection and evaluation of urban sewer. China Architecture &
Building Press.
50. Deutscher Verein des Gas- und Wasserfaches. (2001). Gas leak detection and gas measuring
devices for supervision of gas pipeline systems. Wirtschafts und Verlagsgesellschaft Gas und
Wasser.
51. Housing and Urban-Rural Development of the People’s Republic of China. (2017). CJJ
61: Technical specification for urban underground pipeline detection and survey. China
Architecture & Building Press.
52. American Petroleum Institute. (2007). API RP 1130: Computational pipeline monitoring for
liquids. American Petroleum Institute.
53. Standardization Administration of the People’s Republic of China. (2011). GB/T 27699: Steel
pipeline in-line inspection technical specification. Standards Press of China.
54. American Petroleum Institute. (2015). API RP 1175: Pipeline leak detection program
management. American Petroleum Institute.
55. China Isotope and Radiation Association. (CIRA). (2020). T/CIRA 14: X-ray digital imaging
inspection method for pipeline welds. Standards Press of China.
56. American Petroleum Institute. (2015). API TR 1149: Pipeline variable uncertainties and their
effects on leak detectability. American Petroleum Institute.
57. American Petroleum Institute. (1998). API PUBL 346: Results of range-finding testing of
leak detection and leak location technologies for underground pipelines. American Petroleum
Institute.
58. American Petroleum Institute. (2002). API PUBL 4716: Buried pressurized piping systems
leak detection guide. American Petroleum Institute.
59. American Petroleum Institute. (2017). API RP 1175: Recommended practice for pipeline leak
detection—Program management, and companion guide bundle. American Petroleum Institute.
60. International Organization for Standardization. (2017). ISO 20486: Non-destructive testing—
Leak testing—Calibration of reference leaks for gases. International Organization for Stan-
dardization, Geneva, Switzerland.
Leak testing—Tracer gas method. International Organization for Standardization, Geneva,
Switzerland.
62. International Organization for Standardization. (2017). ISO 18081: Acoustic emission testing
(AT)—Leak detection by means of acoustic emission. International Organization for Standard-
ization, Geneva, Switzerland.
63. American Society for Testing and Materials. (2017). ASTM E432: Standard guide for selection
of a leak testing method. ASTM International, West Conshohocken, PA, USA.
64. American Society for Testing and Materials. (2006). ASTM E479: Preparation of a leak testing
specification. ASTM International, West Conshohocken, PA, USA.
65. National Energy Administration. (2012). SY/T 6889: In-line inspection of pipelines. Petroleum
Industry Press.
66. Det Norske Veritas. (2019). DNVGL-RP-F302: Offshore leak detection. Det Norske Veritas,
Høvik, Norway.
Leak testing—Vocabulary. International Organization for Standardization, Geneva, Switzer-
land.
Chapter 2
Pipeline Inspection Technology
2.1 Introduction
This chapter describes some common inspection methods for pipelines, covering oil,
gas, and water pipelines. This chapter lets readers understand the principles, appli-
cability, advantages, and disadvantages of different methods. According to the prin-
ciple, the inspection can be divided into visual, electromagnetic, acoustic, ultrasonic,
optical, chemical, and other methods.
2.2 Visual Inspection Technology
Visual inspection technologies can be used to look at the internal condition of pipes,
enhancing people’s vision to give a more professional explanation [1, 2]. Before
adopting visual inspection, the following factors need to be considered when selecting
a visual inspection technology:
(1) Whether it can be shut down, isolated, and dewatered;
(2) Whether there are existing access points into the pipeline;
(3) Pipeline operating conditions such as pressure and flow velocity;
(4) Pipeline’s physical configuration (size, alignment, and potential obstructions);
(5) Pipe size;
(6) Pipe length, pipe bends, and pipe slope;
(7) Obstructions inside the pipe [3–8].
Closed-Circuit Television (CCTV) inspection is a widely used condition assess-
ment technology for obtaining inside condition information on water pipes, and it
also belongs to one of the most famous visual inspection technologies [9]. CCTV
emerged in the 1950s and matured in the 1980s. It is always mounted on a self-
propelled vehicle and can be entered into the pipe for camera recording. The techni-
cian can interpret the pipeline condition and determine the method in the next step
https://doi.org/10.1007/978-981-19-6798-6_2
22 2 Pipeline Inspection Technology
based on the recorded video. When the water level in the pipe is high, and CCTV
cannot effectively shoot underwater, the sonar system can be used as a supplement to
scan the underwater mud, foreign objects, and major structural damage, which basi-
cally solves the shortcomings of CCTV. In addition to drainage pipes, Shanghai’s
tap water and gas pipelines also began to use CCTV for inspection [10–12].
Sewer Scanner and Evaluation Technology (SSET) and PANORAMO® 3D
Optical Pipeline Scanner are improved technologies of CCTV [13]. SSET combines
the technical advantages of scanners and gyroscopes, and can provide detailed digital
images. The composition of the SSET, and its viewable range are shown in Figs. 2.1
and 2.2, respectively. SSET is developed by the CORE company and the Tokyo
municipal government sewer service company. It can provide data such as pipe
geometry, vertical and horizontal deviations, structural defects, location, and range
of defects. Moreover, it can automatically complete the analysis and evaluation of
the integrity of the wastewater pipelines. This information can assist engineers and
assessment managers make more reliable and economical repair decisions. Compared
with the existing CCTV technology, the main advantages of SSET are: (1) it can
obtain higher quality data; (2) it can speed up the evaluation process; (3) digital
imaging helps to classify and digitize defect data; (4) marking different defects with
different colors is helpful for rapid identification of defects. Moreover, it can measure
the horizontal and vertical deviation of the pipeline. However, the inspection cost is
too high, about 1.5 to 2 times CCTV. Digisewer, an advanced technology SSET tech-
nology developed by Envirosight LLC, can capture detailed visual data from every
square inch of pipe wall at speeds up to 70 ft/min. In addition, it triples the efficiency
of traditional CCTV, and the generated data is more suitable for fast transmission
and efficient storage (910 m/GB).
PANORAMO® 3D Optical Pipeline Scanner uses two scanning units. Each unit
contains a 185-degree fish-eye lens and a high-resolution digital camera. It can collect
High Intensity
On Board Sensors for Measuring LED Light Array
Inclination and Meander
High Angle "Fish Eye" Lens Provides Simultaneous

Conventional or Custom 360° Side Scan and Forward Looking Images
Transport Configurations
Fig. 2.1 Composition of SSET

2.2 Visual Inspection Technology 23
12 o' clock
Sewer Scanner & Evaluation Technology 3 o' clock 9 o' clock
6 o' clock
Fig. 2.2 Viewable range of Sewer Scanner and Evaluation Technology
data at a rate of 21 m/min without stopping. The number of pipes inspected per
day is twice that of traditional CCTV, while operating costs are reduced by 30%. In
addition, there are some other visual inspection technologies, such as Videoscope, 3D
optical scanning, and laser-based pipe surface profiling. Their applications, merits,
and limitations are shown in Table 2.1.
In the United States, a large part of the old pipes (steel pipes) of urban gas users
have been replaced by plastic pipes. Generally, the pipe diameter of the plastic pipes
at the user end is only 12.7 mm. The fittings on the plastic pipes are usually welded
by hot fusion welding, which will produce large welded joints inside the pipes,
thus reducing the actual inner diameter of the pipes. At this time, large inspection
equipment cannot enter the pipes. However, for the gas pipelines of urban users,
defects and wear are the main inspection objects, and high definition guarantees the
inspection results. Therefore, the Videoscope can be used for inspection.
According to the survey results, visual inspection technology is mainly oriented
toward high precision, faster, and more economical. However, the future direction of
the leading development is the automatic interpretation of inspection data, which can
significantly improve analysis efficiency. In fact, this technology is developing. The
difficulty lies in the accuracy of the automatic identification of defects. In addition,
the CCTV inspection data can be combined with the geographic information system
(GIS), which can significantly enrich the progressive nature and practicability of the
GIS system [21, 22].
Table 2.1 Merits and limitations of various visual inspection methods

Technology Applications Merit Limitations
CCTV Sewers, water 1. Simple, relatively 1. Can only provide
pipelines, gas cheap internal surface defects
pipelines 2. Data is easy to store information
2. The test results are
qualitative and need to be
explained
3. Quantitative rating
requires trained inspectors
4. Pan and tilt are needed,
and the camera must stop
at each defect’s location
5. Tuberculated pipes may
need to be cleaned before
the inspection
6. Cannot be inspected
in-service
7. Requires a special
launching and retrieval
chamber in water mains
SSET Sewers, water 1. Speed is constant 1. The inspection cost is
pipelines, gas and without stopping 1.5 to 2 times that of
pipelines 2. Pan and tilt are not CCTV
needed 2. The same as the
3. Better for limitations 1, 2, 3, 5, 6,
comparison of data and 7 of CCTV
from one year to the
next
4. Better perspective,
higher quality images
Panoramo Sewers, water 1. Inspection speed is 1. The same as the
pipelines, gas fast (21 m/min) limitations 1, 2, 3, 5, 6,
pipelines 2. Cost 30% less than and 7 of CCTV
CCTV
3. Higher inspection
accuracy
Digisewer [14] Sewers 1. Inspection speed is 1. The same as the
fast (21 m/min) limitations 1, 2, 3, 5, 6,
2. Pan and tilt are not and 7 of CCTV
needed
3. File sizes are 3 to 10
times smaller than
other methods
Videoscope [15, Water, gas, and oil 1. Visualization of 1. Can only be used for
16] pipelines (small hidden areas short distance and small
diameter) 2. High quality control diameter pipes
of inspected devices 2. The same as the
3. High speed video limitations 1, 2, and 3 of
capture capability CCTV
4. Remote inspection
(continued)
2.3 Electromagnetic Inspection Technology 25

Technology Applications Merit Limitations
Laser-based pipe Metallic pipelines 1. Prejudgment of 1. Tuberculated pipes need
surface profiling pipeline degradation to be cleaned before the
[17–20] 2. Provides accurate inspection
geometric dimensions 2. Cannot be inspected
for pipe repair in-service
3. Can be applied to a 3. Cannot detect cracks
variety of pipe sizes
2.3 Electromagnetic Inspection Technology
2.3.1 Magnetic Flux Leakage
Magnetic flux leakage (MFL) technology uses the magnetic medium to magnetize the
pipe wall to a saturated state and form a magnetic circuit around the wall. When the
pipe wall is defective or abnormal, part of the magnetic field lines will penetrate the
pipe wall and leak magnetic flux to detect the defect or leakage [23–28]. The principle
of magnetic flux leakage is shown in Fig. 2.3. The flow of pipeline magnetic flux
leakage inspection is shown in Fig. 2.4. According to the electromagnetic theory,
in the no-current area, the magnetic induction intensity is no-curl, which can be
expressed as:
B = −μ0 ∇V (2.1)
where B is magnetic induction intensity; V is scalar magnetic potential; μ0 is

permeability in free space. Applying Gauss’s law to Eq. (2.1) yields Laplace’s
Magnetic flux lines

Steel
Permanent magnet
Steel brushes
Magnetic sensor
Pipe wall
Corrosion pit
Fig. 2.3 The principle of magnetic flux leakage technology [29]

Analytical Waveform
Data storage
processing display
Computer
Multichannel data acquisition

system
Magnetic flux leakage

detection probe
Fig. 2.4 Flow of pipeline magnetic flux leakage inspection [2]
equation:
∇2V = 0 (2.2)
When the measured ferromagnetic material is surrounded by air, the solution

of the Laplace equation can be substituted into Eq. (2.1) to obtain B at any point
P(x, y, z) in the space near the material.
On two interfaces with different magnetic conductivity materials, the tangential
component of the magnetic field intensity vector and the normal component of the
magnetic induction vector are continuous so that the scalar magnetic potential V can
be obtained:

1 ρm x , , y , , z ,
V (x, y, z) = dv
4π |R|
v

1 σm x , , y , , z ,
+ ds (2.3)
4π |R|
s

ρm x , , y , , z = −∇ · M
, (2.4)

σm x , , y , , z , = n · M (2.5)
where R is the position vector from any point on the surface or inside of the ferro-
magnet to the point P(x, y, z) in the ,air;, v and
,
s are the volume and surface area of
the ferromagnet, respectively; ρm x , y , z is the bulk magnetic charge density;
σm x , , y , , z , is surface magnetic charge density; M is magnetization intensity;
n is the unit vector pointing outward perpendicular to the ferromagnetic surface.
Combining Eqs. (2.1) and (2.3), B at P(x, y, z) can be expressed as:

μ0 ρm x , , y , , z ,
B(x, y, z) = Rdv
4π |R|3
v
μ0
σm (x , ,y , ,z , )
+ 4π |R|3
Rds (2.6)
s
Assuming that the relative permeability of the measured pipeline is constant and
the bulk magnetic charge density is ignored, the expression of the magnetic charge
density σm is:
h
2 +1
σm = 5.3 hb2 H0 (2.7)
μb +1
where H0 is magnetic field intensity inside the pipe wall; μ is the relative permeability
of the pipe; h 2 and b are dimension parameters of pipe wall defects. Combining
Eqs. (2.6) and (2.7), it can be known that the magnetic flux leakage signal intensity
generated at the pipe defect is proportional to the magnetic field intensity in the pipe
wall. That is, the overall magnetization of the pipe wall directly affects the magnetic
flux leakage signal characteristics at the pipe defects.
A magnetic sensor is a critical component in magnetic flux leakage, which converts
magnetic signals into electrical signals. Magnetic sensors can be Hall elements,
induction coils, magnetic flux gates, magnetic sensitive transistors, and magnetic
sensitive diodes. Their measuring ranges are shown in Table 2.2.
According to the direction of the magnetic field, magnetic flux leakage inspection
can be divided into axial magnetic field inspection, transverse magnetic field inspec-
tion, radial magnetic field inspection, and rotating magnetic field inspection. Their
characteristics are shown in Table 2.3. Magnetic flux leakage cannot be used in a
small diameter pipeline, and the inner wall of the pipe needs to be cleaned before
inspection because the instrument requires close contact with the pipe wall.
Table 2.2 Measuring range

Magnetic sensor Measurable intensity (Tesla)
of the magnetic sensor [23]
Hall element [10−5 , 10]
Induction coil [10−13 , 10]
Magnetic flux gate [10−12 , 10−13 ]
Magnetic sensitive transistor [10−6 , 10−1 ]
Magnetic sensitive diode [10-6 , 10−1 ]
Table 2.3 Characteristics of four magnetic flux leakage methods

Method Magnetic field direction Feature
Axial magnetic field inspection Axial 1. Sensitive to lateral defects
2. Ability to accurately identify
defect lengths
3. Insensitive to axial defects
4. Difficult to judge the width of
defects
Circumferential magnetic field Circumferential 1. Sensitive to axial defects
inspection 2. As a supplement to
conventional axial magnetic flux
leakage technology
Radial magnetic field Radial 1. More accurate in judging the
inspection length of the defect
Rotating magnetic field Axial and circumferential 1. Combination of axial magnetic
inspection field inspection and
circumferential magnetic field
inspection
2.3.2 Remote Field Eddy Current
The remote field eddy current (RFEC) system can detect cracks in the wall, corrosion
pits, abrasion, and other defects [30]. This inspection technique was proposed in
the 1950s. In 1951, Maclean W. R. obtained the US patent for this technology (ID:
2573799). In 1958, Shell Group of Companies applied the technology for the first time
to corrosion inspection of oil casings. In the late 1980s and early 1990s, the remote
field eddy current technology developed rapidly [31]. The inspection system was
developed, and the built-in probe was used to detect gas and buried pipelines. In 2000,
the American Society for Testing and Materials (ASTM) developed the standard
named “Standard Practice for In Situ Examination of Ferromagnetic Heat-Exchanger
Tubes Using Remote Field Testing” (ASTM E2096-00).
As shown in Fig. 2.5, the remote field eddy current system consists of an exci-
tation coil and one or more inspection coils. The distance between excitation and
inspection coils is 2–3 times the pipe diameter. The magnetic field lines emitted by
the excitation coil diffuse outwardly through the pipe wall, and diffuse the pipe wall
inwardly with surface defects in the remote field zone. The inspection coil receives
the magnetic field lines [32, 33]. There are three zones between the excitation coil
and the inspection coil: direct coupled zone, transition zone, and remote field zone.
The signal-distance characteristic curve of the detector coil can be seen in Fig. 2.6.
(1) Direct coupled zone: Ded < 1.8D. In this region, the magnetic field from the exci-
tation coil interacts with the pipe wall to produce a concentrated eddy current
field, the induced electromotive force decreases drastically as the distance
increases, and the phase changes little.
Indirect energy transmission path
Direct energy path

D
Exciter coil Direct coupled zone Transition zone Remote field zone
Detector coil
0 1D 2D 3D
Fig. 2.5 The principle of remote field eddy current [34]
Fig. 2.6 Signal-distance characteristic curve of the inspection coil [35]
(2) Transition zone: Ded is in the range of 1.8-2D. In this region, there is a large
interaction between the magnetic flux from the field coil and the magnetic flux
caused by the eddy current, the falling rate of the induced electromotive force
decreases, and sometimes, even a slight increase can happen. In the meantime,
the phase will change dramatically.
(3) Remote field zone: Ded is in the range of 2-3D, and amplitude and phase are
reduced at a low rate. The phase lag of the induced electromotive force is
approximately proportional to the thickness of the pipe wall, which the one-
dimensional skin effect phase formula can approximate:
√
θ = 2δ π f μσ (2.8)
where θ is the phase lag of the induced electromotive force; δ is the thickness of the
pipeline; f is excitation frequency; μ is the permeability of the wall material; σ is
the electrical conductivity of the wall material.
Remote field eddy current is established based on Maxwell’s equation, which is
[36–38]:
⎧
⎪
⎪ ∇ × H = Jc + Je
⎨
∇ × E = − j ωB
(2.9)
⎪
⎪ ∇·E =0
⎩
∇·D=ρ
where H is the magnetic field intensity; B is the magnetic induction intensity; D is

the electric displacement vector; E is the electric field intensity; ρ is the bulk current
density; Jc is applied excitation current density; Je is eddy current density; ω is the
angular frequency. According to the magnetic field theory, these parameters have the
following relations:
⎧
⎨ D = εE
B = μH (2.10)
⎩
Je = σ E
Define the vector magnetic potential A to obtain:
∇×A=B (2.11)
∇·A=0 (2.12)
Substituting Eq. (2.11) into Eq. (2.9) and defining the scalar potential function ϕ:
E = −(∇ϕ + j ω A) (2.13)
Substituting Eqs. (2.10), (2.11), and (2.13) into Eq. (2.9) yields:

∇× μ (∇
1
× A) = Jc − σ (∇ϕ + j ω A) (2.14)
From the isotropy of the material, the Coulomb gauge, and the vector equation:
∇ 2 A = −μJc + μσ (∇ϕ + j ω A) (2.15)
According to Maxwell’s equation, we can get:
∇ × J = − ∂ρ
∂t
(2.16)
Substituting Eqs. (2.13) into Eq. (2.16) gives:
−∇ · σ (∇ϕ + j ω A) = − ∂ρ
∂t
(2.17)
∂ρ
Ignoring the displacement current, then ∂t
= 0, and we can get:
∇ · (∇ϕ + j ω A) = 0 (2.18)
Solving Eqs. (2.15) and (2.18), we get:
B=∇×A (2.19)
Je = −σ (∇ϕ + j ω A) (2.20)
The model of remote field eddy current without defects is axisymmetric, and the
mathematical model of remote field eddy current in cylindrical coordinates is:
∂ 2 Aθ 1 ∂ Aθ ∂ 2 Aθ
∂r 2
+ r ∂r
+ ∂ z2
− Aθ
r2
= −μJcθ + j ωμσ Aθ (2.21)
where z is pipe axis direction; r is pipe radial direction; θ is pipe circumferential

direction; Aθ is the circumferential component of vector magnetic potential. Aθ can
be solved by Eq. (2.21), we can get:
⎧
⎨ Br = − ∂∂Azθ
B = Arθ + ∂∂rAθ (2.22)
⎩ r
Jcθ = − j ωσ Aθ
2.3.3 Broadband Electromagnetic
Broadband electromagnetic (BBE) technology was first used in the coal mining
industry in Australia, and now it can be applied to pipeline inspection, especially
water mains. BBE technology can take effect by inducing eddy currents to flow near
the transmitter and transmitting a signal that has a wide frequency spectrum range
(From 50 to 50,000 Hz) [39, 40]. As shown in Fig. 2.7, the transmitter coil conveys
the alternating current (AC) to the surface of the pipe and generates an alternating
magnetic field. Flux lines pass through the pipe wall, generating a voltage across it.
The voltage produces an eddy current in the pipe wall, which induces the secondary
magnetic field. The wall thickness can be estimated indirectly by measuring signal
attenuation and phase delay.
Broadband electromagnetic can only be used to inspect ferrous pipes. Note that
it can also work through thick linings and coatings [41–43]. BBE technology can
detect defects and anomalies in the pipeline. For the prestressed concrete cylinder
pipeline (PCCP), BBE can detect wall thickness, quantify graphitization, and locate
broken wires. The eddy current distributions of the intact pipe and defective pipes
are shown in Fig. 2.8.
Fig. 2.7 Primary field Induced magnetic

distribution from axial field
sensor configuration [39]
Transmitter coil
Fig. 2.8 Eddy current Eddy current

distributions a Without
defect b With defect [39]
(a)
Eddy current
Fracture
(b)
2.3.4 Pulsed Eddy Current System
Pulsed eddy current system (PECS) is used to determine the wall thickness of steel
pipes. A transmitter coil produces a rectangular-shaped eddy current, and each loop
contains a positive pulse and a negative pulse [44, 45]. It produces a rapidly changing
pulsed magnetic field through energization and outage in the excitation coil [46, 47].
Coil
Magnetic field
Overburden
Insulation
Pipe wall
Eddy current
Fig. 2.9 Inspection principle of pulsed eddy current system
At this time, an induced eddy current can be generated and infiltrate to the pipe
wall. The thickness of the wall is related to the eddy current intensity. The receiving
element monitors the attenuation of the eddy current pulse in the metal wall. Thus
the average thickness of the metal can be calculated, and the principle of pulsed
eddy current system is shown in Fig. 2.9. The generating process of the pulsed eddy
current and the picking process of the inspection signal can be seen in Fig. 2.10 , and
the attenuation curve of the wall thinning is shown in Fig. 2.11.
The distribution of current density on the cross-section of the conductor is different
due to the difference in conductivity, permeability, and alternating current frequency.
The distribution of alternating current on the cross-section of a conductor is attenuated
from the conductor surface to the center according to the exponential law [48, 49]. The
relationship between the standard penetration depth and the frequency, conductivity,
and permeability is:
δ= √ 1
π f μσ (2.23)
where δ is standard penetration depth, mm; f is frequency, Hz; μ is relative perme-

ability; σ is conductivity, (µΩ)−1 . Generally, when the thickness of the test piece
is less than 3 δ, an effective signal can be detected, and this thickness is called the
effective penetration depth. Since
ω
f = 2π
(2.24)
Substituting Eq. (2.24) into Eq. (2.23), we get:

Pulse signal Transient induced voltage
Excitation Output
Primary coil Detector coil
Electromagnetic Electromagnetic
induction induction
Pulsed transient magnetic field Regenerative transient magnetic field
Electromagnetic Electromagnetic
induction induction
Pipeline Transient pulsed eddy current
Electromagnetic
induction
Fig. 2.10 The generating process of pulsed eddy current and the picking process of inspection
signal
Fig. 2.11 Attenuation curve

of the wall thinning
Normal condition
Eddy signal strength
Thinning
position of wall
thickness
Time
/
δ= 2
ωμσ
(2.25)
According to the Fourier series, the pulsed eddy current square wave excitation
signal can be decomposed into a fundamental wave component and n harmonic
components, and the fundamental frequency ω1 is:
ω1 = 2π f (2.26)
The frequency spectrum of the n-th harmonic components is:
ω = 2nπ f (2.27)
Substituting Eq. (2.27) into Eq. (2.25), the δ corresponding to the n-th harmonic
can be obtained:
/
δn = nπ 1f μσ (2.28)
It can be seen from Eq. (2.28) that δ decreases with the increase of harmonic
number n without changing the excitation frequency and the permeability and
conductivity of the test piece. The high-frequency component tends to the surface of
the test piece, while the low-frequency component can penetrate the interior of the
test piece, so the low-frequency component contains most of the information of the
test piece.
2.3.5 Ground Penetrating Radar
Ground penetrating radar (GPR) is a technology for locating underground pipes. It

can be divided into high-frequency radar (200–2500 MHz) and low-frequency radar
(10–200 MHz) according to the center frequency of the antenna. High-frequency
radar is mainly used for inspection, and low-frequency radar for exploration [50–53].
GPR transmits high-frequency electromagnetic waves in the form of short pulses into
the ground from a ground transmitting antenna. The high-frequency electromagnetic
wave meets the underground layer or target body with an electrical difference. It then
returns to the ground after receiving reflection, received by the receiving antenna.
The principle and data processing flow chart of GPR are shown in Figs. 2.12 and
2.13 [54–57].
The required time of pulse wave travel (two-way):
√ √
(y+r )2 +(L+0.5x)2 + (y+r )2 +(L−0.5x)2 (2.29)
t= v
where r is the pipe radius. When the ground penetrating radar is above the pipeline
(L = 0), the following equation can be obtained:
√
4(y+r )2 +x 2 (2.30)
t= v
The propagation velocity v of the electromagnetic wave in soil medium can be

expressed as:
v= √c
εr (2.31)
Receiving antenna
Transmitting antenna
x L
Ground T R T R T R
Soil y
v
Pipe
Fig. 2.12 The principle of GPR [58]
Data acquisition Data transmission File editing
File preprocessing
Data processing
Wave filtering Deconvolution Deviation Transform
Hierarchical Dielectric constant

processing calculation
Velocity calculation
Treatment
Parameter calculation
effect
Graphic analysis Graphics

Result output
editing modification notes
Fig. 2.13 Data processing flow chart of GPR [59]

2.4 Acoustic Inspection Technology 37
where v is the propagation velocity of the electromagnetic wave in an object; c is the

propagation velocity of a radar wave in the vacuum; εr is relative dielectric constant
mean value of the medium above the target. The buried depth of the pipeline is:
√
c2 t 2 −0.25εr x 2
y= 4εr
−r (2.32)
In addition to detecting pipelines, GPR can also identify pipeline leaks. When
water or gas pipelines leak in soil, the moisture content of the soil changes. The soil
moisture content θ is closely related to its dielectric constant. When the gas or water
transmission pipeline leaks, the physical properties of the soil covering the pipeline
will change, the leakage rate of the pipeline will change, and the parameters such as
soil porosity and saturated water content will also change. GPR is very sensitive to
the change in soil moisture and can detect subtle differences. After the soil becomes
dry, the following four kinds of received signals can be observed:
(1) the decrease in soil conductivity will increase the penetration depth of GPR
signals;
(2) The decrease in soil conductivity will increase the reflection intensity of the
pipeline signal;
(3) The decrease in soil moisture content will reduce the dielectric properties and
shorten the reflection time of the pulse;
(4) After the pipeline leaks, the wave velocity of the soil above the pipeline will
change.
Studies have shown that the moisture content is linearly related to the square root
of the dielectric constant, namely [60]:
√
θ = 0.128 εr − 0.204 (2.33)
2.4 Acoustic Inspection Technology
2.4.1 Acoustic Emission Method
The acoustic emission method means that the sound sensor is installed outside the
pipe wall in advance. If the pipe leaks, noise will be generated at the leak point,
and the sound sensor installed on the outer wall of the pipe will be received and
amplified [61–64]. The computer software processes the received noise into the rele-
vant entire waveform of the sound, and through the analysis of the entire waveform,
the purpose of monitoring the leakage condition of the pipeline and locating the
leakage point is achieved. The modern acoustic emission inspection system based
on signal processing is shown in Fig. 2.14. This technology is suitable for pipes with
low flow and high pressure. In the inspection of buried pipelines, acoustic emission
Fig. 2.14 Acoustic emission Acoustic emission Acoustic emission

system [66–68] sensor 2 sensor 1
Preamplifier Preamplifier
Signal Signal
collector collector
Computer
technology has higher sensitivity than other inspection methods, and the positioning
of the leakage source is also very accurate [65, 66].
Acoustic emission inspection technology can locate leaks according to time and
signal strength. If the positioning is based on time (see Fig. 2.15), the principle is
to install acoustic emission sensors on both sides of the pipeline [69–71]. Assuming
that the distance from the leak point to sensor 1 is X 1 and the propagation time is t 1 ;
the distance from the leak point to sensor 2 is X 2 and the propagation time is t 2 , and
the propagation speed is v, then:

X1 = L
2
+ 21 (t1 − t2 )v
(2.34)
X2 = L
2
+ 21 (t2 − t1 )v
/
v= 1
(2.35)
a p ρ 1+ Ela
D
p
where a p is reciprocal of pressure; ρ is the density of the medium in the pipeline;

E is the elastic modulus of the pipeline; D is the diameter of the pipeline; l is the
thickness of the outer wall of the pipeline.
The acoustic emission signal also decays as the signal propagates, and there is
a relationship between the acoustic emission signal strength and the distance the
acoustic emission signal travels:
Acoustic Leak point Acoustic

emission sensor 1 emission sensor 2
X2 X1
Fig. 2.15 Schematic diagram of leak location inspection
R L = R0 e−cL (2.36)
where R L is the intensity of the acoustic emission signal captured by the signal
collector at location L; R0 is the intensity of the acoustic emission signal captured
by the signal collector at location 0; c is the attenuation coefficient, which is related
to the diameter, wall thickness, material, medium, etc. of the pipeline, and needs
to be calibrated. The calibration of attenuation coefficient c needs to obtain data
from the test and extract the characteristic parameters from the data through system
identification technology.
When the constant c is known, two acoustic emission sensors can be arranged near
the leak point, which are recorded as L 1 and L 2 , respectively. Accordingly, the leak
intensities R L1 and R L2 of the two sensor positions can be measured by the acoustic
emission sensor and substituted into Eq. (2.35), and the acoustic emission signal
intensity R0 of the leak point can be obtained. Replace the calculated R0 back to
Eq. (2.35), and calculate the distance L between the sensor and the leakage position
with Eq. (2.37):
L= 1
c
ln R0
RL (2.37)
2.4.2 Ultrasonic Method
The ultrasonic method uses the characteristic that ultrasonic waves can be partially
reflected on the metal surface for inspection. When the detector runs in the pipe,
the ultrasonic waves emitted by the detector probe are respectively reflected on the
inner and outer surfaces of the pipe and then received by the detector probe [72–75].
The data processing unit of the detector can calculate the time difference between
the two groups of reflected waves received by the probe multiplied by the ultrasonic
propagation speed to obtain the actual wall thickness of the pipeline.
The frequency of ultrasonic waves is usually greater than 20 kHz. Compared with
audible sound waves (between 20 and 20 kHz), ultrasonic waves have the following
basic characteristics [76–78]:
(1) Good directivity. When the signal source emits ultrasonic waves in a specific
direction, the ultrasonic waves mainly propagate in this direction.
(2) Strong anti-interference ability. Changes in the external environment (such as
light and temperature) have a more negligible effect on ultrasonic waves. There-
fore, ultrasound can still play a better role at night, in a strong light environment,
or in a situation full of electromagnetic wave interference.
(3) Strong penetrating power. When ultrasonic waves propagate in liquids or solids,
the attenuation of ultrasonic waves is not significant, but when they propagate
in air, the attenuation will be considerable. This is because as the ultrasonic
frequency increases, the absorption coefficient of the medium will increase
accordingly. Therefore, as the ultrasonic frequency increases, the penetrating
power of the ultrasonic wave will gradually decrease. The penetration of low-
frequency ultrasonic waves is extreme and can even penetrate thicker steel plates.
(4) Propagation requires a medium. The essence of ultrasonic waves and sound
waves is the same. That is, ultrasonic waves and sound waves are both mechan-
ical waves. The propagation of mechanical waves requires a medium, so the
ultrasonic wave needs a medium during the transmission process.
According to acoustic theory, the wave equation can be expressed as:
∂2 p
− c2 ∂∂ yp2 = 0
2
∂t 2
(2.38)
where p is sound pressure; c is sound velocity; y is the direction of wave propagation.

The relationship between the speed of sound, density, and pressure in the sound field
is as follows:
/ / / /
c = ∂∂ρp = γ ρp = γ RT M
= CCvr RTM
(2.39)
where Cv is the specific heat at constant volume; M is the molar mass of air; R is
the universal constant of air; Cr is the specific heat at constant pressure, and T is the
absolute temperature of the air.
When the pipeline leaks, the internal and external pressure difference is significant,
and the leakage pore size is small, the gas flow rate at the leakage will be significant.
Due to the large flow rate of the leaking gas, the gas at the leaking place has a high
Reynolds coefficient, so a turbulent jet is generated, as shown in Fig. 2.16.
After a series of experiments and analysis, Ma et al. concluded that the factors
affecting sound pressure are pressure difference and leakage diameter. The noise
level formula proposed by them is as follows [79]:

( p1 − p0 )3
L = 80 + 20 log10 D
D0
+ 10 log10 p1 p02 (2.40)
0d B = 20µPa
Fig. 2.16 Schematic Leak point

diagram of jet generated by
leakage [79]
P0
P1
where D is diameter, mm; D0 is reference diameter of 1 mm diameter; L is the total

sound pressure level at 90° to the leakage hole at 1 m away from the leakage; p1 is
the standing pressure of the leakage hole, kPa; p0 is standard atmospheric pressure,
kPa.
2.4.3 Ultrasonic Guided Wave Method
When the ultrasonic wave propagates in the pipeline, it will be reflected back and
forth at the interface, resulting in complex waveform conversion and mutual inter-
ference. This ultrasonic wave guided by the medium boundary is called an ultrasonic
guided wave. The most significant difference between an ultrasonic guided wave
and traditional ultrasonic testing is that the former can test 100% of the material of
a sizeable long-distance pipeline at one test point, while the traditional ultrasonic
can only test this point at one test point. In the ultrasonic guided wave inspection of
the pipeline, the sensor is an essential part of the inspection system [80]. Different
sensors can be selected to stimulate single or different guided wave modes. This
method can be used for long-distance inspection of oil and gas pipelines, which
can save inspection time, reduce labor intensity, and detect the defects of the whole
section of the pipeline [81–83]. In the pipeline-guided wave inspection, the sensors
mainly include piezoelectric, magnetostrictive, and pulsed laser.
The motion equation determines the displacement component of the guided wave
in the pipeline, which is the basis of using a guided wave to detect defects [84–86].
Note that just like the difference in the definition of guided wave and bulk wave, the
difference between the motion equations of guided wave and the bulk wave is that
the motion equations of guided wave need to meet the boundary conditions of the
waveguide.
Gazis first gave an analytical expression for the wave equation in an infi-
nite isotropic hollow cylinder under free stress boundary conditions [87, 88].
Starting from Navier’s motion equation, the equation of motion of the guided wave
propagating in an isotropic elastic hollow cylinder is deduced.
2 →
(λ + μ)∇ ∇ · U→ + μ∇ 2 U→ = ρ ∂∂tU2 (2.41)
Since a hollow cylinder is isotropic, the displacement field is decomposed into

dilatational scalar potential ∅ and isovolumetric vector potential H→ by Helmholtz
decomposition.
U→ = ∇∅ + ∇ × H→ , ∇ · H→ = 0 (2.42)
∇ · ∇∅ = ∇ 2 ∅ (2.43)
∇ · H→ = 0 (2.44)
∇ · ∇ × H→ = 0 (2.45)
∇ × ∇ × ∇∅ = 0 (2.46)
2 →

∇ (λ + μ)∇ 2 ∅ − ρ ∂∂t ∅2 + ∇ × μ∇ 2 H→ − ρ ∂∂tH2 = 0
2
(2.47)
For an infinite hollow cylinder, the gauge invariance condition is applied. In other
words, the volume of the hollow cylinder is constant, so only the boundary conditions
of the inner and outer surfaces are considered, and the boundary conditions at both
ends are not considered.
1 ∂2∅
∇2∅ = c2L ∂t 2 (2.48)
1 ∂ 2 H→
∇ 2 H→ = cT2 ∂t 2
(2.49)
/
λ+2μ
c2L = ρ
(2.50)
/
μ
cT2 = ρ (2.51)
∂2∅ 1 ∂∅ 1 ∂2∅ ∂2∅

∇2∅ = ∂r 2
+ r ∂r
+ r 2 ∂θ 2
+ ∂ z2
(2.52)
H→ = Hr −
→
er + Hθ −
→
eθ + Hz −
→
ez (2.53)

∇ 2 H→ = ∇ ∇ · H→ − ∇ × ∇ × H→ (2.54)
→
∇ 2 H→ = ∇ 2 Hr −
er + Hθ −→
eθ + Hz − →ez

1 1 ∂ Hθ − →
= ∇ 2 Hr − 2 Hr − 2 2 er
r r ∂θ
→
+ ∇ 2 Hθ − r12 Hθ − 2 r12 ∂∂θHr −eθ + ∇ 2 Hz −
→
ez (2.55)
Gazis proposed the expressions of the dilatational scalar potential ∅ and the
isovolumetric vector potential.
∅ = f (r ) . (mθ )ei (kz−wt) (2.56)
Hr = h r (r ).r (mθ )ei (kz−wt) (2.57)
Hθ = h θ (r ).θ (mθ )ei (kz−wt) (2.58)
Hz = h z (r ).z (mθ )ei (kz−wt) (2.59)
where m is the circumferential order of a wave; k is the wave number; f (r ) and h ξ (r )

(ξ = r, θ, z) are coefficients to be found. Functions .(mθ ) and .ξ (mξ )(ξ = r, θ, z)
must only contain sin(mθ ) or cos(mθ ). However, the assumed potential leading
to displacement given by Gazis cannot describe the torsional mode, so Zhang
et al. proposed an alternative solution suitable for the torsional mode [89].
∅ = f (r ) sin(θ m)ei (kz−wt) (2.60)
Hr = h r (r ) cos(mθ )ei(kz−wt) (2.61)
Hθ = h θ (r ) sin(mθ )ei (kz−wt) (m = 0, 1, 2 . . .) (2.62)
Hz = h z (r ) cos(mθ )ei (kz−wt) (2.63)
ω2 ω2
α2 = c2L
− k2, β2 = cT2
− k2 (2.64)
Considering the stress-free boundary conditions:

|
∂u ξ |
σ r ξ |r =a,b = μ ∂r |r =a,b
= 0(ξ = r, θ, z) (2.65)
Substituting Eqs. (2.52)–(2.55) and Eqs. (2.60)–(2.65) into Eqs. (2.48) and (2.49)
yield four Bessel functions. The displacement components in the three directions of
the equation of motion can be obtained by substituting the Bessel function into the
Helmholtz decomposition. Then the stress components can be obtained. Based on the
above results, considering the linear elasticity theory and the stress boundary condi-
tions inside and outside the hollow cylinder, the eigenvalue equation of the dispersion
characteristics of the wave propagation inside the infinite hollow cylinder can be
obtained. The condition for the solution of the eigenvalue equation of the dispersion
characteristic is that the coefficient determinant is zero, so that all wavenumber-wave
velocity relationships can be solved [90–93].
Defect inspection sensors are available in two configurations, namely pulse-echo
and pitch-catch. Pulse-echo method requires one or two sensors to excite and receive
backscattered signals. For the pitch-catch method, at least two sensors are required
for one excitation signal and the other to receive the forward scattering signal, as
shown in Fig. 2.17.
Fig. 2.17 Exciting and receiving methods. a pulse-echo with one transducer, b pulse-echo with
two transducers, c pitch-catch with two transducers [94, 95]
Pulse-echo method is the most commonly used damage inspection method in

non-destructive testing. It relies on the reflected wave echo signal generated by the
guided wave interaction with boundary and defect to detect defects. Generally, the
exciter and the receiver are on the same side. The defect type identification and the
damage degree judgment are based on the mode and amplitude of the echo signal.
Guided wave imaging technology based on the phased array and synthetic focusing
is mainly based on the pulse-echo method [96–100].
The pitch-catch method uses the scattered wave signal generated by the guided
wave interaction with boundary and damage to identify defects. The sensor device
is usually transmitted at one end and received at the other. The received scattered
wave signal contains defect information and other structural feature information, and
then the defects are evaluated by analyzing the amplitude and mode of the scattered
wave. An active structural health monitoring network is formed by pitch-catch and
the distributed sensor array permanently connected to the structure. The pitch-catch
method is usually used to detect micro fatigue cracks and micro corrosion because
these defects are too small to reflect enough echoes to be picked up by the receiver. The
method of pitch-catch is also widely used in guided wave tomography. Pulse-echo
and pitch-catch have the same signal-to-noise ratio (SNR):
u damage 3√3/4 Rdamage R
s= u noise
= 2 p3/2 βδT
= 1.612 p3/2
damage
βδT (2.66)
where s is the signal-to-noise ratio (SNR); β depends on wave pattern and type of
baseline subtraction used in structural health monitoring; Rdamage is the reflection
coefficient of damage; δT is the temperature change. The difference between the two
methods is that the pulse-echo directly receives the signal scattered from the defect
or feature, while the pitch-catch method receives the forward scattered signal.
Transducers are used as actuators (to excite guided waves) and sensors (to
receive guided waves) in nondestructive testing or structural health monitoring
processes. The transducers widely used in the actual pipeline inspection mainly
include piezoelectric transducers, magnetostrictive transducers, and electromagnetic
acoustic transducers.
The piezoelectric transducer based on the piezoelectric principle can realize the
mutual conversion of electrical and mechanical energy. Therefore, the piezoelectric
transducer can be used both as an actor and a sensor. Piezoelectric equations describe
the relationship between mechanical quantity (stress Th and strain Sk , h, k = 1, 2, …,
6) and electrical quantity (electric field strength E i and electric displacement Di , i
= 1, 2, 3). The four sets of piezoelectric equations are as follow:
h-type:

T = c D S − h̃ D
(2.67)
E = −h S + β S D
d-type:

S = s D T + d̃ E
(2.68)
D = dT + ε T E
g-type:

S = s D T + g̃ D
(2.69)
E = −gT + β T D
e-type:

T = c E S − ẽE
(2.70)
E = eS + ε S E
where “ ~ ” is a transpose matrix or vector; c D , c E are elastic constant compo-

nent under constant electrical displacement and constant electric field, respectively;
β S , β T are the component of dielectric isolation ratio at constant strain and constant,
respectively; s D , s E are the flex constant component under constant electric displace-
ment and constant electric field, respectively; ε S , ε T are the dielectric constant
component under constant strain and stress, respectively; h, d, g, e are four constant
components related to stiffness, strain, voltage, and stress, respectively. Piezoelectric
transducers are usually under boundary conditions of mechanical freedom and elec-
trical open circuit, corresponding to the third piezoelectric equation. Several common
deformation modes of piezoelectric vibrators are shown in Fig. 2.18.
The magnetostrictive transducer consists of many nickel (or other magnetostric-
tive material) plates or laminations, which are arranged parallel to one edge of each
laminate, and the edge of each laminate is connected to the bottom of the treatment
tank or other surfaces to be vibrated. The coil is placed around the magnetostric-
tive material. When current is supplied through the coil of a wire, a magnetic field
is generated (just like a high-power wire). This magnetic field causes the magne-
tostrictive material to contract or elongate, thus introducing sound waves into the
application. Due to the physical size limitation of the magnetostrictive transducer
(the frequency depends on the transducer’s length, while the transducer’s length at
higher frequencies needs to be shorter and shorter), it is inherently limited to oper-
ating at a frequency lower than about 30 kHz. The design of the magnetostrictive
transducer leads to its low efficiency. Magnetostrictive systems rely on the double
conversion of electrical energy to magnetic energy and then from magnetic energy
to machinery to generate sound waves. Compared with the piezoelectric system,
a magnetostrictive transducer will need more electric energy to produce the same
amount of ultrasonic vibration. Due to the energy loss in the process of coil heating
and the influence of hysteresis, the efficiency of the magnetic system is usually less
than 50%.
(a) (b)
(c) (d)
(e) (f)
Fig. 2.18 Typical vibration modes of piezoelectric vibrators a Longitudinal extension b Radial
extension c Transverse extension d In-plane poled thickness-sheard e Thickness-poled thickness-
sheard f Thickness-poled face sheard [101]
2.4.4 Echo Impact
When the impact echo method is used for inspection, the tester uses a small hammer
to quickly knock the surface of the tested object to generate a mechanical wave. The
mechanical wave propagates within the component at a specific wave speed. When it
propagates, it will encounter the interface of different materials, and internal defects,
or when it reaches the other end of the component, it will be reflected. A piezoelectric
displacement sensor is placed near the striking point. The sensor converts the surface
displacement of the measured object caused by the reflection of the shock wave into
a voltage–time signal, which is the received reflection waveform [102–104]. After
mathematical transformation, the collected analog signal can complete the time–
frequency domain conversion of the received waveform, which makes it convenient
for researchers to comprehensively analyze the internal damage of the tested object
in the time and frequency domain.
A mechanical wave is also called a stress wave because it is a stress–strain distur-
bance at a certain point on the structure. The frequency range of the stress wave
generated by the steel hammer is wide (but generally less than 80 kHz). Therefore,
Table 2.4 Specific acoustic

Material Specific acoustic impedance (kg/m2 s)
impedance of some materials
Steel 47 × 106
Concrete 7 × 106 –107
Soil 3 × 105 –4 × 106
Water 5 × 105
Air 0.4
in actual operation, the tester is often required to knock several times according to
the different measured objects, and judge whether the generated stress wave meets
the requirements according to the received waveform [105, 106].
The amplitude of the reflected wave depends on the impedance contrast between
the structural element and the material in contact with the element. The reflection
coefficient is used to characterize the amplitude of the reflected wave:
Z t −Z i
R= Z t +Z i (2.71)
where Z i and Z t are the acoustic impedance of structural elements and materials in
contact with opposite surfaces. Table 2.4 summarizes typical Z values for various
materials. The impact source produces P wave, S wave, and Rayleigh wave.
2.4.5 SmartBall
SmartBall is used to locate and estimate the leakage of the natural gas pipeline. It is
an acoustic inspection technology. SmartBall includes a series of acoustic sensors,
accelerometers, magnetometers, ultrasonic transmitters, and temperature sensors,
which roll in the pipeline with the flow of natural gas [107–111]. At first, this tech-
nology could only be used in water transmission pipelines. Since 2010, it has been
demonstrated that SmartBall can be used in oil and gas pipelines. During the tech-
nical demonstration, Pure Technologies conducted 19 experiments, and the range
of locating leak point error is ±2 m. SmartBall can be used for pipelines with a
diameter greater than 100 mm and put into the pipeline through the pigging device
in the natural gas plant. To better use the SmartBall technology, Pure Technologies
has designed the SmartBall analyst software. In addition, to apply the SmartBall to
long-distance natural gas pipelines, Pure Technologies has improved battery and data
storage capacity. After the improvement, the battery endurance can reach 110 h, and
the data storage capacity can reach 16 GB (SmartBall with a diameter of 100 mm)
and 32 GB (SmartBall with a diameter of more than 100 mm).
The advantage of SmartBall technology is that it can be used for pipes of any mate-
rial, and it can realize non-stop inspection under pressure. In addition, the sensitivity
of leak inspection is high, and it can identify natural gas with a leak volume of 0.11
L/min.
The limitation of the SmartBall is that the acoustic activity associated with the
leakage of the SmartBall comes from the pressure difference across the pipe wall,
and the SmartBall will not detect the leak if the pressure difference is negligible. The
sensitivity of all leakage inspection technologies is a function of several variables,
so a fixed threshold cannot be established. Under different working pressures, the
minimum detectable leakage of the SmartBall is different. Therefore, the minimum
detectable leakage needs more experiments. In addition, other ambient noise may
affect the measurement results [112–116].
Tianjin University has established a SmartBall inspection system. The structure
of the electromagnetic signal transmitter antenna of the inspection system is a cylin-
drical coil framework. The outer layer is wrapped with multi-layer enameled wire,
the center of the framework is a magnetic core, and the magnetic core material is
generally a ferrite or silicon steel sheet with high permeability. The distance from
the transmitting antenna to the receiving module is much larger than the size of
the transmitting antenna. The transmitting antenna can be equivalent to a compact
solenoid. The equivalent radius of the compact solenoid is R, and the length is h.
The current flowing through the coil is:
i (t) = I0 cos(ωt) (2.72)
where ω is the angular frequency of alternating current; I0 is the effective value of

alternating current. By studying the magnetic field distribution law of a single-turn
coil in space, the magnetic field distribution of the whole compact solenoid in space
can be obtained by superposition. The magnetic field distribution of a single-turn coil
under current excitation can be equivalent to the magnetic dipole model, as shown
in Fig. 2.19.
The magnetic moment of a single-turn coil can be expressed as:
pm = i (t)Sn (2.73)
Fig. 2.19 Model of z

magnetic dipole [117]
M
I
R O y
x
where S is the cross-sectional area of the coil; n is the magnetic moment direction.
The magnetic moment of a solenoid is the vector sum of the magnetic moments of
each single-turn coil, so the magnetic moment of the entire solenoid can be expressed
as:
Pm = N pm (2.74)
where N is the total number of turns of the solenoid coil.

The rolling model of the SmartBall is shown in Fig. 2.20. In the vertical plane,
the angle between the axis of the transmitter antenna and the pipeline axis is α, and
the position of the receiving antenna is the origin O. The axis of the pipeline is the
x-axis, the horizontal direction is the y-axis, and the vertical direction is the z-axis.
During the movement of the SmartBall, the change of the included angle α needs
to be considered:
P m = Pm (cos α, 0, sin α) (2.75)
Assume that the spatial coordinates of the transmitter antenna are (−x, −y, −z),
the spatial coordinates of the receiving module are (0, 0, 0), and the space vector
from the receiver module to the transmitter antenna is r = (x, y, z).
In practical application, the tracking point is usually set at the mileage stake of
the pipeline, and the receiving antenna will be placed directly above the pipeline.
Therefore, the magnetic field distribution of the electromagnetic signal transmitting
antenna can be expressed by the magnetic field component as [117]:
⎧
⎪
⎪ Bx = Pm μ0 3(x cos α+z sin α)x
− cos α
⎪
⎪ 5 3
⎨ 4π
(x 2 +z 2 ) 2 (x 2 +z 2 ) 2
By = 0 (2.76)
⎪
⎪
⎪
⎪ Pm μ0 3(x cos α+z sin α)x sin α
⎩ Bz = 4π 5 − 3
(x 2 +z 2 ) 2 (x 2 +z 2 ) 2
Fig. 2.20 Rolling model of z

SmartBall [117]
Receive
antenna x
O
y
α
SmartBall
During the rolling process of the SmartBall, the transmitter antenna rotates in
the vertical plane, so the relationship between α and the rolling distance ∆x of the
SmartBall can be deduced as:
2∆x 180◦
α = α0 + D π
(2.77)
where α0 is the initial angle; D is the outside diameter of the SmartBall.
2.4.6 Sonar System Method
The sonar profile system uses sound waves to detect underwater objects. It is mainly
used for water pipelines (CCTV cannot detect underwater objects). The sonar probe
rotates and sends out sonar signals quickly, and then receives the signals from the
pipeline wall or objects in the pipe. After computer processing, the cross-section of
the pipe can be formed. The sonar profile system can calculate the section size, shape,
damage, or defect location of the pipeline. Sediment deposition, sediment thickness,
and waste blockage can also be found below the water surface [118–120].
The sonar system may include an underwater sonar unit, sonar communica-
tion interface unit, sonar image acquisition workstation and software, cable winch
and slip ring, communication cable (long-distance), and deck power supply unit
(optional). High-frequency sonar can provide high-resolution scanning, but its pene-
tration ability is poor. Low-frequency sonar has intense penetration, but limited scan-
ning resolution. Therefore, high-frequency sonar is more suitable for clean water,
and low-frequency sonar can be used for high-concentration muddy water or water
with suspended particles [121, 122].
As shown in Fig. 2.21, a sonar system and CCTV can be used together for pipes
with partial water. CCTV can shoot on the water surface, a sonar system can assist the
underwater inspection process, and underwater images can be displayed and saved
simultaneously. The inspection structure of the sonar and CCTV combined system
is shown in Fig. 2.22.
A sonar system is specially used for monitoring the leakage of underwater oil and
gas pipelines. The system can monitor more than 1 billion cubic feet of seawater
and can cover 360° at a single sensor location without requiring a capable sonar
operator monitoring system. It works by projecting short—time high—bandwidth
ultrasonic pulses into the water and listening to echoes from any object. The system
automatically generates auditory and visual alarms when the leak is detected. Data
on leakage characteristics and location can be displayed on the monitor.
Fig. 2.21 Sonar and CCTV

system in the pipe with part
of water
CCTV
Water
Sonar
Sonar Transporter OR Float
Mainline cable reel
Processor Mixer Cable distance counter
Multi conductor
CCTV system
Fig. 2.22 Inspection structure of Sonar and CCTV combination system

2.4.7 Leakfinder
The Leakfinder system is mainly used for leakage inspection of water pipelines.
The system consists of a leakage sensor, wireless signal transmission system, and
computer. Acoustic sensors (such as accelerometers or hydrophones) are connected
to two contact points on the pipe, such as fire hydrants. Accelerometers are used
to inspect vibration caused by leakage, while hydrophones are used to detect sound
caused by leakage in the water column. Accelerometers are very sensitive to back-
ground noise. Hydrophones are usually used with accelerometers to achieve a better
signal-to-noise ratio [123–125]. The principle of Leakfinder is shown in Fig. 2.23.
The computer calculates the cross-correlation function of the two leakage signals
to determine the time delay (τmax ) between the two sensors [127–129]. Then, the
leak location can be derived from the following equation:
L1 = D−c×τmax
2
(2.78)
L2 = D − L1 (2.79)
where c is the speed of sound propagation in pipes.
Computer
Sensor
Receiver
RF
transmitter
Leak
L1 L2
Fig. 2.23 Principle of Leakfinder [126]

Cable drum and

operations unit
Insertion
Surface device
locator
Sensor head
Fig. 2.24 The Sahara system [5]
2.4.8 Sahara
The Sahara system uses a hydrophone attached to an umbilical cable that runs through
a water pipe to record leakage noise. Location beacons can be tracked on the ground
to mark leaks for excavation and subsequent repair [130–134].
Sahara locates leaks by identifying unique acoustic signals generated by leaks in
pipe walls, joints, or steel welds. The magnitude of leakage can also be estimated from
acoustic signals. Cavitation in the pipe can also be detected by its unique acoustic
characteristics, as shown in Fig. 2.24.
Video and lighting sensors are also provided on the Sahara platform to provide
CCTV inspection of in-service drinking water pipelines. The wastewater main pipe
was also successfully inspected during the inspection by flushing the pipeline with
clean water.
2.5 Optical Inspection Technology
2.5.1 Lidar System
The Lidar system is based on the principle of spectral absorption. Gas molecules
selectively absorb light, and the concentration of gas is obtained by analyzing the
laser’s initial power and echo power [135–138]. An aircraft with a laser emitting
device flies along the natural gas pipeline in the air, tunes the laser to the absorption
wavelength of the detected gas, absorbs a part of the laser energy by the gas, and
2.5 Optical Inspection Technology 55
Fig. 2.25 Schematic

diagram of natural gas leak
inspection of the lidar system
measures the concentration of methane in the atmosphere using the received laser
echo signal, as shown in Fig. 2.25.
The Airborne Natural Gas Emission Lidar (ANTM GEL) system developed by
ITT of the United States includes a high-resolution mapping camera, dial sensor,
digital video camera, and aircraft, and the inspection speed can reach 160 km/h
[139–141]. In 2017, Bridger Photonics, Inc. developed a technology that can display
3D high-quality gas cloud images based on the frequency modulated continuous wave
(FMCW) ranging and simultaneous path integrated gas concentration measurement.
Since 2013, Ball Aerospace has been developing advanced airborne sensors and
expanding the coverage (400 m) of the instrument to ensure the sampling resolution
and positioning accuracy [110, 142].
The inversion process of lidar to the inspection of natural gas leakage is based on
the Lambert–beer theorem:
P = K Se−D(υ) = K Se−α(υ)·2C R (2.80)
where P is the laser echo power, W; S is the initial laser power, W; υ is the laser
frequency, Hz; K is the optical receiving efficiency (the ratio of echo power to initial
power without gas absorption); D(υ) is the absorption path on the optical path, m;
α(υ) is absorption coefficient of unit concentration medium light at frequency υ,
ppm−1 ·m−1 ; C R is the integrated concentration of methane gas path (ppm·m).
Lidar technology is mainly based on the 3.4 and 1.6 µm spectral regions. The
main reason is that the laser light source and detector technology in these two spectral
regions are relatively mature. Another reason is that these two spectral regions can
better avoid water vapor interference.
2.5.2 Diode Laser Absorption Method
The inspection principle of diode laser absorption is similar to that of the lidar
system, except that diode laser is used for illumination. This technique is suitable
for close-range hand-held devices. When the laser beam is directed to the target gas
pipeline, if the natural gas leaks, the beam will be partially absorbed, and the laser
from the leaked air mass is reflected the object after the air mass. To determine the
methane concentration of the reflecting object and detector, the spectrum signal is
received and demodulated by the detector [143]. In 2011, Jiang et al. designed a
methane concentration inspection system based on tunable diode laser absorption
spectroscopy (TDLAS) technology [144]. Eight concentrations of methane were
measured using a long optical path measuring cell. Experiments show that the system
has an excellent linear relationship with methane, and the system is stable and reliable
for a long time [145–149]. The physical basis for quantitative analysis of TDLAS
absorption spectrum is Lambert–beer law:
I (υ) = I0 (υ)e− pS(T )[(ν)ϕ L (2.81)
where I0 (υ) is incident light intensity; I (υ) is transmitted light intensity; ϕ is the
volume ratio of the gas to be measured; S(T ) is the line intensity of the spectral
line, which represents the absorption intensity of the spectral line, is only related to
temperature, cm−2 ·MPa−1 ; p is the total pressure of the gas, MPa; L is the absorption
light path length, cm; [(ν) is a linear function, which represents the shape of the
measured absorption line. This function is related to the temperature, pressure, type
of gas, and content of each component.
The linear intensity of the spectral line can be obtained according to the absorption
line intensity S(T0 ) of a gas molecule at the temperature T0 , to obtain the linear
intensity S(T ) at the temperature T , as shown in Eq. (2.82):
,,
hcν0
0) − hcE 1
− T1 1−e k B T
S(T ) = S(T0 ) Q(T
Q(T )
e kB T 0
hcν0 (2.82)
1−e k B T0
where Q(T ) is the total molecular internal partition function; h is the Planck constant;
c is the speed of light in the vacuum; E ,, is the low energy state energy; k B is
Boltzmann constant; ν0 is the center frequency of the absorption line. The total
molecular internal partition function Q(T ) can be queried by HITRAN molecular
spectrum software. For methane gas in the temperature range of 253 to 333 K,
Eq. (2.83) in polynomial fitting form can also be used for calculation.
Q(T ) = a0 + a1 T + a2 T 2 + a3 T 3 (2.83)
where a0 = −26.479; a1 = 1.1557; a2 = 0.26831 × 10−2 ; a3 = 1.5117 × 10−6 .

The infrared absorption line shape and line width of gas molecules are essen-
tial parameters in quantitative measurement. There are three main absorption lines:
Gaussian line shape, Lorentz line shape, and Voigt line shape. Under high pressure,
the absorption line type is described by Lorentz line type, then:

∆ν L
[(υ) L = 1
π (υ−υ0 )2 +∆υ L2
(2.84)
The relationship between the Lorentz half-width ∆υ L (T ) and the pressure

spreading coefficient γ L is
T0 n
∆υ L (T ) = γ L (T0 ) T
p (2.85)
where p is the total pressure of the tested sample, MPa. The expression for ϕ is:

I (υ)
−ln I0 (υ)
ϕ= (2.86)
pS(T )[(υ)L
In TDLAS technology, if the gas flow rate has a velocity component in the
laser transmission direction, there is a specific offset between the laser receiving
frequency and the actual laser generating frequency, and the difference between the
two frequencies is the Doppler offset [150–155]. The two laser beams incident the
flow field at different incident angles. Due to the Doppler effect, the absorption
spectrum measured by the detector will produce different frequency shifts:
∆υ = υ0 L R ccos θ (2.87)
where L R is the leakage velocity; c is the speed of light; υ0 is the center frequency of
the absorption line; υ is the corresponding frequency offset; θ is the angle between the
laser beam direction and the airflow velocity direction. In the actual measurement,
the absorption is weak due to the short absorption path. In addition, the pressure
change will also cause the shift of the center frequency. Therefore, the frequency
shift of the second harmonic of the absorption spectrum line is used to inverse the
gas velocity, as shown in Fig. 2.26. The flow velocity calculation formula of the
double light path method is:
LR = c∆υ12
υ0 (cos θ1 −cos θ2 ) (2.88)
where ∆υ12 is the frequency shift difference of two absorption spectra.
2.5.3 Thermal Imaging
Thermal imaging inspection technology uses thermal imaging equipment to analyze

the thermal radiation changes around the pipeline. When the natural gas pipeline
leaks, the temperature of the soil around the pipeline will decrease depending on the
Fig. 2.26 Schematic

diagram of double light path
Launcher
velocity measurement [156]
θ2
θ1
Receiving
device
Thermal anomaly
Heat diffusion
Downward seepage
Fig. 2.27 Simplified heat transfer model of heat pipe leakage [161]
throttling effect [157–160]. For a crude oil pipeline that is heated and transported,
the temperature of the surrounding soil will increase when the pipeline leaks. The
representative application products currently include the GasFindIR series gas imager
of FLIR Systems in the United States and the Second Sight series gas imager of the
Bertin Technologies company of France. Zhang et al. [161] proposed a simplified
heat transfer model of heat pipe leakage, as shown in Fig. 2.27. In this model, the
heat transfer process of pipeline leakage can be regarded as an aperiodic non-steady
process. That is, the temperature of the surrounding medium continues to increase
over time, and the scope of the temperature change continues to expand. After a
certain period, the temperature of the surrounding medium gradually approaches the
water temperature in the pipe, and finally, the system reaches thermal equilibrium.
At this time, the temperature distribution remains stable.
This method can be used for mobile vehicles, helicopters, or portable systems,
and can detect several miles to hundreds of miles per day [160–164]. Based on the
characteristics of easy movement, the main components of pipeline thermal imaging
technology include an optical system, infrared inspection system, heat map acquisi-
tion system, preprocessing system, and image display system [165–172]. When the
pipeline thermal imaging technology is in play, the optical system can effectively
concentrate the infrared radiation on the surface of the object to be measured on
the infrared detector, and then perform simple processing through the acquisition
system and the preprocessing system. Finally, the image is displayed in real-time
on the computer terminal through the display module, and the specific structure is
shown in Fig. 2.28.
2.5.4 Spectral Imaging Method
Spectral imaging technology can be further divided into multi-wavelength imaging

technology and hyperspectral imaging technology according to the number of bands.
The spectral imaging method has two modes: absorption mode and emission mode.
The premise of using multi-wavelength emission technology to detect natural gas
concentration is that the temperature of the leaked natural gas is much higher than
that of the surrounding air. Multi-wavelength absorption imaging technology uses a
variety of wavelengths of background radiation to directly image natural gas concen-
tration, even if there is no temperature difference between natural gas and ambient
air [173–177]. Currently, representative applications include the AIRIS imager from
PSI Inc., USA, and the Field-portable Imaging Radiometric Spectrometer (FIRST)
imager from Telops, Inc., Canada [159].
Heatmap display
Pseudo-color transform
Computer processing
system
Image storage
Image analysis
Optical system
Heat map acquisition and pretreatment
Heat map Filtering and

Infrared detector acquisition and temperature USB
correction measurement
Power supply
Fig. 2.28 Hardware system of thermal imaging inspection technology [171]
2.6 Chemical Composition Analysis-Based Method
2.6.1 Sniffer Method
The sniffer method refers to the method of capturing leakage gas and analyzing its
concentration through special instruments. In order to improve efficiency, airborne
equipment is usually used to fly near the pipeline, so that the aircraft passes through
the gas plume [178–184].
2.6.2 Vapor Sampling Method
The vapor sampling method is to sample hydrocarbon vapor near the pipeline. This
method can use a vapor monitoring system or mobile detector. The vapor monitoring
system uses a semi-permeable test tube in parallel with the top of the pipeline to
be monitored. If a pipe leaks, gas can permeate through the pipe’s ethylene–vinyl
acetate membrane and be pumped to a monitoring station for composition testing
[185–188].
2.7 Technology Selection Considerations 61
Table 2.5 Performance indicators of pipeline inspection system [189–191]

Performance indicators Interpretation
Adaptive ability For different environments, transmission media or inspection objects
have corresponding adaptability, improving their performance and
completing tasks according to changing information
Robustness Robustness refers to the ability to accurately complete the inspection
and ensure that the false negatives and false positives are within the
specified standards in the presence of modeling errors, interference,
and noise
Reliability Reliability refers to the ability of the technology to accurately inspect
the pipeline when the pipeline operates within the design range. That
is, the ability of the system to detect abnormal events in the pipeline
and the possibility of the system sending an alarm incorrectly when
no pipeline abnormality occurs
Sensitivity Sensitivity refers to anomalies that the technology can inspect
Accuracy The accuracy is equivalent to reliability
Effectiveness Effectiveness refers to the ability to work continuously
Positioning ability Is it possible to locate the leak and the accuracy of positioning?
Economy Refers to the cost of system construction, operation, and maintenance
Maintainability It refers to whether the system can be repaired quickly or simply
when it fails
Identification ability It refers to the inspection system’s ability to distinguish between
inversion, pressure regulation, pump initiation, pump shutdown, and
leakage
2.7 Technology Selection Considerations
For the pipeline to be inspected, it is necessary to select appropriate technology

and consider the indicators of inspection technology. Through the study of different
documents and standards, 12 indicators need to be investigated for inspection tech-
nology: adaptive ability, robustness, reliability, sensitivity, accuracy, effectiveness,
false-positive rate, response time, positioning ability, economy, maintainability, and
recognition ability. See Table 2.5 for the explanation of each performance indicator.
Various indicators in Table 2.5 should be considered when selecting the appro-
priate inspection technology, and the decision should be made according to the flow of
Fig. 2.29. From the macroscopic point of view, the factors shown in Table 2.6 should
be considered first when selecting inspection tools. In addition, various issues in
Table 2.7 should also be considered. Moreover, the advantage and disadvantages of
inspection methods can be seen in Table 2.8. However, from the performance indi-
cator aspect, each inspection technology’s performance evaluation is shown in Table
2.9.
Determination of the
pipeline type
Determination of
possible technique
Evaluation of technical
feasibility
Feasible
Feasible
techniques
techniques
Evaluation of technical Evaluation of technical

adaptability capability
Available
Available
Economic evaluation
techniques
techniques
Determination of the
technique adopted
Fig. 2.29 Flow chart of selecting inspection methods [2]
Table 2.6 Factors that influence the selection of techniques [2]

Factor Characteristics
Size Large/medium/small
Distance Long/moderate/short
Location Urban/rural/mixed/subsea
Medium Oil/Gas/mixed
On-line inspection Yes/No
Budget Rich/poor
Pipeline condition Good/mixed/near collapse
Strategic importance of pipeline Strategic significance/importance/common
Table 2.7 Problems to be considered in selecting inspection techniques [2]

Item Selection criteria Remark
Technical selection Pipeline type What type of pipeline can be used by
the technique?
Material type What material can be used by the
technique?
Location type Urban pipeline network, long-distance
pipeline, or subsea pipeline?
Access requirement Are there any specific access
requirements (such as power and
launch)?
Pipeline condition-related Is there a limitation if the pipeline is in
limitations bad condition?
Pipeline size-related Is there a limitation if the pipeline
limitations diameter is small?
Destructive/non-destructive Is the pipeline destroyed or not?
Trenchless/excavation Does the road need to be excavated?
On-line inspection/off-line Can it be detected online, or must it be
inspection stopped?
Technical suitability Measured parameters What parameters want to be obtained
(such as leaking sound, pressure drop,
and pressure wave)?
Inspection purpose What is the inspection for (such as the
probability of failure, risk, and level of
service)?
Cooperate with other Is it necessary to cooperate with other
techniques techniques?
Previous use of the technique Is this technique applied in a
comparable situation? What is the
effect?
The maturity of technology Is the technique fully developed? Can
it be used off-the-shelf?
Accuracy/reliability Does the inspection result accurate?
What is the error range? What is the
minimum leakage that can be
detected?
Ease of operation Is it easy to operate?
Inspection efficiency What is the speed of inspection? How
much work can be done every day?
Technical capability Skills required What level of operator skill is needed?
Supporting techniques What technological sophistication is
required needed (such as high-power
computers)?
Standardization Is any standard available?
Table 2.8 Problems to be considered in selecting inspection techniques [2]

Technique Advantage Disadvantage Application
G O W
√ √
Visual inspection 1. Simple, economical, 1. Only the defect
technology and adaptable information on the inner
2. Data is easy to store surface of the pipe can
3. Allows relatively be provided
high scanning speeds 2. The inspection results
and complete offline are qualitative and need
inspection to be explained, while
the quantitative rating
requires professional
inspectors
3. The pipeline may
need to be cleaned
before the inspection
4. Online inspection
cannot be realized
√ √ √
Magnetic flux leakage 1. High inspection 1. The detector shall be
accuracy in close contact with the
2. Service interruption pipe wall
is not required 2. High requirements
for pipeline cleanliness
3. Poor economy
√ √ √
Remote field eddy 1. It can be used to 1. Data interpretation
current inspect broken wires requires professional
and corrosion pits training
2. Online inspection 2. Cleaning may be
can be realized required before
3. There is no need to inspecting
contact the pipe wall
directly
4. Strong adaptability
to pipe size
√ √
Vapor sampling method The response time is 1. The cost is high
short 2. Gases such as biogas
in the soil may cause
false alarms
√ √ √
Broadband 1. The equipment does 1. The inspection
electromagnetic not need to contact the process is
pipe wall time-consuming
2. Not sensitive to 2. Pigging is required
corrosion products for online inspection
(continued)

G O W
√ √ √
Pulsed eddy current 1. Can be operated 1. Data interpretation
system underwater requires professional
2. Not affected by training
coatings or insulating 2. The inspection results
materials are easily affected by
external factors
√ √ √
Ground penetrating 1. It can apply to all 1. The characteristics of
radar materials of the the soil can affect the
pipeline inspection results
2. Inspection speed is 2. Not suitable for pipes
relatively fast with very small
3. No need to excavate diameters
and touch pipes
√ √ √
Acoustic emission 1. The inspection has Mounting the sensor
method higher sensitivity and requires excavation and
accurate positioning may require stopping
2. Can realize online
monitoring
√ √ √
Ultrasonic method 1. Low inspection cost The inspection method
and low power is complex
consumption
2. Fast response and
high sensitivity
√ √ √
Ultrasonic guided wave 1. Shorter inspection 1. The selected
method time and less labor inspection frequency
demand, good must first be obtained
economical from the experiment
2. Especially suitable 2. Professional staff is
for long-distance required to read the
pipeline inspection inspection data
3. The outer layer of the
pipeline, the
inhomogeneity of the
weld, and the severity of
the defect will affect the
echo signal of the
guided wave and the
inspection distance
4. The requirements for
the sensor are high
(continued)

G O W
√ √ √
SmartBall 1. The technology can 1. The acoustic activity
be used for pipes of related to leakage
any material comes from the pressure
2. Online inspection difference on the wall.
can be realized The leak will not be
3. The sensitivity of detected if the pressure
leakage inspection is difference is very small
high 2. The sensitivity of the
inspection is a function
of several variables, so
the fixed threshold
cannot be set up. Under
different working
pressures, the minimum
detectable leakage of
the SmartBall is
different. Therefore,
more experiments are
needed to minimize the
identification of the
leakage
3. Other environmental
noise may affect the
measurement results
√ √ √
Echo impact 1. Strong applicability 1. Embedded parts may
to materials affect wave
2. Easy to operate characteristics and
inspection results
2. Not applicable to
metal
√
Sonar system method 1. Can be used to There are many
monitor submarine oil necessary supporting
and gas pipeline leaks tools
2. Simple operation,
sensitive inspection
3. Can detect small
leaks
√
Leakfinder 1. The technology uses 1. Leakage size cannot
a proprietary enhanced be obtained
correlation method, 2. The larger the pipe
which increases the diameter, the lower the
effectiveness of leak material rigidity, and the
location greater the signal
2. Effective for small attenuation
leaks and high 3. Easy to be disturbed
background noise by low-frequency
vibration
(continued)

G O W
√
Sahara 1. It can be used for The frequency of the
online inspection of required access point is
pipelines determined by the elbow
2. Sensitive to small and flow rate in the pipe
leakage
√
Lidar system The inspection speed is The operation cost is
fast, and the leak high
location is accurate
√
Diode laser absorption 1. It has strong The false alarm rate
anti-interference may be higher
ability, fast inspection
speed, and high
precision
2. It is convenient for
hand-held equipment
and airborne
equipment
3. It can carry out
remote monitoring
√ √ √
Thermal imaging The inspection speed is 1. The equipment is
fast expensive, and the
inspection cost is high
2. If the temperature of
the fluid is consistent
with that of the
surrounding soil,
thermography will not
work
√ √ √
Spectral imaging 1. The false alarm rate 1. Imagers are very
method of the inspection is expensive
very low 2. The data are difficult
2. Options for multiple to explain
platforms
√
Sniffer method 1. The inspection 1. Inspection results
speed is fast may be affected by wind
2. The inspection speed
accuracy is high, and 2. Higher operating
the leak location can costs
be accurately located
3. Labor costs are
lower
Table 2.9 Performance evaluation of some inspection methods (part)
68
Technique Adaptive Positioning Response time Sensitivity Evaluation Continuous False alarm Maintenance Cost
ability accuracy ability monitoring rate requirement
Lidar system Can General N/A General Weak Cannot General High High
Diode laser Can General N/A General Weak Cannot General High High
absorption
Thermal Can General General General General Can High High High
imaging
Spectral Can General General General General Can High High High
imaging
method
Acoustic Cannot General General General Weak Can High General General
emission
method
Ultrasonic Can General Fast High Weak Can General General General
guided wave
method
Ultrasonic Can General Fast High Weak Can General General Low
method
Sonar system Cannot Good Fast High Weak Can Low High High
method
GPR Can Good Fast General Weak Cannot Low General General
SmartBall Can Good Fast Good Weak Cannot Low High High
method
Sniffer method Cannot Good Fast Good Strong Cannot General High High
(continued)
2 Pipeline Inspection Technology
Vapor Can Good Fast Good Weak Cannot High High High
sampling
method
Note The vocabulary used for evaluation are: Good-General-Poor; High-General-Low; Fast-General-Slow; Strong-General-Weak; Can-Cannot; N/A represents
unsure
69
References
1. Latif, J., Shakir, M. Z., Edwards, N., Jaszczykowski, M., Ramzan, N., & Edwards, V. (2022).
Review on condition monitoring techniques for water pipelines. Measurement, 110895.
2. Marlow, D., Heart, S., Burn, S., Urquhart, A., Gould, S., Anderson, M., Cook, S., Ambrose,
M., Madin, B., & Fitzgerald, A. (2007). Condition assessment strategies and protocols for
water and wastewater utility assets. Water Environment Research Foundation.
3. Tuccillo, M. E., Jolley, J., Martel, K., & Boyd, G. (2010). Report on condition Assessment of
Wastewater Collection Systems. United States Environmental Protection Agency.
4. Thomson, J., & Wang, L. (2009). Condition assessment of ferrous water transmission and
distribution systems-state of technology review report. National Risk Management Research
Laboratory, Cincinnati, OH.
5. Liu, Z., Kleiner, Y., Rajani, B., Wang, L., & Condit, W. (2012). Condition assessment technolo-
gies for water transmission and distribution systems. United States Environmental Protection
Agency (EPA).
6. Duran, O., Althoefer, K., & Seneviratne, L. D. (2002). State of the art in sensor technologies
for sewer inspection. IEEE Sensors Journal, 2(2), 73–81.
7. Guo, W., Soibelman, L., & Garrett, J. H., Jr. (2009). Automated defect detection for sewer
pipeline inspection and condition assessment. Automation in Construction, 18(5), 587–596.
8. Mirats Tur, J. M., & Garthwaite, W. (2010). Robotic devices for water main in-pipe inspection:
A survey. Journal of Field Robotics, 27(4), 491–508.
9. Haurum, J. B., & Moeslund, T. B. (2020). A Survey on image-based automation of CCTV
and SSET sewer inspections. Automation in Construction, 111, 103061.
10. Li, D., Cong, A., & Guo, S. (2019). Sewer damage detection from imbalanced CCTV
inspection data using deep convolutional neural networks with hierarchical classification.
Automation in Construction, 101, 199–208.
11. Moradi, S., Zayed, T., & Golkhoo, F. (2019). Review on computer aided sewer pipeline defect
detection and condition assessment. Infrastructures, 4(1), 10.
12. McKim, R. A., & Sinha, S. K. (1999). Condition assessment of underground sewer pipes
using a modified digital image processing paradigm. Tunnelling and Underground Space
Technology, 14, 29–37.
13. Zhang, X., Zhao, P., Hu, Q., Wang, H., Ai, M., & Li, J. (2019). A 3D reconstruction pipeline
of urban drainage pipes based on multiviewimage matching using low-cost panoramic video
cameras. Water, 11(10), 2101.
14. Meijer, D., Scholten, L., Clemens, F., & Knobbe, A. (2019). A defect classification method-
ology for sewer image sets with convolutional neural networks. Automation in Construction,
104, 281–298.
15. Ondrey, G. (2007). Pinpoint problem areas of sewers with this sidewall scanner. Chemical
Engineering, 114(12), 51–52.
16. Zhang, D., Jackson, W., Dobie, G., West, G., & MacLeod, C. (2022). Structure-from-motion
based image unwrapping and stitching for small bore pipe inspections. Computers in Industry,
139, 103664.
17. Duran, O., Althoefer, K., & Seneviratne, L. D. (2007). Automated pipe defect detection
and categorization using camera/laser-based profiler and artificial neural network. IEEE
Transactions on Automation Science and Engineering, 4(1), 118–126.
18. Duran, O., Althoefer, K., & Seneviratne, L. D. (2003). Pipe inspection using a laser-based
transducer and automated analysis techniques. IEEE/ASME Transactions on Mechatronics,
8(3), 401–409.
19. Liu, Z., & Krys, D. (2012). The use of laser range finder on a robotic platform for pipe
inspection. Mechanical Systems and Signal Processing, 31, 246–257.
20. Huang, W., & Kovacevic, R. (2012). Development of a real-time laser-based machine vision
system to monitor and control welding processes. The International Journal of Advanced
Manufacturing Technology, 63(1), 235–248.
References 71
21. Ghavami, S. M., Borzooei, Z., & Maleki, J. (2020). An effective approach for assessing risk
of failure in urban sewer pipelines using a combination of GIS and AHP-DEA. Process Safety
and Environmental Protection, 133, 275–285.
22. Sinha, S. K., & Knight, M. A. (2004). Intelligent system for condition monitoring of
underground pipelines. Computer-Aided Civil and Infrastructure Engineering, 19(1), 42–53.
23. Shi, Y., Zhang, C., Li, R., Cai, M., & Jia, G. (2015). Theory and application of magnetic flux
leakage pipeline detection. Sensors, 15(12), 31036–31055.
24. Pham, H. Q., Tran, B. V., Doan, D. T., Pham, Q. N., Kim, K., Kim, C., Terki, F., & Tran,
Q. H. (2018). Highly sensitive planar Hall magnetoresistive sensor for magnetic flux leakage
pipeline inspection. IEEE Transactions on Magnetics, 54(6), 1–5.
25. Ege, Y., & Coramik, M. (2018). A new measurement system using magnetic flux leakage
method in pipeline inspection. Measurement, 123, 163–174.
26. Bubenik, T. A., Nestlroth, J. B., Eiber, R. J., & Saffell, B. F. (1997). Magnetic flux leakage
(MFL) technology for natural gas pipeline inspection. NDT and E International, 1(30), 36.
27. Liu, B., Cao, Y., Zhang, H., Lin, Y. R., Sun, W. R., & Xu, B. (2015). Weak magnetic flux
leakage: A possible method for studying pipeline defects located either inside or outside the
structures. NDT and E International, 74, 81–86.
28. Peng, X., Anyaoha, U., Liu, Z., & Tsukada, K. (2020). Analysis of magnetic-flux leakage
(MFL) data for pipeline corrosion assessment. IEEE Transactions on Magnetics, 56(6), 1–15.
29. Makar, J., & Chagnon, N. (1999). Inspecting systems for leaks, pits, and corrosion. Journal-
American Water Works Association, 91(7), 36–46.
30. Atherton, D. L. (1995). Remote field eddy current inspection. IEEE Transactions on
Magnetics, 31(6), 4142–4147.
31. Lord, W., Sun, Y. S., Udpa, S. S., & Nath, S. (1988). A finite element study of the remote
field eddy current phenomenon. IEEE Transactions on Magnetics, 24(1), 435–438.
32. Haugland, S. (1996). Fundamental analysis of the remote-field eddy-current effect. IEEE
Transactions on Magnetics, 32(4), 3195–3211.
33. Fukutomi, H., Takagi, T., & Nishikawa, M. (2001). Remote field eddy current technique
applied to non-magnetic steam generator tubes. NDT & E International, 34(1), 17–23.
34. Rajani, B., & Kleiner, Y. (2004). Non-destructive inspection techniques to determine structural
distress indicators in water mains. Evaluation and Control of Water Loss in Urban Water
Networks, 47, 21–25.
35. Krzywosz, K., & Ammirato, F. (1999). Performance Based Remote-Field Eddy Current
Examination of High-Pressure Feedwater Heaters. The e-Journal of Nondestructive Testing &
Ultrasonics 4(8).
36. Kim, D., Udpa, L., & Udpa, S. (2004). Remote field eddy current testing for detection of
stress corrosion cracks in gas transmission pipelines. Materials Letters, 58(15), 2102–2104.
37. She, S., Chen, Y., He, Y., & Zou, X. (2021). Optimal design of remote field eddy current
testing probe for ferromagnetic pipeline inspection. Measurement, 168, 108306.
38. Yang, C., Gao, B., Ma, Q., Xie, L., Tian, G. Y., & Yin, Y. (2018). Multi-layer magnetic
focusing sensor structure for pulsed remote field eddy current. IEEE Sensors Journal, 19(7),
2490–2499.
39. Hazelden, G., Ragula, G., & Roubal, M. (2003). The use of broadband electromagnetic tech-
nology for integrity inspection of a 760 mm (30 in.) cast iron and steel line. In 22nd World
Gas Conference (pp. 1–6).
40. Fan, R. H., Xiong, B., Peng, R. W., & Wang, M. (2020). Constructing metastructures with
broadband electromagnetic functionality. Advanced Materials, 32(27), 1904646.
41. Huang, H. (2005). Depth of investigation for small broadband electromagnetic sensors.
Geophysics, 70(6), G135–G142.
42. Nguema, E., Férachou, D., Humbert, G., Auguste, J. L., & Blondy, J. M. (2011). Broadband
terahertz transmission within the air channel of thin-wall pipe. Optics letters, 36(10), 1782–
1784.
43. Corrao, A., Briones, B., VanderSchaaf, R., & Bermudo, J. E. (2015). Condition assessment
methods for 1920s lock-bar steel pipe. In Pipelines 2015 (pp. 931–942).
44. Sophian, A., Tian, G., & Fan, M. (2017). Pulsed eddy current non-destructive testing and
evaluation: A review. Chinese Journal of Mechanical Engineering, 30(3), 500–514.
45. He, Y., Tian, G., Zhang, H., Alamin, M., Simm, A., & Jackson, P. (2012). Steel corrosion
characterization using pulsed eddy current systems. IEEE Sensors Journal, 12(6), 2113–2120.
46. Vasic, D., Bilas, V., & Ambrus, D. (2004). Pulsed eddy-current nondestructive testing of
ferromagnetic tubes. IEEE Transactions on Instrumentation and Measurement, 53(4), 1289–
1294.
47. Tian, G. Y., Sophian, A., Taylor, D., & Rudlin, J. (2005). Multiple sensors on pulsed eddy-
current detection for 3-D subsurface crack assessment. IEEE Sensors Journal, 5(1), 90–96.
48. Tian, G. Y., He, Y., Adewale, I., & Simm, A. (2013). Research on spectral response of pulsed
eddy current and NDE applications. Sensors and Actuators A: Physical, 189, 313–320.
49. Wilson, J., Tian, G. Y., Abidin, I. Z., Yang, S., & Almond, D. (2010). Pulsed eddy current
thermography: System development and evaluation. Insight-Non-Destructive Testing and
Condition Monitoring, 52(2), 87–90.
50. Jol, H. M., & Smith, D. G. (1995). Ground penetrating radar surveys of peatlands for oilfield
pipelines in Canada. Journal of Applied Geophysics, 34(2), 109–123.
51. Hasan, A. E. (2012, May). The use of ground penetrating radar with a frequency 1 GHz to
detect water leaks from pipelines. In Proceedings of the 16th International Water Technology
Conference (IWTC16), Istanbul, Turkey (Vol. 710, p. 16).
52. Yang, H. W., kun Yang, Z., & Kun Pei, Y. (2014). Ground-penetrating radar for soil and
underground pipelines using the forward modeling simulation method. Optik, 125(23), 7075–
7079.
53. Li, H., Chou, C., Fan, L., Li, B., Wang, D., & Song, D. (2019). Toward automatic subsurface
pipeline mapping by fusing a ground-penetrating radar and a camera. IEEE Transactions on
Automation Science and Engineering, 17(2), 722–734.
54. Bimpas, M., Amditis, A., & Uzunoglu, N. K. (2011). Design and implementation of an
integrated high resolution imaging ground penetrating radar for water pipeline rehabilitation.
Water Resources Management, 25(4), 1239–1250.
55. Wang, Y., Jin, H., & Li, G. (2016). Investigation of the freeze–thaw states of foundation
soils in permafrost areas along the China-Russia Crude Oil Pipeline (CRCOP) route using
ground-penetrating radar (GPR). Cold Regions Science and Technology, 126, 10–21.
56. Shen, Y., Lin, Y., Li, P., Fu, Y., & Wang, Y. (2019). Simulation and detection leakage of
underground water pipeline by ground penetrating radar. Journal of Testing and Evaluation,
48(3), 2003–2027.
57. Jin, Y., & Duan, Y. (2020). Wavelet scattering network-based machine learning for ground
penetrating radar imaging: Application in pipeline identification. Remote Sensing, 12(21),
3655.
58. Sudyka, J., & Krysinski, L. (2011). Radar technique application in structural analysis and iden-
tification of interlayer bonding. International Journal of Pavement Research and Technology,
4(3), 176.
59. Sharma, P., Kumar, B., Singh, D., & Gaba, S. P. (2017). Critical analysis of background
subtraction techniques on real GPR data. Defence Science Journal, 67(5), 559–571.
60. Alharthi, A., & Lange, J. (1987). Soil water saturation: Dielectric determination. Water
Resources Research, 23(4), 591–595.
61. Miller, R. K. (1996). Acoustic emission leak detection. Final Report No. R96–472 to the New
Jersey Institute of Technology.
62. Miller, R. K., Pollock, A. A., Watts, D. J., Carlyle, J. M., Tafuri, A. N., & Yezzi, J. J., Jr.
(1999). A reference standard for the development of acoustic emission pipeline leak detection
techniques. Ndt & E International, 32(1), 1–8.
63. Juliano, T. M., Meegoda, J. N., & Watts, D. J. (2013). Acoustic emission leak detection on
a metal pipeline buried in sandy soil. Journal of Pipeline Systems Engineering and Practice,
4(3), 149–155.
64. Ahadi, M., & Bakhtiar, M. S. (2010). Leak detection in water-filled plastic pipes through
the application of tuned wavelet transforms to acoustic emission signals. Applied Acoustics,
71(7), 634–639.
References 73
65. Li, S., Song, Y., & Zhou, G. (2018). Leak detection of water distribution pipeline subject to
failure of socket joint based on acoustic emission and pattern recognition. Measurement, 115,
39–44.
66. Lukonge, A. B., & Cao, X. (2020). Leak detection system for long-distance onshore and
offshore gas pipeline using acoustic emission technology. A review. Transactions of the Indian
Institute of Metals, 73(7), 1715–1727.
67. Quy, T. B., & Kim, J. M. (2020). Leak detection in a gas pipeline using spectral portrait of
acoustic emission signals. Measurement, 152, 107403.
68. Li, C. (2021). Pipeline leakage detection and localization based on acoustic emission signal.
Master’s thesis. Xi’an University of Technology, Xi’an, China.
69. Mostafapour, A., & Davoudi, S. (2013). Analysis of leakage in high pressure pipe using
acoustic emission method. Applied Acoustics, 74(3), 335–342.
70. Banjara, N. K., Sasmal, S., & Voggu, S. (2020). Machine learning supported acoustic emission
technique for leakage detection in pipelines. International Journal of Pressure Vessels and
Piping, 188, 104243.
71. Sharif, M. A., & Grosvenor, R. I. (1998). Internal valve leakage detection using an acoustic
emission measurement system. Transactions of the Institute of Measurement and Control,
20(5), 233–242.
72. Lee, J. R., Jeong, H., Ciang, C. C., Yoon, D. J., & Lee, S. S. (2010). Application of ultrasonic
wave propagation imaging method to automatic damage visualization of nuclear power plant
pipeline. Nuclear Engineering and Design, 240(10), 3513–3520.
73. Ravanbod, H. (2005). Application of neuro-fuzzy techniques in oil pipeline ultrasonic
nondestructive testing. Ndt and E International, 38(8), 643–653.
74. Yan, Y., Liu, D., Gao, B., Tian, G. Y., & Cai, Z. C. (2020). A deep learning-based ultrasonic
pattern recognition method for inspecting girth weld cracking of gas pipeline. IEEE Sensors
Journal, 20(14), 7997–8006.
75. Gao, G., Dang, R., Nouri, A., Jia, H., Li, L., Feng, X., & Dang, B. (2015). Sand rate model
and data processing method for non-intrusive ultrasonic sand monitoring in flow pipeline.
Journal of Petroleum Science and Engineering, 134, 30–39.
76. Yin, P., Cao, X., Zhang, P., Yang, W., Bian, J., & Guo, D. (2020). Investigation of slug flow char-
acteristics in hilly terrain pipeline using ultrasonic Doppler method. Chemical Engineering
Science, 211, 115300.
77. Lee, L. H., Rajkumar, R., Lo, L. H., Wan, C. H., & Isa, D. (2013). Oil and gas pipeline
failure prediction system using long range ultrasonic transducers and euclidean-support vector
machines classification approach. Expert Systems with Applications, 40(6), 1925–1934.
78. Xu, Z. D., Zhu, C., & Shao, L. W. (2021). Damage identification of pipeline based on ultrasonic
guided wave and wavelet denoising. Journal of Pipeline Systems Engineering and Practice,
12(4), 04021051.
79. Liu, Q. H. (2019). Localization algorithm of gas leakage source based on ultrasonic virtual
phased array. Zhengzhou University.
80. Siqueira, M. H. S., Gatts, C. E. N., Da Silva, R. R., & Rebello, J. M. A. (2004). The use
of ultrasonic guided waves and wavelets analysis in pipe inspection. Ultrasonics, 41(10),
785–797.
81. Chen, Y., Chen, Y., Hu, S., & Ni, Z. (2021). Continuous ultrasonic flow measurement for
aerospace small pipelines. Ultrasonics, 109, 106260.
82. Du, D., Wang, G., & Mao, L. (2021). Analysis of acoustic field characteristics to detect internal
pipeline corrosion based on ultrasonic full-focus. Measurement Science and Technology,
33(1), 015007.
83. Mao, Y. M., & Que, P. W. (2006). Application of Hilbert-Huang signal processing to ultrasonic
non-destructive testing of oil pipelines. Journal of Zhejiang University-Science A, 7(2), 130–
134.
84. Ghavamian, A., Mustapha, F., Baharudin, B. T., & Yidris, N. (2018). Detection, localisation
and assessment of defects in pipes using guided wave techniques: A review. Sensors, 18(12),
4470.
85. Guan, R., Lu, Y., Duan, W., & Wang, X. (2017). Guided waves for damage identification in
pipeline structures: A review. Structural Control and Health Monitoring, 24(11), e2007.
86. Cheong, Y. M., Lee, D. H., & Jung, H. K. (2004). Ultrasonic guided wave parameters for
detection of axial cracks in feeder pipes of PHWR nuclear power plants. Ultrasonics, 42(1–9),
883–888.
87. Gazis, D. C. (1959). Three-dimensional investigation of the propagation of waves in hollow
circular cylinders. I. Analytical foundation. The Journal of the Acoustical Society of America,
31(5), 568–573.
88. Gazis, D. C. (1959). Three-dimensional investigation of the propagation of waves in hollow
circular cylinders. II. Numerical results. The Journal of the Acoustical Society of America,
31(5), 573–578.
89. Zhang, L., Luo, W., & Rose, J. L. (2006, March). Ultrasonic guided wave focusing beyond
welds in a pipeline. In AIP Conference Proceedings (Vol. 820, No. 1, pp. 877–884). American
Institute of Physics.
90. Wang, X., Peter, W. T., & Dordjevich, A. (2010). Evaluation of pipeline defect’s characteristic
axial length via model-based parameter estimation in ultrasonic guided wave-based inspection.
Measurement Science and Technology, 22(2), 025701.
91. Eybpoosh, M., Berges, M., & Noh, H. Y. (2017). An energy-based sparse representation of
ultrasonic guided-waves for online damage detection of pipelines under varying environmental
and operational conditions. Mechanical Systems and Signal Processing, 82, 260–278.
92. Muñoz, C. Q. G., Marquez, F. P. G., Lev, B., & Arcos, A. (2017). New pipe notch detection
and location method for short distances employing ultrasonic guided waves. Acta Acustica
united with Acustica, 103(5), 772–781.
93. Eybpoosh, M., Berges, M., & Noh, H. Y. (2016). Sparse representation of ultrasonic guided-
waves for robust damage detection in pipelines under varying environmental and operational
conditions. Structural Control and Health Monitoring, 23(2), 369–391.
94. Carboni, M., Gianneo, A., & Giglio, M. (2015). A Lamb waves based statistical approach to
structural health monitoring of carbon fibre reinforced polymer composites. Ultrasonics, 60,
51–64.
95. Liu, Z., He, C., Wu, B., Wang, X., & Yang, S. (2006). Circumferential and longitudinal
defect detection using T (0, 1) mode excited by thickness shear mode piezoelectric elements.
Ultrasonics, 44, e1135–e1138.
96. Lais, H., Lowe, P. S., Gan, T. H., Wrobel, L. C., & Kanfoud, J. (2018). Characterization of
the use of low frequency ultrasonic guided waves to detect fouling deposition in pipelines.
Sensors, 18(7), 2122.
97. Verma, B., Mishra, T. K., Balasubramaniam, K., & Rajagopal, P. (2014). Interaction of low-
frequency axisymmetric ultrasonic guided waves with bends in pipes of arbitrary bend angle
and general bend radius. Ultrasonics, 54(3), 801–808.
98. Wu, J., Wang, Y., Zhang, W., Nie, Z., Lin, R., & Ma, H. (2017). Defect detection of pipes using
Lyapunov dimension of Duffing oscillator based on ultrasonic guided waves. Mechanical
Systems and Signal Processing, 82, 130–147.
99. Wang, Z., Huang, S., Wang, S., Zhuang, S., Wang, Q., & Zhao, W. (2019). Compressed
sensing method for health monitoring of pipelines based on guided wave inspection. IEEE
Transactions on Instrumentation and Measurement, 69(7), 4722–4731.
100. Mu, J., Zhang, L., & Rose, J. L. (2007). Defect circumferential sizing by using long range
ultrasonic guided wave focusing techniques in pipe. Nondestructive Testing and Evaluation,
22(4), 239–253.
101. Berlincourt, & Don. (1980). Piezoelectric ceramics: characteristics and applications. Journal
of the Acoustical Society of America, 70(6), 1586–1595.
102. Sack, D. A., & Olson, L. D. (1998). Impact echo testing of in-situ precast concrete cylinder
pipe. In J. P. Castronovo & J. A. Clark (Eds.), Pipelines in the Constructed Environment
(pp. 250–259). American Society of Civil Engineers, San Diego, CA.
103. Madryas, C., Moczko, A. T., & Wysocki, L. (2009). Utilizing the Impact-Echo method for
nondestructive diagnostics of atypically located pipeline. In Proceedings of International
Conference on Underground Infrastructure of Urban Areas, Wrocław (pp. 183–192).
References 75
104. Mehdinia, S., Schumacher, T., Song, X., & Wan, E. (2021). A pipeline for enhanced
multimodal 2D imaging of concrete structures. Materials and Structures, 54(6), 1–16.
105. Chiang, C. H., Cheng, C. C., & Hsu, K. T. (2013). Inspection of deteriorated coastal embank-
ments using radar, thermography, and impact-Echo. In Nondestructive Testing of Materials
and Structures (pp. 927–933). Springer, Dordrecht.
106. Datta, S., & Sarkar, S. (2016). A review on different pipeline fault detection methods. Journal
of Loss Prevention in the Process Industries, 41, 97–106.
107. Fletcher, R., & Chandrasekaran, M. (2008, January). SmartBall™: a new approach in pipeline
leak detection. In International Pipeline Conference (Vol. 48586, pp. 117–133).
108. Ariaratnam, S. T., & Chandrasekaran, M. (2010). Development of an innovative free-
swimming device for detection of leaks in oil and gas pipelines. In Construction Research
Congress 2010: Innovation for Reshaping Construction Practice (pp. 588–596).
109. Po, A., & Xing, Y. (2011). Internal free swimming pipeline leakage detection technology—
Smartball. In ICPTT 2011: Sustainable Solutions for Water, Sewer, Gas, and Oil Pipelines
(pp. 996–1005).
110. Ogai, H., & Bhattacharya, B. (2018). Pipe inspection robots for gas and oil pipelines. In Pipe
Inspection Robots for Structural Health and Condition Monitoring (pp. 13–43). Springer.
111. Murray, D., Carroll, C., & Higgins, M. (2009). Evaluating in-service force mains with air
pocket and leak detection technology. In Pipelines 2009: Infrastructure’s Hidden Assets
(pp. 499–507).
112. Wang, F., Lin, W., Liu, Z., Kong, L., & Qiu, X. (2020, May). Non-intrusive leak monitoring
system for pipeline within a closed space by wireless sensor network. In 2020 IEEE Wireless
Communications and Networking Conference (WCNC) (pp. 1–6). IEEE.
113. Paulson, P., & Nguyen, V. (2010). Pipe wall evaluation using acoustic pulses. In Pipelines
2010: Climbing New Peaks to Infrastructure Reliability: Renew, Rehab, and Reinvest (pp. 711–
718).
114. Zaman, D., Tiwari, M. K., Gupta, A. K., & Sen, D. (2020). A review of leakage detection
strategies for pressurised pipeline in steady-state. Engineering Failure Analysis, 109, 104264.
115. Livingston, B., Malach, W. J., & Duchene, C. (2019). A comprehensive condition assess-
ment of a critical water main. pipelines 2019: Condition assessment, Construction, and
Rehabilitation, pp 68–83.
116. Xu, T., Chen, S., Guo, S., Huang, X., Li, J., & Zeng, Z. (2019). A small leakage detection
approach for oil pipeline using an inner spherical ball. Process Safety and Environmental
Protection, 124, 279–289.
117. Shili, C., Dongxiang, W., Shixu, G., et al. (2016). Tracking and localization technology of
spherical pipeline internal detector. Nanotechnology and Precision Engineering, 14(2), 87–93.
118. Zhang, Y., Zhang, H., Liu, J., Zhang, S., Liu, Z., Lyu, E., & Chen, W. (2022). Submarine
pipeline tracking technology based on AUVs with forward looking sonar. Applied Ocean
Research, 122, 103128.
119. Chapman, P., Wills, D., Brookes, G., & Stevens, P. (1999). Visualizing underwater envi-
ronments using multifrequency sonar. IEEE Computer Graphics and Applications, 19(5),
61–65.
120. Tian, W. M. (2008). Integrated method for the detection and location of underwater pipelines.
Applied Acoustics, 69(5), 387–398.
121. Nadimi, N., Javidan, R., & Layeghi, K. (2021). Efficient detection of underwater natural gas
pipeline leak based on synthetic aperture sonar (SAS) systems. Journal of Marine Science
and Engineering, 9(11), 1273.
122. Kasetkasem, T., Tipsuwan, Y., Tulsook, S., Muangkasem, A., Leangaramkul, A., &
Hoonsuwan, P. (2020). A pipeline extraction algorithm for forward-looking sonar images
using the self-organizing map. IEEE Journal of Oceanic Engineering, 46(1), 206–220.
123. Daneti, M. (2008, September). A practical preprocessing treatment for pipeline leak locating
improving. In 2008 IEEE International Conference on Emerging Technologies and Factory
Automation (pp. 9–12). IEEE.
124. Daneti, M. (2008, December). On using phase data information for pipeline leak location. In
2008 IEEE 25th Convention of Electrical and Electronics Engineers in Israel (pp. 494–498).
IEEE.
125. Daneti, M. (2010, November). Taking steps in understanding multipath propagation in fluid
filled pipelines. In 2010 9th International Symposium on Electronics and Telecommunications
(pp. 399–402). IEEE.
126. Hunaidi, O., Wang, A., Bracken, M., Gambino, T., & Fricke, C. (2004, May). Acoustic methods
for locating leaks in municipal water pipe networks. In International conference on water
demand management (pp. 1–14). Jordan: Dead Sea.
127. Kumar, S. S., Abraham, D. M., Behbahani, S. S., Matthews, J. C., & Iseley, T. (2020). Compar-
ison of technologies for condition assessment of small-diameter ductile iron water pipes.
Journal of Pipeline Systems Engineering and Practice, 11(4), 04020039.
128. Fahmy, M., & Moselhi, O. (2009, June). Detecting and locating leaks in underground
water mains using thermography. In Proceedings of the 26th International Symposium on
Automation and Robotics in Construction (ISARC 2009), Austin, TX, USA (pp. 24–27)
129. Fahmy, M., & Moselhi, O. (2010). Automated detection and location of leaks in water mains
using infrared photography. Journal of Performance of Constructed Facilities, 24(3), 242–248.
130. Costello, S. B., Chapman, D. N., Rogers, C. D. F., & Metje, N. (2007). Underground asset loca-
tion and condition assessment technologies. Tunnelling and Underground Space Technology,
22(5–6), 524–542.
131. Mergelas, B., & Henrich, G. (2005). Leak locating method for precommissioned transmission
pipelines: North American case studies. Leakage, 2005, 12–14.
132. Webb, M. C. (2008). Precise leak detection inspection: Technology. Water & Sanitation Africa,
3(1), 53–57.
133. Laven, K., Amyot, C., Knight, M., Liew, P., & Jones, C. (2007). Leak Detection on wastewater
forcemains and siphons in North America using the Sahara® acoustic system. In Pipelines
2007: Advances and Experiences with Trenchless Pipeline Projects (pp. 1–10).
134. Derr, H. R., & Hubbard, P. (2009). In-service inspection of wastewater force mains. In
Pipelines 2009: Infrastructure’s Hidden Assets (pp. 249–260).
135. Kumar, G. A., Patil, A. K., Patil, R., Park, S. S., & Chai, Y. H. (2017). A LiDAR and
IMU integrated indoor navigation system for UAVs and its application in real-time pipeline
classification. Sensors, 17(6), 1268.
136. Li, Y., & Ibanez-Guzman, J. (2020). Lidar for autonomous driving: The principles, challenges,
and trends for automotive lidar and perception systems. IEEE Signal Processing Magazine,
37(4), 50–61.
137. Minato, A., Joarder, M. M. A., Ozawa, S., Kadoya, M., & Sugimoto, N. (1999). Development
of a lidar system for measuring methane using a gas correlation method. Japanese Journal of
Applied Physics, 38(10R), 6130.
138. Murdock, D. G., Stearns, S. V., Lines, R. T., Lenz, D., Brown, D. M., & Philbrick, C. R. (2008).
Applications of real-world gas detection: Airborne Natural Gas Emission Lidar (ANGEL)
system. Journal of Applied Remote Sensing, 2(1), 023518.
139. Thorpe, M. J., Kreitinger, A., Seger, E., Greenfield, N., Wilson, C., Trey, P., Kreitinger,
S., Gordon, S., Schmitt, R., & Roos, P. (2017, May). Gas Mapping LiDAR for large-area
leak detection and emissions monitoring applications. In 2017 Conference on Lasers and
Electro-Optics (CLEO) (pp. 1–2). IEEE.
140. Bartholomew, J., Lyman, P., Weimer, C., & Tandy, W. (2017, August). Wide area methane
emissions mapping with airborne IPDA lidar. In Lidar Remote Sensing for Environmental
Monitoring 2017 (Vol. 10406, pp. 34–47). SPIE.
141. Tan, Y., Li, S., & Wang, Q. (2020). Automated geometric quality inspection of prefabricated
housing units using BIM and LiDAR. Remote Sensing, 12(15), 2492.
142. Ékes, C. (2017). New Developments in Multi-Sensor Condition Assessment Using LiDAR,
Sonar, and CCTV. In Pipelines 2017 (pp. 23–29).
143. Iseki, T., Tai, H., & Kimura, K. (2000). A portable remote methane sensor using a tunable
diode laser. Measurement Science and Technology, 11(6), 594.
References 77
144. Jiang, Y., Cai, T., & Zhu, Y. (2011). Methane monitoring system based on absorption spec-
troscopy of tunable diode laser. Journal of Electronic Measurement and Instrument, 25,
265–271.
145. Hao, X., Xu, J., & Li, H. (2016). Summary of trace leakage detection technology for natural
gas transmission pipeline based on TDLAS. Journal of North China Institute of Science and
Technology, 13, 60–64.
146. Lackner, M. (2007). Tunable diode laser absorption spectroscopy (TDLAS) in the process
industries—A review. Reviews in Chemical Engineering, 23(2), 65–147.
147. Werle, P. O., Mücke, R., & Slemr, F. (1993). The limits of signal averaging in atmospheric
trace-gas monitoring by tunable diode-laser absorption spectroscopy (TDLAS). Applied
Physics B, 57(2), 131–139.
148. Dong, L., Tittel, F. K., Li, C., Sanchez, N. P., Wu, H., Zheng, C., Yu, Y., Sampaolo, A., &
Griffin, R. J. (2016). Compact TDLAS based sensor design using interband cascade lasers
for mid-IR trace gas sensing. Optics express, 24(6), A528–A535.
149. Wagner, S., Fisher, B. T., Fleming, J. W., & Ebert, V. (2009). TDLAS-based in situ measure-
ment of absolute acetylene concentrations in laminar 2D diffusion flames. Proceedings of the
Combustion Institute, 32(1), 839–846.
150. Liu, C., Xu, L., Chen, J., Cao, Z., Lin, Y., & Cai, W. (2015). Development of a fan-beam
TDLAS-based tomographic sensor for rapid imaging of temperature and gas concentration.
Optics Express, 23(17), 22494–22511.
151. Nwaboh, J. A., Werhahn, O., Ortwein, P., Schiel, D., & Ebert, V. (2012). Laser-spectrometric
gas analysis: CO2 –TDLAS at 2 µm. Measurement Science and Technology, 24(1), 015202.
152. Bao, Y., Zhang, R., Enemali, G., Cao, Z., Zhou, B., McCann, H., & Liu, C. (2020). Relative
entropy regularized TDLAS tomography for robust temperature imaging. IEEE Transactions
on Instrumentation and Measurement, 70, 1–9.
153. Meng, Y., Liu, T., Liu, K., Jiang, J., Wang, R., Wang, T., & Hu, H. (2014). A modified empirical
mode decomposition algorithm in TDLAS for gas detection. IEEE Photonics Journal, 6(6),
1–7.
154. Liu, C., Cao, Z., Lin, Y., Xu, L., & McCann, H. (2018). Online cross-sectional monitoring
of a swirling flame using TDLAS tomography. IEEE Transactions on Instrumentation and
Measurement, 67(6), 1338–1348.
155. Zhu, X., Yao, S., Ren, W., Lu, Z., & Li, Z. (2019). TDLAS monitoring of carbon dioxide with
temperature compensation in power plant exhausts. Applied Sciences, 9(3), 442.
156. Xu, J. D., Hao, X., Li, H., & Yang, G. Y. (2017). Theoretical probe on the TDLAS detection
technology for minor leakage of natural gas transmission pipeline. Journal of Safety and
Environment, 17(6), 2195–2199.
157. Weil, G. J. (1993, May). Non contact, remote sensing of buried water pipeline leaks using
infrared thermography. In Water Management in the 90s: A Time for Innovation (pp. 404–407).
ASCE.
158. Möllmann, K. P., & Vollmer, M. (2017). Infrared thermal imaging: Fundamentals, research
and applications. Wiley.
159. Li, J. K., Jin, W. Q., Wang, X., Jin, M., Dun, X., & Chen, J. (2014). Review of gas leak infrared
imaging detection technology. Infrared Technology, 36(7), 513–520.
160. Soldan, S., Welle, J., Barz, T., Kroll, A., & Schulz, D. (2014). Towards autonomous robotic
systems for remote gas leak detection and localization in industrial environments. In Field
and Service Robotics (pp. 233–247). Springer.
161. Yanbo, Z., Ruifeng, R., Peng, L., Xulong, Y., & Shuai, W. (2020). Experimental study on
flaw detection of buried heat pipeline based by infrared thermal. Chinese Journal of Scientific
Instrument, 41(6), 161–170.
162. Janssens, O., Schulz, R., Slavkovikj, V., Stockman, K., Loccufier, M., Van de Walle, R., &
Van Hoecke, S. (2015). Thermal image based fault diagnosis for rotating machinery. Infrared
Physics and Technology, 73, 78–87.
163. Tong, K., Wang, Z., Si, L., Tan, C., & Li, P. (2020). A novel pipeline leak recognition method of
mine air compressor based on infrared thermal image using IFA and SVM. Applied Sciences,
10(17), 5991.
164. Thusyanthan, I., Blower, T., & Cleverly, W. (2016, November). Innovative uses of thermal
imaging in civil engineering. In Proceedings of the Institution of Civil Engineers-Civil
Engineering (Vol. 170, No. 2, pp. 81–87). Thomas Telford Ltd.
165. Janssens, O., Loccufier, M., & Van Hoecke, S. (2018). Thermal imaging and vibration-
based multisensor fault detection for rotating machinery. IEEE transactions on industrial
informatics, 15(1), 434–444.
166. Jadin, M. S., & Ghazali, K. H. (2014, March). Gas leakage detection using thermal imaging
technique. In 2014 UKSim-AMSS 16th International Conference on Computer Modelling and
Simulation (pp. 302–306). IEEE.
167. Li, Y., Zhang, Y., & Fan, Y. (2018, May). A quantitative analysis method for pipeline
defect inspection based on infrared thermal imaging. In 2018 2nd IEEE Advanced Informa-
tion Management, Communicates, Electronic and Automation Control Conference (IMCEC)
(pp. 430–434). IEEE.
168. Janssens, O., Loccufier, M., Van de Walle, R., & Van Hoecke, S. (2017). Data-driven imbalance
and hard particle detection in rotating machinery using infrared thermal imaging. Infrared
Physics and Technology, 82, 28–39.
169. Adefila, K., Yan, Y., & Wang, T. (2015, May). Leakage detection of gaseous CO2 through
thermal imaging. In 2015 IEEE International Instrumentation and Measurement Technology
Conference (I2MTC) Proceedings (pp. 261–265). IEEE.
170. Saragadam, V., Dave, A., Veeraraghavan, A., & Baraniuk, R. G. (2021). Thermal image
processing via physics-inspired deep networks. In Proceedings of the IEEE/CVF International
Conference on Computer Vision (pp. 4057–4065).
171. Zhang, L., Shi, Q. T., Li, G., Guo, K., Xue, X. Q., & Zhou, J. X. (2022). Research on
application of automatic pipeline inspection technology based on thermal image recognition.
Petrochemical Industry Technology, 2, 124–125.
172. Sousa, M. J., Moutinho, A., & Almeida, M. (2020). Thermal infrared sensing for near real-time
data-driven fire detection and monitoring systems. Sensors, 20(23), 6803.
173. Althouse, M. L., & Chang, C. I. (1995, February). Chemical vapor detection and mapping with
a multispectral forward-looking infrared (FLIR). In Optical Instrumentation for Gas Emis-
sions Monitoring and Atmospheric Measurements (Vol. 2366, pp. 108–114). International
Society for Optics and Photonics.
174. Senthilkumar, T., Jayas, D. S., & White, N. D. G. (2015). Detection of different stages of
fungal infection in stored canola using near-infrared hyperspectral imaging. Journal of Stored
Products Research, 63, 80–88.
175. Bennett, C. L., Carter, M. R., & Fields, D. J. (1995, September). Hyperspectral imaging in the
infrared using LIFTIRS. In Infrared Technology XXI (Vol. 2552, pp. 274–283). International
Society for Optics and Photonics.
176. Marinelli, W. J., & Green, B. D. (1996). Infrared imaging volatile organic carbon field sensor.
Optical Remote Sensing for Environmental and Process Monitoring, 2883, 245.
177. Smith, B. W., Laubscher, B. E., Cooke, B. J., LaDelfe, P. C., Harlander, J. M., Howard, J. W., &
Milligan, S. (1999, July). IRISHS: The infrared imaging spatial heterodyne spectrometer: a
new pushbroom Fourier transform ultraspectral imager with no moving parts. In Infrared
Technology and Applications XXV (Vol. 3698, pp. 501–509). SPIE.
178. Arai, K., & Yoshida, H. (2014). Primary flow meter for calibrating a sniffer test leak artefact
by a pressure rise method.Metrologia, 51(5), 522.
179. Mount, D. J. (2014). Trends in Leak Testing. Quality, 53(3), 25.
180. Erizaldy, A., & Nurhajati, N. (2017). Leak detection on water distribution networks using
helium gas. Civil and Environmental Research, 9(11), 20–29.
181. Itoh, A., Oka, K., Kakudate, S., Obara, K., Taguchi, K., Tada, E., Tesini, A., Shibanuma, K., &
Haange, R. (1997, October). Development of bore tools for blanket cooling pipe connection
in ITER. In 17th IEEE/NPSS Symposium Fusion Engineering (Cat. No. 97CH36131) (Vol. 2,
pp. 921–924). IEEE.
182. Ishikawa, T., Morimoto, T., Yokozeki, T., Morino, Y., & Aoki, T. (2004). Pressurization of
CF/Epoxy model tank at LN2 temperature and identification of leak path formation in tank
References 79
wall. In 45th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials

Conference (p. 1839).
183. Sakiguchi, S., Kawano, K., Hiyama, T., Imahashi, K., Okayama, J., Kawasaki, T., & Tsuji,
H. (1995). Helium leak test of extremely low temperature testing facility (No. KEK-PROC--
95–2).
184. Michalski, R. (2015). Everything leaks: Leak detection is an art form that requires, in some
cases, special equipment, training and patience. Quality, 54(10), 12NDT–12NDT.
185. Thompson, G., & Golding, R. (1993). Pipeline leak detection using volatile tracers. Leak
Detection for Underground Storage Tanks, 1161, 131–138.
186. McHugh, T., Beckley, L., Sullivan, T., Lutes, C., Truesdale, R., Uppencamp, R., Cosky, B.,
Zimmerman, J., & Schumacher, B. (2017). Evidence of a sewer vapor transport pathway at
the USEPA vapor intrusion research duplex. Science of the Total Environment, 598, 772–779.
187. Sheltami, T. R., Bala, A., & Shakshuki, E. M. (2016). Wireless sensor networks for leak
detection in pipelines: A survey. Journal of Ambient Intelligence and Humanized Computing,
7(3), 347–356.
188. Sun, J., Xiao, Q., Wen, J., & Zhang, Y. (2016). Natural gas pipeline leak aperture identification
and location based on local mean decomposition analysis. Measurement, 79, 147–157.
189. Wu, J., Li, H., Wang, L., Xu, B., Wang, H., Cai, Y., Liu, S., & Tao, Z. (2017). Performance eval-
uation indices of pipeline leak monitoring systems. Oil and Gas Storage and Transportation,
36, 209–213.
190. Puust, R., Kapelan, Z., Savic, D. A., & Koppel, T. (2010). A review of methods for leakage
management in pipe networks. Urban Water Journal, 7(1), 25–45.
191. Misiūnas, D. (2008). Failure monitoring and asset condition asssessment in water supply
systems. Vilniaus Gedimino technikos universitetas.
Chapter 3
Pipeline Health Monitoring Technology
3.1 Introduction
Different from pipeline inspection, monitoring refers to real-time monitoring of the

same object for a long time and mastering its change law. In addition, monitoring
has certain timeliness, primarily referring to on-site sampling and monitoring. In
pipeline engineering, monitoring usually needs to rely on the long-term installation
of sensing equipment on the pipeline and real-time data collection to identify the
health status of the pipeline. This chapter introduces common pipeline monitoring
methods, including software-based and hardware-based methods [1].
3.2 Optical Fiber Sensing
The working principle of distributed optical fiber sensing technology is to use optical
fiber sensing to receive a signal and send it out and back in real-time (see Fig. 3.1)
[2–7]. Compared with the traditional nodal optical fiber sensor, the distributed optical
fiber sensing technology can measure the spatial parameters and their corresponding
variables of thousands of scattering points on the forward optical fiber in real-time to
achieve distributed measurement. Through the specific calculation form of contin-
uous function, relatively accurate measurement is carried out for the real-time moni-
toring of each node position on the surface of the entire optical fiber base. Because
this technology can obtain the continuous distribution of measured information in
space and time, it is very suitable for large-scale engineering structure monitoring,
long-distance pipeline monitoring, large-scale regional security, and other fields. At
first, optical fiber sensing technology was mainly used in bridges and tunnels. Later,
the technology began to be used in pipeline safety monitoring. Figure 3.2 shows a
typical layout of optical fibers for monitoring pipelines.
https://doi.org/10.1007/978-981-19-6798-6_3
82 3 Pipeline Health Monitoring Technology
Power
Monitoring station
Supply
Ultra narrow
Acoustooptic EDFA
lineweight Circulator
modulator amplifier
emitter
Signal Data acquisition Photodet EDFA

generator and storage module ector amplifier
Sensing optical cable for

equipment along the line
Pipeline
Fig. 3.1 Principle of distributed optical fiber monitoring [8]. Note EDFA denotes erbium-doped
optical fiber amplifier
Optical fiber
Mileage stake identification
Backfill top line
Common backfill
area
Cavity compaction
backfill area
Optical fiber
Sand gravel cushion
Fig. 3.2 A typical layout of optical fibers for monitoring pipelines [8]
3.2 Optical Fiber Sensing 83
The widely used distributed fiber sensing methods include optical time domain
reflection (OTDR), interferometric optical fiber sensor, and fiber Bragg grating
(FBG).
3.2.1 Optical Time Domain Reflection (OTDR)
The basic principle of optical time domain reflection technology is that the light
emitted by the light source produces backscattering in the forward transmission along
the optical fiber, and the backscattering light intensity attenuates with the increase
of distance. Distance is proportional to time when the speed of light is constant.
Therefore, according to the backscattered light intensity detected by the detector and
the time when it reaches the detector, the initial backscattered light intensity at any
point along the fiber path can be known to carry out leakage inspection [9].
The backscattering of light includes Rayleigh scattering, Raman scattering, and
Brillouin scattering [10–17]. Among them, Rayleigh scattering is caused by the
inhomogeneity of the refractive index of the optical fiber caused by the inhomogeneity
of the optical fiber material. It is an elastic scattering between light and matter,
and there is no frequency shift in the scattering process. Raman scattering is an
optical phonon produced by the interaction between light and the vibration of matter
molecules. Brillouin scattering is an acoustic phonon produced by the periodic change
of optical fiber material density. Both Raman scattering and Brillouin scattering are
inelastic. It is found that Rayleigh scattering does not produce a frequency shift,
but the intensity of Rayleigh scattering is the largest. Both Brillouin scattering and
Raman scattering have frequency shifts. The frequency shift of Brillouin scattering
is smaller than that of Raman scattering, but the intensity of Brillouin scattering is
greater than that of Raman scattering. The part of the spectrum whose frequency is
lower than that of the incident light is called anti-Stokes light, and the part whose
frequency is greater than that of the incident light is called Stokes light. Figure 3.3
shows the analysis of backscattered light.
In the backscattering of light, the intensity of Rayleigh scattering is relatively
large. When the external physical field disturbs the optical fiber, the intensity of
the backscattering light here will decrease significantly. Therefore, the change of
backscattered light intensity can be measured to detect whether the fiber is disturbed.
Brillouin and Raman backscattered light intensity is weak, so it is not suitable to
measure the light intensity change, but its frequency shift is related to temperature
and strain. Therefore, the temperature or strain of the external physical field can be
detected by measuring the frequency shift change.
3.2.1.1 Distributed Optical Fiber Sensor Based on Rayleigh Scattering
When a leak occurs somewhere in the pipeline, the optical fiber is disturbed, which
weakens the backscattered light intensity there. The leak point can be located
Rayleigh scattering
Anti-
Stokes stokes
Brillouin scattering
Light intensity
Raman scattering
Raman scattering
Frequency
Fig. 3.3 Backscattered light [18, 19]
according to the backscattered light intensity detected by the detector and the time
it reaches the detector. This method is based on weak Rayleigh scattering [20].
However, the structure of this system is complex, and it takes a long time to obtain
a high signal-to-noise ratio, and the test accuracy is not high. In order to improve
this inspection technology, the sensing fiber can be made into a unique structure. For
example, an optical fiber can be attached to a specific polymer. When the leaking
fluid acts on the polymer, it expands, causing the fiber to bend with the polymer and
incur bend losses, thereby reducing the backscattered light intensity there. Figure 3.4
shows a typical sensing system based on Rayleigh scattering.
Fig. 3.4 Sensing system Detection

based on Rayleigh scattering laser
processor
[18]
Rayleigh Optical coupler

scattering
Measured physical field

3.2.1.2 Distributed Optical Fiber Sensor Based on Brillouin Scattering
When light passes through the fiber, photons, and phonons generated by self-heating
motion in the fiber will inelastically collide, resulting in spontaneous Brillouin scat-
tering. The frequency of the scattered light changes relative to the frequency of the
incident light, and the magnitude of this change is related to the scattering angle
and the material properties of the fiber. The fiber material properties related to
the frequency of Brillouin scattered light are mainly affected by temperature and
strain. Therefore, distributed temperature and strain measurement can be achieved
by measuring the frequency shift of the back Brillouin scattering light of the pulsed
light [21, 22]. Figure 3.5 is a typical structure of a distributed optical fiber tempera-
ture/strain sensor based on spontaneous Brillouin scattering. The laser pulse gener-
ator emits a beam of pulsed light, which is injected into the sensing fiber through
the fiber coupler. When the pulsed light is transmitted in the fiber, the spontaneous
Brillouin backscattered light can be detected at the pulsed light-emitting end of the
fiber, and the time delay between the backscattered light and the pulsed light provides
a measurement of the position information of the fiber.
The stimulated Brillouin scattering technique was first proposed by Horiguchi
et al. [24]. The typical structure of a distributed optical fiber sensor based on this
technique is shown in Fig. 3.6.
The tunable lasers at both ends of the fiber inject a pulsed light (pump light) and a
continuous light (probe light) into the sensing fiber. When the frequency shift is equal,
the Brillouin amplification effect will be produced in this region (when the two beams
of pump light propagating in the opposite direction in the fiber, and the frequency
Laser pulse generator
Signal detection unit coupler
Sensing fiber
Fig. 3.5 Distributed fiber optic sensor structure based on spontaneous Brillouin scattering [18, 23]
Fig. 3.6 Distributed fiber

optic sensor structure based
on stimulated Brillouin Pump light
scattering [18, 23]
Signal detection unit coupler
Sensing fiber
Detection laser
difference between the two is equal to the Brillouin frequency shift, the weaker
pump signal will be amplified by the stronger one. The pump signal is amplified,
called Brillouin stimulated amplification), and the energy transfer occurs between
the two beams of light. Since the Brillouin frequency shift has a linear relationship
with temperature and strain, the energy on each small section of the fiber can be
determined by detecting the power of the continuous light coupled from one end
of the fiber while continuously adjusting the frequencies of the two laser beams.
The frequency difference corresponding to the maximum transfer can be obtained to
obtain temperature and strain information and realize distributed measurement [25].
However, the main problem with the stimulated Brillouin scattering and spon-
taneous Brillouin scattering technology is that the frequency stabilization of the
laser requires a high light source and control system. Since the Brillouin frequency
shift is sensitive to temperature and strain changes, it is necessary to distinguish the
frequency shift caused by temperature and strain.
3.2.1.3 Distributed Optical Fiber Sensor Based on Raman Scattering
A laser pulse is injected into the fiber, which produces Raman scattering due to
inelastic scattering. Among them, the back Raman scattering includes anti-Stokes
light with a frequency lower than the incident light frequency and Stokes light with
a frequency higher than the incident light frequency due to the conversion between
molecular energy levels. The anti-Stokes light intensity is stronger than the Stokes
light and is more sensitive to temperature. When the fiber temperature increases,
the backscattered light intensity also increases. In the distributed optical fiber sensor
based on Raman scattering, the light pulse from the semiconductor laser is coupled
into the sensing fiber through the coupler, the scattered light after temperature modu-
lation enters the beam splitter through the coupler, and then the Stokes scattering
is filtered out respectively. The light and anti-Stokes scattered light are converted
into electrical signals through a two-way photoelectric converter, and then the elec-
trical signals are amplified and input into the computer. After data processing, it is
calibrated as temperature [26].
Although the distributed optical fiber temperature sensing technology based on
Raman scattering is relatively mature and widely used in many fields, it still has
some limitations: the working wavelength of Raman scattering is about 900 nm,
and the attenuation of this wavelength segment of optical fiber is relatively large.
The Raman scattering light intensity is inherently weak, so the sensing distance is
relatively short.
3.2.2 Fiber Bragg Grating (FBG)
Fiber grating has been one of the fastest-growing passive devices in recent years. Fiber
gratings are optical waveguides whose core refractive index changes periodically by
utilizing the photosensitivity of fiber materials. It is a wavelength-selective reflector
and the wavelength of the reflected signal changes due to temperature and strain.
This change in reflected wavelength is called wavelength shift. Many other physical
quantities can be sensed by using the two effects of temperature and strain of fiber
grating (fiber grating as a sensitive element) [27–29].
According to the Bragg diffraction principle, when light beams with multiple
wavelengths are transmitted to the grating through the optical fiber, the grating will
selectively reflect the light of a specific wavelength (Bragg wavelength λ B ). The
research shows that the change of the physical quantity at the grating has a good
repeatability and linearity with the change of the wavelength λ B at the center of the
fiber grating. In this way, the change in the physical quantity of the damaged part
can be determined by monitoring the change value of the reflected λ B , to form the
online monitoring of the damaged parts. Figure 3.7 is the schematic diagram of fiber
grating [30–34].
The fiber grating quasi-distributed sensing system mainly consists of a fiber
grating sensor array, a fiber grating sensor network analyzer, and a data processing
Ultraviolet interference pattern
Optical fiber
Incidence
Transmission
Reflection
Grating Cladding Fiber core
Fig. 3.7 Schematic diagram of fiber Bragg grating [35]
center. The key technologies in the entire quasi-distributed sensing system are the
engineering technology of the sensor, the demodulation technology of the fiber
grating sensor network analyzer, and the preparation and packaging technology of
the fiber sensor head.
A quasi-distributed temperature sensing system is formed by arranging several
fiber Bragg grating temperature sensors with different periods into an array according
to specific rules. The continuous broadband light from the light source is transmitted
to the fiber Bragg grating temperature sensor through an optical cable. The fiber
Bragg grating selectively sends back a corresponding narrow-band light to the wide-
band light and returns to the optical detector through the same transmission optical
cable. The central wavelength of different narrow-band light returned by each sensor
is measured through the subsequent receiving system, and the fiber Bragg grating
network analyzer analyzes the measured values. Because the central wavelength
range of narrow-band optical signals returned by multiple sensors is different, these
sensors can be connected in series to realize multi-point simultaneous measurement,
which significantly simplifies the layout of sensors and outgoing lines and avoids the
inconvenience of point-by-point measurement in the past.
The fiber Bragg grating sensor is a wavelength-modulated sensor. Its demodula-
tion methods include spectral analysis, wavelength scanning, optical filtering, and
coherent correlation. The fiber Bragg grating sensing system uses optical signals
for measurement and transmission and realizes nonelectric on-site inspection. At the
same time, the system uses grating technology, and the inspection signal is character-
ized by the central wavelength of the optical signal, which overcomes the shortcom-
ings of the traditional optical sensor that depends on the light intensity, and realizes
the digital inspection with good stability and long service life.
3.2.3 Interferometric Optical Fiber Sensor
Interferometric optical fiber sensing technology utilizes optical fibers to be sensed

by the physical field being monitored, such as temperature, rotation, pressure, or
vibration. The principle is to delay the phase of the light guide and change the
intensity of the output light through the change of the phase to know the change
of the measured physical field. Interferometry applied to leak inspection mainly
includes Sagnac interference technology, Mach-Zehnder interference technology,
and all-fiber white light interference technology.
3.2.3.1 Leak Inspection Technology Based on Sagnac Fiber

Interferometer
The distributed optical fiber acoustic sensor system based on the Sagnac effect
comprises a light source, optical fiber loop, photoelectric converter, coupler, lock-in
amplifier, signal processing, and PZT phase modulator. After being split by the 2
× 2 coupler, the light emitted by the light source propagates along the clockwise
and counterclockwise directions of the fiber ring of the Sagnac interferometer and
passes through the relevant leak points successively. The impact force generated by
the fluid leakage in the pipeline acts on the optical fiber, causing the optical fiber to
change along the length and causing the phase of the light-guiding light propagating
in the optical fiber to be modulated. Finally, the two modulated lights interfere with
the photoelectric converter. Due to the difference in time between the forward and
reverse light passing through the leak point, the phase of the two transmitted light
waves changes. The magnitude of the phase difference between the two beams is
directly proportional to the position of the leakage point and the phase change rate
of the light wave caused by the leakage noise. The leakage point can be located by
using this basis [36].
When the pipeline leaks, the impact force generated by the airflow acts on the
optical fiber. The length, core diameter, and refractive index of the optical fiber will
change, causing the phase change of the propagating light wave [37, 38, 39]. Let the
length of the optical fiber affected by the leakage noise be l, and the phase change
caused by it be:
( ) ( ) ( )
∆l ∂β ∂β
∆θ = β∆l + l∆β = βl +l ∆n + l ∆r (3.1)
l ∂n ∂r
where ∆θ is the phase change, rad; β is the propagation constant of light wave in
optical fiber; r is the radius of the optical fiber core, m; n is the refractive index of
the fiber core. The principle of the leakage inspection system of natural gas pipeline
based on the Sagnac optical fiber interferometer is shown in Fig. 3.8.
Suppose the total length of the optical fiber is, the distance between the leakage
point and one end of the optical fiber is, and the one cycle delay time of light wave
Leak location R Leak
Single mode fiber
Total length of fiber L
Coupler
Light source
Detector Signal processing Computer
Fig. 3.8 Principle of natural gas pipeline leak inspection system based on Sagnac fiber optic
interferometer [40]
propagating along the optical fiber ring is:
Ln
t= (3.2)
c
where L is the length of the optical fiber ring, m; c is the propagation velocity of light
wave in the medium, m/s. When the leakage occurs at the end of the fiber ring, the
delay time difference caused by the propagation of two mutually propagating light
waves in the fiber ring from the leakage point to the endpoint is:
−n(L − 2R)
τd = (3.3)
c
where R is the distance from the leakage point to the end of the optical fiber ring, m;
τd is the phase modulation time difference between the two beams, s. From Eq. (3.3),
it can be deduced that:
( )
L − nc τd
R= (3.4)
2
3.2.3.2 Leak Inspection Technology Based on Mach-Zehnder

Fiber-Optic Interferometer
Based on the Mach Zehnder optical fiber interferometer principle and three single
film optical fibers in the optical cable, a distributed micro-vibration test sensor is
formed to test the leakage noise along the pipeline. Among them, two optical fibers
constitute two optical arms of the optical fiber interferometer, two optical fibers are
test fibers, and the third optical fiber is used for signal transmission. The continuous
light wave emitted by the light source (LD) is divided from one end of the sensor
into two light waves with the light intensity of 1:1, which propagate simultaneously
in two optical fibers and converge at the other end of the optical fiber sensor to form
an interference signal. The third optical fiber transmits the interference signal to the
photoelectric detector (PD) and converts the optical signal into an electrical signal.
The signal is processed by the amplification and filter circuit and then transmitted to
the computer through analog-to-digital conversion for further signal processing and
analysis [25, 41–43].
When the test optical cable is affected by the leakage noise, the two test optical
fibers in the optical cable will produce stress and strain, so the two coherent light
waves propagating in the two sensing optical fibers will produce phase changes. The
light fields of the two coherent light waves propagating along the two test fibers are
simple harmonic vibrations, which can be expressed as:
ψ1 = A1 cos[ωt + s1 (t) + ϕ1 ] (3.5)
ψ2 = A2 cos[ωt + s2 (t) + ϕ2 ] (3.6)
where ψ1 and ψ2 are the light field strengths of the two coherent light waves; A1
and A2 are the light field amplitudes; ω is the angular frequencies of the light waves;
ϕ1 and ϕ2 are the initial phases of the two light waves; s1 (t) and s2 (t) are the optical
phase modulation amounts of the two light waves. The superimposed light field is:
A2 = A21 + A22 + 2 A1 A2 cos[s1 (t) − s2 (t) + (ϕ1 − ϕ2 )] (3.7)
Suppose the intensities of the two coherent light waves are I1 , I2 , then I1 = A21 ,
I2 = A22 , ∆s(t) = s1 (t) − s2 (t), ∆ϕ = ϕ1 − ϕ2 . The light intensity after the
interference of the two coherent light waves is:
√
I = I1 + I2 + 2 I1 I2 cos[∆s(t) + ∆ϕ] (3.8)
Let I0 be the total light intensity input into the two test fibers; α be the mixing
efficiency of two coherent light waves, then we have
I (t) = I0 1 + α cos[∆s(t) + ∆ϕ] (3.9)
If the only alternating current light intensity is considered, Eq. (3.9) can be
simplified to:
I (t) = I0 α cos[∆s(t) + ∆ϕ] (3.10)
The light intensity signal is converted into a current signal by a photodetector,

and the AC quantity of the photocurrent is
i (t) = K I0 α cos[∆s(t) + ∆ϕ] (3.11)

where ∆ϕ is usually a constant π2 , called DC bias. The photocurrent and the inspection
phase change slope at this bias is the largest, resulting in the highest inspection
sensitivity. Therefore, the test signal is a function of the phase modulation difference
∆s(t) of the two coherent light waves. Due to the different arrangement positions
of the two test optical fibers in the optical cable, the stress and strain produced by
the test optical cable are also different when the test optical cable is subjected to
leakage noise. Therefore, the phase changes produced by the two coherent light
waves propagating in the two sensing optical fibers are not precisely the same. By
detecting the change of interference light signal in real-time, the micro-vibration
signal generated by the pipeline leakage noise along the distributed optical fiber
sensor can be detected to realize the real-time pipeline leakage monitoring.
The distributed optical fiber technology for pipeline leakage inspection uses two
groups of light waves propagating in opposite directions in the sensing fiber simulta-
neously to detect the vibration signals at both ends of the sensor. When the vibration
signal along the pipeline causes the test optical fiber to produce stress and strain,
resulting in the light wave phase modulation, the light wave generating phase modu-
lation propagates to both ends of the sensor along the optical fiber. The time difference
of the vibration inspection signals at both ends of the sensor is detected by two photo-
electric detectors, and the location of the leakage can be accurately calculated. The
positioning formula can be expressed as
( )
L − v t2 − t1 − Lv
X= (3.12)
2
where X is the distance from the leakage point to the starting station; L is the detected
pipe length; t1 is the time when the photoelectric detector at the front end of the sensor
detects the pipeline leakage signal; t2 is the time when the photoelectric detector
at the end of the sensor detects the pipeline leakage signal; v is the propagation
speed of the light wave in optical fiber, where v = c/n is the speed of light in
vacuum, n is the refractive index of optical fiber. The time difference between the
two inspection signals can be obtained through the correlation operation of the two
inspection signals. The inspection signals at both ends of the distributed optical fiber
sensor can be expressed as:
{
x1 (t) = S(t) + n 1 (t)
(3.13)
x2 (t) = αS(t − τ0 ) + n 2 (t)
where x1 (t) and x2 (t) are signals detected by two detectors, respectively; S(t) is
the pipeline leakage signal; n 1 (t) and n 2 (t) are noise; α is the scale factor; τ0 is the
time delay between two test signals. Generally, S(t), n 1 (t), and n 2 (t) are incoherent
stationary random processes, so the cross-correlation function of x1 (t) and x2 (t) is:
{∞
R(τ ) = x1 (t)x2 (t + τ )dt (3.14)
−∞
3.3 Signal-Based Method 93
There is a peak value when τ = τ0 in the cross-correlation function shown in

Eq. (3.14), so the time difference between the two test signals can be determined by
obtaining the τ value corresponding to the maximum value. The two test signals are
converted into discrete signals after A/D conversion. If the selected sampling period
is Ts , the test signals at both ends of the sensor can be expressed as discrete x1 (n)
and x2 (n) respectively, where n is an integer. The finite data segment with sampling
point N in the same period of two test signals is taken for correlation operation, and
the cross-correlation function can be expressed as
N −1
1 ∑
R(m) = x1 (n)x2 (n + m) (3.15)
N n=0
where m is an integer. Calculate the point m = m 0 at which the maximum value

of R is obtained, and then the time difference m 0 Ts of the two test signals can be
obtained. Bring the time difference into the above equations to obtain the position
of the vibration signal along the pipeline.
3.3 Signal-Based Method
3.3.1 Volume/Mass Balance Method
According to the law of conservation of mass, when there is no leakage in the pipeline,
the mass flow of fluid into the pipeline should equal the mass flow out of the pipeline.
If a leak occurs in the pipeline, when the leakage reaches a certain amount, the inlet
and outlet will form a significant flow difference, as shown in Eq. (3.16) and Fig. 3.9
[44–46]. Therefore, the inspection of the input and output flow at multiple points
of the pipeline, or the inspection of the flow at the pump stations at both ends of
the pipeline and the signals are aggregated to form a mass flow balance image. The
degree of leakage and the approximate location can be determined according to the
changing characteristics of the image [47–50].
Q i (t) − Q 0 (t) = ∆Q(t) (3.16)
where Q i (t) is the flow of the initial station; Q 0 (t) is the flow of the terminal station;
∆Q(t) is the flow difference between initial station and terminal station.
The temperature, pressure, and density of the oil product may change during
the transportation process. For this reason, measuring the pipeline leakage by the
volume/mass balance method is not perfect in the actual project and needs to be
combined with other methods. To improve the precision and sensitivity of oil and gas
pipeline leak inspection, people have improved the traditional volume/mass balance
Fig. 3.9 Schematic diagram

of volume balance method ΔQ(t)
With leak
[51]
No leaks No leaks
method. That is, using the dynamic volume/mass balance method, the model of this
method is more complex, and the calculation volume is larger.
3.3.2 Negative Pressure Wave Method
When a leak suddenly occurs somewhere in the pipeline, a transient pressure drop will
be caused at the leak point, creating a negative pressure wave [52–56]. The sensors
at both ends of the leakage point can determine the leakage location according to the
change in the pressure signal and the time difference between the negative pressure
wave generated by the leakage and the upstream and downstream waves. It is assumed
that the length of the measured pipeline is L (see Fig. 3.10), the fluid velocity is v,
and the negative pressure wave velocity is a. When the leakage occurs at X meters
from the initial station of the pipeline, the time for the negative pressure wave to
reach the initial station from the leakage point is t 1 , and the time to reach the end
station is t 2 , then:
X
t1 = (3.17)
a−v
L−X
t2 = (3.18)
a+v
Then:
X L−X
t1 − t2 = − (3.19)
a−v a+v
The distance from the initial station to the leakage point can be expressed as:
1[ ( )]
X= L(a − v) + ∆t a 2 − v 2 (3.20)
2a
Initial station
End station
X L-X
a a
v
t1 t2
Fig. 3.10 Schematic diagram of the negative pressure wave method [51]
The wave velocity is often regarded as a constant value in the traditional method
based on the negative pressure wave method. However, it is unreasonable to regard
negative pressure wave velocity as a constant for leakage location in the real applica-
tion. This is because the actual negative pressure wave propagation speed is affected
by the pipeline environment and other conditions, and has a very close relationship
with the density, elasticity, and pipeline material of the medium [57, 58]. Based on the
particle momentum equation, fluid continuity equation, and mass conservation equa-
tion, relevant scholars have obtained the wave velocity calculation formula, which
can be described as:
/
K /ρ
v= (3.21)
1 + (K /E)(D/e)C
where v is wave velocity of negative pressure wave, m/s; C is the correction factor; D
is the inner diameter of the pipe, m; e is the wall thickness, m; ρ is the fluid density,
kg/m3 ; K is the bulk modulus of fluid, Pa; E is the elastic modulus of the pipeline,
Pa. In addition, some parameters in Eq. (3.21) are also affected by temperature.
Therefore, Eq. (3.21) can be written as:
/
K (T )/ρ(T )
v(T ) = (3.22)
1 + (K (T )/E)(D/e)C
where T is temperature. K is the compressibility of fluid, which is reciprocal to the

compressibility coefficient of fluid. Taking crude oil pipeline as an example, K of
crude oil can be calculated by Eq. (3.23) [51]:
1010
K (T ) = ( ) (3.23)
exp 0.51992 + 0.0023662T + 846,596
ρ02
+ 2366.6
ρ02
where ρ0 is the standard density of crude oil, kg/m3 .

The wavelet transform technique is used to extract the signal of the transient
negative pressure wave, and the feature points of the measuring point signal at both
ends are captured, thus improving the inspection accuracy of the method.
Based on the negative pressure wave inspection method, there are two positioning
methods: a differential algorithm that can quickly capture the waveform feature points
(negative pressure front peaks reach between pressure measurement points), and a
positioning method by determining the peak point of the correlation function (the
polarity Related introduction of leak location technology). The combination of these
two methods can improve the accuracy of leak location [59].
Wang et al. [60] used several pressure sensors to pick up negative pressure wave
signals. By recording the sequence of negative pressure waves by two groups of
pressure sensors in the pumping station, the real cause of the negative pressure wave
is determined jointly according to this order. This method can effectively reduce the
false alarm rate and improve positioning accuracy. Zhang et al. [61] put forward a
hybrid technology based on real-time transient modeling and the negative pressure
wave method. It has been proved by experiment that leak conditions in this method
can be distinguished from normal regulation conditions. Jia et al. [62] proposed a
deformation method based on the negative pressure wave method—circumferential
strain method, which measures the change of the internal pressure of the pipe indi-
rectly by measuring the change of the pipe’s hoop strain. This method can measure
the energy decay curve of the negative pressure wave to locate the leak according to
the intersection point of the curve.
3.3.3 GPS Time Label Method
When using the negative pressure wave method for leak location, in order to accu-
rately measure the time difference, a GPS time label method may be used. The
GPS synchronization time pulse signal is based on the negative pressure wave and
strengthens the signal synchronization relationship of each sensor data acquisition.
The negative pressure waves upstream and downstream of the pipeline leakage point
are determined by the conversion of sampling frequency and time label. By using the
difference in the time label of the leakage characteristic signal detected upstream and
downstream of the leakage point, the position of the pipeline leakage can be deter-
mined. The leak inspection accuracy of the pipeline leak inspection system based
on the GPS time label method can reach 1% of the total pipeline length, and the
accuracy is increased by three times compared with the traditional method [63].
3.3.4 Pressure Point Analysis Method
Pressure point analysis is a method used to detect leaks in gases, liquids, and some
multiphase flow pipes. The principle is to detect the pressure and flow change rate
in pipelines. When the pipeline is in a steady-state, pressure, velocity, and density

distributions do not change over time. As the equipment (pump or compressor) power
increases or decreases, changes in the velocity, pressure, and density distribution of
the fluid are continuous [64, 65]. Once the steady-state is disturbed by accident,
the pipeline will transition to a new stable state, and the fluid will change its flow
rate and pressure over time. If an accident occurs at a certain point in the pipeline,
its initial leak characteristics will be transmitted to the end of the pipeline (or any
other inspection location) within a certain time. The delivery time depends on the
distance from the accident site to the inspection point, and the sound propagates
speed in the pipeline. When a leak occurs, the pipeline completes the transition to
a new steady state. The momentum and impulse theorem determines the transition
time, and the time required to complete the change is from several minutes to ten
minutes. The pressure point analysis method does not require a transition between
constant steady-state and is suitable for the current pipeline operation. The pressure
point analysis method is usually used as a part of the SCADA system. Practice proves
that its application effect is good.
3.3.5 Cross Correlation Analysis
The walls of oil and gas pipelines are all elastomers. When a fluid leaks, the fluid
is subjected to pressure jets to induce elastic waves and propagate along the wall.
The elastic wave signal at two pipeline points is detected, and the cross-correlation
function is analyzed. The related delay technology can be used to determine whether
the leakage occurs and the leakage location [66]. The measuring points are arranged
at both ends of the test pipe section, and their cross-correlation function is taken.
From the maximum delay time and the sound transmission speed of the pipe, the
distance of the leaking point from the two measuring points can be measured. The
key to cross correlation analysis is to obtain the relationship between the pressure
wave induced by leakage and the vibration characteristics of pipelines. In addition,
a piezoelectric ceramic acceleration sensor can be used to expand the inspection
range. The signal-to-noise ratio and accuracy of the detected signal can be improved
by using the techniques of generalized correlation and multi-spectrum analysis [67].
Signal correlation refers to the correlation between two signals or the linear
relationship between variables. In statistics, it represents the covariance cov(x, y)
between two random vectors x and y. In signal processing, cross correlation is an
angle used to express the similarity between two signals. It is usually used to find
the characteristic h in the unknown signal by comparing it with the known signal, a
function of time between two signals. For discrete functions f i and gi , the correlation
is defined as:
∑
f i ∗ gi = f j gi+ j (3.24)
j
where * indicates conjugate. For continuous signals f (x) and g(x), the cross-
correlation can be defined as:
{
f (x) ∗ g(x) = f (t)g(x + t)dt (3.25)
where the integral in Eq. (3.25) is the integral over the entire possible time domain,
and the cross-correlation is essentially analogous to the convolution of two functions.
The autocorrelation of each individual function occurs when x in the functions f (x)
and g(x) take the same value. The similarity of the entire waveform is analyzed
through cross-correlation of each point of similar waveforms at the starting and end
stations.
However, the data of the starting and end stations are collected, and the sampling
rate is high. Each waveform includes tens of thousands of points. Since it would be
time-consuming to analyze the correlation for each point, it is necessary to find a
simple and effective way to define the correlation. Therefore, the Pearson correlation
coefficient can measure the degree of correlation between two signals [68].
3.3.6 Transient Test-Based Technique
The transient test-based technique (TTBT) refers to a series of methods for locating
leak locations based on transient measurement data. It can detect not only leaks,
but also defects such as partial blockages [69]. TTBTs can be subdivided into four
categories: (1) the transient reflection method, (2) the system response method, (3)
the transient damping method, and (4) the inverse transient method. The transient
reflection method is capable of identifying leaks and other possible faults based
on measured reflected pressure waves [70–72]. In the system response method, the
whole pressure signal in the frequency domain is analyzed by comparing the impulse
response of the faulty pipeline with that of the intact pipeline [73–78]. The transient
damping method is based on additional pressure decay caused by pipeline faults, so
the method focuses on a particular feature of the pressure signal in the frequency
domain [79, 80]. In the inverse transient method [81], the defect is characterized
by minimizing the difference between the measured data and the numerical model
results based on the transient control equation during the calibration process [82].
In addition, unreasonable assumptions must be overturned based on convincing
results in laboratory and field trials. That is, large pressure waves must be generated
for reliable fault inspection. In particular, small amplitude pressure waves have been
shown to allow reliable fault inspection [83, 84]. A portable pressure wave maker
can produce small and drastic control pressures [80].
The basic idea of the transient model method is to establish a real-time dynamic
model of the pipeline with measurable parameters (such as pressure, temperature,
flow, and differential pressure) as boundary conditions, and use the model to estimate
the parameters of the pipeline system online. When the deviation of the calculation
result exceeds the given threshold, it is judged that there is leakage. Otherwise, there is
no leakage. The equations used in dynamic simulation include momentum equation,
continuity equation, and energy equation. This method requires the installation of
flow meters, pressure sensors, and temperature sensors at the outlet, inlet, and along
the pipeline to measure parameters. Figure 3.11 is the schematic diagram of the
transient model method.
According to the transient flow theory, the flow equation of a non-branching,
homogeneous pipe can be obtained. The continuity equation is [86, 85]:
∂(ρ Fω) ∂(ρ F)

+ =0 (3.26)
∂x ∂t
The momentum equation is:
Actual pipeline
Data communication
SCADA database
Measured data
SCADA control interface
Feedback Data
Leakage status analysis and treatment
Output of pipeline model
Pipeline model
Model input
Fig. 3.11 The principle of the transient test-based technique [85]

( )
∂(ρω) ∂ ρω2 + P ( )
+ = −ρ λω2 /2D + g sin α (3.27)
∂τ ∂x
[ ( )]
∂Q ∂ ω2
− (ρωF) = (ρ F) u + + gS dτ
∂x ∂τ 2
[ ( )]
∂ ω2
+ (ρωF) u + + gs (3.28)
∂x 2
The gas state equation in the volume element is:
P = Zρ RT (3.29)
The boundary conditions of the starting and ending points of the pipe are:
{
Q 1 = f (P1 , T1 )
(3.30)
Q 2 = f (P2 , T2 )
where ρ is the density of the gas, kg/m3 ; ω is the flow rate of the gas, m/s; F is
the cross-sectional area of pipe, m2 ; t is a time variable, s; x is the variable along
the length of the pipe, m; g is gravity acceleration m/s2 ; α is the inclination angle
between the pipe and the horizontal plane, rad; λ is hydraulic friction coefficient; D
is the inner diameter of the pipe, m; P is the gas pressure in the pipeline, Pa; u is the
internal energy of the gas, J/kg; S is the height of pipeline position, m; Q is the heat
released by unit mass gas to the outside, J/kg.
The recurrence relation between flow along the pipeline and time and distance
can be solved by numerical solution, and then the pressure and flow values at any
time at each point along the pipeline can be predicted. The model is used to estimate
the parameters of the pipeline system online. When the deviation of the calculation
results exceeds the given threshold, it is judged that there is leakage. Otherwise, it
is judged that there is no leakage. The practice has proved that this method can be
used to judge the leakage of pipelines.
Because of the compressibility of the gas, it is theoretically believed that when
the leakage is small to a certain extent, the leakage will only cause the local pressure
at the leakage point to decrease, and have little impact on the pressure at the far ends.
Only when the leakage increases to a certain extent, the pressure at the beginning
and end of the pipeline will change. Leakage will cause an increase in upstream flow
and a decrease in downstream flow at the leakage point. When the starting pressure
is constant, the leakage will also reduce the pressure along the pipeline. Figure 3.12
shows the pressure and flow changes before and after the leakage point.
Fig. 3.12 Changes in

pressure and flow before and P
after leakage [87]
P1
P2
1 X 2
(a)
QS1
QC1 QC2
QS2
1 X 2
(b)
Temperature, pressure, and flow measuring instruments are installed at each test
point to measure temperature, pressure, and flow. When the transmitter is installed,
the corresponding parameter signals can be transmitted remotely to realize the remote
control. This can be realized in the existing supervisory control and data acquisition
(SCADA) system without additional equipment.
Assuming X is the distance from the leak point to the upstream sensor, then:
D Q 1 = Q S1 − Q C1 (3.31)
D Q 2 = Q S2 − Q C2 (3.32)
where DQ is the difference between the upstream and downstream theoretical flow
and the actual flow, then the total leak rate is:
Q L = D Q1 − D Q2 (3.33)
The Bernoulli equation for stable flow in a simple gas transmission pipeline is:
dP ω2 dx dω2
− =λ + + gds (3.34)
ρ 2 d 2
When the terrain fluctuation height difference does not exceed 200 m, the pressure
drop consumed by overcoming the height difference accounts for a small proportion
in the above formula, so the influence of the elevation change can be ignored, then
Eq. (3.34) can be transformed into:
dP ω2 dx
− =λ (3.35)
ρ 2 d
Let Eq. (3.35) be combined with the gas state equation and flow equation:
⎧
⎪ ω2 dx
⎨− ρ = λ 2 d
dP
P = Zρ RT (3.36)
⎪
⎩
Q = wρπ4 d
2
We can get:
{p2 {L
− Pd P = C Q 2 dl (3.37)
p1 0
where C = 8λZ RT
π 2d5
is approximately constant. Integrating Eq. (2.7), we get:
{L
P12 − P22 = C Q 2 dl
0
{X {L
= C(Q + D Q1) dx +2
C(Q − D Q2)2 dx (3.38)
0 X
If Q is replaced by the calculated flow in Eq. (3.38) and Q + D Q1 and Q − D Q2

are replaced by the actual upstream and downstream flows, the positioning formula
of the leakage point is:
D Q2(2Q − D Q2)
X=( ) L (3.39)
D Q1 + D Q 2 (2Q + D Q1 − D Q2)
This method is based on the assumption of steady-state flow and does not consider
the flow change and pressure distribution of the pipeline under transient condi-
tions. The practice has proved that the gas flow in the actual gas transmission
pipeline is constantly changing, and the flow rate of the endpoint is often greater
than that of the starting point. The location of the leakage point calculated by the
positioning Eq. (3.39) is negative, which is obviously inconsistent with the actual
results. Therefore, this method cannot be used to locate the leakage point accurately.
For transient flow conditions, leak inspection is usually performed using pressure
gradient methods. Taking the starting point pressure and flow data measured on site
as boundary conditions, the pressure change curve along a pipeline can be simulated.
Taking the end-point pressure and flow data measured on site as boundary conditions,
the pressure change curve along a pipeline can also be simulated. These two pressure
change curves must have an intersection, which is theoretically the pipeline’s leakage
point, as shown in Fig. 3.13.
After the system finds the alarm, it will locate the leakage point according to
the real-time monitoring curve. Assuming a leakage accident occurs at x0 , after a
certain period, the disturbance will be transmitted to both ends of the pipeline. The
straight-line SE is the square distribution of pressure along the whole pipeline when
there is no leakage, and the straight-line AB is the pipeline pressure distribution
obtained from the front to back leakage monitoring of the pipeline using the starting
point boundary conditions after the leakage. It can be seen from the figure that the
simulation results from the starting point to x0 are close to the actual situation of
pipeline operation, while the simulation results from x0 to L deviate from the actual
situation. In addition, the farther away from x0 , the greater the deviation.
Fig. 3.13 Square line chart P2

of gas pipeline pressure [87]
S
A
C
M E
D
B
0
x0 L x
Similarly, the pipeline is simulated from back to front with the operating parame-
ters of the pipeline’s endpoint as the boundary conditions. As shown in the straight-
line CD, the simulation results from x0 to L are close to the actual situation of pipeline
operation. When straight-lines AB and CD intersect at a point, it can be concluded
that the corresponding pipe has leaked at x0 . The positioning accuracy depends on
the distance step of the pipe section. The positioning accuracy can be improved by
reducing the pipe section’s distance step, but the pipe section, but the distance step
of the pipe section cannot be reduced without limitation. Generally, 1% of the pipe
length is taken.
The transient test-based method has the following advantages:
(1) It can establish a mathematical model for the dynamic characteristics of the
fluid (flow, temperature, and pressure), and consider the physical characteris-
tics of the pipeline (such as pipe length, diameter, and wall thickness) and the
characteristics of the transmission medium (such as viscosity and density). In
addition, the model can distinguish instrument error, normal transient operation,
and leakage.
(2) It can detect the normal pipeline filling and pressure relief process through
volume compensation. It can also be used during pipeline startup and shutdown;
(3) The alarm threshold can be adjusted according to the current operating
conditions to reduce the false alarm rate;
(4) Accurate positioning, capable of detecting less than 1% leakage, and strong
adaptability.
However, this method also has the following disadvantages:
(1) The model is complex, and the workload of modeling and calculation is quite
large;
(2) It is required to accurately know the flow, pressure, and temperature values
at the input and output ports and the pressure and temperature values at the
intermediate measurement points. There are many measurement data, and there
will always be errors and uncertainties in the actual measurement, resulting in
a high false alarm rate;
(3) The inspection and positioning accuracy of the system largely depends on the
accuracy of the model and measuring instruments.
3.3.7 State Estimation Method
The state estimation method is suitable for small leak inspection and location. Based
on the mass balance equation, momentum equilibrium equation, energy balance equa-
tion, state equation, and other mechanisms, a nonlinear distributed parameter system
model is usually linearized by the difference or characteristic line method. Then,
the state estimator is designed to estimate the state of the system, and the estimated
value is the basis of leak inspection. The estimator can be an observer or a Kalman
filter [88]. Jiménez Cabas [89] summarized the development of the state estimation
method for 2010–2017 and further classified it into seven categories: extended Luen-
berger observer, adaptive state observer, sliding mode method, nonlinear recursive
observer, Kalman filter-based method, high gain observer, exponential observer. This
section briefly introduces the development status of five of them, which are widely
used.
(1) Adaptive state observer
An adaptive state observer is developed according to the principle of conserva-
tion of mass and momentum. In the leak inspection system with the observer
as the core, the leak location and size can only be measured at both ends of the
pipeline [90].
(2) Sliding mode method
Sliding mode observer is widely used in dynamic control, observation, and
pipeline leakage inspection. Navarro et al. [91] established a model based on
a robust sliding mode differentiator for leak inspection of water pipelines. The
simulation results show that the method can reduce the chattering effect. Negrete
and Verde [92] used a sliding modal observer to inspect multiple leaks in the
pipeline. The simulation and experiments proved that the technology has good
robustness. Espinoza-Moreno et al. [93] compared the application effects of
sliding mode observer and Algebraic method in water pipeline leakage inspec-
tion through experiments and showed satisfactory results. Carvajal-Rubio et al.
[94] based on sliding mode observer to detect the leakage of plastic water
pipeline and proved its effectiveness through experiments.
(3) Nonlinear observer
Verde [95] proposed a leak inspection method that only requires sensors installed
at the end of the pipeline. In this method, two minimum order nonlinear
observers generate residuals, and the flow and pressure at the beginning and
end of the pipeline are taken as outputs. Besancon et al. [96] proposed a leak
inspection method based on a direct online observer. Experiments show that
the method is effective. Torres et al. [97] demonstrated the non-linear observer
method for pipeline monitoring and analyzed two observer methods and two
applications.
(4) Kalman filter-based method
Kalman filter-based method includes traditional Kalman filter, extended Kalman
filter, and adaptive Kalman filter. Through literature review, it is found that it is
the most concerned method of scholars [98–104]. Emara-Shabaik et al. [105]
used a modified extended Kalman filter combined with feedforward calculation
to predict the leakage. The numerical simulation results show that the method
can quickly detect small leaks in the pipeline. Khulief and Emara-Shabaik [106]
validated the adaptive multimodel state estimation technology that combines
modified Kalman filter with feedforward computation by establishing an exper-
imental platform, which shows that the method is effective. Torres et al. [107]
proposed a new water hammer equation model for pipeline leak inspection, and
verified the effectiveness of the method by simulation. Dos Santos et al. [108]
simulated the high-pressure gas pipeline as a linear parameter varying system
and built a model to detect gas leakage. Actual data and numerical simulation
proved the validity of the model. Guillén et al. [109] changed the boundary
conditions of the conventional model, taking the inlet pump pressure and the
end pressure of the pipeline as the boundary conditions. The numerical simu-
lation and experimental results show that the method has a good application
prospect. Choi et al. [110] analyzed the influence of the adaptive Kalman filter on
sampling interval and proposed an algorithm that can adjust sampling interval.
The algorithm is applied to the water pipeline in Jeongeup City, South Korea,
and the results show high inspection accuracy. Verde et al. [111] proposed a new
method to detect single leakage in pipeline, which is based on the design idea of
a continuous extended Kalman filter. The numerical simulation and experiment
show that the method has good application prospects. Delgado-Aguiñaga et al.
[112] proposed a method for detecting multiple leaks in pipelines. This method
needs to install pressure and flow sensors at the end of the pipeline. Experiments
verify the performance of this method.
(5) High gain observer
High-gain observers are widely used for output feedback control in nonlinear
systems. Dulhoste et al. [113] proposed a friction model for leak inspection
based on a non-linear observer, in which more complex friction models were
considered, and field tests were carried out. Torres et al. [114] proposed a high-
gain observer method to detect and locate liquified petroleum gas (LPG) pipeline
leakage. The method has been proved to be effective by actual verification.
For pipeline monitoring methods, the explanation of performance indicators is shown

in Table 3.1 [115–118]. The advantages, disadvantages, and performance evaluation
of these technologies are shown in Tables 3.2 and 3.3.
Table 3.1 Performance indicators of pipeline mornitoring system

Performance indicators Interpretation
Adaptive ability Transmission media or monitoring objects have the appropriate
adaptability for different environments and can improve their
performance and complete tasks according to the changing
information
Robustness Robustness refers to the ability to continue working or provide helpful
information in the event of changes in operating conditions or data
loss. The robustness of the monitoring system can be divided into
function loss due to signal interruption, loss of system reliability
caused by changes in the pump and valve state, and loss of system
reliability during the start of the stabilization process
Reliability Reliability refers to the ability of the leak monitoring system to detect
pipeline leakage when operating within the design range accurately.
That is, the ability of the system to detect pipeline leakage events and
the possibility of a system error sending a leak alarm when no
pipeline leakage occurs. Reliability is characterized by two
performance indicators: leak alarm rate and false alarm frequency
Sensitivity The sensitivity of the pipeline leakage monitoring system is a
comprehensive measure of the leakage that can be detected by the
system and response time, namely the leakage monitoring sensitivity
and the alarm response time
Accuracy The accuracy refers to the accuracy of pipeline leakage parameters
estimated by the system. That is, the accuracy of leakage alarms and
additional information such as leak location and leakage rate
Effectiveness Effectiveness refers to the ability to work continuously
False alarm rate Is it possible to detect leaks accurately, and the rate of alarms due to
operational errors and equipment failures is low?
Response time The time required from the beginning of the leak to the detection of
the leak by the system
Positioning ability Is it possible to locate the leak and the accuracy of positioning?
Economy Refers to the cost of system construction, operation, and maintenance
Maintainability It refers to whether the system can be repaired quickly when the
system fails
Identification ability It refers to the inspection system’s ability to distinguish between
inversion, pressure regulation, pump initiation, pump shutdown, and
leakage
Table 3.2 Advantages and disadvantages of leakage mornitoring methods for oil and gas pipelines
G O W
√ √ √
Optical fiber 1. Online monitoring can The location of the leak
sensing be realized point on the
2. The anti-interference cross-section of the
ability is strong, and the pipeline will
measuring range is large significantly affect the
3. Small leaks can be sensitivity of the
detected inspection
√ √ √
Volume/mass There is no need to 1. The negative pressure
balance method establish the wave velocity is greatly
mathematical model of influenced by the
the pipeline. The principle medium, pipe material,
of inspection is simple and temperature, and
the wave velocity is
difficult to calculate
accurately
2. It is challenging to set
the appropriate pipe
leakage threshold
3. The sensitivity to
small leaks is poor
√ √ √
Negative pressure 1. This method is 1. It requires that the
wave method sensitive and accurate occurrence of a leak is
2. There is no need to sudden
establish a mathematical
model of the pipeline
3. The principle is simple,
and its applicability is
strong
√ √ √
Pressure point 1. Easy to use and quick Cannot detect small
analysis method to install leaks
2. Can be used for
gas-liquid multi-phase
flow pipes with high
sensitivity
√ √ √
GPS time tag The precision of /
method inspection and positioning
is high
(continued)

G O W
√ √ √
Cross correlation 1. The leakage inspection 1. It is very insensitive
analysis is sensitive and accurate to small leaks
2. Only need to detect the 2. In long-distance
pressure signal pipelines, especially
plastic pipes, the sound
transmission is
considerably damped
3. The sensor is easily
disturbed
√ √ √
State estimation The positioning accuracy 1. Only applicable to
method is high. The calculation the inspection and
speed is fast, and it is positioning of small
convenient for real-time leaks
processing 2. The inspection speed
of the natural gas
pipeline is slow, and the
flowmeter needs to be
set up
√ √ √
Transient 1. The accuracy and The inspection of small
test-based reliability of oil pipeline leaks is not sensitive
technique leak inspection are high
2. Suitable for large leak
inspection
110
Table 3.3 Performance evaluation of some monitoring methods

Optical fiber Can General N/A General Weak Can General High High
sensing
Volume/mass Cannot Poor General Poor Weak Can High Low Low
balance method
Negative Can Good Fast General Weak Can High General General
pressure wave
method
GPS time tag Can Good Fast General Weak Can High General General
method
Pressure point Cannot Poor General General Weak Can High General General
analysis method
Cross Can General Fast General Weak Can General General General
correlation
analysis
State estimation Can General Slow General General Can High High High
method
Transient Can General General General General Can Low High General
test-based
method
3 Pipeline Health Monitoring Technology
References 111
References
1. Lu, H., Iseley, T., Behbahani, S., & Fu, L. (2020). Leakage detection techniques for oil and
gas pipelines: State-of-the-art. Tunnelling and Underground Space Technology, 98, 103249.
2. Lu, P., Lalam, N., Badar, M., Liu, B., Chorpening, B. T., Buric, M. P., & Ohodnicki, P. R.
(2019). Distributed optical fiber sensing: Review and perspective. Applied Physics Reviews,
6(4), 041302.
3. Barrias, A., Casas, J. R., & Villalba, S. (2016). A review of distributed optical fiber sensors
for civil engineering applications. Sensors, 16(5), 748.
4. Ren, L., Jiang, T., Jia, Z. G., Li, D. S., Yuan, C. L., & Li, H. N. (2018). Pipeline corrosion and
leakage monitoring based on the distributed optical fiber sensing technology. Measurement,
122, 57–65.
5. Maraval, D., Gabet, R., Jaouen, Y., & Lamour, V. (2016). Dynamic optical fiber sensing with
Brillouin optical time domain reflectometry: Application to pipeline vibration monitoring.
Journal of Lightwave Technology, 35(16), 3296–3302.
6. Rogers, A. J. (1991, September). Distributed optical fiber sensing. In Chemical and medical
sensors (Vol. 1510, pp. 2–24). SPIE.
7. Shatalin, S. V., Treschikov, V. N., & Rogers, A. J. (1998). Interferometric optical time-domain
reflectometry for distributed optical-fiber sensing. Applied Optics, 37(24), 5600–5604.
8. Zhang, W., Wu, Q., Zhang, Z., Tian, Y., Yang, Y., Xu, X., Ma, J., & Li, J. (2022). Application
of distributed optical fiber pipeline monitoring technology in long-distance water conveyance
project. Water & Wastewater Engineering, 48(6), 124–129.
9. Li, J., Zhou, X., Yin, Z., Wang, C., Xu, Y., Zhang, J., & Zhang, M. (2021). Reconstruction
compression correlation demodulation for Raman optical time domain reflection. Advanced
Photonics Research, 2(10), 2100047.
10. Takada, K., Himeno, A., & Yukimatsu, K. (1991). Phase-noise and shot-noise limited oper-
ations of low coherence optical time domain reflectometry. Applied Physics Letters, 59(20),
2483–2485.
11. Breteler, R. F. K., Van der Tol, J. J., Felicetti, M., Sasbrink, B., & Smit, M. K. (2011). Photonic
integrated Brillouin optical time domain reflection readout unit. Optical Engineering, 50(7),
071111.
12. Tateda, M., & Horiguchi, T. (1989). Advances in optical time domain reflectometry. Journal
of Lightwave Technology, 7(8), 1217–1224.
13. Klein Breteler, R. F., van der Tol, J. J. G. M., Felicetti, M., Sasbrink, G. D. J., & Smit, M.
K. (2011). Photonic integrated Brillouin optical time domain reflection readout unit. Optical
Engineering, 50(7), 07111–1.
14. Froggatt, M., & Bowen, W. (1998). Optical time-domain reflectometry in optical fiber with
reflection delay time matched to the period of the optical frequency modulation. Applied
Optics, 37(10), 1731–1734.
15. Barnoski, M., Rourke, M., Jensen, S. M., & Melville, R. T. (1977). Optical time domain
reflectometer. Applied Optics, 16(9), 2375–2379.
16. Feng, N. N., & Huang, W. P. (2004). An efficient computation scheme for time-domain
reflection at optical waveguide discontinuities. IEEE Photonics Technology Letters, 16(2),
461–463.
17. Personick, S. D. (1977). Photon probe—An optical-fiber time-domain reflectometer. The Bell
System Technical Journal, 56(3), 355–366.
18. Huang, Z., & Liu, D. (2006). Optical fiber sensing technique for application in oil field.
Chinese Journal Engineering Geophysics, 3(6), 473–477.
19. Allwood, G., Wild, G., & Hinckley, S. (2016). Optical fiber sensors in physical intrusion
detection systems: A review. IEEE Sensors Journal, 16(14), 5497–5509.
20. Qin, Z. (2013). Distributed optical fiber vibration sensor based on Rayleigh backscattering.
University of Ottawa (Canada).
21. Bao, X., Dhliwayo, J., Heron, N., Webb, D. J., & Jackson, D. A. (1995). Experimental and
theoretical studies on a distributed temperature sensor based on Brillouin scattering. Journal
of Lightwave Technology, 13(7), 1340–1348.
22. Glišić, B., Posenato, D., & Inaudi, D. (2007, April). Integrity monitoring of an old steel
bridge using fiber optic distributed sensors based on Brillouin scattering. In Nondestructive
characterization for composite materials, aerospace engineering, civil infrastructure, and
homeland security 2007 (Vol. 6531, pp. 210–217). SPIE.
23. He, C., Hang, L., & Wu, B. (2006). Application of distributed optical fiber sensing technique
in pipeline leak detection. Transducer and Microsystem Technologies, 25(9), 8–14.
24. Horiguchi, T., Shimizu, K., Kurashima, T., Tateda, M., & Koyamada, Y. (1995). Development
of a distributed sensing technique using Brillouin scattering. Journal of Lightwave Technology,
13(7), 1296–1302.
25. Jinno, M., & Matsumoto, T. (1992). Nonlinear Sagnac interferometer switch and its
applications. IEEE Journal of Quantum Electronics, 28(4), 875–882.
26. Dakin, J. P., Pratt, D. J., Bibby, G. W., & Ross, J. N. (1985). Distributed optical fibre Raman
temperature sensor using a semiconductor light source and detector. Electronics Letters,
13(21), 569–570.
27. Bolognini, G., Park, J., Soto, M. A., Park, N., & Di Pasquale, F. (2007). Analysis of distributed
temperature sensing based on Raman scattering using OTDR coding and discrete Raman
amplification. Measurement Science and Technology, 18(10), 3211.
28. Othonos, A. (1997). Fiber Bragg gratings. Review of Scientific Instruments, 68(12), 4309–
4341.
29. Hill, K. O., & Meltz, G. (1997). Fiber Bragg grating technology fundamentals and overview.
30. Chen, J., Liu, B., & Zhang, H. (2011). Review of fiber Bragg grating sensor technology.
Frontiers of Optoelectronics in China, 4(2), 204–212.
31. Liang, W., Huang, Y., Xu, Y., Lee, R. K., & Yariv, A. (2005). Highly sensitive fiber Bragg
grating refractive index sensors. Applied Physics Letters, 86(15), 151122.
32. Giles, C. R. (1997). Lightwave applications of fiber Bragg gratings. Journal of Lightwave
Technology, 15(8), 1391–1404.
33. Mihailov, S. J. (2012). Fiber Bragg grating sensors for harsh environments. Sensors, 12(2),
1898–1918.
34. Perez, I. M., Cui, H., & Udd, E. (2001, August). Acoustic emission detection using fiber Bragg
gratings. In Smart structures and materials 2001: Sensory phenomena and measurement
instrumentation for smart structures and materials (Vol. 4328, pp. 209–215). SPIE.
35. Chen, Z., Zhang, L., Wang, Z., & Liang, W. (2007). Method of leakage detection for gas
pipelines based on distributed optic fiber sensors. Transducer and Microsystem Technologies,
26(7), 108–110.
36. Erdogan, T., Mizrahi, V., Lemaire, P. J., & Monroe, D. (1994). Decay of ultraviolet-induced
fiber Bragg gratings. Journal of Applied Physics, 76(1), 73–80.
37. Starodumov, A. N., Zenteno, L. A., Monzon, D., & De La Rosa, E. (1997). Fiber Sagnac
interferometer temperature sensor. Applied Physics Letters, 70(1), 19–21.
38. Liang, H., Sun, M., & Jin, Y. (2013). Twist sensor based on Sagnac single-mode optic fiber
interferometer. Optik, 124(24), 6676–6678.
39. Kurmer, J. P., Kingsley, S. A., Laudo, J. S., & Krak, S. J. (1993, March). Applicability of a
novel distributed fiber optic acoustic sensor for leak detection. In distributed and multiplexed
fiber optic sensors II (Vol. 1797, pp. 63–71). SPIE.
40. Hu, Z., Zhang, G., He, J., Zhang, L., & Zhou, J (2003). Leak detection on gas pipeline with the
distributed fiber-optic sensing technology. Journal of Transducer Technology, 22(10), 48–53.
41. Sun, Q., Liu, D., Wang, J., & Liu, H. (2008). Distributed fiber-optic vibration sensor using a
ring Mach-Zehnder interferometer. Optics Communications, 281(6), 1538–1544.
42. Lee, B. H., Kim, Y. H., Park, K. S., Eom, J. B., Kim, M. J., Rho, B. S., & Choi, H. Y. (2012).
Interferometric fiber optic sensors. Sensors, 12(3), 2467–2486.
References 113
43. Zhou, Y., Jin, S., Zhang, Y., & Sun, L. (2005). Study on the distributed optical fiber sensing
technology for pipeline leakage detection. Journal of Optoelectronics·Laser, 16(8), 935–938.
44. Kamenev, O. T., Kulchin, Y. N., Petrov, Y. S., Khiznyak, R. V., & Romashko, R. V. (2016).
Fiber-optic seismometer on the basis of Mach-Zehnder interferometer. Sensors and Actuators
A: Physical, 244, 133–137.
45. Yuan, Y., Cheng, Y., Yang, J., Zhang, H., Lu, D., Lv, Y., Peng, F., Li, H., Zhang, X., Jiang, F., &
Yuan, L. (2017). Suppression of interference noise caused by Fresnel reflection in all-fiber
white-light interferometer. Applied Optics, 56(31), 8732–8737.
46. Kim, J. H. (2008). An all fiber white light interferometric absolute temperature measurement
system. Sensors, 8(11), 6825–6845.
47. Stouffs, P., & Giot, M. (1993). Pipeline leak detection based on mass balance: Importance of
the packing term. Journal of Loss Prevention in the Process Industries, 6(5), 307–312.
48. Sheltami, T. R., Bala, A., & Shakshuki, E. M. (2016). Wireless sensor networks for leak
detection in pipelines: A survey. Journal of Ambient Intelligence and Humanized Computing,
7(3), 347–356.
49. Boaz, L., Kaijage, S., & Sinde, R. (2014, July). An overview of pipeline leak detection and
location systems. In Proceedings of the 2nd Pan African International Conference on Science,
Computing and Telecommunications (PACT 2014) (pp. 133–137). IEEE.
50. Sandberg, C., Holmes, J., McCoy, K., & Koppitsch, H. (1989). The application of a continuous
leak detection system to pipelines and associated equipment. IEEE Transactions on Industry
Applications, 25(5), 906–909.
51. Zong, Z. (2009). The design of oil pipeline leak detection system. Master’s Thesis, Harbin
University of Science and Technology, China.
52. Lu, W., Liang, W., Zhang, L., & Liu, W. (2016). A novel noise reduction method applied in
negative pressure wave for pipeline leakage localization. Process Safety and Environmental
53. Li, J., Zheng, Q., Qian, Z., & Yang, X. (2019). A novel location algorithm for pipeline
leakage based on the attenuation of negative pressure wave. Process Safety and Environmental
54. Ma, C., Yu, S., & Huo, J. (2010, August). Negative pressure wave-flow testing gas pipeline leak
based on wavelet transform. In 2010 International Conference on Computer, Mechatronics,
Control and Electronic Engineering (Vol. 5, pp. 306–308). IEEE.
55. Chen, Q., Shen, G., Jiang, J., Diao, X., Wang, Z., Ni, L., & Dou, Z. (2018). Effect of rubber
washers on leak location for assembled pressurized liquid pipeline based on negative pressure
wave method. Process Safety and Environmental Protection, 119, 181–190.
56. Wang, J., Zhao, L., Liu, T., Li, Z., Sun, T., & Grattan, K. T. (2016). Novel negative pressure
wave-based pipeline leak detection system using fiber Bragg grating-based pressure sensors.
57. Liu, B., Jiang, Z., Nie, W., Ran, Y., & Lin, H. (2021). Research on leak location method
of water supply pipeline based on negative pressure wave technology and VMD algorithm.
Measurement, 186, 110235.
58. Yang, J., Qingxin, Y., & Guanghai, L. (2007, May). Leak identification method for buried
gas pipeline based on spatial-temporal data fusion. In 2007 IEEE International Conference
on Control and Automation (pp. 774–777). IEEE.
59. Wang, X., Huang, K., Zhi, X., Lu, Y., & Chi, H. (2008). Present stage of the development of
oil-gas pipeline leak-detection technologies. Pipeline Technique and Equipment, 16, 24–26.
60. Wang, Z., Wang, H., Fu, L., Mu, S., & Wang, L. (2015). Pipeline detection method
based on multiple-pressure sensor and negative pressure wave. Transducer and Microsystem
Technologies, 34, 115–118.
61. Zhang, T., Tan, Y., Zhang, X., & Zhao, J. (2015). A novel hybrid technique for leak detection
and location in straight pipelines. Journal of Loss Prevention in the Process Industries, 35,
157–168.
62. Jia, Z. G., Ren, L., Li, H. N., Ho, S. C., & Song, G. B. (2015). Experimental study of pipeline
leak detection based on hoop strain measurement. Structural Control and Health Monitoring,
22(5), 799–812.
63. Liang, W., Zhang, L. B., & Wang, Z. H. (2004, January). State of research on negative pressure
techniques applied to leak detection in liquid pipelines. In International Pipeline Conference
(Vol. 41766, pp. 2261–2265).
64. bin Md Akib, A., bin Saad, N., & Asirvadam, V. (2011, March). Pressure point analysis for
early detection system. In 2011 IEEE 7th International Colloquium on Signal Processing and
its Applications (pp. 103–107). IEEE.
65. Fiedler, J. (2014). An overview of pipeline leak detection technologies. American School of
Gas Measurement Technology (ASGMT), Peabody, MA, US.
66. Beck, S. B. M., Curren, M. D., Sims, N. D., & Stanway, R. (2005). Pipeline network features
and leak detection by cross-correlation analysis of reflected waves. Journal of Hydraulic
Engineering, 131(8), 715–723.
67. Davoodi, S., & Mostafapour, A. (2014). Gas leak locating in steel pipe using wavelet trans-
form and cross-correlation method. The International Journal of Advanced Manufacturing
Technology, 70(5), 1125–1135.
68. Guo, C., Shi, K., & Chu, X. (2022). Cross-correlation analysis of multiple fibre optic
hydrophones for water pipeline leakage detection. International Journal of Environmental
Science and Technology, 19(1), 197–208.
69. Meniconi, S., Brunone, B., Ferrante, M., & Massari, C. (2011). Transient tests for locating
and sizing illegal branches in pipe systems. Journal of Hydroinformatics, 13(3), 334–345.
70. Brunone, B. (1999). Transient test-based technique for leak detection in outfall pipes. Journal
of Water Resources Planning and Management, 125(5), 302–306.
71. Misiunas, D., Vitkovsky, J., Olsson, G., Simpson, A., & Lambert, M. (2005). Pipeline break
detection using pressure transient monitoring. Journal of Water Resources Planning and
Management, 131(4), 316–325.
72. Meniconi, S., Brunone, B., Ferrante, M., Capponi, C., Carrettini, C. A., Chiesa, C., Segalini,
D., & Lanfranchi, E. A. (2015). Anomaly pre-localization in distribution–transmission mains
by pump trip: Preliminary field tests in the Milan pipe system. Journal of Hydroinformatics,
17(3), 377–389.
73. Kim, S. H. (2005). Extensive development of leak detection algorithm by impulse response
method. Journal of Hydraulic Engineering, 131(3), 201–208.
74. Liou, C. P. (1998). Pipeline leak detection by impulse response extraction. Journal of Fluids
Engineering, 120(4), 833–838.
75. Mpesha, W., Hanif Chaudhry, M., & Gassman, S. L. (2002). Leak detection in pipes by
frequency response method using a step excitation. Journal of Hydraulic Research, 40(1),
55–62.
76. Lee, P. J., Vítkovský, J. P., Lambert, M. F., Simpson, A. R., & Liggett, J. A. (2005). Leak
location using the pattern of the frequency response diagram in pipelines: A numerical study.
Journal of Sound and Vibration, 284(3–5), 1051–1073.
77. Duan, H. F., Lee, P. J., Ghidaoui, M. S., & Tung, Y. K. (2011). Leak detection in complex series
pipelines by using the system frequency response method. Journal of Hydraulic Research,
49(2), 213–221.
78. Gong, J., Zecchin, A., Simpson, A., & Lambert, M. (2014). Frequency response diagram for
pipeline leak detection: Comparing the odd and even harmonics. Journal of Water Resources
Planning and Management, 140(1), 65–74.
79. Wang, X., Lambert, M., Simpson, A., Liggett, J., & Vítkovsky, J. (2002). Leak detection
in pipelines using the damping of fluid transients. Journal of Hydraulic Engineering, 128,
697–711.
80. Brunone, B., Meniconi, S., & Capponi, C. (2018). Numerical analysis of the transient pressure
damping in a single polymeric pipe with a leak. Urban Water Journal, 15(8), 760–768.
81. Liggett, J. A., & Chen, L. C. (1994). Inverse transient analysis in pipe networks. Journal of
Hydraulic Engineering, 120(8), 934–955.
82. Covas, D., & Ramos, H. (2010). Case studies of leak detection and location in water pipe
systems by inverse transient analysis. Journal of Water Resources Planning and Management,
136(2), 248–257.
References 115
83. Meniconi, S., Brunone, B., Ferrante, M., & Massari, C. (2011). Small amplitude sharp pressure
waves to diagnose pipe systems. Water Resources Management, 25(1), 79–96.
84. Meniconi, S., Brunone, B., Frisinghelli, M., Mazzetti, E., Larentis, M., & Costisella, C. (2017).
Safe transients for pipe survey in a real transmission main by means of a portable device: The
case study of the Trento (I) supply system. Procedia Engineering, 186, 228–235.
85. Wang, S., & Carroll, J. J. (2007). Leak detection for gas and liquid pipelines by online
modeling. SPE Projects, Facilities & Construction, 2(2), 1–9.
86. Li, Y. (2017). The research of subsea pipeline leak detection and location system based on
SPRT and pressure gradient method. Masters Thesis, Zhejiang University, China.
87. Zhang, H., & Li, C. (2005). Leak detection technique for gas pipeline with unequal
temperatures. Petroleum Engineering Construction, 31(1), 25–27.
88. Verde, C., & Torres, L. (2017). Modeling and monitoring of pipelines and networks. Springer.
89. Jiménez Cabas, J. A. (2018). Liquid transport pipeline monitoring architecture based on state
estimators for leak detection and location (Master’s thesis). Universidad del Norte.
90. Aamo, O. M. (2015). Leak detection, size estimation and localization in pipe flows. IEEE
Transactions on Automatic Control, 61(1), 246–251.
91. Navarro, A., Begovich, O., Sanchez-Torres, J. D., Besançon, G., & Murillo, J. A. P.
(2012, June). Leak detection and isolation using an observer based on robust sliding mode
differentiators. In World Automation Congress 2012 (pp. 1–6). IEEE.
92. Negrete, M. A., & Verde, C. (2012). Multi-leak reconstruction in pipelines by sliding mode
observers. IFAC Proceedings Volumes, 45(20), 934–939.
93. Espinoza-Moreno, G., Begovich, O., & Sanchez-Torres, J. (2014, August). Real time leak
detection and isolation in pipelines: A comparison between sliding mode observer and
algebraic steady state method. In 2014 World Automation Congress (WAC) (pp. 748–753).
IEEE.
94. Carvajal-Rubio, J. E., Begovich, O., & Sánchez-Torres, J. D. (2015, October). Real-time leak
detection and isolation in plastic pipelines with equivalent control based observers. In 2015
12th International Conference on Electrical Engineering, Computing Science and Automatic
Control (CCE) (pp. 1–6). IEEE.
95. Verde, C. (2004). Minimal order nonlinear observer for leak detection. Journal of Dynamic
Systems, Measurement, and Control, 126(3), 467–472.
96. Besancon, G., Georges, D., Begovich, O., Verde, C., & Aldana, C. (2007, July). Direct observer
design for leak detection and estimation in pipelines. In 2007 European Control Conference
(ECC) (pp. 5666–5670). IEEE.
97. Torres, L., Besançon, G., Georges, D., Navarro, A., & Begovich, O. (2011). Examples
of pipeline monitoring with nonlinear observers and real-data validation. In 8th Interna-
tional Multi-Conference on Systems, Signals & Devices (SSD 2011) (p. v). Ecole Nationale
d’Ingenieurs de Sfax.
98. Doney, K. (2007). Leak detection in pipelines using the extended Kalman filter and the
extended boundary approach (Doctoral dissertation).
99. Navarro, A., Begovich, O., Besançon, G., & Dulhoste, J. F. (2011, September). Real-time
leak isolation based on state estimation in a plastic pipeline. In 2011 IEEE International
Conference on Control Applications (CCA) (pp. 953–957). IEEE.
100. Gong, J., Cai, J., Li, X., & Song, S. (2007, August). Research on state estimation of oil
pipeline considering adaptive extended Kalman filtering. In 2007 International Conference
on Mechatronics and Automation (pp. 1294–1298). IEEE.
101. Modisette, J. P. (2013, April). State estimation of pipeline models using the ensemble Kalman
filter. In PSIG Annual Meeting. OnePetro.
102. Okeya, I., Kapelan, Z., Hutton, C., & Naga, D. (2014). Online burst detection in a water
distribution system using the Kalman filter and hydraulic modelling. Procedia Engineering,
89, 418–427.
103. Durgut, İ, & Leblebicioğlu, M. K. (2016). State estimation of transient flow in gas pipelines
by a Kalman filter-based estimator. Journal of Natural Gas Science and Engineering, 35,
189–196.
104. Delgado-Aguinaga, J. A., Begovich, O., & Besançon, G. (2016, October). Varying-parameter
modeling and extended Kalman filtering for reliable leak diagnosis under temperature vari-
ations. In 2016 20th International Conference on System Theory, Control and Computing
(ICSTCC) (pp. 632–637). IEEE.
105. Emara-Shabaik, H. E., Khulief, Y. A., & Hussaini, I. (2002). A non-linear multiple-model state
estimation scheme for pipeline leak detection and isolation. Proceedings of the Institution of
Mechanical Engineers, Part I: Journal of Systems and Control Engineering, 216(6), 497–512.
106. Khulief, Y. A., & Emara-Shabaik, H. E. (2006). Laboratory investigation of a multiple-model
state estimation scheme for detection and isolation of leaks in pipelines. Proceedings of the
Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering,
220(1), 1–13.
107. Torres, L., Besancon, G., & Georges, D. (2008, December). A collocation model for water-
hammer dynamics with application to leak detection. In 2008 47th IEEE Conference on
Decision and Control (pp. 3890–3894). IEEE.
108. Dos Santos, P. L., Azevedo-Perdicoúlis, T. P., Jank, G., Ramos, J. A., & de Carvalho, J.
M. (2011). Leakage detection and location in gas pipelines through an LPV identification
approach. Communications in Nonlinear Science and Numerical Simulation, 16(12), 4657–
4665.
109. Guillén, M., Dulhoste, J. F., Besançon, G., Rubio, I., Santos, R., & Georges, D. (2014, June).
Leak detection and location based on improved pipe model and nonlinear observer. In 2014
European Control Conference (ECC) (pp. 958–963). IEEE.
110. Choi, D., Kim, S. W., Choi, M. A., & Geem, Z. (2016). Adaptive Kalman filter based on
adjustable sampling interval in burst detection for water distribution system. Water, 8(4), 142.
111. Verde, C., Torres, L., & González, O. (2016). Decentralized scheme for leaks’ location in a
branched pipeline. Journal of Loss Prevention in the Process Industries, 43, 18–28.
112. Delgado-Aguiñaga, J. A., Besançon, G., Begovich, O., & Carvajal, J. E. (2016). Multi-leak
diagnosis in pipelines based on extended Kalman filter. Control Engineering Practice, 49,
139–148.
113. Dulhoste, J. F., Besançon, G., Torres, L., Begovich, O., & Navarro, A. (2011, December).
About friction modeling for observer-based leak estimation in pipelines. In 2011 50th IEEE
Conference on Decision and Control and European Control Conference (pp. 4413–4418).
IEEE.
114. Torres, L., Verde, C., Besancon, G., & Gonzalez, O. (2014). High-gain observers for leak
location in subterranean pipelines of liquefied petroleum gas. International Journal of Robust
and Nonlinear Control, 24(6), 1127–1141.
115. Marlow, D., Heart, S., Burn, S., Urquhart, A., Gould, S., Anderson, M., Cook, S., Ambrose,
M., Madin, B., & Fitzgerald, A. (2007). Condition assessment strategies and protocols for
water and wastewater utility assets. Water Environment Research Foundation.
116. Wu, J., Li, H., Wang, L., Xu, B., Wang, H., Cai, Y., Liu, S., & Tao, Z. (2017). Performance
evaluation indices of pipeline leak monitoring systems. Oil & Gas Storage and Transportation,
36, 209–213.
117. Puust, R., Kapelan, Z., Savic, D. A., & Koppel, T. (2010). A review of methods for leakage
management in pipe networks. Urban Water Journal, 7(1), 25–45.
118. Misiūnas, D. (2008). Failure monitoring and asset condition assessment in water supply
systems. Vilniaus Gedimino technikos univer-sitetas.
Chapter 4
Health Monitoring Technology Based on
Artificial Intelligence
4.1 Introduction
Artificial intelligence is a branch of Computer Science in the 1950s. Its main goal is
to study human intelligence through computer modeling and simulation, and to make
computers more useful by solving complex problems like humans. Machine learning
is a branch of artificial intelligence. It uses computers to simulate human learning
behavior to acquire new knowledge or skills. It can reorganize the existing knowledge
structure according to the new knowledge to continuously improve its performance.
Machine learning is the core of artificial intelligence and the fundamental way to
realize computer intelligence [1–8]. Its application spreads over all fields of artificial
intelligence and gradually develops across fields. Machine learning based on data is
an essential aspect of modern intelligent technology. Since 1950, people have been
studying machine learning. In recent decades, many machine learning models have
been established, such as the artificial neural network model, naive Bayesian model,
random forest model, support vector machine, correlation vector machine, and deep
learning model.
4.2 Classic Models
4.2.1 Linear Regression
As an important method in the field of statistics, linear regression is mainly used

to describe the relationship between independent variables and dependent vari-
ables. Linear regression mainly includes unitary linear and multiple linear regression
models. Due to the small number of independent variables, the complexity of the
unitary linear regression model is simple. In addition, the neglect of independent
variables with high correlation may lead to the underfitting of the model [9–14]. The
linear regression model is:
https://doi.org/10.1007/978-981-19-6798-6_4
118 4 Health Monitoring Technology Based on Artificial Intelligence
yi = α + βxi + ei (4.1)
where yi is the outcome; xi is the value of the predictor for the i-th individual; α and
β are the parameters in the model. ei is the random error term [15–20].
4.2.2 Naive Bayes
The naive Bayes model is the simplest and most effective classification model among
Bayes models. The model is simple, accurate, and fast. Its performance is equivalent
to that of neural networks and decision trees. The naive Bayes classifier is a super-
vised learning method based on the assumption that features are independent of each
other. Although this assumption is often unsatisfactory, naive Bayes still shows high
accuracy and efficiency in practical applications [21–25].
Because naive Bayes has a poor classification effect when the number of attributes
is large or the correlation between attributes is significant. Some methods, such as
semi-naive Bayes, can moderately improve the original naive Bayes by considering
partial correlation [26–32].
4.2.3 Artificial Neural Network
Artificial neural network (ANN) is the most popular model category in the field of
machine learning. It is not a specific model but a general framework. ANN has many
types and variants, and it can be classified as dynamic and static models [33–43].
Radial basis function neural network (RBFNN) is a common model in ANN.
RBFNN is a feedforward neural network with the unique best approximation [44].
RBFNN usually has only three layers, including the input layer, hidden layer, and
output layer, as shown in Fig. 4.1. The prediction of RBFNN firstly maps the low-
dimensional input to the high-dimensional space of the middle layer. Secondly, the
hidden layer chooses the radial basis function for conversion, and then classifies the
output layer and calculates the linear combination, to realize the mapping relationship
between input and output.
The commonly used radial basis function uses Euclidean distance and Gaussian
function, which is expressed as follows [45]:

||x − μi ||2
ϑ(||x − μi ||) = exp − (4.2)
2σ 2
where μi represents center point of the Gaussian function of the i-th node of the
hidden layer; σi represents the width parameter of the i-th node; wi j represents the
hidden to output weight corresponding to the i-th hidden node; M represents the total
number of hidden nodes.
4.2 Classic Models 119
Fig. 4.1 Three-layer x1 x2 xn

structure of RBFNN Input layer
Hidden layer
Output layer
The network output is given as:

M
yj = wi, j ϑ(||x − μi ||2 ), j = 1, 2, . . . , P (4.3)
i=1
4.2.4 Kernel-Based Model
4.2.4.1 Support Vector Machine
The support vector machine (SVM) is a supervised learning method based on statis-
tics, which can solve classification and regression problems with small samples
and nonlinear characteristics [46–51]. Its main idea is to map the data to a high-
dimensional space, transform the nonlinear problem into a linear problem and use
the linear problem-solving method to solve the original nonlinear problem (Fig. 4.2).
Some detailed theories for SVM can be found in the literature [47].
In regression analysis, assume that the training set is
{(xi , yi )|xi ∈ R n , yi ∈ R, i = 1, . . . , m} (4.4)
where xi is the input vector; yi is the output vector.

The optimal linear decision function constructed in high-dimensional space is
f (x) = ω T ϕ(x) + b (4.5)
where ϕ(x) is nonlinear mapping function; ω is weighted vector; b is deviator.

Based on the principle of structural risk minimization, introducing the slack
variables ξi , ξi∗ , then the linear regression function can be expressed as
Fig. 4.2 Support vector x2
2
machine
w
=1
w× x-b
0
b=
w×
x-
-1
b=
x-
w×
w
x1
b
w
m

min 0.5||ω||2 + C ξi + ξi∗ (4.6)
i=1
⎧
⎨ yi − ω T ϕ(x) − b ≤ ε + ξi
s.t. ω T ϕ(xi ) − yi + b ≤ ε + ξi∗ (4.7)
⎩
ξi , ξi∗ ≥ 0
where C is the penalty factor; ε is the insensitive loss function.

The Lagrange function is introduced and converted into the dual form:

n
n

max[−0.5 αi − αi∗ α j − α ∗j K xi , x j
i=1 j=1

n
n
+ αi (yi − ε) − αi∗ (yi − ε)] (4.8)
i=1 i=1
⎧
⎨
n
n
αi = αi∗
s.t. i=1 i=1 (4.9)
⎩
0 ≤ αi ≤ C, 0 ≤ αi∗ ≤ C

where K xi , x j is the kernel function.
Assume that the optimal solutions obtained by Eq. (4.8) are αi and αi∗ , then the
regression function can be expressed as
n

f (x) = αi − αi∗ K xi , x j + b (4.10)
i=1
The selection of kernel functions is a vital issue of the SVM model, and different
kernel functions can lead to different generalization and learning abilities of predic-
tion models [52–57]. The three kernel functions used frequently include polynomial
kernel function, radial basis function (RBF), and Sigmoid kernel function. Because
the characteristic space corresponding to the RBF is infinite, it is sure that the sample
can be linearly separable under the condition of the finite sample, and the RBF is
shown in Eq. (4.11).

xi −x 2j
K xi , x j = ex p − 2σ 2
(4.11)
where σ is the width of the Gaussian radial basis function.
4.2.4.2 Relevance Vector Machine
The RVM was proposed by Tipping [58]. It is a supervised learning algorithm based
on the sparse Bayesian learning theory [59–63]. This method combines Bayesian
theory, Markov property, maximum likelihood estimation, and other theories. It
ensures the model’s sparsity by introducing a Gaussian prior distribution with zero
mean value assigned to the weight vector by hyperparameters. RVM and support
vector machine (SVM) have the same functional form, and they can both solve the
regression problem with small samples, but RVM avoids some of the shortcomings of
SVM [64–68]. For example, the kernel function of RVM is not restricted by Mercer
conditions, and the selection range of kernel functions is wider; RVM does not need
to use cross-validation to obtain hyperparameters, and its generalization ability is
better.
N
Let x j j=1 be the input vector, o = [o1 , o2 , . . . , o N ]T be the target vector. The
functional relationship between them can be expressed as Eq. (4.12). Then RVM
model can be expressed as Eq. (4.13).
on = y(x, w) + n (4.12)

N
y(x, w) = w j K x, x j + w0 (4.13)
j=1

where w is the weight vector; K x, x j is kernel function; y(x, w) is the output of
the prediction model; n is the Gaussian white noise.
In the sparse Bayesian framework, on is distributed independently, the likelihood
function of vector o is expressed as Eq. (4.14). To avoid overfitting, RVM assigns a
zero-mean Gaussian prior distribution to the weight vector w, as shown in Eq. (4.15).
According to Eqs. (4.13)–(4.14), the posterior distribution of weight parameter vector
w is calculated by Eq. (4.15):
N
||o−ψw2 ||

p o|w, σ 2 = √ 1
exp − 2σ 2 (4.14)
2πσ 2

N
p(w|α) = N w j |0, α1j (4.15)
j=0
N +1
√1
p w|o, α, σ 2 = /|
2π
| (4.16)
| |
| exp − 21 (w−µ)T 1
w−µ |
where ψ is the N × (N + 1) matrix; ψmn = K (xm , xn−1 ) (m /= 1) and ψmn = 1

(m = 1); σ 2 is a hyperparameter of RVM; α is another hyperparameter of RVM;
T −1
= ψ B ψ + A ; µ = ψT Bo; A = diag(α0 , α1 , α2 , · · · , α N ); B = σ 2 I N .
The marginal likelihood of hyperparameters can be obtained by integrating the
weights, as shown in Eq. (4.17). RVM can be transformed into the problem of
maximizing Eq. (4.15) by solving α and σ 2 .

N exp − 1 oT 1
−1 ψT o
p o|α, σ 2 = √12π √ 2 −1 B−1 +ψA (4.17)
|B +ψA−1 ψT |
4.2.4.3 Kernel-Based Nonlinear Extension of the Arps Decline Model

(KNEA)
In addition to the two typical kernel methods of SVM and RVM, we introduce a new
and less common kernel method named KNEA in this chapter. The Arps decline
model is currently the most widely used oil and gas production prediction model.
However, the essence of the Arps decline model is a univariate predictive model
because it only reflects the variation of the predicted content over time [69–74].
In 2016, Ma and Liu [75] used the kernel method to establish a predictive model
(see Fig. 4.3) that describes the nonlinear relationship between predicted content
and influencing factors. The model can be used not only for reservoir production
prediction, but also for predicting other energy systems with dynamic characteristics
such as solar radiation [76] and reference evapotranspiration [77]. Its expression is:
g(x) = ag(x − 1) + h(u(x)) + b (4.18)
The nonlinear influencing factor vector is mapped to the high dimensional feature
space to have a linear representation:
h(i (x)) = ω T ξ (u(x)) (4.19)
Equation (4.20) can be expressed as:
g(x) = ag(x − 1) + ω T ξ (u(x)) + b (4.20)

K(u(2),)
λ2
K(u(3),)
λ3
u(x) λ4 ag(x-1)
K(u(4),)
b Z-1
λn
g(x)
K(u(n),) +
Fig. 4.3 KNEA model [71]
In the parameter estimation of the nonlinear model, the following optimization

problem is established by using the kernel method:

N
min J (a, ω, e) = 0.5a 2 + 0.5||ω||2 + 0.5C ex2 (4.21)
x=2
s.t.g(x) = ag(x − 1) + ω T ξ (u(x)) + b + ex (4.22)
Establish the Lagrangian function of Eqs. (4.23) and (4.24):
L(a, ω, b, e, C) = J (a, ω, e)

N
− λx ag(x − 1) + ω T ξ (u(x)) + b + ex − g(x) (4.23)
x=2
The Karush–Kuhn–Tucker condition of Eqs. (4.23) and (4.24) is:

⎧ n
⎪
⎪ ∂L
=0=a− λx g(x − 1)
⎪
⎪ ∂a
⎪
⎪ x=2
⎪
⎪ ∂L n
⎪
⎨ ∂ω = 0 = ω −
⎪ λx ξ [u(x)]
x=2
⎪ ∂L n (4.24)
⎪
⎪ =0= λx
⎪
⎪
∂b
⎪
⎪
x=2
⎪
⎪
⎪
∂L
∂ex
= 0 = ex − λCx
⎩ ∂L
∂λx
=0
The following linear equation can be derived:

T

0 1n−1 b 0
= (4.25)
1n−1 R λ g2|n
T
where 1n−1 = [1, 1, . . . , 1]n−1 T
; λ = λ1 , λ2 , . . . , λn−1 ; g2|n =
[g(2), g(3), . . . , g(n)]T ; R = Ω + G + C1 In−1 ; G i j = g(i )g( j ); Ωi j = ξ (u(i )) ×
ξ (u( j )).
The matrix Ω can be represented by a kernel function K (·, ·) that satisfies the
Mercer condition, namely:

−[u(i )−u( j)]2
Ωi j = K (u(i ), u( j )) = exp 2σ 2
(4.26)
Therefore, the KNEA model can also be expressed as:

n
g(x) = ag(x − 1) + λ j K (u( j ), u(x)) + b (4.27)
i=2
The discrete function can be defined as:

n
φ(x) = λ j K (u( j ), u(x)) + b (4.28)
j=2
The solution of the KNEA model can be obtained by using the recursive method:

x
g(x) = a x−1 g(1) + a x−v φ(v) (4.29)
v
4.2.5 Decision Tree Method
Decision tree is a common supervised learning method, and it is often used in classi-
fication problems, as shown in Fig. 4.4 [78–83]. The typical model development in
the decision tree system is shown in Fig. 4.5. In this section, the theories of random
forests and extreme gradient boosting (XGBoost) are briefly introduced.
4.2.5.1 Random Forest
Random forest is an integrated learning method for classification and regression

(Fig. 4.6). It is one of the representative of ensemble learning, and it is an additive
model based on bagging algorithm. Different from bagging, when constructing each
tree, random forest uses a random sample predictor before each node segmentation,
which can reduce bias. It has the following characteristics: a) The introduction of two
Fig. 4.4 Decision tree

x<5
Yes No
y<3 y<6
Yes No
Yes No
1 2 x<9 5
Yes No
3 4
Decision tree
Random forest
Grandient XGBoost
Boosting tree
Fig. 4.5 From the decision tree to XGBoost [83]
randomness makes random forest not easy to fall into overfitting, and has excellent
noise immunity; b) It can process data of high dimension (many features) without
feature selection; c) It has fast training speed and is easy to be parallelized, so it is
relatively simple to implement [84–96]. More details about random forests can be
found in the literature [95].
Fig. 4.6 Random forest [97] treen
yn
tree2
y2 Averaging y
x
y1
tree1
4.2.5.2 XGBoost
XGBoost was proposed by Chen and Guestrin (2016) and is based on the C++
language [98–109]. The model has achieved great success since its appearance and
is always seen in the top models in various data mining competitions. XGBoost
can integrate multiple weak learning machines into one strong learning machine by
iterating and generating multiple trees, and it has the following features: (a) It can
automatically utilize the multithreading of the CPU for parallelism, while improving
the algorithm to improve accuracy, and this is the most prominent feature of XGBoost;
(b) It is a lifting learning algorithm based on the decision tree model and can process
sparse data automatically; (c) large amounts of data can be processed at high speed.
In the XGBoost model, the tree model adopts an additive model

K
ŷ = f k (xi ), f k ∈ F (4.30)
k=1
The objective function is

L(φ) = l ŷ, yi + Ω( f k ) (4.31)
i k
where Ω( f ) = γ T + 0.5λw 2 , w = (w1 , w2 , . . . , wk ).

Because learning all tree parameters at once is challenging, XGBoost uses an

additive strategy that learns the parameters of one tree at a time:
⎧
⎪
⎪ ŷi(0) = 0
⎪
⎪
⎪ (1) (0)
⎨ ŷi(2) = ŷi(1) + f 1 (xi )
⎪
ŷi = ŷi + f 2 (xi ) (4.32)
⎪
⎪ ..
⎪
⎪ .
⎪
⎪
⎩ (t)
ŷi = ŷi(t−1) + f t (xi )
The XGBoost algorithm uses the stepwise forward additive model as the gradient
boosting algorithm. The difference is that the gradient boosting algorithm is a negative
gradient that learns a weak learner to approximate the loss function. The XGBoost
algorithm first finds the second-order Taylor approximation of the loss function at
that point, and then minimizes the approximation loss function to train the weak
learner (Fig. 4.7). Therefore, the objective function can be expressed as
n

L (t) = l yi , ŷi(t−1) + f t (xi ) + Ω( f t ) (4.33)
i=1
Using the second-order Taylor expansion, the following function can be obtained
n

L (t) = gi f t (xi ) + 0.5h i f t2 (xi ) + Ω( f t ) (4.34)
i=1
4.2.6 Deep Learning
Deep learning is an emerging technology in the machine learning system. Although

it became popular in the 21st century, it actually appeared as early as the 1940s.
Deep learning refers to a method model that uses deep neural networks to implement
machine learning. Compared with other machine learning methods, deep learning
is a modeling structure that can automatically find features and patterns from data
and can capture highly complex behaviors. The motivation to study deep learning
is to establish neural networks, which can simulate the human brain for analysis
and training, and imitate the mechanism of the human brain to explain data, such as
images, sounds, and texts [110–116].
Some common deep learning models include convolutional neural networks
(CNNs) [117–131], deep neural networks (DNNs) [132–142], recurrent neural
networks (RNNs) [143–152], and long short-term memory (LSTM) [153–164].
DNNs mainly include deep belief networks (DBNs) [165–178], stacked autoencoder
(SAE) [179–190], and deep convolution neural networks (DCNNs) [191–199].
Node 2 prediction
Tree 1 splits to generate

residuals
Node 1 prediction Feature 1 division
value
Tree 1
Node 3 prediction
Node 3 prediction
Tree 2 splits to generate

residuals
Node 1 prediction Feature 2 division
value
Tree 2
Node 3 prediction
Fig. 4.7 XGBoost [97]
4.3 Optimizers
The traditional machine learning model usually needs to be used with the optimizer
because of its own limitations. Some commonly used optimizers include particle
swarm optimization algorithm, ant colony optimization algorithm, and some new
algorithms such as grey wolf optimization algorithm and whale swarm optimization
algorithm (see Fig. 4.8).
4.3.1 Fruit Fly Optimizer
The fruit fly optimizer (FFO) was first proposed by Pan (2012) [217]. It is an algorithm
that seeks global optimization based on the performance of fruit fly foraging behavior
4.3 Optimizers 129
Fig. 4.8 Frequently used

meta-heuristic algorithms
[200–215]. Note SCA is Sine
Cosine Algorithm; WOA is
whale optimization
algorithm; MFO is
moth–flame optimization;
GWO is grey wolf
optimization; BA is bat
algorithm; CS is cuckoo
search; PSO is particle
swarm optimization; ACA is
ant colony algorithm; SA is
simulated annealing; GA is
genetic algorithm [216]
[218–220]. The FFO has a simple optimization mechanism, only uses fewer input
parameters, has information-sharing characteristics, and has strong robustness and
adaptability. Although this algorithm has only appeared for about ten years, it has
been widely concerned and applied in various fields. Its flow chart is shown in Fig. 4.9.
Step 1: Initialize the parameters: The maximum number of iterative evolutions is

Gen_max, and the population size is set to Pop_size. The position of the population
of the initial randomly distributed fruit fly is X _axis, Y _axis.
Step 2: Assigning each fruit fly to any direction (RandomV alue) that seeks food
through the sense of smell.
X (i ) = X − axis + RandomV alue

(4.35)
Y (i ) = Y− axis + RandomV alue
where RandomV alue is a random number in the interval [0, 1].

Step 3: At first, the position of the food source cannot be accurately determined.
Therefore, it is necessary to first estimate the starting distance Dist between the
individual and the initial set point, and then bring it into the calculation value Si
for calculating the concentration, which is the reciprocal of Dist:
√
Dist = X (i )2 + Y (i )2 (4.36)
Start
Initialize fruit fly swarm location Init X_axis, Init Y_axis
Give the random direction and distance for X i X axis RandomValue

the search of food using osphresis Y i Y axis RandomValue
2 2
Calculate the distance to the origin (Dist) Dist X i Y i
and smell concentration judgment value (S) Si Dist -1
Substitute S into fitness function, and find

Smell i fit Si
the smell concentration Smell(i)
Find the maximal smell concentration value

bestSmellbestIndex max Smell
(bestSmell)
Smellbest bestSmell
Keep the bestSmell value and x,y coordinate X axis X bestIndex
Y axis Y bestIndex
Yes
Less than the Use vision and fly to the position
maximum iterations? with the highest smell concentration
No
Output optimal solution
End
Fig. 4.9 Flow chart of fruit fly optimizer [221]
Si = 1
Dist
(4.37)
Step 4: By substituting S i into the evaluation function of the taste concentration

(also called the fitness function), the concentration value Smell(i ) above each
fruit fly position can be calculated:
4.3 Optimizers 131
Smell(i ) = f it(Si ) (4.38)
Step 5: The optimal taste concentration judgment value best Smell is memorized
and stored in its X , Y coordinate position.
Step 6: The stored new individual position is placed in steps 2–5, and a value
of the best taste concentration in the fruit fly population is calculated, and the
concentration and position coordinates of the fruit fly are recorded. It is judged
whether the value of the optimal taste concentration is better than the value after
the last iteration.
Step 7: If the number of iterations is less than the set maximum number of iter-
ations Gen_max or the target precision of the pair is not achieved, steps 2–5 are
performed again.
⎧
⎨ Smellbest = best Smell
X b = X (bestindex) (4.39)
⎩ −
Y− b = Y (bestindex)
4.3.2 Grey Wolf Optimizer
Grey wolf optimizer is an intelligent algorithm put forward by Mirjalili et al. based
on the hierarchy and predatory behavior of wolves, as shown in Fig. 4.10. Wolves
system can be divided into four levels (see Fig. 4.11), the highest level is the head wolf
(α), followed by the substitute wolf (β) when the head wolf is not available, the third
level is the wolf (δ) who obeys the command of α and β, and the bottom level, ω, is
responsible for balancing the internal relations of the population. Take each grey wolf
individual in the population as a solution, select the wolves (α, β, δ, respectively) of
the current optimal solution, superior solution, and suboptimal solution. During the
hunting process, the wolves are guided by α, β, and δ to approach the food position:
→ = C→ · X→ p (t) − X→ (t)
D (4.40)
X→ (t + 1) = X→ p (t) − A→ · D
→ (4.41)
A = 2→
a · r→1 − a→ (4.42)
C→ = 2→
r2 (4.43)
D
Dα δ
Prey
δ
Dβ
R a3
a1
C3
C1
α
β
a2
C2
Fig. 4.10 Principle of the GWO algorithm [70]
Fig. 4.11 Hierarchy of grey

wolves [73]
where D → represents the distance between the grey wolf and prey; A→ and C→ represent
−→
coefficient vectors; X p and X→ represent the position vectors of the prey and grey
wolf, respectively; t represents the current iteration.
Subsequently, under the leadership of the wolf leader, the prey is hunted, and the
positions of the wolves are updated by the following mechanism:
4.3 Optimizers 133
⎧−
⎪ → ||− → − → → ||
⎪
⎪ D α = | C 1 · X α − X|
⎪
⎪ −
→ |−→ −
→ |
⎪ Dβ = |C2 · X β − X→ ||
⎪
⎪
|
⎪
⎪
⎨−→ ||− → − → |
|
Dγ = |C3 · X γ − X→ | (4.44)
⎪
⎪ −
→ − → − →− →
⎪ X 1 = X α − A 1 Dα
⎪
⎪
⎪ −
⎪
⎪
→ − → − →−
X 2 = X β − A 2 Dβ
→
⎪
⎪
⎩ −
→ −
→ −→ −→
X 3 = X γ − A 3 Dγ
−
→ −→ −→
−
→ X1+X2+X3
(4.45)
X p (u + 1) = 3
4.3.3 Whale Optimization Algorithm
Whale optimization algorithm (WOA) is a meta-heuristic algorithm proposed by

Seyedali Mirjalili and Andtew Lewis in 2016 based on whale predation [222, 223].
The predation method of whales is the bubble net predation method. First, the whales
sneak into the deep water, move upwards in a spiral path, and constantly spit out
bubbles of different sizes. A series of bubbles form a bubble net and surround the
small fish or shrimp. Finally, the prey is eaten by the whale.
Through practice, WOA has the advantages of simple operation and few param-
eters. In this algorithm, there are three stages: search for prey, encircling prey and
bubble-net predation. Suppose the best candidate solution is the target prey or close to
the optimal solution. Therefore, after defining the best whale position, other whales
will swim towards the whale’s position to update their position. The distance between
the whale individual and the optimal whale position is
| |
→ = ||C→ · −
D
→ |
X ∗ (t) − X→ (t)| (4.46)
−
→
where t is current iteration; X ∗ (t) is the location of the best whale in the t generation;
X→ (t) is the position of the individual whale in the t generation; C→ is the oscillation
factor, its expression is
C→ = 2 · r→ (4.47)
Whales are updated according to the location of humpback whales

−
→
X→ (t + 1) = X ∗ (t) − A→ · D
→ (4.48)
where A→ is convergence factor, its expression is

A→ = 2→
a · r→ − a→ (4.49)
where r→ is a random number between [0, 1].

There are two strategies for whale local search stage. The first strategy is the
shrinking encircling mechanism, in which the location updating of whales is achieved
by Eq. (4.48). The range of A→ at this stage is realized from 2 to 0 as a→ decreases
linearly. A→ represents a random number between [−a, a]. The second strategy is spiral
updating position. The whale first calculates its distance to its prey, then spirals up
and spits out bubbles. The mathematical expression of the predatory behavior is
→' ||−
− → |
|
D = | X ∗ (t) − X→ (t)| (4.50)
−
→' −
→ −
→
X (t + 1) = D ' · ebl · cos 2πl + X ∗ (t) (4.51)
−
→
where D ' is the distance from the i-th whale to the prey (optimal solution), l is a
random value between [−1, 1]; b is a spiral constant.
Since whales have two predation strategies, assuming that the probability of
adopting one of them is 50%, the mathematical model is
−
→' bl −
→
→ D · e · cos 2πl + X ∗ (t) p ≥ 0.5
X (t + 1) = −
→ (4.52)
X ∗ (t) − A→ · D
→ p < 0.5
where p is a random number between [0, 1].

In order to avoid local optimum, the whale will also enter the global search phase,
the mathematical model of this phase is
| |
D→ = ||C→ · X→ rand (t) − X→ ||
(4.53)
X→ (t + 1) = X→ rand − A→ · D →
where X→ rand is the location of a random whale in the current population.
4.3.4 Nondominated Sorting Genetic Algorithm II
The Nondominated sorting genetic algorithm (NSGA) was proposed by Srinivas and
Deb in 1993 [224–228]. It is based on genetic algorithm and introduces two core
techniques to achieve multi-objective optimization: (1) Classify individuals using
the principle of non-dominated sorting; (2) Calculate the virtual crowding distance
of each body. It has strong search ability and good robustness. However, the NSGA
algorithm has the disadvantage of high computational complexity. In 2002, Deb
et al. proposed the NSGA-II algorithm, which reduced the algorithm’s computational
4.3 Optimizers 135
Fast non-dominated sorting and virtual crowding

Start distance calculation for contemporary population Pt
The first N individuals are selected to produce the

Parameters initialization
parent population Pt+1
Randomly generate initial population P0

No
Is the maximum number of iterations
reached?

distance calculation for contemporary population Pt
Yes
Genetic manipulation End
The sub population Qt is obtained
Rt=Pt+Qt
Fig. 4.12 Implementation process of the NSGA-II algorithm [229]
complexity by introducing fast non-dominated sorting technology and elite retention

technology [224]. Its implementation process is shown in Fig. 4.12.
In Fig. 4.12, the virtual crowding distance calculation can make the solution
set evenly distributed in the target space and ensure global optimization. In the
two-objective problem, the crowding distance can be calculated by Eq. (4.54).
D[i + 1] · f 1 − D[i − 1] · f 1
D[i]d =
f 1max − f 1min
f 2 −D[i−1]· f 2
+ D[i+1]·
f max − f min (4.54)
2 2
where D[i]d is the d-th objective function value of the i-th individual; f mmax and
f mmin are the maximum and minimum values of the population under the objective
function m, respectively.
4.3.5 Multi-objective Grey Wolf Optimizer
Multi-objective grey wolf optimizer (MOGWO) is developed based on the GWO

[230, 231]. MOGWO has two changes compared to GWO. First, the update method
has changed, and an archive is introduced to store the current best individual. After
each iteration, the new individual generated is compared with the individual in the
archive. In addition, to avoid too many similar individuals, all individuals are grouped
according to the distance of the objective function value. Secondly, the selection
mechanism of the leader wolf has changed. That is, using roulette to directly select
the leader wolf in the archive, solving the problem that it is difficult to directly
determine three non-dominant solutions through Pareto method. The probability of
each hypercube can be calculated by Eq. (4.55). More information can be found in
the literature [230].
Pi = 1
L ic (4.55)
where c is a constant; L i is the number of Pareto optimal solutions; Pi is the

probability of the hypercube.
4.3.6 Multi-objective Salp Swarm Algorithm
The salp swarm algorithm (SSA) is a meta-heuristic algorithm proposed in 2017

to simulate the movement and predation behavior of salp in the ocean (Mirjalili
et al., 2017) [232–234, 235]. The algorithm has the characteristics of simple struc-
ture, strong search ability, and strong robustness. The optimization strategy of SSA
is unique. Some meta-heuristic algorithms that have been frequently used in recent
years, such as grey wolf algorithm and whale optimization algorithm, are used to
search for targets in a “group” manner, while SSA is in the form of “chain”. Thus,
the salp swarm leader does not directly lead other individuals to approach the prey,
but directly affects the individuals next to them, thus forming a chain effect (see
Fig. 4.13). The salp swarm has a stricter hierarchical system and cannot be managed
beyond the level. Due to the biological characteristics of salp swarm, in the opti-
mization of mathematical problems, this idea makes the optimization individuals
have diversity.
The leader’s location update strategy is:

Fs + 2e−16l /L [c2 ubs − c2 lbs + lbs ], c3 ≥ 0
2 2
Ps1 = (4.56)
Fs − 2e−16l /L [c2 ubs − c2 lbs + lbs ], c3 < 0
2 2
where 2e−16l /L can be defined as c1 , which is the most critical parameter in SSA;
2 2
Fs is location of the food in the s-th dimension; c2 ,c3 are random numbers between
4.3 Optimizers 137
Fig. 4.13 Salp chain [235] Salp leader
0 and 1; lbs is lower bound in the s-th dimension; ubs is upper bound in the s-th
dimension; L is maximum number of iterations; l is current iteration.
According to Newton’s theorem of motion, the position update strategy of salp
followers is:
Psi = 21 at 2 + v0 t (4.57)
where Psi is location of the salp leader in the s-th dimension; a is acceleration; t is
time; v0 is initial speed.
The difference between each iteration is 1 and the initial speed is 0, Eq. (4.57)
can be simplified as:
Psi = 21 Psi + 21 Psi−1 (4.58)
where Psi−1 is location of the salp follower in the s-th dimension.

SSA can only store an optimal solution, and cannot solve multi-objective prob-
lems. Based on SSA, multi-objective salp swarm algorithm (MOSSA) is developed,
which can store non-dominated solutions to solve multi-objective problems. There
are two key techniques in the MOSSA’s optimization process: (1) Equipped with
repository to store non-dominated solutions, and use Pareto dominance operator
to compare the stored search agents; (2) use roulette wheel strategy to select food
source. More detailed theory is shown in the literature (Mirjalili et al., 2017). The
implementation process of MOSSA is shown Fig. 4.14.
Start
Initialize the salp swarm
Calculate the fitness for each

salp
Find non-dominated salps and

initialize the repository
Yes Delete a salp and add the

Is the repository full?
non-dominated salp
No End
Select a food source Yes
No
Reach the maximum iteration?
Update c1
Update salp's position
Yes Update the position of salp Amend search agent based on

Is i equal to 1?
leader according to Eq. (5.73) variable upper and lower bounds
No
Update the position of salp
followers according to Eq. (5.75)
Fig. 4.14 The implementation process of MOSSA [236]
4.4 Application Scenarios
4.4.1 Fault Diagnosis
Pipeline fault diagnosis is to judge the abnormal operation of the pipeline through
monitoring, which aims to find out the cause of the fault in advance to avoid the
occurrence of major accidents [237]. In the literature review, the application of
machine learning in pipeline fault diagnosis is mainly divided into leak inspection
and abnormal diagnosis. In the research of leak inspection, three pipelines of oil, gas,
and water are mainly targeted. For instance, Chen et al. used support vector machine
to detect oil pipeline leakage based on the negative pressure wave signal. Practice
shows that the recognition rate can reach 90.33% [238]. Isa and Rajkumar employed
4.4 Application Scenarios 139
Fig. 4.15 The technical

route of ML in pipeline fault Data collection Fault diagnosis
diagnosis
Data processing Model testing
Feature ML model
extraction training
discrete wavelet transform to extract features and utilized support vector machine
to classify the attenuation and frequency changes of the propagating Lamb waves
to identify the thinning of the pipe wall thickness [239]. Mandal et al. proposed a
hybrid model combining artificial bee colony and support vector machine to iden-
tify the leakage of the liquid pipeline, and the recognition accuracy is up to 95.19%
[240]. El-Abbasy et al. utilized various machine learning models to detect oil and
gas pipeline leaks, including artificial neural network, support vector machine, and
linear regression. Among them, artificial neural network has the highest accuracy,
reaching 95.0-98.4% [241]. In recent years, some scholars have adopted models
based on decision trees. Some common models include extreme gradient boosting,
random forest, gradient boosting machine, and adaptive boosting. Scholars usually
use the technical route shown in Fig. 4.15 for fault diagnosis research. To verify
the validity of the model, many scholars use experiments, and a few scholars use
numerical simulations. A general connection diagram of experimental equipment is
shown in Fig. 4.16. Liang et al. [242] designed a numerical simulation model for
pipeline leak inspection in Flowmaster software, as shown in Fig. 4.17, which can be
used as a typical case. Information on some representative studies is shown in Table
4.1.
4.4.2 Risk Prediction
Some scholars used machine learning to conduct research around pipeline risk predic-
tion, and the research content is diverse [273]. Zhu and Zou used GA to optimize
the pipeline’s risk control plan by establishing an optimization objective function
[274]. Parvizsedghy and Zayed used a Neurofuzzy model to predict the economic
consequences of gas pipeline failure with an accuracy rate of 80% [275]. Bagriacik
et al. used LR, RF, and BRT to predict the damage to water pipelines caused by earth-
quakes. The results show that BRT has the best predictive performance [276]. Jiang
and Dong used MLP, RBFNN, SVM and GP to analyze the collision failure risk of
oil and gas pipelines [277]. Giraldo-González and Rodríguez used GBT, SVM, ANN
Fig. 4.16 Schematic Reservoir

diagram of pipeline fault
diagnosis experiment (Note
Lines represent pipes)
Pump Pressure
gauge
Test pipe section
Sensor Transmitter
Oscilloscope Function generator
Pressure
Leak point
Upstream measure node 2 Downstream
Regulating valve
reservoir reservoir
Pressure
measure node 1 Water recovery
Leak valve pond
Fig. 4.17 Leak simulation model
and Bayes to predict the failure of water pipelines [278]. Among them, GBT has the
best prediction performance, with an accuracy rate of 99.52–99.79%. Information
on some representative studies is shown in Table 4.2.
4.4.3 Condition-Related Parameter Prediction
Condition-related parameter prediction refers to forecasting parameters related to

pipeline states, such as corrosion rate, burst pressure, erosion, and deformation.
Specifically, Mashford et al. used support vector machine to predict the grade of sewer
status and obtained 92% accuracy [292]. Shi et al. predicted the pipe’s remaining
wall thickness by multiple machine learning models, such as multiple linear regres-
sion, artificial neural network, random forest, support vector machine, and a stacking
Table 4.1 The application of machine learning in pipeline fault diagnosis

Ref Objective Pipe Model Data size Proportion Error index
type of training
set
[238] Leak Oil SVM 1500 / MAPE =
inspection 9.67%
[239] Abnormal Oil, DWT-SVM / / MAPE =
diagnosis gas 10.35%
[242] Leak Oil WPA-PCA-GMM-BIC / 72.2% /
inspection
[240] Leak Liquid ABC-SVM 400 50.0% MAPE =
inspection 4.81%
[243] Leak Oil PSO-SVM 20 80.0% RMSE =
inspection 0.39
[244] Leak Water ISLMD / / MAPE =
inspection 3.71%
[241] Leak Oil, Linear regression, 1540-4990 / Linear
inspection gas ANN, DT regression:
MAPE =
1.70-7.30%
ANN: MAPE
= 1.60-5.00%
DT: MAPE =
1.80-9.30%
[245] Leak Gas LMD-SVM / / MAPE =
inspection 2.41-3.67%
[246] Leak / Bagging, Boosting / / /
inspection
[247] Leak Oil LMD-LSTSVM / / MAPE =
inspection 5.56%
[248] Fault Water WE-EDNN / / MAPE =
inspection 4.00%
[249] Leak Steam RF, XGBoost, / 70.0% MAPE =
inspection LDA-KNN, SVM 1.00%-6.20%
[250] Leak / BPNN 49 61.2% MAPE =
inspection 1.01%
[251] Leak Fluid PCA-GA-SVM / / MAPE =
inspection 2.27%
[252] Defect/leak Oil, SVM, GBM / 70.0% MAPE =
inspection gas 5.00%
[253] Leak Water PCA-SVM 100 50.0% MAPE =
inspection 2.00%
[254] Leak Water LR, GNB, SVM, MLP / / MAPE =
inspection 4.07-37.95%
(continued)

type of training
set
[255] Leak Water DNN 200 50.0% /
inspection
[256] Leak Oil, DO-SVM 416 91.4% MAPE =
inspection gas, 4.00%
water
[257] Leak Oil SMDPSO-SVM 470 63.8% MAPE =
inspection 0.58%
[258] Abnormal Oil, ANN, DT, SVM / 88.9% ANN: MAPE
diagnosis gas = 3.12%
SVM: MAPE
= 1.38%
DT: MAPE =
2.35%
[259] Abnormal Oil, ANN, SVM 460 75.0% MAPE <
diagnosis gas 10.00%
[260] Abnormal Gas SVM / / MAPE =
diagnosis 13.80%
[261] Leak Water SVM, RVM 191 89.5% SVM: MAPE
inspection = 0.08%
RVM: MAPE
= 0.54%
[262] Leak Heat XGBoost 10,609 70.0% MAPE =
inspection 14.15%
[263] Abnormal / RF / / MAPE =
diagnosis 4.41%
[264] Defect Water Linear model / / RMSE < 0.06
inspection
[265] Leak Gas KNN, SVM, XGBoost, 1680 80.0% MAPE =
inspection AdaBoost, RF 0.40%
[266] Leak Water RF, DT, ANN, SVM 5607, 1869 70.0% MAPE =
inspection 15.00%
[267] Leak Oil MLTA / / MAPE =
inspection 1.40%
[268] Abnormal Oil ANN, SVM 268 70.0% ANN: MAPE
diagnosis = 11.11%
SVM: MAPE
= 16.05%
(continued)

type of training
set
[269] Leak Gas ANN, SVM, RF 2000 75.0% ANN: MAPE
inspection = 0.60%
SVM: MAPE
= 0.40%
RF: MAPE =
0.60%
[270] Leak Water GMM 120 / /
inspection
[271] Leak Water ANN 26,040 70.0% MAPE =
inspection 0.47%
[272] Leak Oil, SVM 720 60.0% MAPE =
inspection water 5.00%
ensemble model. Results reveal that the stacking ensemble model has the best perfor-
mance, with an RMSE of 0.1302 [293]. Some tree-based models are used to predict
the condition distribution and condition rating for sewers, such as RF and GBT. Lu
et al. used a combined model based on a multi-objective optimization algorithm to
predict oil and gas pipelines’ burst pressure while considering the prediction accu-
racy and stability [236]. Peng et al. used a hybrid model that combines principal
component analysis, chaos particle swarm optimization, and support vector machine
to predict the corrosion rate of submarine pipelines, and the prediction accuracy
reached 91.7% [294]. Information on some representative studies is shown in Table
4.3. It shows that in condition-related parameter prediction, in addition to oil, gas,
and water pipelines, some scholars have conducted research on sewers.
4.4.4 Visual Defect Recognition
Visual defect recognition means the recognition of defects in the pipeline based
on computer vision technology. It is usually necessary to use the Closed-Circuit
Television Video (CCTV) inspection method to obtain the image in the pipeline
[304]. As shown in Table 4.4, related research started more than ten years ago, but at
that time, traditional machine learning models, such as ANN and SVM, were used,
and the data scale was small. Since 2018, related research has entered a period of rapid
development, and some scholars have adopted deep learning models, such as CNN,
Faster R-CNN, and YOLO. They can be applied to larger data scales. Moreover,
almost all scholars’ research objects are sewers, and relevant research has not been
carried out in oil and gas pipelines. A general architecture for automatic inspection
of pipeline defects can be seen in Fig. 4.18.
Table 4.2 The application of machine learning in pipeline risk prediction

Refs Objective Pipe type Model Data size Proportion of Error index
training set
[274] Risk level Gas GA / / /
prediction and
control
[279] Failure / SVM / 50.0-90.0% MAPE =
probability 3.80-12.30%
prediction
[280] Failure risk Oil, gas ANN / / /
prediction
[275] Prediction of Gas NFM / 85.0% MAPE =
financial 20.00%
consequences
of failure
[281] Failure Water HBP / / /
likelihood
prediction
[276] Earthquake Water LR, RF, BRT / 90.0% MAPE =
damage 22.00-40.00%
prediction
[282] Risk Oil, gas DT, ANN / / DT: MAPE =
prediction 0.73%
ANN: MAPE
= 1.65%
[283] Risk Oil, gas BN / / /
identification
[284] Failure Water BDT 3743 50.0% MAPE =
prediction 4.00%
[285] Failure rate Water ELM 9508 75.0% RMSE =
prediction 0.09-0.18
[277] Collision Oil, gas MLP, 100 80.0% MAPE =
failure risk RBFNN, 4.00%
analysis SVM, GP
[286] Risk level Gas k-means, / 66.7% /
assessment spectral
clustering,
Hierarchical
clustering,
DBSCAN
[287] Leak and burst Oil, gas PSO-SSCN 3250 40.0-90.0% MSE =
failure 0.02-0.03
prediction
(continued)
4.5 Application Summary 145

Refs Objective Pipe type Model Data size Proportion of Error index
training set
[288] Break Water Regression / 80.0% /
prediction models,
SVM, RF,
LDA
[289] Failure Water LR, SVM 4393 71.4% LR: MAPE =
prediction 0.60-35.30%
SVM: MAPE
=
0.60-41.10%
[278] Failure Water GBT, SVM, / 70.0% GBT: MAPE
prediction ANN, Bayes = 0.21-0.48%
SVM: MAPE
= 0.17-0.53%
ANN: MAPE
= 0.39-1.01%
Bayes: MAPE
= 5.17-6.31%
[290] Break Water XGBoost 61,613 / RMSE =
prediction 12.53
[291] Burst failure Oil, gas KNN, DT, 959 70.0% MAPE =
risk prediction RF, 14.00-26.00%
AdaBoost,
XGBoost,
LGBoost,
CatBoost
4.5 Application Summary
4.5.1 Model Category
According to Tables 4.1, 4.2, 4.3 and 4.4, all used models can be divided into six
categories: SVM-based, ANN-based, tree-based, regression model, deep learning
model, and others, as shown in Fig. 4.19. Based on the number of times they are used,
statistical information as shown in Fig. 4.20 can be obtained. It indicates that SVM-
based models occupy a large proportion in the four research aspects, especially in fault
diagnosis; the application frequency of tree-based models is also high, but they are
not used in visual defect inspection; the application of regression model is relatively
small, and it cannot be applied to visual defect inspection; deep learning models
have a high proportion of use in visual defect inspection, it has a few applications
in fault diagnosis, but not in the remaining two aspects. Table 4.5 summarizes the
characteristics of these model categories.
Table 4.3 The application of ML in condition-related parameter prediction

Refs Objective Pipe Model Data size Proportion Error index
type of training
set
[292] Condition Sewer SVM 1441 75.0% MAPE =
grade 8.00%
prediction
[295] Condition Oil, ANN 1540–4990 80.0% MAPE =
grade gas 1.60%
prediction
[296] Condition Sewer RF 356 70.0% MAPE =
grade 10.00%
prediction
[297] Condition Sewer ANN, SVM 745 81.9% ANN: MAPE
grade = 21.50%
prediction SVM: MAPE
= 27.40%
[293] Remaining Water MLR, ANN, RF, / 80.0% RMSE =
wall SVM, SEM 0.13–0.15
thickness
prediction
[298] Condition Sewer RF 97,547 60.0% MAPE =
distribution 33.33%
prediction
[299] Condition Sewer GBT, KNN 19,766 80.0% GBT: MAPE =
rating 12.60%
prediction KNN: MAPE
= 16.60%
[300] Deformation Water MLR, RT, CIT, MLP, 973 75.0% RMSE < 0.01
prediction SVM, KNN
[301] Condition Sewer GBT 19,766 80.0% MAPE =
rating 12.60%
prediction
[236] Burst Oil, MOSSA-RVM 453 90.0% MAPE =
pressure gas 3.72%
prediction
[294] Internal Oil, PCA-CPSO-SVM 60 85.0% MAPE =
corrosion gas 8.30%
rate
prediction
[302] Erosion Oil, RF 152 70.0% RMSE < 0.01
prediction gas
[303] Burst Oil, PCA-MOGWO-SVM, 453 90.0% MAPE < 4.00%
pressure gas PCA-NSGA-II-SVM
prediction
Table 4.4 The application of machine learning (or deep learning) in visual defect recognition
Ref Pipe type Model Image amount Proportion of Accuracy
training set
[305] Sewer BPNN, RBFNN, 291 / 60.00%
SVM
[306] Sewer RBFNN, MPN, 1529 90.0% 96.00%
SVM
[307] Sewer Faster R-CNN 4000 75.0% 83.00%
[308] Sewer CNN 12,000 62.5% 86.20%
[309] Sewer CNN 47,072 75.0% 96.33%
[310] Sewer CNN 18,333 70.0% 64.80%
[311] Water DFP-SVM 2000 70.0% 92.81%
[312] Sewer YOLOv3 3664 75.0% 85.37%
[313] Sewer SSD, YOLOv3, 3800 80.0% SSD: 54.40%
Faster R-CNN YOLOv3: 74.50%
Faster R-CNN:
76.20%
[314] Sewer SVM 8952 / 90.00%
[315] Sewer Faster R-CNN 3600 90.0% 94.40%
[316] Steam A neural immune 1113 50.0% 98.38%
ensemble learning
algorithm
Step 1: Video files Step 2: Image Step 3: Image

collection extraction preprocessing and
segmentation
Step 6: Output Step 5: Defect Step 4: Feature

detection and extraction
classification
Fig. 4.18 A general architecture for automatic inspection of pipeline defects

SVM-based ANN-based
ELM
NFM
BPNN
SVM RVM RBFNN
SSCN
MLP MPN
BN
Tree-based
GBT
DT Regression Deep learning
XGBoost
CNN
Linear R-CNN
CatBoost
MLR
AdaBoost YOLO
LR DNN
LGBoost
BDT Others
KNN HBP
RT MLTA
RBT GMM
Bagging
CIT GP LDA SEM
RF
Fig. 4.19 Classification of models
4.5.2 Model Framework
Some scholars use the most primitive models, and some are combined models that
combine multiple techniques. According to the model framework, models can be
divided into four categories: (1) Data preprocessing plus primitive machine learning
model (named as D-M form, such as DWT-SVM); (2) Optimizer plus primitive
machine learning model (named as O-M form, such as ABC-SVM); (3) Data prepro-
cessing plus optimizer plus primitive machine learning model (named as D-O-M
form, such as PCA-CPSO-SVM); (4) Primitive machine learning model (named as
M form, such as SVM). Statistics are carried out on four research aspects, as shown
in Fig. 4.21. It implies that the models with M-form framework account for the
majority in the four research aspects; in risk prediction, no author uses D-M and
D-O-M frameworks; in condition-related parameter prediction, no author uses D-M
framework; in visual defect recognition, only one paper uses O-M framework, and
the rest are M-form framework. Therefore, most scholars are still using the original
(a) (b)
15 (25.42%)
13 (35.14%) 3 (8.11%)
10 (16.95%)
3 (5.08%)
2 (3.39%) 7 (18.92%)
5 (8.47%)
9 (24.32%) 5 (13.51%)
24 (40.68%) SVM-based
ANN-based
Tree-based
Regression model
Deep learning model
Others
8 (33.33%) 8 (44.44%)
(c) (d)
2 (8.33%)
4 (16.67%) 3 (12.5%) 2 (11.11%)

4 (22.22%)
7 (29.17%) 4 (22.22%)
Fig. 4.20 The proportion of the various types of models in the literature. a Fault diagnosis; b risk
prediction; c condition-related parameter prediction; d visual defect recognition
Table 4.5 Advantages and disadvantages for different categories of models

Prediction model Advantages Disadvantages
SVM-based models It has good adaptability to small It has slow training speed for large
sample problems sample problems, and is sensitive
to missing data and kernel function
selection
ANN-based models It has strong nonlinear fitting The interpretability of the model is
ability, robustness, nonlinear poor, and it is easy to overfit
mapping ability, and self-learning
ability
Tree-based models It has strong generalization If there is noise in the data, it is
ability, fast training speed, and prone to overfitting
insensitive to missing data
Regression models The model is simple, The selection and expression of
interpretable, and fast in modeling factors are based on experience and
are limited in application
Deep learning models It has strong adaptability, good Training is time-consuming, the
portability, and has advantages in model is complex, and the
computer vision hardware cost is high
(a) (b)
47 (79.66%)
38 (97.44%)
3 (5.08%) 1 (2.56%)
8 (13.56%)
1 (1.69%)
D-M form
D-O-M form
M form
O-M form
(c) (d)
21 (87.5%)
17 (94.44%)
1 (4.17%) 1 (5.56%)
2 (8.33%)
Fig. 4.21 Proportion of models for various frameworks. a Fault diagnosis; b risk prediction; c
condition-related parameter prediction; d visual defect recognition
ML model to carry out related research. However, some traditional models have
limitations, as shown in Table 4.5, which indicates that there is still great room for
developing relevant models in the future.
4.5.3 Data Size and Data Division
Table 4.6 reveals the statistical parameters of the data size in machine learning appli-
cations. It indicates that the maximum and minimum data sizes used in different
studies are quite different. Because the amount of data used in statistics is limited,
the median can reflect the data size that is routinely used. Scholars can refer to Table
4.6 when applying. Moreover, in machine learning applications, the ratio of the
training set to the test set is also crucial. Figure 4.22 demonstrates that the proportion
of the training set is between 60 and 90% in most studies.
Table 4.6 Statistics of data size

Research aspect Minimum Maximum Median
Fault diagnosis 20 26,040 470
Risk prediction 100 61,613 3743
Condition-related parameter prediction 60 97,547 745
Visual defect recognition 291 47,072 3732
Fig. 4.22 The proportion of

training set
4.5.4 Input Variable
Input variables are the core of machine learning modeling, and many scholars have
been exploring the selection of input variables. For different research, the emphasis
of input variables is also different. Table 4.7 lists the inputs used in machine learning
modeling for fault diagnosis research. Most studies are based on wave signals, pres-
sure, and flow rate. The research based on wave signals is the most, and negative
pressure wave and acoustic emission signals are the most widely used. Tables 4.8 and
4.9 list the input variables used in ML modeling for risk prediction and condition-
related parameter prediction, respectively. These variables can be roughly divided
into pipe parameters, operating parameters, failure parameters, and environmental
parameters, as shown in Fig. 4.23. For visual defect recognition, the input of the
model is CCTV image.
Table 4.7 Input of machine learning modeling in fault diagnosis

Ref Wave Pressure Flow Temperature Strain Vibration Pipe Others
signal rate parameter
√
[238]
√
[239]
√
[242]
√ √
[240]
√ √
[243]
√
[244]
√
[241]
√
[245]
√
[246]
√
[247]
√
[248]
√
[249]
√
[250]
√
[251]
√
[252]
√
[253]
√
[254]
√
[255]
√ √ √
[256]
√
[257]
[258] Drilling
parameter
[259] Drilling
parameter
√
[260]
√
[261]
√ √
[262]
√
[263]
√
[264]
√
[265]
√
[266]
√
[267]
[268] Drilling
parameter
√
[269]
√
[270]
√ √
[271]
(continued)

Ref Wave Pressure Flow Temperature Strain Vibration Pipe Others
signal rate parameter
√
[272]
Table 4.8 Input of ML modeling in risk prediction

Ref Pipe parameter Operating Failure Environmental Others
parameter parameter parameter
[274] Loss
√ √
[279]
√ √ √
[280]
√ √
[275]
√ √ √
[281]
√ √
[276]
√
[282]
√ √
[283] Machinery hits
√ √ √
[284]
√ √
[285]
√
[277]
√ √
[286] Construction
parameter
√ √ √
[287]
√ √ √
[288]
√ √ √
[289]
√ √ √
[278]
√ √ √
[290]
√ √
[291]
4.5.5 Error (Accuracy) Indicator
One of the most important tasks after machine learning modeling is error (accuracy)
evaluation. This chapter summarizes the commonly used error (accuracy) indica-
tors for scholars’ reference. In fault diagnosis, risk prediction, and condition-related
parameter prediction, machine learning model is used for regression, which simu-
lates structured data. Equations (4.80)–(4.92) are three error indicators commonly
used in machine learning modeling in these three fields. The smaller their value, the
higher the accuracy of the model. Mean absolute percentage error has the highest
frequency of use. Table 4.10 shows the statistical information of the mean absolute
percentage error for machine learning models in different research areas. Note that
papers that do not show accuracy or do not use accuracy indicators are excluded
Table 4.9 Input of ML modeling in condition-related parameter prediction

Ref Pipe parameter Operating parameter Failure parameter Environmental parameter
√ √
[292]
√ √
[295]
√ √ √
[296]
√
[297]
√ √ √
[293]
√ √
[298]
√ √
[299]
√ √
[300]
√ √ √
[301]
√
[236]
√ √ √
[294]
√ √
[302]
√
[303]
Pipe parameter
Diameter
Wall thickness
Defect parameters
Pipe material
Pipe age
Joint type
Operating Failure parameter

parameter
Break
Pressure Pipe Safety factor
Temperature Leak
Flow rate
Environmental
parameter
Soil parameter
Season
Location
Disaster
Surrounding buildings
Fig. 4.23 Four parameters related to pipelines

Table 4.10 Statistics information of MAPE for ML models in fault diagnosis, risk prediction, and
condition-related parameter prediction
Research aspect Minimum (%) Maximum (%) Average (%)
Fault diagnosis 0.08 16.05 5.30
Risk prediction 0.17 41.10 10.45
Condition-related parameter prediction 1.60 33.33 13.30
from the statistics. It demonstrates that the average mean absolute percentage error
of machine learning model in fault diagnosis research is the lowest, reaching 5.30%.
In 1982, Lewis rated the prediction performance based on MAPE (see Fig. 4.24).
The MAPE less than 10% can be considered as “excellent”, the MAPE between 10
and 20% can be evaluated as “good”, and the prediction performance is “reasonable”
when the MAPE is in the range of 20–50%. If the MAPE is greater than 50%, the
prediction result is “inaccurate”.
In visual defect recognition, machine learning model is used for classification,
which simulates unstructured data. Three indicators shown in Eqs. (4.82)–(4.84)
Fig. 4.24 Prediction

accuracy level based on 0
MAPE recommended by
Lewis
Excellent
10%
Good
20%
Reasonable
50%
MAPE
Inaccurate
are always utilized for evaluating accuracy. Acc has the highest frequency of use.
Its minimum, maximum, and average values are 54.40%, 98.38%, and 82.31%,
respectively.
n |
|
| Rk −Pk |
MAPE = 100%
n | Rk | (4.59)
k=1

n
MAE = 1
n
|Rk − Pk | (4.60)
k=1
/

n
RMSE = 1
n (Rk − Pk )2 (4.61)
k=1
Acc = T P+T N
T P+F P+F N +T N
(4.62)
Pr = TP
F P+T P
(4.63)
RC = TP
T P+F N
(4.64)
/
n
2
Rk −Pk
RMSPE = 1
n Rk
(4.65)
k=1
√ 1 n
(Rk −Pk )2
√
U1 = 1 nn k=1 √ 1 n (4.66)
2
n k=1 Rk + n k=1 Pk2
√n
(Rk −Pk )2
U2 = √k=1
n 2
(4.67)
k=1 Rk

n
Dstat = 1
n
ak × 100% (4.68)
k=1

1, i f [R(k + 1) − R(k)][P(k + 1) − R(k)] ≥ 0
where ak =
0, other wise
| |
100% | Rk −Pk |
MaxAPE = max n | Rk | (4.69)
k

n
MSE = 1
n (Rk − Pk )2 (4.70)
k=1
/
UI =
n
k=1( Rk+1
U
−Pk+1
U
)2 + nk=1 ( Rk+1
L
−Pk+1
L
)2 (4.71)
n n
k=1 ( Rk+1 −Rk ) + k=1 ( Rk+1 −Rk )
U U 2 L L 2
n |
|
| Rk −Pk |
Accuracy = 1 − 100%
n | Rk | (4.72)
k=1
n
k=1 (Rk −Pk )
2
R2 = 1 − n (4.73)
( )
2
k=1 R k −R

n
SSE = (Rk − Pk )2 (4.74)
k=1
n
k=1 (Rk −Pk )
2
IA = 1 − /
n (4.75)
(| Pk −R |+| Rk −R |)
2
k=1
/
n
2
SDE = 1
n
Ek − E (4.76)
k=1
/

n
SRMSE = 1
P
1
n (Rk − Pk )2 (4.77)
k=1

n
2|Rk −Pk |
SMAPE = 100%
n |Rk |+|Pk | (4.78)
k=1

n
BIAS = 1
n (Rk − Pk ) (4.79)
k=1

n
NBIAS = 1
P
1
n (Rk − Pk ) (4.80)
k=1

n
NMAE = 1
P
1
n
|Rk − Pk | (4.81)
k=1
| |
n
| k |
MAAPE = 1
n
arctan| RkR−P
k
| (4.82)
k=1
n
k=1 Ik,α
PICP = n
(4.83)

n
ck = 1, Rk ∈ [L k , Uk ]
FICP = 1 − β − 1
ck (4.84)
n
k ck = 0, other wise

n
FINAW = 1
nr (Uk − L k ) (4.85)
k
√ 1 n
(Pk −Rk )2
TIC = √ 1 nn k=1 √ 1 n (4.86)
n
2
k=1 Pk + n k=1 Rk2
n
k=1 (Rk −Pk )
2
NMSE = n (4.87)
( )
2
k=1 R k −R
n

1−σ̂k2 2
HMSE = 1
n RVk (4.88)
k=1
n
ARV I = k=1 ( Rk+1
U
−Pk+1
U
)2 + nk=1 ( Rk+1
L
−Pk+1
L
)2 (4.89)
n n
k=1 ( Rk+1 −Rk ) + k=1 ( Rk+1 −Rk )
U U 2 L L 2
B1 +B2
HR = n
(4.90)
where MAPE is mean absolute percentage error; Rk is observation at time k; Pk is

forecasting value at time k; MAE is mean absolute error; RMSE is root mean square
error; Acc is defect recognition accuracy, which means the ratio of correctly classified
images to the total number of images; Pr is precision, which means true positive rate;
RC denotes recall, which means sensitivity; T P is the number of true positives; T N
is the number of true negatives;
T P is the number of false positives;
FN is the
number of false negatives; PkL , PkU is k-th predicted intervals; RkL , RkU is k-th
real intervals; σ̂k2 is volatility forecast for the time k; (1 − β) is the expectation
probability; B1 is the number of correct predictions for rise direction; B2 is the
number of correct predictions for fall direction; E is mean value of errors; E k is
k-th error; Ik,α is prediction interval coverage probability index; L k and Uk are
forecasted interval’s lower and upper bounds, respectively; RVk is actual volatility
for the time k; r is range of real value; ARV is average relative variance; BIAS
is arithmetic mean of the errors; Dstat is directional prediction statistics; FICP is
forecasting interval coverage probability; FINAW is forecasting interval normalized
average width; HMSE is heteroscedasticity-adjusted mean square error; HR is hit
ratio; IA is index of agreement; MAAPE is mean arc-tangent absolute percentage
error; MaxAPE is maximum absolute percentage error; MSE is mean square error;
NBIAS is normalized BIAS; NMAE is normalized mean absolute error; NMSE is
normalized mean squared error; NRMSE is normalized root mean square error; PICP
is prediction interval coverage probability; RMSE is root mean square error; RMSLE
is root mean squared logarithmic error; RMSPE is root mean squared percentage
error; RMSRE is root mean squared relative error; SDE is standard deviation of
errors; SMAPE is symmetric mean absolute percentage error; SSE is error sum of
squares; StdE is population standard deviation of error; TIC is Theil’s inequality
coefficient; U1 is Theil U statistic 1; U2 is Theil U statistic 2.
4.5.6 Real-World Applications
The actual application can be carried out according to the process shown in Fig. 4.25.
Data collection is the most critical link, and the data involved in this step can usually
be obtained from design reports and regular inspection reports. Regular inspection
data can be continuously added to the dataset. Extract features based on the collected
data, and then use a machine learning model to predict the depth of pipeline defects.
Data collection
Geological report Soil parameters
Feature engineering
Pipeline design report Pipeline parameters
Periodic inspection
Defect parameters
report
Machine learning
Decision
prediction
Fig. 4.25 Flow chart of real-world application
Managers can determine the maintenance plan and adjust the inspection cycle based
on the predicted value.
The application of ML in the real world needs to be completed in cooperation
with other equipment and technologies. In the process, data acquisition is the first
and necessary link. It usually requires various types of inspection equipment or
monitoring sensors. Data reaches the data center through the wireless transmission
system of the Internet of Things, which usually needs to be coordinated with satellite
systems and ground base stations. Machine learning calculation is carried out in the
data center. It can identify abnormalities based on sound waves, vibration, and strain
signals to realize monitoring, or realize risk prediction and condition assessment
based on inspection reports. The pipeline owner can judge the pipeline condition
according to the calculation results and give the maintenance scheme. In addition,
because machine learning modeling is highly dependent on data, data can be contin-
uously added to the database during the entire life cycle of the pipeline. Therefore,
on-site interventions and measures can be fed back to the database to optimize the
model. Through this technical process, the pipeline owner can quickly complete the
assessment with good results. For example, ML was applied to the break prediction
of a 20-year-old pipeline in the western United States. The prediction result was
compared with the actual situation. In 2018, 48 of the 57 breaks were predicted, with
an accuracy rate exceeding 80%. In another water supply pipeline project, technicians
predicted the breaks in 2018 based on 11 years of pipeline data. They discovered data
loss and anomalies in the automated data audit, and used ML technology to solve
this problem. In the end, they found 172 out of 218 breaks, with an accuracy rate of
about 80% [317].
4.6 Specific Applications
4.6.1 Burst Pressure Prediction [236]
4.6.1.1 Problem Statement
The burst pressure can accurately reflect the pressure limit of the in-service pres-
sure pipeline, and it is also an essential index for pipeline design and maintenance.
Many scholars have used experimental or finite element methods to estimate the burst
pressure of pipelines. Gao et al. [318] used theoretical, experimental, and numerical
simulation methods to determine the steel reinforced flexible pipe’s burst pressure.
Through practice, they found that the difference between the three results is small.
Wang and Zhou [319] modified Goodall model to make it more suitable for corroded
elbows, and finite element analysis (FEA) method was used to verify the method’s
reliability. Wang and Zhou [320] presented a new method for estimating the burst
pressure of thin-walled elbows. Yeom et al. [321] studied the burst pressure of the
X70 steel pipe with artificial processing defects by full-scale experiment. They also
used the FEA method to analyze the burst pressure of pipelines with a single corro-
sion defect. Zheng et al. [322] used experimental methods to analyze the short-term
burst pressure of steel wire wound polyethylene pipes at different temperatures. Li
et al. [323] used experimental methods to study the effects of corrosion defects and
tensile loads on pipeline bursting in high-pressure gas injection pipelines during CO2
flooding. Chmelko et al. [324] determined the burst pressure of X52 steel with corro-
sion defects by numerical simulation and experiment. Brown et al. [325] established
a three-dimensional model and analyzed the burst pressure of the pipeline repaired
by the cured-in-place pipe method. Shuai et al. [326] used full-scale experiments to
obtain the burst pressure of the pipe with depression and the maximum allowable
operating pressure. Allouti et al. [327] used a series of experiments to determine
the dent depth’s influence on the burst pressure of the in-service pipeline. Ghaednia
et al. [328] carried out full-scale experiments to determine the influence of depression
depth and working pressure on the burst strength of pipelines.
The literature review indicates that there are many studies on the burst pressure
of pipelines. The research methods can be divided into the experimental method and
the FEA method. For the experimental method, the burst pressure can be obtained by
adjusting the pressure, so it is the most direct approach. FEA software can be used
to simulate the actual situation of the pipeline to obtain the mechanical behavior of
the material under different conditions, so it is the most economical way. However,
4.6 Specific Applications 161
these two methods also have their shortcomings. If the burst pressure of corroded
pipeline is determined by experiment, it is necessary to obtain pipeline materials
from the site. Data under different working conditions usually requires multiple
experiments, and some general conclusions need to be obtained based on regression
analysis after multiple experiments. These undoubtedly require a lot of workforce
and material resources and are time-consuming. If the FEA method is used to analyze
the pipeline’s burst pressure, it is necessary to simplify the actual situation and make
some assumptions [329]. Some simulation environments are ideal when setting up,
and more experience is required during the analysis process, such as the setting of
boundary conditions and contact conditions. To increase the reliability of the analysis
results, some scholars or teams have adopted both experiment and FEA methods.
Based on the above reasons and considering that there are many data related to the
burst pressure of corroded pipelines, a novel machine learning model is proposed.
The model can be supervised to learn the relationship between pipe parameters,
defect condition, and burst pressure through training based on the existing data.
After training, the model can predict the burst pressure based on the given pipeline
parameters and corrosion conditions.
4.6.1.2 Prediction Model
The model proposed in this work mixes relevance vector machine (RVM) and multi-
objective salp swarm algorithm (MOSSA). The two hyperparameters of relevance
vector machine will directly affect prediction accuracy and stability. The model
proposed in this work uses multi-objective salp swarm algorithm to optimize the orig-
inal relevance vector machine. In the model training process, multi-objective salp
swarm algorithm finds the optimal hyperparameters of relevance vector machine
within the number of iterations through stochastic optimization. The ensemble
model’s pseudocode is shown in Algorithm 4.1, and the flow is shown in Fig. 4.26.
Algorithm 4.1 MOSSA-RVM

T
Input: Training set: Yt1 = y1 , y2 , . . . , yt1 with training set size N1
T
Test set: Yt2 = yt1 +1 , yt1 +2 , . . . , yt1 +t2 with test set size N2
T
Output: Prediction result:Y t2 = yt1 +1 ,
yt1 +2 , . . . , y
t1 +t2
1 Initialize the salp swarm
2b=1
3 While b < maximum iteration number
4 Calculate the fitness for each salp
5 Find non-dominant salps and initialize the repository
6 If the repository is full
7 Use the repository maintenance procedure to delete a salp
Initialize the salp swarm
Calculate the fitness for each

salp
Initialize parameters of RVM
Find non-dominated salps and

initialize the repository
Training RVM
Yes Delete a salp and add the

Is the repository full?
non-dominated salp
Output the best RVM with the
optimal hyperparameters
No
Select a food source

Yes
No
Reach the maximum
iteration?
Update c1
Update salp's position
Yes Update the position of salp Amend search agent based on

Is i equal to 1?
leader variable upper and lower bounds
No
Update the position of salp

followers
Fig. 4.26 The implementation process of MOSSA-RVM

8 Add non-dominated search agent to the repository

9 End
10 Select a food source
11 Update c1
12 For each search agent
13 If i = 1
14 Update the position of salp leader
15 Else
16 Update the position of salp followers
17 End
18 Amend search agent based on variable upper and lower bounds
19 b = b + 1
20 End while
21 Return repository
22 Get the best hyperparameters of RVM

23 Use the trained RVM to predict and get Yt2
4.6.1.3 Prediction Steps
Collected Data and Statistical Description
The data used in this work comes from [330], and the dataset is recollected from
other literature [331–342]. There are 453 sets of data in the dataset, 102 sets of
data are obtained by experiment, and FEA obtains the rest. Each set of data has
nine parameters, including pipe diameter, wall thickness, ultimate tensile strength,
yield strength, elastic modulus, defect depth, defect length, defect width, and burst
pressure. Their statistical description is shown in Table 4.11. The raw data is divided
into a training set and test set according to the ratio of 0.9:0.1. The data in the test
set is known but pretend to be unknown when validating the model.
Data Normalization
Table 4.11 reveals that the statistical descriptions of different parameters are different
and have different magnitudes. Normalization processing is required before predic-
tion to eliminate the influence of data dimensions [83]. In addition, normalization
can also improve the convergence speed of the model. The normalization operation
is performed according to Eq. (4.91).
R Di −R Dmin
N Di = R Dmax −R Dmin (4.91)
Table 4.11 Statistical analysis of all parameters in the dataset

Statistical Unit Maximum Minimum Mean Standard Skewness Kurtosis
description deviation
Pipe mm 2286 42 801.91 366.36 0.29 3.89
diameter
Wall mm 25.40 2.73 14.55 6.01 –0.26 1.79
thickness
Ultimate MPa 802 262 470.59 87.65 –0.26 4.64
tensile
strength
Yield MPa 891 392 616.42 96.78 –0.43 3.46
strength
Elastic MPa 210,700 200,000 206,340.30 2247.63 –1.08 4.17
modulus
Defect mm 15.41 0.00 5.75 3.16 0.66 3.31
depth
Defect mm 5000 0 482.69 792.15 4.32 24.21
length
Defect mm 679.8 0 122.46 67.56 1.38 12.66
width
Burst MPa 57.33 1.37 18.17 7.38 0.54 4.78
pressure
where N Di is i-th normalized data; R Dmax is maximum raw data; R Dmin is minimum
raw data; R Di is i-th raw data.
Model Training, Optimization, and Prediction
The model needs to be trained before making predictions. In this work, the training
and optimization of the model are carried out simultaneously. When MOSSA finds
the best hyperparameters of RVM, the training of RVM also ends. Since the accuracy
and stability of prediction are considered, two optimization objective functions are
established, as shown in Eq. (4.97). Note that training and optimization operations
are only carried out in the training set. The input and output of the model are shown
in Fig. 4.27.
⎧ Nt | |
⎨ | Ot −Pt |
O B J1 = MAPEtraining = 1
| Ot |
min = Nt (4.92)
⎩ t=1
O B J2 = std(Ot − Pt )
where O B J1 is objective function of prediction accuracy; O B J2 is objective function

of prediction stability; MAPEtraining is mean absolute percentage error (MAPE) in
Pipe diameter
Wall thickness
Ultimate tensile strength
Yield strength
Input Output Burst pressure
Elastic modulus
Defect depth
Defect length
Defect width
Fig. 4.27 Input and output of the prediction
Table 4.12 Initialization parameter setting of six prediction models (Lu et al., 2020b)
Predictor Initialization parameters
MOSSA-RVM Penalty factor = 0.6; kernel width = 2; number of search agents = 200;
maximum iterations = 100; Pareto optimal solutions = 100
RBFNN Training goal = 1 × 10-6 ; learning rate = 0.01; maximum iterations = 500
PSO-SVM Penalty factor = 1; kernel width = 1; acceleration factor = 1.5; maximum
iterations = 50; number of search agents = 50; upper speed limit = 0.1; lower
speed limit = -0.1; maximum iterations = 50
WOA-SVM Penalty factor = 1; kernel width = 1; number of search agents = 30;
maximum iterations = 50
SVM Penalty factor = 1; kernel width = 1
NSGA-II-SVM Penalty factor = 1; kernel width = 1; number of search agents = 100;
maximum iterations = 33; Pareto optimal solutions = 100
the training set; Nt is sample size of the training set; Ot is observation value; Pt is
prediction value.
Model Evaluation
Mere accuracy or stability indicators cannot fully reflect the proposed model’s predic-
tion performance because the characteristics of the data are also one of the influencing
factors. Thus, the proposed model needs to be compared with other models. Five
benchmark models are set up in this work: SVM, SVM with particle swarm opti-
mizer (PSO-SVM), SVM with whale optimization algorithm (WOA-SVM), SVM
with non-dominated sorting genetic algorithm (NSGA-II-SVM), and radial basis
function neural network (RBFNN). The initialization parameter settings of the six
models are shown in Table 4.12.
4.6.1.4 Prediction Results
Prediction Accuracy
Figure 4.28 shows the prediction results of six models, the abscissa of each point
represents the actual value, and the ordinate represents the prediction value. The
green line represents that the prediction value is equal to the actual value. If the point
coincides with the green line or is near the green line, it means that the prediction
results are in good agreement with the actual value, and the prediction accuracy is
higher. Apparently, the prediction accuracy of MOSSA-RVM and NSGA-II-SVM is
higher than other models because all the points are on or near the green line. Most
points of RBFNN and PSO-SVM are located near the green line, and only a few points
are far away from the line. Moreover, most of the points of RBFNN are on the same
side (lower side) of the green line, indicating that the model may have under-fitting
or over-fitting problems. Compared with other models, the points of WOA-SVM and
SVM are more scattered around the green line. Table 4.13 lists the prediction accuracy
indicators of the six models to display the prediction accuracy more intuitively. It
implies that the MAPE, RMSE, RMSPE, U1, and U2 of MOSSA-RVM are 3.724%,
0.767 MPa, 9.440%, 0.017, and 0.035, respectively, and each index is the lowest
among the six models. The prediction accuracy of NSGA-II-SVM is slightly worse
than that of MOSSA-RVM. Its MAPE, RMSE, RMSPE, U1, and U2 are 4.142%,
0.769 MPa, 11.000%, 0.017, and 0.035, respectively. The prediction accuracy of
WOA-SVM is the lowest among the six models. Its MAPE, RMSE, RMSPE, U1,
and U2 are 31.520%, 5.779 MPa, 52.832%, 0.132, and 0.261, respectively.
Prediction Stability
Figure 4.29 presents the percentage error at each point in the test set. It indicates
that the percentage error range of MOSSA-RVM is the smallest, and all errors are
within the interval of [?57.93%,7.22%]. The PSO-SVM has the largest percentage
error range, which is [?708.93%, 25.25%]. Figure 4.30 shows the STDPE of six
models. The STDPEs of MOSSA-RVM, RBFNN, PSO-SVM, WOA-SVM, SVM,
and NSGA-II-SVM are 9.07%, 91.07%, 105.24%, 49.19%, 53.00%, and 10.58%,
respectively. It indicates that MOSSA-RVM has the highest prediction stability,
followed by NSGA-II-SVM. PSO-SVM has the worst prediction stability.
4.6.1.5 Discussions
Diebold-Mariano Test
Diebold-Mariano test can more truly reflect the difference in prediction accuracy
between models [343]. Its calculation method is:
Fig. 4.28 Prediction results. a MOSSA-RVM; b RBFNN; c PSO-SVM; d WOA-SVM; e SVM; f

NSGA-II-SVM
Table 4.13 The prediction errors of the six models

Model MAPE (%) RMSE (MPa) RMSPE (%) U1 U2
MOSSA-RVM 3.724 0.767 9.440 0.017 0.035
RBFNN 22.869 4.915 91.165 0.116 0.222
PSO-SVM 25.738 5.385 106.706 0.123 0.243
WOA-SVM 31.520 5.779 52.832 0.132 0.261
SVM 23.776 5.189 54.554 0.121 0.234
NSGA-II-SVM 4.142 0.769 11.000 0.017 0.035
Note Bold represents the data with the best performance
Suppose the two competing models are Z 1 and Z 2, respectively, and the actual
series is yt . The prediction result of Z 1 is ytZ 1 , and the prediction result of Z 2 is ytZ 2 ,
then their prediction errors can be expressed as:

et1 = yt − ytZ 1
(4.93)
et2 = yt − ytZ 2
where et1 is error of model Z 1; et2 is error of model Z 2.

The hypotheses are:

H0 : E L F et1 = E L F et2 (4.94)

H1 : E L F et1 /= E L F et2 (4.95)
where H0 is null hypothesis; H1 is alternative hypothesis; L F is loss function of the

square error.
Diebold-Mariano test statistics is:
q
DM = t=1 [ SL (√
et1 )−SL (et2 )]/q 2
v (4.96)
v 2 /q

where v 2 is an estimation of the variance of L F et1 − L F et2 .
Table 4.14 shows that the forecast level of MOSSA-RVM is quite different from
WOA-SVM, PSO-SVM, SVM, and RBFNN, and is similar to NSGA-II-SVM.
Unlike the previous accuracy and stability analysis results, Diebold-Mariano test
shows that the prediction results of MOSSA-RVM and WOA-SVM are the most
different. However, the prediction accuracy and stability of PSO-SVM are the lowest,
which may be due to the large error at some points.
Fig. 4.29 The percentage error of prediction results. a MOSSA-RVM; b RBFNN; c PSO-SVM; d
WOA-SVM; e SVM; f NSGA-II-SVM
Data Segmentation and Robustness Test
To further test the performance of the proposed model, the robustness of the model
is analyzed. The analysis is mainly carried out from two aspects. First, divide the
453 sets of data according to other proportions, and then compare the prediction
performance. Second, run the model with different initial parameters multiple times
Fig. 4.30 STDPE of six

models
Table 4.14 The

Model Diebold-Mariano test
Diebold-Mariano test of each
benchmark model NSGAII-SVM 0.0421*
WOA-SVM 3.7622*
PSO-SVM 1.2869*
SVM 2.2909*
RBFNN 1.4583*
Note * is 5% significance level
and compare the prediction performance. If the prediction performance is good, then
the robustness of the model is strong. 453 sets of data are divided into training set
and test set according to 0.7:0.3 (Case I), 0.75:0.25 (Case II), 0.8:0.2 (Case III),
0.85:0.15 (Case IV), and the sizes of the test set are 135, 113, 90, 67, respectively.
Their prediction results are shown in Fig. 4.31, and the MAPEs of prediction results
is shown in Fig. 4.32. They indicate that the prediction accuracy is high in the case
of different data segmentation ratios, and the maximum MAPE is 10.61% when the
ratio of training set and test set is 0.7:0.3. Moreover, Fig. 4.32 shows that as the
proportion of training set and test set increases, the prediction accuracy does not
show a monotonically increasing trend. This is because the data size of the test set
of these five cases is not uniform, and they are not comparable. The experimental
results of these cases can only be used to show that the proposed model has good
adaptability in different situations.
On the other hand, by adjusting the initial parameters of the model, including
hyperparameters, number of search agents, and maximum iterations, the model was
run multiple times, and 13 results were obtained. The running results imply that
Fig. 4.31 The prediction results of the test set with different data segmentation ratios. a Case I; b
Case II; c Case III; d Case IV
Fig. 4.32 MAPEs of

prediction results for test set
under different data
proportions
Fig. 4.33 MAPEs of Case 1 and Case 2
MAPE is in the interval of [3.65%, 5.88%], and the average MAPE is 5.21%. Based
on the above analysis from two aspects, it can be concluded that the proposed model
has high robustness.
Training Data Scale
To explore the impact of training data scale on prediction, this paper keeps the test set
unchanged and changes the training data size. In the original case, the data sizes of
test set and training set are 45 and 408, respectively. In this section, two other cases
are set up-Case 1 and Case 2, the data sizes of their training set are 182 and 137,
respectively, and the test set size is unchanged. Note that the selection of training
data for Case 1 and 2 is random, and the calculation result is the average of 5–9 runs
of the program.
The calculation results (see Fig. 4.33) show that the MAPEs of the original case,
Case 1, and Case 2 are 3.724%, 6.716%, and 8.770%, respectively. This indicates
that when the test set remains unchanged, as the size of the training set increases, the
prediction accuracy of the model improves.
More Applications
The proposed model is applied to two other datasets to test the adaptability. These
two datasets come from the literature [344, 345]. The prediction target is still the
burst pressure, but the input is different. The input is not composed of a series of
single parameters, but the ratio of some parameters or other forms, as shown in Table
4.15. The amount of data in the two datasets is quite different. Dataset I has 250 sets
of data, and Dataset II has 30 sets of data. Note that there are 313 sets of data in the
Table 4.15 The specific information of the two datasets

Dataset Data amount Data amount of the test set Input
I 250 24 PD/WT, DD/WT,
√
YS/SMYS,DL/ PD × WT,
UTS/SMYS, YS/UTS,BP
II 30 8 DW × PD/2π , PD/2WT,
√
DD/WT, DL/ PD × WT,BP
Note: PD is pipe diameter; WT is wall thickness; DD is defect depth; YS is yield strength; SMYS
is specific minimum yield stress; DL is defect length; UTS is ultimate tensile strength; BP is burst
pressure
Fig. 4.34 The prediction results of the proposed model in Dataset I and Dataset II. a Dataset I; b
Dataset II
original Dataset I, but some are missing. After removing the missing data, there are
250 sets left.
Figure 4.34 reveals the prediction results of MOSSA-RVM in Dataset I and Dataset
II (test set), it indicates little difference between the prediction value and the actual
value. Table 4.16 shows the prediction accuracy and stability indicators. It implies
that the accuracy and stability indicators of MOSSA-RVM in Dataset I and Dataset
II are at a relatively low level, and the MAPEs are 1.788% and 4.133%, respectively.
According to the prediction performance rating rules delineated by Lewis [346], the
MAPEs of the proposed model in both datasets are less than 10%, and its prediction
performance can be considered “excellent”.
Table 4.16 The prediction

Dataset I II
accuracy and stability index
of the proposed model in MAPE 1.79% 4.13%
Dataset I and Dataset II RMSE 37.647 psi 0.617 MPa
RMSPE 2.36% 5.59%
U1 0.008 0.026
U2 0.016 0.051
STDPE 2.36% 5.42%
4.6.2 Pullback Force Prediction [229]
4.6.2.1 Problem Statement
The horizontal directional drilling (HDD) technique has become the most popular
trenchless installation method for pipelines since it was invented in 1972. It has
fast construction speed and high accuracy. As shown in Fig. 4.35, HDD construction
usually includes three steps. With the development of technology, the length of HDD
construction has increased from the initial tens of meters to the current thousands of
meters. As of November 2020, the longest construction length of the HDD project
has reached 5.2 km, which was completed at Hong Kong International Airport on
May 18, 2018 [347]. Before constructing the HDD project, it is necessary to calcu-
late the pullback force for selecting the crossing equipment and construction process
[348]. The mechanical behavior of pipelines during HDD construction is complex
and changeable, and is related to mud properties, operating parameters, and pipeline
parameters. Thus, it is challenging to estimate the pullback force accurately. Never-
theless, many scholars have studied this issue, which has become a hot topic in this
field. Many empirical models have been used to estimate the pullback force in HDD
projects. The characteristics and applicability of some commonly used models are
listed in Table 4.17.
Some scholars have improved the model in Table 4.17 and have considered more
factors. For example, Cheng and Polak proposed a model embedded in PipeForce
2005 program and analyzed the contribution of fluid resistance, friction, gravity, and
other factors to the pullback force [354]. Ariaratnam et al. considered the crossing
length, geological conditions, and pipe diameter, and proposed a pullback force
prediction model suitable for the fusible polyvinyl chloride pipeline [355]. Slavin
and Najafi improved the ASTM F1962 method, which can be applied when the entry
point and the exit point are not in the same horizontal plane [356]. Cai and Polak
improved the model proposed by Cheng and Polak in 2007, and the improved model
considered more factors [357]. Although these models have achieved high accuracy to
a certain extent, they still have the following shortcomings: (1) The prediction results
are close to the maximum or minimum value of the actual pullback force, but there
are significant differences at some key points. Cai et al. [351] investigated various
models’ prediction accuracy for two HDD projects in China. The prediction results
Fig. 4.35 HDD construction Entry point Exit point

process. a Pilot hole
River
excavation; b reaming; c (a)
pullback [349]
Pilot hole excavation
(b)
Reamer
Reaming
Pipe
(c)
Pullback
Table 4.17 Commonly used models for pullback force estimation in HDD projects
Model Calculation Considerations Applications
assumption
Driscopipe [350] The borehole profile Gravity, inclination Polyethylene pipe
consists of a series of angle of the pipe
connected straight section, friction
lines coefficient
Drillpath method [351] The pipe consists of Normal force, axial Steel pipe or plastic
short sections tension, tilt angle, pipe
connected by joints gravity, friction
that transmit tension coefficient
and compression
PRCI method [352] The maximum Pull and tension at the Steel pipe
tension is at the exit end of the pipe, mud
of the pipe resistance coefficient,
pipe gravity, friction
between soil and pipe
ASTM F1962 method The bending of the The coefficient of Plastic pipe
[353] pipe is ignored. The friction between the
entry and exit points pipeline and the
are at the same height ground, pipe gravity,
the friction coefficient
of the pipe wall, the
buried depth of the
pipeline, the exit angle
of some models are quite different from the actual values. Some models’ prediction
results can only guarantee the accuracy of the maximum or minimum value, but
not the accuracy at other points. (2) The empirical model needs to consider many
parameters, the calculation steps are cumbersome, and the applicability is lacking in
some construction conditions.
The models mentioned above are used to predict the pullback force in the design
phase and cannot be applied in the construction phase. At different phases, the purpose
of forecasting is different. The prediction of the pullback force in the design phase is
mainly to provide a basis for the crossing scheme design and the drilling rig selection,
and the prediction in the construction stage is mainly to ensure safety. At present, there
is not much research on pullback force prediction during HDD construction. In the
context of big data, data-driven models are an extremely effective way. Many scholars
used machine learning and deep learning to solve engineering problems [358–365].
In 2020, Lu et al. [348] proposed a hybrid model to forecast the pullback force during
HDD construction. The model has high prediction accuracy through the test of actual
monitoring data, but it does not consider stability. However, if the overall prediction
accuracy is high, but the prediction at some points has a significant deviation, it cannot
play the safety warning role in the construction process. Therefore, the stability of
the prediction model is also vital. For this reason, this section proposes a hybrid
model based on multi-objective optimization, while considering both the prediction
accuracy and stability, which is used to assist the pullback force prediction during
the HDD construction process.
4.6.2.2 Prediction Model
In some HDD projects, fiber optic sensors monitor the pullback force during construc-
tion, and the distance between monitoring points is usually about 10 m. For an HDD
project with a length of 1–2 km, there are only a few hundred monitoring points, and
the amount of data is small. Thus, we choose the support vector machine (SVM) as the
basic model for prediction [45]. On the other hand, the two hyperparameters in SVM
significantly affect the prediction performance, and the hyperparameters need to be
optimized when training the model. The purpose of the model is to ensure stability
while ensuring accuracy. Therefore, a classic multi-objective optimization algorithm
(nondominated sorting genetic algorithm II–NSGA-II [366]) is used to optimize
SVM, and the objective functions are shown in Eq. (4.97). Many scholars have used
optimization algorithms to improve SVM, as shown in Table 4.18. Some popular
optimization algorithms (such as genetic algorithm and simulated annealing algo-
rithm) are not adopted because they are single-objective optimization algorithms, and
only one objective function can be established. They cannot ensure the stability of the
forecast while guaranteeing accuracy. Moreover, NSGA-II has high computational
efficiency, which can meet the requirements of rapid prediction in the construction
process.
Table 4.18 Common hybrid SVM-based models and applications

Optimizer Hybrid model Objective function Applications
Particle swarm optimization (PSO) PSO-SVM O B J1 [367, 368]
Grey wolf optimization (GWO) GWO-SVM O B J1 [369, 370]
Genetic algorithm (GA) GA-SVM O B J1 [371, 372]
Cuckoo search (CS) CS-SVM O B J1 [373, 374]
Whale optimization algorithm (WOA) WOA-SVM O B J1 [375, 376]
⎧ Nt | |
⎨ | Ot −Pt |
O B J1 = MAPEtraining = 1
| Ot |
min = Nt (4.97)
⎩ t=1
O B J2 = std(Ot − Pt )
where O B J1 is objective function on accuracy; O B J2 is objective function on

stability; MAPEtraining is mean absolute percentage error in the training set; std
is population standard deviation; Ot is observation value; Pt is prediction value; Nt
is the sample size in the training set.
During the execution of the hybrid model, the optimization of NSGA-II and SVM
training are carried out simultaneously. When NSGA-II obtains the best hyperpa-
rameters of SVM within the number of iterations, the training of SVM also ends. The
implementation process of NSGA-II-SVM is shown in Fig. 4.36, and the pseudocode
is shown in Algorithm 4.2.
Algorithm 4.2 NSGA-II-SVM

T
Input: Training set:Yt1 = y1 , y2 , . . . , yt1
T
Test set:Yt2 = yt1 +1 , yt1 +2 , . . . , yt1 +t2
T
Output: Prediction result:Ỹt2 = ỹt1 +1 , ỹt1 +2 , . . . , ỹt1 +t2
1 Initialize parameters of NSGA-II and SVM
2 Generate initial population P0 randomly
3 Access objective values
4 Assign ranks according to Pareto sorting
5 Generate sub population
6 Binary tournament selection
7 Recombination and mutation
8 For each parent population and sub population
9 Assign ranks according to Pareto sorting
10 Generate non-dominated solutions
11 Calculate the crowding distance
12 Loop by adding solutions to the next generation
13 End
Parameters initialization for SVM
Parameters initialization for NSGA-II Training SVM
Output the best SVM with optimal

Randomly generate initial population P0
hyperparameters
Yes
Fast non-dominated sorting and virtual crowding No

Reach the maximum iteration?
Genetic manipulation
The first N individuals are selected to produce the

The sub population Qt is obtained
parent population Pt+1

Rt=Pt+Qt
Fig. 4.36 Implementation process of NSGA-II-SVM
14 Choose points with larger crowding distance

15 Create next generation
16 Repeat steps 6 and 7
17 Get the best hyperparameters of SVM
18 Use the trained SVM to predict and get Ỹt2
19 End
4.6.2.3 Empirical Analysis
HDD Projects Overview
Two HDD projects belong to the Nanjing branch of the Sichuan-East Gas Pipeline
Project. One project crosses the Yangtze River’s main channel and has 1809.8 m,
Y /m
Entry Main channel crossing project

point Rig X /m
1(0, 0) 6(1809.8, -0.2)
R2=1626 m R4=1626 m
2(289.3, -35.6) 5(1648.8, -28.5)
3(503.6, -49.7) 4(1366.6, -49.7)
Y /m
Jiujiang crossing project

Entry Rig
point X /m
1(0,0) 6(2191,0.2)
R2=762 m R4=762 m
5(1849.8,-41.7)
2(62.4,-17.9)
3(272.4,-47.4) 4(1756.9,-47.4)
Fig. 4.37 Crossing curves for two HDD projects [377]
Fig. 4.38 Construction site

photos of two HDD projects.
a Main channel crossing
project; b Jiujiang crossing
project [377]
which is named Main channel crossing project. Most of the stratum is fine silt sand,
with quicksand and gravel distributed locally. The nominal diameter of the crossing
pipeline is 800 mm. The horizontal position at the exit is 0.2 m lower than that
at the entrance. Another HDD project is located in Jiujiang, namely the Jiujiang
crossing project. The crossing length exceeds 2000 m, and the pipe size is D508
× 11.9 (diameter × thickness). Crossing curves of two HDD projects are shown
in Fig. 4.37. Photos of the construction site are shown in Fig. 4.38. Pipeline and
construction parameters of two HDD projects are shown in Tables 4.19 and 4.20,
respectively [348].
The pullback process of two HDD projects adopted the sending ditch method, of
which specific steps are shown in Fig. 4.39. This process can make the pipeline
float and reduce the resistance effectively. The pullback force during the whole
Table 4.19 Pipeline and

Parameter (Unit) Value
construction parameters for
Main channel crossing project Pipeline diameter (mm) 813
Borehole diameter (mm) 1219.5
Mud flow (L/min) 380
Pullback speed (m/s) 0.026
Friction coefficient between pipe and borehole wall 0.3
Pipeline thickness (mm) 15.9
Consistency coefficient (Pa Sn ) 6.44
Mud density (kg/m3 ) 1200
Friction coefficient between pipe and ground 0.2
Fluidity index 0.31
Table 4.20 Pipeline and

Parameter (Unit) Value
construction parameters for
Jiujiang crossing project Pipeline diameter (mm) 508
Pipeline thickness (mm) 11.9
Borehole diameter (mm) 762
Mud flow (L/min) 265
Pullback speed (m/s) 0.052
Friction coefficient between pipe and borehole wall 0.2
Consistency coefficient (Pa Sn ) 6.44
Mud density (kg/m3 ) 1200
Friction coefficient between pipe and ground 0.3
Fluidity index 0.31
construction process was monitored by China Petroleum and Chemical Corporation.

Optical fiber sensors were arranged at four symmetrical positions around the pipe
circumference, and cooperated with the gyroscope to realize real-time monitoring of
pullback force and position. Collected data are shown in Fig. 4.40. 192 observations
were collected in Main channel crossing project, with a maximum value of 276
tons. 235 observations were collected in Jiujiang crossing project. Figure 4.41 shows
the box diagram of two HDD projects’ pullback force data. Tables 4.19, 4.20, and
Fig. 4.41 indicate that the two projects have different construction conditions, and
the pullback force has different statistical characteristics, which lays the foundation
for more general conclusions.
Fig. 4.39 Steps of sending

ditch method
Excavate the ditch
Put the pipe into

the sending ditch
Pour water into

the ditch and
exceed one third
of the pipe
diameter
Perform pullback
operation
Forecasting Steps
Data normalization operation is based on Eq. (4.83), and the normalized data is
shown in Fig. 4.42.
Collected data are divided into a training set and a test set according to the ratio
of 9:1. Observations in the training set are used to train SVM and enable NSGA-II
to find optimal hyperparameters within the number of iterations, and observations in
the test set are used to test the training effect of the model.
The sliding window length determines the prediction structure and affects the
learning mode in model training. In this work, the single-step prediction is used, and
the sliding window length is three. That is, the first three observations are used to
predict the latter. The selection of input items is also essential. The pullback force
is affected by the mud characteristics, soil properties, and pullback speed. However,
only the data of crossing distance and pullback force were collected in the monitoring
Fig. 4.40 Monitored pullback force data. a Main channel crossing project; b Jiujiang crossing
project
process. In this case, only the crossing distance and the previous pullback force are
taken as the input and the pullback force as the output, as shown in Eq. (4.98).
Pt = M(Pt−3 , Pt−2 , Pt−1 , Dt−3 , Dt−2 , Dt−1 ) (4.98)
where Pt is pullback force at time t; Dt is crossing distance at time t; M is prediction

model.
Fig. 4.41 Box diagram of

pullback force for two HDD
projects
After the prediction, the denormalization operation should be performed

according to Eq. (4.99) to obtain the final prediction result.
pi = (1 − pn )R Dmin + pn R Dmax (4.99)
where pi is final prediction value; pn is normalized prediction value.
4.6.2.4 Forecasting Results
The prediction results of the proposed model are compared with two other benchmark
machine learning models to highlight the superiority of the proposed model. Of these
two models, one is a typical representative RBFNN in neural networks, and the other
is SVM optimized by a classic particle swarm algorithm (PSO). The initial parameters
of the three machine learning models are shown in Table 4.21. Figures 4.43 and 4.44
reveal some machine learning models’ prediction results in two HDD projects. They
present that the proposed model’s prediction results in the training set tend to be
linear, indicating that its training effect is the best, and the training effect of RBFNN
and PSO-SVM is similar. The proposed model’s prediction results are closer to y = x
in the test set, which shows that the proposed model’s prediction performance is the
best, while the performance of RBFNN is the worst. Tables 4.22 and 4.23 show the
specific accuracy and the stability indicators. The MAE, RMSE, MAPE, and STDE
of the proposed model in training set for Main channel crossing project are 1.29
tons, 1.43 tons, 0.89%, and 1.41 tons, respectively; in the test set, they are 10.70
tons, 14.74 tons, 4.28%, and 11.14 tons, respectively. The MAE, RMSE, MAPE,
and STDE of the proposed model in training set for Jiujiang crossing project are
0.58 tons, 0.64 tons, 0.59%, and 0.64 tons, respectively; in the test set, they are 5.84
tons, 9.04 tons, 6.23%, and 7.69 tons, respectively. Note that the model’s prediction
performance in the training set is different from that of the test set because the
Fig. 4.42 Normalized data. a Main channel crossing project; b Jiujiang crossing project
prediction in the training set is supervised, and its purpose is to make the model’s
prediction performance better through continuous correction in the training process.
Therefore, the prediction performance of the training set is usually better than that
of the test set. All the indicators of the proposed model are smaller than those of
RBFNN and PSO-SVM, indicating that the prediction accuracy and stability of the
proposed model are better than the benchmark models. Moreover, the prediction
performance of the proposed model is better than that of PSO-SVM. The reasons for
this phenomenon can be summarized as follows: (1) Compared with PSO, NSGA-II
Table 4.21 Initial parameters of each model

Model Initial parameters
NSGA-II-SVM Penalty factor = 1; kernel width = 1; population size = 100; Pareto optimal
solutions = 100; maximum iterations = 50
RBFNN Training goal = 1 × 10-6 ; learning rate = 0.01; maximum iterations = 500
PSO-SVM Penalty factor = 1; kernel width = 1; acceleration factor = 1.5; number of
search agents = 50; upper speed limit = 0.1; lower speed limit = − 0.1;
maximum iterations = 50; maximum iterations = 50
introduces an elitist strategy to keep multiple solutions rather than unique solutions
in each optimization step, which ensures that some excellent populations will not
be discarded in the evolution process. (2) NSGA-II uses the crowding degree and
crowding comparison operator, which makes the individual extend to the whole
Pareto domain evenly, and ensures the diversity of the population.
Fig. 4.43 Forecasting results for Main channel crossing project. a Training set; b test set
Fig. 4.44 Forecasting results for Jiujiang crossing project. a Training set; b test set
Table 4.22 Accuracy and stability indicators (training set)

Project Model MAE (tons) RMSE (tons) MAPE (%) STDE (tons)
Main channel Proposed model 1.29 1.43 0.89 1.41
crossing project RBFNN 5.62 7.62 3.69 7.57
PSO-SVM 6.14 9.36 3.99 9.35
Jiujiang crossing Proposed model 0.58 0.64 0.59 0.64
project RBFNN 3.69 6.94 3.54 6.94
PSO-SVM 3.12 4.23 3.12 4.22
4.6.2.5 Discussions
Robustness Test
Robustness is essential for an optimizer-based prediction model. In this paper, the

robustness of the proposed model is tested by changing the initial parameters and
Table 4.23 Accuracy and stability indicators (test set)

Project Model MAE (tons) RMSE (tons) MAPE (%) STDE (tons)
Main channel Proposed model 10.70 14.74 4.28 11.14
crossing project RBFNN 16.92 21.41 6.94 21.35
PSO-SVM 14.41 17.14 5.93 13.40
Jiujiang crossing Proposed model 5.84 9.04 6.23 7.69
project RBFNN 17.91 19.40 16.96 9.21
PSO-SVM 8.81 11.83 8.99 11.61
running repeatedly. For NSGA-II, Pareto optimal solution number and population
size are two key initial parameters, and their default values are both 100. In the
robustness test, the Pareto optimal solution number varies from 50 to 100, the popu-
lation size varies from 30 to 100, and the number of iterations is fixed at 15 times.
The proposed model was run 21 times in two pullback force datasets. The calculation
results show that:
(1) For the Main channel crossing project, MAPE is in the interval [3.77%, 4.28%],
and STDE is in the interval [10.04, 11.14];
(2) For Jiujiang crossing project, MAPE is in the interval [6.22%, 6.95%], and
STDE is in the interval [5.11, 7.69]. They demonstrate that the proposed
model’s prediction performance does not change significantly when the initial
parameters are changed and run many times, which shows that the model is
robust.
Sliding Window
This section examines the influence of the sliding window length on the prediction
results. The sliding window length in the original case is three. In this section, the
sliding window length is set to 4–7, and the input–output structure of the model is
shown in Table 4.24. The accuracy and stability indexes corresponding to different
sliding window lengths are shown in Fig. 4.45, and the accuracy is represented by
MAPE. The results show that different sliding window lengths will lead to changes
in prediction performance, and it is not a monotonic change.
Objective Function
Two objective functions set in the original case are for prediction accuracy (O B J1 )
and stability (O B J2 ), respectively. In this section, the multi-objective function is
replaced with a single-objective function (O B J1 ) in the model. The prediction accu-
racy and stability of the model corresponding to different objective functions are
shown in Table 4.25. It presents that the prediction accuracy and stability of the
Table 4.24 Prediction structures corresponding to different sliding window lengths

Sliding window length Prediction structure
3 (original case) Pt = M(Pt−3 , Pt−2 , Pt−1 , Dt−3 , Dt−2 , Dt−1 )
4 Pt = M(Pt−4 , Pt−3 , Pt−2 , Pt−1 , Dt−4 , Dt−3 , Dt−2 , Dt−1 )
5 Pt = M(Pt−5 , Pt−4 , Pt−3 , Pt−2 , Pt−1 ,
Dt−5 , Dt−4 , Dt−3 , Dt−2 , Dt−1 )
6 Pt = M(Pt−6 , Pt−5 , Pt−4 , Pt−3 , Pt−2 , Pt−1 ,
Dt−6 , Dt−5 , Dt−4 , Dt−3 , Dt−2 , Dt−1 )
7 Pt = M(Pt−7 , Pt−6 , Pt−5 , Pt−4 , Pt−3 , Pt−2 , Pt−1 ,
Dt−7 , Dt−6 , Dt−5 , Dt−4 , Dt−3 , Dt−2 , Dt−1 )
Fig. 4.45 Accuracy and stability indicators corresponding to different sliding window lengths. a
MAPE; b STDE
Table 4.25 Model prediction accuracy and stability corresponding to different objective functions
Project Objective function Accuracy and stability
indicators (test set)
MAPE (%) STDE (tons)
Main channel crossing project O B J1 and O B J2 (original case) 4.28 11.14
O B J1 8.28 16.17
Jiujiang crossing project O B J1 and O B J2 (original case) 6.23 7.69
O B J1 6.34 7.70
model with multi-objective functions are higher than those of the model with single-
objective function, indicating that the accuracy and stability are taken as optimization
objectives is effective.
Machine Learning Model Versus Empirical Model
This section compares the prediction results of an empirical model with machine
learning models to highlight the advantages of machine learning models. In 2019,
Cai and Polak proposed an improved Polak theoretical prediction model [357]. The
model has many advantages, and its calculation result can be consistent with the
actual pullback force in the overall trend. Figure 4.46 shows the prediction results of
the improved Polak model and the machine learning model. If they are compared in
the test set (1654–1816 m for the Main channel crossing project, 1992–2199 m for
the Jiujiang crossing project), the predictions of machine learning models at some
points are more accurate.
Feasibility of Real-Time Prediction
The real-time prediction of pullback force in HDD construction requires the model to
have a fast calculation speed, which is also the most basic requirement for the model’s
feasibility in practical applications. Take the proposed model as an example, and the
model is implemented in MATLAB R2017b using a Workstation with an Inter(R)
Core (TM) i7-7700HQ CPU @ 2.8 GHz and Windows 10 with 64 bits and an 8.00 GB
RAM environment. Based on the average of 10 calculations, the average running time
for the Main channel crossing project (192 observations) is 12.13 s, and that of the
Jiujiang crossing project (235 observations) is 13.91 s. It indicates that the proposed
model can complete the calculation quickly, and the feasibility of implementation is
high.
Practical Application Considerations
Since only crossing distance and pullback force data are monitored in these two HDD
projects, only two parameters are considered as inputs in the proposed model. More
relevant parameters can be collected as input to improve the model’s prediction accu-
racy and interpretability in practical applications. For example, geological property
is one of the critical factors affecting the pullback force. Researchers can collect soil
parameters at different depths or crossing distances before construction, and import
these parameters (such as soil density and internal friction angle) into the training
model. On the other hand, some parameters with little correlation may affect the
performance of the prediction model. Principal component analysis can be used to
select appropriate parameters as input to improve rationality and interpretability of
the model. A general application approach is shown in Fig. 4.47. Moreover, the appli-
cation method can also be applied to other aspects, such as pipeline fault diagnosis
and condition assessment.
Fig. 4.46 Prediction results of improved Polak model and machine learning models. a Main channel
crossing project; b Jiujiang crossing project
Pipe
Project
Monitor
collection
Data
Soil data Pipe data Drilling fluid data
PCA
x2
extraction
PC2 PC1
Feature
x1
training
Model
Proposed model
Fig. 4.47 A general HDD pullback force prediction approach
Abbreviations
ABC Artificial bee colony

AdaBoost Adaptive boosting
ANN Artificial neural network
BDT Boosted decision tree
BIC Bayesian information criterion
BN Bayesian network
BPNN Back-propagation neural network
BRT Boosted regression tree
CatBoost Gradient boosting with categorical features
CIT Conditional inference tree
CNN Convolutional neural network
SMDPSO Saturated and mix-delayed particle swarm optimization

CPSO Chaos particle swarm optimization
NSGA-II Non-dominated sorting genetic algorithm II
DBSCAN Density-based spatial clustering of applications with noise
DFP Differential flower pollination
DNN Deep neural network
DO Discrete observer
DT Decision tree
DWT Discrete wavelet transform
EDNN Ensemble deep neural network
ELM Extreme learning machine
faster R-CNN Faster region based convolutional neural networks
GA Genetic algorithm
GBM Gradient boosting Machine
GBT Gradient-boosted tree
GMM Gaussian mixture model
GNB Gaussian Naïve Bayes
GP Genetic programming
HBP Hierarchical beta process
ISLMD Improved spline-local mean decomposition
KNN k-nearest neighbor
LDA Linear discriminant analysis
LGBoost Lightboost
LMD Local mean decomposition
LR Logistic regression
LSTSVM Least squares twin support vector machine
MAPE Mean absolute percentage error
ML Machine learning
MLP Multi-layer perceptron
MLR Multiple linear regression
MLTA Machine learning transient alarming
MOSSA Multi-objective salp swarm algorithm
MOGWO Multi-objective grey wolf optimization
MPN Multilayer perceptron network
MSE Mean square error
NFM Neurofuzzy model
PCA Principal component analysis
PSO Particle swarm optimization
RBFNN Radial basis function neural network
RF Random forest
RMSE Root mean square error
RT Regression tree
RVM Relevance vector machine
SEM Stacking ensemble model
SSCN Subspace clustered neural network
References 193
SSD Single-shot detector

SVM Support vector machine
WE Wavelet entropy
WPA Wavelet packet analysis
XGBoost Extreme gradient boosting
YOLO A clever CNN for doing object detection in real-time
References
1. Winston, P. H. (1984). Artificial intelligence. Addison-Wesley Longman Publishing Co., Inc.

2. Worden, K., & Manson, G. (2007). The application of machine learning to structural health
monitoring. Philosophical Transactions of the Royal Society A: Mathematical, Physical and
Engineering Sciences, 365(1851), 515–537.
3. Farrar, C. R., & Worden, K. (2012). Structural health monitoring: A machine learning
perspective. Wiley.
4. Yuan, F. G., Zargar, S. A., Chen, Q., & Wang, S. (2020). Machine learning for structural
health monitoring: Challenges and opportunities. Sensors and Smart Structures Technologies
for Civil, Mechanical, and Aerospace Systems, 2020(11379), 1137903.
5. Flah, M., Nunez, I., Ben Chaabene, W., & Nehdi, M. L. (2021). Machine learning algorithms
in civil structural health monitoring: A systematic review. Archives of Computational Methods
in Engineering, 28(4), 2621–2643.
6. Ying, Y., Garrett, J. H., Jr., Oppenheim, I. J., Soibelman, L., Harley, J. B., Shi, J., & Jin,
Y. (2013). Toward data-driven structural health monitoring: Application of machine learning
and signal processing to damage detection. Journal of Computing in Civil Engineering, 27(6),
667–680.
7. Demarie, G. V., & Sabia, D. (2019). A machine learning approach for the automatic long-term
structural health monitoring. Structural Health Monitoring, 18(3), 819–837.
8. Gardner, P., Fuentes, R., Dervilis, N., Mineo, C., Pierce, S. G., Cross, E. J., & Worden, K.
(2020). Machine learning at the interface of structural health monitoring and non-destructive
evaluation. Philosophical Transactions of the Royal Society A, 378(2182), 20190581.
9. Weisberg, S. (2005). Applied linear regression (Vol. 528). Wiley.
10. Seber, G. A., & Lee, A. J. (2012). Linear regression analysis. Wiley.
11. Montgomery, D. C., Peck, E. A., & Vining, G. G. (2021). Introduction to linear regression
analysis. Wiley.
12. Su, X., Yan, X., & Tsai, C. L. (2012). Linear regression. Wiley Interdisciplinary Reviews:
Computational Statistics, 4(3), 275–294.
13. Maulud, D., & Abdulazeez, A. M. (2020). A review on linear regression comprehensive in
machine learning. Journal of Applied Science and Technology Trends, 1(4), 140–147.
14. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). Linear regression. In An introduction
to statistical learning (pp. 59–128). Springer.
15. Hadi, A. S., & Chatterjee, S. (2009). Sensitivity analysis in linear regression. Wiley.
16. Yao, W., & Li, L. (2014). A new regression model: Modal linear regression. Scandinavian
Journal of Statistics, 41(3), 656–671.
17. Yuan, Z., & Yang, Y. (2005). Combining linear regression models: When and how? Journal
of the American Statistical Association, 100(472), 1202–1214.
18. Goldberger, A. S. (1981). Linear regression after selection. Journal of Econometrics, 15(3),
357–366.
19. Asai, H. T. S. U. K., Tanaka, S., & Uegima, K. (1982). Linear regression analysis with fuzzy
model. IEEE Transactions of Systems Man Cybern, 12, 903–907.
20. Zhao, L., Chen, Y., & Schaffner, D. W. (2001). Comparison of logistic regression and linear
regression in modeling percentage data. Applied and Environmental Microbiology, 67(5),
2129–2135.
21. Webb, G. I., Keogh, E., & Miikkulainen, R. (2010). Naïve Bayes. Encyclopedia of Machine
Learning, 15, 713–714.
22. Jiang, L., Zhang, H., & Cai, Z. (2008). A novel bayes model: Hidden naive bayes. IEEE
Transactions on Knowledge and Data Engineering, 21(10), 1361–1371.
23. Frank, E., Trigg, L., Holmes, G., & Witten, I. H. (2000). Naive Bayes for regression. Machine
Learning, 41(1), 5–25.
24. Lewis, D. D. (1998, April). Naive (Bayes) at forty: The independence assumption in
information retrieval. In European conference on machine learning (pp. 4–15). Springer.
25. Jiang, L., Wang, D., Cai, Z., & Yan, X. (2007, August). Survey of improving naive bayes
for classification. In International conference on advanced data mining and applications
(pp. 134–145). Springer.
26. Zhang, H. (2005). Exploring conditions for the optimality of naive Bayes. International
Journal of Pattern Recognition and Artificial Intelligence, 19(2), 183–198.
27. Kim, S. B., Han, K. S., Rim, H. C., & Myaeng, S. H. (2006). Some effective techniques
for naive bayes text classification. IEEE Transactions on Knowledge and Data Engineering,
18(11), 1457–1466.
28. Calders, T., & Verwer, S. (2010). Three naive Bayes approaches for discrimination-free
classification. Data Mining and Knowledge Discovery, 21(2), 277–292.
29. Rennie, J. D., Shih, L., Teevan, J., & Karger, D. R. (2003). Tackling the poor assumptions of
naive bayes text classifiers. In Proceedings of the 20th international conference on machine
learning (ICML-03) (pp. 616–623).
30. Saritas, M. M., & Yasar, A. (2019). Performance analysis of ANN and Naive Bayes clas-
sification algorithm for data classification. International Journal of Intelligent Systems and
Applications in Engineering, 7(2), 88–91.
31. Rish, I., Hellerstein, J., & Thathachar, J. (2001). An analysis of data characteristics that affect
naive Bayes performance. IBM TJ Watson Research Center, 30, 1–8.
32. Goggins, S. P., Xing, W., Chen, X., Chen, B., & Wadholm, B. (2015). Learning analytics at
“Small” scale: Exploring a complexity-grounded model for assessment automation. Journal
of Universal Computer Science, 21(1), 66–92.
33. Wang, S. C. (2003). Artificial neural network. In Interdisciplinary computing in java
programming (pp. 81–100). Springer.
34. Abiodun, O. I., Jantan, A., Omolara, A. E., Dada, K. V., Mohamed, N. A., & Arshad, H.
(2018). State-of-the-art in artificial neural network applications: A survey. Heliyon, 4(11),
e00938.
35. Gupta, N. (2013). Artificial neural network. Network and Complex Systems, 3(1), 24–28.
36. Zhang, Z. (2018). Artificial neural network. In Multivariate time series analysis in climate
and environmental research (pp. 1–35). Springer.
37. Wu, Y. C., & Feng, J. W. (2018). Development and application of artificial neural network.
Wireless Personal Communications, 102(2), 1645–1656.
38. Dongare, A. D., Kharde, R. R., & Kachare, A. D. (2012). Introduction to artificial neural
network. International Journal of Engineering and Innovative Technology (IJEIT), 2(1), 189–
194.
39. Maind, S. B., & Wankar, P. (2014). Research paper on basic of artificial neural network.
International Journal on Recent and Innovation Trends in Computing and Communication,
2(1), 96–100.
40. Shanmuganathan, S. (2016). Artificial neural network modelling: An introduction. In Artificial
neural network modelling (pp. 1–14). Springer.
41. Gevrey, M., Dimopoulos, I., & Lek, S. (2003). Review and comparison of methods to study the
contribution of variables in artificial neural network models. Ecological Modelling, 160(3),
249–264.
References 195
42. Shahin, M. A., Jaksa, M. B., & Maier, H. R. (2001). Artificial neural network applications in
geotechnical engineering. Australian Geomechanics, 36(1), 49–62.
43. Chiarello, F., Carelli, P., Castellano, M. G., & Torrioli, G. (2013). Artificial neural network
based on SQUIDs: demonstration of network training and operation. Superconductor Science
and Technology, 26(12), 125009.
44. Dhanalakshmi, P., Palanivel, S., & Ramalingam, V. (2009). Classification of audio signals
using SVM and RBFNN. Expert Systems with Applications, 36(3), 6069–6075.
45. Halali, M. A., Azari, V., Arabloo, M., Mohammadi, A. H., & Bahadori, A. (2016). Application
of a radial basis function neural network to estimate pressure gradient in water–oil pipelines.
Journal of the Taiwan Institute of Chemical Engineers, 58, 189–202.
46. Noble, W. S. (2006). What is a support vector machine? Nature Biotechnology, 24(12), 1565–
1567.
47. Suthaharan, S. (2016). Support vector machine. In Machine learning models and algorithms
for big data classification (pp. 207–235). Springer.
48. Pisner, D. A., & Schnyer, D. M. (2020). Support vector machine. In Machine learning
(pp. 101–121). Academic Press.
49. Widodo, A., & Yang, B. S. (2007). Support vector machine in machine condition monitoring
and fault diagnosis. Mechanical Systems and Signal Processing, 21(6), 2560–2574.
50. Meyer, D., Leisch, F., & Hornik, K. (2003). The support vector machine under test.
Neurocomputing, 55(1–2), 169–186.
51. Zhang, L., Zhou, W., & Jiao, L. (2004). Wavelet support vector machine. IEEE Transactions
on Systems, Man, and Cybernetics, Part B (Cybernetics), 34(1), 34–39.
52. Ukil, A. (2007). Support vector machine. In Intelligent systems and signal processing in power
engineering (pp. 161–226). Springer.
53. Pradhan, A. (2012). Support vector machine-a survey. International Journal of Emerging
Technology and Advanced Engineering, 2(8), 82–85.
54. Kim, H. C., Pang, S., Je, H. M., Kim, D., & Bang, S. Y. (2003). Constructing support vector
machine ensemble. Pattern Recognition, 36(12), 2757–2767.
55. Durgesh, K. S., & Lekha, B. (2010). Data classification using support vector machine. Journal
of Theoretical and Applied Information Technology, 12(1), 1–7.
56. Suykens, J. A., & Vandewalle, J. (1999). Least squares support vector machine classifiers.
Neural Processing Letters, 9(3), 293–300.
57. Lee, Y. J., & Mangasarian, O. L. (2001). SSVM: A smooth support vector machine for
classification. Computational Optimization and Applications, 20(1), 5–22.
58. Tipping, M. E. (2001). Sparse Bayesian learning and the relevance vector machine. Journal
of Machine Learning Research, 1(Jun), 211–244.
59. Caesarendra, W., Widodo, A., & Yang, B. S. (2010). Application of relevance vector machine
and logistic regression for machine degradation assessment. Mechanical Systems and Signal
Processing, 24(4), 1161–1171.
60. Widodo, A., Kim, E. Y., Son, J. D., Yang, B. S., Tan, A. C., Gu, D. S., et al. (2009). Fault
diagnosis of low speed bearing based on relevance vector machine and support vector machine.
Expert Systems with Applications, 36(3), 7252–7261.
61. Widodo, A., & Yang, B. S. (2011). Application of relevance vector machine and survival
probability to machine degradation assessment. Expert Systems with Applications, 38(3),
2592–2599.
62. Yan, J., Liu, Y., Han, S., & Qiu, M. (2013). Wind power grouping forecasts and its uncer-
tainty analysis using optimized relevance vector machine. Renewable and Sustainable Energy
Reviews, 27, 613–621.
63. Zio, E., & Di Maio, F. (2012). Fatigue crack growth estimation by relevance vector machine.
Expert Systems with Applications, 39(12), 10681–10692.
64. Kong, D., Chen, Y., Li, N., Duan, C., Lu, L., & Chen, D. (2019). Relevance vector machine
for tool wear prediction. Mechanical Systems and Signal Processing, 127, 573–594.
65. Wei, L., Yang, Y., Nishikawa, R. M., Wernick, M. N., & Edwards, A. (2005). Relevance
vector machine for automatic detection of clustered microcalcifications. IEEE Transactions
on Medical Imaging, 24(10), 1278–1285.
66. Matsumoto, M., & Hori, J. (2014). Classification of silent speech using support vector machine
and relevance vector machine. Applied Soft Computing, 20, 95–102.
67. Wong, P. K., Xu, Q., Vong, C. M., & Wong, H. C. (2011). Rate-dependent hysteresis modeling
and control of a piezostage using online support vector machine and relevance vector machine.
IEEE Transactions on Industrial Electronics, 59(4), 1988–2001.
68. Zhao, H., Yin, S., & Ru, Z. (2012). Relevance vector machine applied to slope stability
analysis. International Journal for Numerical and Analytical Methods in Geomechanics,
36(5), 643–652.
69. Dong, J., Liu, X., Huang, G., Fan, J., Wu, L., & Wu, J. (2021). Comparison of four bio-
inspired algorithms to optimize KNEA for predicting monthly reference evapotranspiration
in different climate zones of China. Computers and Electronics in Agriculture, 186, 106211.
70. Duan, G., Wu, L., Liu, F., Wang, Y., & Wu, S. (2022). Improvement in solar-radiation fore-
casting based on evolutionary KNEA method and numerical weather prediction. Sustain-
ability, 14(11), 6824.
71. Lu, H., Ma, X., & Azimi, M. (2020). US natural gas consumption prediction using an improved
kernel-based nonlinear extension of the Arps decline model. Energy, 194, 116905.
72. Wang, H., Yan, H., Zeng, W., Lei, G., Ao, C., & Zha, Y. (2020). A novel nonlinear Arps decline
model with salp swarm algorithm for predicting pan evaporation in the arid and semi-arid
regions of China. Journal of Hydrology, 582, 124545.
73. Wu, L., Peng, Y., Fan, J., & Wang, Y. (2019). Machine learning models for the estimation of
monthly mean daily reference evapotranspiration based on cross-station and synthetic data.
Hydrology Research, 50(6), 1730–1750.
74. Lu, H., Ma, X., Huang, K., & Azimi, M. (2020). Prediction of offshore wind farm power using a
novel two-stage model combining kernel-based nonlinear extension of the Arps decline model
with a multi-objective grey wolf optimizer. Renewable and Sustainable Energy Reviews, 127,
109856.
75. Ma, X., & Liu, Z. (2018). Predicting the oil production using the novel multivariate nonlinear
model based on Arps decline model and kernel method. Neural Computing and Applications,
29(2), 579–591.
76. Wu, L., Huang, G., Fan, J., Zhang, F., Wang, X., & Zeng, W. (2019). Potential of kernel-based
nonlinear extension of Arps decline model and gradient boosting with categorical features
support for predicting daily global solar radiation in humid regions. Energy Conversion and
Management, 183, 280–295.
77. Huang, G., Wu, L., Ma, X., Zhang, W., Fan, J., Yu, X., et al. (2019). Evaluation of CatBoost
method for prediction of reference evapotranspiration in humid regions. Journal of Hydrology,
574, 1029–1041.
78. Yu, Z., Haghighat, F., Fung, B. C., & Yoshino, H. (2010). A decision tree method for building
energy demand modeling. Energy and Buildings, 42(10), 1637–1646.
79. Pappalardo, G., Cafiso, S., Di Graziano, A., & Severino, A. (2021). Decision tree method to
analyze the performance of lane support systems. Sustainability, 13(2), 846.
80. Saghebian, S. M., Sattari, M. T., Mirabbasi, R., & Pal, M. (2014). Ground water quality
classification by decision tree method in Ardebil region, Iran. Arabian Journal of Geosciences,
7(11), 4767–4777.
81. Mikučionienė, R., Martinaitis, V., & Keras, E. (2014). Evaluation of energy efficiency
measures sustainability by decision tree method. Energy and Buildings, 76, 64–71.
82. Namazkhan, M., Albers, C., & Steg, L. (2020). A decision tree method for explaining house-
hold gas consumption: The role of building characteristics, socio-demographic variables,
psychological factors and household behaviour. Renewable and Sustainable Energy Reviews,
119, 109542.
83. Hatziargyriou, N. D., Contaxis, G. C., & Sideris, N. C. (1994). A decision tree method
for on-line steady state security assessment. IEEE Transactions on Power Systems, 9(2),
1052–1061.
84. Lu, H., Cheng, F., Ma, X., & Hu, G. (2020). Short-term prediction of building energy consump-
tion employing an improved extreme gradient boosting model: A case study of an intake tower.
Energy, 203, 117756.
References 197
85. Pal, M. (2005). Random forest classifier for remote sensing classification. International
Journal of Remote Sensing, 26(1), 217–222.
86. Speiser, J. L., Miller, M. E., Tooze, J., & Ip, E. (2019). A comparison of random forest variable
selection methods for classification prediction modeling. Expert Systems with Applications,
134, 93–101.
87. Probst, P., & Boulesteix, A. L. (2017). To tune or not to tune the number of trees in random
forest. The Journal of Machine Learning Research, 18(1), 6673–6690.
88. Paul, A., Mukherjee, D. P., Das, P., Gangopadhyay, A., Chintha, A. R., & Kundu, S. (2018).
Improved random forest for classification. IEEE Transactions on Image Processing, 27(8),
4012–4024.
89. Shi, T., & Horvath, S. (2006). Unsupervised learning with random forest predictors. Journal
of Computational and Graphical Statistics, 15(1), 118–138.
90. Liu, Y., Wang, Y., & Zhang, J. (2012, September). New machine learning algorithm: Random
forest. In International conference on information computing and applications (pp. 246–252).
Springer.
91. Tang, F., & Ishwaran, H. (2017). Random forest missing data algorithms. Statistical Analysis
and Data Mining: The ASA Data Science Journal, 10(6), 363–377.
92. Ren, S., Cao, X., Wei, Y., & Sun, J. (2015). Global refinement of random forest. In Proceedings
of the IEEE conference on computer vision and pattern recognition (pp. 723–730).
93. Farnaaz, N., & Jabbar, M. A. (2016). Random forest modeling for network intrusion detection
system. Procedia Computer Science, 89, 213–217.
94. Probst, P., Wright, M. N., & Boulesteix, A. L. (2019). Hyperparameters and tuning strategies
for random forest. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery,
9(3), e1301.
95. Rodriguez-Galiano, V. F., Ghimire, B., Rogan, J., Chica-Olmo, M., & Rigol-Sanchez, J.
P. (2012). An assessment of the effectiveness of a random forest classifier for land-cover
classification. ISPRS Journal of Photogrammetry and Remote Sensing, 67, 93–104.
96. Liaw, A., & Wiener, M. (2002). Classification and regression by randomForest. R news, 2(3),
18–22.
97. Lu, H., & Ma, X. (2020). Hybrid decision tree-based machine learning models for short-term
water quality prediction. Chemosphere, 249, 126169.
98. Lu, H., Ma, X., Huang, K., & Azimi, M. (2020). Carbon trading volume and price forecasting in
China using multiple machine learning models. Journal of Cleaner Production, 249, 119386.
99. Chen, T., & Guestrin, C. (2016, August). Xgboost: A scalable tree boosting system. In
Proceedings of the 22nd ACM sigkdd international conference on knowledge discovery and
data mining (pp. 785–794).
100. Qiu, Y., Zhou, J., Khandelwal, M., Yang, H., Yang, P., & Li, C. (2021). Performance evaluation
of hybrid WOA-XGBoost, GWO-XGBoost and BO-XGBoost models to predict blast-induced
ground vibration. Engineering with Computers, 1–18.
101. Mitchell, R., & Frank, E. (2017). Accelerating the XGBoost algorithm using GPU computing.
PeerJ Computer Science, 3, e127.
102. Ren, X., Guo, H., Li, S., Wang, S., & Li, J. (2017, August). A novel image classification method
with CNN-XGBoost model. In International workshop on digital watermarking (pp. 378–
390). Springer.
103. Nobre, J., & Neves, R. F. (2019). Combining principal component analysis, discrete wavelet
transform and XGBoost to trade in the financial markets. Expert Systems with Applications,
125, 181–194.
104. Jiang, H., He, Z., Ye, G., & Zhang, H. (2020). Network intrusion detection based on PSO-
XGBoost model. IEEE Access, 8, 58392–58401.
105. Shi, X., Wong, Y. D., Li, M. Z. F., Palanisamy, C., & Chai, C. (2019). A feature learning
approach based on XGBoost for driving assessment and risk prediction. Accident Analysis &
Prevention, 129, 170–179.
106. Wang, C., Deng, C., & Wang, S. (2020). Imbalance-XGBoost: Leveraging weighted and focal
losses for binary label-imbalanced classification with XGBoost. Pattern Recognition Letters,
136, 190–197.
107. Al Daoud, E. (2019). Comparison between XGBoost, LightGBM and CatBoost using a home
credit dataset. International Journal of Computer and Information Engineering, 13(1), 6–10.
108. Song, K., Yan, F., Ding, T., Gao, L., & Lu, S. (2020). A steel property optimization model
based on the XGBoost algorithm and improved PSO. Computational Materials Science, 174,
109472.
109. Chen, J., Zhao, F., Sun, Y., & Yin, Y. (2020). Improved XGBoost model based on genetic
algorithm. International Journal of Computer Applications in Technology, 62(3), 240–245.
110. Zhou, Y., Li, T., Shi, J., & Qian, Z. (2019). A CEEMDAN and XGBOOST-based approach
to forecast crude oil prices. Complexity.
111. Ye, X. W., Jin, T., & Yun, C. B. (2019). A review on deep learning-based structural health
monitoring of civil infrastructures. Smart Structures and Systems, 24(5), 567–585.
112. Toh, G., & Park, J. (2020). Review of vibration-based structural health monitoring using deep
learning. Applied Sciences, 10(5), 1680.
113. Azimi, M., Eslamlou, A. D., & Pekcan, G. (2020). Data-driven structural health monitoring
and damage detection through deep learning: State-of-the-art review. Sensors, 20(10), 2778.
114. Bao, Y., Tang, Z., Li, H., & Zhang, Y. (2019). Computer vision and deep learning–based data
anomaly detection method for structural health monitoring. Structural Health Monitoring,
18(2), 401–421.
115. Dang, H. V., Tran-Ngoc, H., Nguyen, T. V., Bui-Tien, T., De Roeck, G., & Nguyen, H. X.
(2020). Data-driven structural health monitoring using feature fusion and hybrid deep learning.
IEEE Transactions on Automation Science and Engineering, 18(4), 2087–2103.
116. Kang, D., & Cha, Y. J. (2018). Autonomous UAVs for structural health monitoring using
deep learning and an ultrasonic beacon system with geo-tagging. Computer-Aided Civil and
Infrastructure Engineering, 33(10), 885–902.
117. Mohtasham Khani, M., Vahidnia, S., Ghasemzadeh, L., Ozturk, Y. E., Yuvalaklioglu, M.,
Akin, S., & Ure, N. K. (2020). Deep-learning-based crack detection with applications for the
structural health monitoring of gas turbines. Structural Health Monitoring, 19(5), 1440–1452.
118. Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., et al. (2018). Recent advances
in convolutional neural networks. Pattern Recognition, 77, 354–377.
119. Vedaldi, A., & Lenc, K. (2015, October). Matconvnet: Convolutional neural networks for
matlab. In Proceedings of the 23rd ACM international conference on nultimedia (pp. 689–
692).
120. Lavin, A., & Gray, S. (2016). Fast algorithms for convolutional neural networks. In Proceed-
ings of the IEEE conference on computer vision and pattern recognition (pp. 4013–4021).
121. Kiranyaz, S., Avci, O., Abdeljaber, O., Ince, T., Gabbouj, M., & Inman, D. J. (2021). 1D
convolutional neural networks and applications: A survey. Mechanical Systems and Signal
Processing, 151, 107398.
122. Pinaya, W. H. L., Vieira, S., Garcia-Dias, R., & Mechelli, A. (2020). Convolutional neural
networks. In Machine learning (pp. 173–191). Academic Press.
123. Yamashita, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018). Convolutional neural networks:
An overview and application in radiology. Insights into Imaging, 9(4), 611–629.
124. Teuwen, J., & Moriakov, N. (2020). Convolutional neural networks. In Handbook of medical
image computing and computer assisted intervention (pp. 481–501). Academic Press.
125. Dosovitskiy, A., Tobias Springenberg, J., & Brox, T. (2015). Learning to generate chairs with
convolutional neural networks. In Proceedings of the IEEE conference on computer vision
and pattern recognition (pp. 1538–1546).
126. Kuo, C. C. J. (2016). Understanding convolutional neural networks with a mathematical
model. Journal of Visual Communication and Image Representation, 41, 406–413.
127. Liu, B., Wang, M., Foroosh, H., Tappen, M., & Pensky, M. (2015). Sparse convolutional neural
networks. In Proceedings of the IEEE conference on computer vision and pattern recognition
(pp. 806–814).
128. Cong, I., Choi, S., & Lukin, M. D. (2019). Quantum convolutional neural networks. Nature
Physics, 15(12), 1273–1278.
References 199
129. Khan, S., Rahmani, H., Shah, S. A. A., & Bennamoun, M. (2018). A guide to convolutional
neural networks for computer vision. Synthesis Lectures on Computer Vision, 8(1), 1–207.
130. Ajit, A., Acharya, K., & Samanta, A. (2020, February). A review of convolutional neural
networks. In 2020 international conference on emerging trends in information technology
and engineering (ic-ETITE) (pp. 1–5). IEEE.
131. Abdel-Hamid, O., Mohamed, A. R., Jiang, H., Deng, L., Penn, G., & Yu, D. (2014). Convo-
lutional neural networks for speech recognition. IEEE/ACM Transactions on Audio, Speech,
and Language Processing, 22(10), 1533–1545.
132. Tian, Y. (2020). Artificial intelligence image recognition method based on convolutional
neural network algorithm. IEEE Access, 8, 125731–125744.
133. Montavon, G., Samek, W., & Müller, K. R. (2018). Methods for interpreting and understanding
deep neural networks. Digital Signal Processing, 73, 1–15.
134. Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., et al. (2019).
Evolving deep neural networks. In Artificial intelligence in the age of neural networks and
brain computing (pp. 293–312). Academic Press.
135. Sze, V., Chen, Y. H., Yang, T. J., & Emer, J. S. (2017). Efficient processing of deep neural
networks: A tutorial and survey. Proceedings of the IEEE, 105(12), 2295–2329.
136. Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Müller, K. R. (2021). Explaining
deep neural networks and beyond: A review of methods and applications. Proceedings of the
IEEE, 109(3), 247–278.
137. Katz, G., Huang, D. A., Ibeling, D., Julian, K., Lazarus, C., Lim, R., et al. (2019, July). The
marabou framework for verification and analysis of deep neural networks. In International
conference on computer aided verification (pp. 443–452). Springer.
138. Rath, S. P., Povey, D., Veselý, K., & Cernocký, J. (2013, August). Improved feature processing
for deep neural networks. In Interspeech (pp. 109–113).
139. Zheng, H., Yang, Z., Liu, W., Liang, J., & Li, Y. (2015, July). Improving deep neural networks
using softplus units. In 2015 International Joint Conference on Neural Networks (IJCNN)
(pp. 1–4). IEEE.
140. Nguyen, A., Yosinski, J., & Clune, J. (2015). Deep neural networks are easily fooled: High
confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on
computer vision and pattern recognition (pp. 427–436).
141. Cao, S., Lu, W., & Xu, Q. (2016, February). Deep neural networks for learning graph repre-
sentations. In Proceedings of the AAAI conference on artificial intelligence (Vol. 30, No.
1).
142. Cichy, R. M., & Kaiser, D. (2019). Deep neural networks as scientific models. Trends in
Cognitive Sciences, 23(4), 305–317.
143. Sharma, P., & Singh, A. (2017, July). Era of deep neural networks: A review. In 2017 8th Inter-
national Conference on Computing, Communication and Networking Technologies (ICCCNT)
(pp. 1–5). IEEE.
144. Medsker, L. R., & Jain, L. C. (2001). Recurrent neural networks. Design and Applications,
5, 64–67.
145. Medsker, L., & Jain, L. C. (Eds.). (1999). Recurrent neural networks: Design and applications.
CRC press.
146. Sutskever, I. (2013). Training recurrent neural networks (pp. 1–101). University of Toronto.
147. Graves, A., Mohamed, A. R., & Hinton, G. (2013, May). Speech recognition with deep
recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and
signal processing (pp. 6645–6649). IEEE.
148. Graves, A., Fernández, S., & Schmidhuber, J. (2007, September). Multi-dimensional recurrent
neural networks. In International conference on artificial neural networks (pp. 549–558).
Springer.
149. Caterini, A. L., & Chang, D. E. (2018). Recurrent neural networks. In Deep neural networks
in a mathematical framework (pp. 59–79). Springer.
150. Williams, R. J., & Zipser, D. (1989). A learning algorithm for continually running fully
recurrent neural networks. Neural Computation, 1(2), 270–280.
151. Funahashi, K. I., & Nakamura, Y. (1993). Approximation of dynamical systems by continuous
time recurrent neural networks. Neural Networks, 6(6), 801–806.
152. Anderson, J. A., Silverstein, J. W., Ritz, S. A., & Jones, R. S. (1977). Distinctive features,
categorical perception, and probability learning: Some applications of a neural model.
Psychological Review, 84(5), 413.
153. Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective
computational abilities. Proceedings of the National Academy of Sciences, 79(8), 2554–2558.
154. Graves, A. (2012). Long short-term memory. In Supervised sequence labelling with recurrent
neural networks (pp. 37–45).
155. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation,
9(8), 1735–1780.
156. Van Houdt, G., Mosquera, C., & Nápoles, G. (2020). A review on the long short-term memory
model. Artificial Intelligence Review, 53(8), 5929–5955.
157. Zhu, X., Sobihani, P., & Guo, H. (2015, June). Long short-term memory over recursive
structures. In International conference on machine learning (pp. 1604–1612). PMLR.
158. Malhotra, P., Vig, L., Shroff, G., & Agarwal, P. (2015, April). Long short term memory
networks for anomaly detection in time series. In Proceedings (Vol. 89, pp. 89–94).
159. Fischer, T., & Krauss, C. (2018). Deep learning with long short-term memory networks for
financial market predictions. European Journal of Operational Research, 270(2), 654–669.
160. Zhang, Q., Wang, H., Dong, J., Zhong, G., & Sun, X. (2017). Prediction of sea surface
temperature using long short-term memory. IEEE Geoscience and Remote Sensing Letters,
14(10), 1745–1749.
161. Sherstinsky, A. (2020). Fundamentals of recurrent neural network (RNN) and long short-term
memory (LSTM) network. Physica D: Nonlinear Phenomena, 404, 132306.
162. Zheng, S., Ristovski, K., Farahat, A., & Gupta, C. (2017, June). Long short-term memory
network for remaining useful life estimation. In 2017 IEEE International Conference on
Prognostics and Health Management (ICPHM) (pp. 88–95). IEEE.
163. Hua, Y., Zhao, Z., Li, R., Chen, X., Liu, Z., & Zhang, H. (2019). Deep learning with long
short-term memory for time series prediction. IEEE Communications Magazine, 57(6), 114–
119.
164. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation,
9(8), 1735–1780.
165. Van Houdt, G., Mosquera, C., & Nápoles, G. (2020). A review on the long short-term memory
model. Artificial Intelligence Review, 53(8), 5929–5955.
166. Chen, Y., Zhao, X., & Jia, X. (2015). Spectral–spatial classification of hyperspectral data based
on deep belief network. IEEE Journal of Selected Topics in Applied Earth Observations and
Remote Sensing, 8(6), 2381–2392.
167. Hua, Y., Guo, J., & Zhao, H. (2015, January). Deep belief networks and deep learning. In
Proceedings of 2015 international conference on intelligent computing and internet of things
(pp. 1–4). IEEE.
168. Liu, P., Han, S., Meng, Z., & Tong, Y. (2014). Facial expression recognition via a boosted
deep belief network. In Proceedings of the IEEE conference on computer vision and pattern
recognition (pp. 1805–1812).
169. Sohn, I. (2021). Deep belief network based intrusion detection techniques: A survey. Expert
Systems with Applications, 167, 114170.
170. Kuremoto, T., Kimura, S., Kobayashi, K., & Obayashi, M. (2014). Time series forecasting
using a deep belief network with restricted Boltzmann machines. Neurocomputing, 137, 47–
56.
171. Hassan, M. M., Alam, M. G. R., Uddin, M. Z., Huda, S., Almogren, A., & Fortino, G. (2019).
Human emotion recognition using deep belief network architecture. Information Fusion, 51,
10–18.
172. Ghasemi, F., Mehridehnavi, A., Fassihi, A., & Pérez-Sánchez, H. (2018). Deep neural network
in QSAR studies using deep belief network. Applied Soft Computing, 62, 251–258.
References 201
173. Deng, W., Liu, H., Xu, J., Zhao, H., & Song, Y. (2020). An improved quantum-inspired differ-
ential evolution algorithm for deep belief network. IEEE Transactions on Instrumentation and
Measurement, 69(10), 7319–7327.
174. O’Connor, P., Neil, D., Liu, S. C., Delbruck, T., & Pfeiffer, M. (2013). Real-time classification
and sensor fusion with a spiking deep belief network. Frontiers in Neuroscience, 7, 178.
175. Tian, Q., Han, D., Li, K. C., Liu, X., Duan, L., & Castiglione, A. (2020). An intrusion detection
approach based on improved deep belief network. Applied Intelligence, 50(10), 3162–3178.
176. Wang, H. Z., Wang, G. B., Li, G. Q., Peng, J. C., & Liu, Y. T. (2016). Deep belief network
based deterministic and probabilistic wind speed forecasting approach. Applied Energy, 182,
80–93.
177. Peng, B. S., Xia, H., Liu, Y. K., Yang, B., Guo, D., & Zhu, S. M. (2018). Research on intelligent
fault diagnosis method for nuclear power plant based on correlation analysis and deep belief
network. Progress in Nuclear Energy, 108, 419–427.
178. Lv, Z., & Qiao, L. (2020). Deep belief network and linear perceptron based cognitive
computing for collaborative robots. Applied Soft Computing, 92, 106300.
179. Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief
nets. Neural computation, 18(7), 1527–1554.
180. Gehring, J., Miao, Y., Metze, F., & Waibel, A. (2013, May). Extracting deep bottleneck
features using stacked auto-encoders. In 2013 IEEE international conference on acoustics,
speech and signal processing (pp. 3377–3381). IEEE.
181. Masci, J., Meier, U., Cireşan, D., & Schmidhuber, J. (2011, June). Stacked convolutional auto-
encoders for hierarchical feature extraction. In International conference on artificial neural
networks (pp. 52–59). Springer.
182. Li, P., Chen, Z., Yang, L. T., Gao, J., Zhang, Q., & Deen, M. J. (2018). An improved stacked
auto-encoder for network traffic flow classification. IEEE Network, 32(6), 22–27.
183. Vartouni, A. M., Kashi, S. S., & Teshnehlab, M. (2018, February). An anomaly detection
method to detect web attacks using stacked auto-encoder. In 2018 6th Iranian Joint Congress
on Fuzzy and Intelligent Systems (CFIS) (pp. 131–134). IEEE.
184. Zabalza, J., Ren, J., Zheng, J., Zhao, H., Qing, C., Yang, Z., et al. (2016). Novel
segmented stacked autoencoder for effective dimensionality reduction and feature extraction
in hyperspectral imaging. Neurocomputing, 185, 1–10.
185. Öztürk, Ş. (2020). Stacked auto-encoder based tagging with deep features for content-based
medical image retrieval. Expert Systems with Applications, 161, 113693.
186. Sagheer, A., & Kotb, M. (2019). Unsupervised pre-training of a deep LSTM-based stacked
autoencoder for multivariate time series forecasting problems. Scientific Reports, 9(1), 1–16.
187. Zhou, P., Han, J., Cheng, G., & Zhang, B. (2019). Learning compact and discriminative stacked
autoencoder for hyperspectral image classification. IEEE Transactions on Geoscience and
Remote Sensing, 57(7), 4823–4833.
188. Singh, D., & Mohan, C. K. (2018). Deep spatio-temporal representation for detection of
road accidents using stacked autoencoder. IEEE Transactions on Intelligent Transportation
Systems, 20(3), 879–887.
189. Wang, W., Ooi, B. C., Yang, X., Zhang, D., & Zhuang, Y. (2014). Effective multi-modal
retrieval based on stacked auto-encoders. Proceedings of the VLDB Endowment, 7(8), 649–
660.
190. Li, W., Fu, H., Yu, L., Gong, P., Feng, D., Li, C., & Clinton, N. (2016). Stacked Autoencoder-
based deep learning for remote-sensing image classification: A case study of African land-
cover mapping. International Journal of Remote Sensing, 37(23), 5632–5646.
191. Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of
deep networks. Advances in Neural Information Processing Systems 19, Proceedings of the
Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British
Columbia, Canada, December 4-7, 2006. DBLP.
192. Yoo, H. J. (2015). Deep convolution neural networks in computer vision: A review. IEIE
Transactions on Smart Processing and Computing, 4(1), 35–43.
193. Al-Saffar, A. A. M., Tao, H., & Talab, M. A. (2017, October). Review of deep convolution
neural network in image classification. In 2017 International Conference on Radar, Antenna,
Microwave, Electronics, and Telecommunications (ICRAMET) (pp. 26–31). IEEE.
194. Traore, B. B., Kamsu-Foguem, B., & Tangara, F. (2018). Deep convolution neural network
for image recognition. Ecological Informatics, 48, 257–268.
195. Yang, J., & Li, J. (2017, December). Application of deep convolution neural network. In
2017 14th International Computer Conference on Wavelet Active Media Technology and
Information Processing (ICCWAMTIP) (pp. 229–232). IEEE.
196. Aloysius, N., & Geetha, M. (2017, April). A review on deep convolutional neural networks. In
2017 International Conference on Communication and Signal Processing (ICCSP) (pp. 0588–
0592). IEEE.
197. Liu, M., Shi, J., Li, Z., Li, C., Zhu, J., & Liu, S. (2016). Towards better analysis of deep
convolutional neural networks. IEEE Transactions on Visualization and Computer Graphics,
23(1), 91–100.
198. Sainath, T. N., Kingsbury, B., Mohamed, A. R., Dahl, G. E., Saon, G., Soltau, H., et al. (2013,
December). Improvements to deep convolutional neural networks for LVCSR. In 2013 IEEE
workshop on automatic speech recognition and understanding (pp. 315–320). IEEE.
199. Khan, A., Sohail, A., Zahoora, U., & Qureshi, A. S. (2020). A survey of the recent architectures
of deep convolutional neural networks. Artificial Intelligence Review, 53(8), 5455–5516.
200. Jin, K. H., McCann, M. T., Froustey, E., & Unser, M. (2017). Deep convolutional neural
network for inverse problems in imaging. IEEE Transactions on Image Processing, 26(9),
4509–4522.
201. Mirjalili, S. (2016). SCA: A sine cosine algorithm for solving optimization problems.
Knowledge-Based Systems, 96, 120–133.
202. Abualigah, L., & Diabat, A. (2021). Advances in sine cosine algorithm: A comprehensive
survey. Artificial Intelligence Review, 54(4), 2567–2608.
203. Mirjalili, S., & Lewis, A. (2016). The whale optimization algorithm. Advances in Engineering
Software, 95, 51–67.
204. Gharehchopogh, F. S., & Gholizadeh, H. (2019). A comprehensive survey: Whale optimization
algorithm and its applications. Swarm and Evolutionary Computation, 48, 1–24.
205. Mirjalili, S. (2015). Moth-flame optimization algorithm: A novel nature-inspired heuristic
paradigm. Knowledge-Based Systems, 89, 228–249.
206. Mirjalili, S., Mirjalili, S. M., & Lewis, A. (2014). Grey wolf optimizer. Advances in
Engineering Software, 69, 46–61.
207. Mirjalili, S., Mirjalili, S. M., & Yang, X. S. (2014). Binary bat algorithm. Neural Computing
and Applications, 25(3), 663–681.
208. Yang, X. S., & Deb, S. (2014). Cuckoo search: Recent advances and applications. Neural
Computing and Applications, 24(1), 169–174.
209. Poli, R., Kennedy, J., & Blackwell, T. (2007). Particle swarm optimization. Swarm Intelli-
gence, 1(1), 33–57.
210. Kennedy, J., & Eberhart, R. (1995, November). Particle swarm optimization. In Proceedings
of ICNN’95-international conference on neural networks (Vol. 4, pp. 1942–1948). IEEE.
211. Blum, C. (2005). Ant colony optimization: Introduction and recent trends. Physics of Life
Reviews, 2(4), 353–373.
212. Dorigo, M., Birattari, M., & Stutzle, T. (2006). Ant colony optimization. IEEE Computational
Intelligence Magazine, 1(4), 28–39.
213. Bertsimas, D., & Tsitsiklis, J. (1993). Simulated annealing. Statistical Science, 8(1), 10–15.
214. Van Laarhoven, P. J., & Aarts, E. H. (1987). Simulated annealing. In Simulated annealing:
Theory and applications (pp. 7–15). Springer.
215. Mirjalili, S. (2019). Genetic algorithm. In Evolutionary algorithms and neural networks
(pp. 43–55). Springer.
References 203
216. Whitley, D. (1994). A genetic algorithm tutorial. Statistics and Computing, 4(2), 65–85.
217. Lu, H., Ma, X., Ma, M., & Zhu, S. (2021). Energy price prediction using data-driven models:
A decade review. Computer Science Review, 39, 100356.
218. Pan, W. T. (2012). A new fruit fly optimization algorithm: taking the financial distress model
as an example. Knowledge-Based Systems, 26, 69–74.
219. Fan, Y., Wang, P., Mafarja, M., Wang, M., Zhao, X., & Chen, H. (2021). A bioinformatic
variant fruit fly optimizer for tackling optimization problems. Knowledge-Based Systems,
213, 106704.
220. Sang, H. Y., Pan, Q. K., & Duan, P. Y. (2019). Self-adaptive fruit fly optimizer for global
optimization. Natural Computing, 18(4), 785–813.
221. Yang, Y., Chen, H., Li, S., Heidari, A. A., & Wang, M. (2020). Orthogonal learning harmo-
nizing mutation-based fruit fly-inspired optimizers. Applied Mathematical Modelling, 86,
368–383.
222. Lu, H., Azimi, M., & Iseley, T. (2019). Short-term load forecasting of urban gas using a hybrid
model based on improved fruit fly optimization algorithm and support vector machine. Energy
Reports, 5, 666–677.
223. Mirjalili, S., & Lewis, A. (2016). The whale optimization algorithm. Advances in Engineering
Software, 95, 51–67.
224. Kaur, G., & Arora, S. (2018). Chaotic whale optimization algorithm. Journal of Computational
Design and Engineering, 5(3), 275–284.
225. Deb, K., Agrawal, S., Pratap, A., & Meyarivan, T. (2000, September). A fast elitist
non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II. In
International conference on parallel problem solving from nature (pp. 849–858). Springer.
226. Sheng, W., Liu, K. Y., Liu, Y., Meng, X., & Li, Y. (2014). Optimal placement and sizing
of distributed generation via an improved nondominated sorting genetic algorithm II. IEEE
Transactions on Power Delivery, 30(2), 569–578.
227. Basu, M. (2008). Dynamic economic emission dispatch using nondominated sorting genetic
algorithm-II. International Journal of Electrical Power & Energy Systems, 30(2), 140–149.
228. Dumedah, G., Berg, A. A., Wineberg, M., & Collier, R. (2010). Selecting model parameter
sets from a trade-off surface generated from the non-dominated sorting genetic algorithm-II.
Water Resources Management, 24(15), 4469–4489.
229. Srinivas, N., & Deb, K. (1994). Muiltiobjective optimization using nondominated sorting in
genetic algorithms. Evolutionary Computation, 2(3), 221–248.
230. Lu, H., Iseley, T., Matthews, J., & Liao, W. (2021). Hybrid machine learning for pullback force
forecasting during horizontal directional drilling. Automation in Construction, 129, 103810.
231. Mirjalili, S., Saremi, S., Mirjalili, S. M., & Coelho, L. D. S. (2016). Multi-objective grey
wolf optimizer: A novel algorithm for multi-criterion optimization. Expert Systems with
Applications, 47, 106–119.
232. Zapotecas-Martínez, S., García-Nájera, A., & López-Jaimes, A. (2019). Multi-objective grey
wolf optimizer based on decomposition. Expert Systems with Applications, 120, 357–371.
233. Aljarah, I., Habib, M., Faris, H., Al-Madi, N., Heidari, A. A., Mafarja, M., et al. (2020). A
dynamic locality multi-objective salp swarm algorithm for feature selection. Computers &
Industrial Engineering, 147, 106628.
234. Cheng, Z., & Wang, J. (2020). A new combined model based on multi-objective salp swarm
optimization for wind speed forecasting. Applied Soft Computing, 92, 106294.
235. Ridha, H. M., Gomes, C., Hizam, H., & Mirjalili, S. (2020). Multiple scenarios multi-objective
salp swarm optimization for sizing of standalone photovoltaic system. Renewable Energy, 153,
1330–1345.
236. Mirjalili, S., Gandomi, A. H., Mirjalili, S. Z., Saremi, S., Faris, H., & Mirjalili, S. M. (2017).
Salp Swarm Algorithm: A bio-inspired optimizer for engineering design problems. Advances
in Engineering Software, 114, 163–191.
237. Lu, H., Iseley, T., Matthews, J., Liao, W., & Azimi, M. (2021). An ensemble model based
on relevance vector machine and multi-objective salp swarm algorithm for predicting burst
pressure of corroded pipelines. Journal of Petroleum Science and Engineering, 203, 108585.
https://doi.org/10.1016/j.petrol.2021.108585
238. Zadkarami, M., Shahbazian, M., & Salahshoor, K. (2017). Pipeline leak diagnosis based on
wavelet and statistical features using Dempster-Shafer classifier fusion technique. Process
Safety and Environmental Protection, 105, 156–163. https://doi.org/10.1016/j.psep.2016.
11.002
239. Chen, H., Ye, H., Chen, L. V., & Su, H. (2004, May). Application of support vector machine
learning to leak detection and location in pipelines. In Proceedings of the 21st IEEE instru-
mentation and measurement technology conference (IEEE Cat. No. 04CH37510) (Vol. 3,
pp. 2273–2277). IEEE. https://doi.org/10.1109/IMTC.2004.1351546
240. Isa, D., & Rajkumar, R. (2009). Pipeline defect prediction using support vector machines.
Applied Artificial Intelligence, 23(8), 758–771. https://doi.org/10.1080/08839510903210589
241. Mandal, S. K., Chan, F. T., & Tiwari, M. K. (2012). Leak detection of pipeline: An integrated
approach of rough set theory and artificial bee colony trained SVM. Expert Systems with
Applications, 39(3), 3071–3080. https://doi.org/10.1016/j.eswa.2011.08.170
242. El-Abbasy, M. S., Senouci, A., Zayed, T., Parvizsedghy, L., & Mirahadi, F. (2016). Unpiggable
oil and gas pipeline condition forecasting models. Journal of Performance of Constructed
Facilities, 30(1), 04014202. https://doi.org/10.1061/(ASCE)CF.1943-5509.0000716
243. Liang, W., & Zhang, L. (2012). A wave change analysis (WCA) method for pipeline leak
detection using Gaussian mixture model. Journal of Loss Prevention in the Process Industries,
25(1), 60–69. https://doi.org/10.1016/j.jlp.2011.06.017
244. Ni, L., Jiang, J., & Pan, Y. (2013). Leak location of pipelines based on transient model and
PSO-SVM. Journal of Loss Prevention in the Process Industries, 26(6), 1085–1093. https://
doi.org/10.1016/j.jlp.2013.04.004
245. Zhou, M., Pan, Z., Liu, Y., Zhang, Q., Cai, Y., & Pan, H. (2019). Leak detection and location
based on ISLMD and CNN in a pipeline. IEEE Access, 7, 30457–30464. https://doi.org/10.
1109/ACCESS.2019.2902711
246. Sun, J., Xiao, Q., Wen, J., & Zhang, Y. (2016). Natural gas pipeline leak aperture identification
and location based on local mean decomposition analysis. Measurement, 79, 147–157. https://
doi.org/10.1016/j.measurement.2015.10.015
247. Camacho-Navarro, J., Ruiz, M., Villamizar, R., Mujica, L., & Moreno-Beltrán, G. (2017,
May). Ensemble learning as approach for pipeline condition assessment. In Journal of Physics:
Conference Series (Vol. 842, No. 1, pp. 012019). IOP Publishing. https://doi.org/10.1088/
1742-6596/842/1/012019
248. Lang, X., Li, P., Hu, Z., Ren, H., & Li, Y. (2017). Leak detection and location of pipelines
based on LMD and least squares twin support vector machine. IEEE Access, 5, 8659–8668.
https://doi.org/10.1109/ACCESS.2017.2703122
249. Duong, B. P., & Kim, J. M. (2018, July). Pipeline fault diagnosis using wavelet entropy and
ensemble deep neural technique. In International conference on image and signal processing
(pp. 292–300). Springer. https://doi.org/10.1007/978-3-319-94211-7_32
250. Oh, S. W., Yoon, D. B., Kim, G. J., Bae, J. H., & Kim, H. S. (2018). Acoustic data condensation
to enhance pipeline leak detection. Nuclear Engineering and Design, 327, 198–211. https://
doi.org/10.1016/j.nucengdes.2017.12.006
251. Jia, Z., Ren, L., Li, H., & Sun, W. (2018). Pipeline leak localization based on FBG hoop strain
sensors combined with BP neural network. Applied Sciences, 8(2), 146. https://doi.org/10.
3390/app8020146
252. Ahn, B., Kim, J., & Choi, B. (2019). Artificial intelligence-based machine learning consid-
ering flow and temperature of the pipeline for leak early detection using acoustic emission.
Engineering Fracture Mechanics, 210, 381–392. https://doi.org/10.1016/j.engfracmech.2018.
03.010
253. Chhotaray, G., & Kulshreshtha, A. (2019). Defect detection in oil and gas pipeline: A machine
learning application. In Data management, analytics and innovation (pp. 177–184). Springer.
https://doi.org/10.1007/978-981-13-1274-8_14
References 205
254. Liu, Y., Ma, X., Li, Y., Tie, Y., Zhang, Y., & Gao, J. (2019). Water pipeline leakage detection
based on machine learning and wireless sensor networks. Sensors, 19(23), 5086. https://doi.
org/10.3390/s19235086
255. Shravani, D., Prajwal, Y. R., Prapulla, S. B., Salanke, N. G. R., Shobha, G., & Ahmad,
S. F. (2019, December). A machine learning approach to water leak localization. In 2019
4th International Conference on Computational Systems and Information Technology for
Sustainable Solution (CSITSS) (Vol. 4, pp. 1–6). IEEE. https://doi.org/10.1109/CSITSS47250.
2019.9031010
256. Zhou, B., Lau, V., & Wang, X. (2019). Machine-Learning-Based leakage-event identification
for smart water supply systems. IEEE Internet of Things Journal, 7(3), 2277–2292. https://
doi.org/10.1109/JIOT.2019.2958920
257. Xie, J., Xu, X., & Dubljevic, S. (2019). Long range pipeline leak detection and localization
using discrete observer and support vector machine. AIChE Journal, 65(7), e16532. https://
doi.org/10.1002/aic.16532
258. Wang, C., Zhang, Y., Song, J., Liu, Q., & Dong, H. (2019). A novel optimized SVM algorithm
based on PSO with saturation and mixed time-delays for classification of oil pipeline leak
detection. Systems Science & Control Engineering, 7(1), 75–88. https://doi.org/10.1080/216
42583.2019.1573386
259. Alshaikh, A., Magana-Mora, A., Gharbi, S. A., & Al-Yami, A. (2019, March). Machine
learning for detecting stuck pipe incidents: Data analytics and models evaluation. In Interna-
tional petroleum technology conference (pp. IPTC-19394-MS). OnePetro. https://doi.org/10.
2523/IPTC-19394-MS
260. Abbas, A. K., Flori, R., Almubarak, H., Dawood, J., Abbas, H., & Alsaedi, A. (2019,
September). Intelligent prediction of stuck pipe remediation using machine learning algo-
rithms. In SPE annual technical conference and exhibition (pp. SPE-196229-MS). Society of
Petroleum Engineers. https://doi.org/10.2118/196229-MS
261. Zajam, S., Joshi, T., & Bhattacharya, B. (2019). Application of wavelet analysis and machine
learning on vibration data from gas pipelines for structural health monitoring. Procedia
Structural Integrity, 14, 712–719. https://doi.org/10.1016/j.prostr.2019.05.089
262. Banjara, N. K., Sasmal, S., & Voggu, S. (2020). Machine learning supported acoustic emission
technique for leakage detection in pipelines. International Journal of Pressure Vessels and
Piping, 188, 104243. https://doi.org/10.1016/j.ijpvp.2020.104243
263. Xue, P., Jiang, Y., Zhou, Z., Chen, X., Fang, X., & Liu, J. (2020). Machine learning-based
leakage fault detection for district heating networks. Energy and Buildings, 223, 110161.
https://doi.org/10.1016/j.enbuild.2020.110161
264. Liu, Z., & Li, S. (2020). A sound monitoring system for prevention of underground pipeline
damage caused by construction. Automation in Construction, 113, 103125. https://doi.org/10.
1016/j.autcon.2020.103125
265. Wang, X., Ghidaoui, M. S., & Lee, P. J. (2020). Linear model and regularization for tran-
sient wave?based pipeline-condition assessment. Journal of Water Resources Planning and
Management, 146(5), 04020028. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001205
266. da Cruz, R. P., da Silva, F. V., & Fileti, A. M. F. (2020). Machine learning and acoustic method
applied to leak detection and location in low-pressure gas pipelines. Clean Technologies and
Environmental Policy, 22(3), 627–638. https://doi.org/10.1007/s10098-019-01805-x
267. Alves Coelho, J., Glória, A., & Sebastião, P. (2020). Precise water leak detection using machine
learning and real-time sensor data. IoT, 1(2), 474–493. https://doi.org/10.3390/iot1020026
268. Sfar Zaoui, W., Lauber, T., Pohl, C., Kerk, M., Glaeser, T., & Jelinek, W. (2020, November).
Machine-learning distributed-temperature-sensing-based pipeline leak detection. In Abu
Dhabi international petroleum exhibition & conference (pp. SPE-202905-MS). Society of
Petroleum Engineers. https://doi.org/10.2118/202905-MS
269. Khan, J. A., Irfan, M., Irawan, S., Yao, F. K., Abdul Rahaman, M. S., Shahari, A. R., Glowacz,
A., & Zeb, N. (2020). Comparison of machine learning classifiers for accurate prediction of
real-time stuck pipe incidents. Energies, 13(14), 3683. https://doi.org/10.3390/en13143683
270. Xiao, R., Hu, Q., & Li, J. (2021). A model-based health indicator for leak detection in gas
pipeline systems. Measurement, 171, 108843. https://doi.org/10.1016/j.measurement.2020.
108843
271. Rai, A., & Kim, J. M. (2021). A novel pipeline leak detection approach independent of prior
failure information. Measurement, 167, 108284. https://doi.org/10.1016/j.measurement.2020.
108284
272. Pérez-Pérez, E. J., López-Estrada, F. R., Valencia-Palomo, G., Torres, L., Puig, V., & Mina-
Antonio, J. D. (2021). Leak diagnosis in pipelines using a combined artificial neural network
approach. Control Engineering Practice, 107, 104677. https://doi.org/10.1016/j.conengprac.
2020.104677
273. Saade, M., & Mustapha, S. (2020). Assessment of the structural conditions in steel pipeline
under various operational conditions–A machine learning approach. Measurement, 166,
108262. https://doi.org/10.1016/j.measurement.2020.108262
274. Dawood, T., Elwakil, E., Novoa, H. M., & Gárate Delgado, J. F. (2020). Water pipe failure
prediction and risk models: State-of-the-art review. Canadian Journal of Civil Engineering,
47(10), 1117–1127. https://doi.org/10.1139/cjce-2019-0481
275. Zhu, L. X., & Zou, L. (2005, August). Application of genetic algorithm in decision-making
optimization of underground gas pipeline risk model. In 2005 international conference on
machine learning and cybernetics (Vol. 5, pp. 2988–2992). IEEE. https://doi.org/10.1109/
ICMLC.2005.1527454
276. Parvizsedghy, L., & Zayed, T. (2016). Consequence of failure: Neurofuzzy-based prediction
model for gas pipelines. Journal of Performance of Constructed Facilities, 30(4), 04015073.
https://doi.org/10.1061/(ASCE)CF.1943-5509.0000817
277. Bagriacik, A., Davidson, R. A., Hughes, M. W., Bradley, B. A., & Cubrinovski, M. (2018).
Comparison of statistical and machine learning approaches to modeling earthquake damage
to water pipelines. Soil Dynamics and Earthquake Engineering, 112, 76–88. https://doi.org/
10.1016/j.soildyn.2018.05.010
278. Jiang, F., & Dong, S. (2020). Collision failure risk analysis of falling object on subsea pipelines
based on machine learning scheme. Engineering Failure Analysis, 114, 104601. https://doi.
org/10.1016/j.engfailanal.2020.104601
279. Giraldo-González, M. M., & Rodríguez, J. P. (2020). Comparison of statistical and machine
learning models for pipe failure modeling in water distribution networks. Water, 12(4), 1153.
https://doi.org/10.3390/w12041153
280. Anghel, C. I. (2009). Risk assessment for pipelines with active defects based on artificial
intelligence methods. International Journal of Pressure Vessels and Piping, 86(7), 403–411.
https://doi.org/10.1016/j.ijpvp.2009.01.009
281. Wu, W. (2015). Oil and gas pipeline risk assessment model by fuzzy inference systems and
artificial neural network. Doctoral dissertation, University of Regina. https://instrepo-prod.
cc.uregina.ca/handle/10294/5863. Accessed Nov 15, 2021.
282. Zhou, J., Sun, J., Wang, Y., & Chen, F. (2017). Wrapping practical problems into a machine
learning framework: Using water pipe failure prediction as a case study. International Journal
of Intelligent Systems Technologies and Applications, 16(3), 191–207. https://doi.org/10.
1504/IJISTA.2017.10005998
283. Ouadah, A. (2018, November). Pipeline defects risk assessment using machine learning and
analytical hierarchy process. In 2018 International Conference on Applied Smart Systems
(ICASS) (pp. 1–6). IEEE. https://doi.org/10.1109/ICASS.2018.8651970
284. Guo, X., Zhang, L., Liang, W., & Haugen, S. (2018). Risk identification of third-party damage
on oil and gas pipelines through the Bayesian network. Journal of Loss Prevention in the
Process Industries, 54, 163–178. https://doi.org/10.1016/j.jlp.2018.03.012
285. Winkler, D., Haltmeier, M., Kleidorfer, M., Rauch, W., & Tscheikner-Gratl, F. (2018). Pipe
failure modelling for water distribution networks using boosted decision trees. Structure and
Infrastructure Engineering, 14(10), 1402–1411. https://doi.org/10.1080/15732479.2018.144
3145
References 207
286. Sattar, A. M., Ertu?rul, Ö. F., Gharabaghi, B., McBean, E. A., & Cao, J. (2019). Extreme
learning machine model for water network management. Neural Computing and Applications,
31(1), 157–169. https://doi.org/10.1007/s00521-017-2987-7
287. Wang, Z., & Li, S. (2020). Data-driven risk assessment on urban pipeline network based on a
cluster model. Reliability Engineering & System Safety, 196, 106781. https://doi.org/10.1016/
j.ress.2019.106781
288. Ossai, C. I. (2020). Corrosion defect modelling of aged pipelines with a feed-forward multi-
layer neural network for leak and burst failure estimation. Engineering Failure Analysis, 110,
104397. https://doi.org/10.1016/j.engfailanal.2020.104397
289. Konstantinou, C., & Stoianov, I. (2020). A comparative study of statistical and machine
learning methods to infer causes of pipe breaks in water supply networks. Urban Water
Journal, 17(6), 534–548. https://doi.org/10.1080/1573062X.2020.1800758
290. Robles-Velasco, A., Cortés, P., Muñuzuri, J., & Onieva, L. (2020). Prediction of pipe failures
in water supply networks using logistic regression and support vector classification. Reliability
Engineering & System Safety, 196, 106754. https://doi.org/10.1016/j.ress.2019.106754
291. Snider, B., & McBean, E. A. (2020). Improving urban water security through pipe-
break prediction models: Machine learning or survival analysis. Journal of Environmental
Engineering, 146(3), 04019129. https://doi.org/10.1061/(ASCE)EE.1943-7870.0001657
292. Mazumder, R. K., Salman, A. M., & Li, Y. (2021). Failure risk analysis of pipelines using
data-driven machine learning algorithms. Structural Safety, 89, 102047. https://doi.org/10.
1016/j.strusafe.2020.102047
293. Mashford, J., Marlow, D., Tran, D., & May, R. (2011). Prediction of sewer condition grade
using support vector machines. Journal of Computing in Civil Engineering, 25(4), 283–290.
https://doi.org/10.1061/(ASCE)CP.1943-5487.0000089
294. Shi, F., Liu, Y., Liu, Z., & Li, E. (2018). Prediction of pipe performance with stacking ensemble
learning based approaches. Journal of Intelligent & Fuzzy Systems, 34(6), 3845–3855. https://
doi.org/10.3233/JIFS-169556
295. Peng, S., Zhang, Z., Liu, E., Liu, W., & Qiao, W. (2021). A new hybrid algorithm model for
prediction of internal corrosion rate of multiphase pipeline. Journal of Natural Gas Science
and Engineering, 85, 103716. https://doi.org/10.1016/j.jngse.2020.103716
296. El-Abbasy, M. S., Senouci, A., Zayed, T., Mirahadi, F., & Parvizsedghy, L. (2014). Artificial
neural network models for predicting condition of offshore oil and gas pipelines. Automation
in Construction, 45, 50–65. https://doi.org/10.1016/j.autcon.2014.05.003
297. Harvey, R. R., & McBean, E. A. (2014). Predicting the structural condition of individual
sanitary sewer pipes with random forests. Canadian Journal of Civil Engineering, 41(4),
294–303. https://doi.org/10.1139/cjce-2013-0431
298. Sousa, V., Matos, J. P., & Matias, N. (2014). Evaluation of artificial intelligence tool perfor-
mance and uncertainty for predicting sewer structural condition. Automation in Construction,
44, 84–91. https://doi.org/10.1016/j.autcon.2014.04.004
299. Caradot, N., Riechel, M., Fesneau, M., Hernandez, N., Torres, A., Sonnenberg, H., Eckert, E.,
Lengemann, N., Waschnewski, J., & Rouault, P. (2018). Practical benchmarking of statistical
and machine learning models for predicting the condition of sewer pipes in Berlin, Germany.
Journal of Hydroinformatics, 20(5), 1131–1147. https://doi.org/10.2166/hydro.2018.217
300. Malek Mohammadi, M. (2019). Development of condition prediction models for sanitary
sewer pipes. Doctoral dissertation, The University of Texas at Arlington. https://rc.library.uta.
edu/uta-ir/handle/10106/28665. Accessed Nov 15, 2021.
301. Shi, F., Peng, X., Liu, Z., Li, E., & Hu, Y. (2020). A data-driven approach for pipe deformation
prediction based on soil properties and weather conditions. Sustainable Cities and Society,
55, 102012. https://doi.org/10.1016/j.scs.2019.102012
302. Malek Mohammadi, M., Najafi, M., Salehabadi, N., Serajiantehrani, R., & Kaushal, V. (2020).
Predicting condition of sanitary sewer pipes with gradient boosting tree. In Pipelines 2020
(pp. 80–89). American Society of Civil Engineers. https://doi.org/10.1061/978078448320
6.010
303. Yang, S., Zhang, L., Fan, J., & Sun, B. (2021). Experimental study on erosion behavior
of fracturing pipeline involving tensile stress and erosion prediction using random forest
regression. Journal of Natural Gas Science and Engineering, 87, 103760. https://doi.org/10.
1016/j.jngse.2020.103760
304. Lu, H., Xu, Z. D., Iseley, T., & Matthews, J. C. (2021). Novel data-driven framework for
predicting residual strength of corroded pipelines. Journal of Pipeline Systems Engineering
and Practice, 12(4), 04021045. https://doi.org/10.1061/(ASCE)PS.1949-1204.0000587
305. Lu, H., Iseley, T., Behbahani, S., & Fu, L. (2020). Leakage detection techniques for oil and
gas pipelines: State-of-the-art. Tunnelling and Underground Space Technology, 98, 103249.
https://doi.org/10.1016/j.tust.2019.103249
306. Yang, M. D., & Su, T. C. (2008). Automated diagnosis of sewer pipe defects based on machine
learning approaches. Expert Systems with Applications, 35(3), 1327–1337. https://doi.org/10.
1016/j.eswa.2007.08.013
307. Khodayari-Rostamabad, A., Reilly, J. P., Nikolova, N. K., Hare, J. R., & Pasha, S. (2009).
Machine learning techniques for the analysis of magnetic flux leakage images in pipeline
inspection. IEEE Transactions on Magnetics, 45(8), 3073–3084. https://doi.org/10.1109/
TMAG.2009.2020160
308. Cheng, J. C., & Wang, M. (2018). Automated detection of sewer pipe defects in closed-circuit
television images using deep learning techniques. Automation in Construction, 95, 155–171.
https://doi.org/10.1016/j.autcon.2018.08.006
309. Kumar, S. S., Abraham, D. M., Jahanshahi, M. R., Iseley, T., & Starr, J. (2018). Automated
defect classification in sewer closed circuit television inspections using deep convolutional
neural networks. Automation in Construction, 91, 273–283. https://doi.org/10.1016/j.autcon.
2018.03.028
310. Hassan, S. I., Dang, L. M., Mehmood, I., Im, S., Choi, C., Kang, J., Park, Y., & Moon,
H. (2019). Underground sewer pipe condition assessment based on convolutional neural
networks. Automation in Construction, 106, 102849. https://doi.org/10.1016/j.autcon.2019.
102849
311. Li, D., Cong, A., & Guo, S. (2019). Sewer damage detection from imbalanced CCTV
inspection data using deep convolutional neural networks with hierarchical classification.
Automation in Construction, 101, 199–208. https://doi.org/10.1016/j.autcon.2019.01.017
312. Hoang, N. D., & Tran, V. D. (2019). Image processing-based detection of pipe corrosion using
texture analysis and metaheuristic-optimized machine learning approach. Computational
Intelligence and Neuroscience, 2019, 8097213. https://doi.org/10.1155/2019/8097213
313. Yin, X., Chen, Y., Bouferguene, A., Zaman, H., Al-Hussein, M., & Kurach, L. (2020). A
deep learning-based framework for an automated defect detection system for sewer pipes.
Automation in Construction, 109, 102967. https://doi.org/10.1016/j.autcon.2019.102967
314. Kumar, S. S., Wang, M., Abraham, D. M., Jahanshahi, M. R., Iseley, T., & Cheng, J. C.
(2020). Deep learning?based automated detection of sewer defects in CCTV videos. Journal of
Computing in Civil Engineering, 34(1), 04019047. https://doi.org/10.1061/(ASCE)CP.1943-
5487.0000866
315. Fang, X., Guo, W., Li, Q., Zhu, J., Chen, Z., Yu, J., Zhou, B., & Yang, H. (2020). Sewer
pipeline fault identification using anomaly detection algorithms on video sequences. IEEE
Access, 8, 39574–39586. https://doi.org/10.1109/ACCESS.2020.2975887
316. Wang, M., Kumar, S. S., & Cheng, J. C. (2021). Automated sewer pipe defect tracking in
CCTV videos based on defect detection and metric learning. Automation in Construction,
121, 103438. https://doi.org/10.1016/j.autcon.2020.103438
317. Yu, X., Lu, Y., & Gao, Q. (2021). Pipeline image diagnosis algorithm based on neural immune
ensemble learning. International Journal of Pressure Vessels and Piping, 189, 104249. https://
doi.org/10.1016/j.ijpvp.2020.104249
318. Fitchett, J. C., Karadimitriou, K., West, Z., & Hughes, D. M. (2020). Machine learning for
pipe condition assessments. Journal AWWA, 50–55. https://doi.org/10.1002/awwa.1501
References 209
319. Gao, L., Liu, T., Shao, Q., Fantuzzi, N., & Chen, W. (2020). Burst pressure of steel reinforced
flexible pipe. Marine Structures, 71, 102704.
320. Wang, Q., & Zhou, W. (2019). A new burst pressure model for thin-walled pipe elbows
containing metal-loss corrosion defects. Engineering Structures, 200, 109720.
321. Wang, Q., & Zhou, W. (2019). Burst pressure models for thin-walled pipe elbows.
International Journal of Mechanical Sciences, 159, 20–29.
322. Yeom, K. J., Kim, W. S., & Oh, K. H. (2016). Integrity assessment of API X70 pipe
with corroded girth and seam welds via numerical simulation and burst test experiments.
Engineering Failure Analysis, 70, 375–386.
323. Zheng, J., Shi, J., Shi, J., Zhong, S., Rao, J., Li, G., & Li, X. (2015). Short-term burst pressure
of polyethylene pipe reinforced by winding steel wires under various temperatures. Composite
Structures, 121, 163–171.
324. Li, M., Liu, Z., Zhao, Y., Zhou, Y., Huang, P., Li, X., et al. (2019). Effects of corrosion defect
and tensile load on injection pipe burst in CO2 flooding. Journal of Hazardous Materials,
366, 65–77.
325. Chmelko, V., Garan, M., & Berta, I. (2020). Calculation of burst pressure of pipeline with
local defect. Procedia Structural Integrity, 26, 417–421.
326. Brown, M. J., Moore, I. D., & Fam, A. (2020). Analysis of a cured-in-place pressure pipe
liner spanning circular voids. Tunnelling and Underground Space Technology, 101, 103424.
327. Shuai, Y., Shuai, J., & Di, Y. (2017). Full-scale burst experiments on dented pipeline. Oil &
Gas Storage and Transportation, 36(1), 44–48.
328. Allouti, M., Schmitt, C., Pluvinage, G., Gilgert, J., & Hariri, S. (2012). Study of the influence
of dent depth on the critical pressure of pipeline. Engineering Failure Analysis, 21, 40–51.
329. Ghaednia, H., Das, S., Wang, R., & Kania, R. (2015). Safe burst strength of a pipeline with
dent?crack defect: Effect of crack depth and operating pressure. Engineering Failure Analysis,
55, 288–299.
330. Bao, J., Zhang, S., Zhou, W., & Zhang, S. (2018, September). Evaluation of burst pres-
sure of corroded pipe segments using three-dimensional finite element analyses. In Interna-
tional pipeline conference (Vol. 51869, p. V001T03A043). American Society of Mechanical
Engineers.
331. Amaya-Gómez, R., Munoz Giraldo, F., Schoefs, F., Bastidas-Arteaga, E., & Sanchez-Silva,
M. (2019). Recollected burst tests of experimental and FEM corroded pipelines. Mendeley
Data, v1
332. Andrade, E. Q., & Benjamin, A. C. (2004, January). Structural evaluation of corrosion defects
in pipelines: comparison of FE analyses and assessment methods. In The fourteenth interna-
tional offshore and polar engineering conference. International Society of Offshore and Polar
Engineers.
333. Benjamin, A. C., Vieira, R. D., Freire, J. L. F., & de Castro, J. T. (2000, October). Burst tests
on pipeline with long external corrosion. In International pipeline conference (Vol. 40252, p.
V002T06A013). American Society of Mechanical Engineers.
334. Cronin, D. (2000). Assessment of corrosion defects in pipelines. PhD thesis, University of
Waterloo.
335. Diniz, J., Vieira, R., Castro, J., Benjamin, A., & Freire, J. (2006). Stress and strain analysis
of pipelines with localized metal loss. Experimental Mechanics, 46(6), 765–775.
336. Ma, B., Shuai, J., Liu, D., & Xu, K. (2013). Assessment on failure pressure of high strength
pipeline with corrosion defects. Engineering Failure Analysis, 32, 209–219.
337. Netto, T., Ferraz, U., & Estefen, S. (2005). The effect of corrosion defects on the burst pressure
of pipelines. Journal of Constructional Steel Research, 61(8), 1185–1204.
338. Oh, C. K., Kim, Y. J., Baek, J. H., Kim, Y. P., & Kim, W. S. (2007). Ductile failure analysis of
API X65 pipes with notch-type defects using a local fracture criterion. International Journal
of Pressure Vessels and Piping, 84(8), 512–525.
339. Phan, H., Dhar, A., & Mondal, B. (2017). Revisiting burst pressure models for corroded
pipelines. Canadian Journal of Civil Engineering, 44(7), 485–494.
340. Terán, G., Capula-Colindres, S., Velázquez, J., Fernández-Cueto, M., Angeles-Herrera, D., &
Herrera-Hernández, H. (2017). Failure pressure estimations for pipes with combined corrosion
defects on the external surface: A comparative study. International Journal of Electrochemical
Science, 12, 10152–10176.
341. Wang, N., & Zarghamee, M. S. (2014). Evaluating fitness-for-service of corroded metal
pipelines: Structural reliability bases. Journal of Pipeline Systems Engineering & Practice,
5(1), 04013012.
342. Xu, L. Y., & Cheng, Y. F. (2012). Reliability and failure pressure prediction of various grades
of pipeline steel in the presence of corrosion defects and pre-strain. International Journal of
Pressure Vessels and Piping, 89, 75–84.
343. Yeom, K. J., Lee, Y. K., Oh, K. H., & Kim, W. S. (2015). Integrity assessment of a corroded
API X70 pipe with a single defect by burst pressure analysis. Engineering Failure Analysis,
57, 553–561.
344. Harvey, D., Leybourne, S., & Newbold, P. (1997). Testing the equality of prediction mean
squared errors. International Journal of Forecasting, 13(2), 281–291.
345. Chauhan, V., & Brister, J. (2009). A review of methods for assessing the remaining strength
of corroded pipelines (No. 6781).
346. Choi, J. B., Goo, B. K., Kim, J. C., Kim, Y. J., & Kim, W. S. (2003). Development of limit load
solutions for corroded gas pipelines. International Journal of Pressure Vessels and Piping,
80(2), 121–128.
347. Lewis, C. (1982). Industrial and business forecasting methods: A practical guide to
exponential smoothing and curve fitting. Butterworth-Heinemann.
348. Lu, H., Behbahani, S., Azimi, M., Matthews, J. C., Han, S., & Iseley, T. (2020). Trench-
less construction technologies for oil and gas pipelines: State-of-the-art review. Journal
of Construction Engineering and Management, 146(6), 03120001. https://doi.org/10.1061/
(ASCE)CO.1943-7862.0001819
349. Lu, H., Matthews, J. C., Azimi, M., & Iseley, T. (2020). Near real-time HDD pullback
force prediction model based on improved radial basis function neural networks. Journal of
Pipeline Systems Engineering and Practice, 11(4), 04020042. https://doi.org/10.1061/(ASC
E)PS.1949-1204.0000490
350. Yang, C. J., Zhu, W. D., Zhang, W. H., Zhu, X. H., & Ren, G. X. (2014). Determination of pipe
pullback loads in horizontal directional drilling using an advanced computational dynamic
model. Journal of Engineering Mechanics, 140(8), 04014060. https://doi.org/10.1061/(ASC
E)EM.1943-7889.0000749
351. Polak, M. A., & Lasheen, A. (2001). Mechanical modelling for pipes in horizontal directional
drilling. Tunnelling and Underground Space Technology, 16(S1), 47–55. https://doi.org/10.
1016/S0886-7798(02)00020-2
352. Cai, L., Xu, G., Polak, M. A., & Knight, M. (2017). Horizontal directional drilling pulling
forces prediction methods–A critical review. Tunnelling and Underground Space Technology,
69, 85–93. https://doi.org/10.1016/j.tust.2017.05.026
353. Baumert, M. E., & Allouche, E. N. (2002). Methods for estimating pipe pullback loads
for horizontal directional drilling (HDD) crossings. Journal of infrastructure systems, 8(1),
12–19. https://doi.org/10.1061/(ASCE)1076-0342(2002)8:1(12)
354. ASTM. (2020). Standard guide for use of maxi-horizontal directional drilling for placement
of polyethylene pipe or conduit under obstacles, including river crossings. ASTM F1962.
ASTM. https://doi.org/10.1520/F1962-20
355. Cheng, E., & Polak, M. A. (2007). Theoretical model for calculating pulling loads for pipes
in horizontal directional drilling. Tunnelling and Underground Space Technology, 22(5–6),
633–643. https://doi.org/10.1016/j.tust.2007.05.009
356. Ariaratnam, S. T., Botteicher, R. B., Ghosh, A., Piratla, K., & Vilfrant, E. (2010). Predic-
tive modeling of installation loads for directional drilled fusible PVC pipe. Tunnelling and
Underground Space Technology, 25(6), 766–772. https://doi.org/10.1016/j.tust.2009.09.003
357. Slavin, L. M., & Najafi, M. (2012). Effect of pipe stiffness on maxi-HDD pull loads. Journal of
Pipeline Systems Engineering and Practice, 3(1), 22–31. https://doi.org/10.1061/(ASCE)PS.
1949-1204.0000092
References 211
358. Cai, L., & Polak, M. A. (2019). A theoretical solution to predict pulling forces in horizontal
directional drilling installations. Tunnelling and Underground Space Technology, 83, 313–323.
https://doi.org/10.1016/j.tust.2018.09.014
359. Zhang, N., Shen, S. L., Zhou, A., & Jin, Y. F. (2021). Application of LSTM approach for
modelling stress–strain behaviour of soil. Applied Soft Computing, 100, 106959. https://doi.
org/10.1016/j.asoc.2020.106959
360. Shen, S. L., Njock, P. G. A., Zhou, A., & Lyu, H. M. (2021). Dynamic prediction of jet
grouted column diameter in soft soil using Bi-LSTM deep learning. Acta Geotechnica, 16(1),
303–315. https://doi.org/10.1007/s11440-020-01005-8
361. Lin, S. S., Shen, S. L., Zhou, A., & Xu, Y. S. (2021). Novel model for risk identification
during karst excavation. Reliability Engineering & System Safety, 209, 107435. https://doi.
org/10.1016/j.ress.2021.107435
362. Lin, S. S., Shen, S. L., Zhou, A., & Xu, Y. S. (2021). Risk assessment and management of
excavation system based on fuzzy set theory and machine learning methods. Automation in
Construction, 122, 103490. https://doi.org/10.1016/j.autcon.2020.103490
363. Tinoco, J., Correia, A. G., & Cortez, P. (2014). Support vector machines applied to uniaxial
compressive strength prediction of jet grouting columns. Computers and Geotechnics, 55,
132–140. https://doi.org/10.1016/j.compgeo.2013.08.010
364. Ochmański, M., Modoni, G., & Bzówka, J. (2015). Prediction of the diameter of jet grouting
columns with artificial neural networks. Soils and Foundations, 55(2), 425–436. https://doi.
org/10.1016/j.sandf.2015.02.016
365. Tinoco, J., Gomes Correia, A., & Cortez, P. (2018). Jet grouting column diameter prediction
based on a data-driven approach. European Journal of Environmental and Civil Engineering,
22(3), 338–358. https://doi.org/10.1080/19648189.2016.1194329
366. Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. A. M. T. (2002). A fast and elitist multiob-
jective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2),
182–197. https://doi.org/10.1109/4235.996017
367. Zhou, W., Chen, M., Yang, Z., & Song, X. (2020). Real estate risk measurement and early
warning based on PSO-SVM. In Socio-economic planning sciences (p. 101001). https://doi.
org/10.1016/j.seps.2020.101001
368. Tang, X., Hong, H., Shu, Y., Tang, H., Li, J., & Liu, W. (2019). Urban waterlogging suscep-
tibility assessment based on a PSO-SVM method using a novel repeatedly random sampling
idea to select negative samples. Journal of Hydrology, 576, 583–595. https://doi.org/10.1016/
j.jhydrol.2019.06.058
369. Deng, J., Chen, W. L., Liang, C., Wang, W. F., Xiao, Y., Wang, C. P., & Shu, C. M. (2021).
Correction model for CO detection in the coal combustion loss process in mines based on
GWO-SVM. Journal of Loss Prevention in the Process Industries, 71, 104439. https://doi.
org/10.1016/j.jlp.2021.104439
370. Barman, M., & Choudhury, N. B. D. (2020). A similarity based hybrid GWO-SVM method of
power system load forecasting for regional special event days in anomalous load situations in
Assam, India. Sustainable Cities and Society, 61, 102311. https://doi.org/10.1016/j.scs.2020.
102311
371. Guan, S., Wang, X., Hua, L., & Li, L. (2021). Quantitative ultrasonic testing for near-surface
defects of large ring forgings using feature extraction and GA-SVM. Applied Acoustics, 173,
107714. https://doi.org/10.1016/j.apacoust.2020.107714
372. Li, K., Wang, L., Wu, J., Zhang, Q., Liao, G., & Su, L. (2018). Using GA-SVM for defect
inspection of flip chips based on vibration signals. Microelectronics Reliability, 81, 159–166.
https://doi.org/10.1016/j.microrel.2017.12.032
373. Wang, S. T., Yuan, Y. Y., Zhu, C. Y., Kong, D. M., & Wang, Y. T. (2019). Discrimination of
polycyclic aromatic hydrocarbons based on fluorescence spectrometry coupled with CS-SVM.
Measurement, 139, 475–481. https://doi.org/10.1016/j.measurement.2019.01.087
374. Yin, X., Mo, W., Wang, Q., & Qin, B. (2018). A terahertz spectroscopy nondestructive identi-
fication method for rubber based on CS-SVM. Advances in Condensed Matter Physics, 2018,
1618750. https://doi.org/10.1155/2018/1618750
375. Kong, D., Chen, Y., Li, N., Duan, C., Lu, L., & Chen, D. (2019). Tool wear estimation in
end milling of titanium alloy using NPE and a novel WOA-SVM model. IEEE Transactions
on Instrumentation and Measurement, 69(7), 5219–5232. https://doi.org/10.1109/TIM.2019.
2952476
376. Zhao, C. H., Hu, H. X., Chen, B. J., Zhang, Y. N., & Xiao, J. W. (2019). Bearing fault
diagnosis based on deep learning feature extraction and WOA-SVM state recognition. Journal
of Vibration and Shock, 38(10), 31–48. https://doi.org/10.13465/j.cnki.jvs.2019.10.005
377. Sinopec. (2010). Research report on construction technology of Sichuan-East gas pipeline
project (unpublished).
Chapter 5
Data Preprocessing Technology
in Pipeline Health Monitoring
5.1 Introduction
Many years ago, engineers and scholars began to use a large amount of data obtained
from inspection and monitoring to carry out health monitoring and fault diagnosis
of building structures, and the research and application are mainly concentrated
in the field of bridges and tunnels [1–10]. Big data started relatively late in the
pipeline, but the integration of pipeline safety assessment with big data and machine
learning is an inevitable trend. With the rapid development of pipeline inspection
and environmental monitoring technology, pipeline companies have collected a large
amount of data. Currently, many companies have realized the network management
and electronic recording of massive data in the pipeline system [11].
This chapter introduces some crucial theories in pipeline big data in detail
from correlation analysis, dimension reduction, noise reduction, and missing data
completion.
5.2 Advantages of Big Data
Big data can be defined as a large amount of unstructured or structured data from
various sources. It has the characteristics of large data capacity and various types. Its
data size usually reaches PB (1024 TB) or EB (1024 PB) and the data type may include
various streaming information and images [12]. It has been popular worldwide since
2009 and has grown rapidly for more than ten years. Big data mainly includes four
technologies: cloud computing, data processing framework, storage technology, and
sensing technology. According to enlyft’s statistics [13] on 127,890 companies (as
of May 24, 2019) using big data processing platforms, the most widely used data
processing frameworks in the world are Informatica, Apache Hadoop, and Apache
Hbase, as shown in Fig. 5.1. Among them, Informatica and Apache Hadoop account
for about 50% of the market share.
https://doi.org/10.1007/978-981-19-6798-6_5
214 5 Data Preprocessing Technology in Pipeline Health Monitoring
40,000
35,000
30,000
Quantity
25,000
20,000
15,000
10,000
5,000
0
Fig. 5.1 Usage statistics of the data processing framework
Table 5.1 The amount of

Section The amount of data
data generated in different
sections of the oil and gas Drilling data 0.3 GB/well/day
industry [16] Electric submersible pump monitoring 0.4 GB/well/day
Wireline data 5 GB/well/day
Fiber optic data 0.1 GB/well/day
Seismic data 100 GB/survey
Plant process data 4–6 GB/day
Pipeline inspection 1.5 TB/600 km
Plant atmospheric data 0.1 GB/day
Plant operational data 8 GB/year
Vibration data 7.5 GB/year/customer
Different industries accept big data differently. For example, the promotion of big
data technology in the oil and gas industry is later than in other industries. According
to a survey conducted by IDC Energy in 2012, 70% of oil and gas companies in the
United States are not familiar with the concept and application scenarios of big data
technology [14]. In recent years, the oil and gas industry has begun to pay attention
to applying this technology. A large amount of data is generated every day in the oil
and gas industry, as shown in Table 5.1. If these data are used, they can produce great
value. According to a market report released by Transparency Market Research, the
value of “global oil and gas market big data” is expected to reach 10.935 billion
US dollars by 2026 [15]. Big data composition analysis is the basis of big data
applications. In the oil and gas industry, the composition of big data can be classified
according to different perspectives, as shown in Fig. 5.2. Table 5.2 lists the design
frameworks and practices for the application of big data in the oil and gas industry.
5.2 Advantages of Big Data 215
Fig. 5.2 Classification of big data in the oil and gas industry [14]
Currently, information technology is advancing rapidly, and oil and gas pipelines
have entered the stage of digital pipelines. Digital pipeline pays attention to commu-
nication technology and remote sensing, and now the digital pipeline is developing
towards intelligent pipeline [51]. The intelligent pipeline is an integrated system that
integrates technologies such as IoT, cloud computing, big data analysis, automa-
tion and intelligent control with pipelines based on the life cycle data of pipelines
and the surrounding environment. It is observable, controllable, and adaptable [52].
According to statistics, 87% of oil and gas companies believe that big data anal-
ysis is the most important technology because it is estimated that every 8000 km of
the pipeline will accumulate about 27 megabytes of information about asset health
and operations every 10 years. They can use this type of information to do the life
cycle maintenance of the pipeline [53]. In 2014, GE and Accenture jointly launched
the world’s first “Intelligent pipeline solution” [54]. As shown in Fig. 5.3, the plat-
form can solve the challenges of intelligent pipeline implementation: data fusion, data
visualization, and business process change, to realize real-time risk management,
weather, and external factor analysis. In January 2016, the Columbia Pipeline Group
became the first company to conduct an “Intelligent pipeline solution” [55]. Currently,
the Columbia Pipeline Group has real-time monitoring of more than 15,000 miles
of interstate pipelines. It integrates multiple data, including geographic information
system, work management system, one-call system, and even data from organizations
such as the United States Geological Survey, enabling pipeline threat monitoring,
risk management, and context awareness.
Table 5.2 Application scenarios of big data in the oil and gas industry [17]
Domain Application scenario References
Exploration Seismic data [18, 19]
Micro-seismic data [20]
1D, 2D, and 3D geological maps [21]
Drilling Drilling rig efficiency [22]
Drilling performance [23]
Invisible non-production time [24]
Reduce the risk of drilling operations [25]
Characterize the drill string dynamics [26]
Reservoir engineering Reservoir management application [27]
Closed-Loop Reservoir Management [28]
(CLRM) and Integrated Asset Modeling
(IAM)
Improve the CO2 sequestration [29]
Optimization on heavy oil reservoirs [30]
Reservoir modeling for unconventional [31, 32]
oil and gas resources
Improve the modeling of hydraulically [33]
fractured reservoirs
Optimize the application of EOR projects [34]
Production engineering Conduct automated decline analysis [35]
Production allocation technique [36]
Optimize the performance of electric [37, 38]
submersible pumps (ESPs)
Optimize the performance of rod pump [39]
wells
Improve hydraulic fracturing projects [40]
Conduct field development [41]
Refining Petroleum asset management [42]
Management optimization of a [43]
comprehensive refinery in Spain
Workflow to study the impact of well [44]
completion parameters on well
productivity
Transportation Improve shipping performance [45]
Health and Safety Executive (HSE) Develop an energy efficiency model [46]
during ship operations
Improve the occupational safety of the oil [47, 48]
and gas industry
Develop safety predictive analytics [49]
(continued)
5.3 Data Correlation Theory 217

Domain Application scenario References
Develop software to forecast hazard [50]
events
Fig. 5.3 Intelligent pipeline solution system [56]
5.3 Data Correlation Theory
5.3.1 Chi-Square Test
The Chi-square test is a widely used hypothesis testing method. It belongs to the
category of nonparametric test, which mainly compares the correlation analysis of
two variables. The Chi-square test indicates the deviation degree between the actual
observed value and the theoretical inferred value of the statistical sample [57, 58]. The
chi-square value is determined by the deviation between the actual double measured
value and the theoretical inferred value [59, 60]. The larger the chi-square value, the
greater the deviation between the two. If the two values are equal, the chi-square
value is 0, indicating that the theoretical value is entirely consistent with the actual
value. The chi-square distribution is shown in Fig. 5.4.
If the test assumes that H is “The distribution function of ξ is F(x), and F(x) =
F0 (x)”, while F0 (x) is a specific known distribution function. ξ1 , . . . , ξ N are obtained
from N independent
] observations of ξ . Divide R1 = (−∞, ∞) into m subintervals
(xi−1 , xi , where −∞ = x0 < x1 < · · ]· < xm = +∞. Use vi to represent the
number of ξ1 , . . . , ξ N that fall in (xi−1 , xi . It is easy to get:
Fig. 5.4 Chi-square 1.4

n=1
distribution
n=3
1.2
n=6
n=10
1.0
0.8
f (χ2)
0.6
0.4
0.2
0.0
0 5 10 15 20 25
∑
m
vi = N (5.1)
(i=1)
Let hypothesis H is “F(x) ≡ F0 (x) is correct”, and let pi = F0 (x) − F0 (xi−1 ) >
0. If H is correct, then:
vi
N
= pi (5.2)
Introduce the parameter shown in Eq. (5.1):
∑
m
(vi −N pi )2
η= N pi (5.3)
i=1
When N is sufficiently large, it can be considered that η obeys x 2 (m − 1) distri-

bution. For the existing p > 0, the constant η p can be obtained from the chi-square
distribution table, so that:
( )
P η > ηp = p
100
(5.4)
After substituting the subsample value into η, if η > η p , then the relative reliability
p
100
should negate H .
5.3.2 Information Gain and Information Gain Ratio
Information gain can also be used to measure correlation [61–63]. Entropy is an essen-
tial concept in information gain. Entropy represents the uncertainty of random vari-
ables [64, 65]. Suppose the probability distribution of the discrete random variable
X is:
P(X = xi ) = pi , i = 1, 2, . . . , n (5.5)
The entropy of X is:
∑
n
H (X ) = − pi log pi (5.6)
i=1
Assume that the joint probability distribution of the two-dimensional random

variable (X, Y ) is:
P(X = xi , Y = yi ) = pi j , i = 1, 2, . . . , n; j = 1, 2, . . . , m (5.7)
For a random variable Y given X , the conditional entropy can be defined as:
∑
n
H (Y |X ) = pi H (Y |X = xi ) (5.8)
i=1
where pi = P(X = xi ), i = 1, 2, . . . , n.
Information gain describes the degree of reduction of information uncertainty
under a condition. The information gain of feature A to training data set D can be
defined as:
g(D, A) = H (D) − H (D|A) (5.9)
The greater the information gain, the more the impurity is reduced after the intro-
duction of condition A. The greater the information gain, the greater the correlation
between the two variables.
As a common basis for distinguishing features in training sets, information gain
often selects features with a wide range of values, which is not convincing. The
information gain ratio can solve this problem to a certain extent. The information
gain ratio can be expressed as:
g R (D, A) = g(D,A)
H A (D) (5.10)
∑n |Di | |Di |
where H A (D) = − i=1 |D| log2 |D| ; n is the number of values of feature A.
5.3.3 Covariance
Assuming that the random variables Y and X have a certain linear relationship, there
are real numbers a and b such that Y ≈ a X +b. To maximize this approximation, the
mean square deviation S = E(Y − a X − b)2 is required to be minimized [66–70].
Let the mean error between Y and a X + b be c, then
EY − (a E X + b) = c (5.11)
S can be expanded:
S = E[(Y − EY ) − a(X − E X ) + c]2

[ ]
= E (Y − EY )2 + a 2 (X − E X )2 + c2 − 2a E(X − E X )(Y − EY )
= DY + a 2 D X + c2 − 2a E(X − E X )(Y − EY ) (5.12)
From the necessary conditions for the extreme value of the binary function, we
can get:
⎧
⎪
⎪
⎨ ∂S
∂a
= 2a D X − 2E(X − E X )(Y − EY ) = 0
(5.13)
⎪
⎪
⎩ ∂S
∂c
= 2c = 0
Then we can get:

[ E(X −E X )(Y −EY )
a= DX (5.14)
c=0
E(X −E X )(Y −EY )
aX + b = DX
X +b (5.15)
Combining Eqs. (5.14) and (5.15) yields:
E(X − E X )(Y − EY ) E(X − E X )(Y − EY )

aX + b = X + EY −
DX DX (5.16)
= E(X −EDX X)(Y −EY ) · (X − E X ) + EY
Thus:
[ ]
E(X − E X )(Y − EY )
S=E Y− (X − E X ) − EY
DX
[ ( )]
E(X − E X )(Y − EY ) 2
= DY 1 − √ √ (5.17)
D X · DY
−E X )(Y
E(X√ √ −EY )
To minimize S, D X · DY
must be maximized. Define:
Cov(X, Y ) = E(X − E X )(Y − EY ) (5.18)
ρX Y = √ )
√Cov(X,Y (5.19)
D X · DY
Then:
( )
S = DY 1 − ρx2y (5.20)
Therefore, the larger the |ρ X Y |, the higher the correlation between X and Y .
5.3.4 Correlation Coefficient
The correlation coefficient is a statistical index first designed by statistician Carl

Pearson [71–73]. It is a quantity to study the degree of linear correlation between
variables [74, 75]. There are three commonly used correlation coefficients: Pearson
correlation coefficient, Spearman correlation coefficient [76–78], and Kendall corre-
lation coefficient [79–83]. The Pearson correlation coefficient evaluates the linear
relationship between two continuous variables. Spearman correlation coefficient,
also known as the rank correlation coefficient, is a linear correlation analysis based
on the rank of two variables. It does not require the distribution of original variables.
It is a nonparametric statistical method with a wider range of applications. Kendall
correlation coefficient is an indicator used to reflect the correlation of classifica-
tion variables, which is applicable to the case where both classification variables are
orderly classified. The greater their absolute value, the higher the correlation between
the data. Positive values represent positive correlations between data, and negative
values represent negative correlations. Their expressions are:
cov(X,Y )
PCC = σ X σY (5.21)
∑ N
6× i=1 |R(X i )−R(Yi )|2
SCC = 1 − N 3 −N
(5.22)
2(n c −n d )
K CC = N ×(N −1) (5.23)
where PCC is Pearson correlation coefficient; cov(X, Y ) is covariance between

X and Y; σ X and σY are the standard deviation of X and Y, respectively; SCC is
the Spearman correlation coefficient; N is the number of samples; R(X i ) and R(Yi )
are the ranking of X i and Yi in their respective column vectors; KCC is Kendall
correlation coefficient; n c and n d are the number of concordant pairs and discordant
pairs, respectively.
5.4 Data Dimensionality Reduction Method
5.4.1 Principal Component Analysis
The principal component analysis is a mathematical–statistical method mainly used

for dimensionality reduction, which is widely used in many fields [84–88]. The
principal component analysis is mainly based on orthogonal transformation to trans-
form multiple variables with relevance into several comprehensive variables with a
small number of linear and almost unrelated variables. These comprehensive vari-
ables are the principal components. Therefore, principal component analysis can
solve the problem of too many variables, and the obtained principal components
as new variables not only reduce the number, but also reflect the primary informa-
tion of the original data. In other words, principal component analysis is mainly to
delete the redundant information in the original variables, thus playing the role of
dimensionality reduction [89, 90]. Its principle is as follows:
Suppose the data sample is Y = [y1 , y1 , . . . , yn ]T ∈ R n×m , n is the number of
samples, m is the number of variables. The covariance matrix can be defined as:
YT Y
M= n−1
(5.24)
The statistical analysis model established by the PCA method is as follows:
Y = d 1 q 1T + d 2 q 2T + · · · + d m q mT = D Q T + E (5.25)
where D is the score matrix; Q is the load matrix; E is the residual matrix; d is
the score vector; q is the load vector.
If both ends of Eq. (5.25) are multiplied by q i at the same time, we can get:
Y qi = di (5.26)
From Eq. (5.26), the raw data sample Y ’s projection on the load vector Q can be
obtained. The larger the projection, the more original information contained in the
sample data. As shown in Fig. 5.5, F1 contains the main information in the original
data. Even if F 2 is lost, too much information will not be lost.
5.4.2 Linear Discriminant Analysis
Linear discriminant analysis is a classical data dimensionality reduction algorithm

[91–94]. It uses the Fisher function to maximize the generalized Rayleigh quotient
to find a set of eigenvectors. The obtained eigenvectors are projection matrix, which
minimizes the intra-class variance and maximizes the inter-class variance of the
projected data [95–97]. After the projection of samples, the distance between samples
5.4 Data Dimensionality Reduction Method 223
Fig. 5.5 Geometric x2 F1

significance of principal
component analysis F2
x1
belonging to different classes is more obvious, and the samples belonging to the same
class are more concentrated so that the projected space can have the best separability,
as shown in Fig. 5.6.
Assuming that a group of samples has k categories, and n i is the number of samples
in the i-th category, and the mean value of the samples is:
∑
N
L = (1/N ) xi (5.27)
i=1
LD2
LD1
Fig. 5.6 Schematic diagram of linear discriminant analysis

The average of the samples for category i is:
∑
N
L ki = (1/n i ) xi (5.28)
i=1
The divergence matrix between categories and the divergence matrix with a
category are:
k (
∑ )( )T
Sb = (Ni /N ) L ki − L L ki − L (5.29)
i=1
k ∑ (
∑ )( )T
Sw = 1
N
x m − L ki x m − L ki (5.30)
i=1 xm ∈ki
Based on the Fisher criterion, we can get:
W T Sb W
J (W ) = W T Sw W
(5.31)
Let both sides of the equation differentiate the variable W and make its value zero:
( )
W T Sb W
∂ J (W ) ∂ W T Sw W
=
∂W ∂W (5.32)
Sb W (W T Sw W )−Sw W (W T Sb W )
= =0
(W T Sw W )2
Equation (5.32) needs to meet:

( ) ( )
Sb W W T G w W − Sw W W T Sb W = 0
(5.33)
Sb W = J (W )Sw W
5.4.3 Locally Linear Embedding
Local linear embedding is to represent high-dimensional data through local linear

relations [98–101]. The high-dimensional data sample point X is mapped to the low-
dimensional space for reconstruction, as shown in Fig. 5.7. The relevant theories are
as follows:
Assuming that the number of samples of nearest neighbors is k, the k nearest
neighbors of a sample are selected by Euclidean distance measurement. After finding
the k nearest neighbors of a sample xi , the linear relationship between xi and the k
nearest neighbors is obtained, which becomes a regression problem. Suppose that m
n-dimensional samples (x1 , x2 , . . . , xm ) use the mean square deviation as the loss
Fig. 5.7 Schematic diagram y Three-

of locally linear embedding
dimensional
Two-
y dimensional
function of the regression problem [102–108]:

|| ||2
|| ∑ ||
|| ||
J (w) = min|| x i − wi j xi j || (5.34)
|| j∈Q(i ) ||
2
where Q(i ) is the k nearest neighbor sample sets of i. Normalize the weight
coefficient wi j , the weight coefficient shall meet the following requirements:
∑
wi j = 1 (5.35)
j∈Q(i)
Substitute Eq. (5.35) into Eq. (5.34):

|| ||2
|| ∑ ( )||
|| ||
J (w) = || w x − x j ||
|| j∈Q(i) i j i || (5.36)
2
( ) T ( )
= wiT x i − x j x i − x j wi = wiT z i wi
( )T ( )
where z i = x i − x j x i − x j , wi = (wi1 , wi1 , . . . , wik )2 .
Use the Lagrange multiplier method to solve Eq. (5.34):
z i−1 1k
wi = (5.37)
1Tk z i−1 1k
where 1k is k-dimensional all-one vector. Use wi j to reconstruct the vector y so that

the minimized quadratic form J ( y) is:
|| ||2
∑ |||| ∑
m ||
||
J ( y) = min || wi j yi j ||
|| yi − || (5.38)
j∈Q(i)|| j=1 ||
2
According to the constraint condition Eq. (5.39) and let M = (I − w)T (I − w),
then Eq. (5.38) can be transformed into Eq. (5.40).
⎧
⎪ ∑
m
⎪
⎨ yi = 0
i=1
∑
m (5.39)
⎪
⎪ 1
yi yiT = I
⎩m
i=1
( )
J (Y ) = tr Y MY T (5.40)
5.4.4 Laplacian Eigenmaps
The Laplacian eigenmaps is a typical manifold learning method, which uses the
graph Laplacian concept to calculate a high-dimensional feature set to obtain a low-
dimensional manifold representation [109–123]. The essence of the dimensionality
reduction of the Laplacian eigenmaps is to find an average sense to keep the local
neighborhood information of the data points. That is, the points that are neighbors
in the original high-dimensional feature space should also be neighbors in the low-
dimensional representation.
Given the parameter k, if x i and x j are k-nearest neighbors to each other, then
there is edge connection between x i and x j . Assuming that the weight matrix is W ,
if there are edge connections between x i and x j , then:
( )
||x −x ||2
W i, j = exp − i t j (5.41)
t is the preset parameter. If x i and x j do not have an edge connection, then
W i, j = 0 (5.42)
Let D be a diagonal matrix of order N × N with diagonal elements Di, j =

∑
N
W i, j , L = D − W . From the basic idea of Laplacian eigenmaps, the following
j=1
optimization problems need to be solved:
∑
N
min 1
2
||z i − z j ||2 Wi, j , s.t. Z T D Z = I d (5.43)
Z i, j=1
where Z = [z 1 , z 2 , . . . , z N ]T , I d is d-order identity matrix. Since
∑
N ( )
1
2
||z i − z j ||2 Wi, j = tr Z T L Z (5.44)
i, j=1
Then Eq. (5.43) can be expressed as:

( )
min tr Z T L Z s.t. Z T D Z = I d (5.45)
Z
The generalized eigenvalue problem can be obtained by the Lagrange multiplier

method:
L f = λD f (5.46)
Suppose the N eigenvectors satisfying Eq. (5.46) are f 0 , f 1 , . . . , f N −1 , corre-

sponding to the eigenvalues λ0 , λ1 , . . . , λ N −1 , then the lower dimension of x i is
( )T
expressed as z i = f 1 (i ), f 2 (i ), . . . , f d (i ) . If the class label l(x i ) of x i in the
dataset is known, supervised Laplacian eigenmaps can be obtained. Let
∑N ( )
1 ||x i − x j ||2
AS(x i ) = exp −
N − 1 i/= j, j=1 t
( ) (5.47)
Nw (x i ) = ( {x j |l(x i ) 2= l) x j
exp −||x i − x j || /t > AS(x i )}
Then:
( )
||x −x ||2
W i, j = exp − i t j (5.48)
5.4.5 High Correlation Filtering
When the two data trends are similar, they contain similar information. Therefore,
deleting highly similar data can still meet the input of the machine learning model, and
can reduce the data dimension. The similarity between the two data is determined by
the Pearson correlation coefficient. The greater the value of the correlation coefficient,
the higher the degree of similarity [124]. The flow of this method is shown in Fig. 5.8.
In this process, the Pearson correlation coefficient is used, as shown in Eq. (5.21).
After obtaining the normalized data, assuming that there are d-dimensional data,
the correlation coefficient matrix is obtained:
Start
Data normalization
Get the correlation coefficient

matrix
Data zeroing processing for

matrix
Threshold setting
Get the maximum value of the

matrix
No
End Is it greater than the threshold?
Yes
Select and save a dimension
Delete the correlation coefficient

of the selected dimension
Fig. 5.8 Execution process of high correlation filtering

⎛ ⎞
1 p1,2 ··· p1,d
⎜ p2,1 1 ··· p2,d ⎟
⎜ ⎟
R=⎜ . .. .. .. ⎟ (5.49)
⎝ .. . . . ⎠
pd,1 pd,2 ··· 1
The data on the positive diagonal and the data above the positive diagonal of the
matrix R are zeroed to obtain R ∗ :
⎛ ⎞
0 0 ··· 0
⎜ p2,1 0 · · · 0⎟
⎜ ⎟
R∗ = ⎜ . .. . . .. ⎟ (5.50)
⎝ .. . . .⎠
pd,1 pd,2 · · · 0
5.4.6 Factor Analysis
Factor analysis is a kind of multivariate statistical analysis method which starts from
the study of the internal dependence of the index correlation matrix and attributes
some variables with overlapping information and complex relationships to a few
unrelated comprehensive factors [125, 126]. The basic idea is to group the variables
according to the correlation so that the variables in the same group have a high
correlation. However, the variables in different groups have no correlation or low
correlation, and each group of variables represents a common factor. On the premise
of retaining most of the original indicator information, it can eliminate the correlation
between the original indicators, reconstruct a new interpretable common factor, and
obtain the weight of each indicator, which objectively reflects the relative importance
of each indicator.
⎧
⎪
⎪ X 1 = a11 F1 + a12 F2 + · · · + a1m Fm
⎪
⎨ X 2 = a21 F1 + a22 F2 + · · · + a2m Fm
⎪ .. (5.51)
⎪
⎪ .
⎩
X p = a p1 F1 + a p2 F2 + · · · + a pm Fm
where X 1 , X 2 , . . . , X p are p original variables, which are standardized variables

with a mean of zero and a standard deviation of 1; ai j (i = 1, 2, 3, . . . , p; j =
1, 2, 3, . . . , m) is factor load, which is the correlation coefficient between the i-th
variable and the j-th common factor. F1 , F2 , . . . , Fm are m common factors. The
matrix form of Eq. (5.51) is:
X = AF + aε (5.52)
( )
where X is the original variable; A is the factor loading matrix and A = ai j p×m ;
a is a constant; ε is a special factor.
5.4.7 Independent Component Analysis
Independent component analysis is a high-dimensional signal processing method

that looks at higher-order statistical features between data [127–129]. Its purpose is
to decompose a multidimensional signal into a linear combination of a set of statis-
tically independent random variables. To extract independent features from a one-
dimensional signal, independent component analysis can be applied to some arbitrary
combination of equal-segment segmentations of the signal. Assuming that the data
length of the segment x is m, it can be decomposed into basis functions and mutually
independent coefficients through independent component analysis transformation:
∑
m
x = A·s = ai · si (5.53)
i=1
where s is independence coefficient; A is mixing matrix; ai is basis function. Let

W = A−1 , then:
s =W ·x (5.54)
Since s is required to be an independent random variable, then:

[ ]
Rs = E ssT = I (5.55)
For Eq. (5.56) to hold, Eqs. (5.57) and (5.58) need to be satisfied.
[ ] [ ] [ ]
E ssT = E W x x T W T = W E x x T W T (5.56)
[ ]
E x xT = I (5.57)
W WT = I (5.58)
To extract the feature of independent basis, the mixing matrix A or the separation
matrix W must be trained. Assuming y = wT x, then:
( )T
y = wT x = wT As = AT w s = z T s (5.59)
where y is a linear combination of si .

5.5 Data Noise Reduction Method 231
5.5 Data Noise Reduction Method
5.5.1 Wavelet Transform
Wavelet transform is a transform analysis method, which retains and develops the idea
of localization of short-time Fourier transform, and overcomes the shortcoming that
the window size does not change with frequency [130–132]. Assume ψ(t) ∈ L 2 (R),
and its Fourier transform is ψ̂(ω). When ψ̂(ω) satisfies the condition of Eq. (5.60),
then ψ(t) is a basic wavelet.
| |2
| |
{ |ψ̂(ω)|
Cψ = |ω| dω < ∞ (5.60)
x
After scaling and translation, a wavelet sequence ψa,∆ (t) can be obtained:
( )
ψa∆ (t) − √1 w t−b
|a| a (5.61)
where a is the scale factor; b is the displacement factor. a, b ∈ R and a /= 0.

For any function s(t) ∈ L 2 (R), the continuous wavelet transform is:
{w ( t−b )
Wv (a, b) = <s(t), ψa,b > = √1
|a|
s(t)ψ a
dt (5.62)
−∞
where ψa,b (t) is the conjugate function of ψa.b (t).

The inverse transformation is:
{∞ {− ( )
s(t) = 1
Cψ a 2 s (a,
1
W b)ψ t−b
a
dadb (5.63)
−∞ −∞
Because the wavelet ψa,b (t) generated by the basic wavelet ψ(t) acts as an obser-
vation window for the analyzed signal in the wavelet transform. Therefore, ψ(t)
should also meet the constraints of general functions:
{∞
|ψ(t)|dt < ∞ (5.64)
−∞
To satisfy the complete reconstruction condition, a must be equal to 0 at the origin:
{∗
ψ̂(0) = ψ(t)dt = 0 (5.65)
−∞
The wavelet transform in which the expansion and contraction of the scale factor
a and the displacement of the displacement factor b are continuous values in ψ a,b (t)
is called continuous wavelet transform. In practical signal processing, continuous

wavelet transform needs to be discretized. This discrete change is for the continuous
displacement factor b of the continuous scale factor a, not for the time variable t. In
a continuous wavelet:
( )
ψa,b (t) = √1 ψ t−b
a a (5.66)
where a ∈ R + and a /= 0, b ∈ R. Since a is a positive value in the compatibility

condition, the compatibility condition is:
| |
{∞ ||ψ̂(ω)||
Cψ = |ω| dω < ∞ (5.67)
0
Discretizing a and b, we get:

[ j
a = a0
' (5.68)
b = ka0 b0
The corresponding discrete wavelet function ψi,k (t) can be written as:
( j ) ( )
− j/2 t−ka0 b0 − jn −j
ψ j,k (t) = a0 ψ j = a0 ψ a0 t − kb0 (5.69)
a0
The discretized wavelet coefficients can be expressed as:
{∞
C j,k = s(t)ψ +
j,k (t)dt = <s, ψ j,k > (5.70)
−∞
Its reconstruction formula is:

∑
∞ ∑
∞
s(t) = C C j,k ψ j,k (t) (5.71)
−∞ −∞
Assuming the original signal is f (t), the noisy signal s(t) can be expressed as:
s(t) = f (t) + n(t) (5.72)
where n(t) is white Gaussian noise with the variance of σ 2 . According to the
properties of the wavelet transform, we can get:
W Ts (a, b) = W TS (a, b) + W Tn (a, b) (5.73)
Find the average value to get:

{ }
E nn T = σ 2 I (5.74)
Since the wavelet transform is an orthogonal change matrix, the matrix W after
the wavelet transform is an orthogonal matrix, and the wavelet transforms of s(t),
f (t) and u(t) can be represented by vectors S, F, and N :
⎧
⎨ S = Ws
F = Wf (5.75)
⎩
N = Wn
Based on Eq. (5.73), we can get:
S=F+N (5.76)
Since:
E N = E W n = W En = 0 (5.77)
Then the covariance matrix of N can be expressed as:
Q = E{N N r } = E{W nn r W r } (5.78)
5.5.2 Empirical Mode Decomposition
The empirical mode decomposition (EMD) method has been considered a break-
through in linear and steady-state spectrum analysis based on Fourier transform
since 2000 [133]. This method decomposes the signal according to the time scale
characteristics of the data itself without setting any basis function in advance.
EMD is the core of Hilbert Huang’s transform computation. This method decom-
poses according to its characteristics to obtain a finite number of intrinsic mode func-
tions (IMF) whose frequencies are arranged from high to low. The IMF represents a
simple oscillation pattern with varying amplitudes and frequencies as a function of
time. The screening process of IMF is mainly to eliminate some additional waves and
make the waveform more symmetrical. IMF is defined as follows: in the complete
data, the number of extreme points and zero-crossing points is equal, or the differ-
ence between them is 1; For any point, the mean value of the envelope defined by
the local maximum and the envelope defined by the local minimum is equal to 0.
According to the definition and screening purpose of IMF, identify all local
extreme points and use the cubic spline function to connect all local extreme points,
respectively, to obtain the upper and lower envelopes, which need to cover all data.
Assume the original signal is x(t), m 1 is the mean value of the upper and lower
envelope, and we can get:
h 1 = x(t) − m 1 (5.79)
In the second screening, the first result h 1 is regarded as the original signal, then:
h2 = h1 − m2 (5.80)
Followed by analogy:
h n = h n−1 − m n (5.81)
where n is the decomposition times. The final remaining signal is:
rn = x(t) − h n (5.82)
When rn is a monotone function, no IMF can be screened out. That is, when the
decomposition conditions are not met, the decomposition stops. The final residual
function may not be zero even if the mean value is zero. Assuming that the data has
a particular trend, the final residual function is the same as the trend of the data. The
stopping criterion is usually detected by Cauchy convergence, which requires that
the standard deviation of two adjacent fractional operations is very small. Standard
deviation Sd is defined as:
∑
T
|h n−1 (t)−h n (t)|2
Sd = h 2n (t)
(5.83)
t=0
The standard deviation Sd is always between 0.2 to 0.3. Finally, the original signal
can be expressed as:
∑
n
x(t) = ci + rn (5.84)
i=1
where ci is the IMF obtained by the i-th screening; rn is the residual signal. To ensure
the accuracy of the EMD method, it is not necessary to calculate the average of the
upper and lower envelopes. The process of EMD is shown in Fig. 5.9.
In practice, scholars have found a mode mixing problem in EMD. For this reason,
Huang adds white noise to the signal to be decomposed, and uses the uniform
distribution of the white noise spectrum. When the signal is added to the white
noise background with consistent time–frequency spatial distribution throughout the
whole time–frequency space, the signals of different time scales will be automatically
distributed to the appropriate reference scale. Due to the characteristics of zero mean
noise, the noise will cancel each other after multiple averaging, and the result of the
integrated mean can be taken as the final result. This approach is called ensemble
empirical mode decomposition (EEMD) [134]. The white noise signal is input into
the signal x(t) to obtain:
Start
Input signal
Determine extreme points
Cubic spline fitting upper and

lower envelope
Calculate the mean value of upper

and lower envelopes
h=x(t)-m
h meet IMF conditions? x(t)=h
Yes
No
n=n+1, c(n)=h, r=r-c(n)
End
Fig. 5.9 Execution process of EMD
y(t) = x(t) + k · σx · n(t) (5.85)
where k is the scale factor; σx is the standard deviation of the signal; n(t) is normalized
white noise.
The remaining steps are the same as those of EMD. Although EEMD reduces
the mode mixing problem of EMD, it still has residual noise. Therefore, complete
ensemble empirical mode decomposition with adaptive noise (CEEMDAN) appeared
later [135]. CEEMDAN is a non-linear, non-stationary data processing method based
on EMD and EEMD approaches, with the characteristics of fast calculation speed
and small reconstruction error. The execution process of CEEMDAN is described as
follows:
Let d(t) be the original signal, by adding wn i (t) with a standard normal
distribution, the i-th signal sequence is:
d i (t) = d(t) + ε0 wn i (t), i = 1, 2, . . . I (5.86)

where wn i (t) is white Gaussian noise; d i (t) is the i-th signal sequence; ε is noise
standard deviation; I is the number of tests.
Then the EMD decomposition is performed on the signal after the first test, and
the components obtained by the decomposition are averaged. That is, the first modal
component is:
∑
I
I M F1 (t) = I −1 I M F1i (t) (5.87)
i=1
where I M Fs are intrinsic mode functions; I M Fk is the k-th modal component.

The margin signal of the[first stage] (k = 1) is given as: r1 (t) = d(t) − I M F1 (t).
Then the signal r1 (t)+ε1 E 1 wn i (t) can be further decomposed to obtain the second
modal component:
∑
I { [ ]}
I M F2 (t) = I −1 E 1 r1 (t) + ε1 E 1 wn i (t) (5.88)
i=1
where E k (·) is the k-th modal component obtained by EMD decomposition; r is

the residue.
In the following stages, the k-th margin signal can be calculated as:
rk (t) = rk−1 (t) − I M Fk (t) (5.89)
The (k + 1)-th modal component is computed as:
∑
I
{ }
I M Fk+1 (t) = I −1 E 1 rk (t) + εk E k [wn i (t)] (5.90)
i=1
Repeat Eq. (5.89) until the residue component no longer satisfies the decomposi-
tion condition. Finally, the original signal d(t) is decomposed into:
∑
K
d(t) = I M Fi (t) + R(t) (5.91)
i=1
where R is the final residue.

ICEEMDAN is a method that appeared in 2014, and its purpose is to overcome
two defects of CEEMDAN [136]. This method has high efficiency and can avoid the
generation of spurious modes. Its implementation process is as follows:
(1) Perform I times EMD decomposition on the original signal:
[ ]
s (i ) = s + λ0 E 1 w (i ) (5.92)
where s is the original signal; E k (·) is the k-th mode component generated by EMD;
w (i ) is Gaussian noise; s (i ) is noise added signal; λ is noise amplitude.
Table 5.3 Statistical

Statistical parameter Value
parameters of the data
Maximum 62.53
Minimum 0.89
Mean 9.72
Standard deviation 7.78
Skewness 1.75
Kurtosis 6.98
(2) Calculate the first residue and the first mode:

< [ ]>
r1 = M s (i ) (5.93)
I M F1 = s − r1 (5.94)
where rk is the k-th residue; I M Fk is the k-th mode; M(·) is local average of signal.
(3) Calculate the second residue and the second mode:
[ ]
r2 = <Mr1 + λ1 E 2 w (i) > (5.95)
[ ]
I M F2 = r1 − <Mr1 + λ1 E 2 w (i) > (5.96)
(4) Calculate the k-th residue and the k-th mode:

[ ]
rk = <Mrk−1 + λk−1 E k w (i ) > (5.97)
I M Fk = rk−1 − rk (5.98)
(5) Repeat step (4) until the termination condition of decomposition is satisfied.
Taking a set of data with 2000 observations as an example (the raw data can be
seen in Fig. 5.10, and the statistical parameters of the data are shown in Table 5.3),
we used four methods to decompose the data, and the relevant parameters are set as
shown in Table 5.4. The decomposition results are shown in Fig. 5.11.
5.5.3 Variational Mode Decomposition
Variational mode decomposition (VMD) is an advanced signal processing technique

developed by Dragomiretskiy et al. to decompose a complex signal into a set of
quasi-orthogonal IMFs in a non-recursive manner [138]. A typical VMD process is
to decompose the signal f (t) into several modes u k (t), which can be expressed as:
Table 5.4 Parameter settings

Data decomposition method Parameters
of various data decomposition
methods EMD resolution = 50; residual energy
= 50; gradient step size = 1
EEMD NSD = 0.2; NR = 500; MNSI =
5000
CEEMDAN NSD = 0.2; NR = 500; MNSI =
5000
ICEEMDAN NSD = 0.2; NR = 500; MNSI =
5000
Note NSD denotes noise standard deviation; NR denotes
the number of realizations; MNSI denotes the maximum number
of sifting iterations allowed
Fig. 5.10 2000 observations 80

in the experiment [137]
60
40
20
0
0 200 400 600 800 1000 1200 1400 1600 1800 2000
∑
k
f (t) = u k (t) (5.99)
k=1
where k is the mode number; u k (t) is the narrowband mode, which can be expressed
as:
u k (t) = Ak (t) cos[ϕk (t)] (5.100)
where Ak (t) is instantaneous amplitude; ϕk (t) is the instantaneous phase. The sum
of all the independent modes shall be obtained using the least-squares estimation in
the original signal f (t), where the bandwidth can be expressed as:
[ ]
Bw = δ(t) + j
πt
∗ u k (t) (5.101)
where δ(t) is Dirac distribution; ∗ is convolution operator. The exponential term is

multiplied by the Fourier transform modulation of the mode u k (t), and the spectrum is
transformed into the corresponding estimated central frequency band ωk . Therefore,
it is expressed as:
Fig. 5.11 Data decomposition results a EMD, b EEMD, c CEEMDAN, d ICEEMDAN [137]
'
[( ) ]
Bw = δ(t) + j
πt
∗ u k (t) e− jωk t (5.102)
{ }
where {ωk } = ω1 , ω2 , . . . , ωk are the center frequency of each mode. Finally,
the bandwidth of each mode is obtained by Gaussian smooth demodulation signal:
[|| [( ) ] ||2 }
|| − j ωk t ||
BWuk = ||∂ t δ(t) + j
πt
∗ u k (t) e || (5.103)
2
If each modal component has a finite bandwidth with a different center frequency,
the goal is to minimize the sum of the bandwidth estimates for each modal component
using Eq. (5.104) by solving the framework of the variational constrained model.
[ || [( ) ] ||2 }
∑|| − j ωk t ||
||∂t δ(t) + π t ∗ u k (t) e
j
min ||
u k ,ωk k 2
∑ (5.104)
Subject to uk = f
k
The augmented Lagrangian function form of Eq. (5.104) is:

|| ||2
∑|| [( j
) ]
||
L(u k , ωk , λ) = α || − jωk t ||
||∂t δ(t) + π t ∗ u k (t) e ||
k 2
|| ||2 (5.105)
|| ∑ || ∑
|| ||
+|| f (t) − u k (t)|| + λ, f (t) − uk
|| ||
k 2 k
The alternating direction multiplier method changes each component and its center
frequency to solve the problem and find the optimal solution. The mode u k and the
center frequency ωk are updated as:
k (ω)
û n+1
∑ ∑ n
(5.106)
fˆ(ω) û in+1 (ω)− i >k û in (ω)+ λ 2(ω)
= i <k
1+2α ( )
2
ω−ωkn
{∞
| |
2
0 ω û k (ω) dω
n+1
ωkn+1 = { ∞ n+1 (5.107)
| 2
0 û k (ω) dω |
where û in (ω) is the sum of the mode u k in the spectral domain and the original signal
f ; ωk is the corresponding center frequency; n is the number of iterations.
5.5.4 Singular Spectrum Analysis
Singular spectrum analysis is a method for dealing with nonlinear time series data
[139]. By decomposing the trajectory matrix of the time series, calculating the eigen-
values of the time-series matrix, identifying the smaller eigenvalues as the noise
subspace, and retaining the larger eigenvalues to reconstruct the signal, thereby
realizing the suppression of noise.
The execution process of singular spectrum analysis can be divided into denoising
and reconstruction. In the denoising process, an appropriate window length m needs
to be selected. In general, the value principle of m is m < L/2. Let n = L − m + 1,
and then the original time series can be lag-arranged to get the trajectory matrix:
⎡ ⎤
g1 g2 · · · gn
⎢ g2 g3 · · · gn+1 ⎥
⎢ ⎥
g=⎢ . .. .. ⎥ (5.108)
⎣ .. . . ⎦
gm gm+1 · · · g L
where g is any valid signal component, its covariance matrix S is:
1 ∑
L−τ
Sτ = gτ gτ +t (5.109)
L − τ t=1
0where 0 ≤ τ ≤ m − 1. Perform singular value decomposition on S to obtain

λ1 ≥ λ2 ≥ · · · ≥ λm ≥ 0, and the corresponding eigenvectors are U 1 , U 2 , . . . , U m ,
and we can get:
⎧
⎨ g = ∑ √λ U V T
m
k k k
k=1 √ (5.110)
⎩
V k = g T U k / λk
√
where λk is the k-th singular spectral value. The projection of g on U is:
∑
m
ak,i = gi+ j Uk, j (5.111)
j=1
Uk, j is the j-th element in the eigenvector corresponding to the k-th eigenvalue,
and 0 < i < L − m; ak,i is the time evolution weight. In the reconstruction process,
the first v eigenvalues are selected, and the corresponding {λv } form a new subset,
and then the matrix g can be reconstructed into a time series with the length of L:
⎧
⎪ 1 ∑
i
⎪
⎪ ak,i− j Uk, j , 1 ≤ i ≤ m − 1
⎪
⎪ i
⎪
⎪ j=1
⎨ ∑ m
xk,i = m1 ak,i− j Uk, j , m ≤ i ≤ L − m + 1 (5.112)
⎪
⎪
⎪
⎪
j=1
∑
⎪
⎪
m
⎪ 1
⎩ m−i+1 ak,i− j Uk, j , L − m + 2 ≤ i ≤ L
j=i−L+m
where xk,i is the i-th element in the reconstructed signal corresponding to the k-th
eigenvalue. The execution process of singular spectrum analysis is shown in Fig. 5.12.
Start
Signal
Embedding
Decompose
Diagonal
average SVD
Reconstruct
Grouping
Stop
Fig. 5.12 The execution process of singular spectrum analysis [140]
5.6 Data Exception Elimination and Missing Supplement

Methods
5.6.1 K-nearest Neighbor Substitution Method
K-nearest neighbor is one of the classic statistical methods, which can be used not
only for classification but also for completing missing data [141]. The principle is
to select k instances closest to the data to be classified from the training dataset, and
predict the classification of samples according to their parameters and categories,
as shown in Fig. 5.13. Squares and triangles are known data, and a five-pointed
star represents data to be supplemented. The principle of data completion using
the K-nearest neighbor is shown in Fig. 5.14.
First, calculate the distance between the target data and each data record in the
input training dataset for a given input training dataset containing target data with
missing items. The classical k-nearest neighbor uses the second-order Minkowski
distance to measure the similarity between samples, which is the distance between
points generated in cluster (analysis. It is a )distance measure
( based on quantita-)
tive variables. Define x i = xi1 , xi2 , . . . , xi p and x j = x j1 , x j2 , . . . , x j p are the
observations of samples i and j, and each sample has p different variables, then the
5.6 Data Exception Elimination and Missing Supplement Methods 243
Fig. 5.13 Principle of

K-nearest neighbor
Distance
Fig. 5.14 The execution

process of K-nearest Start
neighbor
Data initialization and building a

fully valued data matrix
Calculate the distance of the

target data from the data records
in the fully valued data matrix
Find the k-nearest neighbors of

the target data
Calculate the nearest neighbor

weights for the target data
Fill in missing data
End
Minkowski distance is:

[ p |
]1/q
∑ |q
di j (q) = | xik − x jk | (5.113)
k=1
where di j is the distance between samples i and j; q is the order; k is the k-th variable
for each sample. According to the q value, Minkowski distance can be divided into
Manhattan distance, Euclidean distance, and Chebyshev distance. When q = 1, the

first-order Minkowski distance is called the absolute value distance or Manhattan
distance:
p |
∑ |
di j (1) = |xik − x jk | (5.114)
k=1
When q = 1, the second-order Minkowski distance is called the Euclidean

distance:
/
p (
∑ )2
di j (2) = xik − x jk (5.115)
k=1
When q tends to infinity, Minkowski distance can be transferred to Chebyshev

distance:
[ p |
]1/q
∑ |q | |
di j (∞) = lim | xik − x jk | = max |xik − x jk | (5.116)
q→∞ k=1 1≤k≤ p
5.6.2 Regression Filling Method
Regression filling is a method that uses regression technology to replace the missing
data [142]. It is realized by establishing the regression model of variables about all
other variables in the data set through the multiple regression method, and predicting
the missing data of the variable with non-standardized results.
The regression filling method uses the relationship between the auxiliary variable
x and the target variable y to establish a regression model. It then uses the known
auxiliary variable information and the regression model to estimate the missing data
of the target variable. In the sample variables, if there is a high correlation between
variable x and variable y, the known data can be used to fit the regression prediction
model to predict the unknown missing data. For example, the missing variables have
a linear regression relationship with the known variables, and the prediction model
is linear obtained according to the least square method. The estimated value of the
i-th missing data can be expressed as:
∑
K
Z i = β0 + βk X ki + ei (5.117)
k=1
where β is the regression coefficient.

5.7 Multi-source Heterogeneous Data Fusion Method 245
5.7 Multi-source Heterogeneous Data Fusion Method
Multi-source heterogeneous data refers to datasets of different sources and structures

[143]. Since data sets from different sources may have different collection methods,
dimensions, expressions, and may have sparsity and inhomogeneity. They need to be
processed before being used to ensure that all data can communicate with each other
and can be accessed on the same platform. In recent years, scholars have conducted
in-depth research on multi-source data processing with the deepening of big data.
In the pipeline engineering, in addition to the pipeline body characteristics (such as
wall thickness and materials) that will affect the structural safety of the pipeline, the
safety of the pipeline in some sections is greatly affected by the environment, such as
the buried pipeline is affected by the surrounding soil, and the submarine pipeline is
affected by the surrounding seawater. For pipelines with corrosion defects, in addi-
tion to collecting the pipeline body data (such as corrosion length and width), it is also
necessary to collect the environmental parameters around the pipeline (such as soil
resistivity and pipe-soil potential). Due to the different sources, categories, charac-
teristics, and expressions of these data, there may be structured, semi-structured and
unstructured data, which belong to multi-source heterogeneous data and are difficult
to be comprehensively used for pipeline security assessment.
With the continuous development of multi-source data fusion technology, the
structure classification method of multi-source data fusion is further optimized and
updated. According to the information extraction degree of multi-source moni-
toring data preprocessing, it is classified into distributed fusion, centralized fusion,
and hybrid fusion. In addition, the fusion structure is classified into data-level
fusion, feature-level fusion, and decision-level fusion according to the data type
and processing process [144].
(1) Distributed fusion
Distributed fusion first processes the collected monitoring data through the sensor
system, then sends the monitoring data to the fusion center for analysis, and finally
obtains the overall situation and estimates the fusion results. Before the observed
data enter the fusion processor, the data must be analyzed and processed to reduce
the burden of the fusion processor. Figure 5.15 shows the structure of distributed
fusion.
(2) Centralized fusion
Centralized fusion first preprocesses the monitoring data collected by the sensor,
then comprehensively analyzes and processes the processed monitoring data in the
fusion center to obtain the characteristics of the target, and finally obtains the fusion
results. Centralized fusion has the advantages of high precision, no information loss,
and high reliability, but it requires high system performance. Figure 5.16 presents
the structure of the centralized fusion.
Feature
Sensor 1 extraction
and analysis
Sensor system 1
Feature
Monitoring Fusion Fusion
Sensor 2 extraction
objectives center results
and analysis
Sensor system 2
Feature
Sensor n extraction
and analysis
Sensor system n
Fig. 5.15 Structure of sensor-level fusion [144]
Minimal data
Sensor 1
processing
Monitoring Minimal data Fusion Fusion

Sensor 2
objectives processing center results
Minimal data
Sensor n
processing
Fig. 5.16 Structure of central-level fusion [144]
(3) Hybrid fusion
Hybrid fusion is a combination of distributed fusion and centralized fusion, so as to

further improve the reliability and accuracy of the system. Compared with distributed
fusion and centralized fusion, the hybrid fusion structure adds a more complex
data processing process, and the monitoring data collected by the sensor must be
transmitted in real-time. Figure 5.17 shows the structure of hybrid fusion.
(4) Data-level fusion
Data-level fusion first performs minimal processing on the data collected by the
sensor, and then fuses the observed data through a fusion algorithm. Finally, features
5.7 Multi-source Heterogeneous Data Fusion Method 247
Sensor level
Sensor 1
processing
Minimal data
processing
Monitoring Sensor level

objectives Sensor 2 Fusion Fusion
processing
center results
Minimal data
processing
Sensor level
Sensor n
processing
Minimal data
processing
Fig. 5.17 Structure of hybrid fusion [144]
describing the target are extracted from the fused data. In the data-level fusion, the
monitoring data collected by the sensor only needs to undergo simple data prepro-
cessing, and the original information of the data is retained to a large extent, so the
fusion information is more complete. However, data-level fusion needs to deal with
a large amount of data information and many uncertain factors, resulting in a long
fusion time and poor stability.
(5) Feature-level fusion
Feature-level fusion is to analyze and classify the observation data collected by the
sensor, extract representative feature data, and then fuse the feature data extracted
from each sensor data to obtain feature data with comprehensive attributes. Feature-
level fusion differs from data-level fusion in that it can fuse information of different
physical quantities observed by sensors. Relatively important information is extracted
in the feature-level fusion, thereby realizing the compression of the data volume,
reducing the pressure of the system, and making it easier to realize real-time
processing. However, feature-level fusion requires feature extraction of the observed
information, and then further fusion through related algorithms, resulting in a part of
data loss, and the accuracy of fusion is lower than that of data-level fusion. Figure 5.18
presents the structure of the feature-level fusion.
(6) Decision-level fusion
Decision-level fusion is to perform feature extraction and state analysis on the data
collected by different types of sensors to realize the identification and classification
of monitoring bodies, and then input the individual results into the fusion algorithm
for decision processing. Figure 5.19 is a schematic diagram of the decision-level
fusion structure. Decision-level fusion can fuse the observed data of different physical
quantities. After feature extraction and state analysis of the observation data, the
Sensor 1 Preprocessing
Feature Consistency
Monitoring Feature
Sensor 2 Preprocessing Relation -level description
objectives extraction
fusion analysis
Sensor n Preprocessing
Fig. 5.18 Structure of feature-level fusion [144]
Sensor 1 Preprocessing
Decision- Consistency
Monitoring Feature State
Sensor 2 Preprocessing level description
objectives extraction analysis
integration analysis
Sensor n Preprocessing
Fig. 5.19 Structure of decision-level fusion [144]
pressure of fusion processing is effectively reduced, thereby improving the processing

and analysis efficiency, and at the same time, the real-time performance of fusion
processing is better achieved.
References
1. Sun, L., Shang, Z., Xia, Y., Bhowmick, S., & Nagarajaiah, S. (2020). Review of bridge
structural health monitoring aided by big data and artificial intelligence: From condition
assessment to damage detection. Journal of Structural Engineering, 5, 04020073.
2. Yang, J., Xiang, F., Li, R., Zhang, L., Yang, X., Jiang, S., Zhang, H., Wang, D., & Liu, X.
(2022). Intelligent bridge management via big data knowledge engineering. Automation in
Construction, 135, 104118.
3. Roshandeh, A. M., Poormirzaee, R., & Ansari, F. S. (2014). Systematic data management for
real-time bridge health monitoring using layered big data and cloud computing. International
Journal of Innovation and Scientific Research, 2(1), 29–39.
4. Wang, T., Bhuiyan, M. Z. A., Wang, G., Rahman, M. A., Wu, J., & Cao, J. (2018). Big data
reduction for a smart city’s critical infrastructural health monitoring. IEEE Communications
Magazine, 56(3), 128–133.
References 249
5. Li, R., Mo, T., Yang, J., Jiang, S., Li, T., & Liu, Y. (2020). Ontologies-based domain knowledge
modeling and heterogeneous sensor data integration for bridge health monitoring systems.
IEEE Transactions on Industrial Informatics, 17(1), 321–332.
6. Domaneschi, M., Casciati, S., Catbas, N., Cimellaro, G. P., Inaudi, D., & Marano, G. C. (2020).
Structural health monitoring of in-service tunnels. International Journal of Sustainable
Materials and Structural Systems, 4(2–4), 268–291.
7. Hashem, I. A. T., Chang, V., Anuar, N. B., Adewole, K., Yaqoob, I., Gani, A., Ahmed, E., &
Chiroma, H. (2016). The role of big data in smart city. International Journal of information
management, 36(5), 748–758.
8. Cai, G., & Mahadevan, S. (2018). Big data analytics in uncertainty quantification: Applica-
tion to structural diagnosis and prognosis. ASCE-ASME Journal of Risk and Uncertainty in
Engineering Systems, Part A: Civil Engineering, 1, 04018003.
9. Liu, Q., Lu, G., Huang, J., & Bai, D. (2020). Development of tunnel intelligent monitoring
and early warning system based on micro-service architecture: The case of AnPing tunnel.
Geomatics, Natural Hazards and Risk, 11(1), 1404–1425.
10. Tang, B., Chen, Z., Hefferman, G., Pei, S., Wei, T., He, H., & Yang, Q. (2017). Incorporating
intelligence in fog computing for big data analysis in smart cities. IEEE Transactions on
Industrial informatics, 13(5), 2140–2150.
11. Lu, H., Guo, L., Azimi, M., & Huang, K. (2019). Oil and Gas 4.0 era: A systematic review
and outlook. Computers in Industry, 111, 68–90.
12. McAfee, A., Brynjolfsson, E., Davenport, T. H., Patil, D. J., & Barton, D. (2012). Big data:
The management revolution. Harvard business review, 90(10), 60–68.
13. enlyft. (2019). Market Share of Big Data products, (Accessed 25 May, 2019). https://enlyft.
com/tech/big-data/
14. Feblowitz, J. (2012). The big deal about big data in upstream oil and gas. (Accessed 25 May,
2019). https://pdfs.semanticscholar.org/74ef/30cde5fd005f17f4e74add86b03a97d8f339.pdf
15. Transparency Market Research. (2018). Big Data in Oil and Gas Market Expected to Reach
US$ 10,935.2 Mn by 2026: Transparency Market Research. (Accessed 25 May, 2019). https://
www.transparencymarketresearch.com/pressrelease/big-data-oil-and-gas-market.htm
16. Subramaniyan, A. (2017). Putting analytics to work for you: Data and Models. (Accessed
25 May, 2019). https://www.bhge.com/system/files/2017-10/D2%20S3%20Putting%20Anal
ytics%20to%20Work.pdf
17. Mohammadpoor, M., & Torabi, F. (2020). Big data analytics in oil and gas industry: An
emerging trend. Petroleum, 6(4), 321–328.
18. Alfaleh, A., Wang, Y., Yan, B., Killough, J., Song, H., & Wei, C. (2015, September). Topolog-
ical data analysis to solve big data problem in reservoir engineering: Application to inverted
4D seismic data. In SPE Annual Technical Conference and Exhibition. Society of Petroleum
Engineers.
19. Roden, R. (2016, October). Seismic interpretation in the age of big data. In 2016 SEG
International Exposition and Annual Meeting. OnePetro.
20. Joshi, P., Thapliyal, R., Chittambakkam, A. A., Ghosh, R., Bhowmick, S., & Khan, S. N. (2018,
March). Big data analytics for micro-seismic monitoring. In Offshore Technology Conference
Asia. OnePetro.
21. Olneva, T., Kuzmin, D., Rasskazova, S., & Timirgalin, A. (2018, September). Big data
approach for geological study of the big region West Siberia. In SPE Annual Technical
Conference and Exhibition. OnePetro.
22. Duffy, W., Rigg, J., & Maidla, E. (2017, March). Efficiency improvement in the bakken
realized through drilling data processing automation and the recognition and standardization
of best safe practices. In SPE/IADC Drilling Conference and Exhibition. OnePetro.
23. Maidla, E., Maidla, W., Rigg, J., Crumrine, M., & Wolf-Zoellner, P. (2018, March). Drilling
analysis using big data has been misused and abused. In IADC/SPE Drilling Conference and
Exhibition. OnePetro.
24. Yin, Q., Yang, J., Zhou, B., Jiang, M., Chen, X., Fu, C., Yan, L., Li, L., Li, Y., & Liu, Z.
(2018, January). Improve the drilling operations efficiency by the big data mining of Real-
Time logging. In SPE/IADC Middle East Drilling Technology Conference and Exhibition.
OnePetro.
25. Johnston, J., & Guichard, A. (2015, May). New findings in drilling and wells using big data
analytics. In Offshore technology conference. OnePetro.
26. Hutchinson, M., Thornton, B., Theys, P., & Bolt, H. (2018, September). Optimizing drilling by
simulation and automation with big data. In SPE Annual Technical Conference and Exhibition.
OnePetro.
27. Bello, O., Yang, D., Lazarus, S., Wang, X. S., & Denney, T. (2017, May). Next generation
downhole big data platform for dynamic data-driven well and reservoir management. In SPE
Reservoir Characterisation and Simulation Conference and Exhibition. OnePetro.
28. Brulé, M. R. (2015, March). The data reservoir: How big data technologies advance data
management and analytics in E&P. In SPE Digital Energy Conference and Exhibition.
OnePetro.
29. Haghighat, S. A., Mohaghegh, S. D., Gholami, V., Shahkarami, A., & Moreno, D. (2013,
September). Using big data and smart field technology for detecting leakage in a CO2 storage
project. In SPE annual technical conference and exhibition. OnePetro.
30. Popa, A. S., Grijalva, E., Cassidy, S., Medel, J., & Cover, A. (2015, September). Intelligent
use of big data for heavy oil reservoir management. In SPE Annual Technical Conference and
31. Lin, A. (2014, September). Principles of big data algorithms and application for unconven-
tional oil and gas resources. In SPE Large Scale Computing and Big Data Challenges in
Reservoir Simulation Conference and Exhibition. OnePetro.
32. Chelmis, C., Zhao, J., Sorathia, V., Agarwal, S., & Prasanna, V. (2012, March). Semiautomatic,
semantic assistance to manual curation of data in smart oil fields. In SPE Western Regional
Meeting. OnePetro.
33. Udegbe, E., Morgan, E., & Srinivasan, S. (2017, October). From face detection to fractured
reservoir characterization: Big data analytics for restimulation candidate selection. In SPE
Annual Technical Conference and Exhibition. OnePetro.
34. Xiao, J., & Sun, X. (2017, September). Big data analytics drive EOR projects. In SPE Offshore
Europe Conference and Exhibition. OnePetro.
35. Seemann, D., Williamson, M., & Hasan, S. (2013, October). Improving resevoir manage-
ment through big data technologies. In SPE Middle East Intelligent Energy Conference and
36. Rollins, B. T., Broussard, A., Cummins, B., Smiley, A., & Dobbs, N. (2017, July). Continental
production allocation and analysis through big data. In SPE/AAPG/SEG Unconventional
Resources Technology Conference. OnePetro.
37. Sarapulov, N. P., & Khabibullin, R. A. (2017, October). Application of big data tools for
unstructured data analysis to improve ESP operation efficiency. In SPE Russian Petroleum
Technology Conference. OnePetro.
38. Gupta, S., Saputelli, L., & Nikolaou, M. (2016, October). Big data analytics workflow to
safeguard ESP operations in real-time. In SPE North America Artificial Lift Conference and
39. Palmer, T., & Turland, M. (2016, October). Proactive rod pump optimization: Leveraging big
data to accelerate and improve operations. In SPE North America Artificial Lift Conference
and Exhibition. OnePetro.
40. Betz, J. (2015). Low oil prices increase value of big data in fracturing. Journal of Petroleum
Technology, 67(4), 60–61.
41. Ockree, M., Brown, K. G., Frantz, J., Deasy, M., & John, R. (2018, October). Integrating
big data analytics into development planning optimization. In SPE/AAPG Eastern Regional
Meeting. OnePetro.
42. von Plate, M. (2016, September). Big data analytics for prognostic foresight. In SPE Intelligent
Energy International Conference and Exhibition. OnePetro.
References 251
43. Brelsford, R. (2018). Repsol launches big data AI project at tarragona refinery. Oil and Gas
Journal, 116.
44. Khvostichenko, D., & Makarychev-Mikhailov, S. (2018, February). Effect of fracturing
chemicals on well productivity: Avoiding pitfalls in big data analysis. In SPE International
Conference and Exhibition on Formation Damage Control. OnePetro.
45. Anagnostopoulos, A. (2018, June). Big data techniques for ship performance study. In The
28th International Ocean and Polar Engineering Conference. OnePetro.
46. Park, S. W., Roh, M. I., Oh, M. J., Kim, S. H., Lee, W. J., Kim, I. I., & Kim, C. Y. (2018,
June). Estimation model of energy efficiency operational indicator using public data based
on big data technology. In The 28th International Ocean and Polar Engineering Conference.
OnePetro.
47. Tarrahi, M., & Shadravan, A. (2016, April). Advanced big data analytics improves HSE
management. In SPE Bergen One Day Seminar. OnePetro.
48. Tarrahi, M., & Shadravan, A. (2016, September). Intelligent HSE big data analytics platform
promotes occupational safety. In SPE Annual Technical Conference and Exhibition. OnePetro.
49. Pettinger, C. B. (2014, June). Leading indicators, culture and big data: Using your data to
eliminate death. In ASSE Professional Development Conference and Exposition. OnePetro.
50. Cadei, L., Montini, M., Landi, F., Porcelli, F., Michetti, V., Origgi, M., & Duranton, S. (2018,
November). Big data advanced anlytics to forecast operational upsets in upstream production
system. In Abu Dhabi International Petroleum Exhibition & Conference. OnePetro.
51. General Electric. (2017). Digital Transformation of Oil & Gas Pipeline Operations. (Accessed
25 May, 2019). http://solutions.geoilandgas.com/ge-predictive-corrosion-management/dig
ital_pipeline_optimization?_ga=2.82912470.1569595665.1558839537-628690186.155883
9537
52. GE Reports Staff. (2017). When Shale Gas Met Software (Accessed 25 May, 2019). https://
www.ge.com/reports/post/97721689407/when-shale-gas-met-software/
53. Farris, A. (2012). How big data is changing the oil & gas industry (Accessed 25 May, 2019).
http://analytics-magazine.org/how-big-data-is-changing-the-oil-a-gas-industry/
54. Accenture. (2016). GE and Accenture Announce Columbia Pipeline Group First to Deploy
Break-Through “Intelligent Pipeline Solution” (Accessed 25 May, 2019). https://newsroom.
accenture.com/news/ge-and-accenture-announce-columbia-pipeline-group-first-to-deploy-
break-through-intelligent-pipeline-solution.htm
55. World Industrial Reporter. (2016). Colombia pipeline the first to use GE, Accenture’s Intel-
ligent Pipeline Solution (Accessed 25 May, 2019). https://worldindustrialreporter.com/col
ombia-pipeline-first-use-ge-accentures-intelligent-pipeline-solution/
56. Hochstein, A., Horn, E., & Palomino, M. (2015, June). Intelligent pipeline solution: leveraging
breakthrough industrial internet technologies and Big data analytics for safer, more efficient oil
and gas pipeline operations. In 10th Pipeline Technology Conference 2015. EITEP Institute.
57. McHugh, M. L. (2013). The chi-square test of independence. Biochemia Medica, 23(2),
143–149.
58. Franke, T. M., Ho, T., & Christie, C. A. (2012). The chi-square test: Often used and more
often misinterpreted. American Journal of Evaluation, 33(3), 448–458.
59. Tallarida, R. J., & Murray, R. B. (1987). Chi-square test. In Manual of pharmacologic
calculations (pp. 140–142). Springer.
60. Berkson, J. (1938). Some difficulties of interpretation encountered in the application of the
chi-square test. Journal of the American Statistical Association, 33(203), 526–536.
61. Kent, J. T. (1983). Information gain and a general measure of correlation. Biometrika, 70(1),
163–173.
62. Azhagusundari, B., & Thanamani, A. S. (2013). Feature selection based on information gain.
International Journal of Innovative Technology and Exploring Engineering (IJITEE), 2(2),
18–21.
63. Roobaert, D., Karakoulas, G., & Chawla, N. V. (2006). Information gain, correlation and
support vector machines. In Feature extraction (pp. 463–470). Springer.
64. Raileanu, L. E., & Stoffel, K. (2004). Theoretical comparison between the gini index and
information gain criteria. Annals of Mathematics and Artificial Intelligence, 41(1), 77–93.
65. Nelson, J. D. (2005). Finding useful questions: On Bayesian diagnosticity, probability, impact,
and information gain. Psychological review, 112(4), 979.
66. Miller, G. A., & Chapman, J. P. (2001). Misunderstanding analysis of covariance. Journal of
abnormal psychology, 110(1), 40.
67. Jin, S., & Yeung, D. S. (2004, June). A covariance analysis model for DDoS attack detection.
In 2004 IEEE International Conference on Communications (IEEE Cat. No. 04CH37577)
(Vol. 4, pp. 1882–1886). IEEE.
68. Cudeck, R. (1989). Analysis of correlation matrices using covariance structure models.
Psychological bulletin, 105(2), 317.
69. Jöreskog, K. G. (1978). Structural analysis of covariance and correlation matrices. Psychome-
trika, 43(4), 443–477.
70. Fornell, C. (1983). Issues in the application of covariance structure analysis: A comment.
Journal of Consumer Research, 9(4), 443–448.
71. Benesty, J., Chen, J., Huang, Y., & Cohen, I. (2009). Pearson correlation coefficient. In Noise
reduction in speech processing (pp. 1–4). Springer.
72. Benesty, J., Chen, J., & Huang, Y. (2008). On the importance of the Pearson correlation
coefficient in noise reduction. IEEE Transactions on Audio, Speech, and Language Processing,
16(4), 757–765.
73. Adler, J., & Parmryd, I. (2010). Quantifying colocalization by correlation: The Pearson corre-
lation coefficient is superior to the Mander’s overlap coefficient. Cytometry Part A, 77(8),
733–742.
74. Sedgwick, P. (2012). Pearson’s correlation coefficient. Bmj, 345.
75. Ahlgren, P., Jarneving, B., & Rousseau, R. (2003). Requirements for a cocitation similarity
measure, with special reference to Pearson’s correlation coefficient. Journal of the American
Society for Information Science and Technology, 54(6), 550–560.
76. De Winter, J. C., Gosling, S. D., & Potter, J. (2016). Comparing the Pearson and Spearman
correlation coefficients across distributions and sample sizes: A tutorial using simulations and
empirical data. Psychological Methods, 21(3), 273.
77. Artusi, R., Verderio, P., & Marubini, E. (2002). Bravais-Pearson and Spearman correlation
coefficients: Meaning, test of hypothesis and confidence interval. The International Journal
of Biological Markers, 17(2), 148–151.
78. Rebekić, A., Lončarić, Z., Petrović, S., & Marić, S. (2015). Pearson’s or Spearman’s
correlation coefficient-which one to use? Poljoprivreda, 21(2), 47–54.
79. Abdi, H. (2007). The Kendall rank correlation coefficient. Encyclopedia of Measurement and
Statistics (pp. 508–510). Sage.
80. Abdullah, M. B. (1990). On a robust correlation coefficient. Journal of the Royal Statistical
Society: Series D (The Statistician), 39(4), 455–460.
81. Puth, M. T., Neuhäuser, M., & Ruxton, G. D. (2015). Effective use of Spearman’s and Kendall’s
correlation coefficients for association between two measured traits. Animal Behaviour, 102,
77–84.
82. Valencia, D., Lillo, R. E., & Romo, J. (2019). A Kendall correlation coefficient between
functional data. Advances in Data Analysis and Classification, 13(4), 1083–1103.
83. Gideon, R. A., & Hollister, R. A. (1987). A rank correlation coefficient resistant to outliers.
Journal of the American Statistical Association, 82(398), 656–666.
84. Abdi, H., & Williams, L. J. (2010). Principal component analysis. Wiley Interdisciplinary
Reviews: Computational Statistics, 2(4), 433–459.
85. Ringnér, M. (2008). What is principal component analysis? Nature Biotechnology, 26(3),
303–304.
86. Wold, S., Esbensen, K., & Geladi, P. (1987). Principal component analysis. Chemometrics
and Intelligent Laboratory Systems, 2(1–3), 37–52.
87. Vidal, R., Ma, Y., & Sastry, S. S. (2016). Principal component analysis. In Generalized
principal component analysis (pp. 25–62). Springer.
References 253
88. Tipping, M. E., & Bishop, C. M. (1999). Probabilistic principal component analysis. Journal
of the Royal Statistical Society: Series B (Statistical Methodology), 61(3), 611–622.
89. Rao, C. R. (1964). The use and interpretation of principal component analysis in applied
research. Sankhyā: The Indian Journal of Statistics, Series A, 329–358.
90. Bollen, J., Van de Sompel, H., Hagberg, A., & Chute, R. (2009). A principal component
analysis of 39 scientific impact measures. PLoS ONE, 4(6), e6022.
91. Izenman, A. J. (2013). Linear discriminant analysis. In Modern multivariate statistical
techniques (pp. 237–280). Springer.
92. Xanthopoulos, P., Pardalos, P. M., & Trafalis, T. B. (2013). Linear discriminant analysis. In
Robust data mining (pp. 27–33). Springer.
93. Ioffe, S. (2006, May). Probabilistic linear discriminant analysis. In European Conference on
Computer Vision (pp. 531–542). Springer.
94. Prince, S. J., & Elder, J. H. (2007, October). Probabilistic linear discriminant analysis for
inferences about identity. In 2007 IEEE 11th international conference on computer vision
(pp. 1–8). IEEE.
95. Sharma, A., & Paliwal, K. K. (2015). Linear discriminant analysis for the small sample size
problem: An overview. International Journal of Machine Learning and Cybernetics, 6(3),
443–454.
96. Li, M., & Yuan, B. (2005). 2D-LDA: A statistical linear discriminant analysis for image
matrix. Pattern Recognition Letters, 26(5), 527–532.
97. Park, C. H., & Park, H. (2008). A comparison of generalized linear discriminant analysis
algorithms. Pattern Recognition, 41(3), 1083–1097.
98. Roweis, S. T., & Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear
embedding. Science, 290(5500), 2323–2326.
99. Chang, H., & Yeung, D. Y. (2006). Robust locally linear embedding. Pattern Recognition,
39(6), 1053–1065.
100. Ridder, D. D., Kouropteva, O., Okun, O., Pietikäinen, M., & Duin, R. P. (2003). Supervised
locally linear embedding. In Artificial neural networks and neural information processing—
ICANN/ICONIP 2003 (pp. 333–341). Springer.
101. Kouropteva, O., Okun, O., & Pietikäinen, M. (2005). Incremental locally linear embedding.
Pattern recognition, 38(10), 1764–1767.
102. Donoho, D. L., & Grimes, C. (2003). Hessian eigenmaps: Locally linear embedding techniques
for high-dimensional data. Proceedings of the National Academy of Sciences, 100(10), 5591–
5596.
103. Pan, Y., Ge, S. S., & Al Mamun, A. (2009). Weighted locally linear embedding for dimension
reduction. Pattern Recognition, 42(5), 798–811.
104. Zhang, S. Q. (2009). Enhanced supervised locally linear embedding. Pattern Recognition
Letters, 30(13), 1208–1218.
105. Wang, J. (2012). Locally linear embedding. In Geometric Structure of High-Dimensional
Data and Dimensionality Reduction (pp. 203–220). Springer.
106. Wang, X., Zheng, Y., Zhao, Z., & Wang, J. (2015). Bearing fault diagnosis based on statistical
locally linear embedding. Sensors, 15(7), 16225–16247.
107. Kouropteva, O., Okun, O., & Pietikäinen, M. (2003, June). Supervised locally linear embed-
ding algorithm for pattern recognition. In Iberian Conference on Pattern Recognition and
Image Analysis (pp. 386–394). Springer.
108. Chen, J., & Ma, Z. (2011). Locally linear embedding: A review. International Journal of
Pattern Recognition and Artificial Intelligence, 25(07), 985–1008.
109. Belkin, M., & Niyogi, P. (2003). Laplacian eigenmaps for dimensionality reduction and data
representation. Neural Computation, 15(6), 1373–1396.
110. Belkin, M., & Niyogi, P. (2001). Laplacian eigenmaps and spectral techniques for embedding
and clustering. Advances in Neural Information Processing systems, 14.
111. Zhu, H., Sun, K., & Koniusz, P. (2021). Contrastive laplacian eigenmaps. Advances in Neural
Information Processing Systems, 34, 5682–5695.
112. Li, B., Li, Y. R., & Zhang, X. L. (2019). A survey on Laplacian eigenmaps based manifold
learning methods. Neurocomputing, 335, 336–351.
113. Tu, S. T., Chen, J. Y., Yang, W., & Sun, H. (2011). Laplacian eigenmaps-based polarimetric
dimensionality reduction for SAR image classification. IEEE Transactions on Geoscience
and Remote Sensing, 50(1), 170–179.
114. Sprekeler, H. (2011). On the relation of slow feature analysis and laplacian eigenmaps. Neural
Computation, 23(12), 3287–3302.
115. Jia, P., Yin, J., Huang, X., & Hu, D. (2009). Incremental Laplacian eigenmaps by preserving
adjacent information between data points. Pattern Recognition Letters, 30(16), 1457–1463.
116. Lewandowski, M., Martinez-del-Rincon, J., Makris, D., & Nebel, J. C. (2010, August).
Temporal extension of laplacian eigenmaps for unsupervised dimensionality reduction of
time series. In 2010 20th International Conference on Pattern Recognition (pp. 161–164).
IEEE.
117. Ye, X., Li, H., Imakura, A., & Sakurai, T. (2020). An oversampling framework for imbalanced
classification based on Laplacian eigenmaps. Neurocomputing, 399, 107–116.
118. Lewandowski, M., Makris, D., Velastin, S. A., & Nebel, J. C. (2013). Structural Laplacian
eigenmaps for modeling sets of multivariate sequences. IEEE Transactions on Cybernetics,
44(6), 936–949.
119. Gerber, S., Tasdizen, T., & Whitaker, R. (2007, June). Robust non-linear dimensionality
reduction using successive 1-dimensional Laplacian eigenmaps. In Proceedings of the 24th
International Conference on Machine Learning (pp. 281–288).
120. Liu, F., Zhang, W., & Gu, S. (2016). Local linear Laplacian eigenmaps: A direct extension of
LLE. Pattern Recognition Letters, 75, 30–35.
121. Wang, J. (2012). Laplacian eigenmaps. In Geometric Structure of High-Dimensional Data
and Dimensionality Reduction (pp. 235–247). Springer.
122. Yu, H., Ding, M., & Zhang, X. (2019). Laplacian eigenmaps network-based nonlocal means
method for MR image denoising. Sensors, 19(13), 2918.
123. Chen, C., Zhang, L., Bu, J., Wang, C., & Chen, W. (2010). Constrained Laplacian Eigenmap
for dimensionality reduction. Neurocomputing, 73(4–6), 951–958.
124. Choi, J., Jin Chang, H., Yun, S., Fischer, T., Demiris, Y., & Young Choi, J. (2017). Attentional
correlation filter network for adaptive visual tracking. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (pp. 4807–4816).
125. Lawley, D. N., & Maxwell, A. E. (1962). Factor analysis as a statistical method. Journal of
the Royal Statistical Society. Series D (The Statistician), 12(3), 209–229.
126. Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory
factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191–
205.
127. Stone, J. V. (2002). Independent component analysis: An introduction. Trends in Cognitive
Sciences, 6(2), 59–64.
128. Comon, P. (1994). Independent component analysis, a new concept? Signal Processing, 36(3),
287–314.
129. Hyvärinen, A., & Oja, E. (2000). Independent component analysis: Algorithms and applica-
tions. Neural Networks, 13(4–5), 411–430.
130. Zhang, D. (2019). Wavelet transform. In Fundamentals of image data mining (pp. 35–44).
Springer.
131. Bentley, P. M., & McDonnell, J. T. E. (1994). Wavelet transforms: An introduction.
Electronics & Communication Engineering Journal, 6(4), 175–186.
132. Antonini, M., Barlaud, M., Mathieu, P., & Daubechies, I. (1992). Image coding using wavelet
transform. IEEE Transactions on Image Processing, 1(2), 205–220.
133. Flandrin, P., Rilling, G., & Goncalves, P. (2004). Empirical mode decomposition as a filter
bank. IEEE Signal Processing Letters, 11(2), 112–114.
134. Lei, Y., He, Z., & Zi, Y. (2009). Application of the EEMD method to rotor fault diagnosis of
rotating machinery. Mechanical Systems and Signal Processing, 23(4), 1327–1338.
References 255
135. Cao, J., Li, Z., & Li, J. (2019). Financial time series forecasting model based on CEEMDAN
and LSTM. Physica A: Statistical Mechanics and Its Applications, 519, 127–139.
136. Kou, Z., Yang, F., Wu, J., & Li, T. (2020). Application of ICEEMDAN energy entropy and
AFSA-SVM for fault diagnosis of hoist sheave bearing. Entropy, 22(12), 1347.
137. Lu, H., Xi, D., Ma, X., Zheng, S., Huang, C., & Wei, N. (2022). Hybrid machine learning
models for predicting short-term wave energy flux. Ocean Engineering, 264, 112258.
138. Bagheri, A., Ozbulut, O. E., & Harris, D. K. (2018). Structural system identification based on
variational mode decomposition. Journal of Sound and Vibration, 417, 182–197.
139. Vautard, R., Yiou, P., & Ghil, M. (1992). Singular-spectrum analysis: A toolkit for short, noisy
chaotic signals. Physica D: Nonlinear Phenomena, 58(1–4), 95–126.
140. Cao, H., Song, Y., Li, Y., Li, R., Shi, H., Yu, J., … & Wang, C. (2018). Reduction of moving
target time-of-flight measurement uncertainty in femtosecond laser ranging by singular
spectrum analysis based filtering. Applied Sciences, 8(9), 1625.
141. Peterson, L. E. (2009). K-nearest Neighbor. Scholarpedia, 4(2), 1883.
142. Sattari, M. T., Rezazadeh-Joudi, A., & Kusiak, A. (2017). Assessment of different methods for
estimation of missing data in precipitation studies. Hydrology Research, 48(4), 1032–1044.
143. Zhang, L., Xie, Y., Xidao, L., & Zhang, X. (2018, May). Multi-source heterogeneous data
fusion. In 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD)
(pp. 47–51). IEEE.
144. Zhang, Y. (2021). Research on multi-source heterogeneous data fusion method for landslide
monitoring. Chang’an University.
Chapter 6
Application and Cases of Pipeline
Inspection and Monitoring
6.1 Introduction
This chapter introduces several inspection and monitoring cases in actual projects,
including CCTV inspection, magnetic flux leakage inspection, remote field eddy
current, and landslide monitoring of pipelines.
6.2 Closed-Circuit Television (CCTV) Inspection
6.2.1 Considerations Before an Inspection
Although the operation of CCTV inspection is simple, the relevant preliminary work
needs to be comprehensively considered. For example, in some scenarios, operations
such as dredging, blocking, ventilation, and cleaning of pipelines are required.
Dredging is an essential task before CCTV inspection. According to the construc-
tion schedule, the construction machinery and equipment shall be inspected, repaired,
and maintained before construction to ensure the regular operation and safe use of
construction equipment. In addition, it is necessary to carry out special technical and
safety training for on-site management personnel, mechanical equipment operators
and construction personnel, and clarify their responsibilities. Signboards shall be set
up in the construction area to indicate the project name, scope, construction party,
supervisor, construction party, designer, the person in charge of the site, supervision
telephone, commencement date, and completion date.
Instrument inspection and biological inspection can be used and the operation can
be carried out only after meeting the requirements for gas inspection and ventilation.
In addition, the inspection wells should also be tested for toxic, harmful, flammable
and explosive gases, and the stay time of operators at the hydrogen sulfide and
methane sites should be shortened. When the air humidity is high, the air pressure
is low, and the oxygen content is low, artificial oxygen supply measures should be
https://doi.org/10.1007/978-981-19-6798-6_6
258 6 Application and Cases of Pipeline Inspection and Monitoring
adopted. Before the construction personnel enters the inspection well, it is necessary
to make the oxygen in the atmosphere enter the inspection well or use the blower for
ventilation to meet the safety requirements.
There are many construction considerations involved in pipeline plugging.
Considering the possibility of additional drainage due to branch pipe discharge or
leakage, the pipeline should be plugged in sections, as shown in Fig. 6.1. After the
pipeline is plugged, use a submersible pump and a mud pump to place it in the
inspection well to dewater the pipeline, as shown in Fig. 6.2.
High-pressure water jet technology is often used to clean pipelines. It uses water
as a medium to produce multi-beam, multi-angle, multi-intensity high-pressure water
jets through special equipment, and cuts, crushes, extrudes and scours the attachments
and plugs in the pipeline to achieve thorough cleaning purpose.
As shown in Fig. 6.3, the water jet is a liquid cone in the air, its diameter increases
with the distance from the nozzle, and the velocity of the water jet presents the
opposite trend. When the water jet impinges on the solid surface, the contact pressure
is generated in the contact zone. The shorter the distance, the greater the contact
pressure. When the contact pressure is close to or greater than the compressive
strength of the viscous material, its induced stress will cause the plastic material to
flow. However, the brittle material will be crushed and broken into multiple structural
Air pump Safety rope
gasbag gasbag
Fig. 6.1 Schematic diagram of segmented blocking

6.2 Closed-Circuit Television (CCTV) Inspection 259
Fig. 6.2 Schematic diagram of pipeline dewatering operation
Drop and fine jet

Nozzle Liquid bubble
Water flow
Fig. 6.3 Dynamic structure of water jet [1, 2]
fragments, and the scale cinder attached to the inner surface will peel off in the impact
zone, and these fragments and scale cinders will be carried away by the water flow.
Secondly, when a continuous water jet pulsates and diffuses in the air, the formed
drops or fine jets will generate local water hammer pressure in impacting the solid
surface.
The high-pressure water truck (Fig. 6.4) fills the two inspection wells involved
in this pipe section, and uses a dredger to mix the sludge in the inspection well
and sewage pipe to dilute the sludge. Sometimes the staff must go down the well
and clean up. Divers must correctly wear fully enclosed underwater diving suits and
fasten safety ropes when working in the well (Fig. 6.5). The protective equipment
worn by operators must meet the requirements of relevant safety regulations. Before
the diver is loaded and launched, all parts and equipment performance (gas supply,
safety rope, intercom) must be approved by both the diver and the guardian before
launching.
6.2.2 Project Description
The inspection was carried out on September 16, 2021. The inspection length of the
pipeline is 176.8 m, which is divided into seven sections. The material of the pipe
Fig. 6.4 High-pressure water truck
is cast iron, which is used to transport rainwater. The pipe diameter is 800 mm. The
specific construction process is shown in Fig. 6.6.
6.2.3 Evaluation Results
The project was evaluated according to “Technical Specification for Inspection and
Evaluation of Urban Sewer” (CJJ 181-2012) [3], and the evaluation results are shown
in Table 6.1. Among them, the level corresponding to the repair index of the pipeline
is shown in Table 6.2, and the defect level is shown in Table 6.3. A sample of the
inspection report is shown in Table 6.4.
6.3 Magnetic Flux Leakage Inspection
The material of XX section pipeline is spiral plus longitudinal submerged arc welded
(LSAW) steel pipe. The total length of this section is 27.55 km, and the pipe diameter
is 813 mm. The grade of steel pipe is L485, and the design pressure is 6.3 MPa. The
wall thickness of the pipe is available in five specifications: 8.8, 11, 12.5, 14.2,
6.3 Magnetic Flux Leakage Inspection 261
Fig. 6.5 Downhole operation
and 16 mm. The transmission medium in the pipeline is purified natural gas. In
order to fully understand the current situation of the pipeline, evaluate the defect
control measures, timely eliminate the potential safety hazards, reasonably use the
maintenance cost, and realize the scientific management of the pipeline, the project
adopted the three-axis high-definition magnetic flux leakage detector to implement
the intelligent internal inspection of the pipeline.
Collect data
Site survey
Water level<30%d Water level Water level≥30%d
Sludge Sludge≥20%d Sludge
Sludge≥20%d Sludge<20%d Sludge<20%d
Plugging, pumping Plugging and

Cleaning
and flushing pumping
CCTV enters the pipeline for

inspection
Image editing, defect

identification
Report preparation
Fig. 6.6 Schematic diagram of CCTV inspection process
6.3.2 Inspector Description
The pipeline magnetic flux leakage internal detector is composed of four parts: power,
measurement, computer, and power supply. Figures 6.7 and 6.8 show the specific
structure of the detector. Among them, the power part mainly relies on the pressure
difference between the pipeline transmission medium and the cup to provide forward
power. The measurement part consists of a steel brush, excitation source, and probe.
After the steel brush magnetizes the pipe wall, a circumferential probe is distributed
in the circumferential direction of the detector. The probe comprises a magnetic
sensor, a single-chip microcomputer control circuit, and related circuit elements. If
Table 6.1 Pipeline condition evaluation results
Pipe section No Diameter Length Material Buried depth Structural defects
(mm) (m) Start Endpoint Mean Maximum Defect Defect Repair Comprehensive
point level density index evaluation
YS10-2 ~ YS10-1 800 2.92 Cast iron 5 0 / / / / / /

YS10-2 ~ YS10-3 800 19.01 Cast iron 5 0 / / / / / /
YS10-3 ~ YS10-4 800 24.31 Cast iron 5 0 / / / / / /
YS41 ~ YS42 800 34.52 Cast iron 5 0 / / / / / /
YS50 ~ YS49 800 35.22 Cast iron 5 0 0.50 0.50 I 0.87 0.35 *
YS50 ~ YS51 800 25.59 Cast iron 5 0 / / / / / /
YS52 ~ YS51 800 35.23 Cast iron 5 0 / / / / / /
Note *Means there is no or slight pipeline defect, and the structural condition is basically not affected, but there is the possibility of potential deterioration. The
structural conditions are basically intact and do not need to be repaired
263
Table 6.2 Repair level of pipes

Level Repair index (RI) Explanation
I RI ≤ 1 The structural condition is basically intact. No need for repairs
II 1 < RI ≤ 4 The structure will not be damaged in the short term, but a repair plan
should be made
III 4 < RI ≤ 7 Structural damage may occur in the short term and should be
repaired as soon as possible
IV RI > 7 Damage to the structure has occurred or is about to occur and should
be repaired immediately
Table 6.3 Defect level of pipes

Level Defect index G Explanation
I G≤1 No or slight impact. The pipeline operation is basically unaffected
II 1<G≤3 There is a certain obstruction to the flow of the pipeline, and the
operation is not affected much
III 3<G≤6 The flow of the pipeline is seriously blocked, and the operation is
significantly affected
IV G>6 The flow of the pipeline is seriously obstructed, and the operation is
about to or has been paralyzed
there is a defect in the pipe wall, magnetic flux leakage will be formed. The probe
will detect and record the magnetic flux leakage information and send the signal to
the calculation section for recording and storage. The calculation section of the in-
line detector mainly completes the control and storage of probe information and
the classification of magnetic flux leakage signal processing. The battery section of
the detector is mainly used to provide a power supply for other parts in the long-
term distance measurement process, so that all parts can work in coordination. The
parameters and reliability indicators of the detector are shown in Tables 6.5, 6.6, 6.7
and 6.8.
6.3.3 Inspection Process
Magnetic flux leakage testing of pipes is usually carried out according to the process
shown in Fig. 6.9.
(1) Preparation before the inspection

Investigate the pipeline according to “Specification for in-line inspection of
oil and gas pipeline” (SY/T 6597-2018) [4], and fill in the summary informa-
tion and characteristics of the pipeline. The survey contents include pipeline
size, wall thickness, length, age, working pressure, relative elevation differ-
ence, and pipeline crossing situation. In addition, it also includes elbows, bevels,
Table 6.4 Inspection results
Video file YS50_YS49_20210916094159.avi Start well No YS50 End well No YS49
Installation 2021–09-16 Start point depth 5m End point depth 0m
year
Pipe type Storm water pipes Pipeline material Cast iron Diameter 800 mm
Inspection Reverse flow Pipe length 35.22 m Inspection length 35.22 m
direction
Repair 0.35 Maintenance index / Inspector Bill.Fung
index
Inspection Wuling Road Inspection date 2021–09-16
location
Distance Defect code Score Level Description of the internal condition of the pipeline Photo
(m)
11.29 m (PL) Rupture 0.5 1 Structural defects. Circumferential 0102 position, and the longitudinal length is 1 m 1
29.51 m (PL) Rupture 0.5 1 Structural defects. Circumferential 1011 position, and the longitudinal length is 29.51 m 2
265
Photo 1 Photo 2
universal universal
Steel brush Probe coupling coupling Mileage wheel Pipeline wall
Power joint Calculation Battery section

section
Fig. 6.7 The structure of the magnetic flux leakage detector
Fig. 6.8 Physical drawing of magnetic flux leakage detector
Table 6.5 Relevant

Parameter Value
parameters of three-axis
high-definition magnetic flux Detector length 3380 mm
leakage detector Detector weight 2580 kg
Maximum inspection distance 300 km
Battery operating time 35 h
Ability to pass elbows ≥3 DN × 90°
Deformability 670 mm
Speed range From 1 to 3 m/s
Pressure range From 0 to 10 MPa
Temperature range From −20 to 70 °C
deformation, intermediate valve chambers, branch lines, tees, special landform

conditions of the pipeline, and the repair and replacement of the pipeline. In
addition to the structural parameters, it is also necessary to investigate the histor-
ical pigging records, the current gas transmission flow, daily gas transmission,
Table 6.6 Reliability requirements for magnetic flux leakage test results
Inspection target Reliability of inspection results
Internal/external defect differentiation POI > 90%
Metal loss characteristics POI > 90%
Corrosion/corrosion group POI > 90%
Pinhole POI > 50%
Grooves and scratches POI > 50%
Dents POI > 90%
Dents with metal loss POI > 50%
Eccentric sleeve POI > 90%
Repair bushing and sleeve POI > 90%
Straight pipe POI > 98%
Valve POI > 98%
Tee POI > 98%
Elbow POI > 98%
Metal objects or ferromagnetic materials adjacent to or in POI > 90%
contact with pipes
Circumferential, straight, and spiral welds POI > 90%
Circumferential weld and spiral weld crack/abnormality POI > 50%
Table 6.7 Inspection accuracy of pipeline base metal area

Property Confidence General Pitting Axial Circumferential
level metal loss (2t*2t) groove groove (2t*4t)
(4t*4t) (4t*2t)
Minimum / 5%t 8%t 8%t 5%t
measurement depth
(inspection rate
90%)
Depth dimension 90% ±10%t ±10%t ±15%t ±10%t
accuracy
Width dimension 90% ±10 ±10 ±15 ±12
accuracy (mm)
Length dimension 90% ±10 ±10 ±12 ±15
accuracy (mm)
Minimum 8
detectable defect
diameter (mm)
Table 6.8 Inspection accuracy of heat-affected zone of girth weld

Property Confidence General Pitting Axial Circumferential
level metal loss (2t*2t) groove groove (2t*4t)
(4t*4t) (4t*2t)
Minimum / 9%t 13%t 15%t 9%t
measurement depth
(inspection rate
90%)
Depth dimension 90% ±15%t ±15%t ±15%t ±15%t
accuracy
Width dimension 90% ±20 ±15 ±20 ±20
accuracy (mm)
Length dimension 90% ±20 ±20 ±20 ±20
accuracy (mm)
Fig. 6.9 Flow chart of

magnetic flux leakage Preparation before inspection
inspection
Pigging
Magnetic flux leakage test
Signal analysis
Excavation verification
Analysis and evaluation of test

data
Final report for magnetic flux

leakage test
Fig. 6.10 Pigging operations
working pressure, transmission pressure difference, and transmission efficiency

of the pipeline.
(2) Pigging
The purpose of pigging is to clean the sundries, liquid accumulation, and dirt
in the pipeline to meet the requirements of subsequent internal inspection (see
Fig. 6.10). Foam pig, two-way pig, steel brush pig, magnetic pig, and caliper
pig are used for this pipeline, and 23 pigging operations have been carried out in
total. According to the pigging results (the dirt of the last pigging was 3.8 kg),
it can be determined that the pipeline meets the requirements of subsequent
magnetic flux leakage testing.
(3) Inspection
To ensure the integrity and location accuracy of pipeline inspection results, pipe
inspection and marking shall be carried out at an interval of less than 1 km along
the pipeline (special parts shall be densely placed). When the pipeline crosses
large rivers, expressways, railways, and other special sections, markers shall
also be set at the places where it crosses or exits. The buried depth (distance
from the ground to the center line of the pipeline) of all the calibration points
shall not exceed 3 m, and the calibration points shall not be set near railways,
highways, and high-voltage lines as far as possible (Fig. 6.11). In addition, it is
also necessary to set calibration points 5 m upstream and 5 m downstream of
the central line of the main valve. According to the above principles, a total of
29 calibration points were set for this pipeline project.
When the detector runs, it is necessary to monitor the whole line to master
its running speed and position. The setting principle of monitoring points along
the whole line is mainly determined according to the length and condition of
the pipeline. The spacing between monitoring points can be large if the pipeline
operates well during pigging. If the number of monitoring points is small or the
number of monitoring personnel is sufficient, separate monitoring personnel can
Based on the test pile, use the pipe patrol instrument

near the test pile to find the right above the pipe, and
record the relative position between this point and
the test pile. The spacing between calibration points
should be less than 1 km.
The marking points are located near mileage piles,

horizontal corner piles , and sign piles, and avoid
areas with large interference such as high-voltage
lines and railway tracks.
Key locations such as pump station, valve chamber,

large crossing and bare pipe must be calibrated. The
marking at both ends of the valve shall be 5 m away
from the valve center.
Tracking points must be set in special sections, and

points must be set in densely populated areas,
reconstruction points, known deformation points and
maintenance points, water source areas, river gullies,
and both ends of crossing. The buried depth of the
marked points should be less than 3 m.
Fig. 6.11 Selection principle of calibration point and monitoring point
be arranged at each monitoring point. If the construction conditions permit, a

feasible monitoring plan can also be developed, and several personnel groups can
track alternately to meet the requirements. According to the above principles, a
total of 7 monitoring points are set up in this pipeline (Fig. 6.12).
After confirming that the pipeline meets the operation requirements of the
magnetic flux leakage detector, the three-axis high-definition magnetic flux
Launcher Receiving
5 km 10 km 15 km ...
station station
Fig. 6.12 Example of spacing for monitoring points

Table 6.9 Defect statistics

Defect External metal Internal metal loss Abnormal Metal additions
characteristics loss circumferential
weld
0–9% 364 2076 290 39
10–19% 52 139
20–29% 2 3
30–39% 0 0
40–49% 0 0
50–59% 0 0
60–69% 0 0
70–79% 0 0
80–100% 0 0
Total number 418 2218 290 39
leakage detector shall be debugged immediately to make all technical indi-

cators meet the requirements of the operating procedures. The detector is sent
from the launcher and enters the receiver after 2 h and 56 min of operation. After
inspection, the recorder operates normally and the inspection data is complete.
The operating speed range of the detector is 1–3 m/s, and the average speed is
2.55 m/s, meeting the inspection requirements.
(4) Signal analysis
According to the magnetic flux leakage test results, 2636 abnormal features of
metal loss were detected in this pipeline, including 4 external metal loss features
and 2218 internal metal loss features. In addition, there are 30 depressions, 290
circumferential weld abnormalities, and 39 metal additions, as shown in Table
6.9.
(5) Excavation verification
According to the magnetic flux leakage test results of this pipeline, 4 metal
loss defects and 2 abnormal welds were selected for excavation verification
in accordance with the “Specification for in-line inspection of oil and gas
pipeline” (SY/T6597-2018) [4]. The signal diagram and excavation checklist
of the excavation point are shown in Figs. 6.13 and 6.14, respectively.
The main contents of this excavation verification project include ground posi-
tioning, pipe trench excavation, anti-corrosion layer peeling, defect inspection
and evaluation, anti-corrosion repair, backfilling, and landform restoration. The
final excavation verification results are shown in Table 6.10.
(6) Analysis and evaluation of test data
At present, the metal loss evaluation methods that are widely recognized inter-
nationally are based on the NG-18 surface defect formula of the American Gas
Association (AGA) Pipeline Research Committee, which is used to calculate the
failure stress of the pipeline with defects, and also used to evaluate the residual
strength of pipelines with defects, such as Eq. (6.1).
Fig. 6.13 Signal diagram of excavation point
[ ]
1− A
S=S ( )A0 (6.1)
1− AA ( M −1 )
0
where M is Folias coefficient, which depends on the axial length of defects and
pipe geometry; S is hoop stress at failure; S is flow stress of materials, which is
a physical property related to yield strength; A is the cross-sectional area of the
defect on the longitudinal plane; A0 : cross-sectional area on the longitudinal
plane without defects.
Metal loss includes loss defects and manufacturing defects, and Eq. (6.1)
is modified to evaluate loss defects and manufacturing defects. In this defect
evaluation, ASME B31G-2012 is selected to evaluate metal loss defects, and
the Shannon criterion is used to evaluate manufacturing defects.
(7) Final report for magnetic flux leakage test
The final report shall include statistical analysis of the inspection data, including
defect distribution law, future corrosion status, applicability evaluation of
defects, preparation of repair plan, and management suggestions. The final
evaluation results of magnetic flux leakage testing of this pipeline are as follows:
1. Under the design pressure of 6.3 MPa and the maximum allowable operating
pressure of 6.3 MPa, no external metal loss defects exceed the allowable size
of the B31G safety evaluation curve and need to be repaired. If the external
metal loss defect increases at the determined full life metal loss rate, there
will be no additional external metal loss to be repaired within 8 years.
2. Under the design pressure of 6.3 MPa and the maximum allowable operating
pressure of 6.3 MPa, there are no internal metal loss defects that exceed the
Feature description
Name XX
Circumferential direction 1:0.5
Length 26 mm
Width 48 mm
Depth 26%wt
Inside / outside INT
ERF 0.81
Dimension type PITT
Thickness 11.0 mm
Inspection mileage 10710.691 m
Feature location
Nearest reference point 171
Upstream circumferential weld number 10060
Distance from upstream circumferential weld 1.225 m
Inspection mileage: 10710.691 m
Reference point name 171 Reference point name 172
Reference point mileage 10538.191 m Reference point mileage 11753.381 m
3.264 m
3.474 m 3.501 m 1.225 m 2.039 m 4.341 m 3.593 m
Flow direction
Fig. 6.14 An example of an excavation checklist

Table 6.10 Excavation verification results

No Absolute Defect type Defect details Conclusion
distance Report Reality Depth Length (mm)
(m)
Report Reality Report Reality
1 2406.357 External External 22% 19.4% 25 85 Basically
manufacturing manufacturing consistent
defect defect
2 9942.825 Abnormal Abnormal / / / / Qualified
circumferential circumferential
weld weld
3 11,070.636 Abnormal Abnormal / / / / Qualified
circumferential circumferential
weld weld
4 16,957.882 Internal metal Internal metal 17% 16.25% 25 22 Basically
loss loss consistent
5 17,995.088 External External 16% 16.3% 31 365 Basically
manufacturing manufacturing consistent
defect defect
6 20,923.811 Internal metal Internal metal 25% 20.11% 16 15 Basically
loss loss consistent
allowable size of the B31G safety evaluation curve and need to be repaired.
If the internal metal loss defect increases at the determined full life metal
loss rate, there will be no new internal metal loss to be repaired within
8 years.
3. 39 metal additions were detected. It is suggested to judge whether it is a
maintenance mark in combination with maintenance and rerouting, conduct
excavation verification, judge the specific type of metal contacting the
pipe wall, and finally decide whether it needs to be repaired and removed.
Confirm the position of the approaching metal external objects and establish
corresponding accounts as the basis for line patrol or monitoring.
4. Among 290 circumferential weld abnormalities, 275 are mild, 14 are mild,
and 1 is moderate. Due to the abnormal NDT results of the two circumfer-
ential welds verified by excavation, there is no excavation verification plan
for the time being. If excavation is required, it is recommended to combine
the needs of the pipeline management party.
5. After the defect repair and regular maintenance of the pipeline are imple-
mented according to the above recommendations, the interval of internal
inspection is 8 years.
6.4 Remote Field Eddy Current 275
Table 6.11 Parameters of

Item Content
YY pipeline
Project name YY
Pipeline length 13.01 km
Transmission medium Sour natural gas (wet gas)
Pipeline location Onshore
Whether it has a pig receiver or Yes
launcher
Type of cathodic protection Forced current
Whether there is internal coating No
Pipe inner diameter 159 mm
Thickness 6 mm
Operating temperature Normal atmospheric
temperature
6.4 Remote Field Eddy Current
The size of the YY pipe is D168.3 × 11 (diameter × thickness), and the material
is L245 seamless steel. The length of the pipeline project is 13.01 km. The remote
field eddy current inspection was carried out to analyze and evaluate the internal
corrosion defects to understand the safety status of the pipeline. The specific pipeline
parameters are shown in Table 6.11.
6.4.2 Inspector Description
Figure 6.15 shows the design parameters of the detector. Table 6.12 lists the operating
technical indicators of the detector. The inspection accuracy parameters are shown
in Table 6.13.
6.4.3 Inspection Details
Table 6.14 shows the overall operation of the detector. Some critical operations are
shown in Fig. 6.16.
Figure 6.17 is the track signal diagram of the same pipe section collected by
the sensor, including color diagram, single sensor track signal diagram, and curved
surface diagram. Three spools with a length of about 30 m can be seen in the figure.
Fig. 6.15 The structure of the remote field eddy current detector
Table 6.12 Technical

Detector length 423 mm (rigid length of
indicators of detector
227 mm)
Detector diameter 155 mm
Passing capacity 1.5D S Type Elbow
Maximum deformation 20%OD
Optimum operating speed range From 0.3 to 7.0 m/s
Temperature range of medium From 0 to 70 °C
Maximum pressure 14 MPa
Sensor size 51 mm
Number of sensors 12
Detector weight About 7 kg
Battery working time 3.5 h
Table 6.13 Accuracy of detector

Parameter (80%wt Conventional metal Pitting corrosion Axial groove Circumferential
confidence) loss groove
Depth quantization ±0.15 t ±0.20 t ±0.20 t ±0.20 t
accuracy
Width quantization ±45 mm ±45 mm ±45 mm ±45 mm
accuracy
Length ±45 mm ±45 mm ±45 mm ±45 mm
quantization
accuracy
Note t is the standard wall thickness of the pipe, mm; wt represents wall thickness
Table 6.14 Detector

Project YY pipeline
operation situation
Launching time 2020-10-21 12:20
Receiving time 2020-10-21 13:06
Effective running time 0 h 46 m
Average running speed 5 m/s
Detector integrity Complete
Inspection data Complete
Whether the operation is successful Successful
Welding and some defect signals can be clearly observed in the figure (marked with
red “0” in the figure).
6.4.4 Inspection Results
A total of 3182 defects were detected in the inspection of YY pipeline, of which

the wall loss of 2541 defects was greater than 10%. The maximum defect depth is
2.3 mm, and the corresponding wall loss reaches 39% of the wall thickness. A total
of 2177 weld signals were detected in this inspection, and many elbow signals were
recorded. Table 6.15 shows the characteristics of the pipeline and Fig. 6.18 shows
the distribution of the proportion of wall damage of defects. The defect loss ratio is
calculated according to the wall thickness of 6 mm. The signals at the starting point
of the pipeline, the tee, and the position of the severe defect are shown in Fig. 6.19.
Figure 6.20 shows the pie chart of the clock azimuth distribution of defects.
In order to verify the inspection accuracy and effectiveness of the detector, 5
defect points were selected for excavation verification in this pipeline (see Fig. 6.21).
According to the internal inspection results, the defect information and excavation
inspection results of the five excavation verification points are shown in Table 6.16.
Among them, the length and width of D56 and D62 defect points did not obtain
effective inspection results.
(1) A total of 3182 internal corrosion defects were detected in the pipeline, including
38% with a depth below 10%wt, 60% with a depth within the range of 0%wt-
20%wt, 2% with a depth within the range of 20%wt-30%wt, and 0.35% with a
depth within the range of 30%wt-40%wt;
(2) Due to its simple structure, small size, and light weight, electromagnetic
eddy current internal inspection still has good trafficability even in gas gath-
ering pipelines with small pipe diameter, low pressure, and complex internal
environment;
(3) Due to the low internal pressure of the gas gathering pipeline as a whole and the
complex internal structure environment, the movement speed of the equipment
(a)
(b)
Fig. 6.16 Remote field eddy current inspection a Start-up; b pig receive; c detector shutdown
(c)
Fig. 6.16 (continued)
Fig. 6.17 Signal collected by the detector

Table 6.15 Characteristic

Pipeline characteristics Number
statistics of inspection results
Circumferential weld 2177
Flange 2
Elbow 355
Tee 3
Internal metal loss 3182
Total 5719
55 (1.7%) 11 (0.3%) <10%wt

10%wt-20%wt
20%wt-30%wt
30%wt-40%wt
1220 (38.3%)
1896 (59.6%)
Fig. 6.18 Distribution diagram of wall thickness loss
in the pipeline during the inspection process is in an unstable state, and the
movement speed may vary in the range of 0–31 m/s;
(4) The inspection rate of corrosion defects in the pipeline by electromagnetic eddy
current internal inspection can reach 100%. According to the test results, the
inspection error of defect depth is within the range of 0.9–5.7% wt. However,
the inspection errors of defect length, width and clock position are large.
(5) The electromagnetic eddy current internal detector is less affected by the moving
speed of the equipment. It can maintain a good defect inspection rate and defect
depth inspection accuracy even under the condition of serious unstable speed.
(a)
(b)
(c)
Fig. 6.19 Inspection signal diagram a Flange of starting point; b tee; c serious defect
00:00-01:00
01:00-02:00
6.9% 6.8%
02:00-03:00
8.3% 7.5% 03:00-04:00
04:00-05:00
05:00-06:00
7% 06:00-07:00
9% 07:00-08:00
08:00-09:00
09:00-10:00
7.9% 10:00-11:00
11:00-12:00
10.6%
8.7%
9.9%
8.6%
8.9%
Fig. 6.20 Pie chart of clock azimuth distribution of defects
Fig. 6.21 Excavation

verification site
6.5 Pipeline Monitoring in the Landslide Section
The total length of ZZ pipeline is 99.66 km. The pipe diameter is 813 mm and
the wall thickness is 11 mm (or 12.7 mm). The pipeline material is L485, the design
pressure is 7.5 MPa, and the design capacity is 1150 × 104 m3 /d. At present, the daily
gas transmission volume of the pipeline is 200–600 × 104 m3 /d, and the operating
6.5 Pipeline Monitoring in the Landslide Section 283
Table 6.16 Excavation verification results for electromagnetic field eddy current inspection
Defect Data source Width Length Depth Percentage Clock Measured Average
number (mm) (mm) (mm) of depth point minimum wall
(%wt) of the wall thickness
defect thickness of the
(mm) pipe near
the defect
(mm)
D7 Internal 120 56 2.04 34 10:00 4.6 5.35
inspection
Actual 70 70 1.38 23 12:00
measurement
D726 Internal 20 10 0.66 11 8:00 5 5.4
inspection
Actual 20 20 1.02 17 2:00
measurement
D2301 Internal 20 10 0.54 9 7:00 4.8 5.45
inspection
Actual 20 20 1.2 20 11:00
measurement
D10 Internal 80 30 1.8 30 4:00 4.95 5.91
inspection
Actual 80 60 1.08 18 1:00
measurement
D1849 Internal 20 10 0.54 9 9:00 5.39 5.95
inspection
Actual 20 20 0.6 10 11:00
measurement
pressure is 4.0–4.5 MPa. Because of the specific characteristics and influencing

factors of geological disaster points, a relatively complete monitoring profile and
monitoring network were established to make it a three-dimensional monitoring
system. In this monitoring project, a total of three guying displacement monitoring
points, three stress–strain monitoring points, one pipe-soil interface thrust monitoring
point, one rainfall monitoring point, one soil moisture content monitoring point, and
three pore water pressure monitoring points were arranged.
6.5.2 Monitoring Details
The layout of monitoring points is shown in Fig. 6.22. Table 6.17 shows the moni-
toring contents of the acquisition unit. The monitoring period is 12 months, from
December 2020 to December 2021. Because the landslide was in the stage of devel-
oping from unstable to overall sliding, the monitoring period of stress was 4 h, the
monitoring period of earth pressure and groundwater level was 1 h, the monitoring
period for surface displacement, soil temperature, and humidity was 6 h, and the
monitoring period of rainfall was 9 min. All data are automatically uploaded, and
stored for analysis and early warning through the data management platform.
According to the parameters such as steel pipe yield strength, the early warning of
this project was divided into three levels. The third-level early warning value was 458
microstrain, the second-level early warning value was 931 microstrain, and the first-
level early warning value was 1404 microstrain. When the monitoring value reaches
the three-level early warning value, the geological disaster alarm information can be
sent out, and relevant measures need to be taken immediately to reach the first-level
early warning value.
Unstable slope
Acquisition unit 3
Acquisition unit 2
Acquisition unit 1
Acquisition
unit 1 Acquisition unit 3
Acquisition unit 2
Acquisition
unit 2
Fig. 6.22 The layout of monitoring points
Table 6.17 Monitoring items and quantities

Acquisition unit number Monitoring content
Acquisition unit 1 Soil temperature and humidity, surface displacement, rainfall
Acquisition unit 2 Pipeline stress, earth pressure, pore water pressure, surface
displacement
Acquisition unit 3 Pipeline stress and pore water pressure
References 285
6.5.3 Monitoring Results
The automatic stress and strain monitoring system was used to monitor the stress
and strain of the pipeline to determine the harmfulness of the unstable slope to the
pipeline to provide a basis for whether to start the emergency rescue. The maximum
microstrain was less than 30, and the landslide was stable. The variation range of
the guying displacement was about 0.12 mm. To grasp the extrusion situation of the
pipeline by the unstable slope in time, an earth pressure box was arranged in the
pipe trench and compared with the initial earth pressure to determine the pressure
value exerted by the slope deformation on the pipeline. The earth pressure changed
smoothly, the force was uniform, and it was stable.
References
1. Xue, S. C., Zhang, Y., Ma, F., Gao, Y., Xue, D. X. (2008). Application of High-pressure Water
Jet Flow Technique in Washing of Internal Surface of High-pressure Pipes. Oil & Gas Storage
and Transportation, 27(9), 45–47.
2. Ma, J., Feng, H., Liang, Y. T., Lei, T. T., Xu, N., Zhang, H. R. (2017). Feasibility Study on
Unblocking Submarine Pipeline with the Coiled Tubing and the High-pressure Water Jet. China
Petroleum Machinery, 45(6), 103-107.
3. Minstry of Housing and Urban-Rural Development of the People’s of China. (2012). CJJ
181-2012 Technical Specification for Inspection and Evaluation of Urban Sewer. China
Architecture & Building Press, Beijng.
4. National Energy Administration. (2018). SY/T 6597-2018 Specification for in-line inspection
of oil & gas pipeline. National Energy Administration, Beijng.

2023 - Pipeline Inspection and Health Monitoring Technology Hongfang

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

2023 - Pipeline Inspection and Health Monitoring Technology Hongfang

Uploaded by

Copyright:

Available Formats

Hongfang Lu · Zhao-Dong Xu ·

Tom Iseley · Haoyan Peng · Lingdi Fu

Tom Iseley Haoyan Peng

ISBN 978-981-19-6797-9 ISBN 978-981-19-6798-6 (eBook)

Nanjing, China Hongfang Lu

Dong Lin, Safety, Environment and Technology Supervision Research Institute of

1 Background and Health Problems of Pipelines . . . . . . . . . . . . . . . . . . . . 1

2.5.3 Thermal Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4.4.4 Visual Defect Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

6.3 Magnetic Flux Leakage Inspection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260

1.2 Pipeline Classification and Construction Status

Fig. 1.1 Conventional pipe

The transportation volume is large and The one-time investment of pipeline

Transportation costs are high.

Good economy. Transportation The temperature and pressure control

Fig. 1.4 Global transmission oil and gas pipeline length

1.3 Pipeline Health Status Globally

Oil (or gas) Gathering pipelines Processing plant

Fig. 1.6 Oil and gas systems

Table 1.2 Characteristics of different types of pipes

Fig. 1.7 The old pipeline

Fig. 1.8 The old pipeline

condition of wastewater infrastructure in most states (94.28% of the 35 states with

1.4 Pipeline Inspection Technology System

Different scholars have different classifications of inspection technologies.

1.5 Technical System of Pipeline Health Monitoring

Unlike pipeline inspection, monitoring is a long-term process. It is an automatic

1.6 Global Pipeline Inspection and Monitoring Standards

Incident rate (times·10 -5·mile -1·yr -1)

Incident rate (times·10-3·mile-1·yr-1)

1-2 years 35-40 years Time

Fig. 1.11 Pipeline failure probability curve (bathtub curve) [34]

Fig. 1.14 The condition of wastewater infrastructure in the United States

Table 1.6 Classification of

Table 1.7 Classification of

Table 1.8 Related standards for pipeline inspection

Table 1.8 (continued)

2.2 Visual Inspection Technology

High Angle "Fish Eye" Lens Provides Simultaneous

Fig. 2.1 Composition of SSET

Sewer Scanner & Evaluation Technology 3 o' clock 9 o' clock

Fig. 2.2 Viewable range of Sewer Scanner and Evaluation Technology

Table 2.1 Merits and limitations of various visual inspection methods

Table 2.1 (continued)

2.3 Electromagnetic Inspection Technology

2.3.1 Magnetic Flux Leakage

where B is magnetic induction intensity; V is scalar magnetic potential; μ0 is

Magnetic flux lines

Fig. 2.3 The principle of magnetic flux leakage technology [29]

Multichannel data acquisition

Magnetic flux leakage

Fig. 2.4 Flow of pipeline magnetic flux leakage inspection [2]

When the measured ferromagnetic material is surrounded by air, the solution

Combining Eqs. (2.1) and (2.3), B at P(x, y, z) can be expressed as:

Table 2.2 Measuring range

Table 2.3 Characteristics of four magnetic flux leakage methods

2.3.2 Remote Field Eddy Current

Indirect energy transmission path

Direct energy path

Fig. 2.5 The principle of remote field eddy current [34]

Fig. 2.6 Signal-distance characteristic curve of the inspection coil [35]

where H is the magnetic field intensity; B is the magnetic induction intensity; D is

Define the vector magnetic potential A to obtain:

∇ 2 A = −μJc + μσ (∇ϕ + j ω A) (2.15)