Proceedings of the ASME 2022 14th International Pipeline Conference
IPC2022
THE WHY AND HOW OF DATA INTEGRATION FOR INTEGRITY MANAGEMENT
PURSUANT TO API TR1178
Jackie Smith Cesar Espinoza
Principal of Integrity Principal Engineer,
Management Systems, ROSEN
ROSEN Houston, TX
Houston, TX
ABSTRACT
Data integration is a foundational element and a regulatory
requirement for pipeline operators as part of an Integrity
Management (IM) program. Having taken effect in July 2020,
the Pipeline Hazardous Materials and Safety Administration’s
(PHMSA) new Gas Mega Rule heavily emphasizes the
requirement for the validation of data and records. As a result,
pipeline operators must face new challenges to ensure they are
in continuous regulatory compliance. Data integration plays a
major role in supporting pipeline operators and engineering
consultants as they adjust to the new requirements set forth in
regulations. The Integrity Management (IM) regulations require
pipeline operators to include the results of the latest as well as
of previous integrity evaluations, including risk assessment
information based on integrated data from the entire pipeline.
Leveraging this data and the analysis thereof is required when
making decisions regarding the course of action to address
pipeline threats.
The American Petroleum Institute (API) Technical Report
(TR) 1178, “Integrity Data Management and Integration,”
provides a compendium of systematic methodologies,
recommendations and processes to spatially integrate and
normalize integrity data to empower pipeline operators to
efficiently analyze integrity-related data to support their
integrity management decisions and programs. Through the
execution of multiple post-ILI service projects, the significance
of this TR has been realized as a cornerstone to support pipeline
operators in their efforts to meet the requirements of new
regulations and achieve compliance.
In the present case study, ILI results from various
inspections gathered from over 179 post-ILI data integration
projects for a mid-size pipeline operator located in the southern
September 26-30, 2022, Calgary, Alberta, Canada
IPC2022-87064
Karen Collins
Head of Business Line,
Education Systems &
Services,
ROSEN Houston, TX
United States – totaling over 5,500 mi of pipe segments and
ranging in diameter from 10” to 24” – were analyzed following
the core processes described in API TR 1178. The datasets –
ranging from hundreds to hundreds of thousands of integrity data
records from different sources including a variety of in-line
inspection (ILI) tool technologies and service providers – were
interpreted to distinguish pipe features or references from
anomalies. Subsequent analyses were applied to the integrated
datasets and delivered per the pipeline operator’s specifications.
This effort greatly supported a seamless transition to
achieve compliance with the new regulations and help to
establish reporting requirements for technology and inspection
service providers. The operator now has a complete historical
ILI database. As it is fully integrated into the GIS system of
record, the pipeline operator is able to transition into advanced
analytics, helping them to spot trends and make sounder integrity
management decisions long-term.
Keywords: ILI data alignment; data integration; corrosion
growth assessment, Geographic Information Systems
1. INTRODUCTION
In the United States, many pipeline operators are regulated
by the Pipeline and Hazardous Materials Safety Administration
(PHSMA). Recent updates to PHMSA 49 CFR (Code of Federal
Regulations) Part 192 Transportation of Natural and Other Gas
by Pipeline: Minimum Federal Safety Standards make it
imperative to know details about pipelines that are regulated.
These details equate to data. Data has become critical for gaining
knowledge about a pipeline operator’s assets. Data and
information derived from data have become critical assets for the
pipeline operator and can be leveraged to improve safety and
adhere to regulations.
1 © 2022 by ASME
 Per API TR 1178 [1], data governance is the foundation for
gaining insight from data, especially pipeline integrity data.
Pipeline operators store and access data in many different ways.
For example PDF and Excel files may be placed on file share
sites and not integrated with enterprise-wide systems, such as
GIS (Geographic Information Systems). Some pipeline operators
may keep data in silos where information is stored and accessed
only within their own departments as others may store data in
enterprise-wide systems, such as a Geographical Information
System (GIS) or an Enterprise Resource Planning (ERP) system.
Pipeline operators often use industry approved data models such
as the Pipeline Open Data Standards (PODS™) or the Utility and
Pipeline Data Model (UPDM) from Esri™.
It is highly recommended that pipeline operators, as
competent stewards of data review API TR 1178 and use the
recommendations therein as a foundation for data management.
With the use of machine learning and predictive analytics
emerging in the pipeline industry, operators should continue to
improve the data management processes.
Per API TR 1178, data integration is critical to successful
integrity management planning. Integration of new ILI with
previous ILI results and pipeline records offer more in-depth
understanding of pipeline integrity threats. Integrating and
aligning multiple integrity datasets provides pipeline operators
with more information and spatial awareness so the integrity of
pipeline assets can be better understood, managed and analyzed.
API TR 1178 addresses the full cycle of ILI and pipeline data
management and integration activities for integrity purposes
including: data quality, data integrity, data models and spatial
alignment methods. It recognizes that, similar to the integrity of
the pipeline, integrity of data is critical for long-term use,
reliability, and key in making informed decisions.
2. Data Alignment
The first step for data integration is proper data alignment.
This will ensure that each ILI call, anomaly or reference type is
aligned or matched to its respective measure or engineering
stationing on the pipeline centerline. API TR 1178 describes
different approaches in the alignment for the purpose of pipeline
integrity. Following API TR 1178 recommendations, pipeline
operators leverage linear referencing in relation to a pipeline
centerline to store the pipeline asset information. Linear
referencing according to Esri™ is “A method for storing
geographic data by using a relative position along an already
existing linear feature; the ability to uniquely identify positions
along a line without explicit X, Y coordinates. Location is given
in terms of a known linear feature and a position, or measure
along it.” [2] Those locations or measures for pipeline operators
are stored in imperial or metric and are how pipeline features,
inspections, surveys and associated changes in attributes, such as
pressure, diameter, class location, pipe components, etc. are
managed along a digital pipeline. Points of interest along the
centerline, such as fittings, valves, launcher and receivers receive
a measure or station value, similar to how a house (a point of
interest) has an address on a road (think of the road as the
pipeline route). It is easier to refer to a valve or launcher receiver
at a stationing or measure value than remember the associated
GPS coordinate or ILI odometer reading.
Furthermore, API 1178 brings awareness to integrating
various datasets. Pipeline operators may provide pipeline
information such as High Consequence Area (HCA) segments,
class location designations as well as other datasets specific to
cathodic protection, above ground surveys, previous ILI and
excavation and repair sites which can be aligned and integrated.
Datasets should be available in spreadsheet form or GIS format
to ease data alignments.
To achieve the required accuracy, the alignment process
must use a list of data points to align such as welds, valves, tees,
taps, casings and sleeves, above ground markers (AGMs) and
bends. Pipeline operators have input on this list and can
comment on certain data, such as the quality of the AGM
information or gaps, which as an example may direct the process
to use more recent valve survey information first. In general, the
more points available for alignment the tighter the alignment.
Also, ILI to ILI alignments can be done, which are required in
corrosion growth assessments. ILI to ILI is a girth weld to girth
weld alignment and are simpler, it is more like comparing apples
to apples whereas a GIS to ILI alignment is comparing apples to
oranges. Also, since ILI to ILI alignment is a girth weld to girth
weld alignment, the amount of welds available to act as control
point allow for more precise alignments.
ILI to GIS alignments however are different. Typically, in
GIS data pipeline records do not relate to specific girth welds.
Other tie-points to align such as: Valves, Tees, Taps, Casings,
AGMs and Bends must be used. ILI to GIS having less precise
control points should be more thoroughly reviewed by the
analyst and should be managed by an established quality control
process.
2.1 Quality Control
Quality control is critical to ensure the ILI data is aligned
with the GIS data. In order to ensure the quality of the alignment,
analysts developed a desired tolerance chart. This means that the
greater the distance between an alignment point of a matched
GIS and ILI point, the less tolerance is given for that match.
Consequently, the more distance, the less tolerance the analyst
would allow and would not match those two points.
An example of what would fit in with the desired tolerance
would include pipe bends called out from the ILI and pipe bends
provided by the pipeline operator. In Figure 1 two pipe bends
are set to match, the two on the left from the GIS are 49 feet apart
and the two ILI are 56 feet apart. This represents an 11 percent
tolerance and is acceptable according to guidelines developed
when doing alignments.
2 © 2022 by ASME

FIGURE 1: DESIRED ALIGNMENT TOLERANCE
There are occasions when alignment cannot be done due to
pipeline re-routing, replacements or removed features from the
GIS, as well as Out-of-date information from surveys and
inaccurate or missing GPS points where the AGM information is
not complete. When these issues occur, updated records are
needed from the pipeline operator to make the alignments.
In addition to variance checking, the data alignment
software includes quality assurance tools used throughout the
alignment process. This includes a graphical alignment view and
two quality view graphs to ensure alignments are accurate. In
addition to the quality assurance tools, it is standard process for
another analyst to check the work and the overall quality of the
alignment by ensuring any aligned points outside the desired
variance are valid.
3. OPERATOR CASE STUDY
ILI data interpretation is completed for alignments and
integrations in most post-ILI services for several small to large
sized pipeline operators. Analysts perform data alignments and
subsequent reporting for several pipeline operators who require
that the ILI data be delivered with GIS pipeline references. As a
case study, ROSEN collaborated with a pipeline operator with
over 5,500 miles of natural gas pipelines in the southern United
States. ROSEN provided these geospatial alignment services,
along with additional in-depth calculations and corrosion growth
analysis, to enhance the operator’s integrity management plans.
During the initial client engagement phase, it was clear the
pipeline operator had many challenges in their integrity planning
processes due to datasets stored in isolation, including GIS data,
Excel sheets with ILI pipe tallies, alignment sheets and pipeline
questionnaires. Additional issues included data inaccuracies,
duplication and redundancies.
To help solve these challenging problems a program of
customized queries executed on GIS aligned ILI results,
regardless of ILI service provider or tool technology was
produced. Outcomes were integrated into a customized
spreadsheet deliverable that supports decision-making processes
related to ILI response, mitigation, and re-assessment intervals
according to the requirements outlined in the pipeline operator’s
Integrity Management Plan.
3.1 Data Requirements
The pipeline operator was responsible for proving pipeline
data inputs necessary for integration to support the query
execution for the final report. A structured data specification was
required to ensure proper exchange of crucial information and
the delivery of timely reports.
Data and information from the GIS was required to be
delivered with attributes such as the order of pipe segments,
beginning and ending engineering and cumulative stationing as
well as equations, class location designations, HCA information,
outer diameter, MAOP, percent SMYS, wall thickness, in service
date, seam weld types, coating information, product type,
method used to determine HCA and pipeline number. This data
and information was leveraged on a pipe joint basis for correct
data alignment as well as for inputs for the query calculations
required.
Features such as bends, flanges, markers, valves, launchers
or receivers, taps, tees, and begin and end point of sleeves were
required to be delivered with attributes of engineering
stationing, bend degrees, descriptions and comments. An analyst
segmented the pipeline data to create a record where each change
in attribute has a measure or point location along the pipeline
centerline. The pipe properties used for segmentation included
class location designation, date installed, MAOP, outside
diameter, wall thickness, and SMYS.
3.2 Query Development
The objective of the customized query development was to
support the pipeline operator in determining next inspection
dates or excavation dates for the metal loss anomalies and
deformation defects reported. The pipeline operator also
required compliance assessments to be performed based on the
anomaly response criteria in 49 CFR 192 using standards such
as ASME B31G. The process developed assigns response
timelines for anomalies detected by ILI run and provides the
pipeline operator with evaluations on metal loss and dent
anomalies.
The queries utilized information from ILI results, feature
box matching corrosion growth rate (CGR) and other
information uploaded into the software that runs the process.
The metal loss anomalies were grown over time using two
distinct growth rates. One was based on 75% of pipe life and the
other on CGRs determined through feature box matching.
In order to augment the process, certain ILI Information was
added to the base ILI run results to execute the queries. Extended
information includes the tool used during the inspection, box
matching CGR results for depth per anomaly, box matching CGR
results for length per anomaly, if available, dent strain analysis
results added as a percentage for the reported dent and the type
of the seam weld, required for execution of a customized query.
In collaboration with the pipeline operator’s integrity
engineering personnel, the team collaborated to design and
develop customized, parametrized queries based on the pipeline
operator’s previous integrity process documents. As one may
expect, new requirements were discovered during workshops.
Integrity queries or calculations on the ILI and GIS integrated
datasets were designed with the intent to have partial results
available as an additional quality assurance measure. In total
fifty-five (55) customized, parametrized queries were created
3 © 2022 by ASME
and grouped into ten (10) categories. Where applicable, queries
use requirements contained within pipeline standards from
ASME and adhered to PHMSA regulatory requirements. The
categories are included in Table 1.
Query Description
Category
Parameter The Parameter queries provide base values
that are utilized in all subsequent tolerance
queries calculations
Tolerance The Tolerance Calculation is developed using
Calculation the tool tolerances provided by the Operator’s
specifications using the tables for 80%
certainty for anomalies located in the body of
the pipe or in girth weld of heat-affected zone.
Pressure The Pressure Calculation is based on the
Calculation pipeline operator’s specifications and
requirements for determining remaining
strength. These queries calculate the Safe and
Fail (Burst) Pressures at the instant that the
ILI is reported making use of the Tolerance,
Parameter and Intermediate queries.
Intermediate queries are the input factors for
the more complex calculations which can be
viewed to ensure quality assurance
Factor The Factor queries utilizes the results from
the Pressure Calculation queries to determine
the current Estimated Repair Factors and
Failure Pressure Ratio factors.
Corrosion The Corrosion Rate calculation queries make
Rate use of Parameter queries and segmentation
values to estimate the Growth rates used later
on for the Pressure Calculation Queries (time
based). The operator previously defined the
calculations.
Pressure The Pressure Calculation queries (time
Calculation based) uses the Corrosion Rate calculation
(Time based) queries to iterate over the years enabling the
calculation of the remaining life for each
defect. Each feature is grown yearly up to
forty (40) years and is validated based on
operator specifications compared to internal
software results.
Remaining The Remaining Life queries will make use of
Life the Pressure Calculation Queries (time based)
to provide in years the remaining life of the
Pipe based on Parameters previously defined.
The Operator previously defined the
calculations in this section.
Dent Each dent is assessed on a scenario basis
including dent location on the pipeline, such
as near a girth weld, metal loss and excessive
bending strain levels.
Metal Loss of The o’clock position of the feature are
Seam Weld compared to the seam-weld o’clock position
to ensure the feature’s position is within the
prescribed proximity. The output scenarios
also consider the seam weld segmentation
type.
Next Based on the results of the assessments
Inspection previously done combined with the
Date Remaining Life Queries a projection of the
Next Inspection Date is provided.
TABLE 1: OPERATOR DEFINED QUERY CATEGORIES
Each query was evaluated to ensure it met the defined
specifications and validated to guarantee the accuracy of the
results. For validation purposes, a sample inline inspection
dataset was created with a small set of artificial features. These
features were manipulated into case scenarios to match the
conditions defined in the query definitions outlined by the
operator. Validation procedures were performed for the queries.
For example, for the dent queries, dent features properties in the
sample dataset were manipulated to meet prescribed condition
while others were altered to fall right outside the threshold. In
most cases the queries were validated based on the pipeline
operator’s remaining strength calculation specifications. Once
the validation results were consolidated, a semantic check was
also performed by an integrity engineer who analyzed the final
deliverable report for accuracy and the correctness of the
algorithms.
Regarding the Next Inspection Date queries, it should be
noted that solely applying individual feature rates from box
matching is an approach that is often under-conservative. This
risk should be considered when using results from the box
matched features to establish re-inspection interval or date. An
integrity engineer should consider this as part of the assessment.
In fact, over the years, there have been some results where
immediate excavations were flagged by the automated process
and those areas needed to be reassessed.
3.3 CGA Requirements
To support the pipeline operator with their integrity
decision-making regarding corrosion growth, a box-to-box
matching corrosion growth assessment (CGA) was also
performed. Following discussions with the pipeline operator, the
4 © 2022 by ASME
corrosion growth methodology of applying individually
calculated corrosion growth rates was reviewed and assessed.
Recent publications have demonstrated that the application
of individual CGRs can lead to inaccurate predictions of future
integrity [3] [4]. An example of a subset of the results from the
paper above are shown in Figure 2, which demonstrates the
paper’s findings of the application of individual corrosion
growth rates was non conservative for a significant proportion of
the feature population compared to a segmented approach.
Therefore, a basic CGR segmentation methodology considered
appropriate to use with box matching in providing a
recommended CGR for each feature was included with the
process. It is important to note that the results of each assessment
were not verified by corrosion diagnosis or the comparison of
signal data unless requested by the pipeline operator.
FIGURE 2: Comparison ILI Measured Depth vs Predicted Depth
when Considering Different CGR Estimation [1]
The estimation of CGRs is a critical input to any pipeline
integrity management program, and integral to determining
remaining life and in turn timescales for future repairs or re-
inspection intervals. The comparisons for the case study were
based on the most recent ILI data and one previous comparable
(equivalent inspection technology) ILI data set. The box
matching procedure and methodology consisted of a comparison
of the features reported in the two most recent inspections.
During this assessment, all reported corrosion was compared
using box matching software, which first performed alignment
of girth welds from multiple ILI runs. This process provided an
overview of how the reported feature depths have changed
between the two inspections.
In addition, segmentation was defined based on the density
of reported features along the length of the pipeline. For external
corrosion this process was semi-automated (where the
automatically defined segments could be adjusted manually
based on engineering judgement) using nearest neighbor
clustering (a modified DBSCAN algorithm [4].) For this method
a search window was defined for each feature, extending
upstream and downstream by a specified distance suitable for the
pipeline being assessed. Each feature was considered moving
downstream along the pipeline. If the number of features within
the window exceeded a set threshold, the feature was considered
to be part of a “concentration”, along with all other features in
the window. Each “concentration” ends when the feature density
falls below the threshold. Segmentation options were reviewed
for internal corrosion on a per project basis and typically, a single
segment was sufficient for the purpose of future integrity
assessment.
A characteristic CGR was estimated for each of the defined
segments, based on the historical depth changes of the features
therein. Depending on the number of features within each
segment along with engineering judgement, the characteristic
CGR may be based on a calculated upper bound (e.g. 95th
percentile) or the maximum measured depth change. Any
features considered to be statistical outliers were removed from
the assessment and considered separately.
4. THE PROCESS
The overall case study process developed for the operator
includes partial automation of the customer Integrity
Management Plan through software customization using data
management tools, python scripts, Excel macros and in-house
technical and engineering knowledge.
As shown below in Figure 3, the GIS data inputs and recent
ILI results were aligned by software automation. Further
software automation integrated the CGA results from the
engineering team. The final deliverables were created
automatically with some manual intervention before delivery to
the pipeline operator.
FIGURE 3: THE PROCESS FOR THE CASE STUDY
5 © 2022 by ASME
 The deliverables to the operator included a standardized data
integration report, inspection results, including CGA findings or
box-to-box matching results and the recommended CGRs for
each corrosion feature, with a brief technical note describing the
methodology.
API TR 1178 discusses different types of data management
models beyond spreadsheets, such as object-relational data
modeling and dimensional modeling. The operator decided to
improve ILI delivery by requesting data in a standard data model.
The data model requested to implement is called The Utility and
Pipeline Data Model (UPDM) and is a template database
provided from Esri™ [5]. In this database, the ILI data can be
delivered for integrating back into a larger ILI database stored on
the operator’s servers. The UPDM database can serve as the
operator’s source of record for storing all ILI results over time
and can allow them to have a system of record. The operator can
in turn leverage historical ILI data for audits, trending and
performing analysis.
Lastly in the process, the pipeline operator requested that a
form be completed for record keeping purposes as part of the
requirements contained within their Integrity Management Plan.
The Excel form is populated by the analyst upon report
finalization before delivery to the pipeline operator.
4.1 Process Improvements
Over the years, ongoing improvements have been integrated
into the above process. A year after working together, It became
more efficient for the pipeline operator to provide the GIS in
shapefiles (a spatial file format) with the same data requirements,
instead of the within an Excel pipe properties and associated data
listing. Asset information provided by the pipeline operator in
shapefile format included pipe properties, HCA segments, class
location designations, fittings such as such as tees and taps,
above ground markers (AGMs) and valves. In order to satisfy
the new requirement, an application specialist developed tools to
convert the shapefiles into formats for the internal software to
continue the process efficiently.
Additionally, in 2020, AGMs were provided to improve the
accuracies of the inertial measurement unit (IMU) data on the
geometry tool inspections. Following guidance of the API TR
1178, ILI runs that collect IMU data enable better accuracy when
mapping the centerlines of pipeline routes and provide valuable
baseline data for the pipeline operator’s assets upon completion
of construction prior to pipeline operations.
5. BENEFITS TO PIPELINE OPERATOR
Since the inception of the project in 2017, the pipeline
operator and ROSEN have collaborated on over 260 data
integration projects with at least 150 of those including the
corrosion growth analysis (CGA) results and roughly 22% of
those inspections being from other ILI service providers. Figure
4 shows the allocation of the projects per year separated by ILI
service providers. Many benefits have resulted from this
collaboration including a streamlined and documented process,
automated quality control procedures, and integration of CGAs
and dent strain reports and support for dig sheet creation for more
in-depth decision making. Data can be integrated for displaying
each pipeline segment managed in the pipeline operator’s
Integrity Management Plan. Also, the data and information can
be integrated into the operators system of record and can be
accessed for PHMSA or state regulatory reports and audits
quickly with traceability.
FIGURE 4: DATA INTEGRATION PROJECTS
Per API TR 1178, the pipeline operator recognized the
importance of integrating the GIS pipeline data and ILI data.
Various checks could subsequently be performed after successful
integration. For gas lines, for example PHMSA requirements
have been included for metal loss anomalies affecting the seam
weld, at vintage pipe with low frequency ERW seam type pipe
on a HCA. The calculation for this query was done using a
combination of ILI and GIS data where the metal loss affecting
the seam was derived from the ILI and HCA information, pipe
vintage and seam type.
Further benefits of the data integration query were realized
when the seam type was unknown, and other conditions were
met and a conservative approach was required. The process
flagged the anomaly as an immediate repair which required the
pipeline operator to take prompt action, such as make a repair.
Meetings were set up between the pipeline operator and subject
matter experts involved with the engineering analysis where
previous excavation, repair and cathodic protection information
were reviewed for the pipeline operator to make decisions for
excavation planning and repairs.
The benefit of using the integrated ILI results with the GIS
pipeline records integrated were that ILI anomalies that could
have potentially been missed from the repair plan were
identified. The process has successfully identified incorrect GIS
records a few times which in turn led to the pipeline operator
confirming and correcting the GIS pipeline records.
6. CONCLUSION
In conclusion, API TR 1178 can offer in depth guidance to
pipeline operators in implementing integrity management plans
by providing detailed recommendations on data integration as
noted. Data integration is the foundation for which other
assessments, such as CGA’s and accurate responses to PHMSA
6 © 2022 by ASME
can be achieved. After five (5) years of collaboration, many
successful improvements have been implemented and the
requirements and deliverables will continue to be refined in the
years to come to keep up with changes in regulation,
technology and in the pipeline industry in general.

ACKNOWLEDGEMENTS
The concept behind the case study detailed in this paper was
based on an internal project at ROSEN between the operator and
the ROSEN Systems Implementation staff, conceived and led by
Karen Collins, Head of Business Line, Education Systems &
Services, with project support from Marcilio Torres, Senior
Application Specialist and Brett Jarvis, Data Integration
Specialist.

REFERENCES
[1] API Technical Report (TR) 1178, 1st Edition,
November 2017 - Integrity Data Management and Integration
[2]https://desktop.arcgis.com/en/arcmap/latest/manage-
data/linear-referencing/what-is-linear-referencing.htm
[3] IPC2016-64424: “Corrosion Growth and Remnant Life
Assessment – How to Pick the Right Approach for your
Pipeline”
[4] Ester, M; Kriegel, H-P; Sander, J and Xu, X (1996). A
Density-Based Algorithm for Discovering Clusters in Large
Spatial Databases with Noise.
[5] https://doc.arcgis.com/en/arcgis-
solutions/latest/reference/introduction-to-gas-and-pipeline-
referencing-utility-network-foundation.htm

7 © 2022 by ASME