A Theoretical Study For Determination Of POD In Simplified Manner For

The Purpose Of Examination System Qualification Of Online/Automatic

NDT

Debdutta Mallik1
More info about this article: http://www.ndt.net/?id=24370

ASNT Level 3 Trainer / Technical Consultant

Velosi(M) Sdn Bhd
No 6-2 Jalan PJS 8/2 Mentari Business Park Bandar Sunway 46150
Petaling Jaya, Selangor, Malaysia.
debduttamallik@yahoo.co.in, d-mallik@velosi.com

ABSTRACT

Automated/Online NDT system has become a part of online production system in Hot Rolling Mill for last few
decades in various parts of the world to enhance productivity as well as to minimize the inspection time.
Acceptance or Rejection of the product solely depends on performance of the system in most of the cases.
Examination system qualification through the method of probability of detection plays a key role to evaluate
accurately the performance of the online NDT system.
This paper deals with various aspects of probability of detection through a simple theoretical approach which
can be easily determined by the NDT personnel working at Hot Rolling Mill especially in Rail Mills.

Keywords: Online NDT, Probability of Detection, Rail, System Qualification

1.0 Introduction:
Automatic/Online NDT (Non Destructive Testing) system has a great importance in
manufacturing industry (finished & semi-finished product) to enhance productivity and to
reduce time and cost of NDT. Performance of NDT system is vital since the decision for
acceptance or rejection is taken based on the result generated by Online NDT System.
Unwanted signal beyond threshold level will cause additional rejection or extra time to
reprocess which increases inspection cost & time resulting additional financial burden to the
manufacturer. Similarly, missing a true signal from a defect may cause a failure to any degree
even a fatal accident which is unrepairable.
Keeping in view of importance of online/automatic NDT system, relevant code/standards are
stipulated in stringent manner.
As per present practice, calibration of test piece having artificial flaw is done at a testing
speed. Once the system is set to detect all flaws then production pieces are processed for
inspection.
Statistical evaluation for POD (Probability of Detection) of the NDT system is not mandatory
as per existing codes & standards but we can refer to various national & International
standards for such evaluation.
However, Performance evaluation of any online/automatic NDT system will help to
understand whether the system can generate a repeatable & reliable result as per requirement
(i.e. set criteria agreed between manufacturer and End User) and will give confidence to both
parties(i.e. Manufacturer & End User).

In this paper, various statistical options for estimation of POD have been briefed and a simple
method for estimation of POD in term of capability of NDT system has been proposed which
can be easily determined by the NDT personnel working at site/plant environment.
Few Important Definitions:

Performance Demonstration: A demonstration of the capabilities of an examination system

to accurately evaluate a specimen with known flaw characteristics in an environment
simulating field conditions.

Probability of Detection (POD): The percentage resulting from dividing the number of
detections by the number of flawed specimens or grading units examined. POD indicates the
probability that an examination system will detect a given flaw.

NDT System: Ensemble that can include hardware, software, materials, and procedures
intended for the application of a specific NDE method. Can range from fully manually
operated to fully automated.

2.0 Common statistical methods for estimation of POD

1. Hit/ miss method
The hit/ miss method gives a binary analysis of a testing signal, whether a defect is detected
(hit) or missed (miss). Consequently, the testing results can be scaled in four possible
configurations depending if a (non-) existing defect is (not) detected:

Table 1 Possible Output of NDT system

The four possible outcome of the online testing
Defect Detected Not detected Probability
True positive (TP) False negative (FN)
TP + FN =
Existing The existing defect is detected The existing defect is not
100 %
(hit) detected (miss)
False positive (FP) True negative (TN)
Non- FP + TN =
A defect is detected even though No defect is detected, where
existing 100 %
it is not existing no defect exists

The POD is equal to the probability of TP and can be calculated in the following way:
POD = P (TP) = TP / (TP+FN) ……………...………………………………………… Eq.1
P (FP) = FP / (FP+TN) ………………………………………………………………… Eq.2
For one specific flaw size ‘a’, this equation can also be represented by the number of positive
tests divided by the total amount of tests:
POD (a) = npos(a) / ntot(a) ……………………...………………………………………… Eq.3
Where,
npos = number of positive tests
ntot = total amount of tests
The hit/ miss method requires a clearly defined hit/ miss criterion, e.g. a defined threshold.

Theoretically the system should detect the signal from TP only but in practical condition
there may be possibility of FN and FP due to some unavoidable uncertainty.
 2. Log – Odd Distribution :

The log odds distribution is mathematically defined as

POD (a) = exp(a + ß.ln(a) / 1 + exp(a + ß.ln(a)………………………………… Eq.4
Where POD (a) is the mean probability of detection of cracks size ‘a’ and ‘a’ and ß are the
parameters of the log odds curve from the calculated data.

Figure – 1 POD vs Relative depth of crack, Ref: SKI Report 2005-03

3. â vs. a method
This method follows regression methodology, POD curve formulation. In this method we get
a relation between signal strength â and discontinuity size, a.
Here,
a = discontinuity size
â = the measured signal response for a given discontinuity size, a.

Figure.2 â vs log (a) showing the relationship of â scatter, noise scatter, and POD.
Ref: Mil Handbook-1823A.
 4. Binomial Method / Approach

In this method, we consider POD and FCP (False Call Probability) as tolerance band called as
‘α bound’ to describe the statistical uncertainty.

The ‘α bound’ are calculated using standard binomial equations, shown below.
Where:
D = Number of detections recorded
N = Number of grading units that contain flaws (for POD calculations) or that are blank (for
FCP calculations)
Pupper = upper α bound
Plower = lower α bound

……………………………………………… Eq.4

………………………………….…………… Eq.5
Where β is a beta distribution with parameters c1 and c2. The sample set for this test is based
on the above binomial equations.
Example: A POD of 95% with a 90% confidence implies that there is a 90% probability that
95% is an underestimate of the true detection probability. In other words, the confidence
level, α describes how reliable the qualification test must be.
To achieve a level of a 90% POD at a 95% confidence level requires a minimum of 29 flaws
out of 29 flaws to be detected.

Table – 2 Total Number of Samples for a Given Number of Misses at a Specified Confidence
Level and POD

Ref: Article – 14, ASME – Sec – V

3.0 Non Conventional Approach for faster evaluation

There are other statistical methods/models available which can be used for estimation of POD
also.
However, it may not be practically possible for NDT inspector at site/plant condition to
estimate POD and to evaluate the NDT system based on complex statistical analysis.
In other way, too many number of sampling will make the test expensive. In view of that a
simple method is proposed to evaluate the performance of the system with an example.
Performance evaluation of Online NDT System (Ultrasonic) of Rail has been taken for
Performance Demonstration/Performance Evaluation. This methodology can be applied for
other NDT system also.
If required by interested parties, different statistical approach can be applied to plot a POD
curve and to estimate value of POD more precisely.
At first we classify this performance evaluation test in three categories which will based on
damage mechanism, location & characteristics of flaw, degree of safety involved and end use
of the product.
A. Low Rigor
B. Intermediate Rigor
C. High Rigor

As an example, NDT system of Rail is considered under ‘High Rigor’ test since highest
degree of safety involved.
Online Ultrasonic Testing of Rail is carried out followed by successful calibration of a Test
Rail having artificial flaw as stipulated in relevant code/standard/specification. Calibration is
done in periodic manner (e.g. shift wise).
Performance or capability of online NDT system is dependent on optimum performance of
Inspector’s skill, hardware, mechanical assembly set up, hydraulic system,
electrical/electronic system etc associated with it. Hence, there is every possibility of
uncertainty of result even the system is being calibrated in every shift. It’s very difficult to
predict the confidence level and probability of detection based on single successful
calibration result. Repeatability & Reproducibility cannot be established by single test result.

Example of few Standards of Online Ultrasonic Testing of Rail is given below:

Table – 3 Sensitivity requirements or Threshold level of different railway standards

Sl No Standards/Specification Reference Reflector/Threshold Level Scanning Coverage
1 IRS – T – 12 – 2009 Head : 1.5 mm dia through hole Not specified
(IRS - Indian Railway Web : 2.0 mm dia through hole
Standard Specification) Web & foot junction : 2.0 mm dia
through hole
Foot : 0.5mm deep, 12.5mm long and
1.0mm wide notch (inclined at 20o with
vertical axis)
2 EN – 13674 – 2011 Head : 2x2 Flat Bottom Hole(Diameter – Head - at least 70 %.
(EN – Euro Norm 2.0mm each) Web - at least 60 %
/European Standard) 2.0mm diameter through hole Foot – As specified
Web: Flat bottomed holes 2.0 mm with sketch in EN –
diameter drilled to centre line of web. 13674.
Foot: 2 mm diameter through hole at
web-foot junction
Note: Performance evaluation or capability study is not stipulated in existing standards.
4.0 Study and Analysis

For this study, test piece with reference reflectors simulated as per IRS-T-12 is considered. 16
nos of probes may be the possible probe configuration to detect all flaws and to scan
maximum area as per given sensitivity requirement or threshold level.
Since this a single event probability (i.e. one true positive signal or one flaw for one probe)
we consider equation no.3:

POD (a) = npos(a) / ntot(a)

Sampling technique has been referred to the table no.1 for 95% confidence with 90% POD.
Respective sigma value is correlated with result for evaluation (i.e capability of the system
for detection).

Table – 4 – Sigma value against different percentage of yield

Sigma level Percent defective Percentage yield

1 69% 31%
2 31% 69%
3 6.7% 93.3%
4 0.62% 99.38%
5 0.023% 99.977%
6 0.00034% 99.99966%
7 0.0000019% 99.9999981%

Table – 5 – Result of performance evaluation based on non real data

Total Respective sigma(σ)

Probability
Probe Hit Miss value
P-1 45 1 0.9783 Better than 3 sigma
P-2 45 1 0.9783 Better than 3 sigma
P-3 46 0 1.0000 Better than 7 sigma
P-4 44 2 0.9565 Better than 3 sigma
P-5 43 3 0.9348 Better than 3 sigma
P-6 45 1 0.9783 Better than 3 sigma
P-7 44 2 0.9565 Better than 3 sigma
P-8 45 1 0.9783 Better than 3 sigma
P-9 46 0 1.0000 Better than 7 sigma
P-10 45 1 0.9783 Better than 3 sigma
P-11 43 3 0.9348 Better than 3 sigma
P-12 44 2 0.9565 Better than 3 sigma
P-13 46 0 1.0000 Better than 7 sigma
P-14 46 0 1.0000 Better than 7 sigma
P-15 45 1 0.9783 Better than 3 sigma
P-16 44 2 0.9565 Better than 3 sigma
Overall Probability 0.9728 Better than 3 sigma
5.0 Conclusion:
This methodology is fast and reliable and does not require complex statistical calculation. We
can easily correlate the sigma level from single event probability to determine the detection
capability of NDT system with respect to sigma values; i.e. performance evaluation of the
system. In other word, we can say that POD can be determined in other way. Frequency of
calibration is also can be set based on the existing performance of the system. Lower the
sigma value means higher the frequency of periodic calibration of test piece.
However, degrees of sigma level, no of sampling, frequency of performance evaluation
depend on mutual agreement between interested parties.
This is a general and fast approach to get an overall idea about performance of the system
immediately after test run.

References and Studies:

1. Article – 14 , ASME Section – V
2. ASTM – E – 2862, Standard Practice for Probability of Detection Analysis for
Hit/Miss Data
3. ASTM – E – 3023, Standard Practice for Probability of Detection Analysis for â
Versus a Data
4. IRS – T – 12 – 2009, Indian Railway Standard Specification For Flat Bottom Rails
5. EN – 13674, Railway applications - Track - Rail - Part 1: Vignole railway rails 46
kg/m and above
6. David S. FORSYTH, TRI/Austin, Austin, TX USA
John C. ALDRIN, Computational Tools, Gurnee, IL USA, Build Your Own POD,
7. E. Ginzel - Materials Research Institute, Canada - Introduction to the Statistics of
NDT
8. David S. FORSYTH, TRI/Austin, Austin, TX USA, John C. ALDRIN,
Computational Tools, Gurnee, IL USA, Build Your Own POD.
9. Raymond B. MABUZA, Department of Nondestructive Testing & Physics, P/Bag
X021, Vanderbijl Park, 1900, Johannesburg, South Africa, Numerical Analysis of
Probability of Detecting Defects in Engineering Materials.
10. Charles Annis , Luca Gandossi, Oliver Martin, Optimal Sample Size for Probability
of Detection Curves
11. ASNT Handbook, Volume – 7
12. ASM Handbook, Volume – 17
13. MIL Handbook, 1823A
14. www.ndt.net
15. http://zfp.cbm.bgu.tum.de