DEPARTMENT OF MEDICAL TECHNOLOGY - FAR EASTERN UNIVERSITY MANILA

METHOD EVALUATION & QUALITY MANAGEMENT

I. Introduction: Quality Management

A. Method Evaluation: Used to verify the acceptability of new methods
Method Evaluation
B. Quality Control: Insurance of a method to remain valid over time
Quality Control
II. Descriptive Statistics
• Used for monitoring test performance and QC. Summarize the data generated by the laboratory
• Sources of Analytic Variability
1. Operator technique
Operator technique 5. Environmental conditions (temperature, humidity)
temperature, humidity
2. Instrument differences
Instrument differences 6. Reagents
Reagents
Test accessories
3. Test accessories 7. Power
Power Surges
Surges
Contamination
4. Contamination 8. Matrix effects (hemolysis,
Matrix effects lipemia, etc.)
hemolysis, lipemia
A. Measure of Center
1. Mean
Mean - Average
2. Median
Median - Middle of the data after the data have been ranked
3. Mode - The most frequent occurring value in the dataset
Mode
B. Measure of Spread
Range - Largest value in the data minus the smallest value
1. Range
2. Standard
Standard Deviation
Deviation (SD, s or σ) - Represents the “average” distance from the mean. Degree of precision of a
measurement
Coefficient of
3. Coefficient ofvariation
variation - Allows comparing SD with different units (mg/dL vs. mmol/L; Na assay vs. K assay)
Assay values 𝐱̅ - x ( 𝐱̅ - x )2 Computation
92 -2
-2 04
04 N = 1010
88 -2 04
04 90
𝐱̅ = 90
90 -0
-0 00
00 ∑ ( 𝐱̅– x)2 = 108
108
87 -3
-3 09
09 ∑(x̅ –x)2
𝑆 = √
94 -4
-4 16 N−1
86 -4
-4 16
16
𝑆 = 3.46
3.46
93 -3 09
09
90 -0
-0 00
00
CV (%) = 100S/x
95 -5
-5 25 3.84
= 3.84%
85 -5 25
25

C. Measure of Shape
1. Gaussian
Gaussian (Normal) distribution “bell curve”
2. 68.3% is between ̅
68.3 XX̅±1SD 95.4% is between X
±1SD ; 95.4 X̅ ±2SD ; 99.7%
̅ X̅±3SD
99.7 is between ̅
X ±3SD

III. Methods Selection and Evaluation

A. Method Selection • Throughput
1. Analytic Sensitivity
Analytic Sensitivity - Ability of a method to detect small • Turnaround time
quantities of an analyte • Cost
2. Analytic Specificity - Ability of a method to detect only
Analytic Specificity • Space needs
• Disposal needs
the analyte it is designed to determine
• Personnel
3. AMR (analytic
AMR analyticmeasurement
measurement range
range) - Linear or dynamic requirements
range. Range of analyte concentrations that can be
directly measured without dilution or concentration.
4. CRR(clinically
CRR clinicallyreportable
reportable range
range). Allows dilution and
concentration, to extend AMR
5. LoD
LoD (limit
limit of
of detection)
detection - Lowest amount of analyte
accurately detected by a method

Aaron Jan Palmares, RMT, MSMT_09.10.21 Page 14

 DEPARTMENT OF MEDICAL TECHNOLOGY - FAR EASTERN UNIVERSITY MANILA

B. Method Evaluation
1. Imprecision
Determination of Imprecision
• Dispersion of repeated measurements about the mean due to analytic analytic (random)
random error.
error
• Random error varies from sample to sample.
Random error
• Causes: instrument
Instrument instability, temp. & reagent
instability, temp. reagent variation
variation,
handling techniques and operator
handling techniques variables.
operator variables.
• Determined by Repeated Analysis study (detects random Random error
error that affects reproducibility)
Repeated analysis study Precision study
a. Repeated analysis study (Precision study): 2 x 2 x 10 study (2 controls are run twice
2 controls twice aa day
day for 10days
10 days)
• E.g. glucose in hyperglycemic (150 mg/dL) and normal range (90 mg/dL)
• Estimate long terms changes occurring over time
Morning Evening Total Pass / Fail
Standard Deviation (SDI) 1
1 22 22 Pass
Pass
2. Determination of Inaccuracy
Inaccuracy
• Difference between a measured measured value
value and its truetrue value systemic error
value due to systemic error (proportional or constant).
• Systemic error - always in one direction
Systemic error
i. Proportional error - The magnitude of error is dependent
Proportional error dependent on analyte
analyte concentration.
concentration.
Constant error
ii. Constant error - The magnitude of error is constant and not not dependent
dependent on analyte concentration.
• Can be determined by recovery study, interference study and comparison of methods
Recovery Ability of a test to measure a known amount of analyte
i. Recovery: analyte (e.g. Ca2+) added to a matrix
matrix (e.g. CSF)
ii. Interference:
Interference Effect of an interferent
interferent (e.g. Hb) on the accuracy of detection of an analyte analyte (e.g. troponin)
Recovery studies
a. Recovery studies - detects proportional
proportional error
error
Ca Measured in Matrix Ca Added Ca2+ Recovered % Recovery
2+ 2+

Baseline 7.50mg/dL
7.50 mg/dL
Sample 1 8.35 mg/dL
8.35 0.95 mg/dL
0.95 0.85
0.85 mg/dL 89%
89%
Sample 2 9.79
9.79 mg/dL 2.38
2.38 mg/dL 2.29
2.29 mg/dL 96%
96%
Interference studies
b. Interference studies - detects constant error
constant error
• Common interference (i. Hemolysis;
Hemolysis; ii. Lipemia;
Lipemia; iii. Bilirubin;
Bilirubin; iv. Anticoagulant;
Anticoagulant; v. Preservatives)
Preservatives)
Sample # Hemolysate (Hg) g/dL Troponin T (units) % Bias
1 00 2.955
2.955
2 0.05
0.05 2.957
2.957 0.068
0.068
3 0.10
0.10 2.959
2.959 0.135
0.135
4 0.15
0.15 2.961
2.961 0.203
0.203
Comparison of methods – detects systemic
c. Comparison-of-methods error
systemic error
• The test method is compared with a reference method
reference method (gold gold standard)
standard
• 40 -100 spx were run every day over 8 -20 days
• A plot of the test-method data (y-axis) vs. the comparative method (x-axis)
deming plot
is generated (Deming plot: Graphical representation)

Type of Error Test Used to Determine

Random
Random error
error Precision Study
Precision Study
Constant error
Constant error Interference studies
Interference studies
Proportional
Proportional error
error Recovery experiment
Recovery experiment
Systemic
Systemic error
error Comparison
Comparison of of methods
methods
Total error
Total error Precision
Precision && COM
COM

Aaron Jan Palmares, RMT, MSMT_09.10.21 Page 15

 DEPARTMENT OF MEDICAL TECHNOLOGY - FAR EASTERN UNIVERSITY MANILA

IV. Quality Control

A. Introduction
1. Check the reproducibility of a method by including control specimens in the run
a. Equipment
Equipment out out of calibration; Reagent may
calibration b. Reagent may be deteriorating c. Technologist
be deteriorating; Technologist may
may have made an
have made an error
error
2. Done by Running QC Materials (Controls) Controls
a. Should be the same same as as the
the specimens
specimens to be tested
b. Should span spanthe
the clinically important
clinically important range of the analyte
range
c. Glucose assays (cutoff values):
i. In reference range (80 mg/dL)
ii. Hypoglycemia (50 mg/dL)
iii. Hyperglycemia (150 mg/dL)
3. Types of Controls
Lyophilized (dehydrated powder) - Reconstituted with its diluents.
a. Lyophilized
b. Stablized
Stablized frozen
frozen controls
controls - Needs to be evaluated for stability
B. Quality Control (QC) Charts
1. Levey-Jennings
Levey-Jennings Control Chart
Control Chart
• Used to detect errors in accuracy and imprecision over time.
a. Randoms errors:
Randoms errors
i. Due to variations
variations inintechnique
technique
b. Systemic errors:
Systemic errors
i. Poorly made (contaminated) standards standards & reagents
reagents
Instrumentation problems (function and calibration)
ii. Instrumentation
iii. Poorly written procedure
procedure
• Control values should be within statistical limits
(expresses as the mean ± SD)
C. Quality Control (QC) Charts
1. Shift and Drift
Shift More than six consecutive values fall in one side of the mean
a. Shift:
b. Trend:
Trend More than six consecutive values steadily increase or decrease.
2. Multirule Procedure (Westgard and Groth)
a. 13s
13s Rule: If 1 control observation exceeds mean ±3s, reject the run for probable random random error
12s
b. 12s Rule: If 1 control observation exceeds mean ±2s, a warning rule. warning rule
c. 2 2s Rule: If 2 consecutive control values are greater than mean ±2s, reject the run for probable systemic
22s systemic error.
d. R4s Rule: If the difference between the 2 controls is ≥4s, reject the run for probable random error.
R4s random
14s Rule: If 4 consecutive clues exceed the same mean ±1s limit, reject the run for probable systemic
e. 14s systemic error.
10x Rule: If 10 consecutive control values fall on 1 side of mean, reject the run for probable systemic
f. 10x systemic error
• Once a rule has been violated, reject the analytic run, take action to remedy errors (find the cause of error, take
reject the analytic run, remedy errors
corrective action, reanalyze control and patient data)

Aaron Jan Palmares, RMT, MSMT_09.10.21 Page 16

 DEPARTMENT OF MEDICAL TECHNOLOGY - FAR EASTERN UNIVERSITY MANILA

V. Reference Interval Studies

A. Definition
Reference interval
1. Reference reference ranges
interval - Referred as reference medical decision
ranges. A pair of medical decision points
points that span the limits
of results expected for a given condition
Normal range
2. Normal range - Range of results between medical decisions levels that correspond to ±2SD
±2SD of results
from a healthy
healthy patient.
patient
Confidence interval
3. Confidence interval - Range of values that include a specific probability,
probability usually 90%
90% or 95%
95%
Medical decision
4. Medical decision level
level - Value for an analyte that represents the boundary between different therapeutic
therapeutic
approaches.
approaches
5. Therapeutic range - Reference interval applied to a therapeutic
Therapeutic range drug.
therapeutic drug
6. Establishing a reference interval - Done when there is no existing assay for an analyte or methodology in the
laboratory. May require from 120 ≈700 study individuals
120 to ≈700
7. Verifying a reference interval (transference)
transference - Confirm the validity of an existing reference interval for an analyte
using the same type of analytic system. Can require 20 study individuals.
B. Categories
Diagnosis a disease
1. Diagnosis disease or a condition
condition
2. Monitoring of a physiologic condition
Monitoring physiologic condition
3. Therapeutic
Therapeutic Management
Management

VI. Diagnostic Efficiency

A. Definitions
1. Analytic Sensitivity - Refers to the lower limit of detection for a given analyte
Analytic Sensitivity
2. Clinical Sensitivity - proportion of individuals with that disease who test positively with the test
Clinical Sensitivity
3. TruePositive
True Positive - patient with
with a condition
condition who are classified
classified by a test to have
have the condition
condition
4. False
False negative
negative - patient with
with a condition who are classified by a test as notnot having
having the condition
B. Measures of Diagnostic Efficiency
Diagnostic sensitivity
1. Diagnostic sensitivity - Ability of a test to detect a given disease or
condition

Diagnostic specificity
2. Diagnostic specificity - Ability of a test to detect the absence of a
given disease or condition

3. Positive
Positive Predictive
Predictive Value
Value - Refers to the probability of an individual
having the disease if the result is outside the reference range

4. Negative Predictive
Negative Predictive Value
Value - Refers to the probability that a patient
does not have a disease if the result is within the reference range

N = 20 Pregnant patient (10) Non-Pregnant patients (10)

hCG Positive TP = 8 FP = 3
hCG Negative FN = 2 TN = 7
PPV = 8/11
8/11 = 72% NPV = 7/9
7/9 = 78%
78% Sensitivity = 8/10
8/10 = 80% Specificity = 7/10
7/10 = 70%

Aaron Jan Palmares, RMT, MSMT_09.10.21 Page 17