Microbiological Method Validation: How Do

We Prove that a Method is Fit for Purpose?

Thomas Hammack
Chief
Microbial Methods Development Branch
Division of Microbiology
Office of Regulatory Science
Center for Food Safety and Applied Nutrition
U.S. Food and Drug Administration
College Park, MD 20740

Equivalence?
24 h, 35C

Does

or

=
24 h, 35C and 42C

 Why validate methods?

Benefits public health and world trade
False negative results are unacceptable
ISO 17025 lab accreditation demands the use of
validated methods

Validation Programs
AOAC Microbiological Guidelines
ISO 16140:2003
FDAs Guidelines for the Validation of Analytical
Methods for the Detection of Microbial Pathogens in
Foods
http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM273418.pdf

AOAC International

1884Association of Official and Analytical Chemists

Methods Validation for Fertilizers

Membership Restricted to Regulatory Chemists

Supported by USDA

Dr. Harvey Wiley -Father of FDA is also considered the Father of AOAC
1892Secretary of AOAC
1898Established AOACs Committee on Food Safety
Housed within FDA

1939Microbiological Sampling of Eggs and Egg products Official Method 939.14 still in
use today

Many AOAC methods Official Methods were developed and validated in FDA Labs

1972FDA published acceptance of AOAC Official Methods in the Federal Register

Allows FDA to use proprietary rapid methods for the analysis of regulatory samples

1979became an independent non-profit organization no longer tied to FDA, but many

FDA employees serve as volunteers

International Organization for

Standardization (ISO)

International non-governmental organization

Started in 1926 as the International Federation of National Standardizing Associations (ISA)

Focused on mechanical engineering

Disbanded in 1942
Reorganized as ISO in 1946
Mission statement is to provide International Standards for Business, Government and Society

1991 ISO and European Committee for Standardization (CEN) signed the Vienna Agreement

Based in Geneva, Switzerland

Comprised of 163 national standards institutes including ANSI in the US

CEN is comprised of the 31 Member States of the European Union

Vienna agreement stipulates that CEN and ISO Standards be identical to enable free trade within the
EU

ISO Technical Committee 34 (TC 34) Subcommittee 9 (SC 9)

ISO exercises its role in the standardization of food microbiological testing methods through TC
34/SC 9
TC 34/SC 9 started standardizing horizontal reference methods for bacteria in foods in the 1970s

FDAs Methods Validation Guidelines

The Science and Research Steering Committee (SRSC), of the Office of Foods
and Veterinary Medicine (OFVM), approved guidance to be used for validation
of microbiological and chemical methods.

Guidelines for the Validation of Analytical Methods for the

Detection of Microbial Pathogens in Foods
http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM273418.pdf

Guidelines for the Validation of Chemical Methods for the FDA

Foods Program
http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM298730.pdf

Scope
These criteria apply to all FDA laboratories that develop and participate in the validation of analytical
food methods for Agency-wide implementation in a regulatory capacity. This includes all research
laboratories, and field labs where analytical methods may be developed or expanded for potential
regulatory use. These documents will supersede all other intra-agency documents pertaining to foodrelated method validation criteria for microbial and chemical analytes. the SRSC will authorize the 18
formation of a Methods Validation Subcommittee (MVS) to serve as the governing body for all method
validation concerns.

FDA and Methods Validation

Method validation is a process by which a laboratory confirms by examination, and provides objective
evidence, that the particular requirements for specific uses of a method are fulfilled. It serves to
demonstrate that the method can detect and identify an analyte or analytes:

In one or more matrices to be analyzed

In one or more instruments or platforms

With a demonstrated sensitivity, specificity, accuracy, trueness, reproducibility, ruggedness

and precision to ensure that results are meaningful and appropriate to make a decision.

Reliably for its intended purpose. Intended purpose categories include, but may not be
limited to emergency/contingency operations; rapid screening and high throughput testing;
and, confirmatory analyses.

After the method developer has conducted experiments to determine or verify a number of
specific performance characteristics that serve to define and/or quantify method
performance.

The Office of Foods and Veterinary Medicine & the SRSC

Roadmap for Microbiological Method Development and Validation

RESEARCH

Organizational
Partnerships
BAM Council
FERN MCC
IFSH
ORA Micronauts

VALIDATION

Micro
Method
Validation
Sub-group

VERIFICATION

IMPLEMENTATION
9

The Method Validation Sub-Group (Microbiology)

ESTABLISHES validation needs and priorities in consultation with the
SRSC-Micro Super-group, FDA Bacteriological Analytical Manual Council (BAM Council), FERN Method
Coordinating Committee, ORA micronauts inter-center working groups and others as appropriate

ADOPTS procedures to govern all administrative processes needed for emergency and nonemergency method validation proposals and studies.

PROVIDES planning, guidance, oversight, and resources to participating laboratories during the
method development and validation process; will be the responsible authority for recommendations,
evaluations and final approval of all validation studies from planning through field implementation.

CONSULTS with other governmental, and independent (commercial, and international) validation
bodies to harmonize validation standards where possible and practices

10

The Method Validation Sub-Group

(Microbiology)
BROAD REPRESENTATION from CFSAN, ORA, CVM, and NCTR with additional
expertise from biostatisticians and QA/QC managers

CURRENT MICRO MVS COMPOSITION:

ORA
Palmer Orlandi (co-Chair), Cathy Burns
CFSAN Thomas Hammack (co-Chair), William Burkhardt, Darcy Hanes
CVM
Beilei Ge
NCTR
Steven Foley
FERN
Don Burr
NCFST (CFSAN Moffett)
Ravinder Reddy

11

METHOD VALIDATION SCOPE OF RESPONSIBILITY

Pathogens, Genetic Material, Toxins and Antigens

Qualitative assays

Analyte

Bacteriological, e.g.

Salmonella spp.

Pathogenic Escherichia coli

Listeria monocytogenes

Shigella spp

Vibrio spp

Campylobacter spp

Applications

Pre- and selective enrichment

Microbial analyte recovery and concentration

Screening, high-throughput, confirmation

Procedures

Phenotypic, e.g.

Biochemical characterization

Antibiotic resistance traits Antigenic characterization

Genetic, e.g.

Microbial toxins

Nucleic acid isolation/concentration

Viral pathogens, e.g.

Polymerase Chain Reaction

Hepatitis A virus

Conventional, Real-time

Norovirus

Reverse transcription

Enterovirus

Parasitic protozoan pathogens, e.g.

Cryptosporidium

Cyclospora cayetanensis

Bioengineered analytes, e.g.

Genetically-modified foods (GMOs)

Sequencing, e.g.

Whole genome

Selective sequencing

Single nucleotide polymorphism (SNP) analysis

Strain-typing applications

Method Validation is Required for

1.

Submission of a new or original method, OR,

2.

Any significant modification of a method that may alter its performance specifications or changes to the
fundamental science of an existing method. Categories include:

Substitutions of reagents/apparatus

Expansion of the scope of an existing method to include additional analytes.

Changes in intended use i.e. screening or confirmatory.

Platform extensions or significant parameter changes e.g. adaptation to another real-time PCR thermal cycler.

Matrix extensions.

Changes to time/temperature incubation periods, or enrichment media.

In cases where the sample preparation and/or the extraction procedure/analytical method is modified from the existing test
procedure and protocol, i.e the new method should demonstrate that the modifications do not adversely affect the precision and
accuracy or bias of the data obtained.

Modification of a methods performance range e.g. specificity, sensitivity beyond previously validated levels.

Levels of Validation
Four levels of performance are defined. The hierarchy of
scrutiny will provide general characteristics on the methods
utility and insights for its intended use, the assessed risk, and
the food-borne illness potential for an analyte-matrix pairing.

Not all methods will or should be validated to meet the requirements of a Level 4: full
collaborative study.

Method Validation Criteria

Level One
The lowest level of validation, with all the work done by one lab. Inclusivity and exclusivity
testing has been tested, but by a limited number of strains. The analyte was tested at a level
based on the intended use of the method, with just normal background flora. There is no aging
of the artificially-inoculated samples and no comparison to an existing reference culture
method. The expectation would be for the originating lab to continue to conduct further testing
to eventually elevate the method to a higher level of validation.

Intended Use: Emergency needs. A method developed for the detection of an analyte,
or a matrix not previously recognized or identified as a threat to food safety or public health. As
the first level in the development of any method designed for regulatory use; performance of the
method at this level of scrutiny will determine, in part, whether further validation is useful or
warranted.

NOTE: Under emergency situations where the rapid development and deployment of a method is needed to
immediately address an outbreak event, Level 1 criteria should be followed as closely as the situation will allow.
Representatives of the MVC and Agency subject matter experts (SMEs) should be in close consultation with the
originating laboratory. Once the crisis has past and it has been determined that there is a need for further
validation, procedures outline in this document will be followed.

Method Validation Criteria

Level Two
This is a more robust study, with the possibility of regulatory strength depending on the
circumstances. The originating lab has done a more comprehensive initial study, with
inclusivity/exclusivity levels at the AOAC Collaborative Study level. If possible, a comparison
has been done to an existing reference culture method. One other independent laboratory has
participated in the collaborative study. Some of the criteria of the study are at a lower level
than the full AOAC study, but still appropriate for the developing method at this stage.

Intended Use: Emergency needs. Slightly higher false-positive rates are acceptable as
all samples analyzed with methods validated to this level will require confirmatory testing.

Method Validation Criteria

Level Three
This is a validation level that should have full regulatory strength. Most of the criteria
followed by the originating lab are at the AOAC level, including inclusivity/exclusivity,
analyte levels, competitor strains, aging, and comparison to existing method when
available. The additional collaborating labs follow many of the criteria of an AOAC
collaborative study.
Intended Use: All methods validated to this level of scrutiny are acceptable for use in
any and all circumstances e.g. confirmatory analyses; regulatory sampling, and
compliance support.

Level Four
This validation level has criteria equivalent to the AOAC Collaborative Study. Any
method reaching this level of validation should be able to be submitted for adoption by
the AOAC as a fully collaborated method.

Microbiology Validation Category Examples

I.

General Qualitative Guidelines for Microbial Analytes

Table 1. ORIGINATING Laboratory Requirements

AOAC

Originating Laboratory
Criteria

Level One: Urgent

usage

Level Two: Independent lab

validation

Level Three: Multiple

lab collaborative

Level Four: Full

collaborative study

# of target organism
(inclusivity)

10 (20 for
Salmonella)

50 (unless 50 aren't
available)a,b

50 (unless 50 aren't
available)a,b

50 (unless 50 aren't
available)a,b

50a,b

# of non-target organism
(exclusivity)

10 strains

30 strainsc

30 strainsc

30 strainsc

30 strainsc

# of foods

1 or mored

1 or mored

1 or mored

1 or mored

Up to 20 foodse

# of analyte levels/food matrixf

set level based on

intended use and
negative control

One inoculated levelf and

uninoculated level

One inoculated levelf

and uninoculated level

One inoculated levelf

and uninoculated level

One inoculated levelf

and uninoculated
level

Replicates per food at each

level tested

20

20

20

20

20

Aging of inoculated samples

prior to testing

No

Yes

Yesg

Yesg

Yesg

Addition of competitor strain

Normal background
flora

In 1 food at +1 log>analyte at
fractional positivef analyte level

In 1 food at +1
log>analyte at
fractional positivef
analyte level

In 1 food at +1
log>analyte at
fractional positivef
analyte level

In 1 food at +1
log>analyte at
fractional positivef
analyte level

No

Yes, if available

Yes, if available

Yes, if available

Yes, if available

Comparison to recognized
method

Collaborative Study

Microbiology Validation Category Examples, continued

Table 2 - General Qualitative Guidelines for Microbial Analytes-Collaborating
Laboratory Requirements

Collaborating Laboratory Criteria

Level Two: Independent lab

validation

Level Three: Multiple lab

collaborative

Level Four: Full

collaborative study

AOAC Collaborative Study

# of laboratories providing usable data

10

10

# of foods

1 or morea

1 or morea

1 or morea

1 to 6 foodsb

# of strains of organism

1 per food

1 per food

1 per food

1 per food

# of analyte levels/food matrixc

2 levels: One inoculated levelc

and uninoculated level

3 levels: One inoculated levelc

one at 1 log higher and
uninoculated level

3 levels: One inoculated

levelc, one at 1 log higher and
uninoculated level

3 levels: One inoculated levelc

one at 1 log higher and
uninoculated level

# of replicate samples/food

8 per analyte level

8 per analyte level

8 per analyte level

6 per analyte level

Aging of inoculated samples prior to testing

Noe

Yesd

Yesd

Yesd

Comparison to Recognized Method

Yes, if available

Yes, if available

Yes, if available

Yes, if available

Microbiology Validation Category Examples, continued

II. General Qualitative Guidelines for Food-borne Microbial Pathogens
That Present Unique Isolation and/or Enrichment Challenges
Table 1. ORIGINATING Laboratory Requirements
Originating Laboratory
Criteria

Category
One: Urgent
usage

Category
Two:
Independent
lab
validation

Category
Three:
Multiple lab
collaborativ
e

Category
Four: Full
collaborativ
e study

Replicates per strain

Comparison to
recognized methoda

No

Yes, if
available

Yes, if
available

Yes, if
available

Table 2. COLLABORATING Laboratory Requirements

Collaborating
Laboratory Criteria
# of laboratories
providing usable datab
Replicates per strain
Comparison to
Recognized Methoda

Level One:
Urgent usage

Level Two:
Independent lab
validation

Level Three:
Multiple lab
collaborative

Level Four: Full

collaborative study

n/a

10

n/a

n/a

Yes, if available

8
Yes, if
available

Yes, if available

Microbiology Validation Category Examples, continued

III. CRITERIA AND GUIDANCE FOR THE VALIDATION OF FDADEVELOPED MOLECULAR-BASED ASSAYS
Inclusivity and exclusivity
Target gene(s) and controls (positive and negative).
Comparison to the Reference Method

IV. FOOD MATRIX EXTENSION FOR VALIDATED MICROBIOLOGY

METHODS
The validation of a method for a food matrix not previously included in a validation study is
necessary to assure that the new matrix will produce neither high false positive (matrix is free
from cross reactive substances) no high false negative rates (matrix is free of inhibitory substances)

Guidance to Support Field Laboratory Expedience

Guidance for the Formal Expansion of Validated Food Matrices
V. PLATFORM EXTENSION

Microbiology Validation Category Examples, continued

VI.

CRITERIA AND GUIDANCE FOR THE VERIFICATION AND VALIDATION

OF COMMERCIALLY- AVAILABLE MICROBIOLOGICAL DIAGNOSTIC KITS AND PLATFORMS
Validation of an Alternative method: Demonstration that adequate confidence is provided when the results
obtained by the alternative method i.e. the commercially-available kit, are comparable to or exceed those
obtained using the reference method using the statistical criteria contained in the approved validation protocol.
Verification: The confirmation by examination and the provision of objective evidence that specified requirements
have been fulfilled.
Criteria
For Kits Fully Validated in a Collaborative Study Monitored by an Independent Accrediting Body
1.

For commercially-available microbiological diagnostic kits whose performance parameters have been fully validated
in a collaborative study monitored and evaluated by an independent accrediting body e.g. AOAC-OMA, AFNOR, etc.

2.

For commercially-available microbiological diagnostic kits whose performance parameters are supported by data
obtained through an abbreviated validation protocol and evaluated by an independent accrediting body e.g. AOACRI.

Current Micro MVS Priorities

I. Hepatitis A Virus
a. Real-time RT-qPCR assay to detect Hepatitis A virus
b. In a food matrix (green onions)
Status: Final report nearing completion

2.

Non-O157:H7

STEC

Screening method to detect non-O157:H7 STECs using the BioPlex

Status: Multi-lab collaborative study has been completed..

3.

Salmonella

Enteritidis

Isolation and Detection of Salmonella Enteritidis (SE) from Whole Shell

Eggs-Cultural and Molecular Applications

4. Norovirus
a. Real-time RT-qPCR assay to detect Noroviruses
b. In a food matrix (molluscan shellfish)
Status: planning phase underway in collaboration with the ORA-CFSAN virology working group

23

Validation of an Identification Method for Shigatoxigenic E. coli Somatic (O) Groups using the
BioPlex Suspension Array System
Method Authors
Andrew Lin
Teresa Lee
Laurie Clotilde
Julie A. Kase
Insook Son
J. Mark Carter
Carol R Lauzon

24

Validation of a STEC Molecular Serotyping Method

STECs are a significant public health concern

Non-O157 STECs are responsible for over 60% of STEC

infections or an estimated 112,000 illnesses in the U.S. each year

Over 74.2% of STEC infections in the U.S. are caused by serogroups O26
(23.9%), O45 (7.8%), O91 (2.3%), O103 (16.7%), O111 (12.6%), O121
(7.5%), and O145 (3.4%)

O26, O103, O111, O121, and O145 are known to cause HC and HUS, and
O45 is associated with HC

Other serogroups that may cause HC and HUS, but are less commonly
isolated, are O91, O113, and O128

25

Validation of a STEC Molecular Serotyping Method

BioPlex Assay Overview

Must be performed on Pure Cultures

Bead-based assay that can be multiplexed

Conventional and real-time PCR assays for STEC O serogroup - limited by

resolution of band sizes
on a gel, or the # of fluorescent channels

96-well plates
Targets = nucleic acid, proteins Ab, antigens, cytokines
Each bead = a separate assay
100 different color-coded beads (magnetic
or polystyrene)

Unique color comes from different ratios of two distinct flourophores

Two lasers, fluidics, optics, detectors
Automated, High-Throughput, Fast, Expandable

26

Validation of a STEC Molecular Serotyping Method

Method Status
In-House Validation Study Completed
Paper
LIB
5 Lab Collaborative Study Successfully Completed
Approved by the BAM Council
27

Validation of a STEC Molecular Serotyping Method

In-house Validation Results Inclusivity Panel

28

Validation of a STEC Molecular Serotyping method

Exclusivity Panel

40

BioPlex Results

Fluorescence signals are quantified for each micro

bead

Signal to background ratios are calculated, where

background is measured using all no template
reactions

Samples were considered to be positive when

signal to background ratio was greater than 5.0
30

Validation of a STEC Molecular Serotyping method

BioPlex Results

31

Validation of a Method for

Salmonella in Whole Shell Eggs
Development and Validation of a Method for the
Detection and Isolation of Salmonella Enteritidis (SE)
from Whole Shell Eggs
Guodong Zhang
Eve Thau
Eric W. Brown
Thomas Hammack
32

Validation of a Method for Salmonella in

Whole Shell Eggs
Background
SE is the second most commonly isolated Salmonella serotype
in the United States
14.18% (1968-1998)
SE is most commonly associated with whole shell eggs and
egg products
FDAs egg rule (74 FR 33030) recommends various preventive
control measures including sampling and sample analysis
FDAs BAM reference culture method for the detection of SE in
whole shell eggs takes 9 days to complete
There are no AOAC Official Methods of Analysis methods for
the detection of SE in whole shell eggs
33

Validation of a Method for Salmonella in

Whole Shell Eggs
Clean surface
Surface disinfection

20 eggs to a container
Hand homogenize sample

Hold at room Temp (20-24oC)

for 96 h

Clean surface
Surface disinfection
20 eggs + 2L TSB
24 h at 35oC
1 ml to 10 ml TT
24 h at 35oC

0.1 ml to 10 ml RV
24 h at 42oC

Streak on XLD, BS, HE

24 h at 35oC

Streak on XLD, BS, HE

24 h at 35oC

TSI, LIA
24 h at 35oC

TSI, LIA
24 h at 35oC

Serological Test

Serological Test

25 ml eggs to 225 ml TSB + ferrous

sulfate; 24 h at 35oC

1 ml to 10 ml TT
24 h at 35oC

0.1 ml to 10 ml RV
24 h at 42oC

Streak on XLD, BS, HE

24 h at 35oC

Streak on XLD, BS, HE

24 h at 42oC

TSI, LIA
24 h at 35oC

TSI, LIA
24 h at 35oC

Serological Test

Serological Test

CURRENT BAM METHOD

PROPOSED BAM METHOD

34

Validation of a Method for Salmonella in

Whole Shell Eggs

35

Validation of a Method for Salmonella in

Whole Shell Eggs

36

Validation of a Method for Salmonella in

Whole Shell Eggs
Method Status
In-House Validation Study of Culture Method
Complete
Manuscript in preparation
Report to BAM Council in preparation

In-House Validation Study of qPCR Methods in

Progress
ABI SE qPCR Method Promising
Evaluation of an internal qPCR method in progress
37

Validation of a Method for Salmonella in

Whole Shell Eggs
Collaborative Studies
Tentative Schedule
Culture Method Alone
Fall 2012

Culture Method Plus qPCR Method

Spring 2013
38