Auditing Techniques

Part 3.
Auditing Techniques for

Detecting Data Integrity Issues
Ronald F. Tetzlaff, Ph.D.
Corporate Vice President,

PAREXEL International, LLC
7855 St. Marlo Fairway Drive,
Duluth, GA 30097
Ron.tetzlaff@parexel.com
+1-770-622-0193 (O)
+ 1-678-522-5116 (C)
Slide 1
Outline
Prepare
Pre-Audit Plan Section 3.1
("Plan-of-Attack")
Predict
Survey Techniques Section 3.2
(Improving the Odds of Detecting)
Detect
Audit Approach Section 3.3
(Identify Gaps by QS)
Slide 2
© 2016 PAREXEL International
Learning Objectives
• At the Completion of This Course, Attendees Will Be Able

To:
Understand Some Effective Techniques for Preparing and
Conducting Data Integrity Audits
Describe the Elements of a Data Integrity Audit Plan
Summarize Prediction (Survey) Techniques to Decide Audit

Scope/Approach
Describe Audit Techniques that can be used to Detect Data

Integrity Issues
Slide 3
Section 3.1
Some Systematic Techniques to
Prepare
for Data Integrity Audits
Slide 4
Where Can Auditors Expect to Find
Data Integrity Violations?
Answer - Just About Anywhere
Slide 5
Audit Purpose is to Assess Conformance to Requirements
Company Discrepancies from

Policies, SOPs, Methods, Company
Specifications, Standards, or Requirements
other Requirements
Observe Identify
Determine
Actual Discrepancies
Requirements Conditions from Requirements
at Site
Regulatory Discrepancies from

Relevant Laws, Regulations, Regulatory
Directives, Standards, Requirements
Guidelines, Policies, etc Slide 6
Being Prepared Means
Knowing the Data Integrity Requirements
(Maintaining Awareness of Relevant Laws,

Regulations and Regulatory Trends)
Slide 7
Auditors Face a Practical Challenges
Keeping Abreast of Requirements
• Regulatory Authorities - Many diverse laws,

regulations, directive, and Regulatory Expectations are
Communicated in Many Places (Guides, Publications,
Speeches, Enforcement actions, etc.)
• Publications - Countless publications (trade association

magazines & references (Textbooks. Journals, etc.) on every
conceivable topic related to data integrity
• TNTC Internet Sites - Seemingly Countless sources
with an Endless Amount of Information, where to Find the
Requirements?
• Time Limitations - Searching/Maintaining Information
from Many Sources Can be Time Intensive
Slide 8
Organizing Available Information for Use

– Amount of Technical & Regulatory Information is Enormous
– Internet Sources Provide Access to Countless Documents, But
Auditors Can Not Afford Time to Conduct Repetitive Searches
– How to Most Efficiently & Effectively Obtain, Organize &
Maintain such a large body of Technical & Regulatory
Information
– Auditors Usually Operate Remotely & Independently (often
without clerical support or ready access to office systems)
– Auditors Need Simple/Efficient Systems for Organizing &
Retrieving Documents for Use during Remote Audits (& while
preparing).
Slide 9
Organizing Available Information for Use
• All Auditors Collect Various References

Published by Regulatory Agencies from the
Literature, but:
– Each Auditor must establish his/her own process or
system for maintaining up-to-date & relevant information
– Effectiveness in keeping abreast of Regulatory
Requirements & Trends is a function of Auditor Discipline
– Many do not have effective/efficient techniques for storing
and organizing for used in Audit Preparation or Audit
Conduct
• Auditors Must Reach a Balance between Use of Hard
Copy and Electronic Documents
Slide 10
Auditor Needs
– Instant Access to the "Right" Quality System/Data
Integrity Information
• Effective Management of Hardcopy References
– Where to file (Photocopies, Periodicals/Magazines,
Text Books, Seminar Handouts, etc
– How to Permit Ready Access When Needed
• Effecient Management of Electronic References
– Where to Store Electronic Files
– How to Permit Ready Access when Needed
– Minimize the Amount of Time for Obtaining &

Managing Auditors Personal Information Systems Slide 11
Auditor Limitations
• Travel to Remote sites limits the Auditor to carrying

Items in Suitcase/Briefcase
• Availability of Time for Audit Preparation including
Managing Information from Planned & Unplanned
Sources
• Subscriptions to Journals
• Website Alerts
• Colleagues (Other Auditors)
• SMEs at Sites Audited
• Magazines/Newspapers
• Symposia/Training Seminars
Slide 12
Knowing What is Required? What Systems,
QS Structure Processes, & Controls are
• Resources, Equipment, Facilities, Systems Needed?
(Objects) for Generating Data (e.g., storage, (How To Collect & Analyze,
retrieval, archiving & backup) Report & Retain
QS Processes Data/Information)
• Series of actions or operations that leads to a
desired result including the Management What Needs to be Done ?
Reviews and Evaluations to confirm accurate (How To Ensure Data are
data are available to show processes perform Accurate, Truthful &
as intended Complete)
QS Outcomes
What is Required Result ?
• Accurate Data to Demonstrate that the quality
of drug product conforms to the Company's (How to Prove Records &
requirements & those of Applicable Reports are Accurate,
Regulatory Authorities Truthful & Complete)
Slide 13
Sources of Data Integrity Requirements
• Keeping Abreast of Data Integrity Requirements

Requires:
• Utilizing Available Information Sources
• Internal Company Sources
• Public Sources (e.g., Internet &
Publications)
• Applying Diligence in Keeping Abreast of
Developments for Data Integrity
• Developing aTracking System that is:
• Practical (simple
• Efficient (Minimal Time)
Slide 14
Some Key Internet Data Integrity Sources
Regulatory Agencies- Laws, Regulations, Directives,
Guidelines, & Guidance)
• FDA Office of Criminal Investigation (OCI)
Department of Justice (DOJ)
• Office of Inspector General (OIG)
• Office of Research Integrity (ORI)
• EU - MHRA & Various Others Agencies
• Standards Bodies - Pharmacopeias, ICH, ISO, ASTM,

ANSI
• Trade Associations - ISPE, PDA, etc.
• Publications & Periodicals - Subscriptions & Internet

See Appendix 2
Slide 15
Introduction - Key Concepts for Data Integrity Audits
Data Integrity Auditors Must Always be Looking For Something

in Particular ("The Issue(s)")
If Auditor Looks for a Specific Data Integrity Lapse ("The

Issue"), they will almost always find it (if it exists)
Therefore, Audit Planning & Preparation boils down to Deciding:

What to Look for?
Where to Look?
Who is Responsible for Preventing or Detecting?
What Methods & Approaches Auditor Intends to Use to
DETECT "The Issue" ?
Slide 16
Introduction - Deciding the Focus
(is Data Integrity the Primary or Secondary Focus)?
– Surveillance Audit
• When Data Integrity Issues are Not Known or Suspected
– Review of Any or All Policies, Procedures, Systems,
Processes and Controls that apply to Preventing and
Detecting Data Integrity Lapses
– For Cause or Directed Audit
• When Data Integrity Issues are Known (or Suspected)
based on:
– Corporate or Local Site Initiative (Survey/Gap Analysis)
– Prior Findings, such as:
» A Violative Regulatory Inspection
» Internal Audit or Customer Audit
» Complaint (Customer or Whistleblower)
Slide 17
Introduction - Always Decide GxP Focus Before Arriving on Site
• Effective Audit Planning Means Always:
• Having a Good Reason for Selecting the

GxP Focus
• For Selected GxP Focus, Identify the
specific Topic(s) that will be audited [i.e
what are "The Issue(s)" to be covered]
• However, Expect to Modify the Audit Plan
after Assessing Findings of the PREDICT
phase, because ..... Predictors can
Improve the Odds by Identifying Areas
where Gaps are Most Likely to Exist
Slide 18
Introduction - Data Integrity Applies to All Sources of GxP Data
GLPs
Conduct of Animal Studies
GCPs
Conduct of IND Studies &
Investigator Records
Data Integrity
Accuracy & Completeness of
GMPs
Original Data & Records for
all GxP Functions Accuracy of Information & Data for all
Pharmaceutical Quality Systems
(Quality Management, Materials, Bldg
& Equip, Production, Pkg & Labeling,
& Laboratory)
Regulatory
Submissions
New Drug Applications, Pharmacovigilance
DMFs, Amendments, (Medical Assessments &
Responses, Letters, etc. Controls for Collection &
Reporting of Post Slide 19
Marketing© Safety
2016 PAREXEL
Data International
Introduction - Decide GxP Focus
(What part of Product Development Lifecycle?)
Scope (GxP) - Decide Data Integrity Audit Scope Based on

Assessment of the Entire Product Lifecycle - What will be
included or Excluded?
• Good Laboratory Practice (GLP)
• Good Clinical Practice (GCP)
• Good Pharmacovigilance Practice (GVP)
• Good Manufacturing Practice (GMP)
• Good Distribution Practices (GDP)
• Good Tissue Practice (GTP)
• Others ...continued
Slide 20
Introduction - Decide GxP Focus
(What part of Product Development Lifecycle?)
Discovery Development Manufacturing Distribution

Clinical Product Marketing
Basic Research Pre-Clinical Production
I II III IIIB IV Launch and Sales
Methods Dev. & Validation
Development Reports
Tech. Transfer
Batch Prod. & Control Documents & Records
Pharmacovigilance
Regulatory Applications
(Time Zero)
Supplements
NDA/ANDA/MAA Adverse Experiences
(New Drug Applications/ Annual Reports
R&D IND Marketing Authorization Slide 21
Field Alerts
© 2016
Application PAREXEL International
Others
Introduction - Decide Areas of Focus
Quality Data
GxP Facilities Systems Functions Applications
Quality
Materials
Facilities/Equip Collect
Regulations Site Production Analyze Regulatory
& Directives) Locations Pkg & Lbg Report Submissions
Laboratories Retain
GxP
Dev Studies - Formulation &
Methods (R&D)
Pre-Clinical Studies (GLPs)
Clinical Investigators (GCP)
Clinical Trial Materials (GMP
Commercial Batch Production

Quality Systems (CMPs &
GDPs)
Pharmacovigilance (PV)
Applications (Regulatory Slide 22

Submissions)
Decide DI GxP Focus Based on Risk Factors
Risk Factors
GxP Prior Previous Current Current QS Others

Focus DI Audit Regulatory Industry Performance
Audit? Findings Priorities Trends Trends
Dev Studies - Formulation

& Methods (R&D)
Pre-Clinical Studies (GLPs)
Clinical Trials - Clinical
Investigators (GCP)
Clinical Trial Materials
Commercial Batch
Production Quality
Systems (CMPs & GDPs)
Pharmacovigilance
Applications (Regulatory
Submissions)
Slide 23
Regulatory Applications
Accuracy Reviews - Assess the Procedures, Systems, Processes &

Controls that are used to verify Accuracy of Information and Data
Submitted to Regulatory Authorities including:
• Prior to Submission - Confirm all records, reports, summaries,
conclusions and other commitments have been documented as
having been verified for accuracy and completeness. Identify
those that were not.
• Previously Filed Information and Data Periodically Confirm
Accuracy and Completeness (i.e., submissions accurately reflect
any changes at the company since the original submission (e.g.,
Amendments/Supplements)
• Scope - See Next Slide
Slide 24
Decide Scope for Regulatory Applications
Pharmacovigilance, CMC
Safety, & Adverse (Records of Inspections & Tests &
Experience Reports Quality System Documents)
Development
ELAs/PLAs/BLAs All Reports
Regulatory
PMAs/510-Ks
Submissions NDAs/ANDAs
Marketing
Clinical Data
Authorization Supplements- (Bioavailability &
Applications Amendments Bioequivalence)
(MAAs)
NDA
Common Annual
Clinical Trial Field Alert
Technical Reports
Study Reports Reports
Documents Slide 25
DMF(s) IND(s) © 2016 PAREXEL International
Specs/Limits
Practices Data
(Records)
Application
Commitments
Procedures Reports
Methods
Regulatory Submissions
Slide 26
• Request Advance Copies of Submissions Filed with

Regulatory Agency(s) Prior to On-Site Visit
– CMC Section of New Drug Application
– Common Technical Document
– Drug Master File (DMF)
– Marketing Authorization
– Clinical Trial Study Protocols (IND) or Data for Clinical
Batches
– Supplements/Amendments & Any other Filing
Audit Tip
Use 70% Reduced Photocopy
(White Space for Annotations)
Slide 27
• For Cause Data Integrity Audit

– If Known or Suspected Issue(s)
– Identify Relevant Drug Product or
Application (New Drug or Marketing
Authorization)
» Identify Key Quality Systems for Audit
Coverage based on Review of
Application (e.g., CMC or Clinical Data
Sections, Supplements or
Amendments, & Deficiency Letters, if
available) Is the Application Approved
or Pending?
» Use Application & Deficiency Letters to
Predict Quality System Issues
Slide 28
Reminder - Regulatory Affairs (RA)Personnel are
Responsible for Application Integrity
Responsibilities - RA is Responsible for Ensuring that

all Information & Data Submitted to FDA (and Other
Regulatory Authorities) is Accurate Truthful and
Complete
Reminders to Regulatory Affairs Personnel:

1. Submitting Inaccurate or Incomplete Information
and Data in NDA/ANDA Applications is grounds
for withholding approval of applications (FDA and
other Regulatory Authorities)
2. Willful submission of False Statements or
Incomplete data is a criminal offense with severe
penalties ...continued
Slide 29
Certification of
Truthfulness &
Accuracy of Data &
Statements in
New Drug Applications
(FDA Form 356h)
Slide 30
RA Must Certify that the Applications Contain
accurate and Truthful Data
FDA Form 356h - FDA New Drug Applications

(NDAs/ANDAs) require a signed certification statement:
"The data and information on this submission have been
reviewed and, to the best of my knowledge, are certified to be
true and accurate,
Warning: A willfully false statement is a criminal offense, U.S. Code,

title 18, section 1001." (Emphasis Added)
Slide 31
What Does Form 356h Certification Mean?
• Requires a positive affirmation that the data and information in the

submission "have been reviewed."
• Imposes a burden on the person signing to establish a
reasonable basis for making such affirmation
• Its not enough to simply sign the certification based on
assumption that someone else has verified accuracy of the data,
Instead Need to:
• Evaluate/assess the procedures, processes and controls that
are followed for verification of data, and
• Define the sources of data and information that are to be
used by the signing official
• Define the details of the review process that is followed for
confirming the accuracy of data, and the documented
evidence needed as basis for verification of accuracy
Slide 32
Certification Signature on form 356h means that;
• The data and information submitted in the application have been
verified as accurate and complete against the raw data and
original source records and documents
• None of the available supporting documentation identified any
unresolved data integrity issues
• Review of the evidence that documents the verification activities
identifies the identity of personnel performing verifications, date of
verifications, and the source(s) of Raw Data and Original Records
• The person signing not aware of any incident or event that calls
into question the validity or reliability of any raw data, record,
report, or other document that contains data or information that is
intended to support a new drug application.
Slide 33
Data Integrity Certification - NDAs/ANDAs (FDA-Form 356h)
"The data and information on this

submission have been reviewed
and, to the best of my knowledge,
are certified to be true and
accurate. Warning: A willfully
false statement is a criminal
offense, U.S. Code, title 18,
section 1001."
Slide 34
Scope (GxP Facilities)
Scope (Facilities) - Inventory Previous Audits for each Facility that Performs
GxP Functions involved in the development, testing and manufacture of
pharmaceutical and biological products for:
• Commercial manufacturing including In-House Locations & contract
manufacturing organizations (CMOs)
• Clinical trial Batches - in support of new drug applications (including, but
not limited to, INDs, BLAs, MAAs, NDA, and ANDAs)
• Active Pharmaceutical Ingredients (APIs) & Excipients
• Contract Services [Laboratories, Contract Research Organizations
(CROs), & Others]
• Methods & Formulation Development (intended to support new drug
applications)
• Vendors & Suppliers of GxP services and data
• Others
Slide 35
Scope Quality Systems
• Prepare Inventory of Known DI Lapses - Examples:
• Regulatory Inspections
• Internal Audits by Customers, Corporate, Third Party Consultants)
• Deviation and Failure Investigations (including CAPAs & Change
Controls)
• Quality System Performance Reviews (including management reviews,
trending reports, Annual Product Review/ Product Quality Reports
(APRs/PQRs), & Other Assessments
• Employee Interviews
• Assess Taken In Response to Known Data Integrity Lapses
• Deviation Root Cause Investigations
• CAPAS
• Risk Analyses
• Data Management Trending Reports
• APRs/PQRs
Slide 36
• Many Others
Scope (GxP Facilities)
• Inventory - Identify, Tabulate and Rate Risks for All Audits for
Past 3-4 Years for Each GxP Facility/Area
• Identify Audits that Included Data Integrity (DI) Focus
• Rate The Risks
• Decisions - Use Ratings to Decide Need to Adjust Planned Audit
Schedules/Frequencies
GxP Area
R& D GLP GCP GVP GMP Application
DI Audit Performed? N N N Y Y N
Risk Score (DI Audit) 5 5 5 2 1 5
Slide 37
Scope Quality Systems
Verify availability of DI Procedures, Systems, Processes & Controls
for each QS
(i.e., is DI Covered?)
Quality Facilities & Pkg & Laboratory
Materials Production
Management Equipment Labeling Control
Annual Product Environmental Contract Packaging & Environmental
Validation
Review Control Management Labeling Monitoring
Audit Laboratory
Management Equipment Materials Processing Investigations
Management Management Control (OOS/OOT/ Lab
Change
Deviations)
Management
Facility & Utility Supplier Technology Laboratory
Complaint Management Management Transfer Management
Management
Deviation
Method Transfer
Management /
CAPA
Reagent and
Document & Standards
Records Management
Management
Sampling and
Lot Disposition Inspection
Personnel Stability
Management
Management Test Performance

Review & Review
Slide 38
Decide Scope for Quality Systems
Specs/Limits
Practices Data
Quality Systems (Records)
Quality Management
Facilities & Equipment
Materials
Production
Packaging & Labeling
Procedures Laboratory Reports
Quality
Systems
Methods
Slide 39
Use Risk Assessment to Identify Quality Systems & Topics

That Have Been Focus of Recent Regulatory Inspections
["The Issue(s)"]
FDA
MHRA
EMA
TGA
HC
PMDA
Others
Slide 40
Try to Decide Quality System Focus Before Arriving on Site
Quality
Management
Facilities &
Laboratory
Equipment
Quality Systems
Packaging
Materials
& Labeling
Production
Slide 41
Prepare
The Audit Plan
(("Plan of Attack")
Slide 42
Prepare the Audit Plan
Some Auditors Prepare Written Audit Plans, Others Do Not
• Intent of Tutorial is Not to:

– Suggest Format, Content, or Details of
Audit Plans for Data Integrity or to
Debate Pros & Cons
• Intent of Tutorial is to:

– Encourage Use Systematic
Preparation Techniques to Identify
Priority Target Issues Before
Beginning the Audit, and
– Emphasize Need to have a Clear
Focus on "The Issue(s)"
Slide 43
Always Decide Audit Focus Before Arriving on Site
• Effective Audit Planning Means Always:
• Having a Good Reason for Selecting QS

System/Area of Focus
• The Issue(s)" - Knowing what is intended to
be covered before arriving on Site
• Expecting to Modify the Audit Plan after
Assessing Findings of the PREDICT phase
(Described in Tutorial Part 02), because .....
Slide 44
"No plan survives contact with the enemy"*
*Helmuth von
Moltke
(1800-1891)
Slide 45
5-Ws Planning the Approach for Data Integrity Audits
WHAT What Quality System(s) & Areas Will be Your

Target Focus [i.e., What is "The Issue(s)"
WHY Why Did You Choose this QS & Area of Focus?
WHO Who is responsible to Prevent & Detect "The

Issue(s) [i.e. Which Department(s)]?
WHEN When Will Planned Focus Areas Be Covered

(Audit Timing)?
WHERE Where Do You Intend to Look (Areas of Focus)?
Slide 46
5-Ws - The "WHATs" for Preparing Data Integrity Audit Plan
• What is Auditor Looking For? (i.e. What is "The Issue")?
• What Quality System Will be Covered (and whether Development, Clinical Trial
Materials, or Commercial Production) ?
• What will be Areas Focus of each QS (Topics) ("The Issues")?
• What are Sources of Regulatory References that Define the Requirements for
each Topic?
• What are Sources of Relevant Technical Information for each of "The Issue(s)"?
• What Potential Issues have been reported at other companies for each of The
Issue(s)" of focus?
• What Has Auditor Found ("The Issue(s)") at other Companies?
• What References for "The Issue(s)" will be Retrieved for Use During the Audit?
• What Audit Tools/Aids will be needed during the Audit?
• What Files, Findings, Reports, Observations, recommendations are available
from previous Audits for the target QS or focus topics ("The Issue")?
Slide 47
5-Ws - The "WHYs" for Preparing Data Integrity Audit Plan
• Why is Auditor Planning to Look for "The Issue") vs Other potential

Issues?
• Why Has One QS vs Others Been Selected for Coverage to detect "The
Issue"?
• Why Have areas of Focus been selected vs those not selected?
• Why Other Companies have Not been Successful in Managing
(Detecting & Preventing) "The Issue"?
• Why does auditor need access to relevant References on regulatory
requirements or technical information about "The Issue" during the
audit?
• Why Auditor benefits from having available Files, Findings, Reports,
Observations, recommendations from previous Audits for the target QS
or focus topics?
Slide 48
5-Ws - The "WHOs" for Preparing Data Integrity Audit Plan
• Who [the Department(s)] is responsible for managing the relevant data

integrity controls for "The Issue" ?
• Who will Auditor need to interview or observe behavior in order to find
"The Issue")?
• Who will be needed to provide information about relevant Company
Requirements (Policies, Procedures, Methods, practices, &
records/documents for The Issue" ?
• Who will be primary points of contact & Who is Responsible for managing
Audit Logistics and lines of communication for The Issue" ?
Slide 49
5-Ws - The "WHENs" for Preparing Data Integrity Audit Plan
• When will Audit begin & end audit (planned duration)?
• When is the time that is most likely for "The Issue" to occur and/or be detected
by auditor?
• When will auditor visit production and control areas to observe employee
behavior and practices versus office performing Office document reviews?
• When is optimum time for auditor to visit various areas to observe employee
behavior and practices (which days of the audit, which shifts)?
• When will Each Department/area be covered during the audit (i.e, tentative
order of coverage/schedule)?
• When will auditor begin to prepare written observations & drafting report?
• When will auditor know the evaluation of "The Issue" is completed & its now
time to move to assessment of "the Next Issue" ?
Slide 50
5-Ws - The "WHEREs" for Preparing Data Integrity Audit Plan
• Where are areas that "The Issue" is most likely to occur &
most likely to be detected by auditor?
• Where will auditor begin looking for "The Issue(s)"?
• Where will auditor determine employee practices for "The
Issue"(on shop floor versus interviews in conference rooms)?
• Where s optimum location to perform interviews to detect
"The Issue" (In Offices vs On Shop floor)?
• Where are records & documents located that are most likely
to exhibit "The Issue"?
• Where are Optimum locations for review of Records and
Documents to detect "The Issue" (in office vs at Employee
Workstations or sources)?
Slide 51
Section 3.2
Predict
Facility Survey Techniques to Decide Scope/Approach

("Improving-the-Odds")
Slide 52
Why Use Predictive Techniques?
• Purpose
– To increase Chances of Finding: "The
Issue" (Data Integrity Discrepancies)
• Why Used ?
– Limited Time Availability
– Limitations of Random Record Selection
– Expedites Review Process
– Improves Odds of Detecting "The Issue"
Slide 53
Limitations of Random Selections by Auditor
Low Probability High Probability
If Randomly Choose If Selectively Choose

Records/Reports Records/Reports
Employee Behavior Employee Behavior
Workstations Workstations
Procedures Procedures
Slide 54
Using "Rule of 3" to Estimate Sample Size for Safety Analysis*
Examples: Equation n = 3/p

Risk Rate Sample
Criteria
(p) size
very common 1/10 30
common 1/100 300
uncommon 1/1000 3000
rare 1/10000 30000
very rare 1/100000 300000
If the underlying risk/error rate is 1% (1/100), a sample size of 300

is required to detect at least 1 event/error with 95% probability
*Courtesy of Martin Roessner, PAREXEL Slide 55

"Test -of-Reasonableness"
• Concept
– All Information & Data (From Every
Source) for "The Issue" is "Tested"
for "Reasonableness"
– Failure to Pass the "Test" = Extra

Scrutiny (Becomes a Predictor for
"The Issue" ) …continued
*Reference: Tetzlaff, R.F., "New Drug Development…Part III, " Pharm Tech, Jan 1993
Slide 56
Predictor Survey Approach
Obtain Information Across

Multiple Quality Systems
Overall Familiarization About: Gather Preliminary
– Employee Behavior
Survey Information
– Workstations
– Records & Reports
(to Decide Audit Focus)
– Procedures
Slide 57
Obtain Subjective Clues from All Areas
• First Impressions
– "2 Second Survey"

– "2,000 Second Survey" Becomes "Audit"
• "The Issue" Something seems "Not Quite

Right"
– Unexpected or Suspicious (Appearance
Information, Results, Behavior, etc.)
Slide 58
Duration of Survey
Timing Avoid "Tipping Your Hand"

(Impromptu Vs. Announced) About Areas of Focus
(Recognize Employees May
– Advantages
Alter Normal Practices if they
– Disadvantages know Details of Planned Audit)
• Keep Duration Short

– Variable Depending on Size Maintain Pace
& Complexity of Facility, e.g.
• Quality Management (1-4 hrs) • Try to Stay Within Planned
• Materials (1-2 hrs) Time Schedule
• Bldg & Equipment (2-4 hrs)
• Avoid Temptation to Start
• Production (1-4 hours)
• Pkg & Labeling (1-4hrs)
"DETECT" Assessment before
• Laboratories (1-4 hours) Finishing Predictor Survey
Slide 59
Concept - "Improving the Odds"
Delay Investigating
If Find Obvious or (Until Completing Survey)
Suspected Data (Avoid Temptation to "Chase"
Integrity Issues Apparent Discrepancies (Some
Exceptions)
Complete the Survey

Examine All/Most Areas
(If Practical) (Stay on Track)
(Strive to Complete Survey before
Starting QS Reviews)
Finalize Audit Plan Analyze Survey Results to

Modify Audit Plan Based on Decide Audit Focus
Survey Predictors [Which Quality System(s), What
Topics, & Depth of Scrutiny ]
Slide 60
Survey (Screening ) Techniques
Identify Handling/Storage
• Workstations
Practices that Could
• Observe Handling & Storage of
Documents & Records by Result in Data or
Operators & Managers in each Information not being
area (Production, Labs, QA, Reported Accurately or
Warehouse, Engineering, etc)
Completely
• Employee Behavior
– Witness Employees "First Impressions" are a
Performing tasks Reliable Predictor of
(Production, Labs,
Warehouse, QA, Areas with High Potential
Engineering, etc) for Data Integrity Issues
– Obtain "First Impression" in
Each Area Examined Slide 61
Survey (Screening ) Techniques
• Records & Reports

• Rapidly Screen large numbers Judge Incidence of
across Many Systems of for Errors, Omissions,
Format, Appearance, Content,
Clarity, Consistency, & Good
Corrections, Blanks,
Documentation Practices (GDP) Write-Overs, etc
("Too Many" vs "Too few"
• Procedures (SOPs, Methods,
Master Production Records,
Specifications, Etc)
• Rapidly Screen a Large "First Impressions" are a
Number across Multiple
Systems for clarity and Reliable Predictor of
completeness of instructions Areas with High Potential
related to GDP, etc. for Data Integrity Issues
• Obtain "First Impression" for
each Quality System Examined
Slide 62
Reminders about Predictors
• Techniques Described in Section 3.2 - PREDICT are

Intended to Help Decide Which QS to Audit & the
Focus
• Avoid the Temptation to "Chase" Data Integrity Lapses
before Completing the Survey (some exceptions)
Slide 63
Production
Laboratories
Areas
Examples of Workstations Predictors
Warehouse Engineering, Utilities

Areas & Maintenance Areas
Slide 64
Contemporaneous Entries
Survey Employee Behavior Across All Departments

• Observe Employees While Working at their
Personal Worksites including:
– On Processing Lines,
– At Laboratory Workbenches
Auditor Tips
– In Warehouse (Receiving & Shipping)
– Engineering (Calibration & Maintenance)
• Are Entries being made Contemporaneously
for Significant Activities ? i.e., per:
– Regulatory Requirements
– SOPs & Other Written Instructions
Slide 65
Sources of Data Integrity Issues
Survey Many Workstations in Each Area/Department

Examine Workstations Where Employees Perform
Tasks:
• Employees Perform Activities, Maintain their
Raw Data or Prepare Records and Reports,
Storage areas including File Cabinets, Desks &
Offices in:
– Production Areas
– Laboratories
– Offices of Supervisors & Managers Who
Review Documents & Records, including QA
– Worksites & Offices of Engineering,
Maintenance & Cleaning Staff, or other Support
Functions Slide 66
– Others © 2016 PAREXEL International
Observe "Real Time" Entries for Observed Activities

• Watch Employees Perform each
Step of their Job to Determine:
– Do they document each significant step
at time of occurrence?
Auditor Tips – Is the Batch Production or Control

Record located in close Proximity to the
activity (or is record located
elsewhere)?
– Are employees recording anything on
scrap paper, Post-It-Notes or personal
diaries?
– Do employees leave the area before
entering results of activity observed by
auditor? Slide 67
Unobtrusively Observe Employees Performing

In-Process Production & Control Activities
• Verify Data Entries on In-Process
Records are in "Real Time," e.g.:
• Be Alert for Missing Entries for
Activities Already Completed
• Search for Entries Made in Advance
- Repetitive Operations (Check
Dates & Times)
- Confirm Signatures, Dates, &
Times Correspond to Current
Stage of each Activity
Slide 68
Observe Employees from Inconspicuous Vantage Points

• Use Peripheral Vision
• Observe Employee Behavior from a “one off
position” e.g., :
– Walk One Packaging Line While Actually
Auditor Tips
Observing Actions of Employees on An Adjacent
Line
– Ask Questions of One Employee While Actually
Observing and Documenting the Actions of
Another Employee Within Auditor's Field Of
Vision
– Use Windows As Mirrors To Watch The Actions
of Personnel Behind The Auditor Slide 69
Appearance
Data Integrity Predictors
Slide 70
Post-it-Notes
Post-It-Notes & Scrap Paper
Data Integrity Predictors

May be Found
"Everywhere"
Slide 71
Bulletin Boards, Walls, Etc.
Examine Items Posted on Bulleting Boards, Walls &

Equipment/Instruments (Look Everywhere)
• Be Alert for:
• Post-it-notes with Raw Data
• Notes about Deviations, OOS
or Other Non-Conformance
Issues
• Memos/Letters Vs. SOPs
• "Cheat Sheets“
– Cycle Parameters
– Limits/specifications
– Obsolete SOPs … continued
Slide 72
Waste Receptacles
Look For Original Records & Discarded Raw Data on Scrap Paper
• Raw Data on Post-it-Notes or
Scrap Paper (Data transcribed
onto Original Record)
• Handwritten Calculations on
Scrap Paper
• Discarded Chromatograms or
Other Forms Containing Raw Data
• In-Process Control
Measurements (Weight Checks or
Inspections)
• Notes About Unusual Events
(Deviations & Discrepancies)
• Other Original Handwritten
Case Studies
Entries
Slide 73
Waste Receptacles
Look For Original Production & Control Records that were
Replaced with Duplicated Version
• Partially Completed Forms for

Batch Records or Analytical
Records after Personnel
Made Multiple Errors or
Corrections
• Replacing Entire Completed

Sections of BPR, Analytical
Record (or Other Record) with
Another Version of the Record
(With Original Discarded in
Trash) Slide 74
2Drawers & File Cabinets
Systematically Examine Contents of Cabinets & Drawers

(in Many or All Areas)
Auditor Tip
Never Open
Cabinets or
Drawers
Ask Escort to
Open
Slide 75
Drawers & File Cabinets
Examine "End-of-Day" Storage Locations at Workstations
• In-Process Production & Control

Records (Not Yet Submitted)
• Verify Real Time Entries
• Missing Entries for
Activities Already
Completed
• Entries Made in Advance of
Activities
• Raw Data Recorded on Post-it-
Notes or Scrap Paper
Slide 76
Examine Many or All Laboratory Workbenches & Cabinets
• Look Critically & Systematically

at:
– Testing Instruments
(Identification of status, asset
Numbers, Use Logs,
– Workbench Drawers & Cabinets
(look inside Many or All)
– Storage Rooms/Cabinets for
Reminder
Sample, Reagents, In-process
Samples Never Open Cabinets or Drawers
Ask Escort to Open Slide 77

Examine Many or All Laboratory Workbenches & Cabinets
• Instrument Use & Maintenance • Identification of Status

Logs – In-Process Samples
– Confirm Presence vs. Absence – Reagents & Lab Supplies
– Observe Storage Location
– Reference Standards
– Verify Whether Activities Are
Being Recorded
Contemporaneously
• Lab Notebooks & Worksheets

• Storage Location
• Contemporaneous Preparation
Slide 78
Identification
ID of In-Process Containers, Equipment, Samples, Reagents, Etc.

(Look "Everywhere")
• ID on Lab Samples,
Reagents & Reference
Apply
Standards
"Test of – Sample Labels Everywhere
(Workbenches, Incubators,
Reasonableness" Refrigerators, Desiccators,
Sample Preparations, etc
Everywhere You
– Dates, Lot Numbers, Product
Look
– Segregation of Samples
(Comingled Storage)
Slide 79
Reviewers & Managers (Offices)
Examine Reviewers Work Stations for "Real Time" Records
– How Do Reviewers
Document Discrepancies or
Deviations ?
• Use of Post-it-Notes,
Scrap Paper
• Entries about
Problems/issues
• Reference to Other
Records
Auditor Tip Look in the Drawers

Slide 80
Examine Storage of Records At Workstations

• Desks
– Personal Vs. "Official"
files
• Project Files
• Notebooks & Diaries
• Daytime Calendars
– Photocopy Notations Vs.
Original Records &
Documents
File Cabinets
Storage Lockers
Counters/Shelves Day-timers &
Drawers Calendars
Slide 81
Recognize Workload Pressure Points
Be Alert for:
– Evidence of Backlog of
Records to Be
Reviewed (e.g., Overdue
Reports)
– Appearance/Organization
• Too Cluttered vs "Too
Neat"
Slide 82
Predictor Survey Techniques for
Records & Reports
Slide 83
Document Retrieval Times
If Documents are Not Provided Quickly

(i.e., Unusual Delays)
– Is Pattern is Seen or Suspected?

– Identify Systems Involved With
Delays
– Note Date & Time & QS for Unusual
Delays (May be Quality related
reason)
– If Delays Seem Suspicious or
Unusual Go to the Originator
– Be Alert for Records that May Have
been Changed or Corrected Prior to
being Provided
Slide 84
Good Documentation Practices (GDP)
Rapid Screening Techniques for Records & Documents
"The Issue" Techniques
GDPs: • Scan Many Document Types
• Inconsistent Entries – Logbooks
– Dates or Times – Worksheets
(Sequence) – Batch Records
• Blank Spaces on Forms & • Scan Many Pages Rapidly
Logbooks – Few Seconds per Page
– Missing Data, signatures – Few minutes per Log Book or
or Dates Stack of Records
• Apply "Test of Reasonableness"
• Handwritten Entries
– Tag Those "Failing" the Test
– Unidentified or Undated
of Reasonableness"
– Altered/overwritten • Assess the Tagged Predictors
– Unexplained Changes – Confirm/Reject The Issue
– Inconsistent Writing Suspected
– Incidence of Errors & – Tabulate Incidence & Types
Corrections for "The Issue"
» "Too Few" vs "Too
Slide 85
Many" © 2016 PAREXEL International
Good Documentation Practices
Undocumented Handwritten Changes or

Alterations on Original Records
– Actions or Results Not Entered Accurately or

Completely
– Omitted Relevant Data or Information on
» BPRs, Laboratory Worksheets, Logbooks
– Entries of False or Misleading Data/Information,
e.g.,
– Pre-Entered Data (Before activity)
– Activities Already Completed w/o
corresponding Entries on Record
– Changing Entries Made by Another
Person
Slide 86
Good Documentation Practices
Examine Records & Documents in All Areas (Everywhere)

• Changes to Original Entries
• w/o Signature, Date or Explanation
• Erasures , Obliterated Entries, White-out
• Signing Records for Activities Not Performed, e.g.
• Reviewers Signing Operator Name when Blank Spaces
found during second Person Review
• Signature & Handwriting Discrepancies
• Handwriting Analysis (Comparison of Writing)
• Same Handwriting – Different Names
• Different Patterns or Ink Colors
• Inconsistent Writing Style (numbers and letters)
• Width of Ball Point Pen Marking (Fine, Medium, Large) Slide 87
Time & Attendance Discrepancies
Examine Records & Documents in Any/All Areas (Everywhere)
Be Alert for Evidence of:

Inconsistencies in Entries on
Batch Production or Testing
Records, e.g.,
Differences in Dates & Times vs
Attendance Records
Computer Security Logs
(Electronic Records)
Employee Entry & Exit Times in
Pass Card Areas
Slide 88
Omitting Relevant Production Data
• Look for Evidence of:
• Failing to Record Deviations or

Incidents on Batch Record
• Not Recording all Significant Steps

in Production process
• Gaps in Batch Production Records

(Blank Spaces)
Slide 89
Omitting Adverse Laboratory Data
• Not Reporting Laboratory Deviations

• Not Recording or Submitting
Laboratory OOS Test Results
• Reporting Only Passing Results

• Substituting Acceptable Test
Results from one Batch as
Representing Data for Another
Batch
Slide 90
Omitting Data from APRs
• Failing to Include Adverse Data (e.g.)

• Rejected Batches
• Deviations
• Changes to Equipment, Processes,
Methods, etc.
• OOS Test Results
• Stability Test Failures
• Customer Complaints
• Adverse Trends for Performance
Metrics
Slide 91
Second Person Verifications
• Evidence of Second Person

Verification
– Signing as “Verified by” or
“Witnessed by” or “Reviewed By”
when Second Person was not
Present at the Time of the event
• Evidence of QA Approval of
Entries on Records Without:
– Verifying Accuracy &
Completeness of Reported Data
or Information
– Confirming Conformance to
Requirements
Slide 92
Rapid Screening Techniques for Uncontrolled Logbooks
("Personal Diaries")
"The Issue" Techniques
Uncontrolled Raw Data: • Scan Many Diaries
• Raw Data for GMP Activities
• Scan Many Pages Rapidly
– Original data transcribed
– Few Seconds per Page
onto Official Record
– Spot/Mark those the seem to
• GMP Related Notes
be Obviously Related to GMP
• Things that happened or
– Spend Only Few minutes per
results for GMP functions
Log Book
such as Telephone
• Apply "Test of Reasonableness"
Conversations or Meetings
– Tag Those "Failing" the Test
abut GMP matters (often
of Reasonableness"
about deviations or • Assess the Tagged Predictors
problems) – Identify and Tabulate Those
• "Personal Notes Targeted for Investigation
• Notes about Personal Matters during DETECT phase of
Unrelated to Anything Audit
Relevant
Slide 93
SOPs - Screening Electronic Data to Predict "The Issue"
Quality System Data Requested Data Provided Search/Sort by [X]
[Fields] to Find "The Issue"
SOPS, Methods List of SOPs (Methods) by Excel Spreadsheet Sort by Dept to Find
Number & Title [Dept, SOP (Method) Expected Topics Not
No & Title ] Listed
SOPS, Methods List of SOPs (Methods) by Excel Spreadsheet Search Title field To
Number & Title [Dept, SOP (Method) Find Expected Topics
No & Title ] Not Listed
SOPS, Methods List of SOPs (Methods) by Excel Spreadsheet Sort on Eff Date field To
Number, Title & Eff Date [Dept, SOP (Method) Find Documents
No, Title & Eff Date ] outside expected or
required revision dates
SOPS, Methods List of SOPs (Methods) by Excel Spreadsheet Merge the 2 Excel
Number, Title & Eff Date [Dept, SOP (Method) Files, Sort on Method
No, Title & Eff Date ] Number, then Filter to
List of Methods Validation Excel Spreadsheet Find Methods having
Reports [Validation Report No, Zero Entries in
Method Number, Titile) Validation Field
Slide 94
Analytical - Screening Electronic Data to Predict "The
Issue"
Quality System Data Requested Data Provided Search/Sort by [X] to
[Fields] Find "The Issue"
Assay-Product X Assay Data for X Excel Spreadsheet Sort on Product, then Lot
Product for X Time [Product, lot #, Assay # , then Assay Value to
Period Values for X time determine if all lots have
period] results
OOS - Product X OOS Results for X Excel Spreadsheet Merge with Assay data
Product(s) for X Time [Product, lot #, Assay base to find OOS results
Period Values for X time not included in the Assay
period database
Slide 95
Deviations - Using Electronic Data to Predict "The Issue"
Quality System Data Data Provided [Fields] Search/Sort by [X] to
Requested Find "The Issue"
Deviations List of Deviations Excel Spreadsheet Sort on Each Field to find
by No., Reason, [Deviation No., Reason, (1) list of all deviations,
Date, Date Date, Date Opened, Date (2) frequency by reason,
Opened, Date Closed)] (3) deviations not closed
Closed) in timely manner
Deviations List of Deviations Extracted Excel Filter Options: Filter
by No.,[Deviation Spreadsheet [Deviation allows separating out key
No., Reason, No., Reason, Date information to
Date Opened, Opened, Date Closed) Brief
Date Closed) rief Description, Root Cause, • detect incidence
Description, Conclusions, Category, ] (prevalence of issues)
Conclusions, • Outliers
Root Cause, Turn On Excel Filters • Categories
Category,
Find Options : Keyword
Searches for DI Related
Terms
Slide 96
Deviations - Using Electronic Data to Predict "The Issue"
Quality System Data Data Provided [Fields] Search/Sort by [X] to
Requested Find "The Issue"
Deviations List of Deviations Excel Spreadsheet Sort on Each Field to find
by No., Reason, [Deviation No., Reason, (1) list of all deviations,
Date, Date Date, Date Opened, Date (2) frequency by reason,
Opened, Date Closed)] (3) deviations not closed
Closed) in timely manner
Deviations List of Deviations Two Excel Spreadsheets: Compare the 2 sorted
by Product, Lot (1) [Deviation No., Product, data bases to Find
Number, Date Lot Number, Reason, Date Released batches with
Opened, Date Opened & Date Closed)] Open Deviations:
Closed) (1) Sort Deviations
(2) Batch Disposition database on Product
Record [Product, Lot then Lot Number, then
Number, Disposition Status Deviation , then Date
(Date Released/Rejected)] Closed
(2) Sort Batch Disposition
Database on Product,
then Lot Number, then
Slide 97
Disposition date
Attendance - Using Electronic Data to Predict "The Issue"
Quality System Data Requested Data Provided Search/Sort by [X] to Find "The
[Fields] Issue"
Attendance Security Access Excel Spreadsheet Sort on [date, employee ID #, entry
log for [Area/Room Number, time, Departing time to find dates
Area/Room X for date, employee ID #, or time each employee was
X Dates entry time, Departing present in the room
time ] for X dates
Compare against Other Records
covering activities for the same
dates to determine if employee(s)
was present on date and times
recorded on batch production or
control record
Attendance Security Access Excel Spreadsheet Sort on Date to have list of dates
log for Site for X [employee ID #, entry Employee was present and
Dates time, Departing time ] compare against Any/All other
for X dates production or control records
Slide 98
Section 3.3
Detect
Audit Approach to identify Data Integrity Gaps
Slide 99
Approach - DETECT
DI Audits May Evaluate Any of Five Levels
Level 1
• Detecting Data Values
Integrity Issues
Involves
Level 2
Assessments at Any Requirements
or All of the 5 levels
• See Also Tutorial Level 3
Functions
Section 2.2 (Quality
Systems Approach
to Data Integrity - Level 4
Actions
Points to Consider
Level 5 Management Oversight

Responsibilities
Slide
100
Approach - DETECT
DI Audits May Evaluate Any of Five Levels
Management Oversight Responsibilities

Evaluate Management's Comparison of Slide
QS Performance Outputs Against Requirements 101
Values - Audit Elements
Quality Culture - Assess Appropriateness &

Completeness of Actions Taken by Senior
Management To Establish & Sustain a Strong
Quality Culture for Data Integrity
• Assess the Actions Taken that Embrace &
Communicate Data Integrity as a Core
Values Company Value
• Is Integrity & Honesty (Ethical Conduct) one
Level 1 of the corporate value statements?
• Review any or All Directives,
Communications, Training or other means of
communicating Management Expectations
• Interview Selected Employees in Key
Departments (at various Organizational
Slide
Levels) © 2016 PAREXEL International 102
Values - Audit Elements
Quality Culture - continued

• Assess Consistency & Uniformity of Understanding of
the Values by Employees. Do Employees at All Levels
Understand the Meaning of Data Integrity?
• Is there a common level of Understanding about
values, beliefs, thinking and behaviors about quality
Values and data integrity?)
• Observe Actions and Behavior of Employees in All
Level 1
Departments to determine consistently with core values
• Be Alert for disconnects where some employee exhibit
behavior and actions or do not possess an
understanding that is consistent with prevailing data
integrity core values
• Is "Hotline" available for Reporting Issues or Concerns?
Slide
© 2016 PAREXEL International 103
Requirements - Audit Elements
Code of Conduct*
• Confirm Availability of a Code of Conduct (or
Equivalent Elements contained in Policies or
Procedures) including:
• Definition of key elements necessary to
ensure the reliability and integrity of data and
information throughout all aspects the
development and production of drug
Requirements products, including::
Level 2 • Corporate Governance/Ethics
• Employee behavior (Ethics)
• Annual Employee Recertification of
Compliance to Code of Conduct
* See PDA, "Elements of a Code of Conduct for Data Integrity in the Pharmaceutical Industry"
http://www.pda.org/scientific-and-regulatory-affairs/regulatory-resources/code-of-
conduct/elements-of-a-code-of-conduct-for-data-integrity-in-the-pharmaceutical-industry
Slide
• Policies & Procedures - Confirm that the
Documents Clearly and Completely define the
meaning of Data Integrity and describe
requirements, e.g.:
• Systems, Processes, & Contols needed
to ensure data accuracy, truthfulness &
completeness (i.e., to Prevent & Detect Data
Requirements Integrity Lapses)
Level 2 • Ethical Behavior for the Actions (Conduct) of all
Employees
• Contemporaneous Data Entry for the
Collection, Analysis, Reporting of all significant
activities & results
• Good Documentation Practices (GDPs)
...continued Slide
Policies & Procedures (Continued)
• Systems, Processes, & Contols (Continued)
• Retention of all Data (Electronic & Paper
Records)
• Document Management
• Training and Education to Instill Data Integrity
Concepts
• Root Cause Investigations
Requirements
• Auditing / Inspections
Level 2
• Periodic review (APR/PQRs)
• Change management
• Control of Data for Outsourced Activities
• Other QMS sub-systems
• Manufacturing Controls for Data Integrity
• Production Processes & Products
• Packaging & Labeling Processes
Slide
...continued
Policies & Procedures (Continued)
• Management Review of QMS Process

Performance for Data Integrity Controls
• QMS Outcomes (Assess Trends for Data Integrity)
• Evaluate Internal and External Data Integrity
Factors by Reviews of:
Requirements • Regulatory Awareness/Trends
Level 2 • Inspection Observations at
• Own Company
• Other Companies
• Internal Audit Trends
• Data Integrity Risk Assessments
• Facilities & Equipment controls for Security
& Accessibility
• CAPAs Effectiveness for Data Integrity Lapses
Slide
Functions - Audit Elements
• Verify Appropriateness of Data

Inputs for each Data Function:
Functions • Collect Data
Level 3 • Analyze Data
• Report Data
• Retain Data
Slide
Functions - Audit Elements
Quality Facilities & Pkg & Laboratory
Materials Production
Management Equipment Labeling Control
Annual Product Environmental Contract Packaging & Environmental
Validation
Review Control Management Labeling Monitoring
Audit Laboratory
Management Equipment Materials Processing Investigations
Management Management Control (OOS/OOT/ Lab
Change
Deviations)
Management
Facility & Utility Supplier Technology Laboratory
Complaint Management Management Transfer Management
Management
Deviation
Method Transfer
Management / Data Functions
CAPA
Reagent and
Document & Collect Employee Conduct (Good Standards
Records Documentation Practices) & Management
Management
Electronic Acquisition
Sampling and
Lot Disposition Inspection
Analyze Paper and Electronic Records
Personnel Stability
Management Report Handwritten & Electronic
Management
Records/Reports Test Performance
Review & Review
Retain Storage, Retrieval & Archiving of
Data Documents & Records (Paper & Slide
Integrity* Electronic) © 2016 PAREXEL International 109
Actions - Audit Elements
Confirm that the Actions (Outputs) for

each Data Function Meet the
Requirements Established in the
Level 2 Policies & Procedures for:
Actions
Level 4
Machine
Machines
s Employees
Employees
Actions to Actions to
Prevent Detect
DI Lapses DI Lapses
Slide
• Equipment Actions (Data Systems)
• Assess Actions Performed by Equipment that generates or

stores paper or electronic data & Records (Machine
Outputs)
Actions • Compare System Inputs/Outputs against Defined

Requirements/Specifications
Level 4
• Review controls for Calibrating Accuracy of
Sensors/Controllers/Recorders
• Confirm Qualification of All Instruments & Hardware
• Verify Validation of All Software Applications Impacting
Data Collection, Analysis, Reporting and Retention
• Review Security and Access Controls to Prevent
Unauthorized Changes to Hardware, Software or Data
that may impact data accuracy & reliability Slide
Employee Actions
Assess Activities Performed by Humans to meet the

requirements for each of the four functions
Actions • Verify Good Documentation Practices followed
Level 4 during Collection, Analysis, Reporting &
Retention of Paper Records
• Assess Controls for security and accessibility to
computer files (electronic Data)
Slide
Management Oversight Responsibilities - Audit Elements
• Policies & Procedures - Confirm availability of Defined

Requirements for Management Oversight Controls to
ensure the accuracy, truthfulness and completeness of
data and information
Oversight
• Risk Management - Review procedures, tools and criteria
Responsibilities for used by Management of assessing the impact of data
Level 5 integrity lapses, and deciding corrective actions and
disposition of impacted product(s)
• CAPAs - Verify Management Oversight of the

appropriateness of planned corrective actions and
confirmation by Management of the effectiveness of
completed corrective actions
Slide
• Oversight Controls - Assess Any/All Available

Oversight Systems used by Management to Prevent
and Detect Data Integrity Lapses
• Determine Management has confirmed Data
Integrity Lapses are Covered By Deviation
Oversight Investigations, root causes identified, and
Responsibilities
Corrective Actions Verified as Effective
Level 5
• Verify Escalation of Significant Data Lapses to
Management for Decisions about Disciplinary
Actions
• Review Management Trending Reports to Verify
that Data Integrity Lapses have been Addressed
Globally
Slide
• Data Integrity Trends - Verify Availability of

Management Assessment Reports for the Performance
of Quality Systems related to data integrity at each site
• Verify that Management Has established Systems for

Evaluating Current Industry Trends
Oversight
Responsibilities • Confirm controls are implemented to apply to the
Level 5 Company's own systems & controls, lessons learned
by other companies
• Internal Data Integrity Audits - Assess Management
Oversight of Data Integrity Issues detected during
Internal Audits including verification of Corrective Actions,
Risk Assessments based on trends, impact & Corrective
Actions
Slide
Periodic Management Reviews - Verify that procedures
require routine periodic reviews of data integrity related
systems, processes and controls
• Be alert for Data Integrity Lapses that may not have

received Management Oversight Reviews
Oversight
Responsibilities
• Verify that Assessments Provide Provisions for
Management Tracking of Timelines, Impact, and
Level 5
Effectiveness of Corrective Actions
• Confirm APRs/PQRs accurately and completely reflect

data integrity known lapses and corrective actions
• Review Minutes of Meetings and/or Risk Assessment

Reports for evidence of Oversight & Appropriate
Conclusions & Decisions in reaction to Known dataSlide
integrity lapses
Periodic Management Reviews - (Continued)
• Data Systems Performance Reviews - Confirm

Management Oversight Reviews include
assessments of the Actions of data systems
assure accuracy of data during data collection,
Oversight analysis, reporting and retention.
Responsibilities
Level 5 • Verify that the reviews document the Evidence
(data/results) that demonstrates consistent
compliance with established requirements and
the evidence demonstrating the thoroughness
and robustness of the security controls to prevent
and detect unauthorized deletions, manipulations
or changes.
Slide
Periodic Management Reviews - (Continued)
• Employee Conduct - Verify Management reviews

include confirmation that employee behavior consistently
conforms to established requirements, including good
documentation practices
Oversight
Responsibilities • Verify that Management has documented evidence to
Level 5 show that data integrity concepts and principles have
been instilled in all employees
• Confirm that Management measures effectiveness by

assessing the frequency and root causes of data
integrity lapses
Slide
Addressing Deliberate Fraud, Falsification,
Manipulation of Data, or other Wrongful Acts
• Management Actions - If Instances have been reported,
confirm Management Adhered to established requirements,
such as:
• Verify Objective Investigation's were performed by
Oversight
Responsibilities Independent Personnel in a Fair and Balanced Manner, to
determine:
Level 5
• Confirm appropriateness of Root cause determinations
• Assess whether the same or similar practices or conditions
have happened before, and extend investigation as
appropriate
• Ascertain whether conditions or practices impact pending
or approved applications, distributed products or other
Quality Systems Slide
Disciplinary Actions for Wrongful Acts
• Oversight Actions - Assess Clarity & Completeness of Policies and
Procedures that define required behaviors
• Policy(s) for Disciplinary Actions - Confirm Communications to All
Employees Regarding Requirements & Consequences of deliberate or
negligent wrongdoing
Oversight • Disciplinary Pocesses Confirm Procedures & criteria for deciding
Responsibilities disciplinary actions have been followed for instances of data integrity
lapses
Level 5
• Documentation - Verify documentation of disciplinary actions is
available for instances of deliberate data falsification, destruction of
data, unauthorized alteration or manipulation of data or other
wrongdoing that compromises data integrity
• Notification - Confirm Employees Adhere to Policy(s) that Requires
Employees to notify management of any incidents or events involving
known or suspected wrongful acts (including data integrity issues)., and
confirm disciplinary actions are invoked when Employees fail to report
Slide
such instances © 2016 PAREXEL International 120
• Accuracy of Regulatory Submissions (NDAs,

ANDAs, MAAs, etc):
• Actions - Confirm that policies & procedures clearly
and completely define the systems, processes and
controls to ensure accurate and truthful data and
Oversight information are filed, including:
Responsibilities
• Prevention & Detection of Any Untrue Statements of
Level 5 Material Fact in any pending or approved Application or
other submission
• Confirmation of Accuracy, Truthfulness and
Completeness of Information & Data Supporting
Regulatory Applications (Comparison of Actual
Conditions and Practices Used During Production
and Control Operations against Commitments in
Pending/Approved Applications
Slide
Maagement Oversight Responsibilities - Points to Elements
Consider
Management Oversight Responsibilities - Audit
Notifying Regulatory Authorities (Actions)

• Correcting Application Integrity Gaps - Confirm availability
of procedures for addressing any instance where
inaccurate or incomplete data or information is known
or suspected to have been submitted to a regulatory
Oversight authority in a pending or approved application.
Responsibilities
Level 5 • Notifying Regulatory Authorities - Review Clarity *
Completeness of procedures for immediate notification
whenever it becomes known that an application
contains and untrue statement of material facts or omits
material facts (false, misleading, inaccurate or
incomplete)
Slide
Notifying Regulatory Authorities
• Self Disclosure Decisions - Management should obtain legal advice
concerning self disclosure of data integrity lapses that do not involve
submission of untrue statements of material fact or omissions of
material facts. Management must decide if it is s prudent to self-
disclose when not required by applicable laws, regulations or
legislative directions
Oversight
Responsibilities • Notification - Establish a Policy/Procedure that requires Employees
to immediately notify management of any incident or event that results
Level 5
in a Regulatory submission that contains an untrue statement of
material fact or omits a material fact. Establish penalties for failure to
report such instances.
Slide
Case Study
Techniques to Detect Data Integrity Lapses
In
Empower
Laboratory Chromatography Systems
Records & Reports
Slide
Auditing Techniques for Empower Software & Electronic Data Files
Review of Empower™ Functionality

& Areas of Auditor Focus
Slide
EMPOWER 3 – Menu Options
Waters Corporation
Slide
Empower Audit Techniques
Data Integrity Lapses Can Be Detected by Effective

Navigation of the Data Acquisition System Software
courtesy Waters Corporation:

https://www.google.com/search?num=100&site=&source=hp&q=waters+empower+chromatography+
data+system&oq=waters+empower+ch&gs_l=hp.1.0.0j0i22i30k1.2121.14527.0.17505.17.14.0.3.3.0.2
52.1759.0j11j2.13.0....0...1c.1.64.hp..1.16.1788...46j0i131k1j0i3k1j0i46k1j0i10k1.8-_3T1AVdmU Slide
Auditor Precautions - Avoid Interfering with Critical Operations
• Avoid any Auditor Actions That
Might Interfere with Operations
– Don't inadvertently cause a change of
alteration of raw data or original records or
Contribute to an Employee Violating an
Established Procedure or Requirement,
such as:
– Computer keyboard -
• If demonstration (or search query) is
needed request that the escort
organize, but only if
– Escort verifies the Computer is in Never touch the
"safe" mode (no possibility to alter keyboard*
or impact data), & Demo or Search
is Performed by Site Personnel *with Limited Exceptions
Slide
Review of Empower Project Folders
• Open Project Folders

& Review:
• Sample Sets for
the Product
• Injections
• Channels
• Results sets
• Signatures
• Filters
• Custom fields
• Audit trails
Waters Corporation
Slide
• Browse Empower Folders - Be Alert for:

• Non-Typical or Unusual Names (e.g., Known Projects vs
Unexpected)
• Discrepancies between entries contained in Lab Notebooks,
Instrument Use Logs, Analytical Worksheets, & Data Files
(including metadata in audit trail files)
• Browse Operating System Folders - Be Alert for:
• Folders that Differ from Those in Empower (different Names or
Locations)
Verify Against
Instrument Use
Logs
Slide
Waters Corporation © 2016 PAREXEL International 130
Be Alert for Non-Typical Project File Names
Examples
• Deleted • Overwrite
• Experiment • X+Product name
• Information • Product name
• Recheck (misspelled)
• Trials • Product Name
• Failed with Extra
Characters)
• 242
• @+Filename
Slide
Compare Folders/Files against Expected Tests
Waters Corporation
Slide
Empower3 – Review Data (Options)
Auditor
Selects Data
Review Options
Selection is
Based on Audit
Focus
Obtain Desired
Records for
Comparison
Against Audit
Trail Data
Waters Corporation
Slide
Review of Empower Audit Trails
• Detecting Trial Injections

• Examine Folders for Non-Typical
or Unusual Names (See previous
Slide)
• Examine Sample Sequence
Names (Non-Typical/Unusual)
• Single Injection Sample Sets
(Orphan Injections
Waters Corporation
• Compare Empower Data (Results)

Against Lab Notebook Entries and
Against entries in Electronic Data
Files (including Audit Trails) Slide
Export Audit Trail
Data to Excel
Format
Search
Sort
Dates
Key Words
Approvals
Rejections
Chronology
Sample ID Nos.
Vial Numbers
See
Demonstration
Waters Corporation
Slide
Conclusion
Audits provide an effective means for
independently verifying the Appropriateness of
Data Integrity procedures, systems, processes, &
controls
Effective Planning & Preparation can Improve the

likelihood of detecting Data Integrity Lapses
Some key elements of a data integrity audit plan

were outlined
Reviewed were some Predictor (Survey)

Techniques to Decide Audit Scope/Approach
Described were some Audit Techniques that can

be used to Detect Data Integrity Issues Slide

Auditing Techniques

Uploaded by

Copyright:

Available Formats

You might also like

Auditing Techniques

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Auditing Techniques

Uploaded by

Copyright:

Available Formats

Part 3.

Auditing Techniques for

Ronald F. Tetzlaff, Ph.D.

Corporate Vice President,

• At the Completion of This Course, Attendees Will Be Able

Describe the Elements of a Data Integrity Audit Plan

Summarize Prediction (Survey) Techniques to Decide Audit

Describe Audit Techniques that can be used to Detect Data

Some Systematic Techniques to

Answer - Just About Anywhere

Company Discrepancies from

Regulatory Discrepancies from

(Maintaining Awareness of Relevant Laws,

• Regulatory Authorities - Many diverse laws,

• Publications - Countless publications (trade association

Organizing Available Information for Use

Organizing Available Information for Use

• All Auditors Collect Various References

– Minimize the Amount of Time for Obtaining &

• Travel to Remote sites limits the Auditor to carrying

• Keeping Abreast of Data Integrity Requirements

• Standards Bodies - Pharmacopeias, ICH, ISO, ASTM,

• Trade Associations - ISPE, PDA, etc.

• Publications & Periodicals - Subscriptions & Internet

Data Integrity Auditors Must Always be Looking For Something

If Auditor Looks for a Specific Data Integrity Lapse ("The

Therefore, Audit Planning & Preparation boils down to Deciding:

• Effective Audit Planning Means Always:

• Having a Good Reason for Selecting the

Scope (GxP) - Decide Data Integrity Audit Scope Based on

Discovery Development Manufacturing Distribution

Methods Dev. & Validation

Clinical Investigators (GCP)

Clinical Trial Materials (GMP

Commercial Batch Production

Applications (Regulatory Slide 22

GxP Prior Previous Current Current QS Others

Dev Studies - Formulation

Accuracy Reviews - Assess the Procedures, Systems, Processes &

• Request Advance Copies of Submissions Filed with

• For Cause Data Integrity Audit

Responsibilities - RA is Responsible for Ensuring that

Reminders to Regulatory Affairs Personnel:

(FDA Form 356h)

FDA Form 356h - FDA New Drug Applications

Warning: A willfully false statement is a criminal offense, U.S. Code,

• Requires a positive affirmation that the data and information in the

"The data and information on this

Management Test Performance

Use Risk Assessment to Identify Quality Systems & Topics

Some Auditors Prepare Written Audit Plans, Others Do Not

• Intent of Tutorial is Not to:

• Intent of Tutorial is to:

• Effective Audit Planning Means Always:

• Having a Good Reason for Selecting QS

WHAT What Quality System(s) & Areas Will be Your

WHY Why Did You Choose this QS & Area of Focus?

WHO Who is responsible to Prevent & Detect "The

WHEN When Will Planned Focus Areas Be Covered