Auditing Systems

Development Lifecycle
Audit Guidelines On How To Review
SDLC Framework
By
Nandasena T(NT) Hettigei
CISA, CISSP, CITP, CPA, CA

Copyrights © NTH 2007

Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building
Minneapolis . MN 1
 Section (1) - Introduction

• Introduction
• Big Picture
• What is SDLC

• Audit Approach
• Audit Scope & Objectives
• Auditing SDLC Framework

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
2
 Section (2) – Audit Process
• Evaluate Adequacy
• Waterfall Model
• Iterative Model
• Agile Model

• Validate Effectiveness
• Validate Common Components
• Project Management
• Auditor’s Role

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
3
 Section 1

Introduction
to
Systems Development Life Cycle

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
4
 Introduction

• Big Picture Blueprint

– Oversight
– Project management
– Development Life Cycle (SDLC)
• What is SDLC
– System or Software?
– How to add value?

SDLC is a methodology/framework that provides a systematic approach to

develop information systems/software while ensuring quality

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
5
 SDLC Audit Approach (1)
Audit Scope and Objectives
• Evaluate adequacy of the methodology
– Ensure system development follows a proven methodology
to maintain consistency, effectiveness and efficiency of the
systems development process in order to maintain the
quality of the outcome.

• Validate effectiveness of the methodology

– Validate by testing and substantiating that risks are
mitigated effectively by consistently adhering to the
methodology/controls.

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
6
 SDLC Audit Approach (2)
Frameworks/Models
• Traditional phase by phase model
– Waterfall model (linear and sequential)
• Iterative model
– RAD (Rapid Application Development)
– JAD (Joint Application Development)
– Spiral Model
– Synchronize-and-stabilize Model
• Agile model (timeboxes )
– ASD (Adaptive Software Development)
– FDD (Feature Driven Development, and DSDM)

( Vendor specific: HP-Mercury, IBM-RUP, Compuware - ASD, etc)

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
7
 Section 2 – Auditing SDLC

Audit Process
2.1 – Evaluate adequacy
2.2 – Validate effectiveness
Reminder - We have been following the standard audit process of:
9 Obtaining an understanding of the control environment
9 Evaluating the adequacy of controls
9 Assessing by testing of controls
9 Substantiating risk of controls objectives not being met
Source - Control Objectives for Information and Related Technology (CoBiT),
IT Governance Institute.

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
8
 Evaluate the Methodology (2.1.1)
Waterfall Model
• Analysis Phase
• Scope definitions
• Requirements Analysis
• Design Phase
• Functional Design
• Technical Design
• Business Process Design (Across all Phases)
• Development Phase
• Build/Coding
• Testing (unit, integration and system testing)
• Performance, Regression and Security testing
• QA testing (UAT)
• Delivery and Transition Phase
• Data conversion and Deployment
• Training and Support

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
9
 Evaluate the Methodology (2.1.2)
Waterfall Model
Recommended for:
– Customization or implementation of ERP or
other business support systems
– Replacement of a legacy system where you
have defined requirements
– Outsource developments with stage gate
payment terms

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
10
 Evaluate the Methodology (2.1.3)
Iterative Model

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
11
 Evaluate the Methodology (2.1.4)
Iterative Model

Recommended for:
– New product (application) development
– Prototype/Business intelligent systems
– Innovative projects/products
– Increment functionalities within a website

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
12
 Evaluate the Methodology (2.1.5)
Agile Model
• Self-contained mini-project
• Each lasting only a few weeks
• Each iteration has it own self-contained
stages of:
– analysis
– design
– development
– testing
– deployment and
– documentation
(Agile aims to reduce risk by breaking projects into small, time-limited modules i.e.
timeboxes)

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
13
 Evaluate the Methodology (2.1.6)
Agile Model

Recommended for:
– Large projects to use as a powerful
method to manage deployments
– Projects that require rapid and significant
change
– Projects where even late changes in
requirements are needed

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
14
 Evaluate Methodology (2.1.7)

• After all, you’ve probably noticed that the

three major development processes share the
same fundamental phases: design,
implementation, integration, testing and
deployment.

• Validating the processes are not different to

one another.

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
15
 Section 2.2 - Validation

• Validating key controls within

common SDLC components

Reminder - We have been following the standard audit process of:

9 Obtaining an understanding of the control environment
9 Evaluating the adequacy of controls
9 Assessing by testing of controls
9 Substantiating risk of controls objectives not being met
Source - Control Objectives for Information and Related Technology (CoBiT), IT
Governance Institute.

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
16
 Validate SDLC Components (2.2.1)

Analysis Validation
Functional 9 Business Case/requirements priorities
Requirements 9 High level use cases and required activities
9 Dependencies and redundancies (Impacted systems)
/Use cases
9 System inputs and outputs – data, interfaces, etc
9 Re-prioritize requirements as needed

Performance 9 Number of simultaneous users and transactions updates

Requirements 9 Scalability / Throughput / Capacity
9 Resource utilization (especially of shared resources)
9 Response time for a transaction
Security 9 Conceptual Access control requirements (SOD vs. Open)
Requirements 9 Conceptual Application Security ( HIPAA, PCI, GLBA, etc)
9 Conceptual System Security (internal vs. www systems)

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
17
 Validate SDLC Components (2.2.2)

Design Validation
Functional 9 Standard FD template that includes:
9 Complexity (High, Medium and Low)
Designs
9 Transaction Volume, Constrains and Dependencies
/Use cases 9 Risk, Controls, Security and Test scenarios

Technical Designs 9 Standard TD template that includes:

9 Reference to related FD and functions
9 Code, Error handling, systems and integration points
9 Data schema or reference to data tables
9 Security designs

Business Process 9 Standard BPD template that includes:

9 Process flows (systems and functions)
Designs
9 Controls, reports and process owners
9 Manual check points and test scenarios
- Revised throughout SDLC phases to accommodate functional changes

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
18
 Validate SDLC Components (2.2.3)

Build Validation
Development/ 9Development standard documentation that includes:
9 Coding standards
Coding 9 Nomenclatures, Comment lines and segments
9 Programming with multi-threading
9 Code reviews (peer reviews and performance reviews)
9 Application security/Source code analysis
9 Input, process and output controls
9 Error handling standards
9 Defects classifications (Showstoppers, Sev 1, etc.)
9 Unit testing, Coding quality control
9 Code version management

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
19
 Validate SDLC Components (2.2.4)

Integration Validation
System 9Integration approach should include:
9 Inventory of FDs and TDs with priorities and dependencies
Integration
9 Integrators, Adaptors and Middleware (MQ series)
9 System architecture, data flow diagrams
9 Integration with vanilla codes or functionalities
9 Iterative vs. Incremental integration
9 Integration Test approach
9 Dependencies (systems and processes)
9 Change and Version Control
9 Error handling

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
20
 Validate SDLC Components (2.2.5)

Testing Validation
Functional 9System Test approach should include:
9 Production like testing environment
Performance and
9 Acceptable defects rate (%)
Security Testing 9 Entry and exit criteria for system test
9 Unit test completed and acceptable defects rate
9 Code certified (if developed by a third party)
9 Functional test scenarios approved by stakeholders
9 Performance testing includes:
9 Number of users, Volume, response time, etc.
9 Security testing includes:
9 Application, Access and System security
9 Rework and retest standards
9 Regression testing

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
21
 Validate SDLC Components (2.2.6)

QA Validation
System/ 9System Quality Assurance approach should include:
9 Requirements quality (functions, performance and security)
Software 9 Defects tracking and trend analysis
Quality 9 Issue tracking and trend analysis system/tools
9 Stage gate sign-off process
Assurance 9 Security settings and role base access controls
9 Automated process workflows
9 System alerts for transaction exceptions
9 Regression testing
9 Performance and stress testing
9 Application and system security testing
9 UAT (user acceptance test) scenarios and testing
9 High availability, failover/recovery and disaster recovery
9 QA exit criteria – Meeting customer/business requirements

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
22
 Validate SDLC Components (2.2.7)

Delivery Validation
Deployment 9 Launch approach & customer impact assessment
9 Deployment timeframe and system down time (impact)
9 Data conversion and validation process
9 Go/No go decision points
9 Failover/recovery during the migration process

Support 9 Post deployment support (30 days – 6 months)

9 Expert teams knowledge transfer
9 Documents repository
9 Training support
9 Defects clearing
9 Problem resolution

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
23
 Validate SDLC Components (2.2.8)

Documentation Validation
Adequate 9 Requirements Documentation (catalogue)
Documentation 9 Design and Development Approach
9 Test and defects management Approach
9 Quality Assurance Approach
9 Deployment and Launch Approach
9 Functional Designs /Use Cases
9 Technical Designs and Data Schemas
9 Business Process Designs
9 Test scripts/scenarios, Issues log and defects log
9 Deployment process with contingency rollback
9 Security settings (access, system and roles)
9 System specification, data sheets and user guides

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
24
 Validate SDLC Components (2.2.9)

Tools Validation
SDLC 9 Change management tools
Tools 9 Quality management tools (e.g. Quality Center)
9 Issue tracking tools (e.g. PVCS)
9 Code version manager (e.g. Subversion)
9 Source code analysis tools (e.g. DevInspect)
9 Application QA tools (e.g. QAInspect)
9 Code migration tools/scripts
9 Validation checklists and standard templates
9 Enterprise target infrastructure (e.g. Tech Blueprint/BOB)
9 Enterprise information security policies & standards
9 Capacity, performances and scalability testing tools
(e.g. LoadRunner)

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
25
 Validate SDLC Components (2.2.10.1)

Roles Validation
Development 9 Architect (software, system and performance)
9 Business Systems Analyst
9 Developer, Code Reviewer, Tester
9 Security Architect
9 Product Manager/Business/process owner
9 Stakeholder
9 Technical Writer
9 Trainer
Quality 9 QA Manager
Assurance 9 QA Analyst
9 Security Analyst
9 Performance Analyst
9 Business SMEs ( Subject Matter Expert)

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
26
Validate SDLC Components (2.2.10.2)

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
27
 Project Management
Project Validation
Project 9 Project management methodology
Management 9 Adequate business engagement in the project
9 Project managers engaged with the stakeholders
9 IT leaders engaged with end users
9 Scope, Schedule and Budget monitoring
9 Interim Merit Reviews
9 Failsafe Approach

Project Risk 9 Project risk management process

Management 9 Organizational alignment (business readiness)
9 Adequate training and communication
9 Defined service levels
9 Defined project delivery process
9 Contingency plan and roll back approach

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
28
 Auditor’s Role

• Auditor Vs. Quality Assurance

– Auditor is not playing the role of quality assurance
• Auditor Vs. Risk Management
– Risk management is a project activity
• Auditor’s Role
– Auditor is a SME (subject matter expert) for risks
and controls (What may go wrong on process and
recommendation to mitigate such risks)

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007

BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
29
 Q&A
References;
1. IS Control Journal – The Auditor's Role in IT Development Projects – NT Hettigei
2. CoBit ; Control Objectives for Information and Related Technology (CoBiT), IT Governance
Institute. URL http://www.itgi.org/

Thank You
3. IT Auditing Standards – Information Systems and Controls Association URL
http://www.isaca.org/Template.cfm?Section=Standards&Template=/TaggedPage/TaggedPageDis
play.cfm&TPLID=29&ContentID=8529
4. ITIL – The ITIL and ITSM Directory – URL http://www.itil-itsm-world.com/
5. CMM – Capability Maturity Model – URL http://www.sei.cmu.edu/cmm/cmms/cmms.html
6. Which Development Method Is Right for Your Project? By Adam Kolawa URL:
http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=3152
7. Models for Managing Projects, IT Lecture Notes by Mark Kelly, McKinnon Secondary College
URL: http://www.mckinnonsc.vic.edu.au/vceit/models/index.htm#agile
8. Internet Security System White Paper: Dynamic Threat Protection: URL
http://documents.iss.net/whitepapers/DynamicThreatProtection.pdf
Download the presentation from ISACA website – URL http://www.mnisaca.org/

Email your questions to – nthettigei@fairisaac.com

Auditing the Systems Development Lifecycle - By NT Hettigei © 2007
BY NT HETTIGEI © 2007- Presented at ISACA Roundtable on 12/12/2007 at KPMG Training Center, Wells Fargo Building Minneapolis . MN.USA
30