Quality Throughout The Software Life Cycle: Jeffrey Feldstein Cisco Systems

Quality Throughout the Software Life
Cycle
Jeffrey Feldstein
Cisco Systems
jbf@cisco.com
jbf@sherpas.com
© 2006 Jeffrey Feldstein. All rights reserved. 1

sherpas.com
Presentation_ID
© 2004 Jeffrey Feldstein. All rights reserved. © 2004 Cisco Systems, Inc. All rights reserved. 2
Agenda
sherpas.com
• Introduce myself
• Product Phases (Overview)
• Test role in each phase
• Conclusion
• Question and Answer

Introduce Myself
sherpas.com
• Software Career: 25 years

• Software Test: 7 years
• Test Automation: 6 years
Current position at Cisco:

Manage a group of 65 test engineers located in San
Jose, India (2 sites) and Netanya, Israel, testing a
next generation Network Management Platform
(Active Network Abstraction)
Question 1
sherpas.com
Software Quality is who’s responsibility?
My answer: Everybody
Quality starts at the beginning and is the

responsibility of Product Management,
Development, Test, Documentation, Program
Management and Support

Question 2
sherpas.com
Can we test quality into the product
My answer: If you figured out a way to do this,

let’s write the book and retire ☺

Life Cycle Phases
sherpas.com
• Requirements Definition
• Development
• Test
Internal
External
• Post-Ship

Life Cycle Phases
sherpas.com
• Development
• Test
Internal
External
• Post-Ship

Life Cycle: Requirements (1 of 4)
sherpas.com
• Authored by Product Manager (Marketing)

• Developer’s role: Figure out how to build the product
Data structures
Algorithms
Resource Utilization
Security
May be difficult to think about testability, coherence and overall
quality
May already be introducing bugs in their thinking

sherpas.com
• Test Role
Find Bugs now (sooner = cheaper)
• Tester’s Concentration during the review

Are the requirements clear and understandable?
Are the requirements testable?
Are their conflicting requirements?
Consider assigning this to one person
Is the overall system cohesive?

sherpas.com
• Example 1:
“The GUI must be intuitive and easy to use”
• Example 2:
All data entry screens should have a button labeled
“View Shopping Car”, which resides in a constant
location in the window.
• Which requirement is testable?

• What are some examples from your experience?

sherpas.com
• Common omissions
Performance Requirements
Hardware sizing, user response times, database
capacities, scalability requirements
Environmental assumptions
Required Operating systems, browsers, web servers,
database engines, other dependencies

Life Cycle Phases
sherpas.com
• Development
• Test
Internal
External
• Post-Ship

Life Cycle: Development (1 of 3)
sherpas.com
• Begins at PRD Approval

• Roughly follows this pattern
Specifications (Functional + others)
Coding
Unit Test
Integration Test
Base-level-performance test
Hand off to test
• Note: All of the testing above is performed by

Development!

sherpas.com
• Functional Specification Review

Is the specification clear and understandable?
Does the specification address all of the requirements
in the PRD?
Is there “extra” functionality (not found in the PRD)?
Is each requirement fully addressed?
Does the specification describe a testable
architecture?
Is all the information required to develop the test
cases present?

sherpas.com
• Encourage informal 1 one 1 between

Development and test engineers
• Have a system for requirements traceability
• Aim for coding the products and the testing
simultanously.

Life Cycle Phases
sherpas.com
• Development
• Test
Internal
External
• Post-Ship

Life Cycle: Test Planning (1 of 3)
sherpas.com
• Occurs during development phase

• Typical Documents authored by test
Test Strategy – authored based on approved PRD,
simultaneous with Functional Specification.
Test plans – contains the detailed test cases (scripts)
to be run.

sherpas.com
• Test Strategy
Sometimes called the “Master Test Plan”
Outlines Automation goals and strategy
Describes tools used during test
Lays out the test lab
Security Vulnerability test approach
Lists major testing types (Functional, system, performance,
scale, stress and soak)
Should be able to scope the test effort when done
Allows other teams to give early feedback to the test team

sherpas.com
• Test Plans
Details for each test case (Automated and Manual)
Test to break orientation
Should contain many negative test cases
Testing in hostile environments (low memory, etc.)
• All documents should be reviewed by Product

Management, Development, and Documentation
• Developers can fix bugs simply by reading the
test plan
Life Cycle: Test Execution (1 of 3)
sherpas.com
• Begins at Handoff to Test (HOT)

• Important scheduling milestone
• Product team defines specific entry criteria
• During test three things happen
Quality improves
Upper management business pressures emerge
Product Managers get anxious
Fatigue sets in for the entire product team

sherpas.com
• Strict Ship Criteria should be established during

the planning phase to avoid bad decisions
• These criteria should be measurable and
objective quality goals
• Agreement should be reached by all teams up
front

sherpas.com
• Some example goals:

All System crash and “must fix” bugs are resolved
All identified and resolved bugs are verified
All customer-found bugs, from previous releases are
verified
Performance goals are met, results are documented
Total number of open bugs not to exceed “XX”
Defect density under a certain threshold
Code and branch coverage test targets are reached
(relative or absolute).
Life Cycle Phases
sherpas.com
• Development
• Test
Internal
External
• Post-Ship

Life Cycle: Post Ship
sherpas.com
• Keep testing after ship

• Opportunity to run automation for an extended
period
• Find additional bugs before the customer finds
them
• Good time to invoke Model-Based Testing
• Perform escape analysis on all bugs: Feedback
to improve future test plans
• Post Project Assessment
Conclusion
sherpas.com
• Product Quality is everybody’s job

• Product development is a team effort
• Plan and set milestones in advance
• All teams agree to phase entry/exit criteria
• Early test involvement leads to higher quality
software shipping sooner
• Test Engineering plays a vital role in every
phase of the product life cycle

Testability – Further Reading
sherpas.com
Software Test & Performance Magazine (January 2006)

http://www.stpmag.com/issues/stp-2006-01.pdf

sherpas.com
© 2004 Jeffrey Feldstein. All rights reserved.

sherpas.com/mbt 28
sherpas.com
Presentation_ID
© 2004 Jeffrey Feldstein. All rights reserved. © 2004 Cisco Systems, Inc. All rights reserved. 29

Quality Throughout The Software Life Cycle: Jeffrey Feldstein Cisco Systems

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Quality Throughout The Software Life Cycle: Jeffrey Feldstein Cisco Systems

Uploaded by

Copyright:

Available Formats

Quality Throughout the Software Life

© 2006 Jeffrey Feldstein. All rights reserved. 1

© 2004 Jeffrey Feldstein. All rights reserved. 3

• Software Career: 25 years

Current position at Cisco:

Software Quality is who’s responsibility?

Quality starts at the beginning and is the

© 2004 Jeffrey Feldstein. All rights reserved. 5

Can we test quality into the product

My answer: If you figured out a way to do this,

© 2004 Jeffrey Feldstein. All rights reserved. 6

© 2004 Jeffrey Feldstein. All rights reserved. 7

© 2004 Jeffrey Feldstein. All rights reserved. 8

• Authored by Product Manager (Marketing)

© 2004 Jeffrey Feldstein. All rights reserved. 9

• Tester’s Concentration during the review

© 2004 Jeffrey Feldstein. All rights reserved. 10

• Which requirement is testable?

© 2004 Jeffrey Feldstein. All rights reserved. 11

© 2004 Jeffrey Feldstein. All rights reserved. 12

© 2004 Jeffrey Feldstein. All rights reserved. 13

• Begins at PRD Approval

• Note: All of the testing above is performed by

© 2004 Jeffrey Feldstein. All rights reserved. 14

• Functional Specification Review

© 2004 Jeffrey Feldstein. All rights reserved. 15

• Encourage informal 1 one 1 between

© 2004 Jeffrey Feldstein. All rights reserved. 16

© 2004 Jeffrey Feldstein. All rights reserved. 17

• Occurs during development phase

© 2004 Jeffrey Feldstein. All rights reserved. 18

© 2004 Jeffrey Feldstein. All rights reserved. 19

• All documents should be reviewed by Product

• Begins at Handoff to Test (HOT)

© 2004 Jeffrey Feldstein. All rights reserved. 21

• Strict Ship Criteria should be established during

© 2004 Jeffrey Feldstein. All rights reserved. 22

• Some example goals:

© 2004 Jeffrey Feldstein. All rights reserved. 24

• Keep testing after ship

• Product Quality is everybody’s job

© 2004 Jeffrey Feldstein. All rights reserved. 26

Software Test & Performance Magazine (January 2006)

© 2004 Jeffrey Feldstein. All rights reserved. 27

© 2004 Jeffrey Feldstein. All rights reserved.

You might also like