TMMi Introduced in 2007 TMMi Foundation

Whyy TMMi?
 A non-profit
non profit making organization
 TMMi is based on TMM (partially published in
 Dedicated to improving test processes and
1996))
practice.
ti
 New CMMI replacing the old SW-CMM
 Focus: development of a common, robust model of
 Need for independent test maturity test process assessment and improvement in IT
assessment organizations – non-commercial test improvement
 Need for standard certification model
 Focus on testing and integration with CMMI  As of Dec. 2009, 329 members representing 32
nationalities

Companies from UK, India, Korea and Japan have used

thi model.
this d l
TMMi TMMi
Page 1 Hareton Leung, 2010 Page 2 Hareton Leung, 2010

TMMi Foundation TMMi Levels

Level 5 Optimization
Provides Defect prevention
Test process optimisation
 A standard staged TMMI model that can be used in isolation Quality control
or in support of other Software Process Improvement Level 4 Management and Measurement
models Test measurement
Software quality evaluation
 An independently managed data repository to support TMMI Advanced peer reviews

assessment method accreditation, assessor and Level 3 Defined

Test organization
assessment certification/validation and validated Test training program
assessment data and certificates Test life cycle and integration
Non-functional testing
 Assessment method accreditation/audit framework for TMMI Peer reviews
in accordance with ISO15504 Level 2 Managed
Test policy and strategy
 Certification
C ifi i and d training/examination
i i / i i process, proceduresd Test planning
and standards for formal, public accreditation of assessors Test monitoring and control
Test design and execution
and lead assessors and the on-going management
management. Test environment
TMMi
Level 1 Initial TMMi
Page 3 Hareton Leung, 2010 Page 4 Hareton Leung, 2010
 Components within each Level 1 Organization
Mat rit Le
Maturity Level
el
• Testing is a chaotic and is often considered a part
Maturity off debugging.
d b i
indicates
Level contains • The organization usually does not provide a stable
en ironment to ssupport
environment pport testing
testing.
Test process Process • Success in these organizations depends on the
capability Area competence and heroics of the people in the
achieves contains
organization.
• Tests are developed in an ad ad-hoc
hoc way after coding
Specific and Practices
(Specific,
is completed.
Generic Goals 5
Generic)) • The objective of testing is to show that the software 4
describes runs without major failures. 3
2
Implementation 1
TMMi TMMi
Page 5 Hareton Leung, 2010 Page 6 Hareton Leung, 2010

Level 1 Organization Level 2 Organization

• Testing becomes a managed process and is clearly
• The
e de
delivered
e ed pproduct
oduct o
often
te does not
ot fulfill
u itsts needs,
eeds, is
s separated from debugging.
debugging
not stable, or is too slow to work with. • Testing is still perceived as being a project phase that
g there is a lack of resources, tools and
• Within testing follows coding.
well-educated staff. • A company-wide or programme-wide test strategy is
• Organizations are characterized by a tendency to over established.
commit, abandonment of processes in a time of crises, • Test plans are also being developed. The test plan
and an inability to repeat their successes. defines test approach and what testing is required,
• Products
P d t tendt d nott tot be
b released
l d on titime, b
budgets
d t are when, how and by whom.
overrun and quality is not according to expectations. 5 • Risk management techniques are used to identify the 5
• No defined process areas
areas. 4 product
d risks
i k b basedd on d
documented d requirements.
i 4
3 3
2 • Testing is monitored and controlled to ensure it is 2
1 going
i according
di tto plan.
l 1
TMMi TMMi
Page 7 Hareton Leung, 2010 Page 8 Hareton Leung, 2010
 Level 2 Organization Generic Goal of Level 2
• The status of the work products and the delivery of GG 2 Institutionalize a Managed Process
testing services are visible to management
management. • GP 2.1
2 1 Establish
E t bli h an organizational
i ti l policy
li
• Testing may still start relatively late in the development • GP 2.2 Plan the process
life cycle
cycle, e
e.g.
g during design or during the coding phase
phase. • GP 2.3 Provide resources Achieving
• Testing is multileveled: there are unit, integration, system • GP 2.4 Assign responsibilities Level 2
and acceptance test levels
levels.
• GP 2.5 Train people
• The main objective of testing is to verify that the product
p
satisfies the specified requirements.
q • GP 2.6 Manage configurations
• Many quality problems occur because testing occurs late • GP 2.7 Identify and involve relevant stakeholders
5 • GP 2.8 Monitor and control the process 5
in the development life cycle. Defects are propagated
4 4
from the requirements and design into code. 3 • GP 2.9 Objectively evaluate adherence 3
• No formal review programs. 2 • GP 2.10 Review status with higher level management 2
1 1
TMMi TMMi
Page 9 Hareton Leung, 2010 Page 10 Hareton Leung, 2010

Level 3 Organization Level 3 Organization

• Testing is fully integrated into the development life
cycle.
y • Now do non-functional testing, e.g. on usability
• Test planning is done at an early project stage, e.g. and/or reliability.
during the requirements phase. • Testing processes are tailored from the
• The organization’s set of standard processes is organization’s set of standard processes to suit a
established and improved over time. particular project or organization unit and therefore
are more consistent except for the differences
• A test organization and a specific test training
allowed by the tailoring guidelines.
program exist, and testing is perceived as a profession.
• Test cases are gathered, stored and managed in a
central database for re-use and regression testing. 5 5
• Basic
B i ttools
l supportt kkey ttesting
ti activities.
ti iti 4 4
3 3
• A formal review program is implemented. Reviews 2 2
take place across the life ccycle.
cle 1 1
TMMi TMMi
Page 11 Hareton Leung, 2010 Page 12 Hareton Leung, 2010
 Generic Goal of Level 3 Level 4 Organization
• Testing is a thoroughly defined, well-founded and
GG 2 Institutionalize a Defined Process measurablebl process.
• GP 3.1 Establish a defined process
p • The organization and projects establish quantitative
objecti es for prod
objectives productct quality
q alit and process performance
• GP 3.2 Collect improvement information and use them as criteria in managing them.
• Product quality and process performance is
Achieving understood in statistical terms and is managed
Level 3 g
throughout the life cycle.
y
• Measures are incorporated into the organization’s
5 5
measurement repository to support fact-based decision
4 4
3 making. 3
2 2
1 1
TMMi TMMi
Page 13 Hareton Leung, 2010 Page 14 Hareton Leung, 2010

Level 4 Organization Level 5 Organization

• Reviews and inspections are considered to be part of
testing and used to measure document quality quality. • Testing
est g iss now
o a completely
co p ete y defined
de ed pprocess
ocess a
and
doone
e
• The static and dynamic testing approach are integrated is capable of controlling the costs and the testing
into one. Reviews are formally used as means to effectiveness.
control quality gates. • Organization continually improves its processes
• Products are evaluated using gqquantitative criteria for based on a quantitative understanding of the common
quality attributes such as reliability, usability and cause off variation
i ti inherent
i h t iin processes.
maintainability. • Improving test process performance is carried out
• An organization wide test measurement program through incremental and innovative process and
provides information and visibility regarding the test technological improvements. The methods and
5 5
process Testing is perceived as evaluation; it consists
process. 4 techniques are optimized and there is a continuous 4
of all life cycle activities concerned with checking 3 focus on fine-tuning and test process improvement. 3
2 2
products and related work products. 1 1
TMMi TMMi
Page 15 Hareton Leung, 2010 Page 16 Hareton Leung, 2010
 Level 5 Organization Some Statistics
• Defect prevention and quality control are practiced.
Statistical sampling,
sampling measurements of confidence According
g to a study
y done by
y Experimentus
p of UK
levels, trustworthiness, and reliability drive the test  Survey 100 companies
process. The test process is characterized by  72.5% were at TMMi level 1 (they are working in a
sampling based quality measurements. chaotic, hero-based way but starting to build project
• A detailed procedure exists for selecting and based processes)
evaluating test tools. Tools support the test process as  27.5% were at TMMi level 2 (they have some
much as possible during test design, test execution, established project based process and are moving
regression testing
testing, test case management,
management etc etc. t
towards
d iimplementing
l ti process att an organizational
i ti l level)
l l)
• Process reuse is also practiced by a process asset 5  More than 70% of respondents do not have metrics in
library
library. 4 place to monitor or manage testing goals
goals.
• Testing is a process with the objective to prevent 3
2
defects. 1
TMMi TMMi
Page 17 Hareton Leung, 2010 Page 18 Hareton Leung, 2010

Development of TMMi References

 Level 2 and 3 have been defined and used
TMMi version 2, http://www.tmmifoundation.org/
p g
 A working
ki group iis d
developing
l i llevell 4 and
d 5 now.
A Model to Assess Testing Process Maturity - Nov 98,
Work Plan Crosstalk,
• Jan 29: Submit draft of ML4 Process Area Preamble (Scope, http://www.stsc.hill.af.mil/crosstalk/1998/11/burnstein.asp
Purpose) and Specific Goals to review committee
• Mar 09: Submit draft of ML5 Process Area Preamble (Scope,
Purpose) and Specific Goals to review committee
• Mar 26: Submit draft of ML4 Specific and Generic Practices We invite you to participate in the development off TMMi
to review committee
• May 04: Publish Maturity Level 4 level 4 and 5.
• Jun 01: Submit draft of ML5 Specific and Generic Practices If interested, please contact
to review committee
Hareton Leung, 27667252, cshleung@comp.polyu.edu.hk
• Jul 13: Publish Maturity Level 5
TMMi TMMi
Page 19 Hareton Leung, 2010 Page 20 Hareton Leung, 2010