Current 5000.

2
Milestone Phase Review Decision Point
A Concept Refinement Initial Technical Review Pre- MS A
A Concept Refinement Alternative System Review Pre- MS A
B Technology Development System Requirements Review MS A
C System Design Integrated Baseline Review MS B
C System Design System Functional Review MS B
C System Design Preliminary Design Review MS B
C System Design Critical Design Review MS B
C System Demonstration Test Readiness Review MS B
C System Demonstration Flight Readiness Review MS B
C System Demonstration System Verification Review MS B
C System Demonstration Functional Configuration Audit MS B
C System Demonstration Production Readiness Review MS B
Post-C Production and Deployment Operational Test Readiness Review MS C
Post-C Production and Deployment Physical Configuration Audit MS C
Post-C Production and Deployment Full Rate Production Decision Review FRP DR
Post-C Operation and Support In-Service Review FRP DR and Later

Adapted from Review Readiness Criteria developed for OUSD(AT&L)

 Reliability or
Review Activity Purpose Sub-Activity Evidence Evaluation
Maintainability

Refined reliability needs stated as broad Reliability requirements stated in system level
measures of effectiveness with thresholds and operational terms. Evaluation of the stated
objectives if possible requirements includes verifying completeness.

Documented reliability assumptions and

Reliability requirement analysis
To demonstrate that the program
reliability requirements are consistent
Reliability
with, and satisfy, customer needs and
expectations
Rationale for the preferred solution
supporting rationale
Key reliability risk estimate Documented risk assessment
Assessment of relative reliability risks associated
with using Commercial Off The Shelf/Non- Risk assessment includes evaluating potential
Developmental Items if anticipated in any of the reliability issues due to environmental or use
candidate system investigated in the Analysis of profile differences
Alternatives

Reliability requirement estimates documented in

Begin notional system reliability requirements
Reliability, Availability, Maintainability - Cost
development
Report supporting the Analysis of Alternatives
Initial Technical
Review Refined maintainability needs stated as broad Maintainability requirements stated in system
measures of effectiveness with thresholds and level operational terms. Evaluation of the stated
objectives if possible requirements includes verifying completeness.

Rationale for the Analysis of Alternatives

Documented rationale
solutions
Risk assessment for the Technology
To demonstrate that the program Documented risk assessment
Development phase
Maintainability requirement maintainability requirements are
Maintainability
analysis consistent with, and satisfy, customer Assessment of relative maintainability risks
needs and expectations associated with using Commercial Off The
Documented Commercial Off The Shelf/Non-
Shelf/Non-Developmental Items if anticipated in
Developmental Items risk assessment
any of the candidate system investigated in the
Analysis of Alternatives

Maintainability requirement estimates

documented in the Reliability, Availability,
Draft system requirements document
Maintainability - Cost Report supporting the
Analysis of Alternatives

Reliability cost drivers documented in the

Key program reliability life cycle cost drivers
identified
Reliability, Availability, Maintainability - Cost
Report. Rough estimates available for cost to
design in various levels of reliability and
associated reductions in life-cycle costs and
logistics footprint.

Assumptions documented in Reliability,

Reliability drivers and risks
identification for preferred
system design
Availability, Maintainability - Cost Report--
To demonstrate that reliability cost including optempo, expected environmental
Reliability assumptions documented
Reliability drivers and risks have been conditions, and Operational Mission
documented Summary/Mission Profile completed. Reliability
analysis part of Analysis of Alternatives.

Reliability experts involved in developing and

Reliability experts members of the program staff
reviewing the cost estimate
Analysis of Alternatives completed (Alternative Documented Analysis of Alternatives with
System Review entry criteria) reliability analysis

Alternative
System Review
 Reliability drivers and risks To demonstrate that reliability cost
identification for preferred Reliability drivers and risks have been
system design documented

Reliability risks documented and deemed Reliability requirements determined to be

manageable within cost estimates achievable within program schedule and budget.
Alternative
System Review Analysis of maintainability cost drivers
documented in Reliability, Availability,
Key program maintainability cost drivers Maintainability - Cost Report. Rough estimates
identified available for cost of various levels of
maintainability and associated reductions in life-
cycle costs and logistics footprint.

Assumptions, including personnel requirements,

Maintainability drivers and risks To demonstrate that maintainability spare part requirements/approach,
identification for preferred Maintainability assumptions, cost drivers, and risks Maintainability assumptions documented determination of maintainability demonstration
system design have been identified and documented requirements, documented in the Reliability,
Availability, Maintainability - Cost Report.

Maintainability experts involved in developing Maintainability experts members of the program

and reviewing the cost estimate staff
Analysis of Alternatives completed (Alternative Documented Analysis of Alternatives with
System Review entry criteria) maintainability analysis

Maintainability risks documented and deemed Maintainability requirements determined to be

manageable within cost estimates achievable within program schedule and budget.

Reliability requirements stated at a confidence

Analysis conducted to assure derived reliability level and documentation of reliability analysis.
requirements are reasonable Risks associated with ability to achieve
requirements within current technology.

Range of satisfactory reliability values

To ensure reliability requirements
Reliability meet customer expectations, needs,
and is affordable as a system
determined establishing a "design space."
User availability requirements and related
Effects of maintainability, support concept, life-
reliability values derived and documented
cycle cost, and logistics support and
maintainability.

Traceability matrix developed relating each

Traceability matrix
Evaluation of concept readiness reliability requirement to specified user need
MS A Decision for transition to Technology
Development Phase Reliability testing identified and incorporated into Quantitative reliability values verifiable by
preliminary program planning analyses, tests, or comparison.

Derived maintainability requirements analyzed Maintainability requirements stated at a

for reasonableness, ability to achieve within confidence level and documentation of
current technology, and risk areas maintainability analysis.
To ensure maintainability
requirements meet customer
Maintainability Maintainability requirements are quantified and
expectations, needs, and is affordable
as a system verifiable by analyses, tests, or comparison.
User availability requirements and related
Built In Test detection requirements
maintainability values derived and documented
documented. Maintainability requirements
allocated to subsystems/subcontractors.

Integrated
Baseline
Review
 Contract reliability requirements documented
(Inherent Availability, Mean Time Between
User reliability requirements translated into
Failure, Mean Time Between Removal, Mean
contractual specifications
Time Between Calendar Failure) at stated
confidence
System specification completed including
System specifications documented
reliability requirements

Initial reliability requirements allocation complete Lower level reliability allocation documented

Reliability Scorecard assessment completed

(scorecard can be found at
Functional baseline established to Scores indicate a strong reliability program.
Reliability https://acc.dau.mil/CommunityBrowser.aspx?
satisfy requirements id=210483&lang=en-US)

Request For Proposal includes a robust

reliability assessment and growth program as an
Request For Proposal clause for reliability
System integral part of design, development, and
System requirements systems engineering
Functional identification
Review Engineering (Data Collection Analysis and
Corrective Action System/Failure Reporting and
Integrated process to identify and mitigate failure
Corrective Action System, sneak circuit,
modes based on engineering and physics of
reliability enhancement testing, etc.) and
failure based approaches
physics of failure based processes documented
and initiated

Contract maintainability requirements

Functional baseline established to
Maintainability
satisfy requirements
User maintainability requirements translated into documented (Inherent Availability, Mean Time
contractual specifications To Repair, Administrative Delay Time, Logistics
Delay Time) at stated confidence
System specification completed including
System specifications documented
maintainability requirements
Maintainability requirements allocated to lower Lower level maintainability allocation
levels documented
Built In Test detection and false alarm risk
Documented analysis of Built In Test detection
balanced
Reliability program plan completed (individual
plan or part of ongoing System Engineering Plan Documented reliability program plan
updates)
Reliability addressed in ongoing System Reliability documented in Systems Engineering
Engineering Plan updates Plan
Engineering (Data Collection Analysis and
Corrective Action System/Failure Reporting and
Assess progress identifying and mitigating
Corrective Action System, sneak circuit,
failure modes based on engineering and physics
reliability enhancement testing, etc.) and
of failure based approaches
physics of failure based process outputs for
review

Reliability modeling and simulation initiated for Reliability model and simulation plan
system and lower levels as required documented

System and component level Developmental

Assurance that design maturity is
Reliability
sufficient to begin detailed design
Testing testing planned and begun as prototypes Developmental Testing test plan
become available
Robust effort initiated to identify and mitigate
failure modes with accelerated testing of
Reliability growth program documented
components followed by operationally-realistic
testing of subsystems and systems

Preliminary System requirements initial

Design Review evaluation
 Assurance that design maturity is
Reliability
sufficient to begin detailed design

Reliability Scorecard assessment updated

(scorecard can be found at
Scores indicate a strong reliability program.
Preliminary System requirements initial https://acc.dau.mil/CommunityBrowser.aspx?
evaluation id=210483&lang=en-US)
Design Review
Robust effort initiated to identify and mitigate
failure modes do to manufacturing variation and Documented production plan
workmanship errors
Robust effort initiated to identify and mitigate
Human factors, Human Systems Integration,
failure modes chargeable to operators and
etc. documented
maintainers (human errors)
Characterization of loads at various component
system locations through modeling or Component load models documented
instrumented prototypes

Maintainability requirement allocation completed

Maintainability of lower level assemblies stated
with Mean Time To Repair assigned at the sub-
at a confidence level
system and assembly levels

Human Systems Integration included as part of Human Systems Integration in System

Systems Engineering
Assurance the maintainability Testability analysis addresses diagnostics and
Maintainability program is sufficiently designed and trade-offs among Built In Test, Automated Test
implemented to begin detailed design Equipment, and other (automated or manual)
test methods
Failure mode identification activities on track
Maintenance risk areas identified, tracked, and
mitigated
Reliability documentation updated based on
results to date
System, component, assembly, and
subassembly reliability testing status
Engineering Plan
Testability analysis documentation
Top level Failure Modes and Effects Analysis
completed, Physics-of-Failure analyses
documented, etc.
Risk assessment includes evaluating potential
maintenance issues or risks
Updated documentation
Results of completed testing and additional
testing identified

Assess progress identifying and mitigating
failure modes based on engineering and physics
based methods, analysis of manufacturing
variation and workmanship errors, and analysis
of failure modes chargeable to operators and
maintainers
Progress review including analysis outputs
(Failure Modes and Effects Analysis, Data
Collection Analysis and Corrective Action
System/Failure Reporting and Corrective Action
System, Fault Tree Analysis, manufacturing
studies, Human Systems Integration, etc.)

Review of reliability program status

Reliability
including achieved results Assess whether reliability growth is on track to
meet requirements based on operationally Reliability status updated
realistic subsystem and system testing

Reliability Scorecard assessment updated

Critical Design Evaluation of completed design
(scorecard can be found at
Scores indicate a strong reliability program.
Review https://acc.dau.mil/CommunityBrowser.aspx?
id=210483&lang=en-US)
Reliability modeling and analyses & Physics of
Documented modeling and analysis
Failure completed
In-service environmental characterization refined
Updated characterization of loads
and completed

Maintainability assigned at system level with Maintainability requirement development

sub-system and assembly level values updated completed with Mean Time To Repair assigned
as required to reflect lessons learned at the system, sub-system, and assembly levels

Review of maintainability program

Maintainability
status including achieved results
 Review of maintainability program Failure Modes Effects Analysis completed
Maintainability
status including achieved results reflecting current configurations, Physics-of-
Failure analyses updated to address current
Failure mode identification activities on track
configurations, and Data Collection Analysis and
Corrective Action System/Failure Reporting and
Corrective Action System systems functioning

Failure Definition and Scoring Criteria sufficiently

Failure Definition and Scoring Criteria
completed to enable system Developmental
documented
Testing evaluation
Maintenance processes identified and
Maintenance processes for system
documented with resources available to support
Developmental Testing documented
system Developmental Testing
Reliability assessment methods of system
Documented assessment methods
Developmental Testing identified
System testing includes integrated reliability Reliability testing part of Test and Evaluation
testing Master Plan
Evaluation of system readiness to
Reliability Reliability models updated and available to
enter Developmental Testing Updated reliability models
Test Readiness update supportability analysis
Review/Flight Design maturity sufficient to
Test requirements traced to user measures Traceability matrix
Readiness begin testing
Review Assess whether reliability growth is on track to
Reliability status updated and included in
meet requirements based on operationally
availability analysis
realistic subsystem and system testing

Remaining reliability risks identified,

documented, and mitigation plan developed, Reliability risk mitigation plan
assigned, and resourced
System maintainability testing as part of the Maintainability testing part of Test and
Determination of maintenance integrated test plan Evaluation Master Plan
Maintainability measures to assist in availability Statistical documentations of demonstrated
calculations Reliability related maintenance measures
maintainability and Built In Test coverage and
documented
false alarm rate
Test results analyzed using statistical measures
System Reliability
Evaluation of system Developmental
with results extrapolated to operational Test report
Testing results
Verification environment if required
Production representative
Review/Functio system complies with Maintenance Developmental Testing results
nal performance specifications Evaluation of demonstrated evaluated against requirements and models
Developmental Testing test report
Configuration Maintainability
maintainability
Audit Maintainability risks documented and deemed
Risk assessment includes maintainability issues
manageable within cost estimates
Production reliability risks reviewed, identified, Key manufacturing factors affecting component
and mitigated reliability identified
Manufacturing process optimization strategies
Description of optimization efforts
implemented
Production System and manufacturing Pilot manufacturing lines set up and tested
Assurance that production process Documentation of Low Rate Initial Production or
Readiness process maturity sufficient to Reliability
will not induce reliability failures
including validation that adequate yields and
prototype production process and capabilities
Review begin initial production reliability are producible
Data Collection Analysis and Corrective Action
System/Failure Reporting and Corrective Action
Support plans and resources documented
System and Test Analyze And Fix resourced
through production
Failure Definition and Scoring Criteria sufficiently
Failure Definition and Scoring Criteria
completed to enable system Operational Testing
documented
evaluation

Evaluation of system readiness to

Reliability
enter Operational Testing
 Maintenance processes identified and
Maintenance processes for system Operational
documented with resources available to support
Testing documented
system Operational Testing
Reliability assessment methods of system
Documented assessment methods
Operational Testing identified
Evaluation of system readiness to
Reliability System testing includes integrated reliability Reliability testing part of Test and Evaluation
enter Operational Testing
testing Master Plan
Reliability models updated and available to
Updated reliability models
update supportability analysis
Operational System maturity sufficient to Test requirements traced to user measures Traceability matrix
Test Readiness begin Initial Operational Test
Review and Evaluation Remaining reliability risks identified,
documented, and mitigation plan developed, Reliability risk mitigation plan
assigned, and resourced

Maintenance demonstration results evaluated

Maintenance demonstration report and analysis
against requirements and models

Evaluation of achieved and

Maintainability risks documented and deemed
demonstrated maintenance Risk assessment includes maintainability issues
manageable within cost estimates
capabilities

Operational Testing maintenance/logistics

Documented support approach for Operational
processes identified, documented, resourced,
Testing
and staff trained

Physical
Configuration
Audit
Reliability Initial Operational Test and Evaluation Operational Testing report, reliability case, risk
Sufficient reliability demonstrated to performance reviewed versus requirements management
Reliability
enter full rate production
System Initial Operational Test Reliability risks identified, mitigation plans
and Evaluation performance Reliability risk mitigation plan
Full Rate and manufacturing processes
developed, assigned, and resourced
Production maturity sufficient to begin full- Maintenance Operational Testing results
rate production Operational Testing test report
System is maintainable within evaluated against requirements and models
Maintainability
program affordability requirements
Maintainability risks documented and deemed
Risk assessment includes maintainability issues
manageable within cost estimates
Ongoing in-service reliability monitoring
Documented reliability monitoring results
including trend analysis
Reliability Review of in-service reliability
In-Service Status check of measured
Obsolescence plan completed and implemented Obsolescence plan
Review performance in actual use
Ongoing in-service maintenance monitoring
Maintainability Review of in-service maintainability Documented maintenance monitoring results
including trend analysis