Technical Reviews Module

Space Systems Engineering, version 1.0

Space Systems Engineering: Technical Reviews Module

 Module Purpose: Technical Reviews

 To understand the purpose and value of conducting

technical reviews.

 To discuss the timing of technical reviews over the

course of a project’s life cycle.

 To consider the entrance criteria and success criteria

for the standard project technical reviews.

 To understand when a technical review is complete.

Space Systems Engineering: Technical Reviews Module 2

 Definition of Technical Reviews

 Technical reviews are scheduled within a project or program to

communicate an approach, demonstrate an ability to meet
requirements, or establish status.

 Technical reviews often serve as control gates for management

to make a go/no-go decision on a project.
• Control gates involve formal examination of a project’s status in
order to obtain approval to proceed.

 Technical reviews may be held for subsystems too; depending

on their size and complexity. Subsystem reviews precede their
corresponding system review.

 When do technical reviews occur?

• Throughout the entire life cycle of a project, as shown on the
following life cycle chart.

Space Systems Engineering: Technical Reviews Module 3

 Reviews in a NASA Project’s Life Cycle

Space Systems Engineering: Technical Reviews Module 4

 The Value of Technical Reviews

 Technical reviews are key development milestones used to

measure progress, assess project maturity and to infuse lessons
from the past. They…
• Provide confirmation of completed effort and readiness to commit
additional resources for the next phase.
• Encourage and establish project discipline with an integrated project
team perspective.
• Identify risks and review mitigation options.
• Describe plans and priorities for the next phase.
 Technical reviews give everyone (design team, non-advocate
discipline experts, customer and consumer) the opportunity to
agree on the current project baseline (requirements, interfaces,
allocations, margins, schedules, risks, budgets, etc.).
• Functional, resource and performance allocations and assumptions
are described and confirmed.
• System development constraints and interfaces are described and
agreed to by both sides.
• Risks and development problems are identified and mitigation options
discussed.

Space Systems Engineering: Technical Reviews Module 5

 More Values of Technical Reviews

 A significant, and perhaps unexpected, value to the development

team is the preparation for a review. Usually everyone is busy in
their own domain - preparing for a review forces the team to
describe what they have done and understand what others are
doing.

While there is great value in the preparation, there is also unique value
in the execution. One Air Force officer once took Dwight D.
Eisenhower’s famous quote:
“Plans are nothing. Planning is everything.”

too seriously and on the day before it was scheduled cancelled the PDR
of a $100 million space mission. After a minor revolt, the PDR was
reinstated and the team benefited from the preparation and the review.

 Periodic project reviews are held to demonstrate that the appropriate

products, accomplishments and plans have been completed before
proceeding to the next phase.
Appropriate products, accomplishments and plans are based on the
lessons of hundreds of past projects (best practices). NASA, for
example, describes standard entrance and success criteria for all
standard milestone reviews.

Space Systems Engineering: Technical Reviews Module 6

 NASA’s Minimum Set of Technical Reviews

 Mission Concept Review (MCR),

 System Requirements Review (SRR) and/or Mission Definition
Review (MDR),
 Preliminary Design Review (PDR),
 Critical Design Review (CDR),
 System Integration Review (SIR),
 Test Readiness Review (TRR),
 Operational Readiness Review (ORR),
 Flight Readiness Review (FRR),
 Post-Launch Assessment Review (PLAR),
 Critical Event Readiness Review (CERR), and
 Decommissioning Review (DR).

While this may seem like a large number of technical reviews, each
has its own focus and proven value.
In addition, since mission lifetimes are several years, typically
there is 6 months or more between major reviews.
Space Systems Engineering: Technical Reviews Module 7
 The Primary Questions of Each Review
 Mission Concept Review - Does the proposed concept meet the
mission need and objectives?
 System Requirements Review and/or Mission Definition Review - Do
the functional and performance requirements and the selected concept
satisfy the mission?
 Preliminary Design Review - Does the preliminary design meet all the
system requirements within acceptable cost, schedule, and risk?
 Critical Design Review - Is the system design mature enough to
proceed with full-scale fabrication, assembly, integration and test?
 System Integration Review - Are the system, facilities, personnel, plans
and procedures ready for system integration?
 Test Readiness Review - Is the project ready to commence with
verification testing?
 Operational Readiness Review - Are all systems hardware, software,
personnel, and procedures in place to support operations?
 Flight Readiness Review - Is the system ready for launch? Are the
ground facilities and personnel ready to support launch?
 Post-Launch Assessment Review - After launch and deployment, are the
spacecraft systems ready to proceed with routine operations?
 Critical Event Readiness Review - Is the project ready to execute the mission’s
critical activities during flight operation?
 Decommissioning Review - Is the system ready to be terminated?
Space Systems Engineering: Technical Reviews Module 8
 Pre-Phase A Reviews

Mission Concept Review (MCR)

 A validation that the mission has clearly established needs, objectives,
and top-level functional/performance requirements, and that at least
one way of conducting the proposed mission is realistic and attainable
within existing or projected technology and ROM cost.
 An internal Center review(s) of all Pre-Phase A activities and products
should be conducted prior to forwarding the Pre-Phase A report to
Headquarters. Technical, management, resources, and scientific
personnel should conduct the review.
Peer Reviews
 Used informally during Pre-Phase A and Phase A.
 The peer group is composed of individuals selected from outside the
project according to their expertise in the applicable disciplines.
 Throughout Pre-Phase A and Phase A, peer reviews should informally
check the evolving mission concept against objectives, requirements,
and constraints.
 Peer reviews help you take advantage of other engineering experience
from colleagues who have worked on different missions. They can point
out issues they confronted that may be similar for your mission concept.

Space Systems Engineering: Technical Reviews Module 9

 Phase A Reviews

System Requirements Review (SRR)

 The primary focus of the SRR is to verify the realism of the functional
and performance requirements, ensure their congruence with the
mission and system configuration, and ensure the mission objectives
can be satisfied.
 The SRR encompasses all major participants (NASA and contractors),
and is chaired by the Project Manager.
 A product from the SRR is the project system specification that is
formally baselined and placed under configuration management control.

Mission Definition Review (MDR)

 A validation that the mission objectives can be satisfied, the partitioning
of the functionality to each of the systems is adequate, the top-level
performance requirements for each system have been defined, and the
technology required to develop the systems and implement the mission
is attainable.
 The MDR is keyed to the end of Phase A and evaluates the mission
definition, system design, operational concepts, schedule, and cost
estimates.

Space Systems Engineering: Technical Reviews Module 10

 Example Entrance Criteria —
System Requirements Review
1. Successful completion of the Mission Concept Review (MCR) and responses made to all
MCR Requests for Actions (RFAs).
2. A preliminary SRR and/or MDR agenda, success criteria, and charge to the board have
been agreed to by the technical team, project manager, and review chair prior to the SRR
and/or MDR.
3. The following technical products for hardware and software system elements are
available to the cognizant participants prior to the review:
a. System Architecture.
b. System requirements document.
c. System software functionality description.
d. Updated concept of operations.
e. Updated mission requirements, if applicable.
f. Baselined SEMP.
g. Preliminary system requirements allocation to the next lower level system.
h. Updated cost estimate.
i. Technology Development Maturity Assessment Plan.
j. Preferred system solution definition including major trades and options.
k. Updated risk assessment and mitigations.
l. Updated cost and schedule data.
m. Logistics documentation (preliminary maintenance plan, etc.).
n. Preliminary human rating plan, if applicable.
o. Software Development Plan (SDP).
p. System safety and mission assurance plan.
q. Configuration management plan.
r. Project management plan.
s. Initial document tree.
t. Verification and validation approach.
u. Preliminary hazard analysis (PHA).
Space Systems Engineering: Technical Reviews Module 11
 Example Success Criteria —
System Requirements Review
1. The resulting overall concept is reasonable, feasible, complete, responsive
to the mission requirements, and is consistent with system requirements
and available resources (cost, schedule, mass power, etc.).
2. The project utilizes a sound process for the allocation and control of
requirements throughout all levels, and a plan has been defined to
complete the definition activity within schedule constraints.
3. Requirements definition is complete with respect to top level mission and
science requirements, and interfaces with external entities and between
major internal elements have been defined.
4. Requirements allocation and flow down of key driving requirements have
been defined down to subsystems.
5. System and subsystem design approaches and operational concepts exist
and are consistent with the requirements set.
6. The requirements, design approaches, and conceptual design will fulfill the
mission needs within the estimated costs.
7. Preliminary approaches have been determined for how requirements will
be verified and validated down to the subsystem level
8. Major risks have been identified, and viable mitigation strategies have
been defined.
Space Systems Engineering: Technical Reviews Module 12
 Constellation Program Office Schedule of SRRs
You are2007
March Here! April May PBS 5/23
Issue ID, Form Entry 3/26 Provide Issue list w/
5/10
Resolution Plan & Impacts
Rev. Issue Table, POC, & Sched. 3/29
Provide Status on Issue Resolution Develop PBS
(every Mon. @ Cx SE&I 8:30 telecon)
4/6 4/9 4/16 4/23 4/30 Presentation 5/14

CLV SRR (send updates to Cx SE&I COB every other day in May)

Issue ID, Form Entry 3/26 Provide Issue list w/

5/10Resolution Plan & Impacts
Rev. Issue Table, POC, & Sched. 4/6
Provide Status on Issue Resolution Develop PBS
(every Mon. @ Cx SE&I 8:30 telecon)
4/9 4/16 4/23 4/30 Presentation 5/14

CEV SRR (send updates to Cx SE&I COB every other day in May)

Issue ID, Form Entry 4/9 Provide Issue list w/

5/10Resolution Plan & Impacts
Rev. Issue Table, POC, & Sched. 4/13
Develop PBS
Provide Status on Issue Resolution 4/23 4/30 Presentation 5/14
(every Mon. @ Cx SE&I 8:30 telecon) 4/16
MO SRR (send updates to Cx SE&I COB every other day in May)

Provide Issue list w/

5/10
Resolution Plan & Impacts
Issue ID, Form Entry 5/4
Rev. Issue Table, POC, & Sched. 5/5 Develop PBS
5/14 Presentation
GO SRR Provide Status on Issue Resolution
(send updates to Cx SE&I COB every other day in May)

Provide Issue list w/

5/10
Resolution Plan & Impacts
Issue ID, Form Entry 5/8
Rev. Issue Table, POC, & Sched. 5/9
Develop PBS
5/14 Presentation
EVA SRR Provide Status on Issue Resolution
(send updates to Cx SE&I COB every other day in May)
Space Systems Engineering: Technical Reviews Module 13
 Constellation Program Office Status - Closure of SRR

RIDs
Board AI  Total RID Count: 6,283
 Total AI Count: 48 Total  6225 Closed
 9 Closed  58 Open
 39 Open • Majority closed prior to PBS
• 30 are “Past Due” • All RIDs should be closed prior to CxP SDR
• 9 are in work, “Not Due Yet”

TBD/TBR
 Total Count: 2,532 Document Document Title Being
Worked
• Received plans for 2,064 TBRs/TBDs in
CxP 70008 Master Integration and Verification Plan (MIVP) 4
72 of 95 documents (76% of docs.) CxP 70023 CxP Prg Design Spec Natural Environments (DSNE) 9
• Need burn-down plans for 196 CxP 70024 CxP HSIR 1
CxP 70036 Cx Environ Qual & Accept Testing Reqs (CEQATR) 5
TBRs/TBDs in 16 documents (17% of CxP 70050 Vol 2 Electrical Power System Spec Vol 2 1
docs.) CxP 70061 C3I Strategic Plan 1
CxP 70067 Vol 1 CxP Program Human-rating Plan Vol 1 19
• 7 documents (274 TBRs/TBDs) will not CxP 70080 Electromagnetic Environmental Effects (E3) Req'ts 6
be updated/baselined until post CxP CxP 70086 Software Verification and Validiation Plan 5
CxP 70135 Structural Design & Verification Requirements 7
PBS. These TBRs/TBDs will not be Total 58
closed prior to CxP PBS
Space Systems Engineering: Technical Reviews Module 14
 Preliminary Design Review (PDR)

 The PDR is not a single review but a number of reviews starting

with the specific component PDRs, followed by the system-level
review.
 The PDR establishes the “design-to” baseline and ensures that
it meets the program, project , system, subsystem, or specific
component baseline requirements.
 The PDR process should:
• Establish the ability of the selected design approach to meet the
technical requirements (i.e., Verifiability/ Traceability);
• Establish the compatibility of the interface relationships of the
specific end item with other interfacing items;
• Establish producibility of the selected design;
• Establish the operability of the selected design;
• Assess compliance with reliability and system safety requirements;
• Establish the feasibility of the approach;
• Address status, schedule and cost relationships.

Space Systems Engineering: Technical Reviews Module 15

 When is a Review Complete?
Reviews are considered complete when the following is accomplished:
a) Agreement exists for the disposition of all Review Item Discrepancies
(RID) and Request for Actions (RFA).
b) The review board report and minutes are complete and distributed.
c) Agreement exists on a plan to address the issues and concerns in the
review board’s report.
d) Agreement exists on a plan for addressing the actions identified out of
the review.
e) Liens against the review results are closed, or an adequate and timely
plan exists for their closure.
f) Differences of opinion between the project under review and the review
board(s) have been resolved, or a timely plan exists to resolve the
issues.
g) A report is given by the review board chairperson to the appropriate
management and governing program management committees
charged with oversight of the project.
h) Appropriate procedures and controls are instituted to ensure that all
actions from reviews are followed and verified through implementation
to closure.

Space Systems Engineering: Technical Reviews Module 16

 Example Agenda for a Project System Design
Review by a Standing Review Board (SRB)
 Purpose of Review & Charge to SRB SRB Chair
 Project Overview & Status Project Manager
 System Engineering & Status Project SE
• Requirements & V&V plans
• Trade studies
• Technical margins
 WBS-level 2 Design State & Status for each area WBS Managers
• System Design
• Key Requirements
• Trade Studies
• Technology Readiness
• Acquisition Strategy & Long Lead
• Logistics & Facilities
• Challenges & Risks
 Integrated System (e.g., power) State & Status for each area Discipline Leads
 Integration & Test Integration Manager
 Safety & Mission Assurance (S&MA) S&MA Manager
 Human Rating Project HR Rep
 Risk Risk Manager
 Schedule Project Planner
 Cost Cost Manager

Space Systems Engineering: Technical Reviews Module 17

 Pause and Learn Opportunity
Student role-play:
You are the chief systems engineer for a New Frontiers-class
mission to Europa. Your PDR is scheduled in 3 months.
 What do you do?
 What do you ask the development team to do?
 What benefits would you expect from having a PDR?
 How might it waste time?

Or
Read and discuss the Crosslink article: The Role of
Independent Assessments for Mission Readiness
(Crosslink_Independent Review.pdf)

Space Systems Engineering: Technical Reviews Module

 Module Summary: Technical Reviews

 Technical reviews are key development milestones used to

measure progress, assess project maturity and to infuse
lessons from the past. They…
• Provide confirmation of completed effort and readiness to commit
additional resources for the next phase.
• Encourage and establish project discipline with an integrated
project team perspective.
• Identify risks and review mitigation options.
• Describe plans and priorities for the next phase.
 There are 11 reviews in the minimum set of technical reviews
for a NASA robotic mission. These reviews cover the entire
mission life — assessing the concepts and designs early;
readiness for test, flight and operations in mid-life and plans for
disposal at the mission’s end.
 These reviews are held to demonstrate that the appropriate
products, accomplishments and plans have been completed
before proceeding to the next phase.
• Appropriate products, accomplishments and plans are based on the
lessons of hundreds of past projects (best practices).

Space Systems Engineering: Technical Reviews Module 19

 Backup Slides
for Technical Review Module

Space Systems Engineering: Technical Reviews Module

 Major Project Reviews Precede
Each Key Decision Point

FORMULATION IMPLEMENTATION
Pre-A A B C D E F
Concept Concept & Preliminary Final System Operations & Closeout
Project Technology Design & Design &
Assembly,
Sustainment
Studies Test, &
Phases Development Technology Fabrication Launch
Completion

A B C D E F
Key
Decision
Points
Mission Concept Review

Systems Requirements Review

Major Mission/System Definition Review
Reviews Preliminary Design Review
Critical Design Review

Independent Cost Systems Integration Review

Estimates
Operational Readiness Review
Flight Readiness Review
Post Launch Assessment
Review
Decommissioning
Review
Space Systems Engineering: Technical Reviews Module 21
 Purpose of Technical Reviews

A technical review is an evaluation of the project, or element

thereof, by a knowledgeable group for the purposes of:
a) Assessing the status of and progress toward accomplishing the
planned activities.
b) Validating the technical tradeoffs explored and design solutions
proposed.
c) Identifying technical weaknesses or marginal design and
potential problems (risks) and recommending improvements
and corrective actions.
d) Making judgments on the activities’ readiness for the follow-on
events, including additional future evaluation milestones to
improve the likelihood of a successful outcome.
e) Making assessments and recommendations to the project
team, Center, and Agency management.
f) Providing a historical record that can be referenced of decisions
that were made during these formal reviews.
g) Assessing the technical risk status and current risk profile.

Space Systems Engineering: Technical Reviews Module 22

 Need for Management Reviews

The progress between life-cycle phases is marked by key decision

points (KDPs). At each KDP, management examines the
maturity of the technical aspects of the project. For example,
management examines whether the resources (staffing and
funding) are sufficient for the planned technical effort, whether
the technical maturity has evolved, what the technical and non-
technical internal issues and risks are, or whether the
stakeholder expectations have changed. If the technical and
management aspects of the project are satisfactory, including
the implementation of corrective actions, then the project can be
approved to proceed to the next phase.

Space Systems Engineering: Technical Reviews Module 23

 SE&I/ T&V Change Requests submitted to
5/23
Configuration Management Dates 4/3/07
Program Baseline Sync
4/24
4/11 GO SRR Board 4/25
3/27 70022 Vol 3
3/1 MS SRR Board 6 weeks 5/1
4/12 Prior to PBS 5/9
CEV SRR Board EVA SRR Board 5/8
70065, 70000
3/20 4/23 5/7 70130
70070-ANX05 Bk 1 4/13 4/16 5/1
70025 FAD 70070-ANX05 Bk3 70022 Vol 8
3/20 70013 70136 70019
N/A
70135 4/10 4/24 Cx Nomen. Plan
3/23 70087 4/16 70133 TBD
4/30
70012 70085
2/15 70022 Vol 1
4/20 5/25
70073 Vol 3 4/2
70138 70008
70044

Elapsed Time

2/4 2/11 2/18 2/25 3/4 3/11 3/18 3/25 4/1 4/8 4/15 4/22 4/29 5/6 5/13 5/20 5/27 6/3 6/10
2/2 6/17
3/23 3/29
4/25 5/4 5/9 5/15 5/16
70009 70151 4/18 4/18
4/2 4/11 4/24 70061 4/30 70024 5/7 70007 70143 70050 Vol 1,
Anx 1 4/9 4/9 70016 70086
70022 Vol 5 70023 Afford. Plan, 4/24 4/24 70145 70009 5/3 70022 Vol 4 TBD 70050 Vol 2
70064
70070-ANX05 Bk 2 4/19 70036 70080 4/24 4/24 4/26
70078
70141 4/24 70144 70022 Vol 2
70137
70132
Submit CR
CxP Doc # Title to CM Comments Org Doc Owner Change Flag

70000 Constellation Architecure Requirements Document, Revision A 5/9/2007 Per update from PRIMO 2/26/07 PRIMO C. Adamek

70007 DRM & Operations Concept Document 5/9/2007 Submit date reflected in the IMS inputs for 3/30. PRIMO B. Teague
70008 Master Integration & Verification Plan 5/25/2007 Date changed in 3/8 verison of IMS - changed color to yellow T&V R. Cox
70009 System Integrated Analysis Plan 4/30/2007 Added. Per IMS Update of 3/14/07 ATA A. Zuniga

70009 Anx 1 System Integrated Analysis Plan, Annex 1 for IDAC-3 3/23/2007 Submit date reflected in the IMS inputs for 3/30. ATA A. Zuniga
Updated per e-mail on 3/27. Expected to be reflected in the IMS
70012 Constellation Reference Architecture Document for IDAC3 3/23/2007 inputs for 4/6. ATA D. McKissock
70013 Systems Engineering Management Plan 4/13/2007 Confirmed by James A. on 2/26 and Dave C. on 3/8. COS-T J. Afarin
70016 Requirements Engineering Management Plan 4/18/2007 Updated per IMS updates for 3/23 PRIMO M. DiGiuseppe
70022 Vol 1 C3I Interoperability Standards Book 4/30/2007 Updated per IMS updates for 3/23 CSI K. Muery
70022 Vol 2 CSI Spectrum and Channel Plan 4/26/2007 Updated with CSI input rec'd 3/19 CSI C. Sham
70022 Vol 3 CSI Master Link Book 4/25/2007 Updated with CSI input rec'd 3/19 CSI J. Brase
70022 Vol 4 CSI Information Representation Spec 5/7/2007 Updated per IMS updates for 3/23 CSI J. Differding
70022 Vol 5 CSI Data Exchange Protocol Specification 4/2/2007 Updated with CSI input rec'd 3/19 CSI M. Stagnaro
70022 Vol 8 Common Command & Control Requirements 5/7/2007 Updated with CSI input rec'd 3/19 CSI M. Severence
Constellation Program Design Specification for Natural
70023 Environments (DSNE) 4/9/2007 Updated per 3/19 IMS inputs E&C K. Hwang
IMS input on 3/9 shows CR drop 5/4 (but this is prior to all updates in).
70024 Constellation Human Systems Integration Requirements 5/4/2007 Will confirm before updating here. HFSIG K. Ess
FAD through RWG only, no board baseline required. B/L date in
70025 Functional Analysis Document N/A schedule of 6/27 is currently out of sync with PBS need dates PRIMO J. W illiams-Byrd
Constellation Program Environmental Qualifications and
70036 Acceptance Testing Requirements (CEQATR). 4/24/2007 Update from Ed Strong, 2/21 T&V E. Strong
Constellation Program Natural Environment Definition for
70044 Design (NEDD) 4/2/2007 Updated per 3/19 IMS inputs which confirmed date E&C L. Smith
Electrical Power System Specification, Volume 1: Electrical Schedule update received 3/9. Doc will be submitted but not
70050 Vol 1 Power Quality Performance for 28 VDC 5/16/2007 baselined in time for PBS Power R. Scheidegger
Electrical Power System Specification, Volume 2: User Schedule update received 3/9. Doc will be submitted but not
70050 Vol 2 Electrical Power Quality Performance for 28 VDC 5/16/2007 baselined in time for PBS Power R. Scheidegger
Updated per e-mail on 3/27. This will need to be updated in the IMS
70061 C3I Strategic Plan 4/25/2007 inputs for 3/30. (was 3/30) CSI P. Paulsen


70064 Supportability Plan 4/9/2007 Updated per IMS inputs from 3/30 (was 3/23/07) SOA K. Watson Slipped

Blue text indicates changes On plan and will be baselined prior to PBS
Plan confirmed but cannot be baselined prior to PBS (CR
from last update submitted within 2 weeks of PBS) or requires IMS update.

Space Systems Engineering: Technical Reviews Module

No plan confirmed
Submitted to CM 24