Integration Readiness Levels
Jennifer M. Long
Naval Air Systems Command
Patuxent River, MD 20670
301-757-7448
jennifer.m.long@navy.mil
AbstractVirtually all successful systems experience
evolution of their technical baseline over time. The
Technology Readiness Assessment (TRA) framework is
structured to assess the maturity of technology elements via
the established metrics of Technology Readiness Levels
(TRLs), which address the risk associated with developing
and transitioning new technologies in the context of system
development.
The TRL methodology may need to be supplemented when
a program is introducing a new configuration item that
doesnt leverage a new technology into a fielded product
baseline. The cost and schedule risk associated with
developing the configuration item and subsequently
integrating it into the product baseline are often
underestimated at the start of the program. The
consequences of this are usually cost and schedule growth
that may threaten the health and executability of the
overarching program.
Systems engineers and program managers need a metric to
help evaluate the cost and schedule risk of integration
efforts prior to committing resources via contract award.
This paper presents a list of questions that establishes a
qualitative program risk assessment framework. The paper
also introduces the Integration Readiness Level metric,
which can be used to evaluate system integration risk based
on integration characteristics.
TABLE OF CONTENTS
1. INTRODUCTION.1
2. DISCUSSION...1
3. CONCLUSIONS...9
4. REFERENCES.9
5. BIOGRAPHY..9
1. INTRODUCTION
1 introducing new technologies into development programs.
Virtually all successful systems experience evolution of the
technical baseline for a variety of reasons, including
obsolescence, technology upgrades, user requirements
changes, planned incremental or spiral development, etc.
This may involve software upgrades, or introduction of new
hardware subsystems, which typically involve software
1
U. S. Government work not protected by U.S. copyright
2
IEEEAC Paper #1214, Version 4, Updated 2011:01:11
1
unique to the items functionality. These changes are often
introduced to a fielded product baseline via the Engineering
Change Proposal (ECP) process, and are managed as mini
programs embedded within a larger production or
sustainment effort. One of the challenges facing the systems
engineer is understanding the cost and schedule risk
associated with introduction of new subsystems to a mature
technical baseline. Often, there are established cost and
schedule figures allocated for the integration effort, which
may be based on considerations that have nothing to do with
its complexity. Examples of this might include meeting a
deadline when the new capability has to be available, or
fitting within a total budget that is considered reasonable or
affordable within the context of the program. However,
there are many factors that can influence the difficulty of an
integration effort, and generating high-fidelity cost and
schedule estimates prior to starting the effort is challenging.
As a result, it is often difficult to predict whether or not the
planned upgrade is executable within the allocated schedule
and budget.
The purpose of this paper is to present a methodology for
predicting the cost and schedule risk associated with
integrating new configuration items to established product
baselines. An Integration Readiness Level (IRL) metric is
proposed that allows a risk assessment based on integration
characteristics of the configuration item of interest. The
methodology is based on the Technology Readiness Level
(TRL) metric [1] in use in the defense industry to assess the
risk associated with introducing new technologies in the
context of development programs. The TRL metric is
presented for background information. The proposed IRL
metric is then discussed, with an example of how it could be
used to assess integration cost and schedule risk.
2. DISCUSSION
Experience within Department of Defense acquisition
programs has demonstrated the risk associated with
For example, history shows that developing a new aircraft
that incorporates evolutionary technology can result in an
effort that grows well beyond the promised cost and
schedule, and an end item whose unit cost far exceeds what
was originally planned. In some cases there may be
performance gaps due to the new technology not being able
to provide anticipated capabilities. This situation typically
triggers a range of unpleasant consequences for the
acquisition organization. Users may be left with a capability
gap if they need the new system to replace a legacy system
that is obsolete or scheduled for disposal, or to provide a
new capability to meet evolving requirements.
The Technology Readiness Assessment (TRA) framework
[1] was developed to assess the maturity of technology
elements via the established metrics of Technology
Readiness Levels (TRLs). The TRL metric assigns a ranking
to the maturity of the technology element using specific
evaluation criteria; the scale can be used to assess the risk of
developing and transitioning new technologies in the
context of a larger system acquisition. Table 1 contains a
description of the Technology Readiness Levels. DoD
policy requires that technologies are at TRL 6 prior to
entering System Design and Development.
The TRL methodology outlined in Table 1 is designed to
assess the maturity of a new technology, i.e., is it far enough
along in development that it can reasonably be included in a
system? This guidance may need to be supplemented when
a program is integrating a configuration item that
incorporates an established technology. The maturity of the
technology is not in question in that case; the risk
assessment should focus on the maturity of the
configuration item and on the difficulty of integrating it into
the overarching system. If the configuration item is not
ready for integration (i.e., in a prototype state or under
development), then there is some level of risk inherent in
trying to run an integration effort concurrently. If the
configuration item is ready for integration, but was
developed for a different application that is not similar to
the current system, then the integration effort may be
difficult.
Qualitative Risk AssessmentThere are some basic
questions that can be considered to establish a qualitative
level of risk for the integration of configuration items (CIs)
into a technical baseline. The questions are general, and
some may not apply to any given integration effort. And, for
any given integration effort, other questions are possible.
This list is intended to illustrate the basic concept of
qualitative risk assessment.
1. Are the configuration items performance requirements
established prior to starting integration, and does a process
exist to place them under configuration control by the
governing program?
Discussion: Performance requirements should be
documented in a specification, which is under the control of
the acquisition organization, to ensure that the contractor
prices a program that includes the necessary work to
develop and integrate a system that performs as intended.
2
Addition of requirements after contract award, or failure to
maintain configuration control over the governing
requirements document, will result in cost and schedule
growth. The magnitude of this is difficult to predict in
advance.
2. Are the basic development milestones (hardware and
software) for the configuration item understood, and is the
infrastructure (contractual and process) in place to track
them at a sufficient level of fidelity to be linked into the
overarching systems integrated master schedule/integrated
master plan at contract award?
Discussion: The overarching system will have dependencies
with the configuration item under development. If the
development milestones are not linked into the overarching
systems program planning tools and control processes, then
the effect of delays in configuration item development will
not be visible at the system level, resulting in unpleasant
surprises when the configuration item fails to meet its
development and delivery milestones.
3. Does the configuration item contain embedded software?
If so, is that software already developed, or is it pending
hardware maturation to accomplish software development
and integration?
Discussion: If the configuration item contains software that
is a development item in and of itself, the cost and schedule
risks are significant unless substantial amounts of time and
funding are allocated for that element of the program. Risk
may actually increase if software is being extensively
modified from a previous application, rather than developed
from scratch for the new application.
4. If the answer to question 3 is yes, does the
configuration items software have to interact with the
overarching systems hardware/software? If so, is there an
ICD already in work, and what is the maturity of the ICD?
Discussion: Adding a new interface to any system typically
results in an iterative integration process that can be time
consuming. If the ICD is still in flux at contract award,
additional time will be required to mature the interface.
5. If the answers to questions 3 and 4 are yes, is there a
reasonable expectation that the (sub)/contractor will perform
at a CMMI Level 3 for the execution of this effort?
Discussion: Lack of process discipline in software
development, particularly as it relates to interface design,
will result in excessive re-work.
 Technology Readiness Level
1. Basic principles observed and
reported.
2. Technology concept and/or application Invention begins. Once basic principles are observed, practical
formulated.
3. Analytical and experimental critical
function and/or characteristic proof of
concept.
4. Component and/or breadboard
validation in laboratory environment.
5. Component and/or breadboard
validation in relevant environment.
6. System/subsystem model or prototype
demonstration in a relevant environment.
7. System prototype demonstration in an
operational environment.
8. Actual system completed and qualified Technology has been proven to work in its final form and
through test and demonstration.
9. Actual system proven through
successful mission operations.
Table 1: Technology Readiness Levels (Ref. 1)
Description
Lowest level of technology readiness. Scientific research
begins to be translated into applied research and development.
Examples might include paper studies of a technology's basic
properties.
applications can be invented. Applications are speculative and
there may be no proof or detailed analysis to support the
assumptions. Examples are limited to analytic studies.
Active research and development is initiated. This includes
analytical studies and laboratory studies to physically validate
analytical predictions of separate elements of the technology.
Examples include components that are not yet integrated or
representative.
Basic technological components are integrated to establish
that they will work together. This is relatively "low fidelity"
compared to the eventual system. Examples include
integration of "ad hoc" hardware in the laboratory.
Fidelity of breadboard technology increases significantly. The
basic technological components are integrated with
reasonably realistic supporting elements so it can be tested in
a simulated environment. Examples include "high fidelity"
laboratory integration of components.
Representative model or prototype system, which is well
beyond that of TRL 5, is tested in a relevant environment.
Represents a major step up in a technology's demonstrated
readiness. Examples include testing a prototype in a high-
fidelity laboratory environment or in simulated operational
environment.
Prototype near, or at, planned operational system. Represents
a major step up from TRL 6, requiring demonstration of an
actual system prototype in an operational environment such as
an aircraft, vehicle, or space. Examples include testing the
prototype in a test bed aircraft.
under expected conditions. In almost all cases, this TRL
represents the end of true system development. Examples
include developmental test and evaluation of the system in its
intended weapon system to determine if it meets design
specifications.
Actual application of the technology in its final form and
under mission conditions, such as those encountered in
operational test and evaluation. Examples include using the
system under operational mission conditions.
3
6. Is the final form factor of the configuration item
significantly different from existing applications or
prototypes? If so, is miniaturization part of the development
effort?
Discussion: Often, a prototype item will be put forth to
demonstrate that the contractor knows how to do this, and
the development effort is assumed to be low risk. This may
be a dangerous assumption, particularly if the item needs to
be miniaturized for use in the intended application. Thermal
control (if electronics are involved) and tolerance
management become critical in this case.
7. Does the contractor have experience with developing
items that are ruggedized to withstand real world electro-
magnetic and mechanical environments?
Discussion: Depending on the application, real world
environments can be tremendously stressing, and the design
techniques to ensure robustness require specialized
knowledge and experience. A contractor whose main
business is designing electronics that normally sit in a living
room may not have the necessary expertise to design
electronics that are used in military applications.
Qualification test failures and costly re-work late in the
program will occur.
8. If the system is carried on and launched from an aircraft,
does incorporation of the configuration item involve mass
properties or external mold line changes that would
invalidate previous analysis and flight test results?
Discussion: Seemingly minor changes in external mold line
or mass properties can trigger extensive analysis and test
efforts. Specifics of the changes need to be socialized with
certification agencies prior to starting the program, so that
necessary analysis and test can be planned.
9. Is the configuration item under development slated for
multiple programs? If so, is there a joint set of
requirements that is overly-conservative based on
skylining requirements from the individual programs?
Discussion: Skylining requirements is easy, designing items
to work in environments that could never realistically occur
may not be. It takes some effort up front to analyze the
individual requirements sets and break them out by
condition, but it may help to avoid design difficulties and
qualification test failures downstream.
4
10. Does the incorporation of the configuration item into the
system trigger any new regulatory processes involving
certification agencies (for example, the FAA)?
Discussion: Certification agencies tend to have a high
demand for their services, with a finite amount of manpower
to accomplish the work. This can pose a schedule risk if not
properly accounted for in the planning. Assuming that
certification activities can happen more quickly than
normal is usually risky.
The preceding discussion seems fairly intuitive. The
questions are all good ones to ask when starting an
integration effort, and careful consideration of the answers
establishes a risk framework for program planning.
However, when integration efforts are being put on contract,
systems engineers may be asked to assess the realism of
the schedule. Contractors are often over-aggressive in their
proposed price and schedule in a highly competitive
environment. Program managers generally want contracted
efforts to be structured as moderate risk to keep the team
working hard to meet the promised cost and schedule. As a
result, there is generally pressure to make the program fit
within pre-established cost and schedule boundaries.
Integration Risk AssessmentThe systems engineer needs
to be able to provide a realistic estimate of the cost and
schedule required to complete the effort, and given that the
answer may not be what the contractor and program
manager want to hear, a qualitative assessment will
probably not be sufficiently convincing. Quantitative criteria
based on metrics are needed to substantiate the argument.
An Integration Readiness Level (IRL) metric is proposed
that relates the integration characteristics of configuration
items to integration risk levels. The metric is structured
along the same lines as the TRL metric, but is intended to
allow assessment of the risk associated with software and
hardware development of configuration items (CIs), rather
than technologies. The discussion herein focuses on cost
and schedule risk. Systems engineers need to be concerned
about performance risk as well, but in most integration
efforts performance can be achieved given enough time and
money, since the basic technologies are well-established. In
that case, the underlying risk actually reaches back to the
resources available to conduct the integration effort.
 Risk Level Integration Readiness Level
High 1. Configuration item (CI) is a
software-intensive system with no
previous baseline to build on. The
interface with the host system is not
defined.
High 2. CI is a software-intensive system
with some re-use possible, and
interface requirements are not
defined.
High 3. CI is a software-intensive system
with a high degree of re-use possible,
but interface requirements are
incomplete.
High 4. CI is a software-intensive system
with a high degree of re-use possible,
and interface requirements are
defined.
High 5. CI software is mature, but needs to
be ported to new hardware
architecture.
High 6. CI software is mature, but needs to
work with modified hardware
architecture.
Med 7. CI software is mature, but changes
are needed for the modified hardware
architecture.
Med 8. CI software is mature and
compatible with the modified
hardware architecture.
Low 9. CI software and hardware work
as is in a very similar system.
Low 10. CI software is embedded, and the
system is fielded.
Table 2: Software IRL Descriptions
Integration Characteristics
The CI is a software-intensive system, no re-use
of code is planned, the supplier has little or no
experience with the current application, and
interfaces with the host system are not defined.
The CI is a software-intensive system, some re-
use of code is planned, the supplier has
experience with a similar application, and
interfaces with the host system are not defined.
The CI is a software-intensive system, a high
degree of re-use of code is planned, the supplier
has experience with a similar application, and
interface documentation is in work.
The CI is a software-intensive system, a high
degree of re-use of code is planned, the supplier
has experience with a similar application, and
interface documentation is baselined.
The CI is being updated to comply with new
hardware architecture, but the basic functionality
of the software is unchanged. A high degree of
integration testing will be required, and the
interface with the host system needs to be re-
designed
The CI is being updated to comply with modified
hardware architecture, and a significant degree of
integration testing will be required. The interface
with the host system needs significant changes.
The CI is being updated to slightly modified
hardware architecture, and a moderate degree of
integration testing will be required. The interface
with the host system needs minor changes.
The CI is being updated to slightly modified
hardware architecture, and only minor integration
testing will be required. The interface with the
host system will work as is.
The CI software does not require any changes,
and the interface with the host system will work
as is.
The CI contains only embedded software that is
already integrated. There is no software interface
with the host system.
5
 Risk Level Integration Readiness Level
High 1. Supplier has access to the
necessary technology.
High 2. Supplier has some experience with
the technology for different
applications.
High 3. Supplier has developed similar
items for different applications.
High 4. Supplier has developed a similar
item for a related application.
Med 5. Supplier has developed a very
similar item for a closely related
application.
Med 6. Supplier has developed a very
similar item for a very similar
application.
Med 7. Supplier has developed a
representative prototype.
Low 8. Supplier has developed a
representative prototype with some
testing.
Low 9. Supplier is using the CI in a
similar system.
Low 10. Supplier is providing the CI to a
fielded system.
Table 3: Hardware IRL Descriptions
Integration Characteristics
The CI supplier owns or has ready access to the
necessary technology, but no experience with
building or testing items.
The CI supplier has developed items that are not
similar to the current CI, for programs with
different applications.
The CI supplier has developed similar CIs for
programs with different applications.
The CI supplier has developed a similar item that
has been used in a program with a related
application, but major changes are needed.
(Example: all circuit cards need to be re-spun).
The CI supplier has developed a very similar item
that has been used in a program with a closely
related application, such that only moderate
changes are needed. (Example: multiple
components, other than CCAs, need to be
changed)
The CI supplier has developed a very similar item
that has been used in a program with a very
similar application, and only minor changes are
needed. (Example: only a few components need
to be changed, mostly for obsolescence rather
than performance)
The CI supplier has already developed a
prototype of the CI, but it hasnt undergone any
electromagnetic or environmental testing.
The CI supplier has already developed a
prototype of the CI, and it has passed testing to
environments that are expected to cause the
highest performance risk.
The CI supplier has already integrated the item
into another system with similar requirements,
and that system is in test with some positive
results.
The item is integrated into a mature system with
similar requirements.
6
The software IRLs merit some discussion. The risk level
associated with software does not drop to low until a
significant degree of maturity is present in the existing item,
and the degree of compatibility with the host application is
well established. Stated another way, any new software
development, software re-work, or new software interface
involves some degree of risk for the program. Software is
the area that is most consistently underestimated from a
risk/difficulty perspective at program inception. One reason
for this is that its difficult to evaluate both the quality of
existing software and the capability of the personnel
assigned to the development effort to perform the necessary
work. Hardware architecture changes are also tricky; porting
code to a new architecture can be extremely difficult,
particularly with some of the older languages like Ada that
are still being used in some sectors. If the electronics
architecture is being re-worked, software engineers should
be involved in the design process so that the updated
architecture is feasible given the framework of the existing
software. If the hardware and software dont readily fit
together, the resulting test-fix-test iteration will be
prolonged and difficult.
Hardware IRLs are somewhat more straightforward. Since
you can see and touch hardware, its easier to assess how
close it is to the desired end state. Items that only involve
hardware, i.e., no embedded software, may encounter
development difficulties, but as long as the established TRL
guidance for technology readiness is observed, the
difficulties should be manageable with easily quantified
levels of risk. This is why a distinction is made in Table 3
between efforts involving changes to circuit cards (high
risk), and changes to other types of components (medium
risk). Circuit cards are a more complex type of hardware
that usually interact with software, and hence carry a higher
level of risk than purely mechanical or electrical devices. In
general, hardware is more predictable, and better
understood, than software. Hardware obeys the laws of
physics, and there are only so many ways that it can fail.
The engineering profession has a long history of solving
these types of problems. However, the readiness for
integration should be assessed to ensure the necessary
cost/schedule resources are allocated for a successful effort.
For planning purposes, the software and hardware risk
levels should be evaluated at contract award to assess cost
and schedule risk for the program. If both software and
hardware fall into the high risk regions, it may not be
feasible to introduce the configuration item unless sufficient
schedule margin and funding resources are available to
accommodate a prolonged development, integration and test
effort. A combination of high and medium would carry a
similar risk, with the risk dropping somewhat for a medium-
medium combination. If both hardware and software fall
into the low risk regions, the configuration item is probably
admissible with low-to-moderate schedule and cost risk.
The various permutations of hardware and software risk
7
levels can be assessed in the context of the specific program
as the first step towards quantifying overall program risk.
Case StudyTo illustrate how IRLs could be used, lets
consider a program that is integrating a new piece of
electronics that adds a new functionality to the system. The
system-level requirements for the new functionality are
documented in a performance specification. The new box
is based on an earlier version thats already in use in a
related, fielded application, but it needs to be miniaturized
to fit into the space available in the current system. The
legacy box is being advertised as a prototype for the new
item, even though it bears little resemblance to the
configuration that will be needed for the current system. The
prototype box has functional software for its current
application, but the software needs to be re-written to work
with the miniaturized version of the hardware. At contract
award, box-level requirements are documented in the item
specification, which is only in draft form, and the interface
control document does not exist. The system specification is
also in draft form.
In this example, there are hardware and software
development efforts associated with maturation of the box
to final form, and modification of the system hardware and
software to host the box, with a follow-on integration and
test effort to incorporate the box and verify the new
functionality is working per the system specification. The
box in this example is essentially the configuration item,
or CI, in the IRL tables. Based on the description, the
software effort would be assigned an IRL level of 2 in Table
2. There was not a lot of specific information provided
about the status of the hardware, but based on the
description its probably at an IRL of 4 in Table 3. With
both software and hardware IRLs in the high risk region, the
effort cannot be expected to go smoothly, either from an
item development perspective or a follow-on integration
perspective. Based on an assumed cost/schedule profile, the
impacts will be discussed.
This is a very typical example of the type of integration
effort that is often underestimated from a complexity
perspective. All thats being proposed is to integrate a new
subsystem to a mature system, and the box supplier already
has a similar item working in a different fielded system. The
box supplier is persuasive in their arguments that the
modification of the box for the new application is low risk,
and integrating it into the system is assumed to be moderate
risk. Based on this assessment, the development/integration
effort is put on contract for 29 months, at a price of $92M.
One of the assumptions embedded in this schedule is that
fully-functional engineering models of the box will be
delivered to the system integrator 6 months after contract
award. The box supplier contractually commits to a delivery
schedule that meets this timeline.
The program plays out very differently in reality. Box
hardware and software are not as applicable to the new
configuration as was initially assumed, and a significant
development effort is required to produce boxes for
integration. The first boxes are delivered to the system
integrator 16 months after contract award (10 months
behind the baseline), and they are early prototypes with
partial software functionality. So, in addition to cost and
schedule growth at the box supplier due to the unplanned
difficulty of the development effort, there is cost and
schedule growth at the system integrator due to not
receiving boxes when they expected them. Since the ICD
was not in place at contract award, it is written in
conjunction with the development effort, and the system
integrator doesnt have a stable baseline to design and
implement their side of the interface. So, when the first
boxes are delivered and connected to the system, even the
minimal functionality available in box software doesnt
work as intended. The system integrator spends
considerable time and money getting the system working to
some degree to support integration testing.
Meanwhile, at the box supplier, system design is continuing
to evolve, as is the host interface. As newer versions of the
box come online they are delivered to the system integrator,
who continues to iteratively refine the interface and mature
system-level functionality. An additional 9 months are
consumed before a box that is suitable for system-level test
in real world environments is delivered. The subsequent
box-to-system integration iteration consumes another ten
months, at the end of which period the first end-to-end
system test is completed. By this time, production
representative boxes are being delivered to the system
integrator, and another iteration is required to integrate the
final configuration. System-level development,
qualification and regression testing takes another 27 months
to complete, and the program finally finishes over 3 years
behind schedule at roughly double the original contract
price.
This example probably sounds extreme, but its based on a
real integration effort that played out as described above. In
the real situation, integration of the box was not the only
change made to the system, so not all of the cost and
schedule growth are necessarily attributed to difficulties
with box development and integration. However, its a fact
that box development and integration turned out to be a
major effort that drove total program cost and schedule far
beyond what was originally planned, to the extent that the
program had to be re-baselined. Obtaining the necessary
funding to complete the effort was extremely difficult, and
depleted resources originally planned for other projects.
Only the high priority (at the resource sponsor level) placed
on the capability being introduced saved the program from
termination. If the program had been terminated, a
significant near-term capability gap for the user would have
resulted. Worthy of note is that the boxes had also been put
on contract to be delivered to another program with a
different system integrator, and that program was unable to
absorb the cost and schedule growth of the box development
effort. That program had less support at the resource
8
sponsor level, and it was terminated after Milestone B. It
should be noted that in the context of the other program, the
technology readiness of the box was evaluated using the
TRL methodology as part of a formal Technology
Readiness Assessment (TRA), and was not assessed to be a
Critical Technology Element since the underlying
technology was well-established. This is an example of how
the IRL methodology could be used to supplement the TRL
methodology for integration applications, since it assesses
the maturity and suitability of the item rather than the
technology.
ApplicationsHow could the list of questions and IRLs
discussed in this paper have prevented these problems?
Questions 1-6, 9 and 10 all have relevance to this case
study. The box performance requirements and ICD were not
fully established at contract award, nor were the system
performance requirements finalized. The prototype box put
forth initially to demonstrate the low-risk nature of the effort
was nothing at all like the final version of the box; major
hardware and software development efforts were required.
The box was intended for use by more than one program,
and the skylined requirements caused some difficulties
with design and qualification, leading to further
requirements iterations. Finally, approvals from external
agencies were required, however, box development and
integration were so time consuming that obtaining these
approvals did not prove to be a major schedule driver. Its
worth mentioning that considerable touch labor was
dedicated to this aspect of the program to ensure the positive
outcome in that regard. This occurred because the risks
associated with external approvals were well-known and
documented in the risk management database at the start of
the program, triggering the dedication of program resources
to the mitigation activities.
Concerning the IRLs, it was stated at the beginning of the
discussion that software would have been assessed at IRL 2
and hardware at IRL 4. In the tables, both of these would
fall into the high risk area, which is an indicator that the
effort probably has a significant degree of complexity. This
assessment alone does not indicate how long the effort will
take, or how much it will cost, but it can help to evaluate the
realism of any estimates that are developed prior to
committing to contract award.
In the example discussed herein, if the IRL tables existed
and had been used by program management/systems
engineering, the cost and schedule proposed for the effort
might not have been accepted. Realistic estimates of the cost
and schedule might have been considered unaffordable at
the time, but perhaps would have sparked more discussion
of capability and requirements. This might have helped to
mature the concept prior to starting development, and
allowed resources to be applied to fleshing out and
documenting a more finalized set of requirements at both
the system and box levels. Also, a better understanding of
the resources required for this effort might have prevented
the termination of the other program that was supposed to
integrate the box; perhaps an alternate material or other
solution could have been found to meet that programs
requirements within their allotted cost and schedule.
Clearly, a thorough and realistic risk assessment up front is
critical to overall program success.
3. CONCLUSIONS
There are numerous examples in the defense industry of
programs that have experienced major cost and schedule
growth for efforts that were assumed to be relatively
straightforward upgrades to fielded product baselines. This
situation occurs because the programs are structured to fit
within available resources and desired capability delivery
dates, without careful consideration of the integration
characteristics discussed herein.
The Integration Readiness Level metric is proposed to
provide systems engineers with a tool to evaluate cost and
schedule risk based on characteristics of the configuration
item being considered for integration. Obviously, there is no
way to attach cost and schedule estimates to the various
Integration Readiness Levels, since those figures will vary
depending on the specifics of the proposed effort. The IRLs
simply provide a means to translate characteristics of the
integration item into risk levels for system integration. The
systems engineer will need to draw on experience and
judgment to arrive at meaningful conclusions concerning
cost and schedule estimates for the integration effort. The
intent is to provide a framework for estimating risk prior to
contract award, so that the program can be set up with an
executable baseline right from the start. It may turn out that
the cost and schedule resources needed to construct an
executable baseline are not available, and its good to know
that as well. Perhaps the new system isnt the only solution,
or the best solution given available resources, for the needed
capability. Its up to the systems engineer to provide those
recommendations to program management to facilitate the
decision process.
The ultimate goal is to dress the program for success by
putting an executable baseline on contract at the start of the
effort. This facilitates an efficient use of available dollars,
and helps to prevent capability gaps within the user
community. As with any tool, it needs to be used with
judgment based on experience to be effective.
The discussion herein is based entirely on experience with
DoD development programs, and as such the list of
questions and integration characteristics is tailored to those
types of engineering applications. It seems likely that this
methodology could be adapted to virtually any industry that
relies on engineered products, however, the list of questions
and specifics of the IRL table would probably need to be
adjusted to account for differences in the type of end
product being developed.
Future work in this area could proceed in various ways.
First, the IRL tables need refinement even for DoD
applications; they were developed by the author based
solely on experience with a few specific efforts, and inputs
from other systems engineers and program managers would
be beneficial to increase the robustness of the metric. The
current tables may have more levels than are needed; it
seems that similar integration characteristics could be
consolidated to increase the separation between the levels
and simplify the descriptions. Its emphasized that the IRL
methodology is not intended to be a direct read-across from
TRLs, and the levels are not a one-to-one mapping of the
rationale expressed in the TRL scale. Second, it may be of
interest to apply this methodology to other technology
sectors. Referring again to the TRL methodology, it is used
by NASA as well as the DoD, and although the levels and
descriptions are similar for the two industries, they are not
identical. If the IRL methodology ever becomes well-
established and agreed to for DoD, it could be extended to
use in other industries.
4. REFERENCES
[1] Defense Acquisition Guidebook, Defense Acquisition
University, www.dag.dau.mil
5. BIOGRAPHY
Jennifer Long is currently
employed by the Department of Defense as a systems
engineer in the field of weapon systems acquisition. She has
20 years of experience in the defense industry in various
technical areas, including structural engineering for
aircraft and spacecraft, military aircraft flight test, and
weapons system acquisition. In 2005 she received the David
Packard Award for Acquisition Excellence. Jennifer holds a
doctoral degree in Civil Engineering, and she is DAU Level
III certified in the Acquisition Professional Fields of
Systems Engineering, Test and Evaluation and Program
Management.