0003 - Managing Experimentation in The Design of New Products

Managing Experimentation in the Design of New Products
Stefan H. Thomke
Technology and Operations Management, Harvard Business School, Soldiers Field Road, Boston, Massachusetts 02163
xperimentation, a form of problem-solving, is a fundamental innovation activity and accounts for a signicant part of total innovation cost and time. In many elds, the economics of experimentation are being radically affected by the use of new and greatly improved versions of methods such as computer simulation, mass screening, and rapid prototyping. This paper shows that a given experiment (and the related trial and error learning) can be conducted in different modes (e.g., computer simulation and rapid prototyping) and that users will nd it economical to optimize the switching between these modes as to reduce total product development cost and time. The ndings are conrmed by a large-scale empirical study of the experimentation process in the design of integrated circuits containing either (1) electrically programmable logic devices (EPLDs); or (2) application-specic integrated circuits (ASICs). In comparing their different experimentation strategies for analogous design projects, I found that the former (EPLD) an approach that utilizes many prototype iterationsoutperformed the latter (ASIC) by factor of 2.2 (in person-months) and over 43 percent of that difference can be attributed to differences in experimentation strategies. The implications for managerial practice and theory are discussed and suggestions for further research undertakings are provided. ( Experimentation Economics ; Product Design and Development ; Computer Simulation ; Prototyping ; Mode Switching ; Iterative Learning )
1. Introduction
Experimentation, a form of problem-solving, is a fundamental innovation process activity and accounts for a signicant part of total innovation cost and time. Research has shown that it consists of trial and error, directed by insight as to the direction in which a solution might lie (Baron 1988, Marples 1961, Allen 1966). Studies of problem-solving in the specic arena of product and process development show iterative trial and error (or, more precisely, trial, failure, learning, correction and retrial) as a signicant feature of design (Newell and Simon 1972, Simon 1969, Smith and Eppinger 1997, Wheelwright and Clark 1992). Experimentation has also been identied as fundamental to the learning process in areas such as technology integration and manufacturing (Bohn 1987, 1995; Adler 1990; Adler and Clark 1991; Pisano 1996; Iansiti 1997).
0025-1909/98/4406/0743$05.00
Copyright 1998, Institute for Operations Research and the Management Sciences
This paper will focus on the area of product design and demonstrate how the economics of experimentation is being radically affected by the use of new and greatly improved versions of methods such as computer simulation and rapid prototyping. I will show that a given experiment (and the related trial and error learning) can be conducted in different modes (e.g., computer simulation and rapid prototyping) and that manager and designers will nd it economical to determine the optimal switching point between these modes as to reduce total development cost and time. My empirical ndings also suggest that such an experimentation strategy can sometimes benet from getting it [the prototype] wrong the rst time; a strategy that contradicts prior work in product development which often prescribes getting it right the rst time as a recipe for improved development performance. This is conrmed by a large-scale empirical study of the
MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998
743
3b2a 0001 Mp 743 Tuesday Jun 16 01:31 PM Man Sci (June) 0001
THOMKE Managing Experimentation in Design of New Products
experimentation process in the design of integrated circuits containing either (1) electrically-programmable logic devices (EPLDs), or (2) application-specic integrated circuits (ASICs). In comparing different experimentation strategies for a large number of analogous design projects, it was found that the former (EPLD)an approach that utilizes many prototype iterationsoutperformed the latter (ASIC) by factor of 2.2 (in person-months) and over 43 percent of that difference can be attributed to differences in experimentation strategies. These ndings are important to managers in many other elds that involve learning by switching between different experimentation modes: software development, chemical process development, and pharmaceutical drug developmentjust to name a few. Since optimal mode switching strategies can result in signicant increases in development performance, the paper will aid managers in understanding the tradeoffs in the use of different experimentation modes and in managing the development process such that these trade-offs are effectively utilized. The paper begins (2) by discussing experimentation as an economic process that is fundamental to innovation and illustrates by example how computer simulation and rapid prototyping are radically affecting experimentation economics. Next, I describe how diminishing returns of iterative experimentation in different modes can result in an optimal mode switching point (3). I then describe study methods and present my empirical ndings (4). Finally, I discuss the implications for managerial practice and theory and suggest further research undertakings (5).
(4) Analyze: One analyzes the result. If the results of a rst iteration are satisfactory, one stops. However, experimentation is usually a matter of repeated trial and error. That is, if analysis shows that the quality of a design can be improved cost-effectively, one modies ones design on the basis of what one has learned and iterates or tries again. As one has the opportunity to improve the design at the end of each iteration, a complete design may involve many iterations until the marginal cost to improve will exceed the marginal benet from the improvement. For example, one might (1) conceive of and design a new, more rapidly deploying airbag for a car; (2) build a prototype of key elements of that airbag as well as any special apparatus needed to test its speed of deployment; (3) run the experiment to determine actual deployment speed; and (4) analyze the result. (If the results of a rst deployment are satisfactory, one stops. However, if analysis shows that the results of the deployment test are not satisfactory, one modies the airbag design on the basis of what one has learned from the test and the analysis and iterates again.)
2. The Economics of Experimentation

The execution of an experiment can be seen as involving a four-step iterative cycle (Figure 1) 1: (1) Design: One conceives of or designs an experiment. (2) Build: One builds the (physical or virtual) apparatus needed to conduct that experiment. (3) Run: One runs the experiment.
1
Other researchers have used related frameworks to study problemsolving in design. Simon (1969) used generator-test cycles and Wheelwright and Clark (1992; chapters 9 and 10) used design-buildtest cycles.
2.1. The Efciency of Experimentation As an aid to study the effectiveness of experimentation modes and strategies, I coin the term experimentation efciency and dene it as the economic value of information learned during an experimental cycle, divided by the cost of conducting the cycle. When an experiment is costly (inexpensive) and the incremental value of information learned is small (large), I dene the experimental efciency to be low (high). Note that in the denition, experimentation efciency involves not only attributes of the experimental technique itself, but attributes and choices made by the experimenter. For example, if a particular designer is ineffective or less able to analyze an experiment (e.g., for lack of experience) and therefore benets less from new information, the experimental efciency will be lower than for someone more experienced ( ceteris paribus ). Also note that experimentation efciency is not a static measure, as it can change as a function of time and the development path chosen. Finally, I would like to point out that economic cost and benet also includes the opportunity cost of time (time-to-market)a variable that has become increasingly important in many industries as it
744
Figure 1
The Framework Views Experimentation as an Iterative Four-Step Learning Cycle
(Changes in Exogenous Information are Due to Unstable User Needs, Rapid Technological Advances, etc.)
can have a very signicant impact on rm prot (Reinertsen 1983, Smith and Reinertsen 1991, Wheelwright and Clark 1992). The following subsections discuss in greater detail the cost and benets that one typically nds in the experimentation process.
2.1.1. Experimentation Cost. The costs of conducting an experimental cycle typically involve the cost and time of using equipment, material, facilities, and engineering resources. These cost can be as high as millions of dollars, such as a prototype of a new car used in destructive crash testing, or it can be as low as a few dollars, such as a chemical compound used in pharmaceutical drug development and made with the aid of combinatorial chemistry (Thomke et al. 1998). I will discuss some of these costs by reference to the four-step experimental cycle that was dened earlier. The cost of building (step 2) an experimentation model depends highly on the available technology and the degree of accuracy that the underlying model is intended to have. For example, modern computer-aided design (CAD) tools can sometimes interface to com-
puter software that converts a design directly into a simulation model. In such cases, the cost of building a model is relatively low as it consists mainly of the investment in conversion tools (xed cost) and the time required to operate them (variable cost). Furthermore, experimentation models can have varying degrees of delity with respect to reality (Bohn 1987, Wall et al. 1991). The value of using incomplete models in experimentation is both to reduce investments in aspects of the real that are irrelevant for the experiment, and to control out some aspects of the real that would affect the experiment in order to simplify the analysis of the test results (step 4). Thus, a model of an airplane used in wind tunnel experiments has no internal design detailsthese are both costly to model and (mostly) irrelevant to the outcome of wind tunnel tests. Sometimes a model to be built is incomplete because one cannot economically incorporate all relevant aspects of the real or does not know them. The incompleteness of a model can lead to unexpected design errors when a given model being used in testing is replaced by a different (and more accurate) model or by the real design in the real environment for the rst time.
745
The cost of analyzing (step 4) the results from step (3) (run) depends to a signicant degree on access to test-related information and the availability of tools that aid in the problem-solving process. For example, consider the discovery of an error during prototype testing and the series of following diagnostic steps to identify the error cause(s). Sometimes a designer has a thorough understanding of a tested prototype and nds the error cause very quickly. Very often, though, subtle errors make the analysis very difcult, especially in cases of great complexity and poor knowledge of causal relationships between system inputs and outputs. As a result, designers have to rely on diagnostic tools and problem-solving methods to aid in their analysis of error symptoms. A very effective analysis tool is the use of computer simulation, since it gives a designer quick access to virtually any information within the realm of the underlying simulation model. In contrast, an analysis of data from prototype testing is more difcult since access to error-related information is typically limited. For example, consider that a real car crash happens very quicklyso quickly that the designers ability to observe details is typically impaired, even given high-speed cameras, well-instrumented cars, and crash dummies. In contrast, one can instruct a computer to enact a virtual car crash as slowly as one likes, can zoom in on any structural element of the car that is of interest, and can observe the forces acting on it and its response to those forces during a crash.
2.1.2. Experimentation Benet. During product development, designers are initially given a set of input and output specications that are used to guide the design of a solution. Often these are not optimal solutions; instead they are satisfactory with respect to a given set of requirements and constraints (Simon 1969). Once a candidate solution is developed, steps (2), (3), and (4) (build, run, and analyze) are used to evaluate that solution and generate information that identies opportunities to improve it. The economic value of such information is the experimentation benet that one derives as a result of conducting experimental cycles. As an example of iterative design improvement, consider the detection, analysis, and removal (elimination)
of design errors.2 An error is a problem in the external operation of a design (operational failure) due to an internal design error. Thus, error elimination consists of nding such operational failures (detection), determine the internal design error that causes the failure (analysis), and modify the design as to remove the internal error (removal). The process of error elimination is fundamental to the design of new products and research on product development has shown that it accounts for a signicant part of total design cost. (Boehm 1981 studied the effort distribution of several software design projects and found that about 40 percent of total design effort is devoted to the detection and elimination of errors.) Not identifying design errors and its consequences can sometimes have catastrophic outcomes, as demonstrated by the Challenger disaster in 1986 (Hauptman and Iwaki 1991), or can jeopardize the commercial success of an otherwise well-designed product such as the well-publicized problems with the Intel Pentium processor (Uzumeri and Snyder 1996). Thus the economic value of detecting an error can be great but is usually difcult to determine prior to actually detecting the error. As very complex designs are usually not completely error-free when they reach the time of release, designers sometimes resort to heuristics that try to estimate the economic value of nding and removing an error and compare it the against the cost of continuing experimentation. For example, Cusumano and Selby (1995) described that software designers at Microsoft track error categories which approximate economic severity (categories 14). Errors of severity 1 cause the
Another example of improvement is the adjustment of design parameters as to optimize design performance (design optimization). Design optimization consists of systematic and planned changes in design parameters as to increase design performance, sometimes with the aid of the statistical design of experiments (Box et al. 1978, Taguchi 1987, Phadke 1989, Montgomery 1991). Error-free designs that operate without operational failure can often be improved with parameter design which is normally performed after design-errors are eliminated, or during later experimental cycles. (I should note, however, that design optimization and error-elimination are sometimes interdependent. For example, if one uses experimentation to maximize design robustness, some errors that are the result of oversensitivity to uncontrollable noise levels may actually be prevented. Thus we nd that a design approach that focuses on robustness early is likely to reduce the number of iterations necessary for error elimination later.)
746
product to halt and therefore all such errors need to be found (very high economic value of information), whereas errors of severity 4 are minor (e.g., cosmetic), and their elimination is of lower economic signicance.
2.2. Experimentation Modes Experimentation can be conducted in different modes , and the absolute and relative efciencies of these modes play an important role in design performance and strategy. As experimentation modes are often driven by technological innovations, it is particularly important to managers to track new developments in this area. Recently, new experimentation modes have been emerging which are radically affecting the absolute and relative efciencies of experimentation and, as a result, overall design performance and strategies. I will illustrate this by reference to two such modes: computer simulation and rapid prototyping. Computer simulation is used as a substitute for real experimentation in elds ranging from the design of drugs (e.g., rational drug design), to the design of mechanical products (e.g., nite element analysis), to the design of electronic products (e.g., computer simulation of digital circuitry), to the analysis of global warming (e.g., climate modeling). An experimenter typically uses simulation in steps (3) and (4) of an experimental cyclerunning an experiment and analyzing the result. The ability to usefully substitute a simulation for a real experiment requires, of course, a simulation model that is accurate from the point of view of a given experimentation purpose. Simulation can have both advantages and disadvantages relative to testing a physical prototype. As an illustration, consider a real explosion in a prototype cylinder of a gasoline-powered car engine that may take milliseconds, while a detailed and less accurate simulation of that same explosion might take minutes or hours on a powerful computer. On the other hand, the analysis of an experiment carried out in simulation is typically much richer. For example, one can collect data from the combustion in a car engine cylinder at only a few points via instrument probes. But one can obtain information from a simulated explosion on variables such as gas temperature and shock wave propagation at any location and at any point in the evolution of the explosion. This allows experimenters to get much more and much better data
per experimental run. Thus subsequent experiments can be designed to be more efcient. Rapid prototyping is used by developers to quickly generate an inexpensive, easy to modify (and often physical) prototype that can be tested against the actual use environment and allows real experimentation. Rapid prototyping techniques can be found in areas ranging from mechanical designs (e.g., stereolithography, three-dimensional printing) to the design of integrated circuits (e.g., electrically programmable logic devices (EPLDs)), to the design of software (e.g., emulation of user-interfaces) (Griesser 1992, Jones 1992, Sachs et al. 1992, Villasenor and Mangione-Smith 1997, Boehm et al. 1984, von Hippel 1994a). Rapid prototyping is often an inexpensive and fast way to achieve step (2) (build) in an experimental cycle while preserving the advantages of real experimentationhigher degrees of accuracy and experimentation speed (step 3) with respect to a given experiment. The utilization of such rapid prototyping techniques has resulted in signicant improvements in development time and cost. With increased competition and shorter product life cycles, accelerated time-to-market and lower development cost have become increasingly important to product development success (Clark and Fujimoto 1991, Smith and Reinertsen 1991, Wheelwright and Clark 1992, Clausing 1993), and rapid prototyping aids in achieving such improved performance.
3. Optimal Mode Switching as an Economic Experimentation Strategy

If new experimentation modes are radically affecting the absolute and relative efciencies of experimentation, then how can managers and designers change their experimentation strategies such that overall performance improves? The following subsections will describe a model of how these modes drive experimentation efciency and introduce the concept of an optimal mode switching point (OSP). The following discussion of technological innovations and their impact on experimentation strategies serves as a foundation for the empirical study (4) and the discussion at the end of the paper.
747
3.1. The Decreasing Efciency of Experimentation As a design progresses and experimental cycles are repeated within a given mode, I propose that the efciency of experimentation decreases due to diminishing marginal returns from experimenting in that mode and that different modes can decrease at different rates. This view is strongly supported by empirical evidence from studies on bug detection in software engineering, which found similar relationships (Boehm 1981, Shooman 1983, Cusumano and Selby 1995). The following discussion will focus on the detection and removal of design errors as a way of measuring experimental yield. I assume the following. The compared modes can detect the same or at least a large group of common errors as a design evolves. This may not always be true for all cases as modes are sometimes designed to exclude aspects that are irrelevant to the experiment (e.g., wind tunnel experiments will not reveal internal airplane design problems) or contain models that focus on a single class of errors (e.g., logic simulation in integrated circuit design). If there is no overlap between error categories, any decreases in one modes efciency would leave the other modes efciency unchanged. The rate of new error introduction is negligible compared to the rate of error detection and removal. If errors are introduced at a signicantly higher rate (but smaller than the rate of error elimination), the efciency would still be decreasing but at a slower rate. As designers learn between projects, the mode trajectories can change from project to project but are still declining. For example, a more experienced engineer may be able to drive out a mode much more quickly by eliminating all design errors with fewer iterations. As an example, consider the use of computer simulation in the detection and elimination of design errors. During each simulation cycle, design errors will be detected and, if the error cause can be determined and removed, eliminated. With an increasing number of simulation cycles, the cumulative number of design errors eliminated is increasing and the pool of residual errors tends to decrease. Therefore the mean time between errors detected increases, implying that the number of tests to be run until another error is detected also
increases. The overall effect is a monotonically decreasing error detection rate, resulting in an increase in run time cost for a given return.3 Thus, the marginal return (errors detected per iteration) from simulation diminishes as a function of experimental cycles. Similarly, if a designer conducted experimental cycles with the aid of prototype testing, she would also experience diminishing returns but, because prototype tests run signicantly faster than simulated tests in this particular example, at a slower rate (see Figure 2). In general, we nd that prototype tests do not always execute experiments faster than computer simulation (e.g. simulating fatigue life performance in aircraft). However, one can still nd many cases where run time differences are many orders of magnitude even with rapid increases in computational speed. (Examples are thermal combustion in engines, crash tests of automobiles, timingspecic behavior in integrated circuits, 4 etc.)
3.2. Optimal Mode Switching as an Economic Experimentation Strategy In many design environments, designers have more than a single experimentation mode available to them. As mentioned earlier, the emergence of the digital computer has allowed designers to choose between computer simulation and physical prototypes to test their designs. But even before digital computers, engineers were known to conduct experiments with the aid of pencil and papera mode best described as mental simulation. Thus they had to decide when to switch between mental experimentation and prototype testing as a way of economizing their learning strategy. As the efciency trajectories of available experimentation modes differ, one would conceivably start experimentation with the mode that is most efcient under initial experimental conditions. Then, as a design pro3
One may be tempted to conclude that errors which are hard to detect are also hard to correct. In a study of 63 errors in software design, Shooman (1983) found no evidence in support of this conclusion. His data showed that there was no correlation between the difculty of error detection and correction.
The relative differences can run as high as 10 10 for timing-specic verication versus real-time runs in the target system (full IC prototypes) (Cooke 1993). While increasing computer speed will certainly narrow that gap, it is still signicant enough to play an important role in selecting experimentation strategies.
748
Figure 2
Switching Between Different Modes as Experimentation Efciencies Decrease at Different Rates
gresses and the modes marginal returns diminish, there may be a point (the optimal switching point) where the trajectories intersect and switching to another (more efcient) mode will be an economic strategy (see Figure 2). Such an experimentation strategy will be referred to as mode switching. Similar to curves illustrating diminishing returns in economic theory, the depicted smoothness in Figure 2 looks much rougher at the microlevel, resulting in occasional spikes or discontinuities that would suggest reswitching to modes previously used. (Occasional reswitching was observed during my studies of integrated circuit design.) As a general example, consider the use of computer simulation and prototype testing in design once more. Computer simulation is usually more efcient with respect to experimental step (2) (building a test model), step (4) (analysis of error symptoms), and step (1) (design modication). In contrast, full prototypes are more accurate and in many cases can execute test runs (step (3)) much more rapidly than simulation can. But since the number of design errors is usually high prior to starting experimentation, the mean-time between errors detected tends to be small and thus most designers will nd it economical to start with simulation. As a design progresses and the error detection rate of simulation declines, the efciency of continuing to simulate will at some point fall below the efciency of prototype testing. Thus, designers will nd it economical to switch from
simulation to prototype testing exactly where their efciency trajectories intersect (the optimal switching point or OSP), and continue iterative experimentation with prototype testing. I propose that nding the optimal switching point can result in signicant improvements in innovation cost and time.
3.3. The Impact of Technological Innovations on the Optimal Mode Switching Point Under what circumstances do we expect the optimal mode switching point to change? Technological innovations can have a direct impact on the cost and the time of experimental trials or on the value of information learned from an experiment. For example, consider the changes that one would expect if the cost and time of building (or modifying) a test model can be reduced (e.g., through the use of rapid prototyping technologies). Recall that during experimental step (2), models are built to test design solutions which are generated during step (1) (design). Since efciency is a function of cost and time, one would expect reductions in the cost and time of building a test model to have an effect on the efciency of conducting an experimental cycle. (I assume that these changes are due to innovation sources exogenous to a particular design process and that changes in model accuracy are negligible.) Thus one would expect the efciency trajectory of prototype testing to shift outwards and, as a result, the optimal
749
Figure 3
Impact of a Decrease in Cost of Building Prototypes on the Optimal Switching Point
mode switching point (OSP) to shift in favor of prototype testing. Figure 3 illustrates such a shift in the OSP between simulation and prototype testing, given that an exogenous innovation has resulted in a reduced build cost and time for design prototypes. As a general example, consider the dramatic improvements in the cost and time of prototyping that have resulted from rapid prototyping technologies. By signicantly reducing the cost and time to build a physical model prototype of a design, they have signicantly improved step (1) (design) and step (2) (build) of an experimental cycle. At the same time, I have observed an increased use of rapid prototyping (relative to other experimentation modes) at earlier points in a design process. Using the model presented in Figure 3, I propose that this shift in the OSP in favor of rapid prototyping has been partially caused by the (exogenous) change in the cost and time to build prototypes.
4. Research Methods
The empirical investigation concentrates on the integrated circuit (IC) based systems design industry, a eld chosen because (1) its overall economic signicance (the overall IC market is over $100 billion per year, and ICs can be found in most products today, ranging from communications networks to childrens toys); and (2) the eld has beneted greatly from the
availability of dramatically improved experimentation modes such as computer simulation and the rapid prototyping of integrated circuits. (Computer simulation enables designers to model and simulate complex system behavior prior to committing to an often expensive hardware prototype. In contrast, rapid prototyping provides designers with a low-cost, rapidly available prototype that can be used to run faster and more accurate experiments.) I elected to focus the study on the design of integrated systems with the aid of two technologies: (1) systems containing application-specic integrated circuits (ASICs) where the cost and time of building and modifying a prototype is high ; and (2) systems containing electrically programmable logic devices (EPLDs) where the cost and time of building and modifying a prototype is low (rapid prototyping; as described in 2.2). The cost of building or modifying an ASIC prototype can easily exceed tens of thousands of dollars because of xed manufacturing costs (known as nonrecurring engineering (NRE) costs) which are typically amortized over many units. EPLD prototypes are usually very inexpensive (a few hundred dollars), can often be erased and reprogrammed when necessary, but face higher variable cost when large volume production is required. One also nds that lead times for an ASIC prototype typically exceed one week whereas EPLD prototypes can be produced (programmed) almost instantly. This
750
is of particular concern to users of integrated circuits who usually come from the fast moving hightechnology industry where time-to-market drives project performance. While ASIC technology has been available to designers for more than a decade, modern EPLDs are a relatively new technologythey were invented in the late 1980s and have rapidly improved since then (Walker 1992). Todays EPLDs are not able to meet the complexity and speed requirements of very high performance integrated circuit designs, but they are quite capable of serving as a substitute for ASICs at the middle and low end of the integrated circuit design spectrum. Data on mode switching strategies were collected in a two-stage process. First , I followed a grounded research approach during which I conducted a eld study at a local high technology rm (Glaser and Strauss 1970). Over a period of three months, I conducted over 30 extensive interviews with designers and constructed a database with 24 design error case histories. The interviews and the data allowed me to verify the signicance of experimentation strategies in integrated circuit design and to develop an in-depth technical understanding of simulation and prototype testing in design practice. Second , I used my advanced understanding to develop a detailed mail questionnaire which was used to collect data on experimentation strategies from several hundred designers throughout the United States (Judd and Kidder 1991). The data analysis employs statistical analysis to determine if in fact the differences in prototyping cost and time result in different experimentation strategies, and if so, whether it has an impact on overall development cost and time.
4.1. Data Collection A mail questionnaire was used to learn about designers decisions and experiences during switching between computer simulation and prototype testing in the design of integrated circuit-based systems. The questionnaire was sent to 1000 designers who were selected from the subscriber database of Integrated Systems Design, a leading technical industry journal that focuses on issues related to ASIC- and EPLD-based design. The sample was divided into two groups (500 EPLD designers and 500 ASIC designers), which were randomly chosen from a pool of designers who qualied for the survey.
Selected participants work either in system design or design and development engineering. They design a large variety of end products, ranging from communications systems to medical electronic equipment, and they are employed in companies all over the United States. Because of a signicant difference in the cost and time to build and to modify a design prototype, I hypothesized that the two groups switching behavior varies signicantly. The questionnaire consists of four sections. In the rst section, designers were asked to provide general information about themselves such as length of their design experience. To learn about their general assessment of simulation and prototype testing and their a priori expectations of each modes effectiveness, designers were asked to compare both these modes along different dimensions (e.g., experimental steps) in the second section. Then, in the third section, the respondents reported on their switching behavior and the related decision variables in their last design projectnot a typical design project. The former approach is more likely to result in a representative sample of projects while the latter would probably lead to a biased sample (e.g., designers may be tempted to select a particularly successful project). In order to test for relationships between design performance and experimentation strategy, the fourth section asked designers for information on prototype iterations and the design effort (in personmonths) invested in their last design project. Time-tomarket is probably the most critical variable in integrated circuit design as product life cycles are very short. With ASIC and EPLD design teams often as small as one to three employees, the measure personmonths gives a good indication of development time as well as effort. Most of the questions related to variables which were hypothesized to inuence switching strategies and design performance. The questions and their phrasing were strongly inuenced by preliminary ndings in the extensive three-month eld study and a questionnaire pretest. The pretest of the questionnaire was conducted with the aid of two networking technology design groups (one EPLD and one ASIC group) at a company in Maryland, and a detailed discussion of the questions ensured that designers in both groups understood the questions as they were intended (Judd et al. 1991). The
751
Table 1
Subsample of Comparable EPLD and ASIC Design Projects

EPLD Variable Mean 11.54 26.08 5.92 4.31 4.33 5.02 St. Dev. 6.50 21.15 1.03 1.79 1.87 1.32 ASIC
n
52 51 52 52 52 52
Mean 12.40 28.66 6.10 4.61 4.29 4.27
St. Dev. 5.85 26.38 1.46 1.91 2.19 1.73
n
41 38 41 41 41 41
p-value
0.501 0.622 0.086 0.407 0.994 0.035
(1) (2) (3) (4) (5) (6)
Design size (k Gates) Percentage of design copied Type (1 analog; 7 digital) Has done similar designs before Run speed was pushed to limit High degree of logic complexity
Comments: Response scale (variables 46): 1: not at all accurate; 7: very accurate. Variables 1 and 2: t-test; variables 36: Mann-Whitney-U test (npar). The statistical tests along several complexity measures show no signicant differences between EPLD and ASIC design projects.
questionnaire was mailed to 1,000 designers (500 ASIC designers and 500 EPLD designers) in August 1994. In 16 cases, the addressee could not be reached or had left the company. Of the 984 designers reached, 463 returned the questionnaire. Sixty-one questionnaires had to be discarded because the designers felt that their background and experiences did not allow them to answer the questions with high condence. (The questionnaires allowed the addressee to indicate lack of condence and return it unanswered.) Eleven questionnaires arrived after the analysis had been completed and were therefore not included in the study. The remaining 391 yielded a response rate of 39.74 percent (38.53 percent for ASIC designers; 40.93 percent for EPLD designers). A comparison of the respondents length of design experience resulted in no signicant difference between the groupsboth had been designing integrated circuits for an average of 10 years. (Some designers had experience in both ASIC and EPLD technology.) As some of the measured variables can be inuenced by design complexity and ASIC designs are on average more complex than EPLD designs, I extracted two subsamples that were compared along several complexity measures 5 (Table 1).
5
A comparison between EPLD and ASIC design projects that are in the two subsamples shows no signicant difference for the rst ve measures. (Variable 6 shows a difference signicant at 5 percent, which would provide some support of higher EPLD complexity.) Earlier it was mentioned that for projects of low to moderate complexity and average speed requirements, integrated circuit designs can often be developed using either technology (ASIC or EPLD), with differences remaining mainly in the volume production cost of ICs. Using input from design engineers, the subsamples were carefully selected to occupy a complexity and performance spectrum where both technologies were feasible choices to designers. Even though the subsamples represented designs that were considered of lower complexity and speed, they accounted for 30.5 percent of the research studys EPLD-based designs and 25.2 percent of all ASIC-based designs. Thus it is reasonable to assume
for the possibility that some designs have a high gate count but are of little complexity because most of the design is copied or replicated. (For example, a memory cell can be repeated many times, quickly increasing the gate count with little incremental effort.) The third variable (type) captured to what degree the design was a mix between analog and digital components. If there was a signicant difference in type, one would have to analyze its impact on design complexity. The fourth variable (has done similar designs before) measured the degree of similar design projects that designers had completed previously. The fth variable (run speed was pushed to limit) and the sixth variable (high degree of logic complexity) captured rated design speed and logic complexity, respectively. In summary, respondents provided six measures that all relate to design complexity.
Based on the analysis of complexity clusters in a histogram of gate densities, the subsamples were selected to contain all designs with densities between 5,000 gates and 25,000 gates. In addition, other measures that could inuence complexity were used. The rst variable (design size) measured the number of gates that were used in the design. The second variable (percentage of design copied) accounted
752
that projects from both subsamples are of similar complexity and that an objective comparison of project performance can be conducted. Where appropriate, results of the study were compared using these subsamples only.
4.2. Results It was proposed that the efciency of experimentation modes can decrease at different rates and that, as a result, designers will nd it economical to determine the optimal switching point (OSP) between these modes. As changes in the cost to build (and/or modify) experimentation models cause a movement of the experimental modes efciency trajectory and thus a shift in the OSP, designers will nd that adjusting their mode switching strategies in the direction of the new OSP will result in signicant reductions of total design cost and time. As shown in the following subsections, the data strongly support these propositions. 4.2.1. Optimal Mode Switching as an Economic Experimentation Strategy. In the pilot research, several interviewees who used EPLD technology reported that they start error elimination with computer simulation and sometimes switch to prototype testing before simulation has detected all the errors that, in theory, it could have detected. This was in line with the hypothesis that simulation is initially more efcient than prototype testing, but designers would switch if the efciency of simulation fell below the efciency of prototype testing. To test that the switching behavior was driven by such economic considerations, I divided the study sample into two groups of designers that face very different experimentation cost: (1) ASIC-based designs with high prototyping costs and time; and (2) EPLD-based designs with relatively low prototyping costs and time. Thus, the hypothesis suggests that, in the case of ASICs, designers will nd it economical to switch very late and EPLD designers would benet from a strategy of switching early. The result would be a signicant difference in the timing of their switching from simulation to prototype testing. To test for differences in timing, I analyzed the designers reported switching behavior in their last project and divided the observed patterns into two categories: (1) early switchingstop simulation and start prototype testing while design errors remain (and, to some
extent, could be detected via continued simulation); and (2) late switchingstart prototype testing after being extremely condent that no design errors remained. Because simulation delity can inuence behavioral choices and mode efciencies, designers were also asked to provide information on switching to prototype testing as a function of model delity. (Pilot research indicated that simulation delity can best be approximated by error categories. Thus, in the case of logic errors the delity of simulation is very high; it is somewhat lower for timing errors, and it is signicantly lower for errors due to signal quality problems.6 ) The variable switching from computer simulation to prototype testing (Table 2) clearly discriminates between the two design approaches: on average, ASIC designers switch signicantly later than EPLD designers for all levels of delity. This in support of the hypothesis that a reduction in prototype build cost and time (as in the case of EPLDs) would result in a shift of the optimal switching point (OSP) in favor of prototype testing (see 3.3). Interestingly, if we examine mode switching as a function of error categories (simulation delity), we nd that the higher the delity of simulation, the more likely designers will switch late. The nding is not surprising since it follows directly from the model: as a modes probability of detecting an error increases, so will its efciency at a given point in time. Thus we nd further support for the presented model. Thus far, we have measured behavioral choices as they are perceived by the respondents. But are the
Field research showed that designers typically categorize errors in the following types: (1) logic errors: errors due to faulty logic behavior of a design (examples: reversed select lines of a decoder; negation of an input signal); (2) timing errors: errors due to faulty temporal behavior of a design (example: insufcient set-up and hold times in a digital circuit); (3) signal quality errors: errors due to the degradation of signal quality during signal transmission (example: introduction of noise in high frequency applications due to excessive line proximity on printed circuit boards). Designers typically have different simulation tools available that focus on the detection and analysis of one of these error categories. Because of the varying difculty of modeling the underlying error-related behavior, different simulation tools vary with respect to effectiveness and probability of adoption during the design process. Models for logic simulation are usually very accurate, whereas signal quality can be modeled only with great difculty.
753
Table 2
Switching from Computer Simulation to Prototype Testing

EPLD Designers ASIC Designers Total 195 (100%) 195 (100%) 195 (100%) Late 105 (57.1%) 102 (57.0%) 38 (21.1%) Early 79 (42.9%) 77 (43.0%) 142 (78.9%) Total 184 (100%) 179 (100%) 180 (100%)
Error Class LOGIC TIMING SIGNAL QUALITY
Late 91 (46.7%) 59 (30.3%) 17 (8.7%)
Early 104 (53.3%) 136 (69.7%) 178 (91.3%)
p-value
0.051 0.000** 0.001**
Comments: p-value: ** p 0.01; * p 0.025; two-tailed Fishers exact test. The propensity to switch to prototypes early increases with (a) decreasing cost and time of prototype changes; and (b) decreasing delity of the underlying simulation model.
measured behavioral choices consistent with the observed switching strategies (i.e., are there differences in the actual use of simulation and prototype testing?)? To shed light on this question, I asked designers to give a numerical count of the number of prototype iterations undertaken in their last project. Since ASIC designers are more likely to switch late, we would expect to see very few prototype iterations. In contrast, the early switching behavior that we observed in EPLD designers would lead to signicant iterative experimentation during prototype testing (ceteris paribus). The variable number of prototype iterations (Table 3) shows that on average, EPLD designers used 13.90 prototype iterations before the design was complete while ASIC designers used 1.49 prototype iterations (signicance of mean difference: p 0.001; samples contain designs of comparable design complexity as described in 4.1.). This result strongly supports earlier ndings that EPLD designers incorporate prototype testing in their experimentation strategy to a much higher degree than
Table 3
ASIC designers do (and, as a result, switch to prototype testing much earlier). A high iteration frequency suggests that EPLD designers do not attempt to get it right the rst time, while ASIC designers do.
4.2.2. Mode Switching and Development Performance. In the previous section, it was demonstrated that technological innovations leading to a different mode efciency trajectory can result in different switching points. I will now show that knowing these different switching points (and managing experimentation accordingly) can matter signicantly with respect to product development performance. To explain why we would anticipate to nd a change in performance, consider the different mode switching strategies that are illustrated in Figure 4. EPLD technology has effectively lowered the cost and time of building prototypes and caused an upward shift in the trajectory of prototype testing. Under conditions of perfect information and unbounded rationality, one would expect a former ASIC designer who now uses EPLD
Comparing Prototype Iterations of EPLD and ASIC Design Projects

EPLD Variable Mean 13.90 St. Dev. 14.77 ASIC
n
51
Mean 1.49
St. Dev. 1.48
n
33
p-value (t-test)
0.000**
Number of prototype iterations
A prototype iteration occurs whenever the designer makes a change to any part of the physical design prototype and subsequently veries it.
754
Figure 4
Differences in Experimentation Efciency Between EPLDs and ASICs
(Shaded area represents expected efciency gain without change in mode switching point; A / B is potential gain if mode switching strategy is adjusted to new OSP )
technology to change her switching point from OSP to OSP and gain a total efciency of areas A / B. In contrast, if no adjustment of mode switching were made, efciency gains would merely amount to area B (area A is lost since it is due to the change in the mode switching point). Given that bounded rationality and imperfect information pervades in most environments (Simon 1955, 1981), the expected efciency gains would lie somewhere between B and A / B. To test for differences in overall performance of EPLD and ASIC design projects, I asked designers to provide an estimate of design effort required in the completion of their last design project (in person-months) and how the design teams effort was distributed among ve different design phases: (1) design specication, (2) design development, (3) design verication, (4) design prototyping, and (5) prototype evaluation. I will describe by reference to these phases what designers typically do in the design of an ASIC and then discuss some of the differences in EPLD design. (1) Design specication : The design process starts with the general formulation of design requirements and is directly followed by chip-level decisions such as estimates of pin counts, design size, etc. These early phases of the design can be best characterized as design specication and often involve meetings with en-
gineers from the ASIC foundry (the vendor). During these meetings, technical issues that relate to the interdependence between design and fabrication are discussed, technical and economic tradeoffs are considered, and contractual agreements between designer (the customer) and vendor are made. (2) Design development : After an agreement is signed, the vendor usually supplies software design libraries that are specic to its ASIC technology. These libraries are needed to start the low-level design. Furthermore, in very complex projects, designers often write a completed set of design documentation before starting the actual design development. Only then, designers develop the low-level design according to specications and with the aid of either schematic entry tools (schematic capture) or higher level design synthesis tools. (3) Design verication : After a design block is completed to the extent that it can be simulated, the designer develops a set of functional patterns (or functional vectors) and runs a logic simulation. If a logic error is detected, it is analyzed for its cause and design changes are made. This process iterates until no logic errors are detected within a specied time period. Logic simulation is followed by timing simulation and again designers iterate until no timing error is detected. Since the
755
ASIC has not been laid out or routed (a process where the ASIC manufacturer converts the design into a manufacturable and process-specic circuit architecture), timing delay models are only rough approximations and are of moderate accuracy. As a consequence, some timing simulations have to be repeated when the routed design is available.7 (4) Design prototyping : After receiving a signed release approval from the customer, the ASIC vendor generates the nal design database which is used to create pattern-generation (PG) tapes. The PG tapes are used to manufacture photolithography masks which contain information on the IC customization layer(s). Then the integrated circuit is fabricated (for a detailed description of the actual fabrication process, see Einspruch 1991). The rst prototypes are tested and packaged by the vendor and delivered to the customer for in-system verication. The steps involved to create a prototype can be best described as design prototyping (from the customer perspective, however, design prototyping will involve only the time spent on coordination activities with the vendor and internal departments that approve the cost of ordering a prototype).
After the design is veried with the aid of computer simulation, the designers have to prepare a design package for the ASIC vendor. Such a design package normally includes (1) a set of test patterns used by the vendor to verify the fabricated ASIC for physical functionality; (2) the translated design les (in a standard format); (3) copies of error reports; and (4) a copy of a design oorplan. Upon receipt of the design package, the ASIC vendor reviews the material and prepares a design specication document that has to be approved and signed by the customer. The handover of a signed specication document is often referred to as the rst sign-off. After receiving the signed document, the vendor develops a design layout and routes the array interconnects (in the case of gate-arrays) which are highly dependent on the fabrication process. The actual lengths of metal interconnects are extracted and used to create an interconnect-delay le which is returned to the customer. In addition, the vendor normally runs its own simulations to verify the manufacturability of the design. Using the interconnect-delay le, the customer can rerun timing simulations with higher modeling accuracy and check for errors that passed earlier timing simulations. If errors are found, the design has to be modied and part of the design process has to be repeated. Once the customer is satised with the design, it is formally released to the vendor by signing a release-to-manufacturing approval (often referred to as the second sign-off). The process of resimulating with more accurate models can still be regarded as part of design verication (or better design reverication).
(5) Design evaluation : Finally, the ASIC designer determines if the prototype works without error which is not always the case (design evaluation). If an error is detected, costly prototype modications are necessary unless a hardware or software workaround can be found. If a workaround is not possible, the design has to be modied, and many of the earlier design and signoff related steps have to be repeated. After the prototype performs as intended, a prototype approval is signed and volume production can begin. Due to the high cost of correcting design errors, ASIC designers rely heavily on simulation to get the prototype right the rst time. At the micro-level, EPLD design follows a similar process but due to the ability to prototype ICs in-house and the low cost and time of prototyping, there are a number of differences when compared to the ASIC design process: Usually, EPLD designs do not require meetings or contractual agreements (e.g., no sign-off) with vendors since the design is not handed over to an outside party. EPLD design software can route a design directly after it is completed and thus the most accurate timing delay estimates available can be used at the start of timing simulation (i.e., no resimulation is necessary with more accurate models). EPLD designers do not have to prepare test cases that ensure the physical functionality of a prototype device since EPLD manufacturers test the devices before delivering them to the customer. While the rst hardware board can be somewhat expensive, prototype modications are normally much less costly and time-consuming than for ASICs. An analysis of project performance data shows that EPLD designers needed an average effort of 8.15 person-months to complete their projects, while ASIC designers needed 17.94 person-months (signicance of mean difference: p 0.01; Table 4). As we will see, at least 43 percent of this difference can be attributed to the different mode switching strategies discussed earlier. If one considers that design effort correlates highly with time-to-market (quite often, these projects are completed by teams consisting of 12 members) and that product life cycles in this industry are very short, then a difference of such magnitude can be regarded a signicant competitive advantage for a rm.
756
Table 4
Comparing Design Effort and Design Phases Between EPLD and ASIC Projects
EPLD Variable Mean 8.15 1.08 2.74 1.52 1.20 1.60 St. Dev. 5.75 0.93 2.70 1.30 1.23 1.69 ASIC
n
47 47 47 47 47 47
Mean 17.94 3.37 5.48 5.75 1.26 2.07
St. Dev. 17.98 4.91 6.38 6.03 1.59 2.40
n
31 31 31 31 31 31
p-value (t-test)
0.006** 0.015* 0.030* 0.001** 0.856 0.353
Design effort (person-months) due to (1) design specication due to (2) design development due to (3) design verication due to (4) design prototyping due to (5) design evaluation
Comments: Only data from interviewees that responded to all six questions were analyzed. p-value: ** p 0.01; * p 0.025 (test: independent samples of unequal variances). Samples are for designs of comparable complexity; see 4.1.
A close examination of design effort (Table 4) shows that the overall performance difference is driven by earlier phases of a project: (1) design specication, (2) development, and (3) verication. EPLD designers required signicantly less resources than ASIC designers in all of these three phases while no signicant difference was found for design prototyping and evaluation. Design specication accounted for 23.4 percent of the total design performance difference. EPLD designers chose to spend signicantly less effort on design specication than their ASIC counterparts (signicance of mean difference: p 5%). While design specication does not involve computer or prototype-assisted experimentation, this nding does suggest a connection between the degree of specication and the expected cost of making design changes. Thus, if the cost and time of modifying a design is low, designers are less inclined to invest effort in reducing the risk of facing design changesafter all, they can change them at a relatively low cost ( ceteris paribus ). The ability to make changes at low cost and time results in higher development exibility and can have a signicant impact on overall design performance (Thomke 1997). This nding is also consistent with studies of prototype-driven design approaches in software engineering where short specication phases combined with early prototypes for user feedback were instrumental in accelerating software projects and increasing user satisfaction (Alavi 1984, Boehm et al. 1984). Another 28 percent of the total performance difference can be attributed to design development. ASIC design-
ers spent signicantly more effort on development than EPLD designers (signicance of mean difference: p 5%). From eld observations, much of the additional effort that ASIC designers invest is related to the high cost and time of making error-related changes to the design prototype(s). First, ASICs are more carefully designed as to reduce the likelihood of running into problems during prototype testing. Second, as a means of minimizing the risk of failure, ASIC vendors require thorough design documentation which in turn requires additional time on behalf of the designer. In contrast, EPLD designers can come up with a design solution and quickly move to experimentation (via simulation or prototype testing) as means of getting rapid feedback. Finally, design verication where most experimentation activity and the related switching between simulation and prototype testing occuraccounts for 43.2 percent of the total performance difference. EPLD designers (mean performance 1.52 person-months) were by a factor of 3.8 more effective than their ASIC counterparts (mean performance 5.75 personmonths) (signicance of mean difference: p 1%). I submit that the difference is due to early switching to prototype testing by EPLD designers and thus being able to move to a higher efciency trajectory more quickly (the resulting efciency gains were shown earlier in Figure 4). Interestingly, the higher number of EPLD prototype iterations that resulted from early switching did not lead to signicantly higher effort during design evaluationthe phase where most prototype testing occurs ( p 10%).
757
Finally, one may ask why designers would choose to select ASIC technology if EPLD technology was technically feasible and allowed faster and more efcient development. A number of alternative explanations were discovered during the eld study. First, designers chose to stay with ASIC technologies because at large volume production, lower variable cost for ASICs would provide signicant savings which are traded against a more inefcient and slower development process. This is supported by data from the two subsamples analyzed: on average, EPLD designers eventually produced 1581 units, whereas ASIC designers had average production volumes of 113,232 units which, incidentally, may have reinforced the risk-averse behavior of ASIC designers. However, a number of small rms have recognized this opportunity and started to specialize on supplying conversion paths from EPLD to ASIC designs. (Designers would use EPLD technology to develop their integrated circuits, have a specialized rm convert it to an ASIC design, and then run volume production in an ASIC foundry.) This has started to result in a gradual migration away from ASICs as a design technology where EPLDs are feasible alternatives even when large production volumes are required. Second, ASIC designers are reluctant to switch to a technology they often consider inferior as it requires them to make an investment in acquiring different skills and, if they became EPLD designers, often a career track with lower pay, less peer recognition and fewer opportunities to grow (e.g., moving into high-end design projects). Nonetheless market forces are pushing toward a proliferation of EPLD technology in the low-to-moderate integrated circuit design segment for many of the reasons that are described in this paper. In summary, the data offer strong evidence for a positive link between the economic performance of a design project and its related experimentation strategy. While the survey cannot determine cause and effect, it can be concluded that experimentation strategies represent an important connection between problem-solving efciency and overall product design performance.
knowledgereceived little attention in the product development and innovation literature, even though it is often one of the most important decisions in development projects that I have studied. Other than integrated circuit design, examples can be drawn from software development (switching from internal testing to beta site testing) (Cusumano and Selby 1995, Boehm 1981), development of chemical processes (switching from pilot plants to large-scale facilities) (Pisano 1996), and many more elds where experiments are carried out with different modes as to economize on differences of efciency trajectories. This paper has shown that, because of declining mode efciencies, determining the point where it is optimal to switch (the OSP) and adjusting the mode switching strategy accordingly can result in signicant improvements of design performance. Thus, managers will nd it economical to provide design resources that aid in the determination of mode switching points and to devise managerial actions as to move designers switching behavior as close as possible to the optimal point. In the following subsections, I will provide some measures at the conceptual and operational levelthat can aid managers in this process.
5. Discussion
The concept of explicitly managing the switching between modes of experimentation hasto the best of my
5.1. Implications for the Adoption of New Experimentation Technologies Innovation practitioners and managers are now adopting and using new experimentation technologies without an explicit understanding of the change in experimentation economics that can follow, and the consequent change in economic design strategies that may result. One objective of this paper is to help managers understand the impact of new and improved experimentation technologies on design performance and formulate managerial actions that would motivate designers to take full advantage of these changes in experimentation economics. To demonstrate the importance of correct managerial actions with a general example, consider the effect of rapid prototyping on a hypothetical design process CHIP that consists of two phases: (1) experimental cycles with computer simulation; and (2) experimental cycles with prototypes (Figure 5). Before the introduction of the rapid prototyping technology, CHIP designers face very high cost and time of
758
Figure 5
Impact of Rapid Prototyping Technology on Design Cost and Time (with and without modications of mode switching strategy)
time reduction would remain untapped unless management understands the impact of rapid prototyping on the mode switching point and revises its managerial actions accordingly. In this particular example, it would imply an earlier switching from simulation to prototype testing and, as a result, a change of the managerial actions that were appropriate for case 1. For example, the prior prototype-works-right-the-rst-time design award would certainly motivate designers to follow experimentation practices that emphasize rst-prototype success (which was appropriate before the availability of the rapid prototyping technology). Thus management should replace the award with some incentive that would motivate designers to switch from simulation to prototype testing before being very certain that no errors remain in their designs. This behavior, though somewhat counterintuitive, will often result in designers getting it [the prototype] wrong the rst time but, at the same time, being able to further improve design performance signicantly. (This nal strategy of switch ing early is depicted in case 3, Figure 5.)
building and modifying design prototypes. Thus an economical experimentation strategy includes the simulation of a design until one can conclude with a high degree of certainty that the rst prototype will operate without failure (case 1, Figure 5). The appropriate managerial action in this rst case would include a development schedule with ample simulation time and resources, and perhaps design performance feedback such as a prototype-works-right-the-rst-time design award. Now consider the availability of a rapid prototyping technology which allows a designer to build (and modify) a prototype at a lower cost and time (case 2, Figure 5). The use of rapid prototyping would most certainly reduce the total cost and time required for prototype testing, even if the managerial actions from case 1 were left in place and the mode switching strategy remained unchanged. However, the ndings in this paper suggest that additional opportunities for a signicant design cost and
5.2. Implications for the Organization of Development Projects Effective product design requires both that all of the organizational groups involved develop the appropriate specialized capabilities, and that the efforts and technologies are effectively integrated (Katz and Allen 1988, Hayes et al. 1988). In the context of switching between experimentation modes, we sometimes nd that the switching involves a hand-off between organizational groups that are specialists in the use of a particular experimentation mode. Such a hand-off is likely to occur with some bias unless the interaction and integration of these groups is managed effectively at both the individual and organizational level. Consider the use of computer simulation and prototype testing in integrated circuit design once again. In large design projects, one sometimes nds that simulation and prototype testing is assigned to different design groups, and that the knowledge and skills of these groups is highly specialized. Thus switching between experimentation modes will require a formal hand-off between these two groups but will often result in handoff biases. Some of the reasons (and possible solutions) are:
759
Specialization and coordination: Quite often, design groups that specialize in simulation know little about prototype testing (and vice versa) and do not have access to real-time information about each others experimentation activities. An objective evaluation of the optimal mode switching (or hand-off) point, however, requires both, knowledge and information on both activities. Thus, managers will nd it benecial (1) to educate engineers from both groups about each others tasks; (2) to continuously collect data related to the efciency of simulation and prototype testing (e.g., by using electronic error elimination protocols); and (3) to periodically evaluate the decision to switch between simulation and prototype testing with the aid of an interdisciplinary group that consists of engineers from both groups. Managing development resources: Design resources (e.g., time, stafng) are often allocated to groups prior to starting a project. However, one cannot reasonably predict the optimal switching point (and, as a result, the total time and resources required) prior to the start of simulation as much of the experimental learning about diminishing efciencies occurs as the project unfolds. Some managers, however, seem to be unaware of this. For example, my empirical investigation of integrated circuit design showed that some designers stop simulation because their preallocated simulation time has expired and that their project schedule requires them to proceed to prototype testing. However, unless experimentation-related design resources and time schedules are dynamically readjusted (based on the learning that occurs during experimentation), it is unlikely that a hand-off from simulation to prototype testing (and vice versa) occurs at the optimal point. Thus, managers ought to understand the tradeoffs and learning that are involved in optimal mode switchingand manage the development process accordingly. Other reasons that may introduce hand-off biases relate to intergroup competition (e.g., the simulation group tries to demonstrate the goodness of simulation by handing-off an error-freebut excessively simulateddesign to prototype testing). This list of reasons is by no means exhaustive but shows some of the organizational challenges that managers should consider in achieving effective product design.
5.3. Implications for the Core Competencies of a Corporation Finally, an important research question is how the success of new and improved experimentation modes will affect the core competencies of a corporation (Prahalad and Hamel 1990). In a study of Silicon Graphics (SGI), a leader in the high-performance server and graphics workstation industry, Iansiti (1995) observed that computer simulations and partial hardware models are essential to Silicon Graphics [exible and fast cycle] development process. Thus, mastering iterative experimentation and the related modes is an important competence of rms such as SGI in a number of industries. As an example of new modes requiring new competencies in industries other than electronics and computers, consider mass screening an experimentation mode that is rapidly changing the economics of pharmaceutical drug design. Mass screening is a very efcient way to achieve step (4) of an experimental cycle, that is the analysis (or, at least, identication) of experimentation results. In this method, one can screen the results of literally tens of thousands of experiments of a suitable type extremely rapidly and cheaply. For example, one can economically and simultaneously screen many bacteria for a gene giving resistance to a given antibiotic by placing a mixture consisting of millions of different types on an agar plate, treating the plate with the antibiotic in question, and then simply observing which of the many bacteria grow into colonies despite the treatment. Those that do grow in the presence of the antibiotic have the type of resistance one is seeking. Mass screenings of this general type are applicable to both unknown and known inputs. An example of an unknown input is when genetic mutations of many unknown types are created in bacteria by subjecting them to x-ray irradiation. Known inputs for mass screening can be created by, for example, techniques such as combinatorial chemistry that systematically and very quickly create thousands of known combinations of simpler chemicals at very low cost (Thomke et al. 1998). The core competence required for combinatorial chemistry and mass screening in the design of pharmaceuticals is built upon (1) the ability to generate large chemical diversity at relatively small cost and time, and (2) a good understanding of a search techniques and experimental analysis. In contrast, the abil-
760
ity to competently perform rational drug designan approach where one tries to create and model on a computer molecules that will have a given biological effectis built upon an understanding of molecular design. In a study of core competencies in ethical drug discovery, Henderson (1993) found that the old style of drug discovery (i.e., large-scale screening of compounds in screens designed to mimic disease states in man, without a good understanding of the specic biochemical and molecular pathways that are responsible for a therapeutic effect) required a group of pharmacologists who design and run screens and a group of organic chemists who were responsible for synthesizing variants of promising compounds. In contrast, new style drug discovery (i.e., rational design) requires the integration of rapidly changing knowledge from scientists with training in many disciplines, ranging from molecular biology to synthetic chemistry. And with the recent dramatic advances in combinatorial chemistry and mass screening, rms will nd that some of the competencies required for rational drug design may prove to be superuous. Thus, research into the impact of new experimentation modes on rm competence can provide new insights on how rms ought to adapt to these changes.8
8
The author would like to thank Eric von Hippel for his contributions to the ideas embodied in this paper. A special thanks goes to my colleagues Ann Carter, Don Clausing, Larry Dennison, Steven Eppinger, Marco Iansiti, Christoph Loch, and Don Reinertsen, and four referees for their insights and comments. The nancial support of the MIT Lemelson Doctoral Fellowship Program and the Harvard Business School Division of Research are gratefully acknowledged.
References
Adler, P., Shared Learning, Management Sci ., 36 (1990), 938957. and K. Clark, Behind the Learning Curve: A Sketch of the Learning Process, Management Sci ., 37 (1991), 267281. Alavi, M., An Assessment of the Prototyping Approach to Information Systems Development, Comm . ACM , June (1984), 556563. Alexander, C., Notes on the Synthesis of Form , Harvard University Press, Cambridge, MA, 1964. Allen, T. J., Studies of the Problem-Solving Process in Engineering Design, IEEE Trans . Engineering Management , 13, 2 (1966), 72 83. Arora, A. and A. Gambardella, Division of Labor and Inventive Activity, Working Paper, Carnegie-Mellon University, 1992. Barron, J., Thinking and Deciding , Cambridge University Press, New York, 1988.
Boehm, B., T. Gray, and T. Seewaldt, Prototyping versus Specifying: A Multiproject Experiment, IEEE Trans . on Software Engineering , May (1984), 290302. , Software Engineering Economics , Prentice Hall, Englewood Cliffs, NJ, 1981. Bohn, R., Learning by Experimentation in Manufacturing, Working Paper No. 88-001, Harvard Business School, Cambridge, MA, 1987. , Noise and Learning in Semiconductor Manufacturing, Management Sci ., 41 (1995), 3142. Bowen, K., K. Clark, C. Holloway, and S. Wheelwright, The Perpetual Enterprise Machine , Oxford University Press, New York, 1994. Box, G., W. Hunter, and S. Hunter, Statistics for Experimenters , Wiley, New York, 1978. Clark, K. and T. Fujimoto, Product Development Performance , Harvard Business School Press, Boston, MA, 1991. Clausing, D., Total Quality Development : A Step-by-Step Guide to WorldClass Concurrent Engineering , ASME Press, New York, 1993. Cooke, L., The FPGA Engineering Revolution, ASIC & EDA , August (1993), 3853. Cusumano, M. and R. Selby, Microsoft Secrets , The Free Press, New York, 1995. Einspruch, N. and H. Jeffrey, Application Specic Integrated Circuit ( ASIC ) Technology , Academic Press, San Diego, CA, 1991. Eisenhardt, K. and B. Tabrizi, Accelerating Adaptive Processes: Product Innovation in the Global Computer Industry, ASQ , March (1995), 84110. Glaser, B. and A. Strauss, The Discovery of Grounded Theory : Strategies for Qualitative Research , Aldine Publishing Company, Chicago, IL, 1970. Griesser, H., Effective Prototyping during Product Development, Masters Thesis, Massachusetts Institute of Technology, Cambridge, MA, 1992. Hayes, R., S. Wheelwright, and K. Clark, Dynamic Manufacturing , The Free Press, New York, 1988. Hauptman, O. and G. Iwaki, The Final Voyage of the Challenger, Harvard Business School Case #9-691-037, Cambridge, MA, 1991. Henderson, R., Flexible Integration as Core Competence: Architectural Innovation in Cardiovascular Drug Development, Mimeo, Stanford University, Stanford, CA, 1993. Iansiti, M., Technology Integration: Making Critical Choices in a Turbulent World, Harvard Business School Press, Boston, MA, 1997. , Shooting the Rapids: Managing Product Development in Turbulent Environments, California Management Rev ., Fall (1995), 3758. Jones, A., Design-to-Cost and Time-to-Market with FPGAs, Components in Electronics , January (1992), 6. Judd, C., E. Smith, and L. Kidder, Research Methods in Social Relations , Harcourt Brace Jovanovich, Fort Worth, TX, 1991. Katz, R. and T. Allen, Organizational Issues in the Introduction of New Technologies, in R. Katz (Ed.), Managing Professionals in Innovative Organizations , Ballinger, Cambridge, MA, 1988. Leonard-Barton, D., Wellsprings of Knowledge , Harvard Business School Press, Cambridge, MA, 1995.
761
Mowery, D. and N. Rosenberg, Technology and the Pursuit of Economic Growth , Cambridge University Press, New York, 1989. Marples, D. L., The Decision of Engineering Design, IRE Trans . on Engineering Management , 2 (1961), 5571. Miles, M. and M. Huberman, Qualitative Data Analysis , Sage Publications, Beverly Hills, CA, 1984. Montgomery, D., Design and Analysis of Experiments , Wiley, New York, 1991. Nelson, R. and S. Winter, An Evolutionary Theory of Economic Change , Harvard University Press, Cambridge, MA, 1982. Newell, A. and H. Simon, Human Problem Solving , Prentice-Hall, Englewood Cliffs, NJ, 1972. Phadke, M., Quality Engineering Using Robust Design , Prentice Hall, Englewood Cliffs, NJ, 1989. Pisano, G., The Development Factory , Harvard Business School Press, Cambridge, MA, 1996. Pralad, C. K. and G. Hamel, The Core Competence of the Corporation, Harvard Business Rev ., MayJune (1990). Reinertsen, D., Whodunit? The Search for New-Product Killers, Electronic Business , July (1983), 6266. Sachs, E., CAD-Casting: Direct Fabrication of Ceramic Shells and Cores by Three Dimensional Printing, Manufacturing Rev ., 5, 2 (1992), 117126. Shooman, M., Software Engineering : Design , Reliability , and Measurement , McGraw Hill, New York, 1983. Simon, H., A Behavioral Model of Rational Choice, Quarterly J . Economics , 69 (1955), 99118. , The Sciences of the Articial , Second Edition, MIT Press, Cambridge, MA, 1969. Smith, P. and D. Reinertsen, Developing Products in Half the Time , Van Nostrand Reinhold, New York, 1991. Smith, R. and S. Eppinger, A Predictive Model of Sequential Iteration in Engineering Design, Management Sci., 43, 8 (1997), 11041120.
Taguchi, G., System of Experimental Design : Engineering Methods to Optimize Quality and Minimize Costs , Kraus International Publications, White Plains, NY, 1987. Thomke, S., The Role of Flexibility in the Development of New Products: An Empirical Study, Research Policy, 26 (1997), 105119. , E. von Hippel, and R. Franke, Modes of Experimentation: An Innovation Process Variable, Working Paper No. 97-052, Harvard Business School, Cambridge, MA, 1998 (forthcoming in Research Policy ). Tyre, M. and O. Hauptman, Effectiveness of Organizational Response Mechanisms to Technological Change in the Production Process, Organization Sci ., 3, 3 (1992), 301320. Ulrich, K. and S. Eppinger, Product Design and Development , McGrawHill, New York, 1994. Uzumeri, M. and C. Snyder, Information Technology and Accelerated Science: The Case of the Pentium Flaw, California Management Rev ., 38, 2 (1996), 4463. Villasenor, J. and W. Mangione-Smith, Congurable Computing, Scientic American, June (1997), 6671. von Hippel, E., Sticky Information and the Locus of Problem Solving: Implications for Innovation, Management Sci ., 40, 4 (1994a), 429439. and M. Tyre, How Learning by Doing is Done: Problem Identication in Novel Process Equipment, Research Policy , 19, 1 (1994b), 112. Walker, R., Silicon Destiny : The History of Application Specic Integrated Circuits and LSI Logic Corporation , C.M.C. Publications, Milpitas, CA, 1992. Wall, M., K. Ulrich, and W. Flowers, Evaluating Prototyping Technologies for Product Design, Working Paper No. 3334-91-MSA, Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA, 1991. Wheelwright, S. and K. Clark, Revolutionizing Product Development , The Free Press, New York, 1992.
Accepted by Ralph Katz ; received February 1996. This paper has been with the authors 2 months for 2 revisions .
762

0003 - Managing Experimentation in The Design of New Products

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

0003 - Managing Experimentation in The Design of New Products

Uploaded by

Copyright:

Available Formats

Managing Experimentation in the Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

2. The Economics of Experimentation

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

The Framework Views Experimentation as an Iterative Four-Step Learning Cycle

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

3. Optimal Mode Switching as an Economic Experimentation Strategy

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

Switching Between Different Modes as Experimentation Efciencies Decrease at Different Rates

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

Impact of a Decrease in Cost of Building Prototypes on the Optimal Switching Point

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

Subsample of Comparable EPLD and ASIC Design Projects

Mean 12.40 28.66 6.10 4.61 4.29 4.27

St. Dev. 5.85 26.38 1.46 1.91 2.19 1.73

(1) (2) (3) (4) (5) (6)

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

Switching from Computer Simulation to Prototype Testing

Error Class LOGIC TIMING SIGNAL QUALITY

Late 91 (46.7%) 59 (30.3%) 17 (8.7%)

Early 104 (53.3%) 136 (69.7%) 178 (91.3%)

Comparing Prototype Iterations of EPLD and ASIC Design Projects

St. Dev. 1.48

Number of prototype iterations

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

Differences in Experimentation Efciency Between EPLDs and ASICs

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

Mean 17.94 3.37 5.48 5.75 1.26 2.07

St. Dev. 17.98 4.91 6.38 6.03 1.59 2.40

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

THOMKE Managing Experimentation in Design of New Products

MANAGEMENT SCIENCE/Vol. 44, No. 6, June 1998

You might also like