With

this module, we move into the domain of the how; from the domain of the stakeholders to the domain of the system. We will examine the acceptance criteria and the synthesis and assessment of alterna=ve system concepts. By the end of the module we will address the genera=on, evalua=on and selec=on of implementa=on concept(s).

 The objec=ves of this module are: to understand the importance of key acceptance criteria to keep the project focused, to consider a range of possible implementa=on or system concepts at the system and sub-system levels and last we will consider the use of the Pugh Concept Selec=on Matrix to inform the concept selec=on decision. The decisions reached during this phase will set the stage for success or hardship in later phases of the system life cycle.

Here is the outline of the Systems Engineering process that we are following. Weve moved from the mission/stakeholder focus (#1) to the Concept of Opera=ons and Opera=onal Architecture focus (blue box #2). Now we focus on key (validated) acceptance criteria and the important opera=onal constraints. It is also important to understand the key external interfaces, e.g. external systems and work groups, and any other opera=onal constraints that might aect the scope of the system (some examples include: security, safety, performance, etc.)

This slide illustrates the feedback between the parts of the Systems Engineering process. Specically, once we have selected our concept (this module) and then dened the Concept of Opera=ons (next module), we can validate key opera=onal scenarios and stakeholder requirements. This can provide an early assessment of preferred concept(s) within an opera=onal context, thereby providing a valida=on of system mission and scope. Note that these processes and feedback are con=nuous.

We are now ready to generate and assess alterna=ve concepts for system implementa=on. This is the rst part of the process where we examine solu=ons of any kind. Prior to this, we have focused on understanding the full range of customers/stakeholders, and their individual needs, including external systems and workgroups.

 This graphic shows the process of transforming a set of quite general, top- level Stakeholder Expecta=ons into a detailed system design. As shown in the gure, at any level of renement, the design concepts ow out of the expecta=ons and requirements (from le] to right in the gure). But its itera=vewe get smarter as we begin to implement the expecta=ons/requirements, so we might see new requirements that we didnt realize we should have had. As we move from the top down in the gure, our requirements and implementa=on concepts become more rened and specic. This is recursive, in that we may learn we need to return to a higher level in the gure, because we forgot something, or because we now see a be`er way of doing things.

As a precursor to developing system concepts and decision criteria, how does the stakeholders world improve once the system is in place? From their expecta=ons, overall project need and mission, we establish a small set of key acceptance criteria cri=cal capabili=es that the system concepts we will be choosing from must meet. We some=mes call these the sacred few. Note the following deni=on of Stakeholder Requirements, from the INCOSE Systems Engineering Handbook, October 2011: Stakeholder Requirements Formally documented and approved stakeholder requirements that will govern the project, including: required system capabili=es, func=ons, and/or services; quality standards; and cost and schedule constraints.

 Acceptance Criteria are dis=lled from the stakeholder expecta=ons and requirements but they are generally at a higher level. Key or cri=cal acceptance criteria are a small number (typically three to ve, but could be as many as 7) that must be met by any solu=on (or solu=on concept). Think of this as a threshold of performance. Also note that it must be agreed upon by all par=es up front, in par=cular the key stakeholders and paying customer. Key Performance Parameters (KPPs) are more typically used as opera=onal measures of performance. That is, they are important metrics post-deployment. Some=mes they are used to assess that the solu=on is accepted by the customer/ stakeholder. In the la`er case, these would be measured pre-deployment.

As an example, President John F Kennedy dened three key acceptance criteria for the Apollo Project. While there were many detailed stakeholder requirements for the system, he said that if it accomplished these three, it would be considered a success. Failure to meet any one of them would be sucient to declare it a failure.

This IBM project manager notes the importance of key acceptance criteria in focusing the cri=cal and core capabili=es of the system.

10

John von Neumann (1903 1957) was a Hungarian-American mathema=cian who made outstanding contribu=ons to a wide variety of subjects, including set theory, func=onal analysis, quantum mechanics, computer science, game theory, and uid dynamics.

11

Lets examine a few case studies that demonstrate the importance of beginning with a good set of acceptance criteria. Each of these case studies begins with a brief overview of the background, describes the key acceptance criteria, lists challenges. Then each discusses results and factors leading to success. We will go over the Mars Explora=on Rover. The Polycom Soundsta=on and Handspring Visor examples are provided in full in the Explore sec=on. Please be sure to review them, as they provide excellent examples of how focused, key acceptance criteria can mo=vate a program along with close working rela=onships with the customers.

12

 The challenge and background for the Mars Explora=on Rovers (the Spirit and Opportunity) was to: search for and characterize rocks and soils that hold clues to past water ac=vity on Mars. They were to follow the water to learn whether condi=ons might have ever been suitable for life on Mars. These were NASAs largest, most complex, most ambi=ous planetary rovers up to that =me, and followed 1997s successful, but more modest Mars Pathnder lander and Sojourner mini-rover. (And also followed the highly-publicized, embarrassing failures of Mars Climate Orbiter and Mars Polar Lander in 1999) The results of this were separate launches in 2003, Spirit and Opportunity arrived safely on dierent sides of the Mar=an surface in January, 2004. Lets look at the goals, opportunity and some key success factors.

13

This slide contains the goals for each rover the key acceptance criteria or vital few cri=cal goals. Press pause and read on your own. Although there is a picture of the solu=on, note that the goals are stated in the language of the stakeholders.

14

 Both rovers have been successful beyond the most op=mis=c expecta=ons for them. Why? Key reasons for the astounding longevity of Spirit and Opportunity include a healthy conserva=sm in the systems engineering, including the opera=ons concepts. And a bit of luck with the teams assump=ons about the opera=ng environment! A key driver of the system life=me es=mates was the belief that the rovers solar arrays would soon become covered with Mar=an dust, eventually killing their source of electrical power. Indeed, Spirit and Opportunity were engulfed in one of the famous Mar=an dust storms. However, Mother Nature provided a surprising an=dote. Dust devils (small but powerful whirlwinds) occasionally come by and clean o the solar arrays, rejuvena=ng them! The rovers actually photographed some of them. And the Mars Reconnaissance Orbiter has seen them from space.

15

16

This second case study includes another example of one of the founders leaving to establish their own product this =me in the telecommunica=ons equipment industry. The acceptance criteria were straighqorward: simple to use rst class full duplex equipment with high quality sound.

17

Success factors include the teams uncompromising commitment to the original objec=ves: simple to use rst class full duplex equipment with high quality sound. Polycom also listened to Intel and other customers. And, nally, the team was mo=vated by the deadline of mid-August 1992 for the Telecon trade show demonstra=on. The full example materials are contained in the Explore sec=on.

18

We show Rodins sculpture, The Thinker here to emphasize the crea=ve nature of the Conceptual Design phase.

19

 The importance of this conceptual design phase step cannot be over- emphasized. No amount of architectural or detailed design rigor can make up for an inferior concept selec=on so it is important to take the =me and ensure a wide-range of concepts are considered. But, again, lets reiterate that understanding the stakeholders key acceptance criteria can play a big role in this selec=on process. So now lets review where the concept selec=on ts into the design phases, and how innova=on plays a key role in crea=ng possible concepts to consider.

20

 The upper le] box (Conceptual System Design) is where we are in the systems engineering framework. At this point, weve established a good understanding of the need for our system, and the stakeholders expecta=ons and requirements. Now well dene a (crea=ve) range of system concepts and then zero-in on the concept we will actually design. In this example, our need is to cross a river. We could brainstorm concepts for how to do this, and come up with the following candidates: a hot air balloon, a bridge, a boat or ship, a tunnel, and even a catapult. Keep in mind, its important to come up with a wide range of concepts to choose from, to ensure that a broad selec=on is considered and innova=on is encouraged. Once weve se`led on a preferred design concept, we will perform Preliminary Design, and eventually Detailed Design.

21

 And remember that the system concept design itself can be an itera=ve process. Many crea=ve processes take the form of alterna=ng periods of expansion and consolida=on. For example, consider a group of people who rst brainstorm a list of issues, then mul=-vote to pick the most important ones, then generate a large number of poten=al ac=ons, then choose a smaller set of ac=ons they will actually take, etc. In the course of evalua=ng a given set of op=ons, new op=ons will arise and should be added to subsequent versions of the matrix. And if =me and money allow, two concepts can be carried forward for risk reduc=on.

22

 We men=oned a crea=ve range of possible solu=ons. Lets consider Innova=on: Some=mes, the best solu=on to the stakeholder needs/expecta=ons is a straighqorward applica=on or extension of exis=ng solu=ons or work processes. However, when that wont do we may need to innovate. Knowing when and where to innovate can be as important as the innova=on itself. Successful innova=on requires a combina=on of: solid technical understanding, systems thinking, and crea=vity. Solid technical knowledge comes from domain knowledge, experience, and learning from the experts. Systems thinking comes from mastering Systems Engineering concepts and principles. The third aspect of innova=on, crea=vity, is some=mes mistakenly thought of as an exclusive, inherent, inborn trait. In reality, methods of crea=ve design can be learned, just like any discipline.

23

This slide describes one method for innova=on called: TRIZ (trees). That stands for the Theory of Inven=ve Problem Solving (TRIZ), from work by Genrikh Altshuller (Russian scien=st, 1926-1998). Pause and read this slide. There are references in the Notes. References: DaVinci and the 40 Answers, by Mark L. Fox. Idealized Design, by Russell Acko, et al.

24

 Another Approach to innova=on is described in the book: Idealized Design by Russell Acko, et al. Imagine a system that meets all the stakeholder expecta=ons perfectly. Then envision how to design and build such a system. Dr. Acko (1919 2009) was a pioneer in Opera=ons Research, Systems Thinking, and Management Science. He was a professor at the Wharton School, University of Pennsylvania. Reference: Idealized Design, by Russell Acko, et al.

25

 Here is an example of dierent concepts for the type of motor used in these early cars. In this case, the concepts were all implemented as a form of tes=ng and verica=on! Some interes=ng facts: In 1906, a Stanley Steamer set a record for the fastest mile in an automobile (28.6 secthis is over 120 mph!). The Fritchle electric car company of Denver pioneered ba`ery technologies, eventually allowing a range of 100 miles, in the early 1900s. The 1893 Duryea was the rst American gasoline- powered car.

26

27

Johnson's famed 'down-to-brass-tacks' management style was summed up by his mo`o, "Be quick, be quiet, and be on =me. He used a small, competent team, believed rmly in understanding the customer and also closely monitored cost. He ran Skunk Works by the Kelly's 14 Rules. Pause and read the slide and Notes for further details of the 14 Rules. Kelly Johnson's 14 Rules of Management The Skunk Works manager must be delegated prac=cally complete control of his program in all aspects. He should report to a division president or higher. Strong but small project oces must be provided both by the military and industry. The number of people having any connec=on with the project must be restricted in an almost vicious manner. Use a small number of good people (10% to 25% compared to the so-called normal systems). A very simple drawing and drawing release system with great exibility for making changes must be provided. There must be a minimum number of reports required, but important work must be recorded thoroughly. There must be a monthly cost review covering not only what has been spent and commi`ed but also projected costs to the conclusion of the program. Don't have the books 90 days late, and don't surprise the customer with sudden overruns. The contractor must be delegated and must assume more than normal responsibility to get good vendor bids for subcontract on the project. Commercial bid procedures are very o]en be`er than military ones. The inspec=on system as currently used by the Skunk Works, which has been approved by both the Air Force and Navy, meets the intent of exis=ng military requirements and should be used on new projects. Push more basic inspec=on responsibility back to subcontractors and vendors. Don't duplicate so much inspec=on. The contractor must be delegated the authority to test his nal product in ight. He can and must test it in the ini=al stages. If he doesn't, he rapidly loses his competency to design other vehicles. The specica=ons applying to the hardware must be agreed to well in advance of contrac=ng. The Skunk Works prac=ce of having a specica=on sec=on sta=ng clearly which important military specica=on items will not knowingly be complied with and reasons therefore is highly recommended. Funding a program must be =mely so that the contractor doesn't have to keep running to the bank to support government projects. There must be mutual trust between the military project organiza=on and the contractor with very close coopera=on and liaison on a day-to-day basis. This cuts down misunderstanding and correspondence to an absolute minimum. Access by outsiders to the project and its personnel must be strictly controlled by appropriate security measures. Because only a few people will be used in engineering and most other areas, ways must be provided to reward good performance by pay not based on the number of personnel supervised. Reference: See the book Skunk Works by Ben Rich.

28

Press pause and read this slide on the concepts for the twin-engine Army Fighter. The Government required bidders to use the Allison V-1710 engine, (a 12- cylinder, liquid-cooled, inline engine genera=ng about 1,300 horsepower). Be`er engines were under development at the =me, but the War Department had spent a great deal of money sponsoring the development and renement of the V-1710, and wanted to recoup its investment.

29

For the full story on the concepts generated for gezng a man on the moon, see the excellent ar=cle at: h`p://www.nasa.gov/centers/langley/news/factsheets/Rendezvous.html

30

For more, see: Voyager, by Burt Rutan and Jeanna Yeager, with Phil Pa`on. Knopf, New York, 1987.

31

All these innovators had common traits: They had a love and passion for their work They understood the state-of-the-art, and what the next logical step would be They were knowledgeablebut not necessarily the expert They were intellectually curious abut also used exis=ng technology where needed They were crea=ve brainstormers They were part of a team of talented people, all of whom were encouraged to contribute their ideas and concepts They viewed the whole problem, using systems thinking. They knew that you must understand the real need/opportunity; you cant view individual parts separately from the whole system; and that you must understand how the system will be used in opera=on They were all successfulbut not always, and they learned from their failures.

32

Lets go through a very simple example to illustrate some of the ideas we have discussed in the class thus far. First of all, there is the iden=ca=on of the need. In this case we have one key stakeholder with a very specic requirement. Lets accept that for the moment.

33

Here are ve possible concepts. Note the range of ideas and crea=vity: Some are more realis=c than others.

34

 Now we perform a simple analysis of each concept. What are the implied key acceptance criteria? Implied key acceptance criteria include: maintaining a separate oce and conference room, maintaining the exterior design, low cost, and aesthe=cally pleasing (that is, looks good). These would be the vital few, but of course, this would be veried by the key stakeholders.

35

 The op=ons for knocking down a wall, cuzng a hole in the wall and building a balcony were eliminated since they violated one or more of the implied key acceptance criteria. So we end up with two nal selec=ons -- either installing a door between the oce and the conference room, or installing a sliding par==on between the two. Then we discover there are also two addi=onal implied criteria the ability to mount a white board is maintained and the wall con=nues to hide the plumbing vents. In the end, it is the installa=on of the door that best meets the chosen decision criteria.

36

Further, in the true spirit of itera=ve development, decisions with regard to the concept or implementa=on approach spur greater renement and clarity on the part of stakeholders with regard to their requirements.

37

 A useful tool for helping us compare alterna=ve concepts at this early stage of design is a matrix, introduced by Stuart Pugh in 1991. The Pugh matrix separates the iden=ca=on of concepts from the criteria used to select between them. Remember that concepts that are compared here should all be capable of achieving the key acceptance criteria (vital few and that the selec=on criteria (rows on the Pugh matrix) should be the most important stakeholder requirements for the system. Keep in mind that the decision making process usually includes three components: alterna=ves, criteria, and rela=ve priori=es. The Pugh matrix in its original form did not include a sense of priori=es. This should not inhibit you from including priori=es if your analysis will signicantly benet from this. Why not just add up the + and to nd out the right answer? While a Pugh matrix is useful in helping us structure the concept selec=on process, Pugh cau=ons against adding the +'s and -'s together to get an algebraic sum. Remember, this is not math. The systems engineers have received stakeholder expecta=ons, but we do not know the full details of each solu=on. When using a Pugh matrix, the preferred concept should be selected "informed by" but not "determined by" the matrix.

38

In this example, the team has provided a forced weigh=ng (e.g. weights sum to 1). This can help avoid over-emphasis of any par=cular criteria (as compared to a pure assigned weight scheme such as 1-5). Also note that the criteria for this selec=on are traced to the stakeholder expecta=ons (rst column). This ensures that the selec=on criteria are important to the stakeholders, and helps if there are changes in expecta=ons later.

39

40

Con=nuing with our case study about Online Purchasing. These are Key Acceptance Criteria (vital few) stakeholder requirements (or even expecta=ons that are higher level than the previously-dened stakeholder requirements). All concepts considered in the next step (Pugh Matrix) MUST meet these. If the concept doesnt meet these, then it is not a valid concept to even consider.

41

These are the selec=on criteria for the Pugh Matrix (rows). These are criteria selected from the stakeholder requirements that will be used in the trade study (Pugh Matrix). Addi=onally these are also traced to the stakeholder requirements and expecta=ons, as dened previously.

42

 These 4 solu=ons were considered for mee=ng the needs, the vital few Key Acceptance Criteria. They become the columns of the Pugh matrix. Note that Solu=ons 3 & 4 have dierent shipping methods as well (the la`er is a big deal, when it comes to company investment vs. customer sa=sfac=on)

43

Here is the resul=ng Pugh Matrix

44

 This slide provides the ra=onale for selec=ng concept #4. The concept selected is the solu=on that sells products from both purchasing company and vendors, providing the customers with a large selec=on of products. Shipping is provided by some external shipping company (e.g. FedEx, USPS, etc). Some other considera=ons in looking at the Pugh: Solu=ons 1&3 provide links to vendor websites, which means availability, security, and customized reports could not be assured. Also, the solu=on would be hard to use for consumers since each vendor site is dierent (no consistency in the look and feel of the solu=on). Using a shipping company (#2, 4) provides our company with an alterna=ve to star=ng up a delivery organiza=on that would meet Department of Transporta=on regula=ons (improves =me-to-market and lowers risk). It also helped minimize maintenance of trucks, airplanes, etc. The Pugh Matrix did point out a risk in our solu=on shipping costs could be an issue, rela=ng to criterion: Inexpensive to develop, operate, and maintain By discovering the risk at this point in the solu=on, the risk can be worked to try to minimize it (via the risk management plan).

45

To summarize, in this module, weve covered the method to generate, evaluate and select system concepts by considering: key acceptance criteria, methods of innova=on to generate concepts, and concept selec=on.

46

47

This second case study includes another example of one of the founders leaving to establish their own product this =me in the telecommunica=ons equipment industry.

48

The acceptance criteria were straighqorward: simple to use rst class full duplex equipment with high quality sound.

49

But in this case the rst prototype was rejected by customers, funding was running out and the so]ware for full duplex sound did not yet exist.

50

What do you no=ce about the Polycom SoundSta=on? This conference phone denitely looks familiar; we have one in our Stevens Ins=tute conference room.

51

Thats probably because there are over a million SoundSta=ons in conference rooms all over the world! Interes=ngly enough Polycom acquired PictureTel a li`le over ten years a]er the start of the new company.

52

Success factors include the teams uncompromising commitment to the original objec=ves: simple to use rst class full duplex equipment with high quality sound. Polycom also listened to Intel and other customers. And, nally, the team was mo=vated by the deadline of mid-August 1992 for the Telecon trade show demonstra=on.

53

The rst case study is about the Handspring Visor. This slide provides background on the product and its inventors.

54

Next, are the objec=ves or acceptance criteria for the product. The original founders of the Palm Pilot including Je Hawkins and Donna Dubinsky - established these four objec=ves for their new product The Handspring Visor. They were looking to provide a lower priced product with more func=onality while maintaining an aggressive product development cycle 1/3 shorter than the previous Palm development cycle.

55

Third are the Challenges. For this product, the specica=on called for complete plug- and-play hot-swapability so that modules loaded and unloaded when you inserted or removed them without a crash or burp in the system.

56

Challenges are followed by the Results. What do you no=ce about the Handspring Visor? The Handspring Visor has the familiarity of the Palm Pilot all the same bu`ons in the same place so that there was no learning curve for Palm users to switch over to the Visor. And the Visor had the added func=onality of a Springboard expansion slot for peripherals previously thought to be a luxury for the original concept of a simple portable organizer. All of this and more for A Lower Priced Palm than Palm. [From interes=ng Pen Compu=ng ar=cle found at: h`p://www.pencompu=ng.com/palm/Reviews/visor1.html]

57

The eort paid o for the original Palm founders. The company IPO brought in $200M within months of the new product reaching the market. The company followed up with the Visorphone - a PDA and a cellphone in one - announcing the product on their website one week before Christmas 2000. [An interes=ng looking device that only an engineer could love]

58

And nally the factors that led to success. In this case a combina=on of the leadership, an experienced and commi`ed team, and a focus on the schedule.

59