The Metropolis Model: A New Logic for System Development

Rick Kazman University of Hawaii and Software Engineering Institute/CMU

Rick.Kazman@gmail.com

Trends

Two trends are transforming our world:

the rise of the socio-technical ecosystem the business trend towards service orientation

The Rise of the Network

Yochai Benklers The Wealth of Networks:

we are in the midst of a radical transformation of how we create our information environment this transformation is restructuring society; models of production and consumption Benkler calls this commons-based peer production

Service-Dominant Logic

Service industries account for 55% of economic activity in the United States Businesses have moved from a goodsdominant view, to a service-dominant view In this new view customers are seen as cocreators of value

A Sea Change

These changes are much more than just a shift from goods to services. They are a reframing of the purpose of the enterprise and its role in value creation. They are creating new phenomena, e.g. super-linear growth in projects, emergent behaviors in systems,

Implications for Software Engineering

Existing system development models are all broken in a crowdsourced world. These models assume:

projects have dedicated finite resources management can manage these resources requirements can be known software is developed, tested, and released in planned increments system growth is sub-linear

A New Model

We suggest that a new model of the software lifecycle is needed: the Metropolis Model. This model helps us think about system creation that is commons-based and peer produced:

E.g. MySpace, YouTube, Facebook, Hi5, Wikipedia, Orkut, and Craigslist

Metropolis Model Characteristics

1. 2. 3. 4. 5. 6. 7. 8.

Mashability Conflicting, Unknowable Requirements Continuous Evolution Focus on Operations Open Teams Sufficient Correctness Unstable Resources Emergent Behaviors

1. Mashability

Old: we make systems difficult to tear apart, for historical, intellectual property and security reasons. New: mashability is a core capability of commons-based peer produced systems

mashups are accepted practice web-services are proliferating (e.g., Google Maps APIs). open source projects make it easy to interoperate; they are nonrival

2. Conflicting, Unknowable Requirements

Old: iterative lifecycles accept requirements change, but in any given iteration they try to collect and analyze those requirements. New: requirements in a peer-produced system emerge from its individuals, operating independently

requirements are never knowable in any global sense they will inevitably conflict, just as the requirements of a citys inhabitants often conflict

3. Continuous Evolution

Old: systems evolved in planned releases New: systems are constantly changing

resources are non-centralized and so a peerproduced system is never stable one can not conceive of its functionality in terms of releases any more than a city has a release parts are being created, modified, and torn down at all times

4. Focus on Operations

Old: focus on development/maintenance New: focus on system operations as a core competency

Google, eBay, Amazon .... as reliable as a public utility implies high availability, scalability, and seamless evolution

5. Open Teams

Old: closed team of developers who work from a consistent set of requirements New: volunteer projects and decentralized production processes with no managers

teams are diverse with differing, sometimes irreconcilable, views even for-profit companies need to lead and motivate knowledge workers

6. Sufficient Correctness

Old: Completeness, consistency, and correctness as goals New: Perpetual beta

collaborative tagging does not depend on widespread agreement among the taggers Wikipedia never claims to be complete or even fully correct

7. Unstable Resources

Old: Resources and their deployments are planned New: Applications that are peer-produced are subject to the whims of the peers

large numbers tend to ameliorate the actions of any individual despite the lack of guarantees, unstable resource pools have resulted in significant computational achievements

8. Emergent Behaviors

Old: Goal was deterministic behavior New: Emergent behavior is normal and desirable

SecondLife, eBay, and MySpace have seen complex human and system behaviors emerge that were beyond the vision and intent of their creators

A New Logic for System Development

Logic: the formal principles of a branch of knowledge; a particular mode of reasoning viewed as valid or faulty -- Merriam-Webster Old models (and their principles) are inadequate What are the principles upon which we can build a new model?

Metropolis Model Structure

Open Source Social Networking

Kernel

Periphery Periphery: Developers Prosumers Masses Masses: Masses: Customers Users

Metropolis Model Principles

1. 2. 3. 4. 5. 6. 7.

Egalitarian Management Bifurcated Requirements Bifurcated Architecture Fragmented Implementation Distributed Testing/V&V Distributed Delivery/Maintenance Ubiquitous Operations

1. Egalitarian Management

Management of projects is not top-down Work is not assigned; people undertake the work they choose Status and rights are earned

2. Bifurcated Requirements

Requirements are bifurcated into: 1) kernel service requirements

deliver little or no end-user value focus on quality attributes and tradeoffs slow to change contributed by the peer network deliver the majority of the function and end-user value rapidly changing

2) periphery requirements

3. Bifurcated Architecture

Architecture is bifurcated into: 1) kernel infrastructure

designed by a small, coherent team highly modular provides the foundation for the achievement of quality attributes enabled by and constrained by the kernel otherwise unspecified

2) peripheral services

4. Fragmented Implementation

The majority of implementation (the periphery) is crowdsourced to the world

using their own tools, to their own standards, at their own pace

Implementers of the kernel are close-knit and highly motivated

5. Distributed Testing/V&V

The kernel must be highly reliable

this is tractable sufficient correctness is the norm

The reliability of the periphery is indeterminate

6. Distributed Delivery/Maintenance

The kernel must be stable:

seldom changing, and when it does change it must be backwards compatible constant stream of independent, uncoordinated releases no notion of a stable system state

At the periphery, perpetual beta is the norm:

7. Ubiquitous Operations

Traditional systemseven highly reliable onescould be taken down occasionally for maintenance and upgrades. Metropolis systems are always on, even when upgrading

upgrades are not ubiquitous

System must scale with the number of users.

Implications of the Metropolis Model

For some projects there are no implications There will always be projects that are:

high security highly proprietary safety critical too burdened by legacy

Implications - 2

However, there is an increasingly important class of projects to which the Metropolis model applies. So, the question is:
what should crowdsourced projects do differently from the way projects are built and managed today?

Implications 3.

Main implication: separation of kernel and periphery Consequences on:

Project management Requirements Architecture Testing/V&V Delivery/Maintenance Operations

Implications: Project Management

A Metropolis project implies a virtual organization. To manage such a project the periphery must share in its success. The project must:

be (largely) self-governing and self-adaptive have a clear task breakdown structure, but a minimum of hierarchy and bureaucracy have technology for communication and coordination

Implications: Requirements

Requirements are primarily asserted by the participants, not elicited from their users. Requirements emerge through their emails, Wikis, and discussion forums.

So such forums must be made available, and the participants should be encourage to participate

Implications: Architecture

The kernel architecture must be built with a small, experienced, motivated team who focus on modularity

enable the parallel activities of the periphery.

There should be a lead architect (or a small number of leads)

manage project coordination have the final say in matters affecting the kernel

Implications: Implementation

The project can guide, but not command, implementation. This implies the need for

a focus on communication, negotiation, and leadership. good tutorials and examples an attention to usability (simplicity, learnability) of the kernel

Implications: Testing/V&V

Kernel must be tested heavily and validated

it is the fabric that holds together the system keep the kernel small have frequent builds

This can, however, be made tractable

Implications: Delivery/Maintenance

Delivery mechanisms must be created that accept:

incompleteness, multiple versions, and incompatibilities

of the installed base as the norm.

Implications: Operations

Operations must focus on high reliability of the kernel

accepting the fact that the periphery will often fail

Monitoring and control mechanisms need to be created

so that bugs in the periphery can not undermine the kernel.

Final Thoughts

Lifecycle models arise in reactions to ambient market conditions. The Metropolis Model is formally capturing a phenomenon that is already occurring: the dramatic rise of commons-based peer production. If an organization wants to take advantage of this it needs to understand it, and needs to understand how to foster it.

Really Final Thoughts

We are now working on creating a lifecycle model that realizes the principles of the Metropolis Model.

Thank You

thoughts? comments? questions?

Backup Slides

Financial Services Examples

E*Trade: http://www.businessweek.com/technology/content/nov2006/tc200 61113_151490.htm Bank of Tokyo-Mitsubishi: http://soa.sys-con.com/node/312331 Bank-Anywhere.com: http://www.bank-anywhere.com/96825/ Forbes Magazine: http://www.forbes.com/technology/2008/07/27/it-shadow-gamestech-cio-cx_dw_0728shadow.html Blogspot: http://businessmashup.blogspot.com/2008/04/mashups-infinancial-arent-just-for.html

Financial Services Mashup Providers

JackBe: http://www.jackbe.com/resources/demos.php Kapow: http://www.kapowtech.com/newsandevents/p r20070319.aspx