From Proc. OPNETWORKS '98 Conference (CD-ROM), Washington, D.C.

, April 24-25 1998, paper 04

OPNET Simulation of Self-organizing Restorable SONET Mesh Transport Networks

TRLabs c/o Dept. of Electrical and Computer Engineering University of Alberta, Edmonton, Alberta, Canada

Demetrios Stamatelakis, Wayne D. Grover

Abstract
Todays digital fiber-optic telecommunication transport networks carry prodigious quantities of information: the equivalent of millions of phone conversations can be carried on a small fiber optic bundle. However, such capacity comes at a price: if not dealt with very promptly, a cable cut can have significant societal and economic consequences. We have devised a novel approach to the network restoration problem; prior to a failure network redundant capacity is preconfigured, using a distributed design protocol, so that any failure can immediately be restored. This method can realize restorable designs with low capacity redundancy and potentially very rapid restoration. The protocol executes in a peer-to-peer manner in the networks nodes eliminating the need for a centralized design mechanism. The protocol is simulated using OPNET modeler and a platform independent Java animation tool is used for realtime replay of the OPNET generated trace data.

1. Introduction
A mesh restorable network is composed of nodes, formed with Digital Crossconnect Systems (DCS), and node-tonode spans which transport digital multiplexed signals. A DCS permits the formation of connections between individual pairs of digital multiplexed signals (e.g. DCS3, STS-1, or STS-n) in a highly configurable manner. A mesh restorable network routes working traffic between node pairs using working paths formed by the DCS. Conventionally, a mesh network becomes restorable by distributing unconfigured spare capacity among the network spans. In the event of a span failure, the severed working traffic is rerouted with dynamically constructed alternate paths formed from the undedicated spare capacity. The pathset is calculated dynamically in response to each failure and differs substantially between failures. The pattern of crossconnections, for any node, varies significantly from failure to failure. The dynamic nature of the restoration pathsets is both an advantage and a disadvantage. Because of it, the restoration algorithm can make the most efficient use of the spare capacity to restore a specific failure; this is why mesh network require relatively little spare capacity compared to other restoration methods. However, time is required to dynamically calculate and form a restoration pathset so a conventional mesh network may not respond to a failure quickly enough for some applications [1]. With this motivation, we have developed a strategy for network operation that should make it possible to achieve much faster restoration times while retaining the desirable capacity efficiency of the mesh-restorable alternative. The method is based on the formation of pre1

configured cycles, called p-cycles, amongst the previously unconnected spare links (e.g., STS-1s or STS-3s) of a mesh-restorable network [2,3,4]. The advantage in speed is that only two DCS nodes have any real-time cross-connection workload for any given failure. Also, the end nodes only have to effect end-node traffic-substitution connections to ports which are identified by the planning process, before failure. The Cycle Preconfiguration Concept Figure 1 illustrates the ways in which an individual pcycle may be used for restoration. In Fig. 1(a) an example of a p-cycle is shown. In (b), a span on the cycle breaks and the surviving arc of the cycle is used for restoration. In (c) and (d), however, we see how the p-cycle can also be accessed for restoration of working paths that are not on the cycle. In fact, cases (c) and (d) are the more advantageous in general, because two restoration paths are obtained from each p-cycle for such failures. Further inspection of the example in Fig. 1 shows in fact that this particular p-cycle provides a single restoration path for nine span failures and two restoration paths for each of six other span failures (the latter are for each of those spans which are not on the cycle, but straddle it). These paths are immediately available and do not require any time to calculate or form.

a) A p-cycle, X

b) A span on the cycle fails, pcycle X contributes one restoration path

c) A span off the p-cycle fails, p-cycle X contributes two paths

d) A span off the p-cycle fails, p-cycle X contributes two paths

FIGURE 1. Use of a p-cycle in restoration

From Proc. OPNETWORKS '98 Conference (CD-ROM), Washington, D.C., April 24-25 1998, paper 04
A surprising recent finding is that with p-cycle restoration the spare capacity design takes little or no additional capacity than conventional mesh restoration [2,3]. This adds considerable practical motivation to pursue this approach further. In this regard one of the key issues to realizing a practical restoration scheme is the complexity of computing and deploying an always-current spare capacity plan to support the restoration mechanism. One way to compute the optimal p-cycle spare capacity and configuration plan is via a centralized Integer Program (IP) solution (see [2,3].) Centralized computation and control is, however, considered a significant practical burden, with the ever present risk of the centralized design configuration being out of date with the real network state. This is why we have sought a way in which the p-cycle configuration of the network can be made autonomous or self-organizing. Outline of paper We proceed as follows: The rest of Section 1 defines some needed terminology. Section 2 then addresses the desire to avoid dependence on centralized control for deployment and maintenance of a the p-cycle state for a network. This is done through a distributed self-planning protocol, which is the main contribution of the paper. Section 3 describes how this protocol was investigated using OPNET simulation. Definitions We use link to denote an individual bidirectional digital carrier signal between adjacent nodes. In general the link is whatever unit of capacity the DCS manipulates for transport management and restoration. For instance, a DS-3 or STS-1. A span is the set of all working and spare links in parallel between adjacent nodes, whether on one or several OC-n systems in parallel. A useful path is a path segment contained in a p-cycle which is related to a failure span in a manner that allows it to contribute to restoration of a given failure. many outgoing statelets but each outgoing statelet can have only one precursor. The DCPC involves nodes acting in either of 2 roles, that of a Cycler node or a Tandem node. The Cycler sources, and later receives, parts of the statelet broadcast pattern it initiates. Each node adopts this role in a round-robin fashion. While in this role it is temporarily in charge of the cycle-exploration process within the network as a whole. When not in the Cycler role, each node plays a Tandem-node role which mediates the statelet broadcast competition. To give an overview, the DCPC first allows each node to explore the network for p-cycle candidates that are discoverable by it. After completion of its role as Cycler, it hands off to the next node in order by a next-node handoff flood-notification. After all nodes have assumed the role of the Cycler once, each posts its best found cycle in a distributed network-wide comparison of results. The competition flood expands through the network as each node locally relays the statelet with the best cycle metric, or asserts its own best if and while the latter is still superior to anything else it has received notice of yet. Eventually, the globally best cycle candidate dominates everywhere. Upon thus learning of the winning candidate, the node which discovered this p-cycle, triggers its formation as a p-cycle. All nodes on the p-cycle update their local tables of restoration switching pre-plans to exploit the new p-cycle. The whole process then repeats without any central control, adding one p-cycle per iteration until a complete deployment of near-optimal pcycles is built. Thereafter, it continually adapts the pcycle set to changes in the working capacity layer.

3. OPNET implementation of the DCPC

The DCPC algorithm is a highly parallel and asynchronous process. There is no centralized control point; the distributed interactions of spatially separate nodes determine the final design. Although the rules which are applied locally at each node are relatively simple, the global interactions and behaviors of many concurrent instances of the DCPC rules are not always obvious. OPNET modeler was an enabling tool to develop the correct global behavior of this protocol. One of the more interesting parts of this simulation is that the DCPC is a state-based protocol; it operates on the basis of state information which is embedded on each link using statelets. OPNET, however, is packet driven. It was required that packets be used to simulate state based messaging. This was done using packets to transmit the statelet information, and registers to receive and store the statelet information. The OPNET model is constructed at three design levels: the network, node, and process level. Each level is discussed in the upcoming sections. A. OPNET Design at the Network Level Figure 2 contains a network level view of one of the networks for which the DCPC was simulated using OPNET.
2

2. Self-organization of the p-cycle state

Here we give an brief overview of the self-organizing strategy we have developed for the autonomous deployment and continual adaptation of the preconfiguration state. A more detailed description is given in [3]. The Distributed Cycle PreConfiguration (DCPC) protocol is based on distributed interaction through statelets. A statelet is embedded on each spare link and contains a number of state fields. Each logical link, as viewed by a node attached to it, has an incoming statelet and outgoing statelet. An incoming statelet arrives at a node on a link, and originates from the adjacent node connected through the link. Each outgoing statelet has an incoming statelet which forms its precursor. An incoming statelet is a precursor to an outgoing statelet if the incoming statelet was cause, under the protocol rules, for the creation of the outgoing statelet. One incoming statelet can be the precursor for

From Proc. OPNETWORKS '98 Conference (CD-ROM), Washington, D.C., April 24-25 1998, paper 04
Each node is connected to other nodes according to the topology of the network being simulated. Although not obvious in the figure, each span is allocated a certain number of working and spare links. Each simulated network also has an artificial Tracer node whose role it is to trace the actions of the other nodes and to print summary information at the end of the simulation run.

FIGURE 3. Node level view of the OPNET model

FIGURE 2. Network level view of the OPNET model

B. OPNET Design at the Node Level Figure 3 contains a node level view of one the network nodes, in this case a degree 3 node. The network node contains 5 user designed modules: Port card modules, Delay modules, a Poll_Timer module, a Xpt_Controller, and a DCPC module. The DCPC module encapsulates all of the statelet processing rules of the DCPC. The network wide interaction of the DCPC modules is what determines the final pcycle design. More detail follows at the Process level. The Port modules terminate a span. They receive and transmit the statelets which are the medium of communication for the DCPC. The information for a newly received statelet is stored within a register corresponding to the link on which the statelet arrived and a receive interrupt is raised. Similarly, when transmitting the statelet information is written into a ports transmit register (by an other module such as the DCPC) and the port is sent a transmit interrupt. The Delay module receives statelet events from its corresponding Port module and delays the transmission of the packet by a time equal to the sum of the insertion delay and the propagation delay. Insertion delay is based on a SONET 64 kb/s line-overhead byte and the propagation delay is calculated at 70% of the speed of light. The Poll_Timer module periodically polls the nodes port cards for incoming statelets and forwards these to the DCPC module. The Poll_Timer also tracks the number of statelets which are processed by the DCPC module in the current polling interval, and uses this to introduce a time delay due to computation. It also relays any statelet broadcast transmissions, required by the DCPC module, to the Port modules subject to the estimated computational delay.

The Xpt_Controller maintains a map of any crossconnections commanded by the DCPC. Basically, this modules only job is to make and unmake crossconnects, at the command of the DCPC, and maintain a record of the crossconnect matrix state. C. OPNET Design at the Process Level There is a Process view corresponding to each of the modules discussed in the previous section. Due to space constraints only the process view of the DCPC module will be discussed. The function of the DCPC module is to implement the statelet broadcasting rules of the DCPC protocol. Figure 4 shows the DCPC process state diagram. The DCPC protocol effects the following stages: exploration of the best p-cycle candidate by each node, competition among each node to discover who found the best cycle candidate, and construction of the best cycle by the winner of the competition. The exploration stage has each network node, in turn, assume the Cycler role. All statelets originate at a Cycler node. After initiation of statelet broadcast the Cycler node samples for incoming statelets. Because a cycle tends to contain more useful paths as it grows in size during the sampling interval, the score of a prospective cycle will generally improve with time under the actions of the Tandem nodes. The Cycler node therefore waits to observe the evolution of the best cycle possible. The Cycler maintains a record of the received statelet with the best score, and corresponding p-cycle. When the sampling time runs out, the Cycler node records the p-cycle candidate with the best score. It then suspends all statelet broadcasts, terminates its role as the Cycler, and emits a Cycler hand-off flood (a statelet with op-code handoff and the node name, on one link of each span.) The hand-off flood is relayed (once only by all nodes, without link persistence, with one copy in each span). When node n hears hand-off flood, n-1 it knows that it is its turn to become Cycler. After a round of cycler action by every node, the last node in the sequence knows all nodes are ready to take part in a network wide comparison of results. The purpose is to find the one globally best p-cycle candidate. This process is triggered by the initiation of a global
3

From Proc. OPNETWORKS '98 Conference (CD-ROM), Washington, D.C., April 24-25 1998, paper 04
broadcasts at the Cycler, which initiates the p-cycle candidate exploration. It also sets up a countdown timer which limits the Cycler to a predetermined sampling interval. The Sample state is unforced and represents the Cyclers low state as the Cycler pauses in this state while waiting for events to process. The Eval state is where the Cycler processes newly arrived statelets and evaluates the cycles they represent. The arrival of cycles with superior scores is recorded here. The CycDone state is entered when the sampling interval lapses and the Cycler role is released. If the node is the last node in the Cycler role assumption sequence then a competition flood is initiated; otherwise, the initialization of the Cycler hand-off flood takes place. The Tandem node logic for the processing of cycle exploration statelets is found in the Prc_Sig and Updt_Tx forced states. Prc_Sig processes newly arrived incoming statelets according to the rules of the DCPC statelet broadcast rules, while Updt_Tx evaluates all incoming statelets which are currently present on the node and updates the outgoing statelets so they are consistent with the DCPC rules. The DCPC rules are structured to favor the rebroadcast of high scoring incoming statelets which can loop back to the Cycler to form closed cycles. The scoring mechanism is designed so that those statelets which represent a route through the network which has a high potential to loop back to the Cycler node and to form a good p-cycle are given a high score. A detailed description of the DCPC broadcast rules is given in [3]. Between the processing of statelets the node settles into the unforced Idle state which represents the low state for this part of the state diagram. The logic for the construction of a p-cycle is contained in the StrtCycle, Wait, Finish, and ContCycle states. The node which starts the construction of the cycle does this in the StrtCycle state. It then settles in the unforced Wait state. Intermediate construction nodes receive and process the cycle construction request in the ContCycle state where they build their portion of the cycle and then relay the construction to the next node along the cycle. When the cycle construction loops back to the initiating node it completes construction in the Finish state. This state is also where the next iteration of cycle exploration and construction is initiated.

FIGURE 4. Process State Diagram of the DCPC module in the OPNET model

competition flood by the last node in the sequence of cycler phases. The initiating node broadcasts a single statelet, containing the nodes name and its best cycles score, on each span. When adjacent nodes receive such a statelet they compare the received best score to their local best score and relay the better of the two into all spans, along with the name of the node who is reporting the better cycle. If scores are equal, precedence is based on ordinal rank of the node names involved. Rapidly, only the single best score is present everywhere, and the node which found this candidate proceeds to initiate its construction. The node associated with the best candidate cycle scans for the first unoccupied spare link on the span joining it to the first node in the route field (which contains a record of the cycles route through the network) of the cycle to be constructed. It sends the route vector on to that node. Each other node along the route makes the specified spare-link cross-connection and forwards the route vector on to the next node in the prescribed cycle. Eventually this process returns to the node which initiated the cycle construction allowing it to form the final cross-connection, closing the cycle. All spare links used by the p-cycle are removed from eligibility in further cycle constructions. Upon completion of the cycle, the node which began the construction either initiates a Cycler hand-off flood commanding the first node to assume the role of the Cycler, or, if it happens to be the first node, it directly assumes the role of the Cycler node. This starts the next iteration of cycle construction. p-Cycles are constructed iteratively until no more cycles are feasible. Process states The Cycler logic is contained in the following states in Figure 4: InitCycler, Eval, Sample, and CycDone. The InitCycler state primarily sets up the outgoing statelet
4

4. Results
The OPNET DCPC simulation was run in 5 test networks which were optimally provisioned to be fully restorable to any span failure using p-cycle restoration with an Integer Linear Program (IP) p-cycle design formulation [2,3]. The test cases, therefore, represent operation with the minimum of spare capacity. The performance of the DCPC in these minimal capacity networks is in Table 1. The figure of merit is restorability which is the percentage of working links over all span failures which can be restored. Two types of restorability are given in the table: p-cycle is the restorability when using only p-cycle restoration, and 2-step is the

From Proc. OPNETWORKS '98 Conference (CD-ROM), Washington, D.C., April 24-25 1998, paper 04
restorability when first using all useful p-cycle restoration paths and then topping the effort up with conventional restoration within the remaining spare capacity. The restorability results are quite good considering the DCPC is a database-free self-organizing process being run in a minimally spared network.
Network Net1 Net2 Net3 Net4 Net5 p-cycle Restorability(%) 100 100 90.94 89.16 83.75 2-step Restorability (%) 100 100 97.16 97.68 95.44

TABLE 1: Performance in minimal-spare test networks

OPNETs statistic gathering facilities were not used in this simulation, as our main concern was in the performance of the p-cycle design which resulted at the end of a simulation run and not in the statistics of the run itself. To complement the OPNET simulation, we have also developed a Java based animation tool which is used to visualize the simulation trace files. The op-codes which the animator interprets describe generic network processes; they are not specific to the visualization of DCPC processes. For example, an op-code would be of the form turn node 5 bright blue and not show node 5 becoming the Cycler. This permits the animation tool to be used to visualize other concepts being worked on within our group; this is convenient as the utility may be used among multiple projects, eliminating the need to custom design an animation tool for each project. Also, as it is written in Java, animations can be conveniently played back on either PC or Unix systems. Figure 6 contains a screen capture of the animator.

Figure 5 is a simulation trace portraying overall operation of the DCPC process, using Net 1 for its simplicity of illustration. The plot shows the score of the best p-cycle candidate seen by any cycler, versus simulated real time. Six main regions appear in the plot, corresponding to the 5 p-cycles formed by DCPC for Net 1, plus a sixth iteration to realize the halting criterion (the existence of no suitable p-cycle candidate). Each of the larger regions correspond to the all-nodes cycler search, comparison, and construction of a single p-cycle. Inside each region, there are 10 individual cycler node explorations and nextnode hand-off floods (Net 1 has 10 nodes).
Cycle Score (numpaths / hopcount)
4 3.5 3 2.5 2 1.5 1 0.5 0 0 1 2 3 4 5 6 7

Global Operation of the DCPC Protocol

FIGURE 6. Screen Capture of the Java-based Animator

References
1. SR-NWT-002514: Digital cross-connect systems in transport network survivability, Bellcore, 1, January 1993.
Time (s) FIGURE 5. Best cycle score vs. time in Net1

2. D. Stamatelakis, Theory and Algorithms for Preconfiguration of Spare Capacity in Mesh Restorable Networks, M.Sc.Thesis, University of Alberta, Spring 1997. 3. W. D. Grover, D. Stamatelakis, Cycle-Oriented Distributed Preconfiguration: Ring-like Speed with Meshlike Capacity for Self-planning Network Restoration, Proc. ICC98, 1998. 4. D. Stamatelakis, W.D. Grover, "Distributed Preconfiguration of Spare Capacity in Closed Paths for Network Restoration," U.S. Patent Pending, July 11, 1997.
5

5. Concluding Discussion
The simulation described in this paper is perhaps quite different from many OPNET simulations by virtue of its parallelism and the state-based interaction model for this application. The OPNET modeler tool was extremely useful in simulation and development of the DCPC protocol. The alternative would have been the design and programming of a discrete event simulation from scratch.