ARTICLE IN PRESS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Information and Software Technology xx (0000) xxxxxx www.elsevier.com/locate/infsof

57 58 59 60 61 62 63

A distributed execution environment for shared java objects

Nadia Erdogan*, Yunus Emre Selcuk, Ozgur Sahingoz
Department of Computer Engineering, Electrical-Electronics Engineering Faculty, Istanbul Technical University, 80686 Ayazaga, Istanbul, Turkey Received 4 March 2002; revised 11 September 2003; accepted 11 September 2003

64 65 66 67 68 69 70 71 72

Abstract This paper discusses the implementation of a distributed execution environment, DJO, which supports the use of shared Java objects for parallel and distributed applications and provides the Java programmer with the illusion of a network-wide shared object space on loosely coupled distributed systems. DJO supports shared objects through an implementation of multiple reader/single writer write-invalidate DSM protocol in software, providing the shared memory abstraction at object granularity. Object distribution and sharing are implemented through the replication mechanism, transparently to application. The system enforces mutual consistency among replicas of an object. The main benets of DJO are enhanced availability and performance due to the replicated object model and easier application design, as the underlying software takes care of distribution and memory consistency issues. q 2003 Published by Elsevier B.V.

73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112

Keywords: Replication; Consistency management; Concurrency control; Distributed object system; Java

In a distributed system, processes and users require access to shared data for meaningful cooperation. Traditional high performance computing often uses message passing to share information in parallel applications on distributed environments [17,22]. With this approach, the developer has the advantage to control communication occurring in these applications and can adjust it to avoid unnecessary latency which can effect overall performance. This control, however, implies a responsibility to plan every communication detail. As a result, application development becomes extremely difcult and time consuming [10]. As an alternative to message passing, the shared memory paradigm offers an attractively simple programming model for application development. It has been extended to distributed memory machines where it is usually referred to as distributed shared memory (DSM) [16,18], DSM provides a logical single address space which is transparently partitioned over a set of physically distinct nodes. Programming with this model removes the need to explicitly consider communications, as this is handled by the DSM service, thus reducing application

* Corresponding author. E-mail addresses: erdogan@cs.itu.edu.tr (N. Erdogan), selcukyu@itu. edu.tr (Y.E. Selcuk), o.sahingoz@hho.edu.tr (O. Sahingoz). 0950-5849/$ - see front matter q 2003 Published by Elsevier B.V. doi:10.1016/j.infsof.2003.09.017

INFSOF 443531/10/200312:18BELLA85331 MODEL 5

EC

TE D

1. Introduction

development complexity. Transparency is achieved by applying consistency models that take care of propagating changes to the shared information in the distributed environment. This paper presents a distributed execution environment for shared Java objects (DJO) with its underlying software architecture [19,20]. DJO is an effort to provide the DSM abstraction at object granularity. It supports the use of shared Java objects for parallel and distributed applications, thus providing the Java programmer with the illusion of a network-wide shared object space on loosely coupled distributed systems. During DJO design, we have adapted some of the techniques used in software based implementations of DSM systems to address the problems for sharing objects in a distributed object system. The design of DJO is based on the replicated object model. Replicated objects are local copies of logically shared Java objects. They allow for object methods to be invoked locally, eliminating the need to contact the remote object. DJO software architecture implements the multiple reader/single writer write-invalidate protocol to provide a memory consistency model which guarantees a consistent view of shared object state that matches programmer expectations. DJO distributed execution environment consists of a collection of interconnected nodes, where each node

PR

ARTICLE IN PRESS
2 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 N. Erdogan et al. / Information and Software Technology xx (0000) xxxxxx

TE D

executes a number of basic components that collectively form the runtime system. The runtime system is responsible for cashing replicas of objects and also for propagating invalidation requests and local updates of an object to the other replicas. An application interested in accessing a shared object contacts its local runtime system, which obtains a replica of the object and maps it into the applications address space. The application can then access the object through local method calls. A critical region encloses method invocations that modify the object state. Applications on other nodes cannot access the replicas of the object until the control of the critical region is released. Entry and exit to a critical region is specied by special operations which alert the system, that, in turn, noties all other nodes that maintain a replica of the shared object through invalidation messages on an entry and saves local changes on object state on an exit. DJO is implemented in Java, and shared objects are a direct extension of Java objects. The main extension is that object state is distributed, which is implemented without modifying the Java runtime environment. The target application domain of the system includes all types of collaborative applications, including groupware and program development environments. The layout of the body of the paper is as follows: Section 2 briey discusses requirements and related work. Section 3 describes the read-replication algorithm on which the design of DJO is based, followed by the consistency model and coherency protocol that have been implemented. Section 4 presents the design of the software architecture. Section 5 provides detailed information on the user interface of the system. Section 6 presents some performance measures, and nally Section 7 concludes the paper with directions for future research.

2. Requirements and related work

A distributed application allows an arbitrary number of running components (programs) across any number of address spaces and nodes, cooperating as peers in implementing the application. Components of an objectbased distributed application communicate and cooperate through shared objects. Shared objects are ne-grained, fully encapsulated passive entities that consist of private internal state and a set of operations by which that state can be accessed and modied. Each shared object is an instance of a class. Users of shared objects have several requirements:

Location transparent access Efcient access High performance High availability Fault tolerance

Implementations of DJO makes active use of replication to address the performance and availability requirements mentioned above. The system transparently maintains replicas of shared Java objects in the address spaces of applications that access them. A replica encapsulates a local copy of the replicated object state and offers an interface to manipulate this state. Access to a replica is through local method invocations. Thus, by allowing a local rather than a remote copy to be accessed, replicas decrease access times, as delays in retrieving and updating the object are minimized. Furthermore, replication of shared objects also improves performance as concurrent access to replicas on several nodes become possible. Replicas improve availability by making it possible for applications to progress even when one or more replicas become temporarily unavailable. Fault tolerance is achieved by ensuring that object data is kept consistent. Loss of any one replica does not result in updates being lost, if other replicas have copies of the same updates. When a system concurrently maintains several copies of the same object, their consistency should be guaranteed. Different levels of consistency exist. Recent studies show that the overhead of coherence protocol messages limits the performance of a system considerably [8,11]. Weak consistency protocols delay moving data and consistency information until absolutely required to do so by the memory model, thus reducing communication requirements signicantly. DJO implements release consistency, which is a variant of weak consistency protocol, through the writeinvalidate policy to lower the cost of coherence. DJO further tries to hide communication cost by using multithreading to overlap communication with computation. Over the past decade, many software DSM systems such as Ivy [15], Midway [5], Munin [6] and TreadMarks [2] have been implemented on top of message passing hardware or on network of workstations. Early systems were pagebased, just providing a linear memory of bytes and relied on the underlying memory management unit and operating system software for implementation. Recent advances in DSM systems are providing increasingly more support for the sharing of objects rather than portions of memory. Several object systems that have been described in the literature provide facilities similar to those provided by DJO. Generally, they are based on new language/operating system solutions whereas DJO is implemented on Java Virtual Machine, thus providing a portable execution environment independent of any hardware/software. Some of these object systems are outlined below. Munin [6] is a shared-variable DSM and offers the application developer a set of consistency maintenance algorithms. Each algorithm is targeted to a group of shared variables with specic characteristics. Indigo [14] is a software DSM system that is implemented on a PVM [22] platform. CVM [12] implements a shared memory system based on a user-level library, which provides the developer with primitives for synchronization and allocation of shared

169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224

INFSOF 443531/10/200312:18BELLA85331 MODEL 5

EC

PR

ARTICLE IN PRESS
N. Erdogan et al. / Information and Software Technology xx (0000) xxxxxx 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 3 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336

3. Read-replication algorithm Several algorithms have been developed for implementing the shared data model [10]. These algorithms can be classied according to the strategy they use to distribute the shared entities: migration or replication. Migration implies that only a single copy of an objects exists at any time, so the object should move to the requesting node for exclusive access. Replication, on the other hand, allows for multiple copies of the same object to reside in several address spaces. Replication reduces the cost of read operations that do not alter the object state, since it is possible to simultaneously execute such operations locally on multiple nodes. However, operations that modify the state of the shared object become more expensive because its replicas have to be invalidated or updated to maintain consistency. If the ratio of reads over writes is large, this extra expense may be offset. DJO implements the Read-Replication: multiple readers/single writer [1] strategy to distribute shared objects. Object invocations are divided into two types: read accesses that do not change the state of the object and write accesses that modify the object. DJO allows for either: multiple nodes with read-only replicas of the shared objectthe object is replicated on two or more nodes and each node has read access to its copy while none of the nodes have write access, or one node with a read/write replica-no two nodes may be modifying separate copies of an object at the same time.

3.1.1. The consistency model: release consistency Consistency models dene the order in which accesses to shared memory are seen by interested parties. A number of different models have been proposed in the literature such as sequential consistency, causal consistency, PRAM consistency, weak consistency, release consistency, and entry consistency [1]. Consistency models can be divided into two major categories: strict models and relaxed models. In general, the stronger the consistency level, the higher is the latency its implementation produces [3]. Strict consistency models order each access operation individually, while relaxed models combine a set of operations and impose an order on these sets. With strict consistency models, every write access results in an invalidation/update operation on all replicas. However, not all applications require to see all updates to a shared object, in which case relaxed consistency models perform better [3]. Relaxed models allow replicas to become inconsistent and perform coherence operations at specic user dened synchronization points within the program. The overall effect is reduced network trafc. One disadvantage, however, is that the programmer is expected to label programs with synchronization operations that act to separate conicting sets of access operations. DJO implements Release Consistency which is a variety of relaxed consistency model [9] to maintain the coherence of shared objects. In a release consistent environment, object invocations that modify the object (write accesses)

INFSOF 443531/10/200312:18BELLA85331 MODEL 5

EC

TE D

PR

memory. The user can choose between a set of given consistency models. Disom [7] presents an object oriented framework for object sharing. Classes are made sharable by inheriting from particular super-classes, which dene methods that facilitate the exchange of updates among interested nodes. The framework provides an implementation of entry consistency. Mushroom [13] supports a framework of replicated objects written in Java. Communication is based on events, and programmers use events to describe a change that effects the replicas of an object. Orca [4] supports object replication and migration with strong consistency guarantees. However, all objects in this system must be written in a special Orca language. Based on the observation that large and complex systems cause overhead that offsets their benets, we have tried to keep the design of DJO as simple as possible. Instead of supporting multiple consistency models, as many other systems do, a single consistency model, one that we think suits groupware applications the most, is provided. We aim at an execution environment that is easy to use and does not burden the application developer with unwanted/unneeded abstractions and functionality. However, if needed, more complex services could be built on top of those provided.

On a request for a read access on a shared object that is currently not local, the system communicates with remote nodes to get a read-only replica of the object into the callers address space, which may only be possible if no writable replica exists in the system. A request for a write access to a shared object that is either not local or for which the local node has no write permission may only proceed after all replicas at other nodes are invalidated, thus preventing them from being accessed. The read-replication algorithm is consistent because a request for a read access always returns a replica with an internal state that reects the results of the most recent write access. 3.1. Consistency maintenance As stated in Section 2, replication improves performance by allowing concurrent access to replicas at multiple nodes. However, if the concurrent accesses are not controlled, they may be executed in an order different from that expected. A memory is coherent if the value returned by a read access is always the value that was expected [1]. Thus, to maintain the coherence of shared objects, a mechanism that controls or synchronizes the accesses is necessary. A consistency model denes a specic kind of coherence provided by the system while a coherency protocol is responsible for managing object data so that the required level of consistency is actually provided.

ARTICLE IN PRESS
4 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 N. Erdogan et al. / Information and Software Technology xx (0000) xxxxxx

require synchronization. Two synchronization operations are dened to differentiate between entry and exit to critical regions which enclose write accesses to shared objects: an acquire operation tells the system that a critical region is about to be entered and a release operation indicates that a critical region has just been excited. DJO requires the programmer to determine, for every operation, if it modies the object to which it applies, and to explicitly use these special operations on the target object to signify the beginning and end of a set of object invocation which result in modications on object state. In the current implementation, each shared object that is to be write accessed is associated with a synchronization variable, actually a lock, to enforce concurrent accesses to happen sequentially. The system meets the following conditions, with the support of the underlying coherency protocol, to achieve release consistency. It does not allow an acquire access to perform until an up-to-date copy of the shared object associated with the lock is brought into the address space of the requesting application. It is ensured that no replica is accessible by any other application even in read access mode while an application is active in a critical region. After a release operation on an object has been completed, all read access requests of other applications are not allowed to proceed until they all receive an updated replica into their address space. 3.1.2. The coherency protocol The coherency protocol is responsible for managing shared objects so that the conditions to provide release consistency are satised. The main issue is the synchronization of write accesses to objects in such a way as to insure no application reads old data once a write access has been completed on some replica of the object. There are two approaches: write-update and write invalidate [21]. Writeupdate broadcasts the effects of all write accesses to all nodes that have replicas of the shared object. This approach is usually considered to be expensive since a broadcast is needed on every write. In the write-invalidate scheme, on the other hand, invalidations are sent and modications are requested. The basic concept is to send an invalidation message to all nodes that hold a replica before doing an update. Applications ask for updates as they need them. DJO adopts the write-invalidate protocol and implements it in the following way:

the other applications from reading object data that is out of date. Any application that requests to read access an object is not allowed to if a writer to the object already exists. With this scheme, updates are propagated only on requests for object access. Therefore, an invalidated replica on a certain node is updated only when it receives an access request. Meanwhile, several updates to the object may have taken place on other nodes before communication for state transfer becomes necessary. Furthermore, it should also be noted that, several invalidated replicas with different state data may exist in the system at any time, but all valid replicas will hold the same object state data.

393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408

4. Replicated objects software architecture Fig. 1 shows the software architecture that implements the replicated shared object design discussed in the previous sections. A runtime system executes on each node and is responsible for replica management, and propagation of updates and invalidation requests. After completing initialization tasks, the system starts listening to a given address for incoming request messages and spawns a new thread to fulll each request. Applications interested in shared objects contact their local runtime systems. A main concern in the design has been to minimize communication overhead, which brings a major limitation

409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447

An object accessed in read-only mode can be replicated on multiple nodes. When an application requests a write access to an object through an acquire operation, a multicast message is sent to all nodes that have a read-only replica of the object to invalidate them, and all invalidations are acknowledged before acquire can proceed. This approach prevents

EC

TE D
Fig. 1. Overview of replicated object software architecture.

PR

448

INFSOF 443531/10/200312:18BELLA85331 MODEL 5

ARTICLE IN PRESS
N. Erdogan et al. / Information and Software Technology xx (0000) xxxxxx 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 5 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560

on performance. We try to hide communication cost by using Java multithreading to overlap communication and computation whenever possible. All data structures and management routines are thread-safe. The software is structured in layers. At the lowest level is the Communication Management Module. It is responsible for providing elementary communication mechanisms. Next, the Coherency Protocol layer provides routines to perform elementary actions such as transferring object state, invalidating replicas, etc. The Synchronization Module at the same level implements a locking mechanism to guarantee exclusive access to shared objects. Finally, at the highest layer, the Consistency Manager is responsible for implementing the consistency semantics. 4.1. Communication management module The Communication Management Module provides local and remote communication support. Communication between system components takes place through messages that carry requests and resulting management information between local and remote threads. A single message format has been used for simplicity. Each message contains information about the request type, the unique object name, IP addresses identifying either the source or destination of the requesting node, and other elds containing management information required by the request type. Each request interprets those elds differently, extracting information relevant to its type. Communication is carried out through UDP sockets and RMI [23] calls. UDP sockets are used for internal messaging between system components. RMI calls are used to contact remote nodes, for example, when transferring the state of an object. Multiple threads are created for communication via sockets. Threads resole the received messages and take the necessary actions, which could be generating a reply message providing the information asked to another thread, or calling a remote method of a runtime system resident on another node. Threads use a particular port to listen to replies to their requests from other threads. Fig. 2, shows the interactions in the communication module. 4.2. The coherency protocol

The coherency protocol is implemented by three management components: memory management component, ownership management component, and distribution management component. 4.2.1. Memory management component Memory on each node in the system is effectively a part of the total shared object space and continues to exist as long as the node takes active part in the DJO environment. The Memory Management Component is responsible for the management of replicas that are resident in the shared address space of each node. It resolves a given object

descriptor, its system-wide unique name, into a location in this address space and processes access operations. Each application has direct access to a local memory region which is private to the application (heap area), and an indirect access to a shared memory region which is the shared address space on the node where storage is allocated for object replicas. An invocation on a shared object requires the copying of the replica from the shared memory region into the local memory region, thus enabling local method calls on the object. The shared memory region on a node is organized into two Java Hashtables, ObjStateTable and ObjInfoTable, that hold, respectively, current state and management information for the replicas that are located on that particular node. ObjStateTable treats objects as if they had no specic type, dealing with them as being of type Object, root class of all classes in Java. Thus, shared objects of any type can be put in the table. As the type information is lost, a cast to the correct form has to be performed while retrieving the object from the table. A valid replica of an object is retrieved from the ObjStateTable on local or remote demand. Modication on state of a replica, either resulting from local invocations or transferred from a remote node, is stored in this table. ObjInfoTable contains management information that is used to locate and transfer object state and to invalidate replicas. An entry in the table consists of the following elds: Name. Applications that share an object call it by the same name, which is unique across all replicated copies. The name of an object is a user dened character string. The hash tables ObjStateTable and ObjInfoTable are actually manipulated using unique object names as the keys. Probable owner. Each shared object in the system has a single owner. The owner of an object is a unique node which is either the creator of the object or holds the only writable copy of the object. The probable owner eld either points to the true owner of an object or provides a hint through which the true owner of an object may be located. Copyset. Copyset is a set of nodes that hold valid replicas of an object. It is maintained by the owner node. Status. A replica of an object may be in one of the following states at any time. invalid: an invalidated replica is present on the node. readable: a valid replica is available and is not locked. writable: a locked replica is available and its other replicas are invalidated.

EC

INFSOF 443531/10/200312:18BELLA85331 MODEL 5

TE D

Fig. 3 shows the resulting state transitions of an object replica on different operations. Memory Management Component also maintains a data structure, ServerList, which keeps information about the set of nodes that are currently participating the system. The ServerList carries addressing information that is needed to contact the run time systems resident on those nodes.

PR

ARTICLE IN PRESS
6 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 Fig. 3. State transition of an object replica. N. Erdogan et al. / Information and Software Technology xx (0000) xxxxxx 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672

Fig. 2. Interactions in the communication management module.

INFSOF 443531/10/200312:18BELLA85331 MODEL 5

4.2.2. Ownership management component The Ownership Management Component is responsible for the administration of the ownership of replica. DJO adopts the dynamic-distributed scheme for ownership management [10]. The owner of an object is not xed and moves around the system, hence introducing the problem of locating it. To address this problem, the concept of probable owner has been used. The probable owner need not own an object but is responsible for tracking its current owner. Each node associates a probable owner with each replica. This information is just a hint. If the relevant eld contains the address of the local node, then that node is the true owner of the object. Otherwise, it provides the beginning of a sequence of nodes through which the true owner may be located. Requests are forwarded until the true owner is reached. The probable owner eld is updated whenever an up-to-date copy of the object is received. The main activity

4.2.3. Distribution management component Distribution Management Component has the functionality required to send and receive replicas of objects from remote nodes and is closely linked to the Communication Module. A transfer operation accepts an objects name and returns its current internal state and management information. It should be noted that, we assume the receiving application to already possess the required operation code for the object (its class declaration) and thus, transfer only object state. Otherwise, the class declarations would need to be dynamically loaded, too. The Distribution Management Component is also responsible for sending out invalidation messages before a write access to an object is permitted to proceed. Copyset is

EC

TE D

PR

of this unit is to identify the location of the true owner of an object.

ARTICLE IN PRESS
N. Erdogan et al. / Information and Software Technology xx (0000) xxxxxx 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 7 729 730 Semantics Registers a new shared object Removes local replica Creates a local replica and permits read-only access Provides a locked local replica and permits caller to proceed into its critical region Updates local replica and releases the lock 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784

passed along with the ownership property as a node becomes the new owner of an object. Invalidation messages are sent to all nodes on the copyset to prevent access to old data. The copyset at different nodes may be different. However, the owner node knows precisely the set of nodes that currently hold a replica of the object. 4.3. Synchronization module Most parallel applications need to use some kind of synchronization mechanism to order and control concurrent access to shared objects in cases where access results in modication of the object state. DJO provides locking primitives to achieve mutual exclusion. When an application possesses a lock on an object, it can be sure that it will be granted exclusive access to a recent copy of the object. Only the owner of an object can lock it. Therefore, ownership of the object has to be gained before applying for a lock. A lock is assigned to an application following its request to acquire an object and is set free on its next release request on that object. It is the programmers responsibility to take care so that an object does not remain locked forever. 4.4. Consistency manager The Consistency Manager implements the release consistency semantics described in Section 4.3, which includes access and synchronization operations on shared objects. The access operations are directed to the underlying coherency protocol. Synchronization operations are, in turn, mapped to corresponding operations provided by the synchronization module. The Consistency Manager also supports acquire and release operations on shared objects. The acquire operation allows entry into a critical region by acquiring a lock from the synchronization module. The operation species the target object as the parameter of the acquire process and, with this information, the synchronization module can determine if it conicts with another action. For example, some other application may already be holding a lock on that particular object. The release operation ends the critical region by notifying the coherence protocol to save the object state in the nodal replica. Local modications made to the object copy in the local memory region as are transmitted to the corresponding entry in the ObjStateTable. After the operation completes, the lock is released.

Table 1 The user interface Interface call Int djo.CreateObject (Objectdsmobject,StringObjName) boolean djo.RemoveObject (String ObjName) Object djo.ReadObject (String ObjName) Object djo.AcquireObject (StringObjName) boolean djo.ReleaseObject (Objectdmsobject,StringobjName)

5. User interface

Naming as well as shared memory space allocation is achieved with the CreateObject call. An application that wishes to introduce a new shared object uses this call to register it within the system. The caller provides the unique name of the object, a string, with the class data, in the parameter list. Any other application on a remote node may share access to the object if it knows this unique name (identity) of the object. The local runtime system consults local and remote directories to see if an object with the given identity already exists. If an object with the given name already exists, the call returns with a nonzero value. If not, an entry is allocated for the object in the ObjInfoTable, where management information is stored. The caller node gets the initial ownership of the object. Similarly, a new entry is allocated in the ObjStateTable, where a copy of the object state is stored. Thus, any remote application that requests access to the object can have a replica of it retrieved from the ObjStateTable. A replica created on a node remains there until it is explicitly removed. 5.2. Remove a shared object boolean djo.RemoveObject (String ObjName) This call removes a local replica and discards all relevant information if it is not in locked state. No action is taken for

Shared Java objects in the distributed environment become accessible regardless of their location through a small yet powerful set of calls that is supported by the underlying software architecture. They implement the release-consistent memory model, allowing new objects and replicas to be created, and replicas to be updated,

INFSOF 443531/10/200312:19BELLA85331 MODEL 5

EC

TE D

PR

Int djo.CreateObject (Object dsmobject, String ObjName)

5.1. Create a shared object

invalidated or destroyed. They return either the desired resulting information or an error code. The user should check the return code to insure correct program ow. The user interface consists of the cells listed in Table 1. The functionality of each is described in detail in the following sections and Appendix A includes sample code that demonstrates their use in program context.

ARTICLE IN PRESS
8 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 N. Erdogan et al. / Information and Software Technology xx (0000) xxxxxx

its replicas on remote nodes. The call returns true on success and false on failure. 5.3. Access a shared object in read-only mode Object djo.ReadObject (String ObjName)

effect. Readers are expected to contact the system by issuing the read-object call before each access (or, after some user-denable period of time) to nd out if they still hold a valid copy of the object and, if not, to have it updated. 5.4. Receive exclusive access to the shared object

841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896

An application issues a ReadObject call before an attempt to read access an object. The ReadObject call brings a copy of the object into the local memory region of the caller so that local method invocations can be carried out. The runtime system returns the requested object if a valid replica in the unlocked state already exists on the shared memory region of the calling node. If not, one of the following cases will be true. (i) It may be the rst time a node which is not the owner of an object is trying to access it. Therefore, the local node does not posses a replica yet. (ii) An invalid replica exists on the node. That is, an access to the object is requested after it has been invalidated due to coherence actions. This means that changes have been made to the state of the object, and these changes should be transmitted to the local replica before it may accessed again. As a replica already exists, the probable owner may be contacted. Case i. A query message is broadcast to all remote nodes on the ServerList, asking for the object. Nodes that hold a valid replica respond to this message. No reply received implies an access error to a nonexistent object, in which case the cell return null. Otherwise, the rst node that responds to the broadcast message is contacted. This node is either the true owner of the target object or, is not the owner but holds a valid copy. In the rst case, if the owner has restricted access to the object by locking it, null is returned. Otherwise, the owner node adds the identity of the caller to the copyset directly. In the second case, the call initiates an operation that locates the true owner and has the identity of the caller added to its copyset. Next, object state and management information are transferred from the owner node to the shared memory region of the caller node. The call proceeds to update the probable owner eld with the current owner of the object, changes object state into readable and returns the object to the caller. Case ii. Starting with the probable owner eld of the object, a sequence of nodes is scanned until a node which is either the true owner of the object or holds a valid copy is located. Next, the call proceeds following the steps described in case i. After the ReadObject call returns with success, the caller receives a copy of the object into its local memory region and may issue local method invocations, which are expected to access the object state in read-only mode as changes in object state will have no global

Object djo.AcquireObject (String ObjName) Through the AcquireObject call, an application gets permission to enter a critical region. If the call returns with success, the caller obtains both the most recent version of the object copy into its local memory region and a lock on the object that ensures exclusive access. From then on, the caller may issue several updating method calls on the object. The call proceeds to consult the true owner of the target object to ask for its ownership if the caller is not already its owner. The current owner rejects the request if it already holds a lock on the object. If this is not the case, the current owner puts the object into invalid state and removes itself from the copyset. This eliminates the invalidation it would have otherwise received from the new owner of the object. It hands over ownership after updating the probable owner eld of the object with the identity of the requesting node and returns object management information along with object state data, which is requested only in the case when a valid replica is not present on the callers address space. Upon receiving the ownership of the target object, the new owner node locks the object, putting it into writable state. However, all valid replicas of the object need to be invalidated before allowing access to the object to guarantee that no conicting copies exist. Invalidation messages are sent to all nodes on the copyset of the object. The call waits for invalidations to be acknowledged to prevent the existence of old copies. After the invalidation process completes, the call purges the copyset as those nodes no longer possess a valid replica, inserts the identity of the local node into the copyset of the object and returns the object to the caller, which can then proceed into its critical region. Now, method invocations that result in changes in the object state may be issued. It should be noted that these modications take place on the copy of the object located in the local memory region of the application and are not transmitted to the replica that resides on the shared memory region of the node (and therefore will have no global effect) until the critical region is exited through a ReleaseObject call. 5.5. To release exclusive access and save modied object state boolean djo.ReleaseObject Dsmobject, String ObjName) (Object

INFSOF 443531/10/200312:19BELLA85331 MODEL 5

EC

TE D

PR

ARTICLE IN PRESS
N. Erdogan et al. / Information and Software Technology xx (0000) xxxxxx 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 9 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970

ReleaseObject call ends activity in a critical region after transmitting local modications of object state to the only replica of the object on the shared memory region. The call proceeds to verify that the caller is true owner of the object and returns error if there is a conict. Next, the entry allocated to the object in the ObjStateTable is replaced with the new state data provided by the caller. Finally, as the object is unlocked, the status eld in the ObjInfoTable is updated to readable to reect the current state. Normally, a lock is released upon an applications request to exit a critical region. Therefore, the application should take special care so that the object would not remain locked forever. The system has no timeout mechanism to automatically release locks.

Read-only access. Time taken to retrieve a (i) locally available or (ii) remotely available valid replica into the address space of the caller process. Receive exclusive-access. The total time taken to receive valid replica in locked state after obtaining ownership of object. It includes the time spent to send invalidation messages to ve valid object replicas and waiting for arrival of acknowledgement messages. Release exclusive-access. Time taken to transmit local object state to nodal object replica and to unlock object. The results presented suggest that DJO can perform well. They were achieved without any specic optimization and it should be possible to make signicant improvements on them. We further plan to run the system on a larger number of nodes to test it for scalability and to observe effects on its performance.

6. Performance evaluation 7. Conclusion and future work We have performed simple tests to evaluate the performance of DJO system on a LAN environment of ve INTEL architecture machines (Pentium II 350 MHz with 128 MB RAM) connected through a 10 Mbps Ethernet and running Windows 98 SE. JDK 1.3 was used in the testing environment. The data was obtained by subtracting the values returned by System.currentTimeMillis( ) method, which was invoked just before and right after DJO calls. The tests were repeated 10 times for 15 different objects and the arithmetic average of the measurements are reported. Table 2 presents the resulting performance gures, in milliseconds. The tests included a simple application that involved concurrent access, either read or write, to shared Java objects by all participating nodes. Objects had a size of approximately 100 bytes. Each node acquired mutual exclusion before a write-access to an object and released it after completion. The measured gures account for the times spent in system calls, as described below. Initialization. Time taken to initialize a local runtime system before a node participates the DJO environment. Most of the measured time is consumed by the initiation of the Java rmiregistery process. Creating new object. Time taken to create an object with a unique name and register it within the system. This paper presents the design and implementation of a distributed execution environment for shared Java Objects. Object sharing is implemented through the replication mechanism. Each application requesting access to a shared object receives a copy into its local address space. Object access is through invocation of methods provided by the object interface. The system enforces mutual consistency among replicas of an object transparently. A prototype implementation of the system has been completed. DJO distributed execution environment has several advantages. One main advantage of the system is that it simplies distributed application design. Programmers may concentrate on the application logic and receive the benets of a distributed execution environment through a simple yet powerful user interface, as issues related to distribution, access and consistency of shared objects are handled transparently by the underlying system. Programmers do not need to write any extra code for object sharing and distribution. The replication strategy employed in the distribution of shared objects not only decreases access times and increases parallelism, thus resulting in better application performance, but also makes the implementation of fault tolerant system possible. Another advantage of the system is its being based on JVM, which ensures portability across a variety of hardware and software platforms. DJO design can be enhanced with certain features. The current implementation does not employ a memory management algorithm for replicas on a node. Since the size of the shared memory region on a node is physically bounded, only a limited number of replicas can be held. The system rejects requests when this limit is reached. However, a better approach would be to remove replicas to free some memory space. As future work, we intend

971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008

Initialization Creating new object Read-only access Local replica Remote replica Receive exclusive-access Release exclusive-access

Operation

Table 2 Measured performance gures of DJO

INFSOF 443531/10/200312:19BELLA85331 MODEL 5

EC

Elapsed time (ms) 2160 30 17 38 215 84

TE D

PR

ARTICLE IN PRESS
10 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 N. Erdogan et al. / Information and Software Technology xx (0000) xxxxxx

to study memory replacement algorithms, concentrating on locating objects that are particularly suitable for removal. Another feature which we wish to include in our work is extending shared objects with support for security and persistence. The creator of the object can specify a security policy during registration and the system can control access accordingly. Providing support for persistent shared objects would also ease code development signicantly.

status djo.CreateObject (pepe, pepe); } catch (Exception e) {} if (status ! 0) {error Sample code to read access the shared object pepe on a node i: int age; int height; pepe1 new penguin( ); pepe1 (penguin) djo.ReadObject (pepe); age pepe1.getAge( ); height pepe1.getHeight( ); Sample code to read/write access the shared object pepe in a critical section on a node j: boolean status; pepe2 new penguin( ); pepe2 (penguin) djo.AcquireObject (pepe); pepe2.setAge (3); pepe2.setHeight (40); status (penguin) djo.ReleaseObject (pepe2, pepe);

1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120

Appendix A Class declaration for shared Java object penguin:penguin.java public class penguin java.io.Serializable { private int age; public int height; private String nickname; public String name; implements

Sample code that creates a shared object named pepe: pepe new penguin( ); pepe.setAge(1); pepe.set.Height(25); try {

INFSOF 443531/10/200312:19BELLA85331 MODEL 5

public penguin( ) { age 0; height 20; } public void setAge (int i) { age i; } public int getAge( ) { return age; } public void setHeight (int i) { height i; } public.int getHeight( ) { return height;}

EC

TE D

References
[1] S.V. Adve, K. Gharachorloo, Shared memory consistency models: a tutorial, IEEE Comput. 29 (12) (1996) 66 76. [2] C. Amza, A.L. Cox, S. Dwarkadas, P. Keleher, H. Lu, R. Rajamony, W. Yu, W. Zwaenepoel, Treadmarks: shared memory computing on networks of workstations, IEEE Comput. 29 (2) (1996) 1828. [3] H. Attiya, J. Welch, Sequential consistency versus linearizability, ACM Trans. Comput. Syst. 12 (2) (1994) 91122. [4] H. Bal, F. Kaashoek, A. Tanenbaum, Orca: a language for parallel programming of distributed systems, IEEE Trans. Software Engng. 18 (3) (1992) 190205. [5] B. Bershad, M. Zekauskas, W. Sawdon, The midway distributed shared memory system, Proc. 38th IEEE Int. Compcon Conf. (1993) 528 537. [6] J. Carter, J. Bennet, W. Zwaenepoel, Implementation and performance of munin, Proc.13th ACM Symp. Oper. Syst. Principles (1991) 152 164. [7] M. Castro, P. Guedes, M. Sequeira, M. Costa, Efcient and exible object sharing, Proc. Int. Conf. Parallel Process. (ICPP96) 1 (1996) 128 137. [8] A.L. Cox, P. Keleher, H. Lu, R. Rajamony, W. Zwaenepoel, Software versus hardware shared memory implementation, Proc. 21st Annu. Symp. Comput. Architect. (1994) 106117. [9] S. Dwarkadas, P. Keleher, A.L. Cox, W. Zwaenepoel, Evaluation of release consistent software distributed shared memory on emerging network technology, Proc. 20th Annu. Symp. Comput. Architect. (1993) 144 155.

PR

ARTICLE IN PRESS
N. Erdogan et al. / Information and Software Technology xx (0000) xxxxxx 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 [10] A. Judge, P.A. Nixon, V.J. Cahill, B. Tangney, S. Weber, Overview of distributed shared memory, Technical Report TCD-CS-1998-24, 1998. [11] P. Keleher, A.L. Cox, W. Zwaenepoel, Lazy consistency for software distributed shared memory, Proc. 19th Annu. Symp. Comput. Architect. (1992) 13 21. [12] P. Keleher, C.-W. Tseng, Enhancing software DSMs for compiler parallelized applications, Proc. 11th Int. Parallel Process. Symp. (IPPS97) (1997). [13] T. Kindberg, G. Coulouris, J. Dollimore, J. Heikkinen, Sharing objects over the internet: the mushroom approach, Proc. IEEE Global Internet (1996) 6771. [14] P. Kohli, M. Ahamad, K. Schwan, Indigo: user-level support for building distributed shared abstractions, Concurr.: Pract. Experience 8 (10) (1998) 129. [15] K. Li, Ivy: a shared virtual memory system for parallel computing, Proc. of ICPP88 2 (1988) 94 101. 11 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 [16] K. Li, P. Hudak, Memory coherence in shared virtual memory systems, ACM Trans. Comput. Syst. 7 (4) (1989) 321359. [17] Message passing interface forum, http://www.mpi-forum.org/ [18] L. Nitzberg, V. Lo, Distributed shared memory: a survey of issues, IEEE Comput. 24 (8) (1991) 52 60. [19] O.K. Sahingoz, Implementation of a DSM system based on read replication algorithm, MSc Thesis, Istanbul Technical University, 1998. [20] Y.E. Selcuk, Implementation of a distributed shared memory system, MSc Thesis, Istanbul Technical University, 2000. [21] P. Stenstrom, A survey of cash coherence schemes for multiprocessors, IEEE Comput. 23 (6) (1990) 1224. [22] V.S. Sunderam, PVM: a framework for parallel distributed computing, Concurr.: Pract. Experience 2 (4) (1990) 315339. [23] Sun Microsystems, Java remote method invocation specication, Technical Report, Sun Microsystems, Mountain View, CA, 1996.

INFSOF 443531/10/200312:19BELLA85331 MODEL 5

EC

TE D

PR