A PEER-TO-PEER MESSAGE EXCHANGE SCHEME FOR LARGE SCALE NETWORKED VIRTUAL ENVIRONMENTS

Yoshihiro KAWAHARA, Hiroyuki MORIKAWA and Tomonori AOYAMA The University of Tokyo, 7-3-1 Hongo Bunkyo-ku, Tokyo, JAPAN {kawahara, mori, aoyama}@mlab.t.u-tokyo.ac.jp information in the virtual environment, dynamic state of entities, and so on. To tackle with this problem, many communication architectures have been proposed according to the requirement of applications. They are in most cases, categorized into two types: a peer-to-peer (SIMNET, NPSNET [3]) and a server/client architecture (RING [4], NetEffect [5]). In a peer-to-peer based architecture, entities send a unicast or multicast message to other entities when an entity state is updated. Though conventional unicast based approach yields O(N2) update messages for N entities and thus does not scale to many participants, multicast based approach scales well when combined with area partitioning scheme called cell based area of interest management. However, the unfortunate fact that multicast is not widely available on the Internet makes it difficult to use this scheme in the commercial applications such as MMORPG. In a server/client based architecture, on the other hand, every entity sends its update message to the central server, where the update message are replicated and forwarded to appropriate entities. Because of this centralized nature, servers can easily become bottleneck point or single point of failure in the system. Multi-server/client architecture or hierarchical architecture, which are often used in commercial online games, relieves the load of the servers by over-provisioning and load balancing only if the entire number of the entity in the system can be estimated beforehand and the service provider can afford to invest in costly servers. In addition, this schemes force developers to invest enormous money on hardware and spend much time on building complex software systems. Therefore it is too costly to develop large-scale Net-VE applications with conventional scale-up scheme. This situation can limit emergence of attractive applications. In this paper, we present a peer-to-peer communication architecture to create a scalable 3D chat application without costly servers or special infrastructure. In the next section, we present our scalable peer-to-peer message exchange scheme. A message exchange scheme enables entities to find other entities one after another without multicast infrastructure or centralized mechanism in steady state. Then our simulation result is presented in section three. Finally some conclusions and future work is given in section four.

Abstract - Massively multiplayer online games and other multi-user based networked applications are becoming a big business today. Such kind of technology has long been researched in the area called Networked Virtual Environments (Net-VE). With the recent deployment of broadband access technologies, the increase of the number of participants is putting pressure on the Net-VE systems. The conventional systems coped with the scalability problem by segmenting players onto multiple servers or multicast groups, limiting the number of users that could interact with each other. These schemes, however, force developers to invest enormous money on hardware and spend much time on building complex software systems. In this paper, we present a fully distributed communication protocol for large-scale networked virtual environments. As the entities in this scheme are directly connected to each other by unicast, no special infrastructure such as multicast or costly central servers is needed. Simulation results show that this architecture is scalable to the increase of the number of participants in the Net-VE system. Keywords - Networked Virtual Environments, peer-to-peer, online gaming

I. INTRODUCTION
Networked Virtual Environment (Net-VE) is a software system in which multiple users are able to interact with each other in the 3D virtual environment created on networked computers [1]. Technologies developed in this area are widely applied to various applications including military simulation, collaborative design, virtual meetings, distant education and online games. In particular, Massively Multiplayer Online Role Playing Games (MMORPG) are wining popularity and the entertainment industry is investing on them. For example, a famous massively multiplayer online role-playing game called EverQuest [2] holds over 400,000 registered users. In such a large-scale application, an enormous number of participants from heterogeneous environments interact with each other in one consistent virtual environment, in real-time. Therefore improving the scalability of the system is considered to be one of the most major research challenges. The dominant factor that affects the scalability of the system lies in the way to exchange the update messages, which include entities ID, location

II.

A PEER-TO-PEER MESSAGE EXCHANGE SCHEME

When designing a large scale networked virtual environment, the way to exchange update message among entities can be the most important issue because they are continuously exchanged while avatars are participating in the application. Update messages include avatars IDs, locations in the virtual environment, properties (name, color, and so on), event data (action data that some avatar is waving to someone), and chat text messages. They have different characteristics and should be treated accordingly. As described in section I, conventional message exchange schemes are based on multicast or client-server model. Multicast based approach scales well when combined with area partitioning scheme called cell based area of interest management. However, the unfortunate fact that multicast is not widely available on the Internet makes it difficult to use this scheme in the commercial applications such as MMORPG. Client-server based scheme are basically an over-provisioning based approach, which requires the precise estimation of participants and their behavior beforehand as well as large investigation on expensive hardware and complex systems. In our approach, we eschewed servers except for bootstrap servers in order to improve scalability without special investment or multicast infrastructure. Each participating entities connects to its neighboring entities, creating overlay network. Presence and status of participating entities are shared among appropriate entities with a support of the overlay network. As entities are directly connected to each other, this scheme can not only keep the latency smaller than server/client based scheme, but also be deployed without multicast infrastructure. More details of this scheme are described below. A. Bootstrap Method A bootstrap method is a consecutive procedure for an entity to participate in the virtual environment for the first time. The entity needs to acquire some information on other entities that are already in the Net-VE through the bootstrap procedure. The bootstrap procedure can be performed in several ways, such as bootstrap server based approach and host cash based approach. We have developed scalable host-cache based bootstrap for Net-VE [7]. Though detail mechanism of this bootstrap scheme is out of the scope of this paper, we assume this scheme is used for bootstrap method. B. Creating An Overlay Network After an entity acquired some other entities, entities establish unicast connections to other entities in order to create overlay network. To decide which entity to connect to, each participating entity classifies the surrounding entities into two groups called Active Entities (AE) and Latent Entities (LE). AE are entities that are regarded as in interaction and need tight communication. LE are entities that have the potential of interaction thus just the existence

and rough information is supposed to be enough. Various metrics such as Euclid distance in the virtual environment and network bandwidth can be used for this classification. Figure 1 illustrates the example classification where the nearest four entities are selected as Active Entities. For example, Active Entities of entity O are {A1, A2, A3, and A4} and Latent Entities are {B1, B2, , and B8}.
B1
B1 B2

B8
B8 A4

A1
A1

B2
B3

B7
B7

A4

A2

A2
B4

B3
Entity Connection Update Data

A3

B6
B6

A3
B5

B4

B5

Area Of Interest

Figure 1. The relationship among entities in the virtual environment

Here, adopting the nearest N entities as Active Entities are considered to be reasonable especially in the crowded circumstances. Imagine you are in a crowd at a cocktail party, everyone is chatting so aloud that you can barely talk to people just standing next to you. And if you find someone you want to talk to is standing a few meters away from you, it is reasonable for you to step forward to him and set him as your Active Entity. The approach of localizing the communication among participating entities to reduce communication bandwidth is called area of interest management (AOIM) and actually adopted in some Net-VE applications. In NPSNET project, the virtual environment is divided into cells to which multicast address are assigned. Communications among entities in the same cell are performed by multicasting. Another implementation shown in NetEffect uses a distributed server approach instead of multicasting to achieve extensibility and flexibility. These approaches require service provider to estimate the total number of users in the application and install sufficient resource on the system. One of the originalities in our approach is to perform this AOIM over the overlay network that is created by participating users. When creating an overlay network, the way to choose appropriate entity to connect to becomes important issue. This procedure is described below. After Active Entities are determined, each entity then establishes a direct connection with their Active Entities and exchanges their update messages. Necessary and sufficient

information such as location information in the virtual environment, status updates, chat message, and other application specific data are exchanged through this connection. As for information of Latent Entities, each entity degrades or compresses its Active Entities minimum required information (network identifier, location information, and so on) into the size small enough. Then they exchange it with other Active Entities. Since Latent Entities are defined as the entities whose information is not immediately needed or existence is unknown so far, rough or degraded information is assumed to be enough. When some Latent Entity is approaching another and more detailed information is required, then they are supposed to recognize each other as Active Entities to establish a direct connection. In this way, it is possible for each entity to find other entities one after another in a fully distributed manner. Since the update messages of Active Entities are directly exchanged with unicast, not only the latency can be kept smaller than server/client based scheme, but also this application can be deployed without multicast infrastructure.

result. Whole architecture of this simulator is shown in Figure 2.

Entity Initializer Simulates Server-Client Architecture

Set up entities' behaviour in the Virtual Environment

Simulates Peer-to-Peer Architecture C/S Entity Processor P2P Entity Processor

Update location for all the entities

Update location for Entity(I)

Calculate the distance among all the entities

YES

Get latest Information for Entity(I).AE(J) Get AE list for Entity(I).AE(J) J < # of AE for Entity(I) Set proper Active Entities for each Entity YES

Set proper Active Entities for each Entity

III.

SIMULATION

In this section, we show the scalability and performance of our scheme by simulation. One of the key issues for creating Net-VE over an overlay network is to keep optimal adjacency relationships of the entities in the virtual environment even if the number of participants in the application is enormous. This simulation is based on the scheme shown in section II and performed as follows: 1. Entities share a common map of the virtual environment beforehand. 2. Each entity holds a list of neighboring entities. (AE List) 3. When an entity enters a virtual environment for the first time, a bootstrap server tells several candidates to the entity and then the entity picks up proper entities to connect to. 4. Entities can walk about in the virtual environment at random. 5. Each entity communicates with the entities in its AE List and exchanges their ID, location information and AE List itself. 6. After arranging the information of the neighboring entities, each entity selects the nearest N entities as new Active Entities. C. Simulator Architecture In order to know the behavior of the entities in the virtual environment, we have developed simulation software that simulates the behavior of the entities in the virtual environment. In this simulator, every entity prowl in a virtual environment randomly and exchanges its location information and network ID among other entities based on conventional server/client based scheme and proposed peer-to-peer based scheme. The entities behavior and their relationship are illustrated graphically shown as a simulation

I < # of Entities

NO Display style Graphical Evaluator

STDOUT

STDOUT

Graphical Renderer

Figure 2. Simulator Architecture

Table 1.
Field Name ID Location AEList[K] Num_of_AEs Timestamp

Parameters of entities
Description Entitys identifier in the network Entitys location in the VE Cached data of Active Entities The number of Active Entities Elapsed time since the entity entered the VE

X,Y,Z ID, Location, Distance, Timestamp

1) Entity Initializer Entity Initializer is a module where every entitys instance is created. Here, every entitys locus and speed are determined so that their location can be determined uniquely by the timestamp. This makes it possible to compare different message exchange schemes under the same condition. This module also simulates the mechanism of bootstrap servers . 2) C/S Entity Processor C/S Entity Processor is a module where client/server based message exchange scheme is simulated. In the server/client model, all the entities in the virtual environment inform a message exchange server of their current location and IDs. Then the server replicates the message and transfers them to the appropriate clients. In this model, C/S Entity processor processes the entities as follows: 1. Calculates the entities location at time T 2. Calculates the distance among all the entities to generate a distance table 3. According to the table, inform the nearest N entities of each entity as Active Entity 3) P2P Entity Processor P2P Entity Processor is the module that simulates the peer-to-peer based message exchange scheme described in section 4. That is all the entities walk about in the virtual environment in an autonomous manner and exchange their Active Entity List among their neighbors. The detailed algorithm is as follows: 1.Updates the location of entity I (Entity(I)) at time T 2.Inquire its neighboring entities (Entity(I).AE(J)) about its current location 3.Then inquire the neighboring entities about the information of their neighboring entities (Entity(Entity(I).AE(J).ID).AE(K))) 4.Calculates the distance among the entities based on the information acquired in the previous step and sort them in order of the distance 5.Set the nearest N enteritis as new Active Entities D. Simulation Results The simulation result can be displayed in two ways. The first one is graphical mode. In the graphical mode, the relationships among entities are visualized in a window. This mode helps to understand the qualitative behavior of the entities in the virtual environment. The second one is text mode. By comparing the detailed status information of the entities under the different message exchange schemes, quantitative analysis can be performed in this mode. 1) Qualitative Analysis Simulation results are shown in Figure 3. Each figure shows the simulation results that were performed by changing two parameters: the number of total entities in the virtual environment and the number of Active Entities each entity holds.

Figure 1. Simulation Results (Graphical)

The left part of each figure represents the relationships among entities in the client/server based architecture. And

the right part represents the one in the peer-to-peer based architecture. Here, square dots represent the entities in the virtual environment. And if an arrow points from entity A to entity B, it means that entity B is one of the Active Entities for entity A. What could be concluded from Figure 1 is that, qualitatively, the relationships among entities in client-server based model and peer-to-peer model resemble each other: that is the peer-to-peer based scheme is able to build almost the optimal adjacency relationships given in the client-server models. 2) Quantitative Analysis In order to evaluate our system quantitatively, now we define Consistency as equation 1 to measure how accurately this scheme establish adjacency relationships among entities by comparing the difference of the Active Entity List of entities between peer-to-peer based simulation and client/server based one. Now Consistency is defined as follows:
Consistency = 1 N

IV.

CONCLUSIONS AND FUTURE WORK

In this paper, we presented a fully distributed message exchange scheme that enables entities to find other entities one after another by exchanging update messages only with their immediate neighbors in the virtual environments. Our simple simulation result shows that this message exchange scheme can build consistent virtual environment in a scalable manner. We believe this communication architecture enables developers to create attractive large-scale Net-VE application without special infrastructure for scaling the system. At the moment, we are implementing 3D chat software SCAMPI using this message exchange scheme and going to show its scalability on the Internet.

REFERENCES
[1] Shinghal, S., and Zyda, M., Networked Virtual Environments, ACM Press, 1999. [2] EverQuest: http://everquest.station.sony.com/ [3] Macedonia, M., Zyda, M., Pratt D.R., and Barham P.T., Exploiting Reality with Multicast Groups: A Network Architecture for large Scale Virtual Environments. Proceedings of IEEE Virtual Reality Annual International Symposium, 1995. [4] Funkhouser, T.A., RING: A Client-Server System for Multi-User Virtual Environments, Computer Graphics (1995 SIGGRAPH Symposium on Interactive 3D Graphics), pp. 85-92., 1995 [5] Tapas K. Das, Gurminder Singh, Alex Mitchell, P. Senthil Kumar, Kevin McGee: Developing Social Virtual Worlds using NetEffect, Proceedings of the 6th Workshop on Enabling Technologies Infrastructure for Collaborative Enterprises, pp. 148-153, 1997 [6] Gnutella: http://www.gnutelliums.com/ [7] Matsumoto, N., Kawahara, Y., Morikawa, H., Aoyama T.: An Interest-Oriented Bootstrap Method for Large-Scale Networked Virtual Enviornments ,IEICE Technical Report IN2002-, July, 2002. . (in Japanese) [8] Kawahara, Y., Matsumoto, N., Morikawa, H., and Aoyama, T., A Distributed Communication Architecture for Networked Virtual Environments, Technical Report of IEICE, IN-2001-229, 2002. (in Japanese)

Q (i )
i =1

P (i )

(1)

N: the number of entities in the virtual environment P(i): the number of entities that exist in the AE List of entity i in both peer-to-peer and server/client model Q(i) : the number of the Active Entities for entity i Figure 4 shows the Consistency of the two schemes. This simulation result shows that the peer-to-peer based message exchange scheme makes it possible for entities to establish almost consistent relationship (>0.9) compared to centralized scheme (optimal), independently of the number of the participants in the virtual environment. That is, the fully distributed message exchange scheme described in Section 2 scales even if the hundreds or thousands of people are participated in a single virtual environment.
Consistency
1

0.9

Consistency

0.8 AE=5 0.7 AE=7 AE=10 0.6 AE=20

0.5 20 50 100 200 500 1000 2000 5000 10000

The number of entities in the virtual environment

Figure 4: Consistency and the number of entities in the virtual environment