A New Persistence Framework

Distributed Simulation

for

Parallel

and

Haiquan Qiao, Rusheng Ju, Ge Li, and Kedi Huang

School of Mechatronics Engineering and Automation,
National University of Defense Technology,
Changsha, Hunan Province, China, 410073
qiao_haiquan@tom.com
Abstract
Persistence is used not only in load-balancing systems but also in simulation
checkpoint and restart, and it is very important in parallel and distributed
simulation. SPEEDES Persistence Framework (SPF) is a layer of the proposed
Standard Simulation Architecture (SSA), and Boost Serialization Library is a library
of Boost for persistence. This paper first describes these two framework, and then
applies them in a parallel and distributed system. The user friendliness and
performance of SPF and BSL are evaluated and analyzed, at last the conclusion is
given that BSL is a more suitable persistence framework for parallel and distributed
simulation.

1. Introduction
Standard Simulation Architecture (SSA) is proposed by Dr. Steinman, aims at high
reusability of parallel and distributed simulation(PADS) software components [4].
SSA is a layered architecture. The interfaces between layers are standardized to
promote portability and interoperability with other models. The SPEEDES [2] and
HLA are combined in WarpIV, which is currently evolving into Standard Simulation
Architecture [4].
Persistence is an important layer in SSA. Persistence is used in PADS to automate
checkpoint/restart and enable load balance algorithms to migrate objects to
different processors. The SPEEDES persistence framework is a prototype
implementation of SSA persistence layer and is proposed to be a standard [4]. In a
preliminary performance test in Qnet, a simulation of a queuing network, the
SPEEDES persistence framework caused qnet running in approximately half the
speed of qnet without persistence framework [7].
The main work of this paper is that a C++ template based persistence framework
BSL(Boost Serialization Library) is tested in a PDES application, it shows almost no
performance lost. The SPEEDES persistence framework is also tested in the same
application for comparison reason. The result shows that BSL has great superiority
over SPEEDES persistence framework. To introduce our work, the SPEEDES
persistence framework and Boost serialization library are introduced first, then the
test and evaluation are presented, conclusion is in the end.

2. SPEEDES Persistence Framework (SPF)

SPF is designed and implemented by Ronald Van Iwaarden [7]. The design is based
on the following consideration: The primary obstacle of saving and restoring C++
objects is fixing the pointers. If a C++ class A contains a pointer points to a class B
instance, and this instance is not restored to exactly the same physical memory
address, the pointer in the restored class A object will point to incorrect data and
this can cause system crash. To solve this problem, all the pointers in class A
instance must be adjusted to correct memory address after class A and B instances
are restored.
A global persistent database is used in SPF to keep persistent memory blocks and
persistent pointers information. There are two primary data structures in the

persistent database: MemoryTree and PointerTree. MemoryTree contains all

persistent memory blocks information and PointerTree contains all persistent
pointers information. Using user interfaces provided by the framework, persistent
memory blocks and persistent pointers can be automatically stored in or deleted
from the database. In simulation initialization stage, user must plug in all the
persistent classes to the persistence database through PLUG_IN_<TYPE> functions,
this will
store the class constructor/destructor function pointers and class sizes to the
persistence database, and the database allocates a unique ID for each plugged in
class.
To pack an object into a buffer, all of the persistent pointers in the object and all
memory blocks to which the pointers point must be found recursively, stored in a
temporary memory and pointer tree, and all memory blocks and pointers along with
other information such as ObjectTypeId, cardinality and tags are written to the
buffer. Reconstructing an object from a buffer is a more
complex procedure. The framework first constructs all objects using their default
constructor, then reads data of objects from the buffer, and calculates the correct
values of persistent pointers and modifies the pointers to correct values.

2.1.

User Interfaces

SPEEDES persistence framework provides users with macros to make a class T

persistent:

These two macros generate the following class definitions and functions.
1) Persistent smart pointer class T_PTR and T_PTR_PTR. They behaves like T* and
T**, with additional actions of registering themselves to the persistent database. In
a persistent class, T_PTR and T_PTR_PTR should be used instead of T* and T**.
2) Memory allocation/deallocation functions. They should be used instead of global
new/delete operators.

These functions call global new/delete internally and register/unregister memory

blocks.
3) Plug in function which stores the constructor/destructor function pointers and
class size in the persistent database.

4) Functions for generating persistent buffer and reconstructing objects.

5) Registration functions. Register regular memory segments and pointers with the
persistent database.

2.2.

Persistent Object Factory

Frequently allocating and de-allocating objects may slow down the programs very
much. SPEEDES persistence framework reduces the overhead by using object
factory. With the persistent object factory, registration with the database occurs
only when the object is first allocated from the heap. When the DELETE function is
called, object memory is not released but added to a free list in object factory. Next
time allocation get memory block from the free list.

3. Boost Serialization Library(BSL)

In the C++ world, the term serialization is used preferring to persistence, for it has
more meanings than persistence such as marshalling data for transmission to
another system. There are many serialization implementations such as
MFC(Microsoft Foundation Class), Boost Serialization Library, CommonC++ Library
[1] and so on. Boost serialization library has more advanced features than others
[1]:
1) Code portability. Depend only on ANSI C++ facilities.
2) Code economy. Exploit features of C++ such as RTTI, templates, and multiple
inheritance, etc.
3) Deep pointer saving and restoring. That is, save and restore of pointers saves
and restores the data pointed to.
4) Proper restoration of pointers to shared data.
5) Serialization of STL containers and other commonly used templates.
6) Data Portability - Streams of bytes created on one platform is readable on any
others.
7) Rich data format. Serialized data can be text, XML, or binary format.
So we use boost serialization library in our experiment.

3.1.

User Interface

We illustrate the BSL user interface through a simple example.

The classes definitions is as below:

There are two methods to make class A and class B serializable, one is intrusive,
and another is non-intrusive. In the intrusive method, class definition is modified as
follows:

In the non-intrusive method, the serialization function is implemented outside the

class definition, and classes are not modified.

The following code shows how to serialize class B instance to a file:

The above code shows how to serialize pointers, STL containers, derived class and
base class. In the code, the & operator is similar to the '<<' and '>>' operator in I/O
stream library and this makes it easy to use. If the saving procedure is different with
the loading procedure, the serialize function must be split into saving and loading

functions. We can see that it is very flexible to select which member to be serialized
and which not.

3.2.

Serialization Procedure

When code oa << pb is executed, BSL write class type id of B to the archive. If the
object pointed by pb has already been written to the archive, then only an object id
and class id need to be written to, else serialization function of B is called, which
then calls the serialization function of std::vector, which further calls the
serialization function of A, until in the end the serialization function of primitive type
is called. The procedure of de-serialization is similar to the above. If the object
pointed by pb has already been constructed, then the object address is searched
and assigned to pb, else the default constructor of B is called and serialization
function is recursively called.

4. SPF vs. BSL

To test SPF and BSL, a simulation engine and an application must be used.
We use sik [6] as the simulation engine, which is written in C++, with no more
than 4000 code lines. sik was developed to be able to efficiently simulate
largescale discrete-event simulations. It uses a micro-kernel design which is
analogous to the one employed in operating systems. sik has successfully been
used in many applications [3, 8].
In sik, a physical process is referred to as a 'Simulator' and is composed of many
logical processes.
Each logical process has its own synchronization algorithm. Logical processes
communicate via discrete events. All logical processes are saved in four priority
queues simultaneously and ordered with different rules.
There are two lists within each LP, one keeps unprocessed events and the other
keeps processed but uncommitted events.
In order to checkpoint a simulation for restart, it's not necessary to store a copy of
the state of the executable, storing the simulation states is enough. As to sik, if
the simulator object is stored, then all logical processes and all events are stored,
that's enough to restart a simulation.
We use Phold as the simulation application for test. Phold is a classic synthetic
benchmark program. In the Phold implementation, there are NLP logical process in
each Simulator. N events are generated at initialization, with random selected
destinations. When a process receives an event, it schedules a new event into the
future to another random destination with a minimum time increment which is
called lookahead. An interesting feature of Phold we can see is that the population
of events is always N during the simulation execution.
The original sik and Phold are modified to add the SPEEDES persistence framework
and Boost serialization library to make Phold serializable, the results are named SPFPhold and BSL-Phold correspondingly. The application executes on a personal
computer with a single CPU:AMD Athlon XP 3000+ 2.17GHz, 256M memory, Debian
GNU/Linux 3.1r0. The program parameters are: the number of LPs NLP=100, event
population N=100000, lookahead=1, simulation stop time tend=100s. Conservative
time synchronization algorithm is used in this test.

4.1 User Friendliness

In some circumstances, user friendliness is more important than performance. We
compare the BSL and SPF as follows on 5 aspects.

 Code modification
To make an application serializable, program code need to be modified. With BSL,
we just add some code but not modify any existing ones. With SPF, some existing
code must be replaced, for example, pointers must be replaced with smart pointers.
Replacement has more impact on the program, for it can cause other replacement.
Flexible
BSL is more flexible than SPF, The reasons are:
a) Using save/load operators &, << and >> in BSL to do checkpoint/restart is more
natural than using functions in SPF.
b) In BSL, we can change the checkpoint file format, and custom the class
identification method.
Checkpoint file format
The SPF checkpoint file is in binary format. It can be read by programs but is not
readable by humans. The BSL checkpoint file format can be XML, plain text, binary,
or any other format defined by users. The XML file is more useful in debugging
program for that we can see every data item clearly.
Debugging
In the procedure of making application serializable, program errors are inevitable
especially for programmers who are not familiar with these two frameworks. Errors
may occur in compile time or run time.
In SPF-Phold, the persistence framework interfaces are generated by macros.
Compiler can only point out the position of macros but not the generated code
which caused errors. In BSL-Phold, the framework code is C++ template based, we
can get the accurate information about which line caused errors.
In our experiment, given a runtime error message, it is difficult to seek the
problematic code in SPF-Phold. For example, when packing objects to a buffer, we
get an error message: Bad recursive memory find. According to [5], this indicates
that the persistent memory segment referenced by a persistent pointer cannot be
found in the database. This error usually occurs when an application deletes
memory referenced by a persistent pointer, but forgets to reset the pointer to NULL
afterwards. The question is that it is not easy to seek the code that deleting the
memory.
In SPF, most errors occur in run time while in BSL most errors occur in compiling
time. So, it's more difficult to debug errors in SPF than in BSL.
compiling time
The BSL-Phold takes a very long compiling time because of C++ template. For
example, the compiling time of Phold, SPF-Phold, BSL-Phold is 22s, 24s, and 140s.
Summary
The comparison of the two frameworks on user friendliness is listed in Table 1.

4.2 Performance
We evaluate the performance of these two frameworks from four aspects:
1) Physical time taken to run the application
2) Time taken to do checkpoint and restart
3) Checkpoint file size
4) Memory demanded by the application
Table 2 shows the test result. The SPF-Phold program has 2 versions, one uses free
lists to manage event objects memory allocation and de-allocation, and the other
one do not use any memory management techniques. Free list is used almost in all
simulation engines for space and time efficiency. Initially, BSL-Phold and SPF-Phold
are evaluated without memory management, the reason is:
1) Persistent object factory provided by SPF does not support C++ polymorphism
mechanisms, which makes it useless in sik.
2) In sik, free list is used to allocate/de-allocate memory for event objects through
user defined
new/delete operators, but BSL does not support user defined new/delete.

SPF-Phold is very slow without memory management. Its running time is 745s.
What's the matter? Is it caused by frequently allocating/de-allocating events object?
We use some special techniques to add object factory to SPFPhold and make the
BSL-Phold support user defined new/delete. From figure 2 we can see this doubles
the speed of SPF-Phold, the run time is now 337s. But the SPF-Phold is still very slow
compared to BSL-Phold. What's the reason? Experiment shows that it is caused by
priority queues. In the priority queue class, date member
of type T** is replaced by type T_PTR_PTR, which is less efficient than T**. When we
replaced T_PTR_PTR with T**, the run time of SPF-Phold declined to 32.8s.

The run time of BSL-Phold is nearly the same as Phold, because BSL has no
overhead on BSL-Phold when executing without checkpoint enabled except the
global variables initialization at program startup. The checkpoint file of SPF-Phold is
large than BSL-Phold's, this is out of our expectation. Analysis shows that SPF-Phold
saves more information than BSL-Phold:
a) Array length information. SPF-Phold saves the length along with the array data to
the file.
b) Tag information. SPF-Phold marks each persistent memory block with a tag
access the block through the tag.
c) Vtable information. It is useless for checkpoint and restart but the vtable pointer
is a part of a C++ class object.
d) In some persistent classes, the data members which can be deduced from other
data members are not necessary to be stored. SPF can't manipulate this situation
and stores all data members of the class object.
The peak memory consumption of SPF-Phold without persistence enabled is 53.8M,
1.7 times more than that of Phold. This is naturally because the persistent database
keeps a lot of additional information. The time taken by SPF-Phold to do checkpoint
is 5.3, double times of that of BSL-Phold, the reason is that the big checkpoint file
takes a long write time. On the other hand, many operations on database must be
done during check-pointing and restarting, such as recursively finding persistent
pointers, adjusting the persistent pointers to correct value, and so on.

5. Conclusion
SPEEDES persistent framework and Boost serialization library are tested and
analyzed. Result shows that the Boost serialization library is user friendly and high
performance while SPEEDES persistent framework needs to be optimized further
Rollback framework is a very important part of PDES systems. Persistent framework
need to work cooperatively with rollback framework. SPEEDES persistence
framework can be combined with a rollback framework [5] to do undo, redo
operations. Boost serialization library does not save any simulation objects state
before or after checkpoint, and it is independent with rollback framework and other
frameworks of SSA. Hence BSL supports rollback framework without any
modification.
Boost serialization library is an open source software, which is tested and evaluated
by users all over the world, and any bugs can be fixed quickly.
From the points of above, we can draw a conclusion that Boost serialization library is
a more suitable persistence framework for parallel and distributed simulation, and
thus is a good candidate to be integrated into the Standard Simulation Architecture.