SERVERware 3 Cluster Edition

The purpose of this document it to explain basic functionality and some of the core underlying
technologies used in Serverware 3.

Serverware 3 is a next generation communication technology solution. It delivers a wide range

of IP services and applications in a redundant, highly available and fault tolerant way.

With the cluster edition, it provides comprehensive management, flexibility, self healing and
scalability of hosts and virtual servers.

The cluster edition consists of 3 or more physical servers:

- Storage server mirror (consists of 2 servers)

- Processing hosts

In addition to the above, a cluster can also have additional backup servers, and a separate
SIPmon host.

Storage mirror

SERVERware storage component consists of two identical servers (active-standby) connected

back-to-back providing a storage service for the virtual servers running on the processing
servers over the SERVERware SAN network. Should the active storage server fails, the standby
server will automatically take over the role of storage serving. There is no disruption of storage
services looking from the view of processing hosts as well as virtual servers running on them.

Processing servers

The processing servers use their processing power to run the virtual instances on the cluster.
They do not use their local storage for anything except for the OS, but rather use the data from
the network storage on which sits on the Storage mirror. The communication between the
processing hosts and the storage mirror is done on the SAN network which is dedicated and
isolated for this specific purpose. Expanding the cluster to provide more processing power for
your VPS’s is as easy as adding additional processing servers to the existing cluster.
The tech

The three of the primary underlying technologies used to power SERVERware 3 are ZFS, iSCSI
and LXC Containers.

ZFS
To provide disk redundancy, fault tolerance and self-healing capabilities to SERVERware 3,
we’ve implemented ​ZFS​ into the setup. ZFS is a combined file system and logical volume
manager, that provides: protection against data corruption, support for high storage capabilities,
efficient data compression, snapshots, and logical raid setups. Apart from the dedicated disk(s)
for the operating system, Serverware will put all the remaining disks on the storage servers into
a ZFS pool, automatically creating the highest level of raid configurable from the amount of
disks remaining.

For example, if each storage server has a total of 4 disks, one is used for the OS, while the
remaining 6 disks (3 from each storage node) are put into a ZFS raid 10. This will provide disk
redundancy via raid 1 mirroring, but also improve disk I/O by striping the data across the mirrors
via raid 0.
For the disks intended for the network storage, we do not recommend using a hardware raid
controller, especially one without a cache battery. If a hardware RAID card is used, the ZFS
data corruption protection will conflict with the controllers protection which can result in data loss
as each would try do do its own repair. If you already have a HW RAID card, we suggest
flashing it into HBA mode, or setting your disks into non-raid/passthrough.

You can find more information about the incompatibility of ZFS and HW RAID here:

https://en.wikipedia.org/wiki/ZFS#ZFS_and_hardware_RAID
In addition to the above, we do not support FAKE/SOFT RAID implementations.

iSCSI

To provide disk redundancy, seamless failovers, and processing scalability, we’ve implemented
iSCSI (via Open iSCSI) into our Mirror and Cluster editions in two separate implementations.

First implementation is on the the storage mirror where the secondary storage server would
export its local disks to iSCSI targets so that the primary server would see them as its own.

For example if ‘STOR1’ has 3 network disks sda, sdb, sdc and ‘STOR2’ has its own sda, sdb
and sdc. Storage server 1 would see the 3 disks from Storage server 2 as local disks sdd, sde,
sdf allowing ZFS to create a pool of 6 disks total because the iSCSI targets and the fact the
disks are in a completely different server would not be visible to it.

Important:​ Considering the Open iSCSI limitation of 1Gb/s per target, this can clearly become a
bottleneck for the I/O operations in a cluster/mirror. For example if you have a high performance
SSD capable of writing 700MiB/s, due to the Open iSCSI, your passthrough cap would be
around a theoretical maximum of 125MiB/s. The solution is to have multiple smaller disks
instead of a single large disk in each server. This allows iSCSI to export a separate target for
each disk in the mirror, and in a mirror of 4 disks per server, the passthrough cap would be
round 500MB/s, and more disks = more bandwidth.

The second implementation is specifically implemented in our Cluster edition and it would export
iSCSI targets for each individual VPS over the SAN network to be used by the processing
servers.

By using iSCSI on the SAN, the processing servers can use the VPS data that is stored on the
on the Storage mirror like it was on its local hard drives. But in case your processing host was to
have a failure the iSCSI target would be exposed to a different processing server in the cluster
and would allow a seamless failover of your VPSs that were running on the failed processing
server.

LXC Containers

The actual virtualization and the possibility of running multiple virtual instances on a single
server is done by using Linux Containers (LXC). The virtualization method allows multiple
isolated linux systems (containers) to run on a single physical server.

LXC, through kernels cgroups functionality, allows various limitations and prioritizations to be
put in place for the individual containers, including CPU, memory, block I/O and networking
amongst others allowing you to restrict the amount of hardware resources one instance can pull.
It also separates them into separate entities in a way that although the hardware resources are
shared amongst them, the instances will not be aware of that.

The network

Serverware 3 cluster edition uses 3 different networks for various purposes:

- LAN network
- SAN network
- RAN network

The ​RAN network (Replication network)

This is a dedicated back to back network between the two storage servers. The network is
created by cabling two or more network interfaces into a bond and connecting them to the
appropriate network interfaces on the other storage server.
Latency is absolutely critical here, and for that reason we require a direct cable connection from
server to server, without any other network nodes between (switches, routers, etc..) It is also
imperative that the RAN network has a great bandwidth, usually achieved by bonding 4 x 1gb
ports, or ideally, 2 x 10gb ports.

The RAN’s primary purpose is to keep the ZFS disk pool synced between the two storage
servers. You can imagine the ZFS pool like a network and software based raid 10 (or similar)
implementation, built using the available disks from both the primary and the secondary storage
server.

Any data that is written by your services (PBXware VPS, Serverware, etc) needs to be written to
both servers in order to provide data redundancy, hence the need for a high bandwidth, low
latency RAN network, any delays here would equate to potential call quality issues across the
cluster.

The ​SAN network (Storage area network)

This is a dedicated network which is implemented only in the Cluster edition of Serverware 3.
The purpose of the network is to allow the communication and access between the processing
servers, and the main storage mirror.

As previously mentioned, the processing hosts do not have any local storage, but rather use the
data stored on the storage mirror for processing. To transfer that data from the storage mirror to
the processing hosts and back, we use the dedicated SAN network. To ensure that the transfer
of the data between the hosts is not the cluster bottleneck, we highly recommend a high
bandwidth/low latency network to be implemented for the SAN. A slow SAN would behave like a
slow or damaged hard drive, showing crippled I/O operations, and causing various issues, most
easily recognized in call quality deterioration.

The ​LAN network (Local area network)

This is your public/internet facing network connection. We create a bridge on top of the interface
to allow the individual VPS’s to share the network. On top of the bridge, serverware will create
virtual interface for each VPS, to allow them to have their own IP addresses.

For a simple cluster that consists of 2 storage servers and 2 processing servers, we require a
total of 6 IP addresses in the LAN network and 5 IPs in the SAN network.
Physical:
- Storage 1
- Storage 2
- Processing 1
- Processing 2
Floating:
- Controller
- Floating storage

Controller IP address will be used to access your Serverware 3 GUI, while the floating will be
pointing to the Storage server that is currently in primary state. In case of failover between the
storage servers, the floating IP address will follow, this way you can always connect to the
primary server irrelevant on which physical server its located.

Any additional servers or VPS’s added to the cluster will require an additional IP address,
including a backup server, sipmon processing server, and any PBXware and other VPS’s that
are going to be running on the cluster.

We highly suggest configuring public IP addresses directly on the interfaces, to avoid potential
issues which may be caused by NAT.

Redundancy

The LAN network is also used for host monitoring! This means if a processing host within a
cluster stops responding to the heartbeat monitor on the LAN interface, Serverware will consider
that host as down, and trigger the failover process for the VPS’s that were running on it before it
stopped responding.

The failover process will check the resource status of the remaining processing hosts in the
cluster and distribute the load accordingly avoiding an overload of a specific host in the cluster
while others still have available resources. The failover process lasts roughly under a minute
depending on how many VPS’s were running on the affected host. The data from each VPS will
be exactly the same as at the point of failure, considering that the data is still sitting safely on
the storage mirror. Once the failed processing host returns to function, it is required to manually
move the VPS’s back to it.

In case a storage server fails, the remaining storage server will go into primary state and will
continue to process the data in standalone mode until the other server returns. The failover
between the storage servers usually lasts a few seconds and any calls active during the failure
will notice a short period of silence, but the call will continue as normal. Once the failed storage
server returns to function, ZFS will resilver the data between them to ensure both servers have
the most uptodate dataset.

During any failover or disruption, serverware will send out an notification email to all Serverware
administrator account emails. To ensure important events are not overlooked, it is critical that
you configure SMTP configuration for Serverware, and confirm its working before putting your
Serverware into production.

Backup server

While the concept of a cluster serverware setup will provide a high level of hardware
redundancy, we highly recommend implementing a backup server to your cluster to provide an
additional level of data protection that will ensure that even software and file system problems
don’t leave your customers severely affected.

This is where the backup server plays a critical role!

With a dedicated backup server, you can configure that you have daily incremental or full
backups of your VPS’s. With this configured you will have the possibility of doing a rollback of a
specific VPS to a certain date.

The backup process is usually configured in such a way that a full backup is done every Sunday
at 02:00 AM, and incremental backups done every other day at the same time.

For more information about our Serverware product, please check out our ​wiki page​, and our
Youtube Channel​.