BroadHop

Quantum Network Suite Policy Server

Performance and Scalability Validation
Introduction

Test Highlights

The European Advanced Networking Test Center

(EANTC) was commissioned by BroadHop to test the
performance and scalability of BroadHops Quantum
Network Suite Policy Server. The goal of the test
engagement was to measure the performance characteristics of BroadHops policy server under real world
conditions.

Gx Interface Performance and Scale

BroadHops solution is an implementation of the relatively new (mandatory since 3GPP UMTS Release 8)
Policy and Charging Rule Function (PCRF). The policy
function in mobile networks dictates the rules a
subscriber must adhere to when using the mobile
network. The PCRF has a central role in the mobile
network it accesses subscriber information databases, charging systems, GGSNs, and LTE gateways.

BroadHop
Quantum Network Suite
Policy Server

Session Performance
Scales linearly

Session Scalability
20 million active sessions

Real World Scale

Meets mobile providers needs
Test Period: October 2010
Quantum Network Suite Policy Server
with QNS5.1.1 Version
2010 EANTC AG

20 Million Active Sessions

28,000 TPS per 16 Blades
1,750 TPS per Blade
Real World Performancea
11,480 TPS per 8 Bladesb
LTE VoIP Performance
13,000 TPS per 8 Bladesb
a. Service Selection, Location Aware, Fair Use, and Time
of Day Policy Rules, on Gx Interface
b. Not tested against the 16 Blades installation, due to test
bed limitation unrelated to the Quantum Network Suite.

Tested Solution and Test Equipment

BroadHop, through cooperation with Hewlett-Packard
(HP), provided an HP C7000 Blade Chassis in which
up to 16 HP BL460c G7 Blade Servers were installed.
Each server was running on two 2.8 Ghz Xeon X566
Six-Core Processors. The tests were executed in HPs
labs in Cupertino, California using Ixias IxCatapult
M500 chassis using 8 mCU5 cards. BroadHops
Quantum Network Suite uses a virtualized architecture
in which several logical components are split between
virtual machines as depicted in Figure 1.

PS

PD

PS
PS

Tested by

PS

Policy Directors

2010

Policy Servers

SM

SM

SM

SM

Active

Passive

Session Manager
Shards

Quantum Network Suite Policy Server

Figure1: Quantum Network Suite

Policy Server Virtual Architecture

EANTC TEST REPORT: BroadHop Quantum Network Suite Policy Server Performance --- Page 1 of 4

Performance and Scalability

Emulated PCEF

...

...

...

We split the tests into two parts - initially we measured

the performance of the Quantum Network Suite with a
minimum number of policy rules, and monitored the
scalability of the solution as the number of Policy
Server blades increased. The second group of tests
involved more extended policies and rules and
included not only performance of the Gx interface, but
also the more advanced Rx interface that is responsible for communication with application functions
(AF). The common measurement metric used
throughout all the tests was Transaction Per Second
(TPS).

Up to 16
Blades

Quantum Network Suite

Policy Server
Gx

Emulated AF

Rx

Device Under Test

Ixia IxCatapult

Gigabit Ethernet

3GPP Interface

Figure 2: Test Bed Setup

Transaction Per Second Definition

We defined a transaction as the reception of a request
and the issuance of its corresponding response by the
PCRF. We record one transaction when the PCEF
sends a Credit Control Request (CCR) on the Gx interface and the PCEF responds with a Credit Control
Answer (CCA) over the same interface. This exchange
is counted as one transaction. By extension, a single
subscriber session is inevitably responsible for two
PCRF transactions one to establish the session and
one to tear down the session. In UMTS networks, each
PDP activation requires at least one PCRF transaction,
more complicated services might require a higher
number of transactions per subscriber.
The TPS performance of a Policy server is a critical
data point for mobile service providers. Knowing the
TPS rate of a solution allows service providers to
correctly dimension their support systems around their
subscriber base. Two important questions should
always be asked when discussing TPS performance:
What is the baseline TPS performance?
Does the TPS performance decrease when more
and more policies are added to the PCRF?
We set off to answer these two questions against
BroadHops Quantum Network Suite.
Session Performance on the Gx Interface
In this test scenario, we measured the Gx transactions
per second for 16 HP Blades running the Quantum
Network Suite. We used 16 blades since this is the
maximum number of blades that could be installed in
the HP blade server we used in the test bed. The size
of the blade chassis is 10 Rack Units (RUs) which
means that up to 4 chassis could be installed in a standard, floor-to-ceiling rack.

From the Quantum Network Suite perspective, initial

transactions on the Gx interface consist of the
following:
1. Receive and decode the Gx CCR (Initial) request.
2. Create a network session for the subscriber within
the Quantum Network Suite or link the Gx session
to the network session if one already exists.
3. Query the Subscriber Profile Repository (SPR) for
the subscribers profile based on MSISDN number.
4. Make a policy decision based on the evaluation of
policy rules and decision tables. Service selection
was performed with 4 pre-defined PCC rules on the
PCRF.
5. Encode and send the Gx CCA (Initial) response.
Through a number of test runs we were able to
measure that the Quantum Network Suite, using a fully
populated HP Blade chassis with 16 blades, was able
to support 28,000 TPS.
We believe that being able to support such a number
of transactions in a quarter of a rack should satisfy any
current mobile service provider requirement.
Performance Linearity on the Gx Interface
When a mobile service provider considers investment
in a new PCRF solution he asks himself what happens
when I require more capacity from the solution?
Providers would like to reassure themselves that when
they invest in a solution to support X subscribers with Y
servers, doubling the number of servers will also
double the amount of subscribers supported by the
solution. In solutions such as BroadHops the blade to
performance ratio is of course especially of interest.

EANTC TEST REPORT: BroadHop Quantum Network Suite Policy Server Performance --- Page 2 of 4

A service provider is entitled to purchase only 8

blades in his blade server and, at a later time, add
more blades. Can he really expect that the performance of the solution will increase linearly?

the Gx diameter session id as a secondary key to

just using the Gx diameter session ID as the
primary session key.
2. Enabling Subscriber Profile Repository (SPR) data
query caching configuration in the Quantum
Network Suite. This configuration optimized the
system to query the SPR on session creation only
and cache for subsequent requests as opposed to
querying the SPR on every session transaction.
After the configuration changes we were able to verify
that the Quantum Network Suite was able to support
20 million active sessions. The figure below depicts
both the results and the test methodology we had to
use: first all sessions were opened, then we waited
several minutes to verify that the Quantum Network
Suite was stable and then terminated the sessions. As
the figure shows, 20 million subscribers were activated, remained active and then closed.

30000

28,062

20000
TPSTPS

14,053

10000

7,011

1,763

12

16

During the opening of the sessions we did monitor

around 1% of the sessions responding back to the
PCEF with diameter server busy response. In the real
world the PCEF would request session activation
again, however, our test setup was limited in this
aspect and so we had to conclude that session
opening was slightly slower than we initially measured
- 26,500 TPS.

Number of Blades
Figure 3: Performance Linearity Results

The results met our expectations as is seen in Figure 3.

The PCRF implementation from BroadHop was indeed
able to scale linearly meeting mobile service
providers needs for predictable and linear behavior.
Since the tests were only focused on a single blade
server we can not confirm or deny linearity beyond
this configuration, however, BroadHop did assure us
that a redundant configuration, across two or more
geographical areas was also possible. Would that
solution behave also in the same linear fashion? What
about adding another blade server in the same rack?
These will perhaps be the subjects of a future test
engagement.
Session Capacity on Gx Interface
A third data point we were trying to validate was the
number of subscribers the BroadHop test setup would
be able to support. BroadHop claimed that with this
setup 20 million active subscriber sessions could be
supported while still maintaining the same session activation rate.
This sounded like a valid challenge to us. While prestaging the test we did discover that the Quantum
Network Suite required two configuration changes in
order to support the test. The changes were relatively
simple:
1. Change in the primary session key from framed IP
of the user session as the primary session key and

Subscribers [in Million]

TPS on Gx Interface

This test aimed to verify this point. We repeated the

same test as we described in the session performance
section, only this time methodically stepping up the
number of blades used: 1, 4, 8, 16.

25
20
15

IP-CAN Session
Initialization

10

IP-CAN Session
Termination

5
0
0

10

20

30

40

50

Test Duration [minutes]

Figure 4: Session Capacity on Gx Interface Results

Session Scalability on Gx and Rx interfaces

For more advanced services, especially those envisioned for Long Term Evolution (LTE) mobile networks,
the application function (AF), which uses the Rx interface to the PCRF, is essential. In this test, we benchmarked the combined Gx and Rx TPS performance
using 8 blades, not 16, due to test bed limitations
unrelated to the Quantum Network Suite. The PCRF
first had to establish the subscriber session, authorize
it, and then create the AF session. In short - at least
twice as many transactions were required per
subscriber using dynamic services across the Rx interface.

EANTC TEST REPORT: BroadHop Quantum Network Suite Policy Server Performance --- Page 3 of 4

10

15

20

25

Test Duration [in minutes]

Figure 5: Gx-Rx Interface TPS Performance Results

As is seen in the figure the performance measured

across the combined Gx and Rx interfaces was only
10% lower than the performance we measured for Gx
interface only. With 13,000 TPS across the Rx interface, using half the number of blades possible in our
test bed, we believe that a mobile service provider has
a solid starting point to offering applications such as
Voice over LTE, that require dynamic policy and
Quality of Service control.

Time of Day Dynamic rules applied for streaming

during peak and off peak times. These rules carried
rule activation and deactivation time attribute
values pairs.
In addition to the above policies we activated 10
million subscribers to add some more realism to the
test. We measured the transaction per second rate on
Gx interface while location-based rules (and additional policies which were not tracked) were applied.
The highest achieved TPS rate in this case was
measured to be 11,480 TPS on Gx interface.

16,000
14,000
12,000
10,000
8,000
6,000
4,000
2,000
0

TPS

Background Subs

4 Blades

TPSTPS Pe

Fair Use A dynamic policy triggered by a 5

Gbytes per month fair use cap. The fair use was
designed to be applied to 5% of subscribers
violating their fair use during a busy hour, which
equates to 312 TPS across the whole system.

Transactions per Second

8 Blades

16000
14000
12000
10000
8000
6000
4000
2000
0

4 Blades

Transactions per Second

In essence the test repeated the same methodology we

described in the previous test cases, including SPR
query and charging and service selection rules,
however, this time the tester emulated the application
function (as seen in Figure 2). The results for our 8
blade setup are shown below:

TPS

20

40

60

Test Duration [in minutes]

Figure 6: Real World Use Case TPS Results

Real World Use Cases

In addition to the performance oriented tests we
executed a complicated test case using 8 blades (not
16 blades due to test bed limitations unrelated to the
Quantum Network Suite). This test case included Gx
credit control exchanges, SPR queries, and 14
dynamic PCC rules created on the PCRF and applied
to the PCEF:
Location Aware Different policies were applied to
home subscribers than to roaming subscribers.
Roaming and home subscribers were distinguished
based on the Mobile Country Code (MCC)/Mobile
Network Code (MNC) contained in the User
Location Information AVP across the Gx interface.
Policy and Service Selection Three subscriber
types were defined in the network: gold, silver,
bronze, each with its own service plan parameters.
These were selected and applied during service
activation.

About EANTC
The
European
Advanced
Networking
Test
Center
(EANTC) offers vendor-neutral
network test services for manufacturers, service providers and
enterprise customers. Primary
business areas include interoperability, conformance and performance testing for IP, MPLS,
Mobile Backhaul, VoIP, Carrier Ethernet, Triple Play,
and IP applications.
EANTC AG
Einsteinufer 17, 10587 Berlin, Germany
info@eantc.com, http://www.eantc.com/
JG, 1.1, 20101215

EANTC TEST REPORT: BroadHop Quantum Network Suite Policy Server Performance --- Page 4 of 4