TS-ONENDS-SW-0075 - Overload Protection

N
Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

Creation Date: 29/03/2016 Version: 1.0
Approver: Gopal Bagri Status: Approved
Approval Date: 06/04/2016 Document Type: Controlled Document
NSN Organization: MN P CS TS Security Classification: Public
Product: One-NDS Base on Template:
Technical Support Note

TS-ONE NDS -SW -0075
Overload Protection
Core Network
One -NDS
Approval date: 06/04/2016
This document contains following

type of information
Informative
Preventive X
Corrective
Additional categorization
Urgent X
Security
Release Upgrade
SW Update
Parameterization
Information is classified as
Internal
Public X
Customer Specific
Nokia Solutions and Networks is continually striving to reduce the adverse environmental effects of its
products and services. We would like to encourage you as our customers and users to join us in working
towards a cleaner, safer environment. Please recycle product packaging and follow the recommendations for
power use and proper disposal of our products and their components.
If you should have questions regarding our Environmental Policy or any of the environmental services we
offer, please contact us at Nokia Solutions and Networks for additional information.
Printed copies are uncontrolled, for information only

- Page 1/11 - © Nokia Solutions and Networks 2015
N

Table of Contents
1. PURPOSE ................................................................................................................................... 3
2. RATE PROTECTION .................................................................................................................. 3
3. FULL QoS ................................................................................................................................... 4
4. APPLICATION (USER) PREFERRED QoS LEVEL MAPPING.................................................... 5
5. TOOLS FOR OVERLOAD CONTROL ......................................................................................... 6
6. HOW TO CHECK IF THE RATE CONTROL IS ENABLED .......................................................... 7
7. DEFINITION OF MAXIMUM RATES ........................................................................................... 7
8. Ldif Generation Example ............................................................................................................. 8
9. END OF PROCEDURE ............................................................................................................. 10
Contact:
Contact your local Nokia support
Summary of changes:
Date Version Status Author of version Comments

29-Mar-2016 0.1 Draft raj.tiwary@nokia.com Initial Version
31-Mar-2016 0.2 Draft raj.tiwary@nokia.com Updated Doc Id
01-Apr-2016 0.3 Draft raj.tiwary@nokia.com Minor Documentation updates
06-Apr-2016 1.0 Approved raj.tiwary@nokia.com Approved Version
Reviewed by:
Date Version Reviewers

06-Apr-2016 1.0 Karhik K, Rijin E, Mark Lowes, Gopal Bagri
Approved by:
Date Approver
06-Apr-2016 Gopal Bagri
This TSN supersedes the TS-ONENDS-SW-0039-C1

N

1. PURPOSE
This TSN reviews the concept of Overload Protection Mechanism implemented in the One-NDS. This is a mandatory
feature. All customers with the One-NDS product must have both LDAP Rate Control and Quality of Service (QoS) for
each application which supports the feature.
Rate control is only part of the overall protection and that QoS is required to give the full level of protection against
signaling storms and other unexpected major network traffic events. Without QoS the DB has no knowledge of the
relative importance of applications or operations, therefore QoS forms a critical part of the overall robustness solution
ensuring that overload gapping decisions are based on traffic importance rather than random chance.
2. RATE PROTECTION
The rate protection is divided into two core functions: Rate Control and Automatic Rate Protection.
Rate control defines static configuration limits on the total number of LDAP operation, LDAP queries and LDAP
updates which are allowed in a second. Any traffic that exceeds these rates will be rejected. Automatic rate protection
is an algorithm which makes the traffic rejection decisions based on the internal system overload metric.
AUTOMATIC RATE
RATE CONTROL
PROTECTION
Request Query rate

Request CPU Metric Request
received test
Total rate accepted processing
test
Update DAPC Metric
rate test
Request
rejected
The focus of this TSN is Rate Control since this is customer specific and needs an ongoing review process.
In principle automatic rate protection can cope with the most overload situations and keep always the node stable.
Nevertheless, the definition of a static rate with Rate Control improves the system behaviour compared to the
automatic rate protection due to the following facts:
- The average LDAP response times are kept lower in overload scenarios when properly configured rate control
settings are defined.
- The number of queue overload and other alarms is lower with properly configured rate control – especially in
situations where the traffic levels are just about to exceed the overload metric limits.
- Rate control can be configured to limit the update rate to a value which matches the typical network latency which
allows the resynchronisation of secondary nodes in a wide variety of load scenarios.
If the rate control is not enabled in the network, there is potential risk of major service impact to subscribers
in case of overload with a high risk of a critical outage. Please make sure that the Rate Control is enabled and
correctly configured with the appropriate values. This configuration is explained in the following chapters.
When the rate control is set properly, the DS will reject all incoming LDAP traffic above the configured rate. This
measure ensures available CPU capacity for the A31 process also during high overload.

N

Tip: For example you can check the current setting for DSA 3:
ldapsearch -x -h localhost -p 16611 -D "cn=nds" -w sdfrun1 -b
"dsControl=ldap,control=sdf,dsaId=3,DC=C-NTDB" -s sub dsLdapOverloadMetric
3. FULL QoS
Without any QoS, a heavily loaded One-NDS system will reject incoming LDAP operations with LDAP busy
operation code on random basis. From the customer and service point of view it may happen that some
important operations cannot be performed. For example the One-NDS Administrator tool might be unusable.
In overload situation system is rejecting all LDAP traffic exceeding the system limits with LDAP error code: LDAP Busy
(51).
The QoS implementation has 10 internal QoS Priority levels available for the applications. These levels are used by
defining Qos Profiles and assigning a profile to an application (i.e. the appropriate LDAP user). The configuration is set
across all DS nodes in the database, thus any overloaded node is performing the rejections based on the levels
assigned to the active applications.
Idea of the QoS levels is very simple. When system is overloaded, the available capacity will be divided between the
applications starting from the highest priority ending to the lowest priority. If there is not enough capacity for all the
traffic, the capacity will be given to the currently active high priority applications starting from the highest priority
application, lower priority traffic will be rejected.
If nothing is configured or all the traffic is generated by the users with the same priority level, traffic is rejected in the
same fashion as in earlier One-NDS installations.
In normal load scenarios low priority traffic does not affect high priority traffic. In extreme overload situations where due
to the traffic bursts automatic rate protection causes more rejections, high priority applications may be affected by long
response times.
Within the object class ‘Priority Profiles’ (dsQoSProfile) can be created pointing to a ‘Priority Profile name’ to an
individual QoS priority levels (1 to 10). Each LDAP user can be assigned to use one of these ‘Priority Profiles’.
The higher the users QoS priority level is, the lower is the probability that the users requests are rejected (LDAP error
51) in case of RDS or BE overload conditions (rate limit exceeded). Any traffic of LDAP users that does not have a
profile assigned will be treated with the default QoS level which is configured as level 5.
For each priority level a default profile name will be created in the database as delivered with schema LDIF files
coming with the Patch Medium.
Default profile names are:
qosPrio10 (Level 10) Highest priority
qosPrio9 (Level 9)
qosPrio8 (Level 8)
qosPrio7 (Level 7)
qosPrio6 (Level 6)
qosPrio5 (Level 5) Default priority
qosPrio4 (Level 4)
qosPrio3 (Level 3)
qosPrio2 (Level 2)
qosPrio1 (Level 1) Lowest priority
The major goals to be achieved when assigning priorities to the ‘standard’ LDAP users are:
 System stability under a wide range of loading conditions
 Guarantees system management functions (@Com, ADM, One-NDS commands) run stable
 HLR is the highest priority application

N

 Giving higher priority to provisioning will decrease the number of rollbacks and thus reduce the load
generated from provisioning
 Manual LDAP operations and scripts must not affect the system with load
Note: The user sdfrun must not be used for command line operations and locally sourced tools that generate more
than a single request. For internal tools the new LDAP user nds shall be used. It has the password ‘sdfrun1’ assigned
per default.
4. APPLICATION (USER) PREFERRED QoS LEVEL MAPPING
QoS Profiles gives the means for the system administrator to protect the critical applications from the load generated
by low priority applications. Additionally it is common for applications to perform multiple sequential operations, all of
which must complete for the underlying action at the service interface to be successful.
An example of this is the HLR Network Registration procedure. When a mobile handset needs to reattach to the
network, it first authenticates itself with HLR and if the authentication phase is successful, it continues with location
update procedure. The messaging from HLR to One-NDS involves a read, then an update for the authentication,
another read and another updated to for the location update. If any of these operations get rejected by the One-NDS
the HLR will reject the Network Registration procedure and the mobile handset will restart the whole procedure.
Full QoS in the context of above example, could enable the HLR to use a lower priority in the initial operation with the
One-NDS for the Network Registration and increase the priority of the subsequent operations. Then if the One-NDS is
under overload conditions any rejection is likely to be early before much work is performed with a higher probability of
completion once the first operation is complete. The chances of a failure at the end are reduced increasing the overall
efficiency of the system under overload conditions.
Solution for the application is preferred QoS Level Mapping. As the system is internally defining 10 QoS priority levels,
those levels can now be used for each individual LDAP requests. This will be achieved by attaching the new LDAP
control to each LDAP request. This control will contain a string (QoSPreference) which is used for the mapping the
QoS level based on the LDAP user and the content of the string.
The dsQosProfile contains the new User preferred QoS levels and how they are then mapped to the system QoS
levels:
dn: dsQoSProfileName=highPriQoS, configType=qos, config=sdf, dsaId=10, ...
objectClass: dsQoSProfile
dsQoSProfileName: highPriQoS
dsQoSLevel: 7
dsQoSLevelMap: lowest|5
dsQoSLevelMap: highest|9
The optional attribute dsQosProfile is called “dsQosLevelMap” and it is a sequence multi-valued attribute where
element 1 is a string value that means something to the applications using that dsQosProfile and then the appropriate
system QoS level that the string value maps to i.e. “important|10” and “unimportant|1”. If the LDAP request from
application does not contain the LDAP control, default QoS level value of QoS Profile is used.
The Operation Specified User preferred Qos level control where each LDAP request can contain optional information
element with QosPreference string (see example below):
controlType 1.3.6.1.4.1.15573.1.2.1.4
criticality BOOLEAN DEFAULT FALSE
controlValue OCTET STRING
The control string is an ASN.1 encoded string:
QoSPreference ::= PrintableString
In overload situations the maximum rate system can handle will be given to the LDAP requests which have the highest
QoS priority level or to the LDAP users which have the highest priority.

N

4.1 ndsQoS tool
Using the command-line tool: ndsQoS to check if there are any applications using unconfigured QoS level mappings or
reset alarm information.
ndsQoS [-o mismatch] [-r alarm|data]
Attached below, a table with the standard profile assignments. Any modification of these values should be
discussed with the application PL.
5. TOOLS FOR OVERLOAD CONTROL
Additional to the built-in One-NDS tools, following two tools are provided, which assist in managing the overload
protection feature.
• manageLoadControl
• max_stats.sh
The tools are available in the zip file load_tools_NDS8_NDS9.tgz stored in the subdirectory “extras” of the
Customer Documentation media.
These tools can be copied to any database node where they are required, (for example: to /home/oamsys) and may be
executed locally as sdfrun user. Unload the tools on the node from prompt with command line: tar –xvzf
load_tools_NDS8_NDS9.tgz to prevent CR/LF problems.
Warnings about Manage Load Control Tool:
1. The option to directly change values in the directory must not be used in live networks, but only at test
installations. Setting values to too low may instantly lead to a permanent system overload.
2. This tool must only be used in lab environments for setting the values. In the field it must be used for
displaying values and for generating an ldif file. Once this file is generated, it can be edited, reviewed, and
can be applied finally with One-NDS Administrator function “Data Management”/“DM Management – DM
Package Admin - Directory data”.

N

6. HOW TO CHECK IF THE RATE CONTROL IS ENABLED
To display the currently set limits of the Rate Controls execute the command below.
In this example the Rate Control values “0” means that is not enabled.
pargo-be02$ ./manageLoadControl –list
manageLoadControl: | old/new | old/new | old/new

manageLoadControl: dsaId | maxUpdateRate | maxQueryRate | maxOpRate
manageLoadControl: -------+---------------+---------------+--------------
manageLoadControl: 1 | 0/- | 0/- | 0/-
7. DEFINITION OF MAXIMUM RATES

For most customers it is recommended that the values from the Performance Configuration Guide for the HW deployed
are used to define the maximum BE DSA update rate. The more complex approach described below requiring testing
and the excel sheet should be reserved for customers with a higher degree of customization / range of applications.
7.1 DEFAULT DEFINITION OF MAXIMUM RATES

For most customers the recommended values that have to be configured for the update rates based on the hardware
deployed are given in the table below
Reference values for maximum Updates
Hardware One-NDS 8.0 MP8 One-NDS 9 SP2 One-NDS 16

RX200-S5 4000 3900 Not Supported
RX200-S6 3500 3900 4300
RX600-S4 3600 3600 Not Supported
SUN X4200 3200 3300 Not Supported
SUN X4250 3300 3000 3200
SUN X4270 3700 3900 4200
IBM Blade HS22 N/A 4000 4200
HP Blade Gen 8 BL460c Not Supported 3900 4800
HP Blade Gen 9 BL460c Not Supported NA 7400
All new hardware not listed here 3500 3900 3900

N

7.2 MANUAL CALCULATION OF MAXIMUM RATES

The method of Manual calculation of the Maximum rates is required for larger customers with complex networks.
The Following chapters from the Performance Configuration Guide provide a detailed description of how to calculate
the maximum rates for BE-DSA and R-DSA, this should be used to calculate the maximum rates for the platform.
For One-NDS 8 MP8 and One-NDS 16, refer to :
Chapter 2.2 Tools for Overload Control

Chapter 2.3 Definition of Maximum Rates for Backend DSA
For One-NDS 9 SP2, refer to:
Chapter 4.2 Tools for Overload Control

Chapter 4.3 Definition of Maximum Rates for Backend DSA
The traffic profile must be obtained from the customer platform (number and type of LDAP operations per minute
versus the CPU usage), this is then used as the input to define the maximum rates.
The Customer Documentation additional files have the folder “Performance Configuration” where you can find the BE-
DSA and R-DSA excel spreadsheets to calculate the respective maximum rates.
Summary:
RDSA : Do not set update or search rates
BEDSA : Do not set search rate, set the update rate to the appropriate value for the HW from the performance guide or
manual calculation for large complex customers as said above.
Considerations:
1. It is currently not planned to define any rates for PGW-DSA. The reason for this is that the LDAP traffic within
PGW-DSA is very limited – only single requests from PGW and Notification Manager to read configuration
data or to write statistic data are issued.
2. If a new DSA with a faster hardware is added or there are deployments with different DSA HW, it may be
necessary to adjust the limits for those DSA.
3. Any alteration of the customer traffic model will affect the rate control design. Review the maximum rates.
4. Review the numbers every 6-12 months.
8. LDIF GENERATION EXAMPLE
After having the definitive Rate Control values you need to generate the ldif files.
Calling the “manageLoadControl” tool with -L option generates an ldif file, which can be applied manually (open
ldap) or via ADM. The file can be reviewed and edited before applying it to the system.
After defining the values for MaxQuery/s the LDIF will be generated for the live network.

N

For the following example the Rate Control max values from the current load traffic mix are:
BE-DSA:
dsLdapMaxUpdateRate: 3100
a) Log on to the node where the manageLoadControl was install as root user and create the LDIF file:
# ./manageLoadControl -L -U 3100 -Q 0 -O 0 --ldif-file /tmp/loadcontrol.ldif
where:
U is the maximum number of Update/s

O is the maximum number of Operation/s
Q is the maximum number of Query/s
This file should now be transferred to a local PC and must be checked thoroughly.
For DSA1 (R-DSA), the values must be deleted as we are not setting anything for RDSA.
Example for BE-DSA (DSA2):
dn: dsControl=ldap,control=sdf,dsaId=2,DC=C-NTDB
changetype: modify
replace: dsLdapMaxUpdateRate
dsLdapMaxUpdateRate: 3100
8.1 Ldif Execution via ADM
a) Open One-NDS Administrator GUI with Option: DM Management → DM Package Admin.
b) Select directory: Subscriber Directory, data type: Directory Data: <all>. Press <Import>.
c) Name the package. For example: “X4270_8Core_RateControl”. Add version number.
d) From File Selection browse the data file: < LDIF File Name> (loadcontrol.ldif)
e) Select LDAP command MODIFY.
f) Press <+Add> and <OK> to complete the import.
g) Select the package and <Activate> the data on R-DSA. Use “LDAP operation determination = Direct (no
Pre-processing)” for the activation.

N

8.2 Post Check

a) Check if the Job is completed successfully in the ADM Job Management.
b) Display the currently set limits of all DSAs (example below)
pargo-be02$ ./manageLoadControl --list

manageLoadControl: | old/new | old/new | old/new
manageLoadControl: dsaId | maxUpdateRate | maxQueryRate | maxOpRate
manageLoadControl: -------+---------------+---------------+--------------
manageLoadControl: 1 | 0/- 0/- | 0/-
After having the maximum rates deployed, the system must be monitored at least for the following week. Within a
redundant configuration any traffic must not be rejected by the rate control. Therefore, use @Com/NetAct every day
(set a filter to see Warnings as well), to check that no traffic rejection warnings/alarms had been generated.
9. END OF PROCEDURE
Please contact the One-NDS Technical Support if there are any questions is concerning this procedure.
This is also documented in the One-NDS Performance Configuration Guide.

N

Disclaimer
This document is intended for use of Nokia Solutions and Networks customers and collaborators
only for the purpose for which this document is submitted by Nokia Solutions and Networks. No
part of this document may be reproduced or made available to the public or to any third party in
any form or means without the prior written permission of Nokia Solutions and Networks. This
document is to be used by properly trained professional personnel. Any use of the contents in this
document is limited strictly to the use(s) specifically created in the applicable agreement(s) under
which the document is submitted. The user of this document may voluntarily provide suggestions,
comments or other feedback to Nokia Solutions and Networks in respect of the contents of this
document ("Feedback"). Such Feedback may be used in Nokia Solutions and Networks products
and related specifications or other documentation. Accordingly, if the user of this document gives
Nokia Solutions and Networks feedback on the contents of this document, Nokia Solutions and
Networks may freely use, disclose, reproduce, license, distribute and otherwise commercialize the
feedback in any Nokia Solutions and Networks product, technology, service, specification or other
documentation.
Nokia Solutions and Networks operates a policy of ongoing development. Nokia Solutions and
Networks reserves the right to make changes and improvements to any of the products and/or
services described in this document or withdraw this document at any time without prior notice.
The contents of this document are provided "as is". Except as required by applicable law, no
warranties of any kind, either express or implied, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose, are made in relation to the
accuracy, reliability or contents of this document. NOKIA SOLUTIONS AND NETWORKS SHALL
NOT BE RESPONSIBLE IN ANY EVENT FOR ERRORS IN THIS DOCUMENT or for any loss of
data or income or any special, incidental, consequential, indirect or direct damages howsoever
caused, that might arise from the use of this document or any contents of this document.
This document and the product(s) it describes are protected by copyright according to the
applicable laws.
Nokia is a registered trademark of Nokia Corporation. Other product and company names
mentioned herein may be trademarks or trade names of their respective owners.
© Nokia Solutions and Networks 2016


TS-ONENDS-SW-0075 - Overload Protection

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

TS-ONENDS-SW-0075 - Overload Protection

Uploaded by

Copyright:

Available Formats

N

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

Technical Support Note

Approval date: 06/04/2016

This document contains following

Printed copies are uncontrolled, for information only

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

Contact your local Nokia support

Date Version Status Author of version Comments

Date Version Reviewers

This TSN supersedes the TS-ONENDS-SW-0039-C1

Printed copies are uncontrolled, for information only

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

Request Query rate

Printed copies are uncontrolled, for information only

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

Printed copies are uncontrolled, for information only

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

4. APPLICATION (USER) PREFERRED QoS LEVEL MAPPING

Printed copies are uncontrolled, for information only

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

4.1 ndsQoS tool

5. TOOLS FOR OVERLOAD CONTROL

Warnings about Manage Load Control Tool:

Printed copies are uncontrolled, for information only

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

6. HOW TO CHECK IF THE RATE CONTROL IS ENABLED

pargo-be02$ ./manageLoadControl –list

manageLoadControl: | old/new | old/new | old/new

7. DEFINITION OF MAXIMUM RATES

7.1 DEFAULT DEFINITION OF MAXIMUM RATES

Reference values for maximum Updates

Hardware One-NDS 8.0 MP8 One-NDS 9 SP2 One-NDS 16

Printed copies are uncontrolled, for information only

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

7.2 MANUAL CALCULATION OF MAXIMUM RATES

For One-NDS 8 MP8 and One-NDS 16, refer to :

Chapter 2.2 Tools for Overload Control

For One-NDS 9 SP2, refer to:

Chapter 4.2 Tools for Overload Control

8. LDIF GENERATION EXAMPLE

Printed copies are uncontrolled, for information only

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

# ./manageLoadControl -L -U 3100 -Q 0 -O 0 --ldif-file /tmp/loadcontrol.ldif

U is the maximum number of Update/s

Example for BE-DSA (DSA2):

8.1 Ldif Execution via ADM

a) Open One-NDS Administrator GUI with Option: DM Management → DM Package Admin.

c) Name the package. For example: “X4270_8Core_RateControl”. Add version number.

e) Select LDAP command MODIFY.

f) Press <+Add> and <OK> to complete the import.

Printed copies are uncontrolled, for information only

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

8.2 Post Check

b) Display the currently set limits of all DSAs (example below)

pargo-be02$ ./manageLoadControl --list

This is also documented in the One-NDS Performance Configuration Guide.

Printed copies are uncontrolled, for information only

Author/Owner: raj.tiwary@nokia.com Document ID: D542192317

Printed copies are uncontrolled, for information only

You might also like