5/19/2020 CoreXL Dynamic Dispatcher in R77.30 / R80.

10 and above

+1-866-488-6691 Contact Us checkpoint.com CheckMates FAQs Blog Welcome: Dicki Nurzamzam  

SELL MARKET LEARN SUPPORT MY CHECK POINT

Support Center > Search Results > SecureKnowledge Details

Search Support Center

CoreXL Dynamic Dispatcher in R77.30 / R80.10 and above Technical Level

Rate This My Favorites Email Print

Solution ID sk105261

Technical Level

Product CoreXL
Version R77.30, R80.10, R80.20, R80.30, R80.40
OS Gaia, SecurePlatform 2.6
Platform / Model All
Date Created 19-May-2015

Solution
Table of Contents:

1. Introduction
2. CoreXL Dynamic Dispatcher
3. Configuration on Security Gateway R77.30
4. Configuration on Security Gateway R80.10 and above
5. Monitoring CoreXL load distribution
6. Limitations
7. FAQ
8. Related documentation
9. Related solutions

(1) Introduction
CoreXL is a performance-enhancing technology for Security Gateways on platforms with multiple CPU cores. CoreXL enhances Security Gateway performance by enabling
the processing CPU cores to concurrently perform multiple tasks.

On a Security Gateway with CoreXL enabled, the Firewall kernel is replicated multiple times. Each replicated copy, or Firewall instance, runs on one processing CPU core.
These Firewall instances handle traffic concurrently, and each Firewall instance is a complete and independent Firewall inspection kernel. When CoreXL is enabled, all the
Firewall kernel instances in the Security Gateway process traffic through the same interfaces and apply the same security policy.

The CoreXL software architecture includes the Secure Network Distributor (SND). The SND is responsible for:

Processing incoming traffic from the network interfaces

Securely accelerating authorized packets (if SecureXL is running)
Distributing non-accelerated packets or Medium Path packets among CoreXL FW kernel instances - this functionality is also referred to as dispatcher

Traffic received on network interface cards (NICs) is directed to a processing core running the SND.

The dispatcher is executed when a packet should be forwarded to a CoreXL FW instance (in Slow path and Medium path - see sk98737 for details) and is in charge of
selecting the CoreXL FW instance that will inspects the packet.

In R77.20 and lower versions, traffic distribution between CoreXL FW instances is statically based on Source IP addresses, Destination IP addresses, and the IP 'Protocol'
type. Therefore, there are possible scenarios where one or more CoreXL FW instances would handle more connections, or perform more processing on the packets
forwarded to them, than the other CoreXL FW instances.

This may lead to a situation, where the load is not balanced across the CPU cores, on which the CoreXL FW instances are running.

To help mitigate the above issue, CoreXL Dynamic Dispatcher feature was introduced in R77.30 Security Gateway.

(2) CoreXL Dynamic Dispatcher

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk105261&partition=Basic&product=Co… 1/5
5/19/2020 CoreXL Dynamic Dispatcher in R77.30 / R80.10 and above
Rather than statically assigning new connections to a CoreXL FW instance based on packet's IP addresses and IP protocol (static hash function), the new dynamic
assignment mechanism is based on the utilization of CPU cores, on which the CoreXL FW instances are running.

The dynamic decision is made for first packets of connections, by assigning each of the CoreXL FW instances a rank, and selecting the CoreXL FW instance with the lowest
rank.

The rank for each CoreXL FW instance is calculated according to its CPU utilization.

The higher the CPU utilization, the higher the CoreXL FW instance's rank is, hence this CoreXL FW instance is less likely to be selected by the CoreXL SND.

The CoreXL Dynamic Dispatcher allows for better load distribution and helps mitigate connectivity issues during traffic "peaks", as connections opened at a high rate that
would have been assigned to the same CoreXL FW instance by a static decision, will now be distributed to several CoreXL FW instances.

When CoreXL Dynamic Dispatcher is enabled, the dynamic decision is always made (even when there is no significant load).

(3) Configuration on Security Gateway R77.30

CoreXL Dynamic Dispatcher is disabled by default. Meaning, CoreXL statically assigns new connections to a CoreXL FW instance based on packet's IP addresses and IP
'Protocol' type (static hash function).

Important Notes:

Before enabling the CoreXL Dynamic Dispatcher, refer to "Limitations" section and to "FAQ" section.

In R77.30, when working with CoreXL Dynamic Dispatcher, the only officially supported modes are:

Mode Number Mode Name Explanation

0 Off
4 Full-DD
Default.
CoreXL Dynamic Dispatcher is completely disabled. CoreXL statically assigns new connections to a CoreXL FW instance
based on packet's IP addresses and IP 'Protocol' type.
Important Note: Supported only after installing the hotfix from sk109772 - R77.30 NGTP, NGTX and HTTPS Inspection
performance and memory consumption optimization.
CoreXL Dynamic Dispatcher is fully enabled.
Firewall Priority Queue feature in each CoreXL FW instance (refer to sk105762) is fully disabled.

CoreXL Dynamic Dispatcher is fully enabled.

9 On
Note: This mode also fully enables the Firewall Priority Queues feature in each CoreXL FW instance (refer to sk105762).

Instructions:

To check the current mode on Security Gateway:

[Expert@HostName]# fw ctl multik get_mode

Example output:

[Expert@R77.30:0]# fw ctl multik get_mode

Current mode is Off
[Expert@R77.30:0]#

To fully enable the CoreXL Dynamic Dispatcher on Security Gateway:

Note: In cluster environment, this procedure must be performed on all members of the cluster. Since a reboot is required, it is recommended to follow the Gaia
Installation and Upgrade Guide - either "Minimal Effort" procedure, or "Zero Downtime" procedure.

1. Run in Expert mode:

[Expert@HostName]# fw ctl multik set_mode 9

Example output:

[Expert@R77.30:0]# fw ctl multik set_mode 9

Please reboot the system
[Expert@R77.30:0]#

2. Reboot (in cluster, this might cause fail-over).

To completely disable the CoreXL Dynamic Dispatcher on Security Gateway:

Note: In cluster environment, this procedure must be performed on all members of the cluster. Since a reboot is required, it is recommended to follow the Installatio
and Upgrade Guide for Gaia Platforms - Chapter 'Upgrading ClusterXL Deployments' - either "Minimal Effort" procedure, or "Zero Downtime" procedure.

1. Run in Expert mode:

[Expert@HostName]# fw ctl multik set_mode 0

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk105261&partition=Basic&product=Co… 2/5
5/19/2020 CoreXL Dynamic Dispatcher in R77.30 / R80.10 and above
Example output:

[Expert@R77.30:0]# fw ctl multik set_mode 0

Please reboot the system
[Expert@R77.30:0]#

2. Reboot (in cluster, this might cause fail-over).

(4) Configuration on Security Gateway R80.10 and above

CoreXL Dynamic Dispatcher is enabled by default. Meaning, CoreXL dynamically assigns new connections to a CoreXL FW instance based on the utilization of CPU cores.

Important Notes:

Starting R80.20 VSX fully support dynamic dispatcher

Refer to "Limitations" section.
Refer to "FAQ" section.

Instructions:

To check the current mode on Security Gateway:

[Expert@HostName]# fw ctl multik dynamic_dispatching get_mode

Example output:

[Expert@R80.10:0]# fw ctl multik dynamic_dispatching get_mode

Current mode is On
[Expert@R80.10:0]#

To enable the CoreXL Dynamic Dispatcher on Security Gateway:

Note: In cluster environment, this procedure must be performed on all members of the cluster. Since a reboot is required, it is recommended to follow the Installatio
and Upgrade Guide - Chapter 'Upgrading ClusterXL Deployments' - either "Minimal Effort Upgrade" procedure, or "Zero Downtime Upgrade" procedure.

1. Run in Expert mode:

[Expert@HostName]# fw ctl multik dynamic_dispatching on

Example output:

[Expert@R80.10:0]# fw ctl multik dynamic_dispatching on

New mode is: On
Please reboot the system
[Expert@R80.10:0]#

2. Reboot (in cluster, this might cause fail-over).

To disable the CoreXL Dynamic Dispatcher on Security Gateway:

Note: In cluster environment, this procedure must be performed on all members of the cluster. Since a reboot is required, it is recommended to follow the Installatio
and Upgrade Guide - Chapter 'Upgrading ClusterXL Deployments' - either "Minimal Effort Upgrade" procedure, or "Zero Downtime Upgrade" procedure.

1. Run in Expert mode:

[Expert@HostName]# fw ctl multik dynamic_dispatching off

Example output:

[Expert@R80.10:0]# fw ctl multik dynamic_dispatching off

New mode is: Off
Please reboot the system
[Expert@R80.10:0]#

2. Reboot (in cluster, this might cause fail-over).

(5) Monitoring CoreXL load distribution

This section describes how administrator can monitor either the current CoreXL load distribution to decide whether to enable CoreXL Dynamic Dispatcher, or the CoreXL
Dynamic Dispatcher in action (after it was enabled):

Note: To monitor the CoreXL Dynamic Dispatcher in action, it must already be enabled with fw ctl multik set_mode 9 command and Security Gateway must already be
rebooted.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk105261&partition=Basic&product=Co… 3/5
5/19/2020 CoreXL Dynamic Dispatcher in R77.30 / R80.10 and above
The examples provided below were taken from the machine without VPN blade or VoIP traffic and with disabled CoreXL Dynamic Dispatcher. These examples show an
excellent reason to enable CoreXL Dynamic Dispatcher to mitigate the load imbalance between CoreXL FW instances:

1. Check on which CPU cores the CoreXL FW instances are running with the fw ctl affinity -l -r command.

Example (CoreXL FW instances are running on CPU 1, CPU 2, and CPU 3):

[Expert@HostName]# fw ctl affinity -l -r

CPU 0: eth0 eth1 eth2
CPU 1: fw_2
CPU 2: fw_1
CPU 3: fw_0
All: cpca status_proxy in.geod fwm cpstat_monitor fwd mpdaemon cpsead cpd cprid
[Expert@HostName]#

2. Check the current CPU utilization by each CoreXL FW instance with the top command.

Note: If the output does not show all CPU cores (if 3rd line shows "Cpu(s):"), then press "1" and then "Shift+W".

Example (Load on CPU 3 is 24% by SoftIRQ; CoreXL FW instance 0 is consuming 18%; other CoreXL FW instances (fw_worker threads) are idle):

Tasks: 118 total, 3 running, 115 sleeping, 0 stopped, 0 zombie

Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 94.0%id, 0.0%wa, 1.2%hi, 4.8%si, 0.0%st
Cpu1 : 0.0%us, 1.2%sy, 0.0%ni, 98.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 40.2%us, 9.2%sy, 0.0%ni, 49.4%id, 0.0%wa, 0.0%hi, 1.1%si, 0.0%st
Cpu3 : 1.3%us, 1.3%sy, 0.0%ni, 73.3%id, 0.0%wa, 0.0%hi, 24.0%si, 0.0%st
Mem: 4078484k total, 4021144k used, 57340k free, 241380k buffers
Swap: 3140696k total, 64k used, 3140632k free, 414744k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

15964 admin 18 0 1644 436 368 R 41 0.0 0:00.42 cp_logrotate
4129 admin 15 0 0 0 0 R 18 0.0 8:23.67 fw_worker_0
15954 admin 15 0 2176 1112 840 R 2 0.0 0:00.09 top
14 admin 10 -5 0 0 0 S 0 0.0 47:01.09 events/0
4995 admin 15 0 207m 66m 24m S 1 1.7 79:04.08 cpd
1 admin 15 0 2044 724 624 S 0 0.0 0:00.43 init
2 admin RT -5 0 0 0 S 0 0.0 3:08.83 migration/0
3 admin 15 0 0 0 0 S 0 0.0 0:02.15 ksoftirqd/0
4 admin RT -5 0 0 0 S 0 0.0 0:00.05 watchdog/0
5 admin RT -5 0 0 0 S 0 0.0 2:46.42 migration/1
6 admin 15 0 0 0 0 S 0 0.0 0:00.01 ksoftirqd/1
7 admin RT -5 0 0 0 S 0 0.0 0:00.36 watchdog/1
8 admin RT -5 0 0 0 S 0 0.0 2:36.56 migration/2
9 admin 17 0 0 0 0 S 0 0.0 0:00.24 ksoftirqd/2

3. Check the distribution of connections across all CoreXL FW instances with the fw ctl multik stat command.

Example (refer to the "Peak" column - CoreXL FW instance 0 processed almost all connections, while other CoreXL FW instances were idle):

[Expert@HostName]# fw ctl multik stat

ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 3 | 1 | 21623
1 | Yes | 2 | 0 | 6
2 | Yes | 1 | 1 | 13
[Expert@HostName]#

(6) Limitations
The CoreXL Dynamic Dispatcher is not supported in the following scenarios:

Security Gateway is configured in VSX mode (not supported on any of the VSs, including VS0), in versions R80.10 and below.
SAM acceleration card is installed on the appliance
Carrier Grade NAT (CGN) is configured
Security Gateway is configured in Monitor Mode (per sk101670)
6in4 tunnel (SIT interface) is configured
Some lines in the $FWDIR/boot/modules/fwkern.conf file are commented out (refer to sk106309).

The following types of traffic are not load-balanced by the CoreXL Dynamic Dispatcher (this traffic will always be handled by the same CoreXL FW instance):

VoIP
VPN encrypted packets

Additional known limitations:

sk108894 - Difficulties in connecting to untrusted sites when both HTTPS Inspection and CoreXL Dynamic Dispatcher are enabled
sk108856 - R77.30 cluster member might go Down after disabling CoreXL Dynamic Dispatcher only on one member

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk105261&partition=Basic&product=Co… 4/5
5/19/2020 CoreXL Dynamic Dispatcher in R77.30 / R80.10 and above

(7) FAQ
When should I enable CoreXL Dynamic Dispatcher?

Note: The thresholds given in this answer are approximate and should be suitable for majority of environments.

Administrator should enable CoreXL Dynamic Dispatcher if:

1. A CoreXL FW instance consumes its CPU core at 85% (and above) even for 1 second.
2. Other CoreXL FW instances consume their CPU cores at 75% (and below) - i.e., the difference in CPU consumption between overloaded and
normally loaded CoreXL FW instances is 10% and more.
In addition, refer to "Limitations" section.

When will Security Gateway benefit from enabling CoreXL Dynamic Dispatcher?

When CPU load is not properly balanced, Dynamic Dispatcher will distribute the load equally between the CPU cores.
Dynamic Dispatcher will mostly benefit on machines with large number of CPU cores (multi core machines). On machine with small number of
CPU cores, the static hash distribution of traffic is good enough, and the benefit of enabling Dynamic Dispatcher is low.

Additional common scenarios, in which enabling CoreXL Dynamic Dispatcher will improve Security Gateway performance, are cases when traffic is
dropped/delayed due CPU load on Security Gateway.
For example, during traffic peaks, during policy installation.
Refer to "Monitoring CoreXL load distribution" section.

Can CoreXL assign some connections statically and some connections dynamically?

By default, CoreXL assigns all connections statically.

If you fully enable the CoreXL Dynamic Dispatcher, then CoreXL assigns all connections dynamically.
In addition, refer to "Limitations" section.

Does affinity of interfaces change when enabling CoreXL Dynamic Dispatcher?

CoreXL Dynamic Dispatcher does not affect affinity of interfaces.

For information about affinity of interfaces when CoreXL is enabled, refer to sk98737 - ATRG: CoreXL - section "(4) Architecture".

(8) Related documentation

Performance Tuning Administration Guide (R77.X, R80.10)

(9) Related solutions

sk61701 - CoreXL Known Limitations
sk98737 - ATRG: CoreXL
sk101878 - CPView Utility
sk105762 - Firewall Priority Queues R77.30 / R80.10 and above
sk109772 - R77.30 NGTP, NGTX and HTTPS Inspection performance and memory consumption optimization
sk52421 - Ports used by Check Point software
sk98348 - Best Practices - Security Gateway Performance
sk106665 - VoIP traffic, or traffic that uses reserved VoIP ports is dropped after enabling CoreXL Dynamic Dispatcher
sk108432 - Issues with traffic passing through Security Gateway with enabled CoreXL Dynamic Dispatcher

Give us Feedback Please rate this document [1=Worst,5=Best]

Enter your comment here

Comment 

©1994-2020 Check Point Software Technologies Ltd. All rights reserved.

Copyright | Privacy Policy

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk105261&partition=Basic&product=Co… 5/5