Document 1377929.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-sta...

Copyright (c) 2024, Oracle. All rights reserved. Oracle Confidential.

Network Tuning Best Practices for Oracle Streams Propagation (Doc ID 1377929.1)

In this Document

Purpose
Scope
Details
References

APPLIES TO:

Oracle Database - Enterprise Edition - Version 9.2.0.8 to 11.2.0.3 [Release 9.2 to 11.2]
Oracle Database Cloud Schema Service - Version N/A and later
Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.

PURPOSE

This documents discusses recommendations to tune the network layer in order improve the performance of This
documents discusses recommendations to tune the network layer in order improve the performance of Oracle Streams
Propagation. This is applicable to both Streams downstream and local capture configurations, and is most applicable for
configurations over a Wide Area Network (WAN).

SCOPE

The information contained in this note targets The information contained in this note targets Oracle support engineers and
Replication administrators to improve the performance of Streams replication in Oracle 9.2 and higher. This note contains
key recommendations for successful implementation of Streams in Oracle database release 9.2 and above.

DETAILS

Customers should be on the latest patchset and it is recommended to apply relevant patches as detailed in :

Document 437838.1 Streams Specific Patches . These patches address the most commonly encountered performance and
stability issues connected with Oracle Streams.

Also, the recommendation related to propagation tuning form the following note should have been implemented as well:

Document 335516.1 Primary Note for Streams Performance Recommendations.

Proper configuration is crucial to Streams performance where propagation is done

over Wide Are Network (WAN),.

When it has been determined that the propagation process is the bottleneck affecting the latency of streams replication.

Symptoms of this may include:

The capture processes is continuously pausing for flow control due to the high number of un-browsed messages in
the queue.

1 of 5 27/01/2024, 01:04
Document 1377929.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-sta...

The capture is enqueueing messages at a rate considerably higher than the one the propagation process is able to
dequeue messages.
The apply coordinator and apply server processes are idle most of the time

Network Tuning Best Practices

Set the TCP window size to its optimal value on all servers

Set the OracleNet RECV_BUF_SIZE and SEND_BUF_SIZE (TCP Socket Buffer Sizes) parameters equal to three
times the Bandwidth Delay Product (BDP). Doing so provides the largest increase in network throughput.

Set theOracleNet Session Data Unit (SDU) size to 32767.

Increase the default SEND and RECEIVE queue sizes associated with networking devices.

Ensure that theOracleNet TCP_NODELAY parameter is set to YES (which is the default value).

TCP WINDOW SIZE

Normally refers to the TCP Receive Window. A TCP Receive Window is a buffer on each side of the TCP connection that
temporarily holds incoming data. The data in this buffer is sent to the application, making more room for additional
incoming data. A device notifies of the current size of its TCP Window in the TCP Header information.

TCP SOCKECT Buffer Size

TCP socket buffer settings will control how much network bandwidth can be used regardless of the bandwidth
available in the network circuit.
Socket buffer sizes need to be increased from their default values to improve utilization of available bandwidth.
When network latency is high, larger socket buffer sizes are needed to fully utilize network bandwidth.
The optimal socket buffer size is three times the size of the Bandwidth Delay Product (BDP).

Bandwidth Delay Product (BDP)

Determines the amount of data that can be in transit in the network.

Is the product of the available bandwidth and the network Round Trip Time (RTT)

BDP = Bandwidth * RTT

Network Round Trip (RTT)

Is the time required for a packet to travel from a specific source to a specific destination and back again.

On a TCP/IP environment, RTT to a specific address can be determined using ping.

Available Bandwidth

Data rate supported by a network connection or interface.

Needs to be estimated using appropriate commercial or open source tools available.

Tuning Example using iperf

2 of 5 27/01/2024, 01:04
Document 1377929.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-sta...

iperf is an open source tool that can be used to measure the bandwidth and quality of a network link. I also allow setting
different TCP WINDOW sizes without having to modify any OS parameters fo tuning purposes.
There are version of the tool for Linux, Solaris, HP/UX, AIX and Windows.

$ iperf -c <IP Address>

------------------------------------------------------------
Client connecting to <IP Address>, TCP port <port no>
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[ 3] local <IP Address> port <port no> connected with <IP Address> port <port no>
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 17.9 MBytes 14.8 Mbits/sec
dl2672@csdbio1 vbios01d /export/home/dl2672

$ iperf -c <IP Address>

-----------------------------------------------------------
Client connecting to <IP Address>, TCP port <port no>
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[ 3] local <IP Address> port <port no> connected with <IP Address> port <port no>
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.4 sec 13.9 MBytes 11.2 Mbits/sec

The TCP WINDOW Size at the source and target database is set to 1MB and with that value the available bandwidth is 11.2
Mbits/sec.

If the window size is increased to 2 MB:

iperf -s -w 2M (source)
iperf -w 2M -c <IP Address> (target)

The available bandwidth increased to 19 Mbits/sec.

To set the TCP Window size permanently the following OS parameters need to be modified:

Solaris
--------

# ndd -set /dev/tcp tcp_max_buf 2097152

# ndd -set /dev/tcp tcp_xmit_hiwat 2097152
# ndd -set /dev/tcp tcp_recv_hiwat 2097152

Linux
-------

1. Set the network core write and receive memory buffer maximums:

net.core.rmem_max = 2097152

3 of 5 27/01/2024, 01:04
Document 1377929.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-sta...

net.core.wmem_max = 2097152

2. Set the TCP/IP minimum, default, and maximum buffer sizes:

net.ipv4.tcp_rmem = 4096 87380 2097152

net.ipv4.tcp_wmem = 4096 65536 2097152

The RTT between the two servers reported by ping is 60ms:

$ ping -s <server name>

PING <server name>: 56 data bytes
64 bytes from <server name> (<IP Address>): icmp_seq=0. time=60. ms
64 bytes from <server name> (<IP Address>): icmp_seq=1. time=60. ms
64 bytes from <server name> (<IP Address>): icmp_seq=2. time=60. ms

With this values, we can calculate the SEND_BUF_SIZE and RECV_BUF_SIZE as folows:

BPD = Bandwidth * RTT

So if the bandwith of your network link is 1000 Mbps and the RTT is 25 msecs BPD will be:

BPD = 1000000000 bps * 0.025 s

BPD = 25000000 bits
BPD = 25000000 bits / 8
BPD = 31250000 bytes

socket buffer size = 3 * BPD

socket buffer size = 3 * 3125000 bytes
socket buffer size = 9375000 bytes

To set the buffer sizes at the Oracle Net level, the following entries will need to be added either to the sqlnet.ora file of the
2 databases or the the listener.ora and tnsnames.ora files:

sqlnet.ora

RECV_BUF_SIZE=9375000
SEND_BUF_SIZE=9375000

tnsnames.ora

streamsdest10g_halinux06.<HOST> =
(DESCRIPTION=
(SDU=32767)
(SEND_BUF_SIZE=9375000)
(RECV_BUF_SIZE=9375000)

4 of 5 27/01/2024, 01:04
Document 1377929.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-sta...

(ADDRESS=(PROTOCOL=tcp)
(HOST=halinux06vip)(PORT=1521))
(CONNECT_DATA=
(SERVER = DEDICATED)
(SERVICE_NAME = streamsd.<HOST>)))

listener.ora
LISTENER=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=sales-server)(PORT=<port no>)
(SEND_BUF_SIZE=9375000)
(RECV_BUF_SIZE=9375000))

Still have questions ?

To discuss this information further with Oracleexperts and industry peers, we encourage you to review, join or start a
discussion in the My Oracle Support Streams and Distributed Database Community Forum.

The goal of this community is to exchange knowledge and concepts about Oracle Streams Advanced Queuing (AQ) and
distributed databases, with special consideration for the components listed below:
- Distributed Databases
- Streams Replication and Advanced Queuing
- Change Data Capture (CDC)
- Advanced Replication
- XA

To provide feedback on this note, click on the "Rate this document" link above.

REFERENCES

NOTE:437838.1 - Streams Recommended Patches

NOTE:335516.1 - Primary Note for Streams Performance Recommendations
Didn't find what you are looking for?

5 of 5 27/01/2024, 01:04