低延迟测试方案

低延迟测试方案
初稿 <徐彬彬>
修订 <徐彬彬>
日期 2013 年 06 月
网络改造升级方案 2013
文档控制
修改记录
日期作者版本修改记录
2013-06-28 徐彬彬 1.0 创建文档
2013-07-26 徐彬彬 1.1 升级驱动固件后的内容更新，优化内容描述补充
2013-08-08 徐彬彬 1.2 确定 netperf 和 sockperf 为测试工具，确定测试方法
审阅
分发
拷贝 No. 姓名单位
目录
一：整体系统环境介绍............................................................................................................................................4
二．服务操作系统安装/运行状态...........................................................................................................................4
三．优化内容............................................................................................................................................................9
四．测试内容..........................................................................................................................................................12
五．测试时间表......................................................................................................................................................14
一：整体系统环境介绍
目前硬件：HP DL380P Gen8 *2
操作系统 Rhel5.8 X64

RBSU 12/14/2012
网卡固件 5719-v1.31 n\NCSI v1.1.15.0
网卡驱动 tg3 3.119
cciss 版本 3.6.28
CPU 数量 2
内核数量每个 CPU 8 core 总计 16 cores
更新后
操作系统 Rhel5.8 X64

RBSU 12/14/2012
网卡固件 5719-v1.31 n\NCSI v1.1.15.0
网卡驱动 tg3 3.125g
cciss 版本 3.6.28-22
CPU 数量 2
内核数量每个 CPU 8 core 总计 16 cores
测试将会采用直连以及交换机互联两种模式进行。
二．服务操作系统安装/运行状态
目前服务器安装使用 ks 脚本 ks-hp-ctp_rhel5.8.cfg
安装软件包列表为
@admin-tools
@base
@chinese-support
@core
@development-libs
@development-tools
@editors
@gnome-desktop
@graphical-internet
@legacy-software-development
@system-tools
Page 4
@text-internet
@base-x
kexec-tools
iscsi-initiator-utils
fipscheck
device-mapper-multipath
sgpio
python-dmidecode
imake
emacs
audit
xorg-x11-utils
xorg-x11-server-Xnest
xorg-x11-server-Xvfb
-nspluginwrapper
分区状况为
clearpart --all --drives=cciss/c0d0
part /boot --fstype ext3 --size=100 --ondisk=cciss/c0d0
part pv.100000 --size=100 --grow --ondisk=cciss/c0d0
volgroup vg00 --pesize=32768 pv.100000
logvol swap --fstype swap --name=lv_swap --vgname=vg00 --size=32768
logvol / --fstype ext3 --name=lvol0 --vgname=vg00 --size=51200
logvol /var --fstype ext3 --name=lvol1 --vgname=vg00 --size=10240
logvol /usr --fstype ext3 --name=lvol2 --vgname=vg00 --size=5120
logvol /tmp --fstype ext3 --name=lvol3 --vgname=vg00 --size=5120

已禁用服务列表
chkconfig anacron off
chkconfig arptables_jf off
chkconfig atd off
chkconfig autofs off
chkconfig cups off
chkconfig cups-config-daemon off
chkconfig firstboot off
chkconfig haldaemon off
chkconfig isdn off
chkconfig kudzu off
chkconfig netfs off
chkconfig nfslock off
chkconfig pcmcia off
chkconfig portmap off
chkconfig rpcgssd off
chkconfig rpcidmapd off
chkconfig sendmail off
chkconfig smartd off
chkconfig snmpd off
chkconfig avahi-daemon off
grub.conf 参数为默认值
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Server (2.6.18-308.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-308.el5 ro root=/dev/vg00/lvol0 rhgb quiet
initrd /initrd-2.6.18-308.el5.img
title Other
rootnoverify (hd1,0)
chainloader +1
sysctl.conf 为默认未修改
# Kernel sysctl configuration file for Red Hat Linux
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel

kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536
# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
三．优化内容
RBSU 优化
当前 RBSU 设定为
参数当前值建议值
Intel Virtualization Technology Enable Disabled
HT Disable Disabled
Turbo mode Enable Disabled
VT-d Disable Disabled
Thermal configuration Max cooling Max cooling
HP Power profile Maximum performance Maximum performance
HP Power Regulator Static High performance mode Static High performance mode
QPI Link power mgmt. Disable Disable
Minimum Processor Idle Power Core State No c-state No c-state
Minimum Processor IdlePower Package State No p-state No p-state
Energy/Performance Bias Max Performance Max Performance
Collaborative Power Control Enable Disabled
DIMM Voltage Preference Optimized for performance Optimized for performance
Dynamic Power Capping Functionality Enable Disabled
Memory Power Savings Mode Balanced Maximum Performance
ACPI SLIT Perferences Disable Enabled
Processor Power and Util Monitoring Enable Disabled
Memory Pre-failure Notification Enable Disabled
Memory Refresh rate 2x 1x
关闭服务建议
按照 Configuration and Tuning HP Proliant Servers for Low-Latency Applications White Paper 中的建议
目前开启的服务中有如下服务可以关闭
acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
avahi-daemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
bluetooth 0:off 1:off 2:on 3:on 4:on 5:on 6:off
cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
cups 0:off 1:off 2:on 3:on 4:on 5:on 6:off
firstboot 0:off 1:off 2:off 3:on 4:off 5:on 6:off
hidd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off

iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off
iscsid 0:off 1:off 2:off 3:on 4:on 5:on 6:off
kudzu 0:off 1:off 2:off 3:on 4:on 5:on 6:off
lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off
mcstrans 0:off 1:off 2:on 3:on 4:on 5:on 6:off
mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
messagebus 0:off 1:off 2:off 3:on 4:on 5:on 6:off
netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
pcscd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
readahead_early 0:off 1:off 2:on 3:on 4:on 5:on 6:off
readahead_later 0:off 1:off 2:off 3:off 4:off 5:on 6:off
restorecond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rhnsd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rpcgssd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rpcidmapd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
sendmail 0:off 1:off 2:on 3:on 4:on 5:on 6:off
smartd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sysstat 0:off 1:off 2:on 3:on 4:off 5:on 6:off
xfs 0:off 1:off 2:on 3:on 4:on 5:on 6:off
yum-updatesd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
内核参数调整
net.ipv4.tcp_timestamps = 1 改为 0
描述：关闭 TCP 时间戳,减少包头的 11 字节。
作用：每个 tcp 包都会有一个发送和回包的时间戳，用以防止 tcp 包在回包延迟过大时导致包乱序。
在速度较慢的广域网中该功能应该开启。在响应速度较好的局域网内，可以关闭该功能。
net.ipv4.tcp_sack = 1 改为 0
描述：selective acks ，ack 包选择性确认，使 TCP 只重新发送丢失的包，不用发送后续所有的包
作用：A 节点向 B 节点发送 10000 个数据包，由于网络原因第 3000~5000 的 2000 个包传输失败。
Sack 会检测到这 2000 个包传送失败，并请求 A 节点重发第 3000~5000 个包。
该功能关闭的状态下则会重发 10000 个包。
net.ipv4.tcp_low_latency = 1
描述：TCP 协议栈的决策,更喜欢更低的延迟而不是更高的吞吐量
作用：rhel 在运行时更倾向于降低延迟或者更高吞吐。0 为更高吞吐，1 为更低延迟。
增加 TCP 最大和默认的 buffer
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.rmem_default=16777216
net.core.wmem_default=16777216
net.core.optmem_max=16777216
增加 TCP buffer 最小，默认和最大值限制
net.ipv4.tcp_wmem="4096 65536 16777216"
net.ipv4.tcp_rmem="4096 87380 16777216"
关闭 nagle 算法, 没有系统层面开关，需要在建立连接时定义 TCP_NODELAY（默认没有 TCP_NODELAY）
调整网卡
使用 ping –c <count> -M do –s <Bytes> <ipaddress>
得到这样的输出
说明 MTU 过大，需要逐步减少-s 后的包大小
直到找到合适的大小能正常发包后
普通网络 MTU 值在 1500，测试得到恰当值后用 ifconfig ethX mtu <specificed value>
或者在/etc/sysconfig/network-script/ifcfg-ethx 中增加 MTU=<specificed value>
网卡驱动参数调整：
tg3 在 3.119 中没有参数，在 3.125g 中有参数可供调整。
parm: tg3_disable_eee:Disable Energy Efficient Ethernet (EEE) support (int)
在/etc/modprobe.conf 中增加一行 options tg3 tg3_disable_eee=1
作用：禁用网卡节能，在进入和退出节能状态时网卡传输效率会降低，所以关闭 EEE 支持。
Grub 修改
在/boot/grub/grub.conf中的kernel行增加nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce

nosoftlockup 防止RHEL记录一个高优先级的线程在一个核上执行时间超过 soft lockup 阈值
intel_idle_max_cstate=0 防止操作系统override 在RBSU中设定的no cstate
mce=ignore_ce 忽略每五分钟Machine Check对于可修正错误的初始化，这可能会导致网络延迟的毛刺
四．测试内容
NETPERF
在两个服务器上安装 nerperf
从 http://www.netperf.org 下载 netperf，最新版本为 2.60
Compilation
./configure --enable-intervals --enable-histogram --enable-burst
make
makeinstall
两台服务器一台作为 Server，另一台作为 Client
1. Server 端执行 netserver -D –f

客户端执行：
TCP latency 测试：netperf -n 16 -H <netperf server ip> -c -C -t TCP_RR -l 120 -T 2,2 -v2 -- -m 64 -k
P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY,MAX_LATENCY
TCP 吞吐量测试：netperf -n 16 -H <netperf server ip> -c -C -t TCP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k
UDP latency 测试：netperf -n 16 -H <netperf server ip> -c -C -t UDP_RR -l 120 -T 2,2 -v2 -- -m 64 -k
UDP 吞吐量测试：netperf -n 16 -H 1.1.1.1 -c -C -t UDP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k
可用参数：
-c 声明在测试过程中包括连接的 establishment 和 tear-down 过程
-d
可用值有:
2 netperf send to netserver
4 netserver send to netperf
6 request /response test
-D TCP_NODELAY 关闭 nagle 算法
-k 选取 keyvalue 作为输出
例子中的值释义为
MIN_LATENCY
The minimum “latency” or operation time (send, receive or request/response

exchange depending on the test) as measured on the netperf side when the global -
j option was specified. Units: Microseconds.
MAX_LATENCY
The maximum “latency” or operation time (send, receive or request/response
P50_LATENCY
The 50th percentile value of “latency” or operation time (send, receive or
request/response exchange depending on the test) as measured on the netperf side
when the global -j option was specified. Units: Microseconds.
P90_LATENCY
P99_LATENCY
MEAN_LATENCY
The average “latency” or operation time (send, receive or request/response
STDDEV_LATENCY
The standard deviation of “latency” or operation time (send, receive or
-t 指定 socket 类型为 stream（数据流）或 dgram（数据报），stream 基于 TCP，dgram 基于 UDP
-T 指定协议
TCP
Select the Transmission Control Protocol
UDP
Select the User Datagram Protocol
SDP
Select the Sockets Direct Protocol
DCCP
Select the Datagram Congestion Control Protocol
SCTP
Select the Stream Control Transport Protocol
udplite
Select UDP Lite
-m 指定发送包的大小（需要-D 参数）
-M 指定接收包的大小（需要-D 参数）
Sockperf
UDP Server: sockperf sr –i <interface ip>
TCP Server: sockperf sr –I <interface ip> --tcp
Client:
TCP test: sockperf pp –i <sockperf server ip> -m <message size> -t <test time> --tcp
UDP test: sockperf pp –i <sockperf server ip> -m <message size> -t <test time>
测试工具 HP-TimeTest
按 cycle 或者 time 测试 cpu 获取 gettimeofday 的延迟，如果大于容差值则记录。
五．测试方式
1.停止服务
可以使用 stopservice.sh 脚本
#!/bin/bash
for SERVICE in acpid alsasound autofs avahi-daemon bluetooth conman cpuspeed cron cups cupsrenice
dhcdbd dnsmasg dund firstboot hidd ip6tables ipmi irda kudzu libvirtd lvm2-monitor mcstrans mdmonitor
mdmpd messagebus multipathd netconsole netfs netplugd nscd oddjobd pand pcscd postfix powersaved
psacct rdisc readahead_early readahead_later restoresecond rhnsd rpcgssd rpcidmapd rpcsvgssd saslauthd
sendmail slpd smartd smbfs suseRegister sysstat wpa_supplicant xfs vpbind yum-updatesd novell-zmd
do
chkconfig --level 2345 off
service stop
done
低延迟测试
可以使用脚本 lowlatencytest
用法：
Lowlatencytest <server ip> <client ip> [message size] [test time] [dir_suffix]
Message size ， test time 是可选项，如果不填则默认用 64 字节 message size，120s 作为 test time
dir_suffix 用作分类测试结果文件夹的后缀，方便用户归档测试结果。
使用 lowlatencytest 前需要保证 netperf 和 sockperf 已经按照文档四.测试内容中的要求编译安装完成。
测试结果都会保存在/root/result/下以测试开始时间+dir_suffix 命名的文件夹中。
六．测试结果
Netperf：
TCP_RR:
[root@FusionIO ~]# netperf -n 16 -H 1.1.1.1 -c -C -t TCP_RR -l 120 -T 2,2 -v2 -- -m 64 -k

MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 1.1.1.1 (1.1.1.1) port 0 AF_INET :
histogram : interval : first burst 0 : cpu bind
P50_LATENCY=85
P90_LATENCY=111
P99_LATENCY=136
MEAN_LATENCY=90.89
STDDEV_LATENCY=16.20
MAX_LATENCY=1096
Histogram of request/response times
UNIT_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_USEC : 0: 0: 0: 0: 0: 0: 0: 87045: 899291: 54667
HUNDRED_USEC : 0: 271578: 753: 705: 480: 0: 1: 0: 0: 0
UNIT_MSEC : 0: 4: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
HIST_TOTAL: 1314524
[root@FusionIO ~]# netperf -n 16 -H 1.1.1.1 -c -C -t TCP_RR -l 120 -T 2,2 -v2 -- -m 64 -k

P50_LATENCY=85
P90_LATENCY=112
P99_LATENCY=136
MEAN_LATENCY=91.45
MAX_LATENCY=1101
UNIT_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_USEC : 0: 0: 0: 0: 0: 0: 0: 35534: 930514: 80409
HUNDRED_USEC : 0: 258071: 730: 713: 500: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 9: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
HIST_TOTAL: 1306480
TCP_STREAM:
[root@FusionIO ~]# netperf -n 16 -H 1.1.1.1 -c -C -t TCP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k

MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 1.1.1.1 (1.1.1.1) port 0 AF_INET : histogram :
interval : cpu bind
P50_LATENCY=-2147483648
P90_LATENCY=1
P99_LATENCY=3
MEAN_LATENCY=0.49
STDDEV_LATENCY=0.88
MAX_LATENCY=249
Histogram of send times
UNIT_USEC : 123001767: 84167415: 2842462: 412223: 416472: 442132: 483974: 392126: 200137: 61589
TEN_USEC : 0: 25228: 1: 0: 1: 0: 0: 0: 0: 2
HUNDRED_USEC : 0: 1: 827: 0: 0: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
HIST_TOTAL: 212446357
[root@FusionIO ~]# netperf -n 16 -H 1.1.1.1 -c -C -t TCP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k

interval : cpu bind
P50_LATENCY=-2147483648
P90_LATENCY=1
P99_LATENCY=2
MEAN_LATENCY=0.47
**MAX_LATENCY=12129
UNIT_USEC : 134543244: 83803588: 541647: 517428: 589499: 430499: 183592: 59129: 20015: 9714
TEN_USEC : 0: 12606: 134: 27: 5: 0: 0: 0: 0: 0
HUNDRED_USEC : 0: 3: 762: 0: 0: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 0: 0: 0: 0: 0: 1: 0: 0: 0
TEN_MSEC : 0: **820: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
**关闭 avahi-daemon 后正常
[root@FusionIO log]# netperf -n 16 -H 1.1.1.1 -c -C -t TCP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k

interval : cpu bind
P50_LATENCY=-2147483648
P90_LATENCY=1
P99_LATENCY=3
MEAN_LATENCY=0.48
STDDEV_LATENCY=0.88
MAX_LATENCY=431
UNIT_USEC : 123927001: 84135793: 2973037: 336144: 429043: 462685: 492576: 375065: 186500: 55632
TEN_USEC : 0: 25646: 2: 0: 2: 0: 0: 0: 0: 0
HUNDRED_USEC : 0: 1: 870: 0: 1: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
UDP_RR
[root@FusionIO log]# netperf -n 16 -H 1.1.1.1 -c -C -t UDP_RR -l 120 -T 2,2 -v2 -- -m 64 -k

MIGRATED UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 1.1.1.1 (1.1.1.1) port 0 AF_INET :
P50_LATENCY=83
P90_LATENCY=93
P99_LATENCY=101
MEAN_LATENCY=84.49
MAX_LATENCY=1091
UNIT_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_USEC : 0: 0: 0: 0: 0: 0: 0: 63722: 1127334: 206581
HUNDRED_USEC : 0: 14246: 821: 690: 428: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 6: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
HIST_TOTAL: 1413828
[root@FusionIO log]# netperf -n 16 -H 1.1.1.1 -c -C -t UDP_RR -l 120 -T 2,2 -v2 -- -m 64 -k

P50_LATENCY=83
P90_LATENCY=107
P99_LATENCY=125
MEAN_LATENCY=89.38
MAX_LATENCY=1088
UNIT_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_USEC : 0: 0: 0: 0: 0: 0: 0: 1: 923761: 218447
HUNDRED_USEC : 0: 192587: 803: 652: 492: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 16: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
HIST_TOTAL: 1336759
UDP_STREAM：
[root@FusionIO log]# netperf -n 16 -H 1.1.1.1 -c -C -t UDP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k

MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 1.1.1.1 (1.1.1.1) port 0 AF_INET : histogram :
interval : cpu bind
P50_LATENCY=1
P90_LATENCY=3
P99_LATENCY=3
MEAN_LATENCY=1.50
STDDEV_LATENCY=1.34
MAX_LATENCY=4125
UNIT_USEC : 823244: 46601028: 19928037: 8350992: 402073: 23482: 62935: 21804: 4716: 192
TEN_USEC : 0: 16: 1: 0: 0: 0: 0: 0: 0: 0
HUNDRED_USEC : 0: 1: 947: 15: 11: 10: 10: 2: 1: 0
UNIT_MSEC : 0: 0: 0: 0: 2: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
[root@FusionIO log]# netperf -n 16 -H 1.1.1.1 -c -C -t UDP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k

interval : cpu bind
P50_LATENCY=1
P90_LATENCY=2
P99_LATENCY=3
MEAN_LATENCY=1.50
STDDEV_LATENCY=1.22
MAX_LATENCY=4183
UNIT_USEC : 462702: 43974581: 24927611: 6322585: 98215: 22242: 63435: 22527: 3046: 107
TEN_USEC : 0: 20: 1: 0: 0: 0: 0: 0: 0: 0
HUNDRED_USEC : 0: 0: 955: 9: 5: 8: 13: 2: 1: 0
UNIT_MSEC : 0: 0: 0: 0: 1: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
TCP_RR:
[root@localhost sockperf-2.5.208]# netperf -n 16 -H 1.1.1.2 -c -C -t TCP_RR -l 120 -T 2,2 -v2 -- -m 64 -k

P50_LATENCY=86
P90_LATENCY=110
P99_LATENCY=134
MEAN_LATENCY=92.47
MAX_LATENCY=1277
UNIT_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_USEC : 0: 0: 0: 0: 0: 0: 0: 2: 877698: 101352
HUNDRED_USEC : 0: 300954: 741: 662: 529: 1: 1: 0: 0: 0
UNIT_MSEC : 0: 22: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
HIST_TOTAL: 1281962
[root@localhost sockperf-2.5.208]# netperf -n 16 -H 1.1.1.2 -c -C -t TCP_RR -l 120 -T 2,2 -v2 -- -m 64 -k

P50_LATENCY=86
P90_LATENCY=110
P99_LATENCY=133
MEAN_LATENCY=91.88
MAX_LATENCY=1083
UNIT_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_USEC : 0: 0: 0: 0: 0: 0: 0: 23817: 880671: 92539
HUNDRED_USEC : 0: 290974: 766: 642: 523: 1: 0: 0: 0: 0
UNIT_MSEC : 0: 3: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
HIST_TOTAL: 1289936
TCP_STREAM:
[root@localhost sockperf-2.5.208]# netperf -n 16 -H 1.1.1.2 -c -C -t TCP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k

interval : cpu bind
P50_LATENCY=-2147483648
P90_LATENCY=1
P99_LATENCY=5
MEAN_LATENCY=0.54
STDDEV_LATENCY=1.17
MAX_LATENCY=349
UNIT_USEC : 111167348: 74045214: 3739705: 749793: 709087: 745439: 509456: 356555: 303755: 206749
TEN_USEC : 0: 375551: 715: 134: 1: 0: 0: 0: 0: 0
HUNDRED_USEC : 0: 1: 705: 158: 0: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
[root@localhost sockperf-2.5.208]# netperf -n 16 -H 1.1.1.2 -c -C -t TCP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k

interval : cpu bind
P50_LATENCY=-2147483648
P90_LATENCY=1
P99_LATENCY=5
MEAN_LATENCY=0.55
STDDEV_LATENCY=1.18
MAX_LATENCY=330
UNIT_USEC : 105421304: 71658364: 4074269: 874154: 642489: 752468: 498656: 351074: 302818: 198232
TEN_USEC : 0: 340009: 1037: 92: 1: 0: 0: 0: 0: 0
HUNDRED_USEC : 0: 1: 660: 199: 0: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
UDP_RR:
[root@localhost sockperf-2.5.208]# netperf -n 16 -H 1.1.1.2 -c -C -t UDP_RR -l 120 -T 2,2 -v2 -- -m 64 -k

P50_LATENCY=82
P90_LATENCY=94
P99_LATENCY=101
MEAN_LATENCY=83.59
MAX_LATENCY=1225
UNIT_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_USEC : 0: 0: 0: 0: 0: 0: 34: 162234: 1042615: 196580
HUNDRED_USEC : 0: 14228: 823: 681: 425: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 4: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
HIST_TOTAL: 1417624
[root@localhost sockperf-2.5.208]# netperf -n 16 -H 1.1.1.2 -c -C -t UDP_RR -l 120 -T 2,2 -v2 -- -m 64 -k

P50_LATENCY=84
P90_LATENCY=108
P99_LATENCY=128
MEAN_LATENCY=89.71
MAX_LATENCY=1093
UNIT_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_USEC : 0: 0: 0: 0: 0: 0: 0: 11909: 894324: 185575

HUNDRED_USEC : 0: 227217: 793: 650: 490: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 11: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
HIST_TOTAL: 1320969
UDP_STREAM:
[root@localhost sockperf-2.5.208]# netperf -n 16 -H 1.1.1.2 -c -C -t UDP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k

interval : cpu bind
P50_LATENCY=2
P90_LATENCY=3
P99_LATENCY=4
MEAN_LATENCY=2.21
STDDEV_LATENCY=1.57
MAX_LATENCY=342
UNIT_USEC : 53127: 14426383: 16875154: 17563499: 3271490: 176342: 56727: 37025: 16120: 2582
TEN_USEC : 0: 1432: 2: 1: 0: 0: 0: 0: 0: 0
HUNDRED_USEC : 0: 0: 674: 269: 0: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
[root@localhost sockperf-2.5.208]# netperf -n 16 -H 1.1.1.2 -c -C -t UDP_STREAM -l 120 -T 2,2 -v2 -- -m 64 -k

interval : cpu bind
P50_LATENCY=2
P90_LATENCY=3
P99_LATENCY=4
MEAN_LATENCY=2.21
STDDEV_LATENCY=1.58
MAX_LATENCY=340
UNIT_USEC : 48764: 14532790: 16739149: 17109761: 3661460: 177269: 56192: 37509: 16073: 2709
TEN_USEC : 0: 1499: 1: 1: 0: 0: 0: 0: 0: 0
HUNDRED_USEC : 0: 0: 685: 257: 0: 0: 0: 0: 0: 0
UNIT_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
>100_SECS: 0
Sockperf
[root@FusionIO ~]# sockperf -pp -i 1.1.1.1 -m 64 -t 120
sockperf is a tool for testing network latency and throughput.
version 2.5.208
Usage: sockperf <subcommand> [options] [args]
Type: 'sockperf <subcommand> --help' for help on a specific subcommand.
Type: 'sockperf --version' to see the program version number.
Available subcommands:
help (h ,?) Display list of supported commands.
under-load (ul) Run sockperf client for latency under load test.
ping-pong (pp) Run sockperf client for latency test in ping pong mod
playback (pb) Run sockperf client for latency test using playback o
throughput (tp) Run sockperf client for one way throughput test.
server (sr) Run sockperf as a server.
For additional information visit our website http://code.google.com/p/sockper
[root@FusionIO ~]# sockperf pp -i 1.1.1.1 -m 64 -t 120
sockperf: == version #2.5.208 ==
sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 1.1.1.1 PORT = 11111 # UDP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: Starting test...
sockperf: Test end (interrupted by timer)
sockperf: Test ended
sockperf: [Total Run] RunTime=120.100 sec; SentMessages=1558173; ReceivedMessages=1558172
sockperf: ========= Printing statistics for Server No: 0
sockperf: [Valid Duration] RunTime=120.000 sec; SentMessages=1556913; ReceivedMessages=1556913
sockperf: ====> avg-lat= 38.495 (std-dev=4.702)
sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
sockperf: Summary: Latency is 38.495 usec
sockperf: Total 1556913 observations; each percentile contains 15569.13 observations
sockperf: ---> <MAX> observation = 222.409
sockperf: ---> percentile 99.99 = 213.332
sockperf: ---> <MIN> observation = 35.897

[root@FusionIO ~]# sockperf pp -i 1.1.1.1 -m 64 -t 120 --tcp
sockperf: == version #2.5.208 ==
sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)
[ 0] IP = 1.1.1.1 PORT = 11111 # TCP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: Starting test...
sockperf: Test end (interrupted by timer)
sockperf: Test ended
sockperf: [Total Run] RunTime=120.104 sec; SentMessages=1446882; ReceivedMessages=1446881
sockperf: ========= Printing statistics for Server No: 0
sockperf: [Valid Duration] RunTime=120.004 sec; SentMessages=1445686; ReceivedMessages=1445686
sockperf: ====> avg-lat= 41.461 (std-dev=7.916)
sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
sockperf: Summary: Latency is 41.461 usec
sockperf: Total 1445686 observations; each percentile contains 14456.86 observations
sockperf: ---> <MAX> observation = 1263.113

sockperf: ---> <MIN> observation = 37.283

低延迟测试方案

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

低延迟测试方案

Uploaded by

Copyright:

Available Formats

低延迟测试方案

2013-07-26 徐彬彬 1.1 升级驱动固件后的内容更新，优化内容描述补充

2013-08-08 徐彬彬 1.2 确定 netperf 和 sockperf 为测试工具，确定测试方法

操作系统 Rhel5.8 X64

操作系统 Rhel5.8 X64

clearpart --all --drives=cciss/c0d0

part /boot --fstype ext3 --size=100 --ondisk=cciss/c0d0

part pv.100000 --size=100 --grow --ondisk=cciss/c0d0

volgroup vg00 --pesize=32768 pv.100000

logvol swap --fstype swap --name=lv_swap --vgname=vg00 --size=32768

logvol / --fstype ext3 --name=lvol0 --vgname=vg00 --size=51200

logvol /var --fstype ext3 --name=lvol1 --vgname=vg00 --size=10240

logvol /usr --fstype ext3 --name=lvol2 --vgname=vg00 --size=5120

logvol /tmp --fstype ext3 --name=lvol3 --vgname=vg00 --size=5120

chkconfig anacron off

chkconfig arptables_jf off

chkconfig atd off

chkconfig autofs off

chkconfig cups off

chkconfig cups-config-daemon off

chkconfig firstboot off

chkconfig haldaemon off

chkconfig isdn off

chkconfig kudzu off

chkconfig netfs off

chkconfig nfslock off

chkconfig pcmcia off

chkconfig portmap off

chkconfig rpcgssd off

chkconfig rpcidmapd off

chkconfig sendmail off

chkconfig smartd off

chkconfig snmpd off

chkconfig avahi-daemon off

title Red Hat Enterprise Linux Server (2.6.18-308.el5)

kernel /vmlinuz-2.6.18-308.el5 ro root=/dev/vg00/lvol0 rhgb quiet

# Kernel sysctl configuration file for Red Hat Linux

# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and

# sysctl.conf(5) for more details.

# Controls IP packet forwarding

# Controls source route verification

# Do not accept source routing

# Controls the System Request debugging functionality of the kernel

# Useful for debugging multi-threaded applications

# Controls the use of TCP syncookies

# Controls the maximum size of a message, in bytes

# Controls the default maxmimum size of a mesage queue

# Controls the maximum shared segment size, in bytes

# Controls the maximum number of shared memory segments, in pages

acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off

autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off

avahi-daemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off

bluetooth 0:off 1:off 2:on 3:on 4:on 5:on 6:off

cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off

crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off

cups 0:off 1:off 2:on 3:on 4:on 5:on 6:off

firstboot 0:off 1:off 2:off 3:on 4:off 5:on 6:off

hidd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off

iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off

iscsid 0:off 1:off 2:off 3:on 4:on 5:on 6:off

kudzu 0:off 1:off 2:off 3:on 4:on 5:on 6:off

lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off

mcstrans 0:off 1:off 2:on 3:on 4:on 5:on 6:off

使用 lowlatencytest 前需要保证 netperf 和 sockperf 已经按照文档四.测试内容中的要求编译安装完成。