■■■■ ■■ ■■■■■■■■ ■2■ 2nd

Edition Brendan Gregg

Visit to download the full and correct content document:
https://ebookstep.com/download/ebook-56256090/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Mon tour du monde en 80 tortues 1st Edition Bernard

Devaux

https://ebookstep.com/product/mon-tour-du-monde-
en-80-tortues-1st-edition-bernard-devaux/

„■■■■ ■■■■ ■ ■■■■■■■■■...“ : ■■■■■■■ ■ ■■■■■ 80-■■■■

■■■■■■■■ ■■■■■■■■■ ■■■■■■■■■ ■■■■■

https://ebookstep.com/download/ebook-56209260/

Eine Anleitung zum Glücklichsein - 80 anregende Impulse

zur psychischen Gesundheit 1st Edition Alexander
Hüttner

https://ebookstep.com/product/eine-anleitung-zum-
glucklichsein-80-anregende-impulse-zur-psychischen-
gesundheit-1st-edition-alexander-huttner/

■■■■ ■■ ■■■■■■■■ ■2■ 2nd Edition Brendan Gregg

https://ebookstep.com/download/ebook-56256090/
Instituto Nacional de Antropología e Historia 80 años
Jaime Bali Editor Adriana Konzevik Editor

https://ebookstep.com/product/instituto-nacional-de-antropologia-
e-historia-80-anos-jaime-bali-editor-adriana-konzevik-editor/

Gesamtausgabe Bd 91 Ergänzungen und Denksplitter IV

ABTEILUNG HINWEISE UND AUFZEICHNUNGEN Martin Heidegger
Mark Michalski Editor

https://ebookstep.com/product/gesamtausgabe-bd-91-erganzungen-
und-denksplitter-iv-abteilung-hinweise-und-aufzeichnungen-martin-
heidegger-mark-michalski-editor/

Die 80 wichtigsten Management- und Beratungstools: Von

der BCG-Matrix zu den agilen Tools Dirk Lippold

https://ebookstep.com/product/die-80-wichtigsten-management-und-
beratungstools-von-der-bcg-matrix-zu-den-agilen-tools-dirk-
lippold/

Green Arrow Especial 80 Años 1st Edition Mariko Tamaki

Javier Rodríguez Tom Taylor Mike Grell Nicola Scott

https://ebookstep.com/product/green-arrow-especial-80-anos-1st-
edition-mariko-tamaki-javier-rodriguez-tom-taylor-mike-grell-
nicola-scott/

Konverter AC AC Prinsip dan Aplikasi Tole Sutikno Tri

Wahono

https://ebookstep.com/product/konverter-ac-ac-prinsip-dan-
aplikasi-tole-sutikno-tri-wahono/
 l

可
8

@化arson ww邯b「o■□pJ●懈c@而.c∏

0
能之
｜

（第2版）
‖

系统｀企业与云可观测性 ｜

愈子工寸出靡触.
回 中国工信出版集团 圈 二＝……●＝≡

｜
………m

｜
 ■

能之 （第2版 ） ｜

尸
统｀企业与云可观测性
百
起牲

徽贸镶嚣
铲蔚.露谱禽瞬磷露飞.≡撬撼
″四余

撼″个繁憾
骤潞哆＝′令
镭严′目亮

々…’□夕吼』》■
…瞬著＝择｀

…∩…嘴′′＂

鞠。
′肉□＝【∏·

［美］B「e∩da∩G
徐章宁吴寒思

藩豫嚼;露…么,罐罐舅卜″
』角

G □

Syste1nsPerf℃r∏1ance
巨∩te『p「｜sea∩dt∩eC｜oud
（Seco∩d巨d｝t｜o∩）

｜ 乞子工扩出硒抛.
PublisMngHouseofE］ect丁onicSI∏du5tIV
北京.BEIJING
 ‖‖‖‖‖‖‖‖】】】】】■■■■■■】■
＄

内容简介

大型介‖M服务、云计算和虚拟计算系统都面临着严峻的性能挑战宁蛔全.国愿知名的性能专家BrcndanGrcgg将
业界验证的方法、工具和指标融汇在一起’足以应对复杂环境的性能分析和调优工作.
本书着力讲述Lm皿的性能,但所论述的性能问题适用于所有的操作系统·你将了解到系统是如何工作与执行的,
还将学习到如何分析和改进系统及应用程序性能的方法·
本书对第1版的内容做了大量的更新,这些更新包括但不限于:近年来Lm皿内核各方面的变化对于资源性能的
影响;云计算架构的主流演进方向;动态跟踪工具的新星（BPF及其前后端技术）;常见性能工具的使用方法的变化等.
需要说明的是’在第1版中进行性能分析所用到的术语、概念和方法’在第2版中几乎没有变化’依然中肯适用’经
得起时间的检验·
本书的目标受众主要是系统管理员及企业与云计算环境的运维工程师。所有需要了解操作系统和应用程序性能
的开发人员、数据库管理员和网站管理员都适合阅读本书.对于刚接触性能优化的学生等人员』本书还提供了包含
G眶gg丰古教学经验的练习题.
／Au小°渝切cdmnslati。nmmmcEngljshlanguagecdjti。Lentitl函Systc酗PeⅡfmmancc:EntcmqS…dˉ呼CP吟S§con9
Edjtion’9780136820150byBrendanGI℃ggopublishedbyPem召onEducatjon,Inc.,publishingasAddjsonˉWesleyPm允ssiona1,
Copyri碘t@2021PcamonEducanon’Inc.
锣AⅡrigh挖rcsc【ved.Nopanofthisbookmaybercproducedor∏ansmjttedinanyibImorbyanymeans,e1ecnDnjcoT
mechanica』,inc1udjngphotocopy】ng》recordingorbya【Vimb∏nations0omgcretricva1systcm,wjthouhpeNmissionhomPearson
Educanon,Inc.
CHINESESIMPLIFIEDlanguageeditionpub1ishedbyPUBLISHINGHOUSEOFELECTRONICS∏qDUSTRY
CO,LID.’C卯yright@2022.
本书简体中文版专有出版权由PeamonEducation（培生教育出版集团）授予电子工业出版社有限公司’未经出版
者书面许可’不得以任何方式复制或抄袭本书的任何部分.专有出版权受法律保护。
本书简体中文版贴有PemsonEduca前on《培生教育出版集团）激光防伪标签,无标签者不得销售·
版权贸易合同登记号图字:01ˉ2021ˉ2413
｜
图书在版编目（c｜尸）数据
性能之戚:系统、企业与云可观测性:第2版／（美）布兰登.格雷格（BrendanGregg）著;徐章宁等译.＿北京:电
子工‖l出版社’ 20227
书名原文:SystemsPeTfbImance:EnteIpriseandtheC1oud,SecondEditjon
ISBN978ˉ7ˉ121ˉ43587斗

I .@性…∏.@布…@徐…IⅡ.@计算机网络Ⅳ.@TP393
中国版本图书馆CIP数据核字（2022）第089986号

责任编辑:张春雨
印 刷:三河市良远印务有限公司
装 订:三河市良远印务有限公司
出版发行:电子工业出版社
北京市海淀区万寿路l73信箱 邮编: l0O036
开 本: 787×980 1／l6 印张: 53.5 字数: 1173千字
版 次: 20l5年8月第1版
2022年7月第2版 、

印 次: 2022年7月第1次印刷
定 价: 23M0元

凡所购买电子工业出版社图书有缺损问题,请向购买书店调换·若书店售缺’请与本社发行部联系,联系及邮购
电话;（010）88254888’88258888.
质量投诉请发邮件至zIts@phej.comcn,盗版侵权举报请发邮件至dbqq@phei.comcn°
本书咨询联系方式;（010）51260888ˉ819’ faq@phei。com.cn.
 ｜
■『『

｝
‖

推荐语
｝
当前云计算等复杂场景下的系统性能问题日益严峻,需要专业的方法才能找到瓶颈
所在,然后才能有针对性地对其做优化’以让业务高并发地运行。本书从性能分析所需
要的内存、文件系统、网络、CPU等方面进行了详细介绍’并结合丰富的案例将读者快
速带入实战状态’作者还总结了一套性能分析的理论和方法’让读者彻底把提高性能的
法宝掌握在手中.另外,本书针对如今火到天际的BPF前后端技术及相应工具也进行了
介绍。本书的作者Gregg是国际知名的性能专家,译者在操作系统和性能领域也是身经
百战、经验丰富,无论你是系统运维人员、研发工程师’还是性能架构师,通过阅读本书’
都能帮你打开解决性能问题的大门°强烈推荐本书!

毛丈安,龙晰社区系统运维SIGMajntainer
●卜｜臼■一巳

系统性能问题-直是我们面临的难题之-°本书从系统性能、优化分析到应用程序
性能及CPU、内存、文件系统、磁盘、网络’并结合案例由浅入探地讲述了性能方面的
知识°本书的作译者都是性能优化领域的专家’实战经验丰富°本书对系统管理员、运
维工程师、研发工程师具有极高的参考价值’阅读本书,收获颇多,力荐!

田国杰（@JackTian）’微信公众号“杰哥的IT之旅,,作者

【
本书详尽地介绍了企业环境和云计算环境下的性能测试方法与工具,能为读者带来
立竿见影的帮助。“无法衡量就无法管理”’本书先对组件性能进行分析与观测,然后在
此之上进行改进°书中的方法与思路能够“授人以渔”, 引领读者在系统性能领域进行

） 长期思考°
优秀的著作离不开译者的付出’本书的译者通过精准的翻译’原汁原味地将这本经
『｝
▲■

｝
 】

IV 推荐语

典技术著作呈现给大家°

『■司■
无论你是系统管理员、云计算运维人员、技术爱好者’还是立志成为系统性能架构师、
专家,这本经典著作都值得推荐给你· ;
』
吕昭波’
《云端架构》作者

｜‖
g

多年前我第一次接触到火焰图这种神奇的性能剖析方法时’BrendanGregg令我惊
为天人’后来读《性能之獭:洞悉系统、企业与云计算》一书更是受益良多.这次再版’
作者补充了近年热门的可观测性、eBPF、Kubenertes等方面的知识’并将第一版中多处
分析方法进＿步细化成具体的观测工具指南。高屋建领的理论配合深入浅出的落地实践,
自
绝对是广大IT工程师的案头必备书籍.

饶深琳’日志易产品副总裁’前微博系统架构师

分析性能问题需要的技术是很广泛的’操作系统、数据库、网络、语言、存储、架
构等方面的知识都需要理解和掌握°而要想具备性能瓶颈分析和定位的能力,掌握操作
系统方面的知识是绝对跳不过去的一个环节。
本书从性能分析的角度出发来理解操作系统’对广大开发人员梳理性能分析思路有
很好的借鉴作用°本书在不同的操作系统模块中提供了实用的分析思路和方法’为应用
程序、CPU、内存、网络、文件系统、磁盘等的性能分析提供了对应的工具’同时给出
了相应的操作实例’最后还结合ˉ些基准测试工具以及BPF工具给出进一步进行系统级
性能分析的操作指导。这是＿本如果要学习性能分析就必然要看的书°
高楼,盾山科技CE◎＼7DGr°up创始人

在计算机技术的发展长河中’性能问题一直是绕不开的话题°现在市面上针对性能
测试的图书有很多’但和本书—样面面俱到的’却是凤毛腆角°本书不仅从操作系统、
CPU、内存、文件系统、磁盘、网络等方面介绍了计算机性能的基础知识’同时还与时俱进’
从云计算等当前热门技术的角度’介绍了如何进行性能优化。除此之外’本书还介绍了
性能测试的相关方法论和性能测试工具’具有很高的实用性°本书的作译者都是性能领
域的专家’在业内享有极高声望。这本经典的著作’非常适合研发人员、运维人员、测
试人员当作性能优化的参考书’力荐!

臻叔’微信公众号“程序员臻叔”’作者
厂■■

｜
｜

推荐序1

大数据、云计算和人工智能都是热门概念,也是现实问题.很多团队都面对类似的
技术抉择:如何用开源软件构建机群、如何选择云服务、如何设计高效的分布式Web服务’
）

卜
或者如何开发高效的分布式机器学习系统’当面对这些问题的时候,最好能从—些重要
的可度量的方面进行定量的分析和比较°可是哪些方面重要’以及如何度量呢?
古往今来有很多重要的系统度量问题’比如如何度量网络繁忙程度’如何得知某台
机器是否还活着’服务中断是因为某个进程锁死了还是机器出问题了’或是我们的机群
中用SSD替换硬盘的比例应该多大’等等·
这些问题的答案都不简单—正确答案往往构建在对操作系统的深刻理解上’甚至
构建在统计学和统计实验的基础上°可是通常我们是在繁重的业务压力下应对这些问题
｜｜〖■厂｜‖｜｜｜■『■尸

的,所以只能尽快找-个“近似’’的测量办法。这终非长久之计—在我们经历了很多
问题之后,可能会发现自己从未深刻理解问题’没有探入思考’没有沉淀经验’没有获
得成长·
如果有意深入理解问题,借助前人的经验是可以事半功倍的·只是关于操作系统的
教科书不能为我们提供足够的基础知识。操作系统如此复杂’几乎涉及计算机科学的所
有方面。学校往往只能基于简化的示范系统教学’比如Minix°但实际工作中使用的操
作系统’比如Linux和Sola汀s’有更多需要学习和理解的地方·
我做分布式机器学习系统有8年了’其间很多时候要面对系统分析的问题。但是坦
p 诚地说,大部分情况下我都只能尽快地找-个“近似”方法’处于没有时间深入琢磨上
述系统问题的窘境。看到本书之后’我不禁眼前＿亮。这本书从绪论之后’就开始介绍“方
‖

法”—概念、模型、观测和实验手段。作者不仅利用操作系统自带的观测工具』还自
己开发了＿套深入分析观测结果的脚本’这就是有名的DTraceTbolkit（大家可以直接
 』勺∩‖□｜』】〗〗■』‖□司‖‖‖｜‖』‖］‖■‖｜』Ⅷ
vI 推荐席↑

找来使用）。本书介绍的实验和观测方法,包括内存、CPU、文件系统、存储硬件、网
络等各个方面.而且,在介绍方法之前会深入介绍系统原理—我没法期望更多了!
很高兴看到这＿经典名著的中文版问世.因为被邀作序’有幸近水楼台先得月,深
受教益·感谢译者和编辑的努力。相信读者朋友们定能从中受益·

」■】叫·夕‖·司』日］｜
王益

Iinkedm高级主任分析师

司』』
■可』‖』‖‖‖‖』‖日■

｜
｛
—β

｝
推荐序2

收到侠少的邀约来写序’多少有点志恋和紧张。志态是因为平时太多时间总花在解
决各类问题上’没有时间好好静下心来读—读书’总结一下发现问题和解决问题的方法;
紧张是因为我好像很久没有写文章了’上一次写长篇大论还是在很多年前,怀疑自己是
◆

不是能够在有限的文字内把系统性能的问题解释清楚。于是,就在这样的背景下’我从
出版方那里拿到了书稿’迫不及待地读了起来°
本书的作者Gr℃gg先生是业内性能优化方面大名鼎鼎的人物,他早年在Sun公司
｛尸‖

的时候是性能主管和内核工程师’也是大名鼎鼎的DTTace的开发人员’要知道DTmce
可是众多∏ace类工具中最著名的’并且先后被移植到很多操作系统上。全书都在讨论
卜

性能优化’我想了想,我每天从事的工作不就是这个吗? SAE上成千上万的开发者们’
他们每天问的问题几乎全部和性能相关:为什么我的APp打开比较慢’为什么我的网站
访问不了’怎么才能看到我的业务中哪个逻辑比较慢……对于这些问题,其实难点不在
于解决它’最难的在于发现并定位它’因为定位了故障点或者性能瓶颈点’解决起来不
是很复杂的事情。
比

对于性能优化’最大的挑战就是性能分析’而性能分析要求我们对操作系统及网络
的性能要了如指掌’明确各个部分的执行时间数量级,并能做出合理的判断,这部分在
书中有很详细的讨论,让读者可以明确地将这些性能指标应用在80 : 20法则上°
工欲善其事’必先利其器。了解系统性能指标后’还需要找到合适的工具对可能存
在的瓶颈进行分析’这要求我们具备全面的知识’涉及CPU性能、内存性能、磁盘性能、
文件系统性能、网络协议栈性能等方方面面。本书详细介绍了诸如DTTace、vmstat、
mpstat、sar、Systemmp等工具,如何将这些工具组合并应用在适当的场景中,是＿门学问,
相信读者会在书中找到答案。顺便说一句’DTrace＋SystemThp帮助SAE解决过非常多

‖
0
｜｜｜■■■■尸｜｜【■
 ■

VIII 推荐序2

的性能疑难杂症,—定会对读者的业务分析有所帮助!
单个进程的性能分析是简单的,因为我们可以定位到系统调用或者库调用级别’然

·】‖｜｜｜｜｜｜■■■■‖
后对照代码很快就能解决问题,但整个业务的性能分析是复杂的’这里面涉及多个业务
单元、庞大的系统组件6最麻烦的是’往往造成性能问题的不是单元本身’而是使单元
和单元相连接的网络服务°这就要求我们有一套科学的分析方法论’帮助定位整个系统
业务中的瓶颈.书中就此介绍了包括随机变动、诊断循环等多种方法’并且介绍了涉及
分析的数学建模和概念°不要忽视数学在性能分析中的作用’在实际应用中’我就曾利

｜｜
用方差和平均值的变化规律科学地分辨过＿套系统到底是否应该扩容°
‖

找到了性能瓶颈,下＿步就是解决问题°当然解决问题的最好办法就是改代码’但是’
当你无法在短时间内修改代码的时候’对系统进行优化也可以实现这一目的。这就要求
我们弄明白系统各个环节的工作原理和联系°本书第3章详细讨论了操作系统’这对读
者是很有用的。因为很多时候我们在不改代码的情况下优化系统’也就是优化内存分配
比例,优化CPU亲密度’尝试各种调度算法’做操作系统层面的各种网络参数调优等°
对于上述所有问题的认识’我相信读者在通读全书后会有不—样的感觉.记住’不 q

要只读-遍,每读＿遍都必有不同的体会·不多说了’我要赶紧再去读一遍!

‖■
丛磊

新浪SAE创始人／总负责人

｜
』

｜
 口】□‖ˉ■厂｜｜｜｜■■■■■■■■尸■■■■■■■■■■『‖‖‖‖‖

推荐序3

人类正在用软件重构这个世界°从20世纪40年代电子计算机出现’到个人电脑风
靡、互联网大行其道’再到如今正兴旺的云计算和移动互联网’还有蓬勃发展的物联网’
所有这些表面上看是计算机硬件变得无处不在’而实质上是软件一步步掌管了这个世界。
短短几十年’软件轻松征服世界!渗透、渗透、再渗透’软件已经进入所有人的生活。
b

不言而喻’软件对这个世界和人类的重要性越来越高。我很负责任地说’软件的健
康与否关系着世界的安危·君不见’几多时’＿个软件漏洞便让全球惊慌.不经意间’
■∏

我们与软件变得休戚与共。
不幸的是’软件的总体健康状况并不乐观’问题很多°
Bug是软件行业的＿个永恒话题’是破坏软件质量的头号大敌。但迄今为止’完全
发现和彻底消除Bug还只是奢望’是不可能实现的目标。换句话来说,掌管着我们生活
的所有软件都是在带着Bug运行的’是真正的带“病”工作。因为每个软件内部都有纵
横交错的无数条路径’CPU经常“奔跑”的那些路径上的Bug早己被发现和移除’所
以软件大多数时候并不发“病”。但也有时候’CPU会遭遇软件中的Bug’发生意外。
目前,我们能做的只有努力多发现Bug,并尽可能找到根源’将其去除。但这并不是一
件容易的事情。
β
除了Bug’性能问题是威胁软件健康的另＿个大敌°简单来说’我们把软件中的错
误归为Bug。把那些在速度、资源消耗、工作量等方面的不满意表现纳为性能问题。用
p

员工考核做比喻’Bug是把＿件事做错了’而性能问题是虽然做对了’但是做得不够好,
U 可能开销太大’用的资源太多,可能完成速度太慢,用的时间太久’还可能完成的工作
量太少’活干得不够多’总之是结果不令人满意’还有必要改进。
举例来说,某支付软件在性能方面存在问题。一旦运行’就会频繁触发大量的缺页
异常,消耗的CPU时间过多’导致不必要的能源浪费°随着软件变得无处不在’软件
■β

的耗电问题已经引起越来越多的关注。在我写这几行文字时,该软件的几个进程仍在—
■『

▲ ＿
 ｜β
X 推荐序3
｜
刻不停地触发着缺页异常’过去两天累积的异常数量已经超过千万,消耗CPU的挣时
间超过1小时°粗略估计’这几个进程至少已经消耗了0.01度电。不要忽视0O1这个
看似微小的数字’将它乘以全国的总用户数’立刻就变成＿个庞大的数字了·
与软件Bug类似,性能问题也可能危害巨大!更可怕的是’性能方面的问题容易触
发隐藏在软件深处的Bug,直接导致软件崩溃或者其他无法预计的故障°
发现Bug根源的过程-般被称为调试（Debug）°纠正性能问题’提高软件性能的
努力被称为调优（TUne）。不论调试’还是调优’都不是简单的事’对软件工程师的技
术要求很高.处理一些复杂的问题时’常常需要多方面的知识,需要对系统有全面了解’
既有大局观’能俯嫩全局’又能探微索隐,深入关键的细节’可谓‘!致广大而尽精微’’。
如果一定要把调试和调优的难度比＿下’调优的难度更大。简单的解释是’调试的
主要目标是寻找Bug,Bug固定存在于软件中的某一点°因此’调试时可以通过断点等
技术让软件停止,慢慢分析。而调优必须关注一个动态的过程’观察＿段时间内的软件
行为。这样＿来,调优时’不可以让软件中断’而需要以统计学的方法或者其他技术对
软件做长时间监测。
记得两年前’曾经有一位同行以饱经沧桑的神情问我:“在中国这样的软件环境里’
做技术的工程师应该如何发展呢?难道都得像你那样写＿本书吗?”坦率地说’我当时
没能给出让这位同行很满意的回答°因为这个问题确实不太容易回答.此事之后’我常
常想起这个困扰着很多同行的问题°多次思考后’我似乎有了比较好的答案。首先要确
认自己是喜欢软件技术的,愿意在技术方向上持续发展。接下来的问题是如何在技术方
向上不断前进’‘‘日日新’又日新”。我的建议是’逐级攀登软件技术的三级台阶:编码、
调试和调优。
代码是软件的根本’一个好的软件工程师必须过代码这一关,写代码时如行云流水’
读代码时穿梭自如、如履平川。以调试器为核心的调试技术是对抗软件Bug的最有力工

日｜
具,是每位技术同行都应该佩戴的一把利剑。调优技术旨在发现软件的性能障碍’让软
件运行得更快、更好°随着对软件性能问题的重视’调优技术的发展也越来越快’新的
工具层出不穷。调优方面的工作和创业机会也在不断增加·几年前’我写作《软件调试》
时’很多人还不太重视调试,但最近几年,软件调试己经逐渐从藏在背后的隐学逐步走
向前台’成为一门显学°可以预见’性能调优也会受到越来越多的重视。 ｜
学习调优技术有很多挑战,很高兴看到有这样一本关于系统优化的好书被引进到国 ‖

内。好友侠少诚邀作序,盛情难却’仓促命笔’词不达意’请诸君海涵’是为序。
司■

张银奎
』□

资深调试专家’《软件调试》和《格蠢汇编》作者
■■ˉ」□口■■引‖』】
∏〗〗‖‖『‖【〖尸『｜｜｜｜｜｜｜｜｜｜｛‖‖｜｜｜｜｜｜｜｜｜｜｜｜□‖『｜｜｜｜｜｜｜｜‖‖【『‖｜

推荐序4

性能调优是每一个系统工程师应具备的最重要的技能’也是衡量其水平高低的不二
法门°Linux是开源的操作系统,这也意味着它本身的可调整范围比较大°近十几年来,
硬件设备日益复杂’互联网应用场景及Web20蓬勃发展的同时’也带来了各种高并发
的业务应用’有些复杂的分布式数据分析系统’单集群的物理服务器数量甚至就能达到
上万台·这使得对系统运行环境的—点点优化带来的收益都可能非常可观。
性能调优这件事情’大家往往都有话想说’技术专家也都有些秘而不宣甚至奉为压

｜
箱底 的“绝活”。但这些“绝活”’往往来自个案所获得的经验°例如’解决了某大型电
商网站的Nginx服务器问题、某MySQL数据库集群性能问题等。这些特定的大型案例’
促使参与其中的技术人员’在某个或某些系统性能优化方面’具有较高水平。
但即使是这些技术专家,也难以解决所有性能问题。这有两方面原因,一方面自身
缺乏 对整个系统运行环境的全局把控能力。技术专家的能力,是通过某些问题点扩散开
来而 获得的’并非基于事先构建好的系统优化的全局观（这也和国内环境有关’大家往
往大 学毕业后就直接开始从事相关工作’即使参与了某些培训课程’也很缺少底层、结
构性的学习）。
这种系统优化的全局观之所以难以形成’一个原因在于“未知的未知”’也就是说
我们 不知道自己不知道°比如’我们可能不知道设备中断其实会消耗大量CPU资源,
因而 忽略了解决问题的关键线索。再比如,初中级DBA可能并不知道应用程序连接
Oracle时’每一个数据库连接（Sessjon）实际上都非常消耗物理内存’成百上千个数据

库连 接长期驻留（看上去状态还是非活动的）’PGA会被消耗殆尽’引起各种异常,成
为性 能问题的罪魁祸首°
另-个原因在于性能问题的根源太过复杂。诚如作者所说,＿来性能是主观的’连
判断 是否有性能问题’都因人而异·例如’磁盘平均读／写时间为1mS’这是好还是不
X∏ 推荐瞬

好?是否需要调优?这实际上取决于开发人员和最终用户（有时还包括领导）的性能预
期·二来系统是复杂的°例如’本来CPU／磁盘／内存各司其职,有了内存缓存（SWAP）
机制’内存不够时可以使用部分硬盘空间来顶替·这看上去很好’但对于数据库系统而言’
是否启用SWAP,本身就是＿个问题。再比如,对于云计算而言’多虚拟机共享物理
机’这进-步增加了问题的复杂度°资源隔离是个技术活儿’现有技术很难做到磁盘Ⅱ
O完全隔离。另外最近非常流行的容器技术’让问题变得尤为复杂。容器即进程’不
像KVM等虚拟机（KVM至少还会进行资源隔离）自带操作系统·容器并不是为IaaS
而生的’仅靠C厘oup等隔离技术能做的非常有限°第三点是有可能多个问题并存.有
时终端用户抱怨系统慢,很可能不仅是由单个原因引起的’例如’业务负载猛增’内网
速度1000MB／s其实已经不够’但没引起注意;或是整体对外交付能力表面上看还正常,

‖
但数据库磁盘I／O非常繁忙;系统还可能正偷偷地进行大量SWAP交换。
以上两方面原因’使得大部分技术专家,即使对系统优化的某些领域确有独到见解’
但说到能否解决所有系统性能问题时,都会显得有些底气不足°但本书作者看起来是个
例外.纵观全书’作者建立了系统性能优化的体系框架’并且骨肉丰满’很明显’他不
仅擅长某方面的性能优化,而且是全方位的专家,加之他作为DITace（＿种可动态检测
进程状态的工具）的主要开发者’使得本书的说服力和含金量大增°
本书首先提及性能优化的方法论和常见性能检测工具的使用,具体内容更是涉及可
能影响Linux系统性能的方方面面’从操作系统、应用程序、CPU、内存、文件系统、
磁盘到网络’无所不包。在以上这些话题的探讨中’作者的表述方法值得称道＿每章
都程式化地介绍了术语、模型、概念、架构、方法、分析工具和调优建议等’这对于因
长期工作形成一定强迫症的某些技术人员,如我自己’阅读时感到赏心悦目’这也从侧
面体现了作者深厚的技术功底和文字驾驭能力°
本书提供的性能优化方法论也令人印象深刻’包括几种常见的错误思考方法’如街
灯法、随意猜测法和责怪他人法。街灯法来自一个著名的醉汉的故事＿醉汉丢了东西,
但只会在灯光最亮的地方寻找。这种头疼医头脚痛医脚、错把结果当原因的事情,相信
很多人都遇到过。大型业务系统上线,大家都围着DBA贡问数据库为什么崩溃了’但 ｜

数据库出问题是结果’数据库本身＿定是问题的根源吗?是否更应该从业务负载、程序
代码性能、网络等方面联合排查?在列举各种不正确的方法后’作者建议采用科学法,
科学法的“套路’’是:描述问题→假设→预测→实验→分析°这种办法的好处是’可以
逐—排除问题’也可以减少对技术专家个人能力和主观判断的依赖° ‖

本书用单独的章节系统性地介绍了操作系统、性能检测方法和各种基准测试,还特
别介绍了作者主导开发的DTrace（在本书的例子中’用DTTace监控SSH客户端当前执
行的每个命令并实时输出）’这使得本书作为工具书的价值更得以彰显。云计算的出现’
对系统优化带来新的挑战。作者作为某云计算提供商的首席性能工程师,带来了一个真
 推荐围ⅪII
｜
｜｜

实的云客户案例分析’包括如何利用本书提及的技术、方法和工具,一步—步地分析和
解决问题。
很多时候’受限于语言障碍’系统工程师往往通过国内BBS、论坛等获得知识’只
‖‖■巴β‖‖【【】〗『

在性能问题确实棘手时’被迫找些英文资料’寻找解决思路。
博文视点出版的这本中文版图书’对于国内广大运维同人而言’实属幸事°这让我
们有机会系统学习和掌握性能优化的各个方面,有机会建立＿种高屋建领的全局观·这
样,在面对复杂系统问题时’不至于手足无措’或只能盲人摸象般试探。另外’虽然
面对日益复杂的硬件设备和高并发的业务应用’问题不是变得简单而是更为复杂’但
Linux系统演化至今’其最基础的体系架构和关键组件并未发生太多改变,这使得这本
好书即使历经多年’价值也毫无衰减,反而历久弥新°
总的来说’如果早些接触到本书’该有多好!

萧田国
触控科技运维总监’高效运维社区创始人

片
■■

0
■

｝ ●

｜
推荐序5

性能的话题’从一开始就是复杂的°性能是＿种典型的非功能需求’然而又贯穿在
任何一种功能需求中,直接影响系统的运行效率和用户体验。也正是由于这一特性,性
能无法简单地通过单-的、直线式的思维来度量和管理’而注定需要以系统工程的方法
来掌握和调整°绝大多数的图书在谈到性能问题时,都是仅从片面的若干现象出发来触
及问题的冰山一角’抑或干脆语焉不详,甚至避而不谈°这也难怪’因为这个话题一旦
展开,就会占用极大篇幅,相对于原先的论题而言就显得喧宾夺主了。然而更重要的原
因也在于,对性能问题有着全面认识’并且能够给出系统化的分析和全栈式论述的作者
实在不多。相关的要求近乎苛刻:既要对系统的每一个部件都了如指掌’又要深入理解
部件之间的协作方式;既要精通系统运行的细节,又要明白取舍逻辑的大局观;既要懂
得现象背后的原理’又要把握从执行开发部署的工作人员直至终端应用用户的需求乃至
心态。
本书以—种奇妙而到位的方式’把高屋建领的视角和脚踏实地的实践结合了起来’
对性能这一复杂、微妙甚至有些神秘的话题进行了外科手术般的解析’读来真是让人感
觉豁然开朗。
全书以罕见的遍历式结构,对软件系统的每一个部件都如鹿丁解牛般地加以剖析’
几乎涉及业务的每一个细节°然而,对这些细节并非简单罗列’而是每-段论述都与具
体的角色和场景紧密结合’取舍之间极见智慧。方法论更不是单说理’而是通过＿个又 d
】‖
一个的具体实例’逐步地建构起来的’并反复运用于各个部件之上,使读者明白原理普
适性的同时也知道怎样举一反三° 0

本书也是难得的UNⅨ／Lmux系统管理员和运维工程师的百科全书式参考手册’相
对于工作在Wmdows上的同行而言’他们获得的知识更加零碎,甚至在很多场合下不得
司（｜｜｜』■一■■■

不求助于网络上的只言片语’并只能通过耗时的、高风险的生产环境实验来取得—手经
［
推荐序5 Xv

｜｜
验数据。本书当然提供了不少趁手的软件工具供读者使用’然而其更大的价值在于心法
的传授,即怎样利用工程师现在就熟悉、现在就可用的工具来迅速地进行性能建模’完
成故障排除和调优的关键步骤。书中的内容非常新’作者见过大世面’是以最与时俱进
的大型云计算系统为出发点来落笔的’对付日常的性能问题完全没有压力’即使是最新
的硬件的性能问题也能找到对应的解决方案。
本书的译者团队阵容强大’成员皆是在底层系统有多年一线工作经验的运维工程师
和开发工程师。徐章宁几乎是以一己之力支撑起PB级数据运维的明星DevOps’而另外
两位也都是手工实现过复杂生产环境中文件系统和网络协议的“大牛’’。可以说,他们
对于性能的认识是经过多年实际工作的考验的,是深刻而且务实的’这为本书翻译在专
业性方面提供了坚实的保证。他们多年养成的认真严谨的工作习惯’和深厚的中英文功
底,更是为该译本的可读性锦上添花°
希望所有的IT从业者都能从本书受益’让天下的系统都能达到性能之颠!

高博
青年计算机学会论坛（YOCSEF）会页,文津奖得主’
《研究之美》译者

‖‖

o卜

卜
 ｜｜
推荐序6

性能问题—直是个热门话题,在单机时代’我们就投入了不计其数的人力物力进行
研究。随着互联网行业的发展’分布式系统开始大量投入应用’对于性能问题的分析、
调优’更是面临着很多前所未有的挑战。特别是如何做到单机性能与集群整体性能的平
衡’以及在各种影响性能的要素之间进行取舍’成为摆在开发运维人员面前的巨大难题°
本书采用自下而上的结构’从底层的操作系统、CPU、磁盘等基础元素开始’到从
工作原理层面分析性能受到的各种不同影响’以及如何评估、衡量各项性能指标’让读

‖｜
者知其然并知其所以然’在面对实际情况时能够更有针对性地做出判断和决定’而不是
机械地、教条地行事°本书还提供了案例’＿步步地展示了实际性能问题的排查调优过程.
读者可以根据案例’结合业务系统实际情况展开工作°此外’本书还对常用的性能分析
工具的使用和扩展做了详细介绍’这对日常工作效率的提升有很大的帮助。
译者徐章宁曾与我在EMC的云存储部门共事多年,其在系统性能排查、调优方面
有着丰富的经验,很高兴他能参与本书的翻译。审稿过程中,我感觉译者不仅忠实地还
原了原著的精华’还融合了自己多年工作中积累的经验教训。
我坚信’本书无论对于开发人员还是运维人员’无论对于设计、编码还是调优工作,

‖
都能发挥重要的参考作用’尤其适合常备于案头。在此诚挚地向大家推荐本书°
‖□

林应

淘宝技术部高级技术专家

｜
｜

‖
推荐序7

性能测量的水相当深,斯坦福大学的JohnOuste『hout教授在“AlwaysMeasureOne
LevelDeeper”一文（《ACM通讯》杂志’2018年第7期）中提到,在他亲历的几十次
系统性能评估中,没有哪次的首批测量结果是正确的’都是先掉进坑里’再慢慢爬出来°
就拿文件I／O来说’你测得的是真实的磁盘性能’还是操作系统中文件系统的性能?
存储是分级的。操作系统有页缓存’高档的磁盘控制器（阵列卡）有带电池备份的写缓存’
机械硬盘上有高速缓存’混合型机械硬盘带有SSD缓存’消费级TLCSSD上带有SLC
缓存。文件UO的性能主要取决于读写操作击中的是哪-级缓存’而缓存颠箍（thrashing）
是系统过载时造成性能急剧下降的重要因素。
如今的计算机系统极度复杂,性能优劣往往违反直觉·不少人可能先入为主地认为
mmap（）比read（）／wrjte（）读写文件更快’因为前者减少了系统调用和内存拷贝。这听上
去很有道理,也正是20世纪80年代UNIX引入mmap（）的理由.但是卡内基梅隆大学
的AndrewCrotty等人通过分析与测量’对此提出相反的观点’他们关于这个观点的论
p 文“AreYOuSureYOuWanttoUseMMAPinYOurDatabaseManagementSystem?”发表在
CIDR2022会议上。
b

性能瓶颈时常发生在意想不到的地方’我们遇到过服务器硬件升级导致性能下降的
案例。新的服务器有更大的网络带宽、更快的CPU和更多的核’我们原本预计它每秒
能处理更多的计算密集型请求°但很快我们发现’其上运行的—个网络服务程序的响应
时间偶尔会飘升,甚至出现超时°初步探查发现此期间CPU和网络带宽的使用率都不高’
远未饱和°进-步分析发现’这个服务程序会按比例抽取＿些客户请求写在本地磁盘,
以供日后分析°由于新机器更强大’它收到的请求也比原来多’但新磁盘的速度没有多
大提升’这就使得磁盘带宽吃紧。如果同一台机器上同时有其他程序大量读写磁盘’那
么就会造成服务程序阻塞在磁盘I／O上’CPU无所事事,响应时间也自然会增加·更糟
糕的是,负载均衡器主要依据CPU繁忙程度来分配新任务’试图让多台服务器的CPU

｜
 XVI∏推荐席7

的使用率一致,以避免出现延迟波动°这个服务程序因为等在磁盘I／O上,反映出CPU
使用率低的假象,结果反而被分到了更多的请求,排在队列里等候处理,最终造成请求
超时.＿开始谁能料到这个计算密集型网络服务程序的性能最终受限于磁盘带宽?排查
性能故障就像阅读侦探小说’当谜底层层揭开的时候’读者会恍然大悟’反思自己怎么
没有早点想到“凶手”是谁呢?系统的性能瓶颈可能出现在CPU、内存、硬盘、网络甚
至总线上’本书第6～10章详细介绍了各种情况的应对方法°
现代CPU不再是常速运行的·功耗是限制CPU主频提高的主要因素,多核CPU
在整体功耗允许的情况下’会把其中某一个或几个核自动‘‘超频”,以提高程序性能°
这使得汪空闲机器上做的单线程基准测试的结果＿般不能简单推广到多核.比如,消费
级12核Alm5900X的单线程memcpy0的速度可达20GB／s’但是双通道DDR4内存

｜
能提供的总带宽约为50GB／s’平摊到每个核上大约才4GB／s’是单核峰值的1／5。我还
曾经试验把一个单线程的程序改成两个线程交替计算’在多核台式机上测试发现性能下
降50％以上’远超我的预期。后来发现CPU为了节能’在工作量不饱和的情况下’自
动降频运行°两个慢速的CPU核来回接力运算当然远差于一个全速的核。本书第12章
介绍了更多性能基准测试的陷陕与防范措施.
体系结构领域的图灵奖得主、加州大学伯克利分校的DavidPatterson教授在《ACM
通讯》2004年第10期发表的‘‘Latencylagsbandwidth”文章中指出’带宽的增长远远
快于延迟的进步’因为把公路修宽比把路修短容易得多°内存带宽大约每5年翻一倍。
2020年的DDR5内存带宽是2000年DDR1带宽的16倍’但访问延迟变化不大’还是
30ns左右’是现在CPU时钟周期的l00倍左右。跟硬盘相比, 内存当然是很快的,但
是跟CPU运算速度相比’内存访问可以说是相当慢的’遍历链表如今可以入选头号
CPU性能“大杀器”°在必要的情况下’程序员可以通过调整内存布局来提高局部性’
以求提高CPU缓存命中率。Google开源的自家C〈ˉ卜基础库中的absl::Hathash一map正
是利用了这一点’实现了比标准库的开链unordered一map快2～3倍的性能。
互联网先驱, TCP拥塞控制算法的发明人VanJacobson博士在2018年netdev0x12
会议的开场主题演讲中提到’从1990年算起’以太网带宽的增速甚至超过了摩尔定律。
康奈尔大学的博士生蔡其哲等人发表在SIGCOMM21上的论文‘‘UnderstandmgHost
NetworkStackOverheads”中的数据表明’经过调优’Linux中单个TCP连接的网络吞
吐量可达5GB／s’memcpy0只比它快几倍°为了实现高效安全可靠的网络传输,消息在
发送之前’常经过压缩（Zstd）、加密（AES）、校验（SHA1）等算法处理。而尽管有
｜｜可

AES／SHA指令加速’CPU执行这些算法的吞吐量还不及TCP速度的一半°如果设计不
当’加密与计算校验和这样的前期准备工作会成为收发网络消息的性能瓶颈’而TCP／ q

IP协议栈反而乐得清闲’这就有点儿膛恤了。我们当然不能因噎废食’为了速度而放
弃安全性·一个办法是把这些耗费CPU和内存带宽的工作下放（offload）到内核和网

｜
｜
》

推荐席7 XIX

卡里去做。Netβjx从2015年开始往FreeBSD内核增加了TLS功能,然后与网卡厂商合
仿

作’把TLS硬件加速做到网卡里·在2019年的EuroBSDCon会议上’NetHix汇报了成
果—“KemelTLSandhaIdwmETLSoⅢoadmFreeBSD13”,然后在2021年汇报了最
新进展～＿“ServmgNetHixVideoat40OGb／sonFreeBSD′’。注意’这里的400Gb／s是多
个连接汇总的吞吐量,而且也没有提及压缩’因为Ne出ix传输的视频数据本身己经是高
｜ 度压缩过的了。受此檄励, Linux也不甘落后’从413版开始逐步支持内核TLS与硬件
加速.

总而言之’提升系统性能与进行代码性能优化是完全不同的领域’它不是找到热点
函数加以改写’而是通过对系统的总体把握与观测,精确定位软硬件性能瓶颈,进而化
解并突破°
本书作者BrendanGregg是全球知名的实战派性能专家,他发明的火焰图是分析
CPU开销的有力工具’如今已是我们日常性能分析的标配.这本书是他多年工作经验的
总结,既有理论深度,又有丰富案例’在同类图书中是较为全面深入的’非常值得深入
阅读°这个新版本我会第＿时间购买,常备案头’随时查阅·

陈硕
《Linux多线程服务端编程:使用muduoC拌网络库》作者
2022年5月28日

｝
【■｜户｜仁

｝
 司

推荐序S

“我的梦想是将计算机性能分析变成一门科学,让我们可以完全了解所有事物的性
能:应用程序、库、内核、管理程序、固件和硬件°’’本书作者BrendanGregg在2019
年AWSre;Invemtalk的开场白中如此说°本书以操作系统为背景,讲解操作系统和应用
程序的性能问题’针对企业环境和云计算环境编写而成.作为一名讲授操作系统课程近
三十年的老师’我在近几年的教学改革中,把讲解操作系统的优化作为教学目标之一。
因此,我对本书就有了一份独特的期待,更期待性能工程成为一门学科’这门学科’将
融合计算机科学、软件工程、统计学以及数学等多门学科°
eBPF作为Linux内核—种革命性的技术’为Lmux内核的发展带来了勃勃生机.自
己在Linux内核教学和科研＿线驰骋二十多年后,看到了eBPF技术给业界带来的新气象’
■

本书新版中专门增加了BPF章节和BPF工具的介绍。
Gregg在Ne出ix积累了丰富的云计算环境下的性能优化的经验后又加入Intel,按
他的说法’“将为从应用程序到硅的所有领域开发新的性能和调试技术°这将跨越所有
xPU（CPU、GPU、ⅣU等）,并对世界产生巨大影响”°本书可作为大家入门性能优化
的起点’以此为基础’一方面深入操作系统内核’另—方面广泛积累应用程序的优化经验’
』司
最后再回归到硬件’你将走入一个更广泛的世界。

陈莉君 ■

西安由『电大学教授
2022年5月26日
q

q
｜
｜

推荐序9

作为高性能计算和云计算系统的设计＼开发和运维人员,任何时候讨论性能优化总
是＿个充满挑战却令人期望的事情。你似乎永远无法知道所设计的系统架构、调度策略
和数据结构是否能达到最理想的状态’但经过各种不断尝试后因取得的性能提升而获得
的满足感却是如此让人陶醉·
本书从性能分析和调优的基础概念和方法入手,沿着操作系统内核、,应用、CPU、
内存、文件系统、磁盘、网络、云计算系统以及调优工具的线索’展开一幅计算机系统
性能优化的全栈式画卷’向读者介绍影响计算机系统性能的关键因素’以及探入理解如
何动手去解决这些性能瓶颈°
作为国家超级计算中心的总工程师’让高性能计算机系统在运行时任务执行最快、
用户在队列的等待时间最短、系统平均无故障时间越长,是我追求的永恒目标。我所带
领的运维与系统开发团队一直在为提高我国超级计算的系统性能努力’这本书让我有推
荐给我们运维团队每—个成员的强烈冲动。毫无疑问’本书的内容将有助于我们深入理
解系统性能背后的知识,建立完整的“系统观’’°
同时我还要将本书推荐给我的研究生们’值得每—位立志做体系结构研究的学生将
其作为入门教材’在动手进行高性能系统和云计算系统优化研究前’对何为性能优化有
＿个全面完整的认识’有助于他们在研究过程中将问题考虑得更全面。
最后我要把这本书推荐给所有从事高性能计算与云计算领域的从业者’无论你是资
深架构师还是运维人员’相信我’本书对你重新思考系统设计、性能优化将会起到很大
的作用,值得一读。

唐卓
0

国家超级计算长沙中心总工程师、湖南大学教授
2022年5月26日

｜
 ｜

推荐序ⅢO

作为＿名讲授操作系统课程的教师’性能一直是课堂中很难的主题’因为你永远
不知道什么样的性能问题在等着你°就在几天前’我在部署一批机器时排查了一台机
器的性能问题’最终发现竟然是USB接口发生了短路（＿个真正的物理bug）’导致
kworker／0:0＋Pm一直处于活动的状态!正是本能地使用了Linux的perf工具（本书在第
13章中详细介绍）帮助我瞬间定位到xhCi相关的代码’将性能问题指向了USB子系统。
在暴力‘‘拆除”USB接口后,问题解除。
在课堂上’我会和学生谈论“理解性能就是理解程序在时间轴上的执行”’但这谈
何容易!我们既想看得清楚’对每条指令执行的踪迹和所属都了如指掌’又不想付出任
何代价,因为任何插入程序中的探针都会干扰程序的执行.在系统中的所有组件都成为
木桶的短板之后’性能调优更是一项艰苦卓绝’但却对生产系统来说至关重要的任务。
幸运的是’今天我们有了许多成熟的性能分析方法和工具°我找不到任何一份资料
可以和BIendanGregg的这本全面｀深入的书籍相提并论.BrcndanGregg作为性能分析 0■

领域首屈一指的专家’毫无保留地将其真知灼见和技术细节与读者分享·读完本书’你
就有了世界上最先进的性能分析“武器库’′,不再惧怕任何系统的性能分析.

蒋炎岩
q

南京大学教师 日

2022年6月4日

」
｜
瞩″

译者简介
徐
＝旱

←丁

嚣
’慰

劲鞍

目前就职于小红书,担任SRE专家工程师’负责混滩工
鳞Ⅺ 程等云原生可观测性项目的研发°曾就职于百度上海研发中
‖｜｜｜‖‖‖‖‖‖』甘】】【■】】】】】〗『’庐～■｜》

撼撼蹿蹿
川《尸丸≡酞萨

葛 赣 心和EMC中国研发中心’担任SRE运维工程师°对于云原
衰
矗
生计算领域发生的一切变革抱有热忱的态度’对大型系统运
猛瞬 维和性能调优有浓厚兴趣。
。沁

吴寒思

囊; 目前就职于eBay中国研发中心’担任软件技术经理, 负
…鱼醒 责广告系统、推荐系统和搜索系统的研发。曾就职于EMC
『田

中国研发中心’担任丈件系统研发工程师°对大数据、机器
■

学习和性能调优有浓厚兴趣°
段盟拭

圃伞巧
…

余亮

目前就职于百度’担任SRE资深研发工程师°负责
幽
确 早钾
混池工程、智能运维等稳定性工程项目的研发°曾就职于
■

Syn。psγs上海研发中心’担任SWE工程师°喜欢钻研架构优
p
化、性能调优等技术。
■厂≈ ■▲

肌曰

存在已知的已知,有些事情我们知道自己知道°
我们也知道有已知的未知,也就是说’我们知道有些事情自己不知道。
但是还有禾知的未知’有些事情我们不知道自己不知道°
—唐纳德.拉姆斯菲尔德’美国前国防部长’2002年2月12日

虽然上述发言在新闻发布会上引来了记者的笑声’但是它总结了一个重要的原则’
适用于任何如地缘政治般复杂的技术系统:性能问题可能来源于任何地方,包括系统中
因你一无所知而不曾检查的地方（未知的未知）。本书将揭示许多这样的领域’并为其
分析提供方法和工具。

关于这-版
我在8年前写了本书第1版’并为第l版设计了很长的“保质期”。各章首先涵盖
的是持久的技能（模型、架构和方法）,其次是将变化较快的技能（工具和调优）作为
实施的例子.虽然工具和调优的例子会过时’但持久的技能可告诉你如何保持更新°
在过去的8年中’Linux有了很大的发展°eBPF作为一种内核技术’为新一代的性
能分析工具提供动力’包括Ne咀ix和Facebook等公司都在使用。我在本书第】版中加
入了BPF章节和BPF工具’同时我还出版了关于这个主题的更深入的参考资料［Gregg
19］°Linux的perf和Ftrace工具也有了很大的发展’我为这两个工具增加了单独的章节。
同时对于Lmux内核新增的许多性能特性和技术’本书也有涉及.驱动云计算虚拟机的
管理程序和容器技术也发生了很大的变化,我对这些内容也进行了更新。
第1版同时涵盖了Linux和So1a面s°如今’Solaris的市场份额已经大幅缩水
［ITJobsWatch20］,因此第2版中基本删除了有关Sola∏s的内容’为更多的Linux内容
■■■■■■■■■
 前言XXV

｜
■

腾出空间。然而’作为一个视角’通过思考替换方案,可以增强你对操作系统或内核的
理解。出于这个原因’本版中包含了一些涉及Solaris和其他操作系统的内容。
在过去的6年里’我一直是Ne诅ix的高级性能工程师,将系统性能思考践行于
｜ Ne田iX的微服务环境。我＿直从事着改善管理程序、容器｀运行时、内核、数据库和应
用程序的性能的工作·我根据需要开发了新的方法和工具’并与云性能和Linux内核工
程的专家合作°这些经验都有助于改进第2版的内容。

关于本书
本书以操作系统为背景讲解操作系统和应用程序的性能问题’针对企业环境和云计
算环境编写而成。本书的目的是帮助你更好地利用自己的系统°
当你的工作与持续开发的应用程序为伍时’你可能会认为内核经过几十年的开发调
整’操作系统的性能问题早已解决’但事情并非如此!操作系统是一个复杂的软件体’
管理着各种不断变化的物理设备,应对着不同的新应用程序的工作负载。内核也在持续
地发展’不断增加新的特性以提高特定的工作负载的性能,随着系统继续扩展’所遇到
的瓶颈被逐一移除。一些内核变化’如2018年的Meltdown漏洞的缓解措施’也会损害
性能°分析并努力提高操作系统的性能是一项可以不断进行的持续任务。在操作系统的
上下文中做应用程序的性能分析’可以找到更多的线索’而这些线索很可能被只针对应
用程序的工具所遗漏’这些我在本书中会进行介绍。

操作系统的范围
本书的重点就是系统性能的研究’以英特尔处理器上基于Linux的操作系统为主要
例子。本书的内容结构也有助于你对于其他内核和处理器的学习。
除非另有说明,具体的Linux发行版本在本书所使用的例子中并不重要。这些例子
大多来自Ubuntu发行版’必要时会有注解说明它与其他发行版的差异.这些例子也取
自各种系统类型:裸机和虚拟化’生产环境和测试环境’服务器和客户设备。
在我的职业生涯中’我与各种不同的操作系统和内核打过交道,这加深了我对操作
系统和内核设计的理解·为了加深你的理解’本书也会提及-些UNⅨ、BSD、Solarjs
和Wjndows的内容。

其他内容
本书中的示例会包括性能工具的截屏’这样做不仅是为了显示数据’而且是为了对
可用的数据类型进行阐释°—般来说’工具展现数据的方式更为直观’很多UNlX早期
风格的工具生成的输出都是相近的’意义常常不言自明。这意味着屏幕截图可以很好地
XxvI前言

传递这些工具的意图,其中的-些可能仅需要极少的附加说明。（如果—款工具需要费
力地进行说明’它就很可能是一个失败的设计! ） ｜
我将触及某些技术的历史,这能提供有用的见解来加探你的理解·除此之外’了解
这个行业的一些重要人物也是很有用的:你很可能会碰到他们或者接触到他们在性能领
域的工作成果。附录E是—份关于“名人录”的清单。
本书中的少数主题在我之前的书《BPF之獭》［Gregg19］中也有涉及:特别是BPF、
BCC、bpItace、tracepoint、l甲robe、uprobe,以及各种基于BPF的工具。你可以参考该
书以获得更多信息。本书中对这些主题的讲解通常是基于那本较早的书’有时候还使用
了相同的文字和例子。
｜
哪些内容未提及
本书着眼于性能°如果你要执行所有的示例任务,有时可能需要做一些系统管理员
的工作’包括软件的安装或编译（这些本书没有提及）°
书中关于操作系统内部总结的内容会在单独的章节中提供详尽的介绍°对性能分析
高阶专题的概述,是为了让你知道这些内容的存在’以便在需要的时候依靠其他的知识
来源进一步学习,可参见本前言末尾的“补充材料与参考”部分°

本书的结构
本书的内容如下°

第↑章,绪论。介绍系统性能分析’总结关键的概念并展示了与性能相关的一些
例子。
第2章’方法°介绍性能分析和调优的背景知识’包括术语、概念、模型、观测和
实验的方法、容量规划、分析,以及统计°
第3章,操作系统°总结了内核内部的性能分析知识°对于解释和理解操作系统的
行为’这些是必要的背景知识°
第4章’观测工具°介绍系统可用的观测工具的类型’以及构建这些工具所基于的
接口和框架.
第5章’应用程序。讨论了应用程序性能的内容’并从操作系统的角度观测应用
程序。
第6章,C尸U°内容包括处理器、核、硬件线程、CPU缓存、CPU互联、硬件互
联及内核调度。
第7章,内存。本章涉及虚拟内存、换页、交换、内存架构、总线、地址空间和内
存分配器°
第8章’文件系统°介绍了文件系统ⅡO性能’包括涉及的不同缓存。
 肖唁XXVI【
｜
第9章,磁盘。内容包括存储设备、磁盘UO工作负载、存储控制器、RAID’以
及内核ⅡO子系统。
第‖0章,网络.内容涉及网络协议、套接字、接口’以及物理连接。
第↑↑章,云计算。介绍了广泛应用于云计算的操作系统级和硬件级虚拟化方法,
以及这些方法的性能开销、隔离和观测特征.这＿章涵盖了管理程序和容器·
第‖2章,基准测试·介绍了如何精确地做基准测试’如何解读别人的基准测试结果.
这是—个棘手的话题.这—章会告诉你怎样避免常见的错误,并试图理解它们°
第↑3章’pe∏·介绍了标准的Lmux剖析器perf（1）’ 以及它的诸多功能°这是
Perf（1）在全书中的使用参考。
第‖4章, 尸trace。介绍了标准的Lmux跟踪器Ftrace’Ftmce特别适用于探索内核
代码的执行。
第↑5章’BP「。讲解了标准的BPF前端—BCC和bpfirace。
第‖6章’案例研究·包含—个来自Netf］iX的系统性能案例’展示了如何从头开始
分析—个生产环境性能难题°

第1～4章提供了必要的背景知识°阅读完这几章后,你可以根据需要参考本书的
其余部分’特别是第5～12章’其中包括具体的分析目标°第13～15章涵盖了高级分
｜ 析和跟踪的内容,对于那些希望更详细地了解＿种或多种跟踪器的人来说’这是可选的
阅读内容.
第16章用讲故事的方式来描绘性能工程师的工作场合°如果你是性能分析的新手,
这＿章作为-个用各种不同工具做性能分析的例子,你可能想先阅读’然后在阅读完其
他章节后再回到这-章°

作为未来的参考
本书聚焦于系统性能分析的背景知识与方法’以期为读者带来长久的价值。
为了做到这—点,许多章都被分为了两部分°一部分的内容是术语、概念和方法（一
般附有标题）’这些内容许多年后应该还依然中肯适用°另一部分的内容是前-部分如
何实现的示例;架构、分析工具,还有可调参数°这部分内容即便有朝—日过时了,作
为示例进行学习也依然是有用的·

跟踪示例
｜｜｜

我们经常需要深入探索操作系统’这项工作要用到内核跟踪工具。
自本书第1版以来’eBPF己经被开发出来并合并到Lmux内核中,为使用BCC和
bpfmce为前端的新＿代跟踪工具提供动力。本书的重点是BCC和bpfhace’还有Lmux内
■伊『 ｜卜

■
XxvI∏前言 』

‖司
核内置的Fhace跟踪器。BPF、BCC和坤fhace在我之前的书中有更深入的介绍［Gregg19］°
本书还涵盖了Linux中的perf’perf是另一个执行跟踪的工具。然而’pe【f被收录
在各章节中’通常用于采样和PMC分析’而不是用于跟踪。
你可能需要或希望选用其他跟踪工具’这很好.本书中的跟踪工具用来展示你能向
系统抛出的问题’这些问题及提出这些问题的方法’往往才是最难知道的·

目标受众
本书的目标受众主要是系统管理员及企业与云计算环境中的运维工程师’所有需要了
解操作系统和应用程序性能的开发人员、数据库管理员和网站管理员也适合阅读本书.
作为在＿家拥有大型计算环境的公司（NetHix）工作的性能工程师’我经常与SRE（站
点可靠性工程师）和开发人员—起工作’他们面临着巨大的时间压力,需要解决多个同

｜ 】
时发生的性能问题°我还参加过NethixCORESRE的轮流值班’亲身经历了这种压力°
对于很多人来说,性能调整不是他们的主要工作,他们了解的知识足够解决当前的问题
就行.由于知道读者的时间有限’因此我把这本书写得尽可能短’并在书的结构上便于
读者跳到特定的章节。
本书的另一个受众群体是学生:本书适合作为系统性能课程的补充教材·在本书的
编写期间（以及开始动笔的多年以前）’我就曾经教授过这样的课程,并帮助学生解诀
仿真的性能问题（事前不会公布答案! ）°这段经历帮我弄清了什么样的材料能最好地
引导学生解决性能问题’这也促成了本书的部分内容°
无论你是不是学生,每章的习题都会带给你一个审视和应用知识的机会。其中有＿
些可选的高阶练习,可能你完成不了（但至少可以启发思维）°
本书涵盖了足够多的知识细节’无论是大公司还是小公司’乃至雇用了不少性能专
职人员的公司’本书都可以满足其需要。对于众多的小公司’日常用到的可能只是书中
的某些部分’但本书作为参考也可备不时之需°
■

排版约定
本书贯穿始终用到的排版约定如下°
｜｜
示例 说明

netifrece1v昆skb0 函数名

iostat（1） man手册第1章引用的命令

read（2） man手册参考的系统调用

malloc（3） man手册参考的C库函数

vmstat（8） ma『】手册参考的管理命令
Documentatjon／ Linux内核源代码树中的文档

｜
卜
‖‖‖‖‖‖‖‖‖‖‖‖‖‖‖‖‖匹■『

前言ⅫⅨ

续表

示例 说明O
kcmel／ Linux内核源代码

｝
囤… Linux内核源代码’文件系统

CONFIG≡ LinuX内核配置选项
rawaIt 命令行输入和输出
mp9tm1 加粗显示的键入命令或关键细节
＃ 超级用户（root）shell提示符
＄ 普通用户（nonˉroot）shell提示符
＾C 命令被中断（Ctrl＋C）

［…l 文本截断

补充材料与参考
参考资料列在每一章的末尾′这让你能够测览与每＿章主题相关的资料°下面列出
的书可以作为学习操作系统背景知识和性能分析学习的更为深入的参考。

0ain91］Jain′R.′ⅧeA沛o／ComP皿tc了胁te沉J化′γbmm∏ceA门α′沁is;几ch门jgu田允厂EXP旧厂j加e∏m′
D“呕门′Meα皿陀晒e刀呜Sj加u′αtjo″′α门αMO血′j′咽》Wiley》1991.
［Vahalia96］Vahalia’U’UMX加tEmα′5:刚eNew∩O门汀e酒,PIentjceHal1′1996。

［Cockcroft98］Cockcroft′A′andPettjt’R.′SM刃代加7砸α门ceα″‘jTlmj呕:加mαnd肋e加t叮门et’
PIentjceHall′1998。

［Mu＄umeciO2］MuSumeci′G。D.’andLoukjdes’M′S）′stBm代加厂y7m门ccⅧ门j呕2ndEdition′
O’Reilly》2OO2.

［BovetO5］Bovet’D.′andCesati’M〗U》‖αc叮m刃αj呕thCLj刀uxKeme』3IdEdition’O’Rei1ly’
2OO5.

［McDouga】lO6a］McDougal1’R.’MauIo’J.′andG【egg’B.′So!叮js化『允7mα＂ceα门‘j乃oIJ;
D7》切“α刀qMDBTPch门jques加了5o′α「jJ】0α门αOPenSoIα厂i5’PrenticeHall′2OO6

［GoveO7］Gove’D′So′α厂jJA〃′jcαtjo门Pmgmmmjn8》P【enticeHalI,2OO7.

［Love10］Love′R。′Lj∏uxKC了门elDc肥′QPme门t′3IdEdition’AddisonˉWesley）2O1O.

［Gregg11a］Gregg′B’andMauro′］》DT》t【仁efDy门α励ic7mcjngj″Omc′eSo′α汀S′MmOSXα″
Fγ伯eB5DpPrenticeHall’2O11·

［Gregg13a］Gregg′B.′印src加5几「／b厂晒α″〔e;E门rc′P门5旧α∏‘jtheC′o皿‘j’PIenticeHall,2O13

｜
（firsteditjon）.

［Gregg19］Gregg’B.’B尸F胜厂／b厂加口门Ce乃ok:L加ux咖tc加∩门αA〃′jcαtjo门Ob5f「vαMj″′
AddisonˉWesleyb2O19·

［I叮ob5Watch2O］IUob5Wat〔h′″So］arlsJobs′＂https:／／www.jt｜ob5watch.couk／）obs／uk／
solaris.do＃demandt【end′acces5ed2O2O.
 Q

致谢

感谢所有购买本书第1版的人,特别是那些把本书第1版作为他们公司的推荐书目
或必读书目的人,是你们对第1版的支持促成了第2版的诞生。谢谢你们‖
本书虽然不是关于系统性能的第一本书’但是是关于系统性能的较新的图书。我想
感谢在我之前出版相关图书的各位作者的工作’我在写作本书时借鉴和参考过.我尤 ｜

其要感谢AdrianCockcroR、JimMaum、RjchardMcDougal1、MjkeLoukides和RajJain’
是你们帮助了我,我也希望能帮助到你们。
我非常感谢每一个对这个版本提供反馈的人°
De∏dreS订aughan在本书写作过程中再次给予我方方面面的支持’包括她用自己多
年技术文案的编辑经验来改进每—页。你所读到的文字来自我们两个人’我们不仅享受
在一起的时光（我们现在已经结婚了）’而且还一起工作°谢谢你!
PhilippMarek是奥地利联邦计算中心的∏取证专家、IT架构师和性能工程师°他
对本书的每＿个主题都提供了早期的技术反馈（这太了不起了）,还指出了第1版文本
中的问题。1983年’PhiⅡpp在6502CPU上开始编程’此后＿直在寻找额外的CPU周期°
谢谢你, Phjlipp’感谢你的专业知识和不懈的努力.
DaleHamel（Shopify）也审查了每一章’为各种云技术提供了重要的见解’并提供
了一个贯穿整本书的观点。感谢你承担这项工作—在帮助编写BPF书之后又承担了这
项工作.
DanielBodm1ann（Isova1ent）为-些章节提供了探入的技术反馈’特别是与网络相 q

关的章节,这帮助我更好地理解其中的复杂性和技术·Daniel是一位Lmux内核维护者’
在内核网络栈和eBPF方面有多年的工作经验。谢谢你’Daniel’谢谢你的专业知识和
0

严谨的工作态度。 ‖

我特别感谢perf维护者AmaldoCarvalhodeMelo（RcdHat）对第13章提供的帮助;
以及F∏ace创建者StevenRostedt（VMware）对第l4章提供的帮助’这两个主题我在
‖

‖
｜｜｜｜
■｜

致谢XXXI

第1版中没有充分涉及。除了他们对本书的帮助,我还很感谢他们在这些高级性能工具
方面的出色工作,我在Netflix公司用这些工具解决了无数生产环境中的问题。
我很高兴能让DominicKay审阅了若干章节’他帮助提升了内容的可读性和技术准
确性。Dommic在我编写第1版时就给予了帮助（在这之前’他是我在SunMicrosystems
从事性能工作的同事）。谢谢你’Dominic。
我当前在NetHix的同事AmerAther对几个章节提供了很棒的建议。在理解复杂
技术方面’Amer是公认的专家工程师°Zacha】yJones（Ve∏zon）也为复杂的技术主题
‖
提供了建议’并分享了他在性能方面的专业知识,用以改进本书°谢谢你们,Amer和
卜｜

Zachary·
不少审核员承担了多个章节的审核工作’并参与了具体议题的讨论。他们是
AlCjandroPro蛆o（亚马逊）、BikashShanna（Facebook）｀CoIyLueninghoener（洛斯阿
拉莫斯国家实验室）、GTegDuIm（亚马逊）、JohnArra副id（O仗omemc）、JustinGaITison
（亚马逊）、MichaelHausenb］as（亚马逊）’以及Pa‖rickCab］e（ThreatStack）。感谢大家
对本书的技术帮助和热情.
｜‖‖卜

还要感谢AdityaSarwade（Facebook）、AndrewGallahn（Ne出ix）＼BasSmit、
GeorgeNevilleˉNeil（JUULLabs）、JensAxboe（Facebook）、JoelFemandes（谷歌）、
RandallStewart（NetHix）、StephaneEranian（谷歌）和T℃keHoilandˉJorgensen（Red
Hat）’感谢你们回复问题与及时的技术反馈°
我的前＿本书—《BPF之颠》—的作者们也间接地为我提供了帮助’因为本版
中的一些材料是基于那本书的。这些材料的改进要感谢AlastairRobe冗son（Yellowb∏ck
Data）、AlexeiStarovoitov（Facebook）、DanjelBorkmann、JasonKoch（Netf］ix）、MaIy
Marchini（Netf1ix）、MasamiHiramatsu（Linaro）、MathieuDesnoyers（EfficiOS）、
YOnghongSong（Facebook）等人°
本书第2版是在第1版的基础上编写的。第1版的致谢中感谢了许多人对这项工
作的支持和贡献;总之’在多个章节中’我得到了AdamLeventhal、CarlosCardenas、
DarrylGove、DominicKay、JerryJelinek、JimMauro、MaxBmning、RichardLowe
和RobcrtMustacchi的技术反馈,我还得到了AdIianCockcro仇、BryanCamTill、Dan
McDonald、DavidPacheco、KeithWesolowski、MarsellKukuⅡevicˉPearce和Paul
｜ Eggleton的反馈和支持°RochBourbonnaiS和RjchamMcDougall间接地提供了帮助’因
为我从他们之前的性能工程工作中学到了很多东西’JasonHo1Ⅱinan在幕后的帮助让第l
版的出版成为可能°

Ljnux内核是复杂且不断变化的’我感谢lwnnet的JonathanCorbet和JakeEdge的
出色工作,他们总结了诸多深刻的技术主题,他们的许多文章都成为本书的参考°
特别感谢P烷arson公司的策划编辑GTegDoench’他的帮助、鼓励和灵活使本书的出
 ■

…∏关于作者

版过程比以往更有效率。感谢内容制作人JulieNahⅡ（Pearson）和项目经理RachelPaul,
感谢他们对细节的关注和对提高图书质量的帮助。感谢文字编辑KimWimpse忱对本书 q

所做的工作’他找到了诸多方法来提升文字质量。
还有’感谢Mitchell’感谢你的耐心和理解。
从编写本书第1版时开始,我就一直担任性能工程师’负责调试栈中的所有问题’
从应用程序到裸机.现在’在性能调优管理程序、分析包括JVM在内的运行时、在生
产环境中使用包括Ftmce和BPF在内的跟踪器’以及应对Nethix微服务环境和Linux 」
内核的快速变化等方面,我有了许多新的经验°前面提及的这些技术不少都没有妥善保 ｜
存文档’考虑到这-版我要达成的目标’这曾让人心生气馁,但我就是喜欢挑战。

‖‖‖‖』■■■■■■■■■』·勺·■■■■■■■■■■■■■■『■■■■■■■■□｜｜■』γ
关于作者

BrendanGregg是计算性能和云计算方面的行业专家°他是NetHix的高级性能架构
师’从事性能设计、评估、分析和调整工作°他是多本技术图书的作者’包括《BPF之
獭》’他获得了USENⅨLISA系统管理杰出成就奖。他还担任过内核工程师、性能负责
人和专业技术培训师’并曾担任USENⅨLISA2018会议的项目联合主席。他开发了可
用于多个操作系统的性能工具,以及包括火焰图在内的性能分析的可视化工具与方法°

■ ＝＝ ＝ ＝＝＝＝＝＝＝＝＝≡＝＝＝＝＝＝■■＝＝＝＝＝＝＝＝＝＝＝≡≈■■ ■■■＝≡■＝＝←≈＝＝＝＝＝＝＝＝＝■■＝＝≡＝～＝＝＝乙＝＝＝

读者服务 ■□
坦瓣耀磷

微信扫码回复:43587

■ 获取本书扩展链接 门

■ 加入运维读者交流群’与更多同道中人互动
口 获取【百场业界大咖直播合集】（持续更新）,仅需1元 （

日
·｜
 目录

第↑章绪论.…………………………………….……….…………………….……….……………………。↑
1′1 系统性能.…….….………………….………….……………………..……….………………………I
1.2人员……………………………………………ˉ.……………………………….………….……………2
0
13活动………….….….……………..….…………………………….………..…….….…….……….…3
l.4视角………….…….…………………………………………….……………………………………….4
15性能工程是充满挑战的…………………………………………....………..……………..……5
L5.1 主观性….………………………………….……….…………………………………………5
1.52复杂性..………………..….…..…………….………….…………..….…….·………….…5
153多个原因.….……ˉ……….…….…………………………….…….……….…….……..…6
l.5.4 多个性能问题.….…………………..….….…….….…………………………………….6
l·6延时...….……….……..…………………….……….…………..……….………………….…………6
1.7可观测性………….……..……………….…….……………………………………….…….….……7
l.7.1 计数器＼统计数据和指标..…….……………….…………….……….…………｀…8
1′7.2剖析…………….…………….….……….….………..………..……….…………………. l0
1。73跟踪………。.……………..…………….…………….…………………………………….. 1l
1.8实验….…….……………………….………………………….……..…...……………………….…. l3
19云计算……….….…….………...……….………………………………..….………………………l4
1.10方法………….……….….….…….….……….……..…….….………..…..………….……….…. l4
l.l0.l Linux性能分析60秒…………….……………………….….….…………………. l5
l.ll 案例研究.…..……….…………………….………….…….……….…………….….…………….］5
l｀ll.l 缓慢的磁盘….………………………………………...…………………ˉ………..…,. l6
l.ll.2软件变吏.…..……….…………….….………….……….…….…...…………………. l8
 ‖

xXXIV目录

〗’’
◆●●
◆●■
◆●●
◆

◆●●
q｛

玲
舌
◆●
凸●■
矽●■

］
111.3史多阅读….……

●■
●●
■●
●■
●
●●
●■
●●
■●
■●
●
●●
●■
■●
●●
■■
●●
●
●●
●●
●■
●●
■■
●●
●●
●●
●●
●●
●●
●●
■●
●■
●●
●●
■●
●●
■●
■●
●■
■●
●■
●●
●■
■●
■●
■■
■■

■
●■
■●
■■
■■
■■
■
■
■
●
●
1.12参考资料………………….

第2章方法………………………………….………………………………………………………………2↑
21 术语…………………………………….……………….…..………………………….…………….22
2.2模型………….….……………………………………………………………….………………,.…...23
22.1 受测系统…………….…………….…………………………………….…………….…23 0

22.2排队系统………………………………….………………………….……….……………23
2.3概念ˉ.……….……………………………………………………..………………………………..….24
23.1延时.……·………….……………………………………………………………………°….24
23.2时间量级……………………………………………………………………………………25
23.3权衡………………………·.………………………………………….°……………….……26
23.4调优的影响…….……………………….………………………….……….………….…27
2.3.5合适的层级………….……。………………………….….….……………………………28
23.6何时停止分析…….………………………………………………………………………28
2.3.7性能椎荐的时间点…….………….…………….………………….…….……………29
23.8 负载与架构…………………..………………….…………….………………………….30
23.9扩展性…………………..………………………………………。…….……………………30
2。3.10指标………………………………·………………………………….………….…………32
2.3.11伎用率.…….………….………….……………………………..………….….…………33
2°3·12饱和度…………,………………………………………………………….………………34
23.13剖析………………….…………………………………….…………….…………………35
2.3.14缓存…。………°…………………。……….………….……………………。………………35
23.l5 已知的未知…………。…………...…………………………..…..…..………..………37
2.4视角.。……………….……….………….…….………………………..……..……………….………37
2.4.1 资源分析…….………………….………….………………………………………………38 0

2.42工作负载分析……………………….………………..………….…….……………..…39
25方法……....….…….…………………..….….………….……………………….…….…….………40
25·1街灯讹方法….….…….………·……………………………………………………….…q2
2.5.2随机变动讹方法……………….……………………·……………·………………….42
2.53责怪他人讹方法….……………………..…….…..…….……..…………..………….43
25.4AdHoc核对清单法…………..……ˉ……..…….…………….………………………43
■
｜

2.5.5 问题陈述法…………………………….……。……………。……….….…………………44
2.5.6科学法……..….….……….…………..……….…….……………………...………….…44
』

｜
 ｜
目录XXXv
｜
■【

2.5.7诊断循环……..………….……………….…………….…….….……….…….…..…….46
｜ 2.5.8工具法.………………………………………………………………………………….…46

｜
2.5.9USE方法.……………………………….………………………….………………………46
2。5·1ORED方法.…………………………………………………………….……………….…52
2.5.11 工作负载特征归纳…………………′.…………………………….…………………53
25.12向下钻取分析………………………………………………………………ˉ….………54
卜

25.13延时分析….….…………………………………………….…….………………………55
2.5.14R方法.……………………….………………….…………………………………………56
25.15事件跟踪………………….….………………………………….….…….。..………..…56
25.16基础线统计.………………………………………….……………………….…………58
25.17静态性能调优………………….……………………………………………………….59
25。18缓存调优……。……………….……………………·.……………………………………59
25.19徽基准测试….………….………………….…………………………….…………..…60
2。5.20性能麓言.………°……………….…·……ˉ……ˉ….……………………………………60
26建模。………..………………………………….………………………………………………………61
2。6.1 企业与云.….………………………………….……………………………………………6l
2.6.2可视化识别…….……………。……….…………………………·………°….…….……°62
Amdahl扩展定律….…….…….………….………………..……………….…………63
卜

2.6.3
2.64通用扩展定律.….…………………………………….………………………………….64
2.6·5排队理论……………………………….ˉ…………………………………………………。65
27容量规划……………………….……………………….……………….………………..……….…68
27.1 资源极限….……·……..……………………….…………….……………………………69
0

2.7.2 因素分析………….…….….……..……….…….………………………………..………70
2.73扩展方案….…………………….…….……….……………………….…….……………71
28统计.…….……….…….…….………….….….….………..…………………………………………72
28.1 量化性能收益。………ˉ…………………………………….………………….…………72 ‖

2.8.2乎均值………………………。………………………………………………………………73
2.83标准方差｀百分位数＼中位数..….……….….…….…………....…….……….74
p

2.8.4变并系数..………｀……………….…….....…………….…….….…..…………..……..75
2.8,5 多重模态分布…….……………………..……….………………….………·………….75
286异常值.…….….…….….…….……..……………….…….…….….…….…….….….…76
2ˉ9监测……….……………。.ˉ…….…………………｀……………………ˉ……………….ˉ………ˉ…·76
29.1 基于时｜司的规律.……..….….…………….….…….….………….…….………….…76
29.2监测产品……….….……………….….……………………………....…………….…...78
…VI巨录

29.3 自启动以来的信息统计………..……………………………………….……………78
2.10可视化…………………………….……………….……………………………….…..……………78
2.1O1 线图……………………….…………°…………..………………………………°….……79
2.1O2散点图….…………………….……………………………………………………………80

＝■■■■尸■
2。1O3热图……………………………………。…………………………………………….….…81

』』■可｜
2.1O°4时间线图……………….…………。………………….………………….………………82
2.1O5表面图.……………………………………….………………………………..…………ˉ83
2·1O。6可视化工具……………………………….………….………………………………83
2』1练习。….……….……………….·………………….…………·…………………………°ˉ…………84
2.12参考资料…………………………….…….………….………………………….….….….………84

“盯朋朋”刨”％％”＂凹仍陌““们们们佣
●
●■■●●■■■●●■●●■●●■■●■■
■■
■
■
●■●■●■●■●●●■■■●■■●●■■
■■■●●●●●■●●●●■●■■●●■■
■
●
●
■
●
子…

■与●●导●●●■■·■】丛■■■丛●□□县＜■■丛■■□丛＜■□丛■‖旦■■】孕·■】丛＜■■丛
第3章操作系统…….….……………….………..……………ˉ…………………｛

■■●■■■●●■●●●●●■●●●■●
◆
●■
●
■

■■■■■●■■■●●●●■●●●●■■
●
■
●
◆■
●
■

◆
●
●
■
导
●…
●■
●
■●

●
■
●●
●
仔
■
●
■
●
■
■
●
3.1 术语…………….………………………….………………..………………。

■
●●
●●●●■●●■●●●●■■■●●●●
●
■
■●

■
●
■●
■
●
●
…
●
◆
●
■
●…
32背景….……………………………….………………….………….…….…

■
◆●
●
■
弓巳

■
●●
■
●
●
●
●
■
●
■…
内核…。….…………………………………………°.……。………

●
■仔
32°1

凸
●
■●

■吁
●●
◇
审
■
●
■
■
●
●
◇“…
内核态与用户态.…………….………….………….………..

●
◆■
●■
3.2.2

●
■由
●
●
●
■
导
■
●

■
●■
系统调用…………………………………….…………………..
3.2.3
●
啡■

●■
●●
●
■
●
●
●
■
■
●………
●●
●●
3.2.q 中断..……………………………………………………ˉ………,。

●■
●
■
■
●
■
■

●■
●
■■

3.2.5 时钟和空闲………………………….…………………………』

■
●
■●
●
●
＄
●
●
●
●

■
●■
■
已●

3.2·6进程….…。…………….………………….…………..………·…』
■■
■
■
●
●
■
●
■
导
●
■啡
■
●●
●
◆●

3·2.7栈.……………………….…………………………………………』
●●
●
●
■
●
■
●
◆
■
●卯
●■
■
●
●■

32.8虚拟内存.…………………….………………………………。
｜
■●■
■
●
■
●
■
●
◆
■■
●
■■
●
◆●

3.2.9调度器…………………….…….…………..………….………° 0
■
●
■
■
甲
●
◆
●
■
◆№…
■■
●
■●

3。2。10文件系统。……。…………………………….…………………
‖卜‖‖■◆■

●
◆●
◆
■
●
●
●
●
●

●●
●
■
■●

32,11缓存………………………..…….……………………………..
导
■●
●
●
■
●
■
■
◆
●
■…
■■
●●

3.2.12 网络……………………。………………………………….……
甲■
■
■
●
■
●
●
甲
◆
●
■的
●
■●
■
●●

32.13设备驱动……………….……………..………………………
●守
●
■
●●
●
■
■
●
■加

●●
●
■■

32.14 多处理器………………….……………………….………….
●
■●
●
■
●
■
●
◆
■
●■
●
■■
◆
◆
●■

32.l5抢占……….……………….………………·……………·.……
●
■
伯
■
巴
●
■
■
●
卯
■
●
■
●
●

3216资源管理………..………………….…………………………
3.2·17可观测性……………………………………。…………。.……
…………………………108
33内核…….…………………………….….….………….………….
3.3.1 UNIX………………………』………….………°…………….……………………………109

33.2 BSD………….……………….………………….….…………………·…………··………110
 巨录…∏

｜
3。3·3 So1aris.…………….…………………………………….……………………….…………l11
3。4 Linux………………·……..……………………….….……………………………………。…………1l1
3。4.l Linux内核斤发……………………。………………………………………….………。1l2
3.42 systemd….。·…….……………。……………………………………。………………….…117
3.43KP∏（meltdown）.……………………………………………………………………118

3。5
3·4·4 eBPF……………,...…………………….…。…………。…。………………………………118
其他主题…….…ˉ………………………….……………………..…….…………….…….…….. l19
｜
3.5.1 PGO内核………°…………………………….…………。……….…………….……….1l9
3.52unikemel………….………….……………………………….………………..…………120
3.5.3微内核和混合内核………………………….…………….·…………………………l20
35.4分布式操作系统.…………………….…….………………………………………….121
3·6 内核比较……….………………………….……….………………….……………………………121
3.7 练习.………………….……………………………………….……..….….………..………………122
3.8 参考资料….……………….………………………………………………..….….……….…ˉ.ˉ…l22
3.8.l 延伸阅读……………….……………………….……….…………………………….…I25

第4章观测工具……….……………ˉ…………………………………………………………….……‖26
4.1 工具范围….…………………………………………………………………………………………127
411静态性能工具.………………….……………….…………………….………….……127
41.2危机处理工具.…….……..……………………………………..……….………….…l28
42工具类型..……………………………..………………………...……………..……………….…129
42.1 固定计数器…………….…………………….……….….………………….…………130
｜▲■｜｜｜｜｜

422剖析……………..….………….……….……….….……………………………………..131
4。2。3跟踪.………………….………………………。……………………….。………….………132
424监测………………………………………·……………………………………………·….133
43监测来源…………………..……………ˉ……………………….………………….………..……135
43.1 ／Proc………….………………………………….……………….…..…………..…….….137
0

43。2 ／sys…………….….………………………………………….………·……….……………l40
卜 433延时核算.…………..………………....…………………….……………………..……］42
4.3.4 netlink…………….………·………………………………………..……….……….……143
43·5 tracepomt…。…。….………………。……………………………….………………。.……l43
0
43·6 kprobes……。.…………….………………………………。………………°………………l48
4.3.7 uprobes………………………………………………………….……….…………………15l
438USDT………………………………。…………·…………………………………….…….152

p
 ｜
…目录

43.9攻件计数器……………….…………………………………………….………………l53
43.10其他观测源………….…………………….………………………………….……….156
44 sar……··……………………….…….…………………·…………….….….……….………………. 158
4.4.1 sar（1）的覆盖范围.………………………………….……….…………….….………158
4.4.2 sar（1）监测.………………………………………….…………………….…….…….…l59
443 saI（1）实时报告….…………………………………………………………….……….162
44.4 saI（1）丈档………………….……………………………………………………….……163
45 跟踪工具……………………………………………….……….………….…………………….…163
4·6 观测工具的观测….…………………………………………….…………….……….…………l64
4·7 练习….………………………….………………………….………………….………….……….…165
4.8 参考资料…………………………………….….……………………..……………………………166
第5章 应用程序.….……………………………………………………………..……·…..………….…‖68
5.1 应用程序基础….…….………….,…….…….…………...…………………….….……………168
5。1.1 目标….·…………………………………….…………。………………°…………………°170
5.1.2常见情况的优化……………………..………….……………….……………………171
5.1.3…可观测性………….……………°…….…,………………………………………………171
5·1°4大O标记法……………………,…………。……………·。…………………………°…172
52应用程序性能技术…..………….…..…………………………….….….…….………………173
52.1 选择I／O尺寸………..……………………………………………,……….…………..l73
5.2。2缓存………………….…….………………………………………….……………………173
5.2.3缓冲区……………………….….….……………………………………….…………….174
5.2。4轮询……….…………….………..………………………….……….………………..…。174
5.2.5并发和并行………….…………………………………………….………..………….174 ‖

52.6非阻塞I／O..………………………』…….…..………………….….……………………178
5.2.7处理器绑定………………….……………..….………………….……….……………179
5。2.8性能咸言.………………………………………………。…·………………………….…179
53编程语言….………………….………….…………………...…….………..,…….……….…,…179
5.3.1 编译型语言.………………….….……………..….….………………….……….……180
532解释型语言…..……….…………………………….….………………….………..….182
53.3虚拟机..……………….…·……….…….……….….………………...…...……………182
53.4垃圾回收.…………°.….………………………°.…,……………°……………..………182
54方法……………………………………………………………′…….……………….………………183
5.4.1 CPU剖析….………….….….……………….….…….…...……………..′….……..…184

］
 巨录XXXIX

5.4.2 oiPCPU分析……………………….ˉ……………….…………….……………………186
5.43 系统调用分析.……·………………..………………………………………………….189
5.4.4USE方法……….…………………………·………………….…….……………………190

｜ 54.5线程状态分析…………………..……………….…………….………….….….….…19l
5.4.6锁分析……………….………………………….…………….…….………….…………195
5.47静态性能调优…….……………….……·………….….………………………..…….196
5.48分布式跟踪.…….…,…………………………….………………………………,.……l96
55观测工具……………….…………………….……….…………….………………..…….………197
5.5·1 perf..………………………………………….…….………………·………………………198
5。5.2 proii1e………………………….…………….…·……………….…………………………201
. 5。5,3 o贤Cputjme°………………….…………………………………….………………………202
）

5·5·4 strace……0………….….…·…….………………….………………………………·…·…203
5.5.5 execsnoop·…………………………………………………………….………………….205
55.6 syscount………………………………………………………………………….…….….206
｝

5.5.7 bp1汀ace…………………………………………………………………………….………207
56明白了…..…….……………………………………….……….…….……………….………….…212
56.1 缺少符号….………………………..…………….……ˉ…………….……….…………212
56.2缺少栈…』……………。………………….……….……………………………………….213
5.7练习.…………..….……….………………………………………….………….…′…….…………214
5.8参考资料……………….….………………………………..………..………………………….…216
0

第6章C尸U.………………………………………………………………….………….…..…。…………2｜8
b

6.1 术语.…….…….…………….………….…….….…….………..………….………….……………2l9
62模型…….….………………….…………….…………………….…………..……….…………….220
6.21 CPU架构………………·…………….…』………….………°…….……….』……….….220
‖

622CPU内存缓存……..……………….……….……..,..,…….….……………………220
623 CPU运行队列………………………………。…………………………………………。221
63概念…….…………………………….….…………………………·….………….……..….………222
6.3.1 时钟频率.………………….…………………….…….…………….…….……….……222
6°3.2指令…。……….……….…………………………….……………………………….…。…222
63°3指令流水线……….………,…‘…………………….…,…………….….….………….223
■
63.4指今宽度.………….………………..………….………….…….….……..……………223
6.3.5指令尺寸……………….……….………………………………..………………………223
63.6 SMT……….………..……………………………………·…………………………….·…224

‖
 ｜｜｜‖■『『】‖‖］‖‖‖‖‖‖勺‖‖‖‖‖‖‖‖‖‖｜』‖｜｜」■可｜｜｜｜‖‖』〗〗〗〗‖勺‖‖‖‖‖｜｜
XL 曰录

63.7ⅢC和CPI。…………………。…….……………………………·…………。……………224
63.8使用率…………………..…………………………………………………………………225
63.9用户时间／内核时间.…………ˉ………….……….……………….………….……225
63.10饱和度………………….…….……….…………………….….……………………….225
63.11抢占……………·……………………..………………………………………………。…226
63.12优先级反转………·……………·…·………….°………………………。……。 ……226
63.13 多进程和多线程.……………….………………………………………ˉ………….。226
63.14字长.…………………………………………。………………….…………….…...……228
.。……228
63.15编译器优化……………….…….………………………….……….….………
..…228
6.4架构………..……………………………..……………..….…
……228
6.4.1硬件..……………………….…….………….……………….…….………………
………238
6.4.2软件.…。……°…………….…………………………….………………………….
………242
6.5方法……….………….·……….…….………………..……………………….……………
6.5。1 工具法………….………。………………….…….…………………·…………….…….·242
6.5·2USE方法….…………….….………………………………,……………………………243
.……243
65.3 负载特征归纳…….………………….….….…………………….·.…….……
………245
65.4剖析…………………………………………………………………………………
6.5.5周期分析………………………………….………….……………………ˉ….…………248
6.5.6性能监测.………….…………。……………………………………………….…………248
…。249
65.7静态性能调优.………………….………………………………………………
6·5.8优先级调优………………..…………………。………………………………… .·…..249
………250
65.9资源控制.….……………………..….………….……………………….……
65.10 CPU绑定…….………………………….…………………….……………….. ….…250
…….·.250
65.11微基准测试.…….………….…….….……….….……………….…..……… 」

.。…。…251
66观测工具.……….…….….……….…………..………………….…….….…………….
………252
6.6.1 uptime……………°…………….……….…·…….…·….………………………
,……255
6.6.2 vmstat………………….………………………………………………·,……….
255
66。3mpstai….……………………………….……。……………………………….….,
….……256
6.6.4 Sar…..………….………….…………………..…………·…………,……….
……257
6.6·5 ps,……·….……。………………….………….….…………………………
…·……258
6。6.6 top.……………………。………………….…………………·………………°……
……259
6.6.7 pidstat…….……………………。……。………………..…。…………°……·……°
………260
66.8 time和ptime………°…………………………………………………….……
……….261
6.6.9 turbostat……,……………°….…………………………………………….……。
■■』■】□□
‖

巨录 XLI

66.10 showboost….。…………………………………………………………………。…°……262
｜｜

6.6·l1 pmcarch……………………………………………………………….………………。…263
6.6.12 t］bstat……….…………….………………….…….。………………………….…………263

6。6.13 perf.…………………。……………………………….…………………….………·……·264
6.6·14 profile…。………·…….,……….….……………………………………………………..274
6.6.15 cpudist……·…….……………………。.……………·…………………….…………….276
｝
6.6.16 runqlat….·………………………….….………………………………….……,……….277
6.6.17 rlmqlen……….………….……………….……………………………·…….……。……278
6·6.18 sofijrqs.°……·…………………………°…………….……°.…………………。……。…279
‖》

6。6.19 hardirqs…….………….………….。…………,…….………………………,…°………280

｜
66·20 bpRrace………·……ˉ…….……….……….……….….……………………………….280
66.2l 其他工具…………………….….……….………….…….…………………….……..283
67可视化.….…….……….….………….……………………………………………….…………….285
6.7.1伎用率热图…….…….…………………………………….……………………………286
672亚秒级偏移量热图….…………….……….…………….…….………………….…286
6。73火焰图……………….…………………………….……………………。…·…………·…287

6.7.4 FlameScope………………….……………·………………..……….…………·……….290
6.8实验….…….…………………………………………………………….……………………………291
6。8.1 AdHoc,………..…………………………………….…………………….….…·…….…291

6.8.2 SysBench.……………………………….…。…………………….………….……·…….291
69调优.….……………..……………….….……………………………………..………….…………292
69.1 编译器选项.……………………….…………………………….………………………293
69。2调度优先级和调度类……………………………….……………………·…………293
6.9.3调度器选项……·…·…….……..′………………………………………………………293
6.9.4调节调速器.….…·.….………….…….…….….…………….……...…….……….…294
6。95能耗状态……….…………………………………………………………………………295
‖

6。9。6 CPU绑定…………………….…….……………………………………………….。……295
6.9.7独占CPU组……………·…………………………………ˉ…………。……。….………295
69.8资源控制………………….……………………………………………….…….….……296
69。9安全启动选项………….….….…………………………………………………。….…296
6.9.l0处理器选项（BIOS调优）.…..….…….….………………….……..….………297
卜 6.10练习………….…....……….…….………….……….…·………….……….…….……..………297
6.ll 参考资料……..…………..…..…..…….….………..……..……….……….………..………...298

｝
】‖■〗■□■日□】】
 ｜

｜
XL∏ 目录

■
｜
第7章 内存…………………………………………………………………………………….………….3O↑

7ˉ1术语……………………………………….…….…….…………………………………………….3O2
72概念…………………………………………………………….….……………………….…………303
7.2.1虚拟内存………………………………….…….……….…………….……………….303
7.2ˉ2换页。….……………….……………………………………………………。………….….303
7。2.3按需换页°…。………………………………………………………….…….……………305
7.2.4过度提交………….………………….……………..……………………………………306
72.5进程交换°。…………………………….…………………………°……°…………°……°306
7.26文件系统缓存用量……….………….……。……………。………….……….………306
7.2.7使用率和饱和度…………………………….…………………………………………307
7°2.8分配器°.…………………………·…………·……………………………….……………307
72。9共享内存…………………….…………………………………………….………ˉ.……307
72.10工作集大小.….………….…….………………………………………………………308
7.2.11 字长………….….………………………...………………….………………………….308
7.3架构………….……….…………………………………….…………….…………………………308
7.3.1硬件………………………….……………………………………°……………………….309
73.2软件……·………………….….….……………………….……………………………….313
73.3进程虚拟地址空间…………………….…….…………………………,……………317

■■■■■■■■■‖‖‖‖‖』
7.4方法…….……..…………….………………………………………………….………….…………320
7.4.1 工具法…………。……………………………………………………….…………………321
7.4.2USE方法……………………….,…….……………………………………·……………322
7.4。3描述使用情况………………………….……………………………….…….…….….323
7.4.牛周期分析………………………….…….………………………………….…….………324
7噎4.5性能监测……………………………….………………………·…………。……….……324

｜
7.4.6泄漏检测………………°……。…………………………………。…………°…………°324
7.4.7静态性能调优…….….……….….………….……………………….………..………325
7,4.8资源控制….….,……..……………………..………………………….………….….…325
7.4.9微基准测试…………….…….….….…………..………………………………………326
7.4。10 内存收缩…………………….…………………°…………。…………………………。.326
75观测工具………,.………….……..…….……,…….……………..….…,..….…………….……326
7.5·l vmstat………·….……,………………………………………·…·………….…·…………327
7.5。2 PSI…………………………………………………….……………·…….…………………328
7,5.3 swapon·…………,,°….…·……………..…,……………………………………….…….329
 自录 Ⅻ■＂｜

7·5。4 sar….…………………….……………………………………………………….…….……329

7。5·5 slabtop………………….…….…….…………………………………………,…………..331
75.6 ∏um鳃tat..…………………·……………………….。…………………………….………332

｜ 7.5·7 ps..………….…….…….……·………………………………………………………..……332
7·58 top…….…………….…….…………………………………….……….………………。…333
7.59pmap…….…………………。·………………………………….………………………….334
7,5.10 perf………….………….………………….………………………·……………….…·…336
7.5·11 drsnoop……………………·……………………………°…………………·……………339
7.5.12wss…….……………………………………….…….….…………………。…….………340

7.5.13 bp世ace…,·…………….·………………………ˉ….………….………。………………34l
75.14其他工具…………….…….………………….….…………………………………….345
7.6调优……….…………………………………….………………………………………………….…348
76.1 可调参数……….……………………………………………………..………………….348
7.6.2 多种页面大小………………….………………………………….…。………………349
7.6.3分配器°°……………….…………。…………………….…。…………………………·…·350
76.4NUMA绑定….………………………………………….….……………….….………350
7。65资源控制…………………..……………….…………...……….………….…….…….351
7.7 练习….,…….…..……………………….….…….………………………….………….………….351
78参考资料…………·……。……….………………………………,………………………….……353

第8章文件系统.ˉ…………………ˉ…….……………………………….………………………………355

8.1 术语……………..………………………………………….…….….….………….……….….……356
｜ 8.2模型.……………….………·.……….………….………………….….………………….…………357
8°2°1 丈件系统接口……….….…………。…….……·………·…………………°……。……357
822丈件系统缓存……..……….…….………..….….………………….…….……·……357
8.2.3二级缓存..….…………….……...………………….…………………………….…….358
83概念…,…...……………….·……..………….…….………….………….….…….………….……358
83.1 丈件系统延时……………….………,.…….………….……….….………….………358
｜ 8.3.2缓存……….………………….…………..…………….….,……………….…….………359
833随机与顺序I／O………………….…..……………………….…………………….….359
83°4预取·.….………………………………………………°………………………………….·360
8.3.5预读.……….……….….....……….………..….……………….,…………..…………..36l
83°6回写缓存….…··……·…·……………·……….·…………….….…………………·…·.361
83.7 同步写…….,·…..…………....…..…….….……………….…………….…………..…362

｜b
 q

XLIV目录

8.3.8裸I／O与直接I／O……°·…………………………。·………………。.…………………362
83.9非阻塞I／O….…………………………………………………………………………….363

■■
83.10 内存映射文件…………….……………………………………….………………….363
83.11元数据………………….……………………..…………………………………………363
83.12逻辑I／O与物理I／O….……………………………………………………………..3“
83.13操作并不乎等………………………….…………………………………ˉ….………366
8.3.14特殊的丈件系统.………………………………………….…………………………367
83.15访问时间戳…………….……………………………………..……………………….367
8.3.16容量.…….……………………………………….…………………………………….…367
84架构…….….…………………….……………….………………….…………………………….…367
84.1 丈件系统I／O栈…………………………………………….……….…………………368
8.4°2VFS…。…………………………….…。………….…………·……·……………………·…368
8.4.3文件系统缓存……….…………………………………………………….……………369
8.4.4文件系统特性……………………………………………………………………….….371
8。4.5文件系统种类………………。…………………………………….……………………373
84.6卷和池、…………….………………………………………………………………………378
85方法.………………………….……………..……………………….…………………………….…379
8.5.1磁盘分析…………………………………….…………………………….……………380
8.5.2延时分析.………….…………………….…………………….….….….………………380
85.3 负载特征归纳….…….…………·.………………,.….…………………………….…382
8.5°4性能监测.………………………………….…………….……………………………。…384
8。5.5静态性能调优………………….……………………………………………………….384
85.6缓存调优.………………………………….…………………。………………………….385
85.7 负载分离……………….……………….….……….…………….……………………..385
8.5ˉ8微基准测试...……………….……………….…….…….…….…….…………………385
86观测工具……………………….…………….…………….…….…………………………………387
8.6·1 mount…怠….…………………………·………….…………………………………………387
8。6.2丘ee……………….…………………°………………………………………………………388
8·6.3 top……°.………………………………………………………………………….…………388 ｛
｜
8.64 vmstat………°…………………………………………..………………….………………389
8·6.5 sar.………….……,….………….…….………………………………………………….…389
8.6·6 slabtop…·…,….……………………………………………….………………·……·……390
8.6·7 strace……………………………………….……….·…………………………………·….390
8.6.8 fatrace……,……………·°…………………….……………………·………………….…391
｜

｜
Another random document with
no related content on Scribd:
The aggregate force that came from Macala would have been
anything between 1,200 to 1,500 strong. Of these, probably not more
than 1,000 were in action, if so many.
Had the whole body entered the stronghold unperceived, the plan
was to rest a day, then begin attacking the surrounding Chiefs and
looting their stock. These Chiefs, although members of their tribes
had become rebels, had themselves refrained from joining. It was,
therefore, supposed that, on conclusion of hostilities, they would visit
their displeasure on those who had rebelled. Consequently, the
intention was to deal vigorously with and kill off these half-hearted
men; such were the tactics of Tshaka, for, after killing the leaders, the
ordinary people, it was found, flocked to the 'conqueror.' The policy,
furthermore, was to lie in wait for small parties of the Government
forces near, but especially in, the forests, and massacre them before
reinforcements could arrive, as, indeed, had almost happened at
Manzipambana.
The reader will naturally wonder what became of the newly-appointed
commanders, Macala and Mganu. Macala thought it wiser to push
into the gorge. He joined Mangati and Ndabaningi, leaving the
headstrong Mehlokazulu and Bambata to look after themselves.
Mganu, however, remained.
On Bambata and Mehlokazulu's suspicions being aroused, scouts
were sent out to examine the ridge on the west, occupied by part of
Barker's force. These returned a few minutes before the Maxim
opened to report troops were really there. The impi was speedily
roused and formed into an umkumbi, that is, the 'circle' Barker had
seen. Bambata then completely lost his head, so much so that
Mganu, in the absence of Macala, was called on to take general
command. He immediately gave such orders as appeared necessary
to meet the situation. His own regiment, Mavalana, was told to charge
at Barker in one direction, and the others to do likewise in another.
But before the men (already arranged in companies) could move
forward (not backward as McKay and Forbes had believed), the
Maxim began—not from the ridge suspected by the rebels, but from a
different one. What followed at this stage has already been related. It
remains to add that, when it became a case of sauve qui peut, the
majority turned and made over the neck in rear as hard as they could.
It flashed across the minds of those familiar with that part of the
country that Dobo was a snare; that being the case, there was
nothing for it but to run the gauntlet towards the waterfall. This, as
has been seen, many accordingly did.
And so, as far as the ordinary rebel could see, the great storm that
was to come turned out to be nothing more than a thunder of artillery
and hail of bullets, brought on by that very race against whom the
mysterious command had been specially directed. Truly, the manner
in which Dinuzulu had directed the elements left much to be desired.

FOOTNOTES:
[230] This word is dissyllabic, and pronounced 'maw-me' (the 'e'
being as in 'met').
[231] Owing to the insanitary state of his camp (the site having
recently been used by three columns), Barker got permission from
McKenzie to move about three miles to the south-east of the
grave, and out of sight of Macala. This had occurred on the
afternoon of the 8th. On the same day, all the supply waggons
(empty) trekked back to Fort Yolland. Little did Barker suppose that
this lucky move would make the enemy believe the column had
vanished as well.
[232] Some fifteen miles of difficult country had been traversed.
The feat was a noteworthy one; it had called for courage and
daring, and well deserved the Distinguished Conduct Medal
afterwards awarded to each of the men.
[233] The mouth of Tate gorge is about a mile west of the mouth of
Mome gorge, and is on the route along which, as hinted in
McKenzie's despatch, the enemy would probably travel. In
Barker's view, it was just possible the enemy, although bound for
Mome, would proceed thither through Tate. It will be seen later,
McKenzie, notwithstanding his written instructions, entertained
similar suspicions.
[234] This is the hill referred to on p. 246.
[235] Orders were given for the searchlight to be kept flashing
throughout the night, to give the enemy the impression that the
troops were quietly resting on Nomangci—a ruse that exactly
served its purpose.
[236] F company (Capt. Forsbrook) was, however, at Mangeni. It
joined the regiment on the 14th June.
[237] It so happened that a squadron of R.H., which, for a few
minutes had gone astray during the gallop on top, had already
been made by Major A.W. Fraser to occupy a portion of the
position in question, and so prevent escapes then already
beginning to occur.
[238] The battalion moved in line of companies, searching the
bush-covered gullies en route; a number of rebels was come
upon. A and H companies lined the river, whilst parties of D, E and
G lined one of the sides of Dobo bush as it was being driven. The
work done by the regiment, especially B and C companies, was
very useful.
[239] It is believed by some that this man escaped.
[240] Cd. 3888, p. 185.
 XV.
STATE OF AFFAIRS AT UMSINGA.—OPERATIONS BY
MURRAY-SMITH'S COLUMN.—FURTHER OPERATIONS BY
UMVOTI FIELD FORCE AND MACKAY'S COLUMN.

Although Nkandhla had been selected by the rebel leaders as their

terrain, that was not to say outbreaks would not occur in other parts
of the Colony as well. Indeed, one of the objects of making
Sigananda's stronghold the principal rallying-point was to encourage
overt acts of rebellion in other parts from knowledge of the fact that,
so long as the fastness was held, it was available as a general
headquarters and place of refuge. As an illustration of this, the
disaffected men of Ngobizembe's tribe who, after being punished in
the middle of March by Leuchars—being by themselves powerless to
resist the troops—made off from Mapumulo and joined Bambata at
Nkandhla. Mehlokazulu and other Chiefs, as has already been seen,
broke from Nqutu and adjoining districts to do the same. The principle
upon which all these men acted seems to have been that, where local
confederates were not strong enough to offer resistance on the spot,
either from want of numbers or suitable fastnesses, they would move
to Nkandhla, but where the prospects were not unfavourable, as at
Umsinga and Mapumulo, each with a teeming Native, and sparse
European, population, they would resolve to try their chances there
and then, in the hope that, by creating additional and widely-
separated areas of disturbance, the difficulties of the troops would be
increased, when other tribes would follow one or other of the
alternatives referred to, always with the ulterior motive of causing the
country to rise en bloc.
In view of the importance of Umsinga and Mapumulo, both districts
being within forty to forty-five miles of Nkandhla by such routes as
Native pedestrians ordinarily travel, it is well to understand the
position at those places.
In Umsinga district there were, in 1906, ten tribes, by far the largest
being that of Kula. The total huts owned by members of his tribe was
upwards of 4,500.[241] The huts (in the same district) of the other nine
tribes together did not amount to 4,000. To so great an extent did
Kula over-shadow the other Chiefs, that it is unnecessary to make
more than passing reference to the latter.
Kula was a young man, grandson of Ngoza, once famous throughout
Natal and Zululand as Sir Theophilus Shepstone's principal induna.
Ngoza, after serving in a position of trust and responsibility for many
years, was appointed Chief over a tract of country vacated in 1858 by
a Chief Matshana[242] to escape arrest. The ward was 450 square
miles in extent, almost the whole falling within one magisterial
division.
The Poll Tax Act and regulations were promulgated to the Natives of
Umsinga in September, 1905, visits being paid by the Magistrate to
four centres for the purpose. The announcement was well received.
The only matter commented on by the Natives was that young men
and boys (over 18) were held liable for the tax, instead of their
fathers, as in the case of the hut tax. It was thought this would
promote greater independence than was already being exhibited
towards parents, and, at the same time, lead to youths retaining their
earnings, instead of handing them over to their fathers or guardians,
as up till then customary, on the plea of having to meet obligations of
their own towards the State.
Kula and the smallest local Chief, Nondubela, soon began to
influence their respective people against paying the tax. Their
intrigues were extended to other Chiefs near them, both in Natal and
Zululand. Early in December (1905), the same two tribes began to
prepare for rebellion. Supplies of assegais, shields and tshokobezi
badges were obtained. The young men of all the Umsinga Chiefs
were called on to pay the poll tax subsequently to 20th January,
though payment, it was explained, might be made at any time
between then and 31st May. Only one of the Chiefs made earnest
and successful endeavours to induce his men to pay, viz. Tulwana, a
man who had always been conspicuous for his loyalty to the
Government. Nondubela instructed his men not to pay, nor did any do
so until after the Rebellion had broken out and several reverses had
been sustained by the rebels. Kula advised his tribe to pay a few
pounds, and so throw dust in the eyes of the Government. £98 out of
£1,500 is all that was paid by his people.
In February, two of the same man's tribe, constables at Tugela Ferry,
were charged and convicted of conspiring to murder the European
police at that place and seize their arms and ammunition. The
conspiracy was exposed by one of Chief Sibindi's men—a fellow
constable.
Kula, a man of intemperate habits, had for long been a source of
annoyance. In 1898, he openly organized his tribe into regiments. He
was reproved for so doing by the Supreme Chief. Between 1898 and
May, 1906, he was repeatedly fined, either for refusing or neglecting
to supply labourers for the Public Works Department. In January, he
was warned by the U.S.N.A. to be more careful. "The Government,"
this officer said to him, "is lenient, but will not put up with annoyance
such as this for ever." In July, 1905, the Governor (Sir Henry
McCallum) paid a visit to Pomeroy. The Chiefs were summoned to
greet him. Kula arrived with a mounted cavalcade and deliberately
galloped past the King's representative without saluting. For this
gross disrespect he was cautioned by His Excellency in person.
On the 4th March, 1906, about fifty men of the tribe residing at Elands
Kraal, under headman Mabulawo, openly took up arms. This caused
all European farmers in the neighbourhood to flee precipitately from
their homes. The impi continued under arms and defied the local
police, with the result that thirty-six of the Police Field Force, under
Sub-Inspector C.R. Ottley, were sent to Umsinga to restore order.
Ottley, however, deemed it inadvisable to attempt more with so small
a force than camp near the court-house. On the 23rd, Kula held a
large beer-drink at his kraal within two miles of the magistracy. An
armed force was there assembled, it was said, for the purpose of
killing the police and court officials. On the night fixed for the
massacre, Kula, it seems, got drunk, when some of the more loyal
headmen of the tribe bound him up, thereby preventing him from
carrying his supposed threat into execution.
After these proceedings had been reported, Kula was summoned to
Pietermaritzburg by the Supreme Chief. He at first hesitated about
complying, but, on being pressed by his headmen, obeyed. Upon
being questioned at headquarters by the Minister for Native Affairs,
he denied everything that had occurred, but, in the face of irrefutable
evidence, was ordered to arrest Mabulawo and all who had been or
were still under arms. Twelve days later, the headman was brought to
the Magistrate, unaccompanied, however, by any of those who had
armed. On the 23rd April, a large impi was organized to release
Mabulawo. The latter was now driven off in Dr. Keith Murray's trap to
Pietermaritzburg. When the impi heard of this, and of the fact that
thirty of the Umsinga Reserves had been mobilized and posted at the
magistracy, they withdrew to their kraals.
Ever since the 4th March, the affairs of the district had been going
from bad to worse. Europeans and loyal Natives were assaulted by
disaffected Natives with impunity, so much so that the police were
directed to desist from attending beer-drinks in uniform, and not to
arrest any Native in the presence of others. By this time, Kula's tribe
had virtually become master of the district, doing whatever appeared
right in its own eyes.
It so happened the Magistrate (Mr. A.E. Harrington) was collecting hut
tax at Keate's Drift when Mr. Cross and party were fired on by
Bambata and others in Mpanza valley (3rd April). Chiefs Silwana and
Sibindi were ordered to arm and prevent the rebel Chief from
crossing into their wards. As soon as the message reached Sibindi,
he did all he could to assist, in fact most of his tribe in Umsinga
division were mobilized within two hours. Silwana's response was
half-hearted. About 11 p.m. the same day, Ottley and twenty men
arrived at the drift from Pomeroy, six hours after hearing of Bambata's
outbreak. The Magistrate, Umsinga, proceeded the next day (4th) to
collect taxes at Tugela Ferry, where, however, but few paid.
The unrest among the Umsinga Natives now became more
accentuated. Mtele, Kula's uncle and principal induna at Elands
Kraal, mobilized the whole of the people under his charge. Nondubela
joined him. The indunas of that portion of Ngqambuzana's tribe,
which was in Umsinga division, were reported by the Chief as
intending to cross into Zululand and join the rebels. All the Europeans
of the district went into lager at Helpmakaar, excepting the court
officials. The Umsinga Reserves were joined by those of Dundee,
Newcastle and Weenen early in May, together with a composite
Militia force under Major W. Murray-Smith, N.M.R.
Kula reported by messengers on the 4th May that Mtele was in open
rebellion. Harrington, in reply, remarked that he had two months
previously told Kula a portion of his tribe was in rebellion, but this had
been denied; "he sends only now to tell me what I knew two months
ago." Before receipt of this message, Kula had openly declared that
he would never come to the court again.
In consequence of a large number of Natives being expected at the
magistracy to pay taxes, a squadron N.M.R. (Capt. P.M. Rattray,
D.S.O.), with a few Reserves under Chief Leader D.C. Uys, was sent
there from Helpmakaar at 6.30 a.m. on the 8th, pending arrival the
same day of Lieut. G.R. Richards, M.L.A., and a squadron (100) N.C.
Kula, probably because he saw troops proceeding to the magistracy,
deemed it prudent to visit it too. Accompanied by a dozen of his
leading men, he arrived shortly before 11 a.m. Almost simultaneously,
Richards rode in with his men. Harrington conferred with the latter,
Rattray and Uys. He pointed out the degree to which Kula was
implicated in the Rebellion, and suggested the man should be
removed from the district. Richards, then in charge of the post,
though not senior officer, decided, with the concurrence of the other
officers, to transfer the Chief to make the statement he had already
made respecting disloyalty in his tribe to the Officer Commanding at
Helpmakaar. Kula was informed of the decision. A few minutes later,
he was required to proceed with Rattray and his men to Helpmakaar.
Six headmen were allowed to accompany the Chief. On arrival at
Helpmakaar, Kula was closely examined by Lieut.-Col. A.T.G. Wales,
who, in his turn, resolved to keep him in custody pending receipt of
orders from the Government.
The Government was naturally placed in an awkward position by
these unexpected and unusual proceedings. Richards, of course, had
made no actual arrest, and, in referring the question of the
advisability of Kula's continued presence in the district to the officer at
Helpmakaar, had done so because the point at issue appeared to be
one not for him but for some higher authority to decide. The
Commandant was surprised at Richards' action, and could he at once
have got into communication with the responsible officers, the
probabilities are that Kula would not have been either arrested or
detained, as everything pointed to his committing himself sooner or
later, when he would have been either captured or shot as a rebel.
Under the circumstances, in view of the then greatly disturbed
condition of the country, the Government caused the Chief to be
conveyed on the 9th from Helpmakaar to Pietermaritzburg, where he
was detained, though not in custody.
Those best competent to judge, firmly believe the arrest or enforced
removal of this important Chief at that particular moment checked the
spread of rebellion in those parts. This, indeed, was subsequently
realized by the Government. Thus, though there was irregularity in
the way in which Kula was removed, the step was justified by peace
being maintained at a time when a rising was imminent in the thickly-
populated district over which he had control.[243]
Kula's brother, Manuka, tried to usurp control of the tribe after the
former's departure. On such endeavour being detected, he was
promptly placed under arrest.

Simultaneously with the dispatch of the Zululand Field Force to

Nkandhla (1st May), squadron A (with the Sydenham troop of B),
N.M.R. (about 110), under Lieut.-Col. H. Sparks, V.D., were detailed
for Mapumulo, whilst the remainder of the regiment (160) proceeded
viâ Dundee to Helpmakaar, under Major W. Murray-Smith. A force,
[244] under the command of Murray-Smith (with Capt. G.T. Hurst as
Staff Officer), left Dundee for Helpmakaar on the 5th. It reached its
destination on the 6th, but moved on the 7th to a new site two miles
off, where a strong lager of wire entanglements was speedily erected.
It was from the foregoing column that the detachments proceeded to
garrison Pomeroy, as already noticed in connection with Kula's
removal. Patrols, too, were sent out in various directions. These
obtained intelligence in confirmation of Mtele and Nondubela being in
open rebellion.
When Wales left for Pietermaritzburg on duty (11th), Murray-Smith
took over the command. Finding the Natives referred to were actually
in rebellion, Murray-Smith decided to attack and prevent them from
inducing others to take up arms. An armed body of 200 to 300 was
discovered at a kraal in a valley some four or five miles from the
camp. As, however, the Commandant's instructions were that the
column should simply proceed to Helpmakaar and, having fortified
itself, await Mackay's arrival, it became necessary for permission to
assume the offensive to be obtained. This Colonel Bru-de-Wold, after
carefully considering the situation, gave in a few hours by telephone.
On the following day (12th May), Rattray was accordingly detailed to
take command of the mounted troops, viz. 60 to 80 N.C. (Lieut. P.W.
Stride); 60 to 80 N.M.R.; about 150 Newcastle, Ladysmith, Dundee
and Estcourt Reserves; and 8 Umsinga Reserves under Chief Leader
A. Müller, the latter being guides and scouts.
Leaving camp before daybreak, the force moved towards Elands
Kraal, some ten miles away. About 9 a.m., the scouts got in touch
with the enemy and exchanged a few shots. Rattray, close at hand,
pushed on at a gallop up a hill and attacked Mtele's impi, about 150
to 200 strong, the latter partly concealed in bushes. The impi, making
neither charge nor stand, was forthwith driven into an adjoining valley,
through which they were promptly pursued by N.C. and N.M.R. as far
as the Buffalo (about two miles), as well as in other directions. On
reaching the river, the Natives scattered more than they had already
done, some crossing into Zululand, whilst others concealed
themselves in as extraordinarily rough country on the Natal side.
The Reserves, with a Maxim (N.M.R.), took ground where the enemy
had first been seen. From such position, a continuous fire was kept
up, which proved very effective in breaking down any opposition that
might have been offered in such extremely rugged and out-of-the-way
country.
The main force with Murray-Smith had, in the meantime, occupied
high ground overlooking the Elands Kraal settlement and the Buffalo
valley. Here the N.F.A., supporting and covering Rattray, opened and
kept up a shrapnel fire on groups of retreating Natives. Such,
however, was ineffective on account of the long range.
The whole action, including pursuit, lasted about two hours. The
troops suffered no casualties; of the rebels, twenty-nine were killed,
eight prisoners were taken, and thirty cattle captured.
On an examination of the scene being made, two camps of war-huts
were found (about a mile apart). Food and clothing belonging to the
insurgents were discovered at the back of the hill on which they were
first sighted.
Two days later, Murray-Smith took a force (including N.C. at Pomeroy,
then having been relieved by N.R.R.) to Nondubela's kraal, at a hill
called Nqoro, near Buffel's Hoek, but the rebels, on the alert in
consequence of the action at Elands Kraal, escaped to Zululand. It
was on the same day that Mackay crossed into Natal at Rorke's Drift.
[245] Had it been possible for him to co-operate, Nondubela and his
force might not have escaped as cheaply as they did. As it was,
Murray-Smith went with a section of his force to Rorke's Drift, then
down the right bank of the Buffalo, whilst another section, under
Rattray, proceeded by road viâ Elands Kraal valley, and, after
crossing Mazabeko stream, converged with the main body on Nqoro
bill (below Fugitives' Drift). On this occasion, all kraals that were
come across belonging to rebels were destroyed. Over forty miles
were covered during the day. The feature of the day's operations,
however, was the splendid performance of the guns (N.F.A.) under
Wilson, which travelled a distance of not less than forty miles.
In addition to tshokobezi badges, it was noticed that strips of white
goat-skin, about an inch broad and tied round the neck, were used by
Natives of these parts to indicate being in rebellion. The prisoners, as
well as the killed, were found wearing them.
As soon as Mackay arrived, he took command and operated as
already described in Chapter XII.
Orders were received from the Commandant on 16th May for 100
N.M.R. and 100 N.R.R.[246] under Murray-Smith, to escort a convoy
of fifty-one waggons (supplies, etc.), from Dundee to Nkandhla.
Leaving on the 17th, the escort, strengthened at Vant's Drift by 100
N.C. under Richards, arrived at Nkandhla viâ Nqutu, Nondweni and
Owen's store, on the 24th. On the following day, the escort, less the
N.C.—detached to become bodyguard to Colonel McKenzie—was
ordered to return with the empty waggons to Dundee and join the
Umvoti Field Force (then under command of Major W.J.S. Newmarch,
U.M.R.) at Greytown, and do so viâ Tugela Ferry. Murray-Smith left
Nkandhla on the 26th and, returning viâ Nqutu, reached Dundee on
the 30th.
The night the escort reached Nqutu (28th May), an absurd scare
arose out of a belief that Mehlokazulu intended attacking the village
the same night. The Basutos had, in consequence, mobilized and
taken refuge at the gaol. As, at this time, Mackay was camped
sixteen miles away at Isandhlwana, it is difficult to understand how
the scare arose.
On arriving at Dundee, Murray-Smith got orders from Leuchars to
proceed by rail to Greytown, leaving N.R.R. at Dundee. He
accordingly entrained on the 1st June, reached Greytown 6 a.m. on
the following day, and joined the U.F.F. at Spitzkop on the 3rd,
temporarily taking command.
Shortly after, Mackay moved into Nqutu district, Zululand (27th May),
to co-operate with Leuchars. Lieut.-Col. J. Weighton was appointed to
command at Helpmakaar.
On being appointed (29th May), to the command of all troops south of
Tugela and in Nqutu district, Zululand, i.e. the whole of Natal plus
Nqutu, Leuchars, as has been seen, visited Helpmakaar to direct
operations from there. He arrived at that place on the 2nd June to find
that 1,000 of Chief Silwana's men had suddenly, and without
reference to the Commandant of Militia, or to himself, been called out
by the Minister for Native Affairs, with orders to be at Pomeroy on the
3rd. Efforts were at once made to secure a European leader. The
men displayed no enthusiasm whatever and were barely civil.
Although called out under the authority of the Native Code as an
ordinary levy, they asked to be armed with rifles and supplied with
blankets. The applications could not be entertained. Warnings were
received that the levy was not to be trusted. It was freely stated that
Silwana had forbidden the men to cross into Zululand. At Gordon
Memorial mission station they looted a European house. On a false
alarm occurring one night, a number of the younger men sprang to
arms with the war-cry, "Usutu!" Under these circumstances, Leuchars
decided to send them back to Weenen. Urgent requests, however,
came from Government and the Magistrate of Weenen to give them a
trial. Mr. G.A. Jackson, who knew the people, was accordingly
appointed to lead them. The order to return to their homes was
withdrawn. They were told Jackson would take them next day (7th) to
a locality where a number of returned rebels of Manuka's section was
in hiding. As there was a tribal feud of long standing between
Silwana's and Kula's people, it was thought the levy would have
undertaken the duty with alacrity. Jackson went to Helpmakaar to
arrange for supplies. On getting back to Pomeroy next morning to
lead the levy on, he found no less than 600 had deserted during the
night. Although about 100 of those remaining volunteered to go
forward, they were carried away by the majority, who, packing up their
bundles, made for their homes. After expressing regret at the
behaviour of their men, the indunas proceeded to do likewise.[247]
By way of facilitating control of the forces in Umsinga and Nqutu
districts, those under Mackay, less the Reserves sent back to
Helpmakaar, were separated from the Helpmakaar Field Force and
formed into a distinct column, with instructions to confine their
operations to the basin of the Buffalo and the north-west portion of
Qudeni mountain. Mackay's column then consisted of the N.C., Right
and Left Wings; N.R., one company; N.F.A., one battery (six guns);
and N.N.H., one squadron.
The Mapumulo garrison was detached from the U.F.F. and became a
separate column under Sparks, with authority to turn out 200 from
each of two loyal tribes to assist in searching for and arresting
returned rebels of Ngobizembe's tribe.
The H.F.F. was now made up of the garrisons of Pomeroy, Fort
Murray-Smith,[248] Helpmakaar, Paddafontein and Dundee. Wales
was appointed to command vice Weighton, who had been ordered to
take over court-martial duties.
Further proof was given by the Transvaal at this juncture of an
earnest desire to assist Natal in her trouble. That patriotic and well-
known organization, the Lancaster and York Association,
Johannesburg, conceived the idea of offering Natal the services of
150 men (twenty-five of them mounted). On approaching the
President, Mr. (now Sir) Abe Bailey, K.C.M.G., the latter not only
approved the proposal, but undertook to defray all expenses of
equipment, clothing, saddlery, etc.—excepting only salaries and food
supplies—out of his own pocket. This generous offer was most
gratefully accepted by the Natal Government. So enthusiastically did
Mr. Joe Bell, Mr. W. Beachy-Head and other members of the
Association take the matter up, ably assisted by the Staff of the
Transvaal Volunteer Headquarters, that after beginning to enrol on
the 1st June, the corps, under the command of Lieut.-Col. Thomas
Peakman, C.M.G., with Capt. G.H. Helbert, as Adjutant, actually
presented itself for inspection at 3 p.m. on the 3rd,[249] and, moving
off by train at 5.15 p.m., reached Dundee at 9.30 a.m. on the
following day, prepared for any service that might be required.
The greatest pains had been taken to select only the most efficient
out of the 1,500 applications handed in. The rapidity with which the
men were enrolled, clothed, equipped and entrained was as
surprising to the people of Johannesburg as it was to the Colony of
Natal. The L and Y, or 'Rosebuds,' as the corps was more familiarly
styled, were directed to attach themselves to Mackay's column. This
they did at Mangeni on the 9th June. Peakman was appointed
second in command of the column, and he and his contingent
remained with it until the end of the operations in Zululand.

Intelligence was received by McKenzie (at Nomangci) on the 9th

June of the presence of a strong rebel force at Kotongweni mountain
(i.e. close to the Tugela, above Watton's Drift). This was probably the
force whose expedition was referred to on p. 315, although the
expedition there dealt with was confined to the neighbourhood of
Watton's Drift. McKenzie ordered the U.F.F. to make a night march on
12th and, crossing next morning near Ngubevu Drift, to work down
Mfongozi valley, where a column from Nkandhla would be ready to
co-operate. Leuchars thereupon re-mobilized the 1st and 2nd
Krantzkop Reserves, instructing van Rooyen to march down the
Dimane valley to the Tugela and prevent the enemy from breaking
into Natal from Kotongweni, on their being pressed on the Zululand
side. The O.C. Helpmakaar, moreover, was directed to co-operate on
the north-west by moving a force of Reserves, N.R.R. and D.L.I. to
look out at the Buffalo for any of the enemy who might be returning to
their homes in that direction. It was at this time (10th), it will be
remembered, that the Mome action took place.
The U.F.F., joined by Sibindi, who had again voluntarily mobilized his
men, crossed at Ngubevu and proceeded to Mfongozi valley, where
touch was got with Mackay, but, in spite of constant efforts,
communication could not be opened up with the column from
Nkandhla until 11 a.m. on the following day. Leuchars operated in
Mfongozi valley both on the 13th and 14th, capturing many cattle.
After he had conferred with Colonel McKenzie, a decision was come
to for both columns to operate at Kotongweni on the following day.
Five hundred of Sibindi's men accordingly went down Mfongozi river
to the Tugela, whilst McKenzie operated from the top of the mountain.
The operations, however, proved unsuccessful.
The O.C. Troops now directed Leuchars to return to Broeder's Hoek,
about twelve miles from Krantzkop magistracy, keeping his own force
in the vicinity of Kombe forest. Mackay was instructed to remain on
Qudeni mountain during a three days' armistice which was
proclaimed at the same time, to afford rebels an opportunity of
surrendering. Captive women were utilized to make the proclamation
known to those who were in hiding.
The U.F.F. marched viâ Ngubevu to Broeder's Hoek. The Reserves at
Helpmakaar were demobilized on 16th and 17th June, with exception
of those of Umsinga, who were retained as garrison at Helpmakaar
lager, it being still unsafe for the women and children who had taken
refuge there to return to their farms.
With the object of relieving some of the men of Mackay's force, two
squadrons B.M.R. and one squadron N.C.(D), were mobilized, and
sent under Arnott to Dundee on the 18th.[250] Major Moe proceeded,
at the same time, with the whole of the N.N.H. to Pomeroy, in order to
bring rebels of Kula's tribe to book, also escort to Pomeroy Chief
Makafula of Nqutu district whom Mackay had been directed to arrest.
The arrest, however, was not made, as Mackay, who had been in
close touch with the Chief, had good grounds for believing he was not
disloyal, as supposed to be the case by those at a distance. In this
view, Mackay was supported by the Magistrate.
Whilst contemplating a demonstration in Silwana's location by
Mackay's column from the Umsinga side, and by the U.F.F. from that
of Greytown, Leuchars received a wire from O.C. Mapumulo reporting
that his convoy had been attacked on the morning of the 19th at
Oglesby's store, near Otimati, when one man had been killed and
another wounded. He thereupon caused his forces to concentrate at
Mapumulo as speedily as possible.
Before describing the position at Mapumulo, a thickly-populated
district which now became the principal focus of rebellion, it will be
necessary to take up the threads once more at Nkandhla and narrate
what happened between the action at Mome gorge (10th) and the
outbreak at Mapumulo just referred to.

FOOTNOTES:
[241] Giving, roughly, a population of 18,000 souls, or about 2,500
fighting men.
[242] The man referred to frequently in this history as Matshana ka
Mondise.
[243] Kula's uncle, Mtele, with the portion of the tribe that rebelled
with him, formed part of the force that attacked Leuchars at
Mpukunyoni.
[244] It was made up of 200 N.C., Right Wing (Major A.C.
Townsend); 160 N.M.R. (Capt. P.M. Rattray); N.F.A. (four guns)
(Major C. Wilson); 55 N.R.R. (Lieut. A. McKenzie); and
departmental details. When en route to Helpmakaar, it was joined
by the following Reserves: 80 Newcastle (Chief Leader Adendorff);
200 Estcourt (Chief Leader A.F. Henderson, C.M.G.); 55 Dundee
(Chief Leader D.C. Uys).
[245] His and Murray-Smith's men met near the Buffalo and
moved back together to the latter's camp.
[246] N.R.R. were relieved at Pomeroy on the 16th by D.L.I. (100)
(Capt. W.P.M. Henderson), the latter having arrived at Helpmakaar
on the preceding day from Dundee with a convoy of supplies.
[247] Report, Colonel G. Leuchars, C.M.G., 23rd Nov. 1906.
[248] That is, the lager, consisting chiefly of wire entanglements,
erected about two miles from Helpmakaar.
[249] By the General Officer Commanding-in-Chief, South Africa.
[250] Only, as will appear later, to proceed at once to Stanger.
 XVI.
CONCLUDING OPERATIONS, NKANDHLA.—VISIT OF
DINUZULU'S INDUNAS TO PIETERMARITZBURG.—
POSITION AT MAPUMULO.—ACTIONS AT OTIMATI AND
PEYANA (HLONONO).

Reference was made towards the close of the preceding chapter to

a combined move by Leuchars, Mackay and a column from Nkandhla
in the direction of Kotongweni on the 15th June. The object was to
drive the enemy with his stock from Qudeni mountain into the valley
of the Mfongozi river. The Nkandhla column, commanded by
McKenzie, consisted of the 'divisional troops' shown at the foot of the
page.[251]
The transport and N.F.A. (pompoms), with an escort of N.R. (three
companies, A, D and E, Major Boyd-Wilson), proceeded on the 12th
viâ Nkandhla and Ensingabantu to Ntingwe. On the same day, the
remainder of the divisional troops and R.H. (temporarily detached
from Royston's Brigade), visited and thoroughly searched Ofeni gorge
and ridge en route. Ofeni[252] is a remarkable chasm, about five miles
to the south of Empandhleni. A small stream that rises there and
descends rapidly to the Insuze, has the same name. The sides of the
chasm, which are over 300 feet in height, are linked together by
means of a tiny, natural bridge but a few feet in width. Makahleleka,
one of Sigananda's many and more important sons, was declared to
be in hiding at this uncanny place. The search, however, resulted in
practically no rebels being found. The troops afterwards proceeded to
Titlestad's store, at Ntingwe, where they bivouacked for the night.
Although the foregoing movement was carried out expressly with the
object of co-operating on the 13th with Leuchars and Mackay, then
near Kotongweni and Qudeni respectively, the plan, in so far as
McKenzie was concerned, was disturbed through receipt of
intelligence to the effect that Bambata, Cakijana and Mangati had
taken refuge in the bush at Macala. To surround the mountain by
daybreak on the 13th then, of course, became the immediate object.
Barker was ordered to co-operate. He was to take up positions on the
south, whilst McKenzie would do likewise in other directions. When
dawn broke and the latter's troops were in the positions assigned,
Barker was found exactly where it was desired he should be, i.e. at
the lower end of Macala bush. The fastness, which lay in a bush at
the top of the mountain, consisted of great masses of rock lying one
on top of the other in such a way as to form, below the surface of the
ground, a network of dark passages, the one communicating with the
other. Only with the greatest difficulty could people who had taken
refuge there be found and, when this occurred, the searchers, on
account of the irregular formation of the labyrinth and its narrow
passages, ran serious risks when dealing with a desperate enemy,
especially one who had reduced the length of his assegai to enable it
to be used with the best effect. A Native levy which had accompanied
the column, was instructed to drive the bush, whilst being supported
by the troops. During this operation, a rebel, who was concealed
under the rocks, stabbed one of the levy in the leg. On the drive,
which was partially successful, coming to an end, the underground
passages were entered and thoroughly searched by the N.C., with
the result that a number of other rebels was killed. It afterwards
transpired that these Natives, when at first they had found
themselves surprised by the troops, ran to the rock 'warren,' never
dreaming "people with boots on," as they put it, would venture to
explore so dark and perplexing a spot. The principal object of the
quest, however, was not attained, though Bambata's witch-doctor,
Malaza, was among the slain. Some 450 cattle were captured during
the day. McKenzie withdrew to Ntingwe, and Barker to near
Cetshwayo's grave.
Whilst the foregoing operations were in progress, the Z.M.R., under
Vanderplank, proceeded to the hill Jokwana, west of Macala, to get in
touch with Leuchars and Mackay, and to advise the former of what
was taking place at Macala. He was, moreover, to co-operate as well
as he could in carrying out the original plan. Owing, however, to the
haze, communication could not be established.
On the day following, 14th June, McKenzie moved up towards Kombe
forest, where he succeeded in getting into communication with
Leuchars. A drive of the combined forces through the valley that lay
between them was accordingly arranged and took place the same
day, but without result. Colonels McKenzie and Leuchars met, when
further combined operations were arranged to take place on the 15th
at Kotongweni, where Mangati and Cakijana were then alleged to be
hiding in caves. On the departure of the O.C. Troops from Nomangci,
Lieut.-Col. J.S. Wylie, D.L.I., was placed in charge of the camp.
Hedges, Calverley and Titlestad were, at the same time, instructed to
try and locate Sigananda, with a view to bringing about his capture or
surrender. One or other of these alternatives appeared imminent. As
a result of the untiring and well-directed efforts of these officers, not
only was the rebel leader's whereabouts discovered, but, on his being
persuaded to surrender, he did so forthwith, not, however, to Wylie,
but to an officer of lower rank. The latter's action, with Wylie in camp,
was inexcusable, and his acceptance of the surrender irregular and
invalid, as, of course, the only person competent to announce the
terms of surrender was the O.C. Troops. Unfortunately, the last-
named did not receive a notification as to what had happened until
twenty-four hours later. He decided that the surrender was to be
unconditional and be accompanied with those of all the Chief's
people, together with their arms. To this Sigananda agreed. On the
16th, he was conveyed by the balance of R.H. at Nomangci to
Empandhleni.
Boyd-Wilson, by making a creditable forced march with the transport,
succeeded in joining McKenzie at Kombe on the 14th.
The combined operations at Kotongweni (15th) proved disappointing.
In the neighbourhood of the camp, however, where the bushes were
searched by N.R., thirteen rebels were shot, whilst a large quantity of
goods, probably looted from European stores in the vicinity, was
discovered. Owing to Mackay not having got in touch with McKenzie
on the 13th, his column was unable to take part in the operations.
In the meantime, reliable intelligence had reached Nomangci of
Bambata having been killed during the action at Mome. Because of a
rumour circulated on the day of the action that he had escaped with a