The Design and Implementation of a
Kernel Monitoring System for Network Servers
Yutaka Nakamura
yutaka-n@rd.center.osaka-u.ac.jp
Shinji Shimojo
shimojo@cmc.center.osaka-u.ac.jp
Cyber Media Center of Osaka University
Suguru Yamaguchi
suguru@is.aist-nara.ac.jp
Nara Institute of Science and Technology
Abstract
From a stand point of server operation, performance
measurement of the server is important. One of the major
parts of the measurement is kernel monitoring, and most
server administrators have been legacy tools that can influ-
ence server performance by the load itself. In this paper,
we describe the design and implementation of a new kernel
monitoring system that does not influence a server perfor-
mance by dividing measurement module and analysis mod-
ule. Furthermore, our monitoring system can be appled to
a distributed server system.
1. Introduction
To keep good quality of service, a server administrator
should tune a server with the best performance and avoid
saturation of the server. However, many administrators still
operate the server systems based on their intuition and ex-
periment that frequently causes miss-configuration of the
servers. Thus, we developed a server performance measure-
ment tool called ENMA [1]. ENMA monitors packets out-
side the server and observes several performance indices of
the server system.
However, there is a problem of packet monitoring; the
obtained information does not show a direct sketch of the
kernel status. To solve this problem, we developed a new
kernel monitoring system and integrated it with ENMA. It
monitors several indices of kernel status and transports them
to a remote machine that analyzes them in real time. Be-
cause it records precise timestamps in the log data, the data
can be merged easily with other data such as the access log
of the server.
Eiji Kawai
eiji-ka@is.aist-nara.ac.jp
Japan Science and Technology Corporation
Hideki Sunahara
suna@is.aist-nara.ac.jp
Nara Institute of Science and Technology
Our kernel monitoring system, called rep2, gives fairly
low influence to the server even in the high loaded state.
We made experiments with actual servers with a huge num-
ber of real users on the Internet. The results show that our
system can make more detailed observation of the server
system without server performance degradation.
2. Design and Implementation
In general, a kernel monitoring program depends on the
operating system on which the program works. To improve
the portability of our system, we divided our system in two
parts, an OS-dependent part and an OS-independent part.
Figure 1 shows the design of our kernel monitoring system.
Server System Analyzing Host
Visualizer
Server
Prosesses Reportor2 Collector
Graph
Generator
Operating System
CPU information
Memory information
Process information
Network information
Figure 1. Kernel Monitoring System (rep2)
The OS-dependent part interacts with the kernel through
the kvm interfaces, because the target server host employs
the Solaris operating system in our experiment. The kvm in-
terface is supported also by other operating systems such as
FreeBSD. There is some information that is not supported
by the kvm interface such as the buffer status of the protocol
stack and input/output packets. We also use kstat and sysctl
interfaces for this kind of information.
3. Case study of Large Scale WWW Server 100

90
IDLE
USER
KERNEL
100

90
IDLE
USER
KERNEL

80 80

70 70

To show that rep2 is effective enough, we give an exam-

CPU Utilization(%)

CPU Utilization(%)
60 60

50 50

ple to apply rep2 to a large scale actual WWW system. The 40

30
40

30

system is for broadcasting the 82nd National High-School 20 20

10 10

Baseball Games of Japan. The system got about 46 million 0

14:45:00 14:55:00 15:05:00 15:15:00 15:25:00 15:35:00 15:45:00
0
14:45:00 14:55:00 15:05:00 15:15:00 15:25:00 15:35:00 15:45:00
Time Time

accesses per day at the maximum.

Figure 3. Time transition of CPU utilization
3.1. System composition

Figure 2 shows the network environment of this experi- 100 100

with rep2

Cumulative Distrivution Function(%)

Cumulative Distribution Function
90 90 without rep2
ment. We prepared four servers and one layer 4 switch. The 80 80
70 70
layer 4 switch applied round robin algorithm to distribute 60 60
50 50
accesses from many clients. The WWW1 uses Apache 40 40
30 30
server system on Solaris 2.7. From WWW2 to WWW4 use 20 20
10 WWW1 10
Chamomile server system that is our original system for this 0
1e-05 0.0001 0.001 0.01 0.1 1
WWW2
10 100
0
2 3 5 7 10 20 30 50 70 100

service. Response Time (sec)

Time(msec)

Internet Figure 4. Cumulative Distribution Function of

Layer 4 Switch(Server Iron) Response time

WWW1 WWW2 WWW3 WWW4

ENMA ENMA ENMA ENMA

response time with and without rep2. The right graph of

Visualization
Figure 4 shows also cumulative distribution of response
time of WWW1 on both situations. Because they do not
Figure 2. Network Environment show any difference, our system is effective enough not to
influence the server performance.

In this experiment, we observe the server using the

4. Conclusion
ENMA as shown in Figure 2. The visualization host ac-
quires the observation data through the control line. Be-
We showed the effectiveness of our kernel monitoring
cause the visualizer can handle data from several server
system. We combined ENMA and rep2, and observed an
hosts, we can see not only each host data but also total data
actual system from both the outside and the inside.
of the system.

3.2. Results Acknoledgement

Figure 3 is the output of CPU utilization on WWW1 and
WWW2. The CPU cycles consumed by kernel activity on
two systems are at the same level. However, User level ac-
tivity of WWW2 is higher than that of WWW1. Accord-
ing to the reference [2], the server system is saturated when
the CPU kernel rate becomes 90%. Therefore, WWW1 and
WWW2 can process more WWW requests.
The left graph of Figure 4 is cumulative distribution of
response time of the two servers. The response time of
Chamomile is a hundred times slower than that of Apache.
One of the reasons of the poor responsibility is in it’s ses-
sion scheduler. Chamomile handles hundreds of sessions
with a few threads, and that causes large queuing delay.
We also investigate performance degradation through ex-
ecuting rep2 on the server system by comparing the server
This work was supported in part by Research for the
Future Program of Japan Society for the Promotion of
Science under the Project ”Integrated Network Architec-
ture for Advanced Multimedia Application Systems”(JSPS-
RFTF97R16301)
References
[1] Yutaka Nakamura, Ken-ichi Chinen, Hideki Sunahara, Sug-
uru Yamaguchi. ENMA: The WWW Server Performance
Measurement System via Packet Monitoring. In Inet’99, San
Jose, CA, June 1999. http://enma.aist-nara.ac.jp/.
[2] J. Almeida, V. Almeida and D. Yates. Measuring the Behav-
ior of a World-Wide Web Server. Computer Science Depart-
ment, Boston University, 1996, October 29