The design and implement of the centralized log gathering and analysis system

Jian-hua Huang, Man-qi Zhang, Yuan-long Jiang

School of Information Science and Engineering
East China University of Science and Technology
Shanghai, China
jhhuang@ecust.edu.cn, yzszmq@163.com, jyl7-2@163.com

Abstract-Logs generated by network devices and systems
provide important information for network management. In
this paper, we describe a centralized syslog system which
gathers and analyzes log messages from a number of routers,
switches and firewalls. The gathered logs are filtered and
categorized with regular expression, and finally stored in a
MySQL database with format. Through the statistics analysis,
feature-based detection on security events, the system can
effectively find out abnormal behavior of network devices and
ensure the network security. Some methods are found out to
allow us to check if the network behavior is unusual. These
perspective methods also provide the basis of network
management and security strategy design for administrators,
thereby strengthen further network management.
Keywords-router; firewall; syslog; regular expression
I. INTRODUCTION
With the continuous and rapid development of the
computer networks and network applications, the security
issues need more special attention. In order to safeguard the
network security, people tend to adopt quite a lot measures.
Among them, the log messages which record various system
events occurred every day in the network devices, play an
important role in network security management.
Shanghai education and research network (SHERNET)
is a large metropolitan computer network, which connects
all the universities in Shanghai. The backbone nodes which
composed a ring topology are connected by 10 NE40E
routers. Each node site consists of a large number of routers,
switches, firewalls, and application servers. Since the
___________________________________
978-1-4673-0089-6/12/$26.00 ©2012 IEEE
SHERNET has been transformed several times, the number,
species and complexity of the network devices are
increasing sharply. In order to find out the network
anomalies and security threats, it is necessary to view all the
log messages generated every day by active networking
devices. It is out of human capabilities. So we should use
some special software tools which help us to skip over
mostly daily event messages to find out the network
anomalies and security threats effectively and accurately.
As the complexity and diversity of the network devices,
a standard network protocol named syslog is applied to
gather log messages from firewalls, routers, switches and
other network devices. For the purpose of solving problems
in practice, a syslog system which can gather, analyze and
store log messages is designed. Network devices in
SHERNET can send their log messages to the centralized
syslog server which filters raw logs and stores parsed results.
By collecting and extracting useful information, the
administrators can keep abreast of the network status and
solve security issues timely.
The rest of the paper is organized as follows. Section II
discusses related works on syslog. Section III presents an
overview of our system. Section IV gives implementation of
the system. Section V gives statistical analysis techniques to
logs. Section VI concludes with a discussion of the results.
II. RELATED WORKS
A. Syslog Protocol
The syslog protocol was introduced by the Computer
Science Research Group (CSRG) at the University of


California-Berkeley as part of the Berkeley Software
Distribution of UNIX. Syslog was designed to provide the
ability to report system events. These events are collected by
a process and then recorded in log files on the local system,
a remote system or both. Administrators can view the log
files, which can help them to keep track of system status and
ensure the normal operation of systems. It is very common
for switches, routers, firewalls, and other devices to be
configured to log data via the syslog protocol [1].
Syslog adopts the user datagram protocol (UDP) as its
underlying transport layer mechanism. The format of a
syslog message which is defined in RFC3164 has three
discernable parts. The first part is called the PRI, the second
part is the HEADER, and the third part is the MSG. They are
showed as follows:
<priority>timestamp hostname tag: content
First is PRI (<priority>). The logs collected via syslog
describe activity and conditions monitored by diverse audit
components which are categorized as facilities in syslog
parlance. An audit component supplies a rating of its
severity. The priority of a log is defined as the pairing of the
corresponding facility and severity:
Priority=facility*8+severity (facility and severity adopt
the decimal digit code)
The HEADER includes timestamp and hostname. The
timestamp field is the local time and is in the format of
"Mmm dd hh:mm:ss". The hostname field contains only the
hostname, the IP address of the originator of the message.
The MSG consists of tag and content. The value of the
tag is the name of the program or process that generated the
message. The content contains the details of the message.
B. Related Studies
The potential utility of log analysis for security event
detection is clear. A report on data breaches investigated by
the Verizon Corp Business Risk Team reported in 2008 that
66% of organizations investigated had sufficient evidence
available within their logs to discover the breach had they
been more diligent in analyzing such resources [2].
The studies of system logs to characterize system and
network behavior have been a focus of research for some
time. Early work on the study of logs focused on statistical

modeling of errors and failure prediction based on the
models. Gu Zhaojun [3] implemented a log analysis system
which gathered and pretreated syslog logs generated by PIX
firewall, then carried on the TopN statistical analysis and
detected security events for realizing the effective
surveillance on the network behavior. Karel Slavicek [4]
described mathematical methods that are trying to be used
for syslog message processing. The goal was to find out
some mathematical description of network behavior which
would allow us to check if the network behavior is usual or
if it needs some special attention. Cisco IOS also provides a
programmable framework that allows us to filter, correlate
and customize system log messages prior to delivery to a
syslog server [5]. The disadvantage of this approach is its
high demand for manual programming work.
The important event messages are sometimes hidden in a
large number of relatively less important and usual event
messages. Some research has been devoted to find out
anomalies accurately and quickly by using data mining of
log message. Justin Myers [6] presented a distributed event
correlation system which performs security event detection,
and evaluates it experimentally, compared with a centralized
alternative. The comparison measures the value in
distributed event correlation by considering network
bandwidth utilization, detection capability and database
query efficiency. Fukuda [7] presented a method to construct
the log time series for anomaly detection in order to
highlight such hidden anomalies, his work focused on the
effectiveness of using a global weight that is based on a
global appearance of a message type in the all data set.
Although many analysis tools have been developed to
find unusual events in a network, the analysis is expensive
and difficult. The reasons for this include the sheer volume
of data to collect, process and store, the difficulty of doing
the actual analysis, the difficulty of effectively reporting the
results. In this paper, we present an effective system for
collecting, categorizing, storing, and analyzing syslog
messages. Some methods which accurately find out
anomalies in SHERNET are discussed.
III. ARCHITECTURE OF THE SYSTEM
A. SHERNET system parts. The received syslog messages will be
SHERNET is a large metropolitan computer network, pretreated, sent to the real-time alarm module, and stored
which connects all the universities in Shanghai. The into the database, in this way we can do further analysis.
backbone nodes which composed a ring topology are made Logs gathered from SHERNET include basic
of 10 NE40E routers. Figure 1 depicts the ring topology of information of IP layer’s dataflow, many of them are
SHERNET. redundant. Therefore, prior to storing them into the database,
the system needs to analyze them and extract the useful field,
SHU SHUFE
which helps to enhance the efficiency of the system and

ECNU FDU
avoid resources wasting as well.
2) Real-time Alarm
In the centralized log gathering and analysis system, in
DHU TJU
addition to the syslog messages that are stored into the
database for further analysis after pretreatment, the real-time
CERNET
running status of the network should be monitored. In order
SJTU SHSMU to discover the heritage of network and activities which

CERNET2
violate security policies, real-time alarms are essential.
3NET
SHNAP
SHMU Real-time alarms are triggered by the real-time alarm
module according to a certain alarm strategy, which consists
Figure 1. The ring topology of SHERNET
of several alarm rules laid down in advance. It can report the
B. Architecture of the System exception of SHERNET to the monitoring center promptly

Our system is designed for gathering logs from routers, to take further measures.

switches, firewalls and some other devices in SHERNET. 3) Database

The system consists of 5 parts: log generators, log receiver, Although the gathered logs had been pretreated, the

database, real-time alarm, and statistical analysis. Its amount is still quite huge. In order to respond rapidly to the

architecture is shown in Figure 2. system when analyzing and processing data from the
database, MySQL is chosen as the log database. The core
Device Real-time thread of MySQL is completely multi-threading, which
Alarm might accelerate the speed of the system. MySQL supports
Device
C++ and many other programming languages, and open
Syslog Statistical


Database database connectivity (ODBC) as well.

Receiver Analysis
Device 4) Statistical Analysis
After collection, pretreatment and storage of log
Figure 2. The architecture of the system
messages is done, we can analyze and audit logs in the
C. Log Generators database next. Due to the amount of logs is quite huge, the
system sets up a log query function, which is user-friendly.
Log generators are devices generated logs. These devices
The TopN statistics analysis and feature-based detection on
are routers, switches, firewalls and some other devices in
security events are carried out.
SHERNET. They report their own events to the syslog
receiver. IV. IMPLEMENT OF THE SYSTEM
1) Syslog Receiver
The centralized log gathering and analysis system was
Network devices in SHERNET send their log messages
designed for gathering logs from routers, switches, firewalls,
to port 514 of the syslog receiver which is the core of all


and some other devices in SHERNET. The whole system
was developed by C++ language, included five modules
which are the log generators, the log receiver, the database,
real-time alarm, and statistical analysis engine. To
effectively monitor the running status of the network, the
system gathered, pretreated syslog logs, sent them to
real-time alarm for examining security event, stored them
with format in MySQL, then carried on statistical analysis
and, finally displayed the results with kinds of formats, so
the administrators can manage easily the network and
institute the security policy.
A. Thread Pool and Database Connection Pool
The thread pool technology provides a sound solution for
both the problem of thread life-cycle overhead and the
problem of resource insufficiency. Thread pool is a form of
multi-threaded processing, tasks to be processed will be
added to the queue, and then the thread is created to execute
these tasks automatically. Thread pool is particularly
suitable for implementing tasks that require multiple threads,
which can eliminate the delay generated by creating the
thread. After applying the thread pool technology, the log
receiver engine becomes more responsive and its
performance gets greatly improved.
In the process of the program accessing to the database, a
connection with the database should be established firstly,
however, the time of establishing database connection is far
more than data handling. In order to avoid consuming time,
our system adopts the database connection pool technique,
which can connect and disconnect the database rapidly. The
database can be shared by different clients; thereby the
efficiency of data handling and the server performance are
improved.
B. Log Parsing
The syslog protocol is quite simple, detailed provisions
are not given in the protocol. Each manufacturer has its own
structure of HEADER and MSG parts. The following is an
example of the syslog format from a HuaWei NE40E router.
<189>Jun 7 05:22:03 2011 Quidway
IFNET/6/UPDOWN: Line protocol on interface
Ethernet0/0/0, changed state to UP

The syslog format of Cisco routers is as follows:
<189>931: *Sep 11
20:07:26: %LINEPROTO-5-UPDOWN: Line protocol on
Interface FastEthernet0/2, changed state to down
Log parsing technology is a light spot of the system; we
adopt a two-layer analysis approach to provide a simple
interface to separate syslog messages for devices from
different manufacturers. Regular expressions are adopted to
conduct the two-layer analysis. In the first layer analysis,
regular expressions are used to separate log messages into
log fields which are saved into the database for statistical
analysis and depth data mining. A rule library which
includes regular expressions to analyze the logs of devices
from different manufacturers is used to the second layer
analysis. The rule library consists of the rules which are used
to find out system abnormalities. Log fields from the first
layer analysis will be submitted to the real-time alarm
module for the second layer analysis, from which network
anomalies and potential attacks can be found out through
matching with the rules in the rule library.
In computing, a regular expression (RE) provides a
concise and flexible means to match strings of text, such as
particular characters, words, or patterns of characters. The
concept of regular expressions was first popularized by
utilities provided by Unix distributions, in particular the
editor ed and the filter grep. A regular expression is written
in a formal language that can be interpreted by a regular
expression processor, which is a program that either serves
as a parser generator or examines text and identifies parts
that match the provided specification.
The following are ARP alarm logs generated by a
SepiaNIPS firewall, which show that the firewall found a
new device and recorded its IP address and MAC address
and some other information.
Mar 17 13:20:59 SepiaNIPS alert:
mac=00:11:92:76:e8:40, ip=192.168.2.2, type=new
Mar 17 13:22:29 SepiaNIPS alert:
mac=00:11:92:76:e8:40, ip=192.168.2.3, type=ipchange
Mar 17 13:26:18 SepiaNIPS alert:
mac=00:11:92:76:e8:40, ip=192.168.2.2, type=new
We use regular expressions to analyze the above syslog
messages. The following regular expressions are written columnar, caky and curve tendency chart. According to the
according to ARP alarm logs: statistical analysis information, the administrators may judge
1> %timereported:::date% whether some abnormal events exist, thereby revise the
2> %hostname% device rules in time.
3> %syslogfacility-text% TopN sort is a course which selects the highest N
4> %msg:R,ERE,1,BLANK,0:mac=([0-9a-f\ :]+)—end elements of a certain standard. Some useful information can
5> %msg:R,ERE,1,BLANK,0:ip=([0-9\ .]+)--end% be gathered from the database to obtain the TopN statistics,
6> %msg:R,ERE,1,BLANK,0:type=([a-z]+)--end% which mainly included SRC_IP, DST_IP, traffic and so on.
The fields of MSG, port and some others obtained
A. SRC_IP
through the first layer analysis are main input messages of
Source IP statistics can help managers discover the most
the second layer analysis, which contains detailed
traffic host in the network. However, when a host’s traffic is
information of logs, such as specific intrusion methods, user
far more traffic than others, this host may be attacked, or be
name and some other information.
attacking, or download something by BT. Figure 3 shows
Two log types and corresponding regular expressions are
the users’ traffic in the network.
illustrated as follows:
1) System Maintain
*matching logs of successful login into a device
Successful\\s*([A-Z]+)\\s*login
* matching logs of failed login into a device
([A-Z]+)\\s*login\\s*failed
2) Attack
* IP spoofing Attack
ip\\s*spoofing\\s*-\\s*WAN\\s*((?:TCP)|(?:UDP)|(?:IG
MP)|(?:ESP)|(?:GRE)|(?:OSPF))
Figure 3. Source IP counts
ip\\s*spoofing\\s*-\\s*
WAN\\s*ICMP)\\s*\\(type:(\\d+),\\s*code:(\\d+)\\ B. DST_IP
ip\\s*spoofing\\s*-\\s*no\\*routing\\s*entry\\s*ICMP)\\s Destination IP reflects the most popular host in the
*\\(type:(\\d+),\\s*code:(\\d+)\\ network. According to the times of the accessed Destination
ip\\s*spoofing\\s*-\\s*WAN\\s*((?:TCP)|(?:UDP)|(?:IG IP, the system carried on the Top10 statistics. The
MP)|(?:ESP)|(?:GRE)|(?:OSPF)) proportion of destination IP is displayed in figure 4.
V. LOG ANALYSIS
Administrators can get a lot of useful information from
the gathered syslog messages. For example, we can find out
which application is error and whether the device is
subjected to the invasion. According to the log data from the
database, some import and useful information was made the
TopN statistics and analyzed security event, which mainly
included facility, severity, source IP, destination IP, protocol,
destination port, traffic and so on, then the system attained
results and displayed them via the table, the linearity, Figure 4. Destination IP counts


 We adopt pattern analysis for parsing the syslog
messages of known structure. Regular expression is used to
search the syslog messages for known pattern of an event.
For example, the default service port of Microsoft SQL
Server is 1433. If a firewall records a lot of tests aiming at
port 1433, there may be password attacks on a SQL Server.
The following log shows that a host addressed 172.20.45.81
is trying the password of the SQL Server addressed
202.120.107.179:
<3>Nov 16 23:44:01 SepiaNIPS kernel:
DROP_DS_AntiDDoS:IN=br0 OUT=br0 PHYSIN=eth1
PHYSOUT=eth0 SRC=172.20.45.81 DST=202.120.107.179
LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14685 DF
PROTO=TCP SPT=45120 DPT=1433 WINDOW=14600
RES=0x00 SYN URGP=0
Compared with analyzing device’s log message one by
one, a centralized log analysis allows aggregation of
messages reporting the same event and summation of
repeating messages. For example if there is physical line
break on line connecting two backbone routers, maybe we
receive three messages from each router. These messages
are “link down”, “line protocol down”, “routing protocol
down”. These six messages report one link interruption
event. The following log shows that line protocol go down
on a router.
<189>Dec 3 09:33:12 2011 Quidway
IFNET/6/UPDOWN: Line protocol on interface
Ethernet0/0/0, changed state to DOWN
When the line is restored the syslog server will receive
six new messages reporting that the line, line protocol and
routing protocol went up on both ends of the line. Figure 5
shows the line breaks of SHERNET on a whole day.
Figure 5. The line breaks between nodes of SHERNET

The syslog messages from all network devices are
gathered together in a database. The statistical analysis
engine can find out a lot of unusual events to support daily
maintenance of SHERNET.
VI. CONCLUSION
Log messages play an important role in network security
management. The main contribution of this paper is to
present a solution for gathering, storing and analyzing a
large amount of log messages from different devices. An
approach which uses regular expressions for parsing log
messages and matching network anomalies is proposed. We
have implemented a centralized log gathering and analysis
system which is used to detect network anomalies, carry on
the statistical analysis, and ensure the network security. Our
future works will include the following 2 aspects:
Large quantities of log messages are gathered to the
syslog server. The system will inevitably need larger storage
space. On the aspect of processing the massive logs, it needs
further study on how to raise its efficiency and accuracy.
Log messages gathered to the syslog server contains
large quantity of information. We should provide the
network a massive information processing platform based on
the high performance data mining. Via association rules
analysis, we can also find out the users’ habits and conduct
further improvement.
REFERENCES
[1] FC3164: The BSD Syslog Protocol[S]. 2001.08.
[2] Baker, Wade, A. Hutton, C. D. Hylender, C. Novak, C. Porter, Bryan
Sartin, Peter Tippett, and J. A. Valentine. Data Breach Investigations
Report. Technical report, Verizon Business RISK Team, 2009.
[3] Gu Zhaojun, Li Yong, Niu Wenjing, China Tianjin. Analysis and
implement of PIX firewall syslog log[C]. Information Management
and Engineering (ICIME), 2010 The 2nd IEEE International
Conference, Chengdu, 16-18 April 2010.
[4] Slavicek, K.; Ledvinka, J.; Javornik, M.; Dostal, O. Mathematical
Processing of Syslog Messages from Routers and
Switches[C] .Information and Automation for Sustainability(ICIAFS
2008). 4th International Conference,2008, Page(s): 463-468.
[5] Anand Deveriya. An Overview of the syslog Protocol[M]. Cisco
Press, 2005.
[6] J. Myers, M. R. Grimaila, and R. F. Mills. Log-Based Distributed
Security Event Detection Using Simple Event Correlator[C]. System
Sciences (HICSS), 2011 44th Hawaii International Conference,
Date:4-7 Jan. 2011.
[7] K. Fukuda. On the use of weighted syslog time series for anomaly
detection[C]. Integrated Network Management (IM), 2011 IFIP/IEEE
International Symposium, Date:23-27 May 2011.