Professional Documents
Culture Documents
AlienVault Building Collector Plugins
AlienVault Building Collector Plugins
Admin Guide
Table of Content
1
Overview..................................................................................................................................................... 4
1.1
1.1.1
1.1.2
1.2
1.2.1
1.2.2
1.3
2
Rsyslog ..............................................................................................................................................10
2.1.1
2.1.2
2.1.3
Filters ........................................................................................................................................10
2.2
2.2.1
2.2.2
Parameters ...............................................................................................................................11
2.3
2.3.1
2.3.2
2.3.3
Parameters ...............................................................................................................................13
2.3.4
2.3.5
2.4
Aliases ...............................................................................................................................................16
2.4.1
Path...........................................................................................................................................16
2.4.2
2.5
Functions .......................................................................................................................................... 16
2.5.1
Path...........................................................................................................................................16
2.5.2
Conversions ..............................................................................................................................16
2.5.3
2.5.4
2.6
Event Fields....................................................................................................................................... 18
2.7
Rules .................................................................................................................................................19
2.7.1
Page 2
Evaluation Order.......................................................................................................................19
Structure ...................................................................................................................................19
2.8.1
2.8.2
2.9
2.9.1
2.9.2
Debugging .................................................................................................................................................22
Appendix...................................................................................................................................................23
5.1
5.2
5.2.1
Scenario ....................................................................................................................................25
5.2.2
5.2.3
5.2.4
5.2.5
5.2.6
Check whether the new entries are written in the new log file...............................................26
5.2.7
5.2.8
5.2.9
5.2.10
5.2.11
5.2.12
5.2.13
Page 3
1 Overview
1.1 OSSIM Agent Role
1.1.1 Event Collection
The collection process involves extracting the data logs from the source systems (Security, OS,
RDBMS, etc.) and allows first steps for event log filtering. At this stage can be decided what is going
to be read by the OSSIM Agent and what is going to be discarded before having an impact on the
system performance.
Before starting to write a plugin some actions to reduce the amount of events could be considered:
Manage the log level settings at the application and managed device level
In deployments with a big amount of analysed data, filtering at the application level
should be done whenever possible
Log Files
Good practice is to use one log file per plugin in order to increase performance. Having just
one generic log file, all the plugins would have to read the same extensive content in order
to catch the few relevant entries.
Using rsyslog it is possible to filter the collected logs based on the syslog tags.
Raw Event
The raw event might be a generic syslog message, an application log, an SNMP trap, the
result of an SNMP or SQL Query or some other kind of information in a more or less
structured form that is appended to a log file.
Example:
dmz01:/var/log/auth.log:
May 30 13:15:52 dmz01 sshd[12980]: Accepted password for root from
192.168.178.20 port 4445 ssh2
Page 4
Normalized Event
There is a certain set of fields which are required in order to ensure a consistent evaluation
and correlation of the events by the OSSIM server. These fields can be populated with
information from the log message or statically through the plug-in.
Example:
ossim-sensor:/var/log/ossim/agent.log:
2010-05-30 13:15:49,441 Output [INFO]: event type="detector" date="1275239752"
sensor="192.168.178.201" interface="eth0" plugin_id="4003" plugin_sid="7"
src_ip="192.168.178.20" src_port="4445" dst_ip="192.168.178.200" dst_port="22"
username="root" log="May 30 13:15:52 dmz01 sshd[12980]: Accepted password for
root from 192.168.178.20 port 4445 ssh2" fdate="2010-05-30 13:15:52" tzone="0"
Page 5
Enriched Event
The OSSIM Server enriches the event with the Priority and Reliability values, which are
specific to the event type (plugin_id) and subtype (plugin_sid), as well as with the Asset
Value which is specific to the Source (asset_src) and the Destination (asset_dst) hosts.
Example:
ossim:/var/log/ossim/server.log:
2010-05-30 06:48:41 OSSIM-Message: Event received: event id="0" alarm="0"
type="detector" fdate="2010-05-30 13:15:52" date="1275239752" tzone="0"
plugin_id="4003" plugin_sid="7" src_ip="192.168.178.20" src_port="4445"
dst_ip="192.168.178.200" dst_port="22" sensor="192.168.178.201" interface="eth0"
protocol="TCP" asset_src="2" asset_dst="2" log="May 30 13:15:52 dmz01
sshd[12980]: Accepted password for root from 192.168.178.20 port 4445 ssh2"
username="root"
Priority
The priority is related to threats and it reflects the importance of a specific attack, having
nothing to do with a specific host or environment. It only measures the relative importance
of the attack itself.
Range: 0 - 5
Default value: 1
Example:
A Unix server running Samba gets attacked by the Sasser worm .
Apart from the fact that the attack wont have an impact on the given environment, it
has the potential to exploit a big security hole and for that reason the priority is
considered as being high.
Reliability
Classical risk-assessment would refer it as "probability ". Since it's quite difficult to
determine how probable it is for a network to be exposed to certain vulnerabilities, the IDS
related reliability approach was considered more appropriate.
Range: 0 - 10
Default value : 1.
Example:
If a host connects to 5 different hosts in the same subnet using port 445, could be a
normal behavior, unreliable for IDS purposes. If connecting to 15 hosts would be
Page 6
Asset Value
It is assigned to both the Source and the Destination Hosts and represents the importance
the host has to the enterprise.
Range: 0 - 5
Default value: 1 (also used for hosts not being defined in the asset database)
Example:
A database server can have an asset value of 5, a development test server an asset
value of 2 and an unknown host in the Internet causing a portscan event would just
have an asset value of 1.
Alarm
Based on the Event Priority (0-5), Event Reliability (0-10) and the Asset Value (0-5), a Risk
Value (0-10) is calculated and for values equal or greater than 1 Alerts are generated.
The Risk is calculated based on the following formula:
Risk = (Priority * Reliability * Asset) / 25
Page 7
First thing to start with is checking which log messages the application generates and eventually
identify sets of logs having a similar structure. Those logs having a similar structure will be where
possible covered by a single collector rule.
o
Best is to copy one existing file and modify its content to match the new application. Should a plugin
exist for a similar application, it is recommended to copy such a file, as there is a good chance that
rules have a similar content and are grouped in a similar way - a generic HTTP-Proxy log will always
contain a URL, a generic Firewall log will contain a Source IP Address and Source Port as well as a
Destination IP Address and Destination Port. Some user defined fields might be defined for a specific
application and the correlation at the server level can be simplified if similar applications use the
same user defined fields.
o
This is the last Rule to evaluate, which catches all the events that cannot be grouped under specific
rules.
o
The Specific rules are defined for specific error conditions or categories of events. There might also
be that one single rule is used to generate different types or subtypes of events.
o
Discard Noise
Events that are considered noise can be discarded by OSSIM by excluding certain event subtypes
(Plugin_SIDs) in the plugin file, by the way the regular expressions are defined or by using policies.
However, the best way to discard events is by filtering them on the monitored device or at syslog
level on the host running the OSSIM Agent.
o
The rules are evaluated alphabetically, which means that all it counts is the name of a rule and not
the position in the plug-in file. The Generic Rule might even be on the first position if the name is
properly chosen. Having rules alphabetically placed after the Generic Rule will have as effect that
the corresponding logs will be evaluated as generic events instead of having the proper event type
and subtype assigned.
o
In order to have a Plugin activated and sending events to the OSSIM server, the path to the plugin file
has to be specified in the Agent configuration file.
Page 8
This is required in order to let the server know which events should be expected and which priority
and reliability values the events should get assigned.
o
Testing
Using the logger command sample logs can be replayed in order to test the operation of the OSSIM
Agent or Server.
Page 9
Page 10
pid:
[event-consolidation]
Enables event consolidation at agent level. It is recommended to use polices instead of this
feature as consolidation at the agent level affects the correlation process.
by_plugin:
enable:
time:
Example:
[event-consolidation]
by_plugin=1001-1150,1501-1550,4001-4010
enable=False
time=10
[log]
Configures the verbose level and the path to the different log files
error:
file:
stats:
verbose:
[output-plain]
Writes in a log file what is being sent to the OSSIM Server (Useful for debugging and
developing purposes)
enable:
file:
[output-server]
Configures the server to which events are sent
Page 11
enable:
ip:
port:
Page 12
enable:
interval:
restart_interval:
Log
Database
mssql
- Microsoft SQL
mysql
- MySQL
SDEE
SnortLog
- Snort logs
WMI
Example:
plugin_id=4003
Page 13
detector
enable:
source:
location:
The file(s) where the logs can be found - can contain multiple
comma-separated files
create_file:
process:
start:
stop:
startup:
shutdown:
exclude_sids=SID List
Example (hp-eva):
process=snmptrapd
start=yes
stop=yes
startup=/etc/init.d/snmpd start
shutdown=/etc/init.d/snmpd stop
exclude_sids=404,200,403
[translation]
string=value
Example (Postfix):
[translation]
sent=10
bounced=11
[Rule IDs Specific Rules]
Here are the events collected and normalized.
event_type=event
regexp=Regular Expression
plugin_sid=Plugin SID
Event_Field=Value
Example(ssh):
[01 - Failed password]
event_type=event
regexp="(\SYSLOG_DATE)\s+(?P<sensor>[^\s]*).*?ssh.*?Failed password for inval
user (?P<user>\S+)\s+from\s+.*?(?P<src>\IPV4).*?port\s+(?P<sport>\PORT)"
plugin_sid=1
date={normalize_date($1)}
Page 14
Page 15
2.4 Aliases
2.4.1 Path
/etc/ossim/agent/aliases.cfg
2.4.2 Predefined Regular Expressions
The predefined regular expressions can be used when creating new plugins.
IPV4= \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}
MAC= \w{1,2}:\w{1,2}:\w{1,2}:\w{1,2}:\w{1,2}:\w{1,2}
PORT= \d{1,5}
TIME=
\d\d:\d\d:\d\d
SYSLOG_DATE= \w{3}\s+\d{1,2}\s\d\d:\d\d:\d\d
SYSLOG_WY_DATE= \w+\s+\d{1,2}\s\d{4}\s\d\d:\d\d:\d\d
To use an Alias in the regular expression use the \IPV4, \MAC, \SYSLOG_DATE, etc.
2.5 Functions
2.5.1 Path
/usr/share/ossim-agent/ossim_agent/ParserUtil.py
2.5.2 Conversions
Page 16
resolv(host):
resolv_ip(addr):
resolv_port(port):
normalize_date(date):
normalize_protocol(proto):
md5sum(datastring):
upper(string):
hextoint(string):
intrushield_sid(sid,name):
all McAfee Intrushield IDs are divisible by 256, and this length
doesn't fit in the OSSIM table ( mcafee_sid =
hextoint(mcafee_sid)/256)
netscreen_idp_sid(msg):
iss_siteprotector_sid(msg):
resolv_iface(iface):
Page 17
Event Type
plugin_sid
Event Subtype
The time the event has been collected from the device
sensor
interface
protocol
src_ip
src_port
dst_ip
dst_port
username
password
filename
Optional
userdata1 userdata9 User defined fields that could be used in custom reports,
correlation directives, etc.
Special types of events and the list of fields that can be used in each event type:
Host-os-event
Host-mac-event
Host-service-event
host
host
host
os
mac
sensor
sensor
vendor
interface
interface
sensor
port
date
interface
protocol
date
service
application
date
Page 18
2.7 Rules
The Rules define the format of each event and how they are normalized. It is composed by a regular
expression and the list of fields that the event will include once it is sent to the OSSIM Server.
In some cases only one regular expression will collect every event coming from one application, in
some other cases more than one rule will be required.
2.7.1 Evaluation Order
Rules are loading in alphabetical order based on the name given to each rule (Rule ID).
Once the log matches the regex of one rule the ossim agent stops processing the event, therefore
generic rules must be the last to be evaluated.
2.7.2 Structure
o
Name / Rule ID
The name of the rule is mandatory
Regular Expression
The regexp field contains the regular expression that defines the format of the events, and
extracts the information to normalize the event.
The regular expression has to be written following Python regular expression syntax:
http://docs.python.org/library/re.html
The information extracted by the regular expression from the log can be accessed by:
Position: (\d\d):(\d\d):(\d\d)
hour={$1}
minutes ={$2}
seconds={$3}
Tags: (?P<hour>\d\d):(?P<minutes>\d\d)(?P<seconds>\d\d)
hour={$hour}
minutes ={$minutes}
seconds={$seconds}
Normalized Fields
As the server must receive normalized events, where IP addresses for instance are using the
IPV4 format and the date uses the format YYYY-MM-DD HH:MM:SS (2010-12-31 22:57:00)
To simplify the process of normalizing events functions are defined (more details on
functions can be found in the Functions section of this document):
resolv()
Translates hostnames into IPV4 addresses (DNS queries)
Page 19
Translations
Used for instance when the Event ID is not numeric, but plugin_sid has to be numeric.
Translations have to be defined inside the [translation] section. The actual translation is
triggered by using the translate() function.
Exclusions
Some events can be filtered during the collection process editing the configuration file for
each plugin:
Page 20
Remove the Plugin ID from the plugin table, should such an entry already exist
Remove the Plugin SIDs from the plugin_sid table, should already exist
To run the script use the following command (please double-check the content of the SQL scripts
and the command line syntax before applying the changes to the database):
ossim-server:/usr/share/doc/ossim-mysql/contrib/plugins# ossim-db < ssh.sql
Example (/usr/share/doc/ossim-mysql/contrib/plugins/ssh.sql):
-- SSHd
-- plugin_id: 4003
DELETE FROM plugin WHERE id = "4003";
DELETE FROM plugin_sid where plugin_id = "4003";
INSERT INTO plugin (id, type, name, description) VALUES (4003, 1, 'sshd', 'SSHd: Secure Shell
daemon');
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES
(4003, 1, NULL, NULL, 'SSHd: Failed password', 3, 2);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES
(4003, 2, NULL, NULL, 'SSHd: Failed publickey', 2, 2);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority,reliability) VALUES
(4003, 99, NULL, NULL, 'SSHd: Generic SSH Event', 1, 1);
Page 21
3 Log files
Generic Syslog
/var/log/syslog (Unix)
/var/adm/messages (Solaris)
To identify where the logs for specific applications or certain logging levels are saved, check the
/etc/syslog.conf or /etc/rsyslog.conf files.
OSSIM Agent
/var/log/ossim/agent.log
OSSIM Server
/var/log/ossim/server.log
4 Debugging
Note: Do never leave an application running in Debug mode in a production environment
OSSIM Agent
ossim-agent vv
OSSIM Server
ossim-server D6
Page 22
5 Appendix
5.1 Regular Expressions
Operator
Meaning
\c
[]
One or any of the characters ; accepts intervals of the type a-z, 0-9, A-Z
[^]
A char different from ; Accepts intervals of the type a-z, 0-9, A-Z
Regular Expression
Matches with
a.b
a..b
[abc]
[aA]
[aA][bB]
[0123456789]
0123456789
[0-9]
0123456789
[A-Za-z]
A B C ... Z a b c ... Z
[0-9][0-9][0-9]
[0-9]*
[0-9][0-9]*
^.*$
A full line
Page 23
Meaning
r*
r+
r?
r{n}
n occurrences of the RE r
r{,m}
r{n,m}
r1|r2
The RE r1 or the RE r2
Regular expression
Matches with
[0-9]+
[0-9]?
empty_string 0 1 2 .. 9
(ab)*
([0-9]+ab)*
Regular expression
Matches with
Equals
\d
[0-9]
\D
[^0-9]
\s
[ \t\n\r\f\v]
\S
[^ \t\n\r\f\v]
\w
[a-zA-Z0-9_]
and _
\W
\Z
End of line
Page 24
[^a-zA-Z0-9_]
localhost
Wed Jul 14 18:49 - 19:21 (00:31)
localhost
Wed Jul 14 19:23 still logged in
localhost
Wed Jul 14 19:23 still logged in
localhost
Wed Jul 14 19:23 - 19:24 (00:00)
localhost
Wed Jul 14 19:23 - 19:24 (00:00)
172.22.22.10 Wed Jul 14 18:38 - 19:24 (00:45)
172.22.22.10 Wed Jul 14 19:24 still logged in
172.22.22.10 Wed Jul 14 19:24 - 19:26 (00:01)
172.22.22.10 Wed Jul 14 19:26 still logged in
172.22.22.10 Wed Jul 14 19:26 - 19:26 (00:00)
Page 25
5.2.6 Check whether the new entries are written in the new log file
opensourcesim:/etc/ossim/agent/plugins# tail -f /var/log/last_logon.log
Jul 14 19:38:49 dmz01 LOGON_EXAMPLE: > root pts/2
localhost
Wed Jul 14 19:38 still logged in
Jul 14 19:38:54 dmz01 LOGON_EXAMPLE: > root pts/2
localhost
Wed Jul 14 19:38 - 19:38 (00:00)
Jul 14 19:38:59 dmz01 LOGON_EXAMPLE: > ossim pts/2
localhost
Wed Jul 14 19:38 still logged in
Jul 14 19:40:51 dmz01 LOGON_EXAMPLE: > ossim pts/2
localhost
Wed Jul 14 19:38 - 19:40 (00:01)
Jul 14 20:15:09 dmz01 LOGON_EXAMPLE: > reboot system boot 2.6.31.6
Wed Jul 14 17:39 - 20:15 (02:35)
Page 26
Page 27
Page 28
2)
3)
Page 29
Rules having the same plugin_sid will only require one SQL statement and plugin_sid defined on the
OSSIM server. Different rules where used just because both IP addresses and hostnames are
returned as sources by the last command.
-- plugin_id: 9001
DELETE FROM plugin WHERE id = "9001";
DELETE FROM plugin_sid where plugin_id = "9001";
INSERT INTO plugin (id, type, name, description) VALUES (9001, 1, 'Example', 'User logons based on the last output');
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 1, NULL, NULL, 'Login: System
console' , 5, 5);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 2, NULL, NULL, 'Logout: System
console' , 5, 5);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 3, NULL, NULL, 'Login: Pseudo
terminal' , 3, 5);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 4, NULL, NULL, 'Logout: Pseudo
terminal' , 3, 5);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 5, NULL, NULL, 'System reboot:
Restarted' , 5, 5);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 99, NULL, NULL, 'Last: Generic
messages' , 1, 1);
After changing the script to reflect the Plugin IDs and SIDs, load the changes with the command:
opensourcesim:/usr/share/doc/ossim-mysql/contrib/plugins# cat example.sql | ossim-db
Page 30
Plugin ID
Plugin SIDs
5.2.11
5.2.12
Page 31
Events
Alarms
Page 32