Symantec Endpoint Protection

Creating Custom IPS Signatures

Andrew Willetts
Technical Product Manager
Endpoint Security

Copyright 2007 All Rights Reserved

Overview of Symantec Endpoint Protection Signature-based Intrusion Prevention
System

The Network Threat Protection component within Symantec Endpoint Protection includes the
following Intrusion Detection/Prevention capabilities:
• Generic Exploit Blocking (GEB)
• Packet- and stream-based IPS
• Custom IPS signatures similar to Snort™

The product includes an Intrusion Prevention System (IPS) Library database that makes it much
easier to set up and manage signature-based Intrusion Detection and Prevention.

Certain attacks are commonly launched against specific applications. The pattern of code that
identifies certain attacks is known as a signature. For example, a signature can be designed to
watch packets of information received for the string “phf” in “GET / cgi-bin/phf?” as an indicator of
a CGI program attack. Each packet is evaluated for that specific pattern.

A collection of Intrusion Prevention signatures make up a signature library, Symantec provides a

preconfigured library of signatures. Although you cannot modify signatures in this library,
administrators can, exclude signatures, and create and manage custom signature libraries. This
can be done to either to supplement or replace the preconfigured Symantec IPS library.

This document will describe how to create and test custom IPS signatures.

Creating Custom Intrusion Prevention System Signature libraries

It is possible to develop custom libraries of signatures to add to the Endpoint Protection Manager
database for use in Intrusion Detection and Prevention. Signature libraries can be imported or
exported, and one or more IPS libraries can be applied to individual groups. Signatures can also
be copied between libraries.

When developing libraries, signatures are organized into signature groups in order to manage
them more easily. At least one signature group must exist in a custom signature library in order to
add signatures to it. The syntax of a signature must be correct or the library will not be loaded.
Custom Intrusion Prevention Library Example

The following exercise shows an example of the process for adding a custom Intrusion Prevention
Library with custom signatures, and applying it to a specific group. In this example, you are
adding a custom Intrusion Prevention Library with a custom signature that detects an attempt to
access MP3 files through FTP or the web.

The format of an MP3 file makes it difficult to detect in network traffic. The common tools used to
retrieve these files, however, use simple commands in their respective protocols, which can be
easily detected. Packet captures of both HTTP and FTP requests for an MP3 file are shown below.

Packet capture of an HTTP GET request

Packet capture of an FTP RETR request

As can be seen above, the packets captured from the HTTP and FTP sessions clearly show these
commands. Understanding the flow of communication between a client and server during the
session is necessary if custom signatures are to be created. Various pieces of information are sent
back and forth between these two systems, contained in TCP packets destined for the appropriate
service (port 80 for HTTP, and 21 for FTP) on the server. Within these packets is a payload
component containing the required information.

The highlighted packets show the HTTP GET and FTP RETR commands utilized by browsers and
FTP clients when downloading files. The FTP command is also used when multiple files are
retrieved using the MGET command. The file name, including the mp3 extension, is present in
both requests, and both protocols insert [CR][LF] characters to mark the end of the request. This
provides sufficient information to create custom signatures to prevent these requests from being
transmitted.

The signatures will need to contain several parameters, including a regular expression that
identifies the specific commands that should be blocked. Regular expressions are patterns of
characters that are compared against data, in this case, the contents of the packet. The
commands we wish to block are clearly contained in these packets. As we do not know the file
name, we need to use a wildcard to match the unknown number of characters between the
command and the filename. We also need to consider the possibility that the command may not
be issues in upper case, and also that the file extension may be capitalized. With this knowledge,
we can build the signature syntax as follows:
• In order to apply this rule to tcp packets : rule tcp :
• Packet must contact tcp-ack flag (reduces false positives) : tcp_flag&ack
• Request must originate on the host : saddr=$LOCALHOST
• For the FTP rule:
o limit to destination port 21 (FTP) : dest=(21)
o descriptive name for rule in logs : msg=”MP3 in FTP” :
o match ASCII text “RETR [.*].mp3[CR][LF]” – not case sensitive
regexpcontent="[Rr][Ee][Tt][Rr] .*[Mm][Pp]3\x0d\x0a"
• For the HTTP rule:
o limit to destination ports 80 and 443 : dest=(80,443)
o descriptive name for rule in logs : msg=”MP3 in HTTP” :
o ASCII text “GET [.*].mp3[CR][LF]” – not case sensitive
regexpcontent="[Gg][Ee][Tt] .*[Mm][Pp]3 .*\x0d\x0a"

rule tcp, dest=(21), tcp_flag&ack, saddr=$LOCALHOST, msg="MP3 detected",

regexpcontent="[Rr][Ee][Tt][Rr] .*[Mm][Pp]3\x0d\x0a"

rule tcp, dest=(80,443), tcp_flag&ack, saddr=$LOCALHOST, msg="MP3 GET in HTTP detected",

regexpcontent="[Gg][Ee][Tt] .*[Mm][Pp]3 .*\x0d\x0a"
Complete content of the custom signatures
 To create the custom Intrusion Prevention Library and Signatures:

1. Click on the Policies button.

2. Under View Policies, click Intrusion Prevention.
3. Click Add Custom Intrusion Prevention Signatures. The Custom Intrusion Prevention Library
dialog box appears.
Note: An example signature group and signature are listed. To view the signature content,
select the signature and click Edit below the Signatures table. If you want to retain the group
for reference but disable it, clear the Enable This Group checkbox under the group name and
description fields. If you want to delete the sample group, select it and click Delete below the
Signature Groups pane.

Create a custom IDS Signature Group

4. Type a name for the custom library, and optionally add a description.
5. At the bottom of the Signature Groups pane (on the left of the dialog box), click Add to add a
new signature group.
6. Type a name for the signature group, and optionally add a description.
7. The group you added is listed in the Signature Groups box and is automatically enabled. If you
do not want it enabled yet, clear the Enable this Group checkbox.
8. Below the Signatures table, click Add. The Add Signature dialog box is displayed. Fill in the
following:
• Type a Name and Description (optional) for the signature.
• Indicate the Severity from 0 (most critical) to 15 (informational only).
• Indicate the direction of traffic to be checked for the signature.
 • e Content fie
Tyype the syntax of the signature in the eld. For details on signatture syntax, see
s
th
he documentt “Symantec Sygate Enterprise Protecction Policy Manager Adm ministration
Guide”, Chaptter 17, ”Setting Up Custo om Signaturees.”
• Iff you want sp nature, click the Add buttton below th
pecific appliccations to trigger the sign he
list of applications. In the Add Applica ation dialog box,
b specify thet Name an nd Descriptioon of
thhe applicatio
on.
• In
ndicate the action
a you wa ure - Allow or
ant to occur as a result of this signatu o Drop:
Allow results in detection
A n only. The atttack is identtified and the
e Agent recoords the
in
nformation in
n a Security log and, optiionally, a Paccket log. Howwever, the atttack is not
blocked.
D
Drop results in the Agentt blocking the
e attack, as well
w as logging it.
• gged, select Write
Iff you want occcurrences of this signature to be log W to Paccket Log.

C
Create the cu
ustom IDS Signatures

9. Click OK. The sign

nature is add
ded to the Sig
gnatures boxx and is enab
bled.
10. Repea
at the steps to add both of the requirred signature
es to the sign
nature group
p.

Custtom IDS Sign

nature Group
p is complete
e
11. Click OK. The new library is added to the Intrusion Prevention Libraries information area. It is
listed in the Custom Intrusion Prevention dialog box in the Location-independent Policies
and Settings for all groups but is not yet enabled for any group. To enable it, you must apply
the library by checking the Enabled box next to the new custom library.

Enable custom signature group under Location-independent Settings

Testing the Custom Intrusion Prevention Signatures

With the Custom Signature Library applied to a group containing a test client, the new IPS Library
will be applied by the Network Threat Protection component of the client the next time an update
is sent to the client (check your group Communications Settings configuration if this does not
occur after just a few seconds – you can also check the System Log on the client to see when the
new library is applied).

Attempt to access an MP3 file through an FTP client. It does not matter if the client is command-
line or GUI based, the same RETR command is used by all clients (that were tested) – the
command will time-out, and the remote server will reset the connection, effectively disconnecting
the client.

Remote host closes connection

Attempt to access an MP3 file through an web browser. The download will not occur and will time-
out after many attempts.

File download never occurs

Timeout occurs after a long delay

The client will record these events in the appropriate logs, and send to the Endpoint Protection
Manager within a couple of minutes. The event can be viewed in the Console in the Firewall
Attacks Logs, and the msg parameter provided in the signature syntax allows easy identification
of these events.

Event details in the Firewall Logs

Additional Resources

For additional details on the IPS signature syntax, please see the SEP documentation.
Additional information on Intrusion Detection and Signatures may also be found on the
internet.

Copyright © 2007 Symantec Corporation. All rights reserved. Symantec, the Symantec Logo,
Symantec Network Access Control, Symantec Sygate Enterprise Protection are trademarks or
registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries.
Other names may be trademarks of their respective owners.

The product described in this document is distributed under licenses restricting its use, copying,
distribution, and decompilation/reverse engineering. No part of this document may be
reproduced in any form by any means without prior written authorization of Symantec
Corporation and its licensors, if any.

THE DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED

CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED
WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR
NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH
DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. SYMANTEC CORPORATION
SHALL NOT BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES IN
CONNECTION WITH THE FURNISHING PERFORMANCE, OR USE OF THIS
DOCUMENTATION. THE INFORMATION CONTAINED IN THIS DOCUMENTATION IS
SUBJECT TO CHANGE WITHOUT NOTICE.

Symantec Corporation
20330 Stevens Creek Blvd.
Cupertino, CA 95014
http://www.symantec.com