Professional Documents
Culture Documents
Detecting Abnormal Traffic: Jennifer Rexford
Detecting Abnormal Traffic: Jennifer Rexford
Jennifer Rexford
Fall 2010 (TTh 1:30-2:50 in COS 302)
Example intrusions
Denial-of-service attacks
Scans
Worms and viruses
Host compromises
Intrusion detection
Monitoring and analyzing traffic
Identifying abnormal activities
Assessing severity and raising alarms 2
Where to Detect Intrusions?
End host: OS or application
Includes logins, file I/O, program executions, etc.
Can work with encrypted traffic and at lower speeds
Avoid extra packet reassembly and ambiguity
Actively defend
Detect problems in real time
Automatically generate a response
E.g., drop the traffic, engage the adversary, fight back
Faster response, but worse handling of false positives
4
How to Detect Intrusions?
The challenge
We dont know all the bad things that could happen
And telling good from bad is fundamentally hard
Anomaly detection
What is usual, is known
What is unusual, is bad
Signature detection
What is bad, is know
What is not bad, is good
5
How to Detect Intrusions?
Anomaly Detection Signature Detection
Patterns Train to create a Codify patterns of
baseline of normal known vulnerabilities
network traffic or attacks
Detection Detect statistically Detect matches to
significant deviations the patterns in the
from normal signatures
Pros Can detect novel Builds on past
(zero day) attacks experiences
Cons May miss low-rate Misses novel attacks;
attacks; high rate of requires continuous
false alarms updates to signatures
6
Anomaly Detection
Traffic volume
Detect deviations in bytes/sec or packets/sec over time
Not effective for detection low-volume attacks
Traffic features
Detect changes in distributions of traffic characteristics
E.g., traffic distribution by IP address, port number,
packet size, TCP flags, etc.
Aids in classifying the anomaly (e.g., DoS vs. port scan)
Detection techniques
Statistical techniques
Machine learning
... 7
Signature Detection
Examples
Excessive login attempts
TCP packet with both SYN and RST set
HTTP with GET /cgi-bin/phf?
Packet processing
Deep-packet inspection
Regular expression matching
8
Hybrid Solution: Arbor Example
anomaly
detection
scrubber
http://www.arbornetworks.com/en/stopping-ddos-attacks.html 9
Challenges
Accuracy
False positives
False negatives
11
Network Security
Denial-of-service mitigation
Pushback and source quenching
Secure Overlay Services
Spam mitigation
Network-based detection of spam
Advantage: ubiquitous
Supported on all networking equipment
Multiple products for polling and analyzing data
Disadvantages: dumb
Coarse granularity of the measurement data
E.g., number of byte/packet per interface per 5 minutes
Cannot express complex queries on the data
Unreliable delivery of the data using UDP
Collecting Traffic Data: Packet Monitoring
Packet monitoring
Passively collecting IP packets on a link
Recording IP, TCP/UDP, or application-layer traces
Advantages: details
Fine-grain timing information
E.g., can analyze the burstiness of the traffic
Fine-grain packet contents
Addresses, port numbers, TCP flags, URLs, etc.
Disadvantages: overhead
Hard to keep up with high-speed links
Often requires a separate monitoring device
Collecting Traffic Data: Flow Statistics
Scope
Medium-grain information about user behavior
Passively monitoring the link or the interface/router
Helpful in characterizing, detecting, diagnosing, and fixing
Outline
Definition of an IP flow (sequence of related packets)
Flow measurement data and its applications
Mechanics of collecting flow-level measurements
Reducing the overheads of flow-level measurement
IP Flows
source dest
input output
source dest
prefix prefix
Monitor
Router Collecting Flow Measurement
Advantage
No need for separate measurement device(s)
Monitor traffic over all links in/out of router (parallelism)
Ease of providing routing information for each flow
Disadvantage
Requirement for support in the router product(s)
Danger of competing with other 1st-order router features
Possible degradation of the throughput of the router
Difficulty of online analysis/aggregation of data on router
Practical application
View from multiple vantage points (e.g., all edge links)
Packet Monitor Collecting Flow Records
Advantages
No performance impact on packet forwarding
No dependence on support by router vendor
Possibility of customizing the thinning of the data
Disadvantages
Overhead/cost of tapping a link & reconstructing packets
Cost of buying, deploying, and managing extra equipment
No access to routing info (input/output link, IP prefix, etc.)
Practical application
Selective monitoring of a small number of links
Deployment in front of particular services or sites
Cache replacement
Remove flow(s) when the flow cache is full
Evict existing flow(s) upon creating a new cache entry
Apply eviction policy (LRU, random flow, etc.)
Long-lived flows
Remove flow(s) that persist for a long time (e.g., 30 min)
otherwise flow statistics dont become available
and the byte and packet counters might overflow
Sampling: Packet Sampling
Packet sampling before flow creation (Sampled Netflow)
1-out-of-m sampling of individual packets (e.g., m=100)
Create of flow records over the sampled packets
Reducing overhead
Avoid per-packet overhead on (m-1)/m packets
Avoid creating records for a large number of small flows
not sampled
timeout
two flows
Conclusions
Flow measurement
Medium-grain view of traffic on one or more links
Advantages
Lower measurement volume than full packet traces
Available on high-end line cards (Cisco Netflow)
Control over overhead via aggregation and sampling
Disadvantages
Computation and memory requirements for flow cache
Loss of fine-grain timing and per-packet information
Not uniformly supported by router vendors