s u c h a s F 5 B i g I P ) o r I P f i x ( f o r t h e n e we r C i s c o s a n d J u n i p e r s ) s h ou l d be

s u p p o r t e d b y a N e t f lo w c o l l e c t o r in 2 0 1 6 . I a l s o t h in k l i b p c a p s u p p o r t sh ou l d
b e i n . F o r a n a ly s i s , s o m e h o w . S o wh a t a bo u t a p p l y i n g t h e m a n u a l f o r S i L K
in the box now 53?

P a y a t t e n t i o n t o it , a n d y ou w i l l e n d u p l ik e t h i s :

config.status: executing silk_summary commands

* Configured package: SiLK 3.12.2

* Host type: x86_64-unknown-linux-
gnu
* Source files ($top_srcdir): .
* Install directory: /usr/local
* Root of packed data tree: /data
* Packing logic: via run-time plugin
* Timezone support: UTC
* Default compression method: SK_COMPMETHOD_NONE
* IPv6 network connections: YES
* IPv6 flow record support: NO
* IPFIX collection support: YES (-L/usr/local/lib -
lfixbuf -lpthread -lgthread-2.0 -pthread -lglib-2.0)
* NetFlow9 collection support: YES
* sFlow collection support: YES
* Fixbuf compatibility: libfixbuf-1.7.1 >=
1.6.0
* Transport encryption support: NO (gnutls not found)
* IPA support: NO
* ZLIB support: YES (-lz)
* LZO support: YES (-llzo2)
* LIBPCAP support: YES (-lpcap)
* C-ARES support: YES (-lcares)
* ADNS support: NO
* Python interpreter: /usr/bin/python
* Python support: YES (-Wl,-z,relro -
Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions -
L/usr/lib -lz -ldl -lutil -lm -Wl,-z,relro -
L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -
pthread)
* Python package destination:
/usr/lib/python2.7/dist-packages
* Build analysis tools: YES
* Build packing tools: YES
* Compiler (CC): gcc
 * Compiler flags (CFLAGS): -I$(srcdir) -
I$(top_builddir)/src/include -I$(top_srcdir)/src/include -
DNDEBUG -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -Wall -W -Wmissing-
prototypes -Wformat=2 -Wdeclaration-after-statement -
Wpointer-arith -fno-strict-aliasing -O3
* Linker flags (LDFLAGS):
* Libraries (LIBS): -llzo2 -lz -ldl -lm

D e pe n d i n g o n y ou r r e q u i r e m e n t s … y ou m ig h t a c t u a l l y w a n t I P v 6 . M a k e s u r e
y o u e n a b le P y t h o n s u p p o r t . W e w i l l u se t h a t .

Let's get SiLK configured with ipt_netflow

T h e r e a r e t w o c on f i g f i l e s a n d t h e y n e e d c o r r e s p o n d i n g e n t r ie s .

Sample: /data/sensors.conf
probe local netflow-v9
listen-on-port 2055
protocol udp
log-flags bad
end probe

group my-network
ipblocks 1.2.3.4/32
ipblocks 192.168.100.0/24
end group

sensor local
netflow-v9-probes local
internal-ipblocks @my-network
external-ipblocks remainder
end sensor

O k i , n o w f or t h e c or r e s p o n d i n g s i l k . c on f .

Sample: /data/silk.conf
version 2

sensor 0 local "local"

class all
sensors local
end class

class all
type 0 in in
type 1 out out
type 2 inweb iw
type 3 outweb ow
type 4 innull innull
type 5 outnull outnull
type 6 int2int int2int
type 7 ext2ext ext2ext
type 8 inicmp inicmp
type 9 outicmp outicmp
type 10 other other

default-types in inweb inicmp

end class

default-class all

path-format "%N/%T/%Y/%m/%d/%x"

packing-logic "packlogic-twoway.so"

V i m , :set paste, c o p y & p a s t e , d o n e .

sudo /etc/init.d/rwflowpack start

Starting rwflowpack: rwflowpack: Ignoring --archive-
directory since no probes use directory polling
[OK]

Now let's query for the data from ipt_netflow

N o w g r a b a c of f e e o r p a c k y ou r g y m b a g … b e c a u se t h i s t a k e s s o m e m in u t e s .
I t ’ s f l o w d a t a . Y o u w il l f i g u r e ou t h ow i m po r t a n t t h a t i s i n j u s t … 5 - 1 2 3 4 5 6 7
seconds.

A l s o … c h e c k netstat -tulpen f o r U D P : 20 5 5 :
udp 0 0 0.0.0.0:2055 0.0.0.0:* 0 18856743 -

T h i s m e a n s y ou c a n a l s o p i p e d a t a f r o m a n ot h e r h o st i n t o t h is “ S i l k o n a
b o x ” . I r e c o mm e n d t o u se d i f f e r e n t p or t f o r e a c h d e v i c e t h ou gh . T h i s w a y y o u
c a n b u i l d p r o f i l e s , a n d ma n a g e t h e se n s o r s e a s i l y .
N et f l ow v 9 pl ea se

sysctl net.netflow.protocol=9

Check.

Wait for it... wait...

/usr/local/bin/rwfilter --sensor=local --proto=0-255 --
pass=stdout --type=all | rwcut | tail

95.211.83.20|148.251.236.208|56923|51413| 6| 12|
2586|FS PA |2016/09/25T13:37:12.900|
0.164|2016/09/25T13:37:13.064|local|
93.169.25.233|148.251.236.208| 2962|51413| 17| 2|
258| |2016/09/25T13:37:13.084|
0.000|2016/09/25T13:37:13.084|local|
105.103.127.116|148.251.236.208|18229|51413| 17| 2|
290| |2016/09/25T13:37:13.004|
0.000|2016/09/25T13:37:13.004|local|
...

N o w … s u r e . G r e p , A W K , s e d , G n u p l o t a n d t i m e f o r A S C I I g r a ph s a n d g o od o ld
C S V f i l e s . O r n o t . I t ’ s n o t t h e s u m me r of 69 . W e h a v e 2 0 1 6 .

Results so far
We have:

• a high performance Netflow source. We can

scale up it’s performance by adding CPU power

to a VM

• a high performance Netflow collector with state

of the art analysis tools - commandline based

though

• the option to also use sFlow and IPFIX if we

want

• a method to generate real-time Netflow from

mirrored traffic, also via an IPtables hack

• a well-suited network analysis toolchain with

100s of functions
Wasn’t that worth 10 minutes?

FlowBat - a Web UI for SiLK

W a i t a m in u t e … t h e r e a l s o i s a n a u t o m a t e d S i L K o n t h e b o x
i n s t a l l e r 1 3 5 … I d i d n ’ t t e l l y ou . W o w , wh a t a …

T h e r e a s o n i s n ot , t h a t I w a n t t o w a s t e y ou r t i m e . T h e r e a s on is t h a t y ou w i l l
r u n i n t o a p r o b l e m o r t w o ; l a t e r . An d t h e r e f o r e we on l y in s t a l l F l o w B a t w i t h
t h e s c r i p t . N ot S i L K . C o n t r o l i s e v e r y t h in g, m a n .

W e h a v e ma d e su r e t h a t ou r S i L K st u f f i s f u n c t i o n a l a l r e a d y . T h i s i s a mu c h
b e t t e r st a r t p o i n t . W h i l e y ou e x e c u t e t h a t s c r i p t , r e a d s o me s ou r c e c od e .
R e a l l y . N o w h e r e ’ s a t r i c k q u e s t i on f o r t h y m a s t e r :
i s localhost 0.0.0.0 o r 127.0.0.1. I t d e pe n d s [ t m ] . D oe s i t ? W h a t a b o u t
o p e n in g http://$host:1800 n o w t o g e t s t a r t e d ?

Y o u w i l l r e a l i z e t h a t F l o w B a t i s n ’ t l i k e t h e F l o w V i e w e r 3 5 . F l o w V i e we r i s a
h a c k y s e t of P e r l w i t h ma n y m a n y m a n y m a n y ma n y m a n y m a n y m a n y m a n y
m a n y bu g s . A n d a 1 99 0 s t y l e W e b U I . F l ow B a t i s m od e r n , a n d a h a c k y s e t o f
Node.js.

C h e c k o u t t h e g r a p h s , a n d g e t c on v i n c e d :

Pasted image1340×593 28.2 KB

Pasted image1714×416 17.9 KB

P r e t t y p i c t u r e s… I a m i n ! T h e r e a s o n w h y I b or e y ou w i t h a l l o f t h i s i s , t h a t
t h i s i s a w e s o me w o r k .

• You can setup FlowBat to collect the

FlowRecords via SSH. It’s possible to include

collectors, external to this installation, without

having to link the systems via an API or

anything complex.

• You can export the records as a CSV and include

them in your daily manual log processing. For

example with Tableau

• There are countless stats you can generate with

FlowBat. Depending on your CIDRs etc. If you

c a n n o t b u i l d t h e q u e r y w i t h rwfilter, c h a n c e s

are good that you are doing it wrong. Really.

• the query builder is advanced. I don’t think that

there are feature gaps.

• FlowBat is fast. Faster then FlowViever if your

know what I mean.

SilK to JSON for regular log crunching

S o m e n e t w or k s a r e m o r e i m p o r t a n t t h a n o t h e r s . F o r t h e n e t wo r k r a n g e s y ou
hold dear, I have a good way to keep an eye on them.

F o l l o w i n g u p m y e a r l i e r b l o g , we c a n se t u p t h e S u m o Co l l e c t o r , f i l e -f o r wa r d
t h e J SO N o u t p u t , a n d p o s t - p r o c e s s t h e r e c o r d s i n t h e c l o u d .
T e c h n i c a l l y t h i s s h ou l d l o ok v e r y s i m i l a r t o h o w we h a n d l e S u r i c a t a ’ s
E V E . j s o n , s in c e t h e J S O N s e a r c h q u e r y o pe r a t o r s r e ma i n t h e s a m e . T h e
a g g r e g a t i o n c a n h a pp e n r e m o t e l y . W e d o n o t h a v e t o d o t h i s m a n u a l ly .

D u e t o t h i s , w e c a n ge t m e t r i c a l N e t f l o w a l e r t s b a se d on t h r ou g h p u t c r it e r i a .
S u c h c r i t e r i a c a n a l so b e b a se d on t h e o pt i o n s f o r a n om a l y d e t e c t i o n , t h e
S u m o se r v i c e o f f e r s . O r o n s i m p l e t h in g s l i k e a v e r a ge a m ou n t o f i n c o m i n g
b y t e s p e r m i n u t e , a mo u n t of c on n e c t i o n s e t c … I t ’ s on t h e n e t wo r k o pe r a t o r s
t o d e f i n e t h e s e , mo s t l y . B a s e d on t h e s t a t i s t i c a l a n a l y s i s of t h e N e t f l o w d a t a ,
a n d e x pe r i e n c e v a lu e s .

H e r e 1 2 i s w h a t I b a s e d m y w o r k f l o w on .

#!/usr/bin/python2
from silk import *
import json
import datetime

def parse_all():
ffile = '/tmp/test.rwf'
flow = SilkFile(ffile,READ)
d = {}

for rec in flow:

d['stime'] = rec.stime.strftime("%Y-%m-%d
%H:%M:%S")
d['icmpcode'] = rec.icmpcode
d['sip'] = str(rec.sip)
d['protocol'] = rec.protocol
d['output'] = rec.output
d['packets'] = rec.packets
d['bytes'] = rec.bytes
d['application'] = rec.application
d['sensor_id'] = rec.sensor_id
d['etime'] = rec.etime.strftime("%Y-%m-%d
%H:%M:%S")
d['classtype_id'] = rec.classtype_id
d['nhip'] = str(rec.nhip)
d['input'] = rec.input
d['icmptype'] = rec.icmptype
d['dip'] = str(rec.dip)
d['sport'] = rec.sport
d['dport'] = rec.dport
print json.dumps(d)
 # print "\n"

def main():
parse_all()

if __name__ == "__main__":
main()

N o w y ou s e e w h y w e n e e d t h a t P y t h o n s u pp o r t i n S i L K . H e r e ’ s w h a t y ou d o
for a cron job:

1. r u n rwfilter, l i k e
t h i s : /usr/local/bin/rwfilter --type=all
--start-date=$(date -u +%Y/%m/%d:%H) --
sensor=local --proto=0-255 --
pass=stdout > /tmp/test.rwf . Y o u c an a d d
a :%M:%S i f yo u f a m il i a r i z e yo u r s e lf
w i t h sTime a nd eTime. A n d t he f l o w i n t e rv a l s .
M o re o n t h at l a t e r.
2. r u n t h e P y t ho n c o d e l i k e t h i s p yt h o n
n e t f l o w _ j so n . p y ( b e c a u s e I h a rd - c o d e d t he
p a t h)

/ u s r / l o c a l / b i n/ r w f i l t e r - - t yp e = a l l - - s t a rt -
d a t e = $ ( d a t e - u + % Y / % m / % d : % H ) - - s e n so r= l o c a l
- - p r o t o = 0 - 2 5 5 - - p a ss = s t d o u t > / t mp / t e s t . r w f
/ u s r / b i n / p y t ho n / o p t / s c r ip t s/ n e t f lo w _ j s o n . p y >
/ t mp / ne t f lo w . j so n
r m / t m p / t e s t . rw f

I r u n t h i s o n c e e v e r y h ou r f o r a st a r t . Y ou a l s o n e e d t o d o s o me t h i n g a b o u t
t h e l a s t b u c k e t o f d a t a , w h i c h i s n ’ t f u l l y c o l l e c t e d y e t . An d … a n d … a n d … y e s .

F o r t h e crontab:
58 * * * * /opt/scripts/netflow.sh >/dev/nul l 2>&1

SiLK to JSON to Sumo - for Netflow aided

security monitoring and data correlation
 • The 1h timeframe is not optimal.
• T h e date B a s h h a c k w i t h t h e m i n u t e i n cron i s

ridiculous. What if it takes longer? Then there

will be no records in the file for 1h.

I w a n t t o se e i f t h i s d a t a i s a s u se f u l a s S u r i c a t a ’ s f l o w i n J SO N . S u r i c a t a ’ s
f l o w r e c or d s h a v e a flow id w h i c h I c a n m a t c h w i t h a n alert
id. E V E b o x 1 8 w o r k s t h i s w a y . T h i s wa y I c a n q u a n t i f y t h e n e t w o r k t r a f f i c
a s s o c i a t e d w i t h t h e s e c u r i t y a l e r t . I n c a se of a le r t s , wh i c h a r e a s s o c i a t e d
w i t h M a l w a r e , t h i s i s u s e f u l i n f o r m a t i on . I t a n s we r s t h e u su a l q u e s t i o n : c a n
y o u a s s o c i a t e a b r e a c h w i t h t h i s ? T o r e p l y t o s u c h a n a n a ly s i s t a sk , y ou
can gr ab th e C ar bon B lack n e tco n e ven ts and sum up the network traffic
t h e s e c a u s e d w it h t h e i n f e c t e d c l i e n t . T h e m or e d a t a i t i s , t h e m o r e l i k e l y …

meh.

This is just a PoC

L e t ’ s p i p e t h e J S O N d a t a in t o Su m o t h i s w a y f or n o w .

Pasted image1417×841 64.3 KB

ToDo

• c r e a t e a n R F C c o m p l i a n t timestamp f o r t h e

Netflow JSON serializer

• find a way to run this every minute

( rwfilter q u e r i e s n e e d t o b e c o m p o s e d w h i c h

take activity timeframes as well

a s sTime a n d eTime) . A timestamp f i e l d s h o u l d

make sense for the data here.

• compare if Suricata’s EVE.json flow records are

more useful than the serialized SiLK records

T h e pr o b l e m w i t h N e t f l o w h e r e i s , t h a t t h is i s n ’ t
j u s t kernel.capture.stats d a t a . T h e se f l o w s ma r k d i r e c t e d
c o m mu n i c a t i o n . T h e r e f or e t h e y d on ’ t g e t r e p o r t e d w i t h t h e i r f i n a l v a lu e s a t
t h e t i me of t h e rwfilter q u e r y . I f t h e c on n e c t i o n i s s t i l l a c t i v e , t h e a m ou n t
o f b y t e s b e t w e e n s ou r c e a n d d e st i n a t i o n s t i l l g r o w s .

Results

• We can setup a modern Netflow collector based

on OpenSource tools

• We can build an analysis VM which is able to

generate Netflow data from mirror traffic

• We can install a nice Netflow dashboard SiLK

web interface

• We can serialize Netflow data with Python, and

send the data into a Log Management tool (WIP)

• We can provide infrastructure to speed up

Incident Response for network focused attacks

W h a t I r e a l l y w a n t t o d o i s g r ou p in g n e t wo r k s a n d h o s t s by a c t i v i t y u s in g
H i d d e n M a r k ov M od e l s . T h i s sh ou l d be a g r e a t a p p l i c a t i on f o r M a c h in e
Learning.
 • Pf-dup on a PfSense gateway for separate
distributed IDS sensors

• creat ed

Sep '16

• 2

replies

• 11

link s

• 2

F o r r e f e r e n c e : i n c a se y o u w a n t t o r u n F l ow B A T o n s u p e r v i s o r 2 0

[program:FlowBat]
command=/bin/bash -c "/home/netflow/FlowBAT/bin/run"
directory=/home/netflow/FlowBAT
autorestart=true
environment=
HOME=/home/netflow
stderr_logfile=syslog
stdout_logfile=syslog

• Netflow data analysis with SiLK and

Pandas11

5 YEARS LATER

C L O SE D O N A P R 2 9 , ' 2 1
Reply