APsera Tech Inc.

Response@The Speed of Thought

Traffic Baselining Case Studies

June 23, 2008

Sampath Prakash
Jeff Jaggernauth
Case Studies Involving ACE Live

 4 Case studies from the past 1-3 years

– TWE (an Entertainment company)
– MBNE (non-profit educational)
– RBI Inc. (legal firm)
– E-Catalogue (European/US retailer)

 For each customer-

– What was the business problem/objectives/motivation?
– What was the network environment?
– What were the key findings from ACE Live analysis?
– What recommendations were given?

Proprietary- APsera Tech Inc. 2

 Objectives- TWE

 AS 400s in LA and Phoenix were consolidated and relocated into

Buffalo, NY
 WTE engagement objectives were to:
– Baseline the current traffic flows into and out of the AS400s servers at LA and
phoenix locations over a period of 7 days.
– Size the network logical and physical circuits to accommodate the traffic when
the AS400 servers are relocated/consolidated to Buffalo.
– Report other observations made during the traffic collection and analysis.

 Back to Index

Proprietary- APsera Tech Inc. 3

 Appliance Connectivity at LA and Phoenix

INTERNET

Management Station
IMA 2 T1 AT&T WAN IMA 2 T1

NJ
AUI SERIAL 0 SERIAL 1 CON AUX AUI SERIAL 0 SERIAL 1 CON AUX
ASYNC 1-8 ASYNC 1-8

2509 2509

Catalyst 2948G-L3
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 1000 Base - X
CONSOLE
49

50
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47

STATUS AUX
PSI RPSU 49 5 0
1 0/ 100/100 ETHERNET
LAYER 3 SWITCH

Procurve
hp procurve
switch 2524
J4813A
25 26 1

7
2

8
3 4 5 6 13 14 15 16 17 18 Link

9 10 11 12 19 20 21 22 23 24 Link
Mode

Mode
25
Tr an cei ver Po rts
26
1 2 3 4 5 6 13 14 15 16 17 18
Cisco 3524 XL Cisco 3500XL
CATALYST 3550
P ower

Ac t FD x M ax !
1 3 5 7 9 11 13 15 17 19 21 23
Fault

Co nsole
R ese t C lea r Sel f
Te st
Fa n
Fa ul t
LED M od e Se le ct
7 8 9 10 11 12 19 20 21 22 23 24 1 3 5 7 9 11 13 15 17 19 21 23 CATALYST 3550
1 2
1 2 SYSTEM
RPS
SYSTEM
RPS
STAT
UTIL
STAT DUPLEX
UTIL SPEED 2 4 6 8 10 12 14 16 18 20 22 24
DUPLEX
SPEED
2 4 6 8 10 12 14 16 18 20 22 24

Procurve
hp procurve 25 26 1 2 3 4 5 6 13 14 15 16 17 18 Link 1 2 3 4 5 6 13 14 15 16 17 18
Tra nce ive r Po rts
switch 2524 Mode 25 26
J4813A
7 8 9 10 11 12 19 20 21 22 23 24 Link
Mode
Pow er

A ct FDx Ma x !
R ese t Cle ar Se lf Fan
F ault
L ED Mo de Sel ec t
Console Tes t Fau lt 7 8 9 10 11 12 19 20 21 22 23 24

1 3 5 7 9 11 13 15 17 19 21 23 CATALYST 3550
1 2
hp procurve 25 26 1 2 3 4 5 6 13 14 15 16 17 18 Lin k Tra nc eive r Po rt s
1 2 3 4 5 6 13 14 15 16 17 18 SYS TE M
switch 2524 RPS
Mod e 25 26
J4813A
7 8 9 10 11 12 19 20 21 22 23 24 Lin k S TAT
Mod e UTIL
DUPL EX
Pow er
SP EE D
2 4 6 8 10 12 14 16 18 20 22 24
Ac t FDx M ax !
Reset Clear Self Fan LED Mo de Select
Console
Fault

T est F a ult
7 8 9 10 11 12 19 20 21 22 23 24

1 3 5 7 9 11 13 15 17 19 21 23 CATALYST 3550
1 2
hp procurve 25 26 1 2 3 4 5 6 13 14 15 16 17 18 Lin k 1 2 3 4 5 6 13 14 15 16 17 18 SYSTEM
T rance ive r Po rts
RPS
switch 2524 Mod e 25 26
J4813 A
7 8 9 10 11 12 19 20 21 22 23 24 Lin k S TAT
Mod e UTI L
DUPLEX
Power
SPE ED
2 4 6 8 10 12 14 16 18 20 22 24
Ac t F Dx Max !
Reset Cle ar Self Fa n
L ED M od e Select
Fault

Console T est F ault

7 8 9 10 11 12 19 20 21 22 23 24

1 3 5 7 9 11 13 15 17 19 21 23 CATALYST 3550
h p procurve 25 26 1 2 3 4 5 6 13 14 15 16 17 18 L in k 1 2 3 4 5 6 13 14 15 16 17 18 1 2
Tr ance iver Por ts
switch 2524 M ode S YSTEM
25 26 RPS
J4813A
7 8 9 10 11 12 19 20 21 22 23 24 L in k
M ode ST AT
UTI L
Power

DUP LEX
SPE ED
Act FDx Max ! 2 4 6 8 10 12 14 16 18 20 22 24
Re set Clear Self F an
Fault
L ED Mod e Se lec t
Console Te st Fau lt 7 8 9 10 11 12 19 20 21 22 23 24

AS 400s Monitor Monitor AS 400s

AS 400s
Management Management
NPS 2000 = Traffic
NP2000 Port Port NP2000
Monitoring Appliance
USERS USERS
CANTON CARSON
Los Angeles Phoenix

Proprietary- APsera Tech Inc. 4

WAN Utilization – Current and with AS400s in Buffalo
Buffalo LA Phoenix
% of 6 % of 3 % of 3
Utilization (Average Hourly) / Sites Mbps Mbps Mbps

Current WAN Traffic

Current Inbound (eliminated AS400s WAN current traffic) 60 (1) 15 40
Current Outbound (eliminated AS400s current WAN
traffic) 30 (1) 10 20
       

AS400s Local
AS400s Local Inbound   10  5
AS400s Local Outbound   15 16 
       

AS400s In Buffalo

New Inbound Totals 68 (4) 30 (2) 56 (2)

New Outbound Totals 46 (5) 20 (3) 25 (3)
Note (1) LA and Phoenix contributions are small % of the
current WAN traffic - periodic bursts      

Note (2) %= current inbound + AS400s outbound      

Note (3) %= current outbound + AS400 inbound      

Note (4) %= Current inbound + (LA + Phoenix (inbound))/2      

Note (5) = %Current outbound +( LA + Phoenix (outbound))/2      

Proprietary- APsera Tech Inc. 5

 Recommendations

 No need for an upgrade at LA

– LA WAN inbound and outbound net traffic utilization would increase to 30% and 20%
from the current 15% and 10%, respectively
 No need for an upgrade at Phoenix
– Phoenix WAN inbound and outbound net traffic utilization would increase to 50%
and 25% from the current 40% and 20%, respectively
 Upgrade Buffalo bandwidth
– New bandwidth with the AS400 servers in Buffalo is close to the 70% threshold
criteria for upgrade
– Observed sustained bursting above 80 % utilization for more than 20% of the time
during the business hours - 5 hours in one day of the week (Friday)
– To accommodate additional traffic associated with new applications hosted in
Buffalo in the future

Proprietary- APsera Tech Inc. 6

 Objectives- MBNE

 MBNE engagement objectives are to:

– Baseline the current traffic flows into and out of Fairfax, Virginia office to and
from the Internet over a period of 7 days.
– Break-up of the traffic flow by protocols (TCP, UDP, and ICMP)
– Identify the top 20 applications based on bandwidth usage
– Identify the top 20 users based on bandwidth usage
– Identify URL’s visited
– Report on
» Server response times (TCP level) as applicable
» Failed TCP connections
» Packet loss
– Report other observations made during the traffic collection and analysis.

 Back to Index

Proprietary- APsera Tech Inc. 7

MBNE Environment Under Study

Checkpoint Firewall
T1 INTERNET

Management Station
Hub

Procurve
Powe r

Fau lt
hp procurve
switch 2524
J4813A

Console
25 26 1

Re set
7
2 3

Cle ar
4

Se lf
Test
5 6 13 14 15 16 17 18 Link

8 9 10 11 12 19 20 21 22 23 24 Link

Fan
Fau lt
A ct FDx Ma x

L ED Mo de Sel ec t
!
Mode

Mode
25
Tra nce ive r Po rts
26
1

7
2

8
3

9
4

10
5

11
6

12
13

19
14

20
15

21
16

22
17

23
18

24
NJ
Management
Monitor Port

NPS 2000 = Traffic

NP2000 hp procurve 25 26 1 2 3 4 5 6 13 14 15 16 17 18 Lin k T ranceive rPo rts
1 2 3 4 5 6 13 14 15 16 17 18
switch 2524 Mode 25 26
J4813 A
7 8 9 10 11 12 19 20 21 22 23 24 Lin k
Mode

Monitoring Appliance
Power

Ac t F Dx Max !
Re set Clea r Self F an
L ED M od e Se lec t
Fault

Console Test F au lt 7 8 9 10 11 12 19 20 21 22 23 24

hp procurve 25 26 1 2 3 4 5 6 13 14 15 16 17 18 Lin k 1 2 3 4 5 6 13 14 15 16 17 18
Tran ce iver Ports
switch 2524 Mode 25 26
J4813 A
7 8 9 10 11 12 19 20 21 22 23 24 Lin k
Mode
Power

Act FDx M ax !
Re set Clear Self Fa n
L ED Mod e Selec t
Fault

Console Te st Fau lt 7 8 9 10 11 12 19 20 21 22 23 24

h p procurve 25 26 1 2 3 4 5 6 13 14 15 16 17 18 L in k Tr ance iver Po rts

1 2 3 4 5 6 13 14 15 16 17 18
switch 2524 M ode 25 26
J4813A
7 8 9 10 11 12 19 20 21 22 23 24 L in k
M ode
P ower

Act FDx Ma x !
Res et Clea r Self F an
Fa ult
L ED Mod e Se lec t
Cons ole T e st Fau lt
7 8 9 10 11 12 19 20 21 22 23 24

Corporate LAN

Servers `

Fairfax, VA

Proprietary- APsera Tech Inc. 8

 Summary of Network Analysis and Baseline

 Overall traffic:
– Inbound traffic is significantly more than the outbound traffic, on the average it is 3 X
more
– A few users contribute to most of the traffic
– Very little UDP and ICMP traffic
– TCP is 98% of the traffic with contributions from
» HTTP (55.3%+), HTTPS (5.9%), SMTP (16.4%), RTSP (11%) and FTP (6%) (full day basis)
» HTTP traffic is 60%+ during business day and 70%+ during busy hour
» During non-business hours SMTP is 57% and FTP is 24%
– Mail (SMTP) and FTP traffic are present throughout the day

Proprietary- APsera Tech Inc. 9

Traffic – Inbound & Outbound (Weekly and Full Day)

Proprietary- APsera Tech Inc. 10

IP 10.30.33.11 Traffic during the business day

Proprietary- APsera Tech Inc. 11

 Recommendations

 No immediate need for bandwidth upgrade

– Observed traffic can be reduced significantly by limiting the access of the high
bandwidth users to certain sites
 Review usage pattern of the following high bandwidth users
– 10.30.33.11 and a few others

 Implement access list on firewall to block traffic to non-work

related sites:
– Mofile
– Sirius and XM radio

 Proactive monitoring of traffic on a regular basis for minimizing

abuse

Proprietary- APsera Tech Inc. 12

 Objectives- RBI Inc.
 Background
– RBI Inc. had gone through a merger with the 2 sites in the same city of Wilmington, DE
– Wilmington1 site was soon to be consolidated into a single combined facility at Wilmington2
– RBI Inc. wanted to understand traffic flows in and out of these two sites in order to provision
adequate bandwidth capacity for the combined facility
 Engagement objectives
– Baseline the current WAN and Internet traffic flows/volumes into and out of Wilmington1 and
WIlmington2 offices over a 3-day period within a business week.
– Break-up of the traffic flow by protocols (TCP, UDP, and ICMP)
– Identify the top 10 applications based on bandwidth usage
– Identify the top 10 users based on bandwidth usage
– Report on observed
» Packet loss
» Failed TCP connections
» Response times (TCP level) as applicable
– Report relevant observations made during the traffic collection and analysis.

 Back to Index

Proprietary- APsera Tech Inc. 13

 Wilmington- 1 and 2, DE, Environment Under Study
DC
Pittsburgh
Firm 2Locations
Buchanan sites Raleigh,
Cat 3548 NC

..
AT&T Firm
Klett 1 sites
Locations
EVPN
WIL
NEW HAR
..
3 T1
MCI INTERNET
Frame Relay
Management Station

C2621

Monitor Hub Hub

NP2000 Management
Port C2621 Monitor C2621

NP2000
Management
Port

USERS

USERS
NP 2000 = Traffic
Monitoring Appliance

2 Logan Square
Wilmington2
1835 Market Street Wilmington1

Proprietary- APsera Tech Inc. 14

Top 10 applications Consumption of Bandwidth- Business
Hours at Wilmington2

Proprietary- APsera Tech Inc. 15

Wilmington 1 and 2 Consolidated Traffic Estimates

  Sites/ Utilization (KBPS)

  Wilmington1   Wilmington2
  WAN INTERNET TOTAL WAN INTERNET TOTAL
Inbound 159 311 470 376 442 818

Outbound
230 1428 1658 500 62 562

After Consolidation
Inbound N/A N/A   535 753 1288

Outbound
N/A N/A   730 1490 2220

Proprietary- APsera Tech Inc. 16

 Summary/Recommendations
 Adequate WAN bandwidth at Wilmington2 for current traffic in both
directions
 MCI FR link at Wilmington1 is underutilized in both directions
 UUNET Internet Link at Wilmington1 is highly utilized in outgoing
direction and lightly utilized in incoming direction
 Existing 3Mbs ATM link at Wilmington2 can initially handle “as is”
consolidated traffic
– Outgoing link expected to have an average utilization of approximately 69%
– Incoming link expected to an average utilization of approximately 35% - moderately utilized
 Baseline traffic at Wilmington2 again after consolidation
 Incidental recommendations
– Classify Citrix traffic into high data class if response time is an issue for Citrix enabled
applications
– Conduct Addressing cleanup to ease network management
– Review routing at each site to ensure that traffic is appropriately directed
– Cabling connections to the switches and routers have become unruly and could be cleaned
up/tagged during consolidation to ease troubleshooting and monitoring.

Proprietary- APsera Tech Inc. 17

Objectives- E-Catalogue
 Background
– E-catalogue wanted to understand the traffic utilization pattern on their Trans-Atlantic DS3
circuit during the year-end holiday season and the immediate period that follows
» Specifically in January 2 nd week, 2008 which has a special sales promotion day in Holland

 Engagement objectives-
– Find answers to:
» How does the Baseline of the current WAN traffic flows/volumes in and out of NY Data Center over a 6-
week period look like?
» How does it change on/before/after January 8-9 period?
» What is the break-up of traffic flows among specific business groups?
» What is the Break-up of the traffic flow by protocols (TCP, UDP, and ICMP)?
» What are the Latencies/Roundtrip times across NY Data center and European destinations?
» What are the top 10 applications based on bandwidth usage?
» Are there any observed Packet losses and Failed TCP connections?
» Are there any other relevant observations made during the traffic collection and analysis period?

Proprietary- APsera Tech Inc. 18

E-Catalogue Network Environment Under Study
Cat 3548

European locations
Key Business Groups: PRODSRV
DEVSRV
EU-AS400
AMSSRV

AT&T DS3 Link

ECRT001
3725

Spanned Capture port

2970 ECSW001
NP2000 Management C
Monitoring Port
Appliance
(New name:
OPNET ACE
Live)
NY Data center
Servers and LAN
Internet Key Business Groups: WEB Servers
BACKEND Servers

AT&T Remote Monitoring PC, NJ-

Web access to NP2000

Proprietary- APsera Tech Inc. 19

 Hourly Utilization during the 6-week period

 There is one 1-hour peak 7 Mb/s (load testing night), one 1-hour peak- 9 Mb/s
(on sales day) and several 1-hour peaks- 4 Mb/s in the 6-week study period
 Average hourly utilization during busy hours is under 3 Mb/s

Proprietary- APsera Tech Inc. 20

Server Response Times during the sales Midnight to Midnight

DEVSRV

AMSSRV

EU-AS400

PRODSRV

Proprietary- APsera Tech Inc. 21

Conclusion/Recommendations

 The Trans-Atlantic DS3 link is low to moderately utilized even

considering the annual sales event (with a 11 Mb/s 5 minute-
peak)
– 3 Mb/s average use on 45 Mb/s circuit with sales day jump to 9Mb/s
– 5-min bursts of 8 Mb/s seem to occur on an average once a day
– A private VPN type of arrangement can be considered
– Or a Bandwidth downgrade is an option

 The link has very low packet losses

 No issues relating to round-trip times, response times, and
failed TCP connections observed during study period
 The key applications- TCP ports 1555 and 1556 and HTTP- are
performing well

Proprietary- APsera Tech Inc. 22