Reviewing ESM components

HP ArcSight Partners Proof of Concept Boot Training

Technical Day-1
Philippe JOUVELLIER- HP ESP | Global Partner Enablement
philippe.jouvellier@hpe.com

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
ESM Main Components
ArcSight Command Center https://<esmserver>:8443 ESM Console Linux, Windows, Mac

ESM USER ROLES

CONTENT & PACKAGES  Dashboards

OPTIONS IdentityView
 Cases
 Active Channels  Data Monitors Application View
 Filters  Active Lists Risk InSight
 Rules  Session Lists Compliance InSight Packages
 Use Cases  IdentityView (50) ThreatDetector
 Reports  Field Sets, … Add-ons (GB, Assets, Devices, …)

PROPERTIES……………………………<ARCSIGHT_HOME>/config/server.properties
CONFIGURATION & LOG FILES LOGS………………………………………<ARCSIGHT_HOME>/logs/default/

Manager with Services and SERVICE NAME DESCRIPTION

ESM SERVER a process executor

execprocsvc
arcsight_web ArcSight Web service

logger_httpd Logger Apache httpd service

logger_servers Logger service

Manager logger_web Logger Web service
Internal CORR-engine
Connector manager ESM Manager service

mysqld MySQL database

prostgresql PostgreSQL database

2 HA Option – IP Cluster
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
iDPU
ESM Resources

Active Actors
Users Cases
Channels Asset Connectors
s
Stages

Customers
Search
Filters
Dashboards
Saved
Searches Files
ESM Manager
Rules
Filters

Reports Integration
Commands
Knowledge
Query Pattern Base
Viewers Discovery
Lists
Notifications
3 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
ESM 6.8c Overview
ESM High Availability (HA)
Active Channel in Web UI (ACC)
Query Speed Improvements with Bloom Filters*
ESM Web Service APIs Support
CFC Connector Capabilities
Correlation Enhancements
Larger Storage Capacity (12 TB)
Transition to Java 7
Support for RHEL and CentOS 6.5
Software Upgrades from ESM 6.0c Patch3 and ESM 6.5c SP1

4 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 ArcSight Default Content Reputation
IdentityView Threat FISMA v5.0 IT Governance Sarbanes-
Resource * ESM v6.5c Express v4.0 Security HIPAA v2.0 JSOX v4.0 NERC v1.0 PCI v4.0
v2.5 Detector v2.0 SP1 v4.0 SP2 Oxley v4.0
Monitor v1.5
Active Channel 97 102 5 35 0 51 44 45 50 42 1 50
Active List 46 48 22 24 0 30 18 26 20 18 39 20
Asset 0 0 0 32 0 0 0 0 0 0 0 0
Dashboard 130 131 14 39 1 85 50 72 64 54 22 64
Data Monitor 322 313 24 96 1 252 130 228 200 160 2 200
Field Set 63 65 7 23 0 14 11 13 15 12 0 15
File 3 5 0 0 0 4 6 4 10 6 0 10
Filter 514 523 69 222 4 255 126 220 167 138 134 168
Integration Command 32 33 5 8 0 0 0 0 0 0 0 0
Integration Configuration 14 15 2 3 0 0 0 0 0 0 0 0
Integration Target 3 3 1 3 0 0 0 0 0 0 0 0
Profile 2 2 4 24 5 0 0 0 0 0 0 0
Query 792 772 131 266 2 492 194 359 210 198 86 212
Query Viewer 172 158 59 74 1 99 0 60 0 0 51 0
Report 558 551 22 162 0 312 195 259 204 197 37 204
Focused Report 88 91 0 0 0 10 7 9 7 7 0 7
Rule 150 179 29 65 0 81 65 75 73 65 73 74
Session List 16 27 0 2 0 2 0 1 1 0 0 1
Trend 70 68 10 8 1 26 0 15 36 0 10 39
Use Case 32 54 9 7 1 73 0 66 0 0 10 0
TOTAL 3 104 3 140 413 1 093 16 1 786 846 1 452 1 057 897 465 1 064
* Not all Resources included in these counts.
IT Governance Sarbanes-
Logger v5.5 PCI v3.01
v4.01 Oxley v4.0
Alerts 0 26 62 0
Reports 147 316 92 129
Filters 96 0 0 0
Total5 © Copyright 2015 Hewlett-Packard
243 Development
342 Company, L.P.
154The information129
contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
ArcSight Event Schema
Event Schema – 400+ columns
• Event data collected, normalized, enhanced for monitoring and mining

Event Lifecycle – schema processing in 7 phases

1. Data Collection and initial schema population
• Acquisition, Filtering, Normalization, and Aggregation of Event Data
• Apply Event Categories
• Apply Customer and Zone from Network Model
2. Network Model Lookup and Priority Evaluation Phase
3. Correlation Evaluation
4. Monitoring and Investigation
5. Workflow
6. Incident Analysis and Reporting
7. Storage and Archive
6 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
CORR Engine
CORR-engine … Deep Dive in Hybrid Storage
All queries, whether related to resources or events, are initially processed by MySQL
MySQL

ARC_LOGGER
plugin engine for MySQL

All events are stored (1) All resources are stored in

EVENTS STORE ESM RESOURCES STORE
in flat files, with the Compressed Flat Files InnoDB engine InnoDB(3) allowing updates
contents indexed by /opt/arcsight/logger/data/logger
and deletes
PostgreSQL(2)

(1) Event storage is arranged by columns rather than rows. This means that the contents of all the name fields are stored together, the same with messages, end
times, etc. All Fields are indexed.

Archive partitions are arranged by the Manager Receipt Time (MRT).

(2) PostgreSQL is an object-relational database management system (ORDBMS). (3) InnoDB is a storage engine for MySQL

8 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 CORR-engine event storage Events flow into the active retention period, and once a day,
CORR-e (e.g. at midnight) events are copied into the archives
DAY
TODAY Events Feed
TODAY
>>>> archiving DAY-7 data @ 00:00 on DAY-8h >>>>>
Yesterday All events time DAY-7
stamped(1) for a DAY-7 >>>> archiving DAY-6 data @ 00:00 on DAY-7 >>>>>
2 days back
particular day DAY-6
DAY-6
e.g. (12:00:00 a.m. to >>>> archiving DAY-5 data @ 00:00 on DAY-6 >>>>>
3 days back
DAY-5 DAY-5 Active
retention 11:59:59 p.m.) are
TIME LINE

>>>> archiving DAY-4 data @ 00:00 on DAY-5 >>>>>

period set grouped together. DAY-4 Jobs
for 8 days
4 days back
DAY-4
(1)Manager
>>>> archiving DAY-3 data @ 00:00 on DAY-4 >>>>>
Receipt DAY-3
5 days back
Time (MRT)
DAY-3 >>>> archiving DAY-2 data @ 00:00 on DAY-3 >>>>>
6 days back DAY-2 DAY-2
>>>> archiving DAY-1 data @ 00:00 on DAY-2 >>>>>
7 days back DAY-1 DAY-1

Events portion of the CORRe storage management system April 30

consists of two major parts: April 29

1. the active jobs Deactivated archive April 28 Archives

2. the archives
ActivatedHP
archive April 27
9 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Restricted. For HP and Partner Internal Use

April 26
Events Storage and Archiving
 CORR-Engine operates on events available in the active retention period (Active “jobs”), and any
offline archives that have been activated.

 When a days’ worth of events reaches the end of the retention period, they drop off of the
retention period’s memory, although their corresponding archive copy is retained indefinitely in
the archives.

 As events flow into ESM, they receive a time stamp at Manager Receipt Time (MRT).

 Retention period is the policy one set for how long to retain events in active memory for
correlation, for example, 30, 60, or 90 days.

 CORR-Engine content (filters, rules, queries, query viewers, active channels, and data monitors)
evaluates the event data stored in this area (Active “jobs”).

10 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
HP ArcSight ESM Console

ArcSight Console is available as:

-Web UI (a.k.a. ArcSight Command Center)
-Java Client
HP ArcSight Web Console

a.k.a. Arcsight Command Center

New User Interface improves Administration
One User Interface to manage all ArcSight Products

13 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
ArcSight Command Center (ACC) – Administration

14 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Logger like GUI in ESM Console

15 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
HP ArcSight Java Console
ArcSight Java Console

!
Console version must be the same as ESM server

Manager name + IP address must be existing in host file

17 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 Console menus

18 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 Inspect / Edit
! Navigator Panel
Accessing resources CCE - Common Condition Editor
Events details & resources editor

Viewer
Dispay events, charts, reports,
dashboards, active channels

19 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 Events Priority
Level
Color coded
(rated 0 to 10)

Lightning
Fired correlation rule

20 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Resource Navigation
User’ Content

Shared Content(1)

User’ Content

Shared Content(1)

21 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 Drill down correlated
alert by accessing
base events

22 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 Event’s
sequence

23 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 Drill down correlated
alert by accessing
trigger rule

24 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 Rule shows-up
in Inspect/Edit
panel

25 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 Integrated
Workflow
Case
Editor

26 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Architecture
Architecture Sizing: Information Gathering

What information do I need to gather to size architectures?

 Event Throughput Requirements (EPS / EPD)
 Event Type Requirements
 Log Retention Requirements
 High Availability / Fail Over Requirements*
 Additional Customer Requirements
Bandwidth Consideration
NAT Considerations
MSSP Considerations
Regional / Global Considerations
Compliance Requirements
Use Case Requirements
28 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
 ESM Manager
Typical ESM Architecture o Java based server
o Only component to communicate with ESM Data base
Workflow
CORR-e
User Interface
Dashboards
ArcSight
Rapports o Store events on a daily basis
Web
Notifications
o All Fields are indexed
ESM Actors ESM ESM Manager o Chronological slices (midnight to midnight)
Console
User Interfaces
ArcMC ArcSight CORR-engine o ESM Console (GUI) based application
Management Command
Center
o Arcsight Command Center (Web 2.0)
Server

Software Connectors On board Connector

SmartConnectors Hardware Appliance Model HP
Import
FlexConnectors Connector DVLabs

LOGS
SOURCES Security Cloud Application Network Physical Server CCTV
Database IDENTITY

29 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Protocols and Ports

30 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
ArcSight ESM Process Management
• Single script to manage all ArcSight services
–Control process dependence and startup sequence
–Restart failed service
• Unified control of all ArcSight services with /etc/init.d/arcsight_services <command>(1)
–/etc/init.d/arcsight_services help
–/etc/init.d/arcsight_services [start | stop | status | …] [all | manager | mysqld | …]

(1) Starting with SUSE Linux support in ESM 6.5c SP1 the path changed for all users, regardless of operating system

31 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Command line application check
/etc/init.d/arcsight_services status | start | stop
All services are running Manager is not running
web service is available web service is mixed_statuses
manager service is available manager service is unavailable
execprocsvc service is available execprocsvc service is unavailable
logger_httpd service is available logger_httpd service is unavailable
logger_logger service is available logger_logger service is unavailable
logger_web service is available logger_web service is unavailable
mysqld service is available mysql service is available
postgresql service is available postgresql service is unavailable

32 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
ESM Linux partition
• ESM software is installed in the /opt partition
 separate partition
 all ArcSight software & data under 1 single directory: /opt/arcsight
 For RHEL, the XFS and EXT4 file system formats are supported.
 For SUSE Linux, the EXT3 file system format is supported
• This directory is owned by user ‘arcsight’
• All arcsight operations should be run as ‘arcsight’, not ‘root’
• Event storage directory: /opt/arcsight/logger/data/logger
• Event archive directory: /opt/arcsight/logger/data/archives
 1 directory per day (don’t forget archiving logs entries)

33 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
A Primer to HA
HA Architecture explained Intranet

PRIMARY (host name esm)

PRIMARY SECONDARY
eth-0 eth-0
Interface configuration 16.103.74.24 16.103.74.224
PACEMAKER PACEMAKER
 16.103.74.24 Primary (eth0) “esm”
 192.168.145.24 (eth1) Service IP
(cluster
 16.103.74.23 cluster (service Ip/name) 16.103.74.23)

ESM
SECONDARY (host name esm1)
Interface configuration
 16.103.74.224 Secondary (eth0) “esm1” File System
 192.168.145.224 (eth1)
 16.103.74.23 cluster (service Ip/name) Interlink cable
Distributed Replicated Distributed Replicated
Block Device eth-1 eth-1 Block Device
192.168.145.24 192.168.145.224

The service ip/name

! address will be the shared
ESM address/hostname Disk 1 Disk 2

35 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
HA and iPDU (optional) Intranet

HP Intelligent Power Distribution Unit

HA Module uses the iPDU to disable one
PRIMARY SECONDARY
eth-0 eth-0
machine if both get into a mode where they 16.103.74.24 16.103.74.224
each think they are the primary. PACEMAKER PACEMAKER
Service IP
This ensures that the failover from one ESM (cluster
16.103.74.23)
to the other goes smoothly
ESM
ESM HA only supports the HP iPDU product
line.
File System

Pacemaker have STONITH iPDU agent that

sent command to power on/off, get info Interlink cable
Distributed Replicated Distributed Replicated
Block Device eth-1 eth-1 Block Device
192.168.145.24 192.168.145.224
iPDU is a server-room-class
power strip whose outlets
! may be turned on and off Disk 1 Disk 2
remotely

iPDU
36 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
STONITH (shoot the other node in the head)
HA architecture
Enabling technology for failover
• Needed when primary is crippled and will not release resources
– Communication problems – primary cannot receive stop request
– Software problems (e.g. out of memory or other resources)

• Ideally STONITH mechanism should be independent of primary hardware/software

– Power control like iPDU
– In some clusters cutting the server off from the network (I/O fencing) is used.

• Default SSH based fallback reboot control far from ideal.

– Will only work if SSH to server, reboot is possible.

37 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Fail Over Illustrated 1/2 Intranet

ESM IP cluster is up and running PRIMARY SECONDARY

eth-0 eth-0
16.103.74.24 16.103.74.224
Primary has :
PACEMAKER PACEMAKER
 Operating system running
 IP cluster pacemaker activated
Service IP
(cluster
 ESM application started 16.103.74.23)

 File system handling write operations onto ESM Pacemaker on Secondary

disk 1 ! detect Primary failure
 Disk 1 operating
 DRBD replicating data block from Disk 1 to File System
Disk 2 (disk level operation)

Interlink cable
Secondary has:
Distributed Replicated Distributed Replicated
 Operating System started Block Device Block Device
eth-1 eth-1
 IP cluster pacemaker activated and 192.168.145.24 192.168.145.224
monitoring Primary
 ESM application stopped
 DRBD handling disk level block replication
Disk 1 Disk 2

38 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Fail Over Illustrated 2/2 Intranet

ESM IP cluster is still up and FAILED HOST PRIMARY

running eth-0 eth-0
16.103.74.24 16.103.74.224
PACEMAKER
Primary has gone down for one of the
following reasons: Service IP
(cluster
 Operating system crashed 16.103.74.23)

 ESM application stopped/crashed ESM

 Hardware failure
 other
Secondary did the following: File System
 Detected Primary failure
 Took over IP cluster alias address Interlink cable
Distributed Replicated
 Started ESM application
eth-1 eth-1 Block Device
 Continued ESM operations 192.168.145.24 192.168.145.224
 DRBD disk level block trying replicating
data block with former Primary disk if still
operating and available
Disk 2

39 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Architecture: Key points

• ESM runs on the primary node.

• The secondary node is in standby mode.
• The data from the primary node is replicated and synced at the block level.
• The service ip/name address is the HA default address.
• Both machines have two interfaces - eth0 for connecting to the intranet and eth1
for disk mirroring.
• If there is a failure on the primary node, the HA on the secondary detects this
fact, and mounts /opt/arcsight, starts up ESM, and adds the service IP to eth0.
After a brief interruption, service is restored via the secondary .

40 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Managing ESM
Managing and Configuring ESM

To Start/Stop ArcSight Services

To Configure ESM Manager

./arcsight setupmanager

42 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Managing ESM license key

To install a license key

Copy the Zip file to the ESM

server

cd /opt/arcsight/manager/bin

./arcsight deploylicense

43 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
ESM Backup Overview
From Administrator’s Guide and ArcSight Command Center User’s Guide
1. Configuration Parameters Backup
• Configbackup
backs up certain essential configuration information such as search settings and the configuration of archives
(not the archives themselves) in configs.tar.gz in opt/arcsight/logger/current/arcsight/logger/tmp/configs
2. Database dump / import
• export_system_tables
Exports database tables by generating 2 files: a temporary parameter file and the actual database dump file,
arcsight_dump_system_tables.sql, which is placed in /opt/arcsight/manager/tmp.
• import_system_tables
Imported file must be the one that export_system_tables utility created
3. Event Data Archiving
• From ArcSight Command Center GUI
Storage and Archive Tab is used to activate and Schedule Archive Jobs
44 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
ArcSight ESM 6.8c Documentation
ESM Product Guides Standard Content Guides
• ESM 101 – Concepts for ArcSight ESM 6.8c • Cisco Monitoring
• Administrator’s Guide – ArcSight ESM 6.8c • Configuration Monitoring
• User’s Guides for • Intrusion Monitoring
• ArcSight Console • IPv6
• ArcSight Web • NetFlow Monitoring
• ArcSight Command Center • Network Monitoring
• Installation and Configuration Guide • ArcSight Core Security, ArcSight
• Release Notes - ArcSight ESM 6.8c Administration, and ArcSight System
• Upgrade Guide - ArcSight ESM • WorkFlow
User Interface Help Facilities
• Searchable Online Product Guides
• Context-sensitive Help

45 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Thank You

Questions ?

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use