Dashboards & Datamonitors

What are Data Monitors and where do you use them?

• ArcSight Data Monitors perform

the following actions:

-Evaluate the event stream and

system health statistics
-Consolidate events with common
elements
-Focus primarily on summarizing
event data graphically and can
provide different types of analysis

• Data Monitors can only be viewed

and visually manipulated (not
edited) from the Dashboard.
Data Monitors

• Data monitors collect summary information on various

data stored in the ArcSight Database; For example, Top
events, most recent event activity, partial rule
occurrences, hourly event counts, or event averages.

• Data Monitors evaluate the event stream, apply filters and

summarizes the things in a graphical format, which can
then be displayed in dashboard.

• Data Monitors consumes a lot of memory

Where do you use Data Monitors – Adding Data Monitors to
Dashboards

The following are the various ways in which you can display Data
Monitors within your Dashboard

– Bar Chart - shows data as a series of proportional bar elements

– Bar Chart Table - a grid of proportional bar elements
– Horizontal Bar Chart - shows data as a series of proportional bar
elements and may include bar segmentation to subdivide the data
– Pie Chart - shows data as a circle with proportional wedges for elements
– Statistics Chart - displays Moving Average data monitors, especially
those that contain and need to arrange multiple graphs in one
– Table - displays data as a grid
– 3D Bar Chart - shows data as a series of proportional bar elements and
may include bar segmentation to subdivide the data
– Tile - arranges individual Moving Average data graphs into separate,
fixed positions on a data monitor, when multiple graphs are present
What types of Data Monitors are available?

The three types of data monitors are:

• Event based data monitors

• Correlation data monitors
• Non-event based data monitors
Event-Based Data Monitors (1 of 2)

Used primarily as a monitoring and investigation tool during the

discovery phase of the correlation process.

There are eight event based data monitors available:

• Asset Category Count — enumerates the number of events that occur per
asset category (by priority within a time interval)
• Event Graph — draws a real time diagram of selected event activity
• Geographic Event Graph — draws a real time geographic map of selected
event activity
• Hierarchy Map — draws an image made up of proportionally sized panels
where each panel represents a group of events selected by group fields
selected in the source node identifier.
Event-Based Data Monitors (2 of 2)

Event Based Data Monitors (continued)

• Hourly Counts — displays the total count of events on an hourly basis

along with their Priority
• Last N Events — displays the most recent events by Priority, Event Name,
Protocol and Category
• Last State — translates complex values Into graphics to enable rapid
observation of results (red light, green light)
• Top Value Counts — displays top events by selected Data Field, total
number of events and the Event Severity
Correlation Data Monitors (1 of 2)

Are also event based and evaluate the event stream, but have the
capability to perform special analytic functions that rules alone
cannot. Work analytically in conjunction with rules.

There are five correlation data monitors available:

• Event Correlation — provides flow volume correlation between two

different event streams (corroborate attacks reported by different
systems)
• Event Reconciliation — correlates events arriving from one sensor with
events arriving from another sensor using a filter and matching fields
• Moving Average — displays the moving average of events
Correlation Data Monitors (2 of 2)

• Session Reconciliation — correlates events based on their occurrence

within a relevant time period
Often established by a session-based event
Used to monitor network devices with longer-term operations

• Statistics — enables you to select other statistical methods in addition to

moving average.
There are four statistical options available:
Average
Standard deviation
Skew
Kurtosis
Non-Event Based Data Monitors

Allows you to evaluate internal statistics associated with ArcSight

resources and their usage.
• Useful to Administrators to monitor the ArcSight system itself.

There are three non event data monitors available:

• System Monitor — provides measurements based on the Manager’s
internal systems Java classes and attributes
• System Monitor Attribute — focuses on a specific attribute of a given
ArcSight Java class
• Rules Partial Match — displays rules that have partial matches and the
total number of partial match events within a specified time frame
Configuring Data Monitors

You can create/edit Data Monitors in multiple ways:

• Data Monitors tab which resides under the Dashboards resource
• From your Dashboard

When creating a Data Monitor, use the Inspect/Edit panel and select the type
of Data Monitor you would like to configure.

• Poorly configured data monitors can lower manager performance: Restrict

who can edit or add data monitors
• Apply permissions to enable data monitors
• Don’t just copy data monitors:
• They can be re-used in multiple dashboards
• They can be displayed differently on the same dashboard
• Disable or remove unused data monitors
Data Monitors

Data monitor Types

Data Monitors
Parameters :-
Bucket Size in secs - Data Units shows the
group of events for the specified time
period.
Number of Buckets - Overall Time Period.
Calculated on need basis.
Example:- To evaluate the most recent hour.
So the Time interval is 1 hour
If Bucket Size is set to 60 secs (1 min)
Number of Buckets = (Time Interval / Bucket size)
= (60 min/1 min)
= 60
If Bucket Size is set to 300 secs (5 min)
Number of Buckets = 12 (60 min/5 min)

Availability Interval - Refresh Interval

Note:- It is wise to keep bucket size small
for the data monitor to perform faster and
effective
Dashboards
Dashboards
Reports
Report Definition

• In writing, a report is a document characterized by

information or other content reflective of inquiry or
investigation, which is tailored to the context of a given
situation and audience.

• The purpose of reports is usually to inform.

Reports

• The purpose of a report is to show information collected to

the reader about certain topics, usually to set targets or to
show a general view on the subject in hand.

• Another purpose is to discuss and analyze ideas and

thoughts on any problems or improvements to be made
and to inform the audience.
Reports

• XLS,PDF,HTML,CSV or RTF
Reports

Name
The name of the field appears as a column heading in the report unless you
specify an alias

Alias
An alternate name that replace the original field name as the column heading
in the report

COL (Column)
Decides the alignment of fields in the report i.e. which column come first

SORT ORD (Sort Order)

Specify which column you want sorted first, second, and so forth, in your
report.

SRT DIR (Sort Direction)

Decides the sort direction (ascending, descending, or none) for each column.
The "none“ option defaults to ascending.
Reports

SRT BY (Sort By)

Sort by data field values, COUNT (by the total number, for numeric values),
SUM (by total values), AVG (by average value), MAX (by maximum values),
or MIN (by minimum values).

GRP BY (Group By)

For grouping (aggregating) the items in the report. When you select a field to
use as a "group by," also choose a Function by which to evaluate the
grouping.

Function
When you select a field to use as a "group by" factor in a report, also choose
a function by which to evaluate the grouping. These are the same functions
described above for SRT BY.

PGE BRK (Page Break)

Select a field if you want page breaks to occur when there are changes in
that field's sorted content. You can select multiple fields.
Reports

Scheduled and
Archived Reports

Archived reports are
retrieved for immediate
viewing, without required
to rerun the report. In
addition, we can
schedule a report for
automatic archiving, on a
yearly, monthly, weekly,
daily, or hourly basis
Data Sources

• Events
• Active/Session Lists
• Notifications
• Cases
• Assets

• Query
or
• Trend Query
Trends

What is a Trend?
• A trend is an ArcSight resource that defines how and over
what time period data will be evaluated for trends.

Trend Characteristics:
• A trend is always based on a query
• The trend results are stored in a trend table in the
ArcSight Database, and are themselves query-able
• Trends can also be used as the primary data source for a
report
Trend Characteristics

• Give a long term perspective on events, vulnerabilities,

incidents, lists, workflow and other metrics
• Concepts for Trends:
• Store / Access self-sufficient data for a long time (months –
years)
• Provide a long term view of security changes
• Schedule regular, automatic execution of selected queries
to work on a fixed period of time
• Support high performance access to data for long time span
reporting
Note: Monthly reports run faster querying trend tables versus event data.
Trend Characteristics

Trend resource encapsulates query, scheduling, table

management, partitioning, etc.

Trend Types:
Interval
• Time range from X to Y
• Example: Events - Top 10 events for each hour
Snapshot
• No time range
• Example: Assets - Top 10 assets by vulnerability count
Trends - Lifecycle

• Create Query
Define Basic type (e.g. event, asset,G)
Select columns, grouping, sorting
• Create Trend
Choose query, duration, G
Choose columns (subset)
Setup schedule
• Create Query
Choose the trend
Select columns, grouping, sorting
• Create Report
Choose the trend-based query
Specify report settings
Trend – Performance Considerations

Consider Granularity / contents of trends

• Build trends using “umbrella” data reporting requirements

setting up a master trend system

• Example: Daily trends across multiple time-zones would

benefit from hourly accumulation
Could be rolled up for each time-zone midnight to midnight

• Example: if supporting multiple business divisions or separate

companies (MSSP), gather all the information in a single table
and run reports filtered by company or division
ArcSight Logger
Introduction to Logger

What is Logger ?

• ArcSight Logger is a high-volume storage appliance designed to

capture Enterprise Log Data from many sources and quickly
perform dynamic, distributed queries and searches across
terabytes of data
• Rack mountable appliance
• Web GUI and CLI (via serial or direct connection)
• Can interact with Storage Access Network (SAN) OR Network
Attached Storage (NAS)
• Function as a standalone appliance OR as part of ArcSight ESM
appliance and can be configured for single unit processing or
multiple Logger Networks
ArcSight Logger Benefits

Performance and scalability

o High volume data collection
o Linearly scalable distributed architecture (hierarchical or peer-to-peer)

Reduced cost of storage

o Compressed storage (Up to 10:1)

Minimal administrative overhead

o Hardened 2U appliance form factor
o No DB administration
o Web based GUI

Audit quality best practices

o Raw data collection
o Log data integrity measures
o Granular role based access controls
o Retention and suspension policies
ArcSight Logger Benefits

Distributed search capabilities

o Device independent, intuitive search taxonomy

High availability architecture

o Backup/restore of configuration data
o Searchable archival of log data to NAS/SAN

Bidirectional integration with ArcSight ESM and ArcSight Connectors

o Leveraging ArcSight Connectors expands device support significantly
o Connector appliances provide complete turnkey solution for Log
Management needs
Initialization of Logger

It is crucial that these resources be created in the order presented

here, and that you do not reboot the Logger unless you are certain of
your Storage Volume and Storage Group choices; these choices
cannot be changed after they are initially set and Logger is rebooted.

• Establishing the RFS Mount

 If you plan to use a remote file system configure here otherwise you
can skip this step
• Storage Volume
 Configure the Storage Volume
• Storage Groups
 Storage Groups support multiple retention policies by defining a
maximum size and number of days to retain events. Events older
than the maximum age specified will be ignored and eventually
overwritten. The oldest events will be overwritten in order to
maintain the specified maximum size of the Storage Group.
Storage Group
Logger Control Panel

• Logger Gauges
o Number of incoming events per second (EPS In)
o Number of outgoing events per second (EPS Out)
o Percentage of the CPU being used

• Monitor
o The Monitor tab displays the real-time and historical status of Receivers,
Forwarders, and Storage, CPU, and disk usage statistics.

• Analyze
o The Analyze tab contains the fields used to query Logger’s saved log
results.
Logger Control Panel

• Reporting
o The Reporting tab provides access to Logger reporting tools, including
functionality to run reports, view saved reports, and create new reports.

• Configuration
o The Configuration tab provides access to basic Logger functions for
setting up the Logger application environment, such as creating a
receiver or disabling an existing forwarder.

• System Admin
o The System Admin tab shows Logger’s system information, and
provides the interface to manage Logger users and network settings.
Setup of Logger
Users and Groups

• ArcSight Logger users are granted access rights and permissions

based on their member-ship in a user group.

• A user group is a set of users with the same set of permissions

• Four types of user groups

• System Admin
• Logger Rights (Basic User Operations)
• Logger Search
• Logger Report
User Groups
Managing Users
Configuring Logger Input and Output

Configuration tab provides options for defining how Logger interacts

with its data sources and data destination

•Receivers
•Forwarders
•ESM Destinations
•Peer Loggers
•Device and Device Groups
Receivers

• UDP Receiver (Syslog)

• TCP Receiver (Syslog)
• File Transfer (SCP, SFTP, FTP)
• File Receiver (requires RFS)
• SmartMessage Receiver
Forwarders
ESM Destinations
Peer Loggers
Devices
Device Groups
Monitor
Analyze
Analyze > Search

• No familiarity log syntaxes required

• Clean and structured viewing of logs
• Active results for quick drill down
Logger Control Panel

• Reporting
o The Reporting tab provides access to Logger reporting tools, including
functionality to run reports, view saved reports, and create new reports.

• Configuration
o The Configuration tab provides access to basic Logger functions for
setting up the Logger application environment, such as creating a
receiver or disabling an existing forwarder.

• System Admin
o The System Admin tab shows Logger’s system information, and
provides the interface to manage Logger users and network settings.