DPE, DPE TRAINING Click this icon to

add a picture
Notice

Please note that the screenshots in this document are provided for
illustrative purposes only and should not be treated as an exact
representation of the Graphical User Interface. It may differ depending on
the software version and installed components. Additionally, the user may
not have access to some functionalities or modules because of the lack of
appropriate privileges
 Agenda

1 Introduction

2 DPE ETL Data Collections

3 Complex Data Collection

4 Statistical Data Collection

5 Threshold

6 Quick View
01
Introduction
Next Generation Services Assurance
Introduction
Phase 1 Phase 2 Phase 3/4

Remedy Atrium CMDB

SOA

SQL

SQL
Comarch NGSA
Comarch OSS Performance Core Components
Management / Comarch SLA
Reference
Performance data
DB
Comarch Service Authentication Reporting System Repository
Service Service & Configuration
Monitoring / Comarch Service Inventory
Service Impact

Comarch Fault Management / Correlation & Known-Event

Processing Engine Database Comarch OSS Statistic Notification &
Event Management Console System Escalation Service

ActiveMQ JMS

Mediation Mediation Mediation Mediation Mediation Mediation Mediation

Device Device Device Device Device Device
Layer
SNMP
SNMP

Syslog
SNMP

SQL
Network
SNMP

Environment

Sensus TK Eltek PSU

z z z
1600E
Sensus TK
Sensus RNI 1600E
Cisco Router Sensus RNI
Network
887 DB
Controller Phase 2
Phase 1 Phase 3/4
Overview
 The Comarch Data Processing Engine(DPE) system is used to manage system
performance monitoring and plays a significant role as a source of network metrics
used to ensure efficient Data Factory Engine.

 The main goal of the DPE module is to collect system performance data with the use
of various protocols. Once collected, data may:
 be analysed,
 be used for the new KPIs calculatation.

 The system is also able to perform a statistical analysis of gathered data, e.g. Average
or Standard Deviation.
System Architecture
The Distributed Architecture consists of
the following components:

 Dispatcher – Broker,
 Multiple instances of Monitoring
Systems,
 Multiple instances of Data Collector
SQL Systems,
 Multiple instances of Adapter Data
Collectors,
 Data Base Layer,
 Mediation Layer.
System Architecture
DFE
Data Integration Layer:
Comarch OSS Console
Reporting Service

 Connects to DPES databases NGSA

Comarch OSS Web GUI

 Extracts performance data in real-time Event Forwarding

or on a regular schedule
SQM Engine

 Calculates KPI’s as statistical KPI Statistic KPI Complex KPI

functions of raw DPES data
Data Integration Layer (ETL Engine)

 Performs mappings of collected

values onto corresponding Managed SQL SNMP CLI File
Mediation
Collection Collection Collection Collection
Objects (resources, services) imported
from external systems (CMDB)
System Architecture
DFE
DFE Core Engine:
Comarch OSS Console
Reporting Service
 Validates thresholds based on
NGSA
complex conditions on different
KPI’s Comarch OSS Web GUI
Event Forwarding

 Triggers actions on threshold SQM Engine

violation events
KPI Statistic KPI Complex KPI

 Builds alarms, according to well- Data Integration Layer (ETL Engine)

defined enrichment rules

SQL SNMP CLI File
Mediation
 Forwards events to external
Collection Collection Collection Collection

systems (NGSA)
System Architecture
 File Data Source – a view definition that gives
File Data Source
an access to Oracle External Table from
performance data stored in plain flat files.
 Database Directory – a place where mediation
uploads files that will be loaded to the DPE
System. Database
 File Mediation – the Comarch Mediation
module that process files from External sources.
It gives a wide options to modify files and its
content. File Mediation is not a part of this File Mediation
training session.
 System Architecture
 Task Scheduler – a component used for task
management
 File Data Collector – an adapter that supports
creating Oracle External Tables from
performance data stored in plain flat files. The
module is responsible for building and
sending requests to SNMP Adapter.
 SNMP Adapter – an adapter that collects data
from devices supporting SNMP. The adapter
uses Management Information Base files
which describe hierarchies and contain object
identifiers (OID). SNMP Adapter gets specific
requests from File Data Collector.
02
DPE ETL DATA COLLECTION
DPE Objects
Objects To Be Created
After the Data Source and the KPI Groups are defined, the user needs to create a DPE
ETL Data Collection.

Data Source KPI Group

DPE ETL DC COMPLEX DC THRESHOLDS

STAT DC
DPE ETL Data Collection
Overview
The data for an ETL data collection is gathered with the use of ETL Processes.
An ETL process extracts data from one source system, based on a single SQL query that
is provided during the process definition.

ETL data collections support the following time aggregation periods and methods:

 Hour  Minimum
 Day  Maximum
 Week  Average
 Month  Count
 Quarter  Sum
 Year
DPE ETL Data Collection
Creating A DPE ETL Data Collection
1. Click the [New Object] 1
button.
2. Select the DPE ETL Data
Collection object type.
3. Click [Next].

3
DPE ETL Data Collection
Creating A DPE ETL Data Collection
1. Type the Name.
2. Set the Relevance Window.
3. Set the Time Granularity.
4. Choose the Incremental 1
Query Type. 2
3
5. Tick the Is Active checkbox.
4
6. Set the Modify Profile. 5
7. Click the [Next] button. 6
7
DPE ETL Data Collection
Creating A DPE ETL Data Collection
1. Click the File Data
Source field. 2 4
1
2. Select the SNMP Data
Source object type.
3. Click [OK]. 3
4. Click the [Search] button.
5. Select the Data Source.
6. Click [OK].

5
DPE ETL Data Collection
Creating A DPE ETL Data Collection
1. Click [Validate Query]
to validate the generated
query and display the
output columns.
2. Click [Show top] to
display the data returned
by query.

1 2
DPE ETL Data Collection
Creating A DPE ETL Data Collection - Query
1. Click the [Advanced]
radio button.
2. Edit the generated query.
1
Remove all columns
except for STIME, MO,
2
HRSTORAGESIZE and
HRSTORAGEUSED from
the query.
DPE ETL Data Collection
Creating A DPE ETL Data Collection - Instance Mapping & Date

1. Click [Select] next to the

Key field. 1
2. Select the MO column.
3. Click the [OK] button. 2
4. Click the [Next] button.

3 4
Remappings
Overview
Two types of remappings:
 Remapping of Keys
 Propagation Tree
DPE ETL Data Collection
Creating A DPE ETL Data Collection – Column To KPI Mapping
1. Click the [All] button. 1 4
2. Select the appropriate DPE
KPI Group (the child KPI 2
Group).
3. Click the [Add] button.
4. Click [Select All]. 3 5
5. Click [OK].
6. Click [Next].

6
DPE ETL Data Collection
Creating A DPE ETL Data Collection – Aggregations
1. Click [Add]. 4. Click [OK].
2. Fill in the Name of the 5. Tick the tick-boxes for AVG, MIN and
Aggregation (5min AGG). MAX aggregations.
3. Click [Select All]. 2
6. Click [Next].

3
1

5 4

6
23
DPE ETL Data Collection
Creating A DPE ETL Data Collection
Do not change the
configuration at this step.

1. Click the [Next] button.

1
DPE ETL Data Collection
Creating A DPE ETL Data Collection – Scheduling, Summary
1. Click the [On arrival of new data] 2. Click [Next].
radio-button. The data collection will
3. Verify the values of the attributes.
run every time when new data is
detected. 4. Click the [Finish] button.

3
1

2
4
DPE ETL Data Collection
Detail Tabs – Kpis, KPI Groups
1. Select the newly created DPE ETL Data
Collection in an Inventory View window.
2. Select the KPIs tab.
1
3. Check the KPIs which are collected by the
DPE ETL Data Collection. 2
4. Select the KPI Groups tab.
5. Check the DPE KPI Groups which are used
by the DPE ETL Data Collection.
3

5
 DPE ETL Data Collection
Detail Tabs – If Runs Tab
1. Select the newly created DPE ETL
Data Collection in an Inventory View
window.
2. Select the If Runs tab. 1
3. Wait for the first Aggregation and ETL
entries. Check the values in the Count
2
Extracted and Count Loaded columns.

3
Troubleshooting

Raw Counters are not calculated in case of:

 Problems with Data Source
 Data Base problems (e.g. cannot extend table-space)
 Subsystem is not running (Monitoring Broker System,
AdapterDataCollector)
 Incorrect Insert Query
 Data Collection Status
 If Runs analysis
03
DPE Complex Data Collections
DPE Objects
Objects To Be Created

Data Source KPI Group

DPE ETL DC COMPLEX DC THRESHOLDS

STAT DC
COMPLEX Data Collection
Overview
 STATISTICAL and COMPLEX KPI DATA COLLECTIONS are used for
further processing of gathered data:
 the former – by using statistical functions,
 the latter – by calculating the value of a KPI from other KPIs.

 Two available methods of data gathering regarding above Collections:

 ETL mechanism usage,
 Another Data Collection usage.
COMPLEX Data Collection
Overview
Complex operations on KPIs:
 Arithmetical operators
 Logical operators
 Standard mathematical functions
 COMPLEX Data Collection
Creating A Complex KPI Data Collection
1. Click the [New Object] 1
button.
2. Select the Complex KPI
Data Collection object
type.
3. Select the template 3
(default).
2
4. Click [Next].

4
COMPLEX Data Collection
Creating A Complex KPI Data Collection – Main Attributes
1. Type the Name.
2. Set the Modify Profile. 1
3. Click [Next].

3
COMPLEX Data Collection
Creating A Complex KPI Data Collection
1. Select the [Use Existing] radio
1
button.
2. Select the previously created 1
KPI Group by using the 3
Search dialog. 3 2
3. Click [Add]. 4
4. Select the first KPI from the 5
4
list of available KPIs
(FreeStorage). 2
5. Repeat steps 3-4 for the second
KPI (PCTStorageUSED).
 COMPLEX Data Collection
Creating A Complex KPI Data Collection
1. Right-click the first KPI in the
KPIs section. Select [Edit] from
the context menu.
 COMPLEX Data Collection
Creating A Complex KPI Data Collection
1. The Expression Editor window opens. Select the Advanced Mode tab.
2. Select the Operators tab.
3. Drag&drop the [-] operator.
4. Select the KPI tab.

1 4
2

3
 COMPLEX Data Collection
Creating A Complex KPI Data Collection
1. Expand the appropriate KPI Groups. 3. Drag&Drop the second KPI.
2. Drag&Drop the first KPI. 4. Select the Text Mode tab.

2
1

3
COMPLEX Data Collection
Creating A Complex KPI Data Collection
1. Verify the expression in the Text Mode.
2. Click [OK].

2
 COMPLEX Data Collection
Creating A Complex KPI Data Collection
1
1. Select the Operators tab.
2. Drag&Drop the [Case] 2
operator.
3. Drag&Drop the [When]
operator.
3
4. Drag&Drop the [<>]
operator.

4
COMPLEX Data Collection
Creating A Complex KPI Data Collection
1. Select the KPI tab. 1
2. Drag&Drop the
2
appropriate KPI
(denominator).
3. Right-click the empty
node. Select Edit value
from the context menu.
4. Enter the value (0). 3
5. Click [OK].
4

5
 COMPLEX Data Collection
Creating A Complex KPI Data Collection
1. Drag&Drop the [*] operator. 3. Drag&Drop the [/] operator. 5. Drag&Drop the appropriate
2. Set the value for the empty 4. Drag&Drop the appropriate KPI KPI (numerator).
node (100). (numerator).

3
1
5

2
 COMPLEX Data Collection
Creating A Complex KPI Data Collection – Else Operator
1. Right-click the [CASE] node. Select Add node from the context
menu.
2. Drag&Drop the [else] operator. 1
3. Set a value for the ELSE branch.

2 3
COMPLEX Data Collection
Creating A Complex KPI Data Collection – Text Expression
1. Click [Test].
2. Enter the value for each KPI
(e.g. 200,70).
3. Click [Test].
4. Verify the returned value in the
Result section.
5. Click [Close].
6. Click [OK].
 COMPLEX Data Collection
Creating A Complex KPI Data Collection
1. Click […] to view more
aggregation periods. Tick
the 5-Minute tickbox in the
Aggregation Periods 1
section.
2. Select AVG Aggregation
Method for each KPI. 2
3. Click [OK].

3
 COMPLEX Data Collection
Creating A Complex KPI Data Collection
Threshold and Actions will not be
defined for the Complex KPI Data
Collection at this step.

1. Click [Next].

1
COMPLEX Data Collection
Creating A Complex KPI Data Collection - Schedulling

1. Select the Dependent on other 3. Create a filter 5. Select the appropriate DPE ETL
data collections radio-button. expression. data collection.
2. Click [Add]. 4. Click [Search]. 6. Click [OK]. 7. Click [Next].

4
1
3
2

7 6
COMPLEX Data Collection
Creating A Complex KPI Data Collection
1. Verify the values of the
attributes.
1
2. Click [Finish].

2
COMPLEX Data Collection
Creating A Complex KPI Data Collection – Predecessors Tab

1. Select the newly created Complex KPI Data Collection in an Inventory View
window.
2. Select the Predecessors tab.
3. Check the related data collections.

3
COMPLEX Data Collection
Creating A Complex KPI Data Collection – If Runs Tab
1. Select the newly created Complex KPI Data Collection in an Inventory View window.
2. Select the If Runs tab.
3. Wait for the first entries with non-empty values in the Count Extracted and Count
Loaded columns.

3
COMPLEX Data Collection
Creating A Complex KPI Data Collection
Complex KPIs are not calculated in case of:
 Problems with formulas (e.g. dividing by zero)
 Problems with predecessors (e.g. ETL Data Collections)
 Problems with schedules (e.g. disables schedule)
 Subsystem is not running (Monitoring Broker System,
AdapterDataCollector)
 If Runs analysis
DPE Objects
Objects To Be Created

Data Source KPI Group

DPE ETL DC COMPLEX DC THRESHOLDS

STAT DC
04
DPE Statistical Data Collection
DPE Objects
Objects To Be Created
 DPE Statistical Data Collection
Overview
A Statistical Data Collection is a data collection using statistical functions.
Available functions (examples):
 Daily Busy Hour
 Outlier detection
 Moving Analysis
 Moving Average
 Moving Delta
DPE Statistical Data Collection
Creating A Statistical Data Collection
1. Click [New Object].
2. Select the Statistical 1

Data Collection object

type.
3. Select the template
(default).
4. Click [Next]. 3
2

4
DPE Statistical Data Collection
Creating A Statistical Data Collection

1. Type the Name. 1

2. Set the Modify Profile.
3. Click [Next].

3
DPE Statistical Data Collection
Creating A Statistical Data Collection
1. Select the Use Existing radio-
button. Click the […] button.
2. Select the previously created KPI 1
Group using the Search Dialog.

2
DPE Statistical Data Collection
Creating A Statistical Data Collection
3. Click [Add]. Select [Add
new KPI] from the
submenu. 6
4. Fill in the Name of the 31 4 5
KPI.
5. Click the […] button. 3 4 7
6. Choose the Moving Delta 6
function using the search
dialog.

8
DPE Statistical Data Collection
Creating A Statistical Data Collection
7. Click the Source KPI
field.
8 10
8. Select Object Type (KPI).
9. Click [OK].
9
10. Select the previously
created Complex KPI
using the search dialog. 7

11. Set the value of the Delay

parameter (e.g. 3). 11

12. Click [Next].

12
DPE Statistical Data Collection
Creating A Statistical Data Collection
Threshold and Actions will not be
defined for the Statistical Data
Collection at this step.

1. Click [Next].

1
DPE Statistical Data Collection
Creating A Statistical Data Collection
1. Click the Schedule field. 5. Select a Schedule (Every 1 minute).
2. Select the On Time Schedule. 6. Click [OK].
3. Click [OK]. 7. Click the [Next] button.
4. Click [Search]. 4
1
2

7 3 6
DPE Statistical Data Collection
Creating A Statistical Data Collection
1. Verify The Values Of The
Attributes.
1
2. Click [Finish].

2
DPE Statistical Data Collection
Creating A Statistical Data Collection
1. Select the newly created Statistical Data Collection in an Inventory View window.
2. Select the If Runs tab.
3. Wait for the first entries with non-empty values in the Count Extracted and Count
Loaded columns.

1
2

3
DPE Statistical Data Collection
Creating A Statistical Data Collection

Data Source KPI Group

DPE ETL DC COMPLEX DC THRESHOLDS

STAT DC
05
DPE Threshold Data Collections
DPE Threshold Data Collections
Overview
 DPE module makes it possible for the user to configure a system action that
will be generated in case of threshold violations or missing data. The action is
usually an alarm.
 ALARM GENERATION TEMPLATE objects are created in order to make it
possible for the user to define the properties of the alarm that will be
generated.
DPE Threshold Data Collections
Generating A New Condition Group
1. Go to the Thresholds and
Actions step 1 2
2. Select [Generate New
Condition Group] from the 3
toolbar.
3. New Condition Group and
Condition are added.
DPE Threshold Data Collections
KPI, Threshold And Severity

1. Select a Condition.
2
2. Click […] to open Select 1
KPI dialog. 5
3. Select a KPI.
4. Click [OK].
5. Set Comparator, Threshold 3
and Severity.

4
DPE Threshold Data Collections
Additional Conditions
1. Right-click a Condition
Group. Select Conditions-
>Simple Condition from
the context menu. 1

2. Select the newly created

Condition.
3. Set the Condition
properties.
2
3

4
DPE Threshold Data Collections
Alarm Generation Template
ALARM GENERATION
TEMPLATE defines the 2
properties of the alarm that will
be generated. 1
1. Select a Condition. 3
2. Select [Add New Action]
from the toolbar. 4
3. AGT Wizard is open.
4. Type the Name.
5. Click [Next].

5
DPE Threshold Data Collections
Alarm Generation Template - Parameters
1. Select a Parameter (e.g. 2
Displayed Text). 1
2. Click [Edit].
3. Edit the Parameter 3
definition.
4. Drag&Drop Managed
Object->External Key
attribute. 4

5. Click [OK].
5 6
6. Click [Next].
DPE Threshold Data Collections
Copy&paste Actions
1. Copy the newly created
Action.
2. Paste the Action to each 1
Condition. 2
2

3
DPE Threshold Data Collections
Condition Groups, Problem ID
1. Select the Complex
KPI Data Collection
in the Inventory
View. 11
2
2. Select Condition
Groups tab. 2
2
3. Check the value of
Problem ID 3
attribute.

3
06
Quick view
Quick View
Thresholds Visualization
 available only for Condition Groups consisting of Simple Conditions (no Pattern,
Time, Advanced Conditions)
 available when only one KPI is selected
 available for many MOs selected
 two visualization modes: ”Show lines” and ”Show lines and areas”
 threshold lines colours the same as alarms colours
 „Cleared” line and area are never drawn
 only current threshold levels are presented
 thresholds displayed the same way for ”>” and ”>=” conditions
Quick View
Thresholds Visualization
To activate the option:
1. Check the Visualize Thresholds checkbox.
Quick View
Displaying Kpis Using Quick View Tab
The Quick View tab allows the user to display KPI values in the context of specific
Managed Objects. The selected values can be subsequently viewed on a chart based on a
pre-defined chart template.
Quick View
Displaying Kpis Using Quick View Tab

1. Open the Bookmarks tab

in the main OSS Console
window
2. Open the Threshold
Configurator Training –
Operators View bookmark
Quick View
Displaying Kpis Using Quick View Tab
The Quick View tab is divided
into the Panels Section with:
1. KPIs
2. Managed Objects
3. Chart Section
4. Options tab used to define
time and threshold display
settings. 1

4
2 3
Quick View
Kpis Multiselection
 when more than one KPI is selected, there can be more than one Y axis drawn
 KPIs have two attributes deciding how many Y axes are needed: ”Y axis” and
”Unit”
 the rules:
Quick View
Kpis Multiselection
Quick View
Kpis Multiselection
Quick View
Kpis Multiselection
Quick View
Object Tree (Area 2)
Quick View
Object Tree (Area 3)
Quick View
Managing Y Axis Names
Quick View
Chart Mode
Quick View
Table Mode
Quick View
Exporting Data
Quick View
Report Component
Possibilty of visualization basic
KPI and DFE/NGSA self-
monitoring charts
Quick View
Report Component
 DFE self-KPI Published Events
Quick View
External Tab In KPI Tab