SITE 2.

1 Training

Reporting & Alarming

Keynote SIGOS GmbH

Klingenhofstraße 50d
90411 Nuernberg, Germany
Fon +49 / 911 / 9 51 68-0
Fax +49 / 911 / 9 51 68-355
www.keynote-sigos.com Testing is our competence
contact@keynote-sigos.com
 overview - SITE report environment

Test Execution on
different locations
Test
Executions

real time
reporting
Measurements
Strategy
Base
Measurements
(Input and Output)
alarming
KPIs

Keynote SIGOS GmbH 2008, page 2

How to create a report?

Keynote SIGOS GmbH 2008, page 3

 Toolbox for Reports, Report Editor

Reports are created with the “Report Editor”.

You can open the Report Editor with the corresponding button in the toolbar.

Edit Report Definition

Together with the Report Editor the Toolbox for Reports will be opened in the
navigation area...

Keynote SIGOS GmbH 2008, page 4

 Toolbox for Reports, Report Editor

You can open the Toolbox for Reports manually with the regarded button in the
navigation bar:

Toolbox for Reports

Keynote SIGOS GmbH 2008, page 5

 Toolbox for Reports, Report Editor

The Toolbox for Reports contains all items you need to define your reports:

Folder to build up a hierarchy in your report definitions

Report Definition defines a new report which consists
of one or more charts
Chart is a diagram which shows one or more
measurements
Alarm defines a threshold on a measurement. If the
threshold is reached a defined action will be
executed.
Measurement Databases contain the measurements of the
testruns. The layout is defined by the Measurement
Strategy.
Measurements are results of the testruns. The calculation of
the measurements is defines in the Measurement
Strategy

Keynote SIGOS GmbH 2008, page 6

 Toolbox for Reports, Report Editor

You can build up a structure in the “Report Editor” by dragging items from the
“Toolbox for Reports” (navigation area) and dropping them in the working area:

Toolbox For Reports Report Editor

Keynote SIGOS GmbH 2008, page 7

 Toolbox For Reports, Report Editor

You can create reports out of these

structures…

Keynote SIGOS GmbH 2008, page 8

 Toolbox For Reports, Report Editor

You create a report by selecting the report definition node in the Report Editor
and pressing the “Show Current Report” button afterwards.

or by using the context menu

Show Current Report

Keynote SIGOS GmbH 2008, page 9

 first report
Demonstration: Display the Rx_Level of the testruns within the last 2 hours

 open the report editor

 drag a new folder from the toolbox for reports to the structure tree
 drag a new report definition from the toolbox beneath the new folder
 drag the KPI Rx_level from the toolbox beneath the new chart

 show current report by opening the context menu of the new report or
selecting the icon from the toolbar.

Keynote SIGOS GmbH 2008, page 10

 Report Definition

When you click on a Report Definition you will find the parameter tree in the
working area:

Keynote SIGOS GmbH 2008, page 11

 Report Definition - size

With the “size” parameter you can define the initial size of the
charts in the report. When set to automatic the value is taken from
the table ConfigurationParameters (report image width and report
image height) in the database sitecu. Default is 500x300)

Keynote SIGOS GmbH 2008, page 12

Report definition – Synchronize charts

Synchronize the timelines across all

charts in the report.

By setting to 'no' it is possible to

define individual timelines for each
chart.

Keynote SIGOS GmbH 2008, page 13

 Report Definition - Report Interval

The timing parameter defines the time range for the report. Only results of
testruns with a Timestamp value within this range will be shown in the report.
The timestamp value is the time when the final verdict of a testrun is set. So in
most cases the Timestamp will match the end time in the TestcaseStatus view.

Keynote SIGOS GmbH 2008, page 14

 Report Definition – Report Interval
• Fixed (account KPIs from a certain start time till a certain end time)
t
start time end time Report generation
t
start time end time Report generation

• Rubberband (account KPIs from a certain start time till time of report generation)
t
start time Report generation
t
start time Report generation

• Window (account all KPIs within a certain time window before report generation)
t
window Report generation
t
window Report generation

Keynote SIGOS GmbH 2008, page 15

 Report Definition - Data Aggregation

In the Report Timing dialog you also

setup the Data Aggregation Interval.
In case you use aggregated
measurements, the Report Interval is
split into slots of the selected
duration.
For each time slot the aggregated
measurements will be combined to a
single data point in the report which is
placed at the end of the time slot.

Note:
If you select “Window” the time range displayed is aligned according to the data
aggregation interval. If you select “Until now” you will force the system to extend
the end point of the report window to the time of the report generation.

Keynote SIGOS GmbH 2008, page 16

 Report Definition - Data Aggregation

3
2
1
t

4
3
2
1
t
Aggregation method: count
min
max
average
Keynote SIGOS GmbH 2008, page 17
 Report Definition - Report Schedule

You can schedule a report to be generated one or more times. Any time a
scheduled report is generated, the report data is stored in the database.
You can also enter an email address which the generated reports will be sent
to.

Keynote SIGOS GmbH 2008, page 18

Report Definition - eMail Report Example

Keynote SIGOS GmbH 2008, page 19

 Report Definition - Filter

The filter parameters allow you to define a

filter for all measurements used in the
charts of the report.
The filter is valid for all measurements
which are taken from the filtered table.
(A filter on the “Main” table will be valid
for all measurements originating from the
same database.)

You can define following filter:

db_name (Measurement Target), db_table (Testcase), column_filter

If “include_invalid” is set to “no” measurements marked as 'invalid' will not be

included in the report.

Keynote SIGOS GmbH 2008, page 20

Report Definition - Column Filter

Keynote SIGOS GmbH 2008, page 21

 Chart Definition

When you click on a Chart Definition you will find the chart parameter tree in the
working area:

Keynote SIGOS GmbH 2008, page 22

 Chart Definition - Chart Type

Chart Types:
marker lines lines marker matrix single

areas bar stack bar group bar percentage matrix multi

Keynote SIGOS GmbH 2008, page 23

Chart Definition – Aggregation Method

The aggregation method is

necessary for the “matrix”
report types

Keynote SIGOS GmbH 2008, page 24

 Chart Definition – Timing

If the parameter “synchronize_charts”

in the report definition is set to “no”,
the timeline (report interval) can be
specified for each chart in the report,
separately.

Keynote SIGOS GmbH 2008, page 25

 Chart Definition - Chart Title

u can enter a title of a chart. The title will be displayed at the top of the chart.
ere are some text macros available (that will be substituted by the actual values):
TART/> : Begin of timing
ND/> : End of timing
ATH/> : Report-definition-Path
AME/> : Report-definition-Name
MEZONE/> : Time zone setting of the report's owner

Keynote SIGOS GmbH 2008, page 26

 Chart Definition - Chart Title

Here you can modify the information

contained in the legend within the chart:

Keynote SIGOS GmbH 2008, page 27

 Chart Definition - Chart Filter

Here you can define a filter for all measurements in the

chart (not in the whole report).
Remember: The filter will only be taken into account if
it is set on the same table as the measurements
actually used in the chart. (The main table will be valid
for all measurements of the same target.)

Keynote SIGOS GmbH 2008, page 28

 Chart Definition - Chart Filter

You can select a background color for

the chart. Default is white.

Keynote SIGOS GmbH 2008, page 29

Chart Definition - Bar Labels

If you choose chart type bar stack or

bar percentage you can enable the
display of labels for each bar. You can
choose between labels showing the
absolute value or the percentage
value.

Keynote SIGOS GmbH 2008, page 30

 KPI Definition

When you click on a KPI Definition you will find the KPI parameter tree in the
working area:

Keynote SIGOS GmbH 2008, page 31

KPI Definition – Data Source

The data source of the KPI can

still be changed.

Changing this parameter have the same

effect as dragging a different KPI,
respectively a KPI from a different KPI
database and/or a different KPI table, into
the chart.

Keynote SIGOS GmbH 2008, page 32

 KPI Definition - Type

You can define a column and/or a test

definition filter on measurements.
The column filter is a string pattern filter
on string value or a numerical filter on
numeric columns on a table which is a
join of the Main table and the testcase
specific table from which the
measurement was taken.
The test definition filter is a filter on the
test definition node in the test editor. Only
results of the given test definition will be
taken of the report.

Keynote SIGOS GmbH 2008, page 33

Difference test definition and column filter on TestDefPath

You can define a test definition filter or you can define a column filter that
filters for certain test definition paths.
Both filters will only allow measurements of a specific test definition. What's
the difference?

column filter testdef filter

Works after moving the test definition

to another position in the structure or no yes
after renaming
Works after export and reimport to the
same location yes no

Keynote SIGOS GmbH 2008, page 34

 KPI Definition - Type

Here you can choose the "type" of the

KPI in regard to it’s role to the chart.
Following types are available:
y1-value The value will be displayed on
the y-axis.
y2-value The value will be displayed on
the y-axis, but a second scale will be
used.
y3-value, y4-value Additional y-scales
like y2-value
x-value-string All other KPIs of the chart
will be aggregated on the different
values of this KPI. These different
values build up the x-axis of the chart.
x-value-numeric Similar to x-value-string
but the points on the x-axis will be
scaled according to the value.

Keynote SIGOS GmbH 2008, page 35

y2-value example

Keynote SIGOS GmbH 2008, page 36

x-value-string example

Keynote SIGOS GmbH 2008, page 37

x-value-numeric example

Keynote SIGOS GmbH 2008, page 38

 KPI Definition - Factor And Scale

In the y_scale_factor and scaling settings you

can adjust scaling setting. You can multiply the
measurement values with a given factor or
adjust the range of the y-axis.
You can also define a label for the axis and a
unit description.

Keynote SIGOS GmbH 2008, page 39

 KPI Definition - Aggregation

There are different aggregation methods

available:

raw no aggregation is done

min/max/avg the minimal (maximal, average)
value of all values in each aggregation
interval is calculated
sum all values in the specific aggregation
interval are summarized
count is the number of database entries
matching all filter criteria

Remember: In the chart the aggregated

values will be displayed at the end of the
aggregation interval.

Keynote SIGOS GmbH 2008, page 40

 KPI Definition - Chart Type

It is possible to setup different chart

types for different measurements
within a chart. Additional to the chart
types you know from the chart
definition there is the default value
'common'. 'common' means that the
chart type is taken from the chart
definition.

Here is an example of a chart

with measurements of
different chart types.

Keynote SIGOS GmbH 2008, page 41

KPI Definition – Alarm Expiry

Here you can define an expiry period.

If there is no KPI data for at least the
duration of the specified period the
severity of a defined alarm will be set to
“indeterminate‘”.
The period can be defined in following
formats: sec, min:sec, h:min:sec. If the
expiry is set to “0” a trapped alarm will
not expire.

Keynote SIGOS GmbH 2008, page 42

Let's have a closer look
at the report...

Keynote SIGOS GmbH 2008, page 43

 Report Properties
If you look at a report you can
adjust the timing settings
temporarily.

Keynote SIGOS GmbH 2008, page 44

 Report Properties
It's also possible to change the
color of the measurements

Keynote SIGOS GmbH 2008, page 45

 Interactive Chart Viewer

With the chart viewer you are

able to browse through the
measurements of your report.

Keynote SIGOS GmbH 2008, page 46

Interactive Chart Viewer
With the cursor you can run
through the results by using the
arrow keys. You will see the
Timestamp and the Value for
each data point.
(If the cursor is at the exact x
position the information is
displayed with the color of the
according KPI. Otherwise the
color of the information is grey
which means, that you see the
information of a data point which
is near to the current position of
the cursor). In case
unaggregated values are used in
the chart you will also see the
TestcaseId.
If you see the TestcaseId you are
able to go to the trace of the
testrun directly.
Keynote SIGOS GmbH 2008, page 47
Interactive Chart Viewer - Drill down

Keynote SIGOS GmbH 2008, page 48

 Measurement Data
From the report dialog or the
interactive chart viewer you can
open the measurement data view.
Here you can see the list of all
values used for the report.

Keynote SIGOS GmbH 2008, page 49

 Measurement Data – Drill down

In the Measurement Data view you can open the traces of the testcases
which resulted in the measurements had the chart consists of.
Note: Drill down is only possible if you use raw data.

Keynote SIGOS GmbH 2008, page 50

 Chart Viewer - Toolbar

Change Report
Properties
Update view of
current Report
Open Interactive
Chart Viewer

Show Save Chart Enable/Disable

Measurement as Picture Cyclic Update
Data
Show Store Report
Report Info to Database
Send link to report
graphic via email

Keynote SIGOS GmbH 2008, page 51

 Chart Viewer - Toolbar

Change Report
Properties
Update view of
Show current Report
Measurement
Data
Save Chart Enable/Disable
Open Trace as Picture Cyclic Update

Keynote SIGOS GmbH 2008, page 52

Alarming

Keynote SIGOS GmbH 2008, page 53

 Alarm Definition

The first step to define an alarm is to drag an alarm item from the “Toolbox For
Reports” beneath a KPI Item of a report in the “Report Editor” via Drag and Drop.

Toolbox For Reports Report Editor

Keynote SIGOS GmbH 2008, page 54

 Alarm Parameter

If you click on an Alarm definition you will find a parameter tree like this in the
working area:

Keynote SIGOS GmbH 2008, page 55

Alarm Definition - Parameter

An alarm can be activated

or deactivated. Alarm traps
will only be sent when the
alarm is activated.

The alarm threshold can be

shown in the report chart.

An alarm has a certain severity.

You can choose between
warning, minor, major, critical

The color of the alarm line in the

report chart can be changed.

Keynote SIGOS GmbH 2008, page 56

Alarm Definition - Threshold

The alarm will be trapped if the

measurement will be above, below,
equal or not equal (condition) a
given threshold for at least count
times.

Keynote SIGOS GmbH 2008, page 57

Alarm Definition - Actions

The alarm text is a user definable

string that will be send together
with other information as alarm
trap

There are different ways to send an

alarm trap. You can send an alarm
email, SMS (additional HW required).
SITE can also send traps to a SNMP
trap host.
There is also the option to call a
certain application or shell script once
the alarm was raised.

Keynote SIGOS GmbH 2008, page 58

alarm email example

Keynote SIGOS GmbH 2008, page 59

 workshop

 Chart showing the average duration of all testcases run in the last two weeks.
 Chart showing the number of testruns for each testcase (TCName) run in the last
two weeks.
 Chart showing the number of passed, failed and incomplete testruns for each
testcase (TCName) run in the last two weeks.

Keynote SIGOS GmbH 2008, page 60