Splunk® IT Service Intelligence Release

Notes 4.1.2
Known issues in Splunk IT Service Intelligence
Generated: 4/08/2019 7:11 pm

Copyright (c) 2019 Splunk Inc. All Rights Reserved

Known issues in Splunk IT Service Intelligence
IT Service Intelligence (ITSI) version 4.1.2 has the following known issues and
workarounds.

Splunk platform issues that impact ITSI compatibility

Publication Issue
Description
date number

• ITSI Event Analytics is incompatible with

Splunk Enterprise version 7.2.0 - 7.2.3.
• On versions 7.0.5 - 7.1.x, 7.2.4, and 7.2.5,
Event Analytics might duplicate events. To
work around this issue, create a limits.conf
2019-02-14 SPL-155648
file at
$SPLUNK_HOME/etc/apps/SA-ITOA/local/ and
add the setting phased_execution_mode =
auto.
• If you do not plan on using Event Analytics,
the above does not apply.
Backup/Restore and Migration Issues

Issue
Date filed Description
number
In a search head cluster environment, the Backup/Restore page only lets
2019-03-11 ITSI-2714
nightly backups. It does not display a list of all other backup files on all ins
ITSI backup times out due to an extremely large number of episode comm

Workaround:
2019-01-03 ITSI-2164
Delete all comments prior to the backup (purge the collections in the KV s
Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI n
in the KV store to archive them faster (the default is 6 months).
2018-10-16 ITSI-1748 You cannot restore an ITSI backup more than once.

Workaround:
This issue occurs because the saved search DA-ITSI-APM-EUEM_Base_
the system. Create the missing saved search manually before restoring th
create a local version of savedsearches.conf and add the following stanza

1
 Issue
Date filed Description
number
[DA-ITSI-APM-EUEM_Base_Search]
description =
search =
request.ui_dispatch_app = itsi
request.ui_dispatch_view = search

If multiple services use one KPI base search, and the total size of your se
ITSI generates an error.
2017-02-10 ITSI-1309
Workaround:
Increase the value for max_size_per_batch_save_mb (50MB is default)
in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore
After migration, shared objects (service analyzers, glass tables, and deep

Workaround:
Use the curl command and create ACLs for each of the shared objects tha
in the KV store collections: itsi_pages and itsi_service_analyzer.

For example:

$ curl -u admin:Splunk3r -k
https://127.0.0.1:8089/servicesNS/nobody/SA-UserAccess/storage/co
-X POST -H "Content-Type:application/json" -d '\{
"obj_id": "XXX-XXX-XXX",
2016-05-02 ITSI-1305 "obj_type": "glass_table",
"obj_app": "itsi",
"obj_storename": "itsi_pages",

"obj_acl": \{
"obj_owner": "nobody",
"read": ["*"],
"write": ["*"],
"delete": ["*"]

},
"object_shared_by_inclusion": "true",
"acl_owner": "nobody"
}'

2
Deep Dive

Issue
Date filed Description
number
When you drill down to a deep dive from the
Predictive Analytics dashboard in Internet Explorer,
the deep dive opens with no lanes because the URL
2018-09-13 ITSI-1556 is too long.

Workaround:
Manually add the KPI lanes to the deep dive.
If you zoom in on a specific time range in a deep dive
2016-12-14 ITSI-525 while using twin-lane comparison, the comparisons
that appear are occasionally offset by up to a minute.
Entities

Issue
Date filed D
number
The curl command to delete all entities times out with a large amount of e

Workaround:
2019-02-19 ITSI-2540 Use the service endpoint to clear all entities. For example:
curl -k -u admin:Chang3d! -X DELETE
https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collecti

2015-02-12 ITSI-1286 When importing entities using Data inputs > IT Service Intelligence CSV Im
Notable Events

Issue
Date filed
number
2019-02-11 ITSI-2458 Disabling an alert action in alert_actions.conf does not remove the option
Episode Review does not generate events if there is no user with the user
of etc/apps/SA-ITOA/metadata/default.meta.
2019-02-07 ITSI-2431
Workaround:
Create a user with the username "admin" with the admin_all_objects capa
2019-01-09 ITSI-2189 Long notable event descriptions are sometimes truncated.
2019-01-03 ITSI-2164

3
 Issue
Date filed
number
ITSI backup times out due to an extremely large number of episode comm

Workaround:
Delete all comments prior to the backup (purge the collections in the KV s
notable event collections in the KV store to archive them faster (the defau
2019-01-02 ITSI-2161 The "Add column" option in the View Settings of Episodes Review does n
Some notable events are added to more than one episode.

Workaround:
For an ITSI search head running Splunk 7.1 or 7.2, create or edit etc/syste
2018-12-10 ITSI-2059 [search]
phased_execution_mode = auto

For an ITSI search head running Splunk 7.3 or later, there is no need to c
2018-12-05 ITSI-2036 The "All Events" tab does not load for an episode created by an aggregati
2018-08-15 ITSI-1182 If notable events are received so quickly that their timestamps are the sam
When your browser and the Splunk server are set to different DST time zo
2017-03-29 ITSI-1299
Workaround:
Set your time zone to something other than "system default" even if you a
2017-03-29 ITSI-1316 Splunkd connection fails due to "no_shared cipher matched" between clie

Workaround:
In order for notable event management and anomaly detection to work wi

• Java 8/JRE 1.8/JDK 1.8*

* Download JCE 8 from [1]

* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir
running the JDK.

• Java 7/JRE 1.7/JDK 1.7*

* Download JCE 7 from [2]

* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir
running the JDK.

4
 Issue
Date filed
number
Update SA-ITOA/local/commands.conf with the following commands:

[itsirulesengine]

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_rules_en
command.arg.3=-DitsiRulesEngine.configurationFile=../default/itsi
command.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC
chunked = true

[itsicorrelationengine]

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_correlat
command.arg.3=-J-XX:+UseConcMarkSweepGC
command.arg.4=-DitsiCorrelationEngine.configurationFile=../defaul
command.arg.5=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.6=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC
chunked = true

Update SA-ITSI-MetricAD/local/commands.conf with the following comma

[mad]

type = custom
command.arg.1=-J-Xmx1G
command.arg.2=-Dlog4j.configurationFile=../default/log4j.xml
command.arg.3=-Dlog4j2.threadContextMap=com.splunk.mad.util.MadTh
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC
= true

ITSI generates duplicate event_ids from the itsi_tracked_alerts index. This

up the value of the event_id field and does not create a GUID for the even
2016-09-08 ITSI-1268
Workaround:
Rename the event_id field.
2016-04-01 ITSI-1346 The 'Ping Host' action does not work when ITSI and Enterprise Security a

Workaround:
1. Add the following stanza to $SPLUNK_HOME/etc/apps/SplunkEnterprise

5
 Issue
Date filed
number

[app_imports_update://update_es]
apps_to_update = (SA-(?!(ITOA|ITSI|IndexCreation|UserAccess)).*)

2. Delete the "import = *" line from [] stanza of $SPLUNK_HOME/et

SA-ITSI-LicenseChecker, SA-IndexCreation, SA-UserAccess.

3. Restart Splunk.

Glass Table

Issue
Date filed Description
number
When you add a predictive model to a glass table,
2018-09-14 ITSI-1567 you cannot use the sparkline or trending value viz
types because the prediction is a static value
KPI Base Searches

Issue
Date filed Description
number
2017-04-13 ITSI-1294 KPI base search does not handle duplicate entity
aliases, causing incorrect group KPI statistics.

Workaround:
1. When Splunk detects duplicate aliases, a warning
message appears in the Messages menu. Click Show
duplicates to open the ITSI Health Check dashboard
which lists the entities with duplicate aliases. (Or click
Dashboards > ITSI Health Check from the ITSI main
menu.)

2. Click Configure > Entities and edit the entity

definitions with duplicate aliases. Keep the alias value
for one of the entities and edit the other to remove the
duplicate alias value.

Note: You can also merge the duplicates by moving

all the fields that differ to one entity, then deleting the

6
 Issue
Date filed Description
number
extra copy.

3. Turn off all module entity discovery searches.

KPI Search Calculation

Issue
Date filed Description
number
If an ITSI admin, who only has access to certain
indexes, creates a KPI and uses the backfill option, the
2018-04-26 ITSI-248
backfill runs through all data and not just the data that
the admin has access to.
Maintenance Window

Issue
Date filed Description
number
The maintenance window UI does not calculate
daylight savings correctly.

2018-04-25 ITSI-277 Workaround:

The maintenance window UI displays the UTC time in
parentheses. Rely on these times for the
maintenance boundaries.
When you navigate back and forth in the Edit
2017-08-08 ITSI-1236 Maintenance Window modal, some information is
populated incorrectly.
Role Based Access Controls

Issue
Date filed Description
number
If an ITSI admin, who only has access to certain
indexes, creates a KPI and uses the backfill option, the
2018-04-26 ITSI-248
backfill runs through all data and not just the data that
the admin has access to.
2018-02-06 ITSI-440

7
 Issue
Date filed Description
number
When itoa_admin, itoa_analyst, itoa_team_admin and
itoa_user roles are added to a new custom role, users
assigned to the custom role do not have the "edit
permissions" capability for saved service analyzers.
Roles inheriting from itoa_admin do not behave like
itoa_admin. For example, the inheriting role cannot edit
permissions on pages such as glass tables, deep
dives, and service analyzers.
2017-10-16 ITSI-437
Workaround:
Make the user a member of the itoa_admin role (rather
than just a member of a role inheriting from it).
Service Analyzer

Issue
Date filed Description
number
Filters with no matching results can't be saved in
2017-10-04 ITSI-1290
the Service Analyzer.
Service Definition

Issue
Date filed Description
number
On Windows 10 on Chrome, some selectors in the
2016-03-28 ITSI-1269
ITSI app do not function.
Teams

Issue
Date filed Description
number
When you filter services on the team details page,
no services match the filter.
2019-03-25 ITSI-2822
Workaround:
Type the filter using only lower case characters.

8
Threshold Templates

Issue
Date filed Description
number
When you run the kvstore_to_json.py mode 3 option
2018-12-05 ITSI-2020 on ALL KPI threshold templates (versus just one), the
KPI does not reflect the changes made.
Predictive Analytics

Issue
Date filed Description
number
2019-01-18 ITSI-2309 Predictive Analytics is not available if ITSI is installed
on Splunk Enterprise version 7.0.x.

Workaround:
Perform one of the following workarounds:

A. Upgrade to Splunk version 7.1.x or later.

B. If you cannot upgrade, modify the Predictive

Analytics macros:

1. Navigate to
$SPLUNK_HOME/etc/apps/SA-ITOA/local
2. Create or edit a macros.conf file.
3. Add the following stanza to the file:

# Macro to train KPI trend models and health

score KPI relations.
[train_kpi_trends(2)]
args=sid,suffix
definition =
`itsi_predictive_analytics_dataset($sid$)`\
| appendpipe [fit LinearRegression
fit_intercept=true now_avg_hs from\
"value_avg:*" into
app:itsi_predict_kpi_hs_$suffix$ | fields -
_time *]\
| fit StandardScaler "value_*" with_mean=true
with_std=true into
app:itsi_predict_kpi_ss_$suffix$\
| `prepare_kpi_trend_data($sid$,$suffix$)`\
| map search="| inputcsv
itsi_predict_kpi_$suffix$.csv | fit
GradientBoostingRegressor \"next30mkpi_$kpiid$\"

9
 Issue
Date filed Description
number
from\
\"SS_*\" \"this_date_*\"
\"last30mkpi_$kpiid$\" \"value_avg: $kpiid$\"
into app:itsi_predict_kpi_$model_suffix$"\
maxsearches=100\
| head 1\
| fields "predicted(*)"\
| rename "predicted(next30mkpi_*)" as *\
| fields - _time\
| foreach * [eval <<FIELD>>=1]\
| untable modelname kpi dummyfield\
| fields - dummyfield
| eval
modelname="itsi_predict_kpi_".replace(kpi, "-",
"_")\
| append [| listmodels\
| search name="itsi_predict_kpi_*_$suffix$"\
| rename name as modelname\
| fields modelname]{code}
4. Save the file and restart Splunk.

5. Verify the fix by training a predictive model for a

small time period (like 7 days).
When you add a predictive model to a glass table, you
2018-09-14 ITSI-1567 cannot use the sparkline or trending value viz types
because the prediction is a static value
When you drill down to a deep dive from the Predictive
Analytics dashboard in Internet Explorer, the deep dive
opens with no lanes because the URL is too long.
2018-09-13 ITSI-1556
Workaround:
Manually add the KPI lanes to the deep dive.
After you delete a Predictive Analytics model through
2018-08-01 ITSI-1105
Lookups, the model still appears in the UI.
Splunk App for Infrastructure Integration

Issue
Date filed Description
number
2018-09-24 ITSI-1654 Only 50,000 entities can be imported from the Splunk
App for Infrastructure.

10
 Issue
Date filed Description
number
Workaround:
By default, the entity integration imports up to 50,000
entities from the Splunk App for Infrastructure. If you
have more than 50,000 entities in Splunk App for
Infrastructure, only the first 50,000 will be imported into
ITSI. Increase the max_rows_per_query setting in
$SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf
under the [kvstore] stanza to import more than 50,000
entities.
Uncategorized issues

Issue
Date filed
number
2019-02-12 ITSI-2471 If ITSI is installed on multiple environments with multiple license masters,

Workaround:
Delete the internal license, install a secondary internal license, and disab

1. Click Settings > Licensing and delete the IT Service Intelligence Int

2. Click Add license and upload the following license key file:

<license>
<signature>o3eXzWryQOQG3M2d1vs9dSn8NsxXbB1HtozqcaTkjo9QhHzZTLFWu
<payload>
<type>fixed-sourcetype</type>
<group_id>Enterprise</group_id>
<quota>107374182400000</quota>
<max_violations>5</max_violations>
<window_period>30</window_period>
<creation_time>1549958400</creation_time>
<label>IT Service Intelligence Internals *DO NOT COPY*</label>
<expiration_time>2163135600</expiration_time>
<features>
<feature>Auth</feature>
<feature>FwdData</feature>
<feature>LocalSearch</feature>
<feature>ScheduledSearch</feature>
<feature>Alerting</feature>
<feature>SplunkWeb</feature>
</features>
<add_ons>
<add_on name="itsi" type="app">

11
 Issue
Date filed
number
<parameter key="size" value="1"/>
</add_on>
</add_ons>
<sourcetypes>
<sourcetype>itsi_notable:*</sourcetype>
</sourcetypes>
<guid>71029F93-1CBD-4201-8D8D-03D0EAD582A0</guid>
</payload>
</license>

3. Click Settings > Data inputs > IT Service Intelligence license check
When you create a multi-KPI alert, the summary index stores the entity_ti

Workaround:
Create a correlation search as an alternative to a multi-KPI alert.

1. Click Configure > Correlation Searches.

2. Click Create New Search > Create Correlation Search.

2018-11-16 ITSI-1941
3. Provide a search name.

4. Enter a search that contains the service ID. For example, `mka_sn_kpin

5. Enter a notable event title and description. For example, %service_nam

6. Configure other fields and click Save to save the correlation search.

7. Go to Episode Review and you should start seeing events.

Correlation searches created by manually editing savedsearches.conf do
ITSI-1287,
2018-06-27
ITSI-793 Workaround:
Do not create correlation searches by manually editing $SPLUNK_HOME
2015-12-01 ITSI-1320 When you install Enterprise Security on a search head with a pre-existing

Workaround:
1. In Splunk Web, go to Settings > Access Controls.

2. Select Roles > admin.

12
 Issue
Date filed
number
3. Add itoa_admin, itoa_analyst, and itoa_user to Selected roles.
4. Click Save.
In a search head cluster environment, if you want to set up a modular inp

2015-03-25 ITSI-1293 Workaround:

Configure modular inputs on individual search head cluster members. To
works, but logs will show error messages on machines where the modula
All ITSI Modules

Publication Issue
Description
date number
When you bulk add services and an error caused
2017-03-21 ITOA-7585 by the racing condition occurs, the incorrect
message "itsi_module does not exist" is displayed.
KPIs do not have consistent backfill settings across
2017-03-07 MOD-979
all modules.
The Analyze KPI button on the Service Details page
2017-01-17 MOD-452
is broken.
The Export to PDF option does not work in the
2017-01-17 MOD-402
drilldown to a module.
The extendable tab XML generator REST endpoint
2017-01-17 MOD-296 is located in DA-ITSI-OS instead of in common
components where it can be used by all modules.
ITSI displays a misleading error message when a
2017-01-17 MOD-591 KPI template contains a field that cannot be
resolved.
There is no upper limit to the number of characters
2017-01-17 MOD-498 a KPI title or description can contain. Long strings
can negatively affect performance.
The Gruntfile.js included in ITSI modules uses
2017-01-17 MOD-309 double quotes instead of single quotes, which does
not conform to the standard for all JavaScript files.
2017-04-17 MOD-2002 When you drilldown from the Events tab, an "Invalid
earliest_time" error occurs.

13
 Publication Issue
Description
date number

Workaround:
Disable drilldown from the Events tab.
Some modules do not have descriptions for saved
2017-01-17 MOD-439
searches.
Application Server Module

Publication Issue
Description
date number
If you reuse the same panel within a dashboard,
2017-01-27 MOD-492 the duplicate panel does not display any event
data.
Cloud Services Module

There are no known issues for this release.

Database Module

Publication Issue
Description
date number
When a lookup is not configured for
2017-01-17 MOD-586 TA-Microsoft-SqlServer, ITSI displays a misleading
error message on the server drilldown page.
End User Experience Module

There are no known issues for this release.

Load Balancer Module

Publication Issue
Description
date number
If you reuse the same panel within a dashboard,
2017-01-27 MOD-492 the duplicate panel does not display any event
data.

14
Operating System Module

Publication Issue
Description
date number
The Storage Free Space % base search runs every
2017-04-13 MOD-555 minute while the Linux df command runs every 5
minutes. This causes data gaps.
Windows data for memory free space is collected
2017-04-10 MOD-1964
at different intervals than the Memory Free % KPI.
Line, stack, and area charts do not display a metric
2017-01-17 MOD-1398 gap when no metrics are available during a time
period.
Storage Module

There are no known issues for this release.

Virtualization Module

There are no known issues for this release.

Web Server Module

Publication Issue
Description
date number
Some KPI ad hoc searches transform data with the
stats command and do not retain time fields. The
2017-03-17 MOD-320
KPIs do not render anything and do not show
thresholding details.
When you add a new tab with panels and refresh the
2017-03-17 MOD-538
page, the page breaks.

15