Citrix Support Runbook V 2.0

Run Book – Document
For
BI Citrix
Version – 4.0
22.05.2022
Citrix Runbook
Version No 1.00
Date 08.11.2021
Submitted by Ram Satendra/Dimpu Barman
Primary Contact Name & Details Ram Satendra
Secondary Contact Name &
Details Dimpu Barman
Preparation and execution update Modification History

and upgrades all Citrix
components (incl backend
infrastructure) per year within 4
weeks (incl. QA Testing,
Production rollout,
documentation) and (whenever
Version Date Name Reviewe Approver Notes / Changes in
r chapters
1.0 08-11- Ram Werner Werner Intial Release by TCS
2021 satendra Worf Worf
2.0 11-12- Ram Werner Werner Updated sections
2021 satendra Worf Worf 6.7/6.8/6.9/6.14/6.15
3.0 02-02- Deepak Werner Werner Updated section 6.11 &
2022 Vasudevan Worf Worf 6.12
4.0 22-May- Neeraj Werner Werner Light house tool
2-22 Sethi Worf Worf updated
5.0 14-Dec- Kiran MCS provision tool,VDI
2022 house
keeping,Controlup,Citrix
cloud migrations
Document Usage – All Embedded documents will ONLY open through Edit>Open in
Desktop App
Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 2 of 57

Citrix Runbook
Table of content
1. Purpose of the document............................................................................................5
2. Support scope.............................................................................................................6
2.1. Support overview..................................................................................................6
2.2. Support Structure..................................................................................................8
2.3. Support Features................................................................................................10
2.4. Support phone numbers.....................................................................................11
3. Service Scope in detail: Level 2................................................................................11
4. Service Scope in detail: Level 3................................................................................19
5. Support Process....................................................................................................... 23
5.1. New Contractor Onboarding Checklist................................................................23
5.2. Ticket handling and Configuration Management................................................24
5.3. How to gain Service now Access........................................................................24
5.4. Health Check & Monitoring Process...................................................................25
6. Support Activities......................................................................................................26
6.1. Incident handling.................................................................................................26
6.2. Consulting Request............................................................................................ 27
6.3. Image management............................................................................................27
6.4. Publishing applications from AppV.....................................................................27
6.5. Compliance check – Lighthouse Tool.................................................................27
6.6. Patch Management for XenApp..........................................................................27
6.7. Support Key Documentation Locations...............................................................28
6.8. Illumio................................................................................................................. 29
6.9 AMPEG Lighthouse Configuration…………………………………………………...…33
6.10 How to comply Virtual OBICs in AMPEG Lighthouse…………………..…………..33
6.11 Virtual OBIC Housekeeping…………………………………………………………...34
6.12 How to create a new Virtual OBIC solution using Machine Creation Services……..35
6.13 Power BI Report for Citrix Applicatio……………………………………………………36
6.14 Profiling solution for APPS and desktop………………………………………………..36
7.0 Escalation Matrix................................................................................................ 36
7.4 Citrix Knowledge Base Articles (KBs).................................................................39
7.5 Citrix issue tracker..............................................................................................39
7.6 Process for Weekly vDisk Update.......................................................................52
7.7 WFM Process.....................................................................................................52
7.8 Service Management Process............................................................................53
7.9 Oncall Process....................................................................................................53
7.10 Naming Convention.........................................................................................53
8.0 Technical Observations..........................................................................................54
9.0 Support collaboration.............................................................................................54
10.0 Business Continuity Planning.............................................................................55
11.0 Inventory.............................................................................................................56
12.0 RACI................................................................................................................... 56
13.0 Vendor Information.............................................................................................57
14.0 Global Operations Support Roles ......................................................................57

Citrix Runbook

Citrix Runbook
1. Purpose of the document

The Purpose of the Runbook is to document the details necessary to maintain,
support, troubleshoot and recover the application. It will be common document used
by Citrix Team and anyone else needing an overview of this Routine. In Particular, it
will give a quick start to new joiners and support team.
It is expected that the reader has proficiency in Pharma domain background and
experience in Citrix. After reading this document reader should be able to understand
high level environment and be able to perform support activities independently. This
one is running document, Periodic updates, review and approval should be done
whenever there is change in any of the process.

Citrix Runbook
2. Support scope
2.1. Support overview
Based on this SOW, services around Application and Desktop virtualization have to be
executed by the Supplier:
 Virtual Desktops (VDI) – pooled and dedicated
 Published desktops
 Published applications incl. MS AppV
BI has a Full Application and Desktop Virtualization technology stack based on Citrix
products in place:
 XenApp
 Xen Desktop
 PVS (Provisioning Services),
 Citrix NetScaler
 Work Space Environment Manager
 FS Logix
 Citrix Cloud
The production Citrix environment primarily consists of XenApp 6.0 and Citrix Virtual
Apps and Desktop (7.6/7.15/1912)
Citrix XenApp 6.0 farms sites are:

1) INH – Ingelheim
2) INH- CSN
3) NAH-North America
Citrix 7.6 farms sites are:

Citrix 7.15 farms sites are:

2) NAH – North America
3) SIH – Singapore.
4) VIE – Vienna
5) SHH – Shanghai
6) BIB – Biberarch
7) FRE – Fremount

Citrix Runbook
Citrix 1912 farms sites are:

Citrix Cloud farms sites are:
Boehringer Ingelheim EU-PRD
There is only one site in Citrix Cloud with On-Prem Application servers in below regions:
Documentation Links
Citrix 7.15 7.15_Architecture
CVADS 1912 1912 Architecture
CVADS 1912 1912_Handbook
Citrix cloud Cloud management Citrix Desktop
Production Citrix Url:
 https://myworkspace.boehringer-ingelheim.com
 https://myworkspace-ap.boehringer-ingelheim.com
 https://myworkspace-am.boehringer-ingelheim.com
 https://myworkspace-eu.boehringer-ingelheim.com
 https://myworkspace-cn.boehringer-ingelheim.com
 https://csnmyworkspace.boehringer-ingelheim.com
The support partners are responsible to provide Level 2/L3 support for the following
areas:
 Virtual Desktops (VDI) – pooled and dedicated

 Published desktops
 Published applications incl. MS AppV
 Citrix NetScaler as access gateway
 Provisioning - PVS/MCS

Citrix Runbook
 Citrix WEM
2.2. Support Structure
2.2.1. Citrix Level 2 Support Team:

The Citrix level 2 team will handle calls, user order requests, System alerts and
tasks related to Citrix from the SNOW queue. Short description of the activities
performed by the level 2 team is listed below:
Support Short description

Level
Level 2 Virtual Desktops (VDI) – pooled and dedicated
- Handle and resolve incoming system events forwarded by 1TOC or other
support teams
- Handle and resolve incoming incidents and How-to questions
- Analyze and resolve incidents regarding untypical behavior of software
within VDI
- Troubleshoot and support users to solve performance issues, e.g. latency
issues with VDI
- Troubleshoot and support users to solve issues regarding Audio and Video
- Execution of Service Requests (Change permissions for VDI, Create
clones in VDI pools, Decommission VDI, ...)
- Update the master image as defined in this SOW
Published desktops
support teams
- Handle and resolve incoming incidents
- Analyze and resolve incidents regarding untypical behavior of software
within published desktop
- Troubleshoot and support users to solve performance issues e.g. latency
issues with published desktop
- Troubleshoot and support users to solve issues regarding Audio and Video
- Add and remove servers to a delivery group / shared desktop after BI
approval
Published applications incl. MS AppV

support teams
- Handle and resolve incoming incidents
- Analyze and resolve incidents regarding untypical behaviour of published

Citrix Runbook

Level
applications if related to Citrix
- Publish applications (without GPO-configuration) on defined servers
- Removal of applications from Citrix console
- Change configuration of existing applications based on request (e.g. add /
change / remove permissions, add / remove server for published
application, change / other settings)
2.2.2. Citrix Level 3 Support Team:

The Citrix Level 3 teams will work on critical issues and tickets escalated by the
level 2 team. The team will also work on preparing new images, working on
problem tickets and working on Changes when necessary. A brief description of
the activities performed by the level 3 team is given below:

Level
Level 3 Virtual Desktops (VDI) – pooled and dedicated
- Handling complex incidents not solved by Level 2
- Working together with Level 2 and with other support teams to resolve
ongoing problems
- Configure datastore (e.g. existing datastore fills up) as needed
- Creation new pools
- Decommission VDI delivery group (with included GPO)
- Update master image 3 times per year (based on release management)
with additional applications which are distributed via SCCM automatically
Published desktops
problems
- Update master image (based on release management) with additional
applications which are distributed via SCCM automatically
- Update / Upgrade master images of shared servers monthly incl. required
security patches and additionally VDA upgrade


Citrix Runbook

Level
problems
- Publish applications (with GPO-configuration and testing) on defined
servers (incl. publishing on several servers for load balancing)
- For shared servers: Update / Upgrade master images of shared servers
- For dedicated servers: Update / Upgrade VDA package
2.2.3. SNOW Workgroup Name

The SNOW workgroup name, which will be used by both the Level 2 and Level 3
teams, is BI-ITINF-AD-MyWorkSpace Support Activities
2.3. Support Features
The following tables describe how support will be delivered based on the GOS support
framework definitions:
Level Topic Additional description based on framework

agreement
Level 1 Support hours Not applicable
Level 1 Supplier delivery Not applicable
locations
Level 2 Support hours 24x7
Level 2 Supplier delivery India
locations – technical
related Majority of agents work primarily for BI
locations – language
related
Level 3 Support hours 19x5 + on-call support
Pre-announced activities at weekends (in

average once per month)
locations
Majority of agents work primarily for BI
2.4. Support phone numbers

Citrix Runbook
To get support, the following phone numbers are available:
Level Phone number for specific topics Supported languages Description

Level 1 Dial-in number for end users N/A
Status clarification with
Status clarification by Business System Leads in IT
Level 2 Data Stewards and back office English Functional domains
Oncall phone no +91 97116 20792

Level 2 Oncall Contact for any P1/P2 : English Oncall support numbers
Escalation phone no :91- Will be mainly triggered by
9873903416/9910338494 Level 2 in case of urgent
Level 3 Oncall Contact for any P1/P2 : English issues
Xmatters for Primary and Secondary on call details.

Click here to open the details.
3. Service Scope in detail: Level 2

Virtual Desktops (VDI) – pooled and dedicated
Modules Description / Examples as extension to framework agreement

Application Handle and resolve incoming system events forwarded by 1TOC or
Monitoring other support teams
Incident - Handle and resolve incoming incidents, e.g.
Handling o Desktop not visible
o Desktop cannot be started (e.g. VDI in maintenance
mode)
o Connection lost
o Printing
o Backend infrastructure related incidents (disk space
issues, performance issues, VDI-spanning incidents)
o Handle incidents regarding software updates (incl.
antivirus) of dedicated and pooled VDIs
- Handle and resolve How-to questions regarding VDIs
- Reboot servers or restart services if required in alignment

Citrix Runbook

with 1TOC, IT functional domain and related system leads
- Analyze and resolve incidents regarding untypical behavior of

software within VDI, e.g. incorrect language setting of Internet
Explorer within VDI but not on a Standard BI PC package
o Analyze root cause of untypical behaviour
o If related to VDI: Troubleshoot and resolution of
untypical behaviour (e.g. language setting is saved
again)
o If not related to VDI: Forward ticket to the responsible
team
- Troubleshoot and support users to solve performance issues,

e.g. latency issues with VDI (incl. coordination with the local
IT department of external user, especially to clarify issues
around routing)
- Troubleshoot and support users to solve issues regarding

Audio and Video (incl. coordination with the local IT
department of external user to identify root cause e.g. local
firewall settings, installation of real time media engine)
Problem Identification, Handling Problems incl. Involvement of Level 3 and 3rd

Handling party Vendor
Service Request Execution Service Requests within 2 days
Handling  Change permission for VDI (e.g. add additional user)
 Create additional clones in VDI pools
 Handle and deliver Non-Standard VDI requests (Note: These
NON-Standards requests are triggered via user order
requests)
 Decommission individual VDI (without GPO)
 Change hardware settings for VDI clients (CPU, RAM, HDD,
…)
Consulting Consult System Lead and other stakeholders in IT as well as

Request business regarding best approach for provisioning of application
(e.g. VDI, published desktop, published application)
Support Regular shift-left to Service Desk
coordination
Change management (= documentation and tracking changes in

Citrix Runbook

SNOW and ITEMS) for Level 2 and Level 3
Configuration management (create csn/ update / delete CIs based

on defined rules in support handbook)
Support Docu- Regularly update of Knowledge articles, Review FAQ documentation

mentation / and Runbooks
Training
Create / update how-to documentation (especially step-by-step
descriptions with screenshots)
Application For dedicated and pooled VDIs: Patching / Update the master image
operation especially security patches, VDA and software updates monthly per
activities VDI separately
- Test provided SCCM packages before distribution and
provide test report

Application Monitoring of successful automated patching of Operating System
Monitoring and Antivirus
- Check in Lighthouse / ReportIT for not successfully patched
Windows Operating System / Antivirus compliance
Supplier has to use Lighthouse / ReportIT to create the specific

reports at least weekly (if not more often required to fulfil KPIs) to
ensure compliance, at least for the following topics:
- Issues with Antivirus and/or SCCM agents on servers
- Missing patch update
- Missing Antivirus patterns
- VDI not assigned to a wave
Incident Incident handling (e.g. patches not installed, VDI not assigned to
Handling patch waves, patch client does not work correctly), Execution of
manual activities in case of an unsuccessful automated patching for
operating system and / or virus protection (with access to Antivirus
tool, no access to SCCM console– issue has to be solved with
other tools provided by BI or manually)
- Create / Update ticket if not existing
- Analyze issue
- Resolve failure (e.g. redeploy patch, manual activation virus
pattern, manual installation patch)

Citrix Runbook

- Check successful patching / virus protection after resolution
Work together with other support teams (e.g. End User Operations,
SCCM team in case of general issues, Security Operation Centre in
case of Virus alerts, Data Centre team) to identify root cause of
issue (e.g. performance issue)
Problem Handling reoccurring issues as problems, if patching failures occurs

Handling on servers
- Present identified problems in weekly and monthly meetings
- Present work around / solutions for problems in weekly and
monthly meetings
Handling every Priority 1 as problem
Support Assignment of VDIs to specific patch waves

coordination
Support Docu- Any issues have to be documented as tickets
mentation /
Training Creation of change documentation
Application Note: A monthly patch wave is assigned to every VDI regarding
operation operating system.
activities - Windows patching:
o No technical actions as long as patching is successful
o Create, handle and resolve incident in case of any
issues
- Antivirus patterns:
o Patterns are deployed automatically – no activities
required by Supplier if deployment is successfully
o Create, handle and resolve incident in case of any
issues
- Execution of Antivirus client patching / updates / upgrade
o Ensure that patches / updates / upgrades are
distributed to all VDIs correctly
o Open, handle and resolve incident in case of any
issues
o Note: If upgrade is done via SCCM, SCCM package is
created by BI, but Vendor has to check if upgrade
was implemented successfully – if not, counter

Citrix Runbook

measures have to be triggered by Supplier (especially
creation and resolution related incident)
Published desktops

Application Handle and resolve incoming system events forwarded by 1TOC or
Monitoring other support teams
Handling o Desktop not visible
o Desktop cannot be started
o Connection lost
o Printing
issues, performance issues, …)
- Analyze and resolve incidents regarding untypical behavior

of software within published desktop, e.g. incorrect language
setting of Internet Explorer within published desktop but not
on a Standard BI PC package
o Analyze root cause of untypical behaviour
o If related to published desktop: Troubleshoot and
resolution of untypical behaviour (e.g. language
setting is saved again)
o If not related to published desktop: Forward ticket to
the responsible team

- Troubleshoot and support users to solve performance

issues, e.g. latency issues with published desktop (incl.
coordination with the local IT department of external user,
especially to clarify issues around routing)
- Troubleshoot and support users to solve issues regarding

Audio and Video (incl. coordination with the local IT

Citrix Runbook

department of external user to identify root cause e.g. local
firewall settings, installation of real time media engine)
- Taking servers into maintenance and back to load balancing
Problem Identification, Handling Problems Incl. Involvement of Level 3 and

Handling 3rd party Vendor
Service Request Execution service request within 1 day
Handling - Add and remove servers to a delivery group / shared
desktop

(e.g. VDI, published desktop, published application)
Support Regular shift-left to Service Desk
coordination
Change management (= documentation and tracking changes in
SNOW and ITEMS) for Level 2 and Level 3
Configuration management (create / update / delete CIs based on

defined rules in support handbook)
Support Docu- - Regularly update of Knowledge articles and Runbooks

mentation / - Create / update how-to documentation (especially step-by-
Training step descriptions with screenshots)
Note: Publishing of AppV packages is automated, Troubleshooting by Supplier is just

required in case of errors.

Application - Handle and resolve incoming system events forwarded by
Monitoring 1TOC or other support teams

Handling o Application not visible
o Application cannot be started
o Connection lost

Citrix Runbook

o Printing
issues, performance issues, …)

- Analyze and resolve incidents regarding untypical behavior

of published application if related to Citrix
- Taking servers into maintenance and back to load balancing
Problem - Identification, Handling Problems Incl. Involvement of Level

Handling 3 and 3rd party Vendor
Service Request Applications on shared servers

Handling - Handle and resolve incoming incidents if application was not
published / updated / upgraded correctly on shared servers
(e.g. troubleshoot to identify issue in App-V package and
consult responsible packaging team how to solve issue or
manually install application if required)
Execution service requests on Applications on dedicated servers

within 2 days
- Publish applications (without GPO-configuration) on defined
servers (incl. publishing on several servers for load
balancing)
- Support System Lead regarding Update / Upgrade of
application
- Change configurations of existing applications (e.g. add /
change / remove permissions, add / remove server for
published application, change / other settings)
- Disable application during a maintenance window
- Note: There will be several requests per application
regarding publishing based on number of environments (e.g.
test, QA, training, production)

Citrix Runbook

Consulting - Consult System Lead and other stakeholders in IT as well as
Request business regarding best approach for provisioning of
application (e.g. VDI, published desktop, published
application)
Support - Regular shift-left to Service Desk

coordination - Change management (= documentation and tracking
changes in SNOW and ITEMS) for Level 2 and Level 3
- Configuration management (create / update / delete CIs
based on defined rules in support handbook)
Support Docu- - Regularly update of Knowledge articles and Runbooks

mentation / - Create / update how-to documentation (especially step-by-
Training step descriptions with screenshots)
Note: All use cases must be recorded as a single ticket.
Languages Description / Examples as extension to framework agreement

English Yes
Spanish No
German No
Mandarin No
Japanese No
French No
Level 2 Language skills / quality have to be delivered as outlined in the Frame

Agreement, Chapter Language skills.

Citrix Runbook
4. Service Scope in detail: Level 3

Virtual Desktops (VDI) – pooled and dedicated

Incident Handling Handle and resolve complex incidents not solved by Level 2
regarding VDIs and backend infrastructure
Trigger incident at Software Vendor in case of not-solvable

incidents incl. regular contact with Software Vendor until
resolution
Problem Handling Working together with Level 2 and with other support teams to
resolve ongoing problems
Proactive proposals of possible improvements to reduce

reoccurring incidents (incl. performance issues) in alignment with
IT functional domain
Minor Change Update/Upgrade and Patching Backend
Requests Preparation and execution 2 updates and 1 upgrade all Citrix
components per year and (whenever needed) Security Patching
Configure datastore (e.g. existing datastore fills up) within 1 day

as needed
Create new pools

- Create master image
- Installation software incl. testing
- Create clones
- Create GPO
Decommission VDI delivery group (with included GPO) within 2

days
Update master image 3 times per year (based on release

management) with additional applications which are distributed
via SCCM automatically
Trigger and Monitoring of required patching with support e.g.

SCCM
Prepare / definition / coordinate / test new SCCM packages

together with BI SCCM Team (Note: BI SCCM Team responsible
to create package) needed for Citrix infrastructure especially VDA
client, Citrix Receiver, Media Engine and Connector and other

Citrix Runbook

Citrix related software
Create / Update and maintain scripts under consideration of rules

in the Support Handbook
(e.g. VDI, published desktop, published application) in case of
complex requirements
Involvement of BI Functional team in case of new / special

requirements
Support Coordination and execution relevant activities regarding
coordination upgrades of Citrix environment (especially update of procedures,
execution any master image related activities if required, train
agents on changes)
Published desktops

Incident Handling Handling complex incidents not solved by Level 2 regarding
published desktops and backend infrastructure

resolution
Problem Handling Working together with Level 2 and with other support teams to
resolve ongoing problems

reoccurring incidents (incl. performance issues) in alignment with
IT functional domain
Minor Change Update master images 3 times per year (based on release
Requests management) with additional applications which are distributed
via SCCM automatically
Master Images share room link with master list.
Link to master images.
Update / Upgrade master images of shared servers monthly incl.

required security patches and additionally VDA upgrade 2 times
per year (based on release management)

Citrix Runbook

provide test report
- Trigger and Monitoring of required patching with support
e.g. SCCM

together with BI SCCM Team (Note: BI SCCM Team responsible
to create package) needed for Citrix infrastructure especially VDA
client, Citrix Receiver, Media Engine and Connector and other
Citrix related software
Master Images share room link with master list.
Create new PVS image

- Create of master image
- Installation software incl. testing
- Create PVS target
- Create
- GPO
Create additional PVS targets for already existing PVS images to

increase capacity
Create / Update and maintain scripts under consideration of rules

in the Support Handbook

(e.g. VDI, published desktop, published application) in case of
complex requirements

requirements
Support Coordination and execution relevant activities regarding
coordination upgrades of Citrix environment (especially update of procedures,
execution any master image related activities if required, train
agents on changes)

Citrix Runbook
Incident Handling Handle and resolve complex incidents not solved by Level 2
regarding published applications and backend infrastructure,
e.g.
Resolve GPO configuration issues and update GPO
configuration if needed

resolution
Problem Handling Working together with Level 2 and with other support teams
to resolve ongoing problems

reoccurring incidents (incl. performance issues) in alignment
with IT functional domain
Minor Change Publish applications (with GPO-configuration and testing) on
Requests defined servers (incl. publishing on several servers for load
balancing)
- Note: There will be several requests per application
regarding publishing based on number of
environments (e.g. test, QA, training, production)
For shared servers: Update / Upgrade master images of

shared servers monthly incl. required security patches and
additionally VDA upgrade 2 times per year
provide test report
For dedicated servers: Update / Upgrade VDA package max.

2 times per year per server (incl. take server offline, execution
of SCCM package, test successful update / upgrade, take
server online)
provide test report
- Align update / upgrade with System Lead
- Define update / upgrade time slots with System Lead

Citrix Runbook
- Further coordination as needed

- Note: No involvement in patching – is done
automatically

together with BI SCCM Team (Note: BI SCCM Team
responsible to create package) needed for Citrix infrastructure
especially VDA client, Citrix Receiver, Media Engine and
Connector and other Citrix related software
Update and maintain scripts under consideration of rules in

the Support Handbook
Create / Update and maintain scripts under consideration of

rules in the Support Handbook
Consulting Request Consult System Lead and other stakeholders in IT as well as
business regarding best approach for provisioning of
application (e.g. VDI, published desktop, published
application) in case of complex requirements

requirements
Support coordination Coordination and execution relevant activities regarding

upgrades of Citrix environment (especially update of
procedures, execution any master image related activities if
required, train agents on changes)
5. Support Process
5.1. New Contractor Onboarding Checklist
Each new joiner needs to follow an onboarding checklist and is required to update
status in the attached template.
New Contractor Checklist document is located at below link:

Citrix Runbook
Techincal_Onboarding_Checklist
5.2. Ticket handling and Configuration Management
The support would be through Calls, System Events, Problems, Known Errors and User
Order Requests. It would be handled by TCS L2/L3 Citrix teams through SNOW tool and
below are the types of tickets which can be lodged through the tool:
• Call: Call (Incident) will be logged by BI Global Service Desk team.

• System Event: System event will be triggered by BI Monitoring tool
1-TOC (BI Monitoring Team) will receive the alert and assign SNOW to relevant
TCS L2/L3 team
• Problem: Problem ticket will be created for every P1 incident reported in the
environment. Repetitive incidents which require Root Cause Analysis would also
be tracked via problem for a definitive solution.
• Known Error: Document problems/solutions for already resolved problem tickets.
• User Order Request: Standard priority requests will be categorized as User order
request.
• Tasks: Orders placed via MyShop will be reported as Tasks in SNOW tool.
SNOW can be accessed through the link MyServices (service-now.com)
5.3. How to gain Service now Access

Below are the steps that shows how we can request for access to SNOW
MyServices - Request / Remove access
Environment: BI-PAAS-SERVICENOW-PROD
Add below roles:

 Change Analyst
 GxP Incident Analyst
 Incident Analyst
 Knowledge Contributor
 Problem Analyst
 Request Management Analyst

Citrix Runbook
5.4. Health Check & Monitoring Process
Though monitoring is out of scope in terms of the support model for TCS Citrix Team and
is maintained by 1TOC team, however, it is vital to have an overview of the Citrix
monitoring being performed.
Citrix team will also monitor the Citrix environment via Citrix Director on needed base for
troubleshooting. Health check report will be sent to BI SMEs thrice a day. Citrix team will
also monitor via Lighthouse report to ensure VDI are compliant for windows update and
Antivirus update.
Attached documents would help in providing an understanding of the working of 1TOC

team and how monitoring is being performed by Stable Net application.
TCS team is receiving daily PVS XenApp and Xen Desktop 7.15/7.6 and 7.24(1912)
Farm Report for across all sites INH, SIH, and NAH. Also working proactively to resolve if
issues occurs.
For more details for troubleshooting and resolve the issue please open documents.
Document Link/Attachment
PVS troubleshooting PVS_Troubleshooting
Health check Health_Check
Monitoring
/Thresholds
configuration
Sample Farm Report
Controlup:
ControlUp is a Digital Experience Monitoring and Optimization platform that transforms

the way IT manages systems, troubleshoots issues, and helps deliver a great user
experience. Developer SaaS-based analytics and management platform designed to
manage server-based computing farms and virtual desktop infrastructure.

Citrix Runbook
ControlUp is tailor-made for systems administrators and helpdesk personnel who

oversee multi-user environments and are required to prevent and troubleshoot
performance issues, application failures and operating system errors
which are capable of achieving the systems administrator’s two primary goals:
1.Quickly identify issues in a complex multi-user environment

2.Resolve these issues in simply and efficiently
ControlUp is a comprehensive system monitoring and management solution which

provides deep visibility into the real-time activity of servers, workstations, users and the
applications they use
The ControlUp monitor service assists with 24/7 monitoring of your assets and alerts
about any abnormal behavior according to a customizable set of incident triggers
ControlUp documentation: COntrolup_Latest .docx
6. Support Activities
Support topic Further information
Incident handling Work on the Incidents and resolve them in the defined SLA
Problem handling Performing detailed analysis on repeating incidents to

identify the root
Change handling Work on Standard and minor change requests.
User order request Work on standard user order requests
handling
On call support Providing support during outside-support-hours for high
outside business priority incidents
hours
Tasks Work on the Tasks and complete them in the defined SLA
The support includes mainly:
- Handling / Resolution of incidents not solved by Global Service Desk

- Identification and handling of problems for recurring incidents
- Execution of service requests as needed by business (e.g. add users, add
permissions)
- Execution consulting requests to advise business regarding simple consulting
topics
- Extend knowledge transfer based on releases / change requests to support
updated application
- Work together with relevant support partners (Global Service Desk, functional
domain, other internal and external support partners, application vendor, …)

Citrix Runbook
6.1. Incident handling

TCS L2/L3 teams will receive incident from BI Global Service Desk or from 1-TOC.
In SNOW, Calls (tickets created by BI Global Service Desk) and System Events (Tickets
created by 1-TOC) are considered as Incidents. TCS L2/L3 teams should ensure that
incidents are handled according to the priority of the ticket.
Incident Management process and guidelines- Various trainings and documents are
available on BI Learning portal (LOS) and other intranet sites (like ITSM, Shareroom).
Below is one of the links which can be referred for understanding BI Incident
Management Process.
6.2. Consulting Request

Consult System Lead and other stakeholders in IT as well as business regarding best
approach for provisioning of application (e.g. VDI, published desktop, published
application) in case of complex requirements. Involvement of BI Functional team in case
of new / special requirements
The initial meeting for a consulting request has to be done within 3 working days after
request via My Shop. The result of this initial meeting has to be:
- Clear advice for simple topics
Clear time plan for resolution consulting request within 2 weeks for complex topics
6.3. Image management
Image management Links

How to - Update a PVS vDisk – my desktop how to update PVS Disk - My Desktop
How to - Update a PVS vDisk for SCCM Cluster How to - Update a PVS vDisk for SCCM Cluster
Dedicated Catalog (MCS) MCS_Dedicated
Pooled Catalog (MCS) MCS_Pooled
6.4. Publishing applications from AppV

Information about publishing applications via App-V and troubleshooting can be found in
the link
6.5. Compliance check – Lighthouse Tool
The Citrix team must use Lighthouse / ReportIT to create the specific reports at least
weekly (if not more often required to fulfil KPIs) to ensure compliance, at least for the
following topics:

Citrix Runbook
- Issues with Antivirus and/or SCCM agents on servers

- Missing patch update
- Missing Antivirus patterns
- VDI not assigned to a wave
Below link contains the document process of identifying (Lighthouse Tool) the machines
that are not compliant (Patch Management): Click_Here
6.6. Patch Management for XenApp
During this phase, the SL interactively install the patch via SCCM or SCCM does an
automatic rollout of the patches on the Terminal Servers according to the regular patch
waves.
The process has 4 Phases that comprise since a new security patch is published from
vendor until the patch is applied to a Terminal Server within the BI infrastructure.
Patch deployment will be triggered and executed by following the patch wave schedules.
The patch management service is harmonized corporate-wide by One IS organization.
Q1: January, February, March

Q2: April, May, June
Q3: July, August, September
Q4: October, November, December
Wave M: Manual patch wave, SL installs interactively the SW package.

Wave 0: Test Servers, QA Servers, redundant production systems
Regular Waves: Rest of world
Please find below link for more information:

Click here
https://boehringer.sharepoint.com/sites/z365iasservices/Lists/IAS%20Citrix%20Products/AllItems.aspx?
p=11
6.7. Support Key Documentation Locations

Below Shareroom link contains technical documents which are referenced by the Citrix
team:
IAS Services - Application Virtualization Services - All Documents (sharepoint.com)
IAS Services - Checklist and How to's - All Documents (sharepoint.com)

Citrix Runbook
6.8. Illumio
Illumio VEN installation in re-usable master images
Illumio, inc.
https://www.illumio.com/
Illumio Adaptive Security Platform (ASP) Virtual Enforcement Node (VEN)
Citrix Ready
https://citrixready.citrix.com/illumio/adaptive-security-platform-asp.html
Background
Illumio has relatively poor documentation with regards to the installation of the VEN software on non-
persistent master images (a.k.a. 'golden images').
Their documentation:
The problem with these instructions is that the Illumio VEN will be activated at next boot.
This is not practical because we perform several restarts during mater image maintenance and re-use
the master image for future maintenance.

Citrix Runbook
It is not feasible having to uninstall the VEN, perform a clean-up and then re-install the VEN again
directly before sealing the master image.
In addition, the master image may be used for multiple environments and not always where Illumio is
used.
For example, MyDesktop internal and MyDesktop DMZ boot from the same master image. Only DMZ
uses Illumio (for now).
Because of these concerns, we implemented some logic in the SCCM package, now it stops and
disable the Illumio VEN services directly after the software gets installed. This enables us to perform
as many reboots as we want, and the possibility to control the activation of the VEN using a
PowerShell script at system startup, triggered by a Scheduled Task.
Prerequisites
1. The CA certs (root/intermediate) are installed on the Windows workload (Citrix server -Golden
Image).
2. The Windows workload (Citrix server -Golden Image) can resolve scp3.illum.io and reach it
on port 443/TCP.
Add the computer account used for the master target device to the latest SCCM Device Collection for
'Illumio_Virtual Enforcement Node Agent':
Illumio_Virtual Enforcement Node_BASE_21.5.11_MUI_R01_SRV_UNI_x64_C_Install
Installation
Install the Illumio VEN Agent software on the Windows workload (Citrix server -Golden
Image).Complete all steps before restarting the server as will pair the Agent and causes problems.
1. Run installation deployment from Software Center:

Latest Package Name - Illumio_Virtual Enforcement Node_BASE_21.5.11_MUI_R01
_SRV_UNI_x64_C_Install
Wait for the installation to complete.

Citrix Runbook
1. Launch an elevated PowerShell shell.

2. Navigate to C:\Program Files\Illumio
3. Execute the illumio-ven-ctl.ps1 script with “status” option:
Startup script
The SCCM package has some internal logic, so right after the installation the services get stopped
and disabled, we will need to re-enable the services and start the Agent manually on the clones.
A Scheduled Task can be used for this, configured directly in the vDisk, to a script at system startup.
Using an Active Directory Group Policy Objects (GPO) is not feasible because it will not apply in time
during the startup process.
An example PowerShell script and Scheduled Task can be found here:
\\ingfs102\data1\XenDesktop\Temp\RamonU\Illumio
Copy 'Start-IllumioVEN.ps1' to C:\Boehringer\Scripts\.
This PowerShell script performs various checks, based on the MyDesktop requirements. Modify this
to suit the environment.
1. Execute the illumio-ven-ctl.ps1 script with 'enable' option:
This re-enables the services.
2. Execute the illumio-ven-ctl.ps1 script with 'start' option:
This starts the Agent.
[OPTIONAL CHECK]
Check the status on the clones to see if services got started and enabled by the startup script:
- On the clone, launch an elevated PowerShell shell.

Citrix Runbook
- Navigate to C:\Program Files\Illumio

- Execute the illumio-ven-ctl.ps1 script with “status” option:
- The status should be “illuminated”, this indicates that the agent on the server got
successfully paired against the Illumio console.
Note:
If the computer name does not match, then
the script ends without making any changes.
Shutdown script
To prevent Agents from appearing as duplicates after rebooting a non-persistent Windows workload,
the illumio-ven-ctl.ps1 script with 'deactivate' option should be run prior to shut down:
This ends the connection to the Illumio PCE and the workload reverts to its pre-Illumio native firewall
settings. A 30 second delay is added to give the VEN enough time to deactivate before Windows
shuts down. This may not be required. There is also the 'unpair' option, which will also uninstall the
software. This should not be used in a non-persistent environment as it will have no effect.
Documentation:
https://docs.illumio.com/asp/20.1/Content/Guides/ven-administration/rollback-deactivate-or-uninstall-
vens/deactivate-and-unpair-vens.html
6.9 AMPEG Lighthouse Configuration
 The global reporting tool for determining the global compliance status of SCCM is Lighthouse.
 We can find all the information related to Access to Light house and dashboard configuration
can be found.
 Google Chrome is the preferred browser in accessing AMPEG Lighthouse.
 Information regarding Lighthouse configuration can be found in the below link

Citrix Runbook
AMPEG Lighthouse Configuration - Citrix.docx (sharepoint.com)
6.10 How to comply Virtual OBICs in AMPEG Lighthouse
-The below attached document will provide us the step-by step guide to comply Virtual OBICs in
Lighthouse.
How To - Comply VDI in AMPEG Lighthouse.docx (sharepoint.com)
1. Order access in lighthouse.
2. Lighthouse can be accessed through this link using the company authorized web browser:
http://bi-asl
To log in, select your proper domain at the “Log on to” combo box, select the radio button
“Network User” and enter your Active Directory account and password
3. Select the “Coverage” section in the “Security Section” combo-box:
4. Select the scope of PC Virtual and hit O
5. Next, select the EU region in the worldwide map. The Protection percentage to be reported is the
corresponding to the line containing “Coverage = Covered”: (88.20% in this exam
6. Click Partially Covered and Not Covered to extract the affected machines.
7. You will received the exported list as zip file. Extract it then open the xml file with excel
and choose as an XML Table.
8. Copy the machine name in the system column.
9. Paste the list of machines to \\inhas72837\C$\boehringer\Scripts\VDICompliance\

lighthouse.csv
10. Run the script \\inhas72837\C$\boehringer\Scripts\VDICompliance\

CVADComplianceScriptManual.ps1 and wait till it finish. Minimum runtime is three hours.

Citrix Runbook
6.11 Virtual OBIC Housekeeping
 The decommissioning process for Virtual OBIC 2.0 internal and contractors will delete virtual
obics which are not in used in the past 60 days. The process is trigger by scripts and scheduled to
run every two weeks using UC4 job scheduling.
 For virtual OBIC temporary trial, the housekeeping process running every week. It will delete the
virtual OBICs which already reach 120 days since it was delivered. The user will get popup
notification on their TempTrial vobic about how many days left before the vOBIC reach 120 days
and get terminated. The popup counting started from on day 1.
Below are the DGs where the decommission is configured and we get autogenerated
email on every Monday to create a request to remove the OBICs from the BIDS Group
- RLBI-INH-PRD1-DG-WKS-W10-DMZ-OBIC20Contractors
- RLBI-INH-PRD1-DG-WKS-W10-INT-OBIC20
- RLBI-INH-PRD1-DG-WKS-W10-INT-OBIC20TempTrial
Attaching email screenshot
-We will get an attached document in the email where the userids will be mentioned
-We have to create request for those users to get them removed from the BIDS Managed
group.
Below link to remove the users from the BIDS Managed group
Service Catalog - BI Service Portal (service-now.com)

Citrix Runbook
https://boehringer.sharepoint.com/:w:/r/sites/z365iasservices/Shared%20Documents/
Application%20Virtualization%20Services/Desktop%20Virtualization/Virtual%20OBIC
%20Housekeeping.docx?d=wa27178b6c7fb476e9bdb1e703c1af224&csf=1&web=1
6.12 How to create a new Virtual OBIC solution using Machine Creation Services
-The below attached document will provide us the step-by step guide to create a new Virtual OBIC
solution using Machine Creation Services tool
How To - MCS Provisioning Tool.docx (sharepoint.com)
1. Go to management server and launch the tool located in C:\boehringer\Scripts\VDIAutomation\

MCS Manual Provisioning Tool\MCSAutomated.exe.
2. Enter the number of machines you need to create then press OK.
3. Enter the destination Machine Catalog then press OK.
4. Enter the destination Delivery Group.
5. Wait for the machines to be created and look out on the logs in CMD.
6. Check if the machines are created. Please take note that newly created machines need at least 4
hours to process all required registration in the organization (SCCM, Command, ServiceNow,
etc.).
6.13 Power BI Report for Citrix Applications
 It is an Application report where we can find all the details about the citrix
applications which are active, inactive, external
 We can also find the details about session count, not used Apps, DG details from
this tool
Below is the link provided for the powerBI inorder to check the application details.
CTXAppReport - Power BI
6.14 Profiling solution for APPS and desktop:

In the citrix environment currently we are using 3 different profile solutions
 Local profile
 UPM profile
 Fslogix profile containers

Citrix Runbook
Local profile:
we have turned on local profiles on most of dedicated servers . Nevertheless we firstly
have to communicated this to the system leads. They have to be aware that we will turn
off roaming profile which has no sense on dedicated clusters with less than 3
servers. Most of them have only 1 server, so more sense to have local profile. Benefits
would be fast logins (as local drives are running over flash disk) and less corruptions due
to not share profile with others solutions. we already enabled (or we will enable) to not
delete profile after logoff the session, therefore the profile will be same as roaming and
they should not lost any setting. We create the local GPO not delete local profile
anymore for now. We can disable roaming profile one by one on Citrix Cloud as they are
configured via WEM.
UPM:
Citrix UPM copies the user profile at session logon. Changes are saved in the
local session and when the user logoff his session registry changes, files and folders in
the profile are saved to the UPM network location.Most the servers are using Citrix UPM
profile.
FSLogix profile containers
Fslogix stores a complete user profile in a single container. At sign in, this container is
dynamically attached to the computing environment using VHD(X) virtual disk.
 Fslogix containers are Microsoft’s recommended profile solution for all non-
persistent environments
 Extremely fast logon times / reduces network and filesystem load
 Places entire user profile in network-based container
 Virtually eliminates profile corruption
 Fslogix provides unique value for Office 365 environments
 Unify technologies (right now we use Fslogix rules for AppMasking for XenApp
and Profile & Office365 container over MyDesktop solution)
 Remove complexity in virtual applications (right now we use Citrix UPM + Fslogix
rules to folder redirection Documents & Desktop & OneDrive)
Some DG like Mydesktop contractors, Mydesktop internal ,RDM are using Fslogix
profiling solution
Below is links for Fslogix

New Citrix profile concept.pptx (sharepoint.com)
FSLogix the new roaming profiles (sharepoint.com)
IAS Services - FSLogix - All Documents (sharepoint.com)
7. Escalation Matrix
7.1 TCS Escalation Matrix:
Escalation Matrix - IO Delivery_Boehringer Ingelheim

Service Escala Sup TCS Point of Contact
Domain tion port Notification/ Contact Email Contact

Citrix Runbook
Escalation
Phone number
Hour Point
zzITINFmyworkspaceDevOps
Level Hot-line
Lync@boehringer- N/A
1 Number
ingelheim.com
Level TCS Citrix
24 x Lync@boehringer- N/A
Citrix 2 Team
7 ingelheim.com
Level Dimpu.barman.ext@boehringe (+91)
Dimpu Barman
3 r-ingelheim.com 9910338494
Level Ram.satendra.ext@boehringer (+91)
Ram Satendra
4 -ingelheim.com 9873903416
Level Hot-line
Lync@boehringer- N/A
1 Number
Manage ingelheim.com
ment Level Ram.satendra.ext@boehringer (+91)
10 x Ram Satendra
Escalati 2 -ingelheim.com 9873903416
5
on Level Ritika.patni.ext@boehringer- +91
Ritika Patni
contact 3 ingelheim.com 8376877595
Level Saurabh Saurabh.aggarwal.ext@boehri +91
4 Aggarwal nger-ingelheim.com 9953005507
7.2 BI Escalation Matrix :
Escalation Matrix - IO Delivery_Boehringer Ingelheim

BI Point of Contact
Service Escalat Notification/
Domain ion Contact Phone
Escalation Contact Email
number
Point
bogdan.stanciu@boehringer- +34 (93) 405-
Level 1
Stanciu,Bogdan ingelheim.com 4825
werner.worf@boehringer- +49 (6132) 77-
Citrix Level 2 Worf,Werner
ingelheim.com 182006
glyn.lipham@boehringer- +1 (203) 798-
Level 3 Lipham,Glyn
ingelheim.com 4598
juho.pulkkinen.ext@boehringer- +34 (93) 404-

Manage Level 1 Pulkkinen,Juho
ingelheim.com 5972
ment
mark.kolb@boehringer- +1 (203) 233-
Escalati Level 2 Kolb,Mark
ingelheim.com 1995
on
glyn.lipham@boehringer- +1 (203) 798-
contact Level 3 Lipham,Glyn
ingelheim.com 4598

Citrix Runbook
Weekend BI Escalation process: If TCS SME not able to resolve outage issue on
weekend, Ram satendra will contact 1TOC team to send notification to escalation group
i.e BI-CITRIX-L4 to get someone involved from BI SME team.
7.3 BI Escalation and Handoff Procedures:
Please find the attached document that clearly describes the procedures that the team
need to follow in cases were team is unable to reach BI user.

Citrix Runbook
Escalation and handoff procedures

Boehringer Ingelheim managed services
Escalations & Handoff

To empower our managed service partners several key escalation procedures have been
developed. They are specific to the delivery of an efficient managed service. If a ticket
becomes “stuck” in another BISM teams queue it is the responsibility of the managed service
partner to bring this to the attention of BI management for internal escalation.
Escalation process for Tasks, UOR, Call
 First attempt – day one
 Try to reach client via Skype and backup with email
 Second attempt – day two
 Third attempt – day three
 Try to reach direct manager via Skype and backup with email to client and manager explaining the
ticket will be closed in 24 hours as we are waiting for a response from the client and there is no action to be
taken on our part. In addition emphasis they can open another ticket with the same request at a later time.
Escalation process for System Events & Problems
 First attempt – day one
 Second attempt – day two
 Third attempt – day three
 Escalate to all BI L4 managers ( zzIT INF IAS Heads )
 Provide Ticket #, description and status so that manager can escalate appropriately.
Handover process between partners or BI groups

If a ticket needs to be processed by a group other than the primary partner (owner of
service) consideration must be taken when forwarding or escalating. Failure to do so will yield
undo delays in the delivery of service.
Primary partner = service that has accepted the ticket
Secondary partner = service that the ticket is be sent or transferred to.
For Problems the primary partner should hold the problem and create a ticket for the other
group to carry out the task needed to help resolve the problem.
 The primary partner should NOT transfer the problem to the other group. The ticket
should be linked to the problem.
 If the ticket is out of SLA then the primary partner should escalate to BI management.
For System Events the primary partner should update the ticket with all findings to date and
the reason they are forwarding the ticket to that group with instructions of what needs to be
done by them.
 Primary transfer ticket to secondary partner queue.
 The primary partner should notify the secondary partner via distribution list that they
are transferring a system event for resolution.

Citrix Runbook
7.4 Citrix Knowledge Base Articles (KBs)
All the knowledge base Articles are now updated in Service now which can be accessed
from below Link and can be referred for any specific known issues..
Knowledge | MyServices (service-now.com)
Some of the commonly known issues are highlighted below.
KB Number Issue
KB0020835 Profile Reset
KB0032379 MyWorkspace - Unable to launch Virtual Application
KB0032734 MyWorkspace : Manually logoff a hang Citrix session?
MyWorkspace - Common Issues troubleshooting [Self-service
KB0032373 support KB for end user]
How to change Language and Regional settings for Citrix
KB0032144 applications.
MyWorkspace : How to clean up c:\ drive on Virtual desktop
KB0033294 (VDI)?
KB0021525 VDI Latency issue, slow VDI
MyWorkspace : Getting error in office applications excel,
KB0038867 outlook, Teams
External Users-Why do I see a black screen on a virtual
KB0026870 application?
MyWorkspace : Getting error in office applications excel,
KB0038867 outlook, Teams
MyWorkspace : How to change language and region settings
KB0033709 for Myworkspace applications/Desktops?
7.5 Citrix issue tracker
We have Citrix issue tracker Share room Issue_Tracker to discuss about the escalation
and known issues with by SME.
We have some known issue and guidance to work on use cases by SME
S.N
Items Action
o

Citrix Runbook
Citrix Client The correct version is 4.9 as of March 7, 2018. Only use the SCCM package and
1
Installation do NOT use manual installation.
Understandin Always understand the issue and take action. If need then forward the call/uor
2
g of Call/UOR to correct queue
html in SNOW
ticket
3 breaking Team has been instructed to paste only text into SNOW updates.
history full
view.
Citrix
application
4 Citrix application publishing tasks need to be close it asap.
publishing
tasks
Step by Step
Monthly updates of PVS/MCS document is placed on SharePoint. Team need to
guide for
5 follow the document and if they have any concern, then need to contact BI
monthly
SME for clarification.
updates
App-V
System event tickets automatically created by UC4 due to issues with App-V
publication
6 import or publication should be sent to queue BI-ITINF-WP-SOFTWARE instead
errors sent to
of SOFTWARE PACKAGING team.
wrong queue
Suspended
7 System Event Remove the suspension before passing the Calls/System Events.
Call 8206554
Consultancy
Software Meeting invites to be sent to SWO to conduct Consultancy session by checking
8
packaging outlook calendar.
request
UOR need to
9 route to Always transfer the Calls/UOR ticket to team who is working on it.
working team
VDIs
incorrectly
10 Do not give permission to user X2 account on VDI.
assigned to
x2 accounts
BaseVM need to update as per BI document. Below things need to be take care
BaseVM of:
11
updated 1) Assign the BaseVM in IAS product.
2) Seal BaseVM after updating it.

Citrix Runbook
Problem
12 ticket needed Problem tickets need to be created when multiple users have an issue.
to be created
13 Call/UOR type Change the IPK status as per the BI Service Management if needed.
VMWare
Multiples VM not need to power on at same time. It can affect VMware host. If
14 hosts load
need 10 VM onetime can be done.
issues
escalation to
System
15 Owner If any server needs to rebooted, then please contact System Lead.
instead of
System Lead
Service
Delivery Call TCS Citrix team members to explain and share the information or update on
16
Meeting the outage received in Service Delivery Call.
quality
Ticket need to be update with full information, So that other team can
17 Ticket Update
understand what we have done and it help them to work on it.
Sophos Identify the VDI which are not getting Sophos/MS updates.
18
updates Work on affected VDI to solve the issue.
Gpo change Update the below sheet whenever making changes on GPO.
19
log \\ingfs102\data1\XenDesktop\Backups\GPO\GPO%20Changelog.xlsx
Package containing only a link to a network share is not accepted.

Package This is not allowed. The package must contain all the binaries from that folder.
20 creation / http://packman/OrderDetail.aspx?OrderId=8087
Consultancy
If App-V was not working then this should go to SSCM cluster first ALWAYS.

Citrix Runbook
Please be very careful with Reply-all on emails. Some emails might have large
Reply-all on distribution lists. So think about is this really something that everybody on that
21
emails distribution list needs to know about. Only send emails to the people who
need to know or do s
Ticket
The users who have the issue cannot handle the firewall request. The firewall
Handling and
22 request has to be done by the owner of the VDI together with an update of the
legacy special
remote access risk assessment.
VDIs
Standard process to get Visual Studio installed on a Virtual OBIC - Personal

Visual Studio
BI/Contractors VDI is the following:
installation
1. Order the VDI
23 on Virtual
2. Issue a SNOW user / Order request to add 60GB D: drive and 8GB of memory
OBIC -
assigned to BI-ITINF-MYWORKSPACE.
Personal
3. Order the Visual Studio via Myshop
Request to
schedule In case there are questions / problems with the Vendor Integration guide TCS
conference needs to be available for a conference call with the external vendor to help
24
call with with getting the setup correct on the vendor side. This cannot be handled by
Vendor IT the Service Desk.
Department
Periodic
The failed desktop OS machines have to be checked once a day at the moment
Check for
because there is not automated monitoring available at this point. The XenApp
25 Failed
servers are being monitored already but VDIs are not.
Desktop OS
Please check on Citrix Director and take action on affected VDIs
machines
Check if the Outage is related with full Citrix farm or individual

Terminal
application/Server affected.
26 Server outage
Close outage for BI-IT-VIRTUAL-WORKSPACES with 0 minutes and created
handling
outages for the individual servers

Citrix Runbook
Add approval For WFM orders in consultancy and Integration, if TCS decides to go to
27 step in dedicated server or local installation, they will need to escalate to SMEs for
consultancies final approval.
Citrix
Troubleshooting need to be done and try to make application work on
Integration
28 App-V/SCCM. All troubleshooting steps need to be mention on UOR and if
Troubleshooti
need inform Shiv or Zamir before passing to SOFTWARE PACKAGING team.
ng
TCS should
check the
application
Application testing need to be done after publishing the application before
launches
29 sending to SOFTWARE PACKAGING team. Application screenshot need to be
before
paste in the email.
handover to
SOFTWARE
PACKAGING
If user not
If user doesn’t join the meeting after the schedule time, then please inform the
30 join the
user via email or skype.
meeting.
http://packman/OrderDetail.aspx?OrderId=7383
Understandin
It is not allowed to create App-V package just with a link to a network share.
31 g packaging
It is ALSO not allow doing so with SCCM package. All binaries must be in the
rules.
package in any case.
client contact If user is offline for more than 2 hours, Please send email to the user keeping
32
protocol team in CC. If user give reply then available shit engineer will contact the user.

Citrix Runbook
User tried to access SAP COPE from the "SAP Logon" in the Windows 10 Start
Menu and then received a Java error. Apparently then it was attempted to add
a server into the Java exceptions list. The right solution of this would be to tell
the user to access the SAP COPE environment through a web browser
http://myworkspace and then use the SAP COPE TSY application there. It is
SAP COPE important that the users are using exactly this published application from T-
33
access Systems to access SAP COPE because it is a special version with special
adjustments and configurations for COPE.
If you see this in tickets then please forward the ticket to the SAP COPE
Support Team BI-ITOPS-SCM-GLOBAL who will then work with the user to
direct them to the right SAP
This product is only to be used in special cases like to assigning an additional

user to an already existing dedicated / Personal VDIs or temporary test VDI.
For that you first have to get the additional user into the BIDS entitlement
using this MyShop product and then assign him/her to the specific VDI in
Studio.
Assigning a) Requirements are two or more people share the VDI.

additional b) Check that second person is added to BIDS group.
34 user to c) Assignee the VDI to the user on Citrix Studio.
Virtual OBIC - d) Raise Myshop product to order the membership in the VDI BIDS entitlement
Personal VDIs groups:
https://myshop.eu.boehringer.com/myshop/ProductDetail.asp?
ProductID=30760&NextPage=
For ordering a new VDI or access to the MyDesktop the regular MyShop
product for end-users has to be used.
Sometime with App-V package, Adobe/MS Office FTA get change on

File Type
PROD/UAT/TAT server after deploying another App-V package. Script is
Association
35 created and placed on the management server INHAS62267 (C:\Temp\Appv\
on App-V
Adobe_Word\Abobe_Word_Script.ps1). Team will run the script after weekly
servers
or if App-V server is reboot to fix FTA issue.
Citrix PVS farm Monitoring script runs every day and team will receive email
Citrix PVS
about PVS farm health checkup. Please follow the Link PVS document.
36 Farm
Monitoring
Troubleshooti Process of registering and decommissioning/scrapping VDI and Citrix server

37 ng for register Configuration Items (CI) in Command, CMDBWeb (servers only) and Patch
and wave is available at LINK.

Citrix Runbook
decommissio
n machines
in
Command/CI
MIT,
CMDBWeb
and Patch
wave
Dedicated
We can add dedicated Sever on Citrix 7.15 Farm either Windows 2016 or 2012.
38 Server adding
VDA must be installed on dedicated Server i.e. 7.15.4000(CU4)
on Farm 7.15
Troubleshooti
If user have slowness issue on VDI and Citrix application. Citrix team must
ng for
39 troubleshoot the issue from nmas tool to see the latency and slowness
slowness
problem on Citrix applications and VDI.
issue
Weekly vDisk Weekly vDisk update process for S1 and S2 Clusters. Process and
40 Update document is mention in point 6.1 in below document.
Process
The profile The issues should be troubleshoot in this way:
reset or
checking the  look in the director for the terminal server where the user has the
policy on user session
local machine  look at the terminal server performance
won’t solve  look at the application performance on the terminal server
the issue as  if the above is normal, check with the SWO if the application has
41 SNOW is performance issue on the backend.
running on
XenApp Issue us related with SNOW then it might be issue related with “Call Search"
server which windows opened which are querying the SNOW data base. Until the search
has nothing view is filled in, SNOW might not respond. In such case SNOW will be hanged
to do with and unresponsive.
local machine
policy.
Currently only one Server VIEAS00131 is assigned for BIOMESS8 application.
BioMES8 - We had word with SWO and shared information with him this criticality. SWO
42 GxP not using said application is not GO Live and load balancing is not requiring at this
LB on Citrix moment. SWO will let us know whenever require and most probably this will
be Go Love on Dec 19.
For SCCM Team should contact BI-ITINF-WPS-PC-PLATFORM for sccm related

43
support requirement.
For AppV TCS team should create new version every month and update the vdisk with
44
vDisk MTD latest patch

Citrix Runbook
This is published on CSN MyWorkspace and it’s not accessible from outside BI
SAP_COPE
45 network. If user need to have access then user must have MyDesktop then
CSN
user can access it.
Problem TCS team will update the problem Calls with the detailed updates on the issues
46 Ticket and will make sure updates are informative enough for other colleague can
handling take over the issue
WFM
handling TCS team will not close the WFM/UOR if the SWO is out of office and keep it
47 when SWO is pending till SWO is back. IF SWO is out of office for long time and we need
OOO for long inputs from their side, handling of this situation is still under discussion
time
TCS team need to perform the health check of all the v-disk in all 3 sites
(INH/NAH/SIH) and ensure the no disk have the status showing “NO Server”
If any disk found with the above status. Steps to be followed –
PVS Health Disk should be properly exported and should be replicated properly (with base
48 Check report available) on all the site stores.
handling
Once it is exported correctly and replicated correctly on all the store servers, it
should look like-
Secondly, If a new V-disk is being created, ensure that the base is replicated to
all the sites so that while creating a version base it already present.
TCS has the capability to directly open tickets with Citrix in case of critical
issues / outages also happening on a weekend when no Boehringer SMEs are
available. TCS L3s are granted this permission on the Citrix Portal.
Vendor During Weekend if there is any P1/P2/Outage and TCS needs vendor support
49
Support Case then TCS L3 can log a case with Citrix Vendor support. SME’s must be informed
via email with the case number and details.
During Weekday’s TCS L3 need to seek approval from BI SME’s for opening
case with Vendor
TCS Engineers are advised to follow escalation matrix while contacting BI
SME’s.
Escalation
50
Matrix Engineer must discuss internally in team and if support from BI SME is required
then TCS L3’s will be contacting BI SME’s. TCS L3 will cascade the information
to team.
51 PowerShell 1. Script is located at location "C:\Boehringer\scripts\

Citrix Runbook
Script created RemoveInactiveUserVDI\VDIReport.ps1" on server INHAS65751.

to remove 2. Search all machines have user ID with wildcard S-1*.
the VDIs of 3. It will not perform any action if any VDI contains user S-1* and other
users whose working IDs. We will have to remove SIDs from manually in such case.
ID have been 4. Put these filtered machines in maintenance mode.
removed 5. If these machines are already in maintenance mode then it will remove
from Active the VDI from DG, Machine Catalog, Active Directory, and VMware.
Director.
BI Service Attached is the document defining the service management process defined by
Process as BI mentioning 3 strike rules and ticket handling
52
designed by
BI TCS – BI Service Processes_Citrix
Attached is the document defining the process, which needs to be followed for
Handoff between the team and for escalations.
Handoff and
Escalation
53
process as
per BI design
New GPO
creation/ Team needs to use XD7.15 management servers (INHAS65751) to create and
54
update update the policies. Detailed document for GPO creation can be find here
process
TCS L3s have access to Citrix portal to raise a vendor support case incase of
Citrix support
major incident. Details of the same can be found here
55 Usage
Process
All P1 case with Citrix should be approved by Bogdan/Arjen/Werner
Details of GxP Incident Management Process can be found in the document
GxP Incident
Management
56
Process
Correct TCS Team is advised to only assign BIDS managed AD group while publishing
57 Entitlement applications. No legacy group should be used as per communication from Alex
assignment Marsal.
58 How to rejoin Please follow below steps:
broken VDI in
domain. 1. Go to 7.15 management server.

Citrix Runbook
2. Check the OU of broken VDI in Active Directory via DSA.msc.
3. Open the ADSI Edit tool.
4. Go to the AD path of Computer Object.
5. Right click of Computer Object and select Property.
6. In Attribute Editor menu, you will find the attribute named "ms-Mcs-
AdmPwd"
7. This attribute contains a value which the local administrator password of

VDI.
8. Login on VDI with this password and rejoin the VDI in domain.
Please keep in mind that we have URL redirection feature in place for our
master images. Please take note to always verify if URL redirection is working
every time you upgrade VDA or Receiver. Always install the receiver first
before the VDA and if you are upgrading VDA, please make sure that it does
Verify URL not upgrade the current receiver installed. If an issue arises, please perform
redirection the following task.
working fine 1. Uninstall the VDA.
59 after 2. Uninstall receiver.
VDA/Receiver
3. Install receiver.
upgrade OR
reinstall 4. Install VDA.
5. Verify in IE explored add ins option that “Citrix URL-Redirection
Helper” set to “Enabled.
Client Local
Printer not
mapping on
60 user session Please add Local Service account in Local Administrator group on the server.
after applying
GPO or Citrix
Policy
61 Problem  Problem record is open for reoccurring issues or multiple INC for same
Management issue.
in Snow  INC needs to be linked with PRB (procedure explained in problem
management guide
 the incidents linked should be put on hold and not closed while the
problem is being researched
 Problem management guide can be find here

Citrix Runbook
System Lead  http://sylwi

62 Wiki/Support  http://24x7
Team link
GPO configuration that allows to devices to register to Azure AD. Please find the
attached email for the information:
AzureAD join
63 terminal
servers
Adding Server
or resource INF-S-005 Additional Component need to be use to raise a change for adding
64
on more servers or resource in production environment.
production.
Weekend BI
If TCS SME not able to resolve outage issue on weekend, Ram satendra will
65 Escalation
contact 1TOC team to send notification to escalation group i.e BI-CITRIX-L4 to
process
get someone involved from BI SME team.
While ordering servers from non-standard Hosting locations, System lead needs
Indispensable
to provide the approval from the OPU Head of IT and the IT INF Head or
66 Server
Delegate. TCS needs to ensure the attachment for MyShop has all the approvals
Request
added before submitting the MyShop product.
Disk space clean-up needs to be performed on the OBIC with the low disk
Weekend
space. Report will be generated and saved on the remote share. Machine needs
67 space clean-
to be powered on so that the scheduled task can run and perform the clean-up.
up on VDI
For documentation Refer to link
Roles:-
 Queue manager will assign the incidents to available shift engineer
 Will answer to inbox emails in absence of engineer
 Will assign the engineer to ongoing chats/issues/escalation
 Ensure daily health check of citrix farms & shift handover
Queue Responsibilities :-
Managers  Read the ticket description and verify the KB articles linked to ticket
68 roles &  Verify L1 troubleshooting has been performed by GSD/EUO else
responsibilitie transfer the incident to concerned team for L1 troubleshooting
s  Resolve issues like maintenance mode on VDI, missing apps/desktop
etc then n there to increase response time
 High priority incidents will be directly assigned & communicated to
engineers. Engineer will perform the troubleshooting on high priority
incident. Priority can be downgraded by engineer after analysis of
impact (user count/business impact)

Citrix Runbook
KB Article Citrix team will ensure KB articles are updated & reviewed for Citrix
69
update environment. Team will review the articles every 4-6 months.
Citrix team needs to ensure CI field is updated in each ticket before closure.
CI Update
70
process
 Documents regarding the Major incident are stored at http://itinfmod .

 During weekdays TCS needs to inform available BI manager’s(Christoff
Brand, Werner Worf , Mark Kolb).
 On weekend 1TOC will notify MOD by following Xmatters process.
 How to decide whether incident will be considered as major incident or
not
Major
71 Incident
Process
72 Teams TCS engineers need to follow Teams etiquette while contacting user
etiquette  Do not type Hi/Hello and wait for customer to respond
 Do not start conversation with Hi & referring to issue of user
 Always professionally greet user and introduce yourself followed with
reason why we are contacting user. Put this in one paragraph and send to
user rather than sending each line to user
 Only call user once agreed for call and arrange translator if customer
requests
Example: -
Hello Frank
hope you are doing well today
This is (yourname) from Citrix team contacting you regarding the case INCXXX.

Citrix Runbook
Let me know if we can check the issue
1) Problem ticket needs to be updated with every action. Like if we call vendor
and not got the response then it needs to be mention in the ticket.
2) Reach vendor via Call instead of waiting for the response via mail. If vendor
ask for a log, setup a meeting with him and ask him to generate the logs in his
Vendor
presence.
communicatio
3) Connect with BI SME via MS Team channel. For any existing issue, don’t
73 n & Problem
create new chat instead of it use old chat as reply option.
ticket
4) We can use Priority support id’s for logging a case with vendor.
handling
5) If not get proper or delay response from vendor, then we need to inform
Bogdan and Werner about it.
6) For any urgent work or issue from BI SME, they can connect with L3 resource
available in the shift like Shiv, Ankur, Amit and Vishal.
7.6 Process for Weekly vDisk Update
 There is no need for a change request to install applications in QA & Prod if the orders are placed
in WFM.
 Any order pushed to Citrix Integration before Friday 11AM should be installed for QA that week.
vDisks should be closed Friday afternoon.
 Citrix team is supposed to deliver all QA orders within 24 hours of assignment by SOFTWARE
PACKAGING team.
 QA will be down for replication on Monday .
· Prod new vdisks versions will be available to users on Monday morning (after weekly reboot).
· For any high priority order on PROD, application has to be published on the same
day under early access servers and the same has to be moved to prod servers on
Monday after weekend reboots.
Document : How to update Vdisk on MTD Site
7.7 WFM Process
Please find the main point for WFM Process:
· TCS Citrix team will raise request for a new server if the server is not shared by SWO.
· If the server is shared by SWO, Citrix VDA and FsLogix has to be installed. But, before proceeding
with this, Exception approval from CItrix SME (Carlos Vilardell) is to be shared.

Citrix Runbook
· TCS Citrix team will not add legacy AD group for published application.
· If any SWO approach to BI SME/TCS Citrix team for asking installing Citrix VDA or publishing any
new application, please inform SWO to register the application first and then submit the software
registration product:
o Software Registration : https://boehringer.service-now.com/bi?
id=sc_cat_item&sys_id=621f687bdb6bb344d6638e240596196b
o Specify Software Installation: https://boehringer.service-now.com/bi?
id=sc_cat_item&sys_id=07466108db7fb744497151c6f49619b4
· Once the 2nd request is raised, SOFTWARE PACKAGING Team will connect with the software
owner and get all the details if the application will go to the dedicated server or the shared
environment.
· Share SOFTWARE PACKAGING team (i.e zsIT INF Software Packaging Services
zsITINFSoftwarePackagingServices@boehringer-ingelheim.com) address with SWO, if they have
addition question.
· Please follow the steps to provide in the How to - Manual Citrix Dedicated Server Provisioning
How to - Manual Citrix Dedicated Server Provisioning - Accenture v2.3.docx (sharepoint.com)
· For Legacy Orders, Citrix team has to sent a mail to SWO in the below attached format.
7.8 Service Management Process
Service Management Process that TCS Citrix team will follows : Link
Service Management team is responsible to ensure that TCS L2/L3 teams are
completely aligned on ITIL processes like Incident, Problem and Change Management.
Service Management team holds end to end responsibility of managing SLA (assigned to
TCS). All relevant reports and dashboards will be published by TCS Service
Management team link
7.9 Oncall Process
The Citrix team will provide support during outside-support-hours for high priority
incidents. Reference link for this is http://24x7/

Citrix Runbook
7.10 Naming Convention
Below Table describes the naming convention for objects used within the BI-IT-Virtual-
Workspaces infrastructure system. It is absolutely mandatory to adhere to this naming
convention when creating:
 Machine Catalogs (MC)

 Delivery Groups (DG)
 Published Applications
Xen App Xen Desktop

Machine catalog names must begin
with “XX_YY_ZZ_WW”
· “XX” describes the object location

(SITE)
Machine Catalogs names must begin with · “YY” describes the environment
“XX_YY_ZZ” (LVD/RVD)
· “ZZ” describes the system version
· “XX” describes the object location (SITE) (OS)
· “YY” described the system version (OS) · “WW” describes the purpose
Machine Catalog · “ZZ” describes the delivery (TYPE) (Usage)
Delivery group names must begin
with “XX_YY”
· “XX” describes the purpose

Delivery group names must begin with (Usage)
“XX_YY_ZZ” · “YY” describes the delivery (TYPE)
DG: Usage-Type
· “XX” describes the object location (SITE)
· “YY” described the relation (L2 NAME) DG description: Delivery group
Delivery Group · “ZZ” describes the environment (CI) name - VLANxxxx
Application root folders: Dedicated,
Shared
Sub root folder structure: CI Name

Application name for user: SL preference
Application Application description for support
Folder teams: CI Name

Citrix Runbook
8.0 Technical Observations

9.0 Support collaboration
For successful support execution, provider has to interact regularly with the following
support teams additionally to the defined meetings in framework agreement:
Meetings Participants Frequency Main activities

Service Delivery Selected - 2 times per Daily - Report current service
Meeting people from on Working Days status
Supplier - Present any current
outages
Defined BI
stakeholders - Take part in current
for service service information
10.0 Business Continuity Planning

The business continuity planning (BCP) is the creation of a strategy through the
recognition of threats and risks facing a company, with an eye to ensure that personnel
and assets are protected and able to function in the event of a disaster. Business
continuity planning (BCP) involves defining potential risks, determining how those risks
will affect operations, implementing safeguards and procedures designed to mitigate
those risks, testing those procedures to ensure that they work, and periodically reviewing
the process to make sure that it is up to date.
With Respect to Covid Pandemic : The resources are distributed to Various locations and
can work remotely too to provide support.
1. How long can you stay in these second sites for Level 1 and Level 2?
BCP documents are made on an assumption of 3 week for Level 2. If the situation will
prolong, TCS will communicate with a detailed plan to client in two weeks’ time.
2. Also, how long will it take to come back to the Gurugram original working location?
The moment the BCP is revoked, the teams can come back to original working location.
3. Secondary location for BCP will be enabled for service delivery in 4-6 hrs after
standard site became unavailable.
Strategy Summary – TCS has framed standard strategy for BI
This section provides a summary of the business continuity strategy and planning
assumptions that support the strategy

Citrix Runbook
Impact Level Definition High Level Continuity Strategy Recovery Location

Gurugram 4 -
There is a partial Dundahera-Twr
disruption on the E
original Facility that TCS Solutions
does not require Private
invocation of the Limited(DDC-5)
recovery location. 2nd Floor,
The critical resources will
Recovery of services is Building No. 2,
move to Training /
maintained in the Tower A, 8th &
1 Conference room in the
Primary Location. 11th Floor,
primary facility for Level 1
Examples of disruptions Building No. 1,
recovery.
include: Tower B, Info
1. Technology failure space, Gurgaon
2. Facility localized Info space Ltd.,
failure (i.e. power, fire) Sector # 21,
but occupancy is not Dundahera,
compromised Gurgaon –
122001
DDC4
There is a disruption
TCS Solutions
that only affects the
Private Limited,
original Facility making
Noida , Delhi
the facility not
Delivery Centre
operational and
(DDC-4)
invocation of the
Ground Floor,
alternative Facility is
Tower No-
required. Key resources will move to
3,Seaview
2 Examples of disruptions DDC4 for recovery in case of
Developers,
include: level 2 situations.
Infospace,Plot
1. Localized facility
No-20 & 21,
damage making
Sector- 135,
occupancy impossible
Gautambudh
2. Technology failures
Nagar, Uttar
that are not expected to
Pradesh,
be resolved to support
India.Noida-
recovery time objectives
201301
11.0 Inventory
Server/VDI inventory can be accessed via Tableau . Few legacy inventory details can be
found in the ASG Remote Desktop and Citrix AppCenter/Studio console.

Citrix Runbook
12.0 RACI
Below document defines the service support RACI for Citrix:
13.0 Vendor Information

TCS L3 team members have access to Citrix & Microsoft vendor support. All P1 cases
with vendor should be approved by Stam,Arjen <arjen.stam@boehringer-
ingelheim.com> and Stanciu,Bogdan <bogdan.stanciu@boehringer-
ingelheim.com>. TCS L3 Engineer may discuss the issue with BI SME’s before opening
medium/low priority case with vendor.
14.0 Global Operations Support Roles

Resource Name Support Roles
Vendor Contract Manager

Ritika Patni Global Support Coordinator
Ram satendra Team Lead
Dimpu Barman Shift Lead
Priyanka Das Shift Lead
Deepak Vasudevan Technical Support Agent
Utsav Bhardwaj Technical Support Agent
Rohit Mehra Technical Support Agent
Satyam Patel Technical Support Agent
Rishabh Kumar Singh Technical Support Agent
Vaishnav LokeshKumar Technical Support Agent
Ramnivas
Kshitij Nirala Technical Support Agent
Neeraj Sethi Technical Support Agent
Shambhavi Devi Technical Support Agent
Deepanshu Srivastava Technical Support Agent

Citrix Support Runbook V 2.0

Uploaded by

Copyright:

Available Formats

You might also like

Citrix Support Runbook V 2.0

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Citrix Support Runbook V 2.0

Uploaded by

Copyright:

Available Formats

Run Book – Document

Preparation and execution update Modification History

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 2 of 57

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 3 of 57

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 4 of 57

1. Purpose of the document

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 5 of 57

Citrix XenApp 6.0 farms sites are:

Citrix 7.6 farms sites are:

Citrix 7.15 farms sites are:

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 6 of 57

Citrix 1912 farms sites are:

Citrix Cloud farms sites are:

Boehringer Ingelheim EU-PRD

Production Citrix Url:

 Virtual Desktops (VDI) – pooled and dedicated

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 7 of 57

2.2. Support Structure

2.2.1. Citrix Level 2 Support Team:

Support Short description

Published applications incl. MS AppV

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 8 of 57

Support Short description

2.2.2. Citrix Level 3 Support Team:

Support Short description

Published applications incl. MS AppV

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 9 of 57

Support Short description

2.2.3. SNOW Workgroup Name

2.3. Support Features

Level Topic Additional description based on framework

Pre-announced activities at weekends (in

2.4. Support phone numbers

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 10 of 57

To get support, the following phone numbers are available:

Level Phone number for specific topics Supported languages Description

Oncall phone no +91 97116 20792

Xmatters for Primary and Secondary on call details.

3. Service Scope in detail: Level 2

Modules Description / Examples as extension to framework agreement

- Handle and resolve How-to questions regarding VDIs

- Reboot servers or restart services if required in alignment

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 11 of 57

Modules Description / Examples as extension to framework agreement

- Analyze and resolve incidents regarding untypical behavior of

- Troubleshoot and support users to solve performance issues,

- Troubleshoot and support users to solve issues regarding

Problem Identification, Handling Problems incl. Involvement of Level 3 and 3rd

Consulting Consult System Lead and other stakeholders in IT as well as

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 12 of 57

Modules Description / Examples as extension to framework agreement

Configuration management (create csn/ update / delete CIs based

Support Docu- Regularly update of Knowledge articles, Review FAQ documentation

Modules Description / Examples as extension to framework agreement

Supplier has to use Lighthouse / ReportIT to create the specific

Confidential Information – Property of Boehringer Ingelheim – Do not Distribute Page 13 of 57

Modules Description / Examples as extension to framework agreement

Problem Handling reoccurring issues as problems, if patching failures occurs