Document ID: CRI

Document Owner: PA L2 Support

Version Number: 1.0

Application Support Control Plan

Credit Risk Interface

Version 1.0
CRI Application Support Control Plan Version 1.0

Document Control Section

Version Number: 1.0

Author
Role Name
L2/L3 Production Support Analyst Venkat
L2/L3 Production Support Analyst Khushbu

Delivery Leads
Role Name
IBM Offshore Delivery Manager Anu Aggarwal
IBM Onshore Delivery Manager Subrata Mondal

Reviewer
Role Name
L2/L3 Production Lead Yogesh Agarwal

Revision History

Version # Description of Changes Issue Date

1.0 Initial Draft Version 4 Aug 2016

Page 2 of 22
CRI Application Support Control Plan Version 1.0

Table of Content

1. OBJECTIVES..........................................................................................................................4
2. Terminology and Acronyms.....................................................................................................5
3. Production support team, responsibilities and operations.......................................................6
3.1 IBM Production Support Team.....................................................................................6
3.2 Production Support Hours of Support and Shift Coverage...........................................6
3.3 Production Support Responsibilities......................................................................................8
3.1.1 Troubleshoot Environment Issues..........................................................................8
3.1.2 Environment Base lining / Data setup....................................................................8
3.1.3 Monitoring................................................................................................................8
3.1.4 Issue Management and Problem Analysis.............................................................8
3.1.5 Change management:..............................................................................................9
3.1.6 SLA............................................................................................................................9
3.1.7 Work Item Definitions.............................................................................................9
3.4 Production Support Review Meetings.................................................................................10
3.4.1 Daily review of Production Issues...............................................................................10
3.4.2 Monthly SLA Severity Review Meeting.....................................................................10
3.4.3 Bi Weekly Production Issue Review Meeting.............................................................11
4 Incident/MIM Management Process......................................................................................12
5 PROBLEM management PROCESS:....................................................................................13
6 Request Process:....................................................................................................................14
7 Root Cause Analysis Process.................................................................................................15
8 Procedures..............................................................................................................................16
8.1 Problem Reporting and Tracking................................................................................16
8.2 Problem Management Procedure................................................................................16
8.2.1 Problem Identification.............................................................................................16
8.2.2 Information Gathering.............................................................................................17
8.2.3 Assessment of Problem............................................................................................17
8.2.4 Resolution................................................................................................................17
8.3 Configuration Management.........................................................................................17
8.4 Software Release Management...................................................................................17
8.5 SQL Jobs.....................................................................................................................18
9 information Security Consideration.......................................................................................19
10 CRI Production support Contact Information........................................................................20
10.1 IBM Production Support Contact Information............................................................20
10.2 Escalation Tree (Highest level to lowest level)...........................................................20
10.3 L1 Team Information...................................................................................................21
11 Reference...............................................................................................................................22

Page 3 of 22
CRI Application Support Control Plan Version 1.0

1. OBJECTIVES
The objective of the Application Support Control Plan (ASCP) is to document the CRI
application Production Support Functions that will be supported during steady state operations.
The plan will define existing procedures required to support day-to-day production support
activities for these applications.

Page 4 of 22
 CRI Application Support Control Plan Version 1.0

2. TERMINOLOGY AND ACRONYMS

Acronyms and terminology used specifically in this document:

No. Terminology / Definition

Acronyms

1 AID Application Information Document

2 ASCP Application Support Control Plan

3 EST Eastern Standard Time

4 IST Indian Standard Time

5 SLA Service Level Agreements

6 SME Subject Matter Expert

7 UAT User Acceptance Test

8 DPE Delivery Project Executive

9 SA System Administrator

10 GFTS Global Functions Technology Services

11 CRI Credit Risk Interface

12 SN Service Now

Page 5 of 22
 CRI Application Support Control Plan Version 1.0

3. PRODUCTION SUPPORT TEAM, RESPONSIBILITIES AND

OPERATIONS
3.1 IBM Production Support Team

Name Role Skills Location

Pradip C Jain Delivery Project Project Management Gurgaon, India
Executive
Anu Aggarwal
Offshore Project Project Management Gurgaon, India
Delivery Manager
Subrata Mondal
IBM Onshore Project Management Warren, USA
Delivery Manager
Yogesh Agarwal L2/L3 Production Technical Gurgaon, India
Support Lead
Khushbu Singhal
L2/L3 Production Technical Gurgaon, India
Support Analyst
Venkat Dasireddy L2/L3 Production Technical Bangalore, India
Support Analyst
Mohit Siddhu L2/L3 Production Technical Gurgaon, India
Support Analyst
Mohit Gupta
L2/L3 Production Technical Gurgaon, India
Support Analyst
Jagdish Nayak
L2/L3 Production Technical Warren, USA
Support Analyst
3.2 Production Support Hours of Support and Shift Coverage
IBM will provide production support coverage for the CRI Application with an integrated team
located at Citi’s site in the Warren (New Jersey) and from IBM’s site in Gurgaon (India). In
order to cover the support hours outlined above, the IBM Production Support Team will be
assigned.

Shifts details are following:

L1 Support:
1. Morning Shift IST
2. General Shift IST
3. Evening Shift IST
4. Night shift IST

L2 Support:
1.India Morning Shift
2.India Evening Shift

Page 6 of 22
CRI Application Support Control Plan Version 1.0

1. US General Shift

Notes:
A) ‘On-site Support’ to be provided from US
B) CRI production changes are done during Green Zones or additional green zone can be taken
with the prior approval of business
C) Coverage on Holidays – the IBM Production Support Team will cover the local shift on an
‘on-call’ basis. Generally, weekend support will be provided by the IBM resources on an on-call
basis for Severity 1 and 2 problem tickets and deployment releases or for Application checkouts
only. L1 can be contacted for support activities

Support Coverage:
a. L1 support coverage will be 24 x 7 and L2/L3 support coverage will be 19*5 days during
Business hours.
b. IBM will also provide support on Saturday/Sunday if requested by the business since it is
a business day in some countries

Below mentioned diagram, shows the Support hours for the CRI application

L1 team will be available 24 * 7.

L2 team’s support hours are depicted in Figure below: -

Shift# EST IST

Shift1 11.00 PM – 08.30 AM 08.30 AM – 6.00 PM
Shift2 04.00 AM – 01.00 PM 01.30 PM - 10.30 PM
Shift3 09.00 AM – 06.00 PM 06.30 M – 03.30 AM

Page 7 of 22
CRI Application Support Control Plan Version 1.0

3.3 Production Support Responsibilities

All Production requests will be tracked through Service Now tool. IBM will then prepare a high-
level breakout of the tasks to be completed for the issue, assign severity levels
The IBM Production Support Team will support the CRI Application as mentioned in section 3.2.

3.1.1 Troubleshoot Environment Issues

a. Support UAT testing and resolve issues in UAT, if the issue is environment related. If the
issue is development related, code changes can be made on approval from on-site SMEs.
b. Support environment related issues in new client sites, if applicable.
c. Support all issues relating to production in accordance with the agreed SLA.
d. Support UAT issues as per request.

3.1.2 Environment Base lining / Data setup

a. Contact development team for issues relating to production for CRI.
3.1.3 Monitoring
a. Monitoring Autosys alerts for the application and reporting issues.
b. Monitoring DB alerts.
3.1.4 Issue Management and Problem Analysis
a. Responsible for reviewing and resolving Production incidents by checking through
logs/screen share.
b. Respond to Production issues and Alerts (Autosys, BPM, within SLAs, as defined in
Contract (SOW)
c. Resolving/escalating data related issues by checking through logs. L2 team has MST
access
d. Identify workarounds in support of Production Issues where possible
e. BO related issue. Check through screen share with user. Try to replicate issue in BO for
standard and custom reports and advise L3 team after initial analysis.
f. If applicable, escalate issues to development team for code-fix and follow-up for
resolution
g. IBM PS Team will notify Citi and IBM's Project Manager in accordance with the Service
Levels defined in the contract.
h. Responsible for creating and updating run books for issues and deployments. Runbooks
will define the issue and resolution so that when issue occurs again, we can refer the runbook.
i. IBM L1 and L2 PS Team will accept tickets only from CRI Business users.

Page 8 of 22
CRI Application Support Control Plan Version 1.0

j. Responsible for updating FAQ documents

3.1.5 Change management:

1.) Following up with the implementation team to start the deployment of change request
only for PROD, send release notes.
2.) Check DB logs, jrules, bpm screenshots) needs to be done by L1/L2 by checking emails
that get sent out as part of deployment. L1/L2 should request jrules/bpm deployment
screenshots from STE team. Coordinating COB activities with various teams.

3.1.6 SLA
As defined in Contract (SOW)

Page 9 of 22
 CRI Application Support Control Plan Version 1.0

3.1.7 Work Item Definitions

This tabular representation defines all the Work Items which the CRI PS team will handle when
delivering on all the responsibilities listed above:
Work Item Definition
Incident  Any problem reported by the Service Now
 Any issue reported through the notification directly from the
application.
 Issue has priority: Sev 1, Sev 2 and Sev 3. (Medium, High ,Low)
 SLAs are clearly defined for Response & Resolution times.
 Please document SLA information here.
Priority Incident response time Incident resolution time
Priority 1 Within 5 minutes of alert the Service All Pri 1 Incidents shall be
Provider should start acting on the
alerts/failure. resolved within 4 hours

Priority 2
Within 15 minutes of alert the service All Pri 2 Incidents shall be
provided should start acting on the
alerts/failure. resolved within 8 hours

Priority 3 Within 1 hour of alert the service pro- All Pri 3 Incidents shall be
vided should start acting on the alerts/
failure. resolved within 2 business days

Priority 4 Within 2 hours of alert the service pro- All Pri 3 Incidents shall be
vided should start acting on the alerts/
failure. resolved within 5 business days


Problem  Any problem, issue or service request reported by Citi internal
teams, e.g. UAT team, Development team, etc. and not routed through the
Monitoring team. There are no SLAs defined, but Support team will
capture the expected resolution time from the requestor.
 . As discussed, problem tickets due date will be provided by the
development team along with the UAT deployment date. We will create a
task of UAT deployment mentioning it’s due date there. L2 will follow up
with the development team on regular basis on status of issue in problem
ticket. The problem ticket will remain in L2 queue.
RCA Root Cause Analysis (RCA) is the activity performed by L2 team in case
of a problem. This has to be done to fix a repeatedly occurring issue.
3.4 Production Support Review Meetings
3.4.1 Daily review of Production Issues

Page 10 of 22
CRI Application Support Control Plan Version 1.0

Purpose Review of Production issues

Location IBM Kolkata
Frequency Daily
Chairperson Anu Aggarwal
Minutes by Team Members
Attendees L2/L3 Team
Agenda Production Issues
Distribution list *GT GLOBAL ETS CRI Support L2/L3

3.4.2 Monthly SLA Severity Review Meeting

Purpose Meeting between IBM team and Application SME/Lead to discuss

SLA and severity related issues
Location TBD
Frequency Monthly
Chairperson L2/L3 Team
Minutes by Off-shore team
Attendees Off-shore team and Application SME/Lead
Agenda Discussion on SLA and severity related issues
Distribution list *GT GLOBAL ETS CRI Support L2/L3

3.4.3 Bi Weekly Production Issue Review Meeting

Purpose Meeting to discuss Production Issues .

Location TBD
Frequency Bi-weekly
Chairperson L2/L3 Team
Minutes by Off-shore team
Attendees Off-shore team and Application SME/Lead
Agenda Discussion on production issues.
Distribution list *GT GLOBAL ETS CRI Support L2/L3

Page 11 of 22
CRI Application Support Control Plan Version 1.0

4 INCIDENT/MIM MANAGEMENT PROCESS

This section details the Issue Management processes that will be used for the
Production support. The Issue Management process is an ongoing process concerned with
addressing any issues raised in the Production by the Key Citi users / Representatives.

Kindly Refer to the “Citi_GPA_Incident_Management_Process_v2.0.doc” for the incident

management process. Document Location is as follows:

https://globalconsumer.collaborationtools.consumer.citigroup.net/sites/CITS/GPM/PSOptimization/SS
%20Deliverables/Forms/AllItems.aspx?RootFolder=%2Fsites%2FCITS%2FGPM%2FPSOptimization%2FSS
%20Deliverables%2FProcess
%20Documents&FolderCTID=0x01200084B3EE0B4856EC4DA5B1E4C793766BF5&View=%7b9FAE4F2D-
6DC6-4DBA-BEDC-139178E3BC08%7d&InitialTabId=Ribbon
%2EDocument&VisibilityContext=WSSTabPersistence

Page 12 of 22
CRI Application Support Control Plan Version 1.0

5 PROBLEM MANAGEMENT PROCESS:

Kindly refer to the “Citi GPA Problem Management Process v1.2.doc” for the problem
management process. Document location is as follows:

https://globalconsumer.collaborationtools.consumer.citigroup.net/sites/CITS/GPM/
PSOptimization/SS%20Deliverables/Forms/AllItems.aspx?RootFolder=%2Fsites%2FCITS%2FGPM
%2FPSOptimization%2FSS%20Deliverables%2FProcess
%20Documents&FolderCTID=0x01200084B3EE0B4856EC4DA5B1E4C793766BF5&View=
%7b9FAE4F2D-6DC6-4DBA-BEDC-139178E3BC08%7d&InitialTabId=Ribbon
%2EDocument&VisibilityContext=WSSTabPersistence

Page 13 of 22
CRI Application Support Control Plan Version 1.0

6 REQUEST PROCESS:
Kindly refer to the “Citi_GPA_Request Management Process v1.5 .doc” for the User Request
flow process.

Kindly refer to the “Citi GPA Service Request Management Process v1 1 .doc” for the Service
Request flow process.

Document location is as follows:

https://globalconsumer.collaborationtools.consumer.citigroup.net/sites/CITS/GPM/PSOptimization/SS
%20Deliverables/Forms/AllItems.aspx?RootFolder=%2Fsites%2FCITS%2FGPM%2FPSOptimization%2FSS
%20Deliverables%2FProcess
%20Documents&FolderCTID=0x01200084B3EE0B4856EC4DA5B1E4C793766BF5&View=%7b9FAE4F2D-
6DC6-4DBA-BEDC-139178E3BC08%7d&InitialTabId=Ribbon
%2EDocument&VisibilityContext=WSSTabPersistence

Page 14 of 22
CRI Application Support Control Plan Version 1.0

7 ROOT CAUSE ANALYSIS PROCESS

This section documents the RCA Process followed by Production Support team.

Trigger for performing RCA:

RCA is performed for an issue if any of the following trigger is satisfied:
a. The issue is identified as a recurring issue
b. The issue is of high impact in nature
c. Issue is due to a miss in process

Detailed Steps:
 Based on the trigger, Production Support team records the entry in to the RCA log in the
"IBM PS Issues and Daily Activities.xls" sheet.

 Broadly RCA issues can be broadly classified under following major categories for
resolution:
Root Cause Category Meaning
Issues raised due to front end problems (like: application behavior,
Application Problems configurations, known applications issues )
Ticket resolved by providing support to existing user like screen
User Support sharing, navigation steps or workaround)
Break/Fix Issues were resolved with some kind of code/configuration change.
Other Application specific issues related to customizations,
Other mappings or duplicate issues.
Withdrawn when incident is converted to request
Job Restarted/Corrected Autosys job rerun after taking the required action.
Application Known Error Performance issues of programs.
Cancelled Incident is auto-closed after 5 days
Capacity Issue Incidents related to hardware or software capacity
Data Management Issues related to data/database
Hardware Support different hardware issues
Network Support Network outage/configuration issues
Server Maintenance Server upgrade/maintenance related issue
System Maintenance System maintenance activities
Application/Job Alerts(Non-
Issue) No action Taken, enhancement required etc.

Page 15 of 22
 CRI Application Support Control Plan Version 1.0

8 PROCEDURES
8.1 Problem Reporting and Tracking
Tools used to report problems:
Tools Location Owner Brief Description
(Maintainer)
Service Now https:// Citi Service Now is used to lock a
servicemanagement.citigro ticket
up.net
Email CTO Global PA Level-1
Support Team
Phone Call On-call weekend support.

IBM L1 and L2 PS Team would accept tickets only from CRI Business/CRI Business U.S.
(Mentioned in section 10.3)
For tickets originating from other users ( the non-designated contacts) related to users not able to do
bookings or find space, IBM L1 and L2 PS team will inform the users to route the ticket to CRI
Operations Service Now group below: -
Service Now Group Name: CTO GL 2T CRI L2 Team, CTO GL 2T CRI L3 Team

8.2 Problem Management Procedure

8.2.1 Problem Identification
a. Upon incident notification or business originated e-mail alerting of the SN incident assigned,
the initial receiver of incident should review if the ticket has been assigned to a proper resolution
group (CTO GL 2T CRI L2 Team-Prod Support Desk) and originated by the appropriate
business contact. Please refer to CRI Business (Section 3.2.7) to validate the designated contacts.
In case incident was originated by the non-designated contacts, ticket needs to be transferred to
CRI Operations SN group (SN Group Name: CTO GL 2T CRI L2 Team, CTO GL 2T CRI L3
Team.

b. Incident status must be altered to “acknowledge” followed by an email to incident

initiator that incident is being looked at.

8.2.2 Information Gathering

Page 16 of 22
CRI Application Support Control Plan Version 1.0

Below is the suggested list of questions to be sent to ticket initiator in order to identify the
problem resolution flow. IBM L1 team to send relevant question to the user.

1. Is the issue impacting client (CRI) only/ virtual (CRI) only/ both virtual (CRI) and client
(CRI)?
2. Are there problems to login?
3. Are there any error messages generated by application?
4. When did this issue start?
5. How many users are getting affected?
6. In case of slowness, is the application consistently slow? If not, does the slowdown
occur during specific processes/functions/screens?
7. What are the delays?
8. Can users replicate the issue?
9. Are any workarounds available?

8.2.3 Assessment of Problem

a. Based on Information Gathering (Section 8.2.2), data should be analyzed and matched
with severity levels defined as per SLA (see below) to determine criticality/severity of the
reported problem.

Severity 1 (High) – Application is down due to database or webserver is not

responding or down, resulting in a critical impact on the operation. No
workaround is available.
Severity 2 (Medium) – Application operates but its operation is severely restricted
where each process/function/screen takes over 1 minute or unable to complete and
MULTIPLE users are affected. No workaround is available.
Severity 3 (Low) – ALL or SOME operate with limitations that are not critical to
the overall operation. For example, a workaround forces a user to use a time
consuming procedure to operate the system; or removes a non-essential feature.
Severity 4 (Non-Support Tickets) – Non-support/non-incident tickets (e.g.,
application enhancements)

8.2.4 Resolution
Based on Information Gathering (Section 8.2.2), resolver must determine which module is
getting affected in order to follow the proper resolution route

8.3 Configuration Management

(NA)

8.4 Software Release Management

(NA)

Page 17 of 22
CRI Application Support Control Plan Version 1.0

8.5 SQL Jobs

(NA)

Page 18 of 22
CRI Application Support Control Plan Version 1.0

9 INFORMATION SECURITY CONSIDERATION

Citi equipment and services (telephone, e-mail, voice-mail, Blackberry, SMS Messaging, Instant
Messaging, computer and remote access systems) are primarily for business purposes. You may
not use these systems in a manner that could be harmful or embarrassing to Citi, or in a manner
that would be illegal or would violate any Citi policies. If your business allows for any personal
communications using these systems, they must be kept to a minimum.
 Always be mindful when composing e-mail messages. Write e-mail messages with the
same degree of care, judgment and discretion as in all other forms of written business
communications.
 You may not disclose personal, confidential or proprietary information regarding Citi, its
customers, or its staff to any unauthorized person through electronic or other means.
 You must have approval from your supervisor and public affairs officer before you post
anything to a personal blog or website that purports to represent Citi or Citi’s businesses in any
way (unless otherwise prohibited by business policy).
 You may not use Citi’s equipment, services or facilities to access personal e-mail
accounts (such as Hotmail or Yahoo! Mail) or non-business related social networking sites (such
as Facebook or MySpace).
 You must never use Citi systems to view, download, store, transmit or exchange
electronic images, video, sound recordings or text of a sexually explicit or provocative nature, or
containing ethnic slurs, racial epithets, hate speech or any other material of a harassing, offensive
or lewd nature. This includes transmitting or exchanging chain letters, “jokes,” pictures, videos
or other communications and stories that are harassing, demeaning or offensive to any individual
or group.
 Citi servers may not be used for the unauthorized downloading of, or use, of any
copyrighted or unlicensed material, including music, unlicensed software, copyrighted images,
video or printed material.
 Use of Citi’s electronic communications resources and equipment must comply with all
applicable laws and the terms of use of any websites accessed.
 Failure to comply with the above and/or with the Citigroup.net Terms of Use is subject to
disciplinary action, up to and including termination of employment.
 Additional information on the use of electronic communications can be found in the Citi
Code of Conduct. Please note that there may be additional requirements and/or restrictions. for
your business or sector

Page 19 of 22
CRI Application Support Control Plan Version 1.0

10 CRI PRODUCTION SUPPORT CONTACT INFORMATION

10.1 IBM Production Support Contact Information
Office
SNo. Name Role Mobile # Email Address
Telephone #
Offshore
1 Pradip jain - 91-9810329970 pradip.jain@citi.com
DPE
Onshore +1-732-586-
2 Subrata Mondal Subrata.mondal@citi.com
DM 8442
Offshore
3 Anu Aggarwal +919810920470 anu.aggarwal@citi.com
DM
yogesh.kumar.agrawal@citi.
Support
4 Yogesh Agarwal +919999173007 com
team lead
Support
Khushbu khushboo.singhal@citi.com
5 team +919811722085
Singhal
member
Venkateswarareddy.dasiredd
Venkat Team
6 +919886700773 y@citi.com
Dasireddy member
Team +919999461269 mohit.siddhu@citi.com
7 Mohit Siddhu
member
Team mohit4.gupta@citi.com
8 Mohit Gupta +919911075200
Member
Team jagdish.umesh.nayak@citi.com
9 Jagdish Nayak
member
10.2 Escalation Tree (Highest level to lowest level)

Name Role Email Address

Pradip C Jain Offshore DPE pradip.jain@citi.com
Subarata Mondal Onshore Delivery Manager Subrata.mondal@citi.com
Offshore Delivery
Anu Aggarwal anu.aggarwal@citi.com
Manager

Page 20 of 22
CRI Application Support Control Plan Version 1.0

10.3 L1 Team Information

Distribution List CTO Global PA Level-1 Support Team

1-855-241-3505 (USA)
Toll Free Number
91-124-4347880 (INDIA)

Support Hours 24 x 7

Resolver Group CTO GLOBAL PA L1 SUPPORT DESK

Page 21 of 22
CRI Application Support Control Plan Version 1.0

11 REFERENCE
IBM Application Information Document (AID) for CRI.

Page 22 of 22