Statement of Work – Required for Optimization and Tuning of Integrated HRMS and

Finance Based ERP Application.

Solution Landscape:

Major Application Components:

 Oracle EBS (Finance, HR and Payroll)
 Oracle Database 12c
 Oracle AM + OID LDAP
 Liferay Portal
 NewGen – OminDoc – Document Management System
User Base
 Nine Lakh End Users of Type 1 primarily working on Liferay Portal Application and
engaging Oracle DB
 Seven Lakh End Users of Type 2 primarily working on Liferay portal application
 Core Application Users Type 3 – 5000 + primarily working on Oracle EBS Application
 Core Application Users Type 4 – 80000+ primarily working on Oracle EBS Application
User Access via Network
 295 Offices containing core users accessing through MPLS Line
 18000+ offices of Core Users accessing through MPLS and internet line combination
 9 Lac end users accessing primarily though internet line connection
Infrastructure Setup
 Data Center, Near Line Data Recovery Center, Data Recovery Center
 Replication connectivity between multiple Data Centers @ 45 Mbps.

Oracle Stack Solution Setup

Database Servers:
Two nodes with 64 cores/node and 1149 GB of memory/node
RAC setup, ASM Disk, HP-UX O/S, HP Superdome servers
EBS Application Server
4 EBS nodes with 24 Cores each and 288 GB memory in each node
HP-UX O/S on HP Superdome Servers
4 Concurrent Managers setup, 1 in each node for batch processing
48 OACOREs setup, 12 in each node

Performance Requirements:
 Meet 15000 concurrent users in Oracle EBS App layer, DB Layer, as well as in Portal layer with
following numbers of concurrency:
o Core users Type 4: 2000 concurrent users, scalable up to 3000 users in 5 years
o Core users Type 4: 10,000 users (non-peak period) & 20,000 users (during peak period)
o Non-core, End Users – Type 1: 2 lakh users, scalable up to 4 lakhs users in 5 Years
o Non-core, End Users – Type 2: 1 Lakh users, scalable up to 3 Lakh users in 5 Years
 RTO = 15 minutes

Sensitivity: Internal & Restricted

  RPO = Zero – No Data Loss
 CPU, Memory and Disk Space Utilization < 65% at any time of all Servers.
 Query (Majorly Reports) Response Time- generated on 256 kbps internet bandwidth:
o Predefined Reports:
 Peak hours < 20 Seconds
 Non-peak hours < 10 Seconds
o Simple query based adhoc reports:
 Simple repots: 30 seconds
 Medium Complexity: 60 seconds
 High complexity: 180 seconds
 Application Response Time – transmitted with 256 Kbps Internet connectivity:
o Upto 500 Kb – 5 Seconds
o 500 Kb to 1 Mb – 10 Seconds
o 1 Mb to 3 Mb – 30 Seconds
o 3.01 Mb to 5 Mb: 60 Seconds
o 5.01 Mb to 10 Mb: 120 seconds
o Greater than 10 Mb: 150 seconds
Current Status
 Some sites are live and some are in transition stage
 Active concurrent users hovering around 3000 currently. Target to reach 15000 concurrency
 Oracle DB RAC:
o CPU utilization average: 30%-50% in non-peak period with 500 to 800 concurrent user
load, and 60%-90% in peak period with 1200 to 2000 concurrent user load
o Memory Utilization: 60% to 70% constant during working hours
 All Security and Mandatory PSU packs till Jan 2020 applied in DB layer.
 Regular House Keeping activities performed – Rebuild Index, Gather Stats (30% Sampling) etc.

Challenges:
 Performance degradation of application typically during Month end:
o Post payroll processing and salary bill preparation process.
o Generation and Viewing of Pay Statement Report by multiple users simultaneously; if it
goes beyond 6000 requests, then concurrent manager crashes and bounce needs to be
taken
o Pay elements entry rectification followed by “Mark for Retry” Option, are huge in
numbers based on business utilization patterns here.
 Huge Archive Log Generation close to 10GB per day and subsequent Transmission to DR and
NLDR by ODG taking long time.
 Data Growth in Oracle Database is huge, which needs to be optimized. Some tables like Payroll
are growing with 1 crores records per month.
 Expectation of Real Time Data reflection by the customer, post changes in payroll entry while
concurrent manager configured to run in scheduled batch mode needs 30 minutes to 1 hour to
run and provide the desired output, in order to manage all requests via 4 Concurrent Managers
without crashing them
 Maintain CPU, Memory Utilization within 65% for all Servers is difficult in peak hours.

Sensitivity: Internal & Restricted

  Delay in Archive Log transfer affects RPO and RTO requirements.
 OA Cores in EBS going to Warning Stage while few of them going to non-responsive stage, during
peak hours.
 More than 1000 core users hitting payroll table simultaneously for their office payroll activities
like re-run of pay roll or pay statement generation, etc. is creating latch events and locks in the
table thereby consuming high CPU, wait times, and sometime even needing full bounce of server
to get out of it.

Work scope:
 Meet the Performance parameters for EBS payroll processing of 15000 concurrent users with 20
pay elements per employee, and 100000 employees on an average per hour payroll processing,
within the stipulated time intervals of application response time mentioned in section above
 Meet the performance parameters for Bill generation post payroll run with 15000 concurrent
users creating/approving bill transactions, for 100000 employees on an average per hour, within
the stipulated time intervals and application response time mentioned in section above
 Optimization of Thread spawning to control latch events, and locks, and keep CPU utilization
under 65% even during peak hour run, with 15000 concurrent users using payroll and bills
transactions combination.
 Optimizing DB and EBS Parameters for better performance of the Application to meet target
15000 concurrent user load in Oracle EBS transactions within the given response times in section
above.
 Analyzing the enterprise solution architecture setup here, to achieve the performance target of
15000 concurrent users in payroll and bills transactions at any given second with less than 65%
CPU and Memory utilization in DB and App layer, by
o Verifying and optimizing EBS and DB architecture of Oracle EBS setup
o Verifying and optimizing WebLogic layer architecture and parameter setting
o Verifying and optimizing the portal layer architecture and parameter setting
 Analyzing the network bandwidth setup here for DC MPLS line, DC internet line and optimize how
to meet the application response time and Query response time given in section above within the
stipulated time and data transmission size
 Analyzing the network bandwidth for DC-DR replication line, and optimizing the setup here and
Archival log file generation and writing time window of redo log files, so that RTO of 15 minutes is
met, as well as number of redo occurrence is not increased substantially in order that CPU
utilization of server does not go up to impact end user experience.
 Analyze the server current sizing and utilization pattern and server health parameters during
peak hour here, and recommend the sizing for 15000 concurrent users during peak hour on
Payroll and bill transactions combination.
 Analyze the reports query written to ensure that SLA parameters of Query response time is met,
as mentioned in section above on 256 KBPS line and with the data transmission specified.
 Support and ensure that the performance optimization activities done is sustainable through
demonstration for 3 months window with the target load of 15000+ concurrent users processing
9 Lac employees and 6 lac pensioners during month end peak time period of payroll and bill
processing

Sensitivity: Internal & Restricted