Night

NIGHT
MONITORING
TASKS
 Contents Page

Night Monitoring and Batch Running…………………………………………………………………………………....2

1.Monitoring EPS…………………………………………………………………………………………………………………..2
a.Event Monitoring………………………………………………………………………………………………………2
b.Interface Status………………………………………………………………………………………………………..3
c.Checking SAF……………………………………………………………………………………………………………..4
2. Offline…………………………………………………………………………………………………………………………..……5
a. Stopping automatic Balance Refresh……………………………………………………………………..….…5
b. Manual Balance refresh (partial)…………………………………………………………………………..…..…6
c. Changing to offline mode………………………………………………………………………………………..……7
3. Changing back to online mode…………………………………………………………………………..………………9
4. Automatic Balance refresh………………………………………………………………………………………...……14
5. Issuer……………………………………………………………………………………………………………………………...14
5.1. Card Request Processing…………………………………………………………………………………….………14
5.2. Generating transaction & balance report of all prepaid cards daily……………………….……16
6. Extract PTLF &TLF……………………………………………………………………………………………………………18
7. Batch Running………………………………………………………………………………………………………………...20
7.1 ACQuirer batch part 1………………………………………………………………………………………………..20
7.2 Interchange batch……………………………………………………………………………………………………..21
7.3 ACQuirer batch part2…………………………………………………………………………………………………22
7.4 Issuer batch……………………………………………………………………………………………………………….23
7.4.1 Streaming issuer batch…………………………………………………………………………………….....23
8. Extract payment order……………………………………………………………………………………………..……..25
9. Commission…………………………………………………………………………………………………………………….26
10. TLF Extraction for Financial Report…………………………………………………………………………..……..27
11. Monitoring report…………………………………………………………………………………………………...……28
12. Timing……………………………………………………………………………………………………………………………28
13. Speeding up CMM batches tips……………………………………………………………………………………..29
14. Pay attention to………………………………………………………………………………………………………..………30
15. Simple troubleshooting for common errors………………………………………………………………..……31
15.1 Tips for failure of job STM015xx & STM002xx issuer jobs in the nightly batch…………….32
15.1.1 Issuer batch failure at stream able job STM015xx jobs (STM01501, STM01502… etc.)…32
15.1.2 Issuer batch failure at streamable job STM002xx jobs (STM00201, STM00202…etc.) …33
15.2 When there is a plastic bulk issue request & if job G00S068B is failed…………………………..34
15.3 Every time please do the following while you are running the nightly issuer batch……….35

1|Page
Night Monitoring and Batch Running
1. Monitoring EPS
a. EVENT Monitoring
i. evt

a. When monitoring event it is better to use MOBA . To see event by color.

b. And take action when out of ordinary events come especially regarding IS, HSM and
T24.
c. Check the below event frequency if it is high. T24 or Eth switch are not responding
transactions are timing out.

External request timed out. Term ID AGD00085

 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><Event
xmlns="http://www.aciworldwide.com"><DateTime>2018-10-19T16:55:35.064+03:00</
DateTime><TaskID>0011206770</TaskID><ThreadID>0000000772</
ThreadID><UniqueEventID>1</UniqueEventID><Generator>CBE1.IS1
</Generator><Severity>Warning</Severity><EventText>INTFETHQ: External request
timed out. Term ID AGD00085 , Seq Num 2614
.</EventText><SubsystemID>2018</SubsystemID><EventNumber>900</EventNumber>
</Event>
 
d. Please give attention for this kind of events. Errors related to IS and timer are critical.

ii. essl
a. When monitoring esstatus keep your eye on queue which are queuing. Especial IS.
b. Check all process are running.
c. For example any spdh or ncr nof is queuing. The nof need restart

2|Page
b. Interface status

i. T24 transaction Performance

a. Approved should be increasing
b. Timeout should not be increasing

3|Page
c. Checking SAF
i. Run /aci_oltp/b24eps/tools/count_saf_main

ii. See result example below. All counts should be less than 20. If not their might be
SAF issues. As first remedy you can restart saf.

/aci_oltp/b24eps/tools
[prod_eps_oltp_62]:-> count_saf_main
2018-11-21T15:56:24.546+03:00 dci.exe(Build:03MAR2017_V02R0.1, PID:16449698) started.
DALCI - Version 2.01.0 SP6

Authenticating user MinteSysAdmin...

User MinteSysAdmin was successfully authenticated with Full Update Access.
count in saf_t24;

8 rows found in: SAF_T24.

count in SAF_BANKNET;

0 row found in: SAF_BANKNET.

count in SAF_CUP;

0 row found in: SAF_CUP.

count in SAF_INTF_ETH;

1 row found in: SAF_INTF_ETH.

4|Page
 count in SAF_INTF_EXTRA3;

2 rows found in: SAF_INTF_EXTRA3.

count in SAF_MDS;

1 row found in: SAF_MDS.

count in SAF_T24_B;

0 row found in: SAF_T24_B.

File: count_saf
- commands processed: 7
- elapsed time: 0.011 seconds

2018-11-21T15:56:27.166+03:00 dci.exe Entered exit handler.

2018-11-21T15:56:27.167+03:00 dci.exe Exit handler completed, process exiting.

you have mail in /usr/spool/mail/ib24

if counts greater than than 20 just do

stopsafm (use your credential to authenticate )
runsafm

2. Offline
a. Stopping automatic Balance Refresh

If the offline should be before 10:00 PM or 11:00 PM. The automatic refresh should be stopped to do
the refresh manually before 07:30 PM to speed back things.

Steps to stop automatic refresh or any scheduled script on crontab

i. Run crontab –e on 10.1.733 you will see the below and takes you to vi mode.

5|Page
 ii. Go to 30 19 * * * /aci_oltp/b24eps/bin/getPBal.ksh line press dd to delete that line
press esc then :wq press enter
iii. Run crontab –l to check if the line removed
iv. Stopping automatic refresh done
v. When the time is passed like when it is 07:35 on the server time you can revert this
change.
vi. Run date command to check the date. If it is passed 07:35 or 19:35
vii. Run crontab –e
viii. Go to the last line press o paste 30 19 * * * /aci_oltp/b24eps/bin/getPBal.ksh press
esc then :wq press enter
ix. Run crontab –l to check if the line add back
x. Thanks
b. Manual Balance refresh (partial)

a. T24 file generation will start at 19:30

b. The File most probably will we complete 20:00. To check

c. From /aci_oltp/b24eps/balanceRefreshFolder/WithOutHeader directory

d. sftp epssftp@10.3.9.54

e. Cd /t24/t24live/bnk/bnk.run/BASE24.BP/

f. If you find a file check the time stamp it should be todays date and time around
20:00.

6|Page
 g. If the time is as expected. sftp the file by executing :- get <the_file_name>

h. After the file ftped start partial refresh. From $BIN directory Execute:-
partialpbalrfshfromt24

i. Check refer1.dat file is created

j. After the file refresh done. Execute:- warmboot_pbal_rfsh to warmboot. This will
take around 2 hours. Check the prfrpt file in $B24OLTP directory. By running: - tail
–f prfrpt you should see “Partial Refresh completes.”---- done

k. Check refresh time stamp is updated with today’s date. System operation > positive
balance

c. Changing to offline mode

l. Carefully Change t24 primary destination routing to STIP exactly like below

(Just save, do not warmboot)

m. WDL

7|Page
 n. PUR

o. BAL

8|Page
 d. Stop t24 and eth nof file

p. stopicexs.old cbe1_hi93_t24.nof && stopicexs.old cbe1_hi93_t24_2.nof &&

stopicexs.old cbe1_hi93_t24_3.nof && stopicexs.old cbe1_hi93_t24_4.nof &&
stopicexs.old cbe1_eth.nof
q. To check that all t24 icexs process are down execute: - ps -ef | grep -i t24 there
should be none.
2. Now warmboot the routing changes.
3. Restart IS. Now we are offline.
4. Check advice are stored on saf data source. If the count is increasing it is good.
a. rundalci
b. count in SAF_T24_B;

3. Changing back to online mode

From offline to online

1. Start only cbe1_hi93_t24_3.nof. by executing: - r cbe1_hi93_t24_3.nof cold

2. Send logon check if it up.

9|Page
 3. Check advice are forwarded from saf data source. If the count is decreasing it is good.

a. Rundalci

b. count in SAF_T24_B;

4. Check advices are approve on t24 side. If it is approving it is good.

a. cd $LOGS

b. sftp epssftp@10.3.9.54

c. cd /t24/t24live/tcs_atm_B24/log

d. ls –ltr

e. get atm_monitor.log

f. quit

g. tail –f atm_monitor.log

h. Check if sample 1230 response is approved.

5. If advice are forwarding and t24 is approving. Carefully change t24 primary destination
routing to t24 exactly like below(Just save, do not warmboot)

a. BAl

10 | P a g e
 b. PUR

c. WDL

11 | P a g e
 6. Start the remaining icexs t24 processes by executing:- r cbe1_hi93_t24.nof cold && r
cbe1_hi93_t24_2.nof cold && r cbe1_hi93_t24_4.nof cold && r cbe1_eth.nof cold

7. Send logon for both stations

12 | P a g e
 5. Now warmboot the routing changes.

6. Restart IS. Now we are online.

7. Check transactions are approve on t24 station 1 side. If it is approving it is good.

a. cd $LOGS

b. sftp epssftp@10.3.9.54

c. cd /t24/t24live/tcs_atm_B24/log

d. ls –ltr

e. get atm_monitor.log

f. quit

g. tail –f atm_monitor.log

h. check if sample 1210 response is approved.

8. Check transactions are approve on t24 station 1 side. If it is approving it is good.

a. cd $LOGS

b. sftp epssftp@10.1.9.46

c. cd /t24/t24live/tcs_atm_B24/log

d. ls –ltr

e. get atm_monitor.log

f. quit

g. tail –f atm_monitor.log

h. check if sample 1210 response is approved.

9. Check all advice are forwarded from saf data source for the last time. If it is zero it is good.

a. Rundalci

b. count in SAF_T24_B;

Done. Thanks

13 | P a g e
 4. Automatic Balance refresh

Check around 1:40 refresh started on esstatus RFSH should be queuing.

Check $B24OLTP/prfrpt report around 4:00

Eg, tail -20 $B24OLTP/prfrpt

5. ISSUER

5.1 Card request Processing

This is the first thing to do on the night job before running the daily batch

Things to do before submitting Daily Issuer Batch:

 
At the Evening @01:00 local time,
1. Open 10.1.7.170 & Select Issuer Region
2. Go to cd $RUNDATA
ls
you will find an Executable file: get_card_files.ksh
run this file like the below:
 
[ESSISSCBEPD]#pwd
/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files
[ESSISSCBEPD]#get_card_files.ksh
 
3. Now Go to Card_files Directory which is on the RUNDATA
(/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files)
 
[ESSISSCBEPD]#pwd
/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files
[ESSISSCBEPD]#cd Card_files
[ESSISSCBEPD]#pwd

14 | P a g e
/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/Card_files
[ESSISSCBEPD]#
 
There You will find 2 Migration_files For the New CARD Request & For Replacement Request like the
below 2 files:
18321305112018_Migration_File_REPL.txt & 18321305112018_Migration_File.txt
These 2 files are as an Example for the Date 05112018 thinking that the above activities are done on
05112018
 
4. Still on the Card_files Directory those 2 files needs to be merged & a new file name will be given
by:
[ESSISSCBEPD]#cat 18321305112018_Migration_File_REPL.txt 18321305112018_Migration_File.txt
> 05112018_Migration_File.txt
 
Now the 2 files are Merged to a Single File Which is 05112018_Migration_File.txt
 
5. Still on the Card_files Directory duplicate records will be checked if there is any & a new file name
will be given for the final file by:
awk '!seen[$0]++' 05112018_Migration_File.txt > Final_05112018_Migration_File.txt
 
Now the Final Migration File (Final_05112018_Migration_File.txt) is Free from Duplicate Issue.
 
6. Now copy This File (Final_05112018_Migration_File.txt) to app_files Directory by:
[cd ESSISSCBEPD]#pwd
/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/Card_files
[ESSISSCBEPD]#cp Final_05112018_Migration_File.txt $RUNDATA/app_files
 
7. Go to app_files directory - Rename the File (Final_05112018_Migration_File.txt) to
Migration_File.txt :
 
[ESSISSCBEPD]#pwd
/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/Card_files
[ESSISSCBEPD]#cd $RUNDATA/app_files
[ESSISSCBEPD]#pwd
icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/app_files
[ESSISSCBEPD]#cp Final_05112018_Migration_File.txt Migration_File.txt
[ESSISSCBEPD]#ls -ltr Migration_File.txt
-rw-r--r-- 1 icmm staff 13245286 Nov 19 19:31 Migration_File.txt

 
8. submit g23s001b
 
This job Might Fail when it gets invalid record after invalid record count of 500 so, when it fails you
just need to resubmit the job (submit g23s001b)
Finally, the job will get completed running with RC 0 (Response code)

15 | P a g e
 5.2 Generating transaction & balance report of all prepaid cards daily
For generating transaction & balance report of all prepaid cards daily before starting the nightly
issuer batch.

To have the same starting time, please start it @12:00 local time

There will be a total of 4 report files

1. Balance
2. Settled (Unstatemented Trxns) Trxns
3. Outstanding Authorisations Trxns
4. Total sum of current balance, Total sum of Outstanding Authorisations amount & Total sum of
available balance group by card type

Note: The Outstanding Authorisations Trxns report is for the system date But, for the Settled
(Unstatemented Trxns) Trxn report the date to be given will be system date -1

The report will be forwarded in Excel to

Azemeraw Ayenew AzemerawAyenew@cbe.com.et

Yohannes Kebede Teklue YohannesKebede@cbe.com.et
YONATAN MEDHANE GEBRESILASSIE YonatanMedhane@cbe.com.et
Wongel Demeke Aserese WongelDemeke@cbe.com.et

Script *******************************

You will use Issuer Database for the report

User: ISSCBEPD

Password: ISSCBEPD

----For Balance report:

select tgen008.ACT_RFE_NBR_TXT,substr(raw_to_char(tgen818.pan_txt),1,16),
TGEN010.MNL_STS_CDE,TGEN010.STT_DTE,TGEN010.END_DTE,
tgen008.crt_bal_amt,tgen050.OSG_ATH_AMT,tgen050.AVE_BAL_AMT from tgen818 right join
tgen016 on substr(tgen818.pan_tkn_sky,1,16) = substr(tgen016.pan_txt,1,16) right join tgen008 on
substr(tgen016.act_idn_sky,1,16) = substr(tgen008.act_idn_sky,1,16) right join tgen050 on
substr(tgen016.act_idn_sky,1,16) = substr(tgen050.act_idn_sky,1,16) RIGHT JOIN TGEN010 ON
substr(TGEN016.PLA_IDN_SKY,1,16) = substr(TGEN010.PLA_IDN_SKY,1,16) where
substr(raw_to_char(tgen818.pan_txt),1,6) like ('419715') or
substr(raw_to_char(tgen818.pan_txt),1,6) like ('475194') or
substr(raw_to_char(tgen818.pan_txt),1,6) like ('471297') or
substr(raw_to_char(tgen818.pan_txt),1,6) like ('227383') or
substr(raw_to_char(tgen818.pan_txt),1,6) like ('925511') or
substr(raw_to_char(tgen818.pan_txt),1,6) like ('970907') ORDER BY
substr(raw_to_char(tgen818.pan_txt),1,16) ;

------For Outstanding Authorisations Trxns:

16 | P a g e
SELECT tgen008.ACT_RFE_NBR_TXT,raw_to_char(tgen818.PAN_TXT),tgen008.ACT_TYP_IDN_CDE,
TGEN010.MNL_STS_CDE,TGEN010.STT_DTE,TGEN010.END_DTE,tgen138.MSG_SRC_CDE,tgen138.N
WK_TYP_CDE,tgen138.RQT_TYP_CDE,tgen138.MSG_TYP_CDE,tgen138.AQR_ITN_IDN_NBR,tgen138.
FWD_ITN_IDN_NBR,tgen138.STA_NBR,tgen138.RTV_RFE_NBR_TXT,tgen138.GMT_DTE_TME_TXT,tg
en138.ATH_DTE,tgen138.TXN_PFE_TXN_CDE,tgen138.MCC_TXT,tgen138.BLG_CRY_CDE,tgen138.OG
L_BLG_AMT,tgen138.BLG_AMT,tgen138.TXN_CRY_CDE,tgen138.OGL_TXN_AMT,tgen138.TXN_AMT,
tgen138.APR_CDE_TXT,tgen138.AQR_CNY_CDE_TXT,tgen138.PSV_DTA_CDE_TXT,tgen138.ATN_CDE
_TXT,tgen138.PCS_CDE_NBR,tgen138.ATH_STS_CDE,tgen138.CRD_ACR_TML_TXT,tgen138.CRD_ACR
_IDN_TXT,tgen138.TXN_DES_TXT,tgen138.DEL_STS_IND,tgen138.ATH_TME,tgen138.CHP_BSE_ATH_
IND,tgen138.PMY_CDR_VERMHD_CDE,tgen138.ITN_IDN_CDE,tgen138.OGL_ATH_FEE_AMT,tgen138
.ATH_FEE_AMT,tgen138.TXN_DMN_CDE,tgen138.PLA_AVN_IND,tgen138.FST_LOD_IND,tgen138.RQ
T_TXN_AMT,tgen138.RQT_BLG_AMT,tgen138.XAC_ACT_NBR_TXT,tgen138.MCH_ATH_AMT,tgen138
.ATH_TYP_CDE,tgen138.FDT_IND,tgen138.DEL_DTE,tgen138.ICL_ATH_NBR,tgen138.APR_ICL_ATH_N
BR,tgen138.ICL_TXN_AMT,tgen138.ICL_BLG_AMT,tgen138.ICL_FEE_AMT,tgen138.LST_UPD_DTE,tge
n138.LST_UPD_TME,tgen138.LST_UPD_UID,tgen138.LST_UPD_NBR,tgen138.P_ITN_ALS_NBR,tgen13
8.P_PTT_NBR

FROM TGEN138

right join tgen016 on substr(tgen016.act_idn_sky,1,16) = substr(tgen138.act_idn_sky,1,16)

right join tgen008 on substr(tgen016.act_idn_sky,1,16) = substr(tgen008.act_idn_sky,1,16)

RIGHT JOIN TGEN010 ON substr(TGEN016.PLA_IDN_SKY,1,16) = substr(TGEN010.PLA_IDN_SKY,1,16)

right join tgen818 on substr(tgen016.PAN_TXT,1,16) = substr(tgen818.pan_tkn_sky,1,16)

WHERE (ATH_DTE = 'dd/mm/YYYY') and substr(tgen138.act_idn_sky,1,16)

IN (SELECT substr(tgen008.act_idn_sky,1,16)

FROM TGEN008 WHERE (ACT_TYP_IDN_CDE = 'AT07') OR (ACT_TYP_IDN_CDE = 'AT08') OR

(ACT_TYP_IDN_CDE = 'AT09') OR (ACT_TYP_IDN_CDE = 'AT13')

OR (ACT_TYP_IDN_CDE = 'AT14') OR (ACT_TYP_IDN_CDE = 'AT15')) order by

tgen008.ACT_TYP_IDN_CDE ;

------Total Balance of prepaid cards group by card type:

Select
substr(raw_to_char(tgen818.pan_txt),1,6),SUM(tgen008.crt_bal_amt),SUM(tgen050.OSG_ATH_AMT
),SUM(tgen050.AVE_BAL_AMT) from tgen818 right join tgen016 on
substr(tgen818.pan_tkn_sky,1,16) = substr(tgen016.pan_txt,1,16) right join tgen008 on
substr(tgen016.act_idn_sky,1,16) = substr(tgen008.act_idn_sky,1,16) right join tgen050 on
substr(tgen016.act_idn_sky,1,16) = substr(tgen050.act_idn_sky,1,16) RIGHT JOIN TGEN010 ON
substr(TGEN016.PLA_IDN_SKY,1,16) = substr(TGEN010.PLA_IDN_SKY,1,16) where
substr(raw_to_char(tgen818.pan_txt),1,6) like ('419715') or
substr(raw_to_char(tgen818.pan_txt),1,6) like ('475194') or
substr(raw_to_char(tgen818.pan_txt),1,6) like ('471297') or
substr(raw_to_char(tgen818.pan_txt),1,6) like ('227383') or

17 | P a g e
substr(raw_to_char(tgen818.pan_txt),1,6) like ('925511') or
substr(raw_to_char(tgen818.pan_txt),1,6) like ('970907') GROUP BY
substr(raw_to_char(tgen818.pan_txt),1,6) ;

6. Extract PTLF &TLF

 LOGING to Base24EPS server (10.1.7.33) Using your credentials

 Check the transaction extract has finished or not

 JQB will stop queuing
 Go to $B24OLTP
o Cd $B24OLTP
o check tx[1,2,3] completed
 eg. tail -1 tx1 check if footer line generated
o check px[1,2,3] completed
 eg. tail -1 px1 check if footer line generated

 Goto $BIN
#Cd $BIN
Run the script called
# sendExtract.ksh

 Make sure the script picks the correct file. It should pick only that day of TLF and
PTLF(one tx[1,2,3])and one px[1,2,3] if not the batch will fail @IXB24EPS

 Then go to CMM server(10.1.7.170)

o Login with your credential and

 choose 2 to Switch to interchange region

18 | P a g e
  Goto /app_files
#Cd /icmmfs/ACI_CMM/ACI01/ICG/CBEPD/files/app_files/

 check the time stamp for the input file ixb24e is changed to the current time and date

7. BATCH RUNNING
Please use a remote server 10.1.64.248

User Name: Administrator

Password: Pa$$w0rd

7.1 Start ACQ PART ONE

 Login CMM server

 Enter your username PWD

19 | P a g e
 o

 Enter 1 for ACQ region

 Go to /icmmfs/ACI_CMM/ACI01/ACQ/CBEPD/files/cosbatch/utility_input_dir
o #cd
/icmmfs/ACI_CMM/ACI01/ACQ/CBEPD/files/cosbatch/utility_input_dir
o Ls –ltr

 Then run PART_ONE.csv

# batch_run.ksh PART_ONE.csv 1

REMARK : (please take care when you choose the option 1 and zero it creates DB
inconsistence )
o batch_run. PART_ONE.csv 1 for fresh start
o batch_run.ksh PART_ONE.csv.csv 0 to resume the batch run if the
batch run is interrupted due to different reason(like if batch failed with
some error , power problem )

20 | P a g e
Then wait till ACQPART ONE is finished (the last job from ACQ PART ONE G00S572B ) it
should end with zero

7.2 RUN INTERCHANGE BATCH

o Still in CMM server

Open new session
enter 2 for ICG region

OR
o Just Do regswitch to switch to interchange region
o Run the command
o Regswitch
o Chose 2 for ICG region

 again Make sure the transaction extract(ixb24e ) is up-to-date (check the date &size) to avoid
duplicate file processing

 Cd $RUNDATA/app_files
   Ls –ltr ixb24e

o Then Go to /icmmfs/ACI_CMM/ACI01/ICG/CBEPD/files/cosbatch/utility_input_dir
o Cd /icmmfs/ACI_CMM/ACI01/ICG/CBEPD/files/cosbatch/utility_input_dir
o Ls –ltr

 batch_run.ksh CBE_ICG_34112.csv 1 //for fresh start

21 | P a g e
 REMARK : (please take care when you choose the option 1 and zero it creates DB
inconsistence )
o batch_run.ksh CBE_ICG_34112.csv 1 for fresh start
o batch_run.ksh CBE_ICG_34112.csv 0 to resume the batch run if
the batch run is interrupted due to different reason(like if batch failed
with some error , power problem )

Wait until INTERCHNGE batch run reaches @ IXDBASEU

You can check by running the command jobdisp in new session
#jobdisp

7.3 ACQ PART TWO

o Still in CMM server

Open new session
enter 1 for ACQ region

OR
o Just Do regswitch to switch to ACQuirer region
o Run the command
o Regswitch
o Chose 1 for ACQ region

o Goto /icmmfs/ACI_CMM/ACI01/ACQ/CBEPD/files/cosbatch/utility_input_dir
 batch_run.ksh PART_TWO.csv 1

o If it finishes successfully the last job will be G00S660B.

22 | P a g e
7.4. Issuer Batch
Go to issuer region and submit the nightly batch

cd $RUNDATA/cosbatch /utility_input_dir

ls

ACI_ISSUER.csv ACI_ISSUER_BASE24EPS.csv ESS_UTL_10_STREAMED_SUITE.csv

ESS_UTL_5_STREAMED_SUITE.csv

ACI_ISSUER_BASE24CLASSIC.csv ACI_ISSUER_HOUSEKEEPING.csv
ESS_UTL_1_STREAMED_SUITE.csv

execute the below command for fresh start of the batch use option 1 for re-run replace the 1 with a
0

batch_run.ksh ESS_UTL_10_STREAMED_SUITE.csv 1

7.4.1 Streaming Issuer nightly batch

Things to do after submitting Daily Issuer Batch:

 
When you submit Issuer Batch :
 
[ESSISSCBEPD]#batch_run.ksh ESS_UTL_10_STREAMED_SUITE.csv 1
 
There are different Jobs inside (ESS_UTL_10_STREAMED_SUITE.csv) & you will see the flow when
they are completed successfully and the batch flow continues with the next job.
Now what you need to do is when STM01501 is started
at this point when Job STM01501 starts running ---->
1. STOP THE BATCH PROCESSING WITH (CTRL+C) TWICE
2. OPEN 10 DIFFERENT SESSIONS & SELECT ISSUER
ANY WHERE (NOT DIRECTORY DEPENDENT) submit the following 10 Jobs on the 10 sessions, which
means on the 1st session (submit STM01501) , on the 2nd session (submit STM01502) , on the 3rd
session (submit STM01503) & the like...
 
/.
STM01501
STM01502
STM01503
STM01504
STM01505
STM01506
STM01507
STM01508
STM01509
STM01510
./
 
WHEN all 10 JOBS finish successfully with RC of 0 go to $RUNDATA/app_files
ls -ltr *batch*

23 | P a g e
-rwxr-xr-x 1 icmm staff 2269 Jul 11 15:13 batch.ctl.110718
-rw-r--r-- 1 icmm staff 140 Jul 25 13:27 batch.txt
-rwxr-xr-x 1 icmm staff 340 Sep 21 09:48 2nd_STM002_batch.ctl.posting
-rwxr-xr-x 1 icmm staff 914 Sep 21 10:16 3rd_STM005_batch.ctl
-rwxr-xr-x 1 icmm staff 842 Sep 21 10:17 sep_21_batch.ctl
-rwxr-xr-x 1 icmm staff 205 Sep 21 10:20 1st_STM015_batch.ctl.pre_posting
-rwxr-xr-x 1 icmm staff 2046 Sep 21 13:29 4th_STM027_batch.ctl
-rwxr-xr-x 1 icmm staff 5363 Sep 21 14:41 5th_STM089_batch.ctl
-rwxr-xr-x 1 icmm staff 843 Sep 27 19:30 STM029_batch.ctl_andu
-rwxrwxrwx 1 icmm staff 7494 Nov 07 10:55 batch.ctl
 
cp 1st_STM015_batch.ctl.pre_posting batch.ctl
 
cd /icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/cosbatch/utility_input_dir and
continue the nightly job with 0
batch_run.ksh ESS_UTL_10_STREAMED_SUITE.csv 0
 
after this is done with 5min max job STM00201 will start running so do the same thing for
STM002XX jobs as well
 
1. STOP THE BATCH PROCESSING WITH (CTRL+C) TWICE
2. ON THE OPENED 10 SESSIONS submit the following 10 Jobs on the 10 sessions, which means
on the 1st session (submit STM00201) , on the 2nd session (submit STM00202) , on the 3rd session
(submit STM00203) & the like...
 
.
STM00201
STM00202
STM00203
STM00204
STM00205
STM00206
STM00207
STM00208
STM00209
STM00210
 
WHEN all 10 JOBS finish successfully with RC of 10 go to $RUNDATA/app_files
ls -ltr *batch*
-rwxr-xr-x 1 icmm staff 2269 Jul 11 15:13 batch.ctl.110718
-rw-r--r-- 1 icmm staff 140 Jul 25 13:27 batch.txt
-rwxr-xr-x 1 icmm staff 340 Sep 21 09:48 2nd_STM002_batch.ctl.posting
-rwxr-xr-x 1 icmm staff 914 Sep 21 10:16 3rd_STM005_batch.ctl
-rwxr-xr-x 1 icmm staff 842 Sep 21 10:17 sep_21_batch.ctl
-rwxr-xr-x 1 icmm staff 205 Sep 21 10:20 1st_STM015_batch.ctl.pre_posting
-rwxr-xr-x 1 icmm staff 2046 Sep 21 13:29 4th_STM027_batch.ctl
-rwxr-xr-x 1 icmm staff 5363 Sep 21 14:41 5th_STM089_batch.ctl
-rwxr-xr-x 1 icmm staff 843 Sep 27 19:30 STM029_batch.ctl_andu
-rwxrwxrwx 1 icmm staff 7494 Nov 07 10:55 batch.ctl
 
cp 2nd_STM002_batch.ctl.posting batch.ctl

24 | P a g e
 
8.4 Resuming Issuer nightly batch

cd /icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/cosbatch/utility_input_dir and continue the nightly

job with 0
batch_run.ksh ESS_UTL_10_STREAMED_SUITE.csv 0

8. EXTRACT the Payment order

After ACQ PART TWO has finished EXTRACT the Payment order
o In Acquirer Region
o Goto $RUNDATA/app_files
o Ls –ltr *sh

You will get the below result and run the two script POsplitter.sh and POsplitterSCH.sh respectively.

9. COMMISSION
 Uploading commission using Windows server
o Login :10.1.64.249 server
 user : administrator
 pass : Pa$$w0rd
 copy the ixgo_0000001 file to E:\Transaction Extract on 10.1.64.249
-> Go to E:\

25 | P a g e
 -> run Transaction_extrn_n_fee_calculation_UI

Application and give the ixgo file that you copied to E:\Transaction Extract as input

-> there will be several files generated in the E:\Transaction Extract folder, rename
g210tpeo_00date to a format like this transaction_fee_month_day
 LOGIN to CMM server on winscp using your credential
 Then
 goto /commissionfs
-> copy
E:\Transaction Extract file\transaction_fee_month_day
->To
/commissionfs

 open putty
 login using your credentials
o cd /commissionfs
o ls –ltr
 check the latest file you coped before is there or not

 sftp epssftp@10.3.9.54
 cd /t24/t24live/bnk/bnk.run/BASE24.POS.TXN
 run the put command for the copied commission file transaction_fee_month_day
o put transaction_fee_month_day

10. TLF Extraction for Financial Report

1. Login to 10.1.64.249 server
1.1. Remote login to (if remote not working check the network)
1.1.1.IP:- 10.1.64.249
1.1.2.User:- Administrator
1.1.3.Password:- Pa$$w0rd
2. Coping /sftp today’s TLF from CMM server
/icmmfs/ACI_CMM/ACI01/ICG/CBEPD/files/extractbkp directory
2.1. Run WinSCP

26 | P a g e
 2.2. Login to 10.1.7.70/170
2.3. Use your credential

Source directory:-
2.3.1. go to /icmmfs/ACI_CMM/ACI01/ICG/CBEPD/files/extractbkp directory
2.3.2.Sort it
2.3.3.Example:- tx217112018213003
2.4. Destination directory
2.4.1.On 249 windows server
2.4.1.1. C:\reconcilation\TLF\DECEMBER
2.4.1.2. Create new folder with today’s date eg. 28122018
2.4.1.3. Paste on C:\reconcilation\TLF\ DECEMBER \28122018directory
2.4.1.4. After coping is done rename it like below
2.4.1.5. Example:- TLF 171120182.txt
3. Run extracting java program
3.1. Go to C:\reconcilation\REVERSAL.v.8 directory
3.2. Run Reversal.jar
3.2.1.Browse the copied file
3.2.2.Click okay
4. Done. Thanks

11.Monitoring Report
Reports

1. Interfaces status (T24, ETH, VISA, MC and Issuer)

2. Resource status of EPS(10.1.7.33/197/187) and CMM(10.1.7.170) servers

IP User Password
10.1.7.33 ib24 EPSprod33!
10.1.7.197 ib24 epsDb197!
10.1.7.187 ib24 esweB24!
10.1.7.170 icm ICMMpass!33
m
3. TLP and PTLF status
4. Whether Acquirer and Issuer batches successfully processed or not
5. Whether card file prepared for personalization or not

27 | P a g e
 6. Whether commission processed or not? If failed reason should be specified
7. Base24 offline/online time if happened
8. Payment Order extract
9. ATM transaction Report (must be sent for Managements and Business team )
10. TLF Extraction for Financial Report
11. Any other valuable information

12.Timing
Task\Time Time
Monitoring Eps The whole monitoring time
Offline Variable
Prepaid cards report Starting @12:00 local
Card request Processing starting 01:00
Automatic Balance Refresh Starts @1:30 finish around 4:00
ACQ ONE You can start @8:00
INTERCHANGE Starts around 8:15 when TLF and PTLF fully extracted
ACQ TWO Start when Interchange is on job ixdbaseu
ISSUER Start when Interchange is on job ixdbaseu
COMMISSION Start when Interchange is on job ixdbaseu
TLF Extraction for FR start after staring Interchange
ACQ Transaction Report start after ACQ two completed
Monitoring Report When you finish everything.

13. Speeding up CMM batches tips

Having parentless processes will slow down the CMM batch so you can check if there is any & kill by
process id.

ps -aef | grep JOB_DISP

Eg

ESSISSCBEPD]#ps -aef | grep JOB_DISP

    icmm 11600088        1  11 09:25:55      - 13:33 /bin/ksh
/icmmfs/ACI_TOOLS/aci/general_utils/ESS_UTL_JOB_DISP.ksh
    icmm 18284620 32047156   0   Oct 29  pts/1  0:00 /bin/ksh
/icmmfs/ACI_TOOLS/aci/general_utils/ESS_UTL_JOB_DISP.ksh
    icmm 22806750 36044970   0 14:10:32  pts/8  0:00 grep JOB_DISP
    icmm 45613208        1  12 09:24:14      - 13:41 /bin/ksh
/icmmfs/ACI_TOOLS/aci/general_utils/ESS_UTL_JOB_DISP.ksh

28 | P a g e
[ESSISSCBEPD]#

Kill the ones that have 1 as a parent process id

kill -9 11600088 45613208      

AND

cd $ESS_TOOLS_DIR
run this script
ESS_UTL_TPE_PROCESS_REPORT.ksh
If it shows you any parentless process kill by that ID

Eg.

[ESSISSCBEPD]#ESS_UTL_TPE_PROCESS_REPORT.ksh

A Unikix TPE Process Report is being created, please wait a few moments...

The Unikix TPE Process Report,

/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/MTP/sys/tpe_process_report_03112018_141209.log has been
created

Do you want to open the log file? Y/N

==> enter Y here

Then it will list all valid preocesses & parentless one's as well if there are any.Keep going down by
hitting enter & you will see the below message for different kinds of unikix process.(valid process's
are listed first @ the top of the report)
The following unikixtran processes do not have a parent unikixmain process:
If there are any parentless process then just as the above one the parent process ID will also be set
to 1 so kill using the process ID

14. Pay Attention to

 Anyone who is monitoring should pay attention to the space & memory of both /home &
/icmmfs on 10.1.7.170 before beginning their work on 10.1.7.170 including doing migration
for new & replacement cards. (the used % space should be < 90 ).
 If any job on issuer fails with a RC of 12 re-submit the job before assuming complete failure
since it might be network or other env’t issue
 Since reprocessing g23s001b Migration job for new & Replacement card file processing will
not reject the file but instead processes as a normal request,Be sure of failure before
resubmitting the job specially for RC code other than 8
 If any job takes time then check the jobdisp (eg.g023s001b)
 If there is only submitted date & the rest are empty, then the job is still running wait
for it to finish no matter how long it takes

29 | P a g e
  If there is only submitted date & completed date with submitted time & completed
time empty, then that is an indication that the job might have failed in that case
communicate the person responsible

15. Simple troubleshooting for common errors

1. If the batch fails at any of jobs with 255 0r 254 .
 Most probably it will be infrastructure issue so please
o Check the network connection : try it with new session
o Storage issue: check the file system /icmmfs and /home used % >90 .
o DB connection issue :check if you can login to DB via sqlplus
/@$ORACLE_CONID

2. If the JOB IXB24ESP fails with 161

o You might miss somethong on step 5 . so please rerun the script on EPS
Run the script called -> sendExtract.ksh
The resume the batch with ZERO (batch_run.ksh CBE_ICG_3401.csv 0)
3. If the job IXIPMI fails with RC 161

30 | P a g e
 cd $RUNDATD/app_files
ls -ltr ixmi*
[ESSICGCBEPD]#cp ixmi.mc.empty.org  ixmi.mc
[ESSICGCBEPD]#cp ixmi.mc.empty.orig   ixmi.predrej.mc

then continue the batch with 0.

4. If the job IXVISAI fails with RC 161

cd $RUNDATD/app_files

ls -ltr ixvi*

cp ixvi.empty ixvi

then continue the batch with 0.

5. If job G00S543B fails on ACQ part 2

Go interchange (regswitch 2)

cd $RUNDATA/app_files

ls –ltr *ixgr*

cp ixgr_000001 ddmmyyyy_ ixgr_000001

(eg. cp ixgr_000001 24112019_ ixgr_000001)

rm ixgr_000001

touch ixgr_000001

Go back to aquirer (regswitch 1) and continue ACQ batch part 2

15.1 Tips for failure of job STM015xx & STM002xx issuer

jobs in the nightly batch

15.1.1 Issuer batch failure at stream able job STM015xx jobs (STM01501,
STM01502… etc.)

Example: if Job STM01501 has failed, open WinSCP on Issuer app_files directory sort by date &
refresh to get the latest files, also check the date the file was generated on to make sure that you
are working with the right file.

31 | P a g e
The directory is /icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/app_files  

you will find 2 files named <ggnsipt.g00b015b.S001> & <gixsopt.S001> Transfer them to your PC

o The .S001 represent the stream number so if STM01502 failed then the files
would have an extension of .S002 …etc.

> Open the ggnsipt.g00b015b.S001 file with HxD editor

> Go to the last line(record) of the File & scroll up slowly until you see a number that looks like a T24
account number, it is 13-digit number (please confirm the digits)

The account number is found around the info that looks like the sample information provided below:
The T24 account number may be divided b/n 2 lines

> Now Open the gixsopt.S001 file, find this Account Number which was taken from the
ggnsipt.g00b015b.S001 file 

> in the gixsopt.S001 file A single record starts with CBE1 followed by a Card Number (CBE1       
4583006155081619...) up to the starting point of the next CBE1

> after finding the account number in this file Scroll down you will see an invalid record starts
with CBE1        9999999999999999999......

> There might be other similar records following this invalid record. You need to remove those
records until you get the next correct record like (CBE1        4583006155081619...)

> Save the file (gixsopt.S001) & backup the file that already exists in app_files with the current date
Orig_date_gixsopt.S001 , then Transfer the updated file back to app_files directory with the WINSCP.

> Now you can resubmit the Job (submit STM01501).

> If the failure reason is not the above (invalid record which starts with CBE1       
9999999999999999999......) then what you need to do is first you will get the final record from
ggnsipt.g00b015b.S001 file then you will search this record using the account number as above on
the next file gixsopt.S001 when you get this record please remove the next record (A single
record starts with CBE1 followed by a Card Number (CBE1        4583006155081619...) up to the
starting point of the next CBE1) & save your file. Backup the file that already exists in app_files
with the current date Orig_date_gixsopt.S001 , then Transfer the updated file back to app_files
directory with the WINSCP.

32 | P a g e
> Now you can resubmit the Job (submit STM01501).

15.1.2 Issuer batch failure at streamable job STM002xx jobs (STM00201,

STM00202…etc.)

NOTE: - First you should only resubmit the job as it might have failed due to env’t
problems then go to sysout directory and check if the record count is changing
after resubmission (steps on how to check record count is described below), only
if the record count remains the same should the below action be taken.

Example: if Job STM00201 has failed, open WinSCP on Issuer app_files directory sort by date &
refresh to get the latest files, also check the date the file was generated on to make sure that you
are working with the right file.

The directory is /icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/app_files  

In this case you will be working with a single file gtmpipt.S001_000*** the file is incremental & will
be generated for each stream of STM002xx jobs gtmpipt.S002_000***, gtmpipt.S003_000*** … etc.

This file has a header of length 728 and each record has a length of 1715 so the idea is to get the
record count that is causing the job to fail from sysout and do a calculation

But first we have to get the record count from sysout directory.

Go to issuer environment type sysout and enter

ls –ltr

Cd STM00201 (if that is the job that failed)

ls (you will see a file with a naming similar to this STM00201_S0030_GFMTLOG_180627_161432)

More STM00201_S0030_GFMTLOG_180627_161432

Then take a look at the record count field b/c it will be used in finding the record that will be
removed.

(Result of record count – 1) * 1874 + 751…. Copy the result from the calculator

33 | P a g e
Then open the file in HXD (Hex Editor)

do Ctrl + g click on the dec option then paste the result from the calculator & press enter, the cursor
will move to a position similar to the below image then remove that record from that position up to
the next record starting (..YYYY (each record starts with ..YYYY and the Julian date is followed e.g. “
..2018295 “)

o Beginning of a new record in yellow, beginning of another record in red

..2018295101380094CBE1.(.¨..um.(.¨..uh .)\.&........ ..230JIMMA DISTRICT JIMMA

ETH 6011...... ..2302202IXS 3397900
099999C99004..AJM00068AJM00068 ..LOCL...... ..230....
NONEINAI .)\ ......................................................... ..................1000040404132
............
.... ........ ..0100000021892 .................. .. 10.. 10.. 10.. 10.. 10..
10.. 10.. 10.. 10.. 10.. 10.. 10.. 10.. 10.. 10.. 10.. 10.. 10.. 10.. 10.. 10.. 11.. 10..
10.. 10.. 11.. 11.. 10.. 10.. 10.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11..
11.. 11.. 11.. 11.. 11.. 11.. 10.. 10.. 10.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 10.. 10..
10.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 11.. 10.. 11..
10.. 10.. 10.. 10..2018295102617444CBE1.

> Save the file (gtmpipt.S001_000***) & backup the file that already exists in app_files with the
current date Orig_date_ gtmpipt.S001_000***, then Transfer the updated file back to app_files
directory using WinSCP.

> Now you can resubmit the Job (submit STM00201).

------------------------------------------------------------------------------------------------

15.2 When there is a plastic bulk issue request & if job G00S068B is failed, inorder to
resume the batch we need to replace mmppdact_000*** file found
@/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/app_files with an empty file after taking a backup of
the original file like orig_ ddmm_mmppdact_000***. There are many empty files
@/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/app_files  directory & when you replace make the
new file name with the next number like if the one with bulk issue request in it is
mmppdact_000407 what you will do is backup this file & replace it with the empty file & named it
like mmppdact_000408

-------------------------------------------------------------------------------------

15.3 Every time please do the following while you are running the nightly issuer batch.
After you have started the batch, there is a Job named G00S009B around one job before the 2nd
stream & 3 jobs after the 1st stream

(…

34 | P a g e
 …….)

After this job has completed, it will generate an output file named gtgr009_00****  @app_files
directory (/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/app_files )

Please check the file & if it has a size you need to replace it with an empty file (this empty file will be
named with the next sequential number) But, if the generated file is already empty you will not do
the file replacement.

let the file with size is gtgr009_001390 so, the empty file will be gtgr009_001391  

There is an empty file you can use @/icmmfs/ACI_CMM/ACI01/ISS/CBEPD/files/app_files  

directory for the replacement.

You can do the replacement as the batch is still running.

35 | P a g e