Google Code Archive - Long-Term Storage For Google Code Project Hosting

3/18/24, 2:41 PM Google Code Archive - Long-term storage for Google Code Project Hosting.
Code
Archive
Export to GitHub
oracleinfrastructure -
R12_2_Installing_R12_12_2_4_on_multinode_MT_and_2_node_RAC.wiki
Table of contents
Installing R12 12_2_4 on multinode MT and 2 node RAC

http://okon1.remik.org:8005
Download binaries
"File Name" + "V46139-01.zip" + "V46140-01.zip" + "V46243-01_1of3.zip" + "V46243-01_2of3.zip" + "V46243-01_3of3.zip" + "V35231-01_1of5.zip" + "V35231-

01_2of5.zip" + "V35231-01_3of5.zip" + "V35231-01_4of5.zip" + "V35231-01_5of5.zip" + "V35230-01_1of2.zip" + "V35230-01_2of2.zip" + "V35812-01.zip" +
"V35811-01.zip" + "V35810-01.zip" + "V35809-01.zip" + "V35808-01.zip" + "V35807-01.zip" + "V35806-01_1of3.zip" + "V35806-01_2of3.zip" + "V35806-
01_3of3.zip" + "V35805-01_1of2.zip" + "V35805-01_2of2.zip" + "V35802-01.zip" + "V35804-01_1of2.zip" + "V35804-01_2of2.zip" + "V35803-01_1of3.zip" +
"V35803-01_2of3.zip" + "V35803-01_3of3.zip" + "V35813-01.zip" + "V29856-01.zip" + "V29764-01.zip" + "V46241-01_1of9.zip" + "V46241-01_2of9.zip" +
"V46241-01_3of9.zip" + "V46241-01_4of9.zip" + "V46241-01_5of9.zip" + "V46241-01_6of9.zip" + "V46241-01_7of9.zip" + "V46241-01_8of9.zip" + "V46241-
01_9of9.zip" + "V46499-01.zip" + "V20223-01.zip" + "V29762-01_1of2.zip" + "V29762-01_2of2.zip" + "V29672-01_1of2.zip" + "V29672-01_2of2.zip" "V36831-
01.zip" "V36830-01.zip" "V36829-01.zip" "V36833-01.zip" "V36828-01.zip" "V36824-01.zip" "V36803-01.zip" "V36827-01.zip" "V36826-01.zip" "V36832-01.zip"
"V36823-01.zip" "V36822-01.zip" "V36820-01.zip" "V36819-01.zip" "V36817-01.zip" "V36816-01.zip" "V36815-01.zip" "V36805-01.zip" "V36814-01.zip" "V36813-
"V46193-01.zip" "V46201-01.zip" "V46200-01.zip" "V46214-01.zip" "V46191-01.zip"
Setup hosts
Setup MT
setup a shared NFS directory between all MT hosts, this is where the binaries will be installed
```
rsync -a --progress /STAGING/ebs12patches/*.zip /RYBAK/

Default NFS mount params 13.85MB/s, ```
nfs rw,bg,intr,hard,timeo=600,wsize=32768,rsize=32768,nfsvers=3,tcp,noacl,nolock
.71MB/s
Copying RAC nodes
/vmfs/volumes/datastore1 RAC_11.2.0.3_SE_OEL6.3_openfiler RAC_11.2.0.3_SE_OEL6.3_rac1-user1a RAC_11.2.0.3_SE_OEL6.3_usera2
/vmfs/volumes/piaty_4T
https://code.google.com/archive/p/oracleinfrastructure/wikis/R12_2_Installing_R12_12_2_4_on_multinode_MT_and_2_node_RAC.wiki 1/33
various stuff
MT ```
echo "inventory_loc=/RYBAK/oraInventory" > /etc/oraInst.loc

```
Contents of /etc/hosts
Perform some sanity net throughput tests
Create 1G file
okon1 Local / $ time dd if=/dev/zero of=./trash bs=1024 count=1048576 real 0m18.024s 1073741824 bytes (1.1 GB) copied, 17.8473 seconds, 60.2 MB/s
okon1, NFS /RYBAK $ time dd if=/dev/zero of=./trash bs=1024 count=1048576 1073741824 bytes (1.1 GB) copied, 95.8137 seconds, 11.2 MB/s real 1m35.930s
-> this is too slow, 100Mb card network card used ???
How to check what is the network speed on Synology?
zen (VirtualBox on laptop), NFS /RYBAK ```
time dd if=/dev/zero of=./trash bs=1024 count=1048576

skopiowane 1073741824 bajty (1,1 GB), 44,208 s, 24,3 MB/s real 0m44.289s ```
okon2 (VirtualBox on laptop), NFS /RYBAK ```
time dd if=/dev/zero of=./trash bs=1024 count=1048576

1073741824 bytes (1.1 GB) copied, 33.5686 seconds, 32.0 MB/s real 0m33.581s ```
Create staging area

MOS: Troubleshooting Rapid Install for E-Business Suite Release 12.2 (Doc ID 1378579.1) MD5 Checksums for R12.2 Rapid Install Media (Doc ID
1505510.1)
Download the md5_linux.txt file. $ md5sum --check md5_startCD_12.2.0.48.txt > md5sum_result.txt $ cat md5sum_result.txt | grep -v OK
startCD/Disk1/rapidwiz/TechInstallers/DBInstallHome/fnd/admin/template/txkCfgDB11gR2_tmpl.xml: FAILED (that is OK, we edit this file by hand)
Rapid Install Start Here (Part 1 of 3) V46243-01 Part 1 of 3 3.1G Rapid Install Start Here (Part 2 of 3) V46243-01 Part 2 of 3 81M Rapid Install Start Here
(Part 3 of 3) V46243-01 Part 3 of 3 97M
$ unzip V46243-01_1of3.zip $ unzip V46243-01_2of3.zip $ unzip V46243-01_3of3.zip
Run as root ```
cd startCD/Disk1/rapidwiz
./RapidWizVersion
Version 12.2.0.48
cd bin
./buildStage.sh
1. Create new stage area
2. Linux x86 (64-bit) Specify the directory containing the zipped installation media: /STAGING/oracle_EBS_R12.2.4_linux_x86-64 The set of zip files is
complete. -> this will unzip the files ```
Now, lets see if there are any patches to RapidWiz
MOS: Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID 1320300.1)
Oracle recommends using the latest startCD (currently 12.2.0.49 - Patch 19378467) to install or upgrade to the latest Oracle E-Business Suite Release
12.2.0.
RI STARTCD 12.2.0.49 (Patch) p19378467_R12_GENERIC_1of2.zip 2.0 GB (2129316845 bytes) MD5 1E4B1DAE98F72A964BFE67282060FBEF
p19378467_R12_GENERIC_2of2.zip 3.1 GB (3289152372 bytes) MD5 0636D61EE3B51FD43791430BA13EB098
$ unzip p19378467_R12_GENERIC_1of2.zip $ unzip p19378467_R12_GENERIC_2of2.zip
$ cd startCD/Disk1/rapidwiz $ ./RapidWizVersion Version 12.2.0.49
```
cd bin
./buildStage.sh
2. Copy patches to existing stage area
3. Linux x86 (64-bit) ```
```
cd /STAGING/startCD/Disk1/rapidwiz
./rapidwiz
```
?? unable to add another app node. I am skipping this patch, more issues below.
Installing RAC DB
Pre install check failures

RW-20013: Error: - Not enough free disk space on system:
There is already CRS installed Now: $ pwd /u01 $ ls app _inst
Temp Space Check : test using command: su oracle -c "touch /tmp/test.tst" TEMP directory has write privileges: Mount Point = /tmp
RW-20013: Error: - Not enough free disk space on system: TEMP directory = /tmp required = 2048.0 actual = 7.28515625 **> weird, /tmp is from root ```
pwd
/tmp [root@usera1 tmp]# df -h . Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_rac1-lv_root 22G 14G 6.3G 69% / ```**
cluvfy
```
cat cluvfy_output.10571.lst
You must NOT be logged in as root (uid=0) when running /u01/app/11.2.0/grid/bin/cluvfy. -> :) ```
oracle@usera1 $ /u01/app/11.2.0/grid/bin/cluvfy stage -post crsinst -n usera1,usera2
ERROR: PRVF-7617 : Node connectivity between "usera1 : 192.168.1.50" and "usera2 : 192.168.1.51" failed
ERROR: PRVF-7617 : Node connectivity between "usera1 : 192.168.1.50" and "usera2 : 192.168.1.54" failed TCP connectivity check failed for subnet
"192.168.1.0"
$ /u01/app/11.2.0/grid/bin/cluvfy comp nodecon -i eth1 -n usera1,usera2 Verification of node connectivity was successful.
$ /u01/app/11.2.0/grid/bin/cluvfy comp nodecon -i eth0 -n usera1,usera2 ERROR: PRVF-7617 : Node connectivity between "usera1 : 192.168.1.50" and "usera2
: 192.168.1.51" failed
ERROR: PRVF-7617 : Node connectivity between "usera1 : 192.168.1.50" and "usera2 : 192.168.1.54" failed TCP connectivity check failed for subnet
"192.168.1.0"
He is trying somehow to reach IP for scan listener $ crsctl stat res -t
Old Suff $ srvctl remove database -d test11 $ srvctl remove database -d rac
Disable scan listener $ crsctl stop res ora.LISTENER_SCAN2.lsnr $ crsctl stop res ora.LISTENER_SCAN1.lsnr $ crsctl stop res ora.LISTENER_SCAN3.lsnr
$ crsctl stop res ora.scan2.vip $ crsctl stop res ora.scan1.vip $ crsctl stop res ora.scan3.vip
```
ifconfig -a
Shows no vip (too much? I wanted only to shutdown scan, not regular VIP??) ```
Anyway, again the check $ /u01/app/11.2.0/grid/bin/cluvfy comp nodecon -i eth0 -n usera1,usera2

Port 1521 not free
Yes, I am running already CRS, which is using scan listener and VIP on 1521
```
Private
192.168.255.50 usera1-priv.remik.org usera1-priv 192.168.255.51 usera2-priv.remik.org usera2-priv ```
```
VIP
192.168.1.52 usera1-vip.remik.org usera1-vip 192.168.1.53 usera2-vip.remik.org usera2-vip ```
$ lsnrctl status
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 25-DEC-2014 22:41:25
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ----------------------- Alias LISTENER

Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production Start Date 24-DEC-2014 10:45:53 Uptime 1 days 11 hr. 55 min. 31 sec Trace Level off
Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora Listener Log File
/u01/app/oracle/diag/tnslsnr/usera2/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)
(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.51)(PORT=1521))) (DESCRIPTION=(ADDRESS=
(PROTOCOL=tcp)(HOST=192.168.1.53)(PORT=1521))) Services Summary...
> Ichose different port pool: 5,6
Errors during installation:
RW-50004: An error has occured
$ less /u01/oracle/VIS/11.2.0/appsutil/log/VIS_usera1/12231454.log
Fatal Error: TXK Install Service
oracle.apps.fnd.txk.config.ProcessStateException: OUI process failed : Exit=253 See log for details. CMD= /STAGING/oracle_EBS_R12.2.4_linux_x86-
64/TechInstallMedia/database/database/runInstaller -waitForCompletion -ignoreSysPrereqs -force -silent -responseFile
/u01/oracle/VIS/11.2.0/temp/VIS_usera1/cfgHome/response/DB_HOME/txkDB11gR2_11203.rsp
at oracle.apps.fnd.txk.config.OUIPatchActionNode.processState(OUIPatchActionNode.java:160)
at oracle.apps.fnd.txk.config.PatchActionNode.processState(PatchActionNode.java:187)
Cannot install Oracle Database Home
RW-50010: Error: - script has returned an error: 1 RW-50004: Error code received when running external process. Check log file for details.
MOS: Doc 1590656.1
Additional logs: /u01/oracle/VIS/11.2.0/temp/VIS_usera1/logs/oracle.apps.fnd.txk.install0.log CAUSE: The selected Oracle home contained directories or
files. ACTION: To start with an empty Oracle home, either remove its contents or choose another location.
What Oracle homes are installed in inventory? $ cat /u01/app/oraInventory/ContentsXML/inventory.xml | grep "HOME NAME" <HOME
NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true"> <HOME NAME="OraDb11g_home1"
LOC="/u01/app/oracle/product/11.2.0/dbhome_1" TYPE="O" IDX="2"> <HOME NAME="Ora10g_rdbms" LOC="/u01/app/oracle/product/10.2.0/db_1" TYPE="O" IDX="3">
<HOME NAME="OraDb11g_home2" LOC="/u01/trash" TYPE="O" IDX="4">
Remove all Oracle Homesexcept the grid infra. $ cd /STAGING/oracle11.2.0.4_x86-64/deinstall $ ./deinstall -home /u01/trash $ ./deinstall -home
/u01/app/oracle/product/11.2.0/dbhome_1 $ ./deinstall -home /u01/app/oracle/product/10.2.0/db_1
$ cat /u01/app/oraInventory/ContentsXML/inventory.xml | grep "HOME NAME" <HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1"
CRS="true">
Still the same error:
I try tu run the install by hand oracle.apps.fnd.txk.config.ProcessStateException: OUI process failed : Exit=253 See log for details. CMD=
/STAGING/oracle_EBS_R12.2.4_linux_x86-64/TechInstallMedia/database/database/runInstaller -waitForCompletion -ignoreSysPrereqs -force -silent -
responseFile /u01/oracle/VIS/11.2.0/temp/VIS_usera1/cfgHome/response/DB_HOME/txkDB11gR2_11203.rsp
$ /STAGING/oracle_EBS_R12.2.4_linux_x86-64/TechInstallMedia/database/database/runInstaller -responseFile
/u01/oracle/VIS/11.2.0/temp/VIS_usera1/cfgHome/response/DB_HOME/txkDB11gR2_11203.rsp -> Works fine
$ /STAGING/oracle_EBS_R12.2.4_linux_x86-64/TechInstallMedia/database/database/runInstaller -waitForCompletion -ignoreSysPrereqs -force -silent -

responseFile /u01/oracle/VIS/11.2.0/temp/VIS_usera1/cfgHome/response/DB_HOME/txkDB11gR2_11203.rsp CAUSE: The selected Oracle home contained directories
or files. ACTION: To start with an empty Oracle home, either remove its contents or choose another location.
FATAL [INS-13013] Target environment do not meet some mandatory requirements. CAUSE: Some of the mandatory prerequisites are not met. See logs
for details. /u01/app/oraInventory/logs/installActions2014-12-23_11-11-42PM.log ACTION: Identify the list of failed prerequisite checks from the log:
/u01/app/oraInventory/logs/installActions2014-12-23_11-11-42PM.log. Then either from the log file or from installation manual find the appropriate
configuration to meet the prerequisites and fix it manually.
OK, so I should worry about the FATAL
task resolv.conf Integrity Failed
RVF-5637 : DNS response time could not be checked on following nodes: usera1,usera2 Cause: An attempt to check DNS response time for unreachable
node failed on nodes specified.
MOS: PRVF-5637 : DNS response time could not be checked on following nodes (Doc ID 1480242.1)
FATAL [INS-13013] Target environment do not meet some mandatory requirements.
MOS: config.sh in Silent Mode Fails With INS-13013: Target environment does not meet some mandatory requirements. (Doc ID 1925578.1)
$ /STAGING/oracle_EBS_R12.2.4_linux_x86-64/TechInstallMedia/database/database/runInstaller -waitForCompletion -ignoreSysPrereqs -force -silent -

ignorePrereq -responseFile /u01/oracle/VIS/11.2.0/temp/VIS_usera1/cfgHome/response/DB_HOME/txkDB11gR2_11203.rsp
OK, we need to do some hacking ```
cd /STAGING/oracle_EBS_R12.2.4_linux_x86-64/startCD/Disk1/rapidwiz
TechInstallers/DBInstallHome/fnd/admin/template/txkCfgDB11gR2_tmpl.xml: ```
```
cp
TechInstallers/DBInstallHome/fnd/admin/template/txkCfgDB11gR2_tmpl.x
/var/tmp
vim
diff
/var/tmp
12c12
<
```
restoring scan VIP> yes, that helped!!! With some hacking, but anyway it worked.
ora-01102 cannot mount database in exclusive mode
$ . VIS_usera1.env $ env | grep SID ORACLE_SID=VIS
> this should be VIS1? $ export ORACLE_SID=VIS1
ORA-00132: syntax error or unresolved network name 'VIS1_LOCAL'
```
startup ORA-00119: invalid specification for system parameter LOCAL_LISTENER ORA-00132: syntax error or unresolved network name
'VIS1_LOCAL' ```
OPatchActionNode.processState(OPatchActionNode.java:312)
oracle.apps.fnd.txk.config.ProcessStateException: OPatch process failed : Exit=1 See log for details. CMD= /u01/oracle/VIS/11.2.0/OPatch/opatch napply
- skip_subset -skip_duplicate -verbose -silent -ocmrf /u01/oracle/VIS/11.2.0/temp/VIS_usera1/DBInstallHome/fnd/admin/template/txkForms_ocm.rsp
/u01/ora cle/VIS/11.2.0/temp/VIS_usera1/patches/16342486/16342486/
at oracle.apps.fnd.txk.config.OPatchActionNode.processState(OPatchActionNode.java:312)
When running from hand $ export ORACLE_HOME=/u01/oracle/VIS/11.2.0 $ /u01/oracle/VIS/11.2.0/OPatch/opatch napply -skip_subset -skip_duplicate -verbose
-silent -ocmrf /u01/oracle/VIS/11.2.0/temp/VIS_usera1/DBInstallHome/fnd/admin/template/txkForms_ocm.rsp
/u01/oracle/VIS/11.2.0/temp/VIS_usera1/patches/16342486/16342486/ /bin/java is not a valid executable for this platform. OPatch cannot proceed!
MOS: R122 installation failes
$ cd /u01/oracle/VIS/11.2.0/OPatch $ ./opatch lsinventory /bin/java is not a valid executable for this platform. OPatch cannot proceed! OPatch returns
with error code = 1
```
cd /bin/
ln -s /usr/bin/java
```
$ ./opatch lsinventory Oracle Interim Patch Installer version 11.2.0.3.0 Copyright (c) 2012, Oracle Corporation. All rights reserved. Oracle Home :
/u01/oracle/VIS/11.2.0 Oracle Home Inventory : n/a OPatch version : 11.2.0.3.0 Product information : n/a Log file location :
/u01/oracle/VIS/11.2.0/cfgtoollogs/opatch/opatch2014-12-25_23-27-22PM_1.log
Patch history file: /u01/oracle/VIS/11.2.0/cfgtoollogs/opatch/opatch_history.txt
Lsinventory Output file location : /u01/oracle/VIS/11.2.0/cfgtoollogs/opatch/lsinv/lsinventory2014-12-25_23-27-22PM.txt
------------------------------------------------------------------------------- There are no Interim patches installed in this Oracle Home.
-------------------------------------------------------------------------------
OPatch succeeded.
Again the patching:
$ /u01/oracle/VIS/11.2.0/OPatch/opatch napply -skip_subset -skip_duplicate -verbose -silent -ocmrf

/u01/oracle/VIS/11.2.0/temp/VIS_usera1/DBInstallHome/fnd/admin/template/txkForms_ocm.rsp
/u01/oracle/VIS/11.2.0/temp/VIS_usera1/patches/16342486/16342486/
RW-50004: An error has occured
$ cd /u01/oracle/VIS/11.2.0/appsutil/log/VIS_usera1 $ less 12261002.log
Fatal Error: TXK Install Service
oracle.apps.fnd.txk.config.ProcessStateException: OPatch process failed : Exit=73 See log for details. CMD= /u01/oracle/VIS/11.2.0/OPatch/opatch napply
-skip_subset -skip_duplicate -verbose -silent -ocmrf /u01/oracle/VIS/11.2.0/temp/VIS_usera1/DBInstallHome/fnd/admin/template/txkForms_ocm.rsp
/u01/oracle/VIS/11.2.0/temp/VIS_usera1/patches/16342486/16342486/
at oracle.apps.fnd.txk.config.OPatchActionNode.processState(OPatchActionNode.java:312)
at oracle.apps.fnd.txk.config.PatchActionNode.processState(PatchActionNode.java:187)
> He is trying to patch the Oracle Home DB binaries, but there is none now - he did not istall them yet.
This with 49 rapid install.
Trying with 48
Cleanup
```
show parameter db_name -> upewnij się, że to na pewno baza, którą chcesz skasować :) alter system set cluster_database=FALSE
scope=spfile; # if that is a RAC DB shutdown abort startup mount restrict $ rman target / delete archivelog all; drop database; ```
$ cd /STAGING/oracle11.2.0.4_x86-64/deinstall $ ./deinstall -home /u01/oracle/VIS/11.2.0
Remove entry from oratab (both nodes) ```
vi /etc/oratab
```
```
cat /u01/app/oraInventory/ContentsXML/inventory.xml | grep "HOME

NAME"
```
Kill local listener if still running > not needed, will be done during binaries deinstall
What IS working
RapidWiz 48 + Single node DB + Port pool 5,6 RapidWiz 48 + RAC DB + Port pool 5,6
With port pool different than default he is able to start the listener: ```
ps -ef | grep tns

oracle 3072 1 0 Dec24 ? 00:00:01 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit oracle 3829 1 0 Dec24 ? 00:00:01 /u01/app/11.2.0/grid/bin/tnslsnr
LISTENER_SCAN2 -inherit oracle 3833 1 0 Dec24 ? 00:00:01 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit oracle 13098 1 0 20:55 ?
00:00:00 /u01/oracle/VIS/11.2.0/bin/tnslsnr VIS -inherit ```
otherwise the Grid listener running on 1521 is blocking the port.
What is NOT working
RapidWiz 49 + RAC DB + Port pool or default RapidWiz 49 + Single node DB + Port pool 5,6 or default
Issues: He is trying to apply a patch to DB Oracle Home, but the binaries are not there yest It is not possible to add another MT nodes
Post install issues

restart causes ORA-00119: invalid specification for system parameter LOCAL_LISTENER
```
shutdown abort ORACLE instance shut down. SQL> startup mount restrict ORA-00119: invalid specification for system parameter
LOCAL_LISTENER ORA-00132: syntax error or unresolved network name 'VIS_LOCAL' ```
$ env | grep TNS (nothing)
$ . VIS_usera1.env $ env | grep TNS TNS_ADMIN=/u01/oracle/VIS/11.2.0/network/admin/VIS_usera1
Conclusion: Set properly the environment, not by just using oraenv.
Installing MT
Pre install actions
+ Very large CPu utilisation on DB tier by ologgerd
MOS: Node Eviction due to OLOGGERD High CPU (Doc ID 1636942.1)
Apply Patch 13867435 - OLOGGERD USING A LOT OF RESOURCES on top of 11.2.0.3. The bug is fixed in 11.2.0.4 GI PSU.
Checking GI $ cd /u01/app/11.2.0/grid/OPatch $ opatch lsinventory Invoking OPatch 11.2.0.1.7 Oracle Grid Infrastructure 11.2.0.3.0 There are no Interim
patches installed in this Oracle Home.
Downloading the one for 11.2.0.3.0 OLOGGERD USING A LOT OF RESOURCES (Patch) p13867435_112030_Linux-x86-64.zip 156.7 MB MD5
6204BEE6197A14AE24A0511E16005938
$ md5sum p13867435_112030_Linux-x86-64.zip 6204bee6197a14ae24a0511e16005938 p13867435_112030_Linux-x86-64.zip $ cd gi_patch_13867435 $ unzip

p13867435_112030_Linux-x86-64.zip
Patching the OPatch OPatch patch of version 11.2.0.3.6 for Oracle software releases 11.2.0.x (DEC 2013) (Patch) p6880880_112000_Linux-x86-64.zip
31.5 MB MD5 2AB78CB17174C3A49DF560F05E3352C6
Source GI env $ export PATH=$ORACLE_HOME/OPatch:$PATH $ opatch version OPatch Version: 11.2.0.1.7
``` $ cd /STAGING/oracle11.2.0.4_x86-64 $ md5sum p6880880_112000_Linux-x86-64.zip 2ab78cb17174c3a49df560f05e3352c6

p6880880_112000_Linux-x86-64.zip
chmod g+w /u01/app/11.2.0/grid

$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_/bin/date +%Y%m%d $ unzip -d $ORACLE_HOME p6880880_112000_Linux-x86-64.zip $ opatch
version OPatch Version: 11.2.0.3.6
chmod g-w /u01/app/11.2.0/grid

```
Actual patching - auto, as root $ cd 13867435 $ pwd /STAGING/oracle11.2.0.4_x86-64/13867435 $ cd /STAGING/oracle11.2.0.4_x86-64 $

/u01/app/11.2.0/grid/OPatch/ocm/bin/emocmrsp
$ which opatch /u01/app/11.2.0/grid/OPatch/opatch
Run on both RAC nodes ```
export PATH=$PATH:/u01/app/11.2.0/grid/OPatch
opatch auto /STAGING/oracle11.2.0.4_x86-64/gi_patch_13867435 -ocmrf

/STAGING/oracle11.2.0.4_x86-64/ocm.rsp
```
+ Setup flashback DB
Archivelog? ```
archive log list Database log mode No Archive Mode Automatic archival Disabled Archive destination /u01/oracle/VIS/11.2.0/dbs/archive ```
Set archivelog mode ```
alter system set log_archive_dest_1='location=+DATA' scope=spfile sid='*'; shutdown immediate startup mount alter database archivelog; alter
database open; ```
```
archive log list Database log mode Archive Mode Automatic archival Enabled Archive destination +DATA ```
Flashback DB? ```
select FLASHBACK_ON from v$database;
FLASHBACK_ON
NO ```
Configure flash recovery area ```
alter system set DB_RECOVERY_FILE_DEST='+DATA' scope=spfile; alter system set DB_RECOVERY_FILE_DEST_SIZE=20000M

scope=spfile; ```
Enable ```
STARTUP MOUNT; ALTER SYSTEM SET DB_FLASHBACK_RETENTION_TARGET=4320; # 3 days ALTER DATABASE FLASHBACK ON;
alter database open; ```
Test table create table skarby (name varchar(50), value number, timestampid date not null, scn_number number) tablespace sysaux; insert into skarby
values ('after DB install',1, sysdate, userenv('commitscn')); commit;
column name format a30 set linesize 200 column SCN_NUMBER format 999999999999999; alter session set NLS_DATE_FORMAT = "YYYY/MM/DD
HH24:MI:SS"; select from skarby;
NAME VALUE TIMESTAMPID SCN_NUMBER ----------------------------- ---------- ------------------- ---------------- after DB install 1 2015/01/02 14:00:16
12204575949252
Show earliest SCN in the flashback database window ```
SELECT OLDEST_FLASHBACK_SCN, OLDEST_FLASHBACK_TIME FROM V$FLASHBACK_DATABASE_LOG; ```
Show stats for every hour how much data has the flashback, redo and db to write. ```
select * from V$FLASHBACK_DATABASE_STAT; ```
Creating restore point ```
create restore point after_DB_install guarantee flashback database; ```

Going back to SCN
column name format a30 set linesize 200 column SCN_NUMBER format 999999999999999; alter session set NLS_DATE_FORMAT = "YYYY/MM/DD
HH24:MI:SS"; select from skarby;
12204575951690
insert into skarby values ('failed 2 MT install',1, sysdate, userenv('commitscn')); commit;
```
select * from skarby; ```
12204575951690 failed 2 MT install 1 2015/01/02 13:08:07 12204580463127
Actuall rollback to SCN ```
startup mount flashback database to restore point BEFORE_FAILOVER; or flashback database to SCN 12204575951690; ```
I Prevent RW-50016: Error - {0} Was Not Created
MOS: EBS R12.2.0 Installation on Oracle Linux 6 --Post Install Checks Error RW-50016: Error - {0} Was Not Created (Doc ID 1921643.1)
On DB odes, which are OEL 6
```
vi /etc/security/limits.d/90-nproc.conf
cat /etc/security/limits.d/oracle-ebs-server-R12-preinstall.conf
-> here this is set correctly ```
The cause of the error is that there are not enough of resources devoted to app tiers (mainly RAM). See below, the same error is analysed.
Actuall install
During the DB installation the follwing messages were displayed: ```
Configuration file written to: /u01/oracle/VIS/11.2.0/appsutil/conf_VIS.txt

```
Do this on all 3 MT nodes. ```
vncserver
cd /STAGING/oracle_EBS_R12.2.4_linux_x86-64/startCD/Disk1/rapidwiz
./rapidwiz
```
Provide configuration:
Weblogic Server Password: invent new password for Weblogic user Apps OS User Password: this the Linux password for user applmgr. Apps DB User
Password: this the database password for user APPS, which is by default apps.
```
Second File System logfile -

/RYBAK/oracle/VIS/fs2/inst/apps/VIS_okon1/logs/12262127.log
```
Cleanup (if there is a need to repeat the installation)
$ cd /RYBAK $ mkdir trash_failed2_MT_install $ mv oracle trash_failed2_MT_install/ $ mkdir oracle $ mv oraInventory trash_failed2_MT_install/ $ mkdir
oraInventory $ chmod 777 oraInventory
Post install checks fails

First Install, 3 fails
Turns out that all the below failures are the result of too little resources for MT node.
I tried with 2500M for RAM, 1 CPU and it fails.
Works with min 5GB for RAM.
Virtual Directory
Virtual Directory checking URL = RW-50016: Error: - {0} was not created: File = {1} checking URL =
http://okon1.remik.org:8005/OA_JAVA/oracle/apps/fnd/admin/server/server.xml RW-50016: Error: - {0} was not created: File = {1} checking URL =
http://okon1.remik.org:8005/OA_CGI/FNDWRR.exe RW-50016: Error: - {0} was not created: File = {1}
MOS: Troubleshooting Rapid Install for E-Business Suite Release 12.2 (Doc ID 1378579.1) EBS R12.2.0 Installation on Oracle Linux 6 --Post Install
Checks Error RW-50016: Error - {0} Was Not Created (Doc ID 1921643.1) To Bottom
Login Page
Login Page checking URL = http://okon1.remik.org:8005/OA_HTML/AppsLogin RW-50016: Error: - {0} was not created: File = {1}
MOS: Rapidwiz fails with RW-50016: Error: - {0} was not created: File = {1} (Doc ID 1491399.1)
Make sure that time is in sync on DB and MT tiers
JSP
JSP checking URL = http://okon1.remik.org:8005/OA_HTML/jtfTestCookie.jsp JSP is not responding, waiting 15 seconds and retesting. JSP is not
responding, waiting 15 seconds and retesting. JSP is not responding, waiting 15 seconds and retesting. JSP is not responding, waiting 15 seconds and
retesting. JSP is not responding, waiting 15 seconds and retesting. JSP is not responding, waiting 15 seconds and retesting.
Second install, 1 fail
Only Login Page fails.
This time I used 5G RAM, 2 CPU for one MT node.
It was because of the time it took to generate the login page for the first time. Open a browser, let it generate and retest the issue. It should be fine.
http://okon1.remik.org:8005/OA_HTML/AppsLogin
Post install issues
sqlplus VIS gives ORA-12505: TNS:listener
On MT okon1:
$ tnsping VIS Attempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=usera1)(PORT=1526)) (CONNECT_DATA=(SID=VIS))) OK (0 msec)
$ sqlplus apps@VIS ORA-12505: TNS:listener does not currently know of SID given in connect
This is what autoconfig tries to use:
(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=usera-scan)(PORT=1521)))
(CONNECT_DATA=(SERVICE_NAME=VIS)))
```
scan
192.168.1.54 192.168.1.55 192.168.1.56 usera-scan

```
Scan listener stopped working $ ping usera-scan $ nslookup usera-scan $ nslookup usera-scan.remik.org (all hangs)
Checking DNS (separate server) ```
/etc/init.d/named restart
```
$ nslookup usera-scan Server: 192.168.1.10 Address: 192.168.1.10#53
Name: usera-scan.remik.org Address: 192.168.1.54 Name: usera-scan.remik.org Address: 192.168.1.55 Name: usera-scan.remik.org Address:
192.168.1.56
> OK, works again, issue caused by DNS
missing EBSApps.env
> Run autoconfig
$ ls /RYBAK/oracle/VIS fs1 fs2 fs_ne
$ . /RYBAK/oracle/VIS/fs1/EBSapps/appl/APPSVIS_okon1.env $ env | grep FILE_EDITION FILE_EDITION=run
$ . /RYBAK/oracle/VIS/fs2/EBSapps/appl/APPSVIS_okon1.env $ env | grep FILE_EDITION FILE_EDITION=patch
Running autoconfig. $ cd $INST_TOP/admin/scripts $ ./adautocfg.sh -> Still nothing.
Repeating the MT installation. Because of the bug 13867435 the RAC DB was crashed somewehre during last node installation. All the post install MT
checks were OK, but it looked weird.
adop -status ORA-06550: line 6, column 2
``` $ adop -status Enter the APPS username: apps Enter the APPS password:
where adop_session_id=; * ERROR at line 6: ORA-06550: line 6, column 23: PL/SQL: ORA-00936: missing expression ORA-06550: line 4, column 1:
PL/SQL: SQL Statement ignored
adop exiting with status = 1 (Fail) ```
Do not worry, looks like no adop cycle was run yet.
This will be corrected with first patch, or if you really want to you can run "empty" adop cycle: $ adop phase=prepare $ adop phase=cutover $ adop
phase=cleanup
``` $ adop -status Node Name Node Type Phase Status Started Finished Elapsed
okon1 master PREPARE RUNNING 29-DEC-14 12:58:37 +01:00 APPLY NOT STARTED CLEANUP NOT STARTED CUTOVER NOT STARTED ```
okon2 slave PREPARE RUNNING 29-DEC-14 12:58:37 +01:00 APPLY NOT STARTED CLEANUP NOT STARTED CUTOVER NOT STARTED
okon3 slave PREPARE RUNNING 29-DEC-14 12:58:37 +01:00 APPLY NOT STARTED CUTOVER NOT STARTED CLEANUP NOT STARTED
adop phase=prepare failed to create node-cmd.txt file Permission denied
To
$ adop phase=prepare Fils on: $ perl /RYBAK/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl verifyssh -

contextfile=/RYBAK/oracle/VIS/fs1/inst/apps/VIS_okon1/appl/admin/VIS_okon1.xml -hosts=okon1,okon2,okon3 ERROR DESCRIPTION: ( failed to create node-
cmd.txt file Permission denied )
$ find ./ > file_list.txt $ grep node-cmd.txt file_list.txt ./oracle/VIS/fs1/inst/apps/VIS_okon1/logs/appl/rgf/TXK/node-cmd.txt

./oracle/VIS/fs1/inst/apps/VIS_okon2/logs/appl/rgf/TXK/node-cmd.txt ./oracle/VIS/fs1/inst/apps/VIS_okon3/logs/appl/rgf/TXK/node-cmd.txt
$ ls -l ./oracle/VIS/fs1/inst/apps/VIS_okon1/logs/appl/rgf/TXK/node-cmd.txt -rw-r--r-- 1 root root 28 Dec 27 22:13

./oracle/VIS/fs1/inst/apps/VIS_okon1/logs/appl/rgf/TXK/node-cmd.txt
```
chown -R applmgr:oinstall
/RYBAK/oracle/VIS/fs1/inst/apps/VIS_okon1/logs/appl/rgf/TXK
```
$ perl /RYBAK/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl verifyssh -

contextfile=/RYBAK/oracle/VIS/fs1/inst/apps/VIS_okon1/appl/admin/VIS_okon1.xml -hosts=okon1,okon2,okon3
Setup passwordless login from main MT node to all the others.
$ perl /RYBAK/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl verifyssh -

contextfile=/RYBAK/oracle/VIS/fs1/inst/apps/VIS_okon1/appl/admin/VIS_okon1.xml -hosts=okon1,okon2,okon3 Logfile location
/RYBAK/oracle/VIS/fs1/inst/apps/VIS_okon1/logs/appl/rgf/TXK/verifyssh.log xml output =
/RYBAK/oracle/VIS/fs1/inst/apps/VIS_okon1/logs/appl/rgf/TXK/out.xml $ cat /RYBAK/oracle/VIS/fs1/inst/apps/VIS_okon1/logs/appl/rgf/TXK/verifyssh.log ->
SSH equivalence setup verification completed successfully
Error while executing FS config clone on slave nodes Unable to open forms: FRM-92101: There was a failure in the Forms Server during startup
no response from runtime process
MOS: R12: "FRM-92101:There was a failure in the Forms Server during startup" Error When Attempting to Launch Forms (Doc ID 454427.1)
$ pwd /RYBAK/oracle/VIS/fs1/FMW_Home/webtier/lib32 $ ls -l ldflags -rw-r----- 1 applmgr oinstall 111 Dec 29 19:53 ldflags $ diff
/RYBAK/oracle/VIS/fs1/FMW_Home/webtier/lib/ldflags ldflags
> looks fine
MOS: R12: "FRM-92101: There Was a Failure in the Forms Server During Startup" Error After Fresh R12 Installation (Doc ID 429627.1)
$ pwd /RYBAK/oracle/VIS/fs1/EBSapps/10.1.2/forms/lib $ make -f ins_forms.mk install
> This helps.
$ cd /RYBAK/oracle/VIS/fs2/EBSapps/10.1.2/forms/lib $ make -f ins_forms.mk install

ORA-20008: No Concurrent Manager is defined that can run concurrent program ADZDPATCH
$ adop phase=prepare SQL> SQL> Connected. SQL> select AD_ZD_ADOP.WAIT_FOR_CP_TO_RUN(7437014) from dual * ERROR at line 1: ORA-20008: No Concurrent
Manager is defined that can run concurrent program ADZDPATCH ORA-06512: at "APPS.AD_ZD_ADOP", line 248
MOS: Oracle E-Business Suite Technology Stack and Applications DBA Release Notes for Release 12.2.3 (Doc ID 1606170.1)
In the adop prepare phase, an error message may state: ORA-20008: No Concurrent Manager is defined that can run concurrent program ADZDPATCH.
Error details:This error does not appear consistently. Related bug: 1004803 Workaround: Log in to the Concurrent Program Define form and ensure the
executable is of "SQL*Plus" method. You can pick up the correct executable from the LOV, and save the correct definition
MOS: Enter the RUN file system context file name
Nice links: Sharing The Application Tier File System in Oracle E-Business Suite Release 12.2 (Doc ID 1375769.1) How To Automatically Set the
Current Run or Patch Edition / File System for EBS 12.2 (Doc ID 1545584.1) How To Avoid Entering The Apps Password For Every Autoconfig Run?
(Doc ID 1546336.1) Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone (Doc ID 1383621.1)
Why I can not find the EBSenv file?
/RYBAK/oracle/VIS/EBSapps.env run
On last R12.2.2 install and upgrade to 12.2.3 this file came with
R12.TXK.C.DELTA.4 (Patchset)
adop cycle
$ adop phase=prepare Failed.
Resart all the MT ``` $ adop phase=prepare adop exiting with status = 0 (Success) $ adop -status Node Name Node Type Phase Status Started Finished
Elapsed
okon1 master PREPARE COMPLETED 01-JAN-15 07:34:03 +01:00 01-JAN-15 09:07:28 +01:00 1:33:25 okon2 slave PREPARE COMPLETED 01-JAN-
15 07:34:03 +01:00 01-JAN-15 09:07:28 +01:00 1:33:25 okon3 slave PREPARE RUNNING 01-JAN-15 07:34:03 +01:00 01-JAN-15 09:07:28 +01:00
1:33:25 ```
$ adop phase=cutover [ERROR] First run the tool for prepare and apply phases
That is probably because of the RUNNING status on okon3 node $ adop -status okon3 slave PREPARE FAILED 01-JAN-15 07:34:03 +01:00 01-JAN-15 09:07:28
+01:00 1:33:25
Looking at adop logs

/RYBAK/oracle/VIS/fs_ne/EBSapps/log/adop/3/prepare_20150101_193156/VIS_okon3/TXK_SYNC_migrate_Thu_Jan_1_20_55_28_2015$ less
txkADOPPreparePhaseSynchronize_Thu_Jan_1_20_55_28_2015.log FATAL ERROR PROGRAM :
(/RYBAK/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl) TIME : Thu Jan 1 20:58:36 2015 FUNCTION:
main::migrateCloneComponentStage Level 1 ERRORMSG: /RYBAK/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl did not go through
successfully.
Inreasing RAM to 4G
admanagedsrvctl.sh: exiting with status 1
http://okon1.remik.org:7006/console Username: weblogic
ERROR: Unable to start up the managed server oacore_server3 oacore_server3, State ADMIN Control, Resume oacore_server3, State RUNNING
Back to prepare
$ adop phase=prepare adop exiting with status = 0 (Success)
$ adop -status okon1 master PREPARE COMPLETED okon2 slave PREPARE COMPLETED okon3 slave PREPARE RUNNING
STATEMENT Node:
<okon3>
Status:
<failed>
$ less
/RYBAK/oracle/VIS/fs_ne/EBSapps/log/adop/3/prepare_20150101_232959/VIS_okon3/TXK_SYNC_migrate_Thu_Jan_1_23_59_42_2015/wlsTechStack_FSCloneStageAppsTier_010
Copying /RYBAK/oracle/VIS/fs1/FMW_Home/webtier/inventory/Scripts to /RYBAK/oracle/VIS/fs1/EBSapps/comn/clone//prereq/webtier Could not generate

new Clone stage base area RC-50007: Could not create cloning stage area at /RYBAK/oracle/VIS/fs1/EBSapps/comn/clone/ Raised by
oracle.apps.ad.clone.util.CreateCloningStage
Looks like he tries to run a preclon and it fails.
Trying to run preclone
$ cd $INST_TOP/admin/scripts $ perl adpreclone.pl appsTier
okon1 - ok okon2 - ok okon3 - ok
Again looking at logs from preclone, Failed to copy OUI library
$ less
/RYBAK/oracle/VIS/fs_ne/EBSapps/log/adop/3/prepare_20150101_232959/VIS_okon3/TXK_SYNC_migrate_Thu_Jan_1_23_59_42_2015/wlsTechStack_FSCloneStageAppsTier_010
Testing if this home is s_weboh_oh... This home is not s_weboh_oh...will procede with getOUI method... Copy OUI from
/RYBAK/oracle/VIS/fs1/EBSapps/10.1.2/oui/jlib to /RYBAK/oracle/VIS/fs1/EBSapps/comn/clone//jlib/oui/ RC-00012: Error while creating cloning stage
area. Failed to copy OUI library /RYBAK/oracle/VIS/fs1/EBSapps/10.1.2/oui/jlib to /RYBAK/oracle/VIS/fs1/EBSapps/comn/clone//jlib/oui/ Raised by
oracle.apps.ad.clone.util.CreateCloningStage
Copy OUI from /RYBAK/oracle/VIS/fs1/EBSapps/10.1.2/oui/jlib to /RYBAK/oracle/VIS/fs1/EBSapps/comn/clone//jlib/oui/
And it looks like beeing copied.
MOS: Oracle E-Business Suite Technology Stack and Applications DBA Release Notes for Release 12.2.3 (Doc ID 1606170.1)
Manually run the prepare phase on the slave node using the command: $ adop phase=prepare allnodes=no action=nodb
Errors...
I will run the prepare manually on each node now. $ adop phase=abort okon1 master PREPARE SESSION ABORTED okon2 slave PREPARE SESSION ABORTED okon3 slave
PREPARE SESSION ABORTED
okon1 $ adop phase=prepare allnodes=no action=db fails $ adop phase=abort
$ adop phase=prepare
No idea. Reinstalling everything. Now chosing all the same services on all the nodes.
Post install 3
+ $ adop phase=prepare failed to create node-cmd.txt file Permission denied shutdown all MT startup all MT
Error occurred while executing txkADOPEvalSrvStatus.pl
$ less /RYBAK/oracle/VIS/fs_ne/EBSapps/log/adop/2/adop_20150102_230602.log
$ perl /RYBAK/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl -
contextfile=/RYBAK/oracle/VIS/fs1/inst/apps/VIS_okon1/appl/admin/VIS_okon1.xml -
patchcontextfile=/RYBAK/oracle/VIS/fs2/inst/apps/VIS_okon1/appl/admin/VIS_okon1.xml -phase=prepare -promptmsg=hide -nodelist=okon1 -console=off -
sessionid=2 -timestamp=20150102_230602 -outdir=/RYBAK/oracle/VIS/fs_ne/EBSapps/log/adop/2/prepare_20150102_230602/VIS_okon1
$ perl /RYBAK/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl -
patchcontextfile=/RYBAK/oracle/VIS/fs2/inst/apps/VIS_okon1/appl/admin/VIS_okon1.xml -phase=prepare -nodelist=okon1 -sessionid=2 -
timestamp=20150102_230602 -outdir=/RYBAK/oracle/VIS/fs_ne/EBSapps/log/adop/2/prepare_20150102_230602/VIS_okon1
Program : completed @ Sat Jan 3 00:39:24 2015 Returning FAILURE
> no errors in logs, just returning failure
Someone has similar issue https://community.oracle.com/thread/3591770?start=0&tstart=0
$ adop phase=abort
This is still bothering me: Why I can not find the EBSenv file?
/RYBAK/oracle/VIS/EBSapps.env run
On last R12.2.2 install and upgrade to 12.2.3 this file came with
R12.TXK.C.DELTA.4 (Patchset)
I will try to install latest R12.TXK.C

Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2
MOS: Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2 (Doc ID 1617461.1) Oracle E-Business
Suite Release 12.2: Now Available - Essential Bundle Fixes for AD Delta 5 and TXK Delta 5 (Doc ID 1934471.1)
Path A: Installing a new Oracle E-Business Suite system
1. Source the run edition environment file.
$ . /RYBAK/oracle/VIS/fs1/EBSapps/appl/APPSVIS_okon1.env $ env | grep FILE_EDITION FILE_EDITION=run
2. Apply the latest application tier technology patches to the run file system.
Used rapid install 48, should be pretty current.
To avoid: ERROR The EBS Technology Codelevel Checker needs to be installed on this instance. It is available as Patch 17537119.
I will run Codelevel Checker
On DB tier $ cp /STAGING/ebs12patches/p17537119_R12_GENERIC.zip /tmp $ cd /tmp
$ md5sum p17537119_R12_GENERIC.zip f5603e3d73efa99f54a347e2aca294eb p17537119_R12_GENERIC.zip
$ unzip p17537119_R12_GENERIC.zip
Source DB env $ ./checkDBpatch.sh Enter ORACLE_HOME value : /u01/oracle/VIS/11.2.0 on first node Enter ORACLE_SID value : VIS1 on second node Enter
ORACLE_SID value : VIS2
3. Start up only the Weblogic AdminServer service on the run edition application tier file system.
$ sh $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start
4. Unzip patches and run adgrants.
AD R12.AD.C.DELTA.5 (Patchset) p18283295_R12.AD.C_R12_LINUX.zip 3.2 MB MD5 CBB8D2C0BEC8FBE9C577EFD1814A14EC
November Bundle Patches for AD Delta 5 (Patch) p20034256_R12.AD.C_R12_LINUX.zip 1.3 MB MD5 F3187D71BB4255D4854FC4AF8F18FFED
ADOP IS NOT DROPPING THE OLD INDEX IF A NEW INDEX IS PRESENT ON THE SAME COLUMNS (Patch)
p20075842_R12.AD.C_R12_GENERIC.zip 28.5 KB MD5 DB1F4CE092EBB158EA3ED707620D89CA
R12.TXK.C.DELTA.5 (Patchset) p18288881_R12.TXK.C_R12_GENERIC.zip 57.5 MB MD5 B2AB4F599A070B730D4048BB7038003F
November Bundle Patches for TXK Delta 5 (Patch) p20043910_R12.TXK.C_R12_GENERIC.zip 26.9 MB MD5 8CA68B0148976E9DF144AA4100318C96
TST1226: ADD NODE PICKED EXISTING OHS INSTEAD OF CREATING NEW (Patch) p19803392_R12.TXK.C_R12_GENERIC.zip 54.0 KB MD5
C4A6A85CFFE9E0C7818B99E9A34F0B88
TST1226:FEW MANAGED SERVERS STARTED IN 'ADMIN' MODE (Patch) p20158287_R12.TXK.C_R12_GENERIC.zip 16.8 KB MD5
AB01DCC314CB6219E9624228E0B7E119
ERROR WHEN OPENING FORMS IN IE8 ON MULTI-NODE EBS 12.2.3 (Patch) p19259764_R12.FND.C_R12_GENERIC.zip 316.0 KBMD5
9B504769BA39378FA47DFA6B95899E04
$ cp p18283295_R12.AD.C_R12_LINUX.zip p20034256_R12.AD.C_R12_LINUX.zip p20075842_R12.AD.C_R12_GENERIC.zip p18288881_R12.TXK.C_R12_GENERIC.zip

p20043910_R12.TXK.C_R12_GENERIC.zip p19803392_R12.TXK.C_R12_GENERIC.zip p20158287_R12.TXK.C_R12_GENERIC.zip p19259764_R12.FND.C_R12_GENERIC.zip
/RYBAK/oracle/VIS/fs_ne/EBSapps/patch
$ cd /RYBAK/oracle/VIS/fs_ne/EBSapps/patch $ md5sum * cbb8d2c0bec8fbe9c577efd1814a14ec p18283295_R12.AD.C_R12_LINUX.zip b2ab4f599a070b730d4048bb7038003f

p18288881_R12.TXK.C_R12_GENERIC.zip 9b504769ba39378fa47dfa6b95899e04 p19259764_R12.FND.C_R12_GENERIC.zip c4a6a85cffe9e0c7818b99e9a34f0b88
p19803392_R12.TXK.C_R12_GENERIC.zip f3187d71bb4255d4854fc4af8f18ffed p20034256_R12.AD.C_R12_LINUX.zip 8ca68b0148976e9df144aa4100318c96
p20043910_R12.TXK.C_R12_GENERIC.zip db1f4ce092ebb158ea3ed707620d89ca p20075842_R12.AD.C_R12_GENERIC.zip ab01dcc314cb6219e9624228e0b7e119
p20158287_R12.TXK.C_R12_GENERIC.zip
unzip p18283295_R12.AD.C_R12_LINUX.zip unzip p18288881_R12.TXK.C_R12_GENERIC.zip unzip p19259764_R12.FND.C_R12_GENERIC.zip unzip

p19803392_R12.TXK.C_R12_GENERIC.zip unzip p20034256_R12.AD.C_R12_LINUX.zip unzip p20043910_R12.TXK.C_R12_GENERIC.zip unzip
p20075842_R12.AD.C_R12_GENERIC.zip unzip p20158287_R12.TXK.C_R12_GENERIC.zip
Execute adgrants by following the instructions in the readme of Patch 20034256:R12.AD.C. $ cd /RYBAK/oracle/VIS/fs_ne/EBSapps/patch/20034256/admin
Copy that to db node $ scp adgrants.sql oracle@usera1:/u01/oracle/VIS/11.2.0/appsutil/admin
On DB node run adgrants ```
@?/appsutil/admin/adgrants.sql apps ```

5. On the run edition application tier file system, use hotpatch mode to apply Patch 18283295:R12.AD.C.Delta.5
This is performed outside of the usual adop patching cycle.
$ adop phase=apply patches=18283295,20034256,20075842 hotpatch=yes merge=yes adop exiting with status = 0 (Success)
6. On the run edition application tier file system, use hotpatch mode to apply Patch 18288881:R12.TXK.C.Delta.5
$ adop phase=apply patches=18288881,20043910,19803392,20158287 hotpatch=yes merge=yes adop exiting with status = 0 (Success)
And now!!! Finally we have /RYBAK/oracle/VIS/EBSapps.env
7. Apply Patch 19259764:R12.FND.C (conditional).
Skip this step if you are a new installation customer. > Skipping
Source the run edition environment file.
$ . /RYBAK/oracle/VIS/EBSapps.env run
Again adop cycle
$ adop phase=prepare
Executing "txkADOPValidation" action in all nodes ERROR Node: "okon1" Status: "failed"
ERROR: The following required ports are in use: ---------------------------- 7007 : WLS Admin Server Port Corrective Action: Free the listed ports and
retry the adop operation.
$ telnet okon1 7007 Connected
Stop MT
$ telnet okon1 7007 Connected
?? What is listeneing on that port? $ netstat -tulpn | grep 7007 tcp 0 0 192.168.1.29:7007 0.0.0.0:* LISTEN 13540/java $ ls -l /proc/13540/exe lrwxrwxrwx
1 applmgr oinstall 0 Jan 3 23:58 /proc/13540/exe -> /RYBAK/oracle/VIS/fs2/EBSapps/comn/util/jdk64/bin/java
Killed the processes.
Again. $ adop phase=prepare Data Dictionary Corruption Details - TS Mismatch
Data Dictionary Corruption Details - TS Mismatch
Data Dictionary Corruption Details - TS Mismatch OWBSYS HAB_RT_EXEC_PROC_RUN_METRICS OWBSYS WB_RT_VERSION TS mismatch: 16-
SEP-10 09:13:44 26-NOV-12 18:48:19 OWBSYS WB_RTI_OBJECT_CATALOG OWBSYS WB_RT_VERSION TS mismatch: 16-SEP-10 09:13:44 26-
NOV-12 18:48:19 (...) HINT Follow the instructions in "Fix data dictionary corruption" section of My Oracle Support Knowledge Document 1531121.1 .
MOS: Using the Online Patching Readiness Report in Oracle E-Business Suite Release 12.2 (Doc ID 1531121.1)
``` $ sqlplus apps/apps @$AD_TOP/patch/115/sql/adzddtsfix.sql $ sqlplus apps/apps
exec dbms_utility.invalidate(332021,NULL,0); (...) exec sys.utl_recomp.recomp_parallel ```
UNEXPECTEDError occurred while executing txkADOPPreparePhaseSynchronize.pl
UNEXPECTEDError occurred while executing "perl /RYBAK/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0 /patch/115/bin/txkADOPPreparePhaseSynchronize.pl

-contextfile=/RYBAK/oracle/VIS/fs1/inst/apps/VIS_okon1/appl/ admin/VIS_okon1.xml -
patchcontextfile=/RYBAK/oracle/VIS/fs2/inst/apps/VIS_okon1/appl/admin/VIS_okon1.xml -pro mptmsg=hide -console=off -mode=migrate -sessionid=4 -
timestamp=20150104_093934 -outdir=/RYBAK/oracle/VIS/fs_n e/EBSapps/log/adop/4/prepare_20150104_093934/VIS_okon1"
/RYBAK/oracle/VIS/fs_ne/EBSapps/log/adop/4/adop_okon1_20150104_093934.log
$ less
/RYBAK/oracle/VIS/fs_ne/EBSapps/log/adop/4/prepare_20150104_093934/VIS_okon1/TXK_SYNC_migrate_Sun_Jan_4_11_06_46_2015/txkADOPPreparePhaseSynchronize_Sun_Ja
*******FATAL ERROR******* PROGRAM : (/RYBAK/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl) TIME : Sun Jan 4
11:10:25 2015 FUNCTION: main::migrateCloneComponentStage [ Level 1 ]
START: Creating WLS config archive. Script Executed in 25922 milliseconds, returning status 255 ERROR: Script failed, exit code 255
MOS: START: Creating WLS config archive. ERROR: Script failed, exit code 255 R12.2 Adpreclone Fails When Creating WLS Config Archive : Exit
Code 255 (Doc ID 1942015.1)
Restarting everything.
column HOST format a10 column DOMAIN format a10 ```
select NODE_NAME, NODE_MODE, NODE_ID, HOST, DOMAIN from apps.fnd_nodes order by NODE_NAME; NODE_NAME N NODE_ID
HOST DOMAIN
AUTHENTICATION O 33105 OKON1 O 33102 okon1 remik.org OKON2 O 33103 okon2 remik.org OKON3 O 33104 okon3 remik.org USERA1 O 32101
usera1 remik.org USERA2 O 32102 usera2 remik.org ```
Looks fine.
MOS: Adpreclone Errors On Patch Edition in 12.2.3 With Script failed, exit code 255 (Doc ID 1924636.1)
$ mkdir /RYBAK/temp
Add in profile: export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/RYBAK/temp"
Again. $ adop phase=prepare
perl /RYBAK/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -
patchcontextfile=/RYBAK/oracle/VIS/fs2/inst/apps/VIS_okon1/appl/admin/VIS_okon1.xml -mode=migrate -sessionid=4 -timestamp=20150104_132716 -
outdir=/RYBAK/oracle/VIS/fs_ne/EBSapps/log/adop/4/prepare_20150104_132716/VIS_okon1
START: Creating WLS config archive. Script Executed in 37830 milliseconds, returning status 255 ERROR: Script failed, exit code 255
Checking the Admin Server logs $ less /RYBAK/oracle/VIS/fs1/FMW_Home/user_projects/domains/EBS_domain_VIS/servers/AdminServer/logs/AdminServer.out
<Notice>
<WebLogicServer>
<BEA-000360>
oracle.as.management.opmn.optic.OpticBadConnectException: Could not get secure socket for host okon3.remik.org and port 6205 at
oracle.as.management.opmn.optic.OpmnSSL.getSecureSocket(OpmnSSL.java:52)
MOS: 12.2.3 Rapid Clone Fails With: Script failed, exit code 255 During WLS domain creation (Doc ID 1920683.1)
Remove the /tmp/.ovdlock.tmp file.

$ ls -l /tmp/.ovdlock.tmp -rw-r----- 1 applmgr oinstall 0 Dec 26 23:23 .ovdlock.tmp $ rm /tmp/.ovdlock.tmp
Trying autoconfig on all the MT nodes $ . /RYBAK/oracle/VIS/EBSapps.env run $ cd $INST_TOP/admin/scripts $ ./adautocfg.sh AutoConfig completed
successfully. - on all MT nodes
Starting MT on all the nodes
Aborting adop cycle $ adop phase=abort $ rm /tmp/.ovdlock.tmp
Trying to run preclone on all MT tiers $ cd $INST_TOP/admin/scripts $ perl adpreclone.pl appsTier -> All ok
Again $ adop phase=prepare Node okon2: COMPLETED - Prepare Status: COMPLETED Node okon1: COMPLETED - Prepare Status: COMPLETED Node okon3: COMPLETED -
Prepare Status: COMPLETED
adop exiting with status = 0 (Success)
OMG!!! Finally.
$ adop -status okon1 master PREPARE COMPLETED 04-JAN-15 17:38:30 +01:00 04-JAN-15 18:14:36 +01:00 0:36:06 okon2 slave PREPARE COMPLETED 04-JAN-15
16:41:49 +01:00 04-JAN-15 17:27:35 +01:00 0:45:46 okon3 slave PREPARE COMPLETED 04-JAN-15 17:40:47 +01:00 04-JAN-15 18:25:09 +01:00 0:44:22
$ adop phase=cutover adop exiting with status = 0 (Success)
``` $ adop -status Node Name Node Type Phase Status Started Finished Elapsed
okon1 master PREPARE COMPLETED 04-JAN-15 17:38:30 +01:00 04-JAN-15 18:14:36 +01:00 0:36:06 APPLY COMPLETED FINALIZE COMPLETED
04-JAN-15 18:37:15 +01:00 04-JAN-15 18:37:41 +01:00 0:00:26 CUTOVER COMPLETED 04-JAN-15 18:36:54 +01:00 04-JAN-15 19:22:39 +01:00
0:45:45 CLEANUP NOT STARTED ```
okon2 slave PREPARE COMPLETED 04-JAN-15 16:41:49 +01:00 04-JAN-15 17:27:35 +01:00 0:45:46 APPLY COMPLETED FINALIZE COMPLETED 04-
JAN-15 18:37:15 +01:00 04-JAN-15 18:37:41 +01:00 0:00:26 CUTOVER COMPLETED 04-JAN-15 17:37:00 +01:00 04-JAN-15 18:22:12 +01:00 0:45:12
CLEANUP NOT STARTED
CLEANUP NOT STARTED
$ adop phase=cleanup $ adop -status
Current Patching Session ID: 5
Node Name Node Type Phase Status Started Finished Elapsed -------------- --------------- --------------- --------------- ------------------------------ ---------------------------
--- ------------ okon1 master PREPARE COMPLETED 04-JAN-15 17:38:30 +01:00 04-JAN-15 18:14:36 +01:00 0:36:06 APPLY COMPLETED FINALIZE
COMPLETED 04-JAN-15 18:37:15 +01:00 04-JAN-15 18:37:41 +01:00 0:00:26 CUTOVER COMPLETED 04-JAN-15 18:36:54 +01:00 04-JAN-15
19:22:39 +01:00 0:45:45 CLEANUP COMPLETED 04-JAN-15 20:12:58 +01:00 04-JAN-15 20:13:45 +01:00 0:00:47
CLEANUP COMPLETED 04-JAN-15 20:12:58 +01:00 04-JAN-15 20:13:45 +01:00 0:00:47
CLEANUP COMPLETED 04-JAN-15 20:12:58 +01:00 04-JAN-15 20:13:45 +01:00 0:00:47
Time to take a backup Forms are not opening
I can login fine as sysadmin, but after going to System Administrator / Requests /View Just new window is opening with image Oracle E-businesssuite and
that is it.
And it happens on browser that is able top open forms. Usually case is with end user client browser, as per: Recommended Browsers for Oracle E-
Business Suite Release 12 Doc ID 389422.1
http://okon1.remik.org:8005
What logs should I check?
First Apache logs: $ cd $FMW_HOME/webtier/instances/EBS_web_${TWO_TASK}_OHS1/diagnostics/logs/OHS/EBS_web_${TWO_TASK}
> Nothing.
Lets see weblogic console
http://okon1.remik.org:7006/console http://okon1.remik.org:7007/console
> all servers running fine
Nice list to check on: http://tk-appsdba.blogspot.com/2011/02/oracle-apps-r12-forms-not-opening.html
Always check the DB, and alert log.
OK.
Check file system space.
OK.
Check Apache and Forms log. $LOG_HOME/10.1.3/Apache/Apache/logs and $LOG_HOME/10.1.2/forms/logs
I can find $LOG_HOME/ora/10.1.2/forms which is empty
If you are using R12 with IE8 then check Metalink doc 1069497.1,
for Cross Site Scripting(XSS) settings (make it to relevant tab of your internet options). Also check 389422.1 for recommended browsers and settings.
My browser was working already with R12
Check timestamp for files like $AU_TOP/resource/CUSTOM. ,
this will tell you if someone has modified the pll and then the changes should be reverted back.
$ cd $AU_TOP/resource/ -> nothing new
Try checking the form sessions from your server as below
$ ps -ef | grep frm
$ ps -ef | grep frm applmgr 9066 3065 0 16:25 pts/1 00:00:00 grep frm -> Nothing
This will show the frmsrv and frmweb processes. If it doesn't then the issue is in spawning form session.
You can check the timestamp for frmsrv/frmweb executable, if this has got changed. You can relink this by below. a. Source the Application environment b.
from location $ORACLE_HOME/forms/lib run :- make -f ins_forms.mk install
-- How to restart only forms? $ cd $ADMIN_SCRIPTS_HOME $ ./adformsrvctl.sh start Forms Connect Mode is set to Servlet. Please Change Forms Mode to
socket and then run this script.
Checking the statup logs $ cd $LOG_HOME/appl/admin/log/ $ less adstrtal.log Web Application Services oacore_server1 admanagedsrvctl.sh Started Web
Application Services forms_server1 admanagedsrvctl.sh Started Web Application Services oafm_server1 admanagedsrvctl.sh Started Web Application Services
forms-c4ws_server1 admanagedsrvctl.sh Started
$ cd $ADMIN_SCRIPTS_HOME $ admanagedsrvctl.sh status forms_server1 Validated the passed arguments for the option ebs-get-serverstatus forms_server1 is in
RUNNING mode.
$ admanagedsrvctl.sh stop forms_server1 $ admanagedsrvctl.sh start forms_server1 $ ps -ef | grep frm -> nothing
Relink $ cd $ORACLE_HOME/forms/lib $ pwd /RYBAK/oracle/VIS/fs2/EBSapps/10.1.2/forms/lib $ make -f ins_forms.mk install
Restart $ cd $ADMIN_SCRIPTS_HOME $ admanagedsrvctl.sh stop forms_server1 $ admanagedsrvctl.sh start forms_server1 $ ps -ef | grep frm -> nothing
All MT restart > nothing

In other case if from sessions are there and you are not able to open the form, then check the direct form login as below.
http://server:port/forms/frmservlet - for servlet mode http://server:port/OA_HTML/frmservlet - for socket mode
http://okon1.remik.org:8005/forms/frmservlet - URL validation failed. http://okon1.remik.org:8005/OA_HTML/frmservlet - URL validation failed.
http://okon1.remik.org:8005/forms/ Oracle Application Server 10g Forms Services The Forms Listener servlet is up and running.
If above works then you can go to adadmin and compile jsps, or can have clear cache bounce.
$ adadmin ?? which option
$ $FND_TOP/patch/115/bin/ojspCompile.pl --compile --flush -p 2
If step 6 doesn't work then check for dbc file present at $FND_SECURE.
a. The file should have correct entries for Application Server Id, as it is in FND_NODES table. b. All the passwords guest,applsyspub etc are ok, or change
the guest password using FNDCPASS c. You can correct the dbc file manually, or regenerate using $COMMON_TOP/admin/install/adgendbc.sh --
$ cd $FND_SECURE -> all seams fine
Bounce forms server and retry.
http://okon1.remik.org:8005/OA_HTML/jsp/fnd/aoljtest.jsp Error 404--Not Found Works after jsp recompilation!!!
http://okon1.remik.org:8005/OA_HTML/ WSRP Producer Test Page
MOS: Oracle E-Business Suite R12 – Login Page showing a blank page Occuring Intermittently 1491845.1
Restart forms $ cd $ADMIN_SCRIPTS_HOME $ admanagedsrvctl.sh stop forms_server1 $ admanagedsrvctl.sh start forms_server1 -> nothing
Shutdown MT
Relink $ cd $ORACLE_HOME/forms/lib $ make -f ins_forms.mk install
Startup MT > nothing
There has got to be some logs about it
Apache $ cd $FMW_HOME/webtier/instances/EBS_web_${TWO_TASK}_OHS1/diagnostics/logs/OHS/EBS_web_${TWO_TASK}
MOS: Troubleshooting Forms Applet issues in E-Business Suite (Doc ID 1300317.1)
This is bothering me $ ps -ef | grep frm -> Nothing
Forms run-time process (11i - frmweb60 / R12 - frmweb) started by the Forms Server/Servlet and 'assigned' to the Forms Applet. Forms Server/Servlet not
able to spawn a new Forms run-time child process side issue.
I am using forms servlet now.
Applied all the required patches to DB and Forms and Reports
As advised by: Oracle E-Business Suite Release 12.2: Consolidated List of Patches and Technology Bug Fixes (Doc ID 1594274.1) - described in
separate section.
The problem still remains.
MOS: Troubleshooting Forms Applet issues in E-Business Suite (Doc ID 1300317.1) How To Collect And Use Forms Trace (FRD) in Oracle
Applications Release 12 (Doc ID 373548.1)
$ cp $FORMS_WEB_CONFIG_FILE /var/tmp $ vi $FORMS_WEB_CONFIG_FILE serverURL=/forms/lservlet to serverURL=/forms/lservlet/session
Latest file(s) in directory (recursively) $ find . -type f -printf '%T@ %pn' | sort -n | tail -1 | cut -f2- -d" "
./fs2/FMW_Home/user_projects/domains/EBS_domain_VIS/servers/AdminServer/logs/AdminServer.log
./fs2/FMW_Home/user_projects/domains/EBS_domain_VIS/servers/oacore_server2/logs/oacore_server2.log
./fs2/FMW_Home/webtier/instances/EBS_web_VIS_OHS1/config/OPMN/opmn/states/.opmndat
./fs2/FMW_Home/webtier/instances/EBS_web_VIS_OHS1/diagnostics/logs/OHS/EBS_web_VIS/dms_metrics_29518_0.shm
./fs2/FMW_Home/webtier/instances/EBS_web_VIS_OHS1/diagnostics/logs/OHS/EBS_web_VIS/access_log
./fs2/FMW_Home/user_projects/domains/EBS_domain_VIS/servers/forms_server1/logs/forms_server1.log
./fs2/FMW_Home/user_projects/domains/EBS_domain_VIS/servers/forms-c4ws_server1/logs/forms-c4ws_server1.log
./fs_ne/inst/VIS_okon3/logs/appl/conc/log/VIS_0105.mgr ./fs_ne/inst/VIS_okon1/logs/appl/conc/log/FNDCPGSC2728674.txt
./fs_ne/inst/VIS_okon1/logs/appl/conc/log/FNDCPGSC2728673.txt
Nothing. Lets swich to socket mode
MOS: Using Forms Socket Mode with Oracle E-Business Suite Release 12 (Doc ID 384241.1)
Section 2: Enabling Forms Socket Mode
Stop MT
$ cat $CONTEXT_FILE | grep s_formsport <forms_port oa_var="s_formsport" oa_type="PORT" base="9000" step="1" range="-1" label="Forms
Port">9006</forms_port>
Run the following command to enable Forms Socket Mode: $ $FND_TOP/bin/txkrun.pl -script=ChangeFormsMode -contextfile=$CONTEXT_FILE -mode=socket -
port=9006 -runautoconfig=No
Autoconfig $ cd $INST_TOP/admin/scripts $ ./adautocfg.sh
Start MT
Check whether Forms Server is running: $ cd $ADMIN_SCRIPTS_HOME $ ./adformsrvctl.sh status You are running adformsrvctl.sh version 120.19.12020000.5
Checking status of FORMS Server (Socket Mode) ... Forms Server (Socket) is running as PID 9514
So it looks I am running Forms in Server mode $ ps -ef | grep frm applmgr 9514 1 0 00:46 pts/1 00:00:00 frmsrv host okon1 port 9006 pool 1 exe frmweb
log /RYBAK/oracle/VIS/fs2/inst/apps/VIS_okon1/logs/ora/10.1.2/forms/socket.log em 1 applmgr 9526 9514 0 00:46 pts/1 00:00:00 frmweb
webfile=5,0,okon1_9006,/RYBAK/oracle/VIS/fs2/inst/apps/VIS_okon1/logs/ora/10.1.2/forms/socket.log
But the problem remains the same.

Enable some apache debugging
$ cd $FMW_HOME/webtier/instances/EBS_web_${TWO_TASK}_OHS1/config/OHS/EBS_web_${TWO_TASK} $ vim httpd.conf ErrorLog "|${ORACLE_HOME}/ohs/bin/rotatelogs

${ORACLE_INSTANCE}/diagnostics/logs/${COMPONENT_TYPE}/${COMPONENT_NAME}/error_log 86400" LogLevel debug OraLogMode apache
restart Apache $ cd $ADMIN_SCRIPTS_HOME ./adapcctl.sh stop ./adapcctl.sh start
Checking logs $ cd $FMW_HOME/webtier/instances/EBS_web_${TWO_TASK}_OHS1/diagnostics/logs/OHS/EBS_web_${TWO_TASK}
Jan 06 10:44:33 2015 debug mod_proxy_balancer.c(1012): proxy: Entering byrequests for BALANCER (balancer://oacorecluster_oajava)
Looks like this apache mode is spreading connections all over the MT nodes.
Jan 06 11:19:05 2015 debug proxy_util.c(2011): proxy: HTTP: has acquired connection for (okon3.remik.org) Jan 06 11:19:05 2015 debug
proxy_util.c(2067): proxy: connecting http://okon3.remik.org:7207/OA_HTML/classes/oracle/apps/fnd/jar/fndctx.jar to okon3.remik.org:7207 Jan 06
11:19:05 2015 debug proxy_util.c(2193): proxy: connected /OA_HTML/classes/oracle/apps/fnd/jar/fndctx.jar to okon3.remik.org:7207
http://okon1.remik.org:8005/OA_JAVA/oracle/apps/fnd/jar/fndforms.jar
Trying to troubleschoot the socket mode again
MOS: How To Check If The Forms Server Is Configured In Servlet Or Socket Mode On 11i Or R12 (Doc ID 417216.1)
okon1 $ cat $CONTEXT_FILE | grep -i s_frmConnectMode <forms_connect oa_var="s_frmConnectMode">socket</forms_connect>
okon2, okon3 $ cat $CONTEXT_FILE | grep -i s_frmConnectMode <forms_connect oa_var="s_frmConnectMode">servlet</forms_connect>
So I got this confused. Changin it everywhere for consistency
Shutdown all MT
On okon2, okon3
Run the following command to enable Forms Socket Mode: $ $FND_TOP/bin/txkrun.pl -script=ChangeFormsMode -contextfile=$CONTEXT_FILE -mode=socket -
port=9006 -runautoconfig=No
On okon1, okon2, okon3 $ cat $CONTEXT_FILE | grep -i s_frmConnectMode <forms_connect oa_var="s_frmConnectMode">socket</forms_connect>
Start all MT
MOS: R12: Forms Runtime Diagnostics (FRD), Tracing And Logging For Forms In Oracle Applications (Doc ID 438652.1)
On all MT nodes $ grep -e connectMode -e serverURL ./oracle/VIS/fs2/inst/apps/VIS_okon1/ora/10.1.2/forms/server/appsweb.cfg serverURL=

connectMode=socket
Enabling tracing
``` $ vi /RYBAK/oracle/VIS/fs2/inst/apps/VIS_okon1/ora/10.1.2/forms/server/appsweb.cfg $ vi
/RYBAK/oracle/VIS/fs2/inst/apps/VIS_okon2/ora/10.1.2/forms/server/appsweb.cfg $ vi
/RYBAK/oracle/VIS/fs2/inst/apps/VIS_okon3/ora/10.1.2/forms/server/appsweb.cfg
Sub argument for otherparams

from record= to record=collect ```
Page with menu visible after loging: http://okon1.remik.org:8005/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE

Going back to servlet mode
MOS: Using Forms Socket Mode with Oracle E-Business Suite Release 12 (Doc ID 384241.1)
Section 3: Enabling Forms Servlet Mode
$ $FND_TOP/bin/txkrun.pl -script=ChangeFormsMode -contextfile=$CONTEXT_FILE -mode=servlet -runautoconfig=no
Start MT
Page with menu visible after loging: http://okon1.remik.org:8005/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE
Apache error log: Jan 06 20:26:03 2015 debug proxy_util.c(2067): proxy: connecting
http://okon3.remik.org:7207/OA_HTML/classes/oracle/apps/fnd/jar/fndctx.jar to okon3.remik.org:7207 Jan 06 20:26:03 2015 debug proxy_util.c(2193):
proxy: connected /OA_HTML/classes/oracle/apps/fnd/jar/fndctx.jar to okon3.remik.org:7207 Jan 06 20:26:03 2015 debug mod_proxy_http.c(1851): proxy:
header only Jan 06 20:26:03 2015 debug mod_headers.c(756): headers: ap_headers_output_filter() Jan 06 20:26:03 2015 debug proxy_util.c(2029):
proxy: HTTP: has released connection for (okon3.remik.org) Jan 06 20:26:03 2015 debug mod_proxy_balancer.c(622): proxy_balancer_post_request for
(balancer://oacorecluster_oajava)
The same problem. No error, just nothing happens.
MOS: How To Diagnose: Workflow Monitor Status Diagram Failure to Display in Release 11i and 12 (Doc ID 956416.1)> no luck
./oracle/VIS/fs2/EBSapps/10.1.2/j2ee/home/config/oc4j.properties
Checking weblogic logs
$ tail -f /RYBAK/oracle/VIS/fs2/FMW_Home/user_projects/domains/EBS_domain_VIS/servers/oacore_server1/logs/oacore_server1.log
Nothing, which means that it does not come to oacore_server1 yet. The problem is somehere else.
But I can see logs appearing in $ less /RYBAK/oracle/VIS/fs2/FMW_Home/user_projects/domains/EBS_domain_VIS/servers/forms_server1/logs/forms_server1.log
Which means that it is comming here!!!
```
<[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <005307P9mw2Fg4WFLzATOA0002NV00000z>
<1420576961306>
```
MOS: FRM-92050 or FRM-92101 When Re-Opening Forms using Internet Explorer 8 On 12.2.3 (Doc ID 1932415.1)
Not exactly the same issue, but what the hell.
ERROR WHEN OPENING FORMS IN IE8 ON MULTI-NODE EBS 12.2.3 (Patch) p19259764_R12.FND.C_R12_GENERIC.zip 316.0 KB (323597 bytes)
SHA-1 F8487F0C61D7F04BE23D9C4D2591C15D72B2F48E MD5 9B504769BA39378FA47DFA6B95899E04
I skipped this patch before. "Skip this step if you are a new installation customer."
I will install it anyway.
I do not want to go through all the adop cycle (time) I go with the not supported, but available option of applying patch to running edition, as it was done in
11i.
$ cd /RYBAK/oracle/VIS/fs_ne/EBSapps/patch $ adop phase=apply apply_mode=downtime patches=19259764 adop exiting with status = 0 (Success)
The same issue on the beginning, now after login the same login page appears.
After login the same login page appears.
This is weird. Lets check if this is 12.2.4
Does it have 12.2.3 patch installed?
```
select BUG_ID, BUG_NUMBER, LAST_UPDATE_DATE from APPLSYS.AD_BUGS where BUG_NUMBER = '17020683'; no rows selected ```
That is kind of disappointment.
Looks like someone is having similar issue: Forms applet not opening in R12.2 https://community.oracle.com/thread/3632312?start=0&tstart=0
Consolidated List of Patches and Technology Bug Fixes
MOS: Oracle E-Business Suite Release 12.2: Consolidated List of Patches and Technology Bug Fixes (Doc ID 1594274.1)
Section 2: Database Patches and Bug Numbers
I have used Rapid install 12.2.0.48, applying those named "Not included in any startCD"
$ export PATH=$ORACLE_HOME/OPatch:$PATH $ opatch lsinventory $ opatch apply -silent -ocmrf /STAGING/oracle11.2.0.4_x86-64/ocm.rsp
+ 16989137
DRG-10761 ON SETTING PROCEDURE ATTRIBUTE OF PROCEDURE_FILTER PREFERENCE (Patch) p16989137_112030_Generic.zip 11.6 KB MD5
1E1E104D439529DB17B539B76D0308D1 $ md5sum p16989137_112030_Generic.zip 1e1e104d439529db17b539b76d0308d1 p16989137_112030_Generic.zip
``` $ cd /STAGING/DB_11.2_patches/16989137 $ opatch apply -ocmrf /STAGING/oracle11.2.0.4_x86-64/ocm.rsp For a RAC environment, perform these
steps on only one node.
@?/sqlpatch/16989137/postinstall.sql ```
+ 17875948
MERGE REQUEST ON TOP OF 11.2.0.3.0 FOR BUGS 16587934 17654622 (Patch) p17875948_112030_Linux-x86-64.zip 570.2 KB MD5
F7FD987FC9BA4BADACD0E2BB3EC592A9 $ md5sum p17875948_112030_Linux-x86-64.zip f7fd987fc9ba4badacd0e2bb3ec592a9 p17875948_112030_Linux-x86-
64.zip
$ cd /STAGING/DB_11.2_patches $ opatch apply
+ 17912217
TRACKING BUG FOR LRG 9760292 (Patch) p17912217_112030_Linux-x86-64.zip 130.1 KB MD5 0C5329CBC9F13FD39B8A4523140EF0E7 $ md5sum
p17912217_112030_Linux-x86-64.zip 0c5329cbc9f13fd39b8a4523140ef0e7 p17912217_112030_Linux-x86-64.zip
$ opatch apply
+ 18419770
KEY-PRESERVED TABLE IN JOIN VIEW NOT RECOGNIZED WHEN JOINED TABLES HAVE UNQ INDX (Patch) p18419770_112030_Linux-x86-64.zip
48.7 KB MD5 C2CDA21D8CB8ECA41634946574B3D270
$ md5sum p18419770_112030_Linux-x86-64.zip c2cda21d8cb8eca41634946574b3d270 p18419770_112030_Linux-x86-64.zip
$ opatch apply
+ 18614015
UTL_RECOMP DOES NOT COMPILE INVALID OBJECTS MV WITH STATUS 3 (Patch) p18614015_112030_Generic.zip 16.1 KB MD5
EE491C41F86BA4265DB9A78A3CC0DC4B $ md5sum p18614015_112030_Generic.zip ee491c41f86ba4265db9a78a3cc0dc4b p18614015_112030_Generic.zip
$ opatch apply
+ 18685209
ORA-600 [KKOGBRO: NO KKOAPTYP] (Patch) p18685209_112030_Linux-x86-64.zip 42.3 KB MD5 2071A4D8EAC8AF8256B9FAE2BE5C9EEB $ md5sum

p18685209_112030_Linux-x86-64.zip 2071a4d8eac8af8256b9fae2be5c9eeb p18685209_112030_Linux-x86-64.zip
$ opatch apply
+ 19078951
MERGE REQUEST ON TOP OF 11.2.0.3.0 FOR BUGS 19063027 14751895 (Patch) p19078951_112030_Linux-x86-64.zip 1.7 MB MD5
FCEEF7DF6E4EA76FE0068181A2397553 $ md5sum p19078951_112030_Linux-x86-64.zip fceef7df6e4ea76fe0068181a2397553 p19078951_112030_Linux-x86-64.zip
$ opatch apply
19393542
TIMESTAMP MISMATCHES ON RULE SET & EVALUATION CONTEXT (Patch) p19393542_112030_Linux-x86-64.zip 129.1 KB MD5
BFC7972C63F215355EBDA2297751F9F1 $ md5sum p19393542_112030_Linux-x86-64.zip bfc7972c63f215355ebda2297751f9f1 p19393542_112030_Linux-x86-64.zip
$ opatch apply
Back to ../
There were relinks on remote nodes. Remember to check the binary size and timestamp on the nodes 'usera2' . $ ls -l $ORACLE_HOME/bin/oracle -rwsr-s--
x 1 oracle oinstall 232837794 Jan 5 22:32 /u01/oracle/VIS/11.2.0/bin/oracle -rwsr-s--x 1 oracle oinstall 232837794 Jan 5 22:32
/u01/oracle/VIS/11.2.0/bin/oracle
Post install sql run (only in one patch)
For a RAC environment, perform these steps on only one node. ```
@?/sqlpatch/16989137/postinstall.sql ```
Technology Codelevel Checker needs to be installed on this instance. It is available as Patch 17537119. Source DB env $ ./checkDBpatch.sh Enter
ORACLE_HOME value : /u01/oracle/VIS/11.2.0 Enter ORACLE_SID value : VIS1 Enter ORACLE_SID value : VIS2 All the required one-offs are present in Oracle
Database Home
Section 3: Application Tier Technology Patches and Bug Numbers
We used startCD RapidWizVersion Version 12.2.0.48
Section 3.1: Forms and Reports 10.1.2.3 Patches and Bug Numbers
Start an online patching cycle. $ adop phase=prepare
Apply the patches to patch edition.
Here we have patches for versions 10.1.2.3.0 or 10.1.0.5.0. Which is the one I need?
We have 10.1.2.3
Decide on patch edition $ export ORACLE_HOME=/RYBAK/oracle/VIS/fs1/EBSapps/10.1.2 $ export ORACLE_HOME=/RYBAK/oracle/VIS/fs2/EBSapps/10.1.2
$ . /RYBAK/oracle/VIS/EBSapps.env patch $ env | grep ^ORACLE_HOME ORACLE_HOME=/RYBAK/oracle/VIS/fs2/EBSapps/10.1.2
$ export PATH=$PATH:$ORACLE_HOME/OPatch $ opatch lsinventory 43) Patch-es installed
18186693 MERGE REQUEST ON TOP OF 10.1.2.3.2PSU FOR BUGS 17303472 7711752 16806432 (Patch) p18186693_101232_LINUX.zip 4.7
MB MD5 124414E306680C379F1E826C77101756 $ cd /STAGING/ebs12patches/18186693 $ opatch apply
18620223 APPS6: RECORD LOCKING SUSPENDING FIRING OF FORMULA-CALCULATION (Patch) p18620223_101232_LINUX.zip 13.4 KB
MD5 7C937F34A4DE519F1650985E3EF6A960 $ md5sum p18620223_101232_LINUX.zip 7c937f34a4de519f1650985e3ef6a960 p18620223_101232_LINUX.zip
$ opatch apply
19434967 MERGE REQUEST ON TOP OF 10.1.2.3.2PSU FOR BUGS 19059675 6161079 10226288 (Patch) p19434967_101232_LINUX.zip 4.8
MB MD5 1265E88888764BF284592833570414FF $ md5sum p19434967_101232_LINUX.zip 1265e88888764bf284592833570414ff p19434967_101232_LINUX.zip
$ opatch apply
cd ../ ?? SPECIAL INSTRUCTION FOR formsapp.ear ??
$ opatch lsinventory | grep -e 18186693 -e 18620223 -e 19434967 1) Patch 19434967 applied on Mon Jan 05 23:22:02 CET 2015
2) Patch 18620223 applied on Mon Jan 05 23:20:29 CET 2015 Bug fixes: 18620223
?? Patch 18186693 not reported ?
$ opatch lsinventory
Section 3.2: Patches and bug numbers for Oracle Fusion Middleware (FMW) 11.1.1.6
$ export ORACLE_HOME=$IAS_ORACLE_HOME $ export PATH=$ORACLE_HOME/OPatch:$PATH $ which opatch /RYBAK/oracle/VIS/fs2/FMW_Home/webtier/OPatch/opatch
$ opatch lsinventory Oracle Home : /RYBAK/oracle/VIS/fs2/FMW_Home/webtier Oracle WebTier and Utilities CD 11.1.1.6.0
We already have those from rapid install 46 and 47: 10638758, 13643211, 14616819, 16199894, 17325559
Installing those Not included in any startCD 13055259 WIP
17555224
17639414
19945419
../
Something is too slow
Sequential test $ time dd if=/dev/zero of=/RYBAK/testfile bs=16k count=56384 $ time dd if=./testfile of=/dev/null bs=16k
Random test https://www.binarylane.com.au/support/articles/1000055889-how-to-benchmark-disk-i-o
$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --
rwmixread=75
MT1
Sequential test on NFS shared storage $ time dd if=/dev/zero of=/RYBAK/testfile bs=16k count=56384 923795456 bytes (924 MB) copied, 28.7702 seconds,
32.1 MB/s real 0m28.845s
$ time dd if=./testfile of=/dev/null bs=16k 923795456 bytes (924 MB) copied, 23.838 seconds, 38.8 MB/s real 0m23.859s
Random test on NFS shared storage $ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 -
-size=1G --readwrite=randrw --rwmixread=75 read : io=784996KB, bw=737876B/s, iops=180, runt=1089391msec write: io=263580KB, bw=247758B/s, iops=60,
runt=1089391msec
MT2
Sequential test on NFS shared storage $ time dd if=/dev/zero of=./testfile bs=16k count=56384 923795456 bytes (924 MB) copied, 27.6094 seconds, 33.5
MB/s real 0m27.868s
Random test on NFS shared storage read : io=784996KB, bw=754412B/s, iops=184, runt=1065512msec write: io=263580KB, bw=253311B/s, iops=61,
runt=1065512msec
MT3
Sequential test on NFS shared storage $ time dd if=/dev/zero of=./testfile bs=16k count=56384 923795456 bytes (924 MB) copied, 90.1964 seconds, 10.2
MB/s real 1m30.468s
Random test on NFS shared storage read : io=784996KB, bw=758551B/s, iops=185, runt=1059698msec write: io=263580KB, bw=254700B/s, iops=62,
runt=1059698msec
--- This 10.2 MB/s is suspicious and suggests network card that is using 100Mb and not 1000Mb. Turns out this was true, that was actually an
ESX VM machine that was converted from VirtualBox and network card was in "Flexible" mode which turned out to be slow. Changed to
VMXNET 3 and now it looks better.
$ time dd if=/dev/zero of=./testfile bs=16k count=56384 923795456 bytes (924 MB) copied, 25.5814 seconds, 36.1 MB/s real 0m25.769s
$ time dd if=./testfile of=/dev/null bs=16k 923795456 bytes (924 MB) copied, 0.149666 seconds, 6.2 GB/s - this is too good to be true, some cache real
0m0.159s
$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --
rwmixread=75 test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 read : io=784996KB, bw=1088.2KB/s, iops=272, runt=721418msec
write: io=263580KB, bw=374132B/s, iops=91, runt=721418msec
on DB
Sequential test on Local FS $ time dd if=/dev/zero of=./testfile bs=16k count=56384 923795456 bytes (924 MB) copied, 9.99278 s, 92.4 MB/s (probably some
cache) real 0m16.974s
$ time dd if=./testfile of=/dev/null bs=16k 923795456 bytes (924 MB) copied, 34.0162 s, 27.2 MB/s real 0m34.068s
Random test on Local FS $ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=1G --
readwrite=randrw --rwmixread=75 read : io=436944KB, bw=394880B/s, iops=96, runt=1133079msec write: io=146428KB, bw=132331B/s, iops=32, runt=1133079msec
Test on ASM, probably with ORION but now I want something trivial ```
create tablespace USERS datafile '+DATA' size 2G autoextend on maxsize 20G; Fri Jan 09 15:24:05 2015 create tablespace USERS datafile
'+DATA' size 2G autoextend on maxsize 20G Fri Jan 09 15:24:57 2015 Completed: create tablespace USERS datafile '+DATA' size 2G
autoextend on maxsize 20G ```
set timing on create table perf tablespace users as select from ( select from dba_objects union all select from dba_objects union all select from
dba_objects union all select from dba_objects union all select from dba_objects union all select from dba_objects ); Elapsed: 00:00:39.78
create table perf2 tablespace users as select from ( select from perf union all select from perf union all select from perf union all select from perf union
all select from perf union all select from perf union all select from perf union all select from perf ); Elapsed: 00:02:00.52
create table perf3 tablespace users as select from ( select from perf2 union all select from perf2 union all select from perf2 union all select from perf2
union all select from perf2 union all select from perf2 union all select from perf2 union all select from perf2 );
Elapsed: 00:22:37.64
This is weird, I see something like this: sdc 97% 15.4 15.8|RRRRRRRRRRRRRRRRRRRRRRRRRWWWWWWWWWWWWWWWWWWWWWWW>|│
│sdc1 97% 15.4 15.8|RRRRRRRRRRRRRRRRRRRRRRRRRWWWWWWWWWWWWWWWWWWWWWWW>|│ │sdd 16% 0.5 0.0|RRRRRRRRW > |│
│sdd1 16% 0.5 0.0|RRRRRRRRW >
Disk /dev/sdc: 256.0 GB, 255986761728 bytes Disk /dev/sdd: 5268 MB, 5268045824 bytes
```
/usr/sbin/oracleasm listdisks
DATA1 DATA2 ```
```
/usr/sbin/oracleasm querydisk -p DATA1

Disk "DATA1" is a valid ASM disk /dev/sdd1: LABEL="DATA1" TYPE="oracleasm" ```
```

Disk "DATA2" is a valid ASM disk /dev/sdc1: LABEL="DATA2" TYPE="oracleasm" ```
> so it is OK.
After changing all the patchcords to Cat 6:
```
create table perf2 tablespace users (...) Elapsed: 00:01:41.96 ```
```
create table perf3 tablespace users (...) Elapsed: 00:13:53.54 - 148430208 rows ```
ORION test (just reads, not to destroy data on disks)
``` $ mkdir orion_test $ cd orion_test $ vi orion.lun /dev/sdd1 /dev/sdc1 $ which orion /u01/oracle/VIS/11.2.0/bin/orion
/u01/oracle/VIS/11.2.0/bin/orion -run simple

Maximum Small IOPS=91 @ Small=1 and Large=0 Small Read Latency: avg=10870 us, min=129 us, max=136578 us, std dev=6348 us @ Small=1 and
Large=0 ```
Minimum Small Latency=10870 usecs @ Small=1 and Large=0 Small Read Latency: avg=10870 us, min=129 us, max=136578 us, std dev=6348 us @
Small=1 and Large=0 Small Read / Write Latency Histogram @ Small=1 and Large=0 Latency: # of IOs (read) # of IOs (write) 0 - 1 us: 0 0 2 - 4 us: 0 0 4 -
8 us: 0 0 8 - 16 us: 0 0
- 32 us: 0 0
32 - 64 us: 0 0 64 - 128 us: 0 0
8 - 256 us: 171 0

256 - 512 us: 270 0 512 - 1024 us: 2 0
24 - 2048 us: 0 0
2048 - 4096 us: 21 0 4096 - 8192 us: 1244 0 8192 - 16384 us: 3403 0
384 - 32768 us: 338 0

32768 - 65536 us: 65 0 65536 - 131072 us: 2 0
1072 - 262144 us: 1 0

Checking how is it on DB on local disk, not RAC with openfiler.
```
/usr/sbin/oracleasm listdisks
DATA1
Disk "DATA1" is a valid ASM disk /dev/sdc1: LABEL="DATA1" TYPE="oracleasm" ```
``` $ mkdir orion_test $ cd orion_test $ vi orion.lun $ pwd /home/oracle/orion_test $ which orion
cd /home/oracle/orion_test
/u01/app/oracle/product/11.2.0/dbhome_1/bin/orion -run simple

Maximum Large MBPS=224.38 @ Small=0 and Large=2 ```
Maximum Small IOPS=524 @ Small=5 and Large=0 Small Read Latency: avg=9530 us, min=16 us, max=69419 us, std dev=17829 us @ Small=5 and
Large=0
Minimum Small Latency=1956 usecs @ Small=1 and Large=0 Small Read Latency: avg=1956 us, min=14 us, max=44873 us, std dev=3845 us @
Small=1 and Large=0 Small Read / Write Latency Histogram @ Small=5 and Large=0 Latency: # of IOs (read) # of IOs (write) 0 - 1 us: 0 0 2 - 4 us: 0 0 4 -
8 us: 0 0 8 - 16 us: 1 0
- 32 us: 13523 0
32 - 64 us: 5339 0 64 - 128 us: 4565 0
8 - 256 us: 95 0
256 - 512 us: 24 0 512 - 1024 us: 9 0
24 - 2048 us: 24 0
2048 - 4096 us: 856 0 4096 - 8192 us: 2535 0 8192 - 16384 us: 3656 0
Much better here. Even thou it is a very simple one node DB.
12.2.3 patch is taking way too much time, almost 24h
60 2220972027 SYSTEM 14840 SELECT RIDX.OWNER OWNER , RIDX.INDEX_NAME REVISED_INDEX , RIDX.TABLE_OWNER TABLE_OWNER ,
RIDX.TABLE_NAME TABLE_NAME , OIDX.INDEX_NAME ORIGINAL_INDEX , CON.CONSTRAINT_NAME CONSTRAINT_NAME FROM DBA_INDE
```
select sql_text from v$sqltext where hash_value=2220972027 order by piece asc; ```
```
EXEC DBMS_STATS.GATHER_DICTIONARY_STATS; exec DBMS_STATS.GATHER_FIXED_OBJECTS_STATS; ```
already cutover
ADZDWRKR.sql
/RYBAK/oracle/VIS/fs_ne/EBSapps/log/adop/11/cutover_20150108_185610/VIS_okon1/log ```
tail -f cutover.log
```
Starting phase 0 (A0): con
There are now 2 jobs remaining (current phase=A0): 0 running, 1 ready to run and 1 waiting.
Assigned: file ADZDWRKR.sql on worker 1 for product ad username APPLSYS.
MOS: ADZDWRKR.sql Is Running On 8 Workers For Long Time (Doc ID 1946877.1)
Start/Stop MT
http://okon1.remik.org:7006/console
Start
. /RYBAK/oracle/VIS/EBSapps.env run
okon1 $ . /RYBAK/oracle/VIS/fs1/EBSapps/appl/APPSVIS_okon1.env $ cd $ADMIN_SCRIPTS_HOME $ ./adstrtal.sh apps/apps
okon2 $ . /RYBAK/oracle/VIS/fs1/EBSapps/appl/APPSVIS_okon2.env $ cd $ADMIN_SCRIPTS_HOME $ ./adstrtal.sh apps/apps
okon3 $ . /RYBAK/oracle/VIS/fs1/EBSapps/appl/APPSVIS_okon3.env $ env | grep FILE_EDITION FILE_EDITION=run $ cd $ADMIN_SCRIPTS_HOME $ ./adstrtal.sh

apps/apps
Stop
. /RYBAK/oracle/VIS/EBSapps.env run
okon3 $ . /RYBAK/oracle/VIS/fs1/EBSapps/appl/APPSVIS_okon3.env $ env | grep FILE_EDITION FILE_EDITION=run $ cd $ADMIN_SCRIPTS_HOME $ ./adstpall.sh

apps/apps
okon2 $ . /RYBAK/oracle/VIS/fs1/EBSapps/appl/APPSVIS_okon2.env $ cd $ADMIN_SCRIPTS_HOME $ ./adstpall.sh apps/apps
okon1 $ . /RYBAK/oracle/VIS/fs1/EBSapps/appl/APPSVIS_okon1.env $ cd $ADMIN_SCRIPTS_HOME $ ./adstpall.sh apps/apps
Nice links:
Sharing The Application Tier File System in Oracle E-Business Suite Release 12 (Doc ID 384248.1) Section 4: Adding a Node to a Shared
Application Tier File System Troubleshooting RapidClone issues with Oracle Applications R12 (Doc ID 603104.1) Oracle® E-Business Suite
Installation Guide: Using Rapid Install Best Practices for Minimizing Oracle EBS R12.2.n Upgrade Downtime Presentation (PDF 222 KB) Cloning
Oracle E-Business Suite Release 12.2 with Rapid Clone (Doc ID 1383621.1) 5.4 Deleting an Application Tier Node Configuring Applications Node
Services in Oracle E-Business Suite Release 12 (Doc ID 406558.1) Using the Online Patching Readiness Report in Oracle E-Business Suite Release
12.2 (Doc ID 1531121.1) Troubleshooting Forms Applet issues in E-Business Suite (Doc ID 1300317.1) Using Forms Socket Mode with Oracle E-
Business Suite Release 12 (Doc ID 384241.1) R12: Forms Runtime Diagnostics (FRD), Tracing And Logging For Forms In Oracle Applications (Doc ID
438652.1) NOTE:389422.1 - Recommended Browsers for Oracle E-Business Suite Release 12 NOTE:393931.1 - Deploying JRE (Native Plug-in) for
Windows Clients in Oracle E-Business Suite Release 12 NOTE:421245.1 - E-Business Suite Diagnostics References for R12 NOTE:364439.1 - Tips and
Queries for Troubleshooting Advanced Topologies Best Practices for Minimizing Oracle EBS R12.2.n Upgrade Downtime Whitepaper (PDF 675 KB)
Latest file(s) in directory (recursively) $ find . -type f -printf '%T@ %pn' | sort -n | tail -1 | cut -f2- -d" "
Verify application tier failover
Change the profile FND: Diagnostics to Yes on site and user sysadmin level to enable database host info.
Checking how it works now
Login as sysadmin System Administrator -> Requests -> View / Help -> About Oracle Application
At this test: Machine : usera1.remik.org
Login to usera1.remik.org ```
shutdown abort ```
Receivin errors from forms: FRM-40735, Closing forms.
Trying to login again. Blank new browser window appears, with only a name 'Window' and nothing happens. This is the moment where forms were
supposed to open.
Loging to master app tier.
$ tnsping VIS Attempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.56)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.54)

(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.55)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VIS) (INSTANCE_NAME=VIS1))) OK (10 msec)
$ sqlplus apps ORA-12521: TNS:listener does not currently know of instance requested in connect descriptor
There is SERVICE_NAME=VIS and? INSTANCE_NAME=VIS1 - weird
Fixing manually
He tries to connect using scan listener hosts, but not scan listener name, weird too.
$ nslookup usera-scan Name: usera-scan.remik.org Address: 192.168.1.55 Name: usera-scan.remik.org Address: 192.168.1.56 Name: usera-scan.remik.org
Address: 192.168.1.54
On usera1: eth0:2 inet addr:192.168.1.55
On usera2: eth0:1 inet addr:192.168.1.54
Ok, that could work.
But this instance name?
Changing tnsnames form: VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.56)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)

(HOST=192.168.1.54)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.55)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VIS)
(INSTANCE_NAME=VIS1) ) )
to: VIS= (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.56)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.54)

(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.55)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME=VIS) ) )
Now I can login with sqlplus $ sqlplus apps/apps
Trying to login from application and open forms that works!!
So easy? Once again from start. It is working.
Reverting changes done to listeners during: DBCA fails Can not Connect to Database/Instance in Multiple Listener Environment ```
alter system set local_listener='VIS1_LOCAL' sid='VIS1'; alter system set local_listener='VIS2_LOCAL' sid='VIS2'; ```
OK, it worked again.
Checking how will the tnsnames.ora on app tier look like after autoconfig $ cd $ADMIN_SCRIPTS_HOME $ ./adautocfg.sh
Now the tnsname.ora that we edidted before is overwritten.
Reading some docs
Actually better idea ist to follow: MOS: Using Oracle 11g Release 2 Real Application Clusters and Automatic storage management with Oracle E-
Business Suite Release 12.2 (Doc ID 1453213.1) Concurrent Processing - How To Ensure Load Balancing Of Concurrent Manager Processes In PCP-
RAC Configuration (Doc ID 762024.1) Configuring and Managing E-Business Application Tier for RAC (Doc ID 1311528.1)
Verify listener status
Local listener, Source DB env $ cd /u01/oracle/VIS/11.2.0/appsutil/scripts/VIS1_usera1 $ ./addlnctl.sh status VIS Listener Parameter File
/u01/oracle/VIS/11.2.0/network/admin/VIS1_usera1/listener.ora Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.52)
(PORT=1526))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.50)(PORT=1526))) Services Summary... Service "VIS" has 1 instance(s). Instance "VIS1",
status READY, has 1 handler(s) for this service... Service "VIS1" has 1 instance(s). Instance "VIS1", status UNKNOWN, has 1 handler(s) for this
service... Service "ebs_patch" has 1 instance(s). Instance "VIS1", status READY, has 1 handler(s) for this service... The command completed successfully
Grid listener, source grid env $ lsnrctl status Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.50)(PORT=1521))) (DESCRIPTION=(ADDRESS=
(PROTOCOL=tcp)(HOST=192.168.1.52)(PORT=1521))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM1", status READY, has 1 handler(s) for
this service... The command completed successfully
Scan listener, source grid env $ srvctl status SCAN_listener SCAN Listener LISTENER_SCAN1 is enabled SCAN listener LISTENER_SCAN1 is running on node
usera2 SCAN Listener LISTENER_SCAN2 is enabled SCAN listener LISTENER_SCAN2 is running on node usera1 SCAN Listener LISTENER_SCAN3 is enabled SCAN
listener LISTENER_SCAN3 is running on node usera1
$ srvctl config scan SCAN name: usera-scan, Network: 1/192.168.1.0/255.255.255.0/eth0 SCAN VIP name: scan1, IP: /usera-scan/192.168.1.54 SCAN VIP name:
scan2, IP: /usera-scan/192.168.1.55 SCAN VIP name: scan3, IP: /usera-scan/192.168.1.56
$ lsnrctl status LISTENER_SCAN2 Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora Listening Endpoints Summary... (DESCRIPTION=
(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN2))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.55)(PORT=1521))) Services Summary... Service "VIS" has
2 instance(s). Instance "VIS1", status READY, has 1 handler(s) for this service... Instance "VIS2", status READY, has 1 handler(s) for this service...
Service "ebs_patch" has 2 instance(s). Instance "VIS1", status READY, has 1 handler(s) for this service... Instance "VIS2", status READY, has 1
handler(s) for this service... The command completed successfully
Performing permanent changes
Following: MOS: Using Oracle 11g Release 2 Real Application Clusters and Automatic storage management with Oracle E-Business Suite
Release 12.2 (Doc ID 1453213.1)
Following: 4.7.2 Set Up Load Balancing
$ env | grep TWO_TASK TWO_TASK=VIS
Modify the $CONTEXT_FILE ``` $ cp $CONTEXT_FILE ${CONTEXT_FILE}lbdate -I $ vim $CONTEXT_FILE $ diff $CONTEXT_FILE
${CONTEXT_FILE}lbdate -I 399c399
< VIS_BALANCE
<TWO_TASK oa_var="s_tools_twotask" osd="unix">VIS</TWO_TASK>
414c414
< VIS_BALANCE
<TWO_TASK oa_var="s_weboh_twotask" osd="unix">VIS</TWO_TASK>
```
Autoconfig $ cd $ADMIN_SCRIPTS_HOME $ ./adautocfg.sh
$ env | grep TWO_TASK TWO_TASK=VIS_BALANCE
``` $ sqlplus apps/apps
show parameter instance_name instance_name string VIS1 ```
```
shutdown abort on VIS1 ```
```
startup on VIS1 shutdown abort on VIS2 ```
Conclusion: > all work nicely
Checking the application failover. > works fine, after DB1 fails, next login goes to DB2
Note! I have not changed the s_cp_twotask as advised by (Doc ID 762024.1), which is for 12.1.
After setting TWO_TASK to the load balancing alias, the ADOP cleanup and cutover phases will fail. In order to workaround this problem:
ADOP Cleanup: Copy the adalldefaults.txt file from $APPL_TOP/admin/
<sid>
/ to the $APPL_TOP/admin/
directory on RUN file system. $ diff $APPL_TOP/admin/VIS/adalldefaults.txt $APPL_TOP/admin/VIS_BALANCE/adalldefaults.txt -> files exist and are the same
WIP: Check if adop cycle will update the context file on the patch edition
Check all the relevant places to confirm that we use balance connection
Checks as per: Using Oracle 11g Release 2 Real Application Clusters and Automatic storage management with Oracle E-Business Suite Release 12.2
(Doc ID 1453213.1) 5.2.3 Application Tier Install
$ grep s_apps_jdbc_connect_descriptor $CONTEXT_FILE <jdbc_url oa_var="s_apps_jdbc_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=

(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=usera-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=VIS)))</jdbc_url>
$ grep s_apps_jdbc_patch_connect_descriptor $CONTEXT_FILE <patch_jdbc_url oa_var="s_apps_jdbc_patch_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTION=

(ADDRESS=(PROTOCOL=tcp)(HOST=usera-scan)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=ebs_patch)(INSTANCE_NAME=VIS1)))</patch_jdbc_url>
$ cat $FND_SECURE/VIS.dbc | grep APPS_JDBC_URL APPS_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=

(PROTOCOL=tcp)(HOST=usera-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=VIS)))
Requests with status: Inactive No Manager
If I shutdown the DB1, then the Standard Manager stops to process the requests.
Following MOS: Configuring and Managing E-Business Application Tier for RAC (Doc ID 1311528.1)
3.1.1 Steps to configure Parallel Concurrent Processing 1. Change the context variables s_cp_two task to
<sid>
BALANCE alias in the context file. ####
``` $ cp $CONTEXT_FILE ${CONTEXT_FILE}_s_cp_two_date -I $ vim $CONTEXT_FILE $ diff $CONTEXT_FILE ${CONTEXT_FILE}_s_cp_two_date -I

400c400
< VIS_BALANCE
<CP_TWOTASK oa_var="s_cp_twotask">VIS</CP_TWOTASK>
```
Target node/queue unavailable
2. To enable reviver modify s_cp_reviver to enabled.
$ grep s_cp_reviver $CONTEXT_FILE <cp_reviver oa_var="s_cp_reviver">enabled</cp_reviver>
3. Run Autoconfig and verify that the configuration files tnsnames.ora andlistener.ora files have FNDSM entries for all nodes.
Autoconfig $ cd $ADMIN_SCRIPTS_HOME $ ./adautocfg.sh
$ cd /RYBAK/oracle/VIS/fs1/inst/apps/VIS_okon1/ora/10.1.2/network/admin $ grep -i FNDSM tnsnames.ora FNDSM_OKON3_VIS= FNDSM_OKON1_VIS= FNDSM_OKON2_VIS=
4. Restart the application listener process on all application nodes.
Restart MT
7. In Parallel Concurrent Processing implementation when the concurrent tier and database tier are different, then there are two options to provide high
availability, depending upon the value of the profile 'Concurrent:PCP Instance Check ' .

Google Code Archive - Long-Term Storage For Google Code Project Hosting

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Google Code Archive - Long-Term Storage For Google Code Project Hosting

Uploaded by

Copyright:

Available Formats

3/18/24, 2:41 PM Google Code Archive - Long-term storage for Google Code Project Hosting.

Installing R12 12_2_4 on multinode MT and 2 node RAC

"File Name" + "V46139-01.zip" + "V46140-01.zip" + "V46243-01_1of3.zip" + "V46243-01_2of3.zip" + "V46243-01_3of3.zip" + "V35231-01_1of5.zip" + "V35231-

rsync -a --progress /STAGING/ebs12patches/*.zip /RYBAK/

/vmfs/volumes/datastore1 RAC_11.2.0.3_SE_OEL6.3_openfiler RAC_11.2.0.3_SE_OEL6.3_rac1-user1a RAC_11.2.0.3_SE_OEL6.3_usera2

echo "inventory_loc=/RYBAK/oraInventory" > /etc/oraInst.loc

Perform some sanity net throughput tests

How to check what is the network speed on Synology?

zen (VirtualBox on laptop), NFS /RYBAK ```

time dd if=/dev/zero of=./trash bs=1024 count=1048576

okon2 (VirtualBox on laptop), NFS /RYBAK ```

time dd if=/dev/zero of=./trash bs=1024 count=1048576

Create staging area

$ unzip V46243-01_1of3.zip $ unzip V46243-01_2of3.zip $ unzip V46243-01_3of3.zip

Run as root ```

Now, lets see if there are any patches to RapidWiz

RI STARTCD 12.2.0.49 (Patch) p19378467_R12_GENERIC_1of2.zip 2.0 GB (2129316845 bytes) MD5 1E4B1DAE98F72A964BFE67282060FBEF

p19378467_R12_GENERIC_2of2.zip 3.1 GB (3289152372 bytes) MD5 0636D61EE3B51FD43791430BA13EB098

$ unzip p19378467_R12_GENERIC_1of2.zip $ unzip p19378467_R12_GENERIC_2of2.zip

$ cd startCD/Disk1/rapidwiz $ ./RapidWizVersion Version 12.2.0.49

Pre install check failures

There is already CRS installed Now: $ pwd /u01 $ ls app _inst

oracle@usera1 $ /u01/app/11.2.0/grid/bin/cluvfy stage -post crsinst -n usera1,usera2

He is trying somehow to reach IP for scan listener $ crsctl stat res -t

Anyway, again the check $ /u01/app/11.2.0/grid/bin/cluvfy comp nodecon -i eth0 -n usera1,usera2

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 25-DEC-2014 22:41:25

Copyright (c) 1991, 2011, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ----------------------- Alias LISTENER

> Ichose different port pool: 5,6

Errors during installation:

Fatal Error: TXK Install Service

Cannot install Oracle Database Home

MOS: Doc 1590656.1

Still the same error:

$ /STAGING/oracle_EBS_R12.2.4_linux_x86-64/TechInstallMedia/database/database/runInstaller -waitForCompletion -ignoreSysPrereqs -force -silent -

OK, so I should worry about the FATAL

task resolv.conf Integrity Failed

FATAL [INS-13013] Target environment do not meet some mandatory requirements.

$ /STAGING/oracle_EBS_R12.2.4_linux_x86-64/TechInstallMedia/database/database/runInstaller -waitForCompletion -ignoreSysPrereqs -force -silent -

$ . VIS_usera1.env $ env | grep SID ORACLE_SID=VIS

> this should be VIS1? $ export ORACLE_SID=VIS1

ORA-00132: syntax error or unresolved network name 'VIS1_LOCAL'

MOS: R122 installation failes

Patch history file: /u01/oracle/VIS/11.2.0/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/oracle/VIS/11.2.0/cfgtoollogs/opatch/lsinv/lsinventory2014-12-25_23-27-22PM.txt

------------------------------------------------------------------------------- There are no Interim patches installed in this Oracle Home.

Again the patching:

$ /u01/oracle/VIS/11.2.0/OPatch/opatch napply -skip_subset -skip_duplicate -verbose -silent -ocmrf

$ cd /u01/oracle/VIS/11.2.0/appsutil/log/VIS_usera1 $ less 12261002.log

Fatal Error: TXK Install Service

This with 49 rapid install.

$ cd /STAGING/oracle11.2.0.4_x86-64/deinstall $ ./deinstall -home /u01/oracle/VIS/11.2.0

Remove entry from oratab (both nodes) ```

cat /u01/app/oraInventory/ContentsXML/inventory.xml | grep "HOME

ps -ef | grep tns

otherwise the Grid listener running on 1521 is blocking the port.

What is NOT working

Post install issues

$ env | grep TNS (nothing)

$ . VIS_usera1.env $ env | grep TNS TNS_ADMIN=/u01/oracle/VIS/11.2.0/network/admin/VIS_usera1

Conclusion: Set properly the environment, not by just using oraenv.

Pre install actions