Errors while applying 12.2.5 RUP Patch in Oracle E-Business Suite

Issue

Patch application failed with errors and unable to proceed with 12.2.5 RUP

Error Messages

ERROR-1

Attempting upload of Context file and templates to database...ERROR: InDbCtxFile.uploadCtx() : Exception : Error executng BEGIN fnd_gsm_util.upload_context_file(:1,:2,:3,:4,:5); END;: 1; Oracle error -1400: : cannot insert NULL into ("APPLSYS"."FND_OAM_CONTEXT_FILES"."NODE_NAME") has been detected in FND_GSM_UTIL.upload_context_file.
oracle.apps.ad.autoconfig.oam.InDbCtxFileException: Error executng BEGIN fnd_gsm_util.upload_context_file(:1,:2,:3,:4,:5); END;: 1; Oracle error -1400: ORA-01400: cannot insert NULL into ("APPLSYS"."FND_OAM_CONTEXT_FILES"."NODE_NAME") has been detected in FND_GSM_UTIL.upload_context_file.
at oracle.apps.ad.autoconfig.oam.InDbCtxFile.uploadCtx(InDbCtxFile.java:281)
at oracle.apps.ad.autoconfig.oam.CtxSynchronizer.uploadToDb(CtxSynchronizer.java:332)
at oracle.apps.ad.tools.configuration.FileSysDBCtxMerge.updateDBCtx(FileSysDBCtxMerge.java:757)
at oracle.apps.ad.tools.configuration.FileSysDBCtxMerge.updateDBFiles(FileSysDBCtxMerge.java:228)
at oracle.apps.ad.context.CtxValueMgt.processCtxFile(CtxValueMgt.java:1721)
at oracle.apps.ad.context.CtxValueMgt.main(CtxValueMgt.java:787)

ERROR-2

[PROCEDURE] [START 2016/02/24 12:56:08] Updating adop session status 
[UNEXPECTED]Unable to find appltop_id for host ebs1225 in database

ERROR-3

I then tried running AutoConfig for both DB and APPS Tiers but getting following errors:

DB Tier Autoconfig Logfile
--------------------------

Logfile: /u01/oracle/VIS/12.1.0/appsutil/log/VIS_ebs1225/02250530/NetServiceHandler.log 
java.sql.SQLIntegrityConstraintViolationException: ORA-01400: cannot insert NULL into ("APPLSYS"."FND_NODES"."NODE_NAME")
ORA-06512: at "APPS.FND_APP_SYSTEM", line 1368
ORA-06512: at "APPS.FND_NET_SERVICES", line 863
ORA-06512: at line 1 Attempting upload of Context file and templates to database...ERROR: InDbCtxFile.uploadCtx() : Exception : Error executng BEGIN fnd_gsm_util.upload_context_file(:1,:2,:3,:4,:5); END;: 1; Oracle error -1400: ORA-01400: cannot insert NULL into ("APPLSYS"."FND_OAM_CONTEXT_FILES"."NODE_NAME") has been detected in FND_GSM_UTIL.upload_context_file.
oracle.apps.ad.autoconfig.oam.InDbCtxFileException: Error executng BEGIN fnd_gsm_util.upload_context_file(:1,:2,:3,:4,:5); END;: 1; Oracle error -1400: ORA-01400: cannot insert NULL into ("APPLSYS"."FND_OAM_CONTEXT_FILES"."NODE_NAME") has been detected in FND_GSM_UTIL.upload_context_file.
at oracle.apps.ad.autoconfig.oam.InDbCtxFile.uploadCtx(InDbCtxFile.java:281)
at oracle.apps.ad.autoconfig.oam.CtxSynchronizer.uploadToDb(CtxSynchronizer.java:332)
at oracle.apps.ad.tools.configuration.FileSysDBCtxMerge.updateDBCtx(FileSysDBCtxMerge.java:757)
at oracle.apps.ad.tools.configuration.FileSysDBCtxMerge.updateDBFiles(FileSysDBCtxMerge.java:228)
at oracle.apps.ad.context.CtxValueMgt.processCtxFile(CtxValueMgt.java:1721)
at oracle.apps.ad.context.CtxValueMgt.main(CtxValueMgt.java:787)
FAILED

Apps Tier Autoconfig Logfile
----------------------------

Executing script in InstantiateFile:
/u01/oracle/VIS/fs1/inst/apps/VIS_ebs1225/admin/install/afcpnode.sh

declare
*
ERROR at line 1:
ORA-01400: cannot insert NULL into ("APPLSYS"."FND_NODES"."NODE_NAME")
ORA-06512: at "APPS.FND_CONCURRENT", line 1628
ORA-06512: at line 1
ORA-06512: at line 109
ORA-01400: cannot insert NULL into ("APPLSYS"."FND_NODES"."NODE_NAME")
ORA-01400: cannot insert NULL into ("APPLSYS"."FND_NODES"."NODE_NAME")

ERROR-4

Used the following command to restart the patch application:
adop phase=apply apply_mode=downtime patches=19676458 restart=yes

But it immediately threw the following error:
*******FATAL ERROR*******
PROGRAM : (/u01/oracle/VIS/fs1/EBSapps/appl/ad/12.0.0/bin/adzdoptl.pl)
TIME : Tue Mar 1 11:35:13 2016
FUNCTION: ADOP::GlobalVars::_GetMandatoryArgs [ Level 1 ]
ERRORMSG: Cannot identify any valid application tier nodes in
ADOP_VALID_NODES table. Ensure AutoConfig has been run on all nodes.

ERROR-5

Worker status seen using adctrl

Control
Worker Code Context Filename Status
------ -------- ----------------- -------------------------- --------------
1 Run AutoPatch R120 Wait
2 Run AutoPatch R120 AHL_ENDECA_RB_V.xdf Running
3 Run AutoPatch R120 CST_END_TXN_REG_INV_V.xdf Running
4 Run AutoPatch R120 AHL_SIMULATION_FORECAST_V. Running
5 Run AutoPatch R120 Wait
6 Run AutoPatch R120 Wait
7 Run AutoPatch R120 Wait
8 Run AutoPatch R120 Wait
9 Run AutoPatch R120 Wait
10 Run AutoPatch R120 Wait
11 Run AutoPatch R120 AHL_UE_ORGS_DEPTS_V.xdf Running
12 Run AutoPatch R120 Wait
13 Run AutoPatch R120 AHL_PLANNING_FACTORS_V.xdf Running
14 Run AutoPatch R120 Wait
15 Run AutoPatch R120 Wait
16 Run AutoPatch R120 Wait
17 Run AutoPatch R120 Wait
18 Run AutoPatch R120 Wait
19 Run AutoPatch R120 Wait
20 Run AutoPatch R120 Wait
21 Run AutoPatch R120 Wait
22 Run AutoPatch R120 Wait
23 Run AutoPatch R120 Wait
24 Run AutoPatch R120 Wait
25 Run AutoPatch R120 AHL_ENDECA_INST_DTLS_V.xdf Running
26 Run AutoPatch R120 Wait
27 Run AutoPatch R120 Wait
28 Run AutoPatch R120 Wait
29 Run AutoPatch R120 Wait
30 Run AutoPatch R120 Wait
31 Run AutoPatch R120 Wait
32 Run AutoPatch R120 Wait

Investigation Done

Old definition for FND_OAM_CONTEXT_FILES and FND_NODES synonyms in apps schema

Dont's

Do NOT run the following since its applicable prior to R12.2 versions only:

+ MOS Doc Id 260887.1 to delete all data from system tables such as FND_NODES, FND_OAM_CONTEXT_FILES through the execution of following SQL:

SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;

Resolution

Step 1. Please recreate the synonyms :
drop synonym apps.fnd_nodes;
create synonym apps.fnd_nodes for applsys.fnd_nodes;
drop synonym apps.fnd_oam_context_files;
create synonym apps.fnd_oam_context_files for applsys.fnd_oam_context_files;

Step 2. Change the worker status to restart using adcrtl utility

Step 3. Restart the patch:
adop phase=apply apply_mode=downtime patches=19676458 abandon=no restart=yes

Environment

+ Oracle E-Business Suite R12.2.0 to R12.2.5 upgrade

+ Oracle Linux 5.x (x86-64)

What is ADOP Utility and what are its functions?
API Error While Creating A Project From Project Te...