Issue

Upgrade patch 17919161 is taking very long time to apply (upgrading from R12.2.0 to 12.2.4). It has been already over 2 days but some scripts seem to got hung under the patch.

Investigation Done

1. Open a new terminal > login as Application Tier owner

2. Start 'adctrl' utility and list down the worker status. You will see that 3 workers are shown the status of 'RUNNING' with the following 3 scripts:

adobjcmp.sql

appostimportaad.sql

fapostimportaad.sql

3. It looks like there this issue is being possible caused by Invalid Objects in the database APPS schema.

Resolution

There are two ways under which this issue can be resolved:

Option 1

Skip the workers that are running the above jobs. This way rest of the jobs will continue and the patch will be fully applied.

1. 1. Open a new terminal > login as Application Tier owner

2. Start 'adctrl' utility and list down the worker status. You will see that 3 workers are shown the status of 'RUNNING' with the following 3 scripts:

adobjcmp.sql

appostimportaad.sql

fapostimportaad.sql

3. Choose option 2 'Tell worker to restart a failed job', and then give the workers againt which the above scripts are hanging. Once you give this option the worker status will change from 'Running' to 'Failed'.

4. Now choose option 8 to skip a worker (this option is not shown by default and hidden). So simply type 8 when it ask for the list of options. At this stage the above workers status will change from 'Failed' to 'Skip & Restart'.

5. Go back to the terminal session where the patch was running. Kill it by 'Ctrl+C'. And then restart the patch using the following command:

adop phase=apply apply_mode=downtime patches=17919161 abandon=yes

6. This time the patch may not hang and will complete successfully. On the contrary, if the patch hangs again then follow above OPTION 1.

Option 2 - 

1. Open a new terminal > login as Application Tier owner

2. Run 'adadmin' utility > Choose option 3 'Compile Reload Applications Database Entity Menu' > Choose option 1 'Compile APPS Schema'.

3. When it ask you to drop the 'FND_INSTALL_PROCESSES' table then choose 'yes'. Then it will take few hours to recompile all database objects (including invalid).

4. Once completed, exit from adadmin, and then restart the above patch application using command:

adop phase=apply apply_mode=downtime patches=17919161 abandon=yes

5. This time the patch may not hang and will complete successfully. On the contrary, if the patch hangs again then follow above OPTION 1.

Environment

• Oracle E-Business Suite Upgrade from R12.2.0 to R12.2.4
• Patch # 17919161
• Oracle Linux Operating System