Apply Bundle Patch on ODA HA

This doc describe the step by step procedure to apply patch on ODA HA.


Component SERVER to be patch :-
Operating System, ILOM,BIOS, Hardware Management Pack, IPMI, OAK, ASR and local disks.
For virtualized environment, it also includes the Dom0 updates.
Oracle Grid Infrastructure (GI) Stack including ASM, ACFS and TFA.

Component STORAGE to be patch:-
Shared storage including HDD, SSD, Controller, and Expander

Component DATABASE to be patch:-
RDBMS stack

The patch bundle must be applied to the SERVER components first, followed by STORAGE and then the DATABASE.


1. Download the latest bundle patch from MOS note 888888.1
ODA Patch 12.1.2.11.0 Patch 26080564

2. Copy the downloaded patch bundles to both the ODA nodes.
 On a virtualized environment, copy the patch bundle to the ODA_Base (Dom1).

use scp or winscp

3. Check the current version of all components of ODA.

[root@abc0 oak]# oakcli show version -detail
Reading the metadata. It takes a while...
System Version  Component Name            Installed Version         Supported Version
--------------  ---------------           ------------------        -----------------
12.1.2.9.0
                Controller                11.05.03.00               Up-to-date
                Expander                  0342                      Up-to-date
                SSD_SHARED                E12B                      Up-to-date
                HDD_LOCAL                 SF04                      Up-to-date
                HDD_SHARED                0B25                      Up-to-date
                ILOM                      3.0.16.22.d r83408        3.0.16.22.f r100119
                BIOS                      12010311                  Up-to-date
                IPMI                      1.8.12.4                  Up-to-date
                HMP                       2.3.5.2.5                 Up-to-date
                OAK                       12.1.2.9.0                Up-to-date
                OL                        6.8                       Up-to-date
                OVM                       3.2.11                    Up-to-date
                GI_HOME                   12.1.0.2.161018(2400      Up-to-date
                                          6101,23854735)
                DB_HOME                   12.1.0.2.161018(2400      Up-to-date
                                          6101,23854735)

4. Unpack both patch bundles on both Nodes.

[root@abc0 oak]# oakcli unpack -package /tmp/p26080564_1212110_Linux-x86-64_1of2.zip
Unpacking will take some time,  Please wait...
Successfully unpacked the files to repository.
[root@abc0 oak]# oakcli unpack -package /tmp/p26080564_1212110_Linux-x86-64_2of2.zip
Unpacking will take some time,  Please wait...
Successfully unpacked the files to repository.

5. Verify that existing system can be patched

[root@abc0 oak]# oakcli update -patch  12.1.2.11.0 --verify
INFO: 2017-07-19 07:52:47: Reading the metadata file now...
                Component Name            Installed Version         Proposed Patch Version
                ---------------           ------------------        -----------------
                Controller                11.05.03.00               Up-to-date
                Expander                  0342                      Up-to-date
                SSD_SHARED                E12B                      Up-to-date
                HDD_LOCAL                 SF04                      Up-to-date
                HDD_SHARED                0B25                      Up-to-date
                ILOM                      3.0.16.22.d r83408        3.0.16.22.f r100119
                BIOS                      12010311                  Up-to-date
                IPMI                      1.8.12.4                  Up-to-date
                HMP                       2.3.5.2.5                 2.3.5.2.8
                OAK                       12.1.2.9.0                12.1.2.11.0
                OL                        6.8                       Up-to-date
                OVM                       3.2.11                    3.4.3
                GI_HOME                   12.1.0.2.161018(2400      12.1.0.2.170418(2517
                                          6101,23854735)            1037,25942196)
                DB_HOME                   12.1.0.2.161018(2400      12.1.0.2.170418(2517
                                          6101,23854735)            1037,25942196)

6. Apply the patch for server components.This will patch the Infra and GI components.
At the end of the patching process it will re-boot the server. In no particular order which node can be patched first.  Do not run the commands on both nodes at the

same time.


[root@abc0 oak]# oakcli update -patch  12.1.2.11.0 --server --local
INFO: Local patch is running on the Node <0>
INFO: ***************************************************
INFO: ** Please do not patch both nodes simultaneously **
INFO: ***************************************************
INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Local Node may get rebooted automatically during the patch if necessary
Do you want to continue: [Y/N]?: Y
INFO: User has confirmed for the reboot
INFO: All the VMs except the ODABASE will be shutdown forcefully if needed
Do you want to continue : [Y/N]? : Y
INFO: Running pre-install scripts
INFO: Running prepatching on local node
INFO: Completed pre-install scripts
INFO: Running patching on local node
INFO: Stopping local VMs, repos and oakd...
INFO: Shutdown of local VM, Repo and OAKD on node <0>.
INFO: Stopping OAKD on the local node.
INFO: Stopped Oakd on local node
INFO: Waiting for processes to sync up...
INFO: Stopping local VMs...
INFO: Stopping local shared repos...
INFO: Patching ODABASE components
INFO: 2017-07-24 05:57:14: Patching ODABASE Server Components (including Grid software)

INFO: 2017-07-24 05:57:14: ------------------Patching HMP-------------------------
INFO: 2017-07-24 05:57:16: HMP is already Up-to-date
INFO: 2017-07-24 05:57:16: /usr/lib64/sun-ssm already exists.

INFO: 2017-07-24 05:57:16: ----------------------Patching OAK---------------------
INFO: 2017-07-24 05:57:17: OAK is already upgraded or running with the latest version

INFO: 2017-07-24 05:57:20: ----------------------Patching JDK---------------------
INFO: 2017-07-24 05:57:20: JDK is already upgraded or running with the latest version

INFO: 2017-07-24 05:57:20: ----------------------Patching IPMI---------------------
INFO: 2017-07-24 05:57:21: IPMI is already upgraded or running with the latest version

INFO: 2017-07-24 05:57:21: ------------------Patching OS-------------------------
INFO: 2017-07-24 05:57:38: OS is already updated
INFO: 2017-07-24 05:57:43: Clusterware is running on local node
INFO: 2017-07-24 05:57:43: Attempting to stop clusterware and its resources locally
SUCCESS: 2017-07-24 05:59:00: Successfully stopped the clusterware on local node

INFO: 2017-07-24 05:59:00: ------------------Patching Grid-------------------------
INFO: 2017-07-24 05:59:02: Checking for available free space on /, /tmp, /u01
INFO: 2017-07-24 05:59:11: Shutting down Clusterware and CRS on local node.
INFO: 2017-07-24 05:59:11: Shutting down CRS on the node...
SUCCESS: 2017-07-24 05:59:14: Successfully stopped CRS processes on the node
INFO: 2017-07-24 05:59:14: Checking for running CRS processes on the node.
INFO: 2017-07-24 05:59:35: Starting up CRS and Clusterware on the node
INFO: 2017-07-24 05:59:35: Starting up CRS on the node...
SUCCESS: 2017-07-24 06:02:15: CRS has started on the node

INFO: 2017-07-24 06:02:20: Patching the GRID Home on the local node...
INFO: 2017-07-24 06:02:20: Updating OPATCH...
Verifying Opatch version for home:</u01/app/12.1.0.2/grid> on local node.
Expecting version:<12.2.0.1.8>
Opatch version on local node is <12.2.0.1.8>
INFO: 2017-07-24 06:02:24: Patching the GRID Home on the local Node...
INFO: 2017-07-24 06:02:24: Performing local patching...
INFO: 2017-07-24 06:03:11: Performing the conflict checks...
SUCCESS: 2017-07-24 06:03:31: Conflict checks passed for all the Homes
INFO: 2017-07-24 06:03:31: Checking if the patch is already applied to the GRID homes
INFO: 2017-07-24 06:04:01: Home is not Up-to-date
SUCCESS: 2017-07-24 06:05:14: Successfully stopped the Database consoles
SUCCESS: 2017-07-24 06:05:19: Successfully stopped the EM agents
INFO: 2017-07-24 06:05:25: Applying patch on /u01/app/12.1.0.2/grid Homes
INFO: 2017-07-24 06:05:25: It may take upto 15 mins. Please wait...
SUCCESS: 2017-07-24 06:22:05: Successfully applied the patch on the Home : /u01/app/12.1.0.2/grid
SUCCESS: 2017-07-24 06:22:24: Successfully started the Database consoles
SUCCESS: 2017-07-24 06:22:24: Successfully started the EM Agents
INFO: 2017-07-24 06:22:25: ASM is running in Flex mode
INFO: 2017-07-24 06:22:25: Patching DOM0 Server Components
INFO: 2017-07-24 06:22:25: Attempting to patch OS on Dom0...
INFO: 2017-07-24 06:22:46: Clusterware is running on local node
INFO: 2017-07-24 06:22:46: Attempting to stop clusterware and its resources locally
SUCCESS: 2017-07-24 06:23:51: Successfully stopped the clusterware on local node

SUCCESS: 2017-07-24 06:46:28: Successfully updated the device OVM to 3.4.3
INFO: 2017-07-24 06:46:28: Attempting to patch the HMP on Dom0...
INFO: 2017-07-24 06:46:53: The device HMP is already running with the version 2.3.5.2.8 on Dom0
INFO: 2017-07-24 06:46:53: Attempting to patch the IPMI on Dom0...
INFO: 2017-07-24 06:47:14: Successfully updated the IPMI on Dom0
INFO: 2017-07-24 06:47:18: Attempting to patch the local storage on Dom0...
INFO: 2017-07-24 06:47:18: Stopping clusterware on local node...
INFO: 2017-07-24 06:47:43: Disk : c0d0 is already running with ST95000N SF04
INFO: 2017-07-24 06:47:43: Disk : c0d1 is already running with ST95000N SF04
INFO: 2017-07-24 06:48:07: Patch not found for the Controller : c0
INFO: 2017-07-24 06:48:27: Attempting to patch the ILOM on Dom0...
SUCCESS: 2017-07-24 07:06:20: Successfully updated the device Ilom to 3.0.16.22.f r100119
SUCCESS: 2017-07-24 07:06:20: Successfully updated the device BIOS to 12010311

INFO:  patching summary on local node
INFO: 2017-07-24 07:06:20:  HMP is already updated
INFO: 2017-07-24 07:06:20:  OAK is already upgraded
INFO: 2017-07-24 07:06:20:  JDK is already upgraded
INFO: 2017-07-24 07:06:20:  IPMI is already upgraded
INFO: 2017-07-24 07:06:20:  OS is already updated
SUCCESS: 2017-07-24 07:06:20:  Successfully applied the patch on the Home /u01/app/12.1.0.2/grid
SUCCESS: 2017-07-24 07:06:20:  Successfully updated the device OVM
INFO: 2017-07-24 07:06:20:  Local storage patching summary on Dom0...
SUCCESS: 2017-07-24 07:06:20:  Successfully upgraded the local storage
SUCCESS: 2017-07-24 07:06:20:  Successfully updated the device Ilom
SUCCESS: 2017-07-24 07:06:20:  Successfully updated the device BIOS

INFO: Running post-install scripts
INFO: Running postpatch on local node
INFO: restart OAKD on local node
INFO: 2017-07-24 07:14:04: Some of the components patched on node <192.168.16.27>
INFO: 2017-07-24 07:14:04: require node reboot. Rebooting the node
INFO: Rebooting Dom0 on local node
INFO: 2017-07-24 07:14:04: Running /tmp/dom0reboot on node 0
[root@abc0 oak]#
Broadcast message from root@abc0
        (unknown) at 7:18 ...

The system is going down for power off NOW!


7. Verify the updated version of the all components.

[root@abc0 oak]# oakcli show version -detail
Reading the metadata. It takes a while...
System Version  Component Name            Installed Version         Supported Version
--------------  ---------------           ------------------        -----------------
12.1.2.11.0
                Controller                11.05.03.00               Up-to-date
                Expander                  0342                      Up-to-date
                SSD_SHARED                E12B                      Up-to-date
                HDD_LOCAL                 SF04                      Up-to-date
                HDD_SHARED                0B25                      Up-to-date
                ILOM                      3.0.16.22.f r100119       Up-to-date
                BIOS                      12010311                  Up-to-date
                IPMI                      1.8.12.4                  Up-to-date
                HMP                       2.3.5.2.8                 Up-to-date
                OAK                       12.1.2.11.0               Up-to-date
                OL                        6.8                       Up-to-date
                OVM                       3.4.3                     Up-to-date
                GI_HOME                   12.1.0.2.170418(2517      Up-to-date
                                          1037,25942196)
                DB_HOME                   12.1.0.2.161018(2400      12.1.0.2.170418(2517
                                          6101,23854735)            1037,25942196)

8. Apply the patch to the storage components.
Storage patching should only be attempted after the server patching is successful.
No ‘-local’ option for storage.  If required, both nodes will be rebooted.
If ‘proposed patch version’ for storage components are all up-to-date from ‘oakcli update –patch 12.1.2.11 --verify’, then we can skip the storage update.

oakcli update -patch  12.1.2.11.0 --storage


9.Apply the database patch – to update databases to the latest PSU for the same release.
If patching database with ‘--local’ parameter, the order or which node to be patched does not matter.  Please do not run the command at the same time on both nodes.

The database patching is rolling upgradable.


[root@abc0 oak]# oakcli update -patch  12.1.2.11.0  --database --local
INFO: Local patch is running on the Node <0>
INFO: ***************************************************
INFO: ** Please do not patch both nodes simultaneously **
INFO: ***************************************************
INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Running pre-install scripts
INFO: Running prepatching on local node
INFO: Completed pre-install scripts
INFO: Running patching on local node
INFO: 2017-07-24 08:23:31: ------------------Patching DB-------------------------
INFO: 2017-07-24 08:23:31: Clusterware running on local node
INFO: 2017-07-24 08:23:31: Getting all the possible Database Homes for patching...
INFO: 2017-07-24 08:23:34: home version: 12.1.0.2.0
INFO: 2017-07-24 08:23:43: Patching 12.1.0.2 Database Homes on the local Node

Found the following 12.1.0.2 homes possible for patching:

HOME_NAME                      HOME_LOCATION
---------                      -------------
OraDb12102_home1               /u01/app/oracle/product/12.1.0.2/dbhome_1

[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]

Would you like to patch all the above homes: Y | N ? : Y
INFO: 2017-07-24 08:25:27: Updating OPATCH...
Verifying Opatch version for home:</u01/app/oracle/product/12.1.0.2/dbhome_1> on local node.
Expecting version:<12.2.0.1.8>
Opatch version on local node is <12.2.0.1.8>
INFO: 2017-07-24 08:25:34: Performing local patching...
INFO: 2017-07-24 08:29:31: Performing the conflict checks...
SUCCESS: 2017-07-24 08:29:52: Conflict checks passed for all the Homes
INFO: 2017-07-24 08:29:52: Checking if the patch is already applied on any of the Homes
INFO: 2017-07-24 08:30:01: Home is not Up-to-date
SUCCESS: 2017-07-24 08:30:03: Successfully stopped the Database consoles
SUCCESS: 2017-07-24 08:30:08: Successfully stopped the EM agents
INFO: 2017-07-24 08:30:14: Applying patch on /u01/app/oracle/product/12.1.0.2/dbhome_1 Homes
INFO: 2017-07-24 08:30:14: It may take upto 15 mins. Please wait...
SUCCESS: 2017-07-24 08:43:31: Successfully applied the patch on the Home : /u01/app/oracle/product/12.1.0.2/dbhome_1
SUCCESS: 2017-07-24 08:43:31: Successfully started the Database consoles
SUCCESS: 2017-07-24 08:43:31: Successfully started the EM Agents

INFO:  patching summary on local node
SUCCESS: 2017-07-24 08:43:32:  Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1

INFO: Executing /tmp/pending_actions on local node


10. Verify the Database updated version of the database.

[root@abc0 oak]# oakcli show version -detail
Reading the metadata. It takes a while...
System Version  Component Name            Installed Version         Supported Version
--------------  ---------------           ------------------        -----------------
12.1.2.11.0
                Controller                11.05.03.00               Up-to-date
                Expander                  0342                      Up-to-date
                SSD_SHARED                E12B                      Up-to-date
                HDD_LOCAL                 SF04                      Up-to-date
                HDD_SHARED                0B25                      Up-to-date
                ILOM                      3.0.16.22.f r100119       Up-to-date
                BIOS                      12010311                  Up-to-date
                IPMI                      1.8.12.4                  Up-to-date
                HMP                       2.3.5.2.8                 Up-to-date
                OAK                       12.1.2.11.0               Up-to-date
                OL                        6.8                       Up-to-date
                OVM                       3.4.3                     Up-to-date
                GI_HOME                   12.1.0.2.170418(2517      Up-to-date
                                          1037,25942196)
                DB_HOME                   12.1.0.2.170418(2517      Up-to-date
                                          1037,25942196)


For more details pls refer to README file attached to the bundle patch.