How to perform a rolling upgrade in batches, using root user automation, while managing services across upgraded and non-upgraded nodes
Upgrading Oracle Grid Infrastructure in a production RAC environment requires more than following the standard installation procedure.
One of the key challenges is coordinating cluster node upgrades
while maintaining database service availability and minimizing operational
disruption.
With Oracle Grid Infrastructure 26ai, batch-based rolling upgrades
provide an approach to upgrade cluster nodes in stages, rather than treating
the entire cluster as a single upgrade operation.
In this technical walkthrough, I explore a practical four-node
Oracle RAC scenario, divided into two upgrade batches, covering:
v How batch-based rolling upgrades work.
v The role of root user automation and `rootupgrade.sh`.
v How RAC database instances are handled during individual node upgrades.
v How to plan service relocation between upgrade batches.
v Practical `gridSetup.sh -upgrade -batchInfo` command examples.
v Pre-upgrade validation using CVU and installer dry-run mode.
v Post-upgrade verification and handling unreachable cluster nodes.
The walkthrough also highlights important operational restrictions
and considerations that should be understood before starting a production
upgrade.
Ø My objective is to go beyond
documenting upgrade commands and focus on the operational decisions required to
execute a controlled Grid Infrastructure upgrade.
The important distinction is that Oracle describes two
execution approaches: using root automation to upgrade
selected batches of nodes, or running root scripts manually,
which upgrades the entire cluster rather than allowing individual
nodes to be selected or deselected.
Imagine a four-node RAC cluster:
All four nodes run the existing Grid Infrastructure version and host RAC instances.
For a 4-node Oracle RAC cluster upgrading Grid Infrastructure
to 26ai, I would plan the work in two batches, with two nodes per
batch, assuming Oracle supports that batch configuration for your specific
source version and upgrade path.
The objective is to upgrade Grid Infrastructure in
stages, move services where appropriate, and maintain database availability
through the remaining RAC instances wherever the architecture permits.
Example: rolling upgrade in two batches
This illustrates the batching concept; actual batch sizes and
service placement depend on the supported upgrade path and cluster design.
The purpose is to upgrade the cluster in stages instead of
treating every node as one simultaneous maintenance operation.
Validate the first batch and arrange service placement before
continuing.
Complete the remaining batch, then validate the entire cluster and
its services (CVU pre-upgrade validation).
2. Why does Oracle recommend
root user automation?
Root user automation allows you to:
- Divide
cluster nodes into upgrade batches.
- Start
upgrades for those batches.
- Move
services from nodes running the old version to upgraded nodes between
batches.
- Automate
execution of rootupgrade.sh, including stopping and restarting instances
as required.
Without root user automation, Oracle says you cannot select or
deselect individual nodes for the upgrade in this manner; the upgrade applies
to the entire cluster.
3. What happens to the RAC
database instances?
Oracle recommends leaving RAC instances running while upgrading
Oracle Clusterware. However, when the root upgrade script starts on an
individual node, the database instance on that node is shut down, and
rootupgrade.sh starts it again.
These two statements are not contradictory:
- At
cluster level: You do not need to manually shut down
all RAC instances before beginning the upgrade.
- At node
level: The instance on a node undergoing its
upgrade is shut down and restarted as part of that process.
The other nodes may continue serving workloads, provided the
cluster has sufficient capacity and the application can use the remaining
instances and services.
That does not guarantee zero downtime. Service
placement, application connection handling, workload capacity, and the exact
supported upgrade procedure all matter.
4. Why must you not add nodes
during a rolling upgrade?
Note that Oracle does not support attempting to add additional
nodes during a rolling upgrade.
For example, if you start with four nodes and plan to add a fifth
node during the upgrade, do not combine that node-addition operation with the
rolling upgrade. Complete the upgrade according to the supported procedure,
then handle node expansion as a separate planned activity.
5. What should you do before
the maintenance window?
A practical checklist derived from these restrictions is:
- Confirm
the supported upgrade path and prerequisites.
- Validate
storage, patches, and cluster readiness.
- Define
the node batches and service relocation plan.
- Check
application connectivity and the capacity of the nodes that will remain
available.
- Decide
whether to use root user automation and validate the execution procedure.
- Establish
post-upgrade checks for cluster resources, RAC instances, services, and
application connectivity.
The key operational lesson: Rolling upgrades reduce the need
to upgrade the entire cluster at once, but they do not eliminate instance
restarts or automatically guarantee application continuity. The upgrade
sequence and service placement must be designed around the workload.
Step-by-step procedure for
4-Node RAC (Upgrading 19c to 26ai)
The following is a planning runbook, not a copy-paste-ready
production procedure. Exact commands and batch selection depend on the source
Grid Infrastructure release, OS, and supported upgrade path.
Phase 1: Pre-upgrade
preparation
Step 1. Establish the current state on all four nodes.
As the Grid owner, check the current cluster and ASM status:
olsnodes -n
crsctl check cluster
-all
crsctl stat res -t
srvctl status
database -d <DB_UNIQUE_NAME>
asmcmd lsdg
Record the existing Grid home, database homes, services, ASM disk
groups, and current resource placement.
Step 2. Validate prerequisites.
- Confirm
the supported upgrade path to 26ai.
- Check
mandatory patches and run CVU pre-upgrade validation.
- Verify
OCR and voting-file storage locations.
- Plan the
migration away from ASMFD if configured.
- Address
GIMR removal if configured.
- Confirm
backups and a documented recovery procedure.
Step 3. Prepare the new Grid home on the cluster nodes.
Extract the 26ai Grid Infrastructure image into the intended new
Grid home. Confirm ownership, permissions, available space, and required
configuration consistency.
Grid Infrastructure upgrades are out-of-place; do not overwrite
the existing Grid home.
mkdir -p
/u01/app/23.0.0/grid
chown grid:oinstall
/u01/app/23.0.0/grid
cd
/u01/app/23.0.0/grid
unzip -q
download_location/grid_home.zip
You can verify that the permissions required for installing Oracle Clusterware has been configured on the nodes node1 and node2 by running a command similar to the following (Upgrading 19c to 26ai).
$
/u01/app/23.0.0/grid/runcluvfy.sh stage -pre
crsinst \
-upgrade -rolling \
-src_crshome /u01/app/19.0.0/grid
\
-dest_crshome
/u01/app/23.0.0/grid \
-dest_version 23.0.0.0.0 \
-fixup -verbose
Oracle Grid Infrastructure dry-run upgrade mode
enables you to check system readiness for upgrade.
Ø
Starting with Oracle Grid
Infrastructure 19c, the Oracle Grid Infrastructure installer enables you to
perform a dry-run upgrade to check readiness of the system for upgrade. To
perform Oracle Grid Infrastructure dry-run upgrade, create a new Grid home with
the necessary user group permissions, extract the Oracle Grid Infrastructure
26ai gold image to the new Grid home, and then start the installer with, dryRunForUpgrade
flag.
Dry-run upgrades are not supported on Oracle Grid Infrastructure
for a standalone server (Oracle Restart) configuration.
Note: The installer does not perform an actual upgrade in the dry-run
upgrade mode. You can relaunch the installer, without any flag, from any of the
cluster nodes to upgrade Oracle Grid Infrastructure if dry-run is successful.
The installer performs the following tasks in dry-run upgrade
mode:
- Validates
storage and network configuration for the new release
- Checks
if the system meets the software and hardware requirements for the new
release
- Checks
for the patch requirements and apply necessary patches before starting the
upgrade
- Writes
system configuration issues or errors in the gridSetupActions<timestamp>.log
log file
The Grid infrastructure dry-run upgrade flow is similar to a
regular upgrade, but the installer does not run any configuration tool.
Step 4. Run the dry-run upgrade.
Use the documented installer workflow:
$
/u01/app/23.0.0/grid/gridSetup.sh -dryRunForUpgrade
Review the installer logs and resolve reported issues before the
actual upgrade. Oracle recommends configuring root script automation during
this preparation.
Phase 2: Execute the rolling
upgrade
Step 5. Define the batches.
For example, if your supported upgrade procedure permits two
batches:
|
Batch |
Nodes |
Action |
|
1 |
Node 1, Node 2 |
Upgrade first |
|
2 |
Node 3, Node 4 |
Upgrade after the first batch |
The node where you run Oracle Universal Installer must belong to
Batch 1.
Oracle permits up to three batches.
Four-node example: two nodes per batch
Assume your cluster has these hostnames:
|
Node |
Batch |
|
rac01 ; local node running gridSetup.sh |
1 |
|
rac02 |
1 |
|
rac03 |
2 |
|
rac04 |
2 |
The batch-map format shown in Oracle response-file templates is a
comma-separated list of node:batch pairs.
Applying that format to this example gives:
rac01:1, rac02:1,
rac03:2, rac04:2
The corresponding command would be:
cd /u01/app/26.0.0/grid
./gridSetup.sh
-upgrade \
-ORACLE_BASE /u01/app/grid -batchInfo rac01:1, rac02:1, rac03:2, rac04:2
-executePrereqs
./gridSetup.sh
-upgrade -ORACLE_BASE /u01/app/grid \
-batchInfo rac01:1,rac02:1,rac03:2,rac04:2 -dryRun
$NEW_GRID_HOME/gridSetup.sh -upgrade
-ORACLE_BASE /u01/app/grid \
-batchInfo rac01:1,rac02:1,rac03:2,rac04:2
-executeConfigTools \
-executeRootScript
-configMethod ROOT
How the batches work
- Batch 1: rac01
and rac02 are assigned to the first upgrade batch.
- Batch 2: rac03
and rac04 are assigned to the second batch.
- The
local node, rac01, must be in Batch 1.
- With
root script automation configured, the installer can pause between batches
so services can be relocated before continuing.
Do not assume that specifying -batchInfo alone completes
the upgrade or guarantees service continuity. Follow the installer prompts and
the documented root-script/configuration workflow.
One final recommendation: inspect the response-file template shipped with your actual 26ai installation image for the batchInfo description and examples. That is the best way to confirm the expected mapping format for the exact installer you will run.
Step 6. Start the actual upgrade.
Run the upgrade using the supported OUI workflow or the documented
gridSetup.sh -upgrade options. The -batchInfo option specifies
the cluster-node batch map; use the exact syntax and mapping required by the
26ai installer for your environment.
Ø
Do not assume that running the
installer once automatically means both batches have completed.
Relaunch the installer from the same Grid home without any flag to
start an actual upgrade.
cd
/u01/app/23.0.0/grid
$
/u01/app/23.0.0/grid/gridSetup.sh -upgrade -ORACLE_BASE /u01/app/grid [-ignoreDownNodes] -executeConfigTools -executeRootScript -configMethod ROOT
Provide the root user password when prompted.
Because the Oracle Grid Infrastructure home is in a different
location than the former Oracle Clusterware and Oracle ASM homes,
update any scripts or applications that use utilities, libraries, or other
files that reside in the Oracle Clusterware and Oracle ASM homes.
Note: If you need
to apply patches before performing the upgrade, then use the OPatch utility to
apply in-place patches on each
cluster node.
Step 7. Execute the root upgrade script for Batch 1.
Follow the installer’s instructions and use root user automation
where supported. The upgrade script stops and restarts the database instance on
the node being processed.
Verify that the first batch has completed successfully before
proceeding.
Step 8. Validate and manage services between batches.
Check the cluster resources, database instances, service
placement, and application connectivity. Relocate services where required by
your plan, ensuring that the remaining nodes have sufficient capacity.
Step 9. Upgrade Batch 2.
Proceed only when the upgrade workflow permits it and Batch 1 has
reached the required state. Execute the root upgrade operations as directed by
the installer.
Phase 3: Post-upgrade
validation
Step 10. Confirm the final cluster state.
Check:
crsctl check cluster
-all
crsctl stat res -t
srvctl status
database -d <DB_UNIQUE_NAME>
srvctl status
service -d <DB_UNIQUE_NAME>
Also verify ASM disk groups, OCR and voting files, database
connectivity, service failover behavior, and the relevant installer and upgrade
logs.
Step 11. Complete required post-upgrade tasks.
Review the 26ai documentation for any required changes to OCR
backup locations, configuration entries, and scripts that reference the old
Grid home. Retain the previous home until the upgrade has been validated and
the applicable retention and recovery requirements have been satisfied.
Important: The 2+2 layout above is an example, not a guarantee that this
batch plan is supported for every source release or configuration. Before using
it in production, confirm the exact upgrade path and batch restrictions for
your environment based on official Oracle docs.
Completing an Oracle Clusterware Upgrade when Nodes Become
Unreachable
If some nodes become unreachable in the middle of an upgrade, then
you cannot complete the upgrade, because the upgrade script (rootupgrade.sh)
did not run on the unreachable nodes. Because the upgrade is incomplete, Oracle
Clusterware remains in the previous release.
- Confirm
that the upgrade is incomplete by entering the following command:
$ crsctl query crs
activeversion
The active release should be the source release.
- To
resolve this problem, run the rootupgrade.sh command with the -force flag
using the following syntax:
$Grid_home/rootupgrade -force
For example:
#
/u01/app/23.0.0/grid/rootupgrade -force
This command forces the upgrade to complete. The force cluster
upgrade has the following limitations:
- All
active nodes must be upgraded to the newer release.
- All
inactive nodes (accessible or inaccessible) may be either upgraded or not
upgraded.
- For
inaccessible nodes, after patch set upgrades, you can delete the node
from the cluster. If the node becomes accessible later, and the patch
version upgrade path is supported, then you can upgrade it to the new
patch version.
- Verify
that the upgrade has completed by using the following command:
$ sqlplus / as sysasm
SQL> SELECT banner_full FROM v$version;
# Verify CRS active version & state
$ crsctl query crs activeversion -f
$ crsctl query crs softwareversion
# Repeat on remaing
Nodes.
Oracle Clusterware
active version on the cluster is [23.0.0.0.0]. Cluster upgrade state: NORMAL.
The cluster upgrade
state is [FORCED]. The cluster active patch level is [0].
The active release
should be the upgraded release.
The upgrade state FORCED indicates that Oracle Clusterware
upgrade was completed forcibly on all reachable nodes, while one or more
cluster nodes were unreachable. You can either upgrade or delete
the unreachable cluster nodes to change the cluster upgrade state from FORCED to NORMAL.
Root-user automation script
The following is a root-run orchestration wrapper for Oracle Grid Infrastructure 26ai. It performs basic checks and launches the installer as grid, with Oracle’s root-script automation options enabled.
Save as /root/run_gi_upgrade.sh:
#!/bin/bash
set -Eeuo pipefail
# Run this wrapper as root.
if [[ $EUID -ne 0 ]]; then
echo "ERROR: Run this script as root."
exit 1
fi
# Adjust these paths for your environment.
GRID_USER="grid"
GRID_HOME="/u01/app/26.0.0.0/grid"
ORACLE_BASE="/u01/app/grid"
BATCH_MAP="/u01/app/26.0.0.0/grid/batch_map"
LOG_DIR="/var/log/gi_upgrade"
LOG_FILE="${LOG_DIR}/gridSetup_upgrade_$(date +%Y%m%d_%H%M%S).log"
mkdir -p "$LOG_DIR"
chmod 750 "$LOG_DIR"
# Validate required files.
for file in "${GRID_HOME}/gridSetup.sh" "$BATCH_MAP"; do
if [[ ! -r "$file" ]]; then
echo "ERROR: Required file is missing or unreadable: $file"
exit 1
fi
done
# Ensure the Grid owner exists.
id "$GRID_USER" >/dev/null 2>&1 || {
echo "ERROR: User $GRID_USER does not exist."
exit 1
}
echo "Starting GI upgrade preparation at $(date)"
echo "Grid home: $GRID_HOME"
echo "Batch map: $BATCH_MAP"
echo "Log file: $LOG_FILE"
# Run installer as the Grid Infrastructure owner.
# Oracle root-script automation is enabled through the documented flags.
runuser -u "$GRID_USER" -- \
"$GRID_HOME/gridSetup.sh" \
-upgrade \
-ORACLE_BASE "$ORACLE_BASE" \
-batchInfo "$BATCH_MAP" \
-executePrereqs \
-executeRootScript \
-configMethod ROOT \
2>&1 | tee -a "$LOG_FILE"
rc=${PIPESTATUS[0]}
if [[ $rc -ne 0 ]]; then
echo "ERROR: gridSetup.sh failed with exit code $rc"
exit "$rc"
fi
echo "gridSetup.sh completed successfully at $(date)"
Make it executable and run it:
chmod 750 /root/run_gi_upgrade.sh
/root/run_gi_upgrade.sh
Oracle documents -executeRootScript -configMethod ROOT as supported upgrade options.
References:
************ Alireza Kamrani ************
No comments:
Post a Comment