The Evolution of Zero-Downtime Upgrades:
From Transient Logical Standby
To
DBMS_ROLLING & FPP
Database upgrades are a necessary part
of maintaining a secure, performant, and feature-rich IT infrastructure.
However, the process is often fraught with the challenge of minimizing or
eliminating downtime. For Oracle Database administrators, the solution has long
resided in the powerful capabilities of Oracle Data Guard and the concept of a Rolling
Upgrade, Autoupgrade.jar or FPP server.
A rolling upgrade allows a database to
be upgraded or patched while the application remains connected and operational,
leveraging a standby database to absorb the downtime. The historical path of
this enhancement, specifically using a Transient Logical Standby (TLS), is a
fascinating case study in Oracle's commitment to automation and simplification,
evolving from complex manual scripts to a fully automated PL/SQL package.
The Basic solutions to upgrade database are:
Ø Oracle AutoUpgrade Tool
Ø Transportable tablespaces (TTS) export
and import, using the Oracle Database feature full transportable export/import,
or the traditional TTS mode
Ø Oracle Data Pump Export/Import, using
either dump files or network mode with database links
Ø The Original Export/Import utilities
Ø To learn more about the above
solutions, you can refer to: https://www.oracle.com/europe/a/ocom/docs/upgrade-migrate-tech-brief.pdf
But in this document,
I will focus on others ways as a historical
tool for Upgrade of Oracle Database.
The foundation of the zero-downtime
upgrade path lies in the ability to temporarily convert a physical standby
database into a logical standby. This conversion is the critical step that
allows the standby to run a different version or patch level of the Oracle
software than the primary database, a state known as Mixed Version Support.
The core, multi-step process involved:
Ø 1.Stopping Redo Apply on the physical
standby.
Ø 2.Converting the Physical Standby to
Logical Standby using the ALTER DATABASE ACTIVATE LOGICAL STANDBY DATABASE
command, often with the crucial KEEP IDENTITY clause. The KEEP IDENTITY clause
ensures the database maintains its unique identity, allowing it to participate
in a single, one-time switchover to become the new primary.
Ø 3.Upgrading the Logical Standby to the
target version or patch set.
Ø 4.Synchronizing the logical standby
with the primary.
Ø 5.Performing a Switchover to make the
upgraded logical standby the new primary.
Ø 6.Upgrading the Old Primary (now the
new standby) to complete the process.
Transient Logical Rolling Upgrade Process
The Role of ./physru
To manage the complexity and potential
for human error in this manual sequence, Oracle introduced the physru (Physical
Standby Rolling Upgrade) shell script. This script, typically found in the
$ORACLE_HOME/bin directory, was designed to automate the conversion and
switchover steps.
While a significant step forward, the
physru method was still prescriptive and required careful execution. As noted
in Oracle documentation of the time, the script often needed to be called
multiple times to manage the different stages of the rolling upgrade process,
making it an automated but still intricate procedure. It was a tool that
required a deep understanding of the underlying Data Guard mechanics.
The transient logical process gets its
name from the fact that it begins and ends with a physical standby database
while temporarily being converted to a logical standby database for the
upgrade. The process can also be used to reduce downtime for other types of
planned maintenance, such as partitioning a table, or encrypting data with
Transparent Data Encryption, etc.
A script provided by Oracle named physru
automates many of the manual steps along with additional levels of validation
that occur automatically throughout the process.
The physru script requires six
parameters:
The script prompts for the SYSDBA password at
the beginning of each execution of the script. You can execute the script from
any Unix/Linux node as long as that node has access to SQL*Plus and SQL*Net
connectivity to the databases involved in the rolling upgrade.
More info:
https://www.oracle.com/technetwork/database/availability/database-rolling-upgrade-3206539.pdf
Phase 2: Formalizing the Transient Logical Standby (TLS)
The concept of the Transient Logical
Standby became the formal name for this upgrade methodology. The key was the
temporary nature of the logical standby role, which was only needed to
facilitate the upgrade before the database was switched over and potentially
converted back to a physical standby (or upgraded to the new primary).
This era solidified the use of the
KEEP IDENTITY clause, which was essential for maintaining the database's unique
ID (DBID) and ensuring a smooth, one-time transition. This made the TLS a
reliable path for both major version upgrades and applying large patch sets,
such as those that required a full database catalog upgrade.
Despite the formalization, the process
remained a multi-step, command-line driven procedure. It demanded meticulous
attention to detail, including managing initialization parameters, ensuring all
prerequisites were met, and manually monitoring the synchronization process.
The complexity meant that while downtime was reduced, the risk of procedural
error remained high.
More info:
Phase 3: The Modern Era - Automation with DBMS_ROLLING
(Oracle 12cR2 and beyond)
This is an Innovated and latest enhancements
for Transient Logical Standby (TLS) method that need Standby and Primary to changing
roles in an automated instruction Package.
The most significant enhancement to
the rolling upgrade process came with the introduction of the DBMS_ROLLING
PL/SQL package, starting in Oracle Database 12c Release 2 and becoming the
standard in 18c and 19c. This package represents a paradigm shift from
script-based automation to native, procedural automation.
The DBMS_ROLLING package abstracts
away the intricate manual steps of conversion, upgrade, and switchover into a
few simple, high-level procedure calls. It manages the entire lifecycle of the
rolling upgrade, significantly reducing the operational complexity and the
potential for human error.
The process is streamlined into three
primary stages:
|
Procedure |
Description |
|
DBMS_ROLLING.START_ROLLING_OPERATION |
Prepares
the Data Guard configuration for the rolling upgrade, automatically handling
the conversion to a transient logical standby. |
|
DBMS_ROLLING.SWITCHOVER |
Executes
the switchover to make the upgraded standby the new primary, with minimal
interruption. |
|
DBMS_ROLLING.FINISH_ROLLING_OPERATION |
Cleans
up the configuration and converts the old primary (now the standby) back to a
physical standby, completing the upgrade cycle. |
This package is often used in
conjunction with Active Data Guard and is integrated into tools like Oracle
Enterprise Manager Cloud Control, providing a graphical, guided experience. The
shift to DBMS_ROLLING aligns with the modern preference for reduced hard
parsing—in this context, reducing the need for manual, complex command-line
parsing and execution in favor of a simple, robust API.
NOTE: The Transient
Logical Standby is Deprecated
in 26ai.
Command-line Upgrade,
(catctl.pl/dbupgrade) and the Database Upgrade Assistant (DBUA) are deprecated. Oracle
recommends using AutoUpgrade or FPP.
More info:
Phase 4: The Next Generation - AutoUpgrade Utility
(Oracle 18c & beyond)
While DBMS_ROLLING revolutionized the rolling
aspect of the upgrade, the Oracle Database AutoUpgrade Utility (autoupgrade.jar)
introduced in Oracle 18c and matured in 19c, revolutionized the upgrade aspect
itself. AutoUpgrade is a command-line tool designed to automate and simplify
the entire upgrade process, whether it's a single database or a Data Guard
configuration.
The AutoUpgrade utility identifies
issues before upgrades, performs pre- and postupgrade actions, deploys
upgrades, performs postupgrade actions, and starts the upgraded Oracle
Database.
Oracle recommends that you download
the most recent version of the AutoUpgrade Utility from My Oracle Support
Document 2485457.1, and use autoupgrade.jar to prepare for and to
deploy your upgrade. The AutoUpgrade utility is designed to automate the
upgrade process, both before starting upgrades, during upgrade deployments, and
during postupgrade checks and configuration migration. You use AutoUpgrade
after you have downloaded binaries for the new Oracle Database release, and set
up new release Oracle homes. When you use AutoUpgrade, you can upgrade multiple
Oracle Database deployments at the same time, using a single configuration
file, customized as needed for each database deployment. Starting with Oracle
Database 21c, when you have an existing target release CDB, you can use
AutoUpgrade to convert a non-CDB Oracle Database to a PDB on the target release
CDB during the upgrade.
With AutoUpgrade, when you have an
existing target release CDB, you can use AutoUpgrade to convert a non-CDB
Oracle Database to a PDB on the target release CDB during the upgrade.
Starting with AutoUpgrade version
22.1, Oracle provides REST APIs that enable you to perform upgrades remotely
over SSH using Oracle REST Data Services (ORDS) or Oracle Cloud Infrastructure
(OCI) REST API. The ORDS database API is a database management and monitoring
REST API embedded into Oracle REST Data Services. The OCI REST API is enabled
by configuring the REST Adapter connection to use the OCI Signature Version 1
security policy. You can now use these features to run AutoUpgrade upgrades
remotely over HTTPS or HTTP.
Caution:
Before you run AutoUpgrade to complete
the conversion and upgrade. Oracle strongly recommends that you create a full
backup of your source database, and complete thorough testing of the upgrade.
There is no option to roll back to the non-CDB Oracle Database state after
AutoUpgrade starts this procedure.
More info:
AutoUpgrade
Utility System Checks
The utility's most powerful feature is
its comprehensive set of system checks performed in the analyze mode. These
checks are crucial for identifying and resolving potential issues before the
upgrade begins, drastically reducing the risk of failure and unexpected
downtime. Key checks include:
Ø Pre-Upgrade Readiness: Verifying that
the database is in a state ready for upgrade, including checking for
unsupported features, invalid objects, and dictionary conflicts.
Ø Parameter Validation: Reviewing
initialization parameters for compatibility with the target Oracle version and
suggesting optimal values.
Ø Component Status: Ensuring all
database components (e.g., Oracle Text, Spatial) are valid and properly
configured.
Ø Time Zone File Version: Checking and
updating the time zone file to the latest version, a common source of
post-upgrade issues.
Ø Data Guard Integration: Automatically
detecting Data Guard configurations and ensuring the upgrade plan adheres to
best practices for standby databases.
By performing these checks,
AutoUpgrade shifts the focus from reactive troubleshooting to proactive
preparation, making the upgrade process more predictable and reliable.
Phase 5: Enterprise-Scale Management:
ZDU (Zero-Downtime Upgrade) - Fleet Patching and
Provisioning (FPP)
The final and most advanced step in
this historical progression is Fleet Patching and Provisioning (FPP). FPP is a
centralized management framework designed for large enterprises with hundreds
of databases, often running on engineered systems like Exadata.
The Oracle Fleet Patching and
Provisioning (Oracle FPP) Server is a highly available software provisioning
system that uses Oracle Automatic Storage Management (Oracle ASM), Oracle
Advanced Cluster File System (Oracle ACFS), application VIPs, and other components.
Using Oracle Fleet Patching and
Provisioning, which automates and orchestrates database upgrades, you can
upgrade an Oracle RAC or Oracle RAC One Node database with no disruption in
service.
Note:
Ø The Zero Downtime Upgrade (ZDU)
feature of Oracle Fleet Patching and Provisioning (FPP) is deprecated
in Oracle AI Database.
ZDU uses either Transient Logical Standby or
Oracle GoldenGate classic. However, Transient Logical Standby does not
support the latest data types, and the Oracle GoldenGate classic
implementation is deprecated.
Ø Instead of ZDU, Oracle recommends that
you use FPP in combination with the DBMS_ROLLING PL/SQL package:
You can use FPP for the provisioning and
deployment of new Oracle homes, and you can use DBMS_ROLLING as a
streamlined method of performing rolling upgrades.
The DBMS_ROLLING package is
closely integrated with Application Continuity to help to minimize disruptions
in the database tier, and mask rolling patching activities.
Oracle provide an enhanced solution called by ZDU
(that is a powerful and zero downtime migration solution that works based on Oracle
Fleet Patching and Provisioning (FPP)) that can follow by two routines:
You can find more info about these:
Ø Zero-Downtime Upgrade
https://docs.oracle.com/en/database/oracle/oracle-database/26/fppad/zero-downtime-upgrade.html
Ø Running a Zero-Downtime Upgrade Using
Oracle GoldenGate for Replication
Ø Running a Zero-Downtime Upgrade Using
Oracle Data Guard for Replication
Deploying Oracle
software using Oracle FPP has the following advantages:
- Enables you to create and manage Oracle
Real Application Clusters (Oracle RAC), single instance, and Oracle Real
Application Clusters One Node (Oracle RAC One Node) databases. You can
deploy single-instance databases with or without Oracle Grid Infrastructure.
- Ensures standardization and enables high
degrees of automation with gold images and managed lineage of deployed
software.
- Supports change management. With
standardized Oracle homes, an administrator has better control of the
hosted Oracle software and can easily manage the mass deployment,
patching, and upgrade of the software through a single location for change
management.
- Minimizes downtime during patching and
upgrades, eases rollbacks, and makes provisioning for large systems easier
and more efficient.
- Simplifies maintenance and patching of
database software.
- Reduces the cumulative time to patch
software images, since a single Oracle home may be used for many database
instances.
The zero-downtime upgrade process is resumable,
restartable, and recoverable should any errors interrupt the process. You
can fix the issue then re-run the command, and Oracle Fleet Patching and
Provisioning continues from the error point. Oracle also provides hooks at the
beginning and end of the zero-downtime upgrade process, allowing call outs to
user-defined scripts, so you can customize the process.
You can use the zero-downtime upgrade
process to upgrade databases that meet the following criteria:
- Oracle AI Database release: Oracle RAC and Oracle RAC One Node,
with the following upgrade paths:
- 11.2.0.4 to: 12.1.0.2, 12.2, 18c
- 12.1.0.2 to: 12.2, 18c, 19c, 21c
- 12.2 to: 18c, 19c, 21c
- 18c to: 18c, 19c, 21c
- 19c to: 21c, 26ai
- 21c to: 26ai
- Oracle Fleet Patching and Provisioning
management: The source
database home can either be unmanaged (not provisioned by Oracle Fleet
Patching and Provisioning service) or managed (provisioned by Oracle Fleet
Patching and Provisioning service)
- Database state: The source database must be in archive
log mode.
FPP is not an upgrade tool itself, but
a deployment and orchestration layer that manages the entire lifecycle of
Oracle/Grid Homes, databases and data guard also. It allows administrators to:
Ø Standardize: Define a "Gold Image" (a fully
patched and configured Oracle Home).
Ø Scale: Deploy the Gold Image and orchestrate the
upgrade across an entire fleet of databases with a single, high-level command.
Ø Integrate: FPP implicitly leverages the power of
AutoUpgrade and DBMS_ROLLING to perform the actual upgrade and rolling
switchover steps in a highly controlled and repeatable manner.
For modern, high-availability
environments like the Oracle AI Database, FPP represents the pinnacle of
operational efficiency, ensuring that upgrades are not just zero-downtime, but
also standardized and scalable across the entire organization.
Ø Oracle recommends deploying the Oracle
FPP Server in a multi-node cluster so that it is highly available. Oracle FPP
Server supports single-node deployment, but it is not recommended.
Note:
The Oracle FPP Server needs an Oracle
Clusterware installation, thus you cannot configure Oracle FPP Server on an
Oracle Restart server.
Ø The Oracle FPP Server cluster is a
repository for all data, of which there are primarily two types:
·
Gold
images
·
Metadata
related to users, roles, permissions, and identities
The Oracle FPP Server acts as a
central server for provisioning Oracle AI Database homes, Oracle Grid
Infrastructure homes, and other application software homes, making them
available to the cluster hosting the Oracle FPP Server and to the Oracle FPP
Client clusters, and rhpclient-less targets.
Users operate on the Oracle FPP Server
or Oracle FPP Client to request deployment of Oracle homes or to query gold
images. When a user makes a request for an Oracle home, specifying a gold
image, the Oracle FPP Client communicates with the Oracle FPP Server to pass on
the request. The Oracle FPP Server processes the request by taking appropriate
action to instantiate a copy of the gold image, and to make it available to the
Oracle FPP Client cluster using available technologies such as Oracle Advanced
Cluster File System (Oracle ACFS) and local file systems.
Ø Starting with Oracle Grid
Infrastructure 26ai, Oracle FPP enables you to automatically patch Oracle Data
Guard standby databases before patching their primary databases. Also, we can upgrade
Oracle and Grid Simultaneously.
Ø This procedure patches Oracle Grid
Infrastructure and Oracle AI Databases on the cluster to the latest patch level
without cluster downtime.
Oracle Fleet Patching and Provisioning
(Oracle FPP) automatically discovers the standby databases, associated with
primary databases, and patches the standby databases before patching the
primary database using a single command.
Note:
If the Oracle Database Guard databases
are registered on a different client, which can be in different data center,
then those servers must be register peers of each other.
See all features for
FPP in:
https://docs.oracle.com/en/database/oracle/oracle-database/26/fppad/fleet-patching-provisioning.html
Historical
Enhancement Path: A Summary
The evolution of Oracle's upgrade
solutions is a clear narrative of increasing automation, simplification, and
scalability.
|
Era |
Solution |
Primary Focus |
Automation
Level |
Key Enhancement |
|
Pre-10g |
Manual TLS |
Establishing Mixed Version Support |
Low |
Core concept of rolling upgrade. |
|
10g/11g |
./physru Script |
Scripting the TLS process |
Medium |
Automated conversion and switchover. |
|
12cR2+ |
DBMS_ROLLING |
Procedural Zero-Downtime Mechanism |
High |
Native PL/SQL API for rolling upgrade. |
|
18c+ |
AutoUpgrade Utility |
Upgrade Workflow Orchestration &
Pre-Checks |
Very High |
Proactive validation and end-to-end
management. |
|
12cR2+ |
Fleet Patching & Provisioning (FPP) |
Enterprise-Scale Deployment &
Standardization |
Ultimate |
Centralized management of the entire fleet. |
Historical
Enhancement Path: A Comparison
The evolution of the rolling upgrade
process demonstrates a clear trend toward simplification and automation,
directly translating to reduced risk and lower operational costs.
|
Feature |
Era |
Method |
Complexity |
Automation
Level |
Key
Enhancement |
|
Manual
TLS |
Pre-10g |
SQL
commands |
High |
Low |
Established
the core concept of mixed version support. |
|
physru Script |
10g/11g |
Shell Script |
Medium |
Medium |
Automated the multi-step conversion and
switchover process. |
|
DBMS_ROLLING |
12cR2+ |
PL/SQL
Package |
Low |
High |
Fully
automated the entire lifecycle, reducing human error and operational
complexity. |
In-Depth
Scenario Analysis: When to Choose Which Method
The choice of upgrade method depends
entirely on the business requirement for downtime, the size of the environment,
and the existing infrastructure. The following scenarios illustrate the modern
best practice for Oracle upgrades, focusing on AutoUpgrade and RHP/FPP.
|
Scenario |
Business Requirement |
Best Choice |
Reasoning |
|
Scenario
1: Mission-Critical Database |
24/7
availability; Data Guard environment; near-zero downtime required. |
AutoUpgrade
orchestrating DBMS_ROLLING |
This is
the gold standard for zero-downtime. AutoUpgrade (autoupgrade.jar)
manages the entire workflow, including pre-checks and post-checks, and then
calls the DBMS_ROLLING package to execute the transient logical
standby conversion and the final, near-instantaneous switchover. |
|
Scenario
2: Scheduled Maintenance Window |
Corporate
Data Warehouse; acceptable 8-hour weekend maintenance window. |
AutoUpgrade
Utility (autoupgrade.jar) |
The
modern best practice for any single database upgrade, even with a maintenance
window. AutoUpgrade provides superior pre-upgrade analysis, automated
fix-ups, and a highly reliable, repeatable command-line process. It is the
successor to DBUA and manual scripts, offering a much safer and more
efficient path. |
|
Scenario
3: Fleet-Wide
Standardization |
Cloud
Provider or Large Enterprise; hundreds of databases; need for consistent,
repeatable, and automated patching/upgrades. |
RHP
(Rapid Home Provisioning) or FPP (Fleet Patching and Provisioning) |
For
massive scale, the focus shifts to infrastructure management. RHP/FPP
is the orchestration layer that manages "Gold Images" of Oracle/Grid
Homes. It deploys the new Oracle Home and then uses AutoUpgrade (which
in turn can use DBMS_ROLLING for Data Guard pairs) to execute the upgrade
across the entire fleet consistently and automatically. This tool provides to
Upgrade Grid and oracle Simultaneously. |
Clarification on
RHP/FPP:
RHP (Rapid Home
Provisioning) is the
underlying technology that enables FPP (Fleet Patching and Provisioning).
It is the most advanced solution for managing the Oracle software lifecycle,
ensuring that all databases are running on standardized, pre-validated Oracle
Homes. It represents the final step in the historical evolution, moving beyond
just upgrading a single database to managing the entire database fleet.
Conclusion:
The journey from the multi-step,
script-driven physru method to the streamlined, procedural DBMS_ROLLING
package, and finally to the comprehensive management of the AutoUpgrade
utility, is a testament to the continuous enhancement of Oracle Data Guard.
What was once a high-stakes, complex operation requiring expert-level knowledge
has been transformed into a robust, automated feature.
For DBAs today, understanding this
history provides context for the power of the modern toolset. It highlights the
significant effort Oracle has invested in making zero-downtime upgrades
accessible, reliable, and a standard practice for maintaining mission-critical
systems.
The combination of AutoUpgrade and DBMS_ROLLING
is the culmination of years of refinement, ensuring that database upgrades can
be performed with confidence and minimal impact on the business.
- DBMS_ROLLING is not a replacement for dbupgrade;
it's a high-availability strategy that uses an
upgrade method internally. It is a specialized tool for minimizing
downtime by leveraging Data Guard.
- dbupgrade is the core, non-interactive engine
that performs the actual in-place upgrade of the data dictionary.
- DBUA is a user-friendly GUI wrapper around
the dbupgrade engine, designed to make the process safer and
easier for interactive, one-off upgrades.
Choosing the right method is about
balancing the business need for availability against the technical complexity
and resources required for the upgrade.
The journey from the multi-step,
script-driven physru method to the streamlined, procedural DBMS_ROLLING package
is a testament to the continuous enhancement of Oracle Data Guard. What was
once a high-stakes, complex operation requiring expert-level knowledge has been
transformed into a robust, automated feature.
For DBAs today, understanding this
history provides context for the power of DBMS_ROLLING. It highlights the
significant effort Oracle has invested in making zero-downtime upgrades
accessible, reliable, and a standard practice for maintaining mission-critical
systems. The modern DBMS_ROLLING package is the culmination of years of
refinement, ensuring that database upgrades can be performed with confidence
and minimal impact on the business.
References:
[3] Oracle Documentation.
DBMS_ROLLING. (Detailing the modern PL/SQL package for rolling upgrades).
[8] My
Oracle Support AutoUpgrade Tool (Doc ID 2485457.1)
[9] Upgrade your Database – NOW!
Mike Dietrich's Oracle Database Upgrade Blog
[10] Database
Preupgrade tool (via autoupgrade.jar) check list (Doc ID 2380601.1)
