Oracle RAC AI: 12
things I would check before calling a cluster “stable”
I recently
revisited Oracle’s classic recommendations for stabilizing RAC environments.
The
interesting part is that the fundamentals have not changed much.
RAC can provide excellent availability
and scalability, but RAC itself does not make the whole
platform highly available! The network, storage, OS, virtualization layer,
application connectivity and operational procedures still matter.
For an
Oracle Database AI RAC environment, these are the 12 areas DBAs would review
first:
1- Start
with the complete architecture
Don't look
only at the database.
Check the
entire path:
Application
→ DNS/SCAN → VIP/Listener → RAC Service → Database → ASM → Storage → Network
A failure
anywhere in this chain can look like a “database outage”.
Oracle also
emphasizes that true high availability requires the surrounding infrastructure
to be designed without single points of failure.
2- Keep
Grid Infrastructure simple
Oracle Grid
Infrastructure provides the foundation for RAC through Oracle Clusterware +
ASM.
Avoid
introducing additional clustering layers unless there is a real architectural
requirement.
Fewer moving
parts generally means fewer places to troubleshoot when something goes wrong.
3- Treat
the private interconnect as mission-critical
The RAC
interconnect is not just another network.
Latency,
packet loss, congestion, NIC failures and network configuration problems can
directly affect RAC performance and stability.
For modern
RAC deployments, I would specifically validate:
• NIC
redundancy
• Network bandwidth
• MTU consistency
• switch redundancy
• packet loss
• latency/jitter
• NIC errors/drops
• OS and switch configuration
Oracle's
current documentation recommends redundant interconnect usage for RAC
environments.
4-
Eliminate storage single points of failure
ASM
simplifies shared storage management, but ASM cannot compensate for a single
failed storage path.
Validate:
• Multiple
HBAs / initiators
• Multipathing
• Redundant storage paths
• Storage-controller redundancy
• Consistent performance across paths
• ASM disk discovery
• Storage latency during peak load
The question
shouldn't simply be:
“Is ASM healthy?”
It should
be:
“Can I lose a storage
path, controller or component without taking the database down?”
5- Don't
underestimate SCAN, VIP and Services
Client
connectivity is part of RAC HA.
Review:
• SCAN
configuration and DNS
• VIPs
• Listeners
• Remote listener configuration
• Database Services
• Connection load balancing
• Runtime Load Balancing
• FAN / Application Continuity where applicable
A RAC
database can be completely healthy while applications are unable to connect
because the connectivity layer is broken.
6- Design
Services around workloads
One of the
mistakes I still see is using the default database service for everything.
OLTP,
reporting, batch jobs and administration don't necessarily have the same
requirements.
Create
services based on application workloads and use RAC workload-management
capabilities to control where and how workloads run.
Oracle's RAC
documentation specifically describes services as a mechanism for grouping
applications with common attributes, priorities and service-level requirements.
Also using Load
balancing options in your Services design using work load and Application
characteristics:
Oracle Real
Application Clusters (Oracle RAC) offers two options for load balancing:
client-side load balancing and server-side load balancing. Sessions connect to
an Oracle RAC instance using Single Client Access Network (SCAN) and a
user-defined service name. You can configure a service to run on all or a
subset of Oracle RAC instances. By default, SCAN redirects the sessions to the
local listener. The SCAN listener directs a connection request to the best
instance that currently hosts the service. The redirection is based on the -clbgoal and -rlbgoal settings
for the service.
Smart
Connection Rebalance minimizes
resource conflicts. It ensures that workloads accessing similar objects are
directed to a minimal set of instances. These workloads benefit from reduced
inter-instance network messages and reduced data block transfers over the
private network. This feature ensures optimal load balancing and performance.
This feature
performs real-time monitoring of different workloads and attempts to
transparently relocate service-based connections across Oracle RAC instances.
This relocation significantly improves database performance.
The Smart
Connection Rebalance Recommendation engine optimizes workload distribution and
enhances performance in Oracle Real Application Clusters (Oracle RAC)
environments. The engine continuously analyzes the activity of each database
service. It assesses whether enabling the Smart Connection Rebalance feature at
the service level can deliver measurable improvements. These improvements
include better throughput and response times.
7-
Capacity planning is more than CPU and RAM
Adding RAC
nodes doesn't automatically mean linear performance improvement.
You need to
consider:
CPU
Memory
I/O
Interconnect bandwidth
Global Cache traffic
Storage latency
Connection count
Workload distribution
RAC synchronization overhead
A cluster
with plenty of CPU can still have poor performance because of interconnect or
storage contention.
8-
Validate patching and upgrade readiness
Don't wait
for a production incident to discover that your GI/RDBMS combination has a
known issue.
Before
patching:
• Validate
the current cluster health
• Check compatibility
• Review known issues
• Verify free space
• Test the procedure
• Have a rollback/recovery plan
• Prefer rolling approaches when supported
And keep in
mind that Oracle RAC AI has newer HA capabilities, including two-stage
rolling updates for eligible patches.
9- Make
diagnostics part of the architecture
When a RAC
node suddenly reboots at 03:00, historical evidence becomes extremely valuable.
I would
proactively collect and retain information about:
• OS
CPU/memory pressure
• Network errors and latency
• Storage latency
• Clusterware events
• Node evictions
• Database alerts
• ASM events
• AWR/ASH information
• RAC wait events
Oracle's
Autonomous Health Framework (AHF) and diagnostic tooling should be part
of the operational model, not something installed after the first major
incident.
10- Test failures, don't just document
them
A HA
architecture that has never been tested is an assumption.
Test
scenarios such as:
• RAC node
failure
• Public NIC failure
• Interconnect failure
• Storage path failure
• Listener failure
• SCAN/DNS issues
• Instance crash
• Storage latency
• Network congestion
• Planned maintenance
The
objective isn't simply proving that RAC survives.
The
objective is measuring:
Detection
time → Failover time → Application recovery → User impact
11-
Remember that RAC is only one layer of MAA
RAC protects
availability inside the database cluster.
It is not a
complete disaster-recovery strategy.
For serious
mission-critical environments, RAC should be considered as one component of a
broader Oracle Maximum Availability Architecture (MAA) strategy,
potentially together with Data Guard / Active Data Guard, Application
Continuity, backup/recovery and other HA capabilities.
Oracle
currently positions RAC as part of its broader MAA architecture.
https://www.oracle.com/database/technologies/maximum-availability-architecture
Doc ID: Document
1344678.1
12- Anti-Virus
Software
- Be sure to disable Anti-Virus
even before downloading the installation zip files or any Oracle patch.
- Disable Anti-Virus software
running on cluster nodes before and for the entire duration of the
installation or patching of Oracle. (be mindful of manual reboots during
this time).
·
Anti-Virus software may of course be re-enabled
following the installation or patching, but the following should be excluded
from being scanned:
- Oracle Software
directories (including Oracle Home(s) and the Oracle Base)
- OCFS filesystems
- ACFS filesystems
- Network scanning of the private interconnect
My
practical RAC rule:
Don't ask:
❌ “Is my RAC database up?”
Ask:
✅ “Can I lose a server, NIC, network path,
storage path, instance or database service, and what exactly happens to the
application?”
Ø That is a much
better definition of RAC stability.
Oracle RAC AI
documentation:
Oracle Real Application Clusters Administration and Deployment Guide
Oracle
Real Application Clusters — official Oracle page
https://www.oracle.com/database/real-application-clusters
#Oracle
#OracleDatabase #OracleRAC #OracleAI #GridInfrastructure #ASM #HighAvailability
#Database #DBA #MAA #DataGuard
