Oracle database Maximum Performance on Exadata

 ⛳️        Oracle database          

       Maximum performance

                   on Exadata.

               

A total review of features

Hello everyone, 

My name is Alireza Kamrani and I have collected some useful informations about abilities of implementing Oracle database on Exadata and explaining its features to achieve maximum performance, I am sharing this article with you in this post and I hope this will be useful for you.

⛳️Overview of Exadata Machine:

Oracle Exadata Database Machine (Exadata) is an engineered system that delivers the ideal platform for running Oracle Database workloads at an optimal cost. Its scale-out architecture integrates the latest processors, intelligent storage servers, cutting-edge caching technologies, and advanced networking features. Customers can harness unique hardware and software optimizations of Exadata by running Oracle Autonomous Database and Oracle Exadata Database Service on dedicated Exadata Cloud Infrastructure on Oracle Cloud Infrastructure (OCI) and in Microsoft Azure data centers through the Oracle Database@Azure offering. Additionally, through Oracle Exadata Cloud@Customer, which uses the same architecture and infrastructure as Oracle Exadata Database Machine, customers can bring the performance and economics of Oracle Exadata Database Service and the fully managed Oracle Autonomous Database into their data centers, thereby assisting in meeting data residency or application architecture requirements. Lastly, both the Zero Data Loss Recovery Appliance and Zero Data Loss Autonomous Recovery Service utilize Exadata hardware and software; they simplify Oracle Database backup processes, accelerate database recovery, and increase protection against cyberattacks. 

Here I represent some new features on Exadata:

🚩Persistent Storage Index

Storage indexes help Oracle Exadata System Softwareto avoid unnecessary I/O operations. By avoiding unnecessary I/O, affected operations perform better. Also, overall system throughput increases because the saved I/O bandwidth can be used to perform other work.

Oracle Exadata System Software creates and maintains a storage index in shared memory on Exadata Storage Server. For each storage region, the storage index automatically tracks minimum and maximum column values and set membership. Storage index functionality works transparently with no user input. There is no need to create, drop, or tune the storage index.

Starting with Oracle Exadata System Software release21.2.0, the storage index can persist across cell restarts and upgrades. When the feature is enabled, the storage index shared memory segments are written to the storage server system disks during a graceful shutdown, and these are automatically restored during restart. In the event of a cell server crash or offload server crash, the storage index shared memory segments are retained by the operating system and are immediately available when the software service restarts.

🚩Persistent Columnar Cache

The columnar cache is a portion of Exadata Smart Flash Cache that contains data in columnar format. Data in Exadata Smart Flash Cache is automatically converted to columnar format and stored in the columnar cache to aid query performance.

Before Oracle Exadata System Software release 21.2.0, metadata for managing the columnar cache is lost when the cell fails or shuts down. Consequently, the columnar cache must be rebuilt following each cell restart.

Starting with Oracle Exadata System Software release21.2.0, the columnar cache can persist across cell restarts and upgrades. This feature saves resources associated with rebuilding the cache and enables more consistent query performance immediately after a cell restart.

🚩IORM Cluster Plan

Oracle Exadata System Software release 21.2.0extends I/O Resource Management (IORM) to include the cluster plan. The IORM cluster plan specifies how storage server resources are allocated across multiple Oracle Grid Infrastructure clusters. Each directive in a cluster plan specifies an allocation to an Oracle Grid Infrastructure cluster, rather than an individual database or consumer group.

For example, consider a system hosting two clusters;clusterA and clusterB. Now, imagine a cluster plan with share-based resource allocation where clusterAhas three shares, and clusterB has one share. In that case, and in the absence of any other IORM plans, all of the databases running in clusterA share 75% of the I/O resources, while the databases in clusterBshare the remaining 25%.

🚩Options to Preserve Redundancy while Dropping a Physical Disk

Oracle Exadata System Software release 21.2.0introduces the MAINTAIN REDUNDANCY and NOWAIToptions to the CELLCLI ALTER PHYSICALDISK DROP FOR REPLACEMENT command. 

The MAINTAIN REDUNDANCY option streamlines Exadata Storage Server administration by automatically rebalancing data before dropping the corresponding ASM disks. In previous releases, maintaining redundancy required additional administrator intervention to manually drop the corresponding ASM disks and rebalance the data prior to using the ALTER PHYSICALDISK DROP FOR REPLACEMENT command.

The NOWAIT option allows the ALTER PHYSICALDISK command to complete immediately while the DROP FOR REPLACEMENT operation runs asynchronously in the background.

🚩Smart Scan Metadata Sharing

During Smart Scan operations, Oracle Database sends query metadata to the Exadata Storage Servers. The query metadata contains information like predicated columns, projected columns, predicates, database session information, bloom filters, and so on, which is required to offload the query to the storage servers. Historically, during parallel query execution, each Oracle Database parallel query server process sends serialized and compressed query metadata to the storage server, where it is uncompressed and deserialized before use. Furthermore, for each parallel query, a significant portion of the metadata is common to all of the parallel query server processes.

🚩Faster Upgrades with ILOM Pre-Staging

The upgrade process for Oracle Exadata System Software typically includes a firmware upgrade for Integrated Lights Out Manager (ILOM). Historically, the ILOM firmware upgrade was performed as a serial process, taking approximately 30% of the total software upgrade time on each physical server (database server or storage server).

🚩Automatic Recovery from Disk Controller Cache Failure

High Capacity (HC) and Extended (XT) Oracle Exadata Storage Server models have a disk controller that contains a write-back cache, which is separate and distinct from the write-back flash cache.

A multiple-bit ECC error on the disk controller cache can crash the server, resulting in the loss of unflushed data in the disk controller cache and leaving stale data on the disks. If not handled carefully, such errors are severe and can result in data loss.

Oracle Exadata System Software release 21.2.0supports automatic recovery following a disk controller cache failure. Where possible, this feature automatically detects the problem and copies data from other mirrors to recover all of the disks in the failed server.

🚩Oracle ACFS I/O Caching in Flash Cache

To optimize I/O efficiency, small sequential writes to Oracle Advanced Cluster File System (Oracle ACFS) are automatically combined into 1 MB chunks. However, before Oracle Exadata System Software release 21.2.0, these larger writes would bypass the cache and go directly to disk.

Commencing with Oracle Exadata System Softwarerelease 21.2.0, all Oracle ACFS I/O is eligible for caching in Exadata Smart Flash Cache.

This feature enhances Oracle ACFS performance, especially for applications such as Oracle GoldenGatetrail files, having an I/O profile dominated by small sequential writes and fast subsequent reads.

🚩Deprecation of USB Images

Commencing with the September 2022 Oracle Exadata System Software release updates (versions 21.2.16 and 22.1.3), USB images are deprecated.

After the deprecation, USB images are no longer available for imaging or re-imaging an Oracle Exadata Database Machine server and are not included in the Secure Eraser package (secureeraser_label.zip). Instead of USB image files, use the PXE option with ISO image files.

🚩Deprecation of RAM Cache

Commencing with the September 2022 Oracle Exadata System Software release updates (versions 21.2.16 and 22.1.3), the cell RAM Cache feature is deprecated.

Existing RAM Cache installations continue to operate as they did in previous releases. However, new RAM Cache creation requests are not allowed after the deprecation.

The cell RAM Cache feature was introduced in Oracle Exadata System Software version 18.1.0

⛳️Exadata scalability features:

Scale-out architecture enables easy growth

Exadata Database Machine X10M’s scalable design lets customers size initial deployments to meet current needs and add incremental database and storage servers to support new or increasing requirements.

Scalable database processing runs demanding workloads

Customers can utilize as few as 16 licensed CPU cores on Exadata Database Machine X10M or scale up consumption to thousands of cores to meet the needs of demanding workloads. Consolidating databases on fewer systems and managing them together reduces database infrastructure and management complexity by letting more resources be managed as a single unit.

Scalable storage for large databases and data warehouses

Exadata Database Machine X10M’s high-capacity storage server has 22% more disk storage capacity than the previous generation system, supporting up to 1.2 PB of usable, triple-mirrored storage and more than 1 TB/second of analytics throughput in a single rack. Multirack scalability helps meet the database storage and analytics needs of large organizations.

🚩All-flash storage scaling accelerates latency-sensitive workloads

All-flash Exadata X10M Extreme Flash storage servers provide up to 630 TB of usable, triple-mirrored storage capacity, a 2.4X increase over previous generations. In addition, the new Extreme Flash storage servers increase Exadata Smart Flash Cache capacity to 27.2 TB, further accelerating Exadata Smart Software features.

🚩Scalability improves database consolidation

Exadata Database Machine X10M provides high compute and storage density, letting organizations consolidate their Oracle Database workloads onto smaller configurations that cost less and use less data center space.

🚩Oracle Linux advantages for Oracle Database on Exadata

Before focusing on Exadata, it's important to note that Oracle provides a choice between two Linux kernels—the Red Hat Compatible Kernel (RHCK) and Oracle Linux Unbreakable Enterprise Kernel (UEK)—and that UEK is meticulously tailored to provide high performance for Oracle Database and Exadata. Oracle's Exadata, Database, and Linux engineering teams collaborate closely to enhance the performance of UEK. This includes tuning system calls and C library interfaces and integrating cutting-edge features, all leading to extraordinary query processing times and accelerated applications. 

🚩Oracle Linux with UEK is optimized for Exadata

Exadata is exclusively powered by Oracle Linux with UEK. Exadata systems boast a distinctive feature—the operating system (OS) installed on them is a tailored, pared-down version of the standard Oracle Linux distribution with UEK. The minimized UEK, a mere 80MB in size, and the operating system includes only the necessary packages (approximately 1070 packages), device drivers, and features required to run Oracle Database on Exadata infrastructure and Exadata System Software (ESS). Ultimately, this results in a smaller installation footprint, fewer updates, and reduced potential attack surface. 

🚩Exadata System Software (ESS)

Oracle Exadata Storage Server is a highly optimized storage server that runs Oracle Exadata System Software to store and access Oracle Database data. 

With traditional storage, data is transferred to the database server for processing. In contrast, Oracle Exadata System Software provides database-aware storage services, such as the ability to offload SQL and other database processing from the database server, while remaining transparent to the SQL processing and database applications. Oracle Exadata storage servers process data at the storage level, and pass only what is needed to the database servers.

Oracle Exadata System Software is installed on both the storage servers and the database servers. Oracle Exadata System Software offloads some SQL processing from the database server to the storage servers. Oracle Exadata System Software enables function shipping between the database instance and the underlying storage, in addition to traditional data shipping. Function shipping greatly reduces the amount of data processing that must be done by the database server. Eliminating data transfers and database server workload can greatly benefit query processing operations that often become bandwidth constrained. Eliminating data transfers can also provide a significant benefit to online transaction processing (OLTP) systems that include large batch and report processing operations.

The hardware components of Oracle Exadata Storage Server are carefully chosen to match the needs of high-performance processing. The Oracle Exadata System Software is optimized to take maximum advantage of the hardware components. Each storage server delivers outstanding processing bandwidth for data stored on disk, often several times better than traditional solutions.

Oracle Exadata storage servers use state-of-the-art RDMA Network Fabric interconnections between servers and storage. Each RDMA Network Fabric link provides bandwidth of 40 Gb/s for InfiniBand Network Fabric or 100 Gb/s for RoCE Network Fabric. Additionally, the interconnection protocol uses direct data placement, also referred to as direct memory access (DMA), to ensure low CPU overhead by directly moving data from the wire to database buffers with no extra copies. The RDMA Network Fabric has the flexibility of a LAN network with the efficiency of a storage area network (SAN). With an RDMA Network Fabric network, Oracle Exadata eliminates network bottlenecks that could reduce performance. This RDMA Network Fabric network also provides a high performance cluster interconnection for Oracle Real Application Clusters (Oracle RAC) servers.

Each storage server contains persistent storage media, which may be a mixture of hard disk drives (HDDs) and flash devices. There are several configurations of storage server each configured to maximize some aspect of storage based on your requirements.

For example, Exadata takes advantage of Remote Direct Memory Access (RDMA) to facilitate direct memory access from the database servers to the storage servers. The required kernel and system packages for this compelling feature are not merely included in Oracle Linux with UEK for Exadata use; they are carefully integrated, rigorously tested, and delivered to help ensure superior performance and security. Exadata's singular focus on being the ideal platform for Oracle Database, combined with co-engineering with the Oracle Linux team, translates to exceptional performance, improved upgrade time, and hardened system security.

🚩Superior data throughput and scalability

Exadata utilizes an RDMA over Converged Ethernet (RoCE) network fabric to connect each database server to every storage server. This network provides low latency and ample bandwidth for rapid data access and transfer rates. Beyond speed, the RoCE fabric includes capabilities such as zero packet loss messaging, direct data access without CPU involvement, and KVM-based virtualization security. 

As database workloads grow, the RoCE network fabric plays a central role in Exadata's scale-out architecture. Customers can add additional databases and/or storage servers online to increase performance and capacity. Should the demand exceed the capacity of a single physical rack, multiple racks can be interconnected without downtime to meet the performance, scalability, and availability needs of any environment. 

RDMA is an integral part of the high-performance architecture of Exadata and has been continuously enhanced with each successive generation of Exadata, underpinning Exadata-only technologies such as Exadata RDMA Memory (XRMEM) Data Accelerator and Exafusion Direct-to-Wire Protocol. Moreover, it optimizes the efficiency of complex, mission-critical OLTP and analytics workloads, as well as modern applications using JSON, blockchain, spatial, graph, and vector processing within Oracle Database.

To fully leverage Oracle Linux with the minimized UEK and harness the comprehensive features for which Exadata is renowned, specialized software known as ESS is installed and operated on all database and storage servers within the Database Machine. ESS enables a range of features such as XRMEM Data Accelerator, Exadata Smart Flash Cache (Flash Cache), Smart Scan, Storage Indexes, Hybrid Columnar Compression, Database In-Memory Columnar Cache on Storage Servers, and many more. 

RDMA (Remote Direct Memory Access) allows one computer to directly access data from another without operating system or CPU involvement, and has high bandwidth and low latency. The network card directly reads from and writes to memory with no extra copying or buffering and with very low latency. RDMA is an integral part of the Exadata high-performance architecture, and has been tuned and enhanced over the past decade, underpinning several Exadata-only technologies such as Exafusion Direct-to-Wire Protocol.

Exadata RDMA Memory (XRMEM) in the Exadata Storage Servers is leveraged as a shared read accelerator. The XRMEM Data Accelerator is a memory cache tier in front of Flash Cache, enabling orders of magnitude lower latency accessing remotely stored data. By utilizing RDMA to access memory remotely, XRMEM Data Accelerator bypasses the network and I/O stack, eliminating expensive CPU interrupts and context switches, and reducing latency by more than 10x, from 200 µs to less than 17 µs. Smart Exadata System Software also ensures data is mirrored across storage servers, which provides additional fault tolerance. Exadata's unique end-to-end integration between Oracle Database and Exadata Storage Servers automatically caches the hottest data blocks efficiently between the buffer cache in database servers and XRMEM and Flash Cache in storage servers. XRMEM is a shared storage tier across all of the storage nodes, which means the aggregate performance of this cache can be dynamically used by any database instance on any database server. This is a significant advantage over general-purpose storage architectures, which preclude sharing storage resources across database instances.

Exafusion Direct-to-Wire Protocol is a database optimization for Exadata that allows Oracle Real Application Cluster (RAC) Databases to send and recieve messages directly between instances circumventing the local operating system and network stack. Exafusion leads to response time improvements and greater database scalability by reducing the processing requirements of these inter-instance messages.

In an Oracle RAC envirnoment, each database instance caches data blocks in its System Global Area (SGA) to satisfy queries and transactions. If another instance in the cluster requires that block, it is passed between instances.

After one instance caches data, any other instance within the same cluster database can acquire a block image from another instance in the same database faster than by reading the block from disk. Cache Fusion moves current blocks between instances rather than re-reading the blocks from disk. When a consistent block is needed or a changed block is required on another instance, Cache Fusion transfers the block image directly between the affected instances. Oracle RAC uses the private interconnect for interinstance communication and block transfers.

Exafusion Direct-to-Wire Protocol is an optimization to the way Oracle RAC implements Cache Fusion on Exadata, enabling Oracle Database 12.1.0.2 (BP13) and higher to directly interface with the RDMA over Converged Ethernet (RoCE) network cards bypassing the Operating System kernel, and network software thereby improving performance for OLTP workloads significantly

🚩Lower OLTP I/O latency from data caching in remote memory 

The XRMEM Data Accelerator is an automatically managed, shared read-accelerator memory cache tier in front of Flash Cache, which enables orders of magnitude lower latency when accessing remotely stored data. Oracle Database uses RDMA to remotely access the XRMEM Data Accelerator memory, bypassing the network and I/O stack on both the database and storage servers. This eliminates costly CPU interrupts and context switches and reduces latency by more than 10X, from 200 μs to as low as 17 μs. Exadata's end-to-end integration between Oracle Database and Exadata Storage Servers automatically caches the hottest data efficiently between the database buffer cache and Flash Cache in the storage servers, thereby increasing the efficacy of RDMA-enabled Oracle Linux with UEK.

🚩Superior performance produced by intelligent data management

Exadata Storage Servers incorporate PCIe Flash Cards, which function as a caching tier between Exadata RDMA Memory and persistent storage, commonly referred to as Exadata Smart Flash Cache. This caching mechanism is used in conjunction with the XRMEM Data Accelerator to automatically cache frequently accessed data, while less frequently accessed data remains on disk. This approach combines the high I/O rates and fast response times of flash with the large capacity and cost efficiency of disk. Exadata possesses a unique understanding of database workloads and can intelligently avoid caching data that negatively impacts overall performance. 

Smart Scan, also known as SQL Offload to Storage and exclusive to Exadata, pushes SQL query processing down to the storage servers. It performs tasks such as filtering, column selection, joins, and aggregation within the storage tier, eliminating the need to bring all queried data to the database tier to begin with. Conventional storage arrays only deliver a fraction of the aggregate bandwidth available from their flash drives due to the network link between the database server and storage becoming a significant bottleneck.

A key database format-aware storage server optimization is the Storage Index. When smart scans run on storage servers, they populate an in-memory metadata cache that records the ranges of column values stored within each of the data units accessed by the scan. This metadata can be used by future smart scans to determine, based on their filter predicates, whether to visit or skip a certain data unit. As a result, there can be a significant reduction in the number of storage I/O operations issued by the smart scan, depending on the selectivity of the filter.  

🚩Reduced overheads through optimized communication

Exafusion is the next-generation networking protocol uniquely available on Oracle Exadata. Oracle Database utilizes Exafusion to eliminate context switching and minimize OS overhead through direct-to-wire messaging, yielding a 3X improvement in latency compared to non-Exadata platforms relying on traditional messaging models. Additionally, there is a substantial reduction in CPU utilization associated with inter-instance communications. Overall, this acceleration in messaging directly benefits application performance, particularly for those deployed on Oracle Real Application Clusters (RAC).

When deployed on Exadata, Oracle Database maintains an In-Memory Commit Cache. Each Oracle RAC instance maintains a cache of local transactions and commit state, which remote instances can read using RDMA. This commit cache facilitates bulk transaction state lookup, which reduces single transaction lookup messages and substantially improves both query and DML processing time.

🚩Enhanced high-availability

In addition to the Maximum Availablity Architecturefeatures and best practices that Exadata is built on, Exadata Instant Failure Detection distinctively enhances Oracle Database availability. By leveraging RDMA, instead of TCP/IP heartbeat messages, it rapidly detects—sub-second detection—any failures in a database or storage server. Without Exadata Instant Failure Detection, server failure detection typically takes a minute or more, during which time the application may experience brownouts or inconsistent performance.

🚩Database consolidation on Exadata

Isolation is crucial in hosted and shared service provider and DevTest environments. Consolidated environments running on Exadata X10M may also use KVM-based Virtual Machines (VM Guests) and Secure RDMA Fabric Isolation for strong isolation between workloads. When using virtualization, Exadata can securely deploy multiple VM clusters—running either the same or different Exadata software, grid infrastructure, or database versions—all on the same set of database servers.

🚩Elevated security without interruption

Exadata database servers leverage the Oracle Ksplice zero-downtime capability of Oracle Linux to apply security patches while the OS stays online, meaning that no workloads or systems need to be interrupted. For the Exadata Database Machine, customers can use Ksplice to apply updates to the kernel on database servers. For Exadata Cloud@Customer, when possible, updates deployed by Oracle are applied to running systems without downtime through the use of Ksplice. For other updates that require a component restart, Oracle performs the component restart in a rolling fashion to help ensure service availability during the update process.

Exadata's Linux configuration exceeds 90% on the Defense Information Systems Agency (DISA) Security Technical Implementation Guide (STIG)Security Content Automation Protocol (SCAP)benchmark right out of the factory. Coupled with Exadata-optimized Linux security features such as Secure Boot, Advanced Intrusion Detection Environment (AIDE), Security Enhanced Linux (SELinux), LDAP, Kerberos support, Federal Information Processing Standard (FIPS) 140-2 support, Secure Computing (seccomp) on Storage Servers, and the full suite of Oracle Database security features, Exadata delivers end-to-end data protection without compromising performance or availability. 

🚩Fueling business success worldwide

Today, thousands of businesses spanning industries such as retail, finance, and telecommunications are running their most critical and demanding database workloads on Exadata, powered by Oracle Linux. Irrespective of the workload—whether it's OLTP, data warehousing, or machine learning—the dynamic combination of Exadata, Oracle Database, and Oracle Linux, empowers our customers globally to handle mission-critical workloads in a deployment model tailored to their requirements.

************************************

Sincerely,

Alireza Kamrani 

Oracle technical Consultant.