A Practical Guide to CPU-Related Issues in Oracle Database Hosting within a Virtualized Environment

A Practical Guide to CPU-Related Issues in Oracle Database Hosting  within a Virtualized Environment 

What is steal CPU time?

‘Steal time’ (or ‘Stolen’ time) is relevant only in cloud environments or VMWare environments, where multiple virtual machines will be run on one underlying physical host. In such circumstances, CPU resource will be shared amongst the multiple virtual machines. The hypervisor is a technology that will distribute the underlying physical host’s CPU resources and other resources amongst the virtual machines.

“Steal time” is the percentage of time a virtual machine CPU waits document for a real CPU while the hypervisor is servicing other virtual machines.

The impact depends heavily on the workload. A lightly used background service may tolerate occasional steal without noticeable effects, while a latency-sensitive API, database, build system, or real-time workload may experience slower responses much sooner.

High steal can contribute to increased response times, slower request processing, growing queues, lower throughput, or longer batch jobs. The effect becomes more visible when the application already needs most of the CPU capacity available to it.

 

CPU Steal Time vs. CPU Usage vs. I/O wait

CPU steal and I/O wait can both make applications appear slow, but they describe different problems.

CPU Steal is Not the Same as CPU Usage.

CPU usage measures how much CPU time the guest operating system is spending on application or kernel work. CPU steal measures CPU time the guest wanted but did not receive from the hypervisor.

CPU state	What it means
User	CPU running application code
System	CPU running kernel code
Idle	CPU has no work to run
I/O wait	CPU idle while work is waiting on I/O
Steal	VM has runnable work but CPU time is being used elsewhere by the hypervisor

This distinction matters during troubleshooting. Increasing application CPU usage usually means processes inside your VM are doing more work, while increasing steal suggests the bottleneck may exist outside the VM.

I/O wait represents time associated with tasks waiting for I/O operations, such as disk activity, to complete.

If steal is high, investigate virtualization and CPU capacity. If I/O wait is high, disk or storage performance may deserve more attention.

Looking at both metrics together helps avoid chasing the wrong bottleneck.

Ideally, steal should remain very low or close to zero. A small temporary increase may have little impact, while sustained steal becomes more important when it coincides with higher application latency, lower throughput, rising load, or growing request queues.

Avoid interpreting CPU steal alone. Look at it together with CPU utilization, load, traffic, application latency, throughput, errors, and infrastructure changes.

 

Your Oracle Database VM may have enough vCPUs. But does it actually own the physical CPU resources it needs?

One often-overlooked cause of Oracle performance degradation in virtualized environments is CPU Steal Time.

Imagine this infrastructure:

• Physical host: 16 physical CPU cores
• Total allocated resources: 40 vCPUs
• Oracle Database VM: 8 vCPUs
• Other VMs: Application, Postgres, and Monitoring

Your Oracle processes are ready to execute, but the hypervisor cannot immediately schedule them on physical CPUs.

The result?

Ø  CPU Steal Time increases.

This becomes particularly important for Oracle Data Guard environments.

Consider a standby database:

• Redo arrival rate: 500 MB/s
• Normal redo apply capacity: 600 MB/s
• CPU contention reduces apply capacity to 400 MB/s.

Now:

Redo Arrival: 500 MB/s
Redo Apply: 400 MB/s
Result: Apply Lag increases.

Interestingly, Transport Lag might remain close to zero.

The standby receives redo successfully, but cannot apply it quickly enough.

 

How do we investigate CPU Steal Time?

From the Oracle Linux guest:

ü  vmstat 1

ü  mpstat -P ALL 1 10

ü  top

ü  sar -u -P ALL 1

ü  Using web-based root cause analysis tools like yCrash

Pay particular attention to:

• %steal: CPU time the guest was unable to obtain.
• %idle: Actually idle CPU time.
• ***Per-vCPU steal distribution: An aggregate average can hide contention on individual vCPUs.

Also correlate these metrics with following commands + AWR report:

iostat -xz 1

And Oracle Data Guard:

SQL> SELECT NAME, VALUE, TIME_COMPUTED FROM V$DATAGUARD_STATS

WHERE NAME IN ('transport lag', 'apply lag');

But monitoring alone is not the complete solution.

There is another architectural question:

How can we guarantee that the physical CPU cores allocated to an Oracle VM will not be consumed by other VMs?

The answer is Hard CPU Partitioning / Exclusive CPU Allocation.

Three concepts must be distinguished:

1. CPU Reservation: Guarantees a specified amount of CPU capacity.
2. CPU Affinity / Pinning: Restricts where a VM's vCPUs can execute.
3. CPU Isolation / Exclusive Assignment: Prevents other workloads from using the designated physical CPU resources.

 

For example, on a 32-logical-CPU host:

PHYSICAL HOST

|

|-- CPU 0-3    --> Hypervisor / Host

|

|-- CPU 4-11   --> ORACLE VM (Dedicated)

|

|-- CPU 12-19  --> Application VM

|

|-- CPU 20-25  --> Postgres VM

|

|-- CPU 26-31  --> Monitoring VM

 

However, CPU pinning alone does not guarantee exclusivity. The other VMs must also be prevented from executing on Oracle's assigned CPUs.

For KVM/libvirt, this can involve:

• vcpupin for guest vCPU placement.
• emulatorpin for QEMU emulator threads.
• iothreadpin for I/O threads.
• Host CPU isolation and dedicated CPU sets.
• NUMA and physical-core alignment.
• Consideration of Hyper-Threading sibling CPUs.

For VMware, CPU affinity alone is not sufficient to guarantee exclusive physical CPU access. CPU reservation, latency sensitivity, and the supported exclusive resource-allocation mechanisms must be evaluated together.

 

CPU Steal Time in Kubernetes

Kubernetes itself does not create CPU steal, but Kubernetes worker nodes are often virtual machines. If the underlying VM experiences CPU contention at the hypervisor level, every pod running on that node can be affected.

This can make troubleshooting confusing. Several pods may become slower at the same time even though none of them has reached its Kubernetes CPU limit and the node's normal CPU metrics do not immediately explain the degradation.

Checking host-level steal time can reveal that the problem exists below Kubernetes. If high steal aligns with increased pod latency or slower workloads on one node, the underlying VM or physical host may deserve investigation.

CPU steal should therefore be monitored at the node level, then correlated with Kubernetes workloads, application latency, pod CPU usage, and node load.

 

The important takeaway:

CPU Steal Time does not mean Oracle is consuming CPU. It means Oracle wanted CPU execution time but the hypervisor did not provide it.

And this is why a VM can appear to have allocated vCPUs while still experiencing significant performance degradation.

More vCPUs do not automatically mean more available physical CPU capacity.

 

General solution:

Because CPU steal happens outside the guest operating system, optimizing application code inside the VM may not solve it. The response depends on why the hypervisor is withholding CPU.

Moving the workload to a different host or VM can help if the issue is host-level contention. In environments you control, reducing the number of competing VMs, increasing CPU allocation or priority, or changing resource quotas can also reduce contention. Red Hat recommends increasing guest CPU priority or quota, or running fewer guests on the host when steal is high.

In cloud environments, another option may be moving from shared CPU infrastructure to an instance type with more predictable or dedicated compute capacity. Scaling out can also reduce the CPU demand placed on each individual VM.

Before making changes, confirm that CPU steal actually correlates with the application problem. Moving to a larger instance will not fix slow storage, database contention, network latency, or inefficient application code.

 

CPU steal is most useful when viewed over time rather than as a one-time reading. Continuous monitoring helps identify whether a spike was brief or whether the VM repeatedly loses CPU during busy periods.

A useful dashboard can combine steal time with CPU user time, system time, idle time, load average, application latency, request throughput, and errors. If steal increases while application latency rises and idle CPU falls, CPU contention becomes a much stronger investigative lead.

You should also compare affected hosts. If one VM shows unusually high steal while identical instances handling the same workload remain healthy, the difference may point toward the underlying host or resource allocation rather than the application itself.

In a virtual environment, CPU cycles are shared across virtual machines on the server. If your VM displays a high %st in top (steal time), this means CPU cycles are being taken away from your VM to serve other purposes. You may be using more than your share of CPU resources or the physical server may be over-sold. Move the VM to another physical server. If steal time remains high, try giving the VM more CPU resources.

 

This is technically possible with VMware PowerCLI and vCenter APIs to make a trigger to CPU contention detection and make a preferred operation after it, such add some vCPU to Database vm. It is an interesting automation idea for Oracle infrastructure, but I would make one important change:

Ø  Do not automatically add vCPUs simply because CPU Steal Time is high.

In VMware, I would use CPU Ready as the primary hypervisor-side scheduling metric and correlate it with guest CPU utilization.

Why?

Situation	Correct action
High CPU utilization + low CPU Ready	Consider adding vCPUs
High CPU Ready + high host CPU utilization	Investigate host contention; consider migration or adding host capacity
High CPU Ready + low guest CPU utilization	Investigate scheduling, CPU limits and VM sizing
High CPU utilization + high CPU Ready	Investigate both VM sizing and host capacity before scaling

CPU Ready means the VM was ready to execute but ESXi could not schedule it on a physical CPU.

 

My architectural recommendation for critical Oracle workloads:

Do not treat an Oracle production VM like an ordinary shared application VM.

Consider:

• Dedicated physical CPU cores.
• Exclusive CPU allocation.
• NUMA-aware vCPU and memory placement.
• Memory reservation to avoid memory contention.
• Hypervisor-level CPU Ready and scheduling monitoring.
• Guest-level CPU Steal monitoring.
• Correlation with Oracle AWR and Data Guard Apply Lag.

The objective is not simply to allocate more vCPUs.

It is to ensure that the physical CPU capacity allocated to Oracle is actually available when Oracle needs it.

 

A question for infrastructure and database architects:

In your production virtualization environments, how do you protect latency-sensitive Oracle workloads?

1. CPU Shares / Reservations
2. CPU Affinity / Pinning
3. Dedicated Physical CPU Cores / Hard Partitioning
4. Dedicated Physical Hosts

I believe this is an important discussion between DBA, Linux, and virtualization teams.

 

Recommended Actions for High CPU Steal Time

In virtualized environments, sustained high CPU steal time indicates that a VM is not receiving the physical CPU execution time it needs.

General recommendations include:

• Check physical host CPU utilization and hypervisor scheduling metrics.
• Investigate CPU oversubscription and contention.
• Consider migrating the VM to a less congested physical host.
• Review CPU allocation and reservations. Increase vCPUs only if sufficient physical capacity is available.

For high-TPS, latency-sensitive Oracle databases, consider a stronger approach:

• Dedicate physical CPU cores to the Oracle VM.
• Implement CPU pinning and exclusive CPU isolation.
• Ensure other VMs cannot consume Oracle's dedicated cores.
• Align vCPUs and memory with NUMA topology.
• Avoid memory overcommit for critical database workloads.

For particularly demanding workloads, a dedicated physical database server may be justified.

The objective is not simply to allocate more vCPUs, but to guarantee that physical CPU resources are available when Oracle needs them.