Transactional Event Queues (TxEventQ) and Advanced Queuing (AQ) VS Apache Kafka

Transactional Event Queues (TxEventQ) and Advanced Queuing (AQ)

Alireza Kamrani

Oracle Advanced Queuing is a messaging platform built into Oracle Database that is used for application workflows, microservices, and event-triggered actions. Oracle Database 19c introduces Oracle Transactional Event Queues, unifying the best of messaging and pub/sub for events with Kafka interoperability.

Messaging and pub/sub

• High throughput (~1 M+ messages/sec on RAC) with parallel (partitioned) event streams within a queue and in-memory message cache

• Exactly-once transactional message delivery, where DML operation in database and message are part of same transaction

• Total order of messages within each event stream and multiple event streams per queue are supported

• Pub/sub (topics) and send/receive messaging (queues) supported in one infrastructure

• Async consumption of messages with notification or poll time-outs

Language support

• Multiple language APIs supported for producers and consumers: Java, Java with JMS, PL/SQL, C/C++, Python, Node.js, REST, and CLI

• Kafka Java Client support for replacing Kafka broker with TxEventQ as the message broker

• JMS standard open source API support for JMS 1.1+ (some features of JMS 2.0, such as multiple consumers, delayed messages, and more)

Operational features

• TxEventQ gateway supports interoperability with Kafka

• Queue statistics and continuous real-time monitoring with Prometheus/Grafana

• Database security for encryption of messages in the queue table

• Rule-based subscribers used to filter messages on the wire based on fast execution of rules

• Arbitrary seek back in the event stream, new or existing consumers can consume messages from an earlier offset

• Message delay, priority, expiry, and queue-to-queue propagation allow flexibility in building modern applications and workflows

Pub/sub of events in modern app dev environment

Developers of event-driven microservices and workflows rely on a high-throughput reliable messaging platform. Oracle TxEventQ provides robust real-time messaging, streaming events, and pub/sub with multiple publishers and multiple consumers. High-throughput installations of TxEventQ can achieve approximately 100 billion messages per day on an 8-node Oracle Real Application Clusters (RAC) database.

Transactional Event Queues

Oracle Transactional Event Queues (TxEventQ) is a messaging platform built into Oracle Database that combines the best features of messaging and pub/sub systems.

TxEventQ was introduced as a rebranding of AQ Sharded Queues in Oracle Database 21c, evolving from the Advanced Queuing (AQ) technology that has been part of Oracle Database since version 8.0. TxEventQ continues to evolve in Oracle Database 23ai, with Kafka Java APIs, Oracle REST Data Services (ORDS) integration, and many more features and integrations.

TxEventQ is designed for high-throughput, reliable messaging in event-driven microservices and workflow applications. It supports multiple publishers and consumers, exactly-once message delivery, and robust event streaming capabilities. On an 8-node Oracle Real Application Clusters (RAC) database, TxEventQ can handle approximately 1 million messages per second, demonstrating its scalability.

TxEventQ differs from traditional AQ (now referred to as AQ Classic Queues) in several ways:

• Performance: TxEventQ is designed for higher throughput and scalability.

• Architecture: TxEventQ uses a partitioned implementation with multiple event streams per queue, while AQ is a disk-based implementation for simpler workflow use cases.

• Enhanced interoperability with Apache Kafka, JMS, Spring Boot, ORDS, and other software systems.

Key features of Oracle Database Transactional Event Queues include:

• Transactional messaging: Enqueues and dequeues are automatically committed along with other database operations, eliminating the need for two-phase commits.

• Exactly once message delivery.

• Strict ordering of messages within queue partitions.

• Retention of messages for a specified time after consumption by subscribers.

• Capability to query message queues and their metadata by standard SQL.

• Transactional outbox support: This simplifies event-driven application development for microservices.

For developers, TxEventQ can be integrated into modern application development environments using Oracle Database. It’s particularly useful in microservices architectures and event-driven applications where high-throughput, reliable messaging is crucial. The Kafka-compatible Java APIs allow developers to use existing Kafka code with minimal changes, simply by updating the broker address and using Oracle-specific versions of KafkaProducer and KafkaConsumer.

Oracle Database Transactional Event Queues are free to use with Oracle Database in any deployment, including Oracle Database Free.

AQ and TxEventQ use cases

• Application workflows

Retail delivery guarantee workflow

Delivery companies use one-time passwords (OTP) to verify the delivery address and person, using Oracle AQ as the backbone for message exchange between the vendor app, deliverer, and the customer. See code in the Oracle AQ LiveLab.

• Event-driven applications

ATM banking second-factor authentication

Banks require an OTP to validate their accounts for withdrawal in many countries. Learn how to use Oracle AQ to create a messaging infrastructure to transfer the OTP between the ATM server code and the client mobile app of the customer.

Oracle AQ is used for coordinating the OTP from the ATM machine with the customer’s registered mobile phone.

• Microservices

Food delivery app on Oracle Autonomous Database

GrabDish is a food delivery app that uses Oracle AQ for communicating between the order, inventory, and delivery microservices. Check out the code in Simplify Microservices with Oracle AQ LiveLab.

Compare Results :

Use TxEventQ if you want Kafka-like pub/sub messaging within Oracle, with full transactional safety and tight integration. It's a good choice for incremental modernization.

Use Kafka if you're building distributed systems or need to integrate multiple non-Oracle platforms, and are OK managing Kafka infrastructure.

Use AQ only if you have legacy apps relying on it, or for very simple queuing needs within Oracle.