TDMQ for RocketMQ is a distributed high-availability message queue service. Built upon the infrastructure of Apache RocketMQ 4.x and 5.x, it offers various product forms that allow clients of version 4.4.x or later to connect without modifications. It also features the underlying strengths of compute-storage separation and flexible scalability. TDMQ for RocketMQ supports a throughput of up to millions of TPS and is suitable for various large-scale, low-latency online messaging scenarios with high reliability requirements.
Product Features
Ops-Free
Supports one-click cluster creation for out-of-the-box use.
Supports rapid scaling by TPS specification, offering ultimate elasticity.
5.x TPS supports an elastic range to handle traffic spikes.
5.x storage is Serverless, supporting on-demand use and pay-as-you-go billing.
Rich Features
Supports multiple message types: transactional, scheduled, delayed, and ordered messages.
Supports multiple consumption modes: tag filtering, SQL 92 filtering, clustering consumption, and broadcasting consumption.
Observability
Provides comprehensive monitoring metrics to quickly identify issues.
Supports visualized message tracing to connect upstream and downstream services for better troubleshooting and issue pinpointing.
High Availability
Supports cross-availability zone (AZ) deployment to withstand IDC-level failures.
Containerization enables automatic restart within seconds, with no capacity or data damage during single-node downtime.
Supports cluster-level traffic throttling protection to enhance cluster health.
Supports message replication across clusters to meet disaster recovery requirements.
Security Control
Integrates with Tencent Cloud Cloud Access Management (CAM) for granular operation and resource-level permission management.
Provides a visualized access control list (ACL) management interface for controlling message sending and receiving permissions.
Fully compatible with open-source RocketMQ, requiring no business code modifications.
Provides a metadata migration tool to reduce migration costs.
Provides a hitless cluster migration solution with low intrusion and rollback capability, minimizing business intrusion.
Technical Architectures
The following diagrams demonstrate the system deployment architectures of TDMQ for RocketMQ.
5.x Clusters
4.x Clusters
The TDMQ for RocketMQ 5.x series introduces the new gRPC protocol and Proxy components, implementing an architecture featuring storage-compute separation. This significantly changes both the Ops and usage of RocketMQ.
The concepts involved are as follows:
Producer cluster: a client-side application responsible for producing and sending messages.
Consumer cluster: a client-side application responsible for subscribing to and consuming messages.
NameServer cluster: a server-side application responsible for route addressing and broker heartbeat registration. To ensure high availability, it is deployed across AZs by default.
Heartbeat registration: The NameServer acts as a registry. Each component machine periodically reports its status to the NameServer. If a machine fails to report within the timeout period, the NameServer will consider it faulty and remove it from the available list.
Route addressing: Each NameServer stores both the complete routing information of the broker cluster and the queue information for client queries. Producers and consumers use the NameServer to acquire the complete routing information of the broker cluster, which then allows for message delivery and consumption.
Proxy cluster: a new elastic, stateless proxy service that is deployed across AZs by default to ensure high availability. It splits the responsibilities of the 4.x broker, extracting computation logic such as client protocol adaptation, permission management, and consumption management.
Broker cluster: Compared with the 4.x series, the broker in the 5.x series is more focused on the continuous enhancements of storage capabilities. To ensure high availability, it is deployed across AZs by default.
The concepts involved are as follows:
Producer cluster: a client-side application responsible for producing and sending messages.
Consumer cluster: a client-side application responsible for subscribing to and consuming messages.
NameServer cluster: a server-side application responsible for route addressing and broker heartbeat registration.
Heartbeat registration: The NameServer acts as a registry. Each component machine periodically reports its status to the NameServer. If a machine fails to report within the timeout period, the NameServer will consider it faulty and remove it from the available list.
Route addressing: Each NameServer stores both the complete routing information of the broker cluster and the queue information for client queries. Producers and consumers use the NameServer to acquire the complete routing information of the broker cluster, which then allows for message delivery and consumption.
Broker cluster: a server-side application responsible for receiving, storing, and delivering messages. It supports the primary-secondary multi-replica mode where the deployment of secondary nodes is optional. The actual high reliability of data in the production environment on the public cloud directly depends on the three replicas of the cloud disk.
Management cluster: a server-side application with a visual management console, responsible for the Ops of the entire cluster, such as metadata management.
