This document lists the common concepts and definitions in TDMQ for RocketMQ.
Broadcasting Consumption
In the broadcasting consumption mode, each message is pushed to all registered consumers in the cluster to ensure that the message is consumed at least once by each consumer. This mode is suitable for scenarios where each message needs to be processed by every consumer in the cluster.
Clusters
A cluster is a resource dimension in TDMQ for RocketMQ. Resources such as topics and groups are completely isolated across different clusters. Each cluster has resource limitations, such as the total number of topics and message retention time.
Clustering Consumption
In the clustering consumption mode, each message only needs to be processed by any of the consumers in the cluster. This mode is suitable for scenarios where each message only needs to be processed once.
Consumers
A consumer is an entity that receives and processes messages in TDMQ for RocketMQ. It is usually integrated into the business system to obtain messages from the server and convert the messages into information that can be perceived and processed by business logic.
Consumer Offsets
A message is not removed from the queue immediately after it has been consumed by a consumer. TDMQ for RocketMQ records the offset of the last consumed message based on each consumer group. Such an offset is defined as the consumer offset.
Dead Letter Queues
A dead letter queue is a special type of message queue used to centrally process messages that cannot be consumed normally. If a message cannot be consumed after a specified number of retries in the retry letter queue, TDMQ will determine that the message cannot be consumed under the current situation and deliver it to the dead letter queue.
In actual scenarios, messages may fail to be consumed due to prolonged service downtime or network disconnections. In such cases, messages are not discarded immediately. The dead letter queue persists such messages for a long period. After you obtain a solution, you can create a consumer to subscribe to the dead letter queue to process such messages.
Message Filtering
A consumer can filter messages by subscribing to specified message tags to ensure that it only receives the filtered messages. The calculation and matching of filtering rules are completed on the TDMQ for RocketMQ server.
Groups
Groups include producer groups and consumer groups.
Producer group: a collection of the same type of producers that send the same type of messages with the same sending logic. If a producer sends transactional messages and crashes afterward, the broker will contact other producer instances in the producer group to commit or roll back the transaction.
Consumer group: a collection of the same type of consumers that consume the same type of messages with the same consumption logic. It can ensure load balancing and fault tolerance in the message consumption process. Consumer instances in a consumer group must subscribe to the same topics.
Message Keys
A message key is a message-oriented index property in TDMQ for RocketMQ. By setting the message key, you can quickly find the corresponding message content.
Messages
A message is the physical carrier of information transmitted by the messaging system. It is the smallest unit of the produced or consumed data. A producer encapsulates the load and extended properties of business data into messages and sends the messages to the server. Then, the server delivers the messages to the consumer based on the relevant semantics.
Message Backlogs
Message backlogs occur in scenarios where the producer has sent messages to the TDMQ for RocketMQ server but the consumer fails to normally consume all these messages promptly due to its consumption capability limit. In this case, the unconsumed messages will be backlogged on the server. The message heap data is collected once every minute. After the message retention time (3 days by default) elapses, the unconsumed messages will no longer be backlogged on the server because they have been deleted by the server.
Message Queues
A message queue (also known as a message partition) is a physical entity for message storage, and a topic can contain multiple queues. Messages in a queue can only be consumed by one consumer rather than multiple consumers in one consumer group.
Message Queue Offsets
Messages are stored in multiple queues of a specified topic based on the order in which they arrive at the TDMQ for RocketMQ server. Each message has a unique coordinate of type Long in the queue, which is defined as the message queue offset.
Message Retention Policies
A message retention policy defines the retention time for which messages are persisted on the TDMQ for RocketMQ server. Messages exceeding the retention time will be automatically cleaned.
Message Tags
Tags are used to classify different types of messages in the same topic. Topic and tag are basically the first-level and second-level classifications of messages, respectively.
Message Traces
The message trace refers to the complete linkage information formed by aggregating data such as timestamps and locations from each relevant node. It records the entire lifecycle of a message from the time it is sent by the producer to the time it is received and processed by the consumer. With this feature, you can track the entire trace of a message, starting from its production by a producer, its storage and distribution within the TDMQ for RocketMQ server, and finally its consumption by one or more consumers. This helps you troubleshoot any problems that may occur during message processing.
Message Types
Messages are classified by message transmission characteristic for message type management and security verification. TDMQ for RocketMQ has four message types: normal messages, ordered messages, transactional messages, and scheduled/delayed messages.
Message Queries
The Tencent Cloud TDMQ for RocketMQ console and SDK provide robust message query features, supporting message searches by message ID, message key, or time range, which is critical for business troubleshooting and issue diagnosis.
Namespaces
A namespace is a unit for logically isolating message topics and resource groups. In TDMQ for RocketMQ, namespaces serve as a crucial boundary for resource management and permission control.
Normal Messages
Normal messages are a basic type of message. After normal messages are delivered to a specified topic by the producer, they will be consumed by consumers that subscribe to this topic. A topic with normal messages is sequence-insensitive. Therefore, you can use multiple topic partitions to improve the message production and consumption efficiency. This approach performs best when dealing with high throughput.
Ordered Messages
Ordered messages are advanced messages provided by TDMQ for RocketMQ. For a specified topic, messages are published and consumed in strict accordance with the First In First Out (FIFO) principle, which means that the messages sent first are consumed first, and the messages sent later are consumed later.
Producers
A producer in TDMQ for RocketMQ is a messaging entity that creates messages and sends them to the server. It is typically integrated into the business system to encapsulate data as messages and send them to the server.
Retry Queues
A retry queue is designed to ensure that messages are consumed normally. If no normal response is received after a message is consumed by the consumer for the first time, the message will enter the retry queue. When a certain number of retries are reached, the retries are stopped, and the messages will be delivered to the dead letter queue.
In actual scenarios, messages may not be processed promptly due to temporary issues that will recover after a short period of time, such as network jitter and service restarts. In such scenarios, the retry mechanism of the retry queue can be a good solution.
Resetting a Consumer Offset
Resetting the consumer offset means resetting a consumer group's consumer offset for subscribed topics within the persistent storage period of messages based on the time axis. After the offset is reset, the consumers will receive the messages that are produced after the set time point.
Scheduled/Delayed Messages
After a producer sends a message, it is not immediately available to consumers but will be delivered to consumers at a specific time or after a specific delay.
Topics
A topic represents a specific type of message. Each topic contains several messages. It is the basic unit for message subscription in TDMQ for RocketMQ.
Transactional Messages
Transactional messages are messages that support distributed transactions. TDMQ for RocketMQ provides a complete solution for transaction messages, ensuring eventual data consistency in distributed systems.
VPC Access
Virtual Private Cloud (VPC) access refers to the network connection method between your producer/consumer client and the TDMQ for RocketMQ server. It is recommended that the client and the TDMQ instance reside in the same VPC within the same region to achieve optimal network performance and security.