High Cluster Reliability
Download
フォーカスモード
フォントサイズ
TDMQ for CKafka (CKafka) provides cluster-level high-availability capabilities. Through cross-availability zone (AZ) deployment and instance AZ migration features, it effectively enhances the stability and disaster recovery capability of the messaging service.
Cluster-Level High-Availability Capability Description
High-Availability Capability | Applicable Version | Description | Reference Documentation |
Cross-AZ deployment | Serverful advanced and pro editions Serverless mode | When purchasing a CKafka Serverful instance in a region with 3 or more AZs, you can choose to deploy it across multiple AZs, so that partition replicas are forcibly distributed across nodes in different AZs. This deployment mode ensures that your instance can continue to provide services normally, even if a single AZ becomes unavailable. The Serverful pro edition supports selecting up to four AZs. The Serverful advanced edition supports selecting up to two AZs. For a CKafka Serverless instance, you only need to select a region. The system then automatically deploys it across AZs, simplifying configuration and ensuring high availability. | |
Instance AZ change | Serverful pro edition | Migrate an instance to another AZ in the same region, after which all properties, configurations, and connection addresses of the instance will not be changed. Migrate an instance from one AZ to another: The AZ where the instance resides becomes fully loaded or experiences other issues that affect instance performance. Migrate an instance from one AZ to multiple AZs: Enhance the disaster recovery capability of the instance and achieve cross-IDC disaster recovery. |
Cross-AZ Deployment
Deployment Architecture
The cross-AZ deployment architecture of CKafka is divided into the network layer, data layer, and control layer.
Network Layer
CKafka exposes a VIP to clients. After connecting to the VIP, a client obtains metadata about topic partitions. This metadata typically consists of addresses, which are mapped one-to-one through different ports on the same VIP. The VIP can fail over to another AZ at any time. When an AZ becomes unavailable, the VIP automatically drifts to another available node in the same region, enabling cross-AZ disaster recovery.
Data Layer
The data layer of CKafka adopts the same distributed deployment mode as native Kafka. In this mode, multiple data replicas are distributed across different Broker nodes, and these nodes are deployed in different AZs. When a partition is processed, a Leader-follower relationship exists between different nodes. If the Leader becomes abnormal, the cluster Controller elects a new partition Leader to handle requests for that partition.
For clients, when an AZ becomes unavailable due to an exception, if the Leader of a topic partition is located on a Broker node in the unavailable AZ, established connections may time out or be disconnected. After the Leader node of that partition fails, the Controller (if the Controller node fails, the remaining nodes will elect a new Controller node) will select a new Leader node to provide services. The Leader switching typically completes within seconds (the exact duration is proportional to the number of cluster nodes and metadata size). Clients periodically refresh topic partition metadata to connect to the new Leader node for production and consumption.
Control Layer
The control layer of CKafka adopts the same technical solution as native Kafka. It relies on ZooKeeper (ZK) for broker node service discovery and cluster controller election. For CKafka instances supporting cross-AZ deployment, ZK nodes of the ZK cluster are deployed across three AZs (or IDCs). If ZK nodes in any AZ become faulty or disconnected, the entire ZK cluster can still provide services normally.
Advantages and Disadvantages of Cross-AZ Deployment
Advantages
Cross-AZ deployment can significantly enhance the disaster recovery capability of a cluster. When force majeure risks such as unexpected network instability or power outages/restarts occur in a single AZ, it ensures that clients can resume message production and consumption after a brief reconnection wait.
Disadvantages
In cross-AZ deployment, partition replicas are distributed across multiple AZs, so message replication involves additional cross-AZ network latency compared with single-AZ deployment. This latency directly increases client write time for production when the client ACK parameter is greater than 1 or equal to -1 or all. Currently, cross-AZ latency in major regions such as Guangzhou, Shanghai, and Beijing typically ranges from 10 ms to 40 ms.
Analysis of Cross-AZ Deployment Scenarios
Single AZ Unavailable
After a single AZ becomes unavailable, as explained in the previous section on the principles, clients will get disconnected and reconnected. After clients get reconnected, services can still be provided normally.
Since the management API service currently does not support cross-AZ deployment, if a single AZ becomes unavailable, you may be unable to create topics, configure the access control list (ACL) policies, or view monitoring metrics via the console. However, this does not affect production and consumption for existing businesses.
Network Isolation Between AZs
If network isolation occurs between AZs, meaning that they cannot communicate with each other, a split-brain scenario may arise in the cluster. Nodes in both AZs will continue to provide services, but data written in one AZ will be considered dirty data after the cluster recovers.
Consider the following scenario: The cluster controller node and one ZK node in the ZK cluster experience network isolation from other nodes. At this point, the other nodes will re-elect a new controller (since most nodes in the ZK cluster maintain normal network communication, a controller can be elected successfully). However, the controller that experiences network isolation still considers itself the controller node, resulting in a split-brain scenario in the cluster.
Client writes in this scenario need to be considered on a case-by-case basis. For example, when the client ACK policy is set to -1 or all and the number of replicas is 2, assuming the cluster has 3 nodes, the distribution ratio after a split-brain event will be 2:1. Writes to partitions where the original Leader is on the node in the 1-node AZ will report an error, while writes on the other side will succeed. However, if there are 3 replicas and ACK=-1 or all, writes will fail on both sides. In this case, further handling measures need to be determined based on specific parameter configurations.
After the cluster network recovers, clients can resume production and consumption without any manual intervention. However, as the server will normalize the data again, data from one split-brain node will be truncated directly. For the multi-replica cross-region data storage mode, this truncation does not result in data loss.
Multi-AZ Disaster Recovery Limits
Capacity Limits
CKafka achieves multi-AZ disaster recovery by distributing underlying resources across multiple AZs. When an AZ fails, CKafka switches the Partition Leader to a Broker node in another AZ. This reduces the underlying resources that handle traffic, leading to capacity issues. Below is the resource availability status when an AZ fails:
For a two-AZ disaster recovery instance, usable capacity = total instance capacity/2. To ensure normal use, customers need to reserve 100% redundant resources.
For a three-AZ disaster recovery instance, usable capacity = total instance capacity/3 * 2. To ensure normal use, customers need to reserve 50% redundant resources.
For a four-AZ disaster recovery instance, usable capacity = total instance capacity/4 * 3. To ensure normal use, customers need to reserve 33.3% redundant resources.
For an n-AZ disaster recovery instance, usable capacity = total instance capacity/n * (n - 1). To ensure normal use, customers need to reserve redundant resources of 1/(n - 1).
Instance Parameter Configuration Limits
CKafka supports modifying the topic-level configuration
min.insync.replicas in the console. This configuration takes effect when ack = -1 is set for the client, ensuring that a message is only acknowledged as successfully produced after being synchronized by min.insync.replicas replicas simultaneously (for example, if the number of topic replicas is 3 and min.insync.replicas is set to 2, a message needs to be synchronized by at least 2 replicas before being considered successfully produced).Therefore, the Topic configuration must meet
min.insync.replicas <= number of AZs to ensure production availability during an AZ failure, and the Topic configuration must meet min.insync.replicas < number of Topic replicas to ensure production availability during a single-node failure. The configuration rule is: min.insync.replicas < number of Topic replicas <= number of AZs.Two-AZ instance: Number of topic replicas = 2; min.insync.replicas = 1
Three-AZ instance: Number of topic replicas = 2; min.insync.replicas = 1
Three-AZ instance: Number of topic replicas = 3; min.insync.replicas <= 2
Client Parameter Configuration Limits
When you use a CKafka instance deployed with multi-AZ disaster recovery, if an AZ fails, newly started clients may resolve to a faulty node upon the first connection. To enable clients to quickly reconnect to a healthy node, clients in different programming languages require corresponding parameter configurations.
Java Client
Versions earlier than 2.7.0: The connection timeout is 120s, and reconnection is slow.
Versions 2.7.0 and later: The
socket.connection.setup.timeout.ms parameter (default value: 10s) is supported to control connection timeout. When the client load is relatively manageable and there is a requirement for recovery time after an AZ failure, it is recommended to set socket.connection.setup.timeout.ms to 2-5s so that connections to healthy nodes can be retried more quickly when a failure occurs. For parameter details, see Java SDK Practice Tutorial.C++ Client (librdkafka)
For librdkafka client version 1.9 and later, the
socket.connection.setup.timeout.ms parameter (default value: 30s) is supported to control connection timeout. When the client load is relatively manageable and there is a requirement for recovery time after an AZ failure, it is recommended to set socket.connection.setup.timeout.ms to 2-5s so that connections to healthy nodes can be retried more quickly when a failure occurs. For parameter details, see librdkafka SDK Practice Tutorial.Note:
For disaster recovery drills, it is recommended to use librdkafka client version 2.10.0 or later.
Go Client
Confluent Go: Refer to the librdkafka configuration.
Sarama Go: The
config.Net.DialTimeout parameter (default value: 30s) is supported to control connection timeout. When the client load is relatively manageable and there is a requirement for recovery time after an AZ failure, it is recommended to set config.Net.DialTimeout to 2-5s so that connections to healthy nodes can be retried more quickly when a failure occurs. For parameter details, see Sarama Go Practice Tutorial.Note:
For disaster recovery drills, it is recommended to use Confluent Go client version 2.10.0 or later.
Python Client
The Python kafka-python client performs DNS lookups and retries at the second level automatically, without requiring additional configuration.
フィードバック