When you use TDMQ for CKafka (CKafka), you should pay attention to key metrics, including bandwidth, storage, and the number of partitions. In general scenarios, these metrics affect the load capacity of the cluster.
Due to differences in business scenarios, during the actual usage and operation of a cluster, the load of the cluster may be affected by various factors, such as message size, whether messages are compressed, whether transactional messages are used, and the message sending-receiving ratio. Therefore, it is not comprehensive enough to simply use the bandwidth and storage ratio of the cluster as the sole judgment metrics for cluster scale-out.
To better ensure the stable operation of business and rationally plan and manage cluster capacity, CKafka has added the cluster load metric in Advanced Monitoring. You can combine metrics such as bandwidth utilization and cluster load to gain a more comprehensive understanding of the load status of the current cluster, and use this as the reference information for whether to scale out the CKafka cluster.
Scenarios
CKafka Pro Edition
Capacity Planning Scheme
Viewing Metrics
Reference Policies
Instance production bandwidth percentage and instance consumption bandwidth percentage
It is recommended that bandwidth utilization should not exceed 70%.
Suggestions on cluster load metrics
Single-availability zone (AZ) deployment. When a cluster is deployed in a single AZ, it is recommended to keep the maximum cluster load value at around 70%.
Multi-AZ deployment. When a cluster is deployed across multiple AZs, a certain level of redundancy needs to be considered so that if an unexpected exception occurs in one AZ, the remaining AZs can handle the business load normally. For example:
Two-AZ deployment: When a single AZ is unavailable, the cluster has half of its nodes remaining. Considering the 70% utilization, it is recommended to keep the normal load of the cluster below 35%.
Three-AZ deployment: When a single AZ is unavailable, the cluster has 2/3 of its nodes remaining. Considering the 70% utilization, it is recommended to keep the normal load of the cluster below 47%.
For business stability, it is recommended to reserve about 30% of the business traffic as a buffer when purchasing or scaling out a cluster.
Scale-Out Scenario Example 1
Bandwidth utilization is below 70%, cluster load is above 70%, and the deployment mode is single-AZ deployment.
Cluster Status
The cluster bandwidth specification is 3000 MB.
Bandwidth utilization is 50% and cluster load is 90%.
Scale-Out Target
Reduce the cluster load rate from 90% to 60%.
Scale-Out Scheme
Scale-out factor: Reduce cluster load from 90% to 60%. The formula for calculating the scale-out factor is: 90%/60% = 1.5.
Reserved buffer: 30%.
The formula for calculating the specification after scale-out is: 3000 MB x Scale-out factor x Reserved buffer = 3000 MB x 1.5 x (1 + 30%) = 5850 MB, which can be rounded up to 6000 MB to scale out the specification.
Scale-Out Scenario Example 2
Bandwidth utilization is above 70%, cluster load is below 70%, and the deployment mode is single-AZ deployment.
Cluster Status
The cluster bandwidth specification is 3000 MB.
Bandwidth utilization is 90%, and cluster load is 50%.
Scale-Out Target
Reduce bandwidth utilization from 90% to 60%.
Scale-Out Scheme
Reduce bandwidth utilization from 90% to 60%. The formula for calculating the scale-out factor is: 90%/60% = 1.5.
Reserved buffer: 30%.
The formula for calculating the specification after scale-out is: 3000 MB x Scale-out factor x Reserved buffer = 3000 MB x 1.5 x (1 + 30%) = 5850 MB, which can be rounded up to 6000 MB to scale out the specification.