Tencent Cloud Kafka (CKafka) is a distributed, high-throughput and highly scalable messaging system that is fully compatible with open-source Apache Kafka API v0.9 and v0.10. Based on the publish/subscribe model, CKafka enables async interaction between the message producer and consumer by decoupling the messages and thereby eliminating wait time. CKafka supports data compression and offline and real-time data processing, making it ideal for collecting compressed logs and aggregating monitoring data.
Overview
Benefits
Open-source Component Compatibility
CKafka is fully compatible with Apache Kafka v0.9 and v0.10, completely eliminating the costs associated with cloudification. In addition, it works well with open-source upstream and downstream components and seamlessly supports Kafka Streams, Kafka Connect and KSQL.
Upstream and Downstream Ecosystems
CKafka can interconnect with more than 13 Tencent Cloud products such as EMR, COS, CIS, SCS, SCF and CLS, enabling fast one-click deployment.
High Reliability
CKafka boasts high clustering performance and outperforms open-source solutions in terms of productivity. Furthermore, its distributed deployment solution guarantees cluster stability.
High Scalability
CKafka clusters are horizontally scalable and instances can be seamlessly upgraded. The underlying system automatically scales elastically according to the business scope without affecting the user experience.
Business Security
Tenants are isolated at the network level so that the network access of instances is naturally isolated among different accounts. CAM authentication for management streams and SASL permission control for data streams are supported for strict access control.
Unified OPS Monitoring
CKafka provides a complete set of OPS services empowered by the Tencent Cloud platform, including multi-dimensional monitoring and alarm services such as tenant isolation, access control, message retention query and consumer details.
Features
Message Decoupling
Peak Shifting
Horizontal Scaling
One-time Production for Repeated Consumption
Message Decoupling
CKafka effectively decouples the relationship between the message producer and consumer, allowing you to independently scale or modify the production-consumption processing procedure as long as they follow the same interface constraints.
As a result, CKafka can reliably replace traditional messaging middleware. Aside from its ability to decouple producer-consumer relationships and cache unprocessed messages, CKafka features higher throughput, a stronger partition replication mechanism and better fault tolerance.
Peak Shifting
The ability of the system to respond to access surges is critical. However, if traffic surges are uncommon, resources invested based on peak traffic will be wasted.
CKafka ensures that critical system components are able to cope with sudden access surges, eliminating the risk of complete system crash due to request overload.
Horizontal Scaling
As the message processing procedure is decoupled, the efficiency of message queuing and processing can be effectively improved simply by horizontally scaling the processing procedure, enabling a fully flexible solution.
A for implementation, one CKafka topic can be divided into multiple partitions and distributed to one or more brokers.
A consumer can subscribe to one or more partitions, while the producer is responsible for evenly distributing messages to the corresponding partition. Therefore, adding more brokers can scale the cluster horizontally. Generally, the more brokers, the higher the cluster throughput.
One-time Production for Repeated Consumption
CKafka supports multiple modes such as queue and publish/subscribe. CKafka topic supports partitioning. Different partitions can reside in different brokers to effectively increase the throughput. In addition, CKafka features a multi-queue mode in which it adopts the strategy of the consumer group, i.e., one topic stores only one copy of data at a node, and different consumer groups maintain their own consumption records. This is ideal for scenarios where the messages are produced once and consumed by multiple consumer groups.
Features
Message Decoupling
CKafka effectively decouples the relationship between the message producer and consumer, allowing you to independently scale or modify the production-consumption processing procedure as long as they follow the same interface constraints.
As a result, CKafka can reliably replace traditional messaging middleware. Aside from its ability to decouple producer-consumer relationships and cache unprocessed messages, CKafka features higher throughput, a stronger partition replication mechanism and better fault tolerance.
Peak Shifting
The ability of the system to respond to access surges is critical. However, if traffic surges are uncommon, resources invested based on peak traffic will be wasted.
CKafka ensures that critical system components are able to cope with sudden access surges, eliminating the risk of complete system crash due to request overload.
Horizontal Scaling
As the message processing procedure is decoupled, the efficiency of message queuing and processing can be effectively improved simply by horizontally scaling the processing procedure, enabling a fully flexible solution.
A for implementation, one CKafka topic can be divided into multiple partitions and distributed to one or more brokers.
A consumer can subscribe to one or more partitions, while the producer is responsible for evenly distributing messages to the corresponding partition. Therefore, adding more brokers can scale the cluster horizontally. Generally, the more brokers, the higher the cluster throughput.
One-time Production for Repeated Consumption
CKafka supports multiple modes such as queue and publish/subscribe. CKafka topic supports partitioning. Different partitions can reside in different brokers to effectively increase the throughput. In addition, CKafka features a multi-queue mode in which it adopts the strategy of the consumer group, i.e., one topic stores only one copy of data at a node, and different consumer groups maintain their own consumption records. This is ideal for scenarios where the messages are produced once and consumed by multiple consumer groups.
Scenarios
Log Analysis System
Streaming Data Processing Platform
CKafka can work seamlessly with EMR to build a complete log analysis system. Logs are first collected by the agent deployed on the client, and the data is aggregated to CKafka. The data is then computed and consumed multiple times by the backend big data suite such as Spark, and the original logs are cleaned, stored or graphically displayed.
CKafka can be used together with Stream Compute Service (SCS) for real-time/offline data processing and exception detection to meet the needs of different scenarios:
Real-time data is analyzed and displayed and exceptions are detected to quickly locate system issues.
Historical consumption data is stored and analyzed offline for secondary data processing and trend report generation.
Pricing
CKafka supports two billing modes: pay-as-you-go and monthly subscription. For more information, see Billing Overview.