An Apache Kafka-compatible distributed, high-throughput, highly scalable messaging service
Tencent Cloud Kafka (CKafka) is a distributed, high-throughput, highly scalable messaging system that is fully compatible with open-source Apache Kafka API v0.9 and v0.10. Based on the publish/subscribe model, CKafka enables async interaction between the message producer and consumer by decoupling the messages and thereby eliminating wait time. CKafka supports data compression and offline and real-time data processing, making it ideal for collection of compressed logs and aggregation of monitoring data.
CKafka is fully compatible with Apache Kafka v0.9 and v0.10, completely eliminating the costs associated with cloudification. In addition, it works well with open-source upstream and downstream components and seamlessly supports Kafka Streams, Kafka Connect and KSQL.
CKafka can interconnect with more than 13 Tencent Cloud products such as EMR, COS, CIS, SCS, SCF and CLS, enabling fast one-click deployment.
CKafka boasts high clustering performance and outperforms open-source solutions in terms of productivity; furthermore, its distributed deployment scheme guarantees cluster stability.
CKafka clusters are horizontally scalable, and instances can be seamlessly upgraded. The underlying system automatically scales elastically according to the business scope, and user experience will not be affected.
Tenants are isolated at the network level, so that the network access of instances is naturally isolated among different accounts; CAM authentication of management streams and SASL permission control of data streams are supported for strict access control.
CKafka provides a complete set of OPS services empowered by the Tencent Cloud platform, including multi-dimensional monitoring and alarming services such as tenant isolation, access control, message retention query and consumer details.
CKafka effectively decouples the relationship between message producer and consumer, allowing you to independently scale or modify the production-consumption processing procedure as long as they follow the same interface constraints.
As a result, CKafka can reliably replace traditional messaging middleware. Aside from the abilities to decouple producer-consumer relationships and cache unprocessed messages, CKafka features a higher throughput, stronger partition replication mechanism and better fault tolerance.
The ability of the system to respond to access surges is critical. However, if this kind of traffic surges are uncommon, resources invested based on peak traffic will be wasted.
CKafka ensures that critical system components are able to cope with sudden access surges, eliminating the risk of complete system crash due to request overload.
As the message processing procedure is decoupled, the efficiency of message queuing and processing can be effectively improved simply by horizontally scaling the processing procedure, enabling a fully flexible solution.
As for implementation, one CKafka topic can be divided into multiple partitions and distributed to one or more brokers.
A consumer can subscribe to one or more partitions, while the producer is responsible for evenly distributing messages to the corresponding partition. Therefore, adding more brokers can scale the cluster horizontally. Generally, the more brokers, the higher the cluster throughput.
CKafka supports multiple modes such as queue and publish/subscribe. CKafka topic supports partitioning. Different partitions can reside in different brokers to effectively increase the throughput. In addition, CKafka features a multi-queue mode, where it adopts the strategy of consumer group, i.e., one topic stores only one copy of data at a node, and different consumer groups maintain their own consumption records. This is ideal for scenarios where the messages are produced once and consumed by multiple consumer groups.
CKafka can work seamlessly with EMR to build a complete log analysis system. Logs are first collected by the agent deployed on the client, and the data is aggregated to CKafka. The data is then computed and consumed multiple times by the backend big data suite such as Spark, and the original logs are cleaned, stored or graphically displayed.
CKafka can be used together with Stream Compute Service (SCS) for real-time/offline data processing and exception detection to meet the needs of different scenarios:
Real-time data is analyzed and displayed and exceptions are detected for quick locating of system issues.
Historical consumption data is stored and analyzed offline for secondary processing of data and generation of trend reports.
CKafka supports Kafka Connect, KSQL and native Kafka components for real-time data processing and conversion. It can interconnect with various upstream and downstream components through Kafka Connect for tasks such as database synchronization and inter-cluster hot-cold data synchronization. Moreover, it can perform simple analysis and conversion of real-time data based on KSQL, eliminating the need for complicated coding and effectively reducing system complexity.
CKafka is pay-as-you-go, you only pay for what you use. For more details, please see the pricing doc.