Message Queue CKafka

An Apache Kafka-compatible distributed, high-throughput, highly scalable messaging service

Overview

Tencent Cloud Kafka (CKafka) is a distributed, high-throughput, highly scalable messaging system that is fully compatible with open-source Apache Kafka API v0.9 and v0.10. Based on the publish/subscribe model, CKafka enables async interaction between the message producer and consumer by decoupling the messages and thereby eliminating wait time. CKafka supports data compression and offline and real-time data processing, making it ideal for collection of compressed logs and aggregation of monitoring data.

Benefits

Open-source Component Compatibility

CKafka is fully compatible with Apache Kafka v0.9 and v0.10, completely eliminating the costs associated with cloudification. In addition, it works well with open-source upstream and downstream components and seamlessly supports Kafka Streams, Kafka Connect and KSQL.

Upstream and Downstream Ecosystems

CKafka can interconnect with more than 13 Tencent Cloud products such as EMR, COS, CIS, SCS, SCF and CLS, enabling fast one-click deployment.

High Reliability

CKafka boasts high clustering performance and outperforms open-source solutions in terms of productivity; furthermore, its distributed deployment scheme guarantees cluster stability.

High Scalability

CKafka clusters are horizontally scalable, and instances can be seamlessly upgraded. The underlying system automatically scales elastically according to the business scope, and user experience will not be affected.

Business Security

Tenants are isolated at the network level, so that the network access of instances is naturally isolated among different accounts; CAM authentication of management streams and SASL permission control of data streams are supported for strict access control.

Unified OPS Monitoring

CKafka provides a complete set of OPS services empowered by the Tencent Cloud platform, including multi-dimensional monitoring and alarming services such as tenant isolation, access control, message retention query and consumer details.

Features

CKafka decouples message production and consumption, providing scalable, low-latency, high-performance messaging capabilities.
Message Decoupling

CKafka effectively decouples the relationship between message producer and consumer, allowing you to independently scale or modify the production-consumption processing procedure as long as they follow the same interface constraints.

As a result, CKafka can reliably replace traditional messaging middleware. Aside from the abilities to decouple producer-consumer relationships and cache unprocessed messages, CKafka features a higher throughput, stronger partition replication mechanism and better fault tolerance.

Scenarios

Log Analysis System

CKafka can work seamlessly with EMR to build a complete log analysis system. Logs are first collected by the agent deployed on the client, and the data is aggregated to CKafka. The data is then computed and consumed multiple times by the backend big data suite such as Spark, and the original logs are cleaned, stored or graphically displayed.

Streaming Data Processing Platform

CKafka can be used together with Stream Compute Service (SCS) for real-time/offline data processing and exception detection to meet the needs of different scenarios:

Real-time data is analyzed and displayed and exceptions are detected for quick locating of system issues.

Historical consumption data is stored and analyzed offline for secondary processing of data and generation of trend reports.

Data Synchronization and Conversion

CKafka supports Kafka Connect, KSQL and native Kafka components for real-time data processing and conversion. It can interconnect with various upstream and downstream components through Kafka Connect for tasks such as database synchronization and inter-cluster hot-cold data synchronization. Moreover, it can perform simple analysis and conversion of real-time data based on KSQL, eliminating the need for complicated coding and effectively reducing system complexity.

Pricing

CKafka is pay-as-you-go, you only pay for what you use. For more details, please see the pricing doc.