TDMQ for CKafka (CKafka) is a high-throughput scalable distributed messaging system that is fully compatible with Apache Kafka API 0.9、2.8、3.2.Based on the publish/subscribe pattern, CKafka allows producers and consumers to interact asynchronously without waiting for each other through message decoupling. It has many strengths, such as high availability, data compression, and offline/real-time data processing, making it suitable for scenarios such as compressed log collection, monitoring data aggregation, and stream data integration.
TDMQ
TDMQ for CKafka
A high-performance and highly reliable messaging system compatible with Apache Kafka.
Overview
Benefits
Open-Source Component Compatibility
Feature 100% compatibility with Apache Kafka v0.9–3.2, work well with upstream and downstream open-source components such as Kafka Streams and Kafka Connect, completely eliminating the costs associated with cloudification.
Upstream and Downstream Ecosystems
Supports data synchronization with on-premises self-built and cross-cloud Kafka clusters, integrates with cloud products like ES, and enables low-cost one-click deployment of data flow pipelines.
High Reliability
Surpass the productivity of open-source solutions and provide a distributed deployment to ensure cluster stability.
High Scalability
Support automatic horizontal scaling of clusters and seamless upgrade of instances without affecting the user experience.
Business Security
Isolate tenants at the network level among different accounts, and support CAM for management streams and SASL for data streams to enhance security.
Unified OPS Monitoring
Provide a comprehensive set of Ops services, including multidimensional monitoring and alarm services such as tenant isolation, access control, message retention query, and consumer details query.
Features
Message Decoupling
CKafka effectively decouples the relationship between message producer and consumer, allowing you to independently scale or modify the production/consumption processing procedure as long as they follow the same API constraints.
Peak Shifting
CKafka can withstand access traffic surges instead of completely crashing due to sudden overwhelming requests, which effectively boosts system robustness.
Sequential Read/Write
CKafka can guarantee the order of messages in a partition. Just like most message queue services, it can also ensure that data is processed in order, greatly improving disk efficiency.
Async Communication
In the scenario where the business does not need to process messages immediately, CKafka provides an async message processing mechanism, that is, when the traffic is high, messages will be put into the queue only and processed after the traffic drops, which significantly relieves the system pressure.
Scenarios
Log Analysis System
Streaming Data Processing Platform
Message storage
Data ETL and dumping
CKafka can work with EMR to create a complete log analysis system. The client-side agent collects logs and aggregates them to CKafka, where the data is computed and consumed repeatedly by the backend big data suite, such as Spark. The original logs are then cleaned, stored, or graphically displayed.
CKafka can work with Stream Compute Service to process data in a real-time or offline manner and detect exceptions in various scenarios:
- Analyze real-time data for exception detection to troubleshoot system issues.
- Store historical consumption data and analyze it offline for secondary processing and trend report generation.
With SCF integrated, CKafka supports customized data processing to meet your message dumping requirements in different scenarios such as log consumption, microservices, and big data analysis.
In some use cases, data from a cache layer component such as Kafka needs to be stored in a downstream system such as CKafka, ES, or COS after ETL. The common practice is to process the data with Logstash, Flink, or custom code and monitor those components to ensure their stable operation. However, in order to operate and maintain the components, it requires learning their syntax, specifications, and technical principles. This incurs significant costs which are unnecessary if all you need is simple data processing.
CKafka Connector comes with lightweight, UI-based, data ETL and dumping capabilities that are simple to configure, making it easier for you to process and dump data to downstream storage systems.
Pricing
TDMQ for CKafka supports two billing modes: pay-as-you-go and monthly subscription. For more information, see Billing Overview.