tencent cloud

Feedback

Data Subscription (for Kafka) Overview

Last updated: 2021-12-27 11:53:42

    Feature Overview

    Data subscription refers to the process where DTS gets the data change information of a key business in the database, converts it into message objects, and pushes them to Kafka for the downstream businesses to subscribe to, get, and consume. DTS allows you to directly consume data through a Kafka client, so you can build data sync features between TencentDB databases and heterogeneous systems, such as cache update, real-time ETL (data warehousing technology) sync, and async business decoupling.

    How to Implement

    The following takes MySQL as an example to describe how data subscription pulls the incremental binlog from the source database in real time, parses the incremental data into Kafka messages, and then stores them on the Kafka server. You can consume the data through a Kafka client. As an open-source messaging middleware, Kafka supports multi-channel data consumption and SDKs for multiple programming languages to reduce your use costs.

    Typical Use Cases

    Data archiving

    By using the data subscription feature of DTS, you can push the updated incremental data in TencentDB to an archive database or data warehouse as a stream in real time.

    Restrictions

    • Currently, the subscribed message content is retained for 1 day by default. The data will be cleared after it has expired. Therefore, it is recommended to consume the data promptly.
    • The region where the data is consumed should be the same as that of the subscribed database.
    • Data subscription to MySQL, MariaDB, and TDSQL for MySQL does not support geometry data types.

    Supported Subscription Types

    DTS allows you to subscribe to databases and tables. Specifically, the following three subscription types are supported:

    • Data update: subscription to DML operations.
    • Structure update: subscription to DDL operations.
    • Full database: subscription to the DML and DDL operations of all tables.

    Supported Advanced Features

    Feature Description Documentation
    SDKs for various programming languages DTS uses the Kafka protocol and supports Kafka client SDKs for multiple programming languages. -
    Metric monitoring and default alarm policy
  • Data subscription metrics can be monitored.
  • Default configuration is supported for data subscription event monitoring to automatically notify you of exceptional events.
  • Supported Events and Metrics
    Multi-Channel data consumption DTS allows creating multiple data channels for a single database, which can be consumed concurrently through a consumer group. -
    Partitioned consumption DTS supports partitioned storage of data in a single topic for concurrent consumption of data in multiple partitions, improving the consumption efficiency. -
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support