Data Subscription

Last updated: 2020-05-06 17:54:20

    DTS provides a binlog-based incremental data subscription feature that allows you to subscribe to incrementally updated data in TencentDB in just a few steps:

    1. Purchase and create a subscription channel for your TencentDB instance in the DTS Console.
    2. Use the DTS data subscription SDK to connect to this subscription channel to subscribe to and consume incremental data.

    How Subscription Works

    Data subscription is implemented by simulating a slave database to get the corresponding binlog content from the master database for analysis in the following architecture. While parsing the binlogs and analyzing data according to the database/table configured in the subscription channel, it has almost no impact on the master database.

    • Currently, the subscribed message content in the past day is retained by default.
    • If subscription is made at the instance level, subsequently added databases/tables will also appear in the subscription channel, so there is no need to additionally configure the channel.
    • If subscription is made at the database level, subsequently added tables will also appear in the subscription channel, so there is no need to additionally configure the channel. In this scheme, if a new database is added, you need to configure the channel.
    • The data subscription service currently supports TencentDB for MySQL 5.6 and 5.7.
    • Data subscription does not support views, triggers, and foreign keys.
    • The initial configuration of data subscription requires adjustment of the relevant binlog_row_image parameter. Data subscription will automatically kill old sessions according to the conditions met to make the parameters take effect immediately.

    Directions

    Step 1. Create a data subscription channel

    1. Log in to the DTS Console, select Data Subscription on the left sidebar, and click Create Subscription to start configuring the subscription channel.

    2. On the pop-up page, select the region where the source TencentDB instance resides and click Buy Now.

    3. After the service is activated, return to the data subscription page, click Initialize configurations, and complete initial configuration for the purchased data subscription channel.

    4. On the configuration page, select the source TencentDB instance and click Next.

    5. Select your desired data sync type and database/table to be synced.

      • The granularity for subscribed objects divides into database level and table level, i.e., you can choose to subscribe to certain databases or tables.

      • The types of subscribed data include data update, structure update, and all.
        If only data update and subscribed object are selected, you can only subscribe to three types of data changes: insert, delete, and update.
        To subscribe to structure update (DDL), you need to select structure change as the sync type. DTS will pull all structure changes in the entire TencentDB instance, and you need to filter out the desired data by using the SDK.
        If all is selected, both data update and structure update will be subscribed to.

    6. After confirming the subscribed object and type, click Start to enable the subscription channel.

    Step 2. Modify the consumption starting time point

    DTS allows you to modify the consumption starting time point at any time during the consumption process. Once the modification is completed, the downstream SDK will begin pulling incremental data starting from the new starting time point.

    Currently, DTS only supports modifying the consumption starting time point in the console. You cannot specify it in the SDK.

    The modification steps are as follows:

    1. Stop the SDK consumption process.
      In the Data Subscription list, click a subscription ID or View subscription settings in the "Operation" column to enter the subscription configuration page. If Consumer Source (IP) is none, the process has stopped.

    2. Modify the consumption starting time point.
      In the Data Subscription list, move the cursor to the consumption starting time point of the subscription and the "Modify consumption start time" icon will appear. Click the icon to enter the modification page.

      The new consumption starting time point must fall within the data range of the subscription channel.

    3. Restart the SDK consumption process.
      After the consumption starting time point is modified, you can restart the local SDK consumption process. Then, the SDK will begin subscribing to incremental data starting from the new time point.

    Step 3. Modify the subscribed object

    DTS allows you to dynamically add/delete subscribed objects during the consumption process.

    • If a subscribed object is added, the subscription channel will pull its incremental data starting from the current time.
    • If a subscribed object is deleted, the SDK will no longer subscribe to its data.

    The modification steps are as follows:

    1. In the Data Subscription list, select More > Modify subscribed object in the "Operation" column to enter the subscription configuration page.

    2. On the subscription configuration page, modify the subscribed object and click Save.

    Step 4. Use the SDK to consume data

    For more information, please see SDK User Guide.

    Sample Subscriptions

    Pulling changes in a subscribed table to a local file
    Pulling changes in a subscribed table to Redis
    Pulling changes in a subscribed table to Kafka

    Was this page helpful?

    Was this page helpful?

    • Not at all
    • Not very helpful
    • Somewhat helpful
    • Very helpful
    • Extremely helpful
    Send Feedback
    Help