Data Subscription

Last updated: 2020-03-04 17:57:55

PDF

Data Transfer Service (DTS) provides a binlog-based incremental data subscription feature that allows subscription of incremental update data from CDB with sePCl simple steps:

  1. In DTS console Purchase and create a subscription to channel for TencentDB instance.

  2. Use DTS data subscription SDK to connect to this subscription channel to subscribe to and consume incremental data.

Subscription principle

Data subscription is analyzed by simulating the corresponding binlog content from the database to the master database. The architecture is shown below. By parsing binlog, and analyzing Database table, which is configured by subscription channel, there is almost no impact on the master database.

  • The content of subscribed messages is currently retained for the last 1 day by default.
  • Subscribe on a case-by-case basis, and the subsequent addition of Database table will also appear in the original subscription channel. There is no need to add a new configuration to the original subscription channel.
  • Subscribe to the library as a unit, and the subsequent new tables under the library will also appear in the original subscription channel. There is no need to add a new configuration operation to the original subscription channel. Under this scheme, if you need to add a new library, you need to add a new configuration operation.
  • The data subscription service currently supports cloud databases MySQL 5.6,5.7.
  • Data subscription does not support views, triggers and Foreign key.
  • The relevant binlog_row_image parameters need to be adjusted for the initial configuration of data subscription, and the data subscription will automatically kill the old session to make the parameters take effect immediately according to the conditions met.
  • Data subscriptions currently support character sets including latin1, utf8, and utf8mb4.

Directions

Step 1: create a data subscription to channel

  1. Login DTS console Select "data subscription" page in the left navigation, and click "Create data subscription" to start subscribing to channel configuration.

  2. On the pop-up page, select the region where the source TencentDB instance resides, and click Buy now.

  3. After Enable succeeds, return to the data subscription page and click "initialize configuration" to initialize the configuration for the purchased data subscription channel.

  4. On the configuration page, select the source TencentDB instance, and click next.

  5. Select your desired synchronization type and database table.

    • The granularity of subscription objects for DTS data subscription includes two levels: database and table. That is, a user may choose to subscribe to certain databases or tables.
    • DTS divides data subscription into two types: data update and structure update.
      When you select only data update and subscription objects, you can only subscribe to three kinds of data changes: insert, delete and update.
      If you want to subscribe to the structure update (DDL), you need to select the structure change in the synchronization type. Once the subscription structure is updated, DTS will pull out all the structural changes of the entire TencentDB instance, and users need to use SDK to filter the data they need.
      Select full instance, that is, data update + structure update.
  6. After determining the subscription object and type, click [Launch] and Launch can subscribe to channel.

Step 2: modify the consumption point in time

DTS supports modifying consumption point in time at any time in the process of consumption. Once the consumer point in time is modified, the incremental data pulled by the downstream SDK starts with the modified consumer point in time.

DTS only supports modifying the consumption time points in the console. You cannot specify consumption time point in the SDK.

The modification steps are as follows:

  1. Stop SDK consumption process.
    In Data subscription For the list, click "View subscription configuration" in the subscription ID or operation column to enter the subscription configuration page. If "Consumer Source (IP)" is none, the process has stopped.

  2. Change Consumption Time Point

    In Data subscription List, move the mouse over the subscription consumer point in time, there will be a "modify consumption point in time" icon, click the icon to enter the modification page.

    The new consumption time point must fall within the data range of the subscription channel.

  3. Restart SDK consumption process.
    You can restart the local SDK consumption process after the consumption time point is modified.

    At this point, the SDK will start subscribing to incremental data from the new consumption time point.

Step 3: modify the subscription object

When using DTS, you can dynamically add/reduce subscription objects during consumption.

  • When you save the modification after a subscription object is added, the subscription channel will pull incremental data of the added subscription object, from the current time.
  • When you save the modification after a subscription object is deleted, the SDK will no longer subscribe to data of this object.

The modification steps are as follows:

  1. In Data subscription List, select "more" > "modify subscription object" in the operation column to go to the subscription configuration page.
  2. On the subscription configuration page, modify the subscription object, and click Save.

Step 4: consume data using SDK

You have activated the CVM service.

Subscription case

Pull the changes in the corresponding subscription table to the local file
Pull the change of the corresponding subscription table to Redis
Pull the change of the corresponding subscription table to Kafka