tencent cloud

Data Transfer Service

Use Instructions

Unduh
Mode fokus
Ukuran font
Terakhir diperbarui: 2026-11-02 17:51:37
Diterjemahkan oleh AI
Category
Description
Sync Object
1. Only basic tables, views, stored procedures, and functions are supported for synchronization.
2. Associated data objects need to be synchronized at the same time. Otherwise, synchronization will fail.
Impact on the source database
1. During data synchronization, DTS uses the account that executes the synchronization task to write the `__tencentdb__` system database to the source database for recording metadata such as transaction marker IDs. Ensure that the source database has read and write permissions on `__tencentdb__`.
To ensure that subsequent data comparison issues can be located, the `__tencentdb__` database will not be deleted from the source database after the synchronization task is completed.
The `__tencentdb__` system database occupies minimal space, approximately 0.01% to 0.1% of the storage capacity of the source database. For example, for a 50 GB source database, the `__tencentdb__` system database occupies approximately 5 MB–50 MB. It employs a single-threaded, connection-waiting mechanism, thus having a negligible impact on source database performance and not preempting any resources.
2. By default, a lock-free synchronization method is adopted. During the full data export phase, no global lock (FTWRL) is applied to the source database; only table-level locks are applied to tables without primary keys.
Requirements for target-side Kafka
1. For the target Kafka version, only the Pro Edition and Standard Edition available on Tencent Cloud are supported, and the version must be 1.1.1 or later. The physical machine Standard Edition, which has been discontinued, is not supported as a target.
2. You need to modify the message retention period and maximum message size in the target Kafka. It is recommended to set the message retention period to 3 days. Data that exceeds the retention period will be cleared, so consume the data promptly within the configured period. The maximum message size refers to the memory size of a single message that Kafka can receive. Set it to a value greater than the maximum size of a single row in the source database table to ensure that all source database data can be delivered to Kafka properly.
3. It is recommended that the target Topic be empty. During the synchronization task, do not write data to the selected Topic on the target side. Otherwise, message disorder and task errors may occur.
Description of data synchronization
1. Only the InnoDB, MyISAM, and TokuDB database engines are supported for synchronization. If tables using other database engines exist, they are skipped by default and not synchronized.
2. A single task can be associated with multiple Topics, but the same Topic cannot be used by multiple sync tasks at the same time. Otherwise, data disorder may occur, and the consumed data may come from multiple different tasks. This may also lead to abnormal data processing and task errors in scenarios such as task retries.
3. During the full synchronization phase, a checkpoint message is inserted into the target Kafka for every 100,000 synchronized records to mark the current data synchronization position.
4. A single message synchronized by DTS to the target Kafka has a performance limit. It is recommended that a single row of data in the source database not exceed 8 MB. Otherwise, an error may be reported during synchronization.
5. If you determine during synchronization that a table will be renamed, the partitioning rule will match the topic and partition based on the new table name when data is delivered to the target kafka.
6. During incremental synchronization, if binlog statements in STATEMENT format are generated in the source database, data synchronization will fail.
7. ALTER VIEW statements are not supported. If such a statement is encountered, the task skips it and does not synchronize it.
8. Synchronizing the entire source instance is not supported. If a database creation operation (CREATE DATABASE) is performed at the source during the subsequent synchronization task, it is recommended that you manually create the corresponding database object at the target before starting the synchronization task, or modify synchronization configurations to add the new database object after the synchronization task is started.
9. If the CREATE TABLE or RENAME TABLE operation is performed at the source during a synchronization task, you should select the entire database where the table resides when selecting synchronization objects during the configuration phase. Otherwise, the newly created or renamed tables will not be synchronized to the target.
10. If the sync task has already started and you modify the number of partitions on the target Ckafka, you need to restart the task by pausing and then resuming it. Only after the restart can data be delivered to the newly added partitions.
Data consumption description
Before a sync task starts, the happenAt field in the consumer Demo displays as 1970-01-01 08:00:00 for existing full data in the source database because no accurate data write time is available. This time field can be ignored. When incremental data is consumed, the time field displays correctly.
Impact of restart
During a sync task, if a task restart occurs (such as modifying synchronization objects, manually retrying after a task error, changing specifications, pausing and resuming a task, or isolating and then de-isolating a task), it may cause data synchronized to the target Kafka to be duplicated.
DTS synchronizes data by the smallest data unit. In the full stage, a block of data in a single table object is the smallest data unit, while in the incremental stage, each marked checkpoint is a data unit. If a data unit has just been synchronized when a restart occurs, no data duplication will be caused. If a data unit is still being synchronized when a restart occurs, the data unit will need to be synchronized again after the restart, which will cause data duplication.
If you are concerned about duplicate data, configure deduplication logic when consuming data.
Operation Restrictions
Do not perform the following operations during synchronization. Otherwise, they will cause the synchronization task to fail.
1. During the full export phase, do not perform DDL operations that change database or table structures in the source database.
2. Do not modify or delete user information (including usernames, passwords, and permissions) and port numbers in the source database and on the target side.
3. Do not perform binlog cleanup operations on the source database.
Data types
1. During the full data synchronization phase, DTS exports data from the source database and imports it to the target Kafka using the utf8mb4 character set to avoid garbled characters caused by unrecognized character sets.
2. Geometry-related data types are not supported. Such data will cause task errors.
Transaction
A single transaction cannot contain both DML and DDL statements at the same time. Such transactions will cause task errors.
HA Switches
If the source database is a non-GTID database, DTS does not support an HA switch on the source. If a switch is performed on the source, the DTS incremental synchronization may be interrupted.
Specify the start time
1. The incremental synchronization task starts synchronizing data from the set time point. The time point can be set within the range from the previous 14 days to the current time.
2. If full and incremental migrations are performed as separate tasks, ensure that no DDL operations (such as CREATE TABLE, ALTER TABLE, DROP TABLE, RENAME TABLE, CREATE INDEX, and DROP INDEX) occur between time point 1, when the full migration completes, and the set time point 2 for incremental migration. Otherwise, the task will report an error.
3. From the set time point 1 to the start point 2 of the incremental task (the task is considered started only after the task step changes from searching for the specified checkpoint to synchronizing increments). The must-knows for operations on the source database are as follows.
It is recommended that you do not perform a primary-secondary switch, add shards, or recreate secondary servers on the source database; otherwise, it may affect the ability of DTS to obtain the GTID checkpoint of the source database.
You can perform DDL operations that are not related to the synchronization objects on the source. However, ensure that the previous DDL operation has been synchronized to the target before you execute a new one. Executing DDL operations too frequently may cause task errors.
You cannot perform DDL operations related to the synchronization objects (such as CREATE TABLE, ALTER TABLE, DROP TABLE, RENAME TABLE, CREATE INDEX, and DROP INDEX) on the source; otherwise, the synchronization task will report an error.
4. Point-in-time synchronization determines the GTID based on the timestamp specified in the Binlog context (SET TIMESTAMP=XXXX). To ensure data synchronization accuracy, it is recommended that users not modify this context.
5. Ensure that the time zone set in the database is consistent with the current console time zone (that is, the browser time zone), or convert the time to the database time zone. Otherwise, the synchronization results for the specified checkpoint may differ from expectations.
6. If the set time point falls within the duration of an XA transaction, that XA transaction will not be synchronized. In this case, it is recommended to set the time point earlier (before the XA START time).

Bantuan dan Dukungan

Apakah halaman ini membantu?

masukan