tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
TencentDB for Redis
    Documentation
    Download PDF

    Migration with DTS

    Last updated: 2023-05-23 10:36:24
    Download PDF

      Basic Information

      Tencent Cloud Data Transmission Service (DTS) is a data transmission service that integrates such features as data migration, sync, and subscription. It helps you migrate your databases to the cloud without interrupting your business and build a high-availability database disaster recovery architecture through real-time sync channels. Its data subscription feature meets your requirements for commercial data mining and async business decoupling.
      DTS for Redis currently supports the data migration feature for you to migrate data to TencentDB in a non-stop manner at a time. In addition, in its full + incremental data migration mode, historical data in the source database written before migration and incremental data written during migration can be migrated together.

      Overview

      The source and target databases supported for data migration through DTS are shown in the table below.
      Source
      Target
      Note
      Self-built Redis database in IDC and CVM
      TencentDB for Redis and TencentDB for KeeWiDB
      Migration to TencentDB for KeeWiDB
      TencentDB for Redis
      Self-built Redis database in IDC and CVM
      Migration from TencentDB for Redis to self-built Redis (migration off the cloud and multi-cloud sync)
      Third-party Redis database
      TencentDB for Redis and TencentDB for KeeWiDB
      Migration to TencentDB for KeeWiDB
      TencentDB for Redis
      TencentDB for Redis and TencentDB for KeeWiDB
      Migration between TencentDB for Redis instances include the following scenarios:
      Migration between TencentDB for Redis instances in different regions
      Migration between TencentDB for Redis instances in the same region
      Migration between TencentDB for Redis instances on different versions
      Migration across Tencent Cloud accounts
      Migration from Tencent Cloud standard architecture to the cluster architecture, or vice versa. Cluster architecture data is stored in a distributed manner, and there are command compatibility issues with the standard architecture. Before migration, see Check on Migration from Standard Architecture to Cluster Architecture.
      Database of other types
      TencentDB for Redis
      Migration from Codis to TencentDB for Redis
      Migration from Tencent istore to TencentDB for Redis

      Version and Architecture

      The supported source databases includes Redis 2.8, 3.0, 3.2, 4.0, 5.0, and 6.2. We recommend that you migrate from an earlier version to a later one to avoid incompatibility.
      Supported cluster architectures include single-node, Redis cluster, Codis, and Twemproxy.
      The source instance supports the SYNC or PSYNC command.

      Restrictions

      The number of databases in the source database must be less than or equal to that in the target database.
      The memory space of the target database must be greater than or equal to 1.5 times the space occupied by the data to be migrated in the source database.
      The target database must support the overwrite mode. If this mode is not enabled, the target database must be empty.
      Only instances in Running status are allowed to be migrated. Instances with uninitialized passwords or other ongoing tasks cannot be migrated.
      The source database must be a secondary node; otherwise, a warning will be reported for the verification item. The alarm can be canceled based on the business conditions of the primary node.
      Note
      DTS can’t migrate data in a loop where data is continuously written, resulting in a write storm.
      DTS doesn’t support resuming syncing from where it is paused as there is no binlog file. The incremental data is synced in the connected memory. Once disconnected, the incremental data will be lost from the memory. Therefore, the task can’t be paused once it is initiated.

      Migration Preparations

      Check whether the status of the source and target database instances is normal.
      Check whether the access types of the source network environment are interconnected.
      Public Network: The source database can be accessed through a public IP.
      Self-Build on CVM: The source database is deployed in a CVM instance.
      Direct Connect: The source database can be interconnected with VPCs through Direct Connect.
      VPN Access: The source database can be interconnected with VPCs through VPN Connections.
      CCN: The source database can be interconnected with VPCs through CCN.
      Check the memory space: The memory space of the target database must be greater than or equal to 1.5 times the space occupied by the data to be migrated in the source database.
      Check the number of databases: The number of databases in the source database must be less than or equal to that in the target database.
      The target database must support the overwrite mode, and you need to enable it when configuring a migration task. If this mode is not enabled, the target database must be empty.
      Note
      DTS will perform verification before starting the migration task. For troubleshooting methods, see Check Item Overview.
      Check whether there are big keys in the source database.
      During the migration, big keys may cause the buffer (client-output-buffer-limit) to overflow, leading to a migration failure.
      For TencentDB databases, you can use DBbrain’s performance optimization feature to quickly analyze big keys. For more information, see Memory Analysis.
      For non-TencentDB databases, use RDBTools to analyze big keys in Redis.
      Evaluate large keys for splitting or cleaning. If you need to retain them, set the source buffer size (client-output-buffer-limit) to infinite.
      config set client-output-buffer-limit 'slave 0 0 0'
      Check the limit on the number of TCP connections in the source Linux kernel
      If the number of concurrent business requests is high, check the limit on the number of connections in the Linux kernel before the migration. If this value is exceeded, the Linux server will actively disconnect from DTS.
      echo "net.ipv4.tcp_max_syn_backlog=4096" >> /etc/sysctl.conf
      echo "net.core.somaxconn=4096" >> /etc/sysctl.conf
      echo "net.ipv4.tcp_abort_on_overflow=0" /etc/sysctl.conf
      sysctl -p
      Check the access permission of the source RDB file directory
      Before the migration, check and make sure that the directory where RDB files are stored in the source database is readable; otherwise, the migration will fail.
      If the RDB file directory is not readable, run the following command in the source database to set "diskless replication". Then, RDB files will be directly sent to DTS for storage, with no need to be stored in the source database first and then sent.
      config set repl-diskless-sync yes
      Check the command compatibility issues when perform migration from standard architecture to the cluster architecture. For directions, see Check on Migration from Standard Architecture to Cluster Architecture for static and dynamic evaluations.
      

      Migration Directions

      Step 1. Create a migration task

      1. Log in to the DTS console, select Data Migration, and click Create Migration Task.
      2. On the DTS page, create a migration task based on the description of the configuration items in the table below, and click Buy Now.
      Configuration Items
      Note
      Service Type
      Select Data Migration.
      Creation Mode
      Select a task creation mode. The following two modes are supported.
      Create task: Create a new task.
      Create similar task: Quickly create a similar task based on an existing migration task.
      Task ID
      When Create Mode selects Create similar task, this parameter needs to be configured.
      In the drop-down list, select a similar migration task ID.
      After configuring this parameter, the configuration items below will be updated based on the configuration in the selected task ID.
      Billing Mode
      Only pay-as-you-go is supported.
      Source Instance Type
      Select the source database type, which cannot be changed after purchase. Here, select Redis.
      Source Instance Region
      Select the source database region. If the source database is a self-built one, select a region nearest to it.
      Target Instance Type
      Select the target database type, which cannot be changed after purchase. Here, select Redis.
      Target Instance Region
      Select the target database region.
      Specification
      Currently, only Xlarge is supported.
      Tag
      Set tag key and value for a migration task, which facilitates efficient task management through tags.
      Task Name
      Name After Creation: After creating the task, you can change its name.
      Name Now: Set a name for the migration task in the input box below.
      Any special symbols other than hyphens are not supported.
      Lowercase letters, uppercase letters, and digits are supported.
      The length limit is as prompted in the console.
      It is recommended to set a name that is easy to identify.
      Terms of Service
      Be sure to check I have read and agree to.
      Quantity
      The number of created tasks, which ranges from 1 to 10.
      Link Configuration Fees
      Free of charge currently.
      3. Return to the data migration task page. In the migration task list, you can view the task status and progress, and wait for the task to be created.

      Step 2. Configure the migration task

      1. In the data migration list, find the created migration task.
      2. In its Operation column, click Configure.
      3. On the Set source and target databases tab of the Modify Migration Task configuration wizard, configure the source database network access type, host address, port, and target database instance information based on the table below. Then, test the connectivity and click Save.
      
      Configuration Category
      Configuration Items
      Note
      Task Configuration
      Task Name
      If the task name has been set when creating a new migration task, this field defaults to the created name.
      If no task name is configured when creating a new task, you need to set an identifiable task name in the input box.
      Any special symbols other than hyphens are not supported.
      Lowercase letters, uppercase letters, and digits are supported.
      The length limit is as prompted in the console.
      Running Mode
      Select Immediate execution or Scheduled execution.
      Immediate execution: Run immediately after the migration task is configured.
      Scheduled execution: You can set a specific execution time, and the task will start running at the set execution time. This method is recommended.
      After the migration task is configured, you can modify the scheduled execution time before the execution time.
      After the scheduled execution time is set for the migration task, but the task needs to be run immediately before the scheduled time, you can click Immediate start in the Operation column of the migration task list.
      Source Database Settings
      Source Database Type
      The source database type selected during purchase, which cannot be changed.
      Region
      The region selected during purchase, which cannot be changed.
      Access Type
      Select the type of network connected to the source database. For a third-party cloud database, you can select Public Network generally or select VPN Access, Direct Connect, or CCN based on your actual network conditions.
      Public Network: The source database can be accessed through a public IP.
      Self-Build on CVM: The source database is deployed in a CVM instance.
      Direct Connect: The source database can be interconnected with VPCs through Direct Connect.
      VPN Access: The source database can be interconnected with VPCs through VPN Connections.
      Database: The source database is a TencentDB database.
      CCN: The source database can be interconnected with VPCs through CCN.
      Node Type
      Select the cluster deployment type for the source database, which can be Single-Node Migration or Cluster Migration.
      Single-Node Migration: Refers to the cluster architecture with one master and multiple replicas.
      Cluster Migration: Refers to the cluster architecture comprised of several shards and multiple replica nodes on each shard.
      Host Address
      This parameter is displayed when Single-Node Migration is selected for Node Type. You need to enter the IP address of the host replica node of the source database in the input box.
      Port
      This parameter is displayed whenSingle-Node Migration is selected for Node Type. You need to enter the port of the source database in the input box.
      Password
      This parameter is displayed when Single-Node Migration is selected for Node Type. You need to enter the access password of the source database in the input box.
      Node Info
      This parameter is displayed when Cluster Migration is selected for Node Type.
      Enter all the shard node addresses and passwords of the source database cluster ("IP:port:password" or "IP:port"), and separate multiple nodes by line breaker.
      It is recommended to migrate data from the replica node (secondary node) of the source database to avoid affecting the business access of the source database.
      Target Database Settings
      Target Database Type
      The target database type selected at the time of purchase, which cannot be changed and is fixed as Redis.
      Region
      The target database region selected during purchase, which cannot be changed.
      Access Type
      Select the network type for the target database access.
      Node Type
      Select the deployment type for target database cluster.
      Single-Node Migration: Refers to the cluster architecture with one master and multiple replicas.
      Redis Cluster Migration: Refers to the native Redis cluster architecture, which does not support proxy architecture.
      Proxy Cluster Migration: Refers to the cluster architecture with proxy, such as Codis and Twemproxy.
      Host Address
      This parameter is displayed when Single-Node Migration is selected for Node Type. You need to enter the IP address of the host node of the target database in the input box.
      Port
      This parameter is displayed when Single-Node Migration is selected for Node Type. You need to enter the port of the target database in the input box.
      Password
      This parameter is displayed when Single-Node Migration is selected for Node Type. You need to enter the access password of the target database in the input box.
      Node Info
      This parameter is displayed when Redis Cluster Migration or Proxy Cluster Migration is selected for Node Type.
      Enter all the shard node addresses and passwords of the target database cluster ("IP:port:password" or "IP:port"), and separate multiple nodes by line breaker.
      4. On the Set migration options and select migration objects tab, confirm that the Migration Type is Full + incremental migration and the Migration Object is Entire instance. Then, you can check whether Enable Overwrite Mode, click Save.
      Note
      Once the replace mode is enabled, the data will be updated to the target database in a replacing manner without clearing the database, and the data is written from the beginning of the file.
      5. On the Verify task tab, task verification will be performed automatically. The verification items include network, instance access password, instance capacity, and read-only target instance.
      If the verification result is Failed, it indicates that a check item failed and the task is blocked. You need to fix the problem and run the verification task again.
      If the verification result is Alarm, it indicates that a check item doesn't completely meet the requirements, and the task can be continued, but the business will be affected. Before continuing the task, you need to assess whether to ignore the alarm or fix the problem as prompted.
      6. After all verification tasks are in Passed status, click Start Task or Start Later.
      Start Task: Immediately start the migration task, even if a scheduled execution time is set, the data migration will be started immediately without waiting.
      Start Later: If you have set the scheduled execution time when configuring the task, you can select Start Later, and the task will start at the execution time.
      7. Return to the data migration task list, and the task will be in “Preparing” status. After running for 1-2 minutes, the data migration task will be started.

      Step 3. View and manage data migration tasks

      In the migration task list, you can view the progress of the migration task. When the status of the migration task changes to Task successful, verify the data on the target database. If the verification is correct, determine the cutover time, and route the business system to the target database
      If the keys of the source and target databases are the same, click Complete in the Operation column to stop the data migration task.
      Manage migration tasks: In the Operation column of the task list, you can perform operations such as retrying, terminating, and pausing the task. For directions, see [Task Management](https://www.tencentcloud.com/document/ product/571/42637!cf9a9bfbaa744e05fc9c8d74b2dec935).

      Event Alarming and Metric Monitoring

      DTS can automatically report event alarms triggered upon migration interruption to keep you informed of any exceptions. For directions, see Configuring Alarm Policy for Data Migration.
      DTS allows you to view the monitoring data of various metrics during migration to understand the performance metrics of the system. For more information, see Viewing Monitoring Metric.

      FAQs

      For common error messages and handling methods during DTS migration, see Common Error Messages.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy