Tencent Cloud

Recent Pages

Migrating Local Data to COS

Last updated: 2020-01-06 14:15:27

Practice Scenario

If you have a local IDC, COS offers the following migration methods to help you quickly migrate massive amounts of local data to COS.

Migration Method	Description
COS Migration(online migration)	COS Migration is an all-in-one tool integrating COS data migration features. You can migrate data to COS quickly after completing simple configurations.
Cloud Data Migration (CDM) (offline migration)	CDM uses a dedicated offline migration device provided by Tencent Cloud to help you migrate your local data to the cloud, solving the problems that arise from online migration from a local IDC to cloud, such as long time, high costs, and low security.

You can determine how to migrate data based on the data volume, IDC egress bandwidth, IDC idle server resources, acceptable completion time, and other factors. The estimated time needed for online migration is shown in the figure below. As can be seen, if data migration needs to take more than 10 days or the amount of data to be migrated exceeds 50 TB, you are recommended to use CDM for offline migration; otherwise, online migration is a better choice.

Data migration may be affected if there is a high number of small files below 1 MB or the performance of disk IO is insufficient.

Migration Practice

COS Migration

The steps are as follows:

Install the Java environment.
Install the COS Migration tools.
Modify the configuration file.
Launch the tool.

For more information, please see COS Migration Tool.

Tips

Below describes how to configure COS Migration to maximize the migration speed:

Adjust the threshold and concurrence of large/small files based on your network environment to implement optimal migration where large files are split into multiple parts and small files are transferred concurrently. Adjust the tool execution time and set a bandwidth limit to ensure that bandwidth usage by data migration will not affect your business operations. Such adjustments can be made by modifying the following parameters in the [common] section of the config.ini configuration file:

Parameter Name	Parameter Description
smallFileThreshold	Threshold for small files. If the size of a file is higher than or equal to this threshold, multipart upload will be used; otherwise, simple upload will be used. The default value is 5 MB.
bigFileExecutorNum	Concurrence of large files, which is 8 by default. If COS is connected to over the public network with low bandwidth, reduce this value.
smallFileExecutorNum	Concurrence of small files, which is 64 by default. If COS is connected to over the public network with low bandwidth, reduce this value.
executeTimeWindow	This parameter defines the time range when the migration tool will execute a task every day, which will enter sleep mode at other times. In sleep mode, the migration will be paused and the migration progress will be retained until the next time window when the migration will be resumed automatically.

Distributed parallel transfer can further speed up the migration. You can install COS Migration on multiple servers and perform migration tasks separately for different data sources.

Cloud Data Migration (CDM)

The steps are as follows:

Go to the CDM Console to submit an application.
After the application is approved, you can wait to receive a device.
After receiving the device, copy your data to it as instructed in the migration device user manual.
Then, submit a return application in the console and wait for Tencent Cloud to migrate your data to COS.

For more information, please see Cloud Data Migration (CDM).

Tips

Below describes how to migrate data to COS offline efficiently and securely.

Configure a 10 Gbps network connection for your IDC. To remove potential constraints of the local data environment on data transfer, you can use a high-performance server as a mount point to maximize the replication speed.
Parallel transfer is the fastest way to transfer data though CDM. You can monitor the CPU and memory utilization of the device and if the current migration speed is lower than expected, you can speed the migration up in the following ways:
- Connect multiple devices to the same CDM device via different network interfaces.
- Connect multiple devices to several CDM devices via different network interfaces.