The following part describes how to configure a fully-managed public network migration task to achieve data migration when the data source is a storage system that supports the S3 protocol (such as Ceph, MinIO, and GCP).
Preparations
Origin Server
Prepare the required AccessKey and SecretKey for S3 protocol interface authentication as per the origin server instructions.
Tencent Cloud COS
1. Create a destination bucket to store the migrated data. For more details, see Creating Bucket.
2. Create a sub-user for migration and grant the required permissions:
2.2 Choose Users > User List in the left sidebar to enter the user list page.
2.3 Create a sub-user, and check the Programming access and Tencent Cloud console access options.
2.4 Search for and check the QcloudMSPFullAccess and QcloudCOSAccessForMSPRole policies.
2.5 Complete the sub-user creation and save the username, login password, SecretId, and SecretKey.
Note:
You can also log in to MSP with your root account. However, for security reasons, we recommend that you create a sub-account, use the sub-account's API key for migration, and delete the sub-account after migration.
2. Click Object Storage Migration in the left sidebar to enter the object storage migration page.
Creating Migration Task
1. On the object storage migration page, click Create task to enter the migration task configuration page and set the migration parameters.
2. Specify the migration task name.
Task name: 1 to 60 characters ([a-z], [A-Z], [0-9] and [_]). You can use this name to view the migration status and progress in the task list.
3. Task size estimation. Please accurately fill in the task size so that Tencent Cloud can better prepare resources. This field is optional.
4. Set the file source for migration.
Service Provider: Select Origin servers supporting S3 protocol for this field.
AccessKey, SecretKey: Enter the AccessKey and SecretKey of the account used for migration in the text boxes.
Bucket name: Enter the name of the bucket to store the migrated data.
Endpoint: Enter the complete S3 protocol endpoint of the source bucket. (HTTP or HTTPS can be included in the domain name. If not included, HTTPS is used by default.) For S3 systems requiring region verification, enter the source region. Otherwise, leave the field blank.
Header migration mode: If the file in the source bucket has headers and tags that need to be retained after migration, please select the retention option or set the replacement rule.
File name filtering rules: Choose to migrate all the files of the specified bucket or migrate only files with the specified prefix.
Time range: Specify the time range. Migrate files that are added or modified within this time period.
Execution Speed, Speed Limit Mode: All public cloud object storage services have speed limits. To ensure business stability, please confirm with the service provider and set the maximum available migration bandwidth in Mbps before migrating.
5. Select the destination for migration.
Service Provider: The default value is Tencent Cloud COS.
SecretId, SecretKey: Enter the SecretId and SecretKey of the sub-user for migration.
Bucket name: After you enter the Key, click Refresh on the right side of the "Bucket Name" drop-down box to view the list of destination COS buckets.
Storage Path: Specify the directory in the destination bucket for migration.
Save to the root directory: Store the files in the source bucket to the root directory of the destination bucket according to the original relative paths.
Save to the specified directory: Store the files in the source bucket to the specified directory according to the original relative paths.
For example:
For files /a.txt and /dir/b.txt in the source bucket, if "dest" is filled in this field, the paths of the two files in the destination bucket will be /dest/a.txt and /dest/dir/b.txt after migration.
If dest/20180901 is filled in this field, the paths of the two files in the destination bucket will be /dest/20180901/a.txt and /dest/20180901/dir/b.txt after migration.
File with the same name: Specify the handling method for files with the same name.
Notes:
If the method is Overwrite, files in the destination bucket will be overwritten directly during migration.
If the method is Skip, judgment will be performed based on the last modification time (LastModified). Specifically:
If the last modification time of the file in the source bucket is later than or the same as that of the file in the destination bucket, then overwrite the file in the destination bucket.
If the last modification time of the file in the source bucket is earlier than that of the file in the destination bucket, skip the file in the source bucket.
Secondary migration is required if the object (file) content is changed during migration.
Storage type: Set the storage class for migrated files according to the migration requirements. Available options: Use standard storage for all, Use Standard Infrequent Access Storage for all, Keep the original storage properties, and Save all as archive storage.
6. Select migration mode.
Mode Selection:
Create a migration task and start fully-managed migration immediately: When fully-managed migration is used, MSP will access the source bucket via the public network for migration after you click "Create and Start".
Create a migration task and download the Agent manually to start migration: When semi-managed migration is used, only task creation is completed after you click "Create and Start". You need to manually download the Agent and deploy it on the origin server before you can start the migration. For Agent deployment, see Using Semi-Managed Migration Agent.
Scheduled Task: Scheduled tasks can be executed repeatedly to synchronize incremental files in the source bucket. Such tasks are executed immediately for the first time and can be triggered later either after a specified interval or according to the scheduling rules set in Custom UNIX cron format. When a scheduled task is executed again, files with the same name as those in the destination bucket will be skipped. In other words, existing files with the same name in the destination bucket are retained, which means that only incremental files are synchronized.
7. Click Create and Start to start the migration task.
Viewing Migration Status and Progress
You can view the status and progress of all file migration tasks on the Object Storage Migration page:
For completed tasks, the message in green indicates the task is completed with all files successfully migrated, while the message in yellow indicates the migration task is completed but some files failed to be migrated.
You can click Retry failed task to retry the migration of failed files involved in the task. Files that have been successfully migrated will not be retried.
You can click Export list of failed files to export the list of files that failed to be migrated.
Estimating File Migration Time
The migration speed is subject to the lowest speed at each stage of the migration process and is affected by the network transmission speed and maximum concurrency level. For more details, see Estimating File Migration Time.
Yes
No
Was this page helpful?