Feature Overview

COS Migration is an all-in-one tool that integrates COS data migration feature. Users can quickly migrate data from various sources to COS through simple configurations and steps. It has the following features:

• Diverse data sources:
  • Local data: migrate locally stored data to COS.
  • Other cloud storage services: Currently, it supports migration from AWS S3, Alibaba Cloud OSS, and Qiniu to COS. More services will be supported in the future.
  • URL list: download and migrate data from specified URLs to COS.
  • Bucket replication: data can be replicated among COS buckets. Cross-account and cross-region replication is supported.
• Checkpoint restart: restarting uploads from checkpoints is supported. For large files, if the upload exits halfway or service failure occurs, you can run the tool again to restart the upload.
• Multipart upload: an object can be uploaded to COS by parts.
• Parallel upload: multiple objects can be uploaded at the same time.
• Migration verification: migrated objects can be verified.

Note：

• COS Migration only supports UTF-8 encoding.
• If you use this tool to upload a file whose name already exists, the older file will be overwritten. File name duplication cannot be checked.

Operating Environments

System environment

Windows, Linux, and macOS.

Software requirements

• JDK 1.8 X64 or above. For more information on JDK installation and configuration, see Java.
• IFUNC needs to be supported on Linux and the binutils version should be later than 2.20.

Directions

1. Get the tool

Download COS Migration here.

2. Decompress the package

Windows

Decompress the package and save it to a directory, for example:

C:\Users\Administrator\Downloads\cos_migrate

Linux

Decompress the package and save it to a directory, for example:

unzip cos_migrate_tool_v5-master.zip && cd cos_migrate_tool_v5-master

Migration tool structure

The structure of the properly decompressed COS Migration tool is as follows:

COS_Migrate_tool
|——conf  #Directory of the configuration file
|   |——config.ini  #Migrate the configuration file
|——db    #Store the record of successful migrations
|——dep   #JAR package complied by the main logic of the program
|——log   #Log generated during tool execution
|——opbin #Script for compiling
|——src   #Source code of the tool
|——tmp   #Temporary file storage directory
|——pom.xml #Project configuration file
|——README  #Readme document
|——start_migrate.sh  #Migration startup script for Linux
|——start_migrate.bat #Migration startup script for Windows

Note：

• The db directory mainly records the IDs of files successfully migrated by the tool. Each migration job will first compare the records in db directory. If the ID of the current file has already been recorded, the current file will be skipped, otherwise it will be migrated.
• The log directory keeps all the logs generated during tool migration. If an error occurs during migration, first check the error.log under this directory.

3. Modify the config.ini file

Before running the migration startup script, modify the config.ini file (path: ./conf/config.ini) first. This file contains the following parts:

3.1 Configure the migration type

type indicates the migration type, which is filled in by users based on their migration needs. For example, to migrate local data to COS, users need to configure type=migrateLocal for [migrateType].

[migrateType]
type=migrateLocal

Currently, the following migration types are supported:

3.2 Configure the migration job

You can configure a migration job based on your actual needs, including information configuration for the destination COS and job-related configurations.

# The common configuration section of the migration tool includes account information to be migrated to the destination COS.
[common]
secretId=COS_SECRETID
secretKey=COS_SECRETKEY
bucketName=examplebucket-1250000000
region=ap-guangzhou
storageClass=Standard
cosPath=/
https=off
tmpFolder=./tmp
smallFileThreshold=5242880
smallFileExecutorNum=64
bigFileExecutorNum=8
entireFileMd5Attached=on
daemonMode=off
daemonModeInterVal=60
executeTimeWindow=00:00,24:00
encryptionType=sse-cos

3.3 Configure the data source

Configure each section according to the migration type described in [migrateType]. For example, if the configuration item of [migrateType] is type=migrateLocal, users only need to configure the [migrateLocal] section.

3.3.1 Configure a local data source migrateLocal

If you migrate from a local system to COS, configure this section. The specific configuration items and descriptions are as follows:

# Configuration section for migration from a local system to COS
[migrateLocal]
localPath=E:\\code\\java\\workspace\\cos_migrate_tool\\test_data
excludes=
ignoreModifiedTimeLessThanSeconds=

3.3.2 Configure an Alibaba Cloud OSS data source migrateAli

If you migrate from Alibaba Cloud OSS to COS, configure this section. The specific configuration items and descriptions are as follows:

# Configuration section for migration from Alibaba Cloud OSS to COS
[migrateAli]
bucket=bucket-aliyun
accessKeyId=yourAccessKeyId
accessKeySecret=yourAccessKeySecret
endPoint= oss-cn-hangzhou.aliyuncs.com
prefix=
proxyHost=
proxyPort=

3.3.3 Configure an AWS data source migrateAws

If you migrate from AWS to COS, configure this section. The specific configuration items and descriptions are as follows:

# Configuration section for migration from AWS to COS
[migrateAws]
bucket=bucket-aws
accessKeyId=AccessKeyId
accessKeySecret=SecretAccessKey
endPoint=s3.us-east-1.amazonaws.com
prefix=
proxyHost=
proxyPort=

3.3.4 Configure a Qiniu data source migrateQiniu

If you migrate from Qiniu to COS, configure this section. The specific configuration items and descriptions are as follows:

# Configuration section for migration from Qiniu to COS
[migrateQiniu]
bucket=bucket-qiniu
accessKeyId=AccessKey
accessKeySecret=SecretKey
endPoint=www.bkt.clouddn.com
prefix=
proxyHost=
proxyPort=

3.3.5 Configure a URL list data source migrateUrl

If you migrate from a specified URL list to COS, configure this section. The specific configuration items and descriptions are as follows:

# Configuration section for migration from a URL list to COS
[migrateUrl]
urllistPath=D:\\folder\\urllist.txt

3.3.6 Configure bucket replication migrateBucketCopy

If you migrate from one COS bucket to another bucket, configure this section. The specific configuration items and descriptions are as follows:

Note：

The account that initiates the migration needs to have permissions to read the source bucket and write to the destination bucket.

# Configuration section for migration from source bucket to destination bucket
[migrateBucketCopy]
srcRegion=ap-shanghai
srcBucketName=examplebucket-1250000000
srcSecretId=COS_SECRETID
srcSecretKey=COS_SECRETKEY
srcCosPath=/

3.3.7 Configure UpYun data source migrateUpyun
When you migrate from a local system to COS, configure this section. The specific configuration items and descriptions are as follows:

[migrateUpyun]
# Migrate from UpYun
bucket=xxx
# ID of the UpYun operator
accessKeyId=xxx
# Password of the UpYun operator     
accessKeySecret=xxx       
prefix=

# UpYun sdk restriction, this proxy will be configured as a global proxy
proxyHost=
proxyPort=

4. Run the migration tool

Windows

Double-click start_migrate.bat to run the tool

Linux

1. Read into configurations from the config.ini file. The run command is:

   sh start_migrate.sh
   

2. Read into configurations from command lines for some parameters. The run command is:

   sh start_migrate.sh -Dcommon.cosPath=/savepoint0403_10/
   

Note：

• The tool supports reading configuration items in two ways: command line or configuration file.
• The command line takes priority over the configuration file, i.e., for the same configuration item, parameters in command lines take priority.
• Reading configuration items from command lines allows users to run different migration jobs at the same time, provided that key configuration items (such as bucket name, COS path, source path to be migrated, etc.) in the two jobs are not exactly the same. Concurrent migration can be achieved because different migration jobs are written into different db directories. Please refer to db information in tool structure above.
• Configuration items are in the format of -D{sectionName}.{sectionKey}={sectionValue}. sectionName is the section name of the configuration file. sectionKey is the name of the configuration item in the section. sectionValue is the value of the configuration item in the section. COS path to which data is migrated to should be in the format of -Dcommon.cosPath=/bbb/ddd.

Migration mechanism and process

Migration mechanism

COS migration tool has a status. Successful migrations will be recorded in the format of KV in leveldb file under db directory. Before each migration, check whether the path to which data is migrated to has been recorded in db directory. If yes and its attribute is the same as that in db, the migration will be skipped. Otherwise, the migration will be executed. The attribute varies by migration type. For local migration, mtime determines whether to migrate. For migration from other cloud storage services and bucket replication, etag and length of the source file determine whether to migrate. Thus, we search for records of successful migrations in db rather than in COS. If a file is deleted or modified via COSCMD or the console rather than the migration tool, the migration tool cannot detect this change and the file will not be re-migrated.

Migration process

1. The configuration file is read, the corresponding configuration section is read according to the migration type, and parameters are checked.
2. The IDs of the files to be migrated are scanned and compared in db directory according to the specified migration type to determine whether upload is allowed.
3. The execution results are printed out during migration, where inprogress indicates migration is in progress, skip indicates skipped, fail indicates failed, ok indicates succeeded, condition_not_match indicates file fails to meet migration conditions (such as lastmodified and excludes) and is skipped. Details about the failure can be viewed in error log. The execution process is shown below:
   
4. Statistics are printed out after the migration is completed, which include the total number of migrated, failed, and skipped files as well as the amount of time consumed. For failures, check the error log, or rerun the migration job as the migration tool will skip successfully migrated files and retry migrating failed ones. The execution result of a migration job is shown below:
   

FAQs

If an exception such as migration failure or execution error occurs when you use COS Migration tool, see COS Migration Tool for troubleshooting.